but the important thing is to know your audience, even if that means you want to create one.>>
I think that is important. Some audiences will allow any unexpectedness as long as they get this one thing (whatever it is) fulfilled-- usually the basic genre expectation (that the murdered will be found out, that the couple will find love, etc). Other audiences are happy when even that expectation is violated, as long as structurally this is whatever they like to read.
I've often thought that genre readers-- because they read more and KNOW the genre expectations-- are going to be less likely to forgive than, say, bestseller-only readers who come into their reading without much experience and expectation. So my sister who reads a lot of mysteries was very unhappy with a big huge bestseller mystery/thriller where the culprit was revealed to be someone who was not a character in the story before that. (I mean, the murderer was a new character, introduced only when he was revealed as the murderer.) Now I suspect that the author wanted to show how random murder can be, how the most careful policing can't account for the crazy murderer, how sometimes things don't make sense. So that would be a valid reason for deciding to violate the norms, right? But presumably genre readers LIKE the genre norms, and maybe need some tradeoff here-- something better about this book that makes up for this, or the result of the violation is so good, it's worth it.
I think that that story actually violated the central norm of mysteries, that murder will out, but also that thought and investigation, not accident, can restore justice. That's a world-view, actually, a belief in the primacy of reason (at least in fiction) that would lead away from an ending that happened more or less by accident. That, of course, makes it all the more provocative when you violate that and say, "Sometimes reason doesn't find the truth!"
That is, I think that we need to respect that genre expectations are usually based in serious thought/feeling approaches, actual principles, not just stodginess. If I read romance because I want to have my belief in love reaffirmed, then I'm probably less likely to be open to a book that SEEMS like a romance but is actually cynical about love. (However, a story that is cynical about love and THEN ends with love being affirmed-- now that will get me every time. :) Also notice that most genre readers might have no problem with a mainstream book that violates genre expectations.
So how do you get around that pre-existing worldview? A couple thoughts:
1) Aim for a genre audience but be judicious about what you're changing. I remember a great workshop by Jennifer Greene and Emily Richards, two romance writers who have always pushed the envelope, where they suggested identifying the major norms of your genre, and realize that if you change too many all at once, you'll probably alienate a lot of readers. For example, if you're going to set this in a dystopia and have the plot concerned with some arcane alien politics, you might decide that it would be too much to make the protagonist unsympathetic besides. If you want to violate the majority of the genre norms, think about whether you really want to write in that genre. No one is making you, after all. :)
2) Aim for a non-genre audience. Genre storytelling structures (the mystery plot, etc.) are remarkably durable-- they are a very good way to tell many stories that are NOT within genre expectations. Think of The Curious Incident of the Dog at Night, and The Name of the Rose. Both of these are self-consciously modeled on the genre stories about Sherlock Holmes, but they are both aimed at a different audience than a genre one. Both were mainstream successes.
3) Take the genre storytelling structure into another genre. There are a couple genre structures that are really portable-- mystery and romance. These show up all the time as plots in other genres. Mysteries, for example, are quite common as external plots in romance, and I keep trying to say that, say, Shards of Honor (nominally sf) is a quite traditional captive/captor romance, or that that first Amelia Peabody mystery was a great romance but only a mediocre mystery. (I am SO annoying!) But since the "destination genre" readers aren't likely to have the same expectations as the "originating genre" readers, you can probably violate expectations without antagonizing readers. For example, in a novel that is marketed as a science fiction novel where, say, the space ship's captain is murdered, an ambivalent ending to the mystery will probably be well accepted.
4) Write such a great story in such a great way that everyone will forgive anything. Just remember that central expectation and keep that in some way (like a mystery novel without a crime might not go over well). Laura Kinsale specializes romances with less sympathetic characters and emotionally problematic conflicts, but her characterization and prose makes up for the dislocating aspects. The Western genre has been revitalized by several authors who have generally kept the sweeping epic setting but have complicated the characters (occasionally even making the "hero" a woman).
Other thoughts? I think thinking of the =structures- rather than genres can really help. That is, take the mystery or thriller or horror structure into that big category called "General Fiction" or even "Literary Fiction," and see if that gives you freedom to innovate. :)