Sad Fiction

Why is the best fiction almost exclusively sad?

I read Kafka's The Metamorphosis (and Other Short Stories, but I haven't finished all the other short stories yet) a couple days ago and it got me thinking about how almost all great writing is marked by an affinity towards tragic ending.

The metamorphosis tells the story of a young man, Gregor Samsa, who has shouldered the burden of providing for his family (his parents in particular). The story begins with Samsa awakening from unusally deep sleep late for work. After glancing over at his clock, he begins to think evaluate his irresponsibility and consider his employer's possible consequences. At the same time, Samsa has awoken in a transformed physical state: he now appears to be a beetle. His skinny and weak legs twice frenetically, and his heavy and muscular midsection make it difficult for his legs to balance his body's overall weight.

Both his family and employer inquire initially as to the reason for his transgressions, instead of even considering the possibility of illness or incapacity. As the secondary characters become aware of Samsa's metamorphosis, the family dynamic is quickly rearranged. The father, mother, and sister all take on independent jobs to fill the financial void that Samsa's incapacity has left them.

Speaking from Samsa's perspective, Kafka details the constantly metamorphosizing relationship between Samsa and his family. While the family initially feels empathy and sorrow (and, admittedly, horror and repulsion) at the loss of their son, as time passes, that empathy turns to disgust and anger at the sad financial state they now face. While Kafka doesn't couch these sentiments in explicit rhetoric, it seems implicit in the way Samsa is spoken about.

Eventually, as it becomes increasingly clear that Samsa's transformation is permanent, and the presence of a large beetle comes at great social cost for his family, they eventually decide that he has become an unbearable burden on the family--Samsa develops a depression and dies.

--
I find the sadness in this story multidimensional. Superficially, Samsa's undeserved death clearly falls under this category. More abstractly, though, there are several very sad features of this story. First is the correlation (pre-plot) between the increasing naturalness of Samsa's role as primary breadwinner and the decreasing level of appreciation he gets from his family--they more or less take him for granted. Second is the inability of his family to tolerate his "ugly side": his addiction to work (perhaps why Kafka chose to transform him into a worker bug). When his ugly side comes with fringe benefits like financial security, his family can tolerate him; when it comes at their expense, a few months is all they can handle. And so on.

--
The point i'm trying to make is that this is considered one of the great literary works of all time. Along with many of the other greats, The Metamorphosis has a very sad ending. In theory, this makes sense. Sadness is generally associated with realism, or at least has a more powerful impact on the sentiments we associate with particular writing. When someone dies, the lesson or theme to a book becomes explicitly clear and is explicitly impacted: this flaw has this mortal consequence.

But, I think this in and of itself carries some serious implications about our society. Why is it that we find the painful so inspiring? Are we in society incapable of being inspired or moved by those who learn their lessons and improve before an otherwise tragic demise? Is this societal masochism? Okay, maybe that's a little extreme, but the point remains. What about sad endings holds such a dear place in our heart?

Sure, we always feel deeply for the dearly departed, but what are the consequences of naturalizing that condition as the most powerful version of storytelling?