The gap between the truly great and the merely good is always widening. There has never been so much music, film, literature, TV, art, theatre and other culture to experience, but audiences must – and critics do – often wonder why so little of it is good enough to last.

It may even be the case that the work that falls between the cracks, because you’re too busy checking out what everyone else is checking out, is what, in time, turns out to be the real deal.

Sometimes the gap between good and great is brought home very vividly. I'm a regular at South by Southwest (SXSW), the music festival held every March in Austin, Texas. It's a binge-gigging opportunity like no other, a chance for new-music fans to catch hundreds of new bands from all over playing venues of every hue. Over the past decade I've probably seen more than 1,000 new acts at the festival, and raved about hundreds of them, but the number of acts that has made a lasting impression is much smaller.

I can remember a few clearly. There was the first sighting of The Black Keys, in 2003, with their prototype swamp rock, sleazy blues and killer riffs. The next year TV on the Radio turned rock on its head. From 2008, there was Justin Vernon preaching the spectral, ethereal Bon Iver gospel and the Fleet Foxes folk revival. Local Natives introduced their harmonies to the world beyond Los Angeles in 2009, and Warpaint popped up on the radar in 2010. Odd Future's visceral live show dominated the dispatches in 2011, and Alabama Shakes and Kendrick Lamar produced key performances last year.

But those memories are the exceptions. Many acts were good, and worthy of further investigation, but only a handful have come good on that potential. Even fewer of the acts that made a splash have yet to generate lasting work.

Tellingly, highlights of recent SXSW festivals have come from vintage acts, such as Bruce Springsteen last year and Prince this year. Standing in La Zona Rosa with a couple of hundred other people watching a turbocharged Prince and his 22-piece band stomp their way through a brassy, brash, bold set a few Saturdays ago will live long in the memory. It was a gig to relish, both for the context – superstars don't usually play in such small rooms – and the content. Prince is an artist with a perfectly pitched throughline, with decades of live shows behind him and well-turned out tunes to play. Unless he decided to throw a strop, it was always going to be a beautiful night.

Of course, it's not a level playing field. The new bands who are the bread and butter of an event such as SXSW can't compete when an act like Prince, Springsteen or Iggy Pop, whose performance many raved about this year, rolls into town trailing all kinds of expectations. The new acts may well go on to take their place in the canon, but they can't yet compete with seasoned masters.

Surely, though, the art of creating searing, exceptional, life-affirming art that will stand the test of time is not simply down to experience and longevity. While performances often benefit from the latter attributes, there’s also a lot to be said for the thrill of the new, of an artist producing work that is unique because no one else has thought of doing it that way. Yet that initial spark doesn’t often translate into anything else, leaving a sense of what could have been. Occasionally, that first colourful burst will be enough, but too often it’s a promise that is never fulfilled because the artist never quite hits those heights again.

Long-term mark
Music isn't the only area where the work that gets highlighted might not make a long-term mark. It remains to be seen, for instance, if Argo and Lincoln are really what people will remember about the past year in movies. Perhaps Benh Zeitlin's marvellous, ambitious, inspiring Beasts of the Southern Wild or Andrew Dominik's downbeat, intricate, detailed crime-noir, Killing Them Softly , will be viewed with more interest in the future.

These films might gain more weight and stature as people discover them in their own time and of their own accord. Many such films are lost in the frantic, frazzled week-of-release circus, in which all promotional activities are concentrated on getting the most exposure possible when the film opens. The trend is also visible when certain albums are released, and re-evaluated in a different light.

We've also seen it with TV. The Wire might be acclaimed as the best thing on the box since the test card, but the series certainly wasn't seen as such when it began. It took a few seasons for the show to develop a fervent audience that found David Simon's ode to Baltimore getting under their skin and who became evangelists for the drama.

Of course, there are times when the work doesn't deserve initial attention – few people will recommend season one of Love/Hate, for example – but the advent of the box set means many TV shows now benefit from a second look.

That predicting which art is going to stick is not as easy as it might first seem is something to consider when compiling end-of-year lists.

Perhaps a far more useful exercise might be to compile an end-of-year list for 10 years ago, to see what we remember. What is striking in the short term may not have quite the same appeal after a decade or more.