I couldn't keep my sticky, six-year-old fingers off it. I think I loved Sgt Pepper's Lonely Hearts Club Band from the moment I became aware of it. The colourful diorama of famous figures on the front. The way it opened out to reveal – okay, let me see can I remember who is who this time – Ringo, John, Paul . . . and George?

And then, on the back, the printed lyrics that revealed a world of tangerine trees and marmalade skies, and a horse called Henry who danced the waltz, and – most fantastically of all – a traffic warden who was actually worthy of love.

But what I loved especially was the cardboard insert with the cutout moustache and military stripes that I was expressly forbidden from cutting out. This was not a toy, I was told. It was the soundtrack to the Sixties.

It was the most definitive artistic statement of a period that my mother and father referred to as “the good old days” and it was to be handled delicately and only under strict parental supervision.

Like most first listeners to the album, I can say I was exposed to the hype before I heard a note of the music. I understood from a very early age that it was the greatest record ever made. And, 40 years later, I’m still discovering new ways in which that statement is true.

One of the facts about the album that still blows my mind is how quickly The Beatles managed to pull it all together. There are no rules to say how long it should take to produce a classic album. Bob Dylan recorded Highway 61 Revisited in six days.

Bruce Springsteen agonised over Born to Run for 15 months, while sound engineers put silver foil on their dental fillings and bit down hard to try to stay awake during 36-hour shifts at the console. Van Morrison laid down the basic tracks for Astral Weeks in a single New York night.

Preternaturally productive

Sgt Pepper was recorded in a preternaturally productive period of 120 days. In January 1967, the Beatles had completed only one track that would appear on the album – Paul McCartney's whimsical music hall ballad When I'm Sixty-Four.

By the end of May, the record was in the shops, in time to become the musical score to the Summer of Love, that all-to-brief season of optimism when young people like my parents really believed they could change the world with their colourful clothes and their happy thoughts.

It could be argued – and it is, of course, endlessly – that there are better Beatles albums. For this listener, Revolver represents a superior collection of songs. But Sgt Pepper was the album that reset the bar in respect of the potentiality of the recording studio and, for good or bad, changed the way people listened to popular music.

That it emerged from just 700 working hours suggests that the band were possessed by the unique energy of the times. Or rather McCartney was, since Sgt Pepper was his vision. And for most of the time they spent realising that vision, he seemed to be the only Beatle enjoying the process.

Later on, John Lennon said he grew to resent the album, seeing it as a Paul McCartney record more than a Beatles one. Ringo Starr idled away hours in the studio playing chess while George Harrison brooded at being frozen out of the creative process.

While Revolver was the sound of the Beatles all on the same page creatively, Sgt Pepper revealed a shift in the intra-band dynamics, in particular the tightening relationship between McCartney and producer George Martin.

In the autumn of 1966, the pair had worked together on the score for the movie The Family Way while the Beatles were on hiatus and considering their future together.

Pointless screaming

The boy band thing was a beaten docket. All the fun had run out of playing to crowds who drowned out their songs with their pointless screaming. And anyway, musically, since the release of 1965's Rubber Soul and 1966's Revolver, they weren't that kind of band any more.

Inspired by Dylan, they were becoming more sophisticated songwriters, experimenting with new themes, forms, sounds and instruments. Tomorrow Never Knows, Eleanor Rigby and Norwegian Wood weren't the kind of songs to send ballpark crowds into a state of pants-wetting hysteria.

They had changed as people, too. As Dylan observed when McCartney played him an early pressing of Sgt Pepper: "Oh, I get it – you don't want to be cute any more."

The penny had dropped much earlier with their fans. The shirts and ties and collarless suits were gone. Like teenagers, the Beatles were suddenly choosing their own clothes and showing signs of being not quite as acquiescent and eager to please as they were in their younger days.

In fact, by the strict behavioural standards that manager Brian Epstein set for them, they were being positively seditious, growing facial hair, calling out the prime minister and the leader of the opposition (by name!) in Taxman – and were those intakes of breath in Girl really the sound of someone toking on a joint?

When McCartney conceived the idea for Sgt Pepper in the autumn of 1966, the album had become the thing. Before the mid-1960s, LPs served no more ambitious purpose than to capture on vinyl the best live takes of a bunch of songs performed in a recording studio – usually singles that fans had already bought, grouted with inferior filler and B-sides.

But in the summer of 1966, in the space of six weeks, the release of three albums that still feature in the top 10 of most authoritative All Time Greatest polls (The Beatles' own Revolver, Dylan's Blonde on Blonde and The Beach Boys' Pet Sounds) redefined the album as a piece of art and the raison d'etre for most serious bands.

Musical alternates

Innovations in recording technology were changing the game. Pet Sounds was Brian Wilson's attempt to best Rubber Soul and to capture on vinyl the sounds he heard in his head when he was stoned.

And it was Pet Sounds that set the bar for McCartney when, on a flight home from a holiday in Kenya, he came up with the concept for an album produced by a band of musical alternates – he eventually decided on an Edwardian marching band – who, shorn of their Fab Four egos, were free to experiment creatively.

While the music would be progressive, McCartney – the most sentimental Beatle – wanted the theme of the album to be nostalgic, featuring songs inspired by their north of England childhoods.

The early sessions, in December 1966, yielded three: McCartney's When I'm Sixty-Four and Penny Lane and Lennon's phantasmagorical masterpiece, Strawberry Fields Forever.

But then EMI, possibly still in denial over the new direction the band were taking, demanded a single to keep the Beatlemaniacs happy. Martin offered them Penny Lane and Strawberry Fields, which were released as a double-A side in February 1967.

It was the first Beatles single in four years to fail to reach the number one spot in the UK – it was kept off the top by Engelbert Humperdinck's Release Me – and EMI began to have serious misgivings about the new psychedelic sound coming out Abbey Road Studio Two.

The upshot of this was that two of the band’s greatest-ever compositions, which fitted the theme perfectly, failed to go on the album, an almost criminal act of musical misappropriation that Martin regretted for the rest of his life.

These days, of course, we can all use drag and drop to create our own preferred versions of Sgt Pepper. My own re-mastering of the album puts Penny Lane and Strawberry Fields at the beginning of side two and consigns When I'm Sixty-Four and Harrison's incongruous, sitar-led sermon, Within You Without You, to my computer's trash bin.

It’s a matter of personal taste, of course. But it is extraordinary to think that, by the end of 1966, the Beatles still didn’t have the final artistic call over what singles they released and which songs went on their albums.

Shot his creative bolt

With Strawberry Fields, Lennon felt he had shot his creative bolt, which presented him with a problem when the band returned to the studio after Christmas. McCartney was flush with new ideas for the album. Getting Better, Fixing a Hole, She's Leaving Home – they poured out of him, which reflected the happier mental space he was inhabiting compared to his songwriting partner.

With his girlfriend away in the US, McCartney was leading a bachelor life in London, attending openings and happenings, swinging with the hippest of hip cats and generally embracing the avant-garde.

Lennon, by contrast, was mired in a life of domestic ennui in suburban Surrey, with a wife and child he would soon abandon. While he hated writing songs to order, he realised that, unless he got to work, he would have nothing on the album, suggesting that the real creative rivalry behind Sgt Pepper wasn't between McCartney and Wilson but between McCartney and Lennon.

It's interesting that all of the songs that Lennon produced in the weeks that followed were inspired by domestic things or items he happened to have to hand. Being for the Benefit of Mr Kite! was prompted by an old circus poster he bought in an antique shop; Good Morning Good Morning by a TV ad for Kellogg's Corn Flakes; Lucy in the Sky with Diamonds by a painting his son, Julian, did at school; and A Day in the Life by a tabloid newspaper report of a bitter child custody battle involving the widow and mother of Tara Browne, the Irish-born Guinness who had introduced McCartney to LSD.

All four Beatles had used acid by then. Lennon was a voracious consumer, although he abstained during the recording of the album.

And while Sgt Pepper would become the record that defined the psychedelic era, and the soundtrack to a summer in which young people turned on, tuned in and dropped out in their droves, it was created by musicians who were stoned rather than hallucinating.

‘Decisive moment’

The record's release, on May 26th, 1967, was greeted by the kind of hype that had never before attended the release of a pop record. To Kenneth Tynan, writing in the Times, it was "a decisive moment in the history of civilization", while Rolling Stone said that not since the Congress of Vienna in 1815 had the world felt so unified by a single event.

Culturally, if there was a point of gravitation that the 1960s was pulling towards, then Sgt Pepper's Lonely Hearts Club Band was it. It captured the revolutionary optimism of a generation on both sides of the Atlantic by managing to be quintessentially English in its subject matter, yet West Coast American in its sound; progressive in its musical ambition, yet retrospective in its themes; an album that could be enjoyed by anyone, stoned or straight.

Fifty years on, there might not be such a strong consensus among music lovers as to its status as pop music’s greatest masterpiece.

The argument is sometimes made that Sgt Pepper marked the moment when popular music lost the plot, that the album inspired a lot of over-wrought, over-produced music that required punk to come along and try to blow music back to Year Zero.

But I’m enamoured by it as much today as I was when my trembling, six-year-old hands first slipped the record from its cover.

When I listen to it now, from the orchestral throat-clearing that precedes the title track, to the ta-da of the final piano chord of A Day in the Life, I try to imagine how my parents must have felt when they first listened to it – a record that chimed with the happy spirit of the times and made people feel that the world had slipped ever so slightly on its axis.