MEMOIR: Must You Go?: My Life with Harold PinterBy Antonia Fraser, Weidenfeld & Nicolson, 328pp. £20

LADY ANTONIA FRASER, the willowy ash-blond daughter of Lord Longford, the former wife of the late Conservative MP Hugh Fraser and the second wife of Harold Pinter, has, a year after the death from cancer of Britain’s most famous and influential modern playwright (on Christmas Eve 2008), written a paean to their enduring love affair and marriage. With unarguable dignity, and a marked lack of irony or self-doubt, Fraser has produced a long, graceful, albeit at times cloying account of their seemingly extraordinarily happy life together.

Must You Go?is based on Fraser's diaries, in which she virtuously charted their busy, fulfilling and deeply loving relationship, interspersed with her elegantly written and occasionally moving retrospective musings. A strangely persistent book, this memoir, like the historical biographies Fraser has been writing since her early 20s, illuminates an era, with its court of characters – the likes of Samuel Beckett, Arthur Miller, Laurence Olivier, John Fowles and Edna O'Brien, who somehow seem as distant as the men in frocks who more usually populate the author's work.

Fraser and Pinter's relationship began after a dinner party hosted by her brother-in-law, Kevin Billington, who had directed Pinter's The Birthday Party. "I was slightly disappointed not to sit next to the playwright, who looked full of energy, with black curly hair and pointed ears, like a satyr," Fraser recalls.

The seating arrangements did not prevent the star-crossed couple from finding each other, despite both being encumbered by other spouses. Pinter was in an apparently difficult marriage to the actor Vivien Merchant (the couple had one son), while Fraser was firmly ensconced in her rambling west London home with her Tory MP husband, six children and various cats. Leaving the dinner party that evening, Fraser stopped by Pinter’s chair to congratulate him on his theatrical success:

“‘Must you go?’ he asked.

“I thought of home, my lift, taking the children to school the next morning, the exhausting past night in the sleeper from Scotland, my projected biography of King Charles II . . . ‘No, it’s not absolutely essential,’ I said.”

Pinter, having escorted Fraser home in his chauffeur-driven car, remained in her house until dawn. “He stayed until six o’clock in the morning with extraordinary recklessness, but of course the real recklessness was mine.”

This is the point when one casts the book aside in a state of bleak incomprehension. Not that Fraser chooses to cavort with a married man but that she would have the energy for such an endeavour at all, with six bleating offspring to care for. Presumably, her domestic life, already propped up by boarding schools, was also staffed by people who grilled the fish fingers and sorted the socks.

By the time the press eventually got its inky mitts on the scandalous story of Harold and Antonia, the couple were sweeping their lives clean of impediments and had embarked on a relationship that would last for 33 years. Fraser's description, written with her trademark aristocratic understatement, of breaking the news of her passion for the saturnine playwright to her husband, Hugh, borders on high comedy: "It was beyond ghastly – I fetched him inside from the garden, where he was smoking and reading the FT."

Later, Fraser fetches Pinter around to help sort out the domestic situation. “Hugh and Harold discussed cricket at length, then the West Indies, then Proust. I started to go to sleep on the sofa, Harold politely went home. Nothing was decided.”

Despite the unintentional comedy, Fraser’s benign tome to her raven-haired lover is, at times, as irritating as getting the silver spoon stuck in one’s craw. Fraser describes herself, the Catholic aristocrat, and Pinter, the Jewish boy made good from London’s East End, as “bohemians”, backing up this unconvincing thesis with such lines as: “In various fish restaurants in Dubrovnik, money worries are discussed.”

Having jettisoned the spouses – Hugh Fraser went off amicably to shoot grouse in the Highlands while Merchant, described by Fraser as an alcoholic, clung on to the corpse of her marriage until shortly before her early death – Pinter moved into Fraser’s Holland Park home. Later, he purchased an adjoining mews to write in, away from the holler of the children during the hols.

The middle section of the book roves from Caribbean vacations to Parisian readings, to play openings in New York City, to sipping Corvo Biancos under a Venetian winter sun, to sojourns in eastern Europe and South America. Everywhere the couple go they are feted by the artistic community and the intellectual elite, due to their political activism as well as to Pinter’s stunning dramatic canon. The names Vaclav Havel and Daniel Ortega (the Sandinista leader in Nicaragua) are scattered among the guest lists to the couple’s home, along with just about every successful actor, director and film-maker of the past half- century.

Despite the occasional pomposity and the sheer privilege that permeate the pages, you can’t help but feel deeply impressed by Pinter and Fraser’s energy, their political engagement and intellectual rigour. Pinter may have shouted louder and more vociferously, but Fraser, although instinctively more conservative, was no lightweight.

Strangely, despite its being embroidered throughout with Fraser’s abiding love for her husband, this book has an absence at its centre. Somehow Pinter goes missing: his moodiness and bouts of depression are touched on, the green shoots of his ideas for new work are documented, his political views are revealed (we discover that, at one stage, he even voted Conservative and supported the Falklands War), his gentle domesticity and uxorious nature are painted in sharp relief to his more truculent and volatile public persona, but ultimately his character remains abstruse, somehow untouchable. It is as if Fraser loved him so instinctively and so entirely that she had no need to analyse the man.

The final section of Must You Go? is the strongest and most affecting, an account of Pinter’s ill health, his 75th-birthday celebrations in Dublin, his Nobel Prize, his struggle to make his political voice heard and Fraser’s gradual realisation that her time with Pinter is running out.

This is a moving account of carefully crafted and cherished love, a theme that is, in itself, unusual.

Hilary Fannin is a playwright and journalist. She is writing a contemporary version of Racine’s Phaedre for Rough Magic Theatre Company with the composer Ellen Cranitch