Brian O’Doherty is a curious man. He was a leading conceptual artist in 1960s New York, and his painting and sculpture are held by the Irish Museum of Modern Art, the Metropolitan Museum of Art, the Smithsonian Institution and the Pompidou Centre.

As a novelist, his book The Deposition of Father McGreevy was shortlisted for the Man Booker Prize in 2000, and in May he published The Crossdresser's Secret, about an 18th-century drag queen.

An art critic, his essay series Inside the White Cube has been in vogue since it appeared in Artforum magazine, in 1976. He also commissioned Roland Barthes's classic essay The Death of the Author, in 1967. And all of this came after training as an oncologist at Cambridge and Harvard; he left medicine to pursue art in his late 20s.

His curiosity doesn’t even switch off on holidays. For nearly 40 years he has been turning his summer house into an artwork. La Casa Dipinta is an 18th-century townhouse in the hilltop town of Todi, in northern Italy, which he bought in 1975 and has been covering in wall paintings since.

It’s at 25 Via della Mura Antiche, approached via the steep pathways that snake around the medieval piazza. It gets about 100 visitors a year, the local guide says. (A tour is €10.) The act of colour-bombing these ancient walls alone deserves more visits than that. Every wall is covered: Dante’s circles of paradise by the bath, with rope installations instead of a television.

O'Doherty and his wife, the art historian Barbara Novak, usually stay here every September, but they haven't been for two years. "I see a lot of doctors these days," he says from his Manhattan apartment, in a worn-out, gravelly voice. It is like talking to Humphrey Bogart.

The artist, who was born in 1928 in Roscommon, and grew up in Bray, Co Wicklow, even now has a heavy schedule. “It means I can live, doesn’t it? It means I can pay the rent.” Was there a moment when he felt he had arrived? “As an artist you’re always arriving,” he says, laughing. “The 1960s and 1970s was the height of my reputation. I was hot. Then you get cold, you cool off. Then you get hot again. For some reason I’m hot again right now. You get neglected, you get rediscovered. That’s the usual metabolism of an artist’s life.”

La Casa Dipinta stands remote from such throes. O’Doherty says he painted it for his wife. Todi is in Umbria, the “green heart of Italy”, but they had no pretty view from their windows. “Barbara particularly missed the landscape around Todi, which is glorious in all seasons,” says O’Doherty. “And you’re in the land of frescoes. I painted her a landscape in the bedroom.” He painted silhouettes of each of them over their bed, tried some installations, and covered the kitchen in ogham script.

In the land of religious pictures, this fifth-century Celtic code is painted in pastel pinks, greens, blues and yellows. The artist learnt ogham at national school, in Loughlinstown, and brought it into his early conceptual art. His ogham sculptures are fetching six figures today.

Here ogham illustrates the words "One here now", written in four languages around the kitchen archway. In 1967 O'Doherty said he wanted to reduce his conceptual art language in his work to just these three words. (Brenda Moore McCann's monograph Brian O'Doherty/Patrick Ireland: Between Categories can tell you more.) It is a stern, somewhat messianic presence in the house, which is also very homely.

Paint and brushes

There are logs in the fireplaces and a chunky vintage telephone that occasionally rings. There are pots of paint and brushes, and unfinished work. In the kitchen an hourglass outline on a wall will require O’Doherty to climb a ladder for completion, and Novak’s profile languishes under the cobwebby arch. O’Doherty is “not at all” bothered by the possibility of not finishing these.

As for living inside a painting, does he not find it relaxing to separate life and work? “The work is the life, isn’t it? I find making art a joyful business, particularly in that house. It’s just flooded with colour. It’s got a very firm intellectual underpinning.”

In an art library that is in progress you can flick through critical volumes, or through pictures of La Casa Dipinta by Italian schoolchildren.

The house is a sweet contrast to O'Doherty's more sensationalist projects. In 1972 he changed his name to Patrick Ireland, in protest against Bloody Sunday and the British military presence in Northern Ireland. He called it Name Change in a performance piece at Project Arts Centre, with his fellow artist Robert Ballagh as witness, then buried the name, in 2008, in a "funeral" at Imma.

Another of his pseudonyms, used as a critic and performance artist, is Mary Josephson. "My mother gave me the name Mary, which I resented profoundly," he says. "I always wanted to write as a woman, whatever a woman is, and each sex is mysterious to the other." Do women write differently from men? "That's a very interesting question," he says and jumps into discussing his novel.

"It's called The Crossdresser's Secret. It's based on the life of the Chevalier D'Eon, born 1728, a fascinating guy. The novel is about gender, the politics of gender, the politics of the age. I've always led parallel lives. I've always kept many, many streams going.

“If you’re in the right circumstances, and given the right permissions, I believe people can do an awful lot, if the culture allows them. But in general cultures tend to limit people to a single profession. My philosophy is that I only come around once, as far as I know, before going into the darkness. If you don’t do it well, it isn’t worth doing, is it?”