Not too long ago, at a party thrown by Aidan Gillen, two actors fell into conversation without knowing they were up for the same part. "Has anyone ever said it to you that you look incredibly like Brendan Behan?" Peter Coonan asked. (It is not a question that everyone will take the right way.) Gary Lydon was not offended: "Yeah, yeah," he said.

Coonan had been immersed in the life and work of Behan while researching a part for Quirke, the television adaptation of the Benjamin Black novels, in which he played a ringer for the Dublin author.

He hoped to play the actual Behan one day, and the two actors spoke about various attempts to put Behan on screen, such as a Peter Sheridan and Sean Penn project mooted in the 1990s. There was also the prospect of a Gaiety Theatre revival of The Borstal Boy, Frank McMahon's adaptation of Behan's famous memoir. The conversation ran into a hesitant silence. Finally, one of them broke it. "Listen, I've been f***ing offered it. Have you been offered it? Are you going to do it?"

Rather than a competition, it was the beginning of a collaboration. Both men play Brendan Behan in Conall Morrison's new staging for Verdant Productions, with Lydon as the elder Behan and Coonan the younger. It is not difficult to find Behan in Gary Lydon's face, sympathetically doughy and often askance. It's harder to find a similarity in Coonan, the south Dublin born actor with green eyes and an easy smile. Then again, as Coonan's relatively young career reminds you, there's more to acting than looking the part.

Violent psychopath

The role Coonan is best known for is the violent psychopath Fran in Stuart Carolan's RTÉ crime drama Love/Hate. Fran, who was introduced in the show's second series, when the programme had already built up a head of steam, began as a low-level smuggler and loan shark with a nasty sideline in arranging dog fights, who was eventually duped and broken down by more senior players. After that, he became something more comical, a try-hard apprentice gangster with a nervy catchphrase: "Coola boola, Nidge. Coola boola." More recently, however, his ferocity reasserted itself.

“This guy is a psychopath, 100 per cent,” agrees Coonan. “But he gradually became more of a psychopath as the story went on.” Fran’s violence found its most lurid expression in the most recent season, in a much-discussed scene involving the smothering of a hapless dentist with a plastic bag. Coonan thinks back on filming the scene with a mixture of excitement and horror: how does a person do something like that?

“I was relishing doing that,” he says, recalling tough encounters in his youth, people “threatening, bullying, putting ownership on a place”. At the same time, his first identification was with the victim.

“That was quite frightening to do, I remember. I needed to get into a dark place that day. I remember sitting on the stairs where it was the darkest in the house, sitting there and being scared myself of the scenario that was about to happen. Taking the fear of the victim and putting that into you to create this venom and menace. You never question it.”

Keeping it in the family

Many actors are taught a method, others learn a technique, and some, like Coonan, discover what kind of actor they are on the job. As a child, the son of a teacher father and an amateur actor mother, Coonan was a talented but unmotivated performer who attended Ann Kavanagh’s Young People’s Theatre and competed regularly in the Feis Maitiú. “It was something I felt able to do. But I stopped that at 16.”

Acting was also bound up with his relationship with his mother, Betty, who died when Coonan was just 12. He mentions her frequently, recalling following her around amateur drama circles to see her perform in plays by Brian Friel and Seán O'Casey, or sitting on her bed and reading her parts with her.

Years later, as a student indifferent to acting in UCD, he was asked to step in for someone in a production of Terence Rattigan's Cause Célèbre via Ann Kavanagh. "I think your mother was in that play," his father told him. Coonan found her tattered script, with her part still marked. She had played Alma, a woman tried with her teenage lover for the murder of her husband, and Coonan was now playing the part of the lover. He still reels with the intensity of the connection, which he took as an omen. "I thought, that means I've got to get back into this game."

In retrospect, it's easy to connect the dots of his career so far: asked to perform in a student play written by his friend – the actor Stephen Jones – Coonan caught the attention of film-maker Mark O'Connor, who cast him as a small-time criminal in the independent movie Between the Canals.

His performance caught the attention of Love/Hate, and has helped generate a resumé of on-screen villainy. He was the best thing in O'Connor's King of the Travellers, a revenge drama in which he played a volatile Mickey Da Bags; has recently finished shooting the fifth series of Love/Hate, as well as The Guarantee, Colin Murphy's film adaptation of his play for Fishamble, in which Coonan plays the former Anglo Irish Bank executive, David Drumm. Does he worry about getting typecast as a crook?

"Yes," he says emphatically. "Very much so. I was definitely conscious [of the need] to do other things, to play with the Corn Exchange, for example." Last year Coonan was almost unrecognisable in Annie Ryan's production of Desire Under the Elms, playing a gawping, mendacious farm hand with a supple physicality. Camera work asks an actor to internalise everything, to communicate in small gestures, where theatre often asks for bigger displays. "Rehearsals were physical; everything was physical," he says. "I had to tell myself, Peter, let go a little bit more, because you're working with people who are trained. Don't be subconscious."

Earlier this year he was offered a film part not unlike his hard-chaw crime roles. He opted instead to go to Belfast's Lyric theatre to play Gar Public in Brian Friel's Philadelphia, Here I Come!, directed by Conall Morrison. "It was tough. For the first week I remember being somewhat embarrassed, too shy to really let it go. Especially the fact that I was playing Gar Public; I knew that was the tough thing, to play within."

Collegial support

Encouragement came from a community of actors; first from an email from Tom Vaughan-Lawlor, who has sustained a versatile theatre career while also playing Love/Hate's kingpin Nidge, and who had played the extroverted, unseen Gar Private. "I hear you're playing Public. It's such a great part," he wrote. "Public is just heart-breaking, the story moves through him."

In one scene, when Gar watches his housekeeper, a maternal figure, for the last time, Coonan channelled his feelings towards his own absent mother. "I didn't have to try and do anything, because the emotion was coming out," he says. But it couldn't be regulated and soon it became harder to reproduce. He turned to another actor friend, Paul Reid, who had played the part recently in London. "Paul said, 'Listen, it happened to me as well.' Some nights he's wondering why he doesn't have an emotional response, other nights he's fighting it back. That was brilliant, to talk to Paul about it and go, don't ever question what you're doing as not being as good another night. It's just different."

Acting is not often celebrated as a supportively collegiate profession, dogged as it is by competitive cliches: "That could be me up there." But Coonan is open about seeking advice and responding to other performers with wide-eyed wonder. "It's your line, Peter," he is prompted on set in Love/Hate when he's dumbstruck by Vaughan-Lawlor's performance, and he is almost apologetic about his reverence for Aidan Gillen.

It’s there, oddly, that his affinity for Behan shines through. Coonan discusses the genius of the man and his self-destruction – the drinker with the writing problem – but also the capacity of a man groomed as a radical republican to despise the British who let his experience in a juvenile borstal opens his eyes. “Open-mindedness – exactly,” says Coonan. “He definitely had that, all through his life.” It’s a good quality and a keen quality to find in an actor too.

Borstal Boy is at the Gaiety Theatre from September 11