They had spent many dark months together, at times shackled to one another by their youthful captors, several of whom taunted them with European accents. The three journalists, who had all been captured in Syria and held by jihadists, joked with each other despite the harshest of conditions. They swapped stories of reporting in other tough places – Libya, Iraq, Afghanistan, Yemen. They shared blankets and whatever scraps of food they were given.

This week the lives of the three men diverged in the cruellest way. In a sun-dappled garden in northern France one of the three, Nicolas Hénin, a French reporter who had been released with three compatriots in April, celebrated his wedding. Days later another of the three, the American freelance photojournalist James Foley, who had gone missing in northern Syria in November 2012, was beheaded by a masked fighter from the group now known as Islamic State. Another militant filmed the killing. At the end of the gruesome video they dragged Steven Sotloff, another American freelancer and the third of the trio, onscreen and warned he could face a similar fate.

I know all three men. Nicolas Hénin and I have been friends for almost 10 years. We met when we were living in the Jordanian capital, Amman, in 2005. Both of us spent months in Libya during the 2011 revolution, where I met James Foley and Steven Sotloff – known to their friends as Jim and Steve.

We often shared cars to get to the front line as it see-sawed across the desert of eastern Libya. Jim Foley stood out for his laid-back nature, his generosity and his ready smile. We had a friend in common who had studied journalism with Jim in Chicago but opted to report on the American educational system instead of war.

Foley was later captured – for the first time – by Muammar Gadafy’s forces and held for several weeks. “It was a kind of siren song that called me out to the front lines,” he told students at his alma mater, Marquette University, in Milwaukee, in December 2011, before returning to the Middle East. “It’s not enough to see it from the distance.”

Steve Sotloff, who describes himself as a “stand-up philosopher from Miami” in his Twitter bio, could be irreverent in both English and Arabic. He became firm friends with several Libyan revolutionaries, including one musician known for bringing his guitar to battle, and who appreciated his droll sense of humour.

All four of us went on to report from rebel-held territories in northern Syria, witnessing from summer 2012 onwards the first signs of how a popular uprising against President Bashar al-Assad became hijacked by extremists who later coalesced into what became Islamic State.

The video

The group’s rise in recent months, including its extraordinary sweep across northern Iraq, has alarmed regional players and prompted US air strikes in an effort to stem its advance. Foley’s executioners claimed they killed him in response to the attacks.

The video of Foley’s beheading, which the increasingly media-savvy Islamic State posted on YouTube for maximum impact on what would have been his 636th day in captivity, opens with footage of President Barack Obama explaining his decision to order the bombing of Islamic State positions. It then cuts to Foley, shaven-headed and dressed in orange clothing, kneeling in an arid landscape. A black-clad militant holds a knife to his throat. Foley’s name appears onscreen in both English and Arabic.

Speaking in a monotone, Jim Foley says his impending death comes in retaliation for the air raids, describing them as “the last nail in my coffin”. He goes on to utter his final words: “I wish I had more time. I wish I could have the hope of freedom and seeing my family once again. But that ship has sailed. I guess, all in all, I wish I wasn’t American.”

After his killing the camera settles on a man clearly identifiable as Steve Sotloff, minus his usual beard and squinting without his wire-rimmed glasses but dressed in the same orange Guantánamo-like clothes as Jim Foley. The masked man grabs his collar and declares: “The life of this American citizen, Obama, depends on your next decision.”

Captured journalists

The last time I met Steve Sotloff was in July last year. We had lunch in Istanbul, where he was then based. I had just returned from a trip to the borderlands between Turkey and Syria, where I had tried to glean any information I could on Nicolas Hénin’s whereabouts.

Hénin had messaged me the day he was captured that June. I was part of a small team of friends and colleagues who had been working on his case since then, reaching out to our network of contacts inside Syria to find out where he was being held and by whom.

That day Sotloff and I discussed all the journalists we knew who had been kidnapped – Nicolas Hénin, Jim Foley and others who have not been publicly identified. I told Steve not to go back to Syria, that it was too dangerous. My advice fell on deaf ears. Days later I was told he had been abducted.

At Nicolas Hénin’s wedding last weekend he and two other former hostages – a French photographer and a Danish photographer – told me that during their long days in captivity he had often recalled my warning to him. They spoke of Jim Foley’s stoicism throughout an ordeal that got worse when their kidnappers discovered while searching his laptop that his brother was in the US air force. “He became a scapegoat,” said Hénin, who shared a room with him for seven months. The others remember Jim trying to make light of the beatings, trying to survive through humour.

All have been knocked sideways by his killing. “He died as I knew him: with great dignity,” Hénin says. “This is not a guy who will complain or whine. He stayed dignified to the end.”

Killings on film

Jim Foley’s shocking murder was a scenario many had feared, suspecting that Islamic State had been deliberately capturing westerners, including journalists and aid workers, to use as possible leverage in the future.

The Committee to Protect Journalists, which Foley told his fellow hostages he hoped to work for when he was released, has called Syria the most dangerous country in the world for journalists. According to its most recent figures, more than 80 reporters have been kidnapped. About 20 are currently missing; many of them are believed to be held by Islamic State.

The tactic of killing foreigners on film is reminiscent of al-Qaeda in Iraq, the group’s forebear, which decapitated a number of westerners between 2004 and 2007. Its Jordanian leader, Abu Musab al-Zarqawi, earned the sobriquet “Sheikh of the Slaughterers” because of his preference for personally beheading captives, who were usually dressed in orange clothing similar to what Foley and Sotloff wore in this week’s video.

Zarqawi’s methods proved too extreme for al-Qaeda’s then deputy leader Ayman al-Zawahiri, who told him to use a gun instead. Zarqawi was later killed in a US strike, but his cadre re-emerged in Syria under the name Islamic State of Iraq and ash-Sham (an Arabic term broadly meaning the Levant), or Isis.

Their ruthless ways of dealing not just with Assad loyalists but with entire communities in rebel-held northern Syria led to al-Qaeda formally breaking all ties to the group this year. In light of its new territorial gains, Isis recently truncated its name to Islamic State.

In Iraq, Islamic State fighters have occupied Tikrit and the northern city of Mosul since early June, plus swathes of the country’s north and west. More than 1.5 million people have been displaced by fighting there since the group’s lightning advance began, more than two months ago, and thousands more have died.

Its leader Abu Bakr al-Baghdadi, who has declared a caliphate across territory that Islamic State controls in Iraq and neighbouring Syria, draws on powerful religious symbolism in an effort to legitimise a group that many view as merely unschooled – but dangerous – rabble.

Foreign, not Syrian

A recent documentary aired by Vice News provided an unprecedented glimpse of life as it is now lived inside Raqqa, the northern Syrian city that Islamic State seized last year and currently considers its capital.

Raqqa is where Nicolas Hénin was captured last year. For months after his kidnapping I was in contact with activists in the city who described to me how Islamic State imposed its will through fear and intimidation.

Demonstrations against the jihadists petered out after they started publicly executing alleged transgressors. “Our Raqqa is just a shadow of what it once was,” one resident said. “These people rule through brute force.”

The Vice documentary shows swaggering Islamic State fighters indoctrinating young children to kill “infidels” and patrolling the city for any signs of behaviour considered “un-Islamic” according to their austere interpretation of the faith.

It also shows scenes of crucified corpses and the severed heads of regime soldiers killed in battle. What is striking about the film is that few of the Islamic State fighters who feature in it have Syrian accents – a great number within the wider movement are foreigners.

Tunisians, Libyans, Iraqis and Saudis constitute the largest nationalities, but there are also Europeans. The militant filmed killing Jim Foley speaks with what sounds like a London accent; he is thought to be one of a number of British jihadists collectively known as “the Beatles”.

A French journalist freed earlier this year said one of his captors sang Charles Aznavour ballads despite the extremists’ denunciation of music. Mehdi Nemmouche , the young French man arrested over the shooting dead of three people at a Jewish museum in Brussels in May, had spent time in Syria. When he was apprehended in southern France, he was carrying weapons wrapped in a sheet scrawled with the name of Isis.

Islamic State may have been birthed in Iraq but it found a whole new purpose in the midst of Syria’s devastating war, which has left more than 170,000 dead and displaced over nine million others in what many consider the worst humanitarian crisis in decades.

Such is Islamic State’s brutality in Raqqa and other parts of northern Syria it is not uncommon to hear locals there repeat conspiracy theories claiming it is in cahoots with Assad, given his regime’s role in facilitating the stream of jihadists to Iraq after the US invasion in 2003.

Threat to the region

Islamic State’s ranks have been swollen by predominantly young men, some even in their teens, from a multitude of countries who are drawn by its uncompromising approach and the camaraderie of battle.

But several former jihadists, particularly those who fought the Soviets in Afghanistan in the 1980s, say they are greatly unsettled by the ascendancy of Islamic State. “These kids have been influenced by the extreme violence and confusion of ideologies they have been exposed to on the internet,” says one man whose father was a prominent figure in the war against the Soviets. “Because of this it is more difficult to control than anything that has gone before.”

An Arab diplomat offers a chilling assessment: “This is a virus that poses not just a security threat but an ideological threat to the whole region.”

The US must now weigh the risks of adopting a more aggressive strategy to rout what Obama this week called the “cancer” of Islamic State against any action that could result in the death of another American.

In an emotional speech Obama paid tribute to Foley and accused Islamic State of “cowardly acts of violence” and abuses including enslavement and sexual assault.

From their home in New Hampshire, Jim Foley’s parents, who had campaigned tirelessly for his release, implored their son’s killers to spare the lives of Steve Sotloff and other hostages. “Like Jim, they are innocents,” his mother, Diane, said. “They have no control over American government policy in Iraq, Syria or anywhere in the world.”