In October 2002, while still a state senator in Illinois, Barack Obama delivered a speech in which he opposed any US invasion of Iraq, declaring, "I don't oppose all wars. I'm just against dumb ones." On the presidential campaign trail in 2008 he pledged to withdraw all US troops from Iraq within 16 months, and he repeated the commitment to leave the country in his first inaugural speech, in January 2009. US withdrawal was finally completed in December 2012, somewhat behind schedule.

Yet despite his long-standing commitment to extricate the US from the disastrous involvement in Iraq into which his predecessor had plunged it, Obama has authorised renewed involvement in the country, which he said could go on for months if necessary. How has the US found itself directly embroiled in Iraq once more?

The immediate answer lies in the rapid advance of the movement that now calls itself the Islamic State. (It was formerly known as the Islamic State in Iraq and Syria – Isis – or the Islamic State of Iraq and the Levant – Isil.) Fears for the lives of thousands of Yazidis – adherents of a minority religion with Islamic as well as pre-Islamic elements – who were directly threatened by Isis, prompted Obama to authorise air strikes as well as humanitarian assistance.

This seems, for now, to have provided some relief for the Yazidis who were trapped on Mount Sinjar, in northern Iraq.

The Maliki factor
But the origins of the dramatic territorial expansion of Isis and the renewed involvement of the US in Iraq lie elsewhere. They are to be found partly in the failings of the Iraqi state since the US invasion of 2003 and, especially, the sectarian approach pursued by Nouri al-Maliki, prime minister of the country since 2006, who announced on Thursday that he is stepping down.

Maliki, a leading figure in the Shia Dawa party, has systematically alienated political opinion in the country, particularly the Sunni minority that had dominated public life since Iraq became independent in 1932. He has overseen a system characterised by the exclusion and harassment of Sunnis, the emergence of shadowy Shia militias suspected of murder, the politicisation and corruption of the judiciary and the security services, and a military apparatus in which battle commands are reportedly for sale to the highest bidder.

Maliki's inability to transcend mistrust of Iraq's Sunni minority has characterised his time in office, during almost all of which he enjoyed the support, however grudging it may have been, of both the US and Iran, the most significant power brokers in the country. The alienation of Sunnis and other Iraqis has created fertile ground for the expansion of radical opposition to central government in Baghdad.

Asad v Isis
But Iraqi political dynamics are only part of the picture. The expansion of Isis is also inextricably linked to the civil war in neighbouring Syria and the policies pursued by the US and its allies there since the eruption of the anti-Asad uprising in 2011.

Fearful of direct involvement, Obama adopted a cautious approach of limited political and military support for moderate opposition to the regime; it proved insufficient to tip the balance in favour of the opposition or to push the conflict towards resolution.

At the same time, allies of the US in the region, especially Saudi Arabia and Qatar, adopted a very different approach, supporting radical jihadist opposition to the Syrian regime.

Saudi Arabia has long been hostile to the Syrian government. There are profound religious differences – the Wahhabi Saudi leadership is sharply antagonistic to the Shia Alawites who dominate the political, military and security establishment in Syria. Saudi Arabia’s oil fields are concentrated in its Eastern Province, in which the country’s own Shia minority – 10-15 per cent of the population – lives. As a result, the Saudi leadership has a long history of mistrust of any sign of Shia assertiveness.

The Saudis would also see Asad’s removal from power as a significant blow to Iran, a key competitor for regional hegemony.

The tiny Gulf state of Qatar, with its population of 300,000 citizens, has also pursued an activist policy in Syria, supporting radical jihadist groups. It does so for a combination of reasons, partly to do with ideological affinity but more to do with opportunism.

In recent decades Qatar's foreign-policy choices – hosting Al Jazeera and the office of the Taliban, supporting the Egyptian Muslim Brotherhood and Hamas, while also hosting the US military's central command in the Middle East – have embodied its desire to distinguish itself from its larger and more powerful neighbours. With the world's third-largest reserves of natural gas, Qatar has had the resources to spread its influence far and wide.

In this environment, groups such as Jabhat al-Nusra and Isis expanded their sphere of influence. Qatar’s largesse was directed, for the most part, at Jabhat al-Nusra; reports of Saudi support for Isis are widespread, despite official denials.

Paradoxically, this has played into the hands of the Syrian regime, which portrays itself as the only alternative to the ascendancy of Islamic extremists in the country. Indeed, despite their mutual opposition, the Asad regime has strengthened both Jabhat al-Nusra and Isis through the purchase of oil and gas from wells in jihadi control.

The Bush policy
To make sense of the complexity of current events and the role of the US, one needs to take a step further back. Under Obama's immediate predecessor, George W Bush, the US adopted a simplistic frame of analysis of the Middle East, which saw it neatly divided into two camps: a moderate pro-American camp that needed to be supported and a militant pro-Iranian camp that needed to be contained. From this perspective, Turkey, Egypt, Saudi Arabia and the smaller Gulf states were friends to be courted and supported. On the other side was the "Axis of Evil": Iran, Syria and their nonstate allies, Hamas and Hizbullah.

Obama's 'new beginning'
At key moments Obama has seemed anxious to repudiate the inherited simplicities of US foreign policy in the region. From the beginning of his term in office, for example, he signalled the priority he would attach to the Arab-Israeli conflict in the context of a new approach to the Middle East.

Soon after his inauguration, Obama named George Mitchell his "special envoy for Middle East peace" and put pressure on a reluctant Binyamin Netanyahu, the Israeli prime minister, to enter talks with the Palestinian leadership while insisting at a meeting with Netanyahu in May 2009 that Israel must stop building settlements before any progress could be made.

In Cairo, in June 2009, Obama spoke of a “new beginning” for the US and Muslims around the world based on the common principles of “justice and progress; tolerance and the dignity of all human beings”. When the Arab uprisings began in January 2011 Obama offered no support to the autocraticZine El Abidine Ben Ali, president of Tunisia and long-standing US ally. A short few weeks later, if somewhat cautiously, Obama supported calls for Hosni Mubarak to step aside as president of Egypt.

Arab uprisings
Indications of a shift in US policy have been belied by subsequent events, however. The early pressure on Israel dissipated almost as soon as Netanyahu refused to do so much as extend a moratorium on the building of settlements.

The uncertainty and instability that the Arab uprisings prompted across the region, especially the increased influence of Islamist movements, led Obama to revert to familiar tropes. The Bahrain uprising, in which the minority Sunni regime was challenged by a broad-based opposition, including the country's Shia majority, was crushed by Saudi tanks in March 2011. In August that year Obama declared that the future of Syria must be determined by its people, but Asad stood in the way.

Yet, in a “friendly” Gulf monarchy, rule by a minority over a majority was not a problem for the US and its allies. The logic of friends who must be supported at all costs and enemies who were to be thwarted was taking hold once more.

A policy that sees the Middle East in terms of moderates and militants blinds the US to the region's underlying dynamics, as Robert Malley and Peter Harling pointed out four years ago in a seminal article in the journal Foreign Affairs.

Friends and enemies
As is clear in relation to Syria, and now Iraq, the "friends" of the US do not always pursue objectives that are aligned with US interests, while its "enemies" sometimes promote goals that are. Saudi Arabia and Iran both see Iraq through the prism of religion, albeit from different sides, while the secular Syrian regime's approach to Iraq is closer to the nonsectarian vision espoused by the US.

Likewise, US rhetoric on human rights and democracy finds little support in the “friendly” states of the Gulf, while Iran, for all its significant limitations, has a more genuinely participatory political system than most in the region.

Until the US is prepared to rethink some of the fundamental bases of its approach to the region, not least its reliance on the conservative and intrinsically undemocratic regimes of the Gulf, its pursuit of a coherent foreign policy in the Middle East will remain illusory.

Vincent Durac lectures in Middle East Politics at University College Dublin