WB Yeats is often viewed as being away with the fairies in the Celtic twilight, whereas James Joyce is considered a man of this world, grounded in the prose of everyday life. Joyce, however, was no stranger to ghosts, or to the grief that takes leave of the senses.

On the night of his mother's funeral in 1903, Joyce kept a vigil with his sister Margaret ("Poppie") for their mother's ghostly return, and although only his sister purported to see the ghost, it closely resembles the revenant of Stephen Dedalus's mother in Ulysses: "In a dream, silently, she had come to him, her wasted body within its loose graveclothes giving off an odour of wax and rosewood."

Notwithstanding his well-known scepticism, Joyce was susceptible to superstition and had an almost primeval fear of thunder: the seismic roar of the heavens 100 letters long precipitates the Fall at the beginning of Finnegans Wake and rumbles throughout the work. As Bloom ruminates in Ulysses: "Something in all those superstitions because when you go out never know what dangers."

Joyce's sense of loss extended to the nightmare of Irish history, and the "old haunts" of the city he carried with him on his exile in mainland Europe. Asked once by Hanna Sheehy-Skeffington why he did not return to Dublin, Joyce replied: "Have I ever left it?" It was perhaps Dublin that never left Joyce: as late as the 1930s he recounted to Constantine Curran that "every day in every way I am walking along the streets of Dublin and along the strand. And 'hearing voices'."

The phantom in Joyce does not belong to the gothic, for that genre only makes sense historically in cultures that have given up the ghost and no longer succumb to haunting. Hamlet and Macbeth are not gothic tales, strictly speaking, for the spectre had not yet passed from life into literature and was an everyday affair, despite the best efforts of the Reformation to dispel superstition. The cultural milieu of Joyce's Ireland had similarly to undergo the full rigours of disenchantment but was no less integrated into modernity for all that. It is not so much that the ghost was general all over Ireland but that belief itself was kept at bay.

Moments of crisis

In Joyce's work, haunting takes place in moments of crisis where the past eludes the private nets of memory but has not yet become public history. It is common to see the ghost as a sign of mental breakdown, but the fault may be less in the psyche than in the social conditions that convert history itself into haunting. According to Sigmund Freud, the ghost is a "projection" of the mind, but that presupposes that a fully formed inner life is already securely in place, which was far from the case in early 20th-century Ireland.

This has important implications for rational attempts to explain away the ghost by relegating it to psychology, for Joyce’s scepticism about the “Viennese school” is as concerned to raise questions about spectres of the self as about spectres from the otherworld. In Joyce, subjectivity itself is a phantom and is no less spectral for being situated inside the head as against shadowy external forms.

Banshee and Holy Ghost

The ghost, in this sense, emanates from an incomplete formation of the self, in the metropolitan, western sense. In his Encyclopaedia, Hegel welcomed the conjunction of "memory" and "interiority" in the German language, but memory, historically, has never been sealed off from the external world, let alone public life, in Ireland. It is not that the ghost is of one's own making but that it does not always make it to the mind in the first place.

Rather than being a given, inner life was an essentially "half-formed thing" (as Eimear McBride might have it) under colonial rule. This is well-captured in the description of Mrs Kernan's belief in "Grace", in Dubliners, socialised to a suburban red-brick existence on the Glasnevin road but yet not quite at one with her beliefs: "Her beliefs were not extravagant. She believed steadily in the Sacred Heart as the most generally useful of all Catholic devotions and approved of the sacraments. Her faith was bounded by her kitchen, but, if she was put to it, she could believe also in the banshee and in the Holy Ghost."

Manifestations of the uncanny along these lines feature repeatedly in Ulysses. Stephen is haunted by the "ghoul" of his dead mother, whom he had forsaken on her deathbed due to his refusal to kneel down and pray. Bloom is haunted by his infant son Rudy, who died 11 days after his birth and who appears at the end of the Circe chapter in what could be a projection of a different kind: a magic lantern image.

Technology is often taken to be at odds with the otherworld, and thus, as Patricia Lysaght wrote in her book The Banshee, electric light did more to dispel the ghost in Ireland than any exercises of reason. It is no coincidence, then, that a power cut in the Gresham Hotel coincides with the manifestation of the ghost of Michael Furey in the hotel room shared by Gabriel and Gretta Conroy at the end of Joyce's story The Dead.

The Gresham showcased its modern electrical conveniences much as hotels advertise wifi access nowadays, but with the outage on January 6th, 1904 (and there were power cuts in Dublin over Christmas that year), the couple had to make do with the “ghostly light” coming in from the gaslight on the streets. We then find out that Michael Furey worked in the gasworks in Galway (and, indeed, Joyce was surely aware that “gas” derived from the Dutch word for “ghost”, as claimed in Skeat’s dictionary, which he possessed).

Parnell and De Wet

Thus for all the physicality of the city and the hard-edged clarity of Joyce’s prose, there is always a recurring suggestion of something offstage, a relationship to presences and absences vaguely apprehended by those walking the streets. At times, it is as if the city itself is a face wherein people may read strange matters, not least the traces of Parnell – “poached eyes on ghost” – that Bloom discerns in the eyes of his nondescript brother, John Howard Parnell, walking up Grafton Street.

There may be a reason for the sudden "appearance" of Parnell, for Bloom had just recalled being caught up in an anti-Boer war demonstration some years before, outside Trinity College. Later, the rumour surfaces in the Eumaeus chapter at the end of Ulysses that Parnell was not dead at all, but may have absconded to South Africa and reinvented himself as Christiaan De Wet: "He changed his name to De Wet, the Boer general." The resemblance between two was striking, and then we remember that De Wet was also on Bloom's mind earlier as he passed Trinity College, just before he saw Parnell's brother: "Up the Boers! – Three cheers for De Wet!"

In Carlos Fuentes's novel Terra Nostra (1975), a character explains his creation of a Theatre of Memory in which unrequited pasts are staged as trial runs for the future: "The images of my theatre bring together all the possibilities of the past, but they also represent all the opportunities of the future, for knowing what was not, we shall know what demands to be: what has not been, you have seen, is a latent event awaiting its moment to be."

The ghost may represent a refusal to let go, as in classical accounts of melancholia, but this may also lead to new attachments and solidarities, as in the transfer of loyalties from Parnell to De Wet, from the cause of Ireland to the cause of the Boers fighting empire thousands of miles away.

Instead of shutting down the past, the ghost in Joyce occupies a transitional zone in which traumatic memory has the potential to act as a resource of hope, providing glimpses of a way forward, even in the darkest of times.

There is always something left over from the past, wrote Joseph Brodsky, and that is the future.

• This is an extract from Luke Gibbons's book Joyce's Ghosts: Ireland, Modernism and Memory (2015), published by University of Chicago Press