For a long time the hippest word in art criticism was “transgressive”. One of the things that defined a work as modern was the breaking of taboos. And there were so many taboos to break: depictions of sex, obscene language, descriptions of bodily functions, blasphemy. Modernity in art was intertwined with a sense of liberation: the artist was the one who dared to say the unsayable, to reveal what was hidden, to make normal what had been shamefully secret. But here’s the problem: in western culture transgression doesn’t feel liberating any more. Its energy has passed to the far right, and in this transition we have a paradox: transgression is being allied to repression.

Modern art defined itself in opposition to mass culture, but also in opposition to the limits imposed by church and state. It represented not just "the shock of the new" but, more obviously, the shock of the shocking. It is impossible for us to appreciate how vile something like Henrik Ibsen's play Ghosts, which turns on hereditary syphilis, seemed to quite intelligent and cultured critics. Ibsen's translator William Archer compiled a lexicon of terms used to describe it in the mainstream English press, among them "abominable, disgusting, bestial, fetid, loathsome, putrid, crapulous, offensive, scandalous, repulsive, revolting, blasphemous, abhorrent, degrading, unwholesome, sordid, foul, filthy, malodorous, noisome."

It's easy, in retrospect, to laugh at these reactions and to write them off as mere reactionary idiocy. But even people who were themselves at the cutting edge of new artistic forms could feel almost physically sick in the presence of work that described the ordinary realities of the human body. Virginia Woolf found Ulysses to be the work of "a queasy undergraduate scratching his pimples" and "ultimately nauseating". She couldn't bring herself to finish it, so we don't know how she would have coped when it gets really filthy, with Molly Bloom's soliloquy.

Language itself was a field of transgression. Well into the 1960s the official stage censor in England, the office of the lord chamberlain, demanded the removal of words we would now consider pretty mild. Joe Orton's Entertaining Mister Sloane, for example, lost "prat", "fart", "shit" and "arse". Samuel Beckett's Waiting for Godot was shorn of "arses", "balls" and "I'd like to pee."

It's easy, too, to forget that western public discourse placed severe limits not just around sex and the body but also around God. Beckett had a terrible time with, as he called him, the lord chamberpot over the very funny prayer sequence in Endgame, which concludes with Hamm's declaration: "The bastard! He doesn't exist." (As Beckett rather reasonably pointed out, this is hardly less offensive to the deity than Jesus's cry "My God, my God, why have you forsaken me?") The controversy around Andres Serrano's Piss Christ, a 1987 photograph of a plastic statue of Jesus immersed in the artist's urine, rumbled on for decades in the United States.

Censorship is no fun, but the transgression of these boundaries gave the avant-garde much of its energy, from Dada to punk, from surrealism to the photographs of Robert Mapplethorpe or Tracey Emin's installation of her unmade bed and stained sheets. Transgression raised the stakes. It gave an edge of danger and courage even to work that was otherwise mediocre. It made audiences and readers feel that they were involved with the artists in an act of rebellion against bourgeois piety and institutional repression.

But this energy declined as the boundaries widened. Capitalist culture turned out to be very good at absorbing the transgressive. The Rolling Stones logo of a tongue sticking out is just another corporate emblem. Sex scenes that would have been scarcely possible in the most outre cinema are standard fare in TV blockbusters like Game of Thrones. The first lesbian kiss in a British television soap opera was in 1994. God (or at least the Christian God) is mocked – regularly. In Fr Jack's ejaculations, "Feck! Arse!" is no more shocking than "Drink! Girls!"

These days transgression doesn't belong to the progressive avant-garde. It belongs to the neofascists. The self-styled alt-right is fuelled by the thrill of transgression. In the guise of a war on political correctness it revels in insult, abuse, shock tactics and the outrageous. It gets its thrills by breaching the taboos erected by progressive culture against homophobia, racism, misogyny and xenophobia. What Seán O'Casey called "throwing stones at stained-glass windows" now appeals mostly to the frat-boy fascists of the internet. But they are no longer marginal: Donald Trump is a walking (or more likely golf-carting) embodiment of transgression. The provocative assault on accepted norms of decency is itself the new normal.

This new form of cultural provocation is not the same as what we must now call the old artistic one. The modernist strategy of transgression was aimed mostly at liberating people from shame about their bodies and their thoughts. It was about expanding the space for human freedom and legitimising groups – homosexual people, for example – who had been placed beyond the pale. The postmodernist alt-right culture of provocation has precisely opposite aims: it uses shame and threat to close down the space for expression and to exclude whole groups from it by delegitimising them. But it poses a very difficult question for contemporary art: in the age of repressive transgression have shock tactics had their day?