Most diverse Booker prize shortlist ever as Hilary Mantel misses out

Hilary Mantel will not win a third Booker prize with the final novel in her Thomas Cromwell trilogy, after American writers made a near clean sweep of this year’s shortlist.

With four writers of colour among its six authors, the shortlist, announced on Tuesday, is the most diverse line-up in the prize’s history. Four debut novelists – Diane Cook, Avni Doshi, Douglas Stuart and Brandon Taylor – are up against the acclaimed Zimbabwean Tsitsi Dangarembga, and the Ethiopian-American Maaza Mengiste for the £50,000 award.

Apart from Dangarembga, all the authors are from the US or hold joint US citizenship. The rules were changed in 2014 to allow any writer writing in English and published in the UK to compete for the award. This has been widely criticised by the British publishing industry, which warned it would lead to its domination by US authors. Two, George Saunders and Paul Beatty, have won the Booker since the rule change.

“No one wins the Booker prize because of who they are. A book wins because of what it does,” said Gaby Wood, literary director of the Booker Prize Foundation as the shortlist was announced. “What has transpired is a testament to the judges’ faith in – among other things – first fictions: they have found these writers to have much to say, and found them to have said it in a way that became even richer on a second reading.”

Dangarembga, who was recently arrested in Harare during a peaceful protest against government corruption and who is due in court on 18 September, made the Booker line-up for This Mournable Body. A sequel to her 1988 novel Nervous Conditions, it follows a woman trying to make a life for herself in post-colonial Zimbabwe.

Mengiste, shortlisted for her story of the ordinary people who rose up during Italy’s 1935 invasion of Ethiopia, The Shadow King, is the first Ethiopian writer to make the Booker shortlist.

Related: Booker nominee Avni Doshi: 'Women feared my ambivalence towards motherhood'

The other novels in the running are Cook’s The New Wilderness, following a mother trying to keep her daughter safe after an environmental disaster; Doshi’s Burnt Sugar, about the toxic relationship between an adult woman and her ageing mother; Taylor’s Real Life, a campus novel tackling racism and homophobia; and Stuart’s Shuggie Bain, the story of a childhood blighted by poverty and addiction in 1980s Glasgow.

“Every year, judging the Booker prize is an act of discovery. What’s out there, how can we widen the net, how do these books seem when compared to one another, how do they fare when re-read? These are questions judges always ask themselves, and each other,” said Wood.

In regard to the number of US authors up for the award, Wood said: “Whether it is a problem that is for others to say, but we don’t think of it as a problem. We think of it as being for readers, and readers don’t check passports.”

Chair of judges, the publisher Margaret Busby, agreed. “We were not conscious of if someone was British or not, we were looking at the books. In the end we ended up judging on the basis of what we thought of the books we were given, not on nationalities,” she said.

The judges read 162 novels to come up with their shortlist. Mantel was not the only major name who made the longlist but missed out on the shortlist: Anne Tyler and Colum McCann also failed to make the final cut.

“It is an absolutely wonderful novel, there’s no question about it,” said judge Lee Child, of Mantel’s novel, The Mirror and the Light. “It’s a trilogy which will live forever. But as good as it was, there were some books which were better.”

“The shortlist of six came together unexpectedly, voices and characters resonating with us all even when very different,” said Busby. “It’s a wondrous and enriching variety of stories, and hugely exciting as well.”

Four of the shortlisted titles come from independent publishers. Taylor’s novel Real Life is published by Daunt Books Originals, an imprint founded only in February; while Cook’s dystopia comes from Oneworld, the press that won two consecutive Bookers, for Marlon James’s A Brief History of Seven Killings and Beatty’s The Sellout. Dangarembga is published by Faber & Faber, and Mengiste by Canongate.

Busby, who became the first black woman publisher in Britain when she co-founded Allison and Busby in 1967, said that many of the novels she and her fellow judges – Child, Sameer Rahim, Lemn Sissay and Emily Wilson - read conveyed “important, sometimes uncannily similar and prescient messages”.

Related: Judging the Booker prize: 'These books are about living under intense pressure'

“The best novels often prepare our societies for valuable conversations, and not just about the inequities and dilemmas of the world – whether in connection with climate change, forgotten communities, old age, racism, or revolution when necessary – but also about how magnificent the interior life of the mind, imagination and spirit is, in spite of circumstance,” Busby said. “We are delighted to help disseminate these chronicles of creative humanity to a global audience.”

Busby said that, in the first 43 years of the prize – now in its 51st year – only two people of colour had served as Booker judges, out of more than 200.

“That’s not to say that under the auspices of the Booker, brilliant decisions have not been made about excellent literature for decades. However, it does matter for there to be opportunities to see culture and creativity from different perspectives,” she said. “Each of us makes judgments through the prism of who we are and what we have learned or internalized. That’s why diversity has always been important. Diversity is reality. The scope of this year’s books has allowed us to luxuriate in skillful storytelling and to be surprised by what unheard voices have to articulate.”

Last year’s winner, Bernardine Evaristo, wrote on Twitter that she was “so excited by this groundbreaking shortlist for the 21st century”. “If you’re looking for fresh perspectives and narratives, surely you’re going to find it among the most underrepresented voices?” she wrote.

The winner will be announced on 17 November, with the traditional dinner at the Guildhall in London attended by the great and good of British literature replaced with an event broadcast from the city’s Roundhouse venue in collaboration with BBC Arts. Last year’s award was jointly – and controversially – won by The Testaments by Margaret Atwood and Girl, Woman, Other by Evaristo, after judges “explicitly flouted” the rules.

• Guardian Live will host an online event with the six shortlisted authors on Wednesday 11 November.