The Wolds prison, which has been run by the security firm since it opened in 1992, will return to the public sector in July 2013, the department said.
G4S, which has been at the centre of the Olympics security shambles, failed to secure the Northumberland prison and the South Yorkshire group of jails - Lindholme, Hatfield and Moorland - it was bidding for.
Prison inspectors found the performance of the Wolds Prison in East Yorkshire deteriorated in 2010.
Concerns were expressed over "the availability of drugs, a lack of staff confidence in confronting poor behaviour, weaknesses in the promotion of diversity and limited work and training provision".
Reporting on its latest inspection in June, Chief Inspector of Prisons Nick Hardwick warned that while there were some improvements, "many of our previous concerns still needed to be addressed effectively".
He also found "the third of single cells that had been doubled up to hold two prisoners were too cramped, lacked sufficient furniture and had poorly screened toilets".
Overall, it was a "mixed report" and the prison found itself "on the cusp of potentially significant change, with competitive tenders for the management of the prison signalling uncertainty about its future".
The prison is a category C training prison holding up to 395 men.
A G4S spokeswoman said: "We are disappointed by today's announcements. As the leading private provider of prison management in the UK, we have 20 years of experience of running prisons for the MoJ.
"Our performance across all six prisons we run has been to a high standard with every aspect of performance either meeting or exceeding the key performance indicators applied by the MoJ.
"We look forward to discussing the contract award decision with the MoJ within the next few days to determine why we were unsuccessful."
G4S shares were down 5% after the announcement.
Six (SNP: ^SIXY - news) contracts to run nine prisons, including eight currently run by the public sector, were put up for competition "to balance the need to increase efficiency" and fulfil the Government's plans for a "rehabilitation revolution".
Justice Secretary Chris Grayling said: "The cost of running our prisons is too high and must be reduced. We can do this by being more innovative and efficient, and without compromising public safety.
"That is why I have decided to take a new approach to how we compete prison services and reduce unit costs across the prison estate that will lead to better value for the tax-payer, linked to more effective services to reduce reoffending."
Collective savings will also be made by putting services such as maintenance and resettlement services out to competition, the MoJ said.
Overall these changes should save £450m over the next six years, the MoJ estimated.
Frances Crook, chief executive of the Howard League for Penal Reform, said the Government's decision to hand over prisons to the private sector was "a mistake of Olympic proportions".
"The Government will seek to deflect criticism of its prison privatisation programme by excluding G4S from the next stage of the bidding process, but the principle of awarding lucrative contracts to private companies running prisons on the cheap remains unchallenged," Ms Crook said.
"Something as important as taking away someone's freedom should only be done by the state, answerable to taxpayers, rather than by international private security firms, answerable only to their shareholders."
More From Sky News