It said it had spent £4 million on chasing the contract but was pulling out after “considering all options”.
The Tory party and Leavers had trumpeted the fact that Britain could return to using blue passports, replacing the EU burgundy-coloured ones, after the UK leaves the bloc next March.
But to the dismay of Eurosceptics, the government opted to award the £490m contract to EU-based rival Gemalto.
Martin Sutherland, chief executive of De La Rue, described the original decision as “shocking” and pledged to appeal.
However, that appeal process was concluded at midnight on Tuesday. De La Rue said it remained “both surprised and disappointed” by the decision of HM Passport Office to award the contract to a competitor.
In a statement, De La Rue added: “As we initially announced, we have looked at all possible avenues open to us, and thoroughly evaluated all our options.
“Following four weeks of intense consideration and clear legal advice, we have taken the decision not to challenge the award of the UK passport contract.
“We have been advised that the grounds for overturning the decision are insufficiently strong to justify continuing this course of action.
“We believe De La Rue submitted the highest quality and technically most secure bid.”
The firm, which makes the new polymer bank notes, also issued a profits warning as it said annual operating income would now be around £60m to £65m.
The new blue passport, which was replaced by the EU burgundy in the late 1980s, will be reintroduced from October next year.