Delays in answering phonecalls to HM Revenue and Customs hotlines cost the taxpayer £136m in the last year.
According to the National Audit Office (NAO) report, delays cost customers £33m in call charges while they waited for HMRC to answer the phone and the estimated value of customer time while they waited was £103m.
The NAO said 20 million calls to HMRC hotlines - many of which are 0845 numbers - were not picked up at all last year.
People who did get through were also waiting longer to speak to an adviser - an average of 282 seconds compared to 107 seconds in 2009/10.
In the first quarter of this year, some 6.5 million people were left holding on for longer than 10 minutes.
"Depending on the tariff they pay their phone company, customers are charged once their call is connected even if they are held in a queue," the report said.
"We estimate that if HMRC improved performance to answer 90% of calls and reduced waiting times, it could save customers around £52m a year.
"HMRC currently plans to spend £34m to achieve this level of performance."
The NAO found that there had been some progress since thousands more staff were drafted in, with the 74% pick-up rate significantly higher than the 48% recorded in 2010/11.
However, the report warned that the figures probably underestimated the issue, as calls are counted as answered even if they do not reach an adviser.
Public Accounts Committee chairman Margaret Hodge said: "When people have no choice but to contact the Revenue to discuss their tax affairs, I find it totally unacceptable that HMRC uses costly 0845 numbers and charges people for the privilege of waiting for the department to pick up."
TaxPayers' Alliance chief executive Matthew Sinclair said: "This report exposes a shameful level of service at HMRC.
"Taxpayers will be outraged that HMRC could let 20 million phone calls go unanswered and yet still claim that it is outperforming some arbitrary target."
An HMRC spokesman said: "In 2010/11 we answered 48% of all call attempts, rising to 74% in 2011/12.
"By late 2012 we were answering over 90% of calls to our contact centres. We are well aware that in the past we have not delivered the standard of service to which we are committed.
"We are determined to build on this progress and we have invested £34m so we can deliver on our improvement targets earlier than planned."