A £24 million offer from security giant G4S to settle an overcharging scandal has been rejected by the Government as officials vowed to "pursue all possible avenues" to recoup more taxpayers' cash.
The under-fire private contractor publicly admitted "unacceptable" failures in the way it charged for tagging offenders - which included bills for keeping tabs on people after they died.
It said an independent review had found no evidence of dishonesty or criminal conduct.
G4S and fellow contractor Serco are the subject of a criminal investigation by the Serious Fraud Office (SFO) over the way they operated the contracts.
The investigation began after a forensic audit ordered by the Government following concerns raised about the system - including by one former G4S employee - exposed widespread failings.
The shock revelations sparked a Government-wide review of all contracts held by Serco and G4S.
A report issued by the National Audit Office (NAO) said the audit - carried out by accountancy firm PWC - had cost £2 million so far.
It found Serco received £15,500 for operating a monitoring unit it had not been able to install in the first place for nearly five years and G4S kept taking money for three years in another case, netting £4.700.
In another case, Serco billed the Ministry of Justice (MoJ) four separate times for the same tag as the individual was under multiple supervision orders, the public spending watchdog said.
G4S said in a statement it had " wrongly considered itself to be contractually entitled to bill for monitoring services when equipment had not been fitted or after it had been removed".
Credit notes have been issued for £23.3 million for the period from 2005 to May 2013, with another for £800,000 to follow for recent months.
There was no evidence that it extended to any of its other Government contracts, said the firm, which has faced repeated criticism of its performance, not least over security for the 2012 Olympics.
G4S group chief e xecutive Ashley Almanza - who faces a grilling by MPs on the issue when he and other Government contractors including Serco appear before the Commons Public Accounts Committee tomorrow - said the offer was " an important step in setting this matter straight" and restoring trust.
"The way in which this contract was managed was not consistent with our values or our approach to dealing with customers," he said.
"Simply put, it was unacceptable and we have apologised to the Ministry of Justice."
The G4S executive who had been due to appear before the committee - UK and Ireland chief executive Richard Morris - has since departed in a widespread change at the top of the firm.
The firm said it was ready for further negotiations with the MoJ if the audit concluded that it had overcharged by more than the sum offered.
Serco has told the NAO it "considers it charged in line with its genuine interpretation of the contract" and that it was "open" with the MoJ throughout.
It has said it will repay anything it agrees it overcharged.
An MoJ spokesman said it was still working with the firms and the independent auditors to work out "what the final sum will be".
"The Secretary of State has been clear: we are determined to secure a refund for the taxpayer," he said.
"We have taken appropriate legal advice and will pursue all possible avenues.
"This matter is now the subject of a criminal investigation. We are not able to comment further at the current time. We will make a further statement when it is appropriate to do so."
Shadow justice secretary Sadiq Khan said: "This report is an important first step in getting to the bottom of what went on in G4S and Serco, with G4S admitting to £24 million-worth of overcharging.
"With the Serious Fraud Office still investigating, neither company should be awarded any more public contracts by David Cameron's Government unless and until both have been given a clean bill of health.
"Failure to do so will further undermine the public's confidence in our justice system under this out-of-touch Government."
Labour MP Keith Vaz, chairman of the Home Affairs Select Committee, said: "After the Olympic shambles, the committee recommended that the Government implement a high-risk register to stop this happening again. Had it been adopted, the overbilling from June may have been prevented."