As many as 11% of motorists reckon they would not pass their driving test if they had to re-take it, an AA/Populus survey shows.
And it is young drivers rather than the older ones who are least confident of re-passing, the poll of 16,961 AA members revealed.
A total of 9% of all those surveyed said they were "quite likely" to fail, while 2% reckoned they were "very likely" to fail.
As many as 15% of 18-24-year-olds said they were quite or very unlikely to pass, compared with only 9% of over-65s.
Londoners were the most confident of passing, with only 9% believing they were quite likely or very likely to fail, while drivers from the Tyne Tees area were the most pessimistic about their chances, with 13% fearing failure.
Men were more confident of passing a re-sat test than women, with 32% of men believing they were "very likely" to succeed compared with only 23% of women.
Overall, 28% of drivers reckoned they were "very likely" to pass a retaken test, with 55% "quite likely" to succeed.
AA president Edmund King said: "It is alarming to think more than three million drivers who have passed their test may not be safe on the roads due to a lack of confidence or competence to such a degree they don't think they would pass their driving test.
"It is particularly worrying so many young drivers think they would not pass a re-test, when it can't have been long since they took their test.
"Driving is a skill for life and, although it is easy to let bad habits form after your test, drivers should make sure their skills are kept polished."