Most doctors need more training to spot the signs of autism, according to a report out today.
The study, from the National Audit Office (NAO), also found that around half of the estimated 400,000 adults in England with autism may be falling through the gaps due to a lack of services.
This is because they do not have a learning disability and services are mostly set up for people with illness, physical or learning disabilities, and mental health problems.
A poll of 1,000 GPs for the report found 80% thought they needed additional guidance and training to identify and manage those with autism more effectively.
Many did not know where to refer people with autism due to a lack of specific services.
"In our survey of GPs, 64% told us that they referred adults with suspected high-functioning autism to adult mental health services, and 19% to learning disability services," the study said.
"(A total of) 12% were not sure where they should refer such patients, and only 10% reported that they would refer them to a specialised autism diagnostic service."
The NAO found that most NHS organisations and local authorities do not know how many people with autism live in their area, something it said should change.
It also called for specialist support services.
The study - called Supporting People with Autism through Adulthood - said: "Providing specialised support could improve outcomes for this group of people and their carers, and potentially enhance value for money, as the costs of establishing such support could be outweighed over time by overall savings."
Tim Burr, spokesman for the NAO, added: "Greater awareness of the numbers of people with autism, as well as better understanding of autism amongst those providing health, social care, benefits, education and employment services, would lead to improved quality of life for those on the autistic spectrum.
"Specialist support and joint working across all areas - clinical, social and employment - could improve the transition from childhood to adult services, make services more effective and improve value for money."
Mark Lever, chief executive of the National Autistic Society (NAS), said: "In the current economic climate the Government cannot possibly ignore the huge cost savings and benefits, identified by their own auditing body, of providing adults with autism with the right support at the right time.
"Neither the Government, people with autism nor the taxpayer are getting value for money from existing autism services and support, leaving those affected by the condition feeling isolated, ignored and often at breaking point. This is simply unacceptable."
Care services minister, Phil Hope, said: "We must ensure that the excellent services out there are the norm and not the exception.
"There is now a national consultation under way and at the end of the year we will publish the first ever autism strategy.
"Our commitment to do this will have the force of law - in the first ever Autism Bill going through Parliament.
"We currently do not know how many people have autism so this year we will begin a research study. We will be able to better target help when we know how many people need it."