Schools Minister Vernon Coaker today rejected proposals for children to begin formal education at the age of six.
The proposed change was included in the Cambridge Primary Review published today.
The authors of the report, the most comprehensive review of primary education in England for 40 years, found children respond better to play-based learning at a young age and said there was no evidence to suggest it would hold them back in later life.
But Mr Coaker told GMTV: "For many of those children coming into school, it is of crucial importance they are in that formal but appropriate type learning environment so they can gather the skills and get the skills they need as they go through life.
"Leaving it to six would leave many of our children, particularly those in disadvantaged areas, it would mean they would start a long way behind others."
He added that many of the review's findings were included in an earlier Government-backed review and had already been acted on.
He said: "We're already reducing the emphasis on the SATs test, which is another point they make.
"But as I say, for the reception children coming into school, for the four-year-olds, it is a play-based curriculum, then moving into year one and obviously there is an emphasis on reading and writing."
Chairman of the review Dame Gillian Pugh said introducing children to too formal a curriculum before they are ready for it can damage their development.
She said: "If they are already failing by the age of four-and-a-half or five, it's going to be quite difficult to get them back into the system again."
Rather than delaying the school starting age, the way children are taught until the age of six should be reformed, the report said.
It said: "Many practitioners believe, again backed by considerable research evidence and the positive examples of many other countries, that the principles that shape effective pre-school education should govern children's experiences in primary school at least until age six, and possibly until age seven."
The report found that in the current system of early years education, "quality is too variable, and too many staff are under-qualified or poorly paid".
In many European countries, children start school later, but in Britain almost all three to four-year-olds are already getting at least 12.5 hours of education per week.
The review, six years in the planning and writing, also criticised the testing system for making the entire curriculum too narrow.
It said SATs should be replaced with assessment across all subjects, and ranking of pupils' grades must be separated from the rating of schools' performance as a whole.
The report said a full review is needed of special needs education, where the classification of a child's requirements is too often "arbitrary".
The Government intends to refurbish or rebuild half of primary schools by 2023, and the report urged ministers to bear in mind the greater need for specialist facilities.
But libraries must not be replaced by banks of computers. The study said: "To see screens as the 21st-century replacement for books is a grave mistake."
There is a case for bringing in specialist teachers to primary schools to focus on certain subjects, moving away from the traditional system where one teacher takes one class for everything.
Long summer holidays could also be off the menu, as they are a throwback to the past, according to the review writers.
The way Ofsted inspects schools should also be reviewed.
The report said that since 1997 the new Labour Government had become too involved in the education system and "central prescription of teaching methods and lesson content should now cease".
The education system has not caught up with diversity in society and the authors "encountered evidence of discrimination against marginalised groups, within the education system as well as in society".
Regional conferences will now be held on the study, and a document drawn up to be sent to the leaders of all three main political parties.