A crackdown on police cautions will follow a review of guidelines, Policing Minister Damian Green said.
Mr Green said a rapid review of guidance on when a police officer can recommend a caution would report to the Home Office in May.
He said the Government's goal is to see the use of cautions restricted to ensure serious criminals would never escape punishment.
Speaking on the BBC Radio 4 Today programme, Mr Green said: "Serious and repeat criminals should not expect to escape with a caution so we are not only launching a review, we have also changed the guidelines to make it clearer serious and repeat criminals should not expect to get a caution.
"It may be that the guidelines were not clear enough in the past and the new guidance we are issuing actually does provide more specific guidance on the exceptional circumstances when you can give a caution even if there is a serious offence committed.
"It may well be something to do with the mental health or the age of the offender. You do have to give that ultimate decision to the police officer involved but I do think in terms of having general confidence in the system it is clear, on the whole, you only want cautions to be used for low level offences for first time offenders and so on."
Chris Eyre, chief constable of Nottinghamshire Police and Acpo (Association of Chief Police Officers) lead on cautions, said caution decisions were not made in a "cavalier" fashion but by "sensible local officers".
He told the same programme: "It's really important we take a measured view on this and make sure everything that goes on is working to ensure our communities are safer.
"Crime is coming down across communities. We are increasing public satisfaction in police services and we are increasing public confidence. That is because considered decisions are being made which are right in the circumstances locally. It's really important we have local context in these things."
Justice Secretary Chris Grayling told The Sun: "I am worried about some recent cases where cautions have been given to criminals who have committed multiple crimes. The number of cautions given for sexual offences is going in the right direction - down - but we need this review to make sure they are only given where truly appropriate."