The DUP leader made the comment as she addressed her party's supporters in Newry and Armagh DUP Association on Friday evening.
Earlier on Friday, alleged former IRA leader Thomas 'Slab' Murphy was sentenced to 18 months in prison for tax evasion.
The 66-year-old bachelor farmer, who denied the charges, owes the Irish exchequer almost 190,000 euro (£147,000) for tax dodging from 1996-2004.
Speaking at a meeting of the DUP's Newry and Armagh Association on Friday evening, Arlene Foster said the people of that area "know him to be a criminal".
"Whilst some people refer to Murphy as a ‘good republican’ the people of this area know him to be a criminal," she said.
"Setting aside the length of the sentence, this man who evaded prosecution for so long, has at long last been put behind bars.
"Most people will celebrate seeing justice done. If you break our laws, regardless of who you are, expect to go to jail."
Murphy, from Ballybinaby, Hackballscross, Co Louth, on the border with Northern Ireland, was sentenced at the high security Special Criminal Court in Dublin as Ireland votes in a general election.
None of the jail term was suspended and he received no fines.
Dressed in a pink shirt, brown jacket and slacks, Murhpy showed little emotion in the dock as the sentence was delivered.
He acknowledged some family members and friends as he was led out of a side door of the court.