The flags protest and related violence had a negative impact on community relations, new research showed.
Less than half of people questioned for a major university survey thought relations between Protestants and Catholics were better now than five years ago.
Hundreds of people were convicted following loyalist demonstrations against the decision of Belfast City Council in December 2012 to restrict the number of days on which the union flag is flown at the city hall.
Grainne Kelly, a researcher, said: "The findings are concerning for community relations and demonstrate that the recent unrest has had a negative impact far beyond people's attitudes to, and tolerance of, markers of identity for each community.
"In the context of the forthcoming introduction of 11 new super councils with increased decision making powers and the 2016 Northern Ireland Assembly elections it remains to be seen how these polarised positions can be reconciled."
The information was contained in the Life and Times Survey, which is carried out annually by the University of Ulster and Queen's University Belfast.
Last year's research found only 45% of those surveyed thought relations between Protestants and Catholics were better now than five years ago. This has fallen from 62% recorded by the survey in 2010.
The research said: "The survey also found evidence of increasing annoyance within both Catholic and Protestant communities regarding displays of identity such as flags, kerb painting and murals."
Opinion on the flying of the Union flag on public buildings showed the Protestant community marginally favoured designated days like royal birthdays only (48%), although 44% believe the emblem should be hoisted year round.
The report added: "Although this is a minority view it runs consistently throughout the Protestant community and for those under 35 years rises to 61%."
The document said 59% of Catholics believed the Union flag should be displayed only on designated days from public buildings while just over a quarter felt it should not be used on the buildings at all.
Catholics and Protestants reported more murals, kerb painting and flags on display than five years ago - the report suggested this indicated increasing levels of annoyance involving these markers of identity.
A total of 1,210 people across Northern Ireland took part in the 2013 survey.
Following Belfast City Council's flags decision loyalists took to the streets to protest, some demonstrations involving violence during which many police officers were injured.
Major flags protests took place on Saturdays from December 2012 and into the first months of the year, adding to what were already difficult economic conditions on the high street and souring the political atmosphere.