Hundreds of impassioned demonstrators - all waving American flags, but separated into two groups by police - have descended on the proposed mosque near ground zero.
Opponents of the plans chanted "No mosque, no way!" and supporters shouted, "We say 'no' to racist fear!"
The two leaders of the construction project, meanwhile, defended their plans, though one suggested organisers might eventually be willing to discuss an alternative site. The other, Imam Feisal Abdul Rauf, said during a Middle East trip that the attention generated by the project is actually positive and he hopes it will bring greater understanding.
The rallies took place around the corner from the cordoned-off old building that is to become a 13-story Islamic community centre and mosque. There were no reports of physical clashes but there were some nose-to-nose confrontations, including a man and a woman screaming at each other across a barricade under a steady rain.
Opponents of the 100 million dollar (£64.1 million) project two blocks from the World Trade Centre site appeared to outnumber supporters. Bruce Springsteen's Born In The USA blared over loudspeakers as mosque opponents chanted.
Steve Ayling, a 40-year-old Brooklyn plumber, said the people behind the mosque project are "the same people who took down the twin towers".
Opponents demand that the mosque be moved farther from the site where more than 2,700 people were killed on September 11 2001. "They should put it in the Middle East," Mr Ayling said.
New York Mayor Michael Bloomberg has fiercely defended plans for the proposed mosque, saying that the right "to practice your religion was one of the real reasons America was founded".
The mosque project is being led by Mr Rauf and his wife, Daisy Khan, who insist the centre will promote moderate Islam.
The dispute has sparked a national debate on religious freedom and American values and is becoming an issue on the campaign trail ahead of the midterm elections. Republicans have been critical of President Barack Obama's stance - he has said the Muslims have the right to build the centre at the site but has not commented on whether he thinks they should.