The "misogyny and patriarchy" in some communities is widening inequality, the author of a Government-commissioned review has warned.
Dame Louise Casey said the country is becoming more divided as it becomes more diverse and highlighted that in some communities women were the subject of "abuse and unequal treatment of women enacted in the name of cultural or religious values".
Communities Secretary Sajid Javid said he would be studying the findings closely but the Ramadhan Foundation condemned the "inflammatory" report.
Dame Louise said the UK could "no longer duck difficult issues" and highlighted the problems some areas had faced because of the pace and scale of immigration.
In her report she acknowledged that elements would be "hard to read", particularly for Muslim communities which already felt under pressure, but she said the country had to face up to "uncomfortable" problems.
The review recommended that schoolchildren should be taught "British values" of tolerance, democracy and respect to help bind communities together amid growing "ethnic segregation".
It also suggested that ministers should consider whether immigrants intending to settle in Britain should swear an "integration oath" on arrival.
It highlighted the plight of women who found themselves marginalised through poor English language skills while being subjected to "coercive control, violence and criminal acts of abuse".
Dame Louise told the BBC Radio 4 Today programme: "I, only last Thursday, was in a community where women who have lived here for years are not allowed out of their house without their men's permission."
She added: "Inequality within certain communities in these highly segregated areas is getting worse, not better."
Dame Louise said she was "tired" of the debate around Muslim women wearing the veil, saying people should be allowed to dress how they wanted.
But she added: "Do I think police officers or midwives should be fully veiled? No, I don't. I want to see their faces - most of us do."
Dame Louise said: "At the end of the day it is not the women in those communities that I have a problem with, it is the men in those communities. It is the misogyny and the patriarchy that has to come to an end.
"Leaders that are not Muslim and are Muslim need to unite around unity in this country. No matter who you are, no matter what creed or colour you are, equality rules."
The review called for more English classes for isolated groups, greater mixing among young people through activities such as sport, and a new oath for holders of public office
The review was originally commissioned by then prime minister David Cameron in 2015 as part of a wider strategy to tackle the "poison" of Islamic extremism.
It found that while Britain has benefited hugely from immigration and the increased ethnic and religious diversity it had brought, there had not been sufficient emphasis on integration.
It called on the Government to back a new programme to strengthen cohesion through promotion of the English language, raising employment levels among the most marginalised groups and "emancipating" women trapped in social isolation.
It highlighted the "huge national, cultural and symbolic value" of British citizenship, urging the promotion of British laws, history and values within the core school curriculum to build "integration, tolerance, citizenship and resilience" in children.
Mr Javid said the UK was home to many cultures and communities " but all of us have to be part of one society - British society".
"So while it's right that we celebrate the positive contribution that diverse groups make to British life, we also need to continue making sure that nobody is excluded from it or left behind," he said.
"To do that, we need to take a serious look at the facts and must not shy away from the challenges we face."
Mohammed Shafiq, chief executive of the Ramadhan Foundation, condemned the report as " inflammatory, divisive, pandering to the agenda of the far right".
"We are saddened that once again British Muslims have become a political football which is bashed from time to time without any regard for the impact this has on individuals who then are subjected to threats and violence," he said.
Harun Khan, secretary general of the Muslim Council of Britain, said: "Any initiative that facilitates better integration of all Britons should be welcomed, and we certainly endorse the few, fair and supportable suggestions proposed by the Casey review.
"This includes the promotion of the English language, sharing of best practice across the nation and a range of measures to tackle exclusion, inequality and segregation in school placements.
"And while we agree that forced marriages, female genital mutilation, honour-based killings and other practices have no place in modern Britain, we would argue that our faith tradition can be deployed to tackle what are essentially cultural practices.
"I hope we can facilitate robust and active conversations in British Muslim communities where we are frank about the challenges facing us and creative enough to meet them head on.
"Sadly, however, I fear that this report could be a missed opportunity. We need to improve integration, and it needs to involve the active participation of all Britons, not just Muslims. As former prime minister David Cameron has stated, 'integration is a two-way street'.
"The report has little discussion on white flight, and could have delved deeper into the economic structural barriers to integration."
Dame Louise insisted her report was designed to kick-start work and dialogue that will help tackle far-right and Islamic extremism.
She told LBC radio: "I think if we don't grab this and say let's have a dialogue, let's work out how we do this, t hen you do give way to the extremists, the far-right, in some areas of the country.
"This is a gift to them and every time these things happen, every time there's some political correctness on the front page of the Daily Mail, you get the far-right loving it.
"It's appalling if they're using it to basically foment racist nastiness and on the other hand you've got Islamic extremists saying nobody has a chance in this country if you're a Muslim. Both those extremes are patently wrong."
Speaking en-route to Bahrain for an official visit, Prime Minister Theresa May said: "The Government commissioned the Casey review... and it is a significant piece of work, there are a lot of recommendations and we will have to look very closely at what she has said and what her recommendations are."
Asked if she had seen the report before it was published, Mrs May said: "I haven't read the review before it was published, no."