A "retreat" of white Britons from areas where minorities live is limiting cultural integration, according to research by think tank Demos.
Analysis of Census 2011 figures show that 45% of ethnic minorities in England and Wales live in areas where less than half the population is white British, while 41% live in wards that are less than half white.
Trevor Phillips, former chairman of the Equalities Commission and Demos Associate, said white Britons choosing not to live in minority-dominated areas "ought to make us a little anxious".
He said: "This very interesting piece of research reveals a number of vital findings about how people in England and Wales are living together. First, it shows a kind of 'Ambridge effect' - a welcome minority advance into areas previously only the preserve of the white majority.
"It also demonstrates a greater degree of ethnic mixing within cities, although unfortunately this appears to be mostly between minorities. What ought to make us a little anxious is the 'majority retreat' it has unearthed - white people leaving minority-led areas and not being replaced - which isn't good news for the cause of integration."
The 4.1 million ethnic minorities who now live in white minority areas is a significant increase from the 2001 Census, when only around one million minority Britons lived in such wards.
A statement from Demos said: "Demos researchers attribute this mainly to white British people choosing not to move to minority-dominated areas, in what Trevor Phillips has described as 'majority retreat'.
"In these areas, departing white British are replaced by immigration or by the natural growth of the minority population. Over time, the end result of this process is a spiral of white British demographic decline."
But integration between ethnic minorities is now more common than it was during the last census in 2001. Demos said minority white areas were generally multi-minority, since new British minorities such as Somalis have taken up housing vacated by established minorities, such as Afro-Caribbeans.
Figures also show that more ethnic minorities are moving to live in white-dominated rural parts of the country, with fewer than 800 wards which are more than 98% white compared with more than 5,000 in 2001.