The tax system should be used to put second homes in the countryside beyond the reach of more families, according to the head of the Campaign to Protect Rural England (CPRE).
Sir Andrew Motion, the former Poet Laureate, said "townies in the countryside" were "gutting" rural communities, and blamed them for helping to create the housing shortage.
He warned that rural areas were under siege from planners who had more more freedom than ever to build on greenfield sites.
The CPRE president did not call for a ban on second home ownership, but told The Times: "I would increase taxes on second homes to make it very expensive.
"I think there's a question about whether second homes mean you have inert dormitory communities in the countryside through most of the week, very often lived in by people who scoot down in their cars, see their smart friends, don't join in the life of the community and don't feed into it.
"They're townies in the countryside, they make sure they're back in London in time to catch the 10 o'clock news on Sunday night. That means rural communities are gutted."
More than 165,000 people have a second home for weekends or holidays, according to the last census, with Cornwall having the greatest number, at 23,000, the newspaper noted.