The British Government is resisting devolving powers to the devolved regions over fears of what Sinn Fein might do in any future Northern Ireland Assembly.
Prime Minister Theresa May has always said Brexit will mean more powers for the Scottish, Welsh and Northern Ireland assemblies.
Scottish Brexit Secretary Mike Russell told the BBC Cabinet Secretary David Lidington had discussed the issue with him earlier this week.
"The reality of the situation is, that certainly David Lindington wanted to assert, was that the whole reason why devolution in Wales and Scotland had to be subverted was out of fear of what Sinn Fein might do in a legitimate assembly," he said.
"The issue here is people's consent and people's votes. If the UK government is so afraid of people voting and consenting, then they should give up democracy all together.
"If they aren't afraid of that, then they should stop making silly allegations and attempting silly smears."
The Scottish Parliament has so far refused to give its consent to the British Government's Brexit legislation as it believes powers should be taken from Brussels and given to Holyrood.
Mr Russell said he also had concerns about the DUP's influence on the British Government.
"We don't take part in Northern Irish politics, that is an issue for people in Northern Ireland, but it does seem that if the DUP is in a position to veto things happening in Westminster - to veto devolved settlements - then we're in a dangerous situation."
Meanwhile, the shadow Northern Ireland Secretary, Tony Lloyd, has said the the British Government must take a longer-term approach in its Brexit customs proposals to avoid a hard border in Ireland,.
Prime Minister Theresa May's cabinet signed off on a "backstop" proposal on Thursday that would see the UK match EU tariffs after 2020, if there is no deal on their preferred customs arrangements.
Mr Lloyd said: "If I was an investor looking to invest in my own constituency here in Rochdale, or for that matter anywhere across Northern Ireland, I'd want a bit more certainty about how long this is going to last.
"We've got to have some real fleshing out, because no hard border is not just for 2019, 2020, it's got to be into the long future."
He added: "We would oppose a hard border, because it is perfectly possible to negotiate with the EU round no hard border keeping the whole of the UK as one within this."