Lord Kilclooney has said his tweet referring to Taoiseach Leo Varadkar as a "typical Indian" was a mistake and he has since removed it.
The peer sparked a racism row when he made the remark online after the Taoiseach's visit to Northern Ireland on Monday.
He was reacting to a BBC report that DUP MP Sir Jeffrey Donaldson said Mr Varadkar's visit showed "disrespect" and was "outside normal protocol" as local politicians had not been informed about it beforehand.
However, the NIO said the visit had followed all protocols.
After facing a barrage of criticism for once again referring to Mr Varadkar's race, the peer said he was "certainly no racist" and was pointing out how Mr Varadkar had upset unionists more than his predecessors had.
At the time he said he did not intend to withdraw the tweet.
However, he yesterday told the BBC he could "understand people reaching the conclusion the tweet was racist".
"And that's why I removed it. I am certainly not racist at all," he said.
He did not apologise but admitted it was a mistake.
Last November he referred to Mr Varadkar as "the Indian". He claimed he used the term as shorthand for the Taoiseach as he "couldn't spell his name" and given Twitter's restrictive character limit.