During a tour of the Museum of Orange Heritage, Mr Varadkar met with the leadership of the Order, including senior Orangemen from the Republic of Ireland.
He posed beside a sign commemorating the 1690 Battle of the Boyne and also paid respects at a memorial window dedicated to the 336 members of the organisation who were murdered during the course of the Troubles.
Mr Stevenson said: "We acknowledge this is a significant moment; as it is the first time a serving leader of the Republic of Ireland government has visited the headquarters of the Grand Orange Lodge of Ireland.
"As a cross-border organisation, we welcome the Taoiseach's direct engagement with our members based in the border counties of the Republic and, in so doing, recognising the longstanding cultural identity of the Orange family in the south.
"It is also important to acknowledge the importance of the Taoiseach paying his respects to those members of our institution, many of whom served in the security forces, who were murdered by terrorists.
"Such a gesture should not be underestimated and will, I believe, be deeply appreciated by many relatives of the deceased, and the Orange membership as a whole."
Irish premier Varadkar said the UK Government's proposals on a backstop position to avoid a hard border in Ireland "fall short" of what the EU require.
He said the concept of a deadline or time-limit on the customs arrangement was not acceptable, insisting any fall-back position to enable free movement over the frontier should only expire at such time that a broader trading deal between the EU and UK is struck.
"Just putting off a hard border for three years or four years or six years or 20 years isn't enough - it has to be permanent," Mr Varadkar said during his visit.
On Boris Johnson's unguarded comments in which he downplayed the significance of the Irish border issue, Mr Varadkar said the Foreign Secretary's assessment was not accurate.
He added: "When I want to know what the view of the British government is, I listen to the prime minister."
The Taoiseach kicked off a day of engagements in Belfast with a private meeting with Baroness Paisley, the widow of former DUP leader Ian Paisley, at a library dedicated to his memory in the east of the city.
Later on Friday he will launch the west Belfast Feile an Phobail at St Mary's University College.
Taoiseach Leo Varadkar has gone off script & is enjoying an extended tour of the museum at Schomberg House. MrVaradkar taking his time. pic.twitter.com/tnr0vMTOCo