He begins by telling me about the ghosts living in his seven-bedroom farmhouse in the Kent countryside - once owned by the late playwright Noel Coward, who died in 1973 - which forms the backdrop to his latest novel, Briefs Encountered.
Shortly after moving into the property six years ago, spooky things started to happen.
"For about four nights in a row, at four o'clock in the morning, a different picture by the same artist fell off a different wall," he says.
"When something like that happens, you can't help but think, 'ooh, that's Noel saying he doesn't like my choice of art'!"
Clary says he's never been scared. In fact, he's been incredibly accepting of his other-worldly lodgers.
"You just have to assert yourself and make sure everyone knows you're the current owner and they're welcome to waft around.
"The house has a pagan feel to it and when things like sewage pipes burst, you do wonder whether it's the house expressing its disapproval.
"At the moment, I've got this terrible smell in the dining room and I think it's a dead rat under the floorboards, but you can read things into that if you choose to."
The novel is set in two time periods - the 1920s and present day. In the Twenties, the main character is Noel Coward, who buys a country house in Kent.
In the present day, a slightly washed-up actor, Richard Stent, buys the same home from a comedian called Julian Clary, who is eager to sell due to ghostly activity.
Indeed, Clary the writer mercilessly mocks the cameo Clary in the book, painting a picture of a totally selfish - rather ghastly character - with lots of barbs pointed directly at himself.
"It's a comedy device. I thought it was fun. If I'd said all those things about Christopher Biggins, the lawyers would have said, 'you can't do that' because of libel. So at last I've found someone I can be rude about... myself," he teases.
The ever youthful Clary (52) whose Nineties TV shows included All Rise For Julian Clary, has been less visible on screen in recent years, apart from a hilarious stint on Strictly Come Dancing in 2004 - in which he and his partner Erin Boag came third - and a string of guest appearances on panel shows.
A lot of the time he's been busy doing panto, however, Clary's first live tour in six years, Lord Of The Mince in 2009 and 2010 - covering the UK, Australia and New Zealand - was a sell-out.
Now another is planned for autumn, entitled Position Vacant: Apply Within, inspired by his long-term partner Ian's recent 18-month absence in Los Angeles.
Clary made the effort to visit Ian while he was working out there, but it wasn't a bed of roses.
"I did endure it for a few weeks, but I find Los Angeles a bit irritating. It's very materialistic, work-orientated and no one is very sincere... I could go on," he says.
His forthcoming tour is all about looking for love. He'll use a process of elimination, and a lot of audience participation, to select a husband for the night.
"The chosen one will be hauled off to my dressing room!" Clary says, smiling wickedly.
"The idea came about when I thought Ian wasn't coming back from America. I thought, 'right, I'll go and find a new husband!'
"Now, rather inconveniently, he's come back."
Has Clary considered tying the knot with Ian in a civil ceremony?
"Some days I think it's a nice idea and then other days it seems too conventional for my liking.
"We've talked about it. Ian thinks it would be a nice day out."
Their relationship also inspired Clary to create a similar long distance situation in the novel, between the washed-up actor and his boyfriend.
"Theirs is very much like my relationship with Ian. We've been together for about nine years.
"Ian works in London and when I'm there, we live together, but I can escape to Kent during the week if he's getting on my nerves.
"I'm sure I'm difficult to live with," Clary continues. "I'm quite self-absorbed and a bit eccentric. I'm not a prima donna and I don't think I'm argumentative. We don't have rows. We just go very quiet."
Though the ghouls haven't prompted him to move, Clary can't see himself living in the farmhouse forever.
Ideally he'd like a penthouse flat in central London, equipped with porters, he confides.
"It's quite tough living in the country. I do have gardeners and handymen but, even so, by lunchtime I'm worn out with all the sweeping, cleaning and wheelbarrowing."
Clary already has an idea for his next novel and for a TV venture, although getting things commissioned is a challenge.
Some might say that's because he blotted his copybook nearly 20 years ago, with his infamous remark at the 1993 British Comedy Awards - Clary joked that he'd been backstage carrying out an extreme sex act on the then-chancellor Norman Lamont.
Though the audience laughed at the time, Clary was slammed by the media and deemed too dangerous for live TV.
"There was the feeling I'd blown it and that my career had been derailed," he has previously said.
Today, Clary is looking forward to returning to the stage.
"I've missed touring," he says. "I love writing but you can't survive on what you get from it.
"If I spend six months alone, working on a book, then I'm desperate to go out, be sociable and the centre of attention."
No midlife crisis then?
"Not at all," Clary insists. "Turning 50 was a milestone for me.
"I love my 50s and one can't help but feel a sense of achievement once you reach them."