Louis Walsh has said that Simon Cowell sacking him because of ageism was a "big mistake".
The Irish judge, 61, who has announced that he will be quitting the talent show after this year's series, was axed in 2007 but quickly reinstated.
He told the Radio Times that his return to the ITV show - and the moment pop star Williams gave him a lap dance last year - were two of his on-screen highlights.
"Big mistake. I was gutted," Louis said of the sacking. "I couldn't believe I was off the show - me and (presenter) Kate Thornton. I was sure I was safe."
The star, who has previously blamed ageism, said: "Somebody in ITV got into Cowell's ear, and told him to make it younger and fresher. It was an age thing."
Louis was replaced by US dancer and choreographer Brian Friedman before he was asked to return.
He told the magazine: "They gave Brian Friedman the job, who was brilliant as a choreographer, but he wears these silly clothes all the time. (Judge) Sharon (Osbourne) sent me a text on day one saying, 'It's not working. You could get your job back.'
"Then I had to wait. Simon rang me and said: 'There is a possibility that you could come back,' but then he said, 'You cannot tell anybody.' You can't tell me not to tell anybody. I like to tell!"
He said his other X Factor highlights were Sharon drenching him with water in 2005, the success of boyband JLS, twins Jedward singing a Britney Spears number as well as getting a lap dance from Robbie when he performed his song Candy on the X Factor results show.
He said of the incident with Sharon: "I said, 'Are you taking Ozzy's drugs?' or something, and she went crazy. It was a terrible thing to do. A live show, all these wires and electricity, and water?! I could've been electrocuted! But that's Sharon. She's good TV. You never know what she's going to say or do."