When a young Louis Theroux was interviewed by Michael Moore for a job on his show TV Nation, friends who knew Moore briefed him on what to say. " Have you seen Roger & Me?" Moore asked, referring to his magnum opus at the time. Theroux knew to answer yes, even though he hadn't. " What did you think of it?" Moore continued. "I absolutely loved it. I have to say, I think it's my favourite film," replied Theroux. Needless to say, he got the job.
Flattery aside, Moore spotted in Theroux something the fledgling documentary-maker had not realised about himself. "I wanted to be this smooth network correspondent, but I couldn't pull that off," Theroux admits. "He must have got that it would be quite funny to take this slightly geeky British twentysomething and put him out of his depth in the American heartland."
Traces of that gauche youth remain. But at 37, Theroux is now an experienced documentary-maker, and has taken that initial concept of a buttoned-up Brit in unfamiliar surroundings and turned it into a genre all his own.
His latest film is set in California's notorious San Quentin prison, an institution made famous by Johnny Cash, who performed there live in 1969. The jail is an overcrowded 19th-century monstrosity, where thousands of prisoners spend 23 hours a day cooped up two to a cell. Looking at Theroux, the quintessential north London intellectual – he wears a beard and spectacles, was educated at Westminster School and at Magdalen College, Oxford, and lives in Harlesden with his partner, Nancy, and their son Albert, nearly two – it is difficult to imagine how he went down with some of California's most dangerous felons.
Given his reputation for humour, taking on the serious subject of prison was a risk. "I do love to have humorous moments in the programmes, but we took the leap of faith that by putting me in that situation we would have light moments as well as dark ones," he says.
Theroux's current obsession is finding out how human beings interact in a different environment. He found the prison to be a microcosm of society, albeit "an unnatural one, where a small number of people are keeping a large number of people at close quarters trying to prevent them from attacking or killing one another or from escaping".
He seems quite enchanted by the fact that many of the prisoners had formed gay marriages, "little domestic scenarios", even some of the ones who, outside of prison, were straight. The highlight of the project for Theroux was interviewing a former member of the notorious Nazi Low Riders gang who had been moved to the protection wing, where he met and fell in love with a Jewish man. "An ex-Nazi and a Jewish guy being lovey-dovey together: it's a strange relationship and it's a metaphor for how we all are in our lives – that we're thrown together, whether it's by prison walls or whatever physical environment you've grown up in," says Theroux.
At the other end of the social spectrum, he is working on a documentary about big-game hunting in South Africa, where rich Americans go for a guaranteed kill. Theroux investigates the idea of trophy hunting as a "leisure lifestyle pursuit". He filmed on game farms, large reserves of between 3,000 and 6,000 acres surrounded by fences. "You can go out there and say, 'I want a kudu, a zebra and a wildebeest and a baboon', and they'll have a price for each one and you'll be basically guaranteed getting it," says Theroux. But the documentary is not relentlessly anti-hunting. As Theroux points out, "People will happily eat chicken that's factory-raised and then they'll get up in arms about an animal that's been running around a large enclosure. As far as it knows, it's free and getting extra feed in the winter so it doesn't go hungry."
True to form, he makes acute observations about the people he films – in this case, the Afrikaners who run the reserves. "They've got Dutch heritage originally and they're country folk and they're quite dour, hard-bitten, with a very dry sense of humour, the opposite of Americans, and they're not natural TV characters. They all wear khaki shorts and khaki shirts and they've got thick legs and they have these schoolboy haircuts." He pauses then adds, "I don't want to caricature them too much, because I liked them."
Theroux's films are so presenter-led that it is easy to imagine that they are all his own work, but he dispels that myth by constantly referring to "we" – the team he works with at Grafton House, a BBC outpost on Euston Road, which is also home to the Arena and Storyville documentary strands. "Because I'm on screen and the shows are called Louis Theroux ...people who are not TV-literate assume that I do more than I do. The films are all directed by someone and they're series-produced and executive-produced and there's an editor. There's a sense in which the Louis Theroux you see on screen is a wittier, more composed character than the real one."
He defines what he does as journalism. In his 2005 book The Call of the Weird, he explored "the idea of to what extent there is an emotional betrayal involved in the act of journalism". In his early television career, Theroux felt the need to keep in touch with the people he made programmes about, but in writing the book he concluded: "There's a lot more transparency in the transaction of being a journalist than I realised to begin with. It's an artificial relationship, and you get to an intimate place very quickly if you're doing the job right. But when you leave there's no understanding that you will be continuing to call and see how they're doing."
Those he kept in touch with included a paranoid conspiracy theorist who appeared in the first episode of Weird Weekends and with whom he had enjoyed "hanging out in Idaho", as well as a male porn star who appeared in a subsequent episode. But he no longer maintains contact. "It's hard enough to keep up with your friends, especially if you've started a family," he says.
After leaving university in 1991, Theroux, who has dual British and US citizenship, decided to go to America, where his famous travel writer father, Paul, and brother Marcel were living. He "knocked around in Boston" for a few months, before getting a job on a newspaper in San Jose, California. From there, he joined Spy magazine in New York, just after Graydon Carter had left to edit Vanity Fair. Several of his Spy colleagues were working with Michael Moore on the pilot for TV Nation and when they heard that the BBC, which was co-funding the programme, was keen for Moore to find a British presenter, they recommended Theroux. The Moore show saw him meet the Ku Klux Klan and head into the Amazon in search of Avon ladies, although he finds looking back at this "juvenilia" a "chastening" experience.
When TV Nation ended, the BBC signed him to a development deal. He pitched the idea of taking the template from TV Nation but "in a longer form and with more warmth". Weird Weekends was born, a series in which Theroux investigated UFOs, porn stars, swingers, gangsta rappers and televangelists.
At the end of the second series, he and his team felt that it was time for a change. One day, Theroux recalls, he was having lunch with friends and it came up in conversation that all of them had written to Jimmy Savile as a child, asking him to "fix it" for them. In the first of a series of programmes entitled Louis Theroux Meets..., he spent two weeks living with Savile. The result was compelling television. Savile turned Theroux's questions back on him, taunting him by calling him "Mr On-the-Ropes", and demanding "next question, next question" – but out of it emerged a fascinating picture of an eccentric individual, a man who still preserves his mother's room exactly as it was before she died.
Theroux also spent time with Neil and Christine Hamilton, and secured a scoop when they were arrested halfway through filming over false accusations that they had been involved in an indecent assault of a young woman in Ilford, Essex. Other subjects included Paul Daniels, Ken Dodd and Max Clifford, who tried to turn the tables on Theroux by luring him to a lap-dancing club to meet Clifford's client Simon Cowell. The next day, Theroux's visit to the club was splashed across the Daily Mirror. "It was quite amazing to be on the other end of it, probably salutary," says Theroux. At the end of the programme, Clifford is overheard trying to plant stories with reporters, and Theroux confronts him. "The editor at the time said I was losing on points the whole way through but knocked him out in the last round," he says with a touch of glee.
The series upped Theroux's profile in the UK, one of the reasons why he now prefers to work in America, where he enjoys a far greater anonymity. "It was quite a stressful way of working, to make those one-on-ones. I haven't and wouldn't rule out doing one again, and I think the right candidate is out there," he says. "The way I like to work is to be invisible. I think a lot of people get into journalism for that reason. You're curious about how other people live. It doesn't feel that natural to talk about yourself."
One potential candidate for a return to the one-on-one is Heather Mills. Theroux first approached her the day her divorce from Sir Paul McCartney came through, which he admits was "perhaps not the most politic time". The week before our interview, he sent her a second request. "She'd be great. I really do think it could do her a lot of favours. Somewhere along the line, her subjectivity has been lost. She's had the full tabloid experience. She's ripe for a more nuanced appraisal."
Theroux's contract with the BBC has just over a year to run. He is considering "striking out and being in the independent sector, having a stake in the programmes – you get a bit more freedom, a bit more control". He is also keen to have a behind-the-scenes role in programmes.
But above all, he believes in "trusting to fate a bit. I've never really planned my career. I've always just drifted from one opportunity to the next."
The Louis Theroux Collection, published by 2entertain, is available now on DVD