Deputy Commons Speaker Nigel Evans has denied accusations of raping and assaulting two young men, saying the allegations were "completely false".
The 55-year-old Tory MP described his arrest on Saturday at his home in the village of Pendleton, Lancashire, and subsequent questioning by police as "the worst 24 hours of my life".
He made clear through his solicitor that he intended to carry on as Deputy Speaker while police continued their investigation. Speaker John Bercow has however agreed that he should be excused his duties chairing the Queen's Speech debate which opens on Wednesday and continues to Wednesday the following week.
Following his release on bail, Mr Evans emerged in the beer garden of the village pub next to his cottage to deliver a brief statement expressing his "incredulity" at what had happened.
He said: "I was interviewed by the police concerning two complaints, one of which dates back four years, made by two people who are well known to each other and until yesterday, I regarded as friends. The complaints are completely false and I cannot understand why they have been made, especially as I have continued to socialise with one as recently as last week.
"I appreciate the way the police have handled this in such a sensitive manner and I would like to thank my colleagues, friends and members of the public who have expressed their support and, like me, a sense of incredulity at these events."
Lancashire Police said Mr Evans, who has been MP for the Ribble Valley constituency since 1992, had been arrested on suspicion of raping one man and sexually assaulting another between July 2009 and March 2013. Following several hours of questioning by detectives at Preston Police Station, he was released on bail to June 19.
Defence Secretary Philip Hammond said of his Deputy Speaker role: "I stick rigidly to the view that we should treat people as innocent until they are proven guilty but it is quite difficult to carry out a sensitive and high profile role while being under this kind of scrutiny."
Foreign Secretary William Hague described Mr Evans, who came out as gay in 2010, as a "long-standing friend" and said that MPs of all parties would be "very sorry to see this situation". Tory MP Andrew Bridgen told Sky News: "I personally think that Nigel should be able to continue as Deputy Speaker while the police, quite rightly, carry on with their investigations."
A statement on the Ribble Valley Conservatives Association website said Mr Evans was "widely liked and respected" and they expected him to continue with his duties in the constituency as normal.