In a dramatic night of results, Tory leader Ruth Davidson secured a shock victory in Edinburgh Central as her party won 31 MSPs - by far its best result ever at the Scottish Parliament and up from 15 in 2011.
Ms Sturgeon's party won 63 of the 129 seats that were up for grabs, failing to secure a second overall majority.
The SNP ended up with six fewer MSPs than the record 69 the party won under Alex Salmond in the 2011 landslide.
Labour endured another disappointing result, with the party securing 24 seats - a drop of 13 from the party's total five years.
Throughout the election campaign, the polls had predicted the SNP would win another overall majority at Holyrood.
However, this failed to materialise, a result which could see the Tories put pressure on the nationalists to scrap their controversial named person scheme.
The Conservative campaign had focused on providing a strong opposition to the SNP and Ms Davidson accepted "many people that have given us their vote for the very first time, not because they're true blue Conservatives, but because there's a job of work they want us to do".
She hailed her victory in Edinburgh Central as an ''incredible result'' and added: ''I hope the message that was resonating was of being a strong opposition, to hold the SNP to account, to saying no to a second independence referendum, to respect the decision that our country made, and to really focus on the things we're paying a government to focus on, on schools, on hospitals, on public services. That's what people want.''
Scottish Labour leader Kezia Dugdale said she was "heartbroken" the Conservatives had overtaken her party - which once dominated Scottish politics - to claim second place in the election.
Last May the party lost all but one of their MPs from Scotland as the SNP swept the board in the general election.
Ms Dugdale said after that result "this election was always going to be tough for the Scottish Labour Party".
She insisted she would remain as leader adding: "I am proud that our campaign rose to the challenge of offering an alternative vision of what could be done in our new, more powerful parliament."
With the constitution now the dominant issue in Scottish politics, she said her "determination to try to move the Scottish debate on" from the arguments of the 2014 independence referendum had cost Labour votes.
She backed income tax rises for not just the wealthiest Scots but basic rate taxpayers, saying this was needed to prevent cuts in public services such as schools and the NHS.
Ms Dugdale added: "There's no doubt that our defeat for the Labour Party is painful but it is not the end of our campaign.
"We will continue to argue for Labour values, Labour ideas and Labour principles.
"The work to renew the Scottish Labour Party so it is fit to serve the people of Scotland continues."
In Wales, Ukip has won its first seats in the Welsh Assembly - while Labour remains on course to retain power in the Senedd for a fifth successive term.
Labour raced ahead of other parties early on in the contest, despite an overall drop in its share of the vote compared with last time.
It has so far won 26 out of 39 declared constituencies. To claim an overall majority it would need 31 seats - although it could still govern via a minority administration, or leading a coalition with the number of AMs as it stands.
However, while Labour's dominance has not come as a surprise to pollsters, the 2016 election in Wales has produced two surprises.
Plaid Cymru leader Leanne Wood rocked Labour by winning the Rhondda seat from government minister Leighton Andrews.
Nigel Farage's Ukip made history after winning their first place in the Senedd - despite the party being opposed to the Assembly's existence until 2013.
Mr Farage told Sky News: "Many traditional Labour voters look at Jeremy Corbyn and look at the Labour party and see that it's gone way to the left - and doesn't look very patriotic.
"It completely refuses to engage with issues such as immigration.
"Unless the Labour party changes course then Ukip will continue to go on growing at the expense of Labour."
While Labour has left its rivals behind, the contest for second place between Plaid and the Conservatives has been neck and neck.
However, Plaid can lay claim to pulling off the biggest scalp of the night after Ms Wood's gamble to go head-to-head against cabinet minister Leighton Andrews paid off.
The former probation officer took 50.6% of the vote as well as causing a massive 21.1% swing in the process.
She said: "To win here, in the Rhondda, to win where you've lived all your life and grown up, is a real honour.
"A new dawn is about to break in Wales. People have voted for change."
English local elections
Labour performed better in local elections. With results available from 78 out of 124 English councils,
Labour has 40 councils, down one, and 768 seats, down 7. The Tories have 19 councils and 467 seats, up 9. The Lib Dems have two councils, 173 seats, up four.
UKIP has 28 seats, up 21.
The Greens have 11 seats, down three.
Labour has held on to two safe seats in parliamentary by-elections.
The Sheffield Brightside and Hillsborough vote triggered by the death of Harry Harpham was won by the former miner's widow Gill Furniss with a majority of 9,590. Chris Elmore won a majority of 8,575 in the Ogmore constituency vacated by former minister Huw Irranca-Davies when he stepped down to fight for a seat in the Welsh Assembly.
Counting for the mayoral races in London are underway and results are expected later on Friday.
Corbyn under fire
David Cameron said the Conservatives had held councils right across the country, won seats in England, and taken control of the city council in Peterborough for the first time.
"The Labour Party have lost touch with the hard-working people they are supposed to represent," he said.
"They are so obsessed with their left-wing causes and unworkable economic policies, they've forgotten that people want jobs, people want livelihoods, people want lower taxes, people want homes they live in and can afford to own - the things that we are now delivering.
"What these elections show is where we are a united, mainstream, One Nation, compassionate Conservative Party we can win, we can serve our country and communities and we can deliver the things that we believe in."
Mr Cameron hailed the "remarkable" result in Scotland which saw the Tories beat Labour, relegating the party to third place in the Holyrood parliament.
"Frankly, it is something of a realignment in Scottish politics," he said.
"If someone had said to me six years ago when I became Prime Minister - or even a couple of years ago - if someone had come into my office and said 'Prime Minister, pretty soon the Conservatives are going to be the second biggest party in Scotland', I would have told them to go away, lie down, to stop taking whatever they were taking, and come back and tell me what they really think, but that is what has happened.
"It's a great day for our party to see us recovering like this in Scotland, and I think it could be a realignment taking place north of the border, all to the good of those of us who want to see a strong and successful United Kingdom."
Labour party leader Jeremy Corbyn insisted the party had defied expectations in England's council elections.
He said: "All across England last night we were getting predictions that we were going to lose councils. We didn't.
"We hung on and we grew support in a lot of places."
Mr Corbyn added: "I want to send this message to our party in Scotland - well done on the campaign you fought, well done on the determination you have shown.
"There is a lot of building to do in Scotland. We are going to be with you. We are going to walk hand in hand with the party in Scotland to build that support up once again so that the Labour tradition in Scotland will be established once again.
"I am sure I can send a message on behalf of everybody here to our colleagues in Scotland - we are with you."
Mr Corbyn praised Gill Furniss for building on the support her late husband Harry Harpham had secured in Sheffield Brightside and Hillsborough as she replaced him in the seat.
He added: "There is a choice. Either you continue this path of cuts, closures and inequality or you collect the uncollected tax and you pay for the services that everybody else needs.
"That is what we understand - I don't think that they (Conservatives) do. This is the Labour way."