It is the fifth draw from eight games for O'Neill while Northern Ireland remain without a win since beating the Faroe Islands in August 2011.
Danny Lafferty did have the ball in the net in the 18th minute but it was ruled out for pushing and the visitors were grateful to goalkeeper Alan Mannus for two very good saves to deny former Coventry striker Michael Mifsud.
Northern Ireland were almost behind in the fifth minute when Andrew Cohen crossed for Andre Schembri, who should have scored but shot straight at Mannus from eight yards. Will Grigg fluffed his lines when his first big chance came along in the 17th minute, a heavy touch allowing keeper Andrew Hogg to block.
Moments later Northern Ireland had the ball in the net when McAuley headed in a Chris Brunt corner but Bulgarian referee Nikolay Yordanov disallowed it for pushing. Mifsud was proving a handful for the visiting defence and his trickery created another opening for Schembri, whose drive beat Mannus but flew just past the angle of post and bar.
Clayton Failla was the next to threaten Mannus' goal with a fierce free-kick from 30 yards that was a yard over the bar. Wingers Niall McGinn and Shane Ferguson were having little joy down the flanks but in the 36th minute McGinn made progress down the right and Steven Davis crossed for Grigg to glance a header wide.
O'Neill made one change at half-time, Craig Cathcart replacing Jonny Evans in the centre of defence. The Ta'qali has traditionally been a good venue for Northern Ireland, with 2005's 1-1 draw between the sides the only one of six previous meetings they did not win.
The visitors almost caught Malta cold as McGinn broke through in the middle but Hogg was off his line quickly to punch clear, while Aaron Hughes then did well to get to the byline on the right and pull the ball back but McGinn scuffed his shot.
In the 56th minute Mannus was called into serious action for the first time when Mifsud unleashed a stinging shot from the right, the keeper palming it away. In the 63rd minute Inverness striker McKay was introduced, winning his first cap when he replaced Ferguson on the right, with Grigg remaining the lone striker.
The visitors were pressing but in the 70th minute they were indebted to Mannus for an excellent stop to keep out Mifsud once again. Pat McCourt and Josh Magennis were introduced late on but there was no way through the Malta defence and the visitors had to settle for a disappointing draw.