But just as Albania looked set to register their first ever point at an international tournament, substitute Griezmann came to France's rescue with less than a minute to go.
Then in stoppage time West Ham star Payet, back on his old Marseille stomping ground, curled in his second goal in two matches to secure their place in the knockout stages.
All France had to show for a worryingly flat first-half performance were two Payet free-kicks, both headed over by Olivier Giroud, and a blocked shot from Anthony Martial after he was put through by Payet.
Albania predictably began by defending deep but, perhaps sensing some French nerves, they began to venture forward and Ermir Lenjani tried his luck from 30 yards but his effort flew high and wide.
Moments later full-back Elseid Hisaj went on the overlap and pinged in a low cross which Hugo Lloris had to gather as Armando Sadiku threatened to get a toe onto the ball.
Ten minutes before half-time Albania won a corner and Ledian Memushaj's low delivery caught the French by surprise, with Lenjani only denied a shock goal by a deflection.
Martial was sacrificed at the break with Paul Pogba, a surprise exclusion from the starting line-up, summoned from the bench.
France improved immediately - not that they could get much worse - and just 30 seconds into the second half Giroud swung in a cross and youngster Kingsley Coman glanced a header just past the post.
Yet they still had to survive another major scare in the 52nd minute when Memushaj beat Hugo Lloris but saw his effort come back off the foot of the post.
The ball rebounded to the floored striker but he knew little about it as it bounced off his head and narrowly wide.
Pogba had a chance to break the deadlock shortly after when he slid on to meet an inviting cross from Payet only to blaze over.
Giroud then headed wide before the Arsenal striker went closer still when he nodded Patrice Evra's cross against the Albanian woodwork.
A late onslaught saw Laurent Koscielny fluff a close-range header, N'Golo Kante denied by a deflection and Andre Pierre Gignac nod wide.
But with 90 minutes almost up Albania's rearguard finally slipped up, leaving Griezmann unmarked to head home Adil Rami's cross.
Then Payet repeated his trick against Romania on the opening night when he jinked into the box and curled home to secure a victory which was far more hard-earned than the scoreline would suggest.
COMMENT RULES: Comments that are judged to be defamatory, abusive or in bad taste are not acceptable and contributors who consistently fall below certain criteria will be permanently blacklisted. The moderator will not enter into debate with individual contributors and the moderator’s decision is final. It is Belfast Telegraph policy to close comments on court cases, tribunals and active legal investigations. We may also close comments on articles which are being targeted for abuse. Problems with commenting? firstname.lastname@example.org