It put the home team at a distinct disadvantage from the outset and prompted fury from the passionate Ravenhill crowd.
While many fans concede Payne's move deserved a card – they believe it should have been yellow instead of the game-changing red.
The Heineken Cup quarter-final was an uncompromising contest, ending with Saracens winning 17-15. The loss was agonising for Ulster fans as the team failed to move to the last four following the controversial decision by the Frenchman.
Ulster had to play most of the game with 14 men after Payne was sent off for his dangerous tackle on Alex Goode, just four minutes in.
Payne's opposite number required several minutes of treatment on the ground after landing on his head and being knocked unconscious. But the red card decision outraged the 18,000-strong crowd as television replays showed the Ulster player had no malicious intent.
Payne remonstrated with Garces to no avail, and afterwards even Saracens director of rugby Mark McCall admitted a yellow card might have sufficed.
With the outcome of the game coming down to a marginal two-point difference, Ulster fans felt their heroes would have been victorious if playing with a full team. Following the game dejected supporters took to Twitter to vent their anger at the referee and to praise their team's efforts.
One man wrote: "I felt gutted and could have murdered that referee! That said, a great night!"
Another fan wrote: "Still absolutely gutted about yesterday. Great atmosphere at Ravenhill! Hugely proud of @UlsterRugby! #SUFTUM."
"Gutted" seemed to be the general feeling among the Ravenhill crowd and the viewing public who felt robbed. "Proud of the @UlsterRugby boys just gutted a referee decision led to us being put of the @Heineken_UK cup #SUFTUM," one fan said.
"So unlucky tonight, referee killed the game before it even started @UlsterRugby #SUFTUM," another added.
"Outstanding show from the 14 @UlsterRugby players on the field for 75 minutes. Less so from the French referee!," another fan said.
BBC Northern Ireland news presenter Donna Traynor summed up the general consensus nicely with her online comments.
"What a mammoth effort by @UlsterRugby team," she wrote.
"Bruised and broken but winners in the eyes of the fans."