The PM later said there were “a lot of morons,” so he would not comment on the incident