Matfield said: "I asked him if he thought people would still support now England is out.
"He said that the people still love rugby and hoped they will give support to some other home nation teams."
Harry was speaking at a reception at Buckingham Palace for people taking part in this year's Rugby World Cup.
The Queen was there to welcome representatives from each of the 20 competing teams along with the Duke of Edinburgh and Princess Royal.
In a speech welcoming around 400 players, coaches, referees and volunteers to the Palace, Harry said it was a privilege to see the tournament being enjoyed by hundreds of millions of people around the world.
In his role as honorary president of England 2015 he said: "It has been one of the most competitive and entertaining World Cups ever.
"Although only half way through, we have witnessed some truly memorable moments, and one or two upsets!" he added, most likely referring to England's defeat against Wales which he attended with his brother.
Standing on a red carpet fringed with gold, upon which the Webb Ellis Trophy had pride of place, Harry said he was hopeful for the future of rugby.
"England may no longer be represented on the pitch, but as a nation we are incredibly proud to be hosting such a successful tournament.
"A showcase for rugby, which - I truly believe - will help grow the game through its excitement and increasing fan base."
The Queen was presented with a specially commissioned participation medal by Bernard Lapasset, chairman of world rugby, and William "Bill" Beaumont, former England captain.
Wearing a purple flower-patterned dress and a pearl necklace, she remarked that the medal looked very smart before greeting team members and volunteers from all over the world.
Kathryn Catto, a World Cup volunteer from Gloucester, chatted with the Queen as she mingled around the chandelier-decked ballroom.
She said that the monarch was concerned about whether the 17 players - who had stood on the stage behind Harry as he gave his speech - would now be allowed to come down and enjoy the party.
Ms Catto said: "She said 'I hope they will be able to come down soon' - she thought they might be a bit bored and asked a man to see if they could come down off the stage so they wouldn't miss out."
"She said it was lovely that people from all around the country were being represented here tonight."
Ms Catto, who was joined by fellow volunteers and friends Rhian Edwards and Patricia Bailey, joked that they had shared a giggle with the Queen after she revealed she was a great fan of the kilts sported on stage by Scotland's Finn Russell and Jonny Gray.
In response to her admiration Mr Russell said: "It is our traditional wear so it is great to hear that and it's good for Scotland."
Former England captain Beaumont echoed Harry's sentiments that rugby fans should continue to support the World Cup.
He said: "It is important we get behind the whole tournament.
"We have two or three weeks to deliver the best World Cup ever.
"People should still support England. Just because they are not in it - it's times like these we need more support."
To disillusioned England fans he left this message: "Get yourselves ready - we will come back."
COMMENT RULES: Comments that are judged to be defamatory, abusive or in bad taste are not acceptable and contributors who consistently fall below certain criteria will be permanently blacklisted. The moderator will not enter into debate with individual contributors and the moderator’s decision is final. It is Belfast Telegraph policy to close comments on court cases, tribunals and active legal investigations. We may also close comments on articles which are being targeted for abuse. Problems with commenting? firstname.lastname@example.org