Barack Obama and the prime ministers of Japan and Australia have committed to deepen their military co-operation and work together on strengthening maritime security in the Asia Pacific.
The meeting, the first since 2007 among leaders of the three allies, risked antagonising China after a week when the US president reached a surprising level of consensus with President Xi Jinping on climate change and trade, and Japan and China took steps to improve their relationship.
China has viewed Mr Obama's efforts to deepen alliances with other countries in the region, particularly on security issues, as an attempt to counter Beijing's rise.
In a joint statement following the meeting, the three leaders said they had agreed to "deepen the already strong security and defence co-operation" between the countries. They also agreed to work on boosting "maritime security capacity building" in a region rife with disputes between China and its neighbours over claims to waters and islands.
However, there were no announcements on specific military exercises or new troop deployments within the region.
White House officials insisted that the three-way talks on the sidelines of the G20 summit were not meant to send a message to China, but in advance of Mr Obama's meeting with Australian Prime Minister Tony Abbott and Japan's Shinzo Abe, the president pressed China to "adhere to the same rules as other nations - whether in trade or on the seas".
"By virtue of its size and its remarkable growth, China will inevitably play a critical role in the future of this region," Mr Obama said in a speech at the University of Queensland. "And the question is, what kind of role will it play?"
An Obama administration official said the three-way meeting had been in the works for a year. Beyond military co-operation, the leaders also discussed the US-led campaign to defeat Islamic State militants in the Middle East, combat the spread of the deadly Ebola virus in west Africa and stop Russia's destabilising actions in Ukraine.
The conflict between Russia and Ukraine has been of particular concern in Australia, which lost 38 citizens when Moscow-backed rebels downed a commercial airliner earlier this year.
Australia was Mr Obama's last stop on a week-long trip that began in China and Burma. He arrived politically weakened at home by the Democratic Party's election defeats on November 4. The president has tried to show the region's leaders that he retains the ability to deliver on promises to deepen US engagement in Asia and the Pacific, an effort he sees as a central part of his foreign policy.
"There are times when people have been sceptical of this rebalance, they're wondering whether America has the staying power to sustain it," Mr Obama said. "I'm here to say that American leadership in the Asia Pacific will always be a fundamental focus of my foreign policy."
In China, he emerged with an ambitious agreement to reduce greenhouse gas emissions, as well as a deal to extend visa lengths and move forward on tariff reductions on hi-tech goods.
But despite those areas of agreement, there are major sources of tension between the powers, particularly Beijing's provocative actions in territorial disputes in waters off its borders.
The conflict between China and Japan over a string of uninhabited islands in the East China Sea has been intense, with Beijing setting up an air defence zone over the area last year that was denounced by the US.
During a trip to Tokyo this year, Mr Obama pledged to fulfil America's treaty obligations to defend Japan if it were to come under attack from China as part of the conflict over the islands.
In recent days, China and Japan have acted to try to ease tensions. Mr Xi and Mr Abe held their first ever meeting in Beijing this week, a discussion the Japanese leader called a "first step" toward reconciling.
Australia, the G20 host, has played an important role in Mr Obama's efforts to bolster the US military presence in the region to be a counterweight to China. During a trip to Australia in 2011, Mr Obama announced a plan to rotate 2,500 US Marines through a military base in the northern city of Darwin.