South Korea said a North Korean patrol boat entered its waters around their disputed maritime border but backed off after nearly an hour following repeated warnings.
Meanwhile, a senior American diplomat cautioned Pyongyang that its bad behaviour would no longer be rewarded.
The naval standoff came amid concerns that the North might try to provoke an armed clash in the area - the scene of deadly naval skirmishes in 1999 and 2002 - to stoke tensions that were already running high after Pyongyang's nuclear test and a barrage of missile launches last week.
The regime has also conducted amphibious assault exercises near the sea boundary and appeared to be preparing for more missile tests, including one believed capable of reaching the US.
The missile is being readied at the new Dongchang-ni launch site on the North's west coast near China.
New commercial satellite images show the facility is ready for use after nearly a decade of construction.
The launch tower and what appears to be construction materials on the launch pad are seen in the images, said Tim Brown, a senior fellow with GlobalSecurity.org.
He speculated the debris may be there to make the pad appear as though it is still under construction.
"The launch pad appears to be operational," Mr Brown said.
The maritime intrusion occurred as US Deputy Secretary of State James Steinberg was in Seoul to coordinate a united response to Pyongyang's belligerence.
South Korean news reports said a delegation of senior US officials was working on financial sanctions against Pyongyang.
Mr Steinberg told South Korean President Lee Myung-bak that "North Korea would be mistaken if it thinks it can make provocations and then get what it wants through negotiation as it did in the past. The US won't repeat the same mistake again", Seoul's presidential office said in a statement.
Mr Steinberg left for China today.