"It would be a serious miscalculation and mistake if North Korea worked to engage in a nuclear test," said Glyn T. Davies, the U.S. special representative on North Korea policy.
"If there is a further provocation such as a nuclear test, there will be swift and sure reaction by the international community."
Davies spoke to reporters in Seoul after meeting with his counterparts to discuss an analysis of North Korea's possible future decisions, The New York Times reported.
The diplomats vowed "united and coordinated actions and responses" against further provocations, Shinsuke Sugiyama of Japan said.
An April 13 rocket launch by North Korea in defiance of the United States and its allies "sent a signal that they cannot be trusted to follow through on their own undertakings and promises," Davies said.
The rocket disintegrated soon after launch, but the U.N. Security Council condemned it as a test of long-range missile technology that violated other resolutions and ordered tightened sanctions.
A different future could be in store for North Korea if it begins to back away from its missile and nuclear weapons programs, Davies said.
"That engagement aspect remains open," Davies said. "Words are no longer, quite frankly, interesting to us. What we want to see is actions from North Korea."
South Korean envoy Lim Sung-nam also called on North Korea's new leadership to take a "different path" toward peace and move away from its provocative course, the Yonhap News Agency said.
"In response to the failed launch, the international community, including the three of us, as well as China and Russia, has been able to send a clear and consistent message to Pyongyang that North Korea should refrain from any further provocations," Lim said. "There will be a different path for them if they make the right decision."
Davies will travel to Beijing Tuesday to meet with Wu Dawei, China's lead nuclear envoy on North Korea, the Times said.
Recent satellite imagery indicated a resumption of activity at North Korea's nuclear test site, prompting analysts to warn the country could be making preparations for its third underground nuclear test.