Authorities from South Sudan and Sudan began formal talks last week in Addis Ababa to discuss issues left over from the deal that brought independence to the south in July.
South Sudan's Cabinet Affairs Minister Deng Alor said Khartoum didn't want international arbitration so talks broke down in Ethiopia.
"We can agree if there's commitment by the north to international arbitration," he was quoted by Bloomberg News as saying. "They said no."
Parts of the border, including the disputed territory of Abyei, are poorly defined. The countries came close to war this year over conflicts regarding disputed oil fields along the border.
Alor said both sides might have to consider a border demarcation determined by third-party mediators if neither of the Sudans can reach an agreement.
The U.N. Office of the High Commissioner for Human Rights announced Friday that Mashood Adebayo Baderin, the rights envoy to Sudan, will embark on his first mission to Sudan beginning Sunday.
He is to "identify areas of technical assistance and capacity building that will help the country to fulfill its human rights obligations," the agency said.