Observers said the decision Monday was a major defeat for human rights advocates who had called Rios Montt's conviction earlier this month evidence that the courts would no longer allow Guatemala's powerful to act with impunity, The New York Times reported.
The attorney general's office was expected to appeal the court's 3-2 ruling this week.
Rios Montt, 86, was sent to prison immediately after the verdict May 10 when a tribunal found him guilty of genocide and crimes against humanity. He was sentenced to 18 years in prison but soon was transferred to a military hospital for medical tests.
Monday's decision returned him to house arrest, where he had been held since January 2012.
The tribunal found Rios Montt, as commander in chief, responsible for the slaughter, rape and forced displacement of the Maya Ixil ethnic group during his 17-month rule in the early 1980s.
The court didn't invalidate the entire trial but ordered that the proceedings begin anew as of April 19, the date a decision by another judge sent the trial into turmoil and caused a suspension, the Times said. By April 19, the tribunal heard all of the prosecution's case and most of the defense's, which still stands. The court's ruling invalidated everything after April 19.
Legal experts told the Times it was unclear what would happen because a repetition of the days post-April 19 would be unlikely because that would basically be double jeopardy.
His co-defendant, Gen. Jose Mauricio Rodriguez Sanchez, was acquitted by the tribunal. The Times said the court's ruling effectively negates his acquittal, leaving unclear whether he would be rearrested.