Authorities said the wildfire, which started Tuesday, had taken two lives while consuming 485 homes and blackening 14,198 acres, The Denver Post reported. About 1,175 personnel were working to control the blaze.
"There's no further loss of life," El Paso County Sheriff Terry Maketa told reporters at a noon news conference.
Hundreds of people might have to wait until Thursday before they will be allowed to return to their homes, the newspaper said.
Maketa counseled the evacuees to be patient.
"Please don't sneak in," Maketa told them Saturday. "When it is safe to open it up, we will open it up."
The identities of the two fire fatalities had not been released.
"I'm thankful we did not have greater loss of life," Maketa said.
The Post said investigators had yet to determine the cause of the fire, though the sheriff said it likely was human-caused.
"This is a crime scene until proven otherwise," the sheriff said. "I won't compromise that by letting people in too soon."
The Black Forest fire is the most destructive in Colorado history, surpassing 2012's Waldo Canyon fire that wiped out 346 homes, the Post said.