The phone call, reported by Rachel "Raquel" Hernandez-Brown in an interview with the San Jose Mercury News, raised questions about whether Orville Brown was distracted to prevent a quick reaction when one of his passengers pounded on the partition to alert him that the Lincoln Town Car was filling with smoke.
The fire broke out in May in the stretch limo carrying nine women. The other four survived.
The passengers were celebrating the marriage of one of the women.
The former wife said he was upset and trying to make amends following a violent outburst police investigated earlier in the day.
"The music was really loud. And I kept yelling, 'I can't hear you. Turn it down,'" she told the newspaper in an article published Sunday. "I said, 'You're not paying attention. You know, like, get off the phone. Stop calling me."
Contacted by the Mercury News Thursday, Orville Brown refused to answer questions, saying he wasn't "authorized to talk to any more reporters; no more interviews."
"We need to follow up on this," Karen Guidotti, chief deputy San Mateo County district attorney, said upon learning of the former wife's comments.
Police had never contacted Hernandez-Brown, the newspaper said.
The California Highway Patrol was expected to issue a report about the fire's cause soon but authorities said they must look into Hernandez-Brown's account before wrapping up their investigation.
For days after the May 4 fire, Brown explained in interviews what happened that night, saying he first thought the women were asking if they could smoke, not that the passenger compartment was filling with smoke. He told the Mercury News he thought it took another 90 seconds before he realized the car was on fire and did everything he could to free the women but the flames were too intense.
Brown credited someone from another car with finally opening one of the limo's back doors.
After he got out of the burning limo, Orville Brown called Hernandez-Brown again to tell her the car was on fire. She said she and her family were shocked that he didn't call police first.
Days later, one of the survivors said Brown did nothing to help the women escape.
2014: The Year in Music [PHOTOS]