O'Neill, 67, makes the revelation in the "Modern Family: E! Entertainment Special" set to air Sunday.
"When I read [the pilot script,] I thought, 'Oh, boy, this is pretty good,'" O'Neill said. "And I called my manager and he said, 'Well, they're out to Craig T. Nelson.'"
"They couldn't really decide between Craig and Ed," casting director Jeff Greenberg said. "They're both wonderful comic actors."
Greenberg said ultimately, an offer was made to Nelson, but they couldn't reach a deal.
"He wanted to do it, but, financially, we were just too apart," Greenberg said.
"About a week later, they called and said, 'We're back to you,'" O'Neill said. "So, I said, 'Make the deal.'"
O'Neill is currently up for his third Emmy nomination for Outstanding Actor in a Comedy series for his role on the show. Nelson is a member of the cast of the drama series "Parenthood."