"We are thrilled," Kelsey, 57, and Kayte, 31, told People.com in a statement. "Mother and child are in excellent health."
The occasion was tinged with sorrow as little Evangeline's twin brother died early in Kayte's pregnancy.
"We were ecstatic earlier this year, when we announced that Kayte was carrying twins. Tragically, we lost the little boy shortly thereafter," the couple said in their statement. "This was not something we cared to make known publicly at the time. It was unspeakably painful and we know that people will understand our desire to keep the news private then, as we know they will respect our privacy in this matter now. A glorious birth with a lingering sadness is ours today. We choose to celebrate the life that has been given us. We proudly introduce our Faith to the world today looking forward to the days ahead and the children yet to come."
Kayte also suffered a miscarriage in 2010. The couple wed in February 2011.
Kelsey Grammer won four Emmys, and was nominated 13 times overall, for his work as Dr. Frasier Crane on TV's "Cheers" and "Frasier." He received a fifth Emmy in 2006 for his voiceover performance in an episode of "The Simpsons."