The couple married in 2005 and are the parents of Isabella, 4, and Gia, 2. Barroso also has another daughter -- Alexis, 11 -- from a previous relationship.
"My wife is about to have our fourth kid and we've drawn a line in the sand," Damon told Parade.com. "This is it. Our lives are full and wonderful and we're done having kids ... Lucy is in her third trimester for the fourth time and I think that's about enough to ask of any woman ... I don't want to turn her into Ethel Kennedy."
The wife of late Sen. Robert F. Kennedy gave birth to 11 children.
Kris and Bruce Jenner divorcing
'Vacation' re-boot starts shooting in Georgia