The 46-year-old actor addressed the issue during a Watch What Happens Live interview Thursday. He and Garner co-starred in 13 Going on 30 in 2004, and were friends until she and Ben Affleck began their relationship.
"We had a great time together," he shares. "I think we would [stay in touch], but then Ben came on the scene and that was the end of that."
Garner and Affleck met on the set of Daredevil in 2003, and started dating in mid-2004. The couple married in June 2005, and have three children together. Ruffalo says he is "unable to confirm" if Affleck prevents Garner from having male friends, but admits it is "sad" he and the actress don't keep in touch.
Ruffalo is currently filming Avengers: Age of Ultron for 2015. Garner will appear in Alexander and the Terrible, Horrible, No Good, Very Bad Day and Men, Women & Children later this year.