Votto sustained the injury after Dallas Keuchel hit him with a pitch in the fourth inning. Votto initially remained in the game, but Kyle Farmer eventually replaced him at first base in the sixth.
Reds manager David Bell said Votto won't require surgery to repair the injured thumb.
"As basic as your breaks can go," Bell told reporters. "It's going to be weeks for sure, but maybe it's three or four weeks and we'll have him back. ... We'll figure it out."
The 37-year-old Votto has a .226 batting average with five home runs and 17 RBIs in 29 games this season. He is one of only four active players with at least 1,800 career MLB games played, joining Albert Pujols, Miguel Cabrera and Yadier Molina.
The Reds are off Thursday and begin a 10-game road trip Friday in Cleveland.
"Maybe it's good that we have the day off [Thursday], and we have several different ways we can go with it," Bell said of the decision to replace Votto. "We'll all sleep on it, and know no matter which way we go, it'll be a good call and the right call.
"We've just got to figure it out. It's all still a little fresh right now."