Bynum, signed by Indiana as a free agent on Feb. 1, totaled eight points and 10 rebounds in just under 16 minutes off the bench.
"It's a different location, a different time in my life. I'm looking forward to just going out and playing. Obviously, the atmosphere is a lot better from the standpoint that we are winning," Bynum said.
David West checked in with 24 points while Paul George and George Hill netted 12 apiece for the Pacers, who bounced back after losing their previous three contests by an average of nearly 20 points per game.
The biggest lead for Indiana was 15 late in the third quarter, but Boston stayed close behind a 16-4 run that stretched into the final stanza, as Sullinger's 3-pointer cut the Celtics' deficit to 69-66.
A Sullinger putback got Boston within 77-76 near the midway point, and Indiana answered with the next six points to keep the visitors at bay.
West's three-point play within under two minutes to go made it a 91-81 game, and the Pacers held on for their first season sweep of Boston since the 1998-99 campaign.
"It was a pretty good effort by our guys," Celtics head coach Brad Stevens said. "Everything went great except the ball going in the basket. That's OK. I feel like we're getting better, I really do, and I think you can see it."
Bynum, traded away from Cleveland and dumped by Chicago in January, entered the game with 4:22 remaining in the first quarter to a rousing ovation and made an impact on his second possession with a two-handed slam.
The Pacers began pulling away late in the second quarter as another midseason addition, Evan Turner, scored seven points on a 13-2 run that extended Indiana's lead to 47-33.