Cleveland won its third in a row in its bid for a playoff spot, getting 6 2/3 innings of one-run pitching out of Justin Masterson (14-9).
Masterson allowed five hits and walked five more, but struck out seven to win for the first time in four starts.
Rich Hill and Cody Allen held the Angels in check until the ninth, when Chris Perez came on to pick up his 20th save.
Hank Conger drove in the only Los Angeles run in the sixth with a double to cut the Angels' deficit in half and the Indians picked up an insurance tally in the ninth when Mike Aviles scored on a wild pitch.
Jerome Williams (5-10) gave up Swisher's home run among six hits over 6 1/3 innings in suffering the loss.