New York escaped a ninth-inning jam to end the Angels' eight-game winning streak. Los Angeles had not lost since dropping a 2-1 decision to Oakland May 21. After going 8-15 in April, the Angels turned in an 18-11 record in May.
Swisher then delivered the final run of the game after Raul Ibanez tripled with one out in the sixth.
Rafael Soriano came on for the Yankees to protect the one-run lead in the ninth and issued a one-out walk to Alberto Callaspo. Albert Pujols reached on an infield single but Callaspo was erased at third on a fielder's choice grounder by Torii Hunter.
A fly ball to left by Mark Trumbo ended the Angels' threat and the game, giving Soriano his sixth save.
Starter Ivan Nova (6-2) earned the win, surviving 6 2/3 innings, during which he gave up five runs on eight hits.
Angels' reliever Hisanori Takahashi gave up the go-ahead run in the sixth and suffered the loss.