The Orioles had tied the game with a pair of runs in the top of the fourth inning, but the Yankees came right back in the bottom of the frame, scoring three times to put the contest away.
Robinson Cano led off with a walk and Swisher (2-for-3, two runs scored) followed by belting a 2-0 pitch from Chris Tillman (2-4) into the right field seats, handing New York the lead.
Brett Gardner added a pair of RBI for the Yankees, who were playing their first home doubleheader since 2003.
Bartolo Colon (8-6) went five innings to get the win, allowing two runs on five hits while walking a pair and striking out six.
Tillman lasted only 4 1/3 frames, surrendering seven runs (five earned) on eight hits for the O's, who fell to 2-9 in their last 11 road games.
Mark Reynolds hit a solo homer for Baltimore.