Cody Fransen and Dave Steckel also scored for the victors.
Gustavsson stopped 15 shots in the second period and 18 in the final frame to post a hard-earned win.
The Capitals, meanwhile, lost for a season-high fourth straight time.
Tomas Vokoun started in net and was pulled in the second period after yielding four goals on 18 shots. Michal Neuvirth relieved and let in three scores on 11 shots the rest of the way.
Brooks Laich found the net for Washington.
Couple calls 9-1-1 over missing hash browns; assault McDonanld's employees
Britney Spears debuts 'Perfume' video