Cody Fransen and Dave Steckel also scored for the victors.
Gustavsson stopped 15 shots in the second period and 18 in the final frame to post a hard-earned win.
The Capitals, meanwhile, lost for a season-high fourth straight time.
Tomas Vokoun started in net and was pulled in the second period after yielding four goals on 18 shots. Michal Neuvirth relieved and let in three scores on 11 shots the rest of the way.
Brooks Laich found the net for Washington.
Ray Liotta sues skin care company over use of likeness
Millions of Getty images now available for free via embed tool