Czech qualifier Radek Stepanek posted a second-round upset of the third-seeded Federer, 3-6, 6-3, 6-2.
In the final first-round matchup, No. 4 Andrei Pavel of Romania overcame Wimbledon finalist Nalbandian, 7-5, 6-3.
After going 0-4 in four straight visits to Gstaad, Federer finally recorded a win over Hicham Arazi of Morocco on Tuesday. But the Swiss No. 1 failed to bounce back after falling in the opening round of Wimbledon.
"The only thing positive thing I can take from this week is that I won one match," Federer said. "I played a bad game in the beginning of the second set and it gave him confidence. He also started to serve much better in the second set."
Stepanek reached his fourth ATP quarterfinal of the year. He also made the last eight at Doha and Estoril, and the semifinals at Munich in April.
"I feel great to beat a top-10 player, especially Roger," Stepanek said. "This was a huge victory for me, the best win of my career."
In the quarterfinals, the 23-year-old Stepanek will face Attila Savolt of Hungary, who posted a 6-3, 3-6, 7-5 win over Karol Kucera of Slovakia.
On Sunday, Nalbandian became the first player in the Open Era to compete in the Wimbledon men's final in his tournament debut.
But he was humbled by world No. 1 Lleyton Hewitt of Australia in his debut on Centre Court.
"It's more the altitude that is different rather than the transition from grass to clay," Nalbandian said. "It's difficult to control the ball."
"It's tough for him to come from grass to clay after such great success," Pavel said. "I expected to beat him. Here you have the altitude and on grass the balls aren't bouncing at all."
Pavel will take on Swiss wild-card Marc Rosset in the second round. Ivan Ljubicic of Croatia and Galo Blanco of Spain will clash in the quarterfinals.
Ljubicic overcame Austria's Stefan Koubek, 6-4, 6-4, and Blanco outdueled French qualifier Paul-Henri Mathieu, 7-5, 7-6 (7-1).
UPI horse racing roundup