Serbia's Troicki, who was seeded third after winning the event in 2010 and reaching the finals last year, was surprised by Tunisian Malek Jaziri 6-1, 3-6, 6-2 at Olympic Stadium.
Troicki was bested by fellow Serb Janko Tipsarevic in the 2011 Moscow finale.
Jaziri will next face the Czech Republic's Lukas Rosol, who bounced Uzbekistan's Istomin 4-6, 6-4, 6-3.
Also in second-round play, eighth-seeded Tatsuma Ito of Japan edged Russian qualifier Konstantin Kravchuk 7-6 (9-7), 2-6, 6-3.
2014: The Year in Music [PHOTOS]