The regional socialist Samajwadi Party was set to win a majority in the 403-seat assembly in northern Uttar Pradesh with a population of 200 million.
The election had been seen as a critical testing ground for Gandhi -- no relation to Mahatma Gandhi -- to prove his ability to assume the mantle of the world's largest democracy with a population of 1.2 billion.
The loss in Uttar Pradesh is also a huge setback for Congress, which leads the coalition federal government in New Delhi.
The party has been paralyzed by scandals and accusations of corruption even as it tackles stubbornly high inflation and the economic slowdown. An impressive showing in the assembly polls in Uttar Pradesh and four other states would have made it easier for the party to push through much-needed reforms in New Delhi.
Speaking to reporters, Rahul Gandhi, who is expected to succeed Manmohan Singh as India's prime minister, said he will take responsibility for the results in the Uttar Pradesh and congratulated the Samajwadi Party and its young leader Akhilesh Yadav for their victory, NDTV reported.
Gandhi said it was too early to explain why the Congress was routed, the Hindustan Times reported.
Congress also appeared headed for a loss in the assembly polls in the states of Punjab and Goa, but was expected to win in Manipur. The counting in Uttarakhand state was not complete.