Castro's departure came Wednesday night, a day after he forgot some of the words while performing "Mr. Tambourine Man."
Speaking to host Ryan Seacrest before learning his fate, Castro said: "It's just getting tough for me. I think my inexperience is coming in."
In telling Castro he had been voted off of the show, Seacrest said, "I almost feel like you're relieved," to which Castro replied: "A little. It's three songs next week."