Torre, who left the Yankees last month after spending 12 years as the team's manager, said it would take some time to get used to his new situation, NY1 reported Saturday.
"I don't think it's really going to hit me until I go to spring training and all of a sudden, I don't see (Yankee players Derek) Jeter, (Jorge) Posada, Mariano Rivera," Torre said during Friday's fundraiser.
Some of Torre's former players also took time during the Chelsea Piers event to raise money to fight domestic violence, to speak of their history with the famed baseball manager.
"Coming up as a young player, him giving me the opportunity to let me play, and I deserve a lot of things that have happened because of him," Jeter said.
Torre told NY1 that while he is now the manager of the Los Angeles Dodgers, New York will always be important to him.
"New York will always be in my blood," the four-time World Series winner said.
2014: The Year in Music [PHOTOS]