The 77-year-old English actor explained Tuesday on the BBC show HARDtalk that he turned down the chance to play the Harry Potter wizard after Harris' death because the late actor didn't approve of him.
"When they called me up and said would I be interested in being in the Harry Potter films, they didn't say in what part," McKellen recounted. "I worked out what they were thinking, and I couldn't. I couldn't take over the part from an actor who I'd known didn't approve of me."
Harris, who played Dumbledore in Harry Potter and the Sorcerer's Stone and Harry Potter and the Chamber of Secrets before his death in 2002, once described McKellen as a "passionless" performer, as HARDtalk host Stephen Sackur brought up in the interview.
McKellen was cast as Gandalf in The Lord of the Rings trilogy prior to Harris' death, and went on to reprise the wizard in The Hobbit movies. Michael Gambon, meanwhile, replaced Harris as Dumbledore in the final six Harry Potter films.
"Sometimes when I see the posters of Mike Gambon, the actor who gloriously plays Dumbledore, I think sometimes it is me," McKellen joked in the interview.
McKellen previously told The Frame that he's "often mistaken for Dumbledore," but "will always be the actor who played the real wizard." He last portrayed Gandalf in The Hobbit: The Battle of the Five Armies, which opened in theaters in 2014.