The Rev. Louie Giglio, an Atlanta minister, advocated for reparative gay therapy and encouraged followers not to let the "homosexual lifestyle" become accepted, NBC News said Thursday.
The New York Times said Giglio called on opposition to the gay rights movement's "aggressive agenda."
The Presidential Inaugural Committee said it was unaware of Giglio's comments, made in the 1990s.
"We were not aware of Pastor Giglio's past comments at the time of his selection and they don't reflect our desire to celebrate the strength and diversity of our country at this Inaugural," said Addie Whisenant, a spokeswoman for the committee.
A replacement who reflects "the administration's vision of inclusion and acceptance for all Americans" will be sought, Whisenant said.
The selection of pastor Rick Warren to give the invocation in 2008 drew criticism from many Obama supporters because of his views on homosexuality, same-sex marriage and abortion.