The Pennsylvania Republican said he has concerns about Holder's role in the 2001 pardon of financier Marc Rich, The Hill reported Wednesday.
Previously this week, Judiciary Committee Chairman Patrick Leahy, D-Vt., said he would like Holder, the deputy attorney general during the Clinton administration, confirmed by the time President-elect Barack Obama takes office Jan. 20.
However, Specter said starting the hearings before Jan. 26 is "not realistic or fair." Specter said he had not drawn conclusions about Holder's nomination, the newspaper reported.
"There are questions which have to be inquired," Specter was quoted by the newspaper as saying.
The Hill reported that Specter met with Holder privately Monday.