Jefferson has been under investigation for 14 months for allegedly accepting bribes in connection with business ventures in Africa. FBI agents raided his Capitol Hill office Saturday night, and there are reports the FBI has videotape of Jefferson allegedly receiving a payoff.
Investigators have said they found $90,000 in a freezer in Jefferson's home.
In a tersely worded letter Wednesday, Pelosi, D-Calif., requested Jefferson's resignation from the committee.
"In the interest of upholding the high ethical standard of the House Democratic Caucus," said Pelosi, "I am writing to request your immediate resignation from the Ways and Means Committee."
Jefferson wrote back to Pelosi, refusing to resign the committee assignment, the Baton Rouge Advocate reported.
"None of the matters reported to be under scrutiny involve issues under jurisdiction of the Ways and Means committee," Jefferson wrote. "Therefore, such a request would be ... perplexing and unreasonable."
Jefferson called Pelosi's request "discriminatory" and said his resignation from the committee would "punish" the people of his district.