David Andrukonis, former chief risk officer at the huge mortgage broker, said he sent top executive Richard Syron a memo in 2004, warning him the company's lending standards had eroded, The New York Times reported Tuesday.

Government-sponsored enterprises Freddie Mac (NYSE:FRE) and the Federal National Mortgage Association, known Fannie Mae (NYSE:FNM), are both part of a massive federal rescue proposal that could cost billions of taxpayer dollars.

But, when warned the company was buying bad loans in 2004, Syron said, "we couldn't afford to say no to anyone," Andrukonis told the Times.

The memo could not be located, Freddie Mac said in a statement. There was also "little to nothing that Freddie Mac could have done to prevent the losses that it is now incurring," company spokesman David Palombi said.

While some are calling for replacing top executives at Freddie Mac and Fannie Mae, "this company has to answer to shareholders, to our regulator and to Congress, and those groups often demand completely contradictory things," Syron said.