The Financial Times reported that the calls came amid the Labor defeat in the by-election in Crewe and Nantwich, a district in Cheshire.
Thursday's election marked the first time in a generation that conservatives gained in a by-election. The Conservative Party took the election with a 7,860-vote victory, the newspaper said.
Conservative leader David Cameron heralded the victory as the "end of new Labor."
Labor member Graham Stringer said it was "the responsibility of senior members of the cabinet" to stand against Brown or risk facing disaster at the next general election. Some members say Jack Straw, the justice secretary and the cabinet's elder statesman, should tell Brown to step down in the interest of the party.
Harriet Harman, the deputy leader of the Labor Party, conceded there were "discordant voices" but said there was no evidence of an organized effort to unseat Brown.
Swim Week Miami Beach 2014 [PHOTOS]