Ofcom is now investigating the matter, the Daily Mail reported Tuesday.
"They expressed concerns about the operation of the voting and many said they were struggling to get through," an Ofcom spokesman said.
Rhydian Roberts had been favored to win the singing competition, but came in second to Leon Jackson.
An "X Factor" spokeswoman denied any voting irregularities and TalkbackThame, which produces the show, told the Mail viewers couldn't get through because of the high volume of calls.
Roberts stayed away in protest from the show's wrap party Monday night.
"Rhydian feels he was stitched-up over the final. He was the hot favorite, and was shocked to hear that many viewers couldn't get through to register their votes for him," a source said.
"He's very disappointed and feels he should have won. He didn't feel he had anything to celebrate last night, so thought it best to stay away until an investigation is held into the phone voting."
Jackson, presenters Dermot O'Leary and Fearne Cotton and judge Louis Walsh all attended the festivities.
Celebrity Families of 2014 [PHOTOS]
Tracy Morgan hits back at Walmart