Production on the third season of the show is under way. The series has been a huge ratings hit in the United Kingdom on ITV and in the United States on PBS.
Set in the early 20th century, the drama follows the lives of the wealthy Grantham family and the many people who keep their great house running.
Smith, 77, plays the sharp-tongued matriarch dowager countess of Grantham.
The Mirror said Smith has alarmed the show's producers by not signing a contract for the anticipated fourth and fifth seasons of "Downton Abbey," even though the show has not been officially renewed that far in advance.
Co-stars Michelle Dockery, Hugh Bonneville, Joanne Froggatt, Brendan Coyle and Elizabeth McGovern have already signed contracts for possible future seasons, the newspaper said, adding Dan Stevens, who plays Matthew Crawley, has not signed a new contract.
"'Downton Abbey' has a history of major surprises, and the producers are not opposed to changing casting within the show to add to the drama," a source told the Mirror. "But it's hard to imagine the series without Maggie Smith or Dan Stevens involved in some way. To write the dowager countess out of the series, or even kill off the character, would be 'Downton's' boldest storyline yet."
Notable deaths of 2014 [PHOTOS]