Gellar is best known for playing the title character on the TV series "Buffy the Vampire Slayer," which ended its seven-season run in 2003.
Screenwriter Charles Randolph told Variety.com Gellar will play a young woman who has "a kind of zealous immaturity -- like a drug addict with a to-do list" in "The Wonderful Maladys," Randolph's show about three adult siblings whose parents died when they were young.
HBO reportedly plans to shoot the single-camera pilot early next year. Randolph and Gellar will be executive producers of the project, along with Brillstein Entertainment Partners, should it be picked up as a series, the entertainment industry trade newspaper said.
"We're on a fast track," Variety.com quoted Brillstein Chief Executive Officer Jon Liebman as saying.
Gellar described the possible new series as "a terrific next chapter for me."