The lawsuit stems from details Cosby included in her new book, "Blonde Ambition: The Untold Story Behind Anna Nicole," Usmagazine.com reported.
"The libel lawsuit was filed this afternoon in the U.S. District Court for the Southern District of New York against Rita Cosby and her publisher, Hachette Book Group USA," Stern's attorney, Lin Wood, was quoted as saying on the magazine's Web site.
First reported by CelebTV.com, the lawsuit asks for $60 million in compensatory and punitive damages.
In the book, Cosby, a former MSNBC anchor, claims Stern and Larry Birkhead, the father of Smith's daughter, Dannielynn, engaged in homosexual acts caught on tape.
The book also alleges Stern and Birkhead made a deal whereas Birkhead would get custody of Dannielynn and Stern would be named executor of Smith's estate.