"They'll have to use their bare hands," Sherman told The New York Post in a story published Tuesday.
Sherman, a portrait photographer, lives in a rent-controlled, $530-a-month studio that Carnegie Hall wants to turn into office space as part of a $150 million overall building renovation.
Sherman's grand apartment has floor-to-ceiling windows, black-and-white checkered floor tiles and a cast-iron staircase, the Post reported, noting Sherman has photographed Andy Warhol, Henry Fonda, Eva Gabor and Paul Newman.
"My memories of people I've photographed are here," said Sherman who has rejected Carnegie's offer to find her a similar apartment in the neighborhood and to pay any differential in rent for the remainder of her life.
"I'm going to keep on living here," said Sherman, known locally as the Dutchess of Carnegie.
2014: The Year in Music [PHOTOS]