The Sanitation Department, which owns the signs, said it won't replace the old, faded signs, the New York Post reported.
The department hasn't replaced any of the signs in 10 years due to budget cutbacks.
Dog feces on sidewalks was banned by the city in 1978 and those caught not cleaning up after their dogs can be given a $250 penalty, DSNY spokesman Vito Turso.
Department of Transportation Commissioner Janette Sadik-Khan, whose agency is taking down the signs, said she's not worried about the streets being covered in dog feces.
"New Yorkers know they need to clean up after their dogs, so I don't foresee any problems," she said.
However, some New Yorkers are concerned about the signs being taken down.
"I'm opposed to this," said Maureen Villapando, 35, a physical therapist. "People need to be made aware that they should clean up after their dogs. I'm concerned they might slip on it and hurt themselves."
"People need reminders," said graphic designer Jane Winter, 58. "The sign make it more official."