The dog was killed Tuesday outside a residence on the city's west side, The Detroit News reported.
Audrejuanna Tucker, 14, said she had just let JoJo, her 7-pound Chihuahua, outside when she was approached by the man who told her she needed to keep the pet on a leash.
As the man walked away, the dog barked at him and was shot, police were told.
Investigators determined that no charges would be filed against the man, who was not identified, because the animal was unleashed and on a public sidewalk.
"We heard that ... the dog had been a problem, that it was harassing residents," said David Rudolph, a spokesman for Detroit Dog Rescue.
But he questioned why the shooting was necessary.
"This was a little Chihuahua; he shot a Taco Bell dog. At what point could it have aggressed in such a way that you felt you had to go out and shoot it?"
Ohio bar shooting arrested, charged with murder
Duggar sisters unveil Christian dating rules in new book