"It was a surprise," Patty Peters, vice president for community relations, told The Columbus Dispatch.
"He stopped eating and drinking Tuesday, and that's when things started taking a turn for the worse."
The animal died Wednesday morning, zoo workers said.
Kijito was being treated for an autoimmune disorder known as Black Rhino Syndrome, and while the disease can be fatal for the species Kijito's symptoms of ulcerative lesions on the skin and in the mouth were not severe and it might not have been the primary cause of death, the zoo said in a statement.
Born in 1993 at the Brookfield Zoo in the Chicago area and brought to Columbus in 1999, Kijito sired one calf, Klyde, born on Jan. 2, 2002.
The zoo still has two female black rhinos, Kulinda and Rosie.
The black rhino has suffered the greatest rate of decline of all rhino species, from an estimated 65,000 in Africa in 1970 to 2,300 by 1993.
Anti-poaching laws and conservation efforts have helped increase the population in Africa to an estimated 4,300 black rhinos, the zoo said.