Khaled's replacement is Prince Fahd bin Abdullah bin Mohammed, who has commanded the Saudi navy since 2002, the news agency SPA said.
SPA said the king issued a royal decree. No reason for the shift was given.
Khaled, 62, graduated from the British military academy, Sandhurst, and was also educated at U.S. military colleges. He commanded Muslim forces during the First Gulf War, when a U.S.-led coalition ended the Iraqi occupation of Kuwait, Middle East Online said.
Khaled became deputy defense minister in 2011, soon after the death of his father, who was the Saudi defense minister for 50 years.
Abdullah, 88, succeeded his half-brother in 2005. His age and apparently poor state of health have raised concerns and both of his former crown princes, a half-brother and full brother, have died since he became king.