Gilani fired Naeem Khalid Lodhi, a retired general, and named a civilian, Nargis Sethi, to replace him, The New York Times reported.
Lodhi was guilty of "gross misconduct and illegal actions," Gilani said.
Gilani's action is expected to increase the already high level of mistrust between the powerful military and the government of President Asif Ali Zardari, the Times said.
Gen. Ashfaq Parvez Kayani, the army chief who would normally have been consulted on any appointment to the defense post, plans to meet Thursday with the top military commanders.
"The army will not react violently, but it will not cooperate with the new secretary defense," a military officer who did not want his name used told the Times.
Gilani, under pressure from the Supreme Court to reopen a corruption investigation of Zardari, instead defended him. Though not directly criticizing the Supreme Court, the prime minister, speaking to a group of lawmakers in Islamabad, said the Pakistan People's Party or PPP, which heads the ruling coalition, has a tradition of neither submitting to coercion nor giving in to dictatorships, the Dawn newspaper reported Wednesday.
The high court ordered the government Tuesday to reopen the investigation. Zardari became president after his wife, former Prime Minister Benazir Bhutto, was assassinated.
The Zardari government is already under pressure over the so-called memo scandal that allegedly sought U.S. help to prevent a military coup in Pakistan. The government has denied any involvement in it but the scandal rages.
Gilani was quoted as saying his party has the right to be in government since it won elections.
"The PPP workers are loyal to their ideology and I appeal to intellectuals to regard PPP and its leaders as an asset and protect them," Dawn quoted him as saying.
The Los Angles Times quoted experts as saying Zardari could be forced out if it's proved he engineered or authorized the memo. The powerful military is already irate about the memo.
The Supreme Court warning to Gilani relates to a 2-year-old order asking the government to reopen money-laundering charges against Zardari. The government had dropped the case after Zardari and the late Benazir Bhutto appealed a court ruling.
But the Gilani government has not yet reopened the case.
NBC reportedly holds celebs hostage to Jimmy Fallon's show
Boston schools pull out free condoms over wrapping complaints