Gates said his decision was based on "careful consideration" of the situation in Afghanistan as well as consultations with U.S. Army Gen. David Petraeus, commander of CENTCOM.
Gates instead called for Lt. Gen. Stanley McChrystal to take the position of the leader of the International Security Assistance Force in Afghanistan as Washington looks for fresh eyes and fresh thinking for the revamped war effort.
Navy Adm. Mike Mullen, chairman of the Joint Chiefs of Staff, joined Gates on Monday, stressing the new strategy in Afghanistan required new military leadership.
McKiernan led the way in calling for an increased troop strength in Afghanistan as U.S. President Barack Obama pushed a dual strategy that calls for non-military strategy to combat the growing regional insurgency.
Lt. Gen. David Rodriguez would join McChrystal as a deputy commander in Afghanistan, Gates told reporters at an afternoon press conference.
Both officials stressed the call for McKiernan's resignation was not linked to ongoing criticism of mounting civilian casualties in Afghanistan allegedly linked to U.S military strikes, saying the decision was based on "nothing specific."
McKiernan served in his position for about a year. McChrystal, meanwhile, served in an administrative position at the Joint Chiefs of Staff for less than a year.
Both Gates and Mullen said the new leadership in Afghanistan would bring counterinsurgency expertise to the position as military strategists look to the methods employed in Iraq as a model for the new Afghan effort.
Gates called on U.S. lawmakers to confirm the nominations as soon as possible.