Trump made the announcement on Twitter Tuesday morning.
"Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!" Trump tweeted.
Tillerson just finished a diplomatic trip to Africa and spoke to British Foreign Secretary Boris Johnson Monday, telling him the U.S. government is "outraged" about the poisoning of former Russian spy Sergei Skripal.
White House press secretary Sarah Sanders told reporters Trump had asked Tillerson to step aside.
Tillerson said his job would terminate at midnight March 31 and that he plans to effectively delegate duties to his deputy, John Sullivan, in the meantime.
"What is most important is ... an orderly and smooth transition," Tillerson said.
During remarks Tuesday afternoon from the State Department, Tillerson praised the department's successes under the Trump administration and said much work is still needed regarding Syria's civil war, unrest in Iraq and the defeat of the Islamic State. China and Russia also should remain a focus.
"Russia must assess carefully as to how its actions are in the best interest of the Russian people," he said. "Continuing on their current trajectory is likely to ... increase isolation."
Tillerson, the former head of ExxonMobil, has been Trump's only secretary of state. His departure comes amid international trade talks and ahead of Trump's meeting with North Korean leader Kim Jong Un.
"I wish Rex a lot of good things. I think he's going to be very happy," Trump told reporters Tuesday. "I think Rex will be much happier now, but I really appreciate his service. But with Mike, we have had a very good chemistry right from the beginning."
Under Secretary of State Steve Goldstein said in a statement that Tillerson did not speak to Trump before his dismissal and was unaware of the reason behind the dismissal. He said Tillerson found out when Trump tweeted about his dismissal Tuesday morning.
"The secretary did not speak to the president this morning and is unaware of the reason, but he is grateful for the opportunity to serve, and still believes strongly that public service is a noble calling and not to be regretted," Goldstein said, adding Tillerson had "every intention of staying."
Goldstein, the fourth-highest ranking official in the State Department, was fired shortly after issuing his statement, which appears to contradict the White House's account of how Tillerson was notified of his own ousting.
Tillerson notably did not mention Trump during his remarks, but thanked State Department employees as well as his colleagues in the Joint Chiefs of Staff and Department of Defense.
"All of us, we know, want to leave this place as a better place for the next generation," he said. "I'll now return to private life, as a private citizen and proud American, proud of the opportunity I've had to serve my country."
Reports about Tillerson possibly being replaced by Pompeo first emerged last year.
Tillerson's relationship with Trump has seen a few bumps in the road. Last year, after a report surfaced that Tillerson referred to the president as a "moron," the State Department chief held a news conference to say he didn't plan to resign.
Tillerson did not deny the report but said at the time he had never considered leaving his diplomatic post -- and strongly supported Trump.
"I am here for as long as the president feels I can be useful," Tillerson said at the time.
Trump has been critical of Tillerson on matters of foreign policy.
When Tillerson announced the administration was attempting to enter diplomatic talks with North Korea -- Trump remarked, "I told Rex Tillerson, our wonderful Secretary of State, that he is wasting his time trying to negotiate with Little Rocket Man. Save your energy Rex, we'll do what has to be done!"
In a statement, Trump said Pompeo has a "proven record of working across the aisle."
Pompeo graduated first in his class at West Point, served with distinction in the U.S. army, graduated with honors from Harvard Law School and served in the U.S. House of Representatives.
"As director of the CIA, Mike has earned the praise of members in both parties by strengthening our intelligence gathering, modernizing our defensive and offensive capabilities, and building close ties with our friends and allies in the international intelligence community," Trump said.
"I have gotten to know Mike very well over the past 14 months, and I am confident he is the right person for the job at this critical juncture. He will continue our program of restoring America's standing in the world, strengthening our alliances, confronting our adversaries and seeking the denuclearization of the Korean Peninsula."
Trump said Pompeo's experience in the military, in Congress, and as leader of the CIA have "prepared him well for his new role and I urge his swift confirmation."
Pompeo released a statement expressing his gratitude to Trump for the position and said the president's leadership has made America safer.
"I look forward to representing him and the American people to the rest of the world to further America's prosperity," Pompeo said. "Serving alongside the great men and women of the CIA, the most dedicated and talented public servants I have encountered, has been one of the great honors of my life. If confirmed, I look forward to guiding the world's finest diplomatic corps in formulating and executing the President's foreign policy."
Pompeo also thanked Tillerson for his service, saying "A great deal has been accomplished over the last 14 months, and I wish him and his family well."
Haspel, deputy director of the CIA, will be nominated to replace Pompeo. She would be the first woman to lead the agency.
She has been a CIA officer for 30 years.
"I am grateful to President Trump for the opportunity, and humbled by his confidence in me, to be nominated to be the next director of the Central Intelligence Agency," Haspel said in a statement. "If confirmed, I look forward to providing President Trump the outstanding intelligence support he has grown to expect during his first year in office"