Miliband, brother of Labor Party leader Ed Miliband, was expected to announce the move Wednesday, sources close to him told the British newspaper.
"David has landed a dream job in charge of a charity in New York and wants to take it," a friend told the newspaper. "But it means leaving politics."
Miliband has close ties to the United States. His wife, Louise Shackelton, a violinist with the London Symphony, was born in the United States and the couple adopted two sons there.
The IRC is an international organization active in 40 countries in areas like refugee resettlement and post-conflict development.
Miliband, 47, was a rising star in the Labor Party. He became Tony Blair's head of policy at the age of 29.
In 2001, Miliband was elected to Parliament from South Shields, a safe Labor district. He spent only a year as a back-bencher before becoming a junior minister and entered the Cabinet after the 2005 election, becoming foreign secretary in 2007.
After Labor lost the 2010 general election, Miliband stood for party leadership. He lost a five-way race to his brother.