"I'm a great believer that politicians are human beings and they need to have holidays," Cameron said in a report from The Guardian.
"If you don't think politicians ought to have holidays, I think you need to have a serious think."
London's Summer Olympics ended Sunday. Athletes and spectators have been heading back to their homelands.
Cameron and his deputy Nick Clegg will be vacationing in the Spanish town of Olmedo, north of Madrid.
Foreign Secretary William Hague will be staying put in London to run the country until he leaves next week, the Telegraph reported.
Home Secretary Theresa May will then be in charge until Cameron returns in the middle of the week.
The mayor of London, Boris Johnson, will head out for vacation Saturday but will return a week later to prepare for the Paralympics, which begin Aug. 29.
The prime minister will return for the Paralympics opening ceremony, and then immediately go back on vacation, this time in North Cornwall.
Labor leader Ed Miliband is also due to go on vacation.