The closing ceremonies will include the passing of the Olympic baton to London, site for the 2012 Summer Olympics, the BBC reported.
The Olympics minister Tessa Jowell will represent Britain at the opening in Beijing in August.
The prime minister had been under pressure in his country to boycott the Olympics because of China's recent crackdown in Tibet and other human rights violations but a Downing Street spokeswoman said Brown was not boycotting the Games since he'd never planned to attend the opening ceremony, the report said.
Olympics Minister Tessa Jowell will represent Britain at the opening, the report said.
The New York Times reported before the decision, it was implied that Brown planned to attend the opening ceremonies. However, his spokeswoman said there has been no change in the prime minister's position.
"I am flummoxed by this media-created storm," the Times quoted her as saying.