Some 250 police officers were on scene as four segments of the "East Side Gallery" were removed to allow access to an area where apartment buildings will be constructed, the BBC reported.
About 72 feet of the wall will be removed, but replaced once construction is complete, said Maik Uwe Hinkel, the investor in the building project.
Angry protests stopped an attempt on March 1 to dismantle sections of the gallery, the longest intact piece of the wall at that once divided East and West Germany.
Berlin mayor Klaus Wowereit had met with the developer and district authorities that control the wall in an attempt to find alternatives, but those talks failed Tuesday night.
Once seen as an eyesore, the section of the Berlin Wall became a tourist attraction in the 1990s when international artists painted it with sometimes irreverent images. It is now seen by those who oppose the area's gentrification as a memorial to Berlin's divided past.