The shooting took place outside a supermarket, the diplomat, who spoke on condition of anonymity, told CNN.
Witnesses said the German national, whose name wasn't reported, was in the parking lot of the upscale supermarket when two gunmen opened fire on him.
"Twenty minutes later more than a dozen German forces accompanied by Yemeni Special Forces arrived to the scene," said Shukri al-Sharabi, a witness to the incident.
The Yememi diplomat suspected the gunmen belonged to al-Qaida in the Arabian Peninsula.
"It looks like these targeted AQAP assassinations are only going to get worse. If this continues at this rate, we'll be heading toward a complete collapse of the security apparatus in Yemen," the diplomat said.
Newt Gingrich fires back at Mandela backlash
Puzzle-maker slips 'Murdoch Is Evil' into Rupert Murdoch's Sunday Telegraph