The two countries have long been embroiled in territorial disputes, with the island of Cyprus being the focal point, and no Greek leader has officially set foot in Turkey since 1959, a BBC correspondent reported from Ankara.
Before departing Athens, Karamanlis told reporters he would reiterate Greece's support for Turkey's membership in the European Union, and said it was the right time to take action to boost bilateral relations, the Turkish newspaper The New Anatolian reported.
He noted in recent years there has been an improvement in bilateral trade and investments, tourism and the two nations have opened a joint gas pipeline.
Annual trade volume between Greece and Turkey was $200 million 10 years ago whereas it now stands around $3 billion, the Turkish Daily News said.
Karamanlis' three-day visit to Ankara will include meetings with his counterpart, Prime Minister Recep Erdogan.