Christian prayer at the building atop Mount Zion, outside the fortress walls of the city, is in conflict with Orthodox Jewish teachings, they claimed.
Protesters gathered at the site -- which is also the tomb of Jewish King David, as well as a 16th-century mosque -- and rabbis with loudspeakers led prayers.
"Under Jewish law it is a big problem. Basically they (Christians) are taking over the place," said Rabbi Avraham Goldstein.
The pope's visit to Jordan and Israel, scheduled for May 24 to 26, comes after attacks of vandalism on Christian sites in Jerusalem, allegedly by far-right Jewish groups. Several Christian churches have been defaced, including a death threat painted in Hebrew on East Jerusalem's Assembly of Bishops at the Notre Dame Center. The site is where the pope is expected to meet with Israeli Prime Minister Benjamin Netanyahu.