After a raucous hearing Tuesday evening, Community Board No. 1 voted 29-1 in favor of the move with 10 abstentions, The New York Times reports.
The board's vote is seen as an important barometer of community sentiment even though it was only an advisory one, the newspaper says.
More than 100 people testified before the panel voted. Some carried pictures of family members who were killed in the attack on the World Trade Center.
"The pain never goes away," C. Lee Hanson, 77, told the board. Hanson's son Peter died in the attack.
Hanson said building a tribute to Islam so close to the World Trade Center would be insensitive.
Others told the board the center would be a monument to tolerance.
Members of Congress to keep receiving porn magazine
Iranian woman stops the execution of son's killer