Ramirez, who received a 50-game suspension on May 7 for violating baseball's substance abuse policy, is fourth among National League outfielders in the first release of All-Star balloting totals. The top three vote-getters will start in the All-Star contest July 14 at Busch Stadium in St. Louis.
"To me, I think, the significance of the All-Star Game is to reward players who had a good first half," Torre told the Los Angeles Times. "We don't always do that because it's a popularity vote, for the most part."
Torre said Ramirez's popularity is responsible for his high vote total.
"I think realistically, he didn't earn, other than from his reputation, to be, this year, in the All-Star Game," Torre said. "It probably isn't the right thing for him to be in it this year."
Torre said he thought Ramirez might agree with him.
Dodgers owner Frank McCourt has told MLB.com it would be OK with him if Ramirez makes the NL squad.
"I'm sure it would be a great honor," McCourt said.
Ramirez was batting .348 with six home runs and 20 RBI in 92 at-bats before he was suspended.
"I'm going to think about that," Commissioner Bud Selig told the Web site late Friday. "If the fans choose someone, I'd like to honor that."