Wimbledon champion Venus, seeded-sixth in Melbourne, lost 2-6, 6-3, 7-5 to 46th-ranked Carla Suarez Navarro of Spain, who reached the French Open quarterfinals last year.
"She was super consistent and aggressive, and just went for the shots," said Venus, 28, whose serve was broken. Venus, who has never won the Australian Open, lost the last five games of the match.
"I wasn't in control of the points," she said. "I'm definitely used to dictating the points a little bit more."
Serena, a three-time Aussie Open winner, had an easier time beating Argentine Gisela Dulko, 6-3, 7-5. Serena will meet Peng Shuai of China in the third round.
In other Thursday matches, fourth-seeded Olympic gold medalist Elena Dementieva of Russia ousted Czech Iveta Benesova 6-4, 6-1, and countrywoman Svetlana Kuznetsova, seeded eighth, beat Germany's Tatjana Malak 6-2, 6-2.