Kelly said he plans to "seek other opportunities," The Arizona Republic reported.
The primary is scheduled for Aug. 28. Former Giffords aide Ron Barber was elected Tuesday to serve out the remainder of her term.
Barber was wounded in the 2011 Tucson shooting spree aimed at the congresswoman that left six people dead. Giffords, who suffered a serious head wound, decided to resign from Congress this year.
Kelly, 30, ran 7 percentage points behind Barber, 66, Tuesday in the Tucson-area district. Kelly lost to Giffords, D-Ariz., in 2010 by about 4 points.
Kelly's departure clears the way for Martha McSally -- the first woman to fly as a combat pilot in the Air Force -- in the GOP field, the newspaper said.
Republicans have a 26,000 voter registration advantage in the district, but its lines have been redrawn since the 2010 Census, creating a district more evenly split between the parties.
State Rep. Matt Heinz said he is still running against Barber in the primary. He said whoever wins in November could represent southern Arizona for the next decade.
"This last term, this was Gabby's seat. ... She should have some significant say in who finishes out the two years that she earned," Heinz said. "But when it comes to the fall, it is only about 70 percent of the same district."
Exploding whale video goes viral on Internet
Puzzle-maker slips 'Murdoch Is Evil' into Rupert Murdoch's Sunday Telegraph