Birkhead, 35, said he knew he and his 21-month-old daughter Dannielynn were meant to live in the home in his home town when he first took a tour of the 10,000-square-foot, six-bedroom house, People.com reported Saturday.
"When I walked in, I said to myself, 'This is our home!'" Birkhead said.
Birkhead, who was proven to be Dannielynn's biological father after Smith's death in February 2007, said he is now planning on selling his Los Angeles home.
He told People.com the decision to part with the home was difficult, given the time he spent there with the Playboy model and her son Daniel, who also died of drug-related causes.
"At first, I couldn't imagine parting with it," Birkhead said. "Then I realized I was probably holding on to it for the wrong reasons, thinking to myself that would ease the pain of losing Anna and Daniel."