Rafael Cruz, a conservative pastor, made the remark in April 2012 but it only recently became public after the magazine Mother Jones posted several videos of the elder Cruz Thursday.
Once it was online, Cruz's remark to a supporter that Obama should "go back to Chicago or Kenya" quickly sparked criticism.
In an interview Friday with CNN affiliate WFAA-TV, Dallas, the senator suggested the outcry had more to do with politics than real anger.
"It was an ill-advised joke and sadly, those who are trying to play the politics of personal destruction are trying to smear him and use that to attack me. That's a shame," Cruz said.
"I love my father," Cruz said. "He is a pastor. He is a man of deep integrity. And he made a joke."
Obama, whose father returned to Kenya not long after his birth, has faced claims since he became a national figure he was not born in the United States. His birth certificate and a contemporary notice in a Honolulu newspaper show he was born in Hawaii, where his parents were students.
Rafael Cruz, an immigrant from Cuba who became fiercely right-wing after being an early supporter of Fidel Castro, was living in Canada and working in the oil business when the senator was born in Alberta.
He has called Obama an "outright Marxist" who "seeks to destroy all concept of God," Mother Jones reported.