"[He] hated me. Hated me. We did a thing called Restoration and he took one look at me and wanted to kill me," Grant told host Jess Cagle, also People magazine's editor-in-chief.
"I don't know [why]," he added. "I was so hurt."
Grant and Downey starred with Sam Neill, David Thewlis, Meg Ryan and Ian McKellen in Restoration, which is based on the Rose Tremaine novel of the same name. Downey responded to Grant's comments in a tweet Thursday.
"A lot has happened over two decades! I respect how Mr. Grant has matured as an artist & voice against violations of privacy. Let's break bread together soon @HackedOffHugh! #burythehatchet2018," the 52-year-old actor wrote.
Grant also said his Music and Lyrics co-star Drew Barrymore hated him "a bit," explaining, "She was very L.A. and I was a grumpy old Londoner." The actor most recently starred as the voice of Phoenix Buchanan in Paddington 2, which opened Friday.