Whitfield had asked whether Rivers' Fashion Police commentary was too "mean," prompting Rivers to storm off, calling the interview "defensive."
Rivers, who denied accusations that her tantrum was a publicity stunt for her new book, Diary of a Mad Diva, told Letterman that Whitfield was "very judgmental, very nasty, very opinionated, very negative."
"It was like my wedding night, ya know?" she added.
But when Rivers switched gears to a joke about Depends, Letterman stood up, removed his jacket and walked off the set.
When he didn't return, Rivers took over the interview, and started questioning herself.
2014: The Year in Music [PHOTOS]