Dylan, 28, posted an open letter on the New York Times website detailing her allegations against Allen last weekend.
But Moses, 36, told People.com he doesn't believe Allen molested his sister and thinks her claims stem from ideas their adopted mother, Mia Farrow, put in Dylan's head after Allen left Mia in 1992 for another of her adopted daughters, 19-year-old Soon-Yi Previn. Allen married Previn in 1997 and they have two adopted teen daughters of their own.
Dylan's sexual-abuse allegations against Allen were investigated two decades ago, but the writer-director was never charged with any crime.
"My mother drummed it into me to hate my father for tearing apart the family and sexually molesting my sister," said Moses, who is now close to Allen and Previn, but estranged from Mia.
"And I hated him for her for years. I see now that this was a vengeful way to pay him back for falling in love with Soon-Yi," Moses said. "Of course Woody did not molest my sister. ... She loved him and looked forward to seeing him when he would visit. She never hid from him until our mother succeeded in creating the atmosphere of fear and hate towards him. The day in question, there were six or seven of us in the house. We were all in public rooms and no one, not my father or sister, was off in any private spaces. My mother was conveniently out shopping. I don't know if my sister really believes she was molested or is trying to please her mother. Pleasing my mother was very powerful motivation because to be on her wrong side was horrible."
Dylan maintained to People.com she is telling the truth.
"This is such a betrayal to me and my whole family," Dylan said of her brother's comments.
"My memories are the truth and they are mine and I will live with that for the rest of my life," she said. "My mother never coached me. She never planted false memories in my brain. My memories are mine. I remember them. She was distraught when I told her. When I came forward with my story she was hoping against hope that I had made it up. In one of the most heartbreaking conversations I have ever had, she sat me down and asked me if I was telling the truth. She said that Dad said he didn't do anything. And I said, 'He's lying.'"
Notable deaths of 2014 [PHOTOS]