Richard Riddle, 52, had left the Hiawatha Church of God in Christ with his wife after the Sunday service when he was shot, investigators said.
Reshard Riddle, the victim's son, approached him and shot him with a handgun, Sean Adams, a church associate pastor, told police.
"It was terrifying," Adams said. "The children were screaming ... . we were afraid to breathe."
Richard Riddle died instantly, The (Ashtabula) Star-Beacon reported Monday.
Approximately 150 other parishioners were leaving the church as the shooting occurred. Many ran back into the building, hiding under pews and in bathrooms, and calling 911.
Reshad Riddle entered the church, yelling that the shooting was "the will of Allah ... the will of God," witnesses told police.
Police officers arrived on the scene just moments after the shooting and arrested Reshad Riddle, Ashtabula Police Chief Robert Stell said. Police did not discuss charges in the case.
No one else was hit by gunfire, Stell said.
"The initial call we received was more along the lines of a mass shooting," he said. "We knew that shots were fired in the church and we thought there were multiple people down. We called other agencies to assist because it was described as a mass shooting and we weren't sure if there were multiple shooters or multiple people wounded."