Frost said he and Key were unsure of the mystery animal's identity until it let out a "meow" and got close enough for them to tell it was a small kitten.
The men pulled the kitten up into their boat, where it shook off the water.
"Never in my life have I seen anything like that," Frost says in the video.
The men then noticed a second, identical kitten jump into the water from the shore and swim toward them. The men steered their boat toward the second feline and picked it up.
"Well there's a first time for everything, isn't there?" Frost says.
Frost said the kittens were friendly and seemed comfortable with riding in the boat.
"People were looking at us out on the water like, 'Oh my God, these people got cats with them,'" Frost told AL.com.
Frost said he and Key gave the kittens to a family on the shore with two young girls.
"Everybody always has a crazy fishing story, and that's all it is, is a story," he said. "If I wouldn't have had this on video, everybody would have thought I was crazy."