Borel, the first jockey to win the Kentucky Derby three times in four years, confronted Castellano near the spot at Churchill Downs where jockeys have their weight recorded following a race.
Castellano was aboard Prince Will I Am in the Breeders' Cup Marathon and moved his horse into the path of two horses while rounding the final turn. One of those horses was A.U. Miner, with Borel aboard.
Prince Will I Am was disqualified after the race, but that was not good enough for Borel, who pointed a finger in anger at Castellano from close range. Castellano then threw a punch at Borel before security officials separated them.
Borel attempted to break free from the hold placed on him by the security men, but finally walked away.
He later said the two had talked and worked out their differences.
Exploding whale video goes viral on Internet
Jordana Brewster on Paul Walker: 'He was an enormous presence in my life'