Fisher, 52, had been hoping to coach the Seminoles on Saturday against Louisiana-Monroe but Florida State officials pushed for a decision. At the same time, Texas A&M regents on Thursday reportedly approved Fisher as coach of the Aggies during a meeting.
Florida State officials are expected to formally announce the change later Friday.
Defensive line coach Odell Haggins will coach Florida State (5-6) on Saturday.
Fisher reportedly will receive a 10-year contract averaging more than $7 million per season. He made $5.7 million this season as part of an extension he signed in December 2016.
One factor that worked in Texas A&M's favor is that Fisher and Aggies athletic director Scott Woodward worked together when Fisher was offensive coordinator at LSU.
Fisher reportedly has grown frustrated with Florida State officials over the failure to improve football facilities.
Fisher had an 83-23 record in eight seasons with the Seminoles and won the 2013 BCS national championship with a 14-0 record.
Swinney was told of Fisher's move to Texas A&M during a press conference for Saturday's ACC title contest between Clemson and Miami.
"Obviously, he feels like it's the right move for him, I don't know," Swinney said when asked. "I know he did an unbelievable job at Florida State. This is a business that's very public, as we all know, and from time to time, you have to make decisions. ... I like Jimbo. It's been a great, competitive journey. I've known him forever, when I was at Alabama and he was at Auburn.
"Florida State is Florida State. They'll hire a great coach. I don't have any doubt about that. And I guess we'll see (Fisher) next year, if that's what he's going to do. We play Texas A&M the next two years. If he is (going to the Aggies), I can't get rid of him. He'll still be a thorn in our side."
Names circulating as possibilities for the Florida State post include Oregon's Willie Taggart and South Florida's Charlie Strong.
Texas A&M fired Kevin Sumlin after he went 51-26 in six seasons as coach.