Speaking in Tallahassee, Scott said he had called U.S. Transportation Secretary Ray LaHood and told him, "I'm not comfortable this is a project we should be doing," The Miami Herald reported.
The governor said the planned Tampa-Lakeland-Orlando link might not yield taxpayers a return on their investment and the funds would be better spent on highways and ports.
Sen. Bill Nelson, D-Fla., said Scott's decision is "turning down 24,000 jobs, when we badly need them."
Rep. John Mica, R-Fla., chairman of the House Transportation and Infrastructure Committee, called the move "a huge setback for the state of Florida."
Scott is the third new Republican governor to spurn the rail funding, following counterparts in Ohio and Wisconsin. Florida's share would have been $2.4 billion.
The governor said cost overruns could cost Florida taxpayers an extra $3 billion, ridership and revenue projections were questionable, and if the project were canceled, the state would have to return the money.
Tea Party members, who backed Scott's campaign, had lobbied him to reject the funds.
Florida bear attack: Black bear mauls woman's face
Man spent 15 hours in jail for plugging electric car into an outlet at a school