Although that belief is widely held across the political spectrum, the public is split along party lines about who is to blame, The New York Times/CBS News poll found.
About 40 percent of the 1,022 people surveyed, including about two-thirds of Democrats, said the IRS acted on its own investigating conservative groups who had applied for tax-exempt status. Slightly more, including 70 percent of Republicans, said the Obama administration was involved.
The poll said people who closely followed the scandal and Tea Party supporters were significantly more likely to believe the actions were politically motivated.
While 60 percent of respondents thought the IRS was wrong in what it did, they split evenly about whether the agency's actions were illegal or just unethical.
Some 25 percent didn't think the IRS did anything wrong. Twice as many Democrats as Republicans thought the IRS acted appropriately.
The nationwide survey was conducted May 31 through June 4. The margin of error was calculated at 3 percent.
2014: The Year in Music [PHOTOS]