Westerly police arrested Joseph Bernatche, of Portland, Maine, July 27 after he took the note with his phone number to the musician's home, RadarOnline.com reported Thursday.
Bernatche's lawyer, Michael Robinson, said his client was never on Swift's property.
"The security guards have a job to do and I don't blame them. They do what they think they have to do. In this case, I think they were just being overly zealous, overly cautious," Robinson said.
Robinson said Bernatche walked up the the gate at the home, handed the security guard the note, asking him to pass it on to Swift and walked away. The guard took the note, read it, wrote down Bernatche's license plate number and called police.
"[Bernatche] was driving away from Ms. Swift's property when he was stopped and arrested by the Westerly police," Robinson said.
Bernatche's son, Brian Bernatche, told the website he was unaware his father was arrested.
"He talks about Taylor all the time, but I can't believe he did this. I talked to him last night and he didn't even mention it!" he said.
Bernatche had a pre-trial conference Friday, RadarOnline said.
Celebrity Families of 2014 [PHOTOS]