Schefter asked Olsen directly if he would consider holding out of training camp.
"I think it's early on right now. I think as the summer goes on and we get some feedback from the organization about how they feel, we'll make the best decision that we can," Olsen said.
"But I'd be lying if I didn't say that we feel very strongly about where we should be in that totem pole of the top tight ends. I'd be hard pressed to find anybody that could make a reasonable argument on the contrary."
He is due $6.5 million in each of the next two seasons, which could be considered a low wage for a tight end whom has made three consecutive Pro Bowls and has at least 77 receptions in each of those campaigns. The 10-year NFL veteran was the No. 31 overall pick in the 2007 NFL Draft. He played his first four season for the Chicago Bears before joining the Panthers in 2011. Olsen has been a huge part of the Panthers' offense since his arrival. He has had more than 100 targets in each of his six seasons with the Panthers as Cam Newton's favorite target.
Two league sources told the Charlotte Observer last week that Olsen wants a pay raise. He currently has a lower salary than Jimmy Graham, Travis Kelce, Jordan Reed, Rob Gronkowski, Zach Ertz and Charles Clay.
"In the ideal world, that'd [a new deal] be great," Olsen told the Observer. "I'm very confident on where I stand in the league and where I belong. Both productivity and all things considered, there's nobody that's been more productive or more consistent than I have. And I'll stand on that until the cows come home."
"You can make of that what you may," Olsen said. "I feel comfortable with where I belong, and hopefully other people do, too."
The Panthers inked Olsen to a three-year, $22.5 million contract extension in 2015. That deal included $12 million guaranteed.
Olsen told Schefter he could play 15 to 16 years. He is the only tight end in NFL history with three consecutive seasons with at least 1,000 receiving yards.