Carolina held the No. 1 overall pick and selected Auburn quarterback Cam Newton.
"I thought at that time I could be No. 1, but (Carolina) didn't see that way, they took Cam Newton and they won a lot of games," Miller said.
General manager Marty Hurney was fired after the Panthers started 1-5 in 2012.
Miller was the No. 2 pick and John Elway's first as a general manager. That draft turned out to be a defining one for the Broncos, who landed five players who became starters in that draft. Newton is the lone player drafted in 2011 still around in Charlotte.
Miller mocked Newton's Superman celebration when he sacked Newton in a game in Nov. 2012, their only career meeting.
Miller said he is forever grateful that Elway used his first ever draft pick to draft the pass rusher out of Texas A&M. Then Houston Texans coach Gary Kubiak said he didn't do a lot of work on Newton leading up to the draft but know Miller all too well since he played at Kubiak's alma mater. The Texans drafted 10th and somehow wound up with All-Pro defensive end J.J. Watt.
"I knew Cam's ability and what he stood for. I knew Von very well because he was right down the street from us," Kubiak said Monday. "When you look back on it now and who's lining up and playing here in a couple of weeks, that tells you it was a pretty darn good draft for both teams."
Newton accounted for 50 touchdowns to win the Heisman Trophy and a national title at Auburn. He was drafted by a team that had just gone 2-14 with the worst offense in the NFL, and a significant issue was quarterback play from Matt Moore and then-rookie Jimmy Clausen.
Coach Ron Rivera called Newton "the most researched player I've ever been around" the day after drafting Newton No. 1, but Hurney was not dissuaded by one existing public opinion that the 6-5, 250-pound quarterback came with what one draft analyst called "a significant bust factor."
Rivera, a defensive coordinator from the San Diego Chargers hired to replace John Fox as head coach in Carolina, had circled several defensive players that might help the Panthers. Julius Peppers left for the Bears in free agency and pass rush was often mentioned during the draft process. Some teams felt Newton could be a risk.
The Auburn offense was simplified -- in Newton's words, he "looked to the sideline, got a play and went fast" and resembled nothing like any NFL system -- but not everyone felt that was a fair criticism. Considering Newton had 30 touchdown passes and 20 rushing scores, he was flawlessly executing what he was asked to do with the ball.
There were skeletons for scouts and general managers such as Hurney and Elway to weigh. Newton began his college career at Florida, transferred to a junior college amidst academic dishonesty allegations and two years later was investigated by the NCAA. Accusations surfaced that his father demanded a $180,000 from a Mississippi State booster to bring his son to that school.
The NCAA ruled that Newton wasn't aware of his father's pay-for-play ploy.
There were other choices beyond Miller for the Panthers to consider if they weren't sold on Newton - LSU cornerback Patrick Peterson, Alabama defensive tackle Marcell Dareus, Georgia wide receiver AJ Green and Watt chief among those alternatives. All four, of course, became Pro Bowl performers after becoming top-10 draft picks.
"He has special qualities," said Hurney in 2011. "We've had a strong feeling for some time that he was the person who we felt could come in and have the most impact on our football team."
Had Hurney listened to offers to move down in hopes of having a pair of first-round picks that year, consider the other quarterbacks drafted in the first round behind Newton were Blaine Gabbert (10th, Jaguars), Christian Ponder (12th, Vikings). Andy Dalton (35th, Bengals) and Colin Kaepernick (36th, 49ers) were second-rounders.
The 49ers, then coached by Jim Harbaugh, looked closely at every quarterback in 2011. Harbaugh referred to Newton as "plutonium-grade raw material."
Harbaugh said, in vintage Harbaugh speak, before the 49ers played the Panthers in 2013 that Newton is truly one of a kind. Broncos cornerback Aqib Talib seconded that opinion on Monday.
"Uniquely talented," Harbaugh said. "My son, Jack Harbaugh, who's just over 14 months, he's outside of the curve - he's outside of the 100th percentile now. He's above the 100 percentile. He's big. He's big. Growing very well. Cam Newton would be further outside of the graph. He's in a world by himself. Tremendously talented."
Here's what Elway thought of Newton in 2011.
"It's very rare that you have three guys who are at least 6-4, 230 and run a 4.6," Elway said one month before the 2011 draft, when the Broncos were coming off of a 4-12 season and Kyle Orton and Tim Tebow comprised Denver's QB room. "You don't have to be that way. You look at Peyton Manning (then with the Colts) and Tom Brady (Patriots), the success they've had. But I think it's getting harder and harder to play that position and not have mobility."
Hurney also drafted Oregon running back Jonathan Stewart (first round, 2008) middle linebacker Luke Kuechly (first round, 2012) and cornerback Josh Norman (fifth round, 2012) and deserves credit for establishing the core of a championship roster. By the time the Panthers turned in the card at Radio City Music Hall in New York on April 28, 2011, the strong majority of the franchise was in agreement: Newton was the team's future.
"I started with eight guys in my mind," Rivera said in 2011. "It became seven and then four, and I tried to ride that for as long as I could. But everything kept pointing back to Cam."