"They want to make a joke out of how hard we work," West told the audience at the Made in America Music Festival in Philadelphia on Saturday.
"It's all fine and all funny and everything, but don't distract from our vision. It's not a joke what we do up here. This music that we do is not a joke. What we do culturally is not a joke," he continued.
Pharoah -- in character as West -- told the audience at the VMAs on August 24 that he was supposed to introduce them to the Artist to Watch, "but there's really only one artist to watch. Me."
"This is a special night. But not as special as me and Kim's wedding," Pharoah continued. "I wore a cape! Y'all probably seen us in the "Bound" video, like [on a motorcycle]...Me and Kim are bigger than the Clintons fam. We so next level, we named our daughter North West after an airline that no longer exists."
While Kardashian and her half sisters Kendall and Kylie Jenner were photographed laughing throughout the performance at the event, West took issue with the remarks and felt he needed to clear the air with Pharoah.
"I called Jay Pharoah right after the MTV VMA Awards. I said, 'I appreciate your show, but let me tell you about my story. Let me tell you about what I went through to get to that position,'" West said on Saturday.
"So we ain't gonna have no black comedians going up stage spoofing people that's working hard to open doors not only for black people, but any creators, anybody that wants to add a contribution to the world."
Pharoah has yet to comment on the rapper's call or his remarks.