The bipartisan cookout, or "brat summit," was meant to include all legislators and staff members, with food and drinks to be donated, The Milwaukee Journal Sentinel said.
"I am hopeful this get-together will forge relationships that will make it easier to work together to help create jobs," Walker said in a statement.
However, three lawmakers declined their invitations to the post-partisan picnic.
"I'm not going because I think that there's no trust now," said state Rep. Helen Roys, D-Madison. "There's no sincerity in his approach … You know, let's actually do something bipartisan first, then once we actually have a little sense that this is sincere … then let's go to the mansion and celebrate that."
Roys' opponent in the Congressional race, Rep. Mark Pocan, D-Madison, agreed in a statement:
"I refuse to be involved in what appears to be a media stunt -- a true bipartisanship will only occur through concrete action made by Governor Walker."
Rep. Steve Nass, R-Whitewater, also declined to go, saying it was because of this recent comment by Democratic Party Chairman Mike Tate: "Scott Walker will see the inside of a jail cell before he sees the inside of another term."
Walker's narrow win in the recall election last week made history as the first time a U.S. governor survived a recall. In the two previous contests, the incumbents were ousted.