The North Korean leader, who last met with Moon during an informal summit at the truce village of Panmunjom, suggested in a letter made public on Thursday the timing is not right for a visit, South Korean news service MoneyToday reported.
"Now is not the time for the leaders of North and South to meet," Kim said.
Kim took issue with "unfavorable public opinion" about North Korea in the South in his letter to Moon, according to local television network KBS.
Pyongyang's state-controlled news agency KCNA issued a statement on Thursday, claiming Moon had requested Kim send a special North Korean envoy if Kim is unable to attend.
"Is it really time for the North and South to meet?" KCNA said. "None of the promises made at Panmunjom, Pyongyang or Mount Paektu have been realized. This is worse than not holding a North-South summit."
KCNA also called for the South's "understanding" as to why Kim could not go to Busan.
The presidential Blue House confirmed Thursday Moon sent the invitation to Kim in response to Kim's condolences for the death of Moon's mother in October.
North Korea has rejected talks with the South and the United States after demanding all sanctions be lifted during Kim's summit with U.S. President Donald Trump in Vietnam in February.
North Korea has also rejected offers of resuming tourism to Mount Kumgang and continuing operations at the Kaesong Industrial Complex.
South Korean television network JTBC reported Thursday the North had rebuffed proposals to exempt Mount Kumgang and Kaesong from sanctions. In Vietnam, Kim had instead asked for all sanctions to be removed.
Pyongyang is under embargoes for nuclear and weapons-related activity.