The King estate filed the suit Wednesday, the 50th anniversary of the 1963 March on Washington and King's "I have a dream" speech, Courthouse News Service reported.
In the lawsuit, the brothers charge the center has been careless in its handling of King memorabilia, leaving historic documents and items at risk of fire, water, theft, mildew and mold.
Attempts to solve the problem by working with Bernice King have failed, the lawsuit says, and there has been a "total breakdown in communication and transparency."
The King estate sent a 30-day notice in August to the center, notifying it that the licensing agreement for King memorabilia was being terminated. The letter said the center could avoid that by placing Bernice King on administrative leave and taking Andrew Young and Alveda King, Martin Luther King's niece, from the board.
The estate is seeking a court order barring the center from making use of the memorabilia once its license expires.
Celebrity Breakups and divorces of 2014 [PHOTOS]