Nelson's assistant, Avery Johnson, moves up on an interim basis.
Nelson, 64, said he is stepping down for family and other reasons, but will remain with the Mavericks as a consultant.
Nelson has a 339-251 record in eight seasons with Dallas. He has coached in the NBA for more than 26 seasons, with a career record of 1,190-880. He is one of just three coaches in the history of the NBA with 1,000 victories.
Nelson has 42 years of NBA experience as a player, coach and general manager. He is the second winningest coach in NBA history, trailing only Lenny Wilkens. Nelson has also coached Milwaukee, Golden State and New York.
McPhee, Cokas 'working on their marriage' after affair
Astronomers offer more expansive view of universe