Nelson's assistant, Avery Johnson, moves up on an interim basis.
Nelson, 64, said he is stepping down for family and other reasons, but will remain with the Mavericks as a consultant.
Nelson has a 339-251 record in eight seasons with Dallas. He has coached in the NBA for more than 26 seasons, with a career record of 1,190-880. He is one of just three coaches in the history of the NBA with 1,000 victories.
Nelson has 42 years of NBA experience as a player, coach and general manager. He is the second winningest coach in NBA history, trailing only Lenny Wilkens. Nelson has also coached Milwaukee, Golden State and New York.