Brown, 39, agreed to a five-year contract with an option for a sixth season. He replaced Sean Kearney, who was fired March 30 after just one year as the Holy Cross coach. The Crusaders were 9-22 last season.
Brown was 95-120 in seven seasons at Mount St. Mary's but the team was 54-44 the last three seasons, including a 2008 trip to the NCAA tournament.
"I am extremely excited about the opportunity to come to Holy Cross and to try to lead the program back to the top of the Patriot League," Brown said. "My family and I are really pleased about the move, and I cannot wait to get started."
"We spent a lot of time on this search," said Holy Cross Athletics Director Richard M. Regan, Jr. "We were very focused on hiring someone with head coaching experience and we were very impressed with Milan both as a coach and a person."