"What we've done today is taken a first step on defense relations between our two countries," Gillard said.
"It isn't fully normalizing defense relationships but it's opening the door."
However, Gillard said the arms embargo would remain for the foreseeable future.
Gillard said Australia has been "very admiring of the remarkable amount of progress ... and we are optimistic of further progress" made by Myanmar toward democracy and the improvement of human rights.
But she said there was ''much to do'' yet in Myanmar, formerly called Burma, a report by The Age newspaper said.
Thein Sein, a former military general in the junta that ruled Myanmar for decades, said he recognized the international community's concerns about human rights in his country but said Myanmar is making a ''transition to peace."
''What we are undertaking has no precedent in modern times,'' said Thein Sein, the first Myanmar leader to visit Australia since 1974.
Thein Sein -- a former junta prime minister from 2007 -- and his Union Solidarity and Development Party of mainly ex military officers, won a general election in November 2010.
Thein Sein's government began the release of several thousand political prisoners in late 2011.
In August Myanmar lifted a ban on more than 2,000 of its citizens, including journalists, allowing those living abroad to return home.
Since Thein's government took office in early 2011, it has been wooing Western business leaders intent on exploiting the country's natural resources, including mineral, offshore oil and gas and forestry resources.
Gillard also said Australia will boost economic ties with resource-rich Myanmar, especially in the area of mining, with the appointment of an Austrade commissioner to the country. She said the move is "all about facilitating business links."
She said Australia is "a great mining nation and we have a competitive advantage with it comes to working with (Myanmar)."
Australia will provide $20 million over two years for the first phase of the new Myanmar-Australia Partnership for Reform, she said.
Mark Wood, a senior Austrade official with experience in South Korea, Hong Kong and Thailand, takes up his post in Yangon, Myanmar's former capital, in May, a statement from Austrade said.
"Austrade's Myanmar office is part of the expanded diplomatic and commercial presence foreshadowed in the government's Asian Century White Paper," Austrade Chief Executive Officer Bruce Gosper said.
"It also reflects Austrade's increased focus on emerging markets around the world, including our new consulate in Bogota, Columbia and new trade commissioners in Accra, Ghana and Sao Paulo, Brazil."
Austrade's Myanmar mission will focus on agribusiness, infrastructure, energy and mining technologies and services, educational training and financial services.
Thein Sein, 67, has two more years to make good on promises of political, social and economic reforms, not to mention improving inter ethnic relations, before he faces the electorate again, in 2015.
In what would be the first real test of his democratic credentials, Thein Sein likely will be in a tough battle with on old adversary, democracy advocate Aung San Suu Kyi, chairman and member of Parliament for the National League for Democracy.
Suu Kyi and her NLD won a 1990 election but the military junta refused to hand over power.
NLD boycotted the 2010 election over technical issues and the fact that Suu Kyi remained ineligible to run because she was under house arrest.
In April last year, the NLD contested 44 seats in by-elections, winning 43 of them in a contest seen largely free and fair by observers, a report by ABC News said.
Celebrity Families of 2014 [PHOTOS]