Both countries have arrested a "substantial number" of fishermen in recent months for poaching, the official Sri Lankan government website NEWS.LK reported.
The release of the detained fishermen was a "goodwill" gesture agreed between Sri Lankan External Affairs Minister Gamini Lakshman Peiris and Indian External Affairs Minister Salman Kurshid during a recent telephone conversation, the NEWS.LK report said.
Sri Lanka released 20 Indian fishermen held in Malakam, near Jaffna in northern Sri Lanka, and India released the same number in Tamil Nadu state, the government statement said. A report by the Indian newspaper The Hindu said each country released 52 fishermen.
A representative of the the Indian rights group Alliance for the Release of Innocent Fishermen welcomed the releases by both countries, but appealed also for the release of the fishermen's boats, The Hindu reported.
Talks continue this week in the Indian capital New Delhi for more releases.
Sri Lanka's Minister of Fisheries and Aquatic Resource Development Rajitha Senaratne and representatives of the detained Sri Lankan fishermen are meeting with India's Minister of Agriculture and Food Processing Industries of India, Sharad Pawar.
The ministers also will discuss arrangements for meetings between the fishermen's associations of Sri Lanka and India.
Fishing rights and ship movements remain a delicate issue for both countries. Many of the fishing boats are small wooden vessels owned by families that depend on fishing for their livelihood.
Arrests of fishermen by one country's coast guard often create an outcry from fishing communities in the other country's coastal towns.
A highly patrolled area for coast vessels of both countries is Palk Strait, around 30 miles wide at its narrowest point between islands, belonging to India's Tamil Nadu state and the Mannar district of Sri Lanka's Northern Province.
Sri Lanka's navy denied allegations by Indian media in January 2012 that its ships attacked Indian fishermen and damaged their boats. The alleged attacks were reported to have happened near the Sri Lankan administered and uninhabited island of Kachchativu between India and northwestern Sri Lanka.
In February 2011, Sri Lankan police took into custody 24 Indian fishermen picked up by the navy on suspicion of fishing within Sri Lankan waters, bringing the number of Indian fishermen held at that time to around 140.
India's coast guard and Indian navy stepped up patrols in Palk Bay, the Gulf of Mannar and Palk Strait -- all of which separate India's southern Tamil Nadu state from northwestern Sri Lanka.