Officials found the handgun in a book bag at KIPP Ujima Village Academy, a public charter school for grades 5 through 8, The Baltimore Sun reported.
Students told school officials they saw the gun Thursday morning and school police found it in the book bag, the newspaper said.
No one was injured in the incident, and the gun was not loaded, but juvenile authorities took the children into custody at the school in northwest Baltimore Thursday.
They were being held at the Baltimore City Juvenile Justice Center while an investigation is undertaken.
Jason Botel, executive director for KIPP Baltimore, which operates the school, said administrators discussed the incident with students and sent a letter to parents.
"Safety is KIPP's number one priority," the letter said. "We are grateful to the KIPP students who stepped forward and to the police for moving quickly to ensure KIPP Ujima could continue to operate safely."
School spokesman Michael Sarbanes said the students' names would not be released but said they were recent enrollees.
Ron Burgundy interviews Peyton Manning on SportsCenter
Texas principal bans speaking Spanish, stirs controversy