Rogers was born in Latrobe, Pennsylvania, a town located 40 miles (65 km) southeast of Pittsburgh. He was born to James and Nancy Rogers; he spent many years as an only child. Early in his life he spent much of his free time with his maternal grandfather, Fred McFeely, and had an interest in puppetry and music. He would often sing along as his mother would play the piano and eventually, at the age of 5, began to play the piano as well.

Following secondary school, Rogers studied at Dartmouth College in Hanover, New Hampshire, between 1946 and 1948 before transferring to Rollins College in Winter Park, Florida, where he received a BA in music composition in 1951.

At Rollins, Rogers met his wife, Sara Joanne Byrd, an Oakland native, whom he married on June 9, 1952. They had two children, James (born in 1959) and John (born in 1961), and three grandsons, the third (Ian McFeely Rogers) born 12 days after Rogers' death. In 1963, Rogers graduated from Pittsburgh Theological Seminary and was ordained a minister in the Presbyterian Church (U.S.A.). Scholastically, he went on to garner 40 more honorary degrees throughout his life. Rogers was also red-green color blind and a vegetarian. He swam every morning, and neither smoked nor drank.