Joe Sherman was born on July 31, 1912, in Raleigh, North Carolina. He was the son of Franklin Jr. and Grace Berry Sherman. At the age of thirteen, he moved along with his family to Clemson, South Carolina, where his father had accepted a position as head of the Entomology and Zoology Department of clemson College. Sherman attended Calhoun-Clemson High School and Webb School in Bell Buckle, Tennessee. He graduated from Clemson College in 1934 with a BS degree in Arts and Sciences.

After graduation in 1934, he became the first director of Clemson's News Bureau, where he compiled sports information, produced motion picture and photographs to promote the school to prospective students, and revived alumni publication work after a twenty year hiatus.

In 1947, Sherman left Clemson for the University of Florida, where he was involved with alumni relations and the athletic department. In 1953, he and his family relocated to New York City, where he worked as general editor for the National Collegiate Athletic Bureau. In 1956, he returned to Clemson where he was the first director of public relations. In 1957 he was instrumental in producing and developing the annual homecoming event, "Tigerama."

Sherman played an important role in the successful integration of Clemson in 1963. He was sent by President R.C. Edwards to the University of Mississippi in order to gather information and learn from mistakes that had been made. Sherman then successfully developed and communicated rules for the news media and visitors to campus during Clemson's peaceful integration. In 1969, Sherman wrote the beloved essay "Something in These Hills." He retired from Clemson University in 1977.

Joe Sherman married his wife, Fay, in 1936. They had three children: Joe Jr., Walter, and Anne. Sherman died on November 4, 1990 and is buried at Cemetery Hill in Clemson, South Carolina.