James L. Caldwell was born on October 22, 1933 to Roger and Frieda Caldwell near the town of Franklin, Texas in Robertson County. James was raised along with his two brothers, Harold and Dale, in Robertson County. James graduated from Franklin High School and later served in the U.S. Marine Corps during the Korean War. After honorably discharging from the Marine Corps, James attended Abilene Christian College on the GI bill and earned hours towards a degree in Education. While attending ACC, James worked summer construction for HB Zachary and was quickly noticed by the management as a future leader and was offered a Foreman's position at a young age. It was in construction that James found a career and eventually owned and operated his own construction business. James would say his greatest achievement occurred when he married the love of his life, Martha Jean McGauhey on August 05, 1966. James and Martha had two sons, James L. Caldwell Jr. and Roger Thomas Caldwell whom they raised in the Aldine area of North Houston. James loved his family dearly and they spent many hours bass fishing and watching football, two of James' favorite pastimes.