Holloway was born in San Jose, California, the second of four boys. His family moved to the Blue Ridge Mountains of Georgia when he was two. In addition to his immediate family, Holloway has a large extended family that includes Baptist preacher Dr. Dale Holloway and author and World War II prisoner-of-war Carl Holloway. He is also the great-grandson of "Lula" Holloway, who wrote a book sometime in the 20th century about the Holloway family. Holloway grew up in the small North Georgia town of Free Home and attended Cherokee High School in Canton, Georgia. He developed an interest in movies at a very young age. He studied at the University of Georgia for a year, but left to pursue a career as a model.