Holdsworth was adopted at the age of 3 after a short stay in foster care and moved to Wilbraham, Massachusetts.[1] He was enrolled in gymnastics by his mother and didn't like the sport until he began get good at it.[1] He attended Minnechaug Regional High School and won the state prep all-around title twice.[1]
Holdsworth was recruited for his gymnastics ability and was offered scholarships from UCLA, Iowa, Illinois, Penn State, Ohio State, Temple, and Cal State Fullerton.[1] He chose to attend UCLA because of their Olympic pedigree, with alumni Peter Vidmar, Mitch Gaylord, and Tim Daggett having just recently won gold medals at UCLA's Pauley Pavilion in the 1984 Summer Olympics.[1] Holdsworth placed first in the country on the pommel horse at the 1986 NCAA Men's Gymnastics Championships, the first freshman in NCAA history to do so.[2][3] He later won a team NCAA national championship with the Bruins in 1987.[4] He was a nominee for the 1989 Nissen-Emery Award.[5]