Born in Pittsburgh, Pennsylvania,^[2] Fite grew up in Texas, where his family had moved early in his childhood.^[3] As a young man he attended evening courses in law for three years, before deciding not to pursue it as a career.^[2] At that point he moved east to study for the ministry at St. Stephen's College, a small Episcopal institution in Annandale-on-Hudson, in New York's Hudson Valley.^[3] Once there, Fite was drawn to the stage at the campus theater, and at the end of his third year he dropped out.

He joined a traveling troupe of actors, and later moved to Woodstock, where he performed with a local theater.^[3] According to an anecdote that his stepson, Tad Richards, relates, Fite discovered his passion for sculpting suddenly one day when, while sitting backstage during a performance, he absentmindedly pulled out his pocketknife and began whittling on a seamstress's discarded spool that had rolled under his chair.^[3]

A recognized sculptor, Fite was invited in 1933 to organize the fine arts program at his alma mater (St. Stephen's), which, in the three years since his departure, had affiliated with Columbia University and been renamed Bard College.^[2] Fite taught there until his retirement in 1969.^[1] He settled across the river at the Maverick art colony outside Woodstock, New York.

In May 1938, Fite purchased an idle bluestone quarry in Saugerties, New York,^[4] a 12-acre site formerly owned by the widow of the last quarrymaster. He designed, engineered and hand-built a fine wooden house at the edge of the quarry grounds, facing the Catskill Mountains, and settled there in High Woods, a rural hamlet within the township of Saugerties, which neighbors Woodstock. Over many years, he embellished his home's exterior with grand necklaces of quarryman's chains, and filled the interior and attached studio with murals, paintings and sculpture, going as far as whittling door handles of arched nudes, so that the building itself is now a museum of Fite's artwork.

That summer he was invited by the Carnegie Institute to do restoration work on ancient Mayan sculpture in Copan, Honduras.^[1] Fite was profoundly influenced by the art and architecture of the Maya, especially by their method of dry-stone construction.^[3] The next spring he began to organize the rubble scattered about the disused quarry, beginning what eventually became his life's work: a sculptured environment of terraces, alleys, ramps, steps and rain-fed pools which he would eventually name "Opus 40", as he estimated it would take him forty years to complete.^[3] Over the decades, Fite single-handedly moved and positioned stones, including at one point a nine-ton bluestone monolith, using ancient Egyptian methods of leverage and hoisting.^[2]

Harvey Fite died in May 1976 while at work on Opus 40,^[1] in the process of completing an attached open-air "theater" at the site's northwestern extreme; he was riding a power lawnmower at the time, and he fell into the quarry from a 12-foot precipice.^[1] He was 72 years old, and had worked alone on his magnum opus for the last 37 years of his life.^[1]