Yarrowsburg sits at the foot of Elk Ridge mountain on the west side of Pleasant Valley, stretching along Reed, Kaetzel and Yarrowsburg roads in southeastern Washington County. Until the late 19th century, Elk Ridge was wholly owned by the Antietam Iron Works. The forest was used as fuel for the furnace and forges after being converted to charcoal. After being harvested and especially after the iron works went bankrupt in the 1850s, the mountain land was sold off in multi-acre lots. Many of the properties in Yarrowsburg are on former iron works property. The village is named for Aquila and Polly Yarrow, free African Americans who bought a small log house on 1 1/4 acres of land in the southeast corner of Yarrowsburg and Reed roads in 1825. The Yarrows were one of several free black families living in Yarrowsburg at the time. It is thought that Aquila worked as a collier for the iron works.^[5] Aquila died unexpectedly in 1832, but Polly lived at the village named for her until November 21, 1885. This was also the site of the only (historical) crossing of Elk Ridge mountain through the steep and rocky Solomon's Gap. Elk Ridge can only be passed otherwise by circling around its north or south ends via Trego or Sandy Hook roads. The road through Solomon's Gap was abandoned in the early 20th century and is no longer a through road. Solomon's Gap would play a role in the strategy of the Civil War battles in the area.^[6] Yarrowsburg is today a rural village of single-family homes.

According to the U.S. Census Bureau, the community has an area of 0.227 square miles (0.59 km²), all land.^[4]