The fort is said to have been built by the Kolhapur Silahara chief Bhoja II. (1178-1193) of Panhala. About 1648 it is mentioned as being in the charge of a Bijapurmokasadar stationed at Wai. [Grant Duffs Marathas, Vol. I, 109.] In 1673 it was taken by Chhatrapati Shivaji Maharaj [Grant Duffs Marathas, Vol. I, 202.]. In 1701 Pandavgad surrendered with Chandan Vandan to Aurangzeb's officers. [Grant Duffs Marathas, Vol. I, 303.]. In 1713 during his flight from Chandrasen Jadhav the Maratha captain or Senapati, Balaji Vishvanath afterwards the first Peshva, being refused shelter by the Sachiv's agent at Sasvad attempted to cross to Pandavgad in the opposite valley. Closely pursued, he contrived to conceal himself until two Marathas, Pilaji Jadhav and Dhumal, then common cavaliers in his service, gathered a small troop of horse and carried him with great difficulty to Pandavgad where he was protected by Shahu's orders. Chandrasen demanded that Balaji should be given up and in case of refusal threatened to renounce his allegiance. Shahu refused to give up Balaji and sent orders to Haibatrav NimbalkarSarlashkar then at Ahmadnagar to march on at once to Satara. Meanwhile, Balaji was in Pandavgad surrounded by Chandrasen's troops. But hearing of Haibatrav's arrival at Phaltan about forty miles east, Chandrasen quitted Pandavgad and marched to Deur about fifteen miles to the south-east [Grant Duffs Marathas, Vol. I, 323.]. During Trimbakji Dengle's insurrection in 1817 Pandavgad was taken by the insurgents. It surrendered in April 1818 to a detachment of the 9th Native Infantry Regiment under Major Thatcher [Bombay Courier, 18 April 1818.].

The Pandavgad caves are situated on a small south-east projection of Pandavgad fort within the limits of Dhavdi village. On taking the path to Pandavgad and reaching the opening in the hills instead of turning up the shoulder of the hill to ascend the fort, the way to the caves goes straight on towards Dhavdi by a well defined footpath which skirts the face of the hill. The small spur with the caves is found at about a distance of 300 yards. The angle it makes with the main spur should be made for and about 200 feet up are the caves. The first is a flat roofed chapel or chaitya about twenty-one feet by seven and about twelve to fourteen feet high. An arched entrance blocked up with mud and stones leads to a relic shrine or daghoba four and a half feet in diameter and six feet high. Its capital (head portion) is lost. Close by is another cave seven feet square, also flat-roofed with an arched entrance and containing a mutilated stone instead of the daghoba and locally said to be a ling. It looks more like a daghoba, being fully three feet in diameter at the base and scarcely a foot at the top. East of Cave II is an eight-celled dwelling cave or vihar about thirty-five feet square and five feet high. The floor has been much silted up with earth brought in by rain water. The original height, as seen from the outside, was probably eight feet. The roof is flat and the rock overhangs four feet making a verandah with an entrance in its back wall about eight feet wide. The cells are two each on the east and west and four on the north, and there is a bed shelf all round. Five yards to the west is a rock-cut cistern six feet deep and nine feet wide holding no water. The Pandavgad caves are in good condition to this date and are visited by a number of people.