Born in Cortona, Passerini was taken under the wing of the powerful Florentine Medici family, after his father, Rosado, was imprisoned for too openly supporting the Medici cause during one of the reversals of power in 15th‑century Florence. Silvio was raised and educated at the court of Lorenzo de' Medici and became very close to Lorenzo's son Giovanni whom he followed even to the battlefront where they fought side by side in France and were both made prisoners. As papal commissioner and envoy for Perugia and Umbria, Passerini amassed a considerable fortune.

When Giovanni became Pope Leo X in 1513, Silvio Passerini became cardinal-bishop of Cortona, with a diocese enlarged at the expense of the archdiocese of Florence and the diocese of Arezzo. He was made regent of Alessandro de' Medici, probably Giovanni's son, as lord of Florence in Giovanni's stead. A great period of wealth and power ensued: the papal historian Pastor noted 55 benefices for Cardinal Passerini recorded in Leo's official register. In Cortona, Cardinal Passerini directed his diocese from the Palazzone on the height above Cortona. Originally the 12th-century Palazzo del Capitano del Popolo, who represented the "tribune della plebe" (the "tribunes of the people"), in 1514 the palace gave way to Cardinal Passerini, who rebuilt it in Renaissance taste c. 1521‑27, and left it, as the Palazzone Passerini, to his heirs (the last of whom donated it, in 1968, to the Scuola Normale Superiore di Pisa, which now uses it to host prominent international meetings of mathematical researchers). It is richly frescoed with events of classical Roman history. While frescoing its chapel, the painter Luca Signorelli fell from the scaffolding and died. Many of the other frescoes are the work of Il Papacello and his assistants.

The Cardinal was a great Renaissance patron. He built three more personal villas: one in the commune of Bettolle, one in Petrignano and the third in Piazzano, the closest to his official residence. He recognized the talent of the sixteen-year-old Giorgio Vasari of Arezzo and supported him to study in Florence. At Florence he commissioned a tapestry from cartoons by Andrea del Sarto and Raffaellino del Garbo, which is conserved in the diocesan museum at Cortona. With the young Alessandro and Ippolito de' Medici in tow, he attended the first performance of Niccolò Machiavelli's comedy La Mandragola, Vasari related.

When the Medici fell in 1527, Silvio Passerini's loyalty toward them also forced him to flee from Florence and Cortona. He died at Città di Castello. His body was later transferred to Rome and buried at his titular parish, San Lorenzo in Lucina.

• Passerini's Palazzone at Cortona
• Miranda, Salvador. "PASSERINI, Silvio (1469-1529)". The Cardinals of the Holy Roman Church. Florida International University. OCLC 53276621.