Vernon had two daughters, Margaret (born c. 1540) and Dorothy (c. 1545 – 1584) by a first wife, Margaret, the daughter of Sir Gilbert Talbois.[3] His second wife was Maude, the daughter of Sir Ralph Longford, whom he married in the 1550s. In 1558, Margaret married Sir Thomas Stanley, a son of the 3rd Earl of Derby. Dorothy fell in love with John Manners (c. 1534 – 4 June 1611), the second son of Thomas Manners, 1st Earl of Rutland. Dorothy and John Manners were married in 1563;[4] there is evidence that Manners's brother-in-law, George Talbot, encouraged the match. Nevertheless, a legend grew up in the 19th century concerning a supposed elopement between Dorothy and John, and this legend has become the source of many novels, dramatisations and other fictional works.[5] If indeed the elopement happened, the couple were soon reconciled with Sir George Vernon, as they inherited Haddon Hall and half of the estate on his death two years later, and John was Sir George's executor.[6][7]
Vernon died in 1565. He was buried in the parish church of Bakewell.[8] Haddon Hall remains in the Manners family to the present day.[9]