After his first encounter with Lord John Grey in the Carryarick Pass, he did not meet the man again until he was appointed the new governor of Ardsmuir Prison, where Jamie was a captive of war, though still a natural leader among the other Jacobite prisoners. They developed a mutual liking for one another through their monthly meetings, during which they discussed the welfare of the prisoners but also talked of literature and played chess. Their relationship became irrevocably altered, however, when Grey's feelings surpassed those of a friend, let alone that of a prison governor for one of his charges, and he made the mistake of acting on his attraction. Jamie rejected him completely, and their relationship was shattered.
When Grey arranged for Jamie to serve his parole in England, rather than be transported to the colonies, Jamie was deeply suspicious and refused to interact with Grey beyond the barest minimum. In early 1758, Grey traveled to Helwater for Geneva Dunsany's funeral, and during his stay encountered Jamie in the chapel at night, apparently holding vigil next to Geneva's coffin.
On another visit, while in pursuit of information about extant Jacobites, Grey also asked Jamie's advice on the matter of his step-brother – and lover – Percy Wainwright, who faced a court-martial and possible execution for the crime of sodomy. Grey's sense of honor, he explained, could not abide his allowing Wainwright to be punished for a crime he, Grey, is also guilty of. Jamie, disgusted by this revelation, dismissed Grey's dilemma along with the notion that men can love each other, as a man may love a woman. After Jamie suggested that Grey's predilections extended to molesting young boys as well, Grey swore he would challenge Jamie to answer for that insult, were the other man armed. Jamie retorted that Grey could never master him, and, furious, Grey assured him that, should he wish it, he could take Jamie to his bed and make him scream. Jamie's reaction was instantaneous and violent; Grey dodged the blow and escaped, though not before seeing in Jamie a devastating vulnerability, realizing that Fraser must have been victim to some similar threat, and worse.
Jamie did not see Grey again for nearly two years. In spring of 1760, Grey's brother Hal summoned Jamie to London for his assistance in deciphering a message written in Gaelic. Jamie assists Lord John in tracking down Gerald Siverly, against whom Grey and his brother have ample evidence of corruption. During their journey to Ireland, they began to repair the damage to their relationship, even while Jamie resented the Duke of Pardloe's high-handed use of him. Back in London, Jamie acted as second in Lord John's duel against Edward Twelvetrees. When Jamie returned to Helwater, Grey offered an olive branch in the form of speaking aloud a chess move, harking back to the early days of their friendship, and Jamie responded in kind.
In September 1764, Lord John told Jamie he intended to marry Isobel Dunsany and become William's stepfather. Jamie offered his body to Lord John, who declined. Jamie kissed him.
In February 1767, at the governor's mansion, Claire saw Jamie leave the main event to speak privately with Lord John Grey, where Lord John gave him a portrait of young William. Claire was shocked to see the latter's look of longing toward Jamie as they embraced.
In October 1768, while traveling to John's late wife's plantation Mount Josiah in Virginia, John detoured with William to Fraser's Ridge, wanting Jamie to have a chance to see his son in person.
In 1769, when John received word from Jamie that he was in search of a man named Roger Wakefield, John utilized his connections to help find the man. He traveled to River Run plantation in early 1770, where he met Jamie's daughter Brianna and, despite her attempt to blackmail him into marrying her, he agreed to announce their engagement, both out of a sense of obligation to her father, as well as to call off Brianna's other unwanted suitors.
John maintained a correspondence with Jamie, helping him whenever and however he could, be it looking for Stephen Bonnet, sending advance funds based on future sales of gemstones, or obtaining various items such as oil of vitriol for Claire or white phosphorous for Brianna.
Even when it became clear that John and Jamie would be on opposite sides of the coming rebellion, there was little that John wouldn't do for Jamie. In July 1776, without hesitation, when Jamie said he was in need of gemstones, John gave him the sapphire ring that once belonged to his first love, Hector. When his step-brother and ex-lover Percy Wainwright brought up Jamie and William in conversation, John was quick to tell Percy to stay away from them both.
In April 1778, when John received word that the Euterpe sank with Jamie on it, he was truly devastated. In his mourning, John did what he thought would be one final service to Jamie: he married Jamie's widow, Claire, to protect her and the rest of Jamie's family from Captain Richardson.