After his first encounter with Lord John Grey in the Carryarick Pass, which continues to be a source of mortification to Grey, he does not meet the man again until he is appointed the new governor of Ardsmuir Prison, where Fraser is a captive of war, though still a natural leader among the other Jacobite prisoners. Though they develop a mutual liking for one another through their monthly discussions of literature and chess matches, Grey's feelings far surpass those of a friend, let alone a soldier's for his prisoner, and makes the mistake of acting on his attraction. Fraser rejects him completely, and their relationship is shattered.
When Grey arranges for Fraser to serve his parole in England, rather than be transported to the colonies, Fraser is deeply suspicious and refuses to interact with Grey beyond the barest minimum. Still, Grey returns to London and does his best not to think of Jamie Fraser, with varying degrees of success. Upon receiving word of Geneva Dunsany's death in childbirth in early 1758, Grey travels to Helwater to pay his respects, and draws certain conclusions when he notices Fraser's thinly veiled grief.
On another visit, while in pursuit of information about extant Jacobites, Grey also asks Fraser's advice on the matter of his step-brother – and lover – Percy Wainwright, who faces court-martial and possible execution for the crime of sodomy. Grey's sense of honor, he explains, could not abide his allowing Wainwright to be punished for a crime he, Grey, is also guilty of. Fraser, disgusted by this revelation, scornfully dismisses Grey's dilemma along with the notion that men can love each other, as a man may love a woman. After Fraser suggests that Grey's predilections extend to molesting young boys as well, Grey swears he would challenge Fraser to answer for that were the other man armed. Fraser retorts that Grey could not master him, and, furious, Grey assures Fraser that, should he wish it, he could take Fraser to his bed and make him scream. Fraser's reaction is instantaneous and violent; Grey dodges the blow and escapes, though not before seeing in Fraser a devastating vulnerability, and realizes that Fraser must have been victim to some similar threat, and worse.
Jamie does not see Grey again for nearly two years. In spring of 1760, Grey's brother Hal summons Fraser to London for his assistance in deciphering a message written in Erse, a then-common term for the Gaelic language.
In September 1764, Lord John tells Jamie he will marry Isobel Dunsany and become William's stepfather. Jamie offers his body to Lord John, who declines. Jamie kisses him.
In February 1767, at the governor's mansion, Claire sees Jamie leave the main event to speak privately with Lord John Grey, where Lord John gives him a portrait of young William, and is shocked to see the latter's look of longing toward Jamie as they embrace.
While traveling during October 1768, to John's late wife Isobel's plantation Mount Josiah in Virginia, John makes the side trip with William to Fraser's Ridge, wanting Jamie to have a chance to see his son in person.
In 1769, when John receives word from Jamie that he is in search of a man named Roger Wakefield, John utilizes his connections to help find the man. He travels to River Run plantation in early 1770, where he meets Jamie's daughter Brianna and, despite her attempt to blackmail him into marrying her, he agrees to announce their engagement, both out of a sense of obligation to her father, as well as to call off Brianna's other unwanted suitors.
John maintains a correspondence with Jamie, helping him whenever and however he can, be it looking for Stephen Bonnet, sending advance funds based on future sales of gemstones, or obtaining various items such as oil of vitriol for Claire or white phosphorous for Brianna.
Even when it becomes clear that John and Jamie will be on opposite sides of the coming rebellion, there is little that John wouldn't do for Jamie. In July 1776, without hesitation, when Jamie says he is in need of gemstones, John gives him the sapphire ring that once belonged to his first love, Hector. When his step-brother and ex-lover Percy Wainwright brings up Jamie and William, John is quick to tell him to stay away from them both.
In April 1778, when John received word that the Euterpe sank with Jamie on it, he was truly devastated. In his mourning, John did what he thought would be one final service to Jamie: marrying Jamie's widow, Claire, to protect her and the rest of Jamie's family from Captain Richardson.