On February 15, 1755, Lord John arrives in Ardsmuir, Scotland, to replace Colonel Harry Quarry as governor of Ardsmuir Prison. He was sent there after his involvement in a near-scandal.
When he learns from Quarry that one of the prisoners under his care is Jamie Fraser, Grey immediately relives the mortification of their encounter nearly a decade before. Despite Quarry's suggestion that Grey meet with Fraser, whom the Jacobite prisoners consider their leader, to discuss matters concerning the prisoners' welfare, Grey stubbornly resolves not to face Fraser again.
This notion becomes short-lived when Grey receives word of a man in the nearby village raving about gold in a mixture of French and Gaelic. Grey realizes he must seek Fraser's assistance, as Fraser is the only man in the prison who speaks both languages, and would have no way of acting on the information obtained. There is also the tantalizing idea that if he, Grey, were able to locate the fabled Frenchman's gold meant for Charles Stuart during the Rising and hand it over to the Crown, he might escape his banishment sooner. Though Fraser agrees to translate, the dying man's words make little to no sense, and he dies early the next morning. Thereafter, Grey finally heeds Quarry's advice and has Fraser summoned to meet with him in his quarters every month, with the endgame of discovering more information about the gold.
The two men, though their acquaintance is necessarily formal, develop a certain rapport – they recognize each other as fellow soldiers, and both enjoy discussing literature and playing chess. However, their fragile bond is shattered when Grey, who has found himself attracted to Fraser and falling in love with him, makes a tentative move on the prisoner. Fraser rejects him utterly, and severs all contact with him. Later, Fraser takes the blame for the presence of an illicit bit of Highland tartan in the prison block, and Grey has no choice but to have Fraser flogged as punishment.
Upon the completion of the fortress renovations at Ardsmuir in 1756, Grey is charged with transporting the prisoners to the American colonies. In Fraser's case, however, he makes an exception, and arranges to have him paroled at Helwater, to be quartered with the Dunsany family, who have a long acquaintance with Grey. Throughout Jamie's parole, Lord John visits Helwater periodically to ensure Fraser's welfare.
In September 1764, Lord John tells Jamie he will marry Isobel Dunsany and become William's stepfather. Jamie offers his body to Lord John, but he declines. Instead, with gratitude, Jamie kisses him on the mouth.
In early 1767, Lord John meets Claire again on a ship to the West Indies, though he does not recognize her as Jamie's wife; he only knows her as Mrs. Malcolm and does not see her in the daylight. Later, after Claire has escaped the ship and Lord John has assumed his new position as governor of Jamaica, Lord John meets Jamie at a ball for his arrival as governor, and finally sees Claire for who she really is. During the ball, John meets with Jamie in private to give him a miniature portrait of his son, William, and Claire observes them, unseen, as they embrace with great feeling. After his initial shock at discovering her true identity, John talks privately with Claire and explains how Jamie came to have a son and how he, John, came to be stepfather to the boy.
Lord John offers the use of his pinnace to Jamie and Claire as they search for their nephew, Young Ian.