Many prisoners died while working to quarry the blocks of stone necessary to build Portland's naval breakwater. Later, during the 1870s, deaths within the prison were recorded at nearly one per week.

On September 25th 1863 George Rawlinson and 269 other convicts found themselves on The Lord Dalhousie en-route to the Swan River Colony

The 912 ton ship was built at Sunderland in 1847. It was employed as a convict transport for Western Australia and left Portland, England on September 25, 1863 bound for the Swan River Colony. She carried the twenty ninth of 37 shipments of male convicts destined for Western Australia. The voyage took 90 days and the Lord Dalhousie arrived in Fremantle on December 28, 1863 with 89 passengers and 270 convicts.

Once in the Colony things appear to have gone a lot better for George. He was married after only three and a half months and received his Ticket of Leave in 9 months.

A ticket of leave was a document of parole issued to convicts, since 1853, transported from the United Kingdom who had served a period of probation, and had shown by their good behaviour that they could be allowed certain freedoms. Once granted a ticket of leave, a convict was permitted to seek employment within a specified district but could not leave the district without the permission of the government or the district's resident magistrate. Each change of employer or district was recorded on the ticket. They could also acquire property. 

Church attendance was compulsory, as was appearing before a Magistrate when required.

Ticket-of-leave men were permitted to marry or to bring their families from Britain, and to acquire property, but they were not permitted to carry firearms or board a ship. They were often required to repay the cost of their passage to the colony.

A convict who observed the conditions of his ticket-of-leave until the completion of one half of his sentence was entitled to a conditional pardon, which removed all restrictions except the right to leave the colony.

While working as a servant at Perth in 1864 he had met Irish immigrant Margaret Coghlan who arrived on the "West Australian" (a bride ship) in 1859.

They were married in Perth Catholic Pro-Cathedral on the 18th of April 1864 and their first child Rosa was born at Perth in 1865. He received  his Conditional Pardon a year later.

Conditional Pardons freed convicts and were granted on the condition that they did not return to England or Ireland. Original copies of the pardons were sent to England and duplicates remained in Australia. Copies were also given to convicts as a proof of pardon. It appears that within less than two years in Australia he was almost a free man.

When obtaining a Certificate of Freedom in 1868 he moved his family to South Australia and effectively broke the connection with his convict past.