Millers Point originally neamed Cockle Bay Point was known as 'Jack the Miller's Point', named after an ex-convict, John Leighton. The area at the north end of Kent Street and the western end of Windmill Street was known as 'The Quarries' and supplied a large part of the early stone for Sydney.

The land on which the Hotel is now situated was orginally part of the Crown Grant to the plasterer William Wells dated 14th May 1836 and part of the Grant (in trust) to Richard Driver dated 30th November 1840. A census of 1834 however, indicates that there were three William Wells living in Kent and Argyle Streets during the 1830s.

The Lord Nelson was constructed during the late 1830s by William Well. Records indicate that Wells may have been a plasterer by trade, however this cannot be confirmed by 180s directories. The architect was Michael Lehane.

Wells is believed to have lived on the opposite corner to the Sailor's Return in the present day Lord Nelson until 1842 when the liquor licence for the hotel was granted. The sandstone blocks are believed to have been quarried from the area at the base of Observatory Hill. The Lord Nelson is the oldest working licensed hotel in Sydney, dating from the time of licence in June 1842.

The Times of November 7th 1805 contains an account of the victory at the Battle of Trafalgar, in which Vice Admiral Lord Viscount Nelson was killed. The battle took place in Cadiz, and the enemy comprised a contingent of thirty three Spanish and French ships under Admirals Villeneuve and Gravina.

Sands Directories dating back to 1858 indicate that the Lord Nelson Hotel was occupied on and off by Charles J Sounders between the years 1860 and 1866. In 1867 it appears that the property was repurchased by William Wells, who had occipied the building from the time of its construction until 1858. He remained in the building until 1870 before John Henwood and Alfred C Wells, possibly a son or relative of William took over. In the following years leading up to the turn of the century, the building was occupied by numerous people. Most noteable for their lengthy occupation were Robert Drysdale, John H Ferguson, William Adams and Patrick Powell.

The property was transferred to the Sydney Storage Company in 1891 and part of it was leased back to Patrick Powell in 1893. A further lease took place in 1900, by the Sydney Storage Company to L H G Oswald. The Oswalds were to construct the Oswald Bond Stores on either side of the building.

The propoerty was resumed by the Minister of Public Works in 1901 and vested in the Sydney Harbour Trust Commissioners in 1911. There were numerous proprietors of the hotel afetr the turn of the century, particularly during the period from 1901 to 1911. from 1911 to 1917 Michael Lehane occupied the building, followed by Terence O'Brien from 1918 to 1928.

On March 15th 1922 approval was given for further alterations to the building. The name of the architect of these modifications cannot be ascertained. It is possible, however, that the work included a remodelling of the second floor accommodation spaces.

In 1931 Tooth and Co Ltd who were leasing the hotel, reported considerable losses in trade due to the high rent payable to the Sydney Harbour Trust. The General Manager of Tooth and Co. wrote to the Solicitor for the Sydney Harbour Trust Commissioners stating that for the last 12 months trade had been continuing to decline. As a result the terms of the lease were renegotiated.

In 1932 the premises were inspected by the District Licensing Inspector Edward G Allen. The following alterations were required to bring the premises up to the requirements of the Liquor Act No.42 of 1912; kitchen and dining room to be made fly proof, floor covering to be replaced in some rooms, electric lights to be repaired to working order and maintained and certain beds and blinds to be replaced.