The land occupied by Young Street Terraces was originally part of the grounds of First Government House. In 1851, the land was granted to Municipal Council of Sydney who subsequently subdivided it for sale. Lots 1 and 2 were sold to OJ Caraher in 1866, then on to W Andrews in 1874, and to builder JW Walker later that year. Walker built the terrace houses circa 1874-5 and leased them as offices to the Mining Board and the Dept of Lands.

Between 1876-81, the Department of Mines opened a mining museum in two rooms on the ground floor of No. 38. In 1881, Walker sold the property to J Robertson. In 1884 the government resumed the property for offices. Various government departments continued to occupy the terraces until 1936. Between 1937-82 the terraces provided temporary accommodation for nurses from Sydney Hospital, with the offices converted into living quarters. The rear of the site was converted into a car park in 1961. In 1982, the Sydney Hospital relinquished the building which remained unoccupied after the completion of the Museum of Sydney and associated conservation work on the terraces.

In late 1997, the building was refurbished by the Dept of Public Works and Services to provide office accommodation for the Sydney Festival and the Historic Houses Trust. The property title was transferred to the Minister for the Arts in 1998.