The former Bendigo Benevolent Asylum was established in 1857 and moved to its permanent home in 1860. The present complex was named the Anne Caudle Centre after a local doctor's wife who launched a public subscription fund to establish the original asylum. Originally intended as a home for the aged, infirm and destitute, the asylum has also housed an industrial school for neglected children and a lying-in (maternity) hospital for destitute women.

The original 1860 building was designed by architect Robert Love. Much of the subsequent nineteenth century design was undertaken by architects Vahland and Getzschmann. New wings were added to the original building in 1863, 1864 and 1870 and the portico was built in 1872. The gatehouse, designed by Getzschmann, was erected in 1862. In 1870-71 a kitchen and laundry were built at the rear of the main building. The Superintendent's residence was commenced in 1879.

The asylum, part of which was designated as an industrial school from 1868 to 1885, has a strong association with the history of child welfare and education in Victoria and illustrates the attempts by government to deal with the problem of destitute and neglected children.

The complex is a particularly large and rare surviving example of a number of benevolent asylums constructed throughout Victoria in the nineteenth century.

With the exception of the gatehouse and the former dining hall (now the concert hall), the interior of the buildings have been extensively altered.