Around 1800 the Parish House, originally called Highcombe, was built in Regency style. It is the sole survivor of several villas built in this style in the area.
One of the first residents was Baron Sir William Congreve. He was in charge of the Royal Military Laboratories at Woolwich and an inventor. Among his inventions was the Congreve Rocket therefore he is considered the father of modern rocket technology. He also invented decimal currency and an efficient steam engine. King George IV frequently visited and partied here.
The house then had a series of residents with links to the Military Academy in Woolwich, until it was bought by Sir William Barlow, an eminent railway Civil Engineer who is famous for investigating the Tay Bridge disaster and designing the replacement Tay Bridge. His research contribute to redesigning the Forth rail bridge, and completing Brunels Clifton Suspension bridge. He also designed and built St Pancras station, at the time the largest spanned roof in the world, and the Midland Railway. He collapsed and died in the entrance hall of the house on November 12th 1902 aged 89.
In 1903 on the 18th July 1903 the Assumptionist Sisters, expelled from Bordeaux, France, bought the house as a refuge and moved in. Their first act was to set up a small chapel in the current parish office, and the first Mass was said by Fr Benedict Caron AA on a temporary altar until a proper one arrived from Bordeaux. The following day, with the sisters and 5 parishioners. within a few weeks over 100 attended Mass and it became clear that a proper church was needed. The priest lived at 99 Charlton Road.
In 1904 work began on building the church which was designed by French architect Mons Gervais of Bordeaux in Roman style with a nave and west aisle. Built by Jones & Sons of Erith to hold 400, it cost £5000. In 1905 the first local girl, Ellen Noon joined the Sisters.
The foundation stone was laid on 27th August 1905 by Bishop Amigo of Southwark. Also in 1905 the parish school opened using the conservatory on the ground floor of the house.
The church opened on 8th September 1906, opened and dedicated to Our Lady of Grace, because the Sisters had discovered there was a local shrine with that dedication in pre-Reformation days. However, no trace of this shrine exists today. However, Bishop Amigo wanted the church named after St Alphege who was martyred by the Danes on Blackheath, but the Sisters wishes prevailed as they provided the funds!
One of the sisters carved a life size statue of Our Lady out of wood and plaster which was installed in the alcove above the high altar where it remained until is was moved outside in the early 1960s. In 1907 the Stations of the Cross were installed.