Allington Castle is a Grade I listed building. Much of the stonework was laid in an intricate herringbone pattern which is still visible today. It was the birthplace in 1503 of the English lyrical poet Sir Thomas Wyatt and in 1521 of his son the rebel leader Thomas Wyatt.
The manor house on the site was fortified by Lord Warden of the Cinque PortsStephen de Penchester in 1281 after a licence to crenellate was issued by Edward I. It was converted to a mansion in 1492 when the Wyatt family acquired the property. Towards the end of the 16th century whilst under the ownership of the Wyatt family the castle was badly damaged by fire, remaining largely derelict until 1905 when it was restored by Sir Martin Conway.