The castle was a Norman double enclosure, whose construction required the demolition of Saxon houses, and was surrounded by moats filled with water diverted from the river. It was once in possession of the de Mandeville family but by 1156 was held by the Beauchamp family.