William De Braose constructed the castle c1070, along with the Norman church, on a natural mound and most of the surviving masonry dates from this time. Except for a period of confiscation during the reign of King John, Bramber Castle remained in the ownership of the De Braose family until the line died out in 1324.
Despite very little surviving, the basic layout of some areas of Bramber Castle can still be identified. The most prominent feature is a large, rugged lump of stone, all that remains of the Gatehouse tower. Still standing to almost its full height, a single window, and some floor joist holes, are clearly visible within the structure. Beyond the Gatehouse are the existing foundations of what is believed to have been living quarters and a guardhouse. The dressed pillars of an entrance can be made out, but the bulk of the remaining walls now consist of only lumps of basic rough stone infill, the better quality dressing stone having long since been quarried away for use elsewhere. Lying to the north of the gatehouse is the original castle motte, its earthen mound rising to a height of some 30 ft (10 m). A short distance away is a section of the curtain wall and, again, this survives to a reasonable height, up to 10 ft (3 m) in places.