The dovecote that can be seen in Cadoxton is the only remnant that has survived of a medieval manorial complex that was present on the site where the present Cadoxton Court has been built. The complex would have once been home to the lords of Cadoxton.
During the medieval period the lands around Cadoxton Court would have looked extremely different and would have been mostly rural. During the late 19th Century quarrying activities would have shaped the steep precipice that can be seen next to the dovecote at the bottom of Gladstone Road.
When the docks were opened a lot of Victorian terraced houses were built, which took up much of the green land around Cadoxton. Cadoxton was founded in the early 12th century by a family named Mitdehorguil. The only surviving remnant from this period is a Norman font which is to be found within the nearby church of St Cadoc.