Duddy was born in Belfast, Northern Ireland, and grew up in a large Protestant family in the "Hammer" area of the city, located in the lower Shankill Road. He had eight siblings. His father was originally from Derry but had settled in Belfast and served in the B Specials during World War II. Duddy was born shortly after the war ended.Wood, Ian S. (2006). Crimes of loyalty: a history of the UDA. Edinburgh University Press. p.3
At the age of 15, Duddy was apprenticed to a letterpress printer. His father held the same occupation.