The Dursleys started it by mistreating an innocent baby and shutting Harry in a cupboard for much of his childhood.

That's not really something Dudley can be held responsible for, though, imho.
The WW does seem to have a worrying tendency to visit the sins of the fathers on the children: witness first reactions to Harry, Malfoy and Dudley just for starters.