I kind of get the feeling that what keeps Malfoy's anger at Harry going is probably the casual way Harry does just slot him into the enemy box and proceed to treat him as such, and like he's beneath him. The rejection started it, sure, but I don't think Malfoy would have been able to maintain truly intense anger over the incident for so many years, if he didn't have Harry looking at him as if he were dirt to remind him of that humiliation.