Eh, maybe I'm just being naive, but per some of JKR's comments and all of that, I got the impression that the primary reason Harry was shipped off to the Dursleys was protection.

After all, look at what the remaining DEs did to the Longbottoms--a pair of fairly bad-ass Aurors (no canon on the bad-ass part, but let's assume). One might be able to assume that, from the simple fact that he's still alive, whatever guards Privet Drive is insanely powerful.

That isn't a complete excuse by any means, but I think it's a perpetually underrated issue.