Hmm, yes, there's that--but another thought I just had on the heels of a comment elsewhere (http://www.livejournal.com/users/pauraque/218660.html?thread=3385380) is that the relationship itself would be a different kind than he's used to thinking of, and that's probably scary too. Right now, as amythis pointed out, he seems to think of girlfriends as people you snog, not people you interact with in any other way; a relationship with Hermione would necessarily entail more than just snogging.