It's also that when people fall for the twins' jokes it's always sort of a sign of weakness in that person.

Yes, I get that sense, too. Unfortunately that is a bully's arrangement of the world. The good people really still see the world in the same sort of terms. It also seems like the same kind of thing as Dumbledore saying Snape is good because it teaches the kids to learn how to deal with Snapes (actually he never said that but JKR said he thinks it). So kids like Neville who aren't getting anything good out of it are just weak. That's their problem.