I think it stems from the fact that I have a hard time reading people and have difficulty knowing if they're serious or joking, unless it is really obvious.

Frankly, I think people often count on that. Many jokes can go one way or the other, so people can hide a lot of cruelty in them. If the person gets upset they can pretend they just meant it as a joke, but erally they can just be mean. The fact that the twins continue to put most of their energy into this stuff at 17 says they just really like causing people discomfort and laughing at them.