They want to protect themselves from harm, people believe in the delusion of a just world for their own psychological well being. As if they didn't, that means they would have to face the possibly that bad outcomes could befall them, that they could fail through no fault of their own, and that ultimately they'll die. If they accept that you're a reject through no fault of your own, they then also must accept that they have no real control over their own lives, and that idea can be scary.
It's funny because, ironically, you can really only believe that the world is just if circumstances during your childhood allow you to do so. The development of a healthy reward system, and the lack of exposure to trauma during formative years are generally both necessary to facilitate a belief in some sort of metaphysical karmic justice. As they age confirmation bias acquired throughout their own lives does the rest, the result is a lack of understanding for people with less fortunate circumstances. The only options they have for dealing with us within their own construct of reality are either to correct the situation through helping us and make the world "just" again through their own input, or to simply assert that we're bad people. Since the former is an impossibility, their only remaining choice is to demonize us, as to preserve the foundation of their worldview.
