Goodness isn't the same as just being responsible. Anyone can support their family, be a good provider. Goodness is that extra step of self-sacrifice. It is being the parent who is gentle with the teenager when she is screaming at your face. It is the neighbor who doesn't complain about the next door mess, but comes over to help them clean it.
Normal people compromise their standards. That is seen as being human. They will give up, be prejudiced, even hate, and that is fine. They understand that being human is weak, that everyone breaks at times. But the good person may never break, never give up. They must always be strong in goodness, always put on a good face. Unfortunately, this means that every good person is an implied accusation to those who get drunk, who give into their hatred, who refuse to forgive, who allow life to break them. The good person is an offense to those who are normal. Thus, the normal person must constantly buffer the good, to force them to display their humanity, to show that they aren't any better than anyone else.
Of course, the good are human. Every human has a certain level of will power. Some have more and some have less. Will can be exercised, like a muscle and it can also atrophy or be overworked and strained. But no matter who we are, we only have so much will power to work with in our lives. We focus our will on certain things, which means we don't have will to put into other aspects of our lives. A surgeon might spend so much focus on precision in cutting and creativity in repairing the human body, that they might not have energy to put into being polite to their patients. A mother may spend so much time and energy on her toddler that she doesn't have enough "get up and go" to clean the house. We may disagree as to how a person spends their limited will and energy, but we must admit that no one can do everything. This is why the normal person is compromised.
And, of course, the good person may very well break. They might get tired of the accusations and strike back. They might neglect their family or have an affair. They might give way to a stroke. They might fall into severe depression or other mental illness. They might become a hermit or become self-focused.
But what happens if a good person doesn't have any severe breaks? What if they just move to a different community? Or, God forbid, suddenly die? It is then that they really become a saint to the community. A saint is only safe after they are no longer available to be an accusation. They can be lauded and statues can be made for them and they can be safely regarded as a community hero. People might even change after they are gone, seeing their strength and goodness as an example. Their struggles and difficulties are seen as a passion, a legend of their goodness.
The only loved saint is the one who is no longer there.