Perhaps as we grow older, we become more aware of our boundaries, and more willing to defend them. And although we're also more aware of the reasons why people do what they do, we also realize that they can also avoid doing those same things, whatever the reason. It's possible that having seen people overcome great difficulties, we see more clearly that people retain agency. Problems may explain, but do not excuse, any behavior which transgresses.
We may understand their behavior more easily, but we also remember.