Maybe the more sensitive you are the more shame you have. You feel (in empathy) that your actions in the past have really hurt people.
Maybe the tougher you are the less shame you have. You will have barged through life expecting everyone else was as tough as you; the empathy being reduced relatively and therefore shame being less. You simply don’t think you hurt people – because you wouldn’t have been hurt thus.
In which case, whether people were hurt or not is independent of how you feel.
And holding onto any shame is only your own perception of events.
It’s worth bearing in mind.