Caring about what others think

I find myself always worrying about how I'm being perceived by others.Most family members feel that I'm distant which I will admit I am.I also grew up around alot of violence and till this day I am still fearful of my father.I personally do not want to have a relationship with him and I know that sounds terrible but thats how I feel.Relatives think I'm wrong for not calling them but I've notice I have more peace when Iam not around them,but I also feel guilt which makes it hard to fully enjoy my life without them so it seems as though theres no way around all of this, because my relatives are in my head daily so its like I'm still dealing with them when they aren't even there.