I think we're related? I have that problem too. Low self-esteem and a fear of...something or another.
I don't know how to get over it...not for other people. You might want to try what I do, and just tell yourself that your fear is irrational, and ignore it.
It's tough, especially since phobias and your reactions to such phobias happen mostly in the sub-consciene, so you're unable to really help how you act.
Just think before you act, and ignore your sub-conscience. Easier said than done.