Does it mean that a Virgo man always attacks me means he hates me?

No, Virgos generally maintain a very cold and unwilling attitude towards people they don't like.

And he is probably doing this to attract your attention. To be honest, he probably likes you.

But because of my face and fear of being rejected by you, I have never dared to express my feelings.

Virgo is a very reserved sign. Unless there is no other way out, they will not take the initiative to express their feelings.