Why are you embarrassed when a man says you have small breasts?

Go back, of course. If a man really tells you that, it's probably a joke. And generally do not mind, of course, does not rule out the meaning of ridicule.

He will definitely like big breasts, and he doesn't mind small breasts!

So you don't have to worry too much about it. Just make a joke with him and chat and go back.