My colleague told a joke and didn't speak.

Of course, it's hard to be teased by colleagues as a joke. Of course, you should also learn to fight back. Sometimes the weaker you are, the more people push their luck. So you should be strong, brave and bold to tease him and let him feel this feeling. Answer blows with blows is usually the best way.