Married colleagues often send me text messages.

A female friend of mine has the same experience as you. The male boss often chats with her and calls her. Because she is the boss, she doesn't directly refuse. He answered the phone and chatted with him. As a result, his wife called my friend a mistress. So I think you'd better make it clear to your boss, so he'd better be fine and leave you alone.