A colleague used to like me and often sent me messages, but I didn't reply, which always affected my family. What should I do?

Don't tell your husband that no matter what your starting point is, your husband will be unhappy. You should consider how your husband feels. What would you think if you were in your position? Smart women don't usually do this. If you really cherish your family, just hack that person and give you no face. If he goes too far, let him face the floor directly in the office and he will never bother you again.