Is it wrong that I don't return to my boss's work after work?

Actually, the answer is simple. It is wrong of you not to reply. In my opinion, if you can't stand from your own point of view, but from the other side or neutral point of view, the answer is self-evident. It's past 10, and the boss is still asking about your work progress in the group. For you, it may be a heartfelt dislike, but from the boss's point of view, it is a concern for the progress of work and trust in you. Think about it this way, not only don't feel a burden, but also a kind of trusted happiness.