I do think that this (and all variants thereof) is of the most unwelcome things a person can say, in any circumstances. If they actually love you and and are happy to have you along, it can be really hurtful to suggest that you don't believe this and are convinced they really wish you elsewhere. And in the ( let's face it, rather more likely) opposite case that they really are only putting up with you out of kindness, it just adds insult to their pain; they can't decently agree, so you're forcing them to declare insincerely 'No, no, it's no bother at all, we love having you!'
Thank people, yes of course. But the more you suspect that you really are a complete bollard to them, the more you should avoid making remarks that oblige them to lie about it.