Conversations can drift to adjacent topics, yeah, but it's not a "gotcha" when someone suddenly changes the topic to the inverse of what was being said, and then acts like they're arguing against you because the thing that you said about the original topic doesn't add up with the new topic. If you change the topic, you need to at least give the other person an opportunity to respond to your new topic, not just assume that their same argument applies.