It is in fact really easy to tell the difference, you just hear more about the times people make the mistake because it’s not noteworthy when somebody goes “that guy’s just staring off into space” and is right. You also likely have a bigger emotional reaction, assuming you’re a guy, to a woman mistakenly thinking a guy is staring at her and being wrong than you do the knowledge that women get stared at a lot, so it makes the first seem like it’s happening more often.
I’ve lived on both sides (trans) and can tell you I didn’t realize it was this common to get really obviously stared at by older men. And the older they are the more likely they are to do it, which is lucky, because I’m much less afraid of a 70 year old man doing anything to me than a 20-40 year old. I find the only thing I can do in that situation is to avoid looking them directly in the eyes, because they take that as a sign to approach.