If we had scene it in the pensieve, you'd be right. Assuming I'm remembering correctly, we mainly just saw glimpses because it was occlumency, so they're still able to say that she was there and just not in those bits Harry got to see.

If I'm remembering it completely wrong, please pretend I never said this. You see, I've only seen the movie once, so I fail as a fan. D: