Yeah, the gesture can be a bit unreliable. I don't think the sensor hardware was really designed to detect direction so registering that is a bit of a hack.
There's not really anything that can be done for it but to practice being exceedingly deliberate with your gesture. Even so, you probably won't get a high success rate.
It's not just you.