The speed from point A to point B on the internet depends on the traffic at all points of the path taken. It has nothing to do with the speed from point C to point D.
IOW, the path from your provider to Google's nearest server could be incredibly busy, while almost everyone else's path is fine. (If you're on cable, and have a dozen or so neighbors on the same segment all downloading movies, your downloads will crawl, while the guy on the next block, on a different segment, won't have any problem.)
It's just how the internet works. Remember, the prime design criterion was "data gets there eventually", not "data gets there fast". It still works like that sometimes, even though we're used to almost real-time video chatting. (If you chat with someone in th next room, you'll see and hear a fractional-second delay, even with the same carrier - that's why the "almost".) Routing (the decision of which path the packets take from the server to your computer) still doesn't usually have "fastest" as a priority.
tl;dr
It's just one of those things you have to learn to live with.
