Ever heard the saying "The chain is only as strong as it's weakest link"? The speaker can only be as good as the signal coming in from the person calling you. I am sure you are talking about multiple calls you have taken on the speaker phone, but perhaps you just haven't been talking to people that have good mics, or have been on speaker phones themselves.
That is all I can think of. Unless you have a defective phone, but you say music sounds great which kinda throws that possibility out of the window