I've had similar issues with various BT headsets and swapping. I've seen it on my Gnex and S3. It seems to havppen randomly.
My recommendation is to unpair your bluetooth, delete data and cache for your bluetooth connections, and re-pair your devices together.
Also, I don't think I've seen it happen with any consistency, like if I haven't reset the phone in a while or if I have a bunch of apps running. I think the software doesn't see the connection getting changed, like maybe a timer is too short, so the code doesn't have a fall back until you hit speakerphone. I *think* if you try to swap it back from speakerphone to the headset speaker, it *should* work.