Unplug and replug headset sometimes does it. Sometimes I have to try it multiple times.
Sometimes a short call, like dial voicemail and hangup, will bring the volume up to normal.
Notifications, while listening to music, will cause the volume to dip again (usually). Set to silent mode while listening to music to prevent this.