You need to make sure the media volume it up and that you are not using an app that is disabling your media volume when you're at work.
The curious thing is that the volume switch usually manipulates the ringer volume, but only after you start a music track or video will it then change to manipulate the media volume. if you try before starting those multimedia apps, then you'll only be changing the ringer volume, which is independent of the media volume.
Also, like I said, check for apps that are automatically turning volumes down based upon location / situation, etc.