Yep, I HATE taking the case off. It's a struggle to take it off without ripping the battery cover off with it.
Try taking off the case from the top right hand corner. I thought about why it was happening and came to a conclusion that is where we open the back cover to access the battery/micro sd card. I tried the top right hand corner so I won't have to fumble around with the on/off button and the volume toggles and it's working well for me.