Sure with a hard case, yeh wear and tear could certainly be a problem in the long term.
I have to ask, why would you need to be constantly removing the phone from the case? Can't be always changing the battery with an S6, because you just can't do that, it's sealed.
My Oppo F7 is in a hard plastic case, but only remove it occasionally clean it, and to swap batteries if needed, but that's been quite rare for me recently.