Most likely that's simply a question of the USB port not providing a high enough ampage for your (large screen?) phone.
Basically, modern large screen phones need quite a bit of juice - my N4 charger is 1.2amp - or the phone will use power faster than low power chargers can provide it. Having said that, you'd normally need to be using the screen for that to be an issue.
You could try checking if your power saving options are lowering the USB power. It might also be the rating of the USB cable. Or, try turning the phone off.
Alternatively, could be an issue with the USB port on the phone or a dying battery - does the phone charge using a wall charger?