Sometimes recovery mode is tricky to get into.
You got the right button combo to do it. I usually start with holding down the home button, then volume up, then power.
Keep holding all 3 until the phone vibrates, then let go of home and volume up only until the screen lights up in recovery, then let go of the power button.
Worst case, you'll have to ODIN back to stock, then re-root.