It has to do with the movie, they explained it later on in the movie. Something to do with how he kept track of everyone who tried to beat him down in prison, also he was a bit of a ... 'legend' if you will, in that he was legally dead after being shot in the head. Which is why dude called him the ghost.