Used to hear about this problem a lot, although not in a while.
Try going into settings > apps and clearing cache and data for the phone and or voicemail app and rebooting. I've also heard of calling and leaving yourself a new voicemail or ask a friend to do it, then check your voicemail, delete that message, and sometimes the icon will stay turned off. A factory reset always removes the voicemail icon but not a useful tip if you keep a lot of data on your phone. You can also try calling customer support and ask if they can try to clear it.