My impression is that some permissions sound more invasive than others. A lot of apps need access to your contacts...email, phone, SMS, Twitter, Facebook. All of those apps need to access your contacts in order to pull up the address, number etc. A lot of apps you may not think need it will also ask for permission to access contacts, but it's usually for the same reason. You may not think the game you installed needs access to your contacts, but a lot of games these days have features that share scores with your friends or something along those lines. It just needs contact info to grab an email address or phone number.
Basically it comes down to whether or not you trust the developer and whether you can think of any reason that app needs access to that information.