This is one area where Android lags behind iOS. iOS's permissions are pretty user friendly - you can disable access to contacts, calendar, photos, etc., for any app. It still allows app to work, but without the features that require that access.
I wish Android would adopt a similar permissions model.