I toyed around with this a little and it seems like the contacts app uses the small image when the image is... well... small. A contact with a large photo (on Google+ or Facebook, for example) will display full screen.
What I have needed to do on some of the contacts, is to actually remove the image from Google contacts (96 pixels square) and it automatically grabs one from one of the other linked contacts.
There doesn't seem to be a nice/obvious way to just tell the contacts app to prefer high-res images or to always use pictures from G+ or FB instead of Google Contacts.