There is no option but i managed to do it, because all my contacts are stored in google.
If yours are as well you can do this.
Go onto a pc and log into gmail.
go to contacts
create a new group, i called mine "people"
Put all your phone number contacts in that group
go to contacts app on nexus
select menu and go to "contacts to display"
hit customize
hit the down arrow next to your gmail account.
uncheck everything, including all contacts.
check your new group, in my case "people"
hit ok
Enjoy!!!
A bit of a long method but it works like a charm for me!