Help Excessive Contacts Listed in Phone App

[Android Question]

I just changed from a Razr to the Moto X. While it appears that I can display only my gmail contacts within the contacts app, my phone application seems to list anyone I have ever called or emailed in my entire life. Needless to say, this makes scrolling for who I actually want to call a real pain.

Does anyone know how I can change my settings in the phone app to only display those in my gmail contacts list?

Thanks very much.