You may have a downloads app on your phone where you can interface your recent downloads and delete them, check your app drawer. Otherwise, a file explorer app would do the same.
Though installing is a secondary issue to downloading the app in the first place, be sure that the option to allow install of apps from unknown sources is unchecked in your main settings menu. (possibly at settings > security) That will at least disallow a brute force download and install from sources other than Google Play.