Interesting. Now I'm starting to get curious about what people mean by "encrypted bootloader". Obviously it contains byte code, otherwise the OMAP wouldn't do anything when you turn it on. Now, there are full disk encryption methods, but with a stored password (since you aren't providing one at power-on), this is obfuscation at best and kinda pointless. My guess is that the bootloader simply verifies the hash and cryptographic signature of images before flashing them, which is generally a good idea. It could do that every boot and give you the bird and refuse to boot if it doesn't match, but, IIRC, phones often use YAFFS, which is sorta rewritable, and would change the hash every time the phone booted.
Now, that brings us to what it prevents. It sounds to this newbie like all that would hinder is using the boot loader to rewrite the flash. But there should be other ways to do that, given root access, and at least a few of these approaches should work. (NB: I don't actually have a smartphone yet, so I'm speaking from my experience with other devices.)
- You could obviously use JTAG, which lets you do just about anything, but it's way too involved for the average user to attempt.
- You could flash from a rooted Linux using mtd or similar, which seems like the best bet.
- You could overwrite the existing root much like upgrading desktop Linux.
- You could modify the bootscript to chroot to the SD card and install whatever you want there.
- You could run a new kernel in userland.
- You could read the bootloader, extract the key, and see if it's of a crackable type (e.g. factoring a short RSA key with a distributed cluster of volunteers), which is likely given that it's an embedded device.
- You could generate a hash collision with a custom ROM, and the bootloader would be none-the-wiser.
- You could replace the key with your own by rewriting those sectors in the flash.
- You could just reflash the bootloader with an open source one (e.g. based on the Pandora's, which uses a similar OMAP SOC and also runs Linux).
In theory, since we have physical access to the hardware, there is no way to completely lock us out. DRM is annoying and complicated, but offers zero cryptographic protection.