It may have pulled the diaphragm out of place. Might try wacking the earpiece against your palm a few times. See if that will jostle it back in place. I don't thing a magnet could pull it far enough out of place to tear it. I'll try with the dead crackberry I have here in my desk.
Nope, doesn't seem to effect these speakers. Looking at them closer, the electromagnet part is actually attached to the diaphragm, so a magnet shouldn't be able to break that part at all.