The ROM is basically the phone's operating system, user interface and pre-installed apps, sounds and images. So when you receive a system update, that is updating the ROM.
Some people like to customise their system software, to make the phone faster, or add features. This is referred to as a "custom ROM". You need to have rooted the phone (gained administrator access) to install these.
JellyBean is the codename for Android 4.1, 4.2 or 4.3 (4.0 was Ice-Cream Sandwich, 3 was Honeycomb, 2.3 was Gingerbread, 2.2 was Froyo, 2.1, etc).
So a JellyBean ROM would be a ROM based on Android 4.1. A custom JellyBean ROM would be one that has been built by an independent developer, while an official one would be one that was built by Samsung.