A custom OS is a ROM built by an enthusiast or team of such rather than the manufacturer. Advantages may include more features or customisation, less bloatware, and sometimes even android versions that had not been officially released for the device. Drawbacks may be that not all are as stable as stock software (some may be as stable or more so, but not all will be), some may have features that don't work, or there may be differences in performance (which can go both ways, e.g. the ROM could be faster but have poorer battery life). There aren't general statements you can make like this, it depends on the particular ROM.
These things are specific to the particular device model you have. Some may have many custom ROMs, some may have none at all. Some are easy to unlock and install custom software on, some are impossible. Without knowing what phone or tablet you have it's impossible to be more specific.
Now your second question seems unconnected to the first. That just sounds like you've used a key combination to enter "download mode", which is a normal feature on many phones (others have something similar called "fastboot mode"). Just reboot the phone to get out of it if that wasn't what you wanted to do.