Take a look at some of the Comic Book reader apps out there. Some of them support PDFs and if not it is trivial to make a "Comic Book" given a series of images. That is assuming your sheet music is a bunch of images...
All you need to do is name your images in order with the same number of digits for the number part.
Bad - Page1.jpg, Page2.jpg, ... Page100.jpg
Bad - Page01.jpg, Page02.jpg, ... Page100.jpg
Good - Page001.jpg, Page002.jpg, ... Page100.jpg
Then zip or rar all the images and rename SheetMusic.zip to SheetMusic.cbz or SheetMusic.rar to SheetMusic.cbr
As to selection of book and chapter... I never kept more than a few on my mobile devices and it sounds like you want to store a lot more... so I'm not 100% on that part. The apps I've used let you browse the file system so you could create a SheetMusic folder and then A, B, etc folders under that...
If you mean managing the books on your computer then you could take a look at Calibre. It is more of a generic eBook manager but I believe it supports CBR/CBZ format books and I believe it can sync to just about anything (including android devices) over USB.
Matt