Personally, no, I don't have a problem. I've usually turned off any automatic brightness, even when using a custom ROM which allows me to set my own levels, but actually I've found the adaptive brightness on 9 to work well enough that I leave it on. The only times I adjust it these days is when showing photos to my Mum, for whom it's better to turn the brightness up a bit (I do remember adjusting it a bit more when I first installed 9, but that was back in the developer previews anyway).
Now I do have a Pixel 2 rather than a 2XL, so there may be differences due to the hardware. But for me it works well enough that I rarely think about it.