I did try rebooting my router and the Chromecast multiple times without any success, I forgot about restarting my phone though.
It isn't a set up mistake as it worked perfectly fine for months. I did have a similar problem a couple of months ago, and I fixed it by factory resetting the Chromecast, I did the same this time as well.
And while it works now, I wonder if there's a better solution out there (if it happens again, I should try rebooting my phone) or if this is a bug that I could report somewhere.