Yeah, the time isn't necessarily tied to your Chromecast's exact location, whatever time you're seeing on your TV screen is just from whatever signal your Chromecast is being supplied to it. If you look at the Google Home app on the mobile device used to set up/manage your Chromecast you'll see a scaled down duplicate image of your TV's screen. It's that Home app that determines what's showing as a backdrop, Google is currently referring to that as 'Ambient Mode' so there's different options in there to mess with, along with App Settings, Account Settings, and Device Settings all buried in different areas of that Home app. (The Home app is being groomed as a multifaceted, central control point for numerous Google services so a lot of that stuff relates to other things.)
Anyway, it's that Device Settings menu you should look into, there are options there for Time Zone and Time Format to look into.