In summary:
Push notifications don’t work with Marshmallow’s Doze. Receiving instant email notifications is very important to me. My workaround is to use a task automation tool to do ‘something’ at a regular interval to prevent the phone from entering Doze at all. The resulting battery usage is more than acceptable for me: about 1% / hour, measured over a 7-8 hour period (at night), with k9 mail connected to my private email (imap idle for push) and Good Work connected to my work email.
In detail:
Doze stops most apps, preventing them from syncing and accessing the network, after a period of inactivity.
No app can be fully exempt from Doze (despite many claims to the contrary).
Android’s official page explains that whitelisting an app (settings – battery – battery optimisation) does not completely exempt it from Doze: https://developer.android.com/train...doze-standby.html#support_for_other_use_cases
Notably, the acceptable use cases https://developer.android.com/training/monitoring-device-state/doze-standby.html#whitelisting-cases do NOT mention email clients!
As of now, no solution has been found to make imap idle work with Doze.
Many sites and discussions mention that you can disable Doze for certain apps, but that’s incorrect; in fact, there are many reports of push email not working with Doze; at the time of writing, the developers of k9 mail https://github.com/k9mail/k-9/issues/857 and of Aqua Mail https://www.aqua-mail.com/forum/index.php?topic=4858.msg28029#msg28029 have not found a solution. That’s a shame, because these two clients use imap idle for push, which I’m a big fun of: no server implementation required for push; I’ve been using it for 6 years on a number of Android devices without ever experiencing the battery drain which is sometimes mentioned as a drawback; additionally, they are among the few apps which keep your data private (http://androidforums.com/threads/email-which-apps-keep-it-private.935578/ ).
Push email not based on imap idle doesn’t work with Doze, either:
As Android’s site explains, receiving high-priority GCM (Google Cloud Messaging) notifications would cause the device from exiting Doze mode; however, there are two issues:
1) Client-only solutions (like k9 mail with imap idle) don’t work with Doze. You need a GCM-based solution, which requires a server implementation, which means you have to trust someone with the password to your email. Even if you’re willing to pay for this service and to trust the provider for your private email, no sensible IT security manager would: Marshmallow has effectively killed proper push notifications for work email. A company might be willing to invest in its own GCM-based solution, but I have no idea how complicated it would be, if data would have to be shared with Google’s server ( a big no-no in most cases), etc.
2) Leaving aside cost and trust, as of now I have not found a single email app which can send high priority GCM notifications. Cloudmagic and others rely on GCM, but do not send high-priority notifications; Cloudmagic has just confirmed this to me in an email, and the Aquamail discussion linked earlier says the same about other apps. This may change in the future, of course.
My solution with a task automation tool
My solution is to use a task automation tool (I use MacroDroid, which is easier to use than Tasker, but I suppose any similar app would do) to get the phone to do something at regular intervals, so as to prevent the phone from entering Doze at all. I have set up this MacroDroid macro:
With these settings, and with k9 email connected to my private email via imap idle for push, and Good Work connected to my work email, I left the phone on a desk overnight (7.5 hours), connected to my home wifi, with keep wi-fi on during sleep set to always, and the battery usage was 1% / hour (as measured by the Gsam battery monitor app). When I woke up, I noticed the led light was flashing, and when I turned the screen on I saw notifications for emails (both k9 and good) received overnight. I checked with gsam (charts -> other) and verified the phone never went into Doze mode. According to Better Battery Stats, the phone was in deep sleep 85% of the time; I would have expected this % to be higher, but I’m not too bothered because 1% / hour is more than acceptable for me, plus I never leave the phone on for 8 hours without touching it (I typically switch it off at night, or leave it in airplane mode if I need an alarm to wake me up). The main thing I was interested in was making sure the phone doesn’t enter doze mode and breaks push emails if, say, I leave the phone on the desk for an hour during a meeting. With this method, I can be sure it doesn’t happen, yet I still have a good battery usage.
How about root?
I understand Doze can be tweaked on rooted devices, but I haven’t looked into this because I need to keep my phone non-rooted, otherwise Good Work, which I use to connect to my work email, won’t work.
Other settings I have (probably not relevant):
Next steps:
The day before, the phone entered Doze mode overnight and, when I woke up, neither k9 nor good had received emails. I had to manually refresh both. Oddly, though, the battery usage when in Doze was higher (2% / hour vs 1% / hour) over a comparable 7.5 hour period.
Final comments:
I find all of this insane and frustrating. Android went from one extreme to the other: from a flawed OS, which caused battery drain because too many apps were running in the background when they were not supposed to, and with users unable to prevent it (see my discussion: http://androidforums.com/threads/wh...nning-in-the-background-android-flaw.1053919/ ) to a flawed OS which, in its misguided attempt at improving battery life, doesn’t let users have the apps they want running in the background when they want! At first I thought it was a perverse way for Google to force users to reply on its GCM system; however, Marshmallow has been around for almost one year, yet no solution exists to get push email to work properly with Doze. If I had wanted a closed eco-system with a dictatorial “my-way-or-the-highway” approach, I would have bought an iPhone! Doze can make sense for tablets, or at least for how I use my tablet (I am fine with not having push on my tablet if it extends battery life, but I need push on my phone), but breaking push without even giving users the option to fully disable Doze is insane!
Thoughts? Suggestions?
Push notifications don’t work with Marshmallow’s Doze. Receiving instant email notifications is very important to me. My workaround is to use a task automation tool to do ‘something’ at a regular interval to prevent the phone from entering Doze at all. The resulting battery usage is more than acceptable for me: about 1% / hour, measured over a 7-8 hour period (at night), with k9 mail connected to my private email (imap idle for push) and Good Work connected to my work email.
In detail:
Doze stops most apps, preventing them from syncing and accessing the network, after a period of inactivity.
No app can be fully exempt from Doze (despite many claims to the contrary).
Android’s official page explains that whitelisting an app (settings – battery – battery optimisation) does not completely exempt it from Doze: https://developer.android.com/train...doze-standby.html#support_for_other_use_cases
An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.
Notably, the acceptable use cases https://developer.android.com/training/monitoring-device-state/doze-standby.html#whitelisting-cases do NOT mention email clients!
As of now, no solution has been found to make imap idle work with Doze.
Many sites and discussions mention that you can disable Doze for certain apps, but that’s incorrect; in fact, there are many reports of push email not working with Doze; at the time of writing, the developers of k9 mail https://github.com/k9mail/k-9/issues/857 and of Aqua Mail https://www.aqua-mail.com/forum/index.php?topic=4858.msg28029#msg28029 have not found a solution. That’s a shame, because these two clients use imap idle for push, which I’m a big fun of: no server implementation required for push; I’ve been using it for 6 years on a number of Android devices without ever experiencing the battery drain which is sometimes mentioned as a drawback; additionally, they are among the few apps which keep your data private (http://androidforums.com/threads/email-which-apps-keep-it-private.935578/ ).
Push email not based on imap idle doesn’t work with Doze, either:
As Android’s site explains, receiving high-priority GCM (Google Cloud Messaging) notifications would cause the device from exiting Doze mode; however, there are two issues:
1) Client-only solutions (like k9 mail with imap idle) don’t work with Doze. You need a GCM-based solution, which requires a server implementation, which means you have to trust someone with the password to your email. Even if you’re willing to pay for this service and to trust the provider for your private email, no sensible IT security manager would: Marshmallow has effectively killed proper push notifications for work email. A company might be willing to invest in its own GCM-based solution, but I have no idea how complicated it would be, if data would have to be shared with Google’s server ( a big no-no in most cases), etc.
2) Leaving aside cost and trust, as of now I have not found a single email app which can send high priority GCM notifications. Cloudmagic and others rely on GCM, but do not send high-priority notifications; Cloudmagic has just confirmed this to me in an email, and the Aquamail discussion linked earlier says the same about other apps. This may change in the future, of course.
My solution with a task automation tool
My solution is to use a task automation tool (I use MacroDroid, which is easier to use than Tasker, but I suppose any similar app would do) to get the phone to do something at regular intervals, so as to prevent the phone from entering Doze at all. I have set up this MacroDroid macro:
- Trigger: regular interval 45 minutes. Using Gsam battery monitor, it seemed my phone entered Doze after about 1 – 1.5 hours of inactivity. The time may be different for your device. You can leave it stationary on a desk, then come back after a few hours, and use Gsam to check when Doze kicked in.
- Actions:
- Keep device awake: screen on for 1 minute
- Launch Good Work (this might not be necessary, to be honest)
- Constraints (ie the macro only runs if): screen off and phone not in call
With these settings, and with k9 email connected to my private email via imap idle for push, and Good Work connected to my work email, I left the phone on a desk overnight (7.5 hours), connected to my home wifi, with keep wi-fi on during sleep set to always, and the battery usage was 1% / hour (as measured by the Gsam battery monitor app). When I woke up, I noticed the led light was flashing, and when I turned the screen on I saw notifications for emails (both k9 and good) received overnight. I checked with gsam (charts -> other) and verified the phone never went into Doze mode. According to Better Battery Stats, the phone was in deep sleep 85% of the time; I would have expected this % to be higher, but I’m not too bothered because 1% / hour is more than acceptable for me, plus I never leave the phone on for 8 hours without touching it (I typically switch it off at night, or leave it in airplane mode if I need an alarm to wake me up). The main thing I was interested in was making sure the phone doesn’t enter doze mode and breaks push emails if, say, I leave the phone on the desk for an hour during a meeting. With this method, I can be sure it doesn’t happen, yet I still have a good battery usage.
How about root?
I understand Doze can be tweaked on rooted devices, but I haven’t looked into this because I need to keep my phone non-rooted, otherwise Good Work, which I use to connect to my work email, won’t work.
Other settings I have (probably not relevant):
- K9 mail, Good work, MacroDroid, WhatsApp and PNF no root (see below) (partially) exempted from Doze (settings – battery – battery optimisation)
- I have installed PNF (Push notification fixer, no root) https://play.google.com/store/apps/details?id=com.andqlimax.pushfixer.noroot&hl=en_GB and set the interval to 5 minutes. This app resets the heartbeat interval to 5 minutes; it helps when WhatsApp messages and other notifications don’t arrive immediately; in fact, it is mentioned in WhatsApp’s help page: https://www.whatsapp.com/faq/en/android/20887936
Next steps:
- I hope the 1%/hour battery usage will stay roughly constant but I’ll keep monitoring the situation.
- I will also try to fine-tune the macro; maybe I can set it to wake the CPU without waking up the screen (MacroDroid has such an option but I haven’t tried it yet).
- I don’t know if MacroDroid or Tasker can be used to exit Doze; some people may be interested in exiting Doze after, say, 2 hours, but I don’t want my phone to enter Doze at all.
- I’ll keep looking for a GCM-based email solution for my private imap email account that can send high priority notifications.
- I’ll keep chasing Good and my employer’s IT team to see if they have a better solution, e.g. if Good can be fully exempted from Doze or send these high-priority notifications
- I want to understand if Whatsapp messages are received when in Doze mode or not.
The day before, the phone entered Doze mode overnight and, when I woke up, neither k9 nor good had received emails. I had to manually refresh both. Oddly, though, the battery usage when in Doze was higher (2% / hour vs 1% / hour) over a comparable 7.5 hour period.
Final comments:
I find all of this insane and frustrating. Android went from one extreme to the other: from a flawed OS, which caused battery drain because too many apps were running in the background when they were not supposed to, and with users unable to prevent it (see my discussion: http://androidforums.com/threads/wh...nning-in-the-background-android-flaw.1053919/ ) to a flawed OS which, in its misguided attempt at improving battery life, doesn’t let users have the apps they want running in the background when they want! At first I thought it was a perverse way for Google to force users to reply on its GCM system; however, Marshmallow has been around for almost one year, yet no solution exists to get push email to work properly with Doze. If I had wanted a closed eco-system with a dictatorial “my-way-or-the-highway” approach, I would have bought an iPhone! Doze can make sense for tablets, or at least for how I use my tablet (I am fine with not having push on my tablet if it extends battery life, but I need push on my phone), but breaking push without even giving users the option to fully disable Doze is insane!
Thoughts? Suggestions?
