Automated Keyboard and Mouse actions

[Pente]

Hi all.

Not sure if this has been covered, and I have been searching for it with no luck.

I'm wondering if there is a way to automate mouse and keyboard actions so that events such as logins etc. can be automated and initiated unattended or called by another app (even if it is just for known sites/tasks etc.). I'd be happy with having to 'record' a macro with mouse/keyboard actions and just have the macro be callable from something such as tasker.

In particular, I have an app for my HVAC system that requires me to enter a password and hit enter in order to log in to the company's servers. From there I can make adjustments to my home heating/cooling system. I'd like to be able to automate this so that tasker or some such app could open the HVAC app, type in my password, and ideally move and click the mouse on the UI presented (it mimics my local thermostat input screen). The tasker trigger could be based on a text message from my home automation controller attached to temp. sensors, for example.

I can't just use a username/password vault type app as I need to actually move the mouse and register clicks etc.

Any help would be greatly appreciated.

-Pente

 

[Rukbat]

Are you writing an app to simulate the functions of the existing app (easier way) or trying to automate the existing app (more difficult way)?

 

[Pente]

Rukbat,

Thanks for the reply.

I was actually not planning to write an app (my coding skills are more than a tad rusty these days), and I'm not sure I could do it for this app anyways, the HVAC company has a pretty tight grip on their communication protocols etc. so I'm not sure one could even get a 'clone' app to even authenticate on their servers, at least, not without far more work than it is worth.

On the windows side of things, there are apps that will record your mouse movement and keystrokes, creating self-contained macros (think batch file) that can then be run. The macros are basically agnostic of the actual application running in that, the macro will run (mouse moves, changes co-ords, any clicks and/or keystrokes are executed, regardless of whether or not the app which has focus is actually receptive to these mouse/keyboard actions. In this respect the macro is dumb, running in an over-lay if you will, regardless of what app is actually in focus.

Think of it this way: the password text box is always at position x,y in my HVAC app, so, I'd like a macro, or event, that would open the HVAC app, move the mouse to co-ords x,y, left click (activate cursor in the textbox), enter my password into the text box, and then <enter/return> to complete the sign-in process. All of this would be autonomous, and unattended.

If this can be done, it is then just a matter of making multiple macros for any/all such actions required ie. mouse-over and left-click on an onscreen button to increase temperature of thermostat.

I hope this is more clear, sorry for initial confusion.

Thanks in advance,

Pente

 

[codesplice]

You may be able to use Tasker in conjunction with the input tap shell command to create a script that would function like you describe. There's a pretty good write-up on that here: How to emulate touch input using Tasker and root - Pocketables

If you do decide to go the Tasker route, let me know and I can move your thread into the Tasker subforum where lots of neckbeards smart people are ready and waiting to assist you.

 

[Pente]

Codesplice...

Tyvm! That is an excellent link and from a quick read-through, exactly what I need for the mouse movement etc. so all I'd need to do is figure out the keystrokes...hopefully there is something already baked-in for that too (I'm yet to really jump into Tasker, but will be starting soon enough)

I will take you up on your offer, and thank you in advance, for moving this thread to the tasker subform...I'm certain you are correct about that being the best place to continue with this.

-Pente

 

[codesplice]

Codesplice...

Tyvm! That is an excellent link and from a quick read-through, exactly what I need for the mouse movement etc. so all I'd need to do is figure out the keystrokes...hopefully there is something already baked-in for that too (I'm yet to really jump into Tasker, but will be starting soon enough)

There's some discussion at the Tasker Google Group on that topic, but I haven't perused it enough to know for sure if it will work for you. It may at least get you pointed in a helpful direction.

I will take you up on your offer, and thank you in advance, for moving this thread to the tasker subform...I'm certain you are correct about that being the best place to continue with this.

-Pente

Done

 

[KitWasHere]

There is a "Type" action under "Input" you could try. Don't use it myself but other users report it as being slow. You need root for that, as well as the "input tap x y" command posted above.

 

[Pente]

Kit and Code...

Thanks for the continuing help and suggestions.

I will certainly look into that forum Code, and the 'type' action. Speed won't really be an issue as long as it isn't glacier-like, as most of my intended plans won't be of a pressing nature.

Much appreciation!

-Pente