Automation

For anyone who wants an app on many Androids to do the same task without a person tapping, and who may not write code.

Automation works on any app. Nothing in the code knows a particular app: a task is recorded or written in terms of what is on the screen, and it runs on every Android you select, each on its own. A device that meets a screen nobody planned for stops and waits for a person while the others carry on.

There are two ways to make a task, and they run the same way:

• Record a task a few times on the live view in the panel and train the recordings into a program.
• Write a flow, a short YAML script, by hand.

For the AI Assistant and the Pilot, which can operate Androids and teach tasks for you, see AI assistants.

How elements are found#

Taps do not use fixed coordinates as the first choice. Before every step the farm reads the screen's view hierarchy, the tree of elements Android keeps for accessibility, and looks for the element the step means.

A trained program carries evidence about each element, not an exact description, because an app update can rename an id, reuse one id for every row of a list, or draw the same view one level deeper. Each kind of evidence adds to a score and the best element above a threshold wins:

An element is acted on at a score of automation.match_threshold (default 3.0) or more. Among equals, the one nearest to where it was recorded wins, which tells two "Go" labels apart.

A flow uses exact selectors instead. The author says precisely what to look for, and a near miss must not be acted on.

How the hierarchy is read is automation.hierarchy_reader. uiautomator is the built-in dump, which waits for the screen to go idle. u2 is a persistent reader that never waits, which matters on screens that never go idle, such as video; install it with pip install "uiautomator2>=3.2" in the farm's virtual environment (the u2 extra). auto uses u2 when it is installed. On a busy screen the persistent reader was about 20 times faster than the built-in dump.

Record a task#

Recording happens in the panel, because it needs the live view. The panel sends each tap to the farm, which reads the screen first, works out which element is under the point, saves a picture of the area, and only then sends the tap. Sampling the screen on a timer would attribute a tap to the previous screen when you tap right after a change. Every step is saved as it happens, so a crash loses nothing.

1. Start an Android and open Automation, then the Record & train tab.
2. Create a dataset: a name for the task and the app's package name. A dataset groups recordings of one task.
3. Choose the dataset and a running Android and press Start recording.
4. Do the task on the live view. Use the buttons beside it for what a tap cannot say: Type a named value, Launch app, Wait, "Next tap only checks the element is visible", Undo last step.
5. Press Stop and save. Record the same task two more times.

What gets recorded about typing matters. What a person typed into a field is content, not identity, so the element's text is blanked for editable and password fields. Typed text is saved literally unless you give it a name: "Type a named value" types the value now but saves only its name, and the value is supplied for each Android when the program runs. The recorder refuses to save literal text when the focused field looks like a password field (or when the screen cannot be read to check), and asks for a named value instead. Names containing words from automation.forbidden_field_words (password, passwd, pin, otp, 2fa, seed, card, cvv, ssn by default) count as secrets.

Recording is a panel feature. The CLI lists datasets (sf auto datasets) and trains them (sf auto train), and the API has the recording routes under /api/automation/record.

Train recordings into a program#

Training is not model training. A handful of demonstrations cannot tune anything. It is a consensus over the recordings, in two passes:

1. Consensus, deterministic. The recordings are lined up step by step by similarity (same kind, same element, how close the taps were, order), because the same tap can be captured with a different element in different recordings. The recording that agrees most with the others is the template. A step is kept when at least half the recordings made it, its element is decided by majority vote, and its coordinates are the median, so one bad capture is outvoted instead of dropping the step. This pass alone produces a runnable program.
2. A model, optional. If a model is configured it writes what each step is for. It can only annotate. It never adds, reorders or invents a step. Without a model the program is equally valid, only less described.

sf auto datasets
sf auto train open-settings
sf auto programs

In the panel, press Train on a dataset, then see the result under Programs, where Steps shows the outline and flags steps that can be found only by position.

Read the warnings training prints: a step that silently vanished is how a program works in the test and fails on the fleet. The warnings say, among others:

• the program was built from a single recording, so nothing could be cross-checked;
• with fewer than three recordings, one bad capture can outvote the rest;
• steps that appeared in fewer than half the recordings were left out;
• steps were not in every recording and will be skipped when their element is missing (optional steps);
• taps have no id or text and can be found only by position, or the recordings tapped far apart at a step with nothing to tell them apart;
• the recordings were made in different apps, or nothing could be derived because they may be of different tasks.

Programs#

A trained program is stored under <home>/automation/programs/<name>/. Each step has a selector voted from the elements touched, the alternatives the other recordings saw, the device coordinates as a fallback, a picture of the area, and whether it appeared in every recording.

When a step runs, the farm resolves it in this order:

1. The step's element by score, then the alternatives the other recordings saw.
2. The recorded coordinates, but only if the saved picture of that area still matches the live screen.
3. A fix remembered for this app and screen.
4. A model, if one is configured, for a bounded number of attempts.
5. A person.

Run a program on a selection:

sf auto run program open-settings 1-10
sf auto run program sign-in tag:eu --var user=demo --secret password

Flows#

A flow is a YAML file of steps. It says what to do in terms of what is on screen and never where. This is the shipped examples/settings-open-entry:

name: settings-open-entry
description: Open Settings, scroll to an entry on its home screen, open it and wait for its page.
package: com.android.settings
variables:
  entry: Accessibility     # the exact title of an entry on the Settings home screen
timeouts:
  step: 30
  run: 180
steps:
  - launch: {fresh: true}
  - wait_for: {id: homepage_title}
    label: wait for the Settings home screen
  - scroll_to: {id: title, text: "{entry}"}
    max_swipes: 10
  - tap: {id: title, text: "{entry}"}
  - wait_for: {id: collapsing_toolbar, desc: "{entry}"}
    label: wait for the page of the entry

Wait for an element, never for a fixed delay. {entry} is a variable: it has a default here, and variables: {name: null} means the run must supply it. The flow does not use the Settings search bar, because the default package trim disables the app behind it; list com.android.settings.intelligence in performance.trim_keep to keep the search bar. Everything is validated when the flow is loaded. An unknown key, a step with two actions or a bad regular expression is an error with its position, not a surprise halfway through a run on forty devices. YAML aliases are refused, and a flow is limited to 256 KB.

Manage flows:

sf auto flows ls                        # includes the shipped examples
sf auto flows show examples/open-app
sf auto flows validate login.yaml       # check without saving
sf auto flows save login login.yaml
sf auto flows rm login
sf auto run flow login 1-10 --var user=demo --dry-run

In the panel: Automation, then the Flows tab. Pick a flow, edit, Validate, Save, Run…. The bundled examples are read-only; change the name to save your own copy.

Shipped examples#

The files are in superfarm/data/flows/examples/, with comments that explain each step.

Step reference#

The block below is generated from the models in superfarm/automation/flows.py by scripts/gen_flow_docs.py. It is current by construction.

Selectors#

A selector describes one element on the live screen. It is a mapping with any of these keys, a bare string (exact visible text, or id:, text:, desc: and regex: prefixes as a shorthand), or @name to use a role taught in the app's profile. Text values may contain {variables}. A selector needs at least one of the identifying or flag keys, and text and text_regex cannot be combined.

Conditions#

A condition says what must be true on screen. A bare selector means visible. A condition names exactly one of these keys.

Steps#

A step names exactly one action: launch, stop_app, tap, long_press, type, clear, key, back, home, swipe, scroll_to, wait_for, wait_gone, assert, sleep, if, repeat, set, screenshot, call, fail. A step that names two, or a key the action does not accept, is refused when the flow is loaded, with its position. back and home can be written as a bare word (- home). These keys work beside every action:

launch#

The bare value is package: launch: <package>. A mapping with the keys below also works.

stop_app#

The bare value is package: stop_app: <package>. A mapping with the keys below also works.

tap#

The value is a selector: tap: <selector>.

long_press#

The value is a selector: long_press: <selector>.

Keys beside the action:

type#

The bare value is text: type: <text>. A mapping with the keys below also works.

clear#

The value is a selector: clear: <selector>.

key#

The bare value is name: key: <name>. A mapping with the keys below also works.

back#

The value is a mapping: back: {key: value}.

home#

Takes no arguments: - home.

swipe#

The bare value is direction: swipe: <direction>. A mapping with the keys below also works.

scroll_to#

The value is a selector: scroll_to: <selector>.

Keys beside the action:

wait_for#

The value is a condition: wait_for: <condition>.

wait_gone#

The value is a selector: wait_gone: <selector>.

assert#

The value is a condition: assert: <condition>.

Keys beside the action:

sleep#

The bare value is seconds: sleep: <seconds>. A mapping with the keys below also works.

if#

The value is a condition: if: <condition>.

Keys beside the action:

repeat#

The bare value is times: repeat: <times>. A mapping with the keys below also works.

These keys may also be written beside the action, as in the shipped examples: until.

Keys beside the action:

set#

The value is a mapping: set: {key: value}.

screenshot#

The bare value is name: screenshot: <name>. A mapping with the keys below also works.

call#

The bare value is flow: call: <flow>. A mapping with the keys below also works.

These keys may also be written beside the action, as in the shipped examples: with.

fail#

The bare value is message: fail: <message>. A mapping with the keys below also works.

Pop-up rules#

overlays in a flow, and in an app profile, are pop-up rules. A rule fires only when its guard element and its dismiss control are both on screen, or when key: back is used with a guard that has an id.

Top-level keys of a flow#

App profile keys#

A profile belongs to one package and is stored as automation/profiles/<package>.yaml.

App profiles#

A profile holds what the farm has been taught about one app, so every flow and trained program for that app shares it. It lives in <home>/automation/profiles/<package>.yaml.

A flow with a package gets that app's roles, overlays and never-tap list in addition to its own. A trained program gets the profile of its package. Roles are taught from a live screen: read the Android's elements, pick one and give it a name; the farm stores a selector for it, or tells you why the element has nothing stable to name it by.

Profiles are managed through the API under /api/automation/profiles/{package} (roles, overlays, never-tap, and teach to turn an element of a live screen into a role). There is no panel page or CLI command for profiles in this version. API reference lists the routes.

Values and pools#

Anything in braces in a flow, {name}, is a value looked up for each Android in this order:

1. The variables of the run: the flow's defaults, then what you passed with --var, then anything a set step captured.
2. The built-ins instance_id and serial.
3. pool.<name>: one line taken from that pool, the same line for the rest of the run.
4. The Android's own value of that name.
5. The shared value of that name.

sf auto values set city Berlin
sf auto values set --instance 3 nickname demo
sf auto values pool-add codes codes.txt        # one value per line
sf auto values ls

Names that look like secrets (password, pin, otp and the other automation.forbidden_field_words) are refused by the store. The values folder is plain files and would end up in backups and model prompts. Pass such a value for one run with --secret NAME, which asks without echo or reads stdin. It lives in that run's memory only. A flow default or a schedule variable with a secret-looking name is refused in the same way.

Only printable ASCII is typed by flows and programs. Text with other characters stops the step with an explanation, because silently typing something else would be worse. The panel's own typing on the live view handles any text by pasting.

Runs#

Starting a run makes one task. Each Android in it has its own state and thread, automation.max_parallel (8) Androids work at the same time, and the rest wait as queued.

sf auto run flow examples/open-app 1-10 --var package=com.android.settings
sf auto run flow examples/open-app 1-10 --var package=com.android.settings --dry-run
sf auto runs                 # all runs
sf auto runs t12             # one run, Android by Android
sf auto resolve t12 4 resume # resume | skip | abort

A dry run reads the screens and matches every element but taps, types and presses nothing, and leaves pools untouched. It reports the steps that would not be possible on the current screen. Use it before the first real run on a fleet.

When an Android needs attention, look at it on the Control page or the live view, fix the screen by hand if that is the answer, and choose Resume (carry on), Skip this step, or Stop this Android. In the panel these are buttons on the run in the Runs tab. An Android that waits longer than an hour gives up and ends as needs_attention. Whether a flow asks a person or skips is its on_unknown setting: stop asks, continue skips. A step marked optional is skipped, and an assert always stops.

Every wait has a deadline (automation.step_timeout 20 s, automation.run_timeout 900 s per Android, and per-flow timeouts). After each action the farm pauses automation.settle_seconds (0.6) before reading the screen again. With automation.human_jitter on (default) tap positions and pauses vary slightly, as a person's do.

The state of runs is kept in memory. It is lost when the daemon restarts, and the daemon keeps the most recent 50 runs. A restart does not stop the Androids themselves.

Schedules#

sf auto schedule add nightly flow examples/open-app all --cron '0 3 * * *' --var package=com.android.settings
sf auto schedule add poll program check-feed tag:eu --every 1800
sf auto schedule ls
sf auto schedule disable s3

A schedule is either --every SECONDS (at least 30) or a cron line minute hour day-of-month month day-of-week, in the host's local time. Fields take *, a number, a range, a list or a step such as */5; day of week is 0 to 7 with 0 and 7 both Sunday. As in cron, when both day fields are restricted either one counts.

Schedules are deliberately small. A schedule never overlaps itself: if its previous run is still going when it comes due again, that occurrence is skipped and counted. Occurrences missed while the daemon was down are not replayed; one run starts if it is overdue. The daemon checks about once a minute. Schedules are stored on disk, and the variables kept with one may not be secrets.

In the panel: Automation, then Schedules.

A model for recovery#

Automation is deterministic without a model (automation.llm.provider: none, the default). A model can do two bounded jobs and nothing else:

• Annotate a trained program: say in plain words what each step is for. It cannot add, remove or reorder steps.
• Recover a stuck step: propose a few actions (tap one of the elements it was shown, press Back or Escape, swipe, wait, ask a person) to clear an obstacle. It points at elements by their number in the list it was given. Coordinates it makes up are not accepted, every tap still passes the never-tap list, and it gets a bounded number of attempts (two per step) before a person is asked.

Screen text is sent to the model as data, with the contents of fields and anything named like a secret removed first. Everything the model returns is parsed as strict JSON and cut down to those shapes; anything else is dropped.

When a recovery works, automation.learn_fixes (on by default) remembers it, keyed by the app and the screen, and replays it on the next device without asking again. The same obstacles turn up on every device of a farm, so a fix learned on one is found by all.

Configure the model with automation.llm.provider (ollama or openai, meaning any OpenAI-compatible endpoint), base_url, model, api_key_secret, timeout, temperature and num_ctx. The same setting is available as the model profile named automation for the Assistant and the Pilot (AI assistants). With no model, a stuck Android goes straight to a person.

Settings#

All under automation., with defaults. Details and ranges are in Configuration reference.

Verified and not verified#

Verified on the test machine: recording three times, training by consensus and replaying the program; the flows interpreter; the never-tap list; dry runs; hand-off of a stuck Android to a person.

Not verified: more than two real Androids driven by a run at once through Super Farm itself (fleet behaviour is covered by unit tests with fakes), Android versions other than 13, and apps other than Android's own Settings and the video app used for measurements.

Next#

• AI assistants: the Assistant and the Pilot teach and run tasks for you.
• MCP server: let an assistant run flows with your approval.
• Apps: install the app first.
• API reference: the routes behind runs, flows, values, profiles and schedules.
• Command reference: sf auto.