A replacement locator can make a test green while directing it to the wrong button. Recovery and correctness are separate decisions. In the Optics local-model experiment, the assertions after each press and the framework fixes matter as much as the headline recovery count.
The developer repository reports 113 correct repairs out of 114 broken locators on Google Clock running on a Pixel 4a, with 27 of 28 test cases passing. These are developer-run experiments. Neither the device nor the model was run for this article.
Recovering intent from a broken name
The task description supplies a screenshot and screen-element rows, then asks for a row ID and action. Matching skip_notification_button to “Not now” requires relating purpose to current UI text.
Locator failure → screen, element rows, intended element → repair candidate
↓
Preserved log ← next-screen assertion ← bounded action execution
This is our operational interpretation. Letting the repair model also rewrite the expected result would compromise the test oracle: the same component could make a wrong action appear correct.
In Optics PR #705, closest_rows in optics_framework/common/ai_self_heal.py adds words split by screen_snapshot.snake_words to the tokens extracted from the resource ID. This changes hints for names such as timerPreset1. Even with identical weights, candidate rows can change. Recovery therefore measures preprocessing, prompting, the model and the execution loop together. We inspected the diff but did not run the upstream tests.
Two experiments answer different questions
| Evaluation | Conditions | Developer-reported result |
|---|---|---|
| Recorded screens | 96 Android/iOS apps; 1,314 tap steps; four-fold cross-validation split by app | Text names 95.5% → 98.6%; purpose names 77.9% → 84.9% |
| Device pilot | Pixel 4a; Google Clock; two test cases; seven break sets run twice | Locators 113/114; test cases 27/28 |
The live-pilot description also reports four Optics bugs found and repaired. Test-case passes were 14/28 before those repairs. That change cannot be attributed to LoRA alone: hint-row handling, name matching, element-name context and actions after completion changed too.
The developer article reports 109 one-step and five two-step repairs, with one two-step repair incorrect. A recovery loop needs a termination condition as well as a candidate selector. The 114 attempts are not 114 independent apps.
Preserve failure before permitting recovery
Start by recording candidates in shadow mode without clicking. Fix the permitted action set, retry budget and termination condition, and retain the original failure. Destructive or externally visible actions can require a separate human decision. These are our operational recommendations, not security guarantees supplied by the project.
Track incorrect clicks, abstentions and next-screen assertion failures separately from correct repairs. Always producing a plausible candidate can hide a regression. Improving accessible names and stable test IDs may be cheaper and more reliable than a recovery model.
A small model does not imply on-phone inference
The developer served the model through vLLM on a GPU server; the usage guide calls for at least 16GB GPU memory. Local hosting also leaves screenshot privacy, logging and retention to the operator.
The gain on purpose-based names justifies a controlled trial of a specialized small model. It does not establish equal performance on other apps, languages, accessibility trees or devices. Evaluate it as a bounded assistant whose actions are checked by independent assertions. We found no sufficiently direct verified course match for a specific promotional CTA.