When screen access is useful
Visual access is most useful when the AI needs to interpret the present situation and no direct game-state integration exists. A coaching companion might identify an enemy, read an inventory screen, notice a puzzle arrangement, or comment on a match.
Screen access can also reduce the effort required to explain a complex scene. It does not automatically make the answer accurate; the system may still misread text, miss off-screen information, or misunderstand game-specific rules.
Other ways a companion can receive context
Player conversation
The player describes the game, objective, build, or problem through text or voice.
Game integration
An API, mod, plugin, or game engine exposes structured state.
Telemetry
The system receives match statistics, events, or performance data.
Commands
An agent acts from explicit instructions and the available action space.
Saved state
A continuity companion restores named records about the correct game and run.
Screen access by companion type
| Type | Need for screen access |
|---|---|
| Coaching | Often useful, but player description or telemetry can substitute |
| In-game | Usually unnecessary because the system already has game-state access |
| Agentic | Depends on integration; structured state may be better than pixels |
| Social | Usually unnecessary |
| Continuity | Usually unnecessary; saved records are the primary context |
What a screen-aware system should disclose
- when capture begins and ends;
- whether audio, other windows, notifications, or overlays are included;
- whether images are stored or processed temporarily;
- which processors receive the data;
- how the user pauses or revokes access;
- how accidental private information is handled.
These are product-specific privacy questions. Verify them through the current official documentation of the system being used.
Pixels versus structured state
Screen vision is broad and works with many games, but it sees only what is visible. Structured game integration can be more precise, but it requires technical access and may be game-specific. Player-provided context gives the user strong control, but requires more explanation. No single input is universally best.
Frequently asked questions
Can a continuity companion work without seeing the game?
Yes. It can preserve and restore player-approved records such as the game, character, goals, decisions, and intended next step.
Is screen access the same as game integration?
No. Screen access interprets pixels. Game integration can expose structured data directly.
Does screen access mean the AI is always watching?
Not necessarily. The exact capture behavior depends on the implementation and should be disclosed clearly.