How do you show an AI agent what is on your screen?
Hold Control and Option anywhere on your Mac. Kirocast records the screen under your pointer and your microphone for as long as you hold: talk about what is wrong, point at it, draw a circle around anything you want looked at. Let go, and a prompt is already on your clipboard naming three files: a small video, a word-level transcript produced on your Mac, and a JSON record of every click, its timing and everything you circled, with the name of the element under it. Paste it into Claude Code, Cursor, Codex or your own agent and it reads what you meant rather than inferring it from pixels.
The hold
Talk mode has no button. It is a hold, and the take lasts exactly as long as the keys are down.
- ⌃⌥
Hold
Press Control and Option together, anywhere, in any app. A small orange circle appears beside your pointer and stays for as long as the keys are down. Kirocast records the display under the pointer and your microphone; nothing else changes on screen.
- 🎙
Speak
Say what is wrong the way you would to a colleague looking over your shoulder. The circle swells with your voice so you know you are being heard. Your words become the instructions file.
- ◯
Highlight
Point at what matters; the pointer leaves a fading trail and a click leaves a ring. Draw a loop around anything and it becomes an annotation: Kirocast reads the element under the circle, its role and its label, and writes both down. None of the ink is in the recording.
- ⌘V
Release
Let go. The prompt is on your clipboard the moment the take is on disk, before the transcript and the render have finished, so releasing and pasting are one motion. A small card in the corner says so and shows what is still being written.
What the prompt names
Three files, by absolute path, in the order an agent should open them. They are still being written when you paste, and the prompt says so, so an agent that opens them a second later waits instead of reporting them missing.
The video
TalkMode_1785864009.mp4
Rendered for a reader that samples stills instead of watching: the long edge capped at 1536 px so 11-pixel UI text stays legible, 5 fps, H.264, and the frame pared back to the screen itself. Privacy masks and click cues stay, because both change what happened. A twelve-second take lands around a megabyte.
What you said
<name>.transcript.json
Your narration as word-level segments on the video’s own timeline, produced on your Mac by the Parakeet speech model. Your agent can quote you at 0:06 and know which click you were describing. The prompt names this file as your instructions.
What happened, and what you circled
<name>.events.json
Every click with a timestamp and normalised coordinates, and every circle you drew as a focus annotation with the name of the element under it (a button, a link, a field, a block of text), alongside any zooms, privacy masks and cuts. All measured on the exported video’s own timeline, so a moment in the file is the same moment in the frame.
What is inside the events file
The format is self-describing on purpose: a model that has never seen a Kirocast file should be able to read one without being told how.
| Key | What it holds |
|---|---|
format, version, conventions | A self-describing header. The file states its own coordinate space and units, so a model needs no external documentation to read it. |
video | The exported file name, duration, resolution and frame rate: the frame of reference every timestamp below is measured against. |
events[].t | Seconds on the exported video’s timeline, not the original capture’s. Trims and cuts are already applied, so t maps straight onto the frame your agent samples. |
events[].type | What the event is: a cursor click, a focus annotation from a circle you drew, a zoom, a privacy mask, a cut. Your interactions and your annotations are kept strictly apart, so a model never reads your emphasis as your behaviour. |
events[].target | For a circle: the kind of element under it (button, link, text, image, field) and its label, read through macOS Accessibility while it was still on screen. Left out rather than guessed when nothing was under the circle. |
events[].x, .y | Normalised 0–1 coordinates in the exported frame, so a click or a circle resolves to a position regardless of the resolution the video was written at. |
effectSemantics | A map explaining what each annotation means. “Focus at 0:08” becomes “the author wanted attention here at 0:08”. |
What your agent does with it
- Find the bug. “You clicked Publish at 0:05, nothing came back for 1.4 seconds, you circled the button at 0:08, and your second click at 0:09 submitted twice.”
- Fix the thing you pointed at. The circle names the element, so “this one” in your narration resolves to a button with a label, not a region of pixels.
- Write the release note from the take you just recorded, in the order the feature was actually demonstrated.
- Locate the moment. “Where in this recording is the export dialog?” answered from the events file, not by scrubbing.
- Turn a walkthrough into written docs, with the click sequence already in order.
Questions people ask
- How do you show an AI agent what is on your screen?
- Hold Control and Option anywhere on your Mac. Kirocast records the screen under your pointer and your microphone for as long as you hold: talk about what is wrong, point at it, draw a circle around anything you want looked at. Let go, and a prompt is already on your clipboard naming three files: a small video, a word-level transcript produced on your Mac, and a JSON record of every click, its timing and everything you circled, with the name of the element under it. Paste it into Claude Code, Cursor, Codex or your own agent and it reads what you meant rather than inferring it from pixels.
- Which AI agents does Talk mode work with?
- Any agent or model that can open a file on your Mac. Claude Code, Cursor, Codex, Copilot, Cline, Windsurf and Zed all qualify, as does a chat window you paste into. The integration is the file system: plain JSON on disk, and a prompt on your clipboard naming all three files by absolute path.
- Do I have to turn Talk mode on?
- It is a hold, not a setting. Kirocast asks once for the Accessibility permission, which it needs to keep Control and Option off your clicks while the keys are down. Otherwise a Control-click would open a context menu in every app. Grant it and the hold works from then on, anywhere.
- Does the circle show up in the recording?
- No. The trail, the click rings and the circle beside your pointer are feedback for you and are never captured. What survives is what the gesture meant: a circle becomes a focus annotation in the events file, with the name of the element under it, and is drawn as a hand-drawn oval in the exported video.
- What if I let go early, or by accident?
- A release under half a second is treated as a slip and discards the take. Anything longer is a take: the prompt is copied and the card appears. Press Done on the card if you did not mean it, or the bin to delete the recording.
- What if I do not say anything?
- The take still goes out with the video and the events file. The prompt tells your agent that the instructions file never appeared and to work from the recording and the annotations.
- Why is the video small and only 5 fps?
- Because of how a model reads video. It samples frames into stills rather than watching in real time, so the thing that decides whether it can read your UI is resolution, and frame rate buys almost nothing. Capping the long edge at 1536 px keeps 11-pixel labels legible; dropping to 5 fps takes a twelve-second recording from twelve megabytes to about one. The polished export for humans is unaffected. That is a separate button on ordinary takes.
- Does anything get uploaded?
- No. The video, the transcript and the events file are written to your Mac, next to each other. Transcription runs on-device. Kirocast copies file paths to your clipboard. What your agent does with them afterwards is between you and your agent.
- Can I get the same files from an ordinary recording?
- Yes. Press Export for AI on the card that appears when any recording stops, and the same video, transcript and events file are written beside each other; Copy prompt puts their paths on the clipboard. Talk mode is that flow with the pressing taken out.
- What does Talk mode cost?
- Nothing beyond the app. Kirocast Personal is a one-time $39 launch purchase, normally $49, and every license unlocks every feature. Try all Kirocast features except export before you pay.