What your agent receives

The prompt

When you let go of ⌃ ⌥, this is what lands on your clipboard, with the full paths to your Exports folder:

I made a recording that contains annotations for you to understand what I need you to do.
Here are my instructions: ~/Movies/Kirocast/Exports/TalkMode_1790274261.transcript.json
Here is the recording: ~/Movies/Kirocast/Exports/TalkMode_1790274261.mp4
Here are the annotation keys, and necessary elements to understand the recording: ~/Movies/Kirocast/Exports/TalkMode_1790274261.events.json

These files are still being written as you read this. If one is missing, wait a few seconds and read it again. If the instructions file never appears, nothing was said during the recording: work from the video and the annotations.

The files are listed in the order to open them. The last paragraph is there because the prompt reaches your clipboard a few seconds before the files are finished, so your agent knows to wait for them.

Your words

The instructions file is the transcript of what you said, made on your Mac. Your words are split into segments at each pause or full stop, and every word carries its own timing:

"segments": [
  {
    "confidence": 0.94,
    "endT": 1.9,
    "t": 0.5,
    "text": "Export does nothing.",
    "words": [
      { "confidence": 0.99, "endT": 0.92, "t": 0.5, "text": "Export" },
      { "confidence": 0.97, "endT": 1.24, "t": 0.92, "text": "does" },
      { "confidence": 0.86, "endT": 1.9, "t": 1.24, "text": "nothing." }
    ]
  }
]

• t and endT: when the segment or word starts and ends, in seconds into the video.
• text: what you said.
• confidence: how sure the speech model was, from 0 to 1. A low score is a hint that a word may be misheard.

The recording

The video is the display you pointed at, sized for a model to read: up to 1536 pixels on the long edge, at 5 frames per second. It is silent: your voice is in the transcript. The frame shows the whole display at every moment, with each click marked by a ping and each circle you drew turned into an oval.

Export for AI lists exactly what the video keeps.

The annotation keys

The events file is the timed record of what happened on screen. Two kinds of entries matter most in a Talk mode take:

{ "button": "left", "clickCount": 1, "t": 2.3, "track": "cursor", "type": "click", "x": 0.612, "y": 0.084 },
{ "attentionPriority": "high", "endT": 6.2, "intent": "highlight_target", "interaction": false,
  "rect": { "height": 0.062, "width": 0.071, "x": 0.577, "y": 0.054 },
  "t": 3.4, "target": { "kind": "button", "label": "Export" }, "track": "effects", "type": "focus" }

• A click ("track": "cursor"): you clicked at x, y, at time t.
• A circle you drew ("type": "focus"): the area you circled, as rect, from t to endT. "interaction": false tells your agent this is something you pointed at, kept apart from the clicks.
• The target: the element under your circle, read from macOS Accessibility. Here, a button labelled Export. It appears when macOS can name the element.

Positions run from 0 to 1, measured from the top-left corner of the display. The events file describes every field and every kind of event.

Reading them together

All three files share one clock, the seconds of the video, so an agent can line them up:

1. Read the transcript for the request itself.
2. Match each sentence to the events at the same t. “Export does nothing”, said at 0.5 s, followed by a click at 2.3 s and a circle around the Export button at 3.4 s.
3. Look at the frame at that moment. The video shows the whole display edge to edge, so a position converts straight to pixels: x × the video’s width, y × its height. In a video 1536 pixels wide, an x of 0.612 is pixel 940 across.

Your spoken words are the instructions. To add something you didn’t say, such as which project to work in, type it after the prompt when you paste.