The gist

Skill Recorder is an open-source desktop application from Microsoft that lets you record yourself performing a task on your computer. It captures screen activity, clicks, app switches, and browser navigation to solve the problem of teaching AI agents new skills. By recording a task once, you can convert it into a reusable procedure or a scheduled automation that an agent can execute on its own, generalizing from your single example.

What it does

Record screen activity, including clicks, app and window switches, and browser URLs.
Capture optional spoken narration and transcribe it on-device using Whisper.
Analyze a recording with GitHub Copilot to reconstruct the user's intent and generate an ordered list of steps.
Generate a reusable skill as a SKILL.md procedure for an AI agent to run on demand.
Create a scheduled or triggered automation based on the recorded procedure.

How it works

A user starts recording via a hotkey and performs a task. The desktop app captures screen activity locally. Once finished, the user can choose to 'Analyze', which sends the event timeline, screen images, and narration text to GitHub's cloud for Copilot to process. The output is a structured procedure that can be saved as a reusable skill or automation. The tool is a source release built locally and requires a GitHub Copilot subscription.

Best for

This tool is best for developers looking to quickly teach an AI agent a new multi-step task that involves interacting with GUIs. It excels at creating reusable automations or agent skills from a single real-world demonstration.

Watch out for

Users must be careful not to record any sensitive information like passwords or API keys, as recording data is sent to GitHub's cloud for analysis. The tool's primary target is macOS, though Windows is also supported.