tapflow already gives the people who use it a zero-install experience. We wanted the person who hosts it to get the same one. So tapflow setup brings the whole host environment up in as close to one command as the toolchain allows.

https://youtu.be/RTLBJIrHf9M

Here are the DX decisions behind it.

doctor diagnoses, setup installs and configures

We split diagnosis from the steps that install and configure.

tapflow doctor is read-only: it checks the prerequisites — Xcode, simctl, a simulator runtime; the SDK, adb, an AVD — and reports. It never changes your machine, so it's safe to run anywhere, anytime. A clean run looks like this:

tapflow doctor

  ✓  Node v20.11.0

  iOS
  ✓  Xcode 16.2
  ✓  xcrun simctl
  ✓  Simulator available (8)

  Android
  ✓  Android SDK: ~/Library/Android/sdk
  ✓  adb found: ~/Library/Android/sdk/platform-tools/adb
  ✓  AVD available: tapflow-phone

  All checks passed.

When something's missing, each failing line carries the exact fix — ⚠ AVD → No AVD found. Run: tapflow setup android — so doctor always points straight at setup.

tapflow setup is the one command allowed to install and configure. The two mirror each other, so the mutating verb lives in exactly one place. Run setup with no argument and it reads the environment:

if (process.platform === 'darwin') platforms.push('ios')
if (resolveAdb() !== null) platforms.push('android')

macOS implies iOS; an existing adb implies you care about Android. If neither signal is there, it asks instead of guessing.

iOS: installed ≠ usable

Xcode can only come from the App Store, so setup doesn't fake it — it opens the right page and waits.

The part we kept getting wrong was that a freshly installed Xcode isn't a working one. Three steps stand between "the app exists" and "xcodebuild runs": point the active developer directory at Xcode (xcode-select -s), accept the license, and finish first launch. setup runs them for you, after asking, since they need sudo. The check that matters isn't "does Xcode.app exist" — it's whether xcodebuild -version actually runs.

Android: a self-contained SDK

This is the decision we're happiest with.

The obvious path is "install Android Studio." We didn't. The host doesn't need a GUI IDE, and depending on one means fighting whatever SDK location, ANDROID_HOME, and AVDs the user already has. Instead setup builds a self-contained SDK under one path we own:

sdkmanager --sdk_root=~/Library/Android/sdk \
  "cmdline-tools;latest" "platform-tools" "emulator" \
  "system-images;android-35;google_apis;arm64-v8a"

After that, every Android binary tapflow touches comes from inside that directory. A couple of details that make it reliable:

• sdkmanager needs a JDK or it won't run, so a Temurin check happens first.

• The system image is google_apis, not the Play Store one, which is unstable the way we drive it.

The one thing that has to outlive the process is ANDROID_HOME on your PATH. setup writes it into your shell rc inside a marker block, and only if it isn't already there — so re-running never duplicates it:

# >>> tapflow android sdk >>>
export ANDROID_HOME="$HOME/Library/Android/sdk"
export PATH="\(ANDROID_HOME/platform-tools:\)ANDROID_HOME/emulator:$PATH"
# <<< tapflow android sdk <<<

The catch it can't remove: the variable isn't in your current shell, so setup tells you to open a new terminal before doctor.

setup prepares; the relay boots

For both platforms, setup stops at "a bootable device exists." It never boots a simulator or emulator. That's the relay's job — it boots the right device on demand when a teammate joins a QA session. Two components owning device lifecycle would just race each other.

One rail under all of it

Every step that changes the machine asks first, and only auto-runs in an interactive terminal. Run setup in CI and instead of curling an install script as root, it prints guidance and exits clean. Nothing in tapflow deletes your data — the only teardown command is reset, which shuts down running simulators.

Honest limitations

• Xcode is still a manual App Store download. There's no API for it; setup automates everything around it.

• The first run usually needs a new shell for the Android PATH to take effect.

• The agent host is a Mac, since that's the only place iOS simulators run. (The relay itself runs on Linux.)

• Still v0.x, so the steps will keep moving as the toolchain shifts.

Takeaway

The downloads were never the hard part. The DX lives in the glue around them — Xcode activation, the PATH that needs a fresh shell, knowing to stop at "bootable" and let the relay boot the rest. Automating a dev environment means automating everything that isn't the install.

Try it

tapflow is MIT licensed.

npm install -g tapflow
tapflow doctor     # what's missing?
tapflow setup      # set it up
tapflow start

• 🔗 GitHub: https://github.com/jo-duchan/tapflow

• 📖 Docs: https://www.tapflow.dev

JPEG has one great property for interactive streaming: every frame is independent and decodes instantly. There's no buffer, no inter-frame dependency. On localhost it feels like you're touching the simulator directly.

It also has one terrible property: size. A full-frame JPEG of a scrolling screen is ~590KB. On a LAN that's 12–16 MB/s, and our relay started dropping 16–27 frames a second under backpressure — visible tearing.

So we did the obvious thing and moved to H.264. Bandwidth dropped roughly 140× on a still screen and 5× while scrolling. Drops nearly vanished.

And the stream felt worse.

This post is about why, and the two fixes that got H.264 back to "feels like direct touch."

The bar: localhost JPEG

Before touching anything I needed a number, not a vibe. So I instrumented the pipeline end to end — a per-stage panel that reports decode→present and glass→glass (capture timestamp to on-screen) latencies live.

One caveat I'll repeat throughout: glass→glass absolute values are only valid on localhost, where capture and display share one clock. decode→present is a same-machine delta and valid anywhere, so I'll lean on it for the cross-environment claims.

Here's the baseline that mattered, measured on localhost:

H.264 decode was ~20× slower than JPEG. On a hardware decoder. That made no sense — until I looked at what the decoder was actually doing.

Fix 1: the decoder was buffering 8 frames for no reason

The transport was clean (~1ms), the input queue was empty. The latency was entirely inside the decoder: it was holding ~8 frames before emitting the first one.

That's a DPB (decoded picture buffer). A decoder reorders frames when B-frames are present — it has to wait for future frames to arrive before it can output the current one in display order. So it buffers up to the level's maximum.

But our encoder is baseline H.264, B-frames off. There is no reordering. The actual reorder depth is zero. The decoder was buffering anyway because the bitstream never told it the reorder depth was zero.

The signal lives in the SPS (sequence parameter set), in the bitstream_restriction flags inside VUI. Our VideoToolbox encoder wasn't setting them, so the decoder fell back to the worst case for the level — max_dec_frame_buffering of ~8 frames at Level 5.0.

The fix is to rewrite the SPS and inject the missing declaration:

max_num_reorder_frames = 0
max_dec_frame_buffering = num_ref_frames

We do this in the agent, on the keyframe SPS, before the frame ever leaves the Mac — so every decoder downstream benefits, not just one browser path:

// agent-core/utils/sps.ts — rewrite the SPS to declare zero reordering
function rewriteLowLatencySps(sps: Uint8Array): Uint8Array {
  const bits = new BitstreamWriter(parseSps(sps))
  bits.vui.bitstreamRestriction = true
  bits.vui.maxNumReorderFrames = 0
  bits.vui.maxDecFrameBuffering = bits.numRefFrames
  return serialize(bits)
}

Result on localhost:

267 → 2.5ms, roughly 100×. The encoder was lying to the decoder by omission, and the decoder defended itself by buffering. One declaration fixed it.

The browser confirms it's receiving the rewrite — the SPS now reports bitstreamRestriction: true, maxNumReorderFrames: 0.

Fix 2: MSE is a buffer you can't turn off

Fix 1 only helps the WebCodecs path. And WebCodecs has a hard constraint: it only runs in a secure context — HTTPS or localhost.

A team using tapflow over their LAN hits it at plain http://<mac-ip>:4000. That's a non-secure context, so the browser can't use WebCodecs. The fallback at the time was MSE (Media Source Extensions): feed the H.264 into a <video> element through a muxer.

The problem is that <video> is a buffer. It's designed for media playback, where a jitter buffer is a feature. For interactive streaming it's structural latency you can't remove. I measured it on localhost by forcing the MSE tier:

~235ms, on the same reorder=0 stream that WebCodecs decoded in 2.5ms. The SPS fix can't reach this — it's the media-element buffer, not the decoder's DPB. I'd already set the muxer's flushingTime to 0. There was nothing left to shave.

So I stopped trying to make MSE fast and removed it.

The decoder layer is now two tiers, picked automatically per environment:

// pickDecoder — secure → WebCodecs, otherwise WASM
export function pickDecoder(): Decoder | null {
  if (isSecureContext && 'VideoDecoder' in window) {
    return new WebCodecsDecoder()      // HW, lowest latency
  }
  if (webgl2Available && wasmSupported) {
    return new WASMDecoder()           // tinyh264, zero-buffer
  }
  return null                          // → fall back to JPEG
}

On non-secure LAN-HTTP, we decode H.264 in WASM (tinyh264). It's a software decoder, so it costs CPU — but it has no media-element buffer at all. That's the whole point: it gives you JPEG's immediacy with H.264's bandwidth, on plain HTTP.

Measured on localhost (the worst case — encoder and decoder share one Mac):

That's on par with the localhost-JPEG baseline (12.4 / 9.4) — the bar we set at the start. Removing MSE also let us drop the muxer dependency entirely.

One constraint this introduces: tinyh264 only decodes baseline H.264. iOS already encodes baseline. For Android we pin scrcpy to baseline (profile:int=1) so both platforms share the exact same HTTP→WASM path. High profile is still available on the WebCodecs (secure) tier.

One more thing: dropping H.264 isn't like dropping JPEG

There's a subtlety the switch exposed. With JPEG, every frame is a keyframe, so dropping a frame under backpressure is harmless — the next one stands alone. With H.264, if you drop a P-frame, every following P-frame references something the decoder never received. A zero-buffer decoder like WASM tinyh264 shears until the next IDR arrives.

So the relay had to become keyframe-aware: once it starts dropping under backpressure, it drops the whole GOP until the next keyframe, rather than handing the decoder a broken reference chain. The keyframe flag rides in our frame envelope, so this needs zero NAL parsing on the relay.

// relay — once dropping, drop until the next keyframe
if (backpressured) {
  if (!frame.isKeyframe) return       // skip P-frames in a broken GOP
  dropping = false                    // keyframe resets the chain
}

Honest limitations

• WASM decode is CPU-bound. At high resolution × fps it hits a CPU ceiling. We mitigate by downscaling the encode resolution — the display is small, so it's a triple win on bandwidth, CPU, and latency.

• The localhost numbers are best-case for latency and worst-case for CPU. On a real LAN the decoder runs on a separate machine. In our cross-machine measurements, scroll p95 climbs to ~50ms on both decoders — at that point the bottleneck is load/transport, not the codec. The decode→present deltas above hold; the glass→glass absolutes do not transfer across two clocks.

• Still v0.x. The decoder tiers and SPS rewrite are in agent-core; expect them to keep moving.

Takeaway

Two bugs, same symptom ("H.264 feels laggy"), completely different causes:

1. The decoder's DPB buffered 8 frames because the SPS didn't declare reorder=0. Fix: rewrite the SPS at the encoder.

2. The media-element buffer in MSE added ~235ms that no encoder flag can reach. Fix: remove MSE, decode in WASM on non-secure contexts.

The lesson I keep relearning: when streaming feels slow, measure each stage before you change the codec. The codec usually isn't the problem — the buffer you didn't know you had is.

Try it

tapflow is MIT licensed.

npm install -g tapflow
tapflow start

• 🔗 GitHub: https://github.com/jo-duchan/tapflow

• 📖 Docs: https://www.tapflow.dev

Unit tests and API tests run in CI automatically. But the thing that actually matters to most users — does tapping this button do the right thing, does this screen look right after this flow, does the deeplink open the correct state — none of that runs automatically. Someone has to open the simulator, walk through the steps, and verify. Every time.

The usual answer is Appium or XCUITest. But those require engineers to write and maintain test code that mirrors the UI, breaks whenever the screen changes, and only runs against builds developers already have locally.

We had a different idea. tapflow already lets humans control a simulator through a browser. What if we gave an LLM the same interface?

The interface a human uses

When a person does QA in tapflow, the loop is:

1. Look at the simulator screen

2. Decide what to do (tap, swipe, type)

3. Do it

4. Look again

This is exactly the perception-action loop that vision-capable LLMs are built for. The model sees a screenshot, reasons about what it shows, decides what action to take, and calls a tool to execute it.

We didn't need to build a new automation layer. We just needed to expose tapflow's existing WebSocket and REST APIs as MCP tools.

What the MCP server does

@tapflowio/mcp-server connects to a running tapflow relay and registers 13 tools that any MCP-compatible client can call:

list_devices       — see all simulators registered on the relay
connect_device     — join a device session
boot_device        — boot a simulator (waits up to 30s for ready state)
screenshot         — capture the current screen
tap                — tap at a pixel coordinate
swipe              — swipe between two coordinates
type_text          — type into the focused field
press_key          — press a keyboard key (Return, Delete, Escape...)
press_button       — press a hardware button (home, lock)
install_app        — install a build from App Center
launch_app         — launch an installed app
list_builds        — list available builds on the relay
disconnect_device  — end the session

Setup is two environment variables:

TAPFLOW_RELAY_URL=wss://your-relay-url
TAPFLOW_TOKEN=your-pat-token
npx @tapflowio/mcp-server

Add it as an MCP server in your client config, and those tools appear in the model's tool list.

How the tools are implemented

Screenshot — the model's eyes

The screenshot tool calls the REST endpoint we added in v0.3.0 (GET /api/v1/sessions/:id/screenshot), gets back a PNG or JPEG buffer, base64-encodes it, and returns it as MCP image content alongside the pixel dimensions:

return {
  content: [
    { type: 'image', data: buf.toString('base64'), mimeType },
    { type: 'text', text: `Screenshot saved: \({filePath} (\){width}×${height}px)` },
  ],
}

The model receives the actual image. It can read text on screen, identify UI elements, notice error states — the same things a human would.

Tap and swipe — normalized coordinates

Here's the part that took a few iterations to get right. The simulator's logical coordinate space is different from screenshot pixel coordinates, and it changes with screen resolution, device type, and scale factor.

Rather than exposing logical coordinates (which the model can't reason about without device-specific knowledge), we have the model work entirely in screenshot pixel space. The tap tool takes pixel coordinates plus the screenshot dimensions, then normalizes internally:

// tools.ts
client.tap(sessionId, x / screenshotWidth, y / screenshotHeight)

The model calls screenshot first, reads the dimensions from the response, then uses those same dimensions when calling tap. This means the model can identify "the button is at roughly pixel 200, 450" from the image and tap it directly — no coordinate system translation required.

Swipe works the same way, with 8 interpolated touch:move events across the duration to simulate a natural gesture:

// client.ts — swipe interpolation
const STEPS = 8
const interval = durationMs / STEPS

this.send({ type: 'input:touch:start', sessionId, payload: { x: startX, y: startY } })
for (let i = 1; i < STEPS; i++) {
  await delay(interval)
  const t = i / STEPS
  this.send({
    type: 'input:touch:move',
    sessionId,
    payload: {
      x: Math.round(startX + (endX - startX) * t),
      y: Math.round(startY + (endY - startY) * t),
    },
  })
}

Async operations over WebSocket

Several tools involve async operations — booting a device, installing an app — where the relay sends a confirmation back over WebSocket after the operation completes.

The client uses a waitFor pattern: register a predicate against incoming messages, return a promise that resolves when a matching message arrives, and reject if a timeout fires first.

// client.ts — waitFor
private waitFor(predicate: (msg) => boolean, timeoutMs: number): Promise<RelayMsg> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => {
      this.waiters.splice(this.waiters.findIndex(w => w.resolve === resolve), 1)
      reject(new Error('Request timed out'))
    }, timeoutMs)
    this.waiters.push({ predicate, resolve, reject, timer })
  })
}

boot_device waits up to 30 seconds. install_app waits 60 seconds. Each resolves on the confirmation message or rejects with the error payload.

What a session looks like

A model running a login flow might do this:

1. list_devices → pick a session
2. connect_device
3. list_builds → find the build to test
4. boot_device
5. install_app
6. launch_app
7. screenshot → see the login screen
8. tap(email field coordinates) → focus the input
9. type_text("test@example.com")
10. tap(password field coordinates)
11. type_text("password")
12. tap(login button coordinates)
13. screenshot → verify the home screen loaded
14. disconnect_device

Each screenshot gives the model a chance to verify state before proceeding. If step 13 shows an error message instead of the home screen, the model knows something went wrong.

Where we are: experimental

The version says 0.3.1-experimental.1 for a reason. The tools work, but the layer needs more hardening before we'd call it reliable.

The core issue is consistency. The same sequence of tool calls should produce predictable behavior every time. Right now it doesn't always — there are timing edge cases where an action fires before the UI has fully settled, device state can drift between steps without the model noticing, and error recovery when something unexpected happens mid-flow is rough.

These are solvable problems, but we want to solve them before presenting this as something teams should build pipelines on.

Where we're going: CI/CD without a QA script

The direction we're aiming at is using the MCP server as the foundation for LLM-driven smoke tests in CI.

The scenario: a new build passes unit tests and gets uploaded to App Center. A CI step spins up the MCP server, points it at the relay, and gives a model a natural-language test spec:

"Install the latest build. Log in with test credentials. Navigate to the cart, add an item, and confirm the checkout screen shows the correct total. Take a screenshot at each step."

The model does the steps, captures evidence, and reports what it saw. No automation code to write. No selectors to maintain when the UI changes. The spec is just a description of what a human would do.

This isn't production-ready yet. The stability work comes first. But the pieces — browser-controllable simulators, screenshot REST endpoint, MCP tool layer — are in place. The question is whether the model can run a flow reliably enough to be trusted in CI without a human verifying each run.

We think it can. That's what we're building toward.

Try the MCP server (experimental)

npm install -g @tapflowio/mcp-server@experimental

You'll need a running tapflow relay and a PAT token with viewer scope. Configure it in your MCP client:

{
  "mcpServers": {
    "tapflow": {
      "command": "npx",
      "args": ["@tapflowio/mcp-server"],
      "env": {
        "TAPFLOW_RELAY_URL": "wss://your-relay-url",
        "TAPFLOW_TOKEN": "your-pat-token"
      }
    }
  }
}

If you try it and hit rough edges, open an issue — that feedback is exactly what's shaping the stability work.

• 🔗 GitHub: https://github.com/jo-duchan/tapflow

• 📖 Docs: https://www.tapflow.dev/guide/mcp-server