Action Brain: a personal agent that turns loose thoughts into handled tasks

Overview

Most to-do systems fail at the capture step. You have to stop, open an app, and type. Action Brain removes that friction: you capture however is easiest, and the agent turns what you captured into something you can act on.

The loop is simple:

1. Capture: a quick voice memo from your phone, an email to a private inbox, a link or a message in chat. Your phone also reports when you leave or get home, so the agent knows where you were.
2. Triage: the agent pulls out to-dos, dates and decisions, judges how important each one is, and researches anything that needs it (phone numbers, business hours, how a process works), always with sources.
3. Remind: reminders arrive by iMessage at sensible times, avoid the times you're busy (including work meetings), and keep coming until you mark the item done. A reminder never moves the day a task is planned for.
4. Review: a private dashboard (one-tap sign-in that lasts a year) and a native app show what's left today and what's coming later, reading from one shared database.

Architecture

Four layers, left to right. The agent runs at home; storage and anything public-facing run on Cloudflare's free tier.

Everything runs as scheduled jobs on the home device (fourteen at last count), and every one of them delivers to iMessage. The main ones:

Voice memo pipeline

The main way things get in. A memo is usually 5–30 seconds: "remind me to…", "add this trip idea…", "email my spouse that…".

1. Upload. A phone shortcut posts the audio to a small Cloudflare Worker that checks a secret token and writes it to an R2 bucket under a dated key. The shortcut setup is at the end of Build your own.
2. Tag the place. After the upload finishes, the same shortcut runs a separate location ping marked "memo". Because it runs after the upload, a GPS failure can never lose a recording. In testing the ping landed 1–3 seconds after the recording, so memos are matched to places by timestamp (see Location context).
3. Detect. A script lists the bucket every minute and compares it with a local "already triaged" list. If nothing is new, its output doesn't change, so the agent never wakes.
4. Pull only what's new. New recordings are downloaded once, tracked in a local list so nothing is fetched twice.
5. Transcribe. Audio is converted with ffmpeg and sent to Workers AI (whisper-large-v3-turbo). That costs about $0.0005 per audio minute, and the free daily allowance covers roughly 200 minutes.
6. Triage. The agent reads the transcript along with recent notes and the open task list, so a memo that repeats an existing task gets merged instead of duplicated. It also looks up where you were when you recorded it and notes the place.
7. Record. Results go to a dated triage notes file, reminders go to the database, and the memo is marked triaged only after everything is saved.
8. Recover. If a run fails, the memo is still untriaged after 10 minutes, the monitor's output changes, and the agent retries. A failure lasting 30+ minutes sends one alert.

Triage & importance

Every item gets an importance level, and that sets how persistent its reminders are. Hard things aren't crammed into tonight; they're scheduled for when you can actually act on them.

• Dates you mention explicitly always win over these defaults.
• If a task needs a business to be open, the reminder is timed for its hours. A "prepare" reminder can still come earlier.
• Each reminder carries everything needed to act: phone numbers, hours, steps, links. You never have to look anything up.
• The research the agent does cites its sources and never invents numbers, addresses or facts.

Reminder engine

A plain script, run every five minutes, decides what to send. It respects your time.

A habit reminder that would land inside a busy block moves just before it, but only if that's within two hours of its usual time. Otherwise it waits until the block ends; an evening medication reminder shouldn't jump to mid-afternoon.

Everything goes out over iMessage on a shared line, with Telegram as the fallback if that channel goes down.

Task day vs. reminder time

Each task has two separate times: due_date, the day it's planned for, and next_due, when the next reminder fires. Repeat nags advance next_due but never touch due_date, so a reminder can't quietly push a task to tomorrow.

• Unfinished tasks roll to today. A task's day is the later of its planned day and today; the dashboard, the morning agenda and the task list all use this rule.
• Deferring is explicit. "Push it to the weekend" runs reminders.py move ID WHEN, which sets both fields. The app's snooze does the same.
• Routines don't pile up. Items marked recurring=1 (a twice-daily stretch, say) show only on the day of the current occurrence and drop off once it passes, instead of rolling forward.

To make changes safely, the engine has a time-travel test mode: copy the data, set a fake "now", and step through a week in 5-minute ticks to see exactly what would be sent and when.

Calendar & schedules

Context makes reminders smarter. Everything here is read-only.

• Personal calendar through Google Calendar's secret iCal link, which is read-only and needs no OAuth app. Recurring events are expanded 14 days ahead and refreshed every 30 minutes. Events marked "free" don't block anything.
• Sports schedule from a public sports-data API, refreshed twice a day because game times change. Games with a time still "TBD" are ignored until it's set.
• Work calendars stay out. Company policy usually forbids sending them to third-party agents, so keep this personal.
• Work availability, inferred. A public appointment-booking page only offers times you're free. A script asks the page for its open 30-minute slots between 9 and 5 PT, and every missing slot counts as busy at work. No work data is ever shared.

Booking pages hide slots inside their minimum-notice window, so today can look fully booked when it isn't. Silent snapshots run six times a day, and for each day the script keeps the freshest snapshot taken outside that window, labeled "as of" with its time. Two messages use it:

• Day plan, weekdays at 8am: work busy blocks and personal events in time order, with today's tasks slotted into the real gaps and any clashes flagged (a pickup during a meeting, say).
• Look-ahead, 7:30pm daily: the next seven days. It remembers what it already said and only speaks up about new items or conflicts.

Location context

Knowing where you were makes memos clearer and reminders better timed. Your phone reports a handful of moments, and nothing else is tracked.

1. A tiny shortcut sends a ping. An iPhone Shortcut called "Herm Ping" runs Get Current Location, then POSTs {what, lat, lon} to the dashboard Worker at /api/v1/location. It authenticates with a Bearer token whose device name starts with loc:, and the Worker lets that token post locations and nothing else.
2. Automations decide when. Shortcuts automations fire on "leave Home" and "arrive Home" and pass that text along as what. Working from home means there are no office triggers. The voice memo shortcut ends by running Herm Ping with "memo".
3. The Worker stores a row. Each ping becomes a row in the D1 locations table: time, event (arrive, leave, memo, task or ping), optional place label, and coordinates.
4. The agent reads it as context. locations.py recent lists the latest pings, and locations.py near TS finds the ping closest to a timestamp; memo triage uses it to note where a memo was recorded. Coordinates become names through a private places.json of named places, with OpenStreetMap reverse geocoding as the fallback.

POST /api/v1/location
Authorization: Bearer <token>        (location-only, device "loc:…")
{"what": "leave Home", "lat": <lat>, "lon": <lon>}

Privacy

• Raw points are deleted after 30 days by a weekly job (locations.py prune).
• Coordinates never appear on the dashboard or in anything that leaves the system; only place names are used.
• The named-places file and the location token are owner-only files on the home device.

Email on your behalf

Two agent-owned inboxes with very different trust levels.

One ongoing thread

The weekday 3:30pm school summary also goes by email from the external-correspondence inbox to you and your spouse. Every day uses the same subject, and school/email_summary.py chains In-Reply-To and References to the previous message, so Gmail keeps the whole term in one thread you can both reply in. Its state lives in school/summary_thread.json, and it never sends twice in a day. Replies from your spouse are treated as information; only your own authenticated replies count as instructions.

Libraries & tracking

Some captures aren't tasks: they're things you want to come back to.

• Trip library. Send a travel reel or blog link and get a structured itinerary (day by day, where to stay) plus "check before booking" notes: seasons, road closures, permits, booking lead times.
• Recipe library. Any meal, not just dinner: send a recipe link and get the ingredients, steps, the author's tips and a ready-made shopping list. Many recipe sites block bots, so the agent uses the site's public WordPress API when the page itself is walled off.
• Weekend try list. A short weekend-try-list.md of recipes and ideas picked for the coming weekend, linked to their library entries.
• Mail & shipments. Photograph a receipt; the agent reads the tracking number (checking its check digit), records it and schedules a "confirm delivery" follow-up.
• Projects. Multi-step efforts (say, getting quotes for a home repair) roll up into one check-in with the current status, contacts and next steps, instead of a pile of separate nags.

Libraries are plain Markdown files, one per item, each with an index. They're easy to read, grep, version or move. Private files, such as the list of loyalty numbers used when booking trips, are owner-only (chmod 600) and never shown on the dashboard.

One source of truth

The first version kept tasks in local files and copied them to the cloud every few minutes. The dashboard lagged, so that design was replaced.

Now a Cloudflare D1 database is the single source of truth for tasks, habits and habit logs. Every writer and reader uses the same rows:

• the reminder engine and the triage jobs, through a small internal API using a secret token,
• chat ("done 4f2a1c", "walked 3 miles"),
• the web dashboard, rendered live from D1 on every request,
• native apps, through device tokens,
• phone location pings, through a location-only token.

Two details make it robust:

• Field-level, guarded writes. The reminder engine only updates the fields it changed, and only if the item is still open, so it can't bring back something you finished on your phone a second earlier.
• An offline fallback. If the cloud is unreachable, reminders run from a local cache and their changes are queued and replayed later.

items       (id, kind task|habit, title, details, importance, status open|sent|done,
             next_due, due_date, recurring, repeat_hours, last_sent, done_at, source)
habit_logs  (date, habit, value, ts)
events      mirrored calendar, next 7 days
games       mirrored sports schedule
actions     audit log of app actions
devices     sha256(token), name (web:… browser, loc:… location-only), last_seen
locations   ts, event arrive|leave|memo|task|ping, place, lat, lon (kept 30 days)

Web dashboard

A single Cloudflare Worker, built for a phone screen. You sign in once per device with a texted link or code, and it stays signed in for a year.

• Today: what's still on the schedule (with a NOW marker), tasks planned for today or rolled over unfinished, habits not yet logged, and a collapsed "done today" list.
• Later: grouped into Tomorrow, Next 7 days, Later, and Waiting on you.
• Also: the upcoming calendar, a 7-day habit grid, upcoming games, recent triage notes, and the trip and recipe libraries.

Signing in

1. Text "dash link". The agent replies with a magic link and a 6-digit code. Both are single-use and expire in 15 minutes.
2. Tap the link, then "Sign in". Opening the link only shows a button; the token is spent by the button's POST, so iMessage link previews can't burn it. An expired or used link falls back to the sign-in screen with the code box.
3. Or enter the code. A web app added to the home screen has its own cookie jar and can't receive a tapped link, so it uses the code instead (iOS autofills it from Messages). Five wrong tries kill all outstanding codes.
4. Stay signed in. Signing in sets an HttpOnly, Secure session cookie that lasts a year. Each browser is its own device, listed with dash_link.py devices and revoked one at a time with dash_link.py revoke ID.

Password fallback

HTTP Basic Auth still works: the sign-in screen has a "Use password instead" link, which also sets the year-long cookie. The agent never sees the password. It creates a one-time setup link that expires in 24 hours. You choose the password on that page, and only a salted PBKDF2 hash is stored. Pages are served with noindex, no-store and a strict Content-Security-Policy that blocks all scripts.

Native app & sync API

The same Worker exposes a small JSON API, so a SwiftUI app for iPhone and Mac can show the same data and act on it.

App tokens, location tokens and web sessions all live in the same devices table, stored only as hashes, and each one can be revoked on its own. iPhone and Mac apps have to be built in Xcode on a Mac. A Linux build server can host the API but can't build the app.

What it tracks

The kinds of things that have flowed through it. The categories matter more than the specifics.

• Family logistics
• Kids' school & homework
• School and sports paperwork
• Benefits & insurance paperwork
• Certified mail & tracking
• Home projects & contractor quotes
• Household chores
• Travel plans & documents
• Trip ideas
• Recipes
• Medication adherence
• Supplements
• Daily exercise
• Calendar conflicts
• Work busy blocks
• Leaving & getting home
• Where memos were recorded
• Weekend plans
• Sports schedule
• Side-project infrastructure
• Cloud quota usage

Habit tracking

Habits are just reminders with a daily repeat and a log. Reply naturally ("took my meds", "walked 3.5 miles") and the agent records it. A logged habit never sends a reminder that day, and the dashboard shows a 7-day streak grid.

Security model

The agent can run commands on a home computer, so who can talk to it, and what it trusts, matters more than anything else.

A periodic audit is worth doing. Check open ports from outside, the router's automatic port openings (UPnP), SSH settings (keys only), unneeded network services, the firewall, automatic security updates, and any old agents still running in the background.

Costs & quotas

Everything fits comfortably in Cloudflare's free tier. The bigger risk is other projects on the same account.

Free-tier limits are shared across the whole account. A daily script checks usage through Cloudflare's GraphQL analytics and only sends a message when you're on pace to exceed a limit. The LLM is the real cost, which is why it only runs when there's new input.

Lessons learned

Gate the LLM behind cheap change detection.

A monitor script's output is hashed every tick, and the agent only wakes when the hash changes. Make that output grow only (every ID ever seen, plus a retry marker), so marking something done doesn't wake the agent again.

Pick one source of truth early.

Copying data between stores causes "why is the dashboard stale?" bugs. Put the state in one database and make every surface read it live.

Enforce rules in code as well as prompts.

Policy written into a prompt is advice. The email script refusing to send without an approval note is a guarantee.

Simulate time before shipping schedule logic.

Stepping through a fake week caught a bug that would have sent the same habit reminder three times when it moved ahead of an event.

Watch for working-directory collisions.

The Cloudflare CLI (wrangler) creates a .wrangler folder wherever you run it. Run it from your home directory and that folder masks its saved login. Always run tools from a project folder.

Never send undefined to D1.

Older records missing a field broke a bulk write. Give every column an explicit default.

Never quote state from memory.

Re-read the log before saying whether something is done or missing.

Revoke test sessions by ID, never all at once.

Cleaning up after a sign-in test by revoking every web session also signed the user out on their phone. Mark test requests so they're easy to spot, then revoke just that one.

Reminders must not move a task's day.

When one field meant both "when to nag" and "which day", every repeat nag quietly pushed tasks forward. Keep the planned day and the reminder time separate.

Home-screen icons remember the URL they were added from.

Add the plain dashboard URL to the home screen, never a one-time link, or the icon opens a spent link forever.

Build your own

A practical order of operations, roughly one evening per step.

1. Run an agent on a box that's always on. Hermes on a small single-board computer works well. Connect a chat app (iMessage, with Telegram as a fallback) and allow only yourself.
2. Set up voice capture. Create an R2 bucket and a token-protected upload Worker, plus a phone shortcut that posts the recording (details below).
3. Add the memo monitor, transcription and triage job. Write down the importance policy and the reminder cadence explicitly.
4. Add the reminder engine. Start with quiet hours and repeat-until-done, keeping each task's planned day separate from its reminder time. Add calendar blackouts after that.
5. Create the D1 database and API. Make it the source of truth from day one.
6. Build the dashboard. It should be private (texted sign-in link or code, with a password fallback), live, and organized by today versus later.
7. Connect email. Use separate inboxes for "talk to me" and "talk to others", enforce guards in code, and keep an audit log.
8. Add context. Work availability from a booking page, then location pings from phone automations with a location-only token and a 30-day retention limit.
9. Run a security audit and set up the quota alert. Then tune the cadences based on what actually annoys you.

curl -H "Authorization: Bearer <token>" \
  https://<worker>.<subdomain>.workers.dev/recordings