Describe a boring web chore in words — and watch a bot click the site itself

Here's the one-line idea: you describe a tedious bit of web busywork in plain words — "log into my account, pull all my receipts for the month into one file" — and the app does it itself, clicking a real website. You watch the cursor move across the screen, open tabs, press buttons. As if someone sat down and figured it out for you.

And this isn't the agent that googles and brings back a table. This is an agent that works with its hands in a real interface — where there's no convenient API, just a website with buttons.

Why this just became possible

"Automating a website" used to mean a script that hunts for a button by its code: "click the element with id submit-btn." Painfully brittle — the site gets redrawn a little, the button moves, the script dies. Only someone comfortable with code could do it, and it needed constant fixing.

In June 2026 Google opened up Computer Use in the Gemini 3.5 Flash model (gemini-3.5-flash). Straight from the docs: using screenshots, the model can "'see' a computer screen, and 'act' by generating specific UI actions like mouse clicks and keyboard inputs." So it looks at a picture of the screen, like you do, and decides where to click on its own — across browser, mobile, or desktop. The button moved? It doesn't care, it simply sees it. That's the new thing this project rides on.

What you'll learn

This is the most honest way to understand how an agent actually works:

• The agent loop. You send a screenshot + a goal → the model returns an action ("click here") → your code executes it → you take a new screenshot → repeat. Until the task is done.
• Normalized coordinates — why the model says "click at 0.5, 0.3" instead of pixels, and how to turn that into a real click.
• Human in the loop. The model flags dangerous steps itself with require_confirmation — payments, deletions, sending messages. On those, your app must ask you before it presses. That's not fussiness — it's what separates a handy helper from a bot that accidentally paid for something.

A ready starter prompt

You can't hand an agent a vague goal — it'll start improvising. Give it a clear task and boundaries:

A weak goal and the agent wanders off clicking at random. A strong one — clear task plus emergency brakes on the risky steps — does exactly what's needed and nothing extra.

What you'll end up with

You open a browser, type the task, and watch the cursor move across a real site on its own: it opened "Receipts," picked June, hit "Download," pulled it all into one file. And at the "Pay subscription" button — it freezes and asks: "Sure you want this?" You're at the controls, but your hands are free.

Honest about the limits: it's still slow and sometimes misses a button. Which is the reason for the golden rule — don't give the agent access to money or other people's accounts unattended. That's what require_confirmation is for: keep a hand on pause. Start on your own test site or a public one, not your bank.

Weekend plan

1. Friday night. Grab a Gemini key and run the Computer Use example from the docs on a safe site — your own test one or a public one. Just give it a goal and watch it click.
2. Saturday. Wrap it in "describe the task in words → watch it work." And wire in confirmation on steps flagged require_confirmation right away.
3. Sunday. Point it at one real boring chore of yours — but no money and no one else's passwords. Exporting, sorting, collecting — stuff you don't mind delegating.

Start with a one-or-two-click task. Save "go through the whole site and do everything" for later — first, get the small thing working reliably.

Source: Gemini API — Computer Use (ai.google.dev)