How to add content moderation to your app — five steps and zero cost

The moment you added an input field to your app, you became a moderator. Comments, reviews, usernames, descriptions, uploaded images: all of it is your responsibility now, not the user's.

And here's the part people miss: you need to check not only what the user writes, but what your model answers. A user can steer it into a reply you'll be the one answering for.

Good news: OpenAI's moderation endpoint is free — no tokens, no bill. Let's do it step by step.

1. Decide what you're checking

Three places, each with its own rules:

• User input — what they sent you. This protects other users, and you.
• Model output — what your app showed the user. This is your reputation and the app stores' requirements.
• Uploaded files — images. The moderation model reads those; audio it does not.

Building your first version? Start with input, add output right after. Two checks usually cover it, and they're the classic guardrails on the way in and on the way out.

2. Make the first call

The model is omni-moderation-latest. It takes text and images, images up to 20 MB.

import OpenAI from "openai";
const client = new OpenAI();

const result = await client.moderations.create({
  model: "omni-moderation-latest",
  input: "text the user submitted",
});

console.log(result.results[0].flagged);

Same thing in Python:

from openai import OpenAI
client = OpenAI()

result = client.moderations.create(
    model="omni-moderation-latest",
    input="text the user submitted",
)
print(result.results[0].flagged)

The key lives in server-side environment variables, not in frontend code — storing keys safely is covered separately. Call it from your backend: otherwise anyone opens the tab's sources and walks off with your key.

3. Don't block on flagged — read the scores

The most common beginner mistake is taking the boolean flagged field and building a ban on it. The response holds much more:

• flagged — the overall "something triggered" verdict;
• categories — which categories triggered;
• category_scores — a number from 0 to 1 per category;
• category_applied_input_types — whether it applied to the text or the image.

There are thirteen categories: harassment and threatening harassment, hate and threatening hate, illicit instructions (and the violent variant), self-harm in three forms — intent, instructions and general — sexual content, sexual content involving minors as its own category, plus violence and graphic violence.

Different categories deserve different reactions. "Sexual" in an adults-only app is normal; "self-harm: intent" calls for a helpline, not a ban. So instead of one yes/no, set up three outcomes:

1. low scores — let it through silently;
2. medium — publish it, but queue it for review;
3. high, or a sensitive category — don't publish.

The documentation says it outright: moderation scores are signals for your policy, not an automatic blocking decision. You pick the boundaries yourself, and that choice isn't technical: raise the threshold and you miss more violations, lower it and you hit innocent users. That's the exact trade-off covered in false positives.

4. Check the model's answer, not just the request

If your app generates text, you can ask for moderation inside the same request — no second call needed:

const response = await client.responses.create({
  model: "gpt-6-astra",
  input: [{ role: "user", content: userText }],
  moderation: { model: "omni-moderation-latest" },
});

response.moderation.input.flagged;   // what the user sent
response.moderation.output.flagged;  // what the model answered

You get scores for both the prompt and the reply in one round trip. The model still generates its answer — whether to show it stays your call.

And a catch the docs warn about: a polite refusal can get flagged too. If the model replies "I won't explain how to make a weapon", the word "weapon" is in its answer — and the category fires. So don't hang an automatic ban on an output flag: log it, look at the category, decide by your own rules.

5. Keep a log and actually read it

The last step, which everyone skips and then spends months fixing. Record every trigger: the text (or its hash), the categories, the scores, and what you did.

Once a week, open the log and read the borderline cases, not the obvious offenders. That's where you'll see the threshold sitting crooked, the model tripping over your community's slang, users learning to route around the filter. Without a log you'll never hear about a single wrongful ban: the person you upset just leaves quietly.

What you end up with

A couple of hours of work buys you: input checks, output checks, three outcomes instead of a blunt ban, and a log that shows where the filter is wrong. Cost: zero — the endpoint is free, you only pay for generation.

What you still don't have is moderation for what matters in your product specifically. Spam links, account resale, users swapping contacts to cut you out — none of that is "hate" or "violence", and universal categories won't catch it. Rules like that need your own classifier on top, and the cheapest place to start is comparing one against a plain list of banned words.

What does it cost, and are there limits?

The moderation endpoint itself is free — no tokens are billed. Rate limits apply, as with any API method; under heavy traffic, check the finished message on submit rather than every keystroke. If you call moderation as a separate request, remember it's an extra network hop — invisible on a comment form, but in streaming you're better off checking the answer in chunks.

What if my app isn't in English?

It works. The multimodal version is noticeably more accurate on non-English languages than the previous one. But verify on your own data: take fifty real messages from your app, run them through, and see where the verdict disagrees with yours. Half an hour of that saves weeks of arguing with users.