--- title: Content moderation description: Rupt reads the messages, listings and posts your users write, returns the abuse categories it finds, and lets a policy weigh them against the account behind them. --- # Content moderation Content moderation in Rupt is one more set of [checks](/docs/v3/concepts/checks) on an [evaluation](/docs/v3/concepts/evaluations). You send the text a user wrote along with the user, Rupt reads it, and the categories it finds join the device, network and email checks in your [policies](/docs/v3/concepts/policies). The verdict lands on the account, not just on the post. That is the difference from a classifier on its own. A scam message from a two-year-old paying account is worth a look. The same message from a day-old account on a browser that already carries three other accounts is a hold and a list entry that catches the next signup. ## How it works 1. Your server calls [evaluate an action](/api/v3/evaluations/evaluate-an-action) with a custom action such as `message` and a `content` block. 2. Rupt reads the text inside a fixed time budget. When it has a conversation id, it also reads the last ten messages it already saw in that conversation. 3. The result becomes five checks on the evaluation. Your policies match on them, mixed with any account check. 4. The verdict comes back in the same response. `deny` blocks, `challenge` sends the author to verify, `review` holds the message for a person, and `add_to_list` on a fingerprint takes a whole ring down. ## Categories Rupt returns every category that applies, each with a severity, a confidence and a short quote from the text as evidence. | Category | What it covers | | ---------------- | --------------------------------------------------------------------------------------------------------- | | `scam` | A plausible offer with an implausible way to pay: deposits before anything is verified, overpayment, fake escrow, pay-to-work. | | `phishing` | Getting someone to hand over credentials, card details or a one-time code. | | `off_platform` | Steering contact or payment off the platform: WhatsApp, Telegram, Zelle, wire, crypto, gift cards, and obfuscated versions of them. | | `pii_harvesting` | Asking for identity documents, SSN, bank details or passwords before any legitimate need. | | `impersonation` | Claiming to be the platform, its staff, a brand, or a real business or person. | | `spam` | Bulk, duplicate, irrelevant or promotional content. | | `prohibited` | Illegal or restricted goods and services, counterfeits, weapons, drugs, adult services. | | `malicious_code` | Scripts that drain wallets, exfiltrate credentials, or obfuscate themselves. | | `harassment` | Threats, extortion, targeted abuse. | | `hate` | Attacks on protected characteristics. | | `sexual` | Explicit content or solicitation. | | `child_safety` | Any sexualisation of minors. Always maximum severity. | | `self_harm` | Intent or instructions. | | `violence` | Incitement or graphic description. | ## Severity - `low`: a mention or a hint with no ask. "Some people pay by Venmo." - `medium`: a clear ask with an off-platform or unverified element. "Can we do Zelle instead?" - `high`: a concrete attempt: a link, account details, a deadline, documents requested. - `maximum`: credential harvesting, child safety, malicious code, or the same ask repeated after the other party pushed back. The evaluation's `content.severity` is the highest severity across its categories. Categories below a 0.6 confidence are dropped rather than returned as noise. ## Context A message rarely makes sense alone. Rupt reads it with the earlier turns of the conversation in two ways: - Send `content.conversation` and Rupt looks back over the last ten messages it already evaluated in that conversation. Nothing else to do. - Send `content.context`, up to ten turns of `{ user, id, text }`, when you want to supply the history yourself. This is the way to start on a conversation that existed before you integrated: send the context on the first call, then let the look-back take over. Explicit context overrides the look-back, so a call that sends fewer turns than Rupt holds gets a weaker read. ## What the checks look like | Check | Type | | -------------------- | -------------------------------------- | | `content_flagged` | boolean | | `content_category` | list of category names | | `content_severity` | `none`, `low`, `medium`, `high`, `maximum` | | `content_confidence` | 0 to 1 | | `content_status` | `reviewed` or `unavailable` | `content_status` is `unavailable` when the check could not finish inside its budget. The policy engine then runs on the account checks alone. A policy on `content_status equals unavailable` decides what happens in that case; sending it to `review` is the usual choice. ## Review The `review` [verdict](/docs/v3/concepts/verdicts) holds the evaluation for a person. It appears in the dashboard's review queue with the text, the evidence highlighted, the earlier messages the model saw, and the account checks. A reviewer allows or denies it with a note, and the decision is sent to your [review.decided](/api/v3/webhooks/review-decided) webhook with the content id you passed in. ## What Rupt stores The first 4,000 characters of the text stay on the evaluation for 180 days, or less under your project's retention setting, so the review queue and the dashboard can show them. The earlier messages used as context are stored only when the message was flagged or the verdict was not `allow`. Text is never echoed back in the API response; you already have it. ## Related - [Content moderation guide](/docs/v3/guides/content-moderation): the policies for a messaging marketplace. - [Checks](/docs/v3/concepts/checks) and [Verdicts](/docs/v3/concepts/verdicts). - [Evaluate an action](/api/v3/evaluations/evaluate-an-action): the API reference.