---
title: Content moderation
---

# Content moderation

User-to-user messaging is where marketplace scams happen: a deposit before a viewing, a move to WhatsApp, a document request that has no business being there. This guide holds those messages before they reach the other party and sends the borderline ones to a person.

## Step 1: Set up login and signup protection

Before anything else here, set up [Signup protection](/docs/v3/fundamentals/signup-protection) and [Login protection](/docs/v3/fundamentals/login-protection). Content checks are strongest next to the account checks those give you: the same message reads differently from a day-old account on a busy device than from a two-year-old paying customer.

## Step 2: Evaluate each message from your server

When a user sends a message, call [evaluate an action](/api/v3/evaluations/evaluate-an-action) from your backend before you deliver it. Use a custom action name such as `message`, pass the author as `user`, and put the text in a `content` block with your message id and the conversation id.

```js
const evaluation = await rupt.evaluate({
  action: "message",
  user: message.senderId,
  ip: request.ip,
  content: {
    type: "text",
    id: message.id,
    conversation: message.threadId,
    text: message.body,
  },
});

switch (evaluation.verdict) {
  case "allow":
    deliver(message);
    break;
  case "review":
    hold(message, evaluation.id);
    break;
  case "challenge":
    hold(message, evaluation.id);
    redirectSender(evaluation.redirect);
    break;
  default:
    drop(message);
}
```

The same call from .NET:

```csharp
var evaluation = await rupt.EvaluateAsync("message", new EvaluateRequest
{
    User = message.SenderId,
    Ip = request.Ip,
    Content = new ContentInput
    {
        Id = message.Id,
        Conversation = message.ThreadId,
        Text = message.Body,
    },
});

switch (evaluation.Verdict)
{
    case "allow":
        Deliver(message);
        break;
    case "review":
        Hold(message, evaluation.Id);
        break;
    case "challenge":
        Hold(message, evaluation.Id);
        RedirectSender(evaluation.Redirect);
        break;
    default:
        Drop(message);
        break;
}
```

With a conversation id, Rupt reads the last ten messages it already evaluated in that thread as context. For a thread that started before you integrated, send `content.context` on the first call and let the look-back take over.

## Step 3: Add the policies

A policy has a trigger (the event it runs on) and a verdict. Add these in your [policies dashboard](https://app.rupt.dev/policies):

| Policy                    | Trigger   | Conditions                                                                                          | Verdict   |
| ------------------------- | --------- | --------------------------------------------------------------------------------------------------- | --------- |
| Hold scam content         | `message` | `content_category` is in `scam`, `phishing`, `pii_harvesting` and `content_severity` is at least `high` | Deny      |
| Review from new accounts  | `message` | `content_flagged` is true and `is_new_user` is true                                                  | Review    |
| Review what was flagged   | `message` | `content_flagged` is true                                                                            | Review    |
| Hold when the check is down | `message` | `content_status` equals `unavailable`                                                             | Review    |

They stack from the top. The obvious scam is **denied** outright and the message is never delivered. A flagged message from a new account and everything else that was flagged goes to **review**, where a reviewer sees the text with the evidence highlighted, the earlier messages, and the account checks. The last policy decides what happens when the content check cannot run: sending it to review keeps a person in the loop instead of silently allowing or denying.

Two more worth adding once the first week of data is in:

| Policy             | Trigger   | Conditions                                                            | Verdict                                      |
| ------------------ | --------- | --------------------------------------------------------------------- | -------------------------------------------- |
| Take the ring down | `message` | `content_category` is in `scam`, `phishing` and `fingerprint_user_count` is at least 2 | Add to list (fingerprint, 90-day TTL) |
| Verify the author  | `message` | `content_category` is in `off_platform` and `is_phone_verified` is false | Challenge                                  |

Pair the list with a `signup` and `login` policy on `in_list` that denies, and the next account from that browser never posts. The challenge sends the author to verify a phone number before the message goes through.

## Step 4: Act on review decisions

Subscribe to the [review.decided](/api/v3/webhooks/review-decided) webhook. It carries your message id, the conversation id and the decision, so the receiver delivers or drops the held message without a lookup.

```js
app.post("/webhooks/rupt", (req, res) => {
  const { event, content, decision } = req.body;
  if (event === "review.decided") {
    if (decision === "allow") deliverHeld(content.id);
    else dropHeld(content.id);
  }
  res.sendStatus(200);
});
```

In ASP.NET:

```csharp
app.MapPost("/webhooks/rupt", async (HttpRequest request) =>
{
    var body = await JsonSerializer.DeserializeAsync<JsonElement>(request.Body);
    if (body.GetProperty("event").GetString() == "review.decided")
    {
        var contentId = body.GetProperty("content").GetProperty("id").GetString()!;
        if (body.GetProperty("decision").GetString() == "allow") DeliverHeld(contentId);
        else DropHeld(contentId);
    }
    return Results.Ok();
});
```

Reviewers work the queue at [app.rupt.dev/reviews](https://app.rupt.dev/reviews), oldest first, filtered by category, severity, user or conversation.

## Related

- [Content moderation](/docs/v3/concepts/content): the categories, severity rubric and context rules.
- [Multi-accounting prevention](/docs/v3/guides/multi-accounting-prevention): the device linkage the ring takedown relies on.
- [Verdicts](/docs/v3/concepts/verdicts): what each verdict asks your server to do.