Content moderation

User-to-user messaging is where marketplace scams happen: a deposit before a viewing, a move to WhatsApp, a document request that has no business being there. This guide holds those messages before they reach the other party and sends the borderline ones to a person.

Step 1: Set up login and signup protection

Before anything else here, set up Signup protection and Login protection. Content checks are strongest next to the account checks those give you: the same message reads differently from a day-old account on a busy device than from a two-year-old paying customer.

Step 2: Evaluate each message from your server

When a user sends a message, call evaluate an action from your backend before you deliver it. Use a custom action name such as message, pass the author as user, and put the text in a content block with your message id and the conversation id.

const evaluation = await rupt.evaluate({
  action: "message",
  user: message.senderId,
  ip: request.ip,
  content: {
    type: "text",
    id: message.id,
    conversation: message.threadId,
    text: message.body,
  },
});

switch (evaluation.verdict) {
  case "allow":
    deliver(message);
    break;
  case "review":
    hold(message, evaluation.id);
    break;
  case "challenge":
    hold(message, evaluation.id);
    redirectSender(evaluation.redirect);
    break;
  default:
    drop(message);
}

The same call from .NET:

var evaluation = await rupt.EvaluateAsync("message", new EvaluateRequest
{
    User = message.SenderId,
    Ip = request.Ip,
    Content = new ContentInput
    {
        Id = message.Id,
        Conversation = message.ThreadId,
        Text = message.Body,
    },
});

switch (evaluation.Verdict)
{
    case "allow":
        Deliver(message);
        break;
    case "review":
        Hold(message, evaluation.Id);
        break;
    case "challenge":
        Hold(message, evaluation.Id);
        RedirectSender(evaluation.Redirect);
        break;
    default:
        Drop(message);
        break;
}

With a conversation id, Rupt reads the last ten messages it already evaluated in that thread as context. For a thread that started before you integrated, send content.context on the first call and let the look-back take over.

Step 3: Add the policies

A policy has a trigger (the event it runs on) and a verdict. Add these in your policies dashboard:

They stack from the top. The obvious scam is denied outright and the message is never delivered. A flagged message from a new account and everything else that was flagged goes to review, where a reviewer sees the text with the evidence highlighted, the earlier messages, and the account checks. The last policy decides what happens when the content check cannot run: sending it to review keeps a person in the loop instead of silently allowing or denying.

Two more worth adding once the first week of data is in:

Pair the list with a signup and login policy on in_list that denies, and the next account from that browser never posts. The challenge sends the author to verify a phone number before the message goes through.

Step 4: Act on review decisions

Subscribe to the review.decided webhook. It carries your message id, the conversation id and the decision, so the receiver delivers or drops the held message without a lookup.

app.post("/webhooks/rupt", (req, res) => {
  const { event, content, decision } = req.body;
  if (event === "review.decided") {
    if (decision === "allow") deliverHeld(content.id);
    else dropHeld(content.id);
  }
  res.sendStatus(200);
});

In ASP.NET:

app.MapPost("/webhooks/rupt", async (HttpRequest request) =>
{
    var body = await JsonSerializer.DeserializeAsync<JsonElement>(request.Body);
    if (body.GetProperty("event").GetString() == "review.decided")
    {
        var contentId = body.GetProperty("content").GetProperty("id").GetString()!;
        if (body.GetProperty("decision").GetString() == "allow") DeliverHeld(contentId);
        else DropHeld(contentId);
    }
    return Results.Ok();
});

Reviewers work the queue at app.rupt.dev/reviews, oldest first, filtered by category, severity, user or conversation.

Related

• Content moderation: the categories, severity rubric and context rules.
• Multi-accounting prevention: the device linkage the ring takedown relies on.
• Verdicts: what each verdict asks your server to do.