Content moderation

Content moderation in Rupt is one more set of checks on an evaluation. You send the text a user wrote along with the user, Rupt reads it, and the categories it finds join the device, network and email checks in your policies. The verdict lands on the account, not just on the post.

That is the difference from a classifier on its own. A scam message from a two-year-old paying account is worth a look. The same message from a day-old account on a browser that already carries three other accounts is a hold and a list entry that catches the next signup.

How it works

1. Your server calls evaluate an action with a custom action such as message and a content block.
2. Rupt reads the text inside a fixed time budget. When it has a conversation id, it also reads the last ten messages it already saw in that conversation.
3. The result becomes five checks on the evaluation. Your policies match on them, mixed with any account check.
4. The verdict comes back in the same response. deny blocks, challenge sends the author to verify, review holds the message for a person, and add_to_list on a fingerprint takes a whole ring down.

Categories

Rupt returns every category that applies, each with a severity, a confidence and a short quote from the text as evidence.

Severity

• low: a mention or a hint with no ask. "Some people pay by Venmo."
• medium: a clear ask with an off-platform or unverified element. "Can we do Zelle instead?"
• high: a concrete attempt: a link, account details, a deadline, documents requested.
• maximum: credential harvesting, child safety, malicious code, or the same ask repeated after the other party pushed back.

The evaluation's content.severity is the highest severity across its categories. Categories below a 0.6 confidence are dropped rather than returned as noise.

Context

A message rarely makes sense alone. Rupt reads it with the earlier turns of the conversation in two ways:

• Send content.conversation and Rupt looks back over the last ten messages it already evaluated in that conversation. Nothing else to do.
• Send content.context, up to ten turns of { user, id, text }, when you want to supply the history yourself. This is the way to start on a conversation that existed before you integrated: send the context on the first call, then let the look-back take over.

Explicit context overrides the look-back, so a call that sends fewer turns than Rupt holds gets a weaker read.

What the checks look like

content_status is unavailable when the check could not finish inside its budget. The policy engine then runs on the account checks alone. A policy on content_status equals unavailable decides what happens in that case; sending it to review is the usual choice.

Review

The review verdict holds the evaluation for a person. It appears in the dashboard's review queue with the text, the evidence highlighted, the earlier messages the model saw, and the account checks. A reviewer allows or denies it with a note, and the decision is sent to your review.decided webhook with the content id you passed in.

What Rupt stores

The first 4,000 characters of the text stay on the evaluation for 180 days, or less under your project's retention setting, so the review queue and the dashboard can show them. The earlier messages used as context are stored only when the message was flagged or the verdict was not allow. Text is never echoed back in the API response; you already have it.

Related

• Content moderation guide: the policies for a messaging marketplace.
• Checks and Verdicts.
• Evaluate an action: the API reference.