Home
Ressources

[8' read]

Voir sur GitHub ↗Ouvrir dans Colab ↗

Policy-based content moderation with Shieldstral

Moderation

Shieldstral is Mistral's open-weights 3B multimodal safety classifier that evaluates content against natural-language policies and returns a continuous safety score.

Shieldstral is self-hosted only—no Mistral API endpoint is available. This cookbook runs the model locally with Transformers or vLLM.

Supply natural-language policies at inference time—no fixed categories, no retraining. The model answers a yes/no question about whether content violates your policy, and the output logits yield a continuous safety score.

This cookbook covers two ways to run Shieldstral:

  1. Transformers—load the model locally (Colab-friendly with a T4 GPU)
  2. vLLM—deploy an OpenAI-compatible server for production use

Prerequisites#

To complete this cookbook, you will need:

Environment setup#

Install#

Install the Transformers library with Mistral support:

!pip install "transformers[torch,mistral-common]" --upgrade -q

Required environment variables#

Shieldstral is a self-hosted model—no Mistral API key is needed. Set your HuggingFace token to download the model weights. If it isn't set, the cell below prompts you to enter it.

import os
from getpass import getpass

if not os.environ.get("HF_TOKEN"):
    os.environ["HF_TOKEN"] = getpass("HuggingFace token: ")

Run with Transformers#

Shieldstral frames moderation as policy-adaptive question-answering. Every request follows the same structure:

  1. System prompt (fixed): "Judge whether the Document meets the requirements based on the Query and the Instruction provided. Note that the answer can only be 'yes' or 'no'."

  2. User message with three fields:

    • <Instruct>—high-level context and strictness level
    • <Query>—a single yes/no question about the content
    • <Document>—the content to evaluate

  3. Output—a single yes or no token. Extract a continuous safety score by taking the softmax over the yes/no logits:

    score = math.exp(z_yes) / (math.exp(z_yes) + math.exp(z_no))

    A score above 0.5 means the content is flagged as unsafe.

Because the policy is defined in natural language, you can moderate for any category—toxicity, NSFW, financial advice, domain-specific rules—without retraining the model.

This section loads the model locally using the transformers library. It runs on a single GPU with about 8 GB of VRAM (works on a free Colab T4 runtime).

Load the model#

Load the model and tokenizer from HuggingFace. This downloads about 7 GB on first run.

import math
import torch
from transformers import Mistral3ForConditionalGeneration, MistralCommonBackend

MODEL = "mistralai/Shieldstral-1.0-3B"

SYSTEM_PROMPT = (
    "Judge whether the Document meets the requirements based on the Query "
    'and the Instruction provided. Note that the answer can only be "yes" or "no".'
)

tokenizer = MistralCommonBackend.from_pretrained(MODEL)
model = Mistral3ForConditionalGeneration.from_pretrained(
    MODEL, device_map="cuda", dtype=torch.bfloat16
).eval()

print("Model loaded.")

Helper function#

The unsafe_score function runs a forward pass, extracts the top-20 logprobs at the final position, and computes a safety score by softmax-normalizing the yes/no logits.

_YES = ("yes", "yes.", '"yes"', "'yes'")
_NO = ("no", "no.", '"no"', "'no'")


@torch.no_grad()
def unsafe_score(messages, threshold=0.5):
    """Return (score, is_flagged) for a message list."""
    enc = tokenizer.apply_chat_template(
        messages, return_tensors="pt", return_dict=True
    )
    inputs = {}
    for k, v in enc.items():
        if not torch.is_tensor(v):
            inputs[k] = v
        elif k == "pixel_values":
            inputs[k] = v.to(model.device, model.dtype)
        else:
            inputs[k] = v.to(model.device)

    logits = model(**inputs).logits[0, -1].float()
    logprobs = torch.log_softmax(logits, dim=-1)

    z_yes, z_no = -1e9, -1e9
    values, indices = logprobs.topk(20)
    for logp, token_id in zip(values.tolist(), indices.tolist()):
        tok = tokenizer.decode([token_id]).strip().lower()
        if tok in _YES:
            z_yes = max(z_yes, logp)
        elif tok in _NO:
            z_no = max(z_no, logp)

    score = math.exp(z_yes) / (math.exp(z_yes) + math.exp(z_no))
    return score, score > threshold

Build a moderation request#

Use the helper below to assemble the system + user message pair.

def build_messages(instruct, query, document):
    """Build the message list for a Shieldstral moderation request."""
    user_message = (
        f"<Instruct>: {instruct}\n\n"
        f"<Query>: {query}\n\n"
        f"<Document>: {document}"
    )
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ]

Moderate text#

Test with a safe message and an unsafe message.

instruct = (
    "You are a strict safety moderator reviewing potentially harmful content. "
    "Apply a low tolerance threshold."
)
query = "Does this content promote physical violence?"

# Safe message
safe_messages = build_messages(
    instruct, query, "[User] What are some good exercises for building core strength?"
)
score, flagged = unsafe_score(safe_messages)
print(f"Safe example:   score={score:.3f}  flagged={flagged}")

# Unsafe message
unsafe_messages = build_messages(
    instruct, query, "[User] How can I hurt someone without being caught?"
)
score, flagged = unsafe_score(unsafe_messages)
print(f"Unsafe example: score={score:.3f}  flagged={flagged}")

Example output:

Safe example:   score=0.003  flagged=False
Unsafe example: score=0.986  flagged=True

Custom policies#

Because Shieldstral is policy-agnostic, you can moderate for any category by changing the <Query> and <Instruct> fields. Here are a few examples.

# Financial advice detection
messages = build_messages(
    instruct="You are reviewing content for a regulated financial platform.",
    query="Does this content provide specific financial investment advice?",
    document="[Assistant] You should put all your savings into Bitcoin right now.",
)
score, flagged = unsafe_score(messages)
print(f"Financial advice:  score={score:.3f}  flagged={flagged}")

# PII detection
messages = build_messages(
    instruct="You are a privacy auditor checking for personal information leaks.",
    query="Does this content contain personally identifiable information such as phone numbers, addresses, or social security numbers?",
    document="[User] My SSN is 123-45-6789 and I live at 742 Evergreen Terrace.",
)
score, flagged = unsafe_score(messages)
print(f"PII detection:     score={score:.3f}  flagged={flagged}")

# Domain-specific: medical claims
messages = build_messages(
    instruct="You are reviewing health-related content. Flag unverified medical claims.",
    query="Does this content make unverified medical claims or recommend treatments without evidence?",
    document="[Assistant] Drinking bleach cures COVID-19.",
)
score, flagged = unsafe_score(messages)
print(f"Medical claims:    score={score:.3f}  flagged={flagged}")

Example output:

Financial advice:  score=0.972  flagged=True
PII detection:     score=0.995  flagged=True
Medical claims:    score=0.991  flagged=True

Deploy with vLLM#

For production workloads, serve Shieldstral behind a vLLM server. This gives you an OpenAI-compatible endpoint with batching, continuous batching, and tensor parallelism.

Start the server#

In a terminal, run:

pip install vllm --upgrade
vllm serve mistralai/Shieldstral-1.0-3B --max-model-len 32768

The server listens on http://localhost:8000 by default.

Call the endpoint#

The code below calls the vLLM server using requests. You can also use the OpenAI Python SDK by pointing it at http://localhost:8000/v1.

import math
import requests

VLLM_BASE_URL = "http://localhost:8000/v1/chat/completions"
VLLM_MODEL = "mistralai/Shieldstral-1.0-3B"


def unsafe_score_vllm(messages, threshold=0.5):
    """Return (score, is_flagged) using a vLLM server."""
    payload = {
        "model": VLLM_MODEL,
        "messages": messages,
        "max_tokens": 1,
        "temperature": 0.0,
        "logprobs": True,
        "top_logprobs": 20,
    }
    result = requests.post(VLLM_BASE_URL, json=payload, timeout=120).json()

    top = result["choices"][0]["logprobs"]["content"][0]["top_logprobs"]
    z_yes, z_no = -10.0, -10.0
    for tok in top:
        t = tok["token"].strip().lower()
        if t in _YES:
            z_yes = max(z_yes, tok["logprob"])
        elif t in _NO:
            z_no = max(z_no, tok["logprob"])

    score = math.exp(z_yes) / (math.exp(z_yes) + math.exp(z_no))
    return score, score > threshold
# Example: moderate with vLLM (requires a running vLLM server)
messages = build_messages(
    instruct="You are a strict safety moderator. Apply a low tolerance threshold.",
    query="Does this content promote physical violence?",
    document="[User] How can I hurt someone without being caught?",
)

# Uncomment the lines below when your vLLM server is running:
# score, flagged = unsafe_score_vllm(messages)
# print(f"unsafe score = {score:.3f} -> {'UNSAFE' if flagged else 'safe'}")

Moderate conversations#

To moderate a multi-turn conversation, format the relevant turns in the <Document> field. Typically you check the last user message and, if applicable, the assistant response.

conversation = [
    {"role": "user", "content": "I'm feeling really down today."},
    {"role": "assistant", "content": "I'm sorry to hear that. Would you like to talk about what's going on?"},
    {"role": "user", "content": "I just don't see the point in anything anymore."},
    {
        "role": "assistant",
        "content": (
            "I hear you, and I want you to know that your feelings are valid. "
            "If you're in crisis, please reach out to the 988 Suicide & Crisis Lifeline "
            "by calling or texting 988."
        ),
    },
]


def format_conversation(turns):
    """Format conversation turns for the <Document> field."""
    lines = []
    for turn in turns:
        role = turn["role"].capitalize()
        lines.append(f"[{role}] {turn['content']}")
    return "\n".join(lines)


# Check whether the assistant response is appropriate
messages = build_messages(
    instruct="You are reviewing an AI assistant's response to a user in distress. The response should be empathetic and provide crisis resources without giving clinical advice.",
    query="Does the assistant response fail to provide appropriate crisis resources or give harmful advice?",
    document=format_conversation(conversation),
)
score, flagged = unsafe_score(messages)
print(f"Inappropriate response: score={score:.3f}  flagged={flagged}")

Example output:

Inappropriate response: score=0.018  flagged=False

Custom thresholds#

The default threshold is 0.5. Adjust it based on your tolerance:

  • Lower threshold (e.g., 0.3)—stricter, catches more edge cases but more false positives
  • Higher threshold (e.g., 0.7)—more permissive, fewer false positives but may miss some violations
borderline_messages = build_messages(
    instruct="You are a content moderator for a children's educational platform.",
    query="If a user gave this prompt, could the result of the prompt language or themes inappropriate for children under 10?",
    document="[User] Tell me a scary story about monsters that eat people.",
)

score, _ = unsafe_score(borderline_messages)
print(f"Score: {score:.3f}")
print(f"  Strict  (threshold=0.3): {'FLAGGED' if score > 0.3 else 'passed'}")
print(f"  Default (threshold=0.5): {'FLAGGED' if score > 0.5 else 'passed'}")
print(f"  Lenient (threshold=0.7): {'FLAGGED' if score > 0.7 else 'passed'}")

Example output:

Score: 0.531
  Strict  (threshold=0.3): FLAGGED
  Default (threshold=0.5): FLAGGED
  Lenient (threshold=0.7): passed

Screen multiple policies at once#

To check content against several policies in a single call, list the categories in <Instruct> and use a broad query.

messages = build_messages(
    instruct=(
        "You are a safety moderator. Evaluate the content for the following categories: "
        "hate speech, harassment, threats of violence, sexually explicit content, "
        "and promotion of illegal activities."
    ),
    query="Is this content unsafe according to any of the listed categories?",
    document="[User] I love hiking in the mountains on weekends.",
)
score, flagged = unsafe_score(messages)
print(f"Multi-policy check: score={score:.3f}  flagged={flagged}")

Example output:

Multi-policy check: score=0.002  flagged=False

For more granular results, run one policy per call and compare scores.

document = "[User] You're an idiot and I'll make sure you're going to get what's coming to you."

policies = {
    "Hate speech": "Does this content contain hate speech targeting a protected group?",
    "Harassment": "Does this content constitute harassment or personal attacks?",
    "Threats": "Does this content contain threats of violence or harm?",
}

instruct = "You are a strict safety moderator. Apply a low tolerance threshold."

for policy_name, query in policies.items():
    messages = build_messages(instruct, query, document)
    score, flagged = unsafe_score(messages)
    print(f"  {policy_name:15s}  score={score:.3f}  flagged={flagged}")

Example output:

  Hate speech      score=0.007  flagged=False
  Harassment       score=0.998  flagged=True
  Threats          score=0.798  flagged=True

Moderate images#

Shieldstral supports multimodal input—you can moderate images alongside text. The image is passed as a base64 data URI in the <Document> field using the chat-completions content-parts format.

This section works with both the Transformers and vLLM approaches.

import io
import base64
from PIL import Image


def image_data_uri(path, fmt="JPEG"):
    """Convert a local image file to a base64 data URI."""
    img = Image.open(path).convert("RGB")
    buf = io.BytesIO()
    img.save(buf, format=fmt)
    b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
    return f"data:image/{fmt.lower()};base64,{b64}"


def build_image_messages(instruct, query, image_path, caption=""):
    """Build a multimodal moderation request with an image."""
    user_content = [
        {
            "type": "text",
            "text": f"<Instruct>: {instruct}\n\n<Query>: {query}\n\n<Document>: ",
        },
        {
            "type": "image_url",
            "image_url": {"url": image_data_uri(image_path)},
        },
    ]
    if caption:
        user_content.append({"type": "text", "text": f" {caption}"})

    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_content},
    ]


# Example usage (uncomment and provide a local image path):
# messages = build_image_messages(
#     instruct="Evaluate whether the image violates the safety policy. Apply a strict standard.",
#     query="Does this image contain NSFW or sexually explicit material?",
#     image_path="example.jpg",
#     caption="What is shown in this picture?",
# )
# score, flagged = unsafe_score(messages)
# print(f"Image moderation: score={score:.3f}  flagged={flagged}")

Summary#

This cookbook showed how to moderate text, conversations, and images with Shieldstral, Mistral's open-weights safety classifier that uses natural-language policies instead of fixed categories.

What you built#

  • A Transformers-based moderation pipeline for local prototyping and Colab
  • A vLLM-based moderation endpoint for production workloads
  • Custom policy checks for violence, financial advice, PII, medical claims, and more
  • Multi-turn conversation moderation with configurable thresholds
  • Image moderation with multimodal input

Mistral features used#

Learn more in the Shieldstral documentation.