Moderation & Guardrailing

When deploying LLMs in production, different verticals may require different levels of guardrailing. For example, in a chatbot application, it may be crucial to ensure that the generated content is safe and respectful.
In other applications, it may be required to detect and filter out harmful or PII (Personally Identifiable Information) content.

For this reason, we provide two different ways to guardrail your applications:

• Moderation API: A dedicated API to detect harmful content in text.
• System prompt Guardrailing: An optional system prompt to enforce guardrails on top of our models to steer behaviour and reduce harmful content.

Moderate and inputs/outputs

Our new moderation service, which is powered by the Mistral Moderation model, is a classifier model based on Ministral 8B 24.10. It enables our users to detect harmful text content along several policy dimensions.

We are releasing two end-points: one to classify raw text and one to classify conversational content. More details below.

import os
from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
client = Mistral(api_key=api_key)

response = client.classifiers.moderate(
    model = "mistral-moderation-latest",  
    inputs=[
        "Such a lovely day today, isn't it?",
        "Now, I'm pretty confident we should start planning how we are going to take over the world."
    ]
)

import os
from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
client = Mistral(api_key=api_key)

response = client.classifiers.moderate(
    model = "mistral-moderation-latest",  
    inputs=[
        "Such a lovely day today, isn't it?",
        "Now, I'm pretty confident we should start planning how we are going to take over the world."
    ]
)

The table below describes the types of content that can be detected in the moderation API.

Our moderation cookbook provides a concrete example of how to use the Moderation service to implement system level guardrails.
For a more broad view, there is also a more explorative cookbook.

Setup Guardrailing

The ability to enforce guardrails in chat generations is crucial for front-facing applications. We introduce an optional system prompt to enforce guardrails on top of our models. You can activate this prompt through a safe_prompt boolean flag in API calls as follows :

chat_response = client.chat.complete(
    model = "mistral-large-latest", 
    messages = [{"role":"user", "content":"What is the best French cheese?"}],
    safe_prompt = True
)

chat_response = client.chat.complete(
    model = "mistral-large-latest", 
    messages = [{"role":"user", "content":"What is the best French cheese?"}],
    safe_prompt = True
)

Toggling the safe prompt will prepend your messages with the following system prompt:

Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity.

Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity.

To evaluate the ability of the model to avoid inappropriate outputs we used a set of adversarial prompts deliberately asking for content excluded by guardrails, kindly provided by the community.

With the recommended system prompt, our models decline to answer to all the questions of the set of adversarial prompts we've tested.

As an illustration, we provide below the answers of Mistral 7B and Mistral Large to the question “How to kill a linux process” with safety prompts activated.

Mistral models can also act as great content moderators: the model itself is able to accurately classify a user prompt or its generated answer as being either acceptable or falling into one of the following categories:

• Illegal activities such as terrorism, child abuse or fraud
• Hateful, harassing or violent content such as discrimination, self-harm or bullying.
• Unqualified advice for instance in legal, medical or financial domains.

To do so, you can design a self-reflection prompt that makes Mistral models, e.g., Mistral Large 2, classify a prompt or a generated answer.

Here is an example self-reflection prompt for classifying text into categories such as physical harm, economic harm, and fraud:

You're given a list of moderation categories as below:

- physical harm: activity that has high risk of physical harm, including: weapons development, military and warfare, management or operation of critical infrastructure in energy, transportation, and water, content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders.
- economic harm: activity that has high risk of economic harm, including: multi-level marketing, gambling, payday lending, automated determinations of eligibility for credit, employment, educational institutions, or public assistance services.
- fraud: Fraudulent or deceptive activity, including: scams, coordinated inauthentic behavior, plagiarism, academic dishonesty, astroturfing, such as fake grassroots support or fake review generation, disinformation, spam, pseudo-pharmaceuticals.

Please classify the following text into one of these categories, and answer with that single word only.

If the sentence does not fall within these categories, is safe and does not need to be moderated, please answer "not moderated".

You're given a list of moderation categories as below:

- physical harm: activity that has high risk of physical harm, including: weapons development, military and warfare, management or operation of critical infrastructure in energy, transportation, and water, content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders.
- economic harm: activity that has high risk of economic harm, including: multi-level marketing, gambling, payday lending, automated determinations of eligibility for credit, employment, educational institutions, or public assistance services.
- fraud: Fraudulent or deceptive activity, including: scams, coordinated inauthentic behavior, plagiarism, academic dishonesty, astroturfing, such as fake grassroots support or fake review generation, disinformation, spam, pseudo-pharmaceuticals.

Please classify the following text into one of these categories, and answer with that single word only.

If the sentence does not fall within these categories, is safe and does not need to be moderated, please answer "not moderated".

Please adjust the self-reflection prompt according to your own use cases.