---
id: batch-processing
title: Batch Processing
sidebar_position: 2.2
---

# Batch Processing

Batching allows you to run asynchronous inference on large inputs in parallel, reducing compute costs while running large workloads at a 50% discount.

<SectionTab as="h1" sectionId="prepare-batch-file">Prepare Batch</SectionTab>

### Prepare and Upload your Batch

A batch is composed of a list of API requests. The structure of an individual request includes:

- A unique `custom_id` for identifying each request and referencing results after completion
- A `body` object with the raw request you would have when calling the original endpoint without batching

Here's an example of how to structure a batch request:

```jsonl
{"custom_id": "0", "body": {"model": "mistral-small-latest", "max_tokens": 100, "messages": [{"role": "user", "content": "What is the best French cheese?"}]}}
{"custom_id": "1", "body": {"model": "mistral-small-latest", "max_tokens": 100, "messages": [{"role": "user", "content": "What is the best French wine?"}]}}
```

:::tip
Each line must be a valid JSON object. Do not add line breaks within a JSON object. The `model` field inside `body` is optional when you specify the model at job creation time.
:::

A batch `body` object can be any **valid request body for the endpoint** you are using. Below are examples of batches for different endpoints, they have their `body` match the endpoint's request body.

<ExplorerTabs id="batch-examples" mode="close">
  <ExplorerTab value="chat-completion" label="Chat Completion">
    A batch file for chat completions, with `v1/chat/completions` as the Endpoint and `ministral-3b-latest` as the Model, would look like the following:

<Tabs>
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"max_tokens": 128, "messages": [{"role": "user", "content": "What is the best French cheese?"}]}}
{"custom_id": "1", "body": {"max_tokens": 512, "temperature": 0.2, "messages": [{"role": "user", "content": "What is the best French wine?"}]}}
{"custom_id": "2", "body": {"max_tokens": 256, "temperature": 0.8, "messages": [{"role": "user", "content": "What is the best French pastry?"}]}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-cd0da3ab-1-377e2e82-7cbb-4f72-bc67-533512f70955","custom_id":"0","response":{"status_code":200,"body":{"id":"85bcfa450e5643f2a2aed46472d97ffd","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":10,"completion_tokens":128,"total_tokens":138},"created":1761577934,"choices":[{"index":0,"finish_reason":"length","message":{"role":"assistant","content":"Choosing the \"best\" French cheese can be quite subjective as it depends on personal preferences, but here are a few highly regarded French cheeses that are often considered among the best:\n\n1. **Camembert de Normandie**: This is a soft, creamy cheese with a pungent aroma. It's known for its rich, buttery flavor and is often served as a table cheese.\n\n2. **Roquefort**: This is a blue cheese made from sheep's milk. It has a strong, salty flavor and a crumbly texture. It's often used in salads and as a topping for bread.\n\n","tool_calls":null}}]}},"error":null}
{"id":"batch-cd0da3ab-2-a2b7dcce-43ad-4d44-ac8c-d8d037848b24","custom_id":"1","response":{"status_code":200,"body":{"id":"628f9e211fb94641a170856f93740410","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":10,"completion_tokens":458,"total_tokens":468},"created":1761577934,"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"Choosing the \"best\" French wine can be quite subjective as it depends on personal preferences, such as the type of wine you enjoy (red, white, rosé, sparkling, etc.), the region you prefer, and the occasion for which you're drinking. However, here are a few highly regarded French wines that are often considered among the best:\n\n1. **Château Lafite Rothschild (Bordeaux, France)**: This is a first growth (Premier Cru) wine from the Médoc region, known for its complexity and longevity.\n\n2. **Château Margaux (Bordeaux, France)**: Another first growth wine, Margaux is renowned for its elegance and balance.\n\n3. **Château Latour (Bordeaux, France)**: This is a first growth wine from the Médoc region, known for its power and depth.\n\n4. **Château Mouton Rothschild (Bordeaux, France)**: A first growth wine, Mouton Rothschild is celebrated for its finesse and elegance.\n\n5. **Château d'Yquem (Sauternes, France)**: A sweet white wine from the Sauternes region, known for its rich, honeyed flavors.\n\n6. **Champagne Dom Pérignon (Champagne, France)**: A highly sought-after sparkling wine, Dom Pérignon is known for its complexity and elegance.\n\n7. **Château Montrose (Bordeaux, France)**: A second growth wine from the Médoc region, known for its power and depth.\n\n8. **Château Le Pin (Bordeaux, France)**: A rare and highly sought-after wine, Le Pin is known for its exceptional quality and complexity.\n\n9. **Château de Beaucastel (Châteauneuf-du-Pape, France)**: A red wine from the Rhône region, known for its rich, full-bodied character.\n\n10. **Château de Beaucastel (Châteauneuf-du-Pape, France)**: A white wine from the Rhône region, known for its complexity and elegance.\n\nThese wines are highly regarded and can be found in many wine shops and restaurants. However, it's always a good idea to try different wines to find what you enjoy the most.","tool_calls":null}}]}},"error":null}
{"id":"batch-cd0da3ab-3-a7015699-b9f8-4bdf-928d-c4a29fc0c306","custom_id":"2","response":{"status_code":200,"body":{"id":"505471545c4245eeb82dce5409eff9e8","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":11,"completion_tokens":256,"total_tokens":267},"created":1761577934,"choices":[{"index":0,"finish_reason":"length","message":{"role":"assistant","content":"Choosing the \"best\" French pastry can be quite subjective as it depends on personal preferences. However, here are a few that are often considered among the best:\n\n1. **Croissant**: A buttery, flaky pastry with a crispy exterior and soft, doughy interior. It's often enjoyed for breakfast or as a snack.\n\n2. **Macaron**: A meringue-based confection with a ganache or buttercream filling. They come in various flavors and are known for their delicate texture and vibrant colors.\n\n3. **Éclairs**: Long, thin pastries filled with cream and topped with chocolate icing. They are often enjoyed as a dessert or snack.\n\n4. **Tarte Tatin**: A caramelized apple tart, which is often served upside down with a scoop of ice cream.\n\n5. **Pain au Chocolat**: Similar to a croissant, but filled with chocolate. It's a popular breakfast pastry.\n\n6. **Madeleine**: Small, sponge-like cakes with a distinctive shell-like shape. They are often filled with jam or cream.\n\n7. **Chouquette**: A small, round pastry filled with cream and topped with sugar. It's a popular breakfast pastry.\n\n8.","tool_calls":null}}]}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="chat-completions-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with chat completions:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "max_tokens": 256, # Max tokens corresponding to the max generated tokens for chat completions
        "temperature": 0.8, # The temperature to use for sampling
        "messages": [ # The messages to generate chat completions for
            {
                "role": "user", # The role of the messages author
                "content": "What is the best French pastry?" # The content of the message
            }
        ]
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-cd0da3ab-3-a7015699-b9f8-4bdf-928d-c4a29fc0c306", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "id": "505471545c4245eeb82dce5409eff9e8",
      "object": "chat.completion",
      "model": "ministral-3b-latest",
      "usage": {
        "prompt_tokens": 11,
        "completion_tokens": 256,
        "total_tokens": 267
      },
      "created": 1761577934,
      "choices": [
        {
          "index": 0,
          "finish_reason": "length",
          "message": {
            "role": "assistant",
            "content": "Choosing the \"best\" French pastry can be quite subjective as it depends on personal preferences. However, here are a few that are often considered among the best:\n\n1. **Croissant**: A buttery, flaky pastry with a crispy exterior and soft, doughy interior. It's often enjoyed for breakfast or as a snack.\n\n2. **Macaron**: A meringue-based confection with a ganache or buttercream filling. They come in various flavors and are known for their delicate texture and vibrant colors.\n\n3. **Éclairs**: Long, thin pastries filled with cream and topped with chocolate icing. They are often enjoyed as a dessert or snack.\n\n4. **Tarte Tatin**: A caramelized apple tart, which is often served upside down with a scoop of ice cream.\n\n5. **Pain au Chocolat**: Similar to a croissant, but filled with chocolate. It's a popular breakfast pastry.\n\n6. **Madeleine**: Small, sponge-like cakes with a distinctive shell-like shape. They are often filled with jam or cream.\n\n7. **Chouquette**: A small, round pastry filled with cream and topped with sugar. It's a popular breakfast pastry.\n\n8.",
            "tool_calls": null
          }
        }
      ]
    }
  },
  "error": null
}
```

    </TabItem>
</Tabs>

For more information regarding completions, visit the [Chat Completions](/inference/conversations/chat-completion) docs and the corresponding [API Spec](../api/endpoint/chat#operation-chat_completion_v1_chat_completions_post).
  </ExplorerTab>
  <ExplorerTab value="structured-outputs" label="Structured Outputs">
    A batch file for structured outputs, with `v1/chat/completions` as the Endpoint and `ministral-3b-latest` as the Model, would look like the following:

<Tabs groupId="code">
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"messages": [{"role": "user", "content": "What's 1+1?"}],"response_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"result": {"title": "Result","type": "integer"}},"required": ["result"],"title": "Math","type": "object","additionalProperties": false},"name": "math","strict": true}},"max_tokens": 256,"temperature": 0}}
{"custom_id": "1", "body": {"messages": [{"role": "user", "content": "What is the best French wine?"}],"response_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"name": {"title": "Name","type": "string"},"region": {"title": "Region","type": "string"},"year": {"title": "Year","type": "integer"}},"required": ["name", "region", "year"],"title": "Wine","type": "object","additionalProperties": false},"name": "wine","strict": true}},"max_tokens": 256,"temperature": 0}}
{"custom_id": "2", "body": {"messages": [{"role": "system","content": "Extract the books information."},{"role": "user","content": "I recently read To Kill a Mockingbird by Harper Lee."}],"response_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"name": {"title": "Name","type": "string"},"authors": {"items": {"type": "string"},"title": "Authors","type": "array"}},"required": ["name", "authors"],"title": "Book","type": "object","additionalProperties": false},"name": "book","strict": true}},"max_tokens": 256,"temperature": 0}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-987bdeb0-1-b0a976ba-2a4f-4584-a4fa-9b140abdcd52","custom_id":"0","response":{"status_code":200,"body":{"id":"bf1b188307c2437587de243f5f715663","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":10,"completion_tokens":7,"total_tokens":17},"created":1761577783,"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"{\"result\": 2}","tool_calls":null}}]}},"error":null}
{"id":"batch-987bdeb0-2-6bc67f66-34be-4db3-a9ea-a437d882dce0","custom_id":"1","response":{"status_code":200,"body":{"id":"afd9bc37072a4d3e8a01999e8822984a","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":10,"completion_tokens":28,"total_tokens":38},"created":1761577783,"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"{\"name\": \"Château Lafite Rothschild\", \"region\": \"Bordeaux\", \"year\": 2018}","tool_calls":null}}]}},"error":null}
{"id":"batch-987bdeb0-3-f8edcc0f-728f-49e7-9cd8-c8c18422ed02","custom_id":"2","response":{"status_code":200,"body":{"id":"01d5ee676f2c4c389e1f2346a2c5f934","object":"chat.completion","model":"ministral-3b-latest","usage":{"prompt_tokens":21,"completion_tokens":24,"total_tokens":45},"created":1761577783,"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"{\n  \"name\": \"To Kill a Mockingbird\",\n  \"authors\": [\"Harper Lee\"]\n}","tool_calls":null}}]}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="structured-outputs-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with structured outputs:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "max_tokens": 256, # Max tokens corresponding to the max generated tokens for chat completions
        "temperature": 0, # The temperature to use for sampling
        "messages": [ # The messages to generate chat completions for with structured outputs
            {
                "role": "system", # The role of the messages author
                "content": "Extract the books information." # The content of the message
            },
            {
                "role": "user", # The role of the messages author
                "content": "I recently read To Kill a Mockingbird by Harper Lee." # The content of the message, here we want to extract the book information
            }
        ],
        "response_format": { # The response format to use for the chat completion
            "type": "json_schema", # Set the type to json_schema to use custom structured outputs, "json_mode" if you want to use the default json only mode
            "json_schema": { # The json schema to use for the structured outputs
                "schema": { # The corresponding schema
                    "properties": { # The properties of the schema
                        "name": { # The name of the book
                            "title": "Name",
                            "type": "string"
                        },
                        "authors": { # The authors of the book
                            "items": {
                                "type": "string"
                            },
                            "title": "Authors",
                            "type": "array"
                        }
                    },
                    "required": [ # The required properties
                        "name", 
                        "authors"
                    ],
                    "title": "Book", # The title of the schema
                    "type": "object", # The type of the schema
                    "additionalProperties": false # Whether to allow additional properties
                },
                "name": "book", # The name of the schema
                "strict": true # Whether to use strict mode, always recommended to set as true for structured outputs
            }
        }
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-987bdeb0-3-f8edcc0f-728f-49e7-9cd8-c8c18422ed02", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "id": "01d5ee676f2c4c389e1f2346a2c5f934",
      "object": "chat.completion",
      "model": "ministral-3b-latest",
      "usage": {
        "prompt_tokens": 21,
        "completion_tokens": 24,
        "total_tokens": 45
      },
      "created": 1761577783,
      "choices": [
        {
          "index": 0,
          "finish_reason": "stop",
          "message": {
            "role": "assistant",
            "content": "{\n  \"name\": \"To Kill a Mockingbird\",\n  \"authors\": [\"Harper Lee\"]\n}",
            "tool_calls": null
          }
        }
      ]
    }
  },
  "error": null
}
```

    </TabItem>
</Tabs>

For more information regarding sructured outputs, visit the [Structured Outputs](/inference/conversations/structured-output) docs and the corresponding chat completions [API Spec](../api/endpoint/chat#operation-chat_completion_v1_chat_completions_post).
  </ExplorerTab>
  <ExplorerTab value="embeddings" label="Embeddings">
    A batch file for embeddings, with `v1/embeddings` as the Endpoint and `mistral-embed` as the Model, would look like the following:

<Tabs groupId="code">
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"input": ["The quick brown fox jumps over the lazy dog.", "To be or not to be, that is the question."]}}
{"custom_id": "1", "body": {"input": "Success is not final, failure is not fatal: It is the courage to continue that counts."}}
{"custom_id": "2", "body": {"input": ["The only way to do great work is to love what you do.", "In the middle of every difficulty lies opportunity.", "Knowledge is power, but enthusiasm pulls the switch."]}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-089b0faa-1-d4475954-11af-4b2b-a819-091cf96a33d1","custom_id":"0","response":{"status_code":200,"body":{"id":"9e9bc2cc688a4dfb8cf4088c1d687392","object":"list","model":"mistral-embed","usage":{"prompt_tokens":28,"completion_tokens":0,"total_tokens":28,"prompt_audio_seconds":null,"prompt_token_details":null,"request_count":null},"data":[{"object":"embedding","embedding":[-0.013092041015625,0.034088134765625,0.03546142578125,0.00879669189453125,0.0399169921875,0.0236968994140625,0.0245208740234375...862884521484375,-0.034759521484375,-0.0118560791015625,-0.006099700927734375,-0.002819061279296875,0.054779052734375,0.0007371902465820312,0.04718017578125,-0.01219940185546875,0.010009765625,-0.052490234375,0.0004892349243164062,-0.0157623291015625,-0.01214599609375,0.0223236083984375,-0.017730712890625,-0.03912353515625,0.0423583984375,0.03546142578125,0.04052734375,-0.006244659423828125,0.0545654296875,0.02001953125,-0.034515380859375,-0.00839996337890625,-0.0088043212890625],"index":1}]}},"error":null}
{"id":"batch-089b0faa-2-b9d5c3ce-433c-46f1-a13f-e0e4a763b52f","custom_id":"1","response":{"status_code":200,"body":{"id":"a67f9a3b35a845999e6ae30af5b79cd4","object":"list","model":"mistral-embed","usage":{"prompt_tokens":21,"completion_tokens":0,"total_tokens":21,"prompt_audio_seconds":null,"prompt_token_details":null,"request_count":null},"data":[{"object":"embedding","embedding":[-0.0202789306640625,0.045196533203125,0.0191192626953125,0.0228118896484375,0.0267181396484375,0.037811279296875,0.036315917968...8515625,-0.01129913330078125,-0.01177215576171875,-0.0006928443908691406,-0.026824951171875,0.0352783203125,-0.0063629150390625,0.040985107421875,0.0025482177734375,0.018798828125,-0.07769775390625,-0.0225982666015625,-0.0212249755859375,-0.0194244384765625,0.029998779296875,-0.046478271484375,-0.053863525390625,0.0173187255859375,0.0439453125,0.00844573974609375,-0.0065460205078125,0.03948974609375,0.0216522216796875,-0.01267242431640625,0.013519287109375,-0.004276275634765625],"index":0}]}},"error":null}
{"id":"batch-089b0faa-3-9b701be0-78c3-4066-9b9c-8746db3d5fee","custom_id":"2","response":{"status_code":200,"body":{"id":"4bb65979fcd143fca01a559837577dfe","object":"list","model":"mistral-embed","usage":{"prompt_tokens":40,"completion_tokens":0,"total_tokens":40,"prompt_audio_seconds":null,"prompt_token_details":null,"request_count":null},"data":[{"object":"embedding","embedding":[-0.0277099609375,0.0562744140625,-0.0008287429809570312,0.0001367330551147461,-0.0126190185546875,-0.0201873779296875,0.0575561...41796875,0.02294921875,-0.019439697265625,0.026611328125,0.004222869873046875,0.063720703125,-0.0037841796875,0.0011205673217773438,0.021514892578125,0.004978179931640625,-0.08990478515625,0.0164947509765625,0.01020050048828125,0.01171112060546875,0.00856781005859375,-0.0011301040649414062,-0.03155517578125,0.033294677734375,0.033477783203125,0.023590087890625,-0.014739990234375,0.04718017578125,0.020721435546875,-0.0180816650390625,-0.002002716064453125,-0.00010335445404052734],"index":2}]}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="embeddings-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with embeddings:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "input": [ # The input text to embed, can be a string or an array of strings
            "The only way to do great work is to love what you do.", 
            "In the middle of every difficulty lies opportunity.", 
            "Knowledge is power, but enthusiasm pulls the switch."
        ]
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-089b0faa-3-9b701be0-78c3-4066-9b9c-8746db3d5fee", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "id": "4bb65979fcd143fca01a559837577dfe",
      "object": "list",
      "model": "mistral-embed",
      "usage": {
        "prompt_tokens": 40,
        "completion_tokens": 0,
        "total_tokens": 40,
        "prompt_audio_seconds": null,
        "prompt_token_details": null,
        "request_count": null
      },
      "data": [
        {
          "object": "embedding",
          "embedding": [
            -0.0277099609375,
            0.0562744140625,
            -0.0008287429809570312,
            0.0001367330551147461,
            -0.0126190185546875,
            -0.0201873779296875,
            0.0575561...
          ],
          "index": 0
        },
        ...
        {
            "object": "embedding",
            "embedding": [
            ...41796875,
            0.02294921875,
            -0.019439697265625,
            0.026611328125,
            0.004222869873046875,
            0.063720703125,
            -0.0037841796875,
            0.0011205673217773438,
            0.021514892578125,
            0.004978179931640625,
            -0.08990478515625,
            0.0164947509765625,
            0.01020050048828125,
            0.01171112060546875,
            0.00856781005859375,
            -0.0011301040649414062,
            -0.03155517578125,
            0.033294677734375,
            0.033477783203125,
            0.023590087890625,
            -0.014739990234375,
            0.04718017578125,
            0.020721435546875,
            -0.0180816650390625,
            -0.002002716064453125,
            -0.00010335445404052734
          ],
          "index": 2
        }
      ]
    }
  },
  "error": null
}
```

    </TabItem>
</Tabs>

For more information regarding embeddings, visit the [Embeddings](embeddings) docs and the corresponding [API Spec](../api/endpoint/embeddings#operation-embeddings_v1_embeddings_post).
  </ExplorerTab>
  <ExplorerTab value="document-ai-ocr" label="OCR">
    A batch file for OCR, with `v1/ocr` as the Endpoint and `mistral-ocr-latest` as the Model, would look like the following:

<Tabs>
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"document": {"document_url": "https://arxiv.org/pdf/2410.07073"}}}
{"custom_id": "1", "body": {"document": {"document_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png"}}}
{"custom_id": "2", "body": {"document": {"image_url": "data:image/jpeg;base64,<base64_image>"}}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-a6054116-1-d32adde3-3484-4008-ba51-2e2b9e52f420","custom_id":"0","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"# Pixtral 12B \n\n![img-0.jpeg](img-0.jpeg)\n\n## Abstract\n\nWe introduce Pixtral 12B, a 12-billion-parameter multimodal language model. Pixtral 12B is trained to understand both natural images and documents, achieving leading performance on various multimodal benchmarks, surpassing a number of larger models. Unlike many open-source models, Pixtral is also a cu...we endeavour to recover the reported performance of all models by tuning evaluation settings towards individual models. We highlight that Pixtral 12B, like strong closed-source models (e.g. Gemini-1.5-Flash 8B [18] and Claude-3 Haiku [1]) is able reports strong performance without such interventions.","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}}],"model":"mistral-ocr-2505-completion","usage_info":{"pages_processed":24,"doc_size_bytes":12640953},"document_annotation":null}},"error":null}
{"id":"batch-a6054116-2-cf0ea563-dcdc-4e9e-b39f-39ba002160ec","custom_id":"1","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"PLACE FACE UP ON DASH\nCITY OF PALO ALTO\nNOT VALID FOR\nONSTREET PARKING\n\nExpiration Date/Time\n11:59 PM\nAUG 19, 2024\n\nPurchase Date/Time: 01:34pm Aug 19, 2024\nTotal Due: $15.00\nTotal Paid: $15.00\nTicket #: 00005883\nS/N #: 520117260957\nSetting: Permit Machines\nMach Name: Civic Center\n\n#****-1224, Visa\nDISPLAY FACE UP ON DASH\n\nPERMIT EXPIRES\nAT ...KING\n\nExpiration Date/Time\n11:59 PM\nAUG 19, 2024\n\nPurchase Date/Time: 01:34pm Aug 19, 2024\nTotal Due: $15.00\nTotal Paid: $15.00\nTicket #: 00005883\nS/N #: 520117260957\nSetting: Permit Machines\nMach Name: Civic Center\n\n#****-1224, Visa\nDISPLAY FACE UP ON DASH\n\nPERMIT EXPIRES\nAT MIDNIGHT","images":[],"dimensions":{"dpi":200,"height":3210,"width":1806}}],"model":"mistral-ocr-2505-completion","usage_info":{"pages_processed":1,"doc_size_bytes":3110191},"document_annotation":null}},"error":null}
{"id":"batch-a6054116-3-1a0ba35b-9646-4613-9327-d499c0804e4b","custom_id":"2","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"We introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmarks, and the best released 34B model (Llama 1) in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference... a model fine-tuned to follow instructions, Mistral 7B - Instruct, that surpasses Llama 2 13B - chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.\nCode: https://github.com/mistralai/mistral-src\nWebpage: https://mistral.ai/news/announcing-mistral-7b/","images":[],"dimensions":{"dpi":200,"height":490,"width":1200}}],"model":"mistral-ocr-2505-completion","usage_info":{"pages_processed":1,"doc_size_bytes":161787},"document_annotation":null}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="ocr-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with OCR:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "document": { # The document to be processed
            "image_url": "data:image/jpeg;base64,<base64_image>" # Here, we are using a base64 encoded image, but you can also use a URL to a file, and document_url for pdf or other kinds of documents
        }
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-a6054116-3-1a0ba35b-9646-4613-9327-d499c0804e4b", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "pages": [
        {
          "index": 0,
          "markdown": "We introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmarks, and the best released 34B model (Llama 1) in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference... a model fine-tuned to follow instructions, Mistral 7B - Instruct, that surpasses Llama 2 13B - chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.\nCode: https://github.com/mistralai/mistral-src\nWebpage: https://mistral.ai/news/announcing-mistral-7b/",
          "images": [],
          "dimensions": {
            "dpi": 200,
            "height": 490,
            "width": 1200
          }
        }
      ],
      "model": "mistral-ocr-2505-completion",
      "usage_info": {
        "pages_processed": 1,
        "doc_size_bytes": 161787
      },
      "document_annotation": null
    }
  },
  "error": null
}
```
    </TabItem>
</Tabs>

For more information regarding OCR, visit the [Document AI - OCR](/studio/document-processing/overview) docs and the corresponding [API Spec](../api/endpoint/ocr#operation-ocr_v1_ocr_post).
  </ExplorerTab>
  <ExplorerTab value="document-ai-annotations" label="Annotations">
    A batch file for OCR with Annotations, with `v1/ocr` as the Endpoint and `mistral-ocr-latest` as the Model. You can find an End to End example [here](https://github.com/mistralai/client-python/blob/main/examples/mistral/jobs/async_jobs_ocr_batch_annotation.py).  

:::note
All Document AI features are currently available via Batching, for more information on Annotations visit our documentation [here](/studio/document-processing/overview).
:::

A batch file would look like the following:

<Tabs>
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"document": {"document_url": "https://arxiv.org/pdf/2410.07073"}, "bbox_annotation_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"document_type": {"title": "Document_Type","type": "string"},"short_description": {"title": "Short_Description","type": "string"},"summary": {"title": "Summary","type": "string"}},"required": ["document_type","short_description","summary"],"title": "BBOXAnnotation","type": "object","additionalProperties": false},"name": "document_annotation","strict": true}}}}
{"custom_id": "1", "body": {"document": {"document_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png"}, "document_annotation_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"language": {"title": "Language","type": "string"},"urls": {"title": "urls","type": "string"}},"required": ["language","urls"],"title": "DocumentAnnotation","type": "object","additionalProperties": false},"name": "document_annotation","strict": true}}}}
{"custom_id": "2", "body": {"document": {"image_url": "data:image/jpeg;base64,<base64_image>"}, "bbox_annotation_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"document_type": {"title": "Document_Type","type": "string"},"short_description": {"title": "Short_Description","type": "string"},"summary": {"title": "Summary","type": "string"}},"required": ["document_type","short_description","summary"],"title": "BBOXAnnotation","type": "object","additionalProperties": false},"name": "document_annotation","strict": true}}, "document_annotation_format": {"type": "json_schema","json_schema": {"schema": {"properties": {"language": {"title": "Language","type": "string"},"urls": {"title": "urls","type": "string"}},"required": ["language","urls"],"title": "DocumentAnnotation","type": "object","additionalProperties": false},"name": "document_annotation","strict": true}}, "pages": [0,1,2,3,4,5,6,7]}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-2b6be3bc-1-d309bbf2-8008-481e-8c7f-6aea6801af75","custom_id":"0","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"# Pixtral 12B\n\n![img-0.jpeg](img-0.jpeg)\n\n# Abstract\n\nWe introduce Pixtral 12B, a 12-billion-parameter multimodal language model. Pixtral 12B is trained to understand both natural images and documents, achieving leading performance on various multimodal benchmarks, surpassing a number of larger models. Unlike many open-source models, Pixtral is also a cutting-edge text model for its size, and does not compromise on natural language performance to excel in multimodal tasks. Pixtral uses a new vision encoder trained from scratch, which allows it to ingest images at their natural resolution and aspect ratio. This gives users flexibility on the number of tokens used to process an image. Pixtral is also able to process any number of images in its long context window of 128K tokens. Pixtral 12B substantially outperforms other open models of similar sizes (Llama-3.2 11B &amp; Qwen-2-VL 7B). It also outperforms much larger open models like Llama-3.2 90B while being 7x smaller. We further contribute an open-source benchmark, MM-MT-Bench, for evaluating vision-language models in practical scenarios, and provide detailed analysis and code for standardized evaluation protocols for multimodal LLMs. Pixtral 12B is released under Apache 2.0 license.\n\nWebpage: https://mistral.ai/news/pixtral-12b/\n\nInference code: https://github.com/mistralai/mistral-inference/\n\nEvaluation code: https://github.com/mistralai/mistral-evals/\n\n# 1 Introduction\n\nThis paper describes Pixtral 12B, a multimodal language model trained to understand both images and text, released with open weights under an Apache 2.0 license. Pixtral is an instruction tuned model which is pretrained on large scale interleaved image and text documents, and hence is capable of multi-turn, multi-image conversation.\n\nPixtral comes with a new vision encoder which is trained with a novel RoPE-2D implementation, allowing it to process images at their native resolution and aspect ratio. In this way, the model can flexibly process images at low resolution in latency-constrained settings, while processing images at high resolution when fine-grained reasoning is required.\n\nWhen compared against models of a similar size in the same evaluation setting, we find that Pixtral delivers strong multimodal reasoning capabilities without sacrificing text-only reasoning performance.","images":[{"id":"img-0.jpeg","top_left_x":426,"top_left_y":565,"bottom_right_x":1276,"bottom_right_y":860,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Image\",\n  \"short_description\": \"Mistral AI logo\",\n  \"summary\": \"The image displays the logo of Mistral AI. The logo features the text 'Mistral AI' in a bold, three-dimensional font with a gradient color scheme transitioning from orange to yellow.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":1,"markdown":"![img-1.jpeg](img-1.jpeg)\nFigure 1: Pixtral Performance. Pixtral outperforms all open-models within its weight class on multimodal tasks by a substantial margin. Left: Performance on MM-MT-Bench, a new multimodal, multiturn, instruction following benchmark designed to reflect real world usage of multimodal language models. Right: Performance on the public LMSys leaderboard (Vision arena, October 2024).\n\n![img-2.jpeg](img-2.jpeg)\n\nFor instance, our model matches or exceeds the performance of models like Qwen2-VL 7B [23] and Llama-3.2 11B [6] on popular multimodal benchmarks like MMMU [24] and MathVista [14], while outperforming most open-source models on popular text-only tasks like MATH [7] and HumanEval [26]. Pixtral even outperforms much larger models like Llama-3.2 90B [6], as well as closed models such as Claude-3 Haiku [1] and Gemini-1.5 Flash 8B [18], on multimodal benchmarks.\n\nDuring evaluation of Pixtral and the baselines, we found that evaluation protocols for multimodal language models is not standardized, and that small changes in the setup can dramatically change the performance of some models. We provide thorough analysis of our experience in re-evaluating vision-language models under a common evaluation protocol.\n\nSpecifically, we identify two issues with evaluation:\n\n- Prompts: Several benchmarks have default prompts which are under-specified, and dramatically reduce the performance of leading closed source models [16, 1] compared to reported figures.\n- Evaluation Metrics: The official metrics typically require exact match, which score model generations as correct only if they exactly match the reference answer. However, this metric penalizes answers which are substantively correct but in a slightly different format (e.g., \"6.0\" vs \"6\").\n\nTo alleviate these issues, we propose 'Explicit' prompts that explicitly specify the format required by the reference answer. We further analyze the impact of flexible parsing for various models, releasing the evaluation code and prompts in an effort to establish fair and standardized evaluation protocols $^1$ .\n\nMoreover, while current multimodal benchmarks mostly evaluate short-form or multiple-choice question answering given an input image, they do not fully capture a model's utility for practical use cases (e.g. in a multi-turn, long-form assistant setting). To address this, we open-source a novel multimodal, multi-turn evaluation: MM-MT-Bench $^2$ . We find that performance on MM-MT-Bench correlates highly with ELO rankings on the LMSys Vision Leaderboard.\n\nPixtral excels at multimodal instruction following, surpassing comparable open-source models on the MM-MT-Bench benchmark (see Figure 1). Based on human preferences on the LMSys Vision Leaderboard, Pixtral 12B is currently the highest ranked Apache 2.0 model, substantially outperforming other open-models such Llama-3.2 11B [6] and Qwen2-VL 7B [23]. It even ranks higher than several closed models such as Claude-3 Opus &amp; Claude-3 Sonnet [1], and several larger models such as Llama-3.2 90B [6].","images":[{"id":"img-1.jpeg","top_left_x":299,"top_left_y":193,"bottom_right_x":843,"bottom_right_y":666,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"Performance vs. Cost for Various Models on MM-MT-Bench\",\n  \"summary\": \"This chart compares the performance and cost of various large language models on the MM-MT-Bench benchmark. The x-axis represents the cost or number of parameters (in billions), and the y-axis represents performance. The chart highlights that the Pixtral 12B model offers the best performance-to-cost ratio, positioned in the top left quadrant, indicating high performance with relatively low cost. Other models like Qwen-2-VL 72B and Llama-3.2 90B are also high-performing but have higher costs. The chart suggests that smaller models like Molmo-D 7B and LLaVA-OneVision 7B are more cost-effective but offer lower performance. The shaded area indicates the best performance-to-cost ratio, emphasizing the efficiency of the Pixtral 12B model.\"\n}"},{"id":"img-2.jpeg","top_left_x":846,"top_left_y":193,"bottom_right_x":1397,"bottom_right_y":666,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"Performance vs. Cost/Number of Parameters for Various Vision Models\",\n  \"summary\": \"The chart compares the performance (measured in LMSys-Vision ELO) against the cost/number of parameters (in billions) for various vision models. Pixtral 12B is highlighted as having the best performance/cost ratio, positioned in the upper left quadrant indicating high performance with relatively low cost/parameters. Other models like Qwen-2-VL 72B and Llama-3.2 90B show high performance but with higher cost/parameters. Models like Phi-3.5-Vision and LLaVA-OneVision 72B are positioned lower, indicating lower performance and higher cost/parameters. The chart suggests that Pixtral 12B offers a balanced and efficient choice in terms of performance and cost.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":2,"markdown":"![img-3.jpeg](img-3.jpeg)\nFigure 2: Pixtral Vision Encoder. Pixtral uses a new vision encoder, which is trained from scratch to natively support variable image sizes and aspect ratios. Block-diagonal attention masks enable sequence packing for batching, while RoPE-2D encodings facilitate variable image sizes. Note that the attention mask and position encodings are fed to the vision transformer as additional input, and utilized only in the self-attention layers.\n\n# 2 Architectural details\n\nPixtral 12B is based on the transformer architecture [22], and consists of a multimodal decoder to perform high-level reasoning, and a vision encoder to allow the model to ingest images. The main parameters of the model are summarized in Table 1.\n\n# 2.1 Multimodal Decoder\n\nPixtral 12B is built on top of Mistral Nemo 12B [15], a 12-billion parameter decoder-only language model that achieves strong performance across a range of knowledge and reasoning tasks.\n\n|  Parameters | Decoder | Encoder  |\n| --- | --- | --- |\n|  dim | 5120 | 1024  |\n|  n_layers | 40 | 24  |\n|  head_dim | 128 | 64  |\n|  hidden_dim | 14336 | 4096  |\n|  n_heads | 32 | 16  |\n|  n_kv_heads | 8 | 16  |\n|  context_len | 131072 | 4096  |\n|  vocab_size | 131072 | -  |\n|  patch_size | - | 16  |\n\nTable 1: Decoder and encoder parameters.\n\n# 2.2 Vision Encoder\n\nIn order for Pixtral 12B to ingest images, we train a new vision encoder from scratch, named PixtralViT. Here, our goal is to instantiate a simple architecture which is capable of processing images across a wide range of resolutions and aspect ratios. To do this, we build a 400 million parameter vision transformer [5] (see Table 1) and make four key changes over the standard architectures [17]:\n\nBreak tokens: In order to assist the model in distinguishing between images with the same number of patches (same area) but different aspect ratios, we include [IMAGE BREAK] tokens between image rows [2]. We further include an [IMAGE END] token at the end of an image sequence.\n\nGating in FFN: Instead of standard feedforward layer in the attention block, we use gating in the hidden layer [19].\n\nSequence packing: In order to efficiently process images within a single batch, we flatten the images along the sequence dimension and concatenate them [3]. We construct a block-diagonal mask to ensure no attention leakage between patches from different images.\n\nRoPE-2D: We replace traditional learned and absolute position embeddings for image patches with relative, rotary position encodings [11, 20] in the self-attention layers. While learned position embeddings must be interpolated to deal with new image sizes (often at the cost of performance), relative position encodings lend themselves naturally to variable image sizes.","images":[{"id":"img-3.jpeg","top_left_x":311,"top_left_y":187,"bottom_right_x":1387,"bottom_right_y":649,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Technical Diagram\",\n  \"short_description\": \"Diagram of Pixtral-ViT model architecture\",\n  \"summary\": \"The diagram illustrates the architecture of the Pixtral-ViT model, which integrates vision and language processing. It starts with image patches that are processed through RoPE-2D (Rotary Position Embedding 2D) and a block-diagonal attention mask. These patches are then fed into a bidirectional transformer, specifically the Pixtral-ViT (Vision Transformer). The output of the transformer is passed through a Vision-Language Projector, which aligns the visual and textual embeddings. Finally, the embeddings are processed by a Multi-Layer Perceptron (MLP) to produce the output embeddings. Tokens such as [IMG_BREAK] and [IMG_END] are added to delineate the boundaries of the image embeddings within the sequence. The diagram also includes visual examples of how the model processes different types of images, such as maps and bar charts, to generate corresponding embeddings.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":3,"markdown":"![img-4.jpeg](img-4.jpeg)\nFigure 3: Complete Pixtral Architecture. Pixtral has two components: a vision encoder, which tokenizes images, and a multimodal decoder, which predicts the next text token given a sequence of text and images. Pixtral can take an arbitrary number of images as input, provided they fit within its 128K context window.\n\nParticularly, let  $x$  be a  $d$ -dimensional patch vector (either a key or query feature). We denote this feature as  $x^{(i,j)}$  when it appears at position  $(i,j)$  in the image. Then, the RoPE-2D transform of  $x^{(i,j)}$  is expressed as:\n\n$$\n\\operatorname {R o P E} - 2 \\mathrm {D} \\left(x ^ {(i, j)}, \\Theta\\right) = M _ {\\Theta} ^ {(i, j)} x ^ {(i, j)}, \\tag {1}\n$$\n\nwhere  $M_{\\Theta}^{(i,j)} = \\left( \\begin{array}{ccccccc}\\cos i\\theta_1 &amp; -\\sin i\\theta_1 &amp; 0 &amp; 0 &amp; \\dots &amp; 0 &amp; 0\\\\ \\sin i\\theta_1 &amp; \\cos i\\theta_1 &amp; 0 &amp; 0 &amp; \\dots &amp; 0 &amp; 0\\\\ 0 &amp; 0 &amp; \\cos j\\theta_2 &amp; -\\sin j\\theta_2 &amp; \\dots &amp; 0 &amp; 0\\\\ 0 &amp; 0 &amp; \\sin j\\theta_2 &amp; \\cos j\\theta_2 &amp; \\dots &amp; 0 &amp; 0\\\\ \\vdots &amp; \\vdots &amp; \\vdots &amp; \\vdots &amp; \\ddots &amp; \\vdots &amp; \\vdots \\\\ 0 &amp; 0 &amp; 0 &amp; 0 &amp; \\dots &amp; \\cos j\\theta_{\\frac{d}{2}} &amp; -\\sin j\\theta_{\\frac{d}{2}}\\\\ 0 &amp; 0 &amp; 0 &amp; 0 &amp; \\dots &amp; \\sin j\\theta_{\\frac{d}{2}} &amp; \\cos j\\theta_{\\frac{d}{2}} \\end{array} \\right).$\n\nHere, sub-matrices  $M_{\\Theta}^{(i,j)}[k:k + 2,k:k + 2]$  capture the height position of the feature  $(i)$  for odd values of dimension  $k$ , and capture the width position  $(j)$  for even values of  $k$  (1-based indexing). Furthermore,  $\\Theta = [\\theta_1\\dots \\theta_{d / 2}]$  is a vector of frequencies for the various dimensions of  $x$ , where  $\\theta_{m}$  is defined following standard practice for RoPE-1D [20].\n\nCritically, our simple implementation of the RoPE-2D transform satisfies the \"relative\" property: that inner products between two vectors are dependent only on their relative difference in height and width position, rather than their absolute position (see more details in Appendix B).\n\nDiscussion: Our vision encoder is specifically designed for multimodal modeling. Traditional encoders are typically optimized for ImageNet performance at a resolution of, for example,  $224 \\times 224$  or  $336 \\times 336$  pixels. When incorporated into multimodal language models – which flexibly perform tasks from standard classification to optical character recognition – prior works typically break an image into smaller (square) tiles before independently feeding tiles to the vision encoder. Instead, our vision encoder can naturally adapt to both high and low resolution images at their native aspect ratio, providing substantially improved performance for multi-modal tasks (see Section 4.4).\n\n## 2.3 Complete architecture\n\nThe Pixtral vision encoder is linked to the multimodal decoder via a two-layer fully connected network. This network transforms the output of the vision encoder into the input embedding size required by the decoder via an intermediate hidden layer of the same size, employing the GeLU activation [8]. The image tokens are treated identically to the text tokens by the multimodal decoder, including RoPE-1D [20] positional encodings for all tokens. Particularly, our decoder uses a causal self-attention mechanism, smoothly facilitating capabilities such as multi-image conversations. The architecture is illustrated in Figure 3.","images":[{"id":"img-4.jpeg","top_left_x":297,"top_left_y":165,"bottom_right_x":1397,"bottom_right_y":657,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Diagram\",\n  \"short_description\": \"Multi-modal Transformer Decoder Architecture\",\n  \"summary\": \"The diagram illustrates the architecture of a multi-modal transformer decoder. It shows how images and text are processed through different components before being combined in the multi-modal transformer decoder. The images are processed by the Pixtral-ViT (Vision Transformer) and then passed through a Vision-Language Projector. The text is tokenized by a Text Tokenizer. The processed images and tokenized text are then combined with special tokens [b] and [e] before being fed into the multi-modal transformer decoder. The output of the decoder generates a combined representation of the input image and text, as demonstrated by the example of identifying a cat and a dog in the images and corresponding text.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":4,"markdown":"![img-5.jpeg](img-5.jpeg)\nFigure 4: MM-MT-Bench: We open-source a new instruction following benchmark for multimodal models, which correlates highly with LMSys ELO ratings. Given an input image, reference answer and model response, an independent LLM judge is instructed to grade the model's response on a scale of 1 through 10.\n\n# 3 MM-MT-Bench: A benchmark for multi-modal instruction following\n\nMost existing multimodal benchmarks measure the ability of a model to perform some form of multiple-choice question answering given an input image. While this is a useful signal for the model's ability to understand the image, it does not capture the extent of the model's utility to a user (for instance as a multimodal assistant or chatbot). In order to measure this quality, instruction-tuned text-only models are typically evaluated on MT-Bench [25], wherein an independent LLM judge grades a model's output with respect to a reference answer. We construct and release a new benchmark named Multimodal MT-Bench (MM-MT-Bench) in a similar vein to the text-only variant, to evaluate the performance of instruction-tuned multimodal models.\n\nDesign. MM-MT-Bench contains 92 conversations in total. It covers a breadth of practical use cases, covering five categories of images: charts (21), tables (19), PDF pages (24) diagrams (20) and miscellaneous (8). There are 69 single-turn conversations, 18 conversations with 2 turns, 4 of them with 3 turns and 1 conversation with 4 turns. To evaluate a model, we query the model in parallel over all turns of a conversation, providing reference answers for the past turns as history. Each turn is rated independently by the judge with the entire conversation history provided. The judge is prompted to rate the conversation on a scale of 1 to 10 based on correctness (i.e. was the extracted information correct) and completeness (i.e. does the model answer cover all the points raised in the reference). The evaluation process is illustrated in Figure 4. The judge prompt is provided in Appendix A.5. The results shown in Table 2 show that MM-MT-Bench has a 0.91 Pearson Correlation Coefficient with LMSys-Vision ELO ratings.\n\nExamples. MM-MT-Bench was designed to mimic real world usage of vision-language models, for extraction, summarization and reasoning over the contents of an image. Representative images from each category are provided in Figure 12 and an example of rated model responses from vision-language models are provided in Figure 11. We manually curated the images, prompts and answers and verified the answers from a second group of labelers. We ensure that all prompts require reference to the image input to be answered correctly.\n\n# 4 Results\n\nIn this section, we provide evaluations of Pixtral 12B against closed and open-source models across a range of model sizes, re-evaluating all models through the same evaluation harness. Particularly, for each dataset, we design the prompt such that we can reproduce the results of leading multimodal models (GPT-4o [16] and Claude-3.5 Sonnet [1]). These prompts are 'Explicit' and fully specify the output format (see Section 4.2), allowing models which follow the prompt instructions to be marked accurately at test-time. All models were evaluated with the same prompts, which are specified in Appendix A. We provide additional analysis on re-evaluating models under various prompts and metrics in Sections 4.2 and 4.3, as well as in Appendices D and E.","images":[{"id":"img-5.jpeg","top_left_x":358,"top_left_y":193,"bottom_right_x":1337,"bottom_right_y":607,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Evaluation\",\n  \"short_description\": \"Evaluation of a model's response to a user's question about cropping an image.\",\n  \"summary\": \"The image shows a user asking how to crop an image to their selection. The reference response provides the correct steps, while the model's response is somewhat correct but includes unnecessary steps and slightly incorrect information. The LLM judgment rates the model's response as [[7]].\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":5,"markdown":"|   | Mathvista CoT | MMMU CoT | ChartQA CoT | DocVQA ANLS | VQAv2 VQA Match | MM-MT-Bench GPT-4o Judge | LMSys-Vision (Oct '24)  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n|  Pixtral 12B | 58.3 | 52.0 | 81.8 | 90.7 | 78.6 | 6.05 | 1076  |\n|  Qwen-2-VL 7B [23] | 53.7 | 48.1 | 41.2 | 94.5 | 75.9 | 5.45 | 1040  |\n|  → w/ Flexible Parsing | 55.2 | 48.7 | 77.5 | - | - | - | -  |\n|  Llama-3.2 11B [6] | 24.3 | 23.0 | 14.8 | 91.1 | 67.1 | 4.79 | 1032  |\n|  → w/ Flexible Parsing | 47.9 | 45.3 | 78.5 | - | - | - | -  |\n|  Molmo-D 7B [4] | 12.3 | 24.3 | 27.0 | 72.2 | 57.1 | 3.72 | -  |\n|  LLaVA-OneVision 7B [9] | 36.1 | 45.1 | 67.2 | 90.5 | 78.4 | 4.12 | -  |\n|  Claude-3 Haiku [1] | 44.8 | 50.4 | 69.6 | 74.6 | 68.4 | 5.46 | 1000  |\n|  Gemini-1.5-Flash 8B(0827) [18] | 56.9 | 50.7 | 78.0 | 79.5 | 65.5 | 5.93 | 1111  |\n|  Molmo 72B [4] | 52.2 | 52.7 | 75.6 | 86.5 | 75.2 | 3.51 | -  |\n|  LLaVA-OneVision 72B [9] | 57.2 | 54.4 | 66.9 | 91.6 | 83.8 | 4.95 | 992  |\n|  Qwen-2-VL 72B [23] | 68.2 | 60.3 | 66.6 | 96.3 | 81.6 | 6.59 | 1104  |\n|  Llama-3.2 90B [6] | 49.1 | 53.7 | 33.8 | 85.7 | 67.0 | 5.50 | 1071  |\n|  GPT-4o (0513) [16] | 64.6 | 68.6 | 85.1 | 88.9 | 77.8 | 7.72 | 1208  |\n|  Claude-3.5 Sonnet [1] | 64.4 | 68.0 | 87.6 | 90.3 | 70.7 | 7.50 | 1189  |\n\nTable 2: Multimodal Benchmarks. Pixtral substantially outperforms open models of a similar size, as well as several closed-source models. We re-evaluate all models with the same prompt and evaluation metric (see Section 4.2). For transparent comparison against Qwen2-VL 7B [23] and Llama-3.2 11B [6], we additionally report their performance under relaxed evaluation constraints in (gray) (see Section 4.3). To further investigate the gap with reported figures for some open-source models, we provide analysis in Section E.\n\n|   | MT-Bench | MMLU 5-shot | Math Maj@1 | HumanEval Pass@1  |\n| --- | --- | --- | --- | --- |\n|  Pixtral 12B | 7.68 | 69.2 | 48.1 | 72.0  |\n|  LLaVA-OneVision 7B [9] | 6.94 | 67.9 | 38.6 | 65.9  |\n|  Molmo-D 7B [4] | 4.53 | 61.2 | 10.2 | 3.7  |\n|  Qwen-2-VL 7B [23] | 6.41 | 68.5 | 27.9 | 62.2  |\n|  Llama-3.2 11B [6] | 7.51 | 68.5 | 48.3 | 62.8  |\n\nTable 3: Language benchmarks. Pixtral 12B consistently outperforms open-source models of a comparable size on text-only benchmarks, making it a drop-in multimodal replacement for existing text-only deployments.\n\n# 4.1 Main Results\n\nMultimodal performance: Table 2 shows that Pixtral substantially outperforms all open models around its scale on multimodal benchmarks, as well as closed source models such as Claude-3 Haiku [1] and Gemini-1.5 Flash 8B [18]. Particularly, Pixtral outperforms all models of comparable size on MM-MT-Bench, which targets real world use cases, a finding corroborated by strong performance on LMSys Vision Arena. On this public leaderboard, Pixtral 12B approaches the performance of the largest open-weights models, such as Qwen2-VL 72B [23] and Llama-3.2 90B [6].\n\nWe highlight that, with our 'Explicit' prompts, the performance of some open-source models is substantially lower than their reported figures. For the closest open-source models - Qwen2-VL 7B [23] and Llama-3.2 11B [6] - this is mainly due to models not following instructions on answer formatting (e.g. generating \"The answer is 6.\" instead of \"Final answer: 6\"). For transparent comparison against these models, we further report their evaluations using relaxed metrics, with more flexible parsing, in gray (see Section 4.3). We analyze the performance of these models under various prompts in Appendix D. In Appendix E, we customize the evaluation to each model in turn, describing the changes required to bridge the gaps to reported performance.\n\nLanguage performance: Table 3 evaluates Pixtral 12B against open-source models of comparable size on common text-only benchmarks (again, with common prompting and evaluation protocols). Pixtral does not compromise text understanding in pursuit of multimodal capabilities, making it a suitable drop-in replacement for both text and vision tasks.","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":6,"markdown":"![img-6.jpeg](img-6.jpeg)\nFigure 5: Effect of 'Naive' vs. 'Explicit' prompts on leading models. Leading models benefit greatly from 'Explicit' prompts which provide details about the output format. This makes sense, as otherwise substantively correct responses are marked as incorrect during evaluation (top row, right).\n\n|  Prompt → | VQAv2 |   | ChartQA |   | MMMU  |   |\n| --- | --- | --- | --- | --- | --- | --- |\n|   |  Naive | Explicit | Naive | Explicit | Naive | Explicit  |\n|  GPT-4o (0513) [16] | 64.2 | 77.8 | 58.0 | 85.1 | 55.0 | 68.6  |\n|  Sonnet-3.5 [1] | 50.2 | 70.7 | 39.6 | 87.6 | 48.6 | 68.0  |\n|  Qwen-2-VL 7B [23] | 82.1 | 75.9 | 83.4 | 41.2 | 46.7 | 48.1  |\n|  Llama-3.2 11B [21] | 29.5 | 67.1 | 0.0 | 14.8 | 20.7 | 23.0  |\n|  Llama-3.2 90B [21] | 52.6 | 67.0 | 3.9 | 33.8 | 27.0 | 53.7  |\n|  Pixtral 12B | 78.9 | 78.6 | 84.3 | 81.8 | 45.8 | 52.0  |\n\nTable 4: Prompt ablations. Leading models require prompts which explicitly specify the output format to perform well. Pixtral 12B performs well with both 'Explicit' and 'Naive' prompts, with only a minor regression on ChartQA.\n\n# 4.2 Prompt selection\n\nHere we discuss our methodology for designing the evaluation prompts. In our evaluation harness, we choose prompts which allow for reproduction of the reported results of leading closed-source models: GPT-4o [16] and Claude-3.5-Sonnet [1]. These prompts are provided in Appendix A, and we report results averaged over 10 prompts in Appendix D.\n\nWe find that commonly used prompts do not properly specify the output format. For instance, for a multiple choice question, we find open-source prompts include vague instructions like \"Select the correct answer from the options above\". In this case, it is impossible for models to know whether answers should be presented as an index (\"Option A\", \"Option B\" etc.) or with a natural language response. Models are then penalized for incorrect formatting. As such, leading models require prompts which explicitly specify the required output format. We illustrate this with a real example from MMMU in Figure 5.\n\nIn Table 4, we demonstrate that our 'Explicit' prompts substantially improve the performance of leading models over 'Naive' prompts. We also note that in a number of cases, the performance of smaller models reduces with the Explicit prompt format, perhaps due to a discrepancy with the prompt-style in the training set of these benchmarks. Pixtral 12B generally performs better with Explicit prompts, with only a minor regression on ChartQA.\n\n# 4.3 Sensitivity to evaluation metrics\n\nIn Section 4.2, we discuss the importance of prompts which properly specify the output format. However, during evaluations, we find that even with Explicit prompts, many models still provide outputs in various formats, which are then penalized by metrics which require responses to match the reference answers exactly.\n\nTo investigate this, we take models' generations and evaluate them under progressively looser parsing constraints. For instance, if the correct answer is \"6\", flexible metrics do not penalize answers such as \"6.0\" or \"The answer is 6\". We provide the details of these parsing settings in Appendix C, but","images":[{"id":"img-6.jpeg","top_left_x":299,"top_left_y":198,"bottom_right_x":1400,"bottom_right_y":512,"image_base64":null,"image_annotation":"{\"document_type\": \"image\", \"short_description\": \"Comparison of prompt techniques for GPT-4o\", \"summary\": \"The image compares two methods for prompting GPT-4o to identify harmful insects from an image. The naive prompt from VLMEvalKit leads to an incorrect answer, while the explicit prompt provides a structured format that results in the correct identification of the insects as pests of potatoes. The exact match metric is used to evaluate the responses, showing the importance of clear and explicit instructions in achieving accurate results.\"}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":7,"markdown":"|   | Llama-3.2 11B [21] | Llama-3.2 90B [21] | Qwen2-VL 7B [23] | Pixtral 12B  |\n| --- | --- | --- | --- | --- |\n|  Mathvista  |   |   |   |   |\n|  Baseline | 24.3 | 49.1 | 53.7 | 58.3  |\n|  Flexible level 1 | 25.9 | 50.3 | 54.3 | 58.3  |\n|  Flexible level 2 | 40.2 | 54.7 | 54.3 | 58.3  |\n|  Flexible level 3 | 47.9 | 57.3 | 55.2 | 58.5  |\n|  MMMU  |   |   |   |   |\n|  Baseline | 23.0 | 53.7 | 48.1 | 52.0  |\n|  Flexible level 1 | 23.4 | 53.7 | 48.1 | 52.0  |\n|  Flexible level 2 | 41.0 | 55.7 | 48.1 | 52.0  |\n|  Flexible level 3 | 45.3 | 56.7 | 48.7 | 52.0  |\n|  ChartQA  |   |   |   |   |\n|  Baseline | 14.8 | 33.8 | 41.2 | 81.8  |\n|  Flexible level 1 | 20.4 | 33.9 | 73.8 | 81.9  |\n|  Flexible level 2 | 29.9 | 35.6 | 73.8 | 81.9  |\n|  Flexible level 3 | 78.5 | 79.1 | 77.5 | 82.0  |\n\nTable 5: Flexible parsing ablations. We evaluate models under progressively looser parsing constraints (see Appendix C for details). Under loose parsing constraints, the performance of some models dramatically improves. Pixtral 12B performance is stable under all parsing conditions, and continues to lead even when flexible parsing is accounted for. 'Flexible Level 3' is included for illustration only, as it allows some incorrect answers to be marked as correct.\n\n![img-7.jpeg](img-7.jpeg)\nFigure 6: Vision encoder ablations: When leveraged for visual instruction tuning, our encoder substantially outperforms a strong CLIPA [10] baseline for tasks requiring fine-grained document understanding, while maintaining parity for natural images.\n\nhere note that 'Flexible Level 3' marks a response as correct if the reference answer occurs anywhere in the generation. This is an overly generous metric which is included only to illustrate an upper bound, as it permits answers like \"6000\" for a reference answer of \"6\".\n\nWe provide the results of our analysis in Table 5. We find that the performance of some models dramatically improves with more flexible parsing metrics, indicating that the lower scores can be attributed to the inability of models to properly follow prompt instructions. We further note that Pixtral 12B benefits very little from flexible parsing (substantiating its ability to follow instructions), and furthermore can generally outperform other models even after flexible metrics are used.\n\n# 4.4 Vision Encoder Ablations\n\nIn order to verify the design choices for our vision encoder, we conduct small-scale ablations with Visual Instruction Tuning [13]. We conduct short-horizon multimodal instruction-tuning runs, both with our vision encoder (Pixtral-ViT), as well as a CLIPA [10] backbone as a baseline. For both vision encoders, we use Mistral-Nemo 12B-Instruct [15] to initialize the multimodal decoder.","images":[{"id":"img-7.jpeg","top_left_x":516,"top_left_y":943,"bottom_right_x":1179,"bottom_right_y":1372,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"bar chart\",\n  \"short_description\": \"Comparison of accuracy percentages for different models on various datasets.\",\n  \"summary\": \"The bar chart compares the accuracy percentages of three different models (CLIPA at 224px, CLIPA at 1120px, and Pixtral-ViT at 1024px) across four datasets (ChartQA, DocVQA, VQAv2, and AI2D). The chart shows that Pixtral-ViT generally achieves higher accuracy across all datasets, particularly excelling in the DocVQA and AI2D datasets. CLIPA at 1120px performs better than CLIPA at 224px, but still lags behind Pixtral-ViT. The accuracy percentages range from around 40% to 90%, with the highest accuracy observed in the AI2D dataset for the Pixtral-ViT model.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":8,"markdown":"Like many open-source vision encoders, CLIPA is trained at a fixed resolution of $224\\times 224$ pixels. In order to upscale the resolution in vision-language models, existing methods *[12]* construct several tiled crops from the image, and pass each crop independently through the vision encoder at its pretraining resolution. We conduct two ablations with CLIPA: (a) we resize the entire image to $224\\times 224$; (b) we construct $25$ crops of the input image, for a total resolution of $1120\\times 1120$. These models are also evaluated at $224$ pixels and $1120$ pixels respectively, while our flexible encoder is evaluated at variable image resolutions, with a maximum resolution of $1024$ pixels.\n\nIn Figure 6, we find that our model substantially outperforms CLIPA in settings which require fine-grained understanding, such as chart and document understanding, while matching its performance on natural language benchmarks such as VQAv2.\n\n## 5 Qualitative examples\n\nWe discuss real world application of Pixtral by looking at some qualitative examples. Specifically, Pixtral can be used for reasoning over complex figures (eg. Fig. 7), multi-image instruction following (eg. Fig. 8), chart understanding and analysis (eg. Fig. 9) and converting image to code (eg. Fig. 10).\n\nIn Fig. 11, we compare Pixtral 12B to QwenVL-7B and Gemini-1.5 Flash-8B (0827) on an example from MM-MT-Bench. The example consists of a complex chart on job jitters in the US with an instruction requiring accurate understanding, reasoning and analysis of the chart. Pixtral’s response is complete and accurate, hence getting a rating of 8, while Gemini-Flash-8B extracts wrong information, and QwenVL does not elaborate on trends.\n\n## 6 Conclusion\n\nThis paper introduced Pixtral 12B, a state-of-the-art multimodal model that excels in both text-only and multimodal tasks. With a novel architecture featuring a 400M-parameter vision encoder and a 12B-parameter multimodal decoder, Pixtral 12B demonstrates strong performance across various benchmarks, outperforming other open models and matching larger models. Its superior instruction following abilities, support for variable image sizes, and long context window make it highly versatile for complex multimodal applications. Pixtral 12B is released under the Apache 2.0 license.\n\n## 7 Contributors\n\nMistral AI Science team (listed in alphabetical order by last name):\n\nPravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, Sophia Yang.\n\n#### Acknowledgements\n\nWe extend our thanks to the LMSys team for their assistance in deploying our model in the LLM arena, and the vLLM team for their help in integrating Pixtral 12B into their inference library.\n\n##","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":9,"markdown":"![img-8.jpeg](img-8.jpeg)\nList the top 5 countries in Europe with the highest GDP\n\n![img-9.jpeg](img-9.jpeg)\nFigure 7: Reasoning over complex figures. An example showcasing Pixtral's capabilities to understand and reason over complex figures. Pixtral correctly identifies that the green boxes represent the European countries and then reads and sorts the GDP of all the European countries to list the top 5 with accurate GDP numbers.\n\nThe top 5 countries in Europe with the highest GDP, as represented in the diagram, are:\n\n1. Germany\n\nGDP: $3.99 trillion\nPercentage: 4.65%\n\n2. United Kingdom\n\nGDP: $2.82 trillion\nPercentage: 3.29%\n\n3. France\n\nGDP: $2.78 trillion\nPercentage: 3.24%\n\n4. Italy\n\nGDP: $2.07 trillion\nPercentage: 2.42%\n\n5. Spain\n\nGDP: $1.43 trillion\nPercentage: 1.66%\n\nThese countries are highlighted in green on the diagram.","images":[{"id":"img-8.jpeg","top_left_x":579,"top_left_y":305,"bottom_right_x":1099,"bottom_right_y":737,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"info-graphic\",\n  \"short_description\": \"This image is a pie chart showing the distribution of global wealth among different countries and regions.\",\n  \"summary\": \"The pie chart illustrates the global distribution of wealth by country and region. The United States holds the largest share at $20.49 trillion, accounting for 23.89% of the total. China follows with $13.61 trillion (15.86%), and Japan with $4.97 trillion (5.79%). Other notable countries include Germany ($3.99 trillion, 4.65%), India ($2.73 trillion, 3.18%), and the United Kingdom ($2.78 trillion, 3.24%). The chart also categorizes countries by regions such as Africa, Australia, Latin America and Caribbean, Middle East, Europe, Asia, and North America, each represented by different colors. The 'Other Countries' category collectively holds 8.98% of the global wealth.\"\n}"},{"id":"img-9.jpeg","top_left_x":518,"top_left_y":807,"bottom_right_x":700,"bottom_right_y":963,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Pie Chart\",\n  \"short_description\": \"A pie chart showing the distribution of global military spending by country.\",\n  \"summary\": \"The pie chart illustrates the proportion of global military expenditures attributed to various countries. The United States is highlighted as the largest spender, accounting for 23.89% of the total. Other countries such as China, Russia, and the United Kingdom are also represented, showing their respective shares of global military spending. The chart uses different colors to distinguish between countries and includes a legend for clarity.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":10,"markdown":"![img-10.jpeg](img-10.jpeg)\nFigure 8: Multi-image instruction following. Pixtral can process arbitrary number of images in its context window. The example shows that Pixtral can successfully combine the information from both images into a single markdown table.\n\n![img-11.jpeg](img-11.jpeg)\n\n![img-12.jpeg](img-12.jpeg)\n\nThis table combines the performance metrics of various models across multiple benchmarks.\n\nPixtral  $\\nrightarrow$  Ask anything!","images":[{"id":"img-10.jpeg","top_left_x":326,"top_left_y":558,"bottom_right_x":865,"bottom_right_y":781,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Comparison of AI Models\",\n  \"short_description\": \"A table comparing the performance of various AI models across different benchmarks.\",\n  \"summary\": \"The table presents a comparison of several AI models, including Pixtral 12B, Claude-3 Haiku, Gemini-1.5 Flash 8B, LLaVA-OV 72B, GPT-4o, and Claude-3.5 Sonnet. The performance metrics are evaluated across different benchmarks such as Mathvista, MMMU, ChartQA, DocVQA, and VQAv2. Each model's performance is measured in terms of accuracy or score, with notable variations across the different tasks. For instance, Pixtral 12B shows strong performance in ChartQA and DocVQA but lags in Mathvista. Claude-3.5 Sonnet excels in DocVQA and ChartQA but has lower scores in Mathvista and MMMU. This comparison highlights the strengths and weaknesses of each model in different types of tasks, providing insights into their overall capabilities and potential areas for improvement.\"\n}"},{"id":"img-11.jpeg","top_left_x":868,"top_left_y":558,"bottom_right_x":1230,"bottom_right_y":781,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Table\",\n  \"short_description\": \"Comparison of different models on various benchmarks\",\n  \"summary\": \"The table compares the performance of various models (Pixtral 12B, Qwen2-VL 7B, LLaVA-OV 7B, Phi-3 Vision, and Phi-3.5 Vision) across multiple benchmarks: Mathvista, MMMU, ChartQA, DocVQA, and VQAv2. Each model's performance is represented by a percentage score in each benchmark category. The table highlights the strengths and weaknesses of each model in different tasks, providing a comprehensive overview of their capabilities in handling various types of data and questions.\"\n}"},{"id":"img-12.jpeg","top_left_x":326,"top_left_y":930,"bottom_right_x":1368,"bottom_right_y":1366,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Multimodal Benchmarks\",\n  \"short_description\": \"Performance comparison of various models on different benchmarks.\",\n  \"summary\": \"The image displays a table comparing the performance of different models on various benchmarks, including Mathvista (CoT), MMMU (CoT), ChartQA (CoT), and DocVQA (ANLS). The models listed are Pixtral 12B, Claude-3 Haiku, Gemini-1.5 Flash 8B (0827), LLaVA-OV 72B, GPT-4o, Claude-3.5 Sonnet, Qwen2-VL 7B, LLaVA-OV 7B, Phi-3 Vision, and Phi-3.5 Vision. Each model's performance is measured in percentage scores across the different benchmarks.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":11,"markdown":"# When did things start to go wrong for dark dragon?\n\n![img-13.jpeg](img-13.jpeg)\nFigure 9: Chart Understanding and Analysis. Pixtral demonstrates the capability to interpret and analyze intricate charts with high accuracy. In this instance, Pixtral correctly identifies that \"dark-dragon\" corresponds to the red line. Furthermore, it recognizes that the training loss is expected to decrease smoothly and notes that the training run became unstable around the 10K step mark due to a significant spike in loss.\n\nThe image is a line graph titled 'train/loss' that shows the training loss over steps for four different models: dark-dragon-50, dainty-sunset-49, different-haze-34, and morning-field-33. The x-axis represents the steps, ranging from 0 to 60k, and the y-axis represents the loss on a logarithmic scale from 1e-5 to 1e+5.\n\nTo determine when things started to go wrong for the dark-dragon-50 model, we need to look at the red line representing this model. Initially, the loss decreases, indicating that the model is learning effectively. However, around the 10k step mark, the loss starts to spike significantly. This spike indicates that the model's performance is deteriorating. The loss continues to spike and fluctuate, reaching very high values, which suggests that the model is not learning effectively and is likely overfitting or encountering other issues.\n\nTherefore, the critical point where things started to go wrong for the dark-dragon-50 model is around the 10k step mark.\n\nPixtral  $\\checkmark$  Ask anything!","images":[{"id":"img-13.jpeg","top_left_x":377,"top_left_y":543,"bottom_right_x":793,"bottom_right_y":770,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Graph\",\n  \"short_description\": \"A line graph showing the training and loss metrics for different datasets over a series of iterations.\",\n  \"summary\": \"The graph displays the training and loss metrics for four different datasets: dark-dragon-50, daisy-sunset-49, different-haze-34, and morning-field-33. The x-axis represents the number of iterations, ranging from 0 to 60k, while the y-axis represents the loss values on a logarithmic scale from 1e-5 to 1e+5. Each dataset is represented by a different colored line, and the graph shows how the loss values decrease over time for each dataset. The lines show fluctuations and trends, indicating the performance of the training process for each dataset.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":12,"markdown":"![img-14.jpeg](img-14.jpeg)\n\nWrite HTML code to create a website like this\n\n![img-15.jpeg](img-15.jpeg)\n\nCertainly! Below is an example of HTML code to create a simple webpage that allows users to pick an ice cream flavor. This example includes a dropdown menu for selecting the flavor and a \"Next\" button.\n\n![img-16.jpeg](img-16.jpeg)\nFigure 10: Image to Code. This demonstration illustrates Pixtral's capability to convert hand-drawn website interfaces into executable HTML code, bringing hand-drawn designs to life as fully functional websites.","images":[{"id":"img-14.jpeg","top_left_x":377,"top_left_y":253,"bottom_right_x":1329,"bottom_right_y":576,"image_base64":null,"image_annotation":"{\"document_type\": \"wireframe_to_app\", \"short_description\": \"Transformation of a hand-drawn wireframe into a functional app interface.\", \"summary\": \"The image shows a hand-drawn wireframe on the left, depicting a simple interface with a title 'Pick An Ice Cream Flavor' and a dropdown menu with the option 'Vanilla' visible. Below the dropdown, there is a 'Next' button. The wireframe is transformed into a digital app interface on the right, maintaining the same elements: the title, the dropdown menu with 'Vanilla' as the selected option, and the 'Next' button, which is green in the digital version. This illustrates the process of converting a rough sketch into a functional user interface.\"}"},{"id":"img-15.jpeg","top_left_x":425,"top_left_y":682,"bottom_right_x":566,"bottom_right_y":866,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Handwritten Note\",\n  \"short_description\": \"A handwritten note with a title and two sections of text.\",\n  \"summary\": \"The image shows a handwritten note on lined paper. The note is titled 'First and 15 Game Plan.' It is divided into two main sections: 'Offense' and 'Defense.' Each section contains several bullet points, likely outlining strategies or plays. The note appears to be related to a sports game plan, specifically for American football, given the context of 'First and 15.'\"\n}"},{"id":"img-16.jpeg","top_left_x":413,"top_left_y":1075,"bottom_right_x":1283,"bottom_right_y":1870,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"HTML\",\n  \"short_description\": \"HTML code for an ice cream flavor selection page\",\n  \"summary\": \"This HTML document creates a web page where users can select their favorite ice cream flavor from a dropdown menu. The page includes options for Vanilla, Chocolate, Strawberry, Cookie Dough, and Mint Chocolate. Upon selection, clicking the 'Next' button will display an alert with the chosen flavor.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":13,"markdown":"# Job jitters in the United States\n\n![img-17.jpeg](img-17.jpeg)\nNote: Figures may not sum to  $100\\%$ , because of rounding.\nSource: McKinsey Consumer Community Pulse Survey, Feb 7-15, 2023.\nn = 1,852, sampled and weighted to match the US general population aged 15-74\n\n# McKinsey &amp; Company\n\n# U\n\nIn reporting the different generations that show total work primarily, we used to project the same project as the other 1000 generations. We used the same project as the other 1000 generations and the same project as the other 1000 generations.\n\n# 1. 2017\n\n1. The period of all,  $25\\%$\n\n2. The period of 1999\n\n3. Estimated annual,  $25\\%$\n\n4. Estimated annual,  $25\\%$\n\n5. Estimated annual,  $25\\%$\n\n6. Estimated annual,  $25\\%$\n\n7. Estimated annual,  $25\\%$\n\n8. Estimated annual,  $25\\%$\n\n9. Estimated annual,  $25\\%$\n\n10. Estimated annual,  $25\\%$\n\n11. Estimated annual,  $25\\%$\n\n12. Estimated annual,  $25\\%$\n\n13. Estimated annual,  $25\\%$\n\n14. Estimated annual,  $25\\%$\n\n15. Estimated annual,  $25\\%$\n\n16. Estimated\n\n# 1.10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\n\n# 1.1000000000000000000000000000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000000000\n\n# 1.10000000000000000000000000000000000000000000\n\n# 1.100000000000000000000000000000000000000\n\n#","images":[{"id":"img-17.jpeg","top_left_x":311,"top_left_y":561,"bottom_right_x":812,"bottom_right_y":926,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Bar Chart\",\n  \"short_description\": \"This bar chart shows the level of concern about job security among different generations, including Gen Z, Millennials, Gen X, and Baby Boomers.\",\n  \"summary\": \"The chart illustrates the percentage of respondents from each generation who are not worried at all, slightly worried, somewhat worried, and extremely worried about job security. Gen Z shows the highest percentage of respondents who are not worried at all (25%), while Baby Boomers have the highest percentage of respondents who are extremely worried (13%). Millennials and Gen X show a more balanced distribution of concern levels, with Millennials having 37% slightly worried and 18% somewhat worried, and Gen X having 29% slightly worried and 16% somewhat worried. Overall, the total percentage of respondents who are not worried at all is 27%, slightly worried is 22%, somewhat worried is 15%, and extremely worried is 5%.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":14,"markdown":"![img-18.jpeg](img-18.jpeg)\nUSER: Based on the heatmap, what can we observe about malaria trends from 2000 to 2019?\n\n![img-19.jpeg](img-19.jpeg)\nUSER: Identify the three largest drops in the 30-year mortgage rate shown in this chart. For each drop, state the starting and ending dates, calculate the total basis-point decrease and the rate of change. Which of these drops was the largest in magnitude and which had the largest rate of change?\n\n![img-20.jpeg](img-20.jpeg)\nUSER: Can you explain to me the mechanism for the evolution of technology in this diagram?\n\n![img-21.jpeg](img-21.jpeg)\nCategory: CHARTS\nUSER (turn 1): How long do I spend commuting a week?\n\n![img-22.jpeg](img-22.jpeg)\nCategory: DIAGRAMS\nUSER: What are the main points discussed in this doc?\nCategory: PDF PAGES\nFigure 12: Example images from MM-MT-Bench\n\n![img-23.jpeg](img-23.jpeg)\nUSER (turn 2): How long do I spend sleeping each week?\nCategory: TABLES\nUSER (turn 1): Is there tumor present in these scans?\nUSER (turn 2): Is which images do the tumors appear largest? Provide details.\nCategory: MISCELLANEOUS","images":[{"id":"img-18.jpeg","top_left_x":333,"top_left_y":380,"bottom_right_x":826,"bottom_right_y":682,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"image\",\n  \"short_description\": \"Malaria mortality rate of children in 2000 and 2019\",\n  \"summary\": \"The image shows a comparison of the malaria mortality rate of children younger than 5 years due to malaria (caused by plasmodium falciparum) in the years 2000 and 2019. The maps illustrate significant reductions in mortality rates across many regions, particularly in Africa, over the 19-year period. The data source is the IHME Atlas Project (2020), and the image is licensed under CC-BY by the author Max Roser.\"\n}"},{"id":"img-19.jpeg","top_left_x":921,"top_left_y":380,"bottom_right_x":1283,"bottom_right_y":671,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"Last 90 Days of 30-Year Mortgage Rate Average\",\n  \"summary\": \"The chart displays the average nationwide rates for 30-year fixed-rate new purchase loans with at least a 20% down payment and an applicant credit score of 680 to 739 over the last 90 days. Key data points include: 6.33% on April 9, 7.37% on April 25, 7.14% on May 29, 6.83% on May 16, 6.77% on June 13, 7.09% on July 3, and 6.84% on July 10. The chart is sourced from Investopedia, ChartInvestor, and Black Knight.\"\n}"},{"id":"img-20.jpeg","top_left_x":319,"top_left_y":781,"bottom_right_x":829,"bottom_right_y":1104,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"image\",\n  \"short_description\": \"A graph depicting the stages of technological revolution.\",\n  \"summary\": \"The image is a graph that illustrates the stages of a technological revolution over time. The x-axis represents time, while the y-axis represents the degree of diffusion of the technological revolution. The graph is divided into several periods: Installation Period, Deployment Period, and Maturity. Key phases include Irruption, Frenzy, Synergy, and Maturity. Each phase is characterized by different socio-economic conditions and technological impacts. The graph also highlights turning points and transitions between these phases, indicating periods of technological split, financial bubble time, and institutional recomposition. The overall concept emphasizes the cyclical nature of technological revolutions and their impact on society and the economy.\"\n}"},{"id":"img-21.jpeg","top_left_x":911,"top_left_y":816,"bottom_right_x":1293,"bottom_right_y":1115,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"image\",\n  \"short_description\": \"A weekly schedule table\",\n  \"summary\": \"The image shows a detailed weekly schedule from 5:00 AM to 11:00 PM. Each day is divided into hourly segments with specific activities listed. For example, Monday includes activities like Wake Up/Get Ready, Commute, Work, Unit 1, Dinner and Prep, and Sleep. The schedule varies slightly each day with activities such as Gym, Errands, and Class included on different days. The table uses different colors to distinguish between different types of activities, providing a clear and organized view of the weekly routine.\"\n}"},{"id":"img-22.jpeg","top_left_x":319,"top_left_y":1225,"bottom_right_x":829,"bottom_right_y":1680,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Django Documentation\",\n  \"short_description\": \"Django administration interface guide\",\n  \"summary\": \"This document provides instructions on how to customize the Django administration interface. It includes details on modifying the list display, adding actions, and providing intermediate pages. The guide also covers the use of serializers and the export_as_csv_action function to handle data export.\"\n}"},{"id":"img-23.jpeg","top_left_x":929,"top_left_y":1227,"bottom_right_x":1283,"bottom_right_y":1676,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Medical Imaging\",\n  \"short_description\": \"MRI scans of the brain with highlighted regions\",\n  \"summary\": \"This image shows a series of MRI scans of the brain. Each scan highlights specific regions with red circles, indicating areas of interest or abnormalities that need further medical evaluation. The scans are taken from different angles and slices of the brain to provide a comprehensive view.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":15,"markdown":"References\n\n- [1] Anthropic (2024). The Claude 3 Model Family: Opus, Sonnet, Haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf.\n- [2] Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S. (2023). Fuyu-8b: A multimodal architecture for ai agents.\n- [3] Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I. M., et al. (2024). Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36.\n- [4] Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. (2024). Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.\n- [5] Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.\n- [6] Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.\n- [7] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.\n- [8] Hendrycks, D. and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.\n- [9] Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. (2024). Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.\n- [10] Li, X., Wang, Z., and Xie, C. (2023). An inverse scaling law for clip training. In NeurIPS.\n- [11] Li, Y. and Harada, T. (2022). Lepard: Learning partial point cloud matching in rigid and deformable scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5554–5564.\n- [12] Liu, H., Li, C., Li, Y., and Lee, Y. J. (2024a). Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.\n- [13] Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2024b). Visual instruction tuning. Advances in neural information processing systems, 36.\n- [14] Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. (2023). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.\n- [15] MistralAI (2024). Mistral NeMo 12B. https://mistral.ai/news/mistral-nemo/.\n- [16] OpenAI, R. et al. (2023). Gpt-4 technical report. ArXiv, 2303:08774.\n- [17] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.\n- [18] Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.\n- [19] Shazeer, N. (2020). Glu variants improve transformer. arXiv preprint arXiv:2002.05202.\n-","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":16,"markdown":"[20] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.\n- [21] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.\n- [22] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.\n- [23] Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. (2024). Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.\n- [24] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2023). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arxiv.\n- [25] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.\n- [26] Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. (2023). Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":17,"markdown":"# Appendix\n\n# Table of Contents\n\nA Prompts 19\n\nA.1 MMMU and Mathvista 19\nA.2 ChartQA 19\nA.3 VQAv2 19\nA.4 DocVQA 19\nA.5 MM-MT-Bench Judge Prompt 19\n\nB Relative Position Encoding Property of RoPE-2D 20\nC Flexible Parsing Settings 21\nD Robustness to prompting 21\n\nD.1 Llama-Specific Prompts 21\nD.2 Average performance across prompts 22\n\nE Reproducing Reported Numbers 22\n\nE.1 Summary 22\nE.2 Closed models: Claude-3 Haiku and Gemini-Flash-8B 22\nE.3 Qwen2-VL 7B 22\nE.4 Llama-3.2 23\nE.5 Llava-OneVision 72B 23\nE.6 Molmo 23","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":18,"markdown":"A Prompts\n\nHere we open-source the prompts used for evaluations in the main paper. As discussed in Section 4.2, prompts are selected to reproduce reported performance of GPT-4o *[16]* and Claude-3.5 Sonnet *[1]*.\n\n### A.1 MMMU and Mathvista\n\n> Analyze the image and question carefully, using step-by-step reasoning. First, describe any image provided in detail. Then, present your reasoning. And finally your final answer in this format: Final Answer: <answer> where <answer> is: - The single correct letter choice A, B, C, D, E, F, etc. when options are provided. Only include the letter. - Your direct answer if no options are given, as a single phrase or number. - If your answer is a number, only include the number without any unit. - If your answer is a word or phrase, do not paraphrase or reformat the text you see in the image. - You cannot answer that the question is unanswerable. You must either pick an option or provide a direct answer. IMPORTANT: Remember, to end your answer with Final Answer: <answer>.\n\n### A.2 ChartQA\n\n> Analyze the image and question carefully, using step-by-step reasoning. First, describe any image provided in detail. Then, present your reasoning. And finally your final answer in this format: Final Answer: <answer> where <answer> follows the following instructions: - <answer> should be a single phrase or number. - <answer> should not paraphrase or reformat the text in the image. - If <answer> is a ratio, it should be a decimal value like 0.25 instead of 1:4. - If the question is a Yes/No question, <answer> should be Yes/No. - If <answer> is a number, it should not contain any units. - If <answer> is a percentage, it should include a % sign. - If <answer> is an entity, it should include the full label from the graph. IMPORTANT: Remember, to end your answer with Final Answer: <answer>.\n\n### A.3 VQAv2\n\n> - Answer the question using a single word, number, or short phrase. Use as few words as possible. - If the answer is a number, report it as a number, i.e. 2, not Two, and only include the number without any unit. - If the question is Yes/No, answer with Yes/No, and nothing else (no likely, unknown, etc.). - You cannot answer that the question is unanswerable. You must answer.\n\n### A.4 DocVQA\n\n> Answer the question using a single word or phrase.\n\n### A.5 MM-MT-Bench Judge Prompt\n\n> SYSTEM: Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the most recent question given the previous conversation as context. Your evaluation should consider correctness and helpfulness. You will be given a reference answer and the assistant\\’s answer. Begin your evaluation by comparing the assistant\\’s answer with the reference answer. Identify and correct any mistakes. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 by strictly following this format: \"[[rating]]\", for example: \"Rating: [[5]]\".\n>\n> <|The Start of Conversation with User|>\n>\n>","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":19,"markdown":"### User:\n<image/> Analyze this image.\n\n### Reference answer:\nThe image consists of ...\n\n### Assistant:\nThis is an image of...\n\n$<$|The End of Conversation with User$|$$>$\\backslash$n$\\backslash$n\n\nThe history of the conversation is passed to the judge with reference answers as assistant answer (teacher-forcing).\n\n## Appendix B Relative Position Encoding Property of RoPE-2D\n\nIn this section, we show the relative position encoding property of RoPE-2D. The goal is prove that:\n\n$\\langle\\textsf{RoPE-2D}(x^{(p,q)},\\Theta),\\textsf{RoPE-2D}(y^{(r,s)},\\Theta)\\rangle=\\langle\\textsf{RoPE-2D}(x^{(p-r,q-s)},\\Theta),\\textsf{RoPE-2D}(y^{(0,0)},\\Theta)\\rangle$\n\nfor any feature $x,y\\in\\mathbb{R}^{d}$ for all positions $p,r\\in\\{0\\dots H\\}$ and $q,s\\in\\{0\\dots W\\}$. To keep the discussion simple, we will illustrate this property for $d=4$ (the extension to higher dimension is straightforward).\n\nRoPE-2D $\\left(x^{(p,q)},\\Theta\\right)$ \\[ =\\begin{pmatrix}\\cos p\\theta_{1}&-\\sin p\\theta_{1}&0&0\\\\\n\\sin p\\theta_{1}&\\cos p\\theta_{1}&0&0\\\\\n0&0&\\cos q\\theta_{2}&-\\sin q\\theta_{2}\\\\\n0&0&\\sin q\\theta_{2}&\\cos q\\theta_{2}\\end{pmatrix}\\cdot\\begin{pmatrix}x_{1}\\\\\nx_{2}\\\\\nx_{3}\\\\\nx_{4}\\end{pmatrix} \\]\n\nRoPE-2D $\\left(y^{(r,s)},\\Theta\\right)$ \\[ =\\begin{pmatrix}\\cos r\\theta_{1}&-\\sin r\\theta_{1}&0&0\\\\\n\\sin r\\theta_{1}&\\cos r\\theta_{1}&0&0\\\\\n0&0&\\cos s\\theta_{2}&-\\sin s\\theta_{2}\\\\\n0&0&\\sin s\\theta_{2}&\\cos s\\theta_{2}\\end{pmatrix}\\cdot\\begin{pmatrix}y_{1}\\\\\ny_{2}\\\\\ny_{3}\\\\\ny_{4}\\end{pmatrix} \\]\n\nNow, we compute\n\n$\\langle\\textsf{RoPE-2D}(x^{(p,q)},\\Theta),\\textsf{RoPE-2D}(y^{(r,s)},\\Theta)\\rangle$\n\n$=$ \\[ (x_{1}\\ x_{2})\\cdot\\begin{pmatrix}\\cos p\\theta_{1}&-\\sin p\\theta_{1}\\\\\n\\sin p\\theta_{1}&\\cos p\\theta_{1}\\end{pmatrix}^{T}\\begin{pmatrix}\\cos r\\theta_{1}&-\\sin r\\theta_{1}\\\\\n\\sin r\\theta_{1}&\\cos r\\theta_{1}\\end{pmatrix}\\cdot\\begin{pmatrix}y_{1}\\\\\ny_{2}\\end{pmatrix} \\]\n\\[ \\quad+(x_{3}\\ x_{4})\\cdot\\begin{pmatrix}\\cos q\\theta_{2}&-\\sin q\\theta_{2}\\\\\n\\sin q\\theta_{2}&\\cos q\\theta_{2}\\end{pmatrix}^{T}\\begin{pmatrix}\\cos s\\theta_{2}&-\\sin s\\theta_{2}\\\\\n\\sin s\\theta_{2}&\\cos s\\theta_{2}\\end{pmatrix}\\cdot\\begin{pmatrix}y_{3}\\\\\ny_{4}\\end{pmatrix} \\]\n$=$ \\[ (x_{1}\\ x_{2})\\cdot\\begin{pmatrix}\\cos p\\theta_{1}\\cos r\\theta_{1}+\\sin p\\theta_{1}\\sin r\\theta_{1}&-\\sin r\\theta_{1}\\cos p\\theta_{1}+\\sin p\\theta_{1}\\cos r\\theta_{1}\\\\\n\\sin r\\theta_{1}\\cos p\\theta_{1}-\\sin p\\theta_{1}\\cos r\\theta_{1}&-\\sin p\\theta_{1}\\cos r\\theta_{1}+\\sin p\\theta_{1}\\sin r\\theta_{1}\\end{pmatrix}^{T}\\cdot\\begin{pmatrix}y_{1}\\\\\ny_{2}\\end{pmatrix} \\]\n\\[ \\quad+(x_{3}\\ x_{4})\\cdot\\begin{pmatrix}\\cos q\\theta_{2}\\cos s\\theta_{2}+\\sin q\\theta_{2}\\sin s\\theta_{2}&-\\sin q\\theta_{2}\\cos s\\theta_{2}+\\sin q\\theta_{2}\\cos s\\theta_{2}\\\\\n\\sin q\\theta_{2}\\cos s\\theta_{2}-\\sin q\\theta_{2}\\cos s\\theta_{2}&-\\cos q\\theta_{2}\\cos s\\theta_{2}+\\sin q\\theta_{2}\\sin s\\theta_{2}\\end{pmatrix}\\cdot\\begin{pmatrix}y_{3}\\\\\ny_{4}\\end{pmatrix} \\]\n$=$ \\[ (x_{1}\\ x_{2})\\cdot\\begin{pmatrix}\\cos\\left((p-r)\\cdot\\theta_{1}\\right)&-\\sin\\left((p-r)\\cdot\\theta_{1}\\right)\\\\\n\\sin\\left((p-r)\\cdot\\theta_{1}\\right)&-\\cos\\left((p-r)\\cdot\\theta_{1}\\right)\\end{pmatrix}^{T}\\cdot\\begin{pmatrix}y_{1}\\\\\ny_{2}\\end{pmatrix} \\]\n\\[ \\quad+(x_{3}\\ x_{4})\\cdot\\begin{pmatrix}\\cos\\left((q-s)\\cdot\\theta_{2}\\right)&-\\sin\\left((q-s)\\cdot\\theta_{2}\\right)\\\\\n\\sin\\left((q-s)\\cdot\\theta_{2}\\right)&-\\cos\\left((q-s)\\cdot\\theta_{2}\\right)\\end{pmatrix}^{T}\\cdot\\begin{pmatrix}y_{3}\\\\\ny_{4}\\end{pmatrix} \\]\n$=$ $\\langle\\textsf{RoPE-2D}(y^{(p-r,q-s)},\\Theta),\\textsf{RoPE-2D}(y^{(0,0)},\\Theta)\\rangle$","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":20,"markdown":"# C Flexible Parsing Settings\n\nIn Section 4.3, we introduce three 'Parsing levels' which evaluate models under progressively looser constraints. While common evaluation metrics reward only exactly the answer format in the ground truth annotation, we seek to relax these requirements and investigate how model performance varies.\n\nBaseline: This setting requires exact following of prompt instructions, with model responses ending in the string \"Final Answer: <answer>\".\n\nFlexible Parsing Level 1: This setting also catches cases where the model ends responses with \"Answer: <answer>\".\n\nFlexible Parsing Level 2: Here we additionally catch cases where the model has added extra markdown formatting. We strip markdown such as: \"**Answer**\", \"**Answer:**\", \"**Answer:**\", \"We find such formatting to be particularly prevalent in Llama-3.2 models [6].\n\nFlexible Parsing Level 3: This is the most generous evaluation setting. Here we mark a response as correct if the ground truth answer appears anywhere in the model's response. For single letter answers, we search the response for \"is <a>\", \"are <a>\", \"<a>\". For single number responses, we search the response for the number both with and without commas.\n\nWe highlight that Flexible Parsing Level 3 is intended to serve as an upper bound, as it may mark incorrect answers as correct.\n\n# D Robustness to prompting\n\n# D.1 Llama-Specific Prompts\n\nIn Section 4.1, we evaluate all models with a common prompt, which allowed us to reproduce the reported figures of GPT-4o [16] and Claude-3.5 Sonnet [1]. This prompt requires models to end responses with \"Final Answer: <answer>\" (see Appendix A for full prompts).\n\nHowever, when evaluating Llama-3.2 models [6], we found that this model family defaults to responding with \"**Answer:** <answer>\" (i.e., with markdown formatting and omission of 'Final', despite the explicit instruction). While the performance degradation due to regex mismatches is mitigated through our flexible parsing strategy (see Section 4.3), we found that Llama-3.2 models performed substantially better when the prompt specifically asks for \"**Answer:** <answer>\" (i.e., respecting its default output format).\n\nIn Table 6, we show the results for models both with the default prompts from Appendix A, and with the Llama-specific prompts (all evaluated under the Exact Match metric). We show that the Llama-specific prompt substantially improves the performance of Llama-3.2 models, particularly for the 11B variant, with over  $15\\%$  jumps on both Mathvista and MMMU. We further note that Pixtral performance is stable across prompts, and leads the 11B variant by a substantial margin.\n\n|   |  | Mathvista | MMMU | ChartQA  |\n| --- | --- | --- | --- | --- |\n|   |  | Exact Match | Exact Match | Exact Match  |\n|  Llama-3.2 11B [6] | Default prompt | 24.3 | 23.0 | 14.8  |\n|   |  Llama-specific prompt | 41.6 | 41.9 | 33.7  |\n|  Llama-3.2 90B [6] | Default prompt | 49.1 | 53.7 | 33.8  |\n|   |  Llama-specific prompt | 57.6 | 58.6 | 34.8  |\n|  Qwen2-VL 7B [23] | Default prompt | 53.7 | 48.1 | 41.2  |\n|   |  Llama-specific prompt | 52.6 | 47.4 | 74.0  |\n|  Pixtral 12B | Default prompt | 58.3 | 52.0 | 81.8  |\n|   |  Llama-specific prompt | 57.7 | 50.8 | 83.8  |\n\nTable 6: Evaluation with Llama-specific prompts. We re-evaluate models with a prompt tailored towards the Llama-3.2 model family [6]. We find that this substantially improves the performance of the 11B variant of the model. Pixtral 12B reports stable performance across both prompts, and maintains a substantial lead over Llama-3.2 11B and Qwen2-VL 7B.</answer></answer></answer></answer></answer></answer></answer></answer></answer>","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":21,"markdown":"# D.2 Average performance across prompts\n\nHere we report average results across a number of prompts. We task Mistral Large v2 with creating 10 versions of the prompt used in the main paper (see Appendix A), with varied wording while keeping instructions explicit. As prior works suffer under stricter parsing constraints, all models are evaluated under 'Flexible Parsing Level 3' for this experiment (see Section 4.3 and Appendix C).\n\nWe find that the trends follow those from the main paper, with Pixtral outperforming models of comparable size, and surpassing Llama-3.2 90B [6] on Mathvista and ChartQA. Pixtral also typically displays lower variance in performance between prompts (shown in gray).\n\n|   | Mathvista | MMMU | ChartQA  |\n| --- | --- | --- | --- |\n|   | Flexible Level 3 | Flexible Level 3 | Flexible Level 3  |\n|  Llama-3.2 11B [6] | 42.1 (±1.9) | 45.3 (±1.0) | 77.2 (±0.8)  |\n|  Llama-3.2 90B [6] | 56.0 (±1.5) | 56.7 (±0.5) | 80.1 (±0.5)  |\n|  Qwen2-VL 7B [23] | 53.7 (±2.1) | 46.9 (±1.9) | 77.0 (±0.8)  |\n|  Pixtral 12B | 56.4 (±1.0) | 49.5 (±1.5) | 83.8 (±0.4)  |\n\nTable 7: Average multimodal performance across prompts. We evaluate models with 10 different prompts, reporting the mean performance, and standard deviations in gray. The trends follow those in the main paper, with Pixtral outperforming open-source models of a comparable size. All models are evaluated with 'Flexible Level 3' parsing (see Section 4.3)\n\n# E Reproducing Reported Numbers\n\nIn Section 4.1 we re-evaluate all models under a common and rigorous protocol. All models are evaluated under the same evaluation metric and with the same prompt, in such a way that frontier models achieve their reported performance.\n\nUnder this common protocol, we found some models substantially underperformed their reported figures. Here, we document the steps required to recover the reported figures of open models, by tuning the evaluation prompt and metric to each model in turn. All results are shown in Table 8.\n\n# E.1 Summary\n\nOur analysis indicates that frontier models, and even smaller closed-source models, are able to recover or exceed their reported figures under the common protocol discussed in Section 4.1. This is achieved through precise following of instructions in the 'Explicit' prompts (see Appendix A).\n\nSmaller, open-source models typically require some degree of prompt tuning and/or adjustment of the evaluation metric, targeted towards the model, to recover reported performance. With such interventions, we are generally able to recover or exceed reported figures.\n\nPixtal 12B, like closed and leading models, is able to follow prompt instructions to report strong performance without targeted interventions. This is substantiated in robust performance across prompts (see Appendix D), as well as strong performance in both LMSys Vision Arena and MM-MT-Bench (see Section 4.1).\n\n# E.2 Closed models: Claude-3 Haiku and Gemini-Flash-8B\n\nWe find we the standardized evaluation protocol roughly matches or exceeds reported figures, with a small gain achieved through flexible parsing. The only exception is for Claude Haiku [1] on ChartQA, where Flexible Parsing Level 3 is required to approach reported performance.\n\n# E.3 Qwen2-VL 7B\n\nWe first simplify the prompt into a one-line instruction, similar to the training set of ChartQA. Next, we provide different prompts depending on the answer format expected. For instance, if the answer is a floating point number, we specify \"Answer with a two decimal place floating point\",","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":22,"markdown":"with analogous prompts for integer and multiple-choice questions. We found that providing a single, unified prompt with all format specifications (as in the prompts in Appendix A) reduces performance.\n\n### E.4 Llama-3.2\n\nWe find that these models default to responses with markdown formatting such as: \"**Answer**\", \"**Answer:**\", \"*Answer: <ANSWER>*\". We find substantial improvement by changing the ‘Explicit’ prompt to request this format (see Appendix D). These models then recover their reported performance after evaluating with Flexible Level 3.\n\nWhen evaluating Llama-3.2 90B on DocVQA , many generations are of the form ‘The answer is <ANSWER>’, which is penalized by the ANLS metric. We strip such prefixes, and this improves DocVQA by $+4.8$.\n\n### E.5 Llava-OneVision 72B\n\nSimilarly to Qwen2-7B *[23]*, we first simplify the prompt into a one-line instruction and provide different prompts depending on the answer format expected. We found that providing a single, unified prompt with all format specifications reduces performance.\n\n### E.6 Molmo\n\nSimilarly to Qwen2-7B *[23]* and Llava-Onevision 7B *[9]*, we first simplify the prompt into a one-line instruction, and provide different prompts depending on the answer format expected. Furthermore, similarly to the intervention for Llama-3.2 *[6]*, we reformat the prompt and relax the evaluation metrics. Molmo models default to ending long responses with \\n\\n<ANSWER>. In long-answer cases, we adjust the evaluation metric to capture this.\n\nFor VQAv2, we apply custom post-processing filters, such as remapping textual output of numerical answers to the integer digits (e.g. Two to 2).\n\n## Appendix","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":23,"markdown":"|   | Mathvista CoT | MMMU CoT | ChartQA CoT | DocVQA 650.5 | VQAv2 VQA Match | MM-MT-Bench GPT-4o Judge | LMSys-Vision (Oct '24)  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n|  Pixtral 12B | 58.3 | 52.0 | 81.8 | 90.7 | 78.6 | 6.05 | 1076  |\n|  Qwen-2-VL 7B [23]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 53.7 | 48.1 | 41.2 | 94.5 | 75.9 | 5.45 |   |\n|  Measured (Custom evaluation, see Section E.3) | 63.7 | 50.6 | 83.4 | 94.5 | 82.1 | - | 1040  |\n|  Reported | 58.2 | 54.1 | 83.0 | 94.5 | - | - |   |\n|  Llama-3.2 11B [6]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 24.3 | 23.0 | 14.8 | 91.1 | 67.1 | 4.79 |   |\n|  Measured (Custom evaluation, see Section E.4) | 47.9 | 46.6 | 78.5 | 91.1 | 67.1 | - | 1032  |\n|  Reported | 51.5 | 50.7 | 83.4 | 88.4 | 75.2 | - |   |\n|  Molmo-D 7B [4]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 12.3 | 24.3 | 27.0 | 72.2 | 57.1 | 3.72 |   |\n|  Measured (Custom evaluation, see Section E.6) | 43.2 | 47.0 | 76.7 | 72.2 | 70.0 | - | -  |\n|  Reported | 51.6 | 45.3 | 84.1 | 92.2 | 85.6 | - |   |\n|  LLaVA-OneVision 7B [9]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 36.1 | 45.1 | 67.2 | 90.5 | 78.4 | 4.12 |   |\n|  Measured (Custom evaluation, see Section E.5) | 63.1 | 48.1 | 80.2 | 90.5 | 83.7 | - | -  |\n|  Reported | 63.2 | 48.8 | 80.0 | 87.5 | - | - |   |\n|  Molmo 72B [4] |   |   |   |   |   |   | -  |\n|  Measured (Exact Match) | 52.2 | 52.7 | 75.6 | 86.5 | 75.2 | 3.51 |   |\n|  Measured (Custom evaluation, see Section E.6) | 61.3 | 52.9 | 82.3 | 86.5 | 75.5 | - | -  |\n|  Reported | 58.6 | 54.1 | 87.3 | 93.5 | 86.5 | - |   |\n|  Llama-3.2 90B [6]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 49.1 | 53.7 | 33.8 | 85.7 | 67.0 | 5.50 |   |\n|  Measured (Custom evaluation, see Section E.4) | 57.5 | 60.2 | 91.7 | 91.5 | 67.0 | - | 1071  |\n|  Reported | 57.3 | 60.3 | 85.5 | 90.1 | 78.1 | - |   |\n|  Claude-3 Haiku [1]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 44.8 | 50.4 | 69.6 | 74.6 | 68.4 | 5.46 |   |\n|  Measured (Custom evaluation, see Section E.2) | 44.8 | 51.3 | 79.8 | 74.6 | 68.4 | - | 1000  |\n|  Reported | 46.4 | 50.2 | 81.7 | 88.8 | - | - |   |\n|  Gemini-1.5-Flash 8B[27] [18]  |   |   |   |   |   |   |   |\n|  Measured (Exact Match) | 56.9 | 50.7 | 78.0 | 79.5 | 65.5 | 5.93 |   |\n|  Measured (Custom evaluation, see Section E.2) | 57.1 | 50.7 | 78.2 | 79.5 | 69.2 | - | 1111  |\n|  Reported | - | 50.3 | - | 73.6 | - | - |   |\n\nTable 8: Reproducing the reported performance of prior models. In Table 2 we conduct fair re-evaluation of all models through the same evaluation harness, with the same prompt and metric. Here, we endeavour to recover the reported performance of all models by tuning evaluation settings towards individual models. We highlight that Pixtral 12B, like strong closed-source models (e.g. Gemini-1.5-Flash 8B [18] and Claude-3 Haiku [1]) is able reports strong performance without such interventions.","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}}],"model":"mistral-ocr-latest","usage_info":{"pages_processed":24,"doc_size_bytes":12640953},"document_annotation":null}},"error":null}
{"id":"batch-2b6be3bc-2-67b9acae-e7a5-4a67-937f-d2432157295e","custom_id":"1","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"PLACE FACE UP ON DASH\nCITY OF PALO ALTO\nNOT VALID FOR\nONSTREET PARKING\n\nExpiration Date/Time\n11:59 PM\nAUG 19, 2024\n\nPurchase Date/Time: 01:34pm Aug 19, 2024\nTotal Due: $15.00\nTotal Paid: $15.00\nTicket #: 00005883\nS/N #: 520117260957\nSetting: Permit Machines\nMach Name: Civic Center\n\nRate: Daily Parking\nPmt Type: CC (Swipe)\n\n#*** -1224, Visa\nDISPLAY FACE UP ON DASH\nPERMIT EXPIRES\nAT MIDNIGHT","images":[],"dimensions":{"dpi":200,"height":3210,"width":1806}}],"model":"mistral-ocr-latest","usage_info":{"pages_processed":1,"doc_size_bytes":3110191},"document_annotation":"{\n  \"language\": \"English\",\n  \"urls\": \"\"\n}"}},"error":null}
{"id":"batch-2b6be3bc-3-cc32d644-1ed5-4a31-b847-0850a682ce07","custom_id":"2","response":{"status_code":200,"body":{"pages":[{"index":0,"markdown":"# Mistral 7B\n\nAlbert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed\n\n![img-0.jpeg](img-0.jpeg)\n\n# Abstract\n\nWe introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmarks, and the best released 34B model (Llama 1) in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B - Instruct, that surpasses Llama 2 13B - chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.\n\nCode: https://github.com/mistralai/mistral-src\n\nWebpage: https://mistral.ai/news/announcing-mistral-7b/\n\n# 1 Introduction\n\nIn the rapidly evolving domain of Natural Language Processing (NLP), the race towards higher model performance often necessitates an escalation in model size. However, this scaling tends to increase computational costs and inference latency, thereby raising barriers to deployment in practical, real-world scenarios. In this context, the search for balanced models delivering both high-level performance and efficiency becomes critically essential. Our model, Mistral 7B, demonstrates that a carefully designed language model can deliver high performance while maintaining an efficient inference. Mistral 7B outperforms the previous best 13B model (Llama 2, [26]) across all tested benchmarks, and surpasses the best 34B model (LLaMa 34B, [25]) in mathematics and code generation. Furthermore, Mistral 7B approaches the coding performance of Code-Llama 7B [20], without sacrificing performance on non-code related benchmarks.\n\nMistral 7B leverages grouped-query attention (GQA) [1], and sliding window attention (SWA) [6, 3]. GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding, allowing for higher batch sizes hence higher throughput, a crucial factor for real-time applications. In addition, SWA is designed to handle longer sequences more effectively at a reduced computational cost, thereby alleviating a common limitation in LLMs. These attention mechanisms collectively contribute to the enhanced performance and efficiency of Mistral 7B.","images":[{"id":"img-0.jpeg","top_left_x":426,"top_left_y":602,"bottom_right_x":1276,"bottom_right_y":893,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Image\",\n  \"short_description\": \"Mistral AI logo\",\n  \"summary\": \"The image displays the logo of Mistral AI, a prominent artificial intelligence company. The logo features the text 'Mistral AI' in a bold, three-dimensional font with a gradient color scheme transitioning from orange to yellow.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":1,"markdown":"Mistral 7B is released under the Apache 2.0 license. This release is accompanied by a reference implementation $^{1}$  facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot $^{2}$ . Integration with Hugging Face $^{3}$  is also streamlined for easier integration. Moreover, Mistral 7B is crafted for ease of fine-tuning across a myriad of tasks. As a demonstration of its adaptability and superior performance, we present a chat model fine-tuned from Mistral 7B that significantly outperforms the Llama 2 13B - Chat model.\n\nMistral 7B takes a significant step in balancing the goals of getting high performance while keeping large language models efficient. Through our work, our aim is to help the community create more affordable, efficient, and high-performing language models that can be used in a wide range of real-world applications.\n\n# 2 Architectural details\n\n![img-1.jpeg](img-1.jpeg)\nFigure 1: Sliding Window Attention. The number of operations in vanilla attention is quadratic in the sequence length, and the memory increases linearly with the number of tokens. At inference time, this incurs higher latency and smaller throughput due to reduced cache availability. To alleviate this issue, we use sliding window attention: each token can attend to at most  $W$  tokens from the previous layer (here,  $W = 3$ ). Note that tokens outside the sliding window still influence next word prediction. At each attention layer, information can move forward by  $W$  tokens. Hence, after  $k$  attention layers, information can move forward by up to  $k \\times W$  tokens.\n\n![img-2.jpeg](img-2.jpeg)\n\n![img-3.jpeg](img-3.jpeg)\n\nMistral 7B is based on a transformer architecture [27]. The main parameters of the architecture are summarized in Table 1. Compared to Llama, it introduces a few changes that we summarize below.\n\nSliding Window Attention. SWA exploits the stacked layers of a transformer to attend information beyond the window size  $W$ . The hidden state in position  $i$  of the layer  $k$ ,  $h_i$ , attends to all hidden states from the previous layer with positions between  $i - W$  and  $i$ . Recursively,  $h_i$  can access tokens from the input layer at a distance of up to  $W \\times k$  tokens, as illustrated in Figure 1. At the last layer, using a window size of  $W = 4096$ , we have a theoretical attention span of approximately  $131K$  tokens. In practice, for a sequence length of  $16K$  and  $W = 4096$ , changes made to FlashAttention [11] and xFormers [18] yield a 2x speed improvement over a vanilla attention baseline.\n\n|  Parameter | Value  |\n| --- | --- |\n|  dim | 4096  |\n|  n_layers | 32  |\n|  head_dim | 128  |\n|  hidden_dim | 14336  |\n|  n_heads | 32  |\n|  n_kv_heads | 8  |\n|  window_size | 4096  |\n|  context_len | 8192  |\n|  vocab_size | 32000  |\n\nTable 1: Model architecture.\n\nRolling Buffer Cache. A fixed attention span means that we can limit our cache size using a rolling buffer cache. The cache has a fixed size of  $W$ , and the keys and values for the timestep  $i$  are stored in position  $i \\mod W$  of the cache. As a result, when the position  $i$  is larger than  $W$ , past values in the cache are overwritten, and the size of the cache stops increasing. We provide an illustration in Figure 2 for  $W = 3$ . On a sequence length of 32k tokens, this reduces the cache memory usage by 8x, without impacting the model quality.","images":[{"id":"img-1.jpeg","top_left_x":302,"top_left_y":644,"bottom_right_x":683,"bottom_right_y":1056,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Diagram\",\n  \"short_description\": \"This image shows a matrix representing 'Vanilla Attention' in a sequence of words.\",\n  \"summary\": \"The image depicts a matrix used to illustrate the concept of 'Vanilla Attention' in natural language processing. The matrix is a 5x5 grid corresponding to the sequence of words: 'The', 'cat', 'sat', 'on', 'the'. Each cell in the matrix indicates the attention score between pairs of words in the sequence. The diagonal cells have a value of 1, indicating full attention to the same word, while the upper triangular section (above the diagonal) is filled with zeros, indicating no attention to future words. The lower triangular section (below the diagonal) is filled with ones, indicating full attention to all previous words. This visualization helps in understanding how attention mechanisms work in processing sequences of words, where each word's representation is influenced by the previous words in the sequence.\"\n}"},{"id":"img-2.jpeg","top_left_x":724,"top_left_y":644,"bottom_right_x":1040,"bottom_right_y":1056,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Diagram\",\n  \"short_description\": \"Sliding Window Attention\",\n  \"summary\": \"The image depicts a matrix used in the context of Sliding Window Attention, a technique in natural language processing. The matrix shows attention weights between words in a sentence, with the sentence 'The cat sat on the' displayed above the matrix. The matrix is filled with binary values (0 and 1), indicating the presence or absence of attention between pairs of words. The diagonal pattern suggests that each word attends to a certain window of previous words, which is a characteristic of Sliding Window Attention mechanisms.\"\n}"},{"id":"img-3.jpeg","top_left_x":1057,"top_left_y":644,"bottom_right_x":1387,"bottom_right_y":1056,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"diagram\",\n  \"short_description\": \"Illustration of effective context length in a multi-layer transformer model.\",\n  \"summary\": \"The image depicts a multi-layer transformer model where each layer processes tokens within a specific window size. The effective context length varies across layers, with the window size decreasing as the layers progress from bottom to top. This indicates that the model's ability to capture context diminishes with each subsequent layer, potentially impacting the model's performance in understanding long-range dependencies.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":2,"markdown":"![img-4.jpeg](img-4.jpeg)\nFigure 2: Rolling buffer cache. The cache has a fixed size of  $W = 4$ . Keys and values for position  $i$  are stored in position  $i \\mod W$  of the cache. When the position  $i$  is larger than  $W$ , past values in the cache are overwritten. The hidden state corresponding to the latest generated tokens are colored in orange.\n\nPre-fill and Chunking. When generating a sequence, we need to predict tokens one-by-one, as each token is conditioned on the previous ones. However, the prompt is known in advance, and we can pre-fill the  $(k,v)$  cache with the prompt. If the prompt is very large, we can chunk it into smaller pieces, and pre-fill the cache with each chunk. For this purpose, we can select the window size as our chunk size. For each chunk, we thus need to compute the attention over the cache and over the chunk. Figure 3 shows how the attention mask works over both the cache and the chunk.\n\n![img-5.jpeg](img-5.jpeg)\nFigure 3: Pre-fill and chunking. During pre-fill of the cache, long sequences are chunked to limit memory usage. We process a sequence in three chunks, \"The cat sat on\", \"the mat and saw\", \"the dog go to\". The figure shows what happens for the third chunk (\"the dog go to\"): it attends itself using a causal mask (rightmost block), attends the cache using a sliding window (center block), and does not attend to past tokens as they are outside of the sliding window (left block).\n\n# 3 Results\n\nWe compare Mistral 7B to Llama, and re-run all benchmarks with our own evaluation pipeline for fair comparison. We measure performance on a wide variety of tasks categorized as follow:\n\n- Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24]\n- World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15]\n- Reading Comprehension (0-shot): BoolQ [8], QuAC [7]\n- Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4\n- Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot)\n- Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only)\n\nDetailed results for Mistral 7B, Llama 2 7B/13B, and Code-Llama 7B are reported in Table 2. Figure 4 compares the performance of Mistral 7B with Llama 2 7B/13B, and Llama  $134\\mathrm{B}^4$  in different categories. Mistral 7B surpasses Llama 2 13B across all metrics, and outperforms Llama 1 34B on most benchmarks. In particular, Mistral 7B displays a superior performance in code, mathematics, and reasoning benchmarks.","images":[{"id":"img-4.jpeg","top_left_x":294,"top_left_y":200,"bottom_right_x":1400,"bottom_right_y":374,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"image\",\n  \"short_description\": \"Image showing the Mistral model's tokenization process over three timesteps.\",\n  \"summary\": \"The image illustrates the tokenization process of the Mistral model over three consecutive timesteps. Each row represents a different sentence being tokenized. At each timestep, the model processes a portion of the sentence, highlighting the tokens in red as they are added to the sequence. The first row shows the tokenization of 'This is an example of ...', the second row shows 'Mistral is a good ...', and the third row shows 'The cat sat on the mat ...'. The progression over the timesteps demonstrates how the model incrementally builds the token sequence.\"\n}"},{"id":"img-5.jpeg","top_left_x":465,"top_left_y":748,"bottom_right_x":1232,"bottom_right_y":1056,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Matrix Representation\",\n  \"short_description\": \"A matrix showing the presence of words in different contexts.\",\n  \"summary\": \"This matrix represents the presence (1) or absence (0) of specific words ('the', 'dog', 'go', 'to') in different contexts: past, cache, and current. The rows correspond to the words, and the columns correspond to the words in the sentence 'The cat sat on the mat and saw the dog'. The matrix is divided into three sections: past, cache, and current, showing how the presence of words changes over these contexts.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":3,"markdown":"![img-6.jpeg](img-6.jpeg)\nFigure 4: Performance of Mistral 7B and different Llama models on a wide range of benchmarks. All models were re-evaluated on all metrics with our evaluation pipeline for accurate comparison. Mistral 7B significantly outperforms Llama 2 7B and Llama 2 13B on all benchmarks. It is also vastly superior to Llama 1 34B in mathematics, code generation, and reasoning benchmarks.\n\n![img-7.jpeg](img-7.jpeg)\n\n|  Model | Modality | MMLU | HellaSwag | WinoG | PIQA | Arc-e | Arc-c | NQ | TriviaQA | HumanEval | MBPP | MATH | GSM8K  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  LLaMA 2 7B | Pretrained | 44.4% | 77.1% | 69.5% | 77.9% | 68.7% | 43.2% | 24.7% | 63.8% | 11.6% | 26.1% | 3.9% | 16.0%  |\n|  LLaMA 2 13B | Pretrained | 55.6% | 80.7% | 72.9% | 80.8% | 75.2% | 48.8% | 29.0% | 69.6% | 18.9% | 35.4% | 6.0% | 34.3%  |\n|  Code-Llama 7B | Finetuned | 36.9% | 62.9% | 62.3% | 72.8% | 59.4% | 34.5% | 11.0% | 34.9% | 31.1% | 52.5% | 5.2% | 20.8%  |\n|  Mistral 7B | Pretrained | 60.1% | 81.3% | 75.3% | 83.0% | 80.0% | 55.5% | 28.8% | 69.9% | 30.5% | 47.5% | 13.1% | 52.2%  |\n\nSize and Efficiency. We computed \"equivalent model sizes\" of the Llama 2 family, aiming to understand Mistral 7B models' efficiency in the cost-performance spectrum (see Figure 5). When evaluated on reasoning, comprehension, and STEM reasoning (specifically MMLU), Mistral 7B mirrored performance that one might expect from a Llama 2 model with more than  $3\\mathrm{x}$  its size. On the Knowledge benchmarks, Mistral 7B's performance achieves a lower compression rate of  $1.9\\mathrm{x}$ , which is likely due to its limited parameter count that restricts the amount of knowledge it can store.\n\nEvaluation Differences. On some benchmarks, there are some differences between our evaluation protocol and the one reported in the Llama 2 paper: 1) on MBPP, we use the hand-verified subset 2) on TriviaQA, we do not provide Wikipedia contexts.\n\n# 4 Instruction Finetuning\n\nTo evaluate the generalization capabilities of Mistral 7B, we fine-tuned it on instruction datasets publicly available on the Hugging Face repository. No proprietary data or training tricks were utilized: Mistral 7B - Instruct model is a simple and preliminary demonstration that the base model can easily be fine-tuned to achieve good performance. In Table 3, we observe that the resulting model, Mistral 7B - Instruct, exhibits superior performance compared to all 7B models on MT-Bench, and is comparable to 13B - Chat models. An independent human evaluation was conducted on https://1lmboxing.com/leaderboard.\n\nTable 2: Comparison of Mistral 7B with Llama. Mistral 7B outperforms Llama 2 13B on all metrics, and approaches the code performance of Code-Llama 7B without sacrificing performance on non-code benchmarks.\n\n|  Model | Chatbot Arena ELO Rating | MT Bench  |\n| --- | --- | --- |\n|  WizardLM 13B v1.2 | 1047 | 7.2  |\n|  Mistral 7B Instruct | 1031 | 6.84 +/- 0.07  |\n|  Llama 2 13B Chat | 1012 | 6.65  |\n|  Vicuna 13B | 1041 | 6.57  |\n|  Llama 2 7B Chat | 985 | 6.27  |\n|  Vicuna 7B | 997 | 6.17  |\n|  Alpaca 13B | 914 | 4.53  |\n\nTable 3: Comparison of Chat models. Mistral 7B - Instruct outperforms all 7B models on MT-Bench, and is comparable to 13B - Chat models.\n\nIn this evaluation, participants were provided with a set of questions along with anonymous responses from two models and were asked to select their preferred response, as illustrated in Figure 6. As of October 6, 2023, the outputs generated by Mistral 7B were preferred 5020 times, compared to 4143 times for Llama 2 13B.","images":[{"id":"img-6.jpeg","top_left_x":299,"top_left_y":213,"bottom_right_x":836,"bottom_right_y":550,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Bar Chart\",\n  \"short_description\": \"This bar chart compares the performance of different language models (Mistral 7B, LLaMA 2 13B, LLaMA 2 7B, and LLaMA 1 34B) across four categories: MMLU, Knowledge, Reasoning, and Comprehension.\",\n  \"summary\": \"The chart shows the accuracy percentages of various language models in different categories. Mistral 7B performs best in MMLU and Comprehension, while LLaMA 1 34B excels in Reasoning. LLaMA 2 13B shows strong performance across all categories but does not lead in any specific category. LLaMA 2 7B generally performs the least well across all categories. The data suggests that different models have varying strengths, with Mistral 7B and LLaMA 1 34B showing particularly high performance in specific areas.\"\n}"},{"id":"img-7.jpeg","top_left_x":858,"top_left_y":215,"bottom_right_x":1397,"bottom_right_y":550,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"bar chart\",\n  \"short_description\": \"This bar chart compares the accuracy percentages of different models (Mistral 7B, LLaMA 2 13B, LLaMA 2 7B, and LLaMA 1 34B) across four categories: AGI Eval, Math, BBH, and Code.\",\n  \"summary\": \"The bar chart illustrates the performance of various models in terms of accuracy across different tasks. Mistral 7B shows the highest accuracy in the AGI Eval category, while LLaMA 1 34B performs best in the Math category. In the BBH category, LLaMA 1 34B again leads, followed closely by Mistral 7B. For the Code category, Mistral 7B and LLaMA 1 34B show similar high accuracy, with LLaMA 2 13B and LLaMA 2 7B trailing behind. The chart highlights the strengths and weaknesses of each model in different tasks, providing insights into their relative performance.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":4,"markdown":"![img-8.jpeg](img-8.jpeg)\n\n![img-9.jpeg](img-9.jpeg)\n\n![img-10.jpeg](img-10.jpeg)\nFigure 5: Results on MMLU, commonsense reasoning, world knowledge and reading comprehension for Mistral 7B and Llama 2 (7B/I3B/70B). Mistral 7B largely outperforms Llama 2 13B on all evaluations, except on knowledge benchmarks, where it is on par (this is likely due to its limited parameter count, which limits the amount of knowledge it can compress).\n\n![img-11.jpeg](img-11.jpeg)\n\n# 5 Adding guardrails for front-facing applications\n\nThe ability to enforce guardrails when it comes to AI generation is important for front-facing applications. In this section, we highlight how to leverage system prompting to optionally enforce output constraints on top of our models. Additionally, we showcase the ability of Mistral 7B to perform fine-grained content moderation, which can be useful to enforce quality content in applications.\n\n# 5.1 System prompt to enforce guardrails\n\nWe introduce a system prompt (see below) to guide the model to generate answers within specified guardrails, similar to the work done with Llama 2. Using this prompt allows the user to move on the Pareto front of model utility / guardrails enforcement, as indicated in Table 4.\n\nAlways assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity.\n\nWe use a set of 175 unsafe prompts for evaluating safety. With the recommended system prompt, the model properly declines to answer  $100\\%$  of the harmful questions.\n\nAs an illustration, we provide in Table 5 the answers of both Mistral 7B - Instruct and Llama 2 Chat 13B to the question How to kill a linux process with system prompts activated. We observe that Mistral 7B provides a correct response while Llama 2 declines to answer. Note that on this specific question, both models answer correctly when system prompts are deactivated.\n\n|  Guardrails | MT Bench  |\n| --- | --- |\n|  No system prompt | 6.84 ± 0.07  |\n|  Llama 2 system prompt | 6.38 ± 0.07  |\n|  Mistral system prompt | 6.58 ± 0.05  |\n\nTable 4: System prompts. Mean official MT Bench score over 10 iterations with standard deviation for Mistral 7B - Instruct. For reference, Llama 2 13B - Chat reports official results of 6.65.\n\n# 5.2 Content moderation with self-reflection\n\nMistral 7B – Instruct can be used as a content moderator: the model itself is able to accurately classify a user prompt or its generated answer as being either acceptable or falling into one of the following categories: Illegal activities such as terrorism, child abuse or fraud; Hateful, harassing or violent content such as discrimination, self-harm or bullying; Unqualified advice for instance in legal, medical or financial domains.","images":[{"id":"img-8.jpeg","top_left_x":465,"top_left_y":202,"bottom_right_x":838,"bottom_right_y":468,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"Comparison of MMLU (%) against Model size (billion parameters) for LLaMA 2 and Mistral.\",\n  \"summary\": \"The chart compares the performance of LLaMA 2 and Mistral models in terms of MMLU (Massive Multitask Language Understanding) percentage against the model size in billion parameters. LLaMA 2 is represented by a red line with circular markers, while Mistral is represented by an orange square marker. The x-axis represents the model size in billion parameters, ranging from 7 to 70 billion. The y-axis represents the MMLU percentage, ranging from 45% to 70%. The chart shows that as the model size increases, the MMLU percentage also increases for LLaMA 2. Mistral is shown at a model size of approximately 7 billion parameters with an MMLU percentage slightly above 60%. A notable point on the chart indicates an effective LLaMA 2 size of 23B (3.3x), which corresponds to a model size of around 34 billion parameters and an MMLU percentage of approximately 65%. This suggests that larger model sizes generally lead to better performance in terms of MMLU percentage for LLaMA 2, and Mistral performs competitively at a smaller model size.\"\n}"},{"id":"img-9.jpeg","top_left_x":851,"top_left_y":204,"bottom_right_x":1225,"bottom_right_y":468,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"This chart compares the reasoning percentages of two AI models, LLaMA 2 and Mistral, based on their model sizes in billions of parameters.\",\n  \"summary\": \"The chart shows a positive correlation between the model size and reasoning percentage for the LLaMA 2 model, with reasoning percentages increasing from approximately 64% to 72% as the model size increases from 7 billion to 70 billion parameters. The Mistral model is represented by a single data point at 13 billion parameters with a reasoning percentage slightly above 68%. The chart also highlights an effective LLaMA 2 model size of 38 billion parameters, which is noted to be 5.4 times larger than the base model size of 7 billion parameters.\"\n}"},{"id":"img-10.jpeg","top_left_x":465,"top_left_y":468,"bottom_right_x":838,"bottom_right_y":726,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"Comparison of knowledge percentage against model size for LLaMA 2 and Mistral models.\",\n  \"summary\": \"The chart compares the knowledge percentage of LLaMA 2 and Mistral models against their respective model sizes in billion parameters. LLaMA 2 shows a linear increase in knowledge percentage as the model size increases from 7 billion to 70 billion parameters. Mistral models are represented by two data points, both showing lower knowledge percentages compared to LLaMA 2 at similar model sizes. The chart also highlights an 'Effective LLaMA 13B (1.9x)' point, indicating a specific comparison or adjustment in the evaluation of the LLaMA 13B model.\"\n}"},{"id":"img-11.jpeg","top_left_x":851,"top_left_y":468,"bottom_right_x":1225,"bottom_right_y":726,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"Chart\",\n  \"short_description\": \"A line chart comparing the comprehension percentage of two language models, LLaMA 2 and Mistral, against their model sizes in billion parameters.\",\n  \"summary\": \"The chart illustrates that as the model size increases, the comprehension percentage also increases for the LLaMA 2 model. The Mistral model is represented by a single data point with a lower comprehension percentage compared to LLaMA 2 at a similar model size. The chart highlights that LLaMA 2 shows significant improvement in comprehension with larger model sizes, and it also indicates an 'Effective LLaMA size 21B (3x)' which suggests a notable increase in comprehension efficiency.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":5,"markdown":"|  Model | Answer  |\n| --- | --- |\n|  Mistral 7B – Instruct with Mistral system prompt | To kill a Linux process, you can use the `kill` command followed by the process ID (PID) of the process you want to terminate. For example, to kill process with PID 1234, you would run the command `kill 1234`. It's important to note that killing a process can have unintended consequences, so it's generally a good idea to only kill processes that you are certain you want to terminate. Additionally, it's generally a good idea to use the `kill` command with caution, as it can be used to terminate critical system processes that are necessary for the operation of the system.  |\n|  Llama 2 13B – Chat with Llama 2 system prompt | As a helpful and respectful assistant, I cannot provide information on how to kill a Linux process. It is not appropriate or ethical to intentionally harm or terminate processes, as it can cause damage to the system and potentially result in data loss. Additionally, it is important to respect the autonomy and security of other users' processes, as they may be critical to their work or personal activities. Instead, I suggest that you explore other options to address any issues or concerns you may have.  |\n\nTable 5: Comparison between Mistral and Llama system prompts. Mistral provides the right answer whereas Llama 2 declines to answer to the question How to kill a linux process.\n\nTo do so, we designed a self-reflection prompt that makes Mistral 7B classify a prompt or a generated answer. We evaluated self-reflection on our manually curated and balanced dataset of adversarial and standard prompts and got a precision of  $99.4\\%$  for a recall of  $95.6\\%$  (considering acceptable prompts as positives).\n\nThe use cases are vast, from moderating comments on social media or forums to brand monitoring on the internet. In particular, the end user is able to select afterwards which categories to effectively filter based on their particular use-case.\n\n# 6 Conclusion\n\nOur work on Mistral 7B demonstrates that language models may compress knowledge more than what was previously thought. This opens up interesting perspectives: the field has so far put the emphasis on scaling laws in 2 dimensions (directly associating model capabilities to training cost, as in [14]); the problem is rather 3 dimensional (model capabilities, training cost, inference cost), and much remains to be explored to obtain the best performance with the smallest possible model.\n\n# Acknowledgements\n\nWe are grateful to CoreWeave for their 24/7 help in marshalling our cluster. We thank the CINECA/EuroHPC team, and in particular the operators of Leonardo, for their resources and help. We thank the maintainers of FlashAttention, vLLM, xFormers, Skypilot for their precious assistance in implementing new features and integrating their solutions into ours. A huge thanks to Tri Dao and Daniel Haziza for helping include Mistral related changes to FlashAttention and xFormers on a tight schedule. We thank the teams of Hugging Face, AWS, GCP, Azure ML for their intense help in making our model compatible everywhere.","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":6,"markdown":"\\text{一}\nLlama 2 13b\n\n1 Health Remaining\n\n\\odot\nMistral 7b\n\n5 Health Remaining\n\n# Can you recommend a foundational and accessible book about quantum physics?\n\nCopy Link to Question\n\nNext Round (Enter) →\n\nRound Winner: mistral-7b-instruct-v0.1\n\n![img-12.jpeg](img-12.jpeg)\nFigure 6: Human evaluation of Mistral 7B – Instruct vs Llama 2 13B – Chat Example. An example of human evaluation from llmboxing.com. The question asks for recommendations of books in quantum physics. Llama 2 13B – Chat recommends a general physics book, while Mistral 7B – Instruct recommends a more relevant book on quantum physics and describes in the contents in more detail.\n\nmistral-7b-instruct-v0.1 Round Winner\n\nA great book for a foundational understanding of quantum physics is \"The Quantum Universe: Everything that Can Happen Does Happen\" by Brian Cox and Jeff Forshaw. This book is accessible and easy to understand, making it a great choice for beginners. It covers all the basics of quantum mechanics, including superposition, entanglement, and the uncertainty principle. The authors also explore the implications of quantum physics for our understanding of the universe, including the nature of time and space. Overall, \"The Quantum Universe\" is an excellent choice for anyone looking to gain a foundational understanding of this fascinating field\n\n# llama-2-13b-chat\n\nCertainly! For a foundational and accessible introduction to quantum physics, I highly recommend \"The Feynman Lectures on Physics\" by Richard Feynman. This classic text provides a clear and engaging overview of the principles and concepts of quantum mechanics, with minimal mathematical formalism and plenty of illustrations and examples. It's a great starting point for anyone looking to gain a deeper understanding of quantum physics without getting bogged down in too much technical detail.","images":[{"id":"img-12.jpeg","top_left_x":736,"top_left_y":803,"bottom_right_x":975,"bottom_right_y":1040,"image_base64":null,"image_annotation":"{\n  \"document_type\": \"image\",\n  \"short_description\": \"A 3D rendering of a letter 'M' with boxing gloves.\",\n  \"summary\": \"The image features a stylized, three-dimensional letter 'M' that appears to be personified with boxing gloves. The letter 'M' is rendered in a vibrant orange color, while the boxing gloves are depicted in a contrasting red. The gloves are positioned as if the letter is ready to box, adding a dynamic and playful element to the design. The background is dark, which makes the colors of the letter and gloves stand out prominently.\"\n}"}],"dimensions":{"dpi":200,"height":2200,"width":1700}},{"index":7,"markdown":"References\n\n- [1] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.\n- [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.\n- [3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n- [4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.\n- [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.\n- [6] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n- [7] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036, 2018.\n- [8] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.\n- [9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.\n- [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n- [11] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.\n- [12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n- [13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.\n- [14] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, 2022.\n- [15] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n- [16] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n-","images":[],"dimensions":{"dpi":200,"height":2200,"width":1700}}],"model":"mistral-ocr-latest","usage_info":{"pages_processed":8,"doc_size_bytes":3749788},"document_annotation":"{\"language\": \"en\", \"urls\": \"https://mistral.ai/news/announcing-mistral-7b/ https://github.com/mistralai/mistral-src\"}"}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="ocr-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with OCR:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "document": { # The document to be processed
            "image_url": "data:image/jpeg;base64,<base64_image>" # Here, we are using a base64 encoded image, but you can also use a URL to a file, and document_url for pdf or other kinds of documents
        },
        "document_annotation_format": {...}, # The document annotation schema, see our Annotations documentation for more
        "bbox_annotation_format": {...}, # The bbox annotation schema, see our Annotations documentation for more
        "pages": [0,1,2,3,4,5,6,7] # The pages to annotate
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-2b6be3bc-3-cc32d644-1ed5-4a31-b847-0850a682ce07", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "pages": [
        {
          "index": 0,
          "markdown": "# Mistral 7B\n\nAlbert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed\n\n![img-0.jpeg](img-0.jpeg)\n\n# Abstract\n\nWe introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmarks, and the best released 34B model (Llama 1) in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B - Instruct, that surpasses Llama 2 13B - chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.\n\nCode: https://github.com/mistralai/mistral-src\n\nWebpage: https://mistral.ai/news/announcing-mistral-7b/\n\n# 1 Introduction\n\nIn the rapidly evolving domain of Natural Language Processing (NLP), the race towards higher model performance often necessitates an escalation in model size. However, this scaling tends to increase computational costs and inference latency, thereby raising barriers to deployment in practical, real-world scenarios. In this context, the search for balanced models delivering both high-level performance and efficiency becomes critically essential. Our model, Mistral 7B, demonstrates that a carefully designed language model can deliver high performance while maintaining an efficient inference. Mistral 7B outperforms the previous best 13B model (Llama 2, [26]) across all tested benchmarks, and surpasses the best 34B model (LLaMa 34B, [25]) in mathematics and code generation. Furthermore, Mistral 7B approaches the coding performance of Code-Llama 7B [20], without sacrificing performance on non-code related benchmarks.\n\nMistral 7B leverages grouped-query attention (GQA) [1], and sliding window attention (SWA) [6, 3]. GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding, allowing for higher batch sizes hence higher throughput, a crucial factor for real-time applications. In addition, SWA is designed to handle longer sequences more effectively at a reduced computational cost, thereby alleviating a common limitation in LLMs. These attention mechanisms collectively contribute to the enhanced performance and efficiency of Mistral 7B.",
          "images": [
            {
              "id": "img-0.jpeg",
              "top_left_x": 426,
              "top_left_y": 602,
              "bottom_right_x": 1276,
              "bottom_right_y": 893,
              "image_base64": null,
              "image_annotation": "{\n  \"document_type\": \"Image\",\n  \"short_description\": \"Mistral AI logo\",\n  \"summary\": \"The image displays the logo of Mistral AI, a prominent artificial intelligence company. The logo features the text 'Mistral AI' in a bold, three-dimensional font with a gradient color scheme transitioning from orange to yellow.\"\n}"
            }
          ],
          "dimensions": {
            "dpi": 200,
            "height": 2200,
            "width": 1700
          }
        },
        ...
        {
          "index": 7,
          "markdown": "References\n\n- [1] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.\n- [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.\n- [3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n- [4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.\n- [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.\n- [6] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n- [7] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036, 2018.\n- [8] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.\n- [9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.\n- [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n- [11] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.\n- [12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n- [13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.\n- [14] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, 2022.\n- [15] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n- [16] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n-",
          "images": [],
          "dimensions": {
            "dpi": 200,
            "height": 2200,
            "width": 1700
          }
        }
      ],
      "model": "mistral-ocr-latest",
      "usage_info": {
        "pages_processed": 8,
        "doc_size_bytes": 3749788
      },
      "document_annotation": "{\"language\": \"en\", \"urls\": \"https://mistral.ai/news/announcing-mistral-7b/ https://github.com/mistralai/mistral-src\"}"
    }
  },
  "error": null
}
```
    </TabItem>
</Tabs>

For more information regarding OCR, visit the [Document AI - Annotations](/studio/document-processing/overview) docs and the corresponding [API Spec](../api/endpoint/ocr#operation-ocr_v1_ocr_post).
  </ExplorerTab>
  <ExplorerTab value="transcriptions" label="Transcriptions">
    A batch file for transcriptions, with `v1/audio/transcriptions` as the Endpoint and `voxtral-mini-latest` as the Model, would look like the following:

<Tabs groupId="code">
    <TabItem value="file" label="Batch File" default>

```bash
{"custom_id": "0", "body": {"file_url":"https://docs.mistral.ai/audio/obama.mp3"}}
{"custom_id": "1", "body": {"file_url":"https://docs.mistral.ai/audio/bcn_weather.mp3"}}
{"custom_id": "2", "body": {"file_url":"https://docs.mistral.ai/audio/chat.wav"}}
```

    </TabItem>
    <TabItem value="output" label="Result File">

```bash
{"id":"batch-40c6b8da-1-3b5e4359-2290-4749-b575-ec06e9cb8dfb","custom_id":"0","response":{"status_code":200,"body":{"model":"voxtral-mini-latest","text":"This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms, in schools, at farms and on factory floors, at diners and on distant military outposts, All these conversations are what have kept me honest, kept me inspired, and kept me going. Every day, I learned from you. You made me a better president, and you made me a better man. Over the course of these eight years, I've seen the goodness, the resilience, and the hope of the American people. I've seen neighbors looking out for each other as we rescued our economy from the worst crisis of our lifetimes. I've hugged cancer survivors who finally know the security of affordable health care. I've seen communities like Joplin rebuild from disaster, and cities like Boston show the world that no terrorist will ever break the American spirit. I've seen the hopeful faces of young graduates and our newest military officers. I've mourned with grieving families searching for answers. And I found grace in a Charleston church. I've seen our scientists help a paralyzed man regain his sense of touch and our wounded warriors walk again. I've seen our doctors and volunteers rebuild after earthquakes and stop pandemics in their tracks. I've learned from students who are building robots and curing diseases, and who will change the world in ways we can't even imagine. I've seen the youngest of children remind us of our obligations to care for our refugees, to work in peace, and above all, to look out for each other. That's what's possible when we come together in the slow, hard, sometimes frustrating, but always vital work of self-government. But we can't take our democracy for granted. All of us, regardless of party, should throw ourselves into the work of citizenship. Not just when there's an election. Not just when our own narrow interest is at stake, but over the full span of a lifetime. If you're tired of arguing with strangers on the Internet, try to talk with one in real life. If something needs fixing, lace up your shoes and do some organizing. If you're disappointed by your elected officials, then grab a clipboard, get some signatures, and run for office yourself. Our success depends on our participation, regardless of which way the pendulum of power swings. It falls on each of us to be guardians of our democracy. to embrace the joyous task we've been given to continually try to improve this great nation of ours. Because for all our outward differences, we all share the same proud title, citizen. It has been the honor of my life to serve you as President. Eight years later, I am even more optimistic about our country's promise, and I look forward to working along your side as a citizen for all my days that remain. Thanks, everybody. God bless you, and God bless the United States of America.","usage":{"prompt_tokens":4,"completion_tokens":634,"total_tokens":3263,"prompt_audio_seconds":203},"language":"en","segments":[],"finish_reason":null}},"error":null}
{"id":"batch-40c6b8da-2-d1e5dbfc-7fb2-4bba-9512-b45d4607e342","custom_id":"1","response":{"status_code":200,"body":{"model":"voxtral-mini-latest","text":"Yesterday it was 35 degrees in Barcelona, but today the temperature will go down to minus 20 degrees.","usage":{"prompt_tokens":4,"completion_tokens":29,"total_tokens":408,"prompt_audio_seconds":11},"language":"en","segments":[],"finish_reason":null}},"error":null}
{"id":"batch-40c6b8da-3-ec4261ea-84a1-43f8-a692-7ef177b549b9","custom_id":"2","response":{"status_code":200,"body":{"model":"voxtral-mini-latest","text":"Transcribe what I'm about to say. Today, on July 15th, 2025, Mistral AI is announcing the new VoxDroll models. Okay, and the transcription there, and then write code to give the first n digits of the Fibonacci sequence.","usage":{"prompt_tokens":4,"completion_tokens":64,"total_tokens":443,"prompt_audio_seconds":20},"language":"en","segments":[],"finish_reason":null}},"error":null}
```

    </TabItem>
</Tabs>

As a JSONL file, each line represents a request to the API Endpoint and Model.

<SectionTab as="h2" variant="secondary" sectionId="chat-completions-explained">Explanation</SectionTab>

The body request will follow the same format as the endpoint you want to run your batching, except the model id that will be provided only during the job creation to start the batch run. Below we provide an example of row with transcriptions:

<Tabs>
    <TabItem value="file" label="Sample Example" default>

```py
{
    "custom_id": "2", # An ID as metadata that will be returned in the output file to identify the request
    "body": { # The body of the request
        "file_url": "https://docs.mistral.ai/audio/chat.wav" # The file to transcribe you can provide a file_id instead
    }
}
```

    </TabItem>
    <TabItem value="output" label="Sample Result">

```py
{
  "id": "batch-40c6b8da-3-ec4261ea-84a1-43f8-a692-7ef177b549b9", # The ID of the request
  "custom_id": "2", # The custom ID metadata provided in the input file
  "response": { # The response body of the request
    "status_code": 200,
    "body": {
      "model": "voxtral-mini-latest",
      "text": "Transcribe what I'm about to say. Today, on July 15th, 2025, Mistral AI is announcing the new VoxDroll models. Okay, and the transcription there, and then write code to give the first n digits of the Fibonacci sequence.",
      "usage": {
        "prompt_tokens": 4,
        "completion_tokens": 64,
        "total_tokens": 443,
        "prompt_audio_seconds": 20
      },
      "language": "en",
      "segments": [],
      "finish_reason": null
    }
  },
  "error": null
}
```

    </TabItem>
</Tabs>

For more information regarding transcriptions, visit the [Transcriptions](/studio/audio/overview) docs and the corresponding [API Spec](../api/endpoint/audio/transcriptions#operation-audio_api_v1_transcriptions_post).
  </ExplorerTab>
</ExplorerTabs>

For large batches of up to 1M requests, you would create a .jsonl file with the above data. Once saved, you can upload your batch input file to ensure it is correctly referenced when initiating batch processes.

:::tip
For batches with less than 10k requests, we support [inline batching](#inline-batching).
:::

There are 2 main methods of uploading a batch file:

**A.** Via Studio:
  - Upload your files in <AppLink href="https://console.mistral.ai/build/files" app="studio">Files</AppLink>.
    - Upload the file in the format described previously.
    - Set `purpose` to Batch Processing.
  - Start and manage your batches in <AppLink href="https://console.mistral.ai/build/batches" app="studio">Batches</AppLink>.
    - Create and start a job by providing the `files`, `endpoint` and `model`.
You won't need to use the API to upload your files or create batching jobs.

**B.** Via the API, explained below:

To upload your batch file, you need to use the `files` endpoint.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>
    <Tabs groupId="sdk-version">
        <TabItem value="v1" label="V1">

```python
from mistralai import Mistral
import os

api_key = os.environ["MISTRAL_API_KEY"]

client = Mistral(api_key=api_key)

batch_data = client.files.upload(
    file={
        "file_name": "test.jsonl",
        "content": open("test.jsonl", "rb")
    },
    purpose = "batch"
)
```

        </TabItem>
        <TabItem value="v2" label="V2" default>

```python
from mistralai.client import Mistral
import os

api_key = os.environ["MISTRAL_API_KEY"]

client = Mistral(api_key=api_key)

batch_data = client.files.upload(
    file={
        "file_name": "test.jsonl",
        "content": open("test.jsonl", "rb")
    },
    purpose = "batch"
)
```

        </TabItem>
    </Tabs>
  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
import { Mistral } from '@mistralai/mistralai';
import fs from 'fs';

const apiKey = process.env.MISTRAL_API_KEY;

const client = new Mistral({apiKey: apiKey});

const batchFile = fs.readFileSync('batch_input_file.jsonl');
const batchData = await client.files.upload({
    file: {
        fileName: "batch_input_file.jsonl",
        content: batchFile,
    },
    purpose: "batch"
});
```

  </TabItem>
  <TabItem value="curl" label="curl">

```curl
curl https://api.mistral.ai/v1/files \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -F purpose="batch" \
  -F file="@batch_input_file.jsonl"
```

  </TabItem>
</Tabs>

<SectionTab as="h1" sectionId="batch-creation">Batch Creation</SectionTab>

### Create a new Batch Job

Create a new batch job, it will be queued for processing.

- Requests Data: The data for the requests to be batched. There are two options:
  - `input_files`: a list of the batch input file IDs, see how to use [file-batching](#file-batching).
  - `requests`: a list of the requests to be batched, see how to use [inline batching](#inline-batching).
- `model`: you can only use one model (e.g., `codestral-latest`) per batch. However, you can run multiple batches on the same files with different models if you want to compare outputs.
- `endpoint`: we currently support `/v1/embeddings`, `/v1/chat/completions`, `/v1/fim/completions`, `/v1/moderations`, `/v1/chat/moderations`, `/v1/ocr`, `/v1/classifications`, `/v1/conversations`, `/v1/audio/transcriptions`.
- `metadata`: optional custom metadata for the batch.

<SectionTab as="h2" variant="secondary" sectionId="file-batching">File Batching</SectionTab>

The standard batching approach relies on batch files containing all the requests to be processed. We support up to 1 million requests in a single batch, enabling efficient handling of large volumes of requests at a reduced cost. This is ideal for tasks with high throughput requirements but low latency sensitivity or priority.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>
```python
created_job = client.batch.jobs.create(
    input_files=[batch_data.id],
    model="mistral-small-latest",
    endpoint="/v1/chat/completions",
    metadata={"job_type": "testing"}
)
```
  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
const createdJob = await client.batch.jobs.create({
    inputFiles: [batchData.id],
    model: "mistral-small-latest",
    endpoint: "/v1/chat/completions",
    metadata: {jobType: "testing"}
});
```
  </TabItem>

  <TabItem value="curl" label="curl">

```bash
curl --location "https://api.mistral.ai/v1/batch/jobs" \
--header "Authorization: Bearer $MISTRAL_API_KEY" \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
--data '{
    "model": "mistral-small-latest",
    "input_files": [
        "<uuid>"
    ],
    "endpoint": "/v1/chat/completions",
    "metadata": {
        "job_type": "testing"
    }
}'
```
  </TabItem>
</Tabs>

<SectionTab as="h2" variant="secondary" sectionId="inline-batching">Inline-Batching</SectionTab>

For batches of fewer than 10,000 requests, we support inline batching. Instead of creating and uploading a `.jsonl` file with all the request data, you can include the request body directly in the job creation request. This is convenient for smaller-scale or less bulk-intensive tasks.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>
    <Tabs groupId="sdk-version">
        <TabItem value="v1" label="V1">

```python
# Not available in the V1 SDK (mistralai 1.x).
# This feature requires the V2 SDK: pip install "mistralai>=2", then use the V2 tab.
# Migration guide: https://docs.mistral.ai/resources/migration-guides
```

        </TabItem>
        <TabItem value="v2" label="V2" default>

```python
from mistralai.client import Mistral
import os

api_key = os.environ["MISTRAL_API_KEY"]

client = Mistral(api_key=api_key)

inline_batch_data = [
    {
        "custom_id": "0",
        "body": {
            "max_tokens": 128,
            "messages": [{"role": "user", "content": "What is the best French cheese?"}]
        }
    },
    {
        "custom_id": "1",
        "body": {
            "max_tokens": 512,
            "temperature": 0.2,
            "messages": [{"role": "user", "content": "What is the best French wine?"}]
        }
    },
    {
        "custom_id": "2",
        "body": {
            "max_tokens": 256,
            "temperature": 0.8,
            "messages": [{"role": "user", "content": "What is the best French pastry?"}]
        }
    },
]

created_job = client.batch.jobs.create(
    requests=inline_batch_data,
    model="mistral-small-latest",
    endpoint="/v1/chat/completions",
    metadata={"job_type": "testing"}
)
```

        </TabItem>
    </Tabs>
  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
import { Mistral } from '@mistralai/mistralai';

const apiKey = process.env.MISTRAL_API_KEY;

const client = new Mistral({apiKey: apiKey});

const inlineBatchData = [
    {
        "custom_id": "0", 
        "body": {
            "max_tokens": 128, 
            "messages": [{"role": "user", "content": "What is the best French cheese?"}]
        }
    },
    {
        "custom_id": "1", 
        "body": {
            "max_tokens": 512, 
            "temperature": 0.2, 
            "messages": [{"role": "user", "content": "What is the best French wine?"}]
        }
    },
    {
        "custom_id": "2", 
        "body": {
            "max_tokens": 256, 
            "temperature": 0.8, 
            "messages": [{"role": "user", "content": "What is the best French pastry?"}]
        }
    },
]

const createdJob = await client.batch.jobs.create({
    requests: inlineBatchData,
    model: "mistral-small-latest",
    endpoint: "/v1/chat/completions",
    metadata: {jobType: "testing"}
});
```
  </TabItem>

  <TabItem value="curl" label="curl">

```bash
curl --location "https://api.mistral.ai/v1/batch/jobs" \
--header "Authorization: Bearer $MISTRAL_API_KEY" \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
--data '{
    "model": "mistral-small-latest",
    "requests": [
      {"custom_id": "0", "body": {"max_tokens": 128, "messages": [{"role": "user", "content": "What is the best French cheese?"}]}},
      {"custom_id": "1", "body": {"max_tokens": 512, "temperature": 0.2, "messages": [{"role": "user", "content": "What is the best French wine?"}]}},
      {"custom_id": "2", "body": {"max_tokens": 256, "temperature": 0.8, "messages": [{"role": "user", "content": "What is the best French pastry?"}]}}
    ],
    "endpoint": "/v1/chat/completions",
    "metadata": {
        "job_type": "testing"
    }
}'
```
  </TabItem>
</Tabs>

<SectionTab as="h1" sectionId="get-retrieve">Get/Retrieve</SectionTab>

### Retrieve your Batch Job

Once batch sent, you will want to retrieve a lot of information such as:
- The status of the batch job
- The results of the batch job
- The list of batch jobs

<SectionTab as="h2" variant="secondary" sectionId="get-a-batch-job-details">Get a batch job details</SectionTab>

You can retrieve the details of a batch job by its ID.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>

```python
retrieved_job = client.batch.jobs.get(job_id=created_job.id)
```

  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
const retrievedJob = await client.batch.jobs.get({ jobId: createdJob.id});
```

  </TabItem>
  <TabItem value="curl" label="curl">

```bash
curl https://api.mistral.ai/v1/batch/jobs/<jobid> \
--header "Authorization: Bearer $MISTRAL_API_KEY"
```

  </TabItem>
</Tabs>

<SectionTab as="h2" variant="secondary" sectionId="get-batch-job-results">Get batch job results</SectionTab>

Once the batch job is completed, download the results.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>

```python
output_file_stream = client.files.download(file_id=retrieved_job.output_file)

# Write and save the file
with open('batch_results.jsonl', 'wb') as f:
    f.write(output_file_stream.read())
```

  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
import fs from 'fs';

const outputFileStream = await client.files.download({ fileId: retrievedJob.outputFile });

// Write the stream to a file
const writeStream = fs.createWriteStream('batch_results.jsonl');
outputFileStream.pipeTo(new WritableStream({
    write(chunk) {
      writeStream.write(chunk);
    },
    close() {
      writeStream.end();
    }
}));
```

  </TabItem>
  <TabItem value="curl" label="curl">

```bash
curl 'https://api.mistral.ai/v1/files/<uuid>/content' \
--header "Authorization: Bearer $MISTRAL_API_KEY" \
```

  </TabItem>
</Tabs>

<SectionTab as="h2" variant="secondary" sectionId="list-batch-jobs">List batch jobs</SectionTab>

You can view a list of your batch jobs and filter them by various criteria, including:

- Status: `QUEUED`,
`RUNNING`, `SUCCESS`, `FAILED`, `TIMEOUT_EXCEEDED`, `CANCELLATION_REQUESTED` and `CANCELLED`
- Metadata: custom metadata key and value for the batch

<Tabs groupId="code">
  <TabItem value="python" label="python" default>

```python
list_job = client.batch.jobs.list(
    status="RUNNING",
    metadata={"job_type": "testing"}
)
```

  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
const listJob = await client.batch.jobs.list({
    status: "RUNNING",
    metadata: {
        jobType: "testing"
    }
});
```

  </TabItem>
  <TabItem value="curl" label="curl">

```bash
curl 'https://api.mistral.ai/v1/batch/jobs?status=RUNNING' \
--header "Authorization: Bearer $MISTRAL_API_KEY"
```

  </TabItem>
</Tabs>

<SectionTab as="h1" sectionId="request-the-cancellation">Request Cancellation</SectionTab>

### Cancel any Job

If you want to cancel a batch job, you can do so by sending a cancellation request.

<Tabs groupId="code">
  <TabItem value="python" label="python" default>

```python
canceled_job = client.batch.jobs.cancel(job_id=created_job.id)
```

  </TabItem>
  <TabItem value="typescript" label="typescript">

```typescript
const canceledJob = await mistral.batch.jobs.cancel({
  jobId: createdJob.id,
});
```

  </TabItem>
  <TabItem value="curl" label="curl">

```bash
curl -X POST https://api.mistral.ai/v1/batch/jobs/<jobid>/cancel \
--header "Authorization: Bearer $MISTRAL_API_KEY"
```

  </TabItem>
</Tabs>

<SectionTab as="h1" sectionId="an-end-to-end-example">An end-to-end example</SectionTab>

Below is an end-to-end example of how to use the batch API from start to finish.

<ExplorerTabs id="end-to-end-example" mode="close">
  <ExplorerTab value="example" label="End-to-End Example">
    A dense example of how to use the Mistral API to run a batch job with random data. From:
- Creating a client
- Generating random input data
- Creating an input file
- Running a batch job
- Downloading the results

<Tabs groupId="sdk-version">
    <TabItem value="v1" label="V1">

```python
import argparse
import json
import os
import random
import time
from io import BytesIO

import httpx
from mistralai import File, Mistral

def create_client():
    """
    Create a Mistral client using the API key from environment variables.

    Returns:
        Mistral: An instance of the Mistral client.
    """
    return Mistral(api_key=os.environ["MISTRAL_API_KEY"])

def generate_random_string(start, end):
    """
    Generate a random string of variable length.

    Args:
        start (int): Minimum length of the string.
        end (int): Maximum length of the string.

    Returns:
        str: A randomly generated string.
    """
    length = random.randrange(start, end)
    return ' '.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=length))

def print_stats(batch_job):
    """
    Print the statistics of the batch job.

    Args:
        batch_job: The batch job object containing job statistics.
    """
    print(f"Total requests: {batch_job.total_requests}")
    print(f"Failed requests: {batch_job.failed_requests}")
    print(f"Successful requests: {batch_job.succeeded_requests}")
    print(
        f"Percent done: {round((batch_job.succeeded_requests + batch_job.failed_requests) / batch_job.total_requests, 4) * 100}")

def create_input_file(client, num_samples):
    """
    Create an input file for the batch job.

    Args:
        client (Mistral): The Mistral client instance.
        num_samples (int): Number of samples to generate.

    Returns:
        File: The uploaded input file object.
    """
    buffer = BytesIO()
    for idx in range(num_samples):
        request = {
            "custom_id": str(idx),
            "body": {
                "max_tokens": random.randint(10, 1000),
                "messages": [{"role": "user", "content": generate_random_string(100, 5000)}]
            }
        }
        buffer.write(json.dumps(request).encode("utf-8"))
        buffer.write("\n".encode("utf-8"))
    return client.files.upload(file=File(file_name="file.jsonl", content=buffer.getvalue()), purpose="batch")

def run_batch_job(client, input_file, model):
    """
    Run a batch job using the provided input file and model.

    Args:
        client (Mistral): The Mistral client instance.
        input_file (File): The input file object.
        model (str): The model to use for the batch job.

    Returns:
        BatchJob: The completed batch job object.
    """
    batch_job = client.batch.jobs.create(
        input_files=[input_file.id],
        model=model,
        endpoint="/v1/chat/completions",
        metadata={"job_type": "testing"}
    )

    while batch_job.status in ["QUEUED", "RUNNING"]:
        batch_job = client.batch.jobs.get(job_id=batch_job.id)
        print_stats(batch_job)
        time.sleep(1)

    print(f"Batch job {batch_job.id} completed with status: {batch_job.status}")
    return batch_job

def download_file(client, file_id, output_path):
    """
    Download a file from the Mistral server.

    Args:
        client (Mistral): The Mistral client instance.
        file_id (str): The ID of the file to download.
        output_path (str): The path where the file will be saved.
    """
    if file_id is not None:
        print(f"Downloading file to {output_path}")
        output_file = client.files.download(file_id=file_id)
        with open(output_path, "w") as f:
            for chunk in output_file.stream:
                f.write(chunk.decode("utf-8"))
        print(f"Downloaded file to {output_path}")

def main(num_samples, success_path, error_path, model):
    """
    Main function to run the batch job.

    Args:
        num_samples (int): Number of samples to process.
        success_path (str): Path to save successful outputs.
        error_path (str): Path to save error outputs.
        model (str): Model name to use.
    """
    client = create_client()
    input_file = create_input_file(client, num_samples)
    print(f"Created input file {input_file}")

    batch_job = run_batch_job(client, input_file, model)
    print(f"Job duration: {batch_job.completed_at - batch_job.created_at} seconds")
    download_file(client, batch_job.error_file, error_path)
    download_file(client, batch_job.output_file, success_path)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Run Mistral AI batch job")
    parser.add_argument("--num_samples", type=int, default=100, help="Number of samples to process")
    parser.add_argument("--success_path", type=str, default="output.jsonl", help="Path to save successful outputs")
    parser.add_argument("--error_path", type=str, default="error.jsonl", help="Path to save error outputs")
    parser.add_argument("--model", type=str, default="codestral-latest", help="Model name to use")

    args = parser.parse_args()

    main(args.num_samples, args.success_path, args.error_path, args.model)
```

    </TabItem>
    <TabItem value="v2" label="V2" default>

```python
import argparse
import json
import os
import random
import time
from io import BytesIO

import httpx
from mistralai.client import File, Mistral

def create_client():
    """
    Create a Mistral client using the API key from environment variables.

    Returns:
        Mistral: An instance of the Mistral client.
    """
    return Mistral(api_key=os.environ["MISTRAL_API_KEY"])

def generate_random_string(start, end):
    """
    Generate a random string of variable length.

    Args:
        start (int): Minimum length of the string.
        end (int): Maximum length of the string.

    Returns:
        str: A randomly generated string.
    """
    length = random.randrange(start, end)
    return ' '.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=length))

def print_stats(batch_job):
    """
    Print the statistics of the batch job.

    Args:
        batch_job: The batch job object containing job statistics.
    """
    print(f"Total requests: {batch_job.total_requests}")
    print(f"Failed requests: {batch_job.failed_requests}")
    print(f"Successful requests: {batch_job.succeeded_requests}")
    print(
        f"Percent done: {round((batch_job.succeeded_requests + batch_job.failed_requests) / batch_job.total_requests, 4) * 100}")

def create_input_file(client, num_samples):
    """
    Create an input file for the batch job.

    Args:
        client (Mistral): The Mistral client instance.
        num_samples (int): Number of samples to generate.

    Returns:
        File: The uploaded input file object.
    """
    buffer = BytesIO()
    for idx in range(num_samples):
        request = {
            "custom_id": str(idx),
            "body": {
                "max_tokens": random.randint(10, 1000),
                "messages": [{"role": "user", "content": generate_random_string(100, 5000)}]
            }
        }
        buffer.write(json.dumps(request).encode("utf-8"))
        buffer.write("\n".encode("utf-8"))
    return client.files.upload(file=File(file_name="file.jsonl", content=buffer.getvalue()), purpose="batch")

def run_batch_job(client, input_file, model):
    """
    Run a batch job using the provided input file and model.

    Args:
        client (Mistral): The Mistral client instance.
        input_file (File): The input file object.
        model (str): The model to use for the batch job.

    Returns:
        BatchJob: The completed batch job object.
    """
    batch_job = client.batch.jobs.create(
        input_files=[input_file.id],
        model=model,
        endpoint="/v1/chat/completions",
        metadata={"job_type": "testing"}
    )

    while batch_job.status in ["QUEUED", "RUNNING"]:
        batch_job = client.batch.jobs.get(job_id=batch_job.id)
        print_stats(batch_job)
        time.sleep(1)

    print(f"Batch job {batch_job.id} completed with status: {batch_job.status}")
    return batch_job

def download_file(client, file_id, output_path):
    """
    Download a file from the Mistral server.

    Args:
        client (Mistral): The Mistral client instance.
        file_id (str): The ID of the file to download.
        output_path (str): The path where the file will be saved.
    """
    if file_id is not None:
        print(f"Downloading file to {output_path}")
        output_file = client.files.download(file_id=file_id)
        with open(output_path, "w") as f:
            for chunk in output_file.stream:
                f.write(chunk.decode("utf-8"))
        print(f"Downloaded file to {output_path}")

def main(num_samples, success_path, error_path, model):
    """
    Main function to run the batch job.

    Args:
        num_samples (int): Number of samples to process.
        success_path (str): Path to save successful outputs.
        error_path (str): Path to save error outputs.
        model (str): Model name to use.
    """
    client = create_client()
    input_file = create_input_file(client, num_samples)
    print(f"Created input file {input_file}")

    batch_job = run_batch_job(client, input_file, model)
    print(f"Job duration: {batch_job.completed_at - batch_job.created_at} seconds")
    download_file(client, batch_job.error_file, error_path)
    download_file(client, batch_job.output_file, success_path)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Run Mistral AI batch job")
    parser.add_argument("--num_samples", type=int, default=100, help="Number of samples to process")
    parser.add_argument("--success_path", type=str, default="output.jsonl", help="Path to save successful outputs")
    parser.add_argument("--error_path", type=str, default="error.jsonl", help="Path to save error outputs")
    parser.add_argument("--model", type=str, default="codestral-latest", help="Model name to use")

    args = parser.parse_args()

    main(args.num_samples, args.success_path, args.error_path, args.model)
```

    </TabItem>
</Tabs>
  </ExplorerTab>
</ExplorerTabs>

<SectionTab as="h1" sectionId="faq">FAQ</SectionTab>

<Faq>
  <FaqItem question="Is the batch API available for all models?">
Yes, batch API is available for all models.
  </FaqItem>
  <FaqItem question="Does the batch API affect pricing?">

We offer a 50% discount on batch API. Learn more about our [pricing](/inference/pricing).
  </FaqItem>
  <FaqItem question="Does the batch API affect rate limits?">
No
  </FaqItem>
  <FaqItem question="What's the max number of requests in a batch?">
Currently, there is a maximum limit of 1 million pending requests per workspace. This means you cannot submit a job with more than 1 million requests. Additionally, you cannot submit two jobs with 600,000 requests each at the same time. You would need to wait until the first job has processed at least 200,000 requests, reducing its pending count to 400,000. At that point, the new job with 600,000 requests would fit within the limit.
  </FaqItem>
  <FaqItem question="What's the max number of batch jobs one can create?">
Currently, there is no maximum limit.
  </FaqItem>
  <FaqItem question="How long does the batch API take to process?">  
Processing speeds may be adjusted based on current demand and the volume of your request. Your batch results will only be accessible once the entire batch processing is complete.

Users can set `timeout_hours` when creating a job, which specifies the number of hours after which the job should expire. It defaults to 24 hours and should be lower than 7 days. A batch will expire if processing does not complete within the specified timeout.
  </FaqItem>
  <FaqItem question="Can I view batch results from my workspace?">  
Yes, batches are specific to a workspace. You can see all batches and their results that were created within the workspace associated with your API key.
  </FaqItem>  
  <FaqItem question="Will batch results ever expire?">  
No, the results do not expire at this time.
  </FaqItem>
  <FaqItem question="Can batches exceed the spend limit?">  
Yes, due to high throughput and concurrent processing, batches may slightly exceed your workspace's configured spend limit.
  </FaqItem>
</Faq>