[2' read]

Image Description Extraction using Mistral's Vision Models

VisionImage understanding+1

In this notebook, we'll use the Mistral API to extract structured image descriptions in JSON format using the ministral-14b-2512 model. We'll send an image URL and prompt the model to return key elements with descriptions leveraging Structured Outputs.

Prerequisites#

Make sure you have an API key for the Mistral AI platform. We'll also show you how to load it from environment variables.

# Install the Mistral Python SDK
!pip install mistralai

# Install the Mistral Python SDK
!pip install mistralai

Setup#

We'll load the Mistral API key from environment variables and initialize the client. Make sure your API key is saved in your environment variables as MISTRAL_API_KEY.

%env MISTRAL_API_KEY=

%env MISTRAL_API_KEY=

from mistralai.client import Mistral
import getpass
import os

if not os.environ.get("MISTRAL_API_KEY"):
    os.environ["MISTRAL_API_KEY"] = getpass.getpass("Mistral API key: ")

client = Mistral(os.environ["MISTRAL_API_KEY"])

# Model specification
model = "ministral-14b-2512"

from mistralai.client import Mistral
import getpass
import os

if not os.environ.get("MISTRAL_API_KEY"):
    os.environ["MISTRAL_API_KEY"] = getpass.getpass("Mistral API key: ")

client = Mistral(os.environ["MISTRAL_API_KEY"])

# Model specification
model = "ministral-14b-2512"

Data Model#

Define a object that will correspond to the expected output of the model. Using structured outputs, we can force the model to answer in a specific format.

from pydantic import BaseModel

class Element(BaseModel):
    name: str
    description: str

class ImageDescription(BaseModel):
    elements: list[Element]
    global_description: str

from pydantic import BaseModel

class Element(BaseModel):
    name: str
    description: str

class ImageDescription(BaseModel):
    elements: list[Element]
    global_description: str

Sending Image URL for Description#

We'll prompt the model to describe the image by providing an image URL. The response will be returned in a structured JSON format with the key elements described.

# Define the messages for the chat API
messages = [
    {
        "role": "system",
        "content": (
            "You are an image description system. Given an image, describe it by providing a list of elements, their names and descriptions. "
            "And end with a global short description of the full image.\n\n"
            "# Response Format\n"
            "Respond strictly in JSON using the following schema:\n\n"
            "{\n"
            '  "elements": [\n'
            "    {\n"
            '      "name": "<short name of the element>",\n'
            '      "description": "<detailed description of the element>"\n'
            "    }\n"
            "  ],\n"
            '  "global_description": "<short overall description of the image>"\n'
            "}"
        )
    },
    {
        "role": "user",
        "content": "Describe the image"
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": "https://docs.mistral.ai/img/eiffel-tower-paris.jpg"
            }
        ]
    }
]

# Call the Mistral API to complete the chat
chat_response = client.chat.parse(
    model=model,
    messages=messages,
    response_format=ImageDescription
)

# Define the messages for the chat API
messages = [
    {
        "role": "system",
        "content": (
            "You are an image description system. Given an image, describe it by providing a list of elements, their names and descriptions. "
            "And end with a global short description of the full image.\n\n"
            "# Response Format\n"
            "Respond strictly in JSON using the following schema:\n\n"
            "{\n"
            '  "elements": [\n'
            "    {\n"
            '      "name": "<short name of the element>",\n'
            '      "description": "<detailed description of the element>"\n'
            "    }\n"
            "  ],\n"
            '  "global_description": "<short overall description of the image>"\n'
            "}"
        )
    },
    {
        "role": "user",
        "content": "Describe the image"
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": "https://docs.mistral.ai/img/eiffel-tower-paris.jpg"
            }
        ]
    }
]

# Call the Mistral API to complete the chat
chat_response = client.chat.parse(
    model=model,
    messages=messages,
    response_format=ImageDescription
)

Showing Result#

We'll now parse the JSON response from the API. Our SDK parses and provides the object out of the box under.parsed, you can access the raw response in .content

# Get the object from the response
image_description = chat_response.choices[0].message.parsed

# Print each element and its description
print(image_description.global_description)
for element in image_description.elements:
    print("Element:", element.name)
    print("Description:", element.description, end="\n\n")

# Access the raw content
print(chat_response.choices[0].message.content)

# Get the object from the response
image_description = chat_response.choices[0].message.parsed

# Print each element and its description
print(image_description.global_description)
for element in image_description.elements:
    print("Element:", element.name)
    print("Description:", element.description, end="\n\n")

# Access the raw content
print(chat_response.choices[0].message.content)

And done ! You know how to use the Mistral Vision capabilities to describe an image by sending an image URL and receiving a structured JSON response. The descriptions provided by the model offer insights into the key elements of the image.