Use BedrockConverse to send image (or document) ChatMessages

[stadeltobi]

Hi,

I'm wondering how I could use BedrockConverse to send an image (or document), rather than a plain text, ChatMessage to the model (in my case, Anthropic Claude Sonnet 3.5). In other words, rather than sending this:

{
    "role": "user | assistant",
    "content": [
        {
            "text": "string"
        }
    ]
}

... I'd like to send this:

{
    "role": "user",
    "content": [
        {
            "image": {
                "format": "png | jpeg | gif | webp",
                "source": {
                    "bytes": "image in bytes"
                }
            }
        }
    ]
}

Both formats are taken from the respective AWS documentation page

From a quick look at the conversion code in bedrock_converse/utils.py it looks like this is currently not supported.

Or am I missing something?

[kimnamu]

Thanks for laying this out so precisely, with both Converse payload shapes side by side and a pointer to the exact conversion function. That made it quick to check against the current code.

Short answer: this is supported now. ImageBlock support landed in llama-index-llms-bedrock-converse 0.4.15 (#18373) and DocumentBlock handling was fixed up in #18791. Put the blocks in ChatMessage.blocks and _content_block_to_bedrock_format in bedrock_converse/utils.py turns them into the image / document content blocks from the Converse docs, raw bytes included.

from llama_index.core.base.llms.types import ChatMessage, TextBlock, ImageBlock, DocumentBlock
from llama_index.llms.bedrock_converse import BedrockConverse

llm = BedrockConverse(model="anthropic.claude-3-5-sonnet-20240620-v1:0", region_name="us-east-1")
msg = ChatMessage(
    role="user",
    blocks=[
        TextBlock(text="Describe the image and quote the document."),
        ImageBlock(image=png_bytes, image_mimetype="image/png"),  # or ImageBlock(path="pixel.png")
        DocumentBlock(data=txt_bytes, document_mimetype="text/plain", title="notes"),  # or DocumentBlock(path="report.pdf", title="report")
    ],
)
resp = llm.chat([msg])

What client.converse(...) actually receives for that message (captured with a mock client on main 0f43c00, llama-index-core 0.14.24, bedrock-converse 0.15.0, boto3 1.40.61; bytes shortened):

{
  "role": "user",
  "content": [
    {"text": "Describe the image and quote the document."},
    {"image": {"format": "png", "source": {"bytes": "<70 raw bytes, head=b'\\x89PNG\\r\\n\\x1a\\n'>"}}},
    {"document": {"format": "txt", "name": "notes", "source": {"bytes": "<29 raw bytes, head=b'The secr'>"}}}
  ]
}

Live call and a few details from the code path

Same message sent for real in us-east-1. Claude Sonnet 4 came back with a "marked as Legacy" error on my account, so this is Sonnet 4.5. The image was a 1x1 PNG and the document a text file containing "The secret word is pineapple."

model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
assistant: The image appears to be blank/white, and the secret word in the document is pineapple.
stopReason: end_turn | usage: {'inputTokens': 83, 'outputTokens': 25, 'totalTokens': 108, 'cacheReadInputTokens': 0, 'cacheWriteInputTokens': 0}

From _content_block_to_bedrock_format in llama_index/llms/bedrock_converse/utils.py:

• Image format comes from image_mimetype (image/png, image/jpeg, image/gif, image/webp). Anything else logs a warning and falls back to png. With ImageBlock(path=...) the mimetype is detected from the file.
• Image blocks are only emitted for role="user". On other roles they are dropped with a warning, which matches what Converse accepts.
• Document format is mapped from document_mimetype via BEDROCK_DOC_MIMETYPE_TO_FORMAT (pdf, csv, doc, docx, xls, xlsx, html, txt, md), falling back to txt. DocumentBlock(path="report.pdf", title="report") produced "format": "pdf" in the same capture.
• title becomes the Converse name field.

[stadeltobi]

very cool, thanks!