add support for Amazon Nova Canvas model #7838

omrishiv · 2025-01-17T21:53:21Z

Title

Add initial support for Amazon Nova Canvas image generation

Relevant issues

Type

🆕 New Feature

Changes

Adds Nova Canvas model to provider list and adds supporting transformations for Bedrock

[REQUIRED] Testing - Attach a screenshot of any new tests passing locally

Added new test TestBedrockNova

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel · 2025-01-17T21:53:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 10, 2025 3:02pm

krrishdholakia · 2025-01-17T23:45:43Z

@omrishiv can you share a screenshot of it passing your testing

omrishiv · 2025-01-17T23:47:29Z

@krrishdholakia is there not one attached to the text? If you look at the bottom you’ll see the test name and it passing.

krrishdholakia · 2025-01-17T23:59:42Z

litellm/llms/bedrock/image/amazon_nova_transformation.py

+        """
+        No additional OpenAI params are mapped for Nova
+        """
+        return []


does amazon nova not support any openai image gen params?

if you could link me to their optional params that would help

imageGenerationConfig":{"cfgScale":8,"seed":42,"quality":"standard","width":1280,"height":720,"numberOfImages":3}

i see:

n -> numberOfImages

quality -> quality [if quality="hd" => quality="premium"]

size -> split into width x height
https://docs.aws.amazon.com/nova/latest/userguide/image-gen-req-resp-structure.html

I’m happy to convert those to the Nova ones, I’ll update for that

krrishdholakia · 2025-01-18T00:05:51Z

litellm/llms/bedrock/image/amazon_nova_transformation.py

+        """
+        Map the OpenAI params to the Bedrock params
+
+        No OpenAI params are mapped for Nova, so directly return the optional_params


@omrishiv could you please map the params listed above.

this also needs to be be used here -

litellm/litellm/utils.py

Line 2273 in 7584369

elif custom_llm_provider == "bedrock":

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

krrishdholakia · 2025-01-18T16:34:55Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+            width, height = _size.split("x")
+            width, height = int(width), int(height)
+        number_of_images = non_default_params.get("n", 1)
+        quality = "premium" if non_default_params.get("quality") == "premium" else "standard"


shouldn't it also handle the openai equivalent value - hd

I apologize, I now understand what this is doing. Can you clarify if intend for this to only handle openAI API parameters or if it should work for both? As in, would I expect quality premium|hd is premium or if we are only handling the openAI ones

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

omrishiv · 2025-01-20T03:33:07Z

I've added the rest of the image generation tasks:

text_image with image conditioning
color guided image generation
image variation
inpainting
outpainting
background removal

I'll push the changes soon

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

omrishiv · 2025-01-21T19:08:12Z

@krrishdholakia when you have a moment, can you please take a look at this again. I added all of the image generation tasks and tests for all of them. Thank you!

ishaan-jaff

overall LGTM, looks like you addressed all points in the review. Can you fix merge conflicts ? @omrishiv

cc @krrishdholakia, can you confirm this has addressed your points too ?

ishaan-jaff · 2025-02-13T01:58:07Z

tests/image_gen_tests/test_image_generation.py

+        return {"model": "bedrock/amazon.nova-canvas-v1:0",
+                "n": 1,
+                "size": "320x320",
+                "imageGenerationConfig": {"cfgScale":6.5,"seed":12},


i'm confused these are not litellm input params. Your tests should be passing these params https://docs.litellm.ai/docs/image_generation#input-params-for-litellmimage_generation

My understanding was that we can pass model specific parameters that can be used downstream https://github.com/BerriAI/litellm/blob/main/litellm/main.py#L3307. Happy to be corrected and address

ishaan-jaff · 2025-02-13T01:59:33Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+        Transform the request body for Amazon Nova Canvas model
+        """
+        task_type = optional_params.pop("taskType")
+        image_generation_config = optional_params.pop("imageGenerationConfig", {})


this is done incorrectly, litellm standardizes to this spec here: https://docs.litellm.ai/docs/image_generation#input-params-for-litellmimage_generation

Same as above I think?

I may be confused by what's on the page

Is this different?

There is no need for a new parameter - imageGenerationConfig

Any non-openai param, is passed directly though via optional_params - https://docs.litellm.ai/docs/completion/provider_specific_params

omrishiv · 2025-02-13T02:11:28Z

I fixed the merge conflict and would be happy to address the custom parameters. Bedrock allows for multiple tasks for the Nova Canvas model so I'd like to support those however makes sense

krrishdholakia · 2025-03-05T20:27:27Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+        """
+        Transform the request body for Amazon Nova Canvas model
+        """
+        task_type = optional_params.pop("taskType")


if this is None - what is the default behaviour? I would assume it's text -> image, given this is for image_generation

krrishdholakia · 2025-03-05T20:28:09Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+            return AmazonNovaCanvasColorGuidedRequest(taskType=task_type,
+                                                      colorGuidedGenerationParams=color_guided_generation_params,
+                                                      imageGenerationConfig=image_generation_config)
+        if task_type == "IMAGE_VARIATION":


@omrishiv this is the wrong endpoint to do image_variation on.

There is already an existing openai endpoint for this - https://platform.openai.com/docs/api-reference/images/createVariation

krrishdholakia · 2025-03-05T20:28:28Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+            return AmazonNovaCanvasImageVariationRequest(taskType=task_type,
+                                                         imageVariationParams=image_variation_params,
+                                                         imageGenerationConfig=image_generation_config)
+        if task_type == "INPAINTING":


this is the wrong endpoint for this.

It should be on image edit - https://platform.openai.com/docs/api-reference/images/createEdit

krrishdholakia · 2025-03-05T20:28:40Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+            return AmazonNovaCanvasOutpaintingRequest(taskType=task_type,
+                                                      outPaintingParams=outpainting_params,
+                                                      imageGenerationConfig=image_generation_config)
+        if task_type == "BACKGROUND_REMOVAL":


same for this - this should also be on image edit - https://platform.openai.com/docs/api-reference/images/createEdit

krrishdholakia · 2025-03-05T20:29:22Z

Please reduce the scope of this PR to just focus on text -> image generation,

We can then address the ability to do

text -> video
image edit

on separate PR's

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

krrishdholakia · 2025-03-06T03:27:01Z

tests/image_gen_tests/test_image_generation.py

+                "n": 1,
+                "size": "320x320",
+                "imageGenerationConfig": {"cfgScale":6.5,"seed":12},
+                "taskType": "TEXT_IMAGE"}


does tasktype always need to be specified?

I would assume if i'm calling the image_generation endpoint, this is implicit

Would you put Color Guided Image Generation into the Edit endpoint? If it's still in image_generation, the task is COLOR_GUIDED_GENERATION

krrishdholakia · 2025-03-06T03:28:08Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+        """
+        Transform the request body for Amazon Nova Canvas model
+        """
+        task_type = optional_params.pop("taskType", "TEXT_IMAGE")


Hey @omrishiv can you handle the case where tastType is set in params and is = None

This is handled by raising the error. None is not a supported taskType. I think if it's set as a parameter, it would be appropriate to expect a valid one. What do you think?

krrishdholakia · 2025-03-06T03:29:06Z

litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py

+        task_type = optional_params.pop("taskType", "TEXT_IMAGE")
+        image_generation_config = optional_params.pop("imageGenerationConfig", {})
+        image_generation_config = {**image_generation_config, **optional_params}
+        if task_type == "TEXT_IMAGE":


nit: is there a need for an if-block here? why not just check earlier on if tasktype != "TEXT_IMAGE" and raise an exception there

Reduces the amount of code inside an if-block

This goes back to the COLOR_GUIDED_GENERATION question. I can remove it for now, but may need it back later

krrishdholakia · 2025-03-06T03:30:42Z

LGTM - planning on merging this into a dev branch later today for further testing

tierfour · 2025-03-07T19:36:29Z

Please merge this!

…-support-for-nova-canvas

…10_2025_p1 add support for Amazon Nova Canvas model (#7838)

add initial support for Amazon Nova Canvas model

Loading
Loading status checks…

236ecbe

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 17, 2025 21:54 View deployment

krrishdholakia reviewed Jan 17, 2025

View reviewed changes

krrishdholakia reviewed Jan 18, 2025

View reviewed changes

adjust name to AmazonNovaCanvas and map function variables to config

Loading
Loading status checks…

dd08d6a

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 18, 2025 16:26 View deployment

tighten model name check

Loading
Loading status checks…

7db6252

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 18, 2025 16:27 View deployment

krrishdholakia reviewed Jan 18, 2025

View reviewed changes

fix quality mapping

Loading
Loading status checks…

6703dbc

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 18, 2025 16:49 View deployment

add premium quality in config

Loading
Loading status checks…

0f261ec

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 18, 2025 16:56 View deployment

support all Amazon Nova Canvas tasks

Loading
Loading status checks…

50a409a

vercel bot deployed to Preview January 20, 2025 16:39 View deployment

remove unused import

Loading
Loading status checks…

e9aefc3

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 20, 2025 16:41 View deployment

merge main

Loading
Loading status checks…

cb449a6

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 20, 2025 23:24 View deployment

add tests for image generation tasks and fix payload

Loading
Loading status checks…

e62b958

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 21, 2025 04:27 View deployment

add missing util file

Loading
Loading status checks…

f84069c

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 21, 2025 04:28 View deployment

update model prices backup file

Loading
Loading status checks…

f17ce63

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview January 21, 2025 04:31 View deployment

omrishiv changed the title ~~add initial support for Amazon Nova Canvas model~~ add support for Amazon Nova Canvas model Jan 22, 2025

omrishiv mentioned this pull request Jan 28, 2025

[Bug]: Unable to use Bedrock Nova Canvas model (Image Generation Model) #7190

Closed

ishaan-jaff approved these changes Feb 13, 2025

View reviewed changes

ishaan-jaff reviewed Feb 13, 2025

View reviewed changes

Merge branch 'main' into initial-support-for-nova-canvas

Loading
Loading status checks…

4af96c1

vercel bot deployed to Preview February 13, 2025 02:11 View deployment

krrishdholakia self-assigned this Feb 21, 2025

krrishdholakia reviewed Mar 5, 2025

View reviewed changes

remove image tasks other than text->image

Loading
Loading status checks…

5ede447

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

vercel bot deployed to Preview March 6, 2025 02:44 View deployment

krrishdholakia reviewed Mar 6, 2025

View reviewed changes

krrishdholakia changed the base branch from main to litellm_dev_contributor_prs_03_10_2025_p1 March 10, 2025 15:00

Merge branch 'litellm_dev_contributor_prs_03_10_2025_p1' into initial…

Loading
Loading status checks…

9650348

…-support-for-nova-canvas

krrishdholakia merged commit 0674491 into BerriAI:litellm_dev_contributor_prs_03_10_2025_p1 Mar 10, 2025
1 of 2 checks passed

vercel bot deployed to Preview March 10, 2025 15:02 View deployment

krrishdholakia added a commit that referenced this pull request Mar 12, 2025

Merge pull request #9101 from BerriAI/litellm_dev_contributor_prs_03_…

Loading
Loading status checks…

ae826be

…10_2025_p1 add support for Amazon Nova Canvas model (#7838)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for Amazon Nova Canvas model #7838

add support for Amazon Nova Canvas model #7838

omrishiv commented Jan 17, 2025

vercel bot commented Jan 17, 2025 •

edited

Loading

krrishdholakia commented Jan 17, 2025

omrishiv commented Jan 17, 2025

krrishdholakia Jan 17, 2025

krrishdholakia Jan 18, 2025

omrishiv Jan 18, 2025

krrishdholakia Jan 18, 2025

krrishdholakia Jan 18, 2025

omrishiv Jan 18, 2025

omrishiv commented Jan 20, 2025

omrishiv commented Jan 21, 2025

ishaan-jaff left a comment •

edited

Loading

ishaan-jaff Feb 13, 2025

omrishiv Feb 13, 2025 •

edited

Loading

ishaan-jaff Feb 13, 2025

omrishiv Feb 13, 2025

omrishiv Feb 13, 2025

krrishdholakia Mar 5, 2025

omrishiv commented Feb 13, 2025

krrishdholakia Mar 5, 2025

krrishdholakia Mar 5, 2025

krrishdholakia Mar 5, 2025

krrishdholakia Mar 5, 2025

krrishdholakia commented Mar 5, 2025

krrishdholakia Mar 6, 2025

omrishiv Mar 6, 2025

krrishdholakia Mar 6, 2025

omrishiv Mar 6, 2025

krrishdholakia Mar 6, 2025 •

edited

Loading

omrishiv Mar 6, 2025

krrishdholakia commented Mar 6, 2025

tierfour commented Mar 7, 2025

add support for Amazon Nova Canvas model #7838

add support for Amazon Nova Canvas model #7838

Conversation

omrishiv commented Jan 17, 2025

Title

Relevant issues

Type

Changes

[REQUIRED] Testing - Attach a screenshot of any new tests passing locally

vercel bot commented Jan 17, 2025 • edited Loading

krrishdholakia commented Jan 17, 2025

omrishiv commented Jan 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omrishiv commented Jan 20, 2025

omrishiv commented Jan 21, 2025

ishaan-jaff left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omrishiv Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omrishiv commented Feb 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krrishdholakia commented Mar 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krrishdholakia Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krrishdholakia commented Mar 6, 2025

tierfour commented Mar 7, 2025

vercel bot commented Jan 17, 2025 •

edited

Loading

ishaan-jaff left a comment •

edited

Loading

omrishiv Feb 13, 2025 •

edited

Loading

krrishdholakia Mar 6, 2025 •

edited

Loading