PolicyDocument not supported for hotswap deployments - full redeployment is required

[antje]

A full deployment is required every time I make a change due to the following error:

The following non-hotswappable changes were found:
    logicalID: ChatDefaultConversationHandlerconversationHandlerFunctionServiceRoleDefaultPolicyXXXXXX type: AWS::IAM::Policy, rejected changes: PolicyDocument, reason: This resource type is not supported for hotswap deployments

...any suggestion on how to workaround this to speed up the development cycle?

[ianacl]

Hi @antje, thanks for opening this issue and for your feedback.

For context, this occurs because the Resource element in the relevant IAM policy is constrained to the foundation model defined in the route -- changing that model results in a full deployment because the CDK doesn't support hotswaps for IAM policies. Changes to the inference configuration and the system prompt are hotswapped.

We're exploring options to allow model changes to be hotswapped.

In the meantime, a less-than-great workaround is to use a custom conversation handler function.

Add the @aws-amplify/backend-ai package.

npm install @aws-amplify/backend-ai

In amplify/data/resource.ts

import { type ClientSchema, a, defineData } from '@aws-amplify/backend';
import { defineConversationHandlerFunction } from "@aws-amplify/backend-ai/conversation";

export const conversationHandler = defineConversationHandlerFunction({
  entry: "./conversationHandler.ts",
  name: "conversationHandler",
  models: [
    { modelId: a.ai.model('Amazon Nova Pro') },
    { modelId: a.ai.model('Amazon Nova Lite') },
    { modelId: a.ai.model('Amazon Nova Micro') },
    { modelId: a.ai.model('Claude 3 Haiku') },
    { modelId: a.ai.model('Claude 3 Opus') },
    { modelId: a.ai.model('Claude 3 Sonnet') },
    { modelId: a.ai.model('Claude 3.5 Haiku') },
    { modelId: a.ai.model('Claude 3.5 Sonnet') },
    { modelId: a.ai.model('Claude 3.5 Sonnet v2') },
    { modelId: a.ai.model('Cohere Command R') },
    { modelId: a.ai.model('Cohere Command R+') },
    { modelId: a.ai.model('Llama 3.1 8B Instruct') },
    { modelId: a.ai.model('Llama 3.1 70B Instruct') },
    { modelId: a.ai.model('Llama 3.1 405B Instruct') },
    { modelId: a.ai.model('Mistral Large') },
    { modelId: a.ai.model('Mistral Large 2') },
    { modelId: a.ai.model('Mistral Small') },
  ],
});

const schema = a.schema({
  chat: a.conversation({
    // changing this to one of the models listed above will result in a hotswap
    aiModel: a.ai.model('Amazon Nova Lite'),
    systemPrompt: 'You are a helpful assistant.',
    handler: conversationHandler,
  )}
    .authorization((allow) => allow.owner())
});

Create a new file amplify/data/conversationHandler.ts

import { handleConversationTurnEvent } from '@aws-amplify/backend-ai/conversation/runtime';

export const handler = handleConversationTurnEvent;

In amplify/backend.ts

import { defineBackend } from '@aws-amplify/backend';
import { auth } from './auth/resource';
import { data, conversationHandler } from './data/resource';

defineBackend({
  auth,
  data,
  conversationHandler,
});