AI Service Deployment Guide

PreviousAI Service Development Guide NextHarbor Guide

Last updated 5 months ago

1. Architecture overview

1.1 Overview

Table 1. Describe component in AI service architecture

No.

Category

Component

Description

AI scope

Note

Client

AI marketplace

Client callback

Get results after a task has been completed

AI App API

Receive request from client (marketplace) and process. Manage stages to solve a specific task.

Storage

S3 storage

Store media data (Including request data from the client and AI result data). Services send media file metadata to each other

AI APP Database

Store user request information for statistics

Redis

Store temporary results of tasks and requests: Will be deleted after a period of time

Queue

RabbitMQ

Store pending tasks

AI service 1

AI service 1 API

Receive requests to process tasks and push tasks into the queue

AI service worker

Pull the task in the queue and process it

AI service 2

AI service 2 API

Receive requests to process tasks and push tasks into the queue

Optional

Whether or not depends on the AI problem

AI service worker

Pull the task in the queue and process it

Optional

Whether or not depends on the AI problem

With a basic AI problem: AI service will require a minimum of 6 components (docker container). The number of components will increase when the AI problem is complex and needs to be divided into multiple stages for processing

Examples for simple AI problems: Person detection, face detection, face embedding, face searching… For each problem like this, 6 components need to be developed and deployed

For more complex problems such as face recognition, it needs to be processed through many steps: face detection, then face embedding and finally searching. There are up to 3 stages in this problem, so there will be up to 3 AI services. Therefore, the number of components that need to be developed and deployed is 10 (add 2 components for each added service)

1.2 AudioCraft service

Audio Craft is an AI service that generates sounds and music based on user descriptions. It is divided into 2 sub-services that operate independently of each other. The AudioGen is used for generating sounds such as: car horns, cat meowing... and the MusicGen specializes in generating music (with melody). AudioCraft's system design is shown in the figure 2

Table 2. Description of sequence diagram and executable code in source code

No.

Description

Executable code

Note

1.1

Client create a request to AI service

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:call

Line number: 198

1.2

Service validation

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:call

Line number: 205

1.3

Create workflow with workflow_id (task_id)

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:call:

Line number: 223

1.4

Response to client status of request

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:call:

Line number: 227

1.5

Sen request to AudioGen API

Module: app_api.app_api

Class:AudioCraftWorkFlow

Function:trigger_request_text2audio:

Line number: 132

1.6

Create a task and push to task queue

Module: audio_core_api.audio_core_api

Class:AudioGenerationAPI

Function:api_generate_audio

Line number: 66

1.7

AudioGen worker pull task and perform

Module: audio_core_api.audio_gen_worker

Class:AudioGenerationSerivce

Function:run

Line number: 28

1.8

Execute callback

Module: audio_core_api.audio_gen_worker

Class:AudioGenerationSerivceCallback

Function:run

Line number: 104

1.9

Store media file on S3 storage

Module: audio_core_api.audio_gen_worker

Class:AudioGenerationSerivce

Function:run

Line number: 54

1.10

Store temporary result on Redis

Module: audio_core_api.audio_gen_worker

Class:AudioGenerationSerivce

Function:run

Line number: 74

1.11

Aggregate final result

Module: app_api.app_api

Class:AudioCraftWorkFlow

Function:trigger_text2audio_callback

Line number: 145

1.12

Store final result on Redis

Module: app_api.app_api

Class:AudioCraftWorkFlow

Function:trigger_text2audio_callback

Line number: 147

1.13

Store request information in database

Module: app_api.app_api

Class:AudioCraftWorkFlow

Function:process_result

Line number: 103

1.14

Execute client callback

Module: app_api.app_api

Class:AudioCraftWorkFlow

Function:process_result

Line number: 94

Table 3. Description of sequence diagram and executable code in source code

No.

Description

Executable code

Note

2.1

Client create a request to to get result based on task_id

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:result

Line number: 235

2.2

Service validation

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:result

Line number: 238

2.3

Retrieve workflow based on task_id (workflow uuid)

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:result

Line number: 258

2.4

Get URL in S3

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:result

Line number: 266

2.5

Response result to client

Module: app_api.app_api

Class:AudioCraftAppAPI

Function:result

Line number: 288

Table 4. Description of sequence diagram and executable code in source code

No.

Description

Executable code

Note

3.1

Client create a request to to get statistical information

Module: aimarket_core_app_base.app

Class:BaseAppAPI

Function:stats

Line number: 112

3.2

Service validation

Module: aimarket_core_app_base.app

Class:BaseAppAPI

Function:stats

Line number: 112

3.3

Query data in database

Module: aimarket_core_app_base.app

Class:BaseAppAPI

Function:get_results_by_user_id

Line number: 117

3.4

Response result to client

Module: aimarket_core_app_base.app

Class:BaseAppAPI

Function:stats

Line number: 120

2. Package

2.1 Docker image

No.

Component

Docker image name

Type

RabbitMQ

rabbitmq:3.8.14-management-alpine

Opensource

Redis

redis:6-alpine

AI APP database

postgres:12-alpine

AI APP API

your_registry_url/your_repo:version

Self-build

AudioGen API

AudioGen worker

MusicGen API

MusicGen worker

2.2 Build docker image

Step 1: Clone the project

$ git clone https://github.com/DopikAI-Labs/aimarket-audiocraft-service.git
# The Git repository URL may change when delivering the source code to the customer. Carefully check the provided URL.

Step 2: Update submodule

$ cd aimarket-audiocraft-service
$ submodule update --init --recursive

If the source code provided is a compressed file, skip the above two steps

Step 3: Build image

Edit script: ./scripts/build_dockerfile.sh

Update the ECR, and version in ./version.json

#!/bin/bash

ECR="your_registry_url/your_repo"
TAG=$(jq -r '.api_version' version.json)
SERVICE="audio_craft"
IMAGE_NAME="${ECR}:${SERVICE}_${TAG}"

docker build -t $IMAGE_NAME -f dockerfile .

echo  image $IMAGE_NAME is built

$ cd aimarket-audiocraft-service

Build image

$ chmod +x ./scripts/build_dockerfile.sh
$ ./scripts/build_dockerfile.sh

If no errors occur, the terminal will show

$ your_registry_url/your_repo:audio_craft_0.1.4 is built

Step 4: Push image to registry

Step 4.1: Login to registry

Install awscli

$ pip install awscli
# or pip3 install awscli
# If you're using a virtual environment created with venv or conda, make sure it's activated before proceeding.

Configure AWS information

$ aws configure
# AWS Access Key ID [None]: Enter your access key ID
# AWS Secret Access Key [None]: Enter your secret  access key 
# Default region name [None]: Enter region
# Default output format [None]: Can skip it

For the next time, there is no need to do the above 2 steps in step 4.1 section

Get the password

$ aws ecr get-login-password
# The terminal will generate a string (password)

$ docker login -u AWS -p <<generated_string_from_above_command>> <<repository_URL_in_ECR>>

Step 4.2: Push docker image

Push docker image

$ docker push docker_image_name:tag

3. Deployment

3.1 Infrastructure requirements

No.

Category

Item

Required

Hardware

CPU

CPU arch

x64

RAM

16G

NVIDIA GPU

12GB

Storage (root)

32GB

Software

Ubuntu 20.04 or later

Docker

26.x.x or later

Docker-compose

1.29.x or later

3.2 Deployment procedure

Step 1: Pull docker image

Step 1.1 Login registry

Do the same as step 4.1 in the package section

Step 1.2 Pull image

$ docker pull docker_image_name:tag
# docker_image_name:tag => It is an pushed image in the package section

Step 2: Deploy service

Make a deployment directory

$ cd ../../../deployment_space_dirctory_path
$ mkdir audio_craft
$ cd audio_craft
$ mkdir v0.1.4
$ cd v0.1.4
# v0.1.4: Change to match the version to be deployed

Create a docker-compose.yml file

$ vim docker-compose.yml 
# Copy ./aimarket-audiocraft-service/docker-compose.yml and past to it

Create .env file

$ vim .env 
# Copy ./aimarket-audiocraft-service/.env and past to it

With the .env file, all environment variables highlighted in red need to be updated

No.

Group

Variable

Default value

Required update

AI Service

APP_API_PORT

8007

AUDIO_GEN_PORT

8081

Optional

MUSIC_GEN_PORT

8082

Optional

MAX_DURATION

Optional

MAX_PROMPT_LEN

100

Optional

CACHE_MODEL_DIR

../model_hub

TRANSFORMERS_CACHE

/model_hub

AUDIOCRAFT_CACHE_DIR

/model_hub

AUDIO_GEN_MODEL

facebook/audiogen-medium

Optional

MUSIC_GEN_MODEL

facebook/musicgen-small

Optional

API_VERSION

0.1.4

Optional

API_NAME

AudioCraft

Optional

Database

POSTGRES_USER

aimarketAdmin

Optional

POSTGRES_PASSWORD

aimarketPassword

Optional

POSTGRES_DB

aimarketDb

Optional

S3 storage

AWS_REGION_NAME

<<SECRET>>

AWS_ACCESS_KEY

<<SECRET>>

AWS_SECRET_KEY

<<SECRET>>

RabbitMQ

RABBITMQ_HOST

aimarket_rabbitmq

Optional

RABBITMQ_VHOST

aiqrbroker

Optional

RABBITMQ_USER

aiqradmin

Optional

RABBITMQ_PASS

aiqrPassword

Optional

RABBITMQ_PORT

5672

Optional

RABBITMQ_UI_PORT

15672

Optional

Redis

REDIS_HOST

aimarket_redis

Optional

REDIS_PORT

6379

Optional

REDIS_DB_NAME

Optional

MarketPlace

MARKETPLACE_TOKEN

<<SECRET>>

MARKETPLACE_CALLBACK

<<SECRET>>

Run service

# make sure you are in the same directory with docker-compose.yml file
$ docker-compose up

Check the terminal to determine if any errors occurred. If not, Ctrl + C to stop running service

Run service in detached mode

$ docker-compose up -d

1. Architecture overview

1.1 Overview

1.2 AudioCraft service

2. Package

2.1 Docker image

2.2 Build docker image

3. Deployment

3.1 Infrastructure requirements

3.2 Deployment procedure

4. Troubleshooting and support