Deploying Python Microservices Application on Kubernetes (aks)

This DevOps project revolves around a Python-based microservices application deployed on Azure Kubernetes Service (AKS). The primary objective is to facilitate the conversion of MP4 videos to MP3 audio files seamlessly. Below is an overview of the architecture that will be deployed.

DevOps Project- AKS application architecture

Architecture Overview:

1. API Gateway:
— Three pivotal endpoints: Login, Upload, Download.
— Users initiate the process by logging in via the Login endpoint.

2. Authentication and Authorization:
— User credentials are validated against PostgreSQL for access verification.
— Successful authentication results in the generation of a JWT token, granting users the privilege to upload videos.

3. Upload Process:
— Authenticated users can upload MP4 videos via the designated endpoint.
— Uploaded videos are stored in RabbitMQ, ensuring efficient and reliable transfer to the subsequent processing step.

4. Video-to-Audio Conversion:
— The Converter Service, a vital microservice, receives videos from RabbitMQ.
— Utilizing advanced conversion algorithms, the service transforms MP4 videos into MP3 audio files.
— Processed data is then securely stored in MongoDB, providing a persistent repository for the converted audio.

5. Containerized Deployment with Helm-Charts:
— The entire solution leverages Kubernetes for container orchestration.
— Helm-charts streamline the deployment of essential components:
— PostgreSQL: Manages user credentials and authentication data.
— RabbitMQ: Facilitates efficient and reliable communication between components.
— MongoDB: Serves as a persistent storage solution for converted audio files.

6. Notification Service:
— Upon successful conversion, users receive notifications via email.
— This ensures users stay informed about the completion of their video-to-audio conversion tasks.

7. Download Endpoint:
— Users can access the Download endpoint to retrieve their converted MP3 audio files.
— Additionally, the application enables users to locally store the audio output for convenient access.

Key Technologies Used:
a. Azure Kubernetes Service (AKS): Provides a scalable and reliable container orchestration environment.
b. Python Microservices: Developed for efficient video-to-audio conversion.
c. JWT Authentication: Ensures secure access to the application.
d. PostgreSQL, RabbitMQ, MongoDB: Deployed via Helm-charts for robust and scalable data management.
e. Notification Service: Enhances user experience through timely notification.

HIGH-LEVEL MICROSERVICES DEPLOYMENT

The application comprises of 4 major microservices: auth-server, converter-module, Database-server (PostgreSQL and MongoDB) and notification server.

• auth-server: Navigate to the auth-server manifest folder and apply the configuration.
• gateway-server: Deploy the gateway-server.
• converter-module: Deploy the converter-module. Make sure to provide your email and password in converter/manifest/secret.yaml.
• notification-server: Configure email for notifications and two-factor authentication (2FA).

LOW-LEVEL STEPS OF DEPLOYING THE APPLICATION

Step-1: Cluster Creation on Azure Aks

a. Login to the Azure portal for the deploying of the Aks cluster

b. Navigate to Kubernetes services and start the deployment of the Azure Kubernetes services cluster.

Kubernetes Cluster Configuration

c. Configure Networking settings and IAM roles under networking section (ensure all necessary ports are open for communication) while configuring the aks cluster. Review + create the cluster and wait for it to be provisioned. Once, the cluster status shows as running/active you can go ahead and add Node pools.

d. Create Node pools and configure them based on the requirements

AKS Node pool configuration

There is option to add Labels and taints if required.

Configuring Taints and Labels on Node Pool

Step-2: Enable Container storage Interface (CSI) drivers on Azure Kubernetes Services

i. follow the steps mentioned in this documentation.

ii. To enable CSI storage drivers on a new cluster, include one of the following parameters depending on the storage system:

--enable-disk-driver allows you to enable the Azure Disks CSI driver.

--enable-file-driver allows you to enable the Azure Files CSI driver.

--enable-blob-driver allows you to enable the Azure Blob storage CSI driver.

--enable-snapshot-controller allows you to enable the snapshot controller.

az aks update -n myAKSCluster -g myResourceGroup --enable-disk-driver --enable-file-driver --enable-blob-driver --enable-snapshot-controller

iii. It may take several minutes to complete this action. Once it’s complete, you should see in the output the status of enabling the driver on your cluster.

Step-3: Deploying your Application on AKS

a. clone the code from this Github repository.

b. Set the cluster context as follows:

az login
az account set --subscription <subscription-id>
az aks get-credentials --resource-group <resource_group_name> --name <aks cluster name 
(this command downloads the credentials and configures the Kubernetes CLI to use them)

Setting Cluster context

Step-4: Managing your deployments (Helm-charts)

1. MongoDB

To install MongoDB, set the database username and password in values.yaml, then navigate to the MongoDB Helm chart folder and run:

cd Helm_charts/MongoDB
helm install mongo .

2. PostgreSQL

Set the database username and password in values.yaml . Install PostgreSQL from the PostgreSQL Helm chart folder and initialize it with the queries in init.sql. For PowerShell users:

cd ..
cd Postgres
helm install postgres .

3. RabbitMQ

Deploy RabbitMQ by running:

cd ..
cd RabbitMQ
helm install rabbitmq .

Step-5: Connecting

1. MongoDB

Connect to the MongoDB instance using:

mongosh mongodb://<username>:<pwd>@<nodeip>:30005/mp3s?authSource=admin

2. PostgreSQL

Connect to the Postgres database and copy all the queries from the “init.sql” file.

psql 'postgres://<username>:<pwd>@<nodeip>:30003/authdb'

3. RabbitMQ

Ensure you have created two queues in RabbitMQ named mp3and video. To create queues, visit <nodeIp>:30004> and use default username guest and password guest.

Step-6: Applying the manifest file for each micro service

1. Auth Service

cd auth-service/manifest
kubectl apply -f .

2. Gateway Service

cd gateway-service/manifest
kubectl apply -f .

3. Converter Service

cd converter-service/manifest
kubectl apply -f .

4. Notification Service

cd notification-service/manifest
kubectl apply -f .

Step-7: Add any mp4 video under assests that you want to convert

https://github.com/Muskantomar001/Python_Microservices_App_on_Azure_Kubernetes_Service_aks.git

Add any video mp4 under assests section of the repo that you want to convert to mp3.

Step-8: Verification of the deployed application

After deploying the microservices, verify the status of all components by running:

kubectl get all 

Step-9: Notification Configuration

For configuring email notifications and two-factor authentication , follow these steps:

1. Go to your Gmail account and click on your profile.
2. Click on “Manage Your Google Account.”
3. Navigate to the “Security” tab on the left side panel.
4. Enable “2-Step Verification.”
5. Search for the application-specific passwords. You will find it in the settings.
6. Click on “Other” and provide your name.
7. Click on “Generate” and copy the generated password.
8. Paste this generated password in converter/manifest/secret.yaml along with your email.

API DEFINITION

Run the application through the following API calls:

1. Login Endpoint :

POST http://nodeIP:30002/login

curl -X POST http://nodeIP:30002/login -u <email>:<password>

Expected Output: Success!

2. Upload Endpoint:

POST http://nodeIP:30002/upload

curl -X POST -F 'file=@./video.mp4' -H 'Authorization: Bearer <JWT Token>' http://nodeIP:30002/upload

Note: Check if you received the ID on your email.

3. Download Endpoint:

GET http://nodeIP:30002/download?fid=<Generated file identifier>

 curl --output video.mp3 -X GET -H 'Authorization: Bearer <JWT Token>' "http://nodeIP:30002/download?fid=<Generated fid>"

This DevOps project not only showcases the power of microservices but also exemplifies the seamless integration of multiple technologies to deliver a robust and user-friendly video-to-audio conversion solution.

Thanks! There will be more cloud computing and DevOps related articles to follow.

About me:

Hi 👋, I’m Muskan Tomar , a passionate Engineer from India working on Cloud & DevOps projects for 3 years now.

Azure Solutions Architect | Azure DevOps |SecurityEngineer| dministrator Associate| Kubernetes & Docker | Terraform

Reach me out on:

LinkedIn: https://www.linkedin.com/in/muskan -tomar-b3566615a/

Email: muskantomar001@gmail.com

Portfolio: muskantomar001.github.io