Reading Time: 4 minutes

When ChatGPT was launched in Nov 2022, it could only process text.

But now we can see all these LLMs can process Images, Voice and Videos as well. All these models have become multi-modal.

Namaste and Welcome to Build It Yourself.

Now a days multi modal LLMs (or Large Multimodal Models) can extract information from images and even answer questions related to them.

For example, if you upload an image of an invoice, the model can extract key details such as amounts, dates, and addresses.

Many businesses use paid tools for this purpose, but in this article, we will build a simple tool that can perform this function, potentially saving you money.

In this tutorial, we will build a Multi-Modal AI Application for Digital Invoice Reader using Anthropic’s multimodal model – Claude 3.5 Sonnet

If you are an entrepreneur or a senior It professional and looking to learn AI + LLM in a simple language, check out the courses and other details – https://www.aimletc.com/online-instructor-led-ai-llm-coaching-for-it-technical-professionals/

Multi-Modal AI Application for Digital Invoice Reader using Claude 3.5 Sonnet.

You can find the code notebook here – https://github.com/tayaln/Multimodality-Extracting-Text-from-digital-invoice

To use it, download and then upload it to your Google Drive and open it as a Google Colab file.

Pre-requisite

– An Open Mind to learn new things

– Anthropic account

– Anthropic API Key

Concepts we discussed in this video

• Anthropic’s Claude 3.5 Sonnet model

• API setup and usage

• Base64 image encoding

• JSON data extraction from images

• Automating invoice processing

Step 1 – Setting Up the Environment

We will use Anthropic’s Claude 3.5 Sonnet model for this project. To get started, follow these steps:

1. Install and Import Anthropic
   • First, install the necessary dependencies and import Anthropic’s API.
2. Set Up API Access
   • Note that Anthropic’s API requires a minimum deposit of $5 to use.
   • Visit their website, add $5 to your account, and you’re ready to go.
3. Select the Model
   • We will use the Claude 3.5 Sonnet model, released on October 22, 2024.

Step 2 – Building the Application

To begin, let’s create a basic application:

1. Uploading an Image
   • If using Google Colab, you can upload an image by navigating to the file section and adding your file.
2. Processing the Image
   • Load the image (e.g., an ice cream bowl) and encode it in Base64 format.
   • Convert the encoding into a string for processing.
3. Sending the Image to the Model
   • Define a prompt asking how many scoops of ice cream are in the image.
   • Send the image to the model with the appropriate request format.
4. Receiving and Displaying the Response
   • The model will analyze the image and return an answer, such as: “This image contains three scoops of chocolate ice cream served in a white bowl, garnished with a fresh strawberry.”

Step 3 – Extracting Data from an Invoice Image

Now, let’s apply the same process to a real-world use case—extracting structured data from an invoice image.

1. Upload an Invoice Image
2. Convert and Encode the Image
3. Define the Prompt
   • Ask the model to generate a JSON object containing key invoice details such as:
     • Dates
     • Dollar amounts
     • Addresses
4. Process the Response
   • The model will return a structured JSON output containing all relevant details, making it easy to integrate into a database or application.

Conclusion

This simple AI-powered tool demonstrates how easy it is to extract valuable data from images using multi-modal AI models.

Many people pay for expensive tools to perform similar tasks, but as developers, we have the power to build these tools ourselves.

With the right approach, creating such applications can be even easier than using pre-existing paid solutions.

I hope you found this tutorial helpful! See you in the next one. Thanks for reading!

what are the key concepts discussed in this video. Just the names, no description

If you have any queries, or suggestions, share them with me on LinkedIn – https://www.linkedin.com/in/nikhileshtayal/

Here is how I learned AI as a non-technical person in 4 months for free.

Let’s learn to build a basic AI/ML model in 4 minutes (Part 1)

Happy learning!