In this tutorial, we will create a personalized Q&A app that can extract information from PDF documents using your selected open-source Large Language Models (LLMs). We will cover the benefits of using open-source LLMs, look at some of the best ones available, and demonstrate how to develop open-source LLM-powered applications using Shakudo.
If you want to skip directly to code, we’ve made it available on GitHub!
Let's start by understanding why developers increasingly prefer open-source LLMs over commercial offerings, like OpenAI's APIs.
Open-source LLMs are highly adaptable. They allow users to modify and optimize the models to cater to their needs. This flexibility enables the LLMs to understand and process unique data effectively. Ecosystems of hugging face, LangChain and Pytorch make open-source models easy to infer and finetune for specific use cases.
Adopting open-source LLMs significantly reduces dependency on large AI providers, giving you the freedom to select your preferred technology stack. This autonomy minimizes issues related to vendor lock-in and fosters an environment of collaboration within the developer community.
Cost efficiency is another vital benefit of employing open-source LLMs. For small-scale use (thousands of requests/day), the OpenAI's ChatGPT API is relatively cost-effective at around $1.30/day. For large-scale use (millions of requests/day), it can quickly rise to $1,300/day. In contrast, open-source LLMs on an NVIDIA A100 cost approximately $4/hour or $96/day.
Open-source LLMs provide better data privacy and security. Unlike third-party AI services, these models allow you to maintain complete control over your data, which minimizes the risk of data breaches. OpenAI offers an enterprise license that allows businesses to use and fine-tune their LLMs. This can help businesses address data privacy concerns by allowing them to train the models on their data. However, the enterprise license is expensive and requires a significant amount of technical expertise to use.
Fine-tuning can be time-consuming and expensive. It can also be difficult to ensure that the model is not biased or harmful. Open-source LLMs still offer the best data privacy and security, allowing businesses to completely control their data and training process.
For applications where real-time user interaction is crucial, the high latency of GPT-4 can be a significant drawback. When optimized and deployed efficiently, open-source models can offer much lower latencies, which makes them more suitable for user interfacing applications.
Open-source LLMs enable companies and developers to contribute to the future of AI. The freedom to control the model's architecture, training data, and training process promotes experimentation with novel techniques and strategies. It allows you to stay updated with the latest developments in AI research and contribute to the AI community by sharing your models and techniques.
When it comes to open-source LLMs, there's a variety to choose from, including top ones like Falcon-40B-Instruct and Guanaco-65b. OpenLLM Leaderboard compares text-generative LLMs on different benchmarks.
MTEB leaderboard similarly compares text-embedding models on different tasks.
For any textual knowledge base (in our case, PDFs), we first need to extract text snippets from the knowledge base and use an embedding model to create a vector store representing the semantic content of the snippets. When a question is asked, we estimate its embedding and find relevant snippets using an efficient similarity search from vector stores. After extracting the snippets, we engineer a prompt and generate an answer using the LLM generation model. The prompt can be tuned based on the specific LLM used.
Experimentation and development are crucial elements in the field of data science. Shakudo's session facilitates the selection of the appropriate computing resources. It provides the flexibility to choose from Jupyter Notebooks, VS Code Server (provided by the platform) or connecting via SSH to use a preferred local editor.
We begin by setting up the models and embeddings that the knowledge bot will use, which are critical in interpreting and processing the text data within the PDFs.
We use the following Open Source models in the codebase:
Hugging faces MTEB leaderboard compares embedding models on different tasks. Instructor XL ranks very highly on this list, even better than OpenAI's ADA.
Open source models used in the codebase are
There are other high-performing open-source models (MPT-7B, StableLM, RedPajama, Guanaco) in the OpenLLM Leaderboard, which can be easily integrated with hugging face pipelines.
Let’s go ahead and first set up SBERT for the embedding model and FLANT5-Base for the generation model. We chose these models because they can run on an 8 core CPU. FastChat-T5 and Flacon-Instruct-7B require GPU. Loading them is similar and is shown in Codebase:
To employ these models, we use Hugging Face pipelines, which simplify the process of loading the models and using them for inference.
The creation of the models is governed by the configuration settings and is handled by the create_sbert_mpnet() and create_flan_t5_base() functions, respectively.
If we want to load Falcon, the pipeline would be as below and its task is ”text-generation” as Falcon is a decoder-only model. We need to allow remote code execution because the code comes from the Falcon author’s repository and not from hugging face.
This setup forms the foundation of the knowledge bot's capability to understand and generate responses to textual input.
In this step, let’s load our PDF and split it into manageable text snippets.
Now, we retrieve relevant snippets based on question embeddings and then construct a prompt to query the LLM.
Finally, we query the LLM using our question. The PDF knowledge bot will return the relevant information extracted from the PDF.
To make the code more organized, we can encapsulate all functionalities into a class.
We can now initialize and run the PdfQA class with the following code:
Shakudo integrates with various tools you can choose to build your front end. For this app, let’s wrap our web application around our PdfQA class with Streamlit, a Python library that simplifies app creation.
Below is the code breakdown:
We start by importing the necessary modules
Now, let’s set the page configuration and have a session state of the class to avoid instantiating the class multiple times in the same session.
To load the model and embedding on the GPU or CPU only once across all the client sessions, we cache the LLM and embedding pipelines.
Create our Steamlit app sidebar to include radio buttons for model selection and a file uploader. Once the file is submitted, It triggers the model loading and PDF ingestion to create a vector store.
Add a text input box for the question. Once we submit the question, it triggers the retrieval of relevant text snippets from the vector store and queries the LLM with an appropriate prompt.
This user interface allows the user to upload a PDF file, choose the model to use and ask a question.
Finally, our app is ready, and we can deploy it as a service on Shakudo. The platform makes the deployment process easier, allowing you to put your application online quickly.
Finally, our app is ready, and we can deploy it as a service on Shakudo. The platform makes the deployment process easier, allowing you to put your application online quickly.
Deploying applications on Shakudo offers enhanced security and control. Unlike many other deployments, Shakudo locks your application behind the SSO or your organization. The services and the self-hosted models run entirely within your cloud tenancy and on your dedicated Shakudo cluster, providing you with the flexibility to avoid vendor lock-in and enabling you to retain control over your applications running in the cloud
To deploy your app on Shakudo, we need two key files: pipeline.yaml, which describes our deployment pipeline, and run.sh, a bash script to set up and run our application. Here's what these files look like.
In this script:
Now, our application is live! We can browse through the user interface to see how it works.
Shakudo Services not only simplifies the deployment of your applications but also has a robust approach to security. Deploying your models within your Virtual Private Cloud (VPC) is one of the most secure ways of hosting models, as it isolates them from the public internet and provides better control over your data.
In this tutorial, we described the advantages of using open-source LLMs over Commercial APIs. We showed how to integrate OSS LLMs Falcon, FastChat, and FlanT5 to query the internal Knowledge Base with the help of Hugging Face pipelines and LangChain.
Hosting and managing open-source LLMs can be a complex and challenging task. Shakudo simplifies LLM infrastructure, saving time, resources, and expertise. For a first-hand experience of our platform, we encourage you to reach out to our team and book a demo.
To understand about the practical applications with OpenAI APIs, we recommend reading our previous post about "Building a Confluence Q&A App with LangChain and ChatGPT" where we showcase a real-world use case, a chatbot to query your confluence directories. For further reading on LangChain, check out CommandBar's in-depth guide.
* The code is adapted based on the work in LLM-WikipediaQA, where the author compares FastChat-T5, Flan-T5 with ChatGPT running a Q&A on Wikipedia Articles.
In this tutorial, we will create a personalized Q&A app that can extract information from PDF documents using your selected open-source Large Language Models (LLMs). We will cover the benefits of using open-source LLMs, look at some of the best ones available, and demonstrate how to develop open-source LLM-powered applications using Shakudo.
If you want to skip directly to code, we’ve made it available on GitHub!
Let's start by understanding why developers increasingly prefer open-source LLMs over commercial offerings, like OpenAI's APIs.
Open-source LLMs are highly adaptable. They allow users to modify and optimize the models to cater to their needs. This flexibility enables the LLMs to understand and process unique data effectively. Ecosystems of hugging face, LangChain and Pytorch make open-source models easy to infer and finetune for specific use cases.
Adopting open-source LLMs significantly reduces dependency on large AI providers, giving you the freedom to select your preferred technology stack. This autonomy minimizes issues related to vendor lock-in and fosters an environment of collaboration within the developer community.
Cost efficiency is another vital benefit of employing open-source LLMs. For small-scale use (thousands of requests/day), the OpenAI's ChatGPT API is relatively cost-effective at around $1.30/day. For large-scale use (millions of requests/day), it can quickly rise to $1,300/day. In contrast, open-source LLMs on an NVIDIA A100 cost approximately $4/hour or $96/day.
Open-source LLMs provide better data privacy and security. Unlike third-party AI services, these models allow you to maintain complete control over your data, which minimizes the risk of data breaches. OpenAI offers an enterprise license that allows businesses to use and fine-tune their LLMs. This can help businesses address data privacy concerns by allowing them to train the models on their data. However, the enterprise license is expensive and requires a significant amount of technical expertise to use.
Fine-tuning can be time-consuming and expensive. It can also be difficult to ensure that the model is not biased or harmful. Open-source LLMs still offer the best data privacy and security, allowing businesses to completely control their data and training process.
For applications where real-time user interaction is crucial, the high latency of GPT-4 can be a significant drawback. When optimized and deployed efficiently, open-source models can offer much lower latencies, which makes them more suitable for user interfacing applications.
Open-source LLMs enable companies and developers to contribute to the future of AI. The freedom to control the model's architecture, training data, and training process promotes experimentation with novel techniques and strategies. It allows you to stay updated with the latest developments in AI research and contribute to the AI community by sharing your models and techniques.
When it comes to open-source LLMs, there's a variety to choose from, including top ones like Falcon-40B-Instruct and Guanaco-65b. OpenLLM Leaderboard compares text-generative LLMs on different benchmarks.
MTEB leaderboard similarly compares text-embedding models on different tasks.
For any textual knowledge base (in our case, PDFs), we first need to extract text snippets from the knowledge base and use an embedding model to create a vector store representing the semantic content of the snippets. When a question is asked, we estimate its embedding and find relevant snippets using an efficient similarity search from vector stores. After extracting the snippets, we engineer a prompt and generate an answer using the LLM generation model. The prompt can be tuned based on the specific LLM used.
Experimentation and development are crucial elements in the field of data science. Shakudo's session facilitates the selection of the appropriate computing resources. It provides the flexibility to choose from Jupyter Notebooks, VS Code Server (provided by the platform) or connecting via SSH to use a preferred local editor.
We begin by setting up the models and embeddings that the knowledge bot will use, which are critical in interpreting and processing the text data within the PDFs.
We use the following Open Source models in the codebase:
Hugging faces MTEB leaderboard compares embedding models on different tasks. Instructor XL ranks very highly on this list, even better than OpenAI's ADA.
Open source models used in the codebase are
There are other high-performing open-source models (MPT-7B, StableLM, RedPajama, Guanaco) in the OpenLLM Leaderboard, which can be easily integrated with hugging face pipelines.
Let’s go ahead and first set up SBERT for the embedding model and FLANT5-Base for the generation model. We chose these models because they can run on an 8 core CPU. FastChat-T5 and Flacon-Instruct-7B require GPU. Loading them is similar and is shown in Codebase:
To employ these models, we use Hugging Face pipelines, which simplify the process of loading the models and using them for inference.
The creation of the models is governed by the configuration settings and is handled by the create_sbert_mpnet() and create_flan_t5_base() functions, respectively.
If we want to load Falcon, the pipeline would be as below and its task is ”text-generation” as Falcon is a decoder-only model. We need to allow remote code execution because the code comes from the Falcon author’s repository and not from hugging face.
This setup forms the foundation of the knowledge bot's capability to understand and generate responses to textual input.
In this step, let’s load our PDF and split it into manageable text snippets.
Now, we retrieve relevant snippets based on question embeddings and then construct a prompt to query the LLM.
Finally, we query the LLM using our question. The PDF knowledge bot will return the relevant information extracted from the PDF.
To make the code more organized, we can encapsulate all functionalities into a class.
We can now initialize and run the PdfQA class with the following code:
Shakudo integrates with various tools you can choose to build your front end. For this app, let’s wrap our web application around our PdfQA class with Streamlit, a Python library that simplifies app creation.
Below is the code breakdown:
We start by importing the necessary modules
Now, let’s set the page configuration and have a session state of the class to avoid instantiating the class multiple times in the same session.
To load the model and embedding on the GPU or CPU only once across all the client sessions, we cache the LLM and embedding pipelines.
Create our Steamlit app sidebar to include radio buttons for model selection and a file uploader. Once the file is submitted, It triggers the model loading and PDF ingestion to create a vector store.
Add a text input box for the question. Once we submit the question, it triggers the retrieval of relevant text snippets from the vector store and queries the LLM with an appropriate prompt.
This user interface allows the user to upload a PDF file, choose the model to use and ask a question.
Finally, our app is ready, and we can deploy it as a service on Shakudo. The platform makes the deployment process easier, allowing you to put your application online quickly.
Finally, our app is ready, and we can deploy it as a service on Shakudo. The platform makes the deployment process easier, allowing you to put your application online quickly.
Deploying applications on Shakudo offers enhanced security and control. Unlike many other deployments, Shakudo locks your application behind the SSO or your organization. The services and the self-hosted models run entirely within your cloud tenancy and on your dedicated Shakudo cluster, providing you with the flexibility to avoid vendor lock-in and enabling you to retain control over your applications running in the cloud
To deploy your app on Shakudo, we need two key files: pipeline.yaml, which describes our deployment pipeline, and run.sh, a bash script to set up and run our application. Here's what these files look like.
In this script:
Now, our application is live! We can browse through the user interface to see how it works.
Shakudo Services not only simplifies the deployment of your applications but also has a robust approach to security. Deploying your models within your Virtual Private Cloud (VPC) is one of the most secure ways of hosting models, as it isolates them from the public internet and provides better control over your data.
In this tutorial, we described the advantages of using open-source LLMs over Commercial APIs. We showed how to integrate OSS LLMs Falcon, FastChat, and FlanT5 to query the internal Knowledge Base with the help of Hugging Face pipelines and LangChain.
Hosting and managing open-source LLMs can be a complex and challenging task. Shakudo simplifies LLM infrastructure, saving time, resources, and expertise. For a first-hand experience of our platform, we encourage you to reach out to our team and book a demo.
To understand about the practical applications with OpenAI APIs, we recommend reading our previous post about "Building a Confluence Q&A App with LangChain and ChatGPT" where we showcase a real-world use case, a chatbot to query your confluence directories. For further reading on LangChain, check out CommandBar's in-depth guide.
* The code is adapted based on the work in LLM-WikipediaQA, where the author compares FastChat-T5, Flan-T5 with ChatGPT running a Q&A on Wikipedia Articles.
.svg)
.jpg)
.svg)
.jpg)
