Balancing Privacy and Performance in LLM

Welcome to the enthralling universe of Generative AI! Recognized as a realm of
artificial
intelligence algorithms with the magic to conjure fresh content based on available data, this
technology has unassumingly become the new frontier for a plethora of industries, including tech,
banking, and media. And, no surprises here, the technological world has openly embraced it in myriad
innovative ways.

Let’s talk about the recent sensation that has been taking the digital world by a storm: ChatGPT. An
innovation by OpenAI, this AI-powered chatbot, which enjoys robust backing from notable investors
like Microsoft, boasts of advanced conversational capabilities that have caught the world’s
attention. The technology has been smoothly integrated into Microsoft’s Bing search engine and Edge
browser, with further plans to weave it into “every layer of the stack,” as revealed by CEO Satya
Nadella. 

The narrative took a new twist in March 2023 when OpenAI introduced GPT-4, the latest in the ChatGPT
software series, currently available to a circle of subscribers and developers. And the competition
is heating up! Alphabet, for instance, is on the brink of launching a conversational AI service,
aptly named Bard.

A Peek into Privacy Concerns with OpenAI

However, when dealing with sensitive data, we wade into a more complex scenario. Imagine your medical
history or bank account details being accessible – these are elements that most of us prefer to
cloak under a veil of privacy. The corporate world is no stranger to this necessity either, where
sensitive data, like proprietary research data, needs to be handled with kid gloves.

A few headlines to ponder:

ChatGPT
under investigation by Canadian privacy watchdog

Germany
considers following Italy in banning ChatGPT

ChatGPT
bug leaked payment data, conversation titles of users, confirms OpenAI

OpenAI gives in to
Italy’s data privacy demands, ending ChatGPT ban

Exploring the In-House Model 🏠

Here’s an alternative: an in-house AI model. However, it does not come without its own set of
challenges. This approach demands a devoted team of data scientists to upkeep and refresh the model,
making it a resource-intensive strategy. But there’s a silver lining – adopting an in-house model
bestows a superior level of control over data and its processing, offering a secure option for
handling sensitive data.

Yet, the dilemma persists. Re-training the data will be confined to your private dataset unless your
data scientists dive into merging the LLM with updates from public sources, ensuring that the model
benefits from external enhancements without transmitting any internal data outwards.

Navigating the Path to Deploy Open Source LLMs Privately

And so, we arrive at the million-dollar question: How does one deploy
open source LLMs on a private network? 🔐

Generative AI applications often utilize common components like Embedding models, vector stores, and
LLMs. Those tech aficionados following Bluetick Consultants’ generative AI journey might have a
grasp on utilizing open source vector stores and Embedding models. The enigma, however, lies in
deploying LLMs while considering aspects like latency and cost.

1. Harnessing Amazon SageMaker’s Might for LLM Deployment

Amazon SageMaker, celebrated as a fully-managed machine learning service, presents a seamless
horizon where data scientists and developers can not only effortlessly build and train machine
learning models but also usher them into a production-ready hosted environment. It arms you with
an integrated Jupyter authoring notebook instance, ensuring smooth access to your data sources
for exploration and analysis, all while liberating you from the shackles of server management.
Moreover, it extends a plethora of common machine learning algorithms, optimized to perform with
utmost efficiency against sizeable data in a distributed environment.

For additional information, see Amazon
SageMaker developer resources.

• Step 1: Craft Your Domain and Step into the Studio

Start by creating a domain and opening the studio, navigating to the SageMaker Jumpstart
option. For our journey today, we’ll deploy the “llama 2 7B” model.

📌 A Quick Detour: Why the llama 2 7B model, you ask? The face-off among LLMs (GPT
3.5 Turbo, Llama 2 7B, and Falcon 7B) unfolds the story clearly. Dive into “The LLM
Face-Off” to explore the nuances.

The LLM Face-Off: GPT 3.5 Turbo, Llama 2 7B, and Falcon 7B

SageMaker defaults its instance type to ml.g5.2xlarge, limiting the downgrade to versions
like ml.g5.xlarge but permitting upgrades to, say, ml.g5.4xlarge or ml.g5.8xlarge to
minimize latency.


• Step 2: Open and Operate Your Notebook

Within the SageMaker studio, initiate a notebook, each operating on an instance type, with
the default being ml.t3.medium.


• Step 3: Examine the Model and Endpoint

Ensure to turn off the studio when it’s idle to conserve resources. Navigate through
SageMaker’s left menu, selecting “Inference.” Here, your locally deployed model, endpoint,
and other configurations stand ready for exploration.


• Step 4:
  Time for Cleanup

Especially crucial for those in the testing or research phases, don’t overlook the cleanup.
Wipe the slate clean and remove all necessary components within “Inference,” deleting both
the model and endpoint.

2. Empowering Deployment with RunPod

RunPod is a cloud computing platform meticulously sculpted for AI and machine learning
applications. With offerings like GPU Instances, Serverless GPUs, and AI Endpoints, RunPod is
steadfast in its mission: making cloud computing not only accessible and affordable but also
rich in features, user-friendly, and an immersive experience. Here, we strive to arm both
individuals and enterprises with pioneering technology, unlocking the colossal potential that AI
and cloud computing bring to the table.
Embark with us on a journey to deploy the Llama 2 7B model using RunPod, ensuring every step is
unveiled, from account creation to API utilization.

• Step 1: Begin with Basics – Account Creation & API Key
  Generation

Commence by creating your RunPod account and weaving through the settings page to generate an
API key. It’s crucial to note that RunPod operates as a prepaid service, so let’s kick
things off by adding $10 to get the ball rolling.


• Step 2: Navigate the “Secure Cloud”

Find your way to the “Secure Cloud” tab, where you’ll be greeted by a myriad of GPU servers,
available with either on-demand or monthly options to cater to your specific needs.

For deploying llama 2 7B model, we’ll use NVIDIA RTX A5000 GPU instance with specifications
24 GB vRAM 27 GB RAM, 7 vCPU


• Step 3: Deploying the Llama 2 7B Model – Let’s Get Technical with
  Python

Now, let’s get our hands a bit dirty in the codes. We will deploy the Llama 2 7B model
utilizing Python and here’s how we’ll do it:

Install the Needed Packages:

 
 !pip install -quiet requests==2.31.0 --progress-bar off
 !pip install -quiet runpod --progress-bar off
                             

Import the relevant modules

 
 import requests
 import runpod
                             

Add your API key generated in step 1

 
 runpod.api_key = "Add your Runpod API Key"
                             

Create a pod on runpod

 
 gpu_count = 1

 pod = runpod.create_pod(
     name="Llama-7b-chat",
     image_name="ghcr.io/huggingface/text-generation-inference:0.9.4",
     gpu_type_id="NVIDIA RTX A5000",
     cloud_type="SECURE",
     docker_args="--model-id TheBloke/Llama-2-7b-chat-fp16",
     gpu_count=gpu_count,
     volume_in_gb=50,
     container_disk_in_gb=5,
     ports="80/http,29500/http",
     volume_mount_path="/data",
 )
                             

To access the API via
Swagger Docs, utilize the URL, as shown below.

 
 SERVER_URL = f'https://{pod["id"]}-80.proxy.runpod.net'
 print(SERVER_URL)
                             

To view this source code –Source code

3. LLM Deployment on Your Private, GPU-Enabled Server! 🚀

In the world of AI and generative algorithms, the notion of privacy isn’t just a luxury—it’s an
imperative.When privacy and control are paramount, hosting LLM models on a private server
provides that extra layer of security and customization that businesses often need.

Why a GPU-Enabled Server?

GPUs (Graphics Processing Units) are designed to handle multiple tasks simultaneously. They’ve
been the powerhouses behind the computational might needed for high-intensity tasks like deep
learning. LLM models, given their intricate architecture and the vast amounts of data they
process, can benefit immensely from the parallel processing capabilities of GPUs.

So, let’s unravel the steps to establish this deployment.

• Step 1: LLM Model – Download It!

Begin by downloading your chosen LLM model. Ensure that the model aligns with your objectives
and has been validated for its performance and reliability.

 
 # Define Model ID
 model_id = "TheBloke/Llama-2-7b-chat-fp16"

 # Load Tokenizer
 tokenizer = AutoTokenizer.from_pretrained(model_id)

 # Load Model
 model = AutoModelForCausalLM.from_pretrained(model_id, 
 cache_dir='/opt/workspace/',
     torch_dtype=torch.bfloat16, trust_remote_code=True, 
 device_map="auto", offload_folder="offload")
                             

OR

 
 model_name_or_path = "TheBloke/Llama-2-7b-chat-fp16"
 model_basename = "llama-2-7b-chat.ggmlv3.q5_0.bin" # the model is in bin 
 format

 model_path = hf_hub_download(repo_id=model_name_or_path, 
 filename=model_basename)
                             

• Step 2: Load Up with Tokenizer or Ctokenizer

With the LLM model in your possession, employ a Tokenizer or Ctokenizer to load it. These
tools assist in preparing your text data in a format that’s comprehensible and usable for
the LLM, ensuring that the model interprets the input effectively to generate desired
outputs.

 
 from langchain.llms import CTransformers
 llm = CTransformers(
         model = model_path,
         model_type="llama",
         max_new_tokens = 512,
         temperature = 0.5
     )
                             

OR

 
 pipeline = transformers.pipeline(
     "text-generation",
     model=model,
     tokenizer=tokenizer,
     device_map="auto",
     max_length=400,
     do_sample=True,
     top_k=10,
     num_return_sequences=1,
     eos_token_id=tokenizer.eos_token_id
 )
                             

• Step 3: Bridge the Gap with a Framework

Introduce a framework that’s specifically designed to form a stable and efficient connection
between the LLM and your programming language. A popular choice in this context is Langchain
– renowned for its ability to seamlessly integrate LLMs with different programming
languages, thereby, enabling an efficient interaction between the two.

 
 # Setup prompt template
 template = PromptTemplate(input_variables=['input'], template='{input}')
 # Pass hugging face pipeline to langchain class
 llm = HuggingFacePipeline(pipeline=pipeline, model_kwargs = 
 {'temperature':0})
 # Build stacked LLM chain i.e. prompt-formatting + LLM
 chain = LLMChain(llm=llm, prompt=template)
                             

Alternatively, you can build an API.

 
 @app.route('/generate_response', methods=['POST'])
 def generate_response():
 try:
     data = request.get_json()
     user_prompt = data['user_prompt']

     # Generate a response using LLMChain
     response = LLM_Chain.run(user_prompt)

     return jsonify({"response": response})

 except Exception as e:
     return jsonify({"error": str(e)}), 500
                             

Eager to explore more? Dive deeper with the following resources where similar deployments
have been carved out meticulously:

Falcon LLM in
Action: A Step-by-Step Tutorial

Exploring Llama 2: From
Installation to Interaction

In these resources, you’ll find a treasure trove of insights, strategies, and step-by-step
guidelines to navigate the exciting world of LLM deployments.

Pricing

Ah, pricing! The crucial decider when it comes to choosing a platform for deploying your LLM. As your
guide through the terrain of LLM deployments, we’re here to dissect the costs related to deploying
open-source LLMs using different cloud platforms, ensuring that your AI adventures don’t burn a hole
in your pocket.

Let’s delve into the nitty-gritty by comparing the pricing, specification, and latency across
different platforms:

Choosing the right platform essentially boils down to balancing your budget against your need for
speed (latency) and computational power (specifications). It’s imperative to ponder upon:

• Will the latency impact user experience?
• Is the computational power sufficient to manage the workload?
• Does the cost align with your budget allocations?

Always ensure to factor in additional costs like data transfer, storage, and potential overage
charges to get a holistic view of your total expenditure.

Conclusion

Embarking upon the intricate pathways of deploying LLMs leads us to the intersection of innovation,
pragmatism, and prudent financial planning. With each model we deploy, a new chapter of digital
communication unfolds, steering us toward a future where our virtual interactions are increasingly
sophisticated, personalized, and secure. It’s vital to remember that every algorithm we bring to
life in the silent depths of our private networks not only empowers our digital endeavors but also
holds the mirror to our ethical and strategic choices in the enthralling realm of Generative AI.

“In every byte of data, there’s a story yet to be told; let’s write it wisely,
let’s write it boldly.”