LangChain Streaming Response Tutorial: Build Real-Time AI Chat in 15 Minutes

March 18, 2026 30 minute read

Get Ready for Super Fast AI Chat!

Have you ever used an AI chatbot and had to wait a long time for the full answer? It can feel a bit slow sometimes, right? Imagine chatting with a friend who pauses for a minute before giving you their entire thought. It’s not very natural.

This is where “streaming responses” come in, making AI chats feel much faster and more real. Instead of waiting for the whole message, you see the AI’s reply appear word by word, just like someone typing. In this LangChain streaming response tutorial, you’ll learn how to build your own real-time AI chat experience. We will explore how to get your AI to talk instantly, making conversations smooth and engaging.

You’re going to discover how to use LangChain, a cool toolkit, to make your AI assistant respond in real-time. We’ll cover everything from the basic setup to showing those words instantly on your screen. Get ready to build your very own quick-responding AI in about 15 minutes! This LangChain streaming response tutorial will guide you every step of the way.

Why Streaming Matters for a Great User Experience

Imagine Waiting…

Think about sending a text message and then waiting a whole minute for the other person’s complete reply. You wouldn’t know if they’re still thinking or if they’ve gone away. This waiting can be a bit boring and make you feel impatient. When you wait for an AI to give its full answer all at once, it can feel similar.

You might even wonder if the AI is broken or just really slow. If an AI chat takes too long, people might get frustrated and stop using it. This is not a great experience for anyone.

Why Real-Time Feels Better

Now, imagine seeing the reply from your AI appear instantly, character by character. It feels much more natural and alive, doesn’t it? This is exactly what streaming responses do for your AI applications. It’s like watching someone type their message directly to you.

Seeing the words appear in real-time makes the AI seem more responsive and smarter. You feel like you’re having an actual conversation, not just waiting for a computer program to finish thinking. This makes using your AI much more enjoyable and engaging, especially when you’re following a LangChain streaming response tutorial to build something interactive.

Understanding Streaming Basics in LangChain

What is “Streaming” Anyway?

Imagine you are trying to drink a glass of water. If you try to drink the whole glass at once, it might be difficult and messy. But if you take small sips, it’s easy and natural. Streaming with AI is kind of like taking small sips.

Instead of the AI giving you one huge, complete answer all at once, it sends you tiny pieces. Each tiny piece is called a “token,” which can be a word or even just part of a word. You get these tokens one after another very quickly.

How LangChain Helps

LangChain is like a Swiss Army knife for building applications with smart AI models. It has many tools to help you connect different AI parts together. One of its super useful tools is for handling these tiny pieces, or “tokens,” when the AI is streaming.

It has special ways to let you catch each token as it comes out of the AI. You can set up “callbacks,” which are like little alarms that go off every time a new token arrives. This makes it much easier to build a responsive chat, which is a key part of any good LangChain streaming response tutorial.

Setting Up Your Environment

Before we dive into the exciting world of streaming with LangChain, you need a place to do your coding. Think of it like setting up your workspace before building a cool LEGO set. We’ll need Python, LangChain, and a way for your code to talk to an AI model like OpenAI.

Getting Started

First, make sure you have Python installed on your computer. Python is a popular programming language that LangChain uses. If you don’t have it, you can download it from the official Python website.

Next, we need to install the LangChain library and the OpenAI library. These are like the special tools you’ll need for your project. The OpenAI library helps your code talk to the powerful AI models made by OpenAI.

Finally, you’ll need an API key from OpenAI. This key is like a secret password that lets your computer use OpenAI’s AI models. Keep it safe and never share it publicly! You can get one from the OpenAI platform website.

Code Snippet: Installation

Open your computer’s command line or terminal program. This is where you type commands for your computer. Type the following command and press Enter to install the necessary libraries:

pip install langchain langchain-openai python-dotenv

The python-dotenv library is helpful for keeping your secret API key safe. It lets you store your key in a special file, so you don’t accidentally put it directly into your code. It’s a good practice for security.

After installing, create a file named .env in the same folder where you’ll write your Python code. Inside this .env file, add your OpenAI API key like this:

OPENAI_API_KEY="YOUR_SUPER_SECRET_OPENAI_API_KEY_HERE"

Remember to replace "YOUR_SUPER_SECRET_OPENAI_API_KEY_HERE" with your actual API key. Now your environment is ready for this LangChain streaming response tutorial!

LLM Streaming Configuration: Making Your AI Talk Instantly

Now that your setup is complete, it’s time to tell your AI model to start streaming. This is the core of our LangChain streaming response tutorial. We need to specifically ask the AI to send its response in tiny pieces, not all at once.

Picking Your AI Brain

For this tutorial, we will use one of OpenAI’s powerful AI models, like GPT-3.5 or GPT-4. These models are great because they are designed to send responses in a streaming fashion. You just need to configure LangChain correctly to enable this feature.

When you set up the AI model in LangChain, there’s a special switch you need to flip. This switch tells the model: “Hey, please send me your answer piece by piece, as soon as you think of it!” This is how we make the AI talk instantly.

Code Snippet: Basic LLM Setup for Streaming

Here’s how you tell LangChain to use an OpenAI model and enable streaming. First, we’ll load our API key using dotenv. Then we’ll create an instance of the ChatOpenAI model.

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI

# Load environment variables from .env file
load_dotenv()

# Get your OpenAI API key
api_key = os.getenv("OPENAI_API_KEY")

# Create the ChatOpenAI model instance
# The 'streaming=True' part is super important!
llm = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo", # You can try "gpt-4" if you have access
    streaming=True,         # This makes the magic happen!
    temperature=0.7         # How creative the AI is (0.0 for less, 1.0 for more)
)

print("LLM configured for streaming!")

The streaming=True part is the secret sauce here. Without it, the AI would wait until it had the entire answer ready before sending anything. With streaming=True, it starts sending tokens as soon as it generates them.

Understanding Callbacks: Your AI’s Messenger

Imagine you’re waiting for a delivery, and you want to know exactly when each part of your order arrives. A “callback” in programming is like a special messenger that tells you exactly that. In LangChain, callbacks are functions that get called at different moments during an AI operation.

For streaming, we use specific callbacks that are triggered every time a new token arrives from the AI. These callbacks are like little listeners waiting for the AI to speak. As soon as the AI utters a new token, the callback “hears” it and can do something with it.

For example, a callback might simply print the token to your screen as it arrives. This is how you see the words appearing one by one in real-time. Callbacks are fundamental for building interactive streaming experiences using LangChain.

Setting Up Streaming Callbacks: Catching Every Word

Now we know that callbacks are like messengers that tell us when a new token arrives. To truly make use of the LangChain streaming response feature, we need to create our own special messenger. This messenger will know what to do with each token as it comes in.

The `BaseCallbackHandler`

LangChain provides a special building block called BaseCallbackHandler. Think of it as a blank template for our messenger. We can create our own custom messenger by building on top of this template. This custom messenger will have specific instructions for what to do when new tokens appear.

The BaseCallbackHandler has several methods (like instructions) that we can fill in. The most important one for streaming is on_llm_new_token. This method is like the messenger’s main job: whenever a new AI token arrives, this is the instruction it follows.

Code Snippet: Simple Streaming Callback

Let’s create a very simple callback that just prints each token as it receives it. This is a crucial step in our LangChain streaming response tutorial.

from langchain.callbacks.base import BaseCallbackHandler
from langchain_core.outputs import LLMResult

# Our custom callback handler for streaming
class MyStreamingCallback(BaseCallbackHandler):
    """Callback handler for streaming LLM responses."""

    def on_llm_new_token(self, token: str, **kwargs) -> None:
        """Runs on new LLM token. Print tokens as they arrive."""
        print(token, end="", flush=True)

    def on_llm_start(self, serialized: dict, prompts: list[str], **kwargs) -> None:
        """Runs when LLM starts running."""
        print("\n--- LLM started streaming ---")

    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        """Runs when LLM ends running."""
        print("\n--- LLM finished streaming ---")

print("Custom streaming callback created.")

In on_llm_new_token, token: str means we expect a piece of text (a string) which is the new token. print(token, end="", flush=True) is important:

end="" makes sure the next token prints right after the current one, without starting a new line.
flush=True forces the text to appear on your screen immediately, instead of waiting for a buffer to fill up. This is key for real-time display.

We also added on_llm_start and on_llm_end to show when the AI starts and finishes talking. These are good for understanding the flow.

Integrating Callbacks with Your LLM

Now that we have our custom streaming messenger (the MyStreamingCallback), we need to tell our AI model to use it. When we created our ChatOpenAI model earlier, there’s a place to plug in our callbacks. This is how the ChatOpenAI model knows who to send the new tokens to.

You pass a list of your callback handlers to the callbacks parameter of the ChatOpenAI model. This tells LangChain, “Whenever you get a new token from the AI, please send it to this list of callback handlers.” This is how your custom logic gets to process each piece of the AI’s response in real time.

# Assuming llm and MyStreamingCallback are defined as above

# Create a new LLM instance, or modify the existing one
# We pass our callback handler inside a list
llm_with_callbacks = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo",
    streaming=True,
    callbacks=[MyStreamingCallback()], # <-- Here's where we add our messenger!
    temperature=0.7
)

print("LLM now configured with our streaming callback.")

Notice that callbacks=[MyStreamingCallback()] is a list, meaning you can have multiple messengers doing different things at the same time. This setup allows your LangChain streaming response tutorial to fully demonstrate real-time output.

Handling Streaming Events: Displaying Tokens in Real-Time

We’ve set up our AI model to stream and created a callback to catch those streaming tokens. Now, it’s time to put it all together and see the magic happen! You will ask the AI a question, and then watch as its answer appears on your screen character by character.

Putting it All Together

With llm_with_callbacks ready, we can send it a message. LangChain will handle sending your message to OpenAI, and then OpenAI will start streaming back the response. Each piece of that response will trigger our MyStreamingCallback. Our callback will then print each token to the console as soon as it arrives.

This process gives you an immediate visual feedback loop. You won’t have to wait for the entire response to be generated. Instead, you’ll see the AI’s thoughts unfold in front of you, making the interaction feel instant and engaging.

Code Snippet: First Streaming Chat

Let’s run a simple example using the invoke method. The invoke method is a straightforward way to send a single message to your LLM. Watch your terminal closely when you run this code!

from langchain_core.messages import HumanMessage
# Assuming llm_with_callbacks and MyStreamingCallback are defined as above

print("\n--- Sending a message to the AI ---")

# We send a HumanMessage to the LLM
# The streaming will happen automatically because of our setup
llm_with_callbacks.invoke([HumanMessage(content="Tell me a short, simple story about a friendly robot.")])

print("\n--- Finished streaming chat example ---")

When you run this Python script, you will see the story appear token by token on your terminal. It’s a fantastic way to observe the LangChain streaming response tutorial in action. The on_llm_start and on_llm_end messages from our callback will bracket the streamed story.

This simple setup shows the core mechanics of real-time AI responses. You are directly observing the output as it’s generated, which is the essence of streaming. This immediate feedback dramatically enhances the user experience for any AI application you build.

Building a Better Callback

Printing to the console is a good start, but what if you want to do more? Maybe you want to collect all the tokens to form the complete answer. Or perhaps you want to send them to a web page so a user can see them in a chat bubble. Your callback can do all these things!

You can modify the on_llm_new_token method to store the tokens in a list or a string variable. This way, after the AI finishes speaking, you’ll have the complete response as well as the real-time display. For a web interface, you’d send each token to the user’s browser using technologies like WebSockets.

Here’s an example of how you might update MyStreamingCallback to collect the full response:

from langchain.callbacks.base import BaseCallbackHandler
from langchain_core.outputs import LLMResult
from typing import List, Any

class MyCollectingStreamingCallback(BaseCallbackHandler):
    """Callback handler for streaming LLM responses that also collects the full response."""

    def __init__(self, answer_container: List[str] = None) -> None:
        """Initialize with an optional container to store tokens."""
        self.answer_container = answer_container if answer_container is not None else []
        self.full_response_parts: List[str] = [] # To store all parts internally

    def on_llm_new_token(self, token: str, **kwargs) -> None:
        """Runs on new LLM token. Prints and collects tokens."""
        print(token, end="", flush=True)
        self.full_response_parts.append(token)
        if self.answer_container is not None:
            self.answer_container.append(token) # Also append to external container if provided

    def on_llm_start(self, serialized: dict, prompts: list[str], **kwargs) -> None:
        """Runs when LLM starts running. Clears previous response."""
        print("\n--- LLM started streaming (Collecting) ---")
        self.full_response_parts = [] # Clear for new response
        if self.answer_container is not None:
            self.answer_container.clear()

    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        """Runs when LLM ends running. Prints the collected full response."""
        print("\n--- LLM finished streaming (Collecting) ---")
        full_text = "".join(self.full_response_parts)
        print(f"\nFull collected response: {full_text}")

# Example usage with the collecting callback:
collected_tokens = []
collecting_callback = MyCollectingStreamingCallback(answer_container=collected_tokens)

llm_with_collecting_callbacks = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo",
    streaming=True,
    callbacks=[collecting_callback],
    temperature=0.7
)

print("\n--- Sending a message to the AI with collecting callback ---")
llm_with_collecting_callbacks.invoke([HumanMessage(content="Tell me a short poem about a fluffy cloud.")])

print("\n--- Finished collecting streaming chat example ---")
# Now you can access the full response from `collected_tokens`
print(f"External collected tokens: {''.join(collected_tokens)}")

This enhanced callback is more practical for real applications, allowing you to both display tokens immediately and reconstruct the full message. It fully demonstrates how versatile the LangChain streaming response tutorial can be.

Building a Basic Chat Interface Setup

So far, we’ve seen tokens stream directly into your computer’s terminal. While this is great for learning, a real chat application usually has a nicer looking interface. We’re talking about a webpage with text boxes and chat bubbles!

Beyond the Console

Imagine building an actual website where users can type messages and see the AI’s response. Printing text directly to the console won’t work there. We need a way to send those streaming tokens from our Python program to a user’s web browser. This is where the real-time aspect becomes exciting.

For a true chat interface, you would typically have a simple text input box where you type your question. Then, there would be a display area, maybe a series of chat bubbles, where the conversation history appears. The goal is to make the AI’s streamed responses update this display area in real-time.

Conceptual Flow

Let’s think about how the pieces would connect in a simple web application using our LangChain streaming response setup.

User Types Message: You type your question into a text box on a webpage and hit Enter.
App Sends to LangChain: Your web browser sends this question to your Python program (the “backend” server).
LangChain Streams Response: Your Python program uses our llm_with_callbacks setup from this tutorial. The AI starts generating its answer token by token.
App Updates Display: Each time our MyStreamingCallback receives a new token, instead of just printing it, it sends that token back to the user’s web browser. The web browser then immediately adds this new token to the chat display, making it appear in real-time.

Example: Simple Web Framework (Flask/Streamlit Concept)

While building a full web application is beyond our 15-minute goal, we can imagine what a simplified Python web framework might do. Frameworks like Flask, FastAPI, or Streamlit are popular for this. They allow your Python code to serve web pages and handle user requests.

For streaming to a web page, WebSockets are often used. A WebSocket is like a special, always-open pipe between your server and the user’s browser. Your server can push new tokens through this pipe to the browser as soon as they arrive.

Here’s a conceptual snippet showing how a Flask app might send tokens:

# This is NOT a complete runnable Flask app, just a conceptual example.
# It shows how streaming tokens from LangChain might be sent to a web client.

from flask import Flask, render_template, request, Response
from flask_socketio import SocketIO, emit
# You would need to install flask and flask-socketio: pip install flask flask-socketio

app = Flask(__name__)
socketio = SocketIO(app)

# Assume llm_with_callbacks and MyStreamingCallback are defined as before
# For a real app, you'd manage LLM initialization carefully.

class WebSocketStreamingCallback(BaseCallbackHandler):
    """Callback handler that emits tokens via WebSocket."""
    def __init__(self, sid: str) -> None:
        self.sid = sid # Session ID for the specific user's connection

    def on_llm_new_token(self, token: str, **kwargs) -> None:
        """Emit each token to the client via WebSocket."""
        # emit sends data to the client connected with 'self.sid'
        socketio.emit('new_token', {'token': token}, room=self.sid)
        print(f"Emitted token to {self.sid}: {token}") # For server-side debugging

    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        """Signal the end of the streaming response."""
        socketio.emit('stream_end', {}, room=self.sid)
        print(f"Stream ended for {self.sid}")

@app.route('/')
def index():
    return render_template('index.html') # A simple HTML file to show the chat

@socketio.on('connect')
def test_connect():
    print('Client connected:', request.sid)
    emit('my response', {'data': 'Connected', 'sid': request.sid})

@socketio.on('disconnect')
def test_disconnect():
    print('Client disconnected:', request.sid)

@socketio.on('send_message')
def handle_message(message_data):
    user_message = message_data['message']
    session_id = request.sid
    print(f"Message from {session_id}: {user_message}")

    # Initialize the LLM with a callback specific to this user's session
    llm_for_session = ChatOpenAI(
        openai_api_key=api_key, # assuming api_key is available
        model="gpt-3.5-turbo",
        streaming=True,
        callbacks=[WebSocketStreamingCallback(sid=session_id)],
        temperature=0.7
    )

    # Invoke the LLM - tokens will be sent via WebSocketStreamingCallback
    llm_for_session.invoke([HumanMessage(content=user_message)])

if __name__ == '__main__':
    # To run: flask --app your_file_name run
    # Or for SocketIO: python your_file_name.py
    socketio.run(app, debug=True)

And a super basic templates/index.html file (you would put this in a folder named templates):

<!DOCTYPE html>
<html>
<head>
    <title>Streaming Chat</title>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.0.1/socket.io.min.js"></script>
    <style>
        #chat-window {
            border: 1px solid #ccc;
            height: 300px;
            overflow-y: scroll;
            padding: 10px;
            margin-bottom: 10px;
        }
        .message {
            margin-bottom: 5px;
        }
        .user {
            color: blue;
        }
        .ai {
            color: green;
        }
    </style>
</head>
<body>
    <h1>Real-Time AI Chat</h1>
    <div id="chat-window"></div>
    <input type="text" id="user-input" placeholder="Type your message...">
    <button onclick="sendMessage()">Send</button>

    <script>
        var socket = io();
        var chatWindow = document.getElementById('chat-window');
        var userInput = document.getElementById('user-input');
        var currentAIMessageElement = null; // To append tokens to the same element

        socket.on('connect', function() {
            console.log('Connected to server!');
        });

        socket.on('new_token', function(data) {
            if (!currentAIMessageElement || currentAIMessageElement.dataset.streamEnded === 'true') {
                currentAIMessageElement = document.createElement('div');
                currentAIMessageElement.classList.add('message', 'ai');
                chatWindow.appendChild(currentAIMessageElement);
                chatWindow.scrollTop = chatWindow.scrollHeight; // Scroll to bottom
            }
            currentAIMessageElement.innerHTML += data.token;
        });

        socket.on('stream_end', function() {
            if (currentAIMessageElement) {
                currentAIMessageElement.dataset.streamEnded = 'true';
            }
        });

        function sendMessage() {
            var message = userInput.value;
            if (message.trim() === '') return;

            // Display user message
            var userMessageElement = document.createElement('div');
            userMessageElement.classList.add('message', 'user');
            userMessageElement.textContent = 'You: ' + message;
            chatWindow.appendChild(userMessageElement);

            // Send message to server
            socket.emit('send_message', { message: message });

            userInput.value = '';
            chatWindow.scrollTop = chatWindow.scrollHeight;
            currentAIMessageElement = null; // Reset for the next AI response
        }

        userInput.addEventListener('keypress', function(e) {
            if (e.key === 'Enter') {
                sendMessage();
            }
        });
    </script>
</body>
</html>

This conceptual example bridges the gap between our Python backend and a frontend. It demonstrates how a custom callback for this LangChain streaming response tutorial could leverage WebSockets to push real-time updates to a web client.

Streaming with Memory: Remembering the Chat

A smart AI chatbot should remember what you talked about before. If you ask, “What is the capital of France?” and then follow up with, “What about Germany?”, the AI should know you’re still talking about capitals. This ability to remember past conversations is called “memory.”

Why Memory is Important

Without memory, every question you ask the AI is like starting a brand new conversation. It forgets everything that was said just moments ago. This makes the chat feel disjointed and less intelligent. For example, if you ask “Summarize the last paragraph”, the AI would have no idea what “the last paragraph” refers to.

Good memory makes an AI chat feel much more natural and helpful. It allows for follow-up questions and more complex discussions. LangChain has excellent tools to manage this chat history, even when you are streaming responses.

Adding Memory to Your Chain

LangChain provides different types of memory. A common and easy one to start with is ConversationBufferMemory. This simply stores the past messages in a buffer (like a temporary storage area). To use memory with streaming, we usually combine our LLM with memory into a “chain.” A ConversationChain is perfect for this.

The ConversationChain takes your LLM and your memory component and links them together. When you send a message to the chain, it first looks at the chat history from the memory. Then, it sends that history along with your new message to the LLM. The LLM then generates a response, and our streaming callback still catches those tokens in real-time.

Code Snippet: Streaming with Memory

Let’s integrate memory into our streaming chat example.

from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from langchain_core.messages import HumanMessage, AIMessage
# Assuming llm and MyCollectingStreamingCallback are defined as above

# Create a memory instance
memory = ConversationBufferMemory(
    memory_key="history", # This is the key the chain will look for
    return_messages=True  # Returns messages as objects, useful for chat models
)

# Initialize a new callback specifically for this chat
# We'll use the collecting one to also see the full response
collected_memory_tokens = []
memory_streaming_callback = MyCollectingStreamingCallback(answer_container=collected_memory_tokens)

# Create the LLM with the new callback
llm_for_memory_streaming = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo",
    streaming=True,
    callbacks=[memory_streaming_callback],
    temperature=0.7
)

# Create the ConversationChain, passing our LLM and memory
conversation_chain = ConversationChain(
    llm=llm_for_memory_streaming,
    memory=memory,
    verbose=False # Set to True if you want to see what LangChain is doing internally
)

print("\n--- Starting a streaming chat with memory ---")

# First message
print("\nUser: Hi there!")
conversation_chain.invoke({"input": "Hi there!"})
# After the response, the memory will store "Human: Hi there!" and "AI: Hello! How can I help you today?"

# Second message, asking a follow-up
print("\nUser: Can you tell me a short story about an alien who loves pizza?")
conversation_chain.invoke({"input": "Can you tell me a short story about an alien who loves pizza?"})
# The AI should remember the "Hi there!" context, though it's not directly related,
# the conversation history is passed.

# Third message, referencing previous topic implicitly
print("\nUser: What was their name again?")
conversation_chain.invoke({"input": "What was their name again?"})
# The AI should remember the alien from the previous story.

print("\n--- Finished streaming chat with memory ---")

# You can inspect the memory manually too
print("\n--- Full Conversation History from Memory ---")
print(memory.load_memory_variables({}))

# Check the collected tokens for the last response
print(f"\nLast response (collected): {''.join(collected_memory_tokens)}")

In this example, the ConversationChain automatically manages the memory. When invoke is called, it fetches the chat history from memory, combines it with the input message, and then sends it to llm_for_memory_streaming. Our callback still ensures that each token appears instantly.

This setup makes your AI much more powerful and personable. It’s a key feature for building engaging applications with the LangChain streaming response tutorial. If you want to dive deeper into different memory types and their uses, you can read our comprehensive guide on LangChain memory types.

Testing Streaming Responses

After putting all the pieces together for your LangChain streaming response, it’s really important to test it. Testing helps you make sure everything works as expected and that your AI chat is truly real-time. Just like checking if your new toy works after you’ve built it.

How to Know it Works

The simplest and most direct way to test if streaming is working is to simply run your code and watch the output. Are the tokens appearing one by one, immediately after you send your message? If you see the text appearing as if someone is typing it, then your streaming setup is likely working correctly.

Also, make sure the full response that eventually appears is exactly what you expect from the AI. Check for any missing words or incomplete sentences. This ensures both real-time display and accurate content.

Measuring Speed

Beyond just “does it work?”, you might want to know “how fast does it work?”. You can measure a few things:

Time to first token (TTFT): This is how long it takes from when you send your message until the very first character of the AI’s response appears. A low TTFT makes the AI feel super responsive.
Total response time: How long it takes for the entire message to be generated and displayed.

You can use Python’s time module to measure these durations. Compare these times to what you would get if you disabled streaming. You should see a noticeable improvement in TTFT with streaming enabled.

import time
from langchain_core.messages import HumanMessage
# Assuming llm_with_callbacks and MyStreamingCallback are defined as above

# Let's create a new callback that prints AND records TTFT and total time
class TimingStreamingCallback(BaseCallbackHandler):
    def __init__(self):
        self.start_time = None
        self.first_token_time = None

    def on_llm_start(self, serialized: dict, prompts: list[str], **kwargs) -> None:
        self.start_time = time.time()
        self.first_token_time = None # Reset for each new call
        print("\n--- LLM started streaming (Timing) ---")

    def on_llm_new_token(self, token: str, **kwargs) -> None:
        if self.first_token_time is None:
            self.first_token_time = time.time()
            print(f"\n(Time to first token: {self.first_token_time - self.start_time:.2f} seconds)")
        print(token, end="", flush=True)

    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        end_time = time.time()
        print(f"\n--- LLM finished streaming (Timing) ---")
        print(f"(Total response time: {end_time - self.start_time:.2f} seconds)")

llm_for_timing = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo",
    streaming=True,
    callbacks=[TimingStreamingCallback()],
    temperature=0.7
)

print("\n--- Testing Streaming Performance ---")
llm_for_timing.invoke([HumanMessage(content="Explain quantum entanglement in simple terms.")])

print("\n--- Testing another query ---")
llm_for_timing.invoke([HumanMessage(content="Tell me a joke about a computer.")])

print("\n--- Finished Performance Test ---")

This timing callback helps you quantify the benefits of your LangChain streaming response setup.

Edge Cases

What happens if the AI gives a very short answer, like just one word? Your streaming should still work, displaying that single word instantly. What if there’s an error on the AI’s side? Your callback should ideally handle this gracefully, perhaps by printing an error message instead of crashing.

It’s good practice to think about these less common scenarios to make your chat application robust. The on_llm_error method in BaseCallbackHandler can be overridden to handle such situations. For instance, you could log the error or send a predefined message to the user.

# Extending our callback to handle errors
class RobustStreamingCallback(TimingStreamingCallback):
    def on_llm_error(self, error: Exception, **kwargs) -> None:
        print(f"\n--- LLM Error Encountered! ---")
        print(f"Error type: {type(error).__name__}, Message: {error}")
        print("--- Attempting to continue or gracefully exit ---")

llm_for_robust_test = ChatOpenAI(
    openai_api_key=api_key,
    model="gpt-3.5-turbo",
    streaming=True,
    callbacks=[RobustStreamingCallback()],
    temperature=0.7
)

print("\n--- Testing Error Handling (simulated, as LLM errors are hard to force) ---")
# To truly test this, you'd need to intentionally cause an error,
# e.g., by providing a bad API key or hitting rate limits.
# For now, we'll just show the structure.
llm_for_robust_test.invoke([HumanMessage(content="Tell me a story.")])

print("\n--- Finished Robustness Test ---")

This systematic approach to testing ensures that your LangChain streaming response tutorial leads to a reliable and fast AI application.

Deployment Considerations: Making Your Chat Live

You’ve built an amazing real-time AI chat on your own computer. That’s a huge step! But what if you want your friends, family, or even the whole world to use it? This is where “deployment” comes in. It means taking your code from your machine and putting it onto a server that is always running and accessible online.

From Your Computer to the World

When your AI chat runs on your local computer, only you can use it. To make it available to others, you need to host it on a server. A server is essentially another computer, but it’s powerful, always on, and connected to the internet. Think of it like moving your LEGO masterpiece from your bedroom to a public display.

This process involves setting up your Python application on the server. The server will then listen for requests from users’ web browsers. When a user visits your website, the server runs your LangChain code to generate and stream AI responses.

Websockets for Real-Time

For a truly real-time chat experience, especially with streaming responses, a technology called WebSockets is often the best choice. Unlike traditional website requests (where the browser asks for something, and the server sends a full answer and then closes the connection), WebSockets keep a continuous, open connection.

Imagine a direct phone line that stays open after you say “hello.” This allows your server to push information (like individual tokens from the AI) to the user’s browser whenever they arrive, without the browser constantly having to ask for updates. This is how the real-time display of tokens works smoothly in a web browser, making your LangChain streaming response powerful.

Choosing a Platform

There are many places (called “cloud platforms”) where you can host your application.

Big Cloud Providers: Companies like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure offer powerful and scalable services. They are great for large applications but can be complex for beginners.
Simpler Options: Platforms like Render, Heroku, or Vercel are often easier for deploying smaller applications. They handle many of the complicated server setups for you.
Streamlit Cloud: If you built your interface with Streamlit, Streamlit Cloud is an incredibly easy way to deploy your app directly.

When choosing, think about how many people you expect to use your chat and how much you’re willing to learn about server management. As your LangChain streaming response application grows, you might need to think about “scaling” – making sure it can handle many users at once.

Security

Security is super important when deploying any application online. Here are a few key things to remember:

Protect your API keys: Never put your OpenAI API key directly into your public code. Use environment variables (like we did with .env) and make sure these are correctly set up on your server.
Validate user input: Don’t trust any input directly from users. Always check it to prevent malicious attacks or unexpected behavior.
Think about privacy: If your chat application stores conversations, make sure you handle that data responsibly and inform your users about it.

By considering these deployment aspects, you can successfully share your real-time AI chat with others.

Troubleshooting Common Streaming Issues

Even with a great LangChain streaming response tutorial, sometimes things don’t go perfectly. If your streaming isn’t working as expected, don’t worry! Here are some common problems and how to fix them.

No Tokens Appearing?

This is usually the first sign that something isn’t right.

Check streaming=True: Double-check your ChatOpenAI setup. Did you forget to set streaming=True? This is the most common reason for no streaming.
Is your API key correct? If your OpenAI API key is wrong or missing, the LLM won’t be able to connect to OpenAI at all. Look for error messages related to authentication.
Is your callback correctly implemented? Make sure your custom callback handler (e.g., MyStreamingCallback) is inheriting from BaseCallbackHandler and that its on_llm_new_token method is spelled correctly and doing something visible (like print).
Did you pass the callback to the LLM? Ensure you included callbacks=[YourCallbackInstance()] when initializing your ChatOpenAI model.

Slow Streaming?

If tokens are appearing, but they’re very slow, it could be a few things:

Internet Connection: A slow internet connection on your side or the server’s side can delay tokens.
LLM Model Choice: Some AI models are faster than others. While gpt-3.5-turbo is generally quick, larger models like gpt-4 can sometimes be slower to generate tokens.
Server Load: If the OpenAI servers are very busy, or if your own server is under heavy load, responses might be delayed.
Network Latency: The physical distance between you (or your server) and OpenAI’s data centers can affect speed.

Partial Responses?

Sometimes, you might see tokens, but the full answer seems to cut off or be incomplete.

Ensure your callback collects all tokens: If you’re trying to reconstruct the full response from your callback, make sure you’re appending every token to a list or string.
Handle on_llm_end: The on_llm_end method in your callback is called when the LLM has completely finished its response. Use this method to finalize your collected response or perform any cleanup. If the connection drops before this is called, you might get an incomplete answer.
Check flush=True: If you’re just printing, make sure flush=True is used. Otherwise, tokens might be buffered and only appear in chunks, giving the illusion of a partial response before the buffer is cleared.

By systematically checking these points, you can usually identify and fix most issues related to your LangChain streaming response setup.

You’re a Streaming Pro!

Congratulations! You’ve successfully navigated this LangChain streaming response tutorial. You started with understanding why real-time AI chat is so much better for users. Then, you learned the fundamental concepts of streaming and how LangChain helps manage those tiny pieces of AI responses, called tokens.

You’ve set up your environment, configured an LLM for streaming, and built custom callback handlers to catch every single word the AI generates. You even learned how to display these tokens in real-time in your console and envisioned how this would work in a web interface. Plus, you integrated memory to make your AI chat feel even smarter and more natural, and understood how to test and deploy your creation.

The power to build highly interactive and responsive AI applications is now at your fingertips. Keep experimenting, keep building, and explore the endless possibilities of real-time AI. The world of instant AI chat is waiting for you!

Share on

X Facebook LinkedIn Bluesky

Coding Rhodes