useChat / handleSubmit -> pass back data other than message?

[peepers-rick]

Hi there,

I'm using the handleSubmit method from the useChat hook and then hitting my POST function at /api/chat, returning a StreamingTextResponse(OpenAIStream(gptResponse)). That seems to add to the messages array, which is great, but I want to pass along additional information that I can render in my app as well. I'd like to pass some JSON that I can render below the chat message as different content cards, similar to how a plugin on ChatGPT might render dynamic cards. It won't be part of the streaming response from the OpenAI API, so I need to be able to inject in onResponse or onFinish somehow?

[MaxLeiter]

Hey @peepers-rick, we're actively working on supporting this use-case.

You can accomplish this now by including your component data in the response headers or adding it to the stream with a custom delimiter (which is what we'll likely due for you in the future).

https://markprompt.com/blog/hybrid-streaming-payloads has details

[peepers-rick]

Awesome, thanks! I was kind of experimenting with this direction but it felt a bit awkward so I abandoned it. It's not super critical now so I'll wait for your team's solution which will certainly be more elegant than my duct tape.

[RobertHH-IS]

More working examples of handling links, content and images, and user feedback with functions and langchaim would be quite helpful. Parsing partial JSON and injecting additional data into streams is messy work.

[lgrammel]

useChat now supports sending data in addition to messages. See #725 for more details.

[MauritsMosterd]

Sorry for my ignorance, but I'm stuck on the same part. And #725 seems to send data to the server. But @ peepers-rick meant receiving extra data in the front end. Same thing I'm stuck on. I can send the data through headers without any problem, but receiving them through 'useChat' seems impossible. Personally, I want to use it for a set of fixed questions, but there are many use cases for it.

Edit: onResponse did the trick, sometimes I'm just blind lol.

[lgrammel]

You can use the StreamData() for this purpose, see https://sdk.vercel.ai/docs/api-reference/stream-data

Example with data in test: https://github.com/vercel/ai/blob/main/packages/core/streams/openai-stream.test.tsx#L335

[MauritsMosterd]

Tried that, but in the frontend it just appended it to the message like "FIXED DATA, message". Ended up using onResponse to get header data :). Thank you very much for your reply!

[lgrammel]

How are you using the functionality on the frontend? Here is a test case that shows how to access the data: https://github.com/vercel/ai/blob/main/packages/core/react/use-chat.ui.test.tsx#L13

[karar-shah]

Oh how exciting! I was in search of same solution as you @MauritsMosterd , browsing through all the chats. But guess what? After just good 30 minutes of exploring each ref. link, I stumbled upon your tiny edit message. Haha! 😂

onResponse(response) {
  // Extracting headers from the response
  const headers = response.headers;
  const rateLimit = headers.get("prompt_tokens"); // Extracting the value of X-RATE-LIMIT(vercel sdk doc) header
  console.log("prompt_tokens ----:", rateLimit);  // rateLimit is a custom variable
},

[tobyab]

Hey! I've been trying to use experimental_StreamData, but have only been getting errors. Are there any other solutions?

[karar-shah]

IDK what exactly you wanna do but this might help!!!

Next.js Backend

import { OpenAIStream, StreamingTextResponse } from "ai";
import OpenAI from "openai";

      const response = await openai.chat.completions.create({
        model: "gpt-3.5-turbo",
        stream: true,
        messages: newPrompt,
      });
      // // Convert the response into a friendly text-stream
      const stream = OpenAIStream(response);
      // // Respond with the stream
      return new StreamingTextResponse(stream, {
        headers: { prompt_tokens: String(tokenList) },
      });

Next.js Frontend

import { useChat } from "ai/react";

  const { messages, input, handleInputChange, handleSubmit, setMessages } =
    useChat({
      body: {
        vocabularyData: moduleData,
        seed: Math.floor(Math.random() * 100) + 1,
      },
      onFinish: async (message) => {
        const enc = encodingForModel("gpt-4-0125-preview"); // js-tiktoken
        const tokenCount = enc.encode(message.content).length;
        console.log(`completion_tokens: ${tokenCount}`);
      },
      onResponse(response) {
        // Extracting headers from the response
        const headers = response.headers;
        const rateLimit = headers.get("prompt_tokens"); // Extracting the value of X-RATE-LIMIT header
        console.log("prompt_tokens ----:", rateLimit); // rateLimit is a custom variable
      },
    });

[cheraff]

useChat / onResponse is failing when trying to access response headers sent by StreamingTextResponse.

I'm able to get StreamingTextResponse to pass back headers no problem, but useChat (onResponse) and the code above by @karar-shah , specifically:

onResponse(response) { // Extracting headers from the response const headers = response.headers; const rateLimit = headers.get("prompt_tokens"); // Extracting the value of X-RATE-LIMIT header console.log("prompt_tokens ----:", rateLimit); // rateLimit is a custom variable }

Isn't working. Console showing correct response headers. onResponse showing
{type: 'basic', url: 'x', redirected: false, status: 200, ok: true, …} body : (...) bodyUsed : true headers : Headers {} ok : true redirected : false status : 200 statusText : "" type : "basic" url : "https://myurl.com"
Everytime.

[karar-shah]

Yup its working fine, make sure you are returning something in header from the API:

eg.

Below is Next.js POST api code snippet.

      // // Convert the response into a friendly text-stream
      const stream = OpenAIStream(response);
      // // Respond with the stream
      return new StreamingTextResponse(stream, {
        headers: { prompt_tokens: String(tokenList) },
      });

[cheraff]

Thanks @karar-shah, I'm streaming back the same way and via dev tools I can see the headers are set in the response, the issue is trying to read them

Stream API route app/api/chat: (confirmed to set response headers via dev tools)

  const stream = OpenAIStream(response);

        return new StreamingTextResponse(stream,  {
            headers: {
                 "Access-Control-Expose-Headers": "*",
                "Content-Type": "text/json",
                "Claim-Refresh": JSON.stringify(claimRefresh),
                "Authorization":  JSON.stringify(req.headers.get('Authorization')),
                "Firebase-Token": JSON.stringify(firebaseToken)
            },
        });

Client file that reads the stream app/page.tsx : (failing to read above)

  const {    messages, input, handleInputChange, handleSubmit } = useChat({
    body: {
       seed: Math.floor(Math.random() * 100) + 1,
    },
    onFinish:   (message) => {
  console.log("onFinish",message);
    },
    onResponse(response) {
      console.log(response);  

      // Extracting headers from the response
      const headers = response.headers;
      console.log("headers ----:", headers);  

    },
    api: '/api/chat',
    headers: {
      "X-User-ID":  `${user?.uid}`,
      "Authorization": `Bearer ${token}`
    }
  });

Using vercel/ai 3.0.24 - what version are you on? Scratching my head trying to see where else this could be losing the response

[karar-shah]

Its working fine nothing wrong, just start it from scratch!!!

"dependencies": {
    "ai": "^3.0.24",
    "next": "14.2.2",
    "openai": "^4.38.2",
    "react": "^18",
    "react-dom": "^18",
    "zod": "^3.23.0"
  }

Interf.tsx

"use client";
import { useChat } from "ai/react";
import React from "react";

export default function Interf1() {
  const { messages, input, handleInputChange, handleSubmit, setMessages } =
    useChat({
      onFinish: async (message) => {
        console.log("done:");
      },
      onResponse(response) {
        // Extracting headers from the response
        const headers = response.headers;
        const rateLimit = headers.get("prompt_tokens"); // Extracting the value of X-RATE-LIMIT header
        console.log("prompt_tokens ---:", rateLimit); // rateLimit is a custom variable
      },
    });
  return (
    <div>
      <div>
        {JSON.stringify(messages.length > 0 ? messages[0].content : "")}
      </div>
      <form
        onSubmit={(e: any) => {
          handleSubmit(e);
          // console.log(messages);
        }}
        className="flex items-center"
      >
        <textarea
          // type="text"
          value={input}
          onChange={handleInputChange}
          className="flex p-2 mr-2 border rounded-lg outline-none focus:border-indigo-400 text-black w-full min-h-[40px]  max-h-[130px] resize-none"
          placeholder="Type your answer..."
        />
        <button
          type="submit"
          className="px-4 py-2 bg-indigo-500 rounded-lg hover:bg-indigo-500 focus:outline-none h-16"
        >
          Send
        </button>
      </form>
    </div>
  );
}

api/chat/route.ts

import { OpenAIStream, StreamingTextResponse } from "ai";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export const runtime = "edge";

export async function POST(req: Request) {
  const response = await openai.chat.completions.create({
    model: "gpt-4-0125-preview",
    stream: true,
    messages: [
      {
        role: "system",
        content: `${"you are a helpful person"}`,
      },
      {
        role: "user",
        content: `${"say hi and introduce yoruself"}`,
      },
    ],

  });
  // // Convert the response into a friendly text-stream
  const stream = OpenAIStream(response);
  // // Respond with the stream
  return new StreamingTextResponse(stream, {
    headers: {
      prompt_tokens: String("this is a test header for prompt_tokens"),
    },
  });
}

[cheraff]

@karar-shah amazing, thank you! That helped me debug and realise the issue came from me not explicitly requesting the headers with headers.get("prompt_tokens")

I had expected console.log output of response and headers objects would have the header array visible, but apparently not.

A simple headers.get solved the issue.

Many thanks!

[jadejamig]

@karar-shah Thank you for your code above, really helped me. Got one more thing to ask if anyone knows how to handle return statements from api/chat/route.ts that is not a StreamingTextResponse.

Example, in my api/chat/route.ts I am doing some checking if a file exists in a database

const file = await prisma.file.findFirst();

if (!file) {
  return new NextResponse("Bad request", {status: 400})
}

// Do some things here

return new StreamingTextResponse(stream);

How do I handle in the frontend/client component if the file does not exist and the returns the bad request response? In the client side I want to show a Toast message.

[karar-shah]

@karar-shah Thank you for your code above, really helped me. Got one more thing to ask if anyone knows how to handle return statements from api/chat/route.ts that is not a StreamingTextResponse.

Example, in my api/chat/route.ts I am doing some checking if a file exists in a database

const file = await prisma.file.findFirst();

if (!file) {
  return new NextResponse("Bad request", {status: 400})
}

// Do some things here

return new StreamingTextResponse(stream);

How do I handle in the frontend/client component if the file does not exist and the returns the bad request response? In the client side I want to show a Toast message.

@jadejamig I am little short on time try manually decoding the stream on the front end. Hopefully this will allow you to return more customized responses and based on the return response run stream decode or not run stream decode.

1. Quick solution:

Return something in header and validate it before showing stream like:
return new StreamingTextResponse(stream, { headers: { prompt_tokens: String(tokenList) }, });

2. If things are more complex:

Then use custom code to decode the stream response and validate, may below code spark any idea for your scenario.

const decoder = new TextDecoder();

export async function decodeStreamResponse(
  response: Response,
  setMessages: any,
  message: any,
  decoder: any,
  id: string,
) {
  if (!response.ok) {
    throw new Error(response.statusText);
  }

  const data = response.body;
  if (!data) {
    return;
  }

  const reader = response.body.getReader();
  let done = false;
  var initialMessage = "";
  while (!done) {
    const { value, done: doneReading } = await reader.read();
    done = doneReading;
    const data = decoder.decode(value);
    initialMessage = initialMessage + data;

    message
      ? setMessages([
          ...message,
          { role: "assistant", content: initialMessage, id: id },
        ])
      : setMessages([{ role: "assistant", content: initialMessage, id: id }]);
  }

  const enc = encodingForModel("gpt-4-0125-preview"); // js-tiktoken
  const tokenList = enc.encode(initialMessage).length;
  console.log(`Token count: ${tokenList}`);
}