Skip to content

Commit

Permalink
community[patch]: invoke callback prior to yielding token (llama.cpp) (
Browse files Browse the repository at this point in the history
…langchain-ai#19392)

**Description:** Invoke callback prior to yielding token for llama.cpp
**Issue:** [Callback for on_llm_new_token should be invoked before the
token is yielded by the model
langchain-ai#16913](langchain-ai#16913)
**Dependencies:** None
  • Loading branch information
sepiatone authored and Dave Bechberger committed Mar 29, 2024
1 parent 5a8cce9 commit 79b89b4
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion libs/community/langchain_community/llms/llamacpp.py
Original file line number Diff line number Diff line change
Expand Up @@ -344,11 +344,11 @@ def _stream(
text=part["choices"][0]["text"],
generation_info={"logprobs": logprobs},
)
yield chunk
if run_manager:
run_manager.on_llm_new_token(
token=chunk.text, verbose=self.verbose, log_probs=logprobs
)
yield chunk

def get_num_tokens(self, text: str) -> int:
tokenized_text = self.client.tokenize(text.encode("utf-8"))
Expand Down

0 comments on commit 79b89b4

Please sign in to comment.