-
Notifications
You must be signed in to change notification settings - Fork 13.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update agents to use tool call messages #20074
Merged
Merged
Changes from all commits
Commits
Show all changes
101 commits
Select commit
Hold shift + click to select a range
d68a71c
rfc: tool calls msg
baskaryan 2319f42
fmt
baskaryan f6048cc
fmt
baskaryan 67e3d1e
fmt
baskaryan 35189ad
merge master
ccurme 0c14c48
ToolMessage --> ToolCallsMessage in openai
ccurme 684e107
lint
ccurme b6dc899
Merge branch 'master' into bagatur/tool_calls_msg
ccurme 2b373cd
factor out parse_partial_json and parse_tool_calls
ccurme 28bc99e
add ToolCallsMessageChunk
ccurme ea74d44
add to openai
ccurme 6dd3ff7
format
ccurme 2692239
lint
ccurme 1849f93
accept None
ccurme 10b2c7b
update test_imports
ccurme 1f637b9
ToolMessage --> ToolOutputMessage in tests
ccurme 3605678
update snapshots
ccurme 9540adf
Merge branch 'master' into bagatur/tool_calls_msg
ccurme 16bdda4
add ToolCall to module init
ccurme 8698a86
add to integration test
ccurme c7d00cc
lint
ccurme 808cf82
update serializable mapping
ccurme 0d6118a
update mistral
ccurme 6c0e3ea
update test_imports
ccurme 508e3bc
ToolOutputMessage --> ToolMessage
ccurme 189600f
update snapshots
ccurme 360f9e2
lint
ccurme ca59616
clean up ToolMessage
ccurme bda88a6
update JsonOutputToolsParser
ccurme ac4a0da
lint
ccurme 0b515c8
Merge branch 'master' into bagatur/tool_calls_msg
ccurme 6d81051
Merge branch 'master' into bagatur/tool_calls_msg
ccurme fec6db2
cr
ccurme c5ec7fd
update ToolCallsMessageChunk.__add__
ccurme 33127d3
move tool calls msgs to ai
ccurme e1fb61b
update AIMessageChunk.__add__
ccurme b1e9235
move parse_tool_calls
ccurme 75d11dc
fix bug
ccurme 9d0176f
lint
ccurme e225577
rename ToolCallsMessage
ccurme c27c9e8
rename ToolCallsMessageChunk
ccurme 5765f48
lint
ccurme 4eb4c34
merge tool calls
ccurme 9089029
fmt (#19968)
baskaryan a8b2733
fix bug
ccurme a29c6b1
catch KeyError
ccurme 3ffcf0a
update mistral streaming
ccurme 1b53ef6
cr
ccurme ed71599
update mistral test
ccurme 48d9355
remove check
ccurme d140449
update docstring
ccurme 5ea8bb4
add tests
ccurme 2060f37
add tests
ccurme ff07346
update fireworks
ccurme 71764a9
update cohere and add test
ccurme cbf66ec
add to openai tests
ccurme f45caa0
fix bug
ccurme a49f23e
Merge branch 'master' into bagatur/tool_calls_msg
ccurme 9a05cba
update groq
ccurme bc85987
update agents
ccurme 2ea5d66
Merge branch 'master' into bagatur/tool_calls_msg
baskaryan 6103cd4
Merge branch 'bagatur/tool_calls_msg' of github.com:langchain-ai/lang…
ccurme 9ff7ae9
Revert "update agents"
ccurme e4ca284
Merge branch 'master' into bagatur/tool_calls_msg
ccurme e012414
update anthropic
ccurme 53138b9
cr
ccurme a79a980
use tool call msgs (#20051)
baskaryan bfe8fe3
update docstrings
ccurme ce684dd
spell check
ccurme 13222fa
export json output parsers in langchain
ccurme e845536
undo stray import changes
ccurme 1f902d2
tweak docstring
ccurme 62c8f6e
update agent tools parser
ccurme 46c9be9
update agents
ccurme 7fcff8b
fix test
ccurme b490c57
move tool calls to AIMessage (#20090)
ccurme 34e629d
merge (delete cohere)
ccurme 94bdf1c
update
ccurme a1feb7d
merge
ccurme f8b3b82
Merge branch 'master' into bagatur/tool_calls_msg
ccurme 52b7531
best effort parsing + handle parsing errors (#20111)
ccurme bb37dd7
fix anthropic
ccurme bcffbc4
Merge branch 'bagatur/tool_calls_msg' into cc/tool_calls_agents
ccurme d5f4f53
update agents
ccurme a809ef4
add create_tools_agent
ccurme d3caec1
add type hint
ccurme 893e9a9
add type hint
ccurme f330a22
Merge branch 'cc/tool_calls_agents' of github.com:langchain-ai/langch…
ccurme 12955b2
type hint
ccurme 225f519
cr
ccurme fa3a8c0
update tool calls to typeddict (#20208)
ccurme 3d243d4
fix docstring
ccurme 0cc3142
update (#20215)
ccurme c3f2805
fix bug
ccurme bbe9c0f
Merge branch 'bagatur/tool_calls_msg' into cc/tool_calls_agents
ccurme 55affdf
update
ccurme 1d937b7
fmt
baskaryan aa8cfa0
fmt
baskaryan 1e0aff8
fmt
baskaryan e621293
tools_agent --> tool_calling_agent
ccurme 3f4015b
Merge branch 'cc/tool_calls_agents' of github.com:langchain-ai/langch…
ccurme File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
60 changes: 3 additions & 57 deletions
60
libs/langchain/langchain/agents/format_scratchpad/openai_tools.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,59 +1,5 @@ | ||
import json | ||
from typing import List, Sequence, Tuple | ||
|
||
from langchain_core.agents import AgentAction | ||
from langchain_core.messages import ( | ||
AIMessage, | ||
BaseMessage, | ||
ToolMessage, | ||
from langchain.agents.format_scratchpad.tools import ( | ||
format_to_tool_messages as format_to_openai_tool_messages, | ||
) | ||
|
||
from langchain.agents.output_parsers.openai_tools import OpenAIToolAgentAction | ||
|
||
|
||
def _create_tool_message( | ||
agent_action: OpenAIToolAgentAction, observation: str | ||
) -> ToolMessage: | ||
"""Convert agent action and observation into a function message. | ||
Args: | ||
agent_action: the tool invocation request from the agent | ||
observation: the result of the tool invocation | ||
Returns: | ||
FunctionMessage that corresponds to the original tool invocation | ||
""" | ||
if not isinstance(observation, str): | ||
try: | ||
content = json.dumps(observation, ensure_ascii=False) | ||
except Exception: | ||
content = str(observation) | ||
else: | ||
content = observation | ||
return ToolMessage( | ||
tool_call_id=agent_action.tool_call_id, | ||
content=content, | ||
additional_kwargs={"name": agent_action.tool}, | ||
) | ||
|
||
|
||
def format_to_openai_tool_messages( | ||
intermediate_steps: Sequence[Tuple[AgentAction, str]], | ||
) -> List[BaseMessage]: | ||
"""Convert (AgentAction, tool output) tuples into FunctionMessages. | ||
|
||
Args: | ||
intermediate_steps: Steps the LLM has taken to date, along with observations | ||
|
||
Returns: | ||
list of messages to send to the LLM for the next prediction | ||
|
||
""" | ||
messages = [] | ||
for agent_action, observation in intermediate_steps: | ||
if isinstance(agent_action, OpenAIToolAgentAction): | ||
new_messages = list(agent_action.message_log) + [ | ||
_create_tool_message(agent_action, observation) | ||
] | ||
messages.extend([new for new in new_messages if new not in messages]) | ||
else: | ||
messages.append(AIMessage(content=agent_action.log)) | ||
return messages | ||
__all__ = ["format_to_openai_tool_messages"] |
59 changes: 59 additions & 0 deletions
59
libs/langchain/langchain/agents/format_scratchpad/tools.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
import json | ||
from typing import List, Sequence, Tuple | ||
|
||
from langchain_core.agents import AgentAction | ||
from langchain_core.messages import ( | ||
AIMessage, | ||
BaseMessage, | ||
ToolMessage, | ||
) | ||
|
||
from langchain.agents.output_parsers.tools import ToolAgentAction | ||
|
||
|
||
def _create_tool_message( | ||
agent_action: ToolAgentAction, observation: str | ||
) -> ToolMessage: | ||
"""Convert agent action and observation into a function message. | ||
Args: | ||
agent_action: the tool invocation request from the agent | ||
observation: the result of the tool invocation | ||
Returns: | ||
FunctionMessage that corresponds to the original tool invocation | ||
""" | ||
if not isinstance(observation, str): | ||
try: | ||
content = json.dumps(observation, ensure_ascii=False) | ||
except Exception: | ||
content = str(observation) | ||
else: | ||
content = observation | ||
return ToolMessage( | ||
tool_call_id=agent_action.tool_call_id, | ||
content=content, | ||
additional_kwargs={"name": agent_action.tool}, | ||
) | ||
|
||
|
||
def format_to_tool_messages( | ||
intermediate_steps: Sequence[Tuple[AgentAction, str]], | ||
) -> List[BaseMessage]: | ||
"""Convert (AgentAction, tool output) tuples into FunctionMessages. | ||
|
||
Args: | ||
intermediate_steps: Steps the LLM has taken to date, along with observations | ||
|
||
Returns: | ||
list of messages to send to the LLM for the next prediction | ||
|
||
""" | ||
messages = [] | ||
for agent_action, observation in intermediate_steps: | ||
if isinstance(agent_action, ToolAgentAction): | ||
new_messages = list(agent_action.message_log) + [ | ||
_create_tool_message(agent_action, observation) | ||
] | ||
messages.extend([new for new in new_messages if new not in messages]) | ||
else: | ||
messages.append(AIMessage(content=agent_action.log)) | ||
return messages |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
76 changes: 23 additions & 53 deletions
76
libs/langchain/langchain/agents/output_parsers/openai_tools.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
102 changes: 102 additions & 0 deletions
102
libs/langchain/langchain/agents/output_parsers/tools.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
import json | ||
from json import JSONDecodeError | ||
from typing import List, Union | ||
|
||
from langchain_core.agents import AgentAction, AgentActionMessageLog, AgentFinish | ||
from langchain_core.exceptions import OutputParserException | ||
from langchain_core.messages import ( | ||
AIMessage, | ||
BaseMessage, | ||
ToolCall, | ||
) | ||
from langchain_core.outputs import ChatGeneration, Generation | ||
|
||
from langchain.agents.agent import MultiActionAgentOutputParser | ||
|
||
|
||
class ToolAgentAction(AgentActionMessageLog): | ||
tool_call_id: str | ||
"""Tool call that this message is responding to.""" | ||
|
||
|
||
def parse_ai_message_to_tool_action( | ||
message: BaseMessage, | ||
) -> Union[List[AgentAction], AgentFinish]: | ||
"""Parse an AI message potentially containing tool_calls.""" | ||
if not isinstance(message, AIMessage): | ||
raise TypeError(f"Expected an AI message got {type(message)}") | ||
|
||
actions: List = [] | ||
if message.tool_calls: | ||
tool_calls = message.tool_calls | ||
else: | ||
if not message.additional_kwargs.get("tool_calls"): | ||
return AgentFinish( | ||
return_values={"output": message.content}, log=str(message.content) | ||
) | ||
# Best-effort parsing | ||
tool_calls = [] | ||
for tool_call in message.additional_kwargs["tool_calls"]: | ||
function = tool_call["function"] | ||
function_name = function["name"] | ||
try: | ||
args = json.loads(function["arguments"] or "{}") | ||
tool_calls.append( | ||
ToolCall(name=function_name, args=args, id=tool_call["id"]) | ||
) | ||
except JSONDecodeError: | ||
raise OutputParserException( | ||
f"Could not parse tool input: {function} because " | ||
f"the `arguments` is not valid JSON." | ||
) | ||
for tool_call in tool_calls: | ||
# HACK HACK HACK: | ||
# The code that encodes tool input into Open AI uses a special variable | ||
# name called `__arg1` to handle old style tools that do not expose a | ||
# schema and expect a single string argument as an input. | ||
# We unpack the argument here if it exists. | ||
# Open AI does not support passing in a JSON array as an argument. | ||
function_name = tool_call["name"] | ||
_tool_input = tool_call["args"] | ||
if "__arg1" in _tool_input: | ||
tool_input = _tool_input["__arg1"] | ||
else: | ||
tool_input = _tool_input | ||
|
||
content_msg = f"responded: {message.content}\n" if message.content else "\n" | ||
log = f"\nInvoking: `{function_name}` with `{tool_input}`\n{content_msg}\n" | ||
actions.append( | ||
ToolAgentAction( | ||
tool=function_name, | ||
tool_input=tool_input, | ||
log=log, | ||
message_log=[message], | ||
tool_call_id=tool_call["id"], | ||
) | ||
) | ||
return actions | ||
|
||
|
||
class ToolsAgentOutputParser(MultiActionAgentOutputParser): | ||
"""Parses a message into agent actions/finish. | ||
|
||
If a tool_calls parameter is passed, then that is used to get | ||
the tool names and tool inputs. | ||
|
||
If one is not passed, then the AIMessage is assumed to be the final output. | ||
""" | ||
|
||
@property | ||
def _type(self) -> str: | ||
return "tools-agent-output-parser" | ||
|
||
def parse_result( | ||
self, result: List[Generation], *, partial: bool = False | ||
) -> Union[List[AgentAction], AgentFinish]: | ||
if not isinstance(result[0], ChatGeneration): | ||
raise ValueError("This output parser only works on ChatGeneration output") | ||
message = result[0].message | ||
return parse_ai_message_to_tool_action(message) | ||
|
||
def parse(self, text: str) -> Union[List[AgentAction], AgentFinish]: | ||
raise ValueError("Can only parse messages") |
Empty file.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't think it matters too much in this particular case since actions aren't really serialized but still technically breaking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea probably shouldn't change namespace
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or should support both
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated