extra_tools argument in create_pandas_dataframe_agent() doesn't seem to be working

[PyroGenesis]

System Info

Platform: Windows Server 2022
Python: 3.11.6
Langchain version: 0.0.306

Who can help?

@agola11 @hwchase17

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Creating a test tool

from langchain.agents import Tool

tools = [
    Tool(
        name="test_tool",
        func=print,
        description="This is a test tool"
    )
]
tools

[Tool(name='test_tool', description='This is a test tool', args_schema=None, return_direct=False, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, handle_tool_error=False, func=<built-in function print>, coroutine=None)]

Setting up the pandas_dataframe_agent

from langchain.agents import create_pandas_dataframe_agent
from langchain.llms import HuggingFacePipeline
import pandas as pd

llm = HuggingFacePipeline.from_model_id(
    model_id="google/flan-t5-small", task="text2text-generation", device=0
)

agent = create_pandas_dataframe_agent(llm, pd.DataFrame(), verbose=True, extra_tools=tools)
agent.tools

[PythonAstREPLTool(name='python_repl_ast', description='A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.', args_schema=<class 'langchain.tools.python.tool.PythonInputs'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, handle_tool_error=False, globals={}, locals={'df': Empty DataFrame
 Columns: []
 Index: []}, sanitize_input=True),
 Tool(name='test_tool', description='This is a test tool', args_schema=None, return_direct=False, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, handle_tool_error=False, func=<built-in function print>, coroutine=None)]

Executing agent with debugging enabled

import langchain
langchain.debug = True

agent.run('What is 2+2?')

[chain/start] [1:chain:AgentExecutor] Entering Chain run with input:
{
  "input": "What is 2+2?"
}
[chain/start] [1:chain:AgentExecutor > 2:chain:LLMChain] Entering Chain run with input:
{
  "input": "What is 2+2?",
  "agent_scratchpad": "",
  "stop": [
    "\nObservation:",
    "\n\tObservation:"
  ]
}
[llm/start] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:HuggingFacePipeline] Entering LLM run with input:
{
  "prompts": [
    "You are working with a pandas dataframe in Python. The name of the dataframe is `df`.\nYou should use the tools below to answer the question posed of you:\n\npython_repl_ast: A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [python_repl_ast]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\n\nThis is the result of `print(df.head())`:\n\n\nBegin!\nQuestion: What is 2+2?"
  ]
}

The prompt from the above log

You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
You should use the tools below to answer the question posed of you:

python_repl_ast: A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [python_repl_ast]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question


This is the result of `print(df.head())`:


Begin!
Question: What is 2+2?

Expected behavior

Where did my custom tool test_tool disappear? I expected it to show up after python_repl_ast?

[dosubot]

🤖

Hello,

Thank you for bringing this to our attention. From your description, it seems like you're expecting the test_tool to be included in the prompt when you run the agent. However, the extra_tools argument in the create_pandas_dataframe_agent() function is used to extend the base tools used by the agent, not to modify the prompt.

The prompt is generated based on the agent type and does not dynamically include the names of the extra tools. In your case, the agent type is AgentType.ZERO_SHOT_REACT_DESCRIPTION, and the prompt is created by the _get_prompt_and_tools() function. This function does not include the names of the extra_tools in the prompt.

If you want to include the test_tool in the prompt, you would need to modify the _get_prompt_and_tools() function to include the names of the extra_tools in the prompt. Here's an example of how you might do this:

def _get_prompt_and_tools(self, extra_tools):
    prompt = "You are working with a pandas dataframe in Python. The name of the dataframe is `df`.\nYou should use the tools below to answer the question posed of you:\n\n"
    for tool in extra_tools:
        prompt += f"{tool.name}: {tool.description}\n"
    # ... rest of the function
    return prompt, tools

This will include the names and descriptions of the extra_tools in the prompt. Please note that this is a simplified example and you might need to adjust it to fit your specific needs.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/langchain/langchain/agents/agent_toolkits/pandas/base.py
• docs/extras/modules/agents/how_to/custom_agent_with_tool_retrieval.ipynb

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[PyroGenesis]

Hmm, I don't really see the point of extending the basic tools but not telling the model about them in the prompt. How is the model supposed to use the new tools if it is not provided information on it?

If you are open to it, I can try and make a pull request that integrates these custom tools into the prompt.

[utkucanaytac]

Hello, i think i managed to add tools to the pandas agent. Please check https://stackoverflow.com/a/77941942/12906920 Maybe it will help.