Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

community [enh] : adds callback handler for Fiddler AI #17708

Merged
merged 1 commit into from
Feb 26, 2024
Merged

community [enh] : adds callback handler for Fiddler AI #17708

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
215 changes: 215 additions & 0 deletions docs/docs/integrations/callbacks/fiddler.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "0cebf93b",
"metadata": {},
"source": [
"## Fiddler Langchain integration Quick Start Guide\n",
"\n",
"Fiddler is the pioneer in enterprise Generative and Predictive system ops, offering a unified platform that enables Data Science, MLOps, Risk, Compliance, Analytics, and other LOB teams to monitor, explain, analyze, and improve ML deployments at enterprise scale. "
]
},
{
"cell_type": "markdown",
"id": "38d746c2",
"metadata": {},
"source": [
"## 1. Installation and Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e0151955",
"metadata": {},
"outputs": [],
"source": [
"# langchain>=0.1.7 langchain-openai fiddler-client"
]
},
{
"cell_type": "markdown",
"id": "5662f2e5-d510-4eef-b44b-fa929e5b4ad4",
"metadata": {},
"source": [
"## 2. Fiddler connection details "
]
},
{
"cell_type": "markdown",
"id": "64fac323",
"metadata": {},
"source": [
"*Before you can add information about your model with Fiddler*\n",
"\n",
"1. The URL you're using to connect to Fiddler\n",
"2. Your organization ID\n",
"3. Your authorization token\n",
"\n",
"These can be found by navigating to the *Settings* page of your Fiddler environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f6f8b73e-d350-40f0-b7a4-fb1e68a65a22",
"metadata": {},
"outputs": [],
"source": [
"URL = \"\" # Your Fiddler instance URL, Make sure to include the full URL (including https://). For example: https://demo.fiddler.ai\n",
"ORG_NAME = \"\"\n",
"AUTH_TOKEN = \"\" # Your Fiddler instance auth token\n",
"\n",
"# Fiddler project and model names, used for model registration\n",
"PROJECT_NAME = \"\"\n",
"MODEL_NAME = \"\" # Model name in Fiddler"
]
},
{
"cell_type": "markdown",
"id": "0645805a",
"metadata": {},
"source": [
"## 3. Create a fiddler callback handler instance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13de4f9a",
"metadata": {},
"outputs": [],
"source": [
"from langchain.callbacks.fiddler_callback import FiddlerCallback\n",
"\n",
"fiddler_handler = FiddlerCallback(\n",
" url=URL,\n",
" org_name=ORG_NAME,\n",
" project_name=PROJECT_NAME,\n",
" model_name=MODEL_NAME,\n",
" auth_token=AUTH_TOKEN,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "2276368e-f1dc-46be-afe3-18796e7a66f2",
"metadata": {},
"source": [
"## Example 1 : Basic Chain"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9de0fd1",
"metadata": {},
"outputs": [],
"source": [
"from langchain_core.output_parsers import StrOutputParser\n",
"from langchain_openai import OpenAI\n",
"\n",
"# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEY\n",
"llm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])\n",
"output_parser = StrOutputParser()\n",
"\n",
"chain = llm | output_parser\n",
"\n",
"# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generated\n",
"chain.invoke(\"How far is moon from earth?\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "309bde0b-e1ce-446c-98ac-3690c26a2676",
"metadata": {},
"outputs": [],
"source": [
"# Few more invocations\n",
"chain.invoke(\"What is the temperature on Mars?\")\n",
"chain.invoke(\"How much is 2 + 200000?\")\n",
"chain.invoke(\"Which movie won the oscars this year?\")\n",
"chain.invoke(\"Can you write me a poem about insomnia?\")\n",
"chain.invoke(\"How are you doing today?\")\n",
"chain.invoke(\"What is the meaning of life?\")"
]
},
{
"cell_type": "markdown",
"id": "48fa4782-c867-4510-9430-4ffa3de3b5eb",
"metadata": {},
"source": [
"## Example 2 : Chain with prompt templates"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2aa2c220-8946-4844-8d3c-8f69d744d13f",
"metadata": {},
"outputs": [],
"source": [
"from langchain.prompts import (\n",
" ChatPromptTemplate,\n",
" FewShotChatMessagePromptTemplate,\n",
")\n",
"\n",
"examples = [\n",
" {\"input\": \"2+2\", \"output\": \"4\"},\n",
" {\"input\": \"2+3\", \"output\": \"5\"},\n",
"]\n",
"\n",
"example_prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\"human\", \"{input}\"),\n",
" (\"ai\", \"{output}\"),\n",
" ]\n",
")\n",
"\n",
"few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
" example_prompt=example_prompt,\n",
" examples=examples,\n",
")\n",
"\n",
"final_prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\"system\", \"You are a wondrous wizard of math.\"),\n",
" few_shot_prompt,\n",
" (\"human\", \"{input}\"),\n",
" ]\n",
")\n",
"\n",
"# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEY\n",
"llm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])\n",
"\n",
"chain = final_prompt | llm\n",
"\n",
"# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generated\n",
"chain.invoke({\"input\": \"What's the square of a triangle?\"})"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
2 changes: 2 additions & 0 deletions libs/community/langchain_community/callbacks/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from langchain_community.callbacks.clearml_callback import ClearMLCallbackHandler
from langchain_community.callbacks.comet_ml_callback import CometCallbackHandler
from langchain_community.callbacks.context_callback import ContextCallbackHandler
from langchain_community.callbacks.fiddler_callback import FiddlerCallbackHandler
from langchain_community.callbacks.flyte_callback import FlyteCallbackHandler
from langchain_community.callbacks.human import HumanApprovalCallbackHandler
from langchain_community.callbacks.infino_callback import InfinoCallbackHandler
Expand Down Expand Up @@ -63,4 +64,5 @@
"SageMakerCallbackHandler",
"LabelStudioCallbackHandler",
"TrubricsCallbackHandler",
"FiddlerCallbackHandler",
]