Langsmith is incorrectly tracing tokens

78 Views Asked by At

I have a simple langgraph chain in place and I noticed that the counting of tokens is oddly off in langsmith in comperison to OpenAI online tokenizer or Python tokenizer:

Langsmith tokens (2,067): enter image description here

Python program:

import tiktoken

def num_tokens_from_string(string: str) -> int:
    encoding = tiktoken.get_encoding("cl100k_base")
    num_tokens = len(encoding.encode(string))
    return num_tokens

test_string = """ Ctrl+c ctrl+v from langsmith trace """
print(num_tokens_from_string(test_string))

Output: 11185

OpenAI tokenizer: enter image description here

Question:

  1. Why are the token counts different for langsmith and OpenAI?
  2. How do I send / set in Langsmith correct counting token methods? For this python code: response = ChatOpenAI().invoke("Hello!")
0

There are 0 best solutions below