feat(nodes-langchain): Add token usage tracking and cost calculation … #22260
Conversation
…for AI Agent nodes Implements automatic token usage tracking and real-time cost estimation for AI Agent nodes, addressing a highly requested community feature. Features: - AgentTokenTracker callback handler that captures token usage from all LLM calls - Automatic accumulation of tokens across multi-turn conversations - Support for both 'tokenUsage' and 'estimatedTokenUsage' fields from different LLM providers - ModelPricingService with LiteLLM CDN integration for up-to-date pricing - 24-hour pricing cache to minimize network overhead - Fallback pricing for 15+ common models (GPT-4, Claude, Gemini) - Graceful handling of unknown models (returns null cost) Output format includes: - promptTokens, completionTokens, totalTokens - isEstimate flag for accuracy indicator - estimatedCost in USD - modelName for transparency Tests: - 26 unit tests for ModelPricingService - 7 unit tests for AgentTokenTracker - Integration tests for token tracking in agent execution - All tests passing (1560 total) Addresses community request from https://community.n8n.io/t/retrieve-llm-token-usage-in-ai-agents/68714
|
|
n8n-assistant
bot
added
community
|
Hey @nsouzaco, Thank you for your contribution. We appreciate the time and effort you’ve taken to submit this pull request. Before we can proceed, please ensure the following: Regarding new nodes: If your node integrates with an AI service that you own or represent, please email [email protected] and we will be happy to discuss the best approach. About review timelines: Thank you again for contributing to n8n. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 9 files
Prompt for AI agents (all 4 issues)
Understand the root cause of the following 4 issues and fix them.
<file name="packages/@n8n/nodes-langchain/utils/agent-execution/AgentTokenTracker.ts">
<violation number="1" location="packages/@n8n/nodes-langchain/utils/agent-execution/AgentTokenTracker.ts:19">
Rule violated: **Prefer Typeguards over Type casting**
Community PR Guidelines require following the TypeScript rule that forbids narrowing with `as`. Here the code coerces the union of `tokenUsage`/`estimatedTokenUsage` into a custom shape via an assertion, so the compiler cannot verify mismatches. Annotate the variable or guard the value instead of asserting.</violation>
<violation number="2" location="packages/@n8n/nodes-langchain/utils/agent-execution/AgentTokenTracker.ts:27">
Token usage entries with 0 prompt or completion tokens are ignored, causing under-reported totals and costs.</violation>
</file>
<file name="packages/@n8n/nodes-langchain/nodes/agents/Agent/agents/ToolsAgent/V3/helpers/runAgent.ts">
<violation number="1" location="packages/@n8n/nodes-langchain/nodes/agents/Agent/agents/ToolsAgent/V3/helpers/runAgent.ts:40">
Rule violated: **Prefer Typeguards over Type casting**
Casting `model` to `any` to read metadata bypasses the “Prefer Typeguards over Type casting” rule. Please add explicit typing (e.g., intersecting `BaseChatModel` with the optional metadata properties) or a type guard instead of erasing the type safety.</violation>
</file>
<file name="packages/@n8n/nodes-langchain/utils/test/modelPricing.test.ts">
<violation number="1" location="packages/@n8n/nodes-langchain/utils/test/modelPricing.test.ts:7">
Tests reset the service cache but never mock `fetch`, so every test hits the live LiteLLM CDN and depends on external pricing data instead of deterministic fixtures.</violation>
</file>
Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR
| async handleLLMEnd(output: LLMResult): Promise<void> { | ||
| // Extract token usage from LLMResult | ||
| // Some providers use 'tokenUsage', others use 'estimatedTokenUsage' | ||
| const tokenUsage = (output?.llmOutput?.tokenUsage || output?.llmOutput?.estimatedTokenUsage) as |
There was a problem hiding this comment.
Rule violated: Prefer Typeguards over Type casting
Community PR Guidelines require following the TypeScript rule that forbids narrowing with as. Here the code coerces the union of tokenUsage/estimatedTokenUsage into a custom shape via an assertion, so the compiler cannot verify mismatches. Annotate the variable or guard the value instead of asserting.
Prompt for AI agents
Address the following comment on packages/@n8n/nodes-langchain/utils/agent-execution/AgentTokenTracker.ts at line 19:
<comment>Community PR Guidelines require following the TypeScript rule that forbids narrowing with `as`. Here the code coerces the union of `tokenUsage`/`estimatedTokenUsage` into a custom shape via an assertion, so the compiler cannot verify mismatches. Annotate the variable or guard the value instead of asserting.</comment>
<file context>
@@ -0,0 +1,66 @@
+ async handleLLMEnd(output: LLMResult): Promise<void> {
+ // Extract token usage from LLMResult
+ // Some providers use 'tokenUsage', others use 'estimatedTokenUsage'
+ const tokenUsage = (output?.llmOutput?.tokenUsage || output?.llmOutput?.estimatedTokenUsage) as
+ | {
+ completionTokens?: number;
</file context>
| ): Promise<typeof tokenUsage & { estimatedCost?: number | null; modelName?: string }> { | ||
| // Get model name from the model instance | ||
| // Try multiple properties as different LLM providers use different naming | ||
| const modelAny = model as any; |
There was a problem hiding this comment.
Rule violated: Prefer Typeguards over Type casting
Casting model to any to read metadata bypasses the “Prefer Typeguards over Type casting” rule. Please add explicit typing (e.g., intersecting BaseChatModel with the optional metadata properties) or a type guard instead of erasing the type safety.
Prompt for AI agents
Address the following comment on packages/@n8n/nodes-langchain/nodes/agents/Agent/agents/ToolsAgent/V3/helpers/runAgent.ts at line 40:
<comment>Casting `model` to `any` to read metadata bypasses the “Prefer Typeguards over Type casting” rule. Please add explicit typing (e.g., intersecting `BaseChatModel` with the optional metadata properties) or a type guard instead of erasing the type safety.</comment>
<file context>
@@ -14,13 +14,59 @@ import {
+): Promise<typeof tokenUsage & { estimatedCost?: number | null; modelName?: string }> {
+ // Get model name from the model instance
+ // Try multiple properties as different LLM providers use different naming
+ const modelAny = model as any;
+
+ const modelNameRaw =
</file context>
| const modelAny = model as any; | |
| const modelAny: BaseChatModel & Partial<{ modelName: unknown; model: unknown; name: unknown; _modelName: unknown; caller: { model?: unknown }; client: { modelName?: unknown }; _modelType: () => unknown }> = model; |
| } | ||
| | undefined; | ||
|
|
||
| if (tokenUsage && tokenUsage.completionTokens && tokenUsage.promptTokens) { |
There was a problem hiding this comment.
Token usage entries with 0 prompt or completion tokens are ignored, causing under-reported totals and costs.
Prompt for AI agents
Address the following comment on packages/@n8n/nodes-langchain/utils/agent-execution/AgentTokenTracker.ts at line 27:
<comment>Token usage entries with 0 prompt or completion tokens are ignored, causing under-reported totals and costs.</comment>
<file context>
@@ -0,0 +1,66 @@
+ }
+ | undefined;
+
+ if (tokenUsage && tokenUsage.completionTokens && tokenUsage.promptTokens) {
+ // Actual token data from API
+ const isEstimate = !!output?.llmOutput?.estimatedTokenUsage && !output?.llmOutput?.tokenUsage;
</file context>
| if (tokenUsage && tokenUsage.completionTokens && tokenUsage.promptTokens) { | |
| if ( | |
| tokenUsage && | |
| tokenUsage.completionTokens !== undefined && | |
| tokenUsage.promptTokens !== undefined | |
| ) { |
| beforeEach(() => { | ||
| // Reset the singleton instance state before each test | ||
| // @ts-expect-error - accessing private property for testing | ||
| modelPricingService.pricingData = null; |
There was a problem hiding this comment.
Tests reset the service cache but never mock fetch, so every test hits the live LiteLLM CDN and depends on external pricing data instead of deterministic fixtures.
Prompt for AI agents
Address the following comment on packages/@n8n/nodes-langchain/utils/test/modelPricing.test.ts at line 7:
<comment>Tests reset the service cache but never mock `fetch`, so every test hits the live LiteLLM CDN and depends on external pricing data instead of deterministic fixtures.</comment>
<file context>
@@ -0,0 +1,231 @@
+ beforeEach(() => {
+ // Reset the singleton instance state before each test
+ // @ts-expect-error - accessing private property for testing
+ modelPricingService.pricingData = null;
+ // @ts-expect-error - accessing private property for testing
+ modelPricingService.lastFetchTime = 0;
</file context>
2 participants
Summary
This PR implements automatic token usage tracking and real-time cost calculation for AI Agent nodes, addressing a highly requested community feature with 20+ active members requesting this functionality.
What it does
AI Agent nodes now automatically track and report:
How it works
Output Example
{ "output": "I found 3 relevant documents for you...", "tokenUsage": { "promptTokens": 320, "completionTokens": 180, "totalTokens": 500, "isEstimate": false, "estimatedCost": 0.000705, "modelName": "gpt-4o-mini" } }Use Cases Enabled
How to test
tokenUsagefield with all tracking dataSupported Models:
Related Linear tickets, Github issues, and Community forum posts
Review / Merge checklist
Implementation Details
Files Changed:
AgentTokenTracker.ts- LangChain callback handler for token capturemodelPricing.ts- Pricing service with CDN integration and fallback datatypes.ts- Extended AgentResult and RequestResponseMetadata typesbuildResponseMetadata.ts- Token accumulation across iterationsprepareItemContext.ts- TokenTracker initializationrunAgent.ts- Cost calculation integrationKey Features:
Pricing Data Source
Breaking Changes
None - this feature is opt-in and fully backward compatible.