Cost Tracking Reference

This reference guide covers advanced topics: configuring custom models, understanding tokenizers, handling reasoning models, and using the Public API to manage model definitions.

Supported Tokenizers

ABV uses tokenizers to infer usage when token counts aren’t provided. The following tokenizers are currently supported:

Model Pattern	Tokenizer	Package	Notes
gpt-4o*	o200k_base	Built-in	Used for GPT-4o models
gpt*	cl100k_base	Built-in	Used for GPT-4, GPT-3.5, and other GPT models
claude*	claude	@anthropic-ai/tokenizer	Not fully accurate for Claude models per Anthropic

According to Anthropic, their tokenizer is not fully accurate for Claude models. For best results, ingest token counts directly from the Anthropic API response.

Predefined Models

ABV maintains a list of predefined popular models with tokenizers and pricing for:

• OpenAI - GPT-4, GPT-3.5, GPT-4o, and other models
• Anthropic - Claude 4 family (Opus, Sonnet, Haiku)
• Google - Gemini models

View the complete list of predefined models in the ABV UI (sign-in required). Navigate to Settings → Model Definitions.

Custom Model Definitions

You can add your own model definitions for:

• Custom or proprietary models
• Fine-tuned models
• Models not yet supported by ABV
• Overriding default pricing

Creating Models via the UI

1. Navigate to Settings → Model Definitions in the ABV UI
2. Click Add Model
3. Configure the model settings (see configuration details below)
4. Save the model definition

Creating Models via the API

Use the Models API to programmatically manage model definitions:

# List all models
GET /api/public/models

# Create a new model
POST /api/public/models

# Get a specific model
GET /api/public/models/{id}

# Delete a model
DELETE /api/public/models/{id}

View the complete Public API documentation

Model Configuration

Model Matching

Models are matched to generations using regular expressions:

Generation Attribute	Model Attribute	Description
model	match_pattern	Regex pattern to match model names

Example patterns:

# Exact match for GPT-4
(?i)^(gpt-4-0125-preview)$

# Match all GPT-4 variants
(?i)^gpt-4.*$

# Match custom model family
(?i)^my-custom-model-.*$

User-defined models take priority over ABV-maintained models. This allows you to override default pricing or add support for new models.

Pricing Configuration

Define pricing per usage type. Usage types must match exactly with the keys in the usage_details object:

{
  "pricing": {
    "input": 0.00001,           // Price per input token in USD
    "output": 0.00003,          // Price per output token in USD
    "cached_tokens": 0.000001   // Price per cached token in USD
  }
}

Usage type matching:

• Usage types are case-sensitive
• Must match exactly: "input" ≠ "Input"
• Supports arbitrary usage types for custom metrics

Tokenization Configuration

For models using the openai tokenizer, specify the tokenization config:

{
  "tokenizer": "openai",
  "tokenization_config": {
    "tokenizerModel": "gpt-4-0125-preview",  // tiktoken model name
    "tokensPerMessage": 4,                   // Required for chat models
    "tokensPerName": -1                      // Required for chat models
  }
}

Copy tokenization config from predefined OpenAI models in the UI. See the OpenAI tokenization guide for more details on tokensPerMessage and tokensPerName.

Complete Model Definition Example

Here’s a complete example of a custom model definition:

JSON

{
  "name": "My Custom GPT-4 Model",
  "match_pattern": "(?i)^my-custom-gpt-4$",
  "tokenizer": "openai",
  "tokenization_config": {
    "tokenizerModel": "gpt-4-0125-preview",
    "tokensPerMessage": 4,
    "tokensPerName": -1
  },
  "pricing": {
    "input": 0.00001,
    "output": 0.00003
  }
}

cURL

curl -X POST "https://app.abv.dev/api/public/models" \
  -H "Authorization: Bearer sk-abv-..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Custom GPT-4 Model",
    "match_pattern": "(?i)^my-custom-gpt-4$",
    "tokenizer": "openai",
    "tokenization_config": {
      "tokenizerModel": "gpt-4-0125-preview",
      "tokensPerMessage": 4,
      "tokensPerName": -1
    },
    "pricing": {
      "input": 0.00001,
      "output": 0.00003
    }
  }'

Reasoning Models

Reasoning models (like OpenAI’s o1 family) require special handling.

Why Inference Doesn’t Work

Reasoning models take multiple internal steps to arrive at a response:

1. Model generates reasoning tokens (internal thought process)
2. Model generates completion tokens (final response)
3. API bills you for: reasoning tokens + completion tokens

The problem: ABV cannot see the internal reasoning tokens, so it cannot accurately infer costs by tokenizing only the visible input and output.

Solution: Always Ingest Usage

For reasoning models, you must ingest usage details from the API response:

# Python example for OpenAI o1
response = openai_client.chat.completions.create(
    model="o1-preview",
    messages=[{"role": "user", "content": "Solve this problem..."}]
)

generation.update(
    usage_details={
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
        "completion_tokens_details": {
            "reasoning_tokens": response.usage.completion_tokens_details.reasoning_tokens
        }
    }
)

Critical: If you don’t ingest usage for reasoning models, ABV cannot calculate accurate costs. The tokenized input + output will significantly underestimate actual usage.

Learn more about OpenAI reasoning models

Historical Data and Model Changes

Retroactive Application

ABV does not retroactively apply model definition changes to historical generations. This includes:

• Price updates
• New tokenizers
• Changed model patterns

Why: Costs are calculated at ingestion time using the model and price information available at that moment. This ensures historical data remains accurate and auditable.

Batch Reprocessing

If you need to apply new model definitions to existing generations:

1. Contact ABV support
2. Request a batch job to reprocess generations
3. Specify the time range and model changes

Batch reprocessing is typically used when:

• Correcting a pricing error
• Adding missing model definitions
• Migrating to new tokenizers

OpenAI Usage Schema Compatibility

ABV supports the OpenAI usage schema for compatibility with OpenAI SDKs and tools.

Automatic Mapping

When you use OpenAI field names, ABV automatically maps them:

OpenAI Field	ABV Field
prompt_tokens	input
completion_tokens	output
total_tokens	total
prompt_tokens_details.*	input_*
completion_tokens_details.*	output_*

Example:

# You send this (OpenAI schema):
usage_details={
    "prompt_tokens": 100,
    "completion_tokens": 50,
    "prompt_tokens_details": {
        "cached_tokens": 20
    }
}

# ABV stores this:
{
    "input": 100,
    "output": 50,
    "input_cached_tokens": 20,
    "total": 150  # automatically calculated
}

This makes it easy to integrate with OpenAI without changing your existing code.

Requesting Official Model Support

Don’t want to maintain custom model definitions? Request official support:

1. Visit the ABV Model Support Request Form
2. Provide model details and documentation
3. ABV will add official support in a future release

Official support includes:

• Automatic tokenization
• Pricing updates
• Maintenance and updates

Troubleshooting

Usage and Cost Missing for Historical Data

Problem: After adding a new model definition, historical generations still show no usage or cost.

Solution: Model definitions are not applied retroactively. Options:

1. Request a batch reprocessing job from support
2. Accept that historical data uses old definitions
3. For future data: Ingest usage directly instead of relying on inference

Incorrect Costs

Problem: Costs don’t match your LLM provider’s billing.

Checklist:

1. Check pricing: Verify the model definition pricing matches your provider’s current rates
2. Check usage types: Ensure usage type keys match exactly (case-sensitive)
3. Check model matching: Verify the regex pattern correctly matches your model name
4. Consider timing: Prices are applied at ingestion time, not retroactively

Best practice: For billing-critical applications, always ingest both usage AND cost from the provider’s API response.

Model Not Matching

Problem: Generations aren’t matching your custom model definition.

Debug steps:

1. Check the model field in your generation matches the match_pattern regex
2. Test the regex pattern in a regex tester
3. Verify user-defined model isn’t being overridden by ABV model
4. Check for typos in the model name

Example:

// Generation model
"model": "gpt-4-custom-v1"

// Match pattern (correct)
"match_pattern": "(?i)^gpt-4-custom-v1$"

// Match pattern (incorrect - won't match)
"match_pattern": "^gpt-4-custom-v1$"  // missing (?i) flag

Tokenization Errors

Problem: Tokenization failing or producing unexpected results.

Solutions:

1. For Claude models: Ingest token counts from API instead of inferring
2. For OpenAI models: Verify tokensPerMessage and tokensPerName are set correctly
3. For custom tokenizers: Ensure the tokenizer is supported (see supported tokenizers table)

API Reference

Models API Endpoints

List all models:

GET /api/public/models

Create a model:

POST /api/public/models
Content-Type: application/json

{
  "name": "string",
  "match_pattern": "string",
  "tokenizer": "string",
  "tokenization_config": {},
  "pricing": {}
}

Get a specific model:

GET /api/public/models/{id}

Delete a model:

DELETE /api/public/models/{id}

View complete API documentation

Next Steps

Implementation Guide

Return to implementation examples

Daily Metrics API

Export usage and cost data