AI bot traffic tracking

Bot traffic is a Talivia Cloud server-side feature. It records recognized crawler requests in a dedicated data set, separate from human pageviews, visitors, conversions, and revenue.

A recorded request shows that a matching crawler requested a URL. It does not prove citation, recommendation, use in an answer, cryptographic identity, or revenue impact.

1. Create a website token

Open Website settings → Bot traffic, enable collection, and generate a token. Each token belongs to one website. Copy it immediately because it is displayed only once, and store it as a server-side secret. Rotating it revokes the previous token.

pnpm add @talivia/bot-traffic

TALIVIA_BOT_TOKEN=<one-time-token>

Next.js middleware or proxy

Next middleware runs before the final page response. Use request-only tracking: NextResponse.next() is not the final page status and must not be reported as one.

import { NextResponse } from 'next/server';
import { trackBotRequestInBackground } from '@talivia/bot-traffic/next';

export function middleware(request, event) {
  trackBotRequestInBackground(
    request,
    { token: process.env.TALIVIA_BOT_TOKEN! },
    event,
  );
  return NextResponse.next();
}

export const config = { matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)'] };

Express

The Express adapter calls next() immediately and reports the final response status after the finish event.

import express from 'express';
import { createExpressBotMiddleware } from '@talivia/bot-traffic/express';

const app = express();
app.use(createExpressBotMiddleware({
  token: process.env.TALIVIA_BOT_TOKEN!,
}));

Cloudflare Workers and Pages

import { withBotTracking } from '@talivia/bot-traffic/cloudflare';

export default {
  fetch: withBotTracking(
    (request, env) => env.ASSETS.fetch(request),
    { token: '<server-token>' },
  ),
};

For Pages Functions, process context.request, then call trackBotRequestInBackground with the final status and pass context as the third argument so waitUntil keeps delivery alive.

Any JavaScript or TypeScript framework

No named adapter is required when the framework exposes standard Request and Response objects:

import { trackBotRequestInBackground } from '@talivia/bot-traffic';

export async function handle(request, context) {
  const response = await yourFrameworkHandler(request);
  trackBotRequestInBackground(
    request,
    { token: process.env.TALIVIA_BOT_TOKEN!, status: response.status },
    context, // optional object with waitUntil(promise)
  );
  return response;
}

If only the incoming request is available, omit status. Request-only tracking still records which page the crawler attempted to fetch. Use withBotTracking only around a handler that returns the final Response. Use the awaited trackBotRequest function for tests or controlled background jobs, not on a latency-sensitive response path.

This generic pattern applies to Hono, Bun, Deno, custom Node.js Fetch servers, serverless functions, and other Fetch-compatible runtimes.

Docker, Cloud Run, and reverse proxies

If request.url contains an internal host such as http://app:3000/docs, set the real public origin:

{
  token: process.env.TALIVIA_BOT_TOKEN!,
  publicOrigin: 'https://example.com',
}

The package replaces only the origin and preserves the request path. publicOrigin must be an http or https origin without credentials, a path, query, or fragment. Talivia still validates the public hostname against the website that owns the token.

PHP, Go, Ruby, and other languages

The JavaScript package is optional. Any server language can send this minimal event to the fixed endpoint:

POST https://talivia.com/v1/bot-traffic
Authorization: Bearer <website-bot-token>
Content-Type: application/json

{
  "schemaVersion": 1,
  "hostname": "example.com",
  "path": "/docs/getting-started",
  "method": "GET",
  "userAgent": "GPTBot/1.0",
  "status": 200
}

status is optional. Talivia Cloud derives provider, bot, category, and verification itself; do not send those fields.

PHP example

Invoke this helper only after the application has sent the response. With PHP-FPM, call fastcgi_finish_request() first when available.

function trackTaliviaBot(array $event, string $token): void {
    $ch = curl_init('https://talivia.com/v1/bot-traffic');
    curl_setopt_array($ch, [
        CURLOPT_POST => true,
        CURLOPT_POSTFIELDS => json_encode($event, JSON_UNESCAPED_SLASHES),
        CURLOPT_HTTPHEADER => [
            'Authorization: Bearer ' . $token,
            'Content-Type: application/json',
        ],
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT_MS => 1500,
    ]);
    curl_exec($ch);
    curl_close($ch);
}

// After your framework has produced and sent the final response:
if (function_exists('fastcgi_finish_request')) {
    fastcgi_finish_request();
}
trackTaliviaBot($event, getenv('TALIVIA_BOT_TOKEN'));

Call the helper only for likely crawler GET or HEAD document requests after the final status is known and the user response has been sent. Send a pathname without query strings or fragments. Do not send cookies, authorization headers, request bodies, visitor identifiers, referrers, or source IP.

Required custom-integration behavior

• Endpoint: https://talivia.com/v1/bot-traffic.
• Authentication: Authorization: Bearer <website-bot-token>.
• Content type: application/json.
• Required fields: schemaVersion, hostname, path, method, userAgent.
• Optional field: status from 100 to 599.
• Methods: document-like GET and HEAD requests.
• Keep payloads at or below 8 KiB and User-Agent at or below 512 characters.
• Strip query strings and fragments from path.
• Use a short timeout and fail open. A telemetry failure must never fail or delay the website response.
• Do not trust User-Agent as proof of identity. Talivia performs central classification and network verification.

After deployment, use the detailed report to review request trends, providers, categories, pages, status codes, missing pages, and evidence-based verification confidence.