# AI bot traffic tracking (https://talivia.com/docs/integrations/bot-traffic)



# AI bot traffic tracking [#ai-bot-traffic-tracking]

Bot traffic is a **Talivia Cloud** server-side feature. It records recognized crawler requests in a dedicated data set, separate from human pageviews, visitors, conversions, and revenue.

A recorded request shows that a matching crawler requested a URL. It does not prove citation, recommendation, use in an answer, cryptographic identity, or revenue impact.

## 1. Create a website token [#1-create-a-website-token]

Open **Website settings → Bot traffic**, enable collection, and generate a token. Each token belongs to one website. Copy it immediately because it is displayed only once, and store it as a server-side secret. Rotating it revokes the previous token.

```bash
pnpm add @talivia/bot-traffic
```

```bash
TALIVIA_BOT_TOKEN=<one-time-token>
```

## Next.js middleware or proxy [#nextjs-middleware-or-proxy]

Next middleware runs before the final page response. Use request-only tracking: `NextResponse.next()` is not the final page status and must not be reported as one.

```ts
import { NextResponse } from 'next/server';
import { trackBotRequestInBackground } from '@talivia/bot-traffic/next';

export function middleware(request, event) {
  trackBotRequestInBackground(
    request,
    { token: process.env.TALIVIA_BOT_TOKEN! },
    event,
  );
  return NextResponse.next();
}

export const config = { matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)'] };
```

## Express [#express]

The Express adapter calls `next()` immediately and reports the final response status after the `finish` event.

```ts
import express from 'express';
import { createExpressBotMiddleware } from '@talivia/bot-traffic/express';

const app = express();
app.use(createExpressBotMiddleware({
  token: process.env.TALIVIA_BOT_TOKEN!,
}));
```

## Cloudflare Workers and Pages [#cloudflare-workers-and-pages]

```ts
import { withBotTracking } from '@talivia/bot-traffic/cloudflare';

export default {
  fetch: withBotTracking(
    (request, env) => env.ASSETS.fetch(request),
    { token: '<server-token>' },
  ),
};
```

For Pages Functions, process `context.request`, then call `trackBotRequestInBackground` with the final status and pass `context` as the third argument so `waitUntil` keeps delivery alive.

## Any JavaScript or TypeScript framework [#any-javascript-or-typescript-framework]

No named adapter is required when the framework exposes standard `Request` and `Response` objects:

```ts
import { trackBotRequestInBackground } from '@talivia/bot-traffic';

export async function handle(request, context) {
  const response = await yourFrameworkHandler(request);
  trackBotRequestInBackground(
    request,
    { token: process.env.TALIVIA_BOT_TOKEN!, status: response.status },
    context, // optional object with waitUntil(promise)
  );
  return response;
}
```

If only the incoming request is available, omit `status`. Request-only tracking still records which page the crawler attempted to fetch. Use `withBotTracking` only around a handler that returns the final `Response`. Use the awaited `trackBotRequest` function for tests or controlled background jobs, not on a latency-sensitive response path.

This generic pattern applies to Hono, Bun, Deno, custom Node.js Fetch servers, serverless functions, and other Fetch-compatible runtimes.

## Docker, Cloud Run, and reverse proxies [#docker-cloud-run-and-reverse-proxies]

If `request.url` contains an internal host such as `http://app:3000/docs`, set the real public origin:

```ts
{
  token: process.env.TALIVIA_BOT_TOKEN!,
  publicOrigin: 'https://example.com',
}
```

The package replaces only the origin and preserves the request path. `publicOrigin` must be an `http` or `https` origin without credentials, a path, query, or fragment. Talivia still validates the public hostname against the website that owns the token.

## PHP, Go, Ruby, and other languages [#php-go-ruby-and-other-languages]

The JavaScript package is optional. Any server language can send this minimal event to the fixed endpoint:

```http
POST https://talivia.com/v1/bot-traffic
Authorization: Bearer <website-bot-token>
Content-Type: application/json

{
  "schemaVersion": 1,
  "hostname": "example.com",
  "path": "/docs/getting-started",
  "method": "GET",
  "userAgent": "GPTBot/1.0",
  "status": 200
}
```

`status` is optional. Talivia Cloud derives provider, bot, category, and verification itself; do not send those fields.

### PHP example [#php-example]

Invoke this helper only after the application has sent the response. With PHP-FPM, call `fastcgi_finish_request()` first when available.

```php
function trackTaliviaBot(array $event, string $token): void {
    $ch = curl_init('https://talivia.com/v1/bot-traffic');
    curl_setopt_array($ch, [
        CURLOPT_POST => true,
        CURLOPT_POSTFIELDS => json_encode($event, JSON_UNESCAPED_SLASHES),
        CURLOPT_HTTPHEADER => [
            'Authorization: Bearer ' . $token,
            'Content-Type: application/json',
        ],
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT_MS => 1500,
    ]);
    curl_exec($ch);
    curl_close($ch);
}

// After your framework has produced and sent the final response:
if (function_exists('fastcgi_finish_request')) {
    fastcgi_finish_request();
}
trackTaliviaBot($event, getenv('TALIVIA_BOT_TOKEN'));
```

Call the helper only for likely crawler `GET` or `HEAD` document requests after the final status is known and the user response has been sent. Send a pathname without query strings or fragments. Do not send cookies, authorization headers, request bodies, visitor identifiers, referrers, or source IP.

## Required custom-integration behavior [#required-custom-integration-behavior]

* Endpoint: `https://talivia.com/v1/bot-traffic`.
* Authentication: `Authorization: Bearer <website-bot-token>`.
* Content type: `application/json`.
* Required fields: `schemaVersion`, `hostname`, `path`, `method`, `userAgent`.
* Optional field: `status` from `100` to `599`.
* Methods: document-like `GET` and `HEAD` requests.
* Keep payloads at or below 8 KiB and User-Agent at or below 512 characters.
* Strip query strings and fragments from `path`.
* Use a short timeout and fail open. A telemetry failure must never fail or delay the website response.
* Do not trust User-Agent as proof of identity. Talivia performs central classification and network verification.

After deployment, use the detailed report to review request trends, providers, categories, pages, status codes, missing pages, and evidence-based verification confidence.