AgentBlog MDX content source

/**
 * `mdxSource`: MDX files on disk. The reference `ContentSource` implementation.
 *
 * ---------------------------------------------------------------------------
 * READ THIS FIRST IF YOU ARE WRITING YOUR OWN ADAPTER
 * ---------------------------------------------------------------------------
 * This file is the worked example. A `supabaseSource`, a `sanitySource`, or a
 * hosted source is the same seven methods with a different `readBundle`. The
 * shape below is deliberate and worth copying:
 *
 *   1. `mdxSource(opts)` resolves its options once and returns a plain object
 *      of closures. No class, no `this`, so a method can be destructured off
 *      the source and passed to `generateStaticParams` without losing binding.
 *   2. Every read goes through `loadBundle`, which returns posts, authors, and
 *      categories together. One trip, everything hydrated. Contract assertion 3
 *      exists because a lazy `author` reference becomes an N+1 against your
 *      database at exactly the moment you have the most posts.
 *   3. Sorting happens once, in `byRecencyThenSlug`, and is total. Contract
 *      assertion 8 exists because unstable ordering makes `generateStaticParams`
 *      churn, which invalidates every cached page on every build.
 *   4. Public methods are wrapped in `guarded`, which turns anything unexpected
 *      into an `AgentBlogSourceError` naming the operation that failed.
 *
 * ---------------------------------------------------------------------------
 * THE PURITY RULE
 * ---------------------------------------------------------------------------
 * Every method here is callable at build time, from `generateStaticParams()` and
 * `generateMetadata()`. No hooks, no `cookies()`, no `headers()`. That is what
 * keeps posts fully prerendered, and full prerendering is what puts the whole
 * article in the first response byte for crawlers that do not run JavaScript.
 *
 * ---------------------------------------------------------------------------
 * WHAT `prerenderStrategy: 'build'` MEANS HERE
 * ---------------------------------------------------------------------------
 * Content lives in the repository, so publishing a post is a git push and a
 * deploy. A new slug is always present the next time `generateStaticParams` runs,
 * so there is nothing extra to do when one appears, and `agentblog.config.ts`
 * does not require a `deployHook`. A database-backed source cannot say that,
 * which is why it declares `'deploy-hook'` instead.
 *
 * @see https://docs.agentblog.dev/reference/content-sources
 */
import { readFile, readdir, stat } from 'node:fs/promises'
import path from 'node:path'

import matter from 'gray-matter'

import { AuthorSchema, CategorySchema, PostSchema } from '@/lib/schemas'
import type { Slug } from '@/lib/schemas'
import { AgentBlogContentError, AgentBlogSourceError } from '@/lib/types'
import type {
  Author,
  Category,
  ContentIssue,
  ContentSource,
  Post,
  PostQuery,
  PublishedPost,
} from '@/lib/types'

/* -------------------------------------------------------------------------- */
/*  Options                                                                   */
/* -------------------------------------------------------------------------- */

export interface MdxSourceOptions {
  /** Directory of .mdx files, relative to the project root. No default. */
  readonly dir: string
  /** Author records, relative to the project root. Defaults to `content/authors.json`. */
  readonly authorsFile?: string
  /** Category records, relative to the project root. Defaults to `content/categories.json`. */
  readonly categoriesFile?: string
  /**
   * Author slug applied to a post whose frontmatter omits `author`. No default,
   * so a post without an author fails validation unless this is set.
   *
   * ---------------------------------------------------------------------------
   * WHY THIS IS AN OPTION AND NOT READ FROM `config.defaultAuthor`
   * ---------------------------------------------------------------------------
   * `agentblog.config.ts` calls `mdxSource(...)` to build its own `source` field,
   * and `lib/config.ts` imports `agentblog.config.ts`. If this file imported
   * `@/lib/config` to read `config.defaultAuthor`, the cycle would close:
   * config imports the source, source imports config. Under a bundler that
   * resolves to a partially initialised module, which surfaces as
   * `config is undefined` at module scope and not as a helpful error.
   *
   * The slug therefore travels as an option, passed by the one file that already
   * owns both sides. Do not "simplify" this by importing the config here.
   */
  readonly defaultAuthor?: string
  /** Fail the build on a malformed post (default) or skip it with a warning. */
  readonly onInvalid?: 'throw' | 'warn'
}

/** `ContentSource.name`, and the prefix on every error this file throws. */
const SOURCE_NAME = 'mdx'

const DEFAULT_AUTHORS_FILE = 'content/authors.json'
const DEFAULT_CATEGORIES_FILE = 'content/categories.json'

/**
 * `.mdx` only, and the `?` that used to be on the `x` was a real bug rather than
 * a convenience.
 *
 * `lib/render-mdx.tsx` compiles every body as MDX, so a `.md` file accepted here
 * was never treated as CommonMark, it was treated as MDX with a different
 * extension. That is fine until a post uses markdown that MDX does not share.
 * `<https://example.com/x>` is a legal CommonMark autolink and is what many
 * editors emit; under the MDX parser the `<` opens a JSX tag and the compile
 * fails with `Unexpected character '/' (U+002F) before local name`, which aborts
 * `next build` for the whole site. `<br>` and `<img src=...>` fail the same way.
 *
 * The alternative was a second, CommonMark pipeline for `.md`. It was rejected:
 * every other surface in the product (the component map, the FAQ and answer
 * capsule components, the docs, the seed posts) says MDX, so a second dialect
 * would mean two renderers, two component stories, and a post whose behaviour
 * depends on its file extension. One dialect, named honestly, is the smaller
 * thing to maintain. `readMdFileNames` below makes the exclusion audible.
 */
const POST_FILE = /\.mdx$/

/** Matched only to warn about it. See `POST_FILE` and `warnAboutMarkdownFiles`. */
const IGNORED_MARKDOWN_FILE = /\.md$/

/* -------------------------------------------------------------------------- */
/*  Internal shapes                                                           */
/* -------------------------------------------------------------------------- */

/** Options after `process.cwd()` resolution. Also the memoization cache key. */
interface ResolvedPaths {
  readonly dir: string
  readonly authorsFile: string
  readonly categoriesFile: string
  readonly defaultAuthor: string | undefined
  readonly onInvalid: 'throw' | 'warn'
}

/**
 * One read of the content directory: every post (drafts included), sorted, with
 * `author` and `category` already hydrated, plus the full record lists.
 *
 * Drafts are kept in the bundle rather than filtered here so that a single cached
 * read can answer both `getAllPosts()` and `getAllPosts({ includeDrafts: true })`.
 */
interface ContentBundle {
  readonly posts: readonly Post[]
  readonly authors: readonly Author[]
  readonly categories: readonly Category[]
}

/**
 * The slice of a Zod schema that the record loader needs.
 *
 * Declared structurally rather than as `z.ZodType<T>` because `AuthorSchema` and
 * `CategorySchema` both use `.default()`, which makes their input type differ from
 * their output type. A structural type accepts both without a cast and does not
 * pin this file to one Zod major version.
 */
interface RecordSchema<T> {
  safeParse(
    value: unknown,
  ):
    | { readonly success: true; readonly data: T }
    | { readonly success: false; readonly error: { readonly issues: readonly ZodIssueLike[] } }
}

interface ZodIssueLike {
  readonly path: readonly (string | number | symbol)[]
  readonly message: string
}

/* -------------------------------------------------------------------------- */
/*  The factory                                                               */
/* -------------------------------------------------------------------------- */

/**
 * Build a content source over a directory of MDX files.
 *
 * ```ts
 * // agentblog.config.ts
 * export default defineConfig({
 *   source: mdxSource({ dir: 'content/blog', defaultAuthor: 'your-name' }),
 *   defaultAuthor: 'your-name',
 *   // ...
 * })
 * ```
 *
 * `defaultAuthor` appears twice on purpose. See `MdxSourceOptions.defaultAuthor`
 * for the import cycle that keeps this file from reading the config itself.
 */
export function mdxSource(opts: MdxSourceOptions): ContentSource<'build'> {
  // Paths resolve against `process.cwd()`, which Next.js sets to the project root
  // for both `next dev` and `next build`. Resolving once here means an option
  // typo surfaces as one clear path in the error message rather than as a
  // different path per call site.
  /*
   * `turbopackIgnore` on the three resolves, and it is not a workaround.
   *
   * Turbopack traces filesystem access statically. It cannot prove which
   * directory `opts.dir` resolves to, so it assumes the worst and traces the
   * whole project, copying your source tree and `public/` next to the server
   * bundle. That inflates every deployment and can trip a platform size limit.
   *
   * A computed path is unavoidable here: the directory comes from the user's
   * `agentblog.config.ts`, so it can never be a literal. The annotation says we
   * take responsibility for declaring the dependency, and the way to declare it
   * is `outputFileTracingIncludes` in `next.config.ts`, which `agentblog init`
   * writes for you and which `doctor` checks. That is the correct mechanism:
   * it names the content directory explicitly instead of tracing everything.
   */
  const paths: ResolvedPaths = {
    dir: path.resolve(/* turbopackIgnore: true */ process.cwd(), opts.dir),
    authorsFile: path.resolve(
      /* turbopackIgnore: true */ process.cwd(),
      opts.authorsFile ?? DEFAULT_AUTHORS_FILE,
    ),
    categoriesFile: path.resolve(
      /* turbopackIgnore: true */ process.cwd(),
      opts.categoriesFile ?? DEFAULT_CATEGORIES_FILE,
    ),
    defaultAuthor: opts.defaultAuthor,
    onInvalid: opts.onInvalid ?? 'throw',
  }

  const load = (): Promise<ContentBundle> => loadBundle(paths)

  async function getAllPosts(query?: PostQuery): Promise<PublishedPost[]> {
    return guarded('getAllPosts', async () => {
      const bundle = await load()
      return selectPosts(bundle.posts, query)
    })
  }

  async function getPost(slug: Slug, query?: PostQuery): Promise<Post | null> {
    return guarded('getPost', async () => {
      const bundle = await load()
      const post = bundle.posts.find((candidate) => candidate.slug === slug)
      // A missing file is `null` so the route can call `notFound()`. Everything
      // else throws, because at build time a source that cannot answer should
      // fail the build rather than ship a sitemap that quietly lost URLs.
      if (!post) return null
      // A draft is "absent" too unless the caller opted in. Drafts are opted into,
      // never out of, so a preview route has to ask.
      if (post.draft && query?.includeDrafts !== true) return null
      return post
    })
  }

  async function getAllCategories(): Promise<Category[]> {
    return guarded('getAllCategories', async () => [...(await load()).categories])
  }

  async function getAllAuthors(): Promise<Author[]> {
    return guarded('getAllAuthors', async () => [...(await load()).authors])
  }

  async function getPostsByCategory(slug: Slug): Promise<PublishedPost[]> {
    return guarded('getPostsByCategory', async () => {
      const bundle = await load()
      return selectPosts(bundle.posts, undefined).filter((post) => post.category.slug === slug)
    })
  }

  async function getPostsByAuthor(slug: Slug): Promise<PublishedPost[]> {
    return guarded('getPostsByAuthor', async () => {
      const bundle = await load()
      return selectPosts(bundle.posts, undefined).filter((post) => post.author.slug === slug)
    })
  }

  async function getRelatedPosts(post: Post, limit: number): Promise<PublishedPost[]> {
    return guarded('getRelatedPosts', async () => {
      const bundle = await load()
      // Never the post itself, never a draft. Both are contract assertion 6.
      const pool = selectPosts(bundle.posts, undefined).filter((c) => c.slug !== post.slug)
      return pickRelated(post, pool, limit)
    })
  }

  return {
    prerenderStrategy: 'build',
    name: SOURCE_NAME,
    getAllPosts,
    getPost,
    getAllCategories,
    getAllAuthors,
    getPostsByCategory,
    getPostsByAuthor,
    getRelatedPosts,
  }
}

/* -------------------------------------------------------------------------- */
/*  Error boundary                                                            */
/* -------------------------------------------------------------------------- */

/**
 * Wrap a public method so that anything unexpected becomes an
 * `AgentBlogSourceError` naming the operation.
 *
 * `AgentBlogContentError` passes through untouched. It already names the offending
 * file and every Zod issue in it, which is strictly more useful than
 * "getAllPosts failed".
 */
async function guarded<T>(operation: string, run: () => Promise<T>): Promise<T> {
  try {
    return await run()
  } catch (error) {
    if (error instanceof AgentBlogContentError) throw error
    throw new AgentBlogSourceError(SOURCE_NAME, operation, { cause: error })
  }
}

/* -------------------------------------------------------------------------- */
/*  The per-process cache                                                     */
/* -------------------------------------------------------------------------- */

/**
 * One entry per resolved directory, holding the in-flight promise rather than the
 * resolved value so that concurrent callers share a single read.
 *
 * A production build calls into this source many times for the same content:
 * `generateStaticParams` once, then `generateMetadata` and the page body once per
 * post, plus `sitemap.ts` and `feed.xml`. Without memoization a 200 post blog
 * re-reads 200 files a few hundred times.
 */
const bundleCache = new Map<string, { stamp: string; bundle: Promise<ContentBundle> }>()

/**
 * Every field of `ResolvedPaths`, not just the three paths.
 *
 * `defaultAuthor` and `onInvalid` are both read by `parsePost` and
 * `normalizeFrontmatter`, so two sources over the same directory that differ
 * only in those two produce genuinely different bundles. Keyed on the paths
 * alone, the second source silently received the first one's bundle: a config
 * with `onInvalid: 'warn'` inherited a bundle built under `'throw'`, and a post
 * relying on a different `defaultAuthor` came back hydrated with the wrong
 * person. Nothing errors, which is why it survived review the first time.
 *
 * NUL as the separator rather than a space, because a directory path is allowed
 * to contain a space and a separator that can appear inside a component is not a
 * separator: dir 'a b' plus authorsFile 'c' and dir 'a' plus authorsFile 'b c'
 * join to the same string. NUL cannot appear in a path on any platform this
 * code runs on.
 */
function cacheKey(paths: ResolvedPaths): string {
  return [
    paths.dir,
    paths.authorsFile,
    paths.categoriesFile,
    paths.defaultAuthor ?? '',
    paths.onInvalid,
  ].join('\u0000')
}

/**
 * A cheap fingerprint of the content: the directory's mtime plus the mtime of
 * each metadata file.
 *
 * A directory's mtime changes when a file is added to it or removed from it, so
 * three `stat` calls are enough to notice that the post set changed, without
 * re-reading every post to find out. Editing a post in place does not change the
 * directory mtime, and that is fine: an edit to a file already in the deployment
 * arrives with a rebuild, and under `next dev` the cache is off entirely.
 *
 * Why this matters at all, given that `mdxSource` declares
 * `prerenderStrategy: 'build'`. The publish webhook revalidates `/sitemap.xml`
 * and `/feed.xml`, and those handlers re-render inside a long-lived server
 * process. Caching for the lifetime of that process means a revalidation
 * regenerates the sitemap from the post set the process started with, so the
 * webhook appears to work and changes nothing. Worse, a production server runs
 * several worker processes, so whether the new post appeared would depend on
 * which worker answered, which is the least debuggable failure available.
 */
async function contentStamp(paths: ResolvedPaths): Promise<string> {
  const parts = await Promise.all(
    [paths.dir, paths.authorsFile, paths.categoriesFile].map(async (target) => {
      try {
        const info = await stat(target)
        return String(info.mtimeMs)
      } catch {
        // A missing metadata file is a real condition that `readBundle` reports
        // properly. Here it only has to be a stable part of the fingerprint.
        return 'missing'
      }
    }),
  )
  return parts.join(' ')
}

async function loadBundle(paths: ResolvedPaths): Promise<ContentBundle> {
  // The tradeoff, stated explicitly: caching is on only when `NODE_ENV` is
  // 'production', which is what `next build` sets. Under `next dev` every read
  // hits the disk again, so saving a file shows up on the next request instead of
  // requiring a server restart. That costs a directory scan per request in dev,
  // which is the correct trade: dev has tens of posts and a human waiting, build
  // has hundreds of posts and no one waiting.
  if (process.env['NODE_ENV'] !== 'production') return readBundle(paths)

  const key = cacheKey(paths)
  const stamp = await contentStamp(paths)
  const cached = bundleCache.get(key)
  if (cached && cached.stamp === stamp) return cached.bundle

  const bundle = readBundle(paths)
  bundleCache.set(key, { stamp, bundle })
  // Never cache a rejection. A transient read failure should be retryable rather
  // than poisoning every later call in the process.
  void bundle.catch(() => {
    const current = bundleCache.get(key)
    if (current?.bundle === bundle) bundleCache.delete(key)
  })
  return bundle
}

/* -------------------------------------------------------------------------- */
/*  Reading                                                                   */
/* -------------------------------------------------------------------------- */

/**
 * The single round trip.
 *
 * Authors and categories are read once, before any post is parsed, and passed to
 * every post as lookup maps. This is the filesystem version of the join that a
 * SQL adapter has to write, and it is what contract assertion 3 is about.
 */
async function readBundle(paths: ResolvedPaths): Promise<ContentBundle> {
  const [authors, categories] = await Promise.all([
    readRecords<Author>(paths.authorsFile, AuthorSchema, 'author'),
    readRecords<Category>(paths.categoriesFile, CategorySchema, 'category'),
  ])

  const authorsBySlug = new Map(authors.map((author) => [String(author.slug), author]))
  const categoriesBySlug = new Map(categories.map((c) => [String(c.slug), c]))

  const entries = (await readdir(paths.dir, { withFileTypes: true })).filter((entry) =>
    entry.isFile(),
  )

  const fileNames = entries
    .map((entry) => entry.name)
    .filter((name) => POST_FILE.test(name))
    // Sort the filenames, not just the posts, so `onInvalid: 'warn'` prints its
    // warnings in the same order on every machine.
    .sort(compareStrings)

  warnAboutMarkdownFiles(
    entries
      .map((entry) => entry.name)
      .filter((name) => IGNORED_MARKDOWN_FILE.test(name))
      .sort(compareStrings),
    paths.dir,
  )

  const sources = await Promise.all(
    fileNames.map(async (name) => ({
      name,
      file: path.join(paths.dir, name),
      raw: await readFile(path.join(paths.dir, name), 'utf8'),
    })),
  )

  const posts: Post[] = []
  for (const source of sources) {
    const post = parsePost(source, authorsBySlug, categoriesBySlug, paths)
    if (post) posts.push(post)
  }
  posts.sort(byRecencyThenSlug)

  // `Category.latestPostDate` is derived, not authored: `sitemap.ts` needs a real
  // `lastModified` for each category hub, and stamping the build time there is a
  // false freshness signal. Deriving it needs the posts, and hydrating the posts
  // needs the categories, so it happens in a second pass and the hydrated
  // references are rewritten to match. One category object, one value.
  const enriched = withLatestPostDates(categories, posts)
  const enrichedBySlug = new Map(enriched.map((c) => [String(c.slug), c]))
  const linked = posts.map((post) => {
    const category = enrichedBySlug.get(String(post.category.slug))
    return category ? { ...post, category } : post
  })

  return { posts: linked, authors, categories: enriched }
}

/**
 * Say out loud that a `.md` file in the content directory is not being published.
 *
 * A warning rather than a thrown error, and the line between the two is what the
 * file most likely is. A `.md` file here is usually an author's `README.md` or a
 * note to themselves, and failing every build in a repository that happens to
 * contain one would be hostile. It is occasionally a post whose extension is
 * wrong, and that case is silent 404 territory, which is exactly the failure
 * mode this whole file is written to avoid. One line on stdout at build time
 * covers both: the note-keeper ignores it, the author who renamed a file finds
 * it in the build log.
 */
function warnAboutMarkdownFiles(names: readonly string[], dir: string): void {
  if (names.length === 0) return
  console.warn(
    `${SOURCE_NAME}: ignoring ${names.length} .md file(s) in ${dir}: ${names.join(', ')}.\n` +
      '  AgentBlog compiles posts as MDX only. Rename to .mdx to publish, or move the file ' +
      'out of the content directory to silence this.',
  )
}

/**
 * Read and validate a JSON array of records.
 *
 * The `location` on every error is the file path, so a bad author record names the
 * file it lives in instead of surfacing as "invalid input" during a build that
 * touched two hundred files.
 */
async function readRecords<T>(file: string, schema: RecordSchema<T>, label: string): Promise<T[]> {
  const raw = await readFile(file, 'utf8')

  let parsed: unknown
  try {
    parsed = JSON.parse(raw)
  } catch (error) {
    throw new AgentBlogContentError(file, [
      { path: [], message: `is not valid JSON: ${String(error)}` },
    ])
  }

  if (!Array.isArray(parsed)) {
    throw new AgentBlogContentError(file, [
      { path: [], message: `must be a JSON array of ${label} records.` },
    ])
  }

  const records: T[] = []
  const issues: ContentIssue[] = []
  for (const [index, entry] of parsed.entries()) {
    const result = schema.safeParse(entry)
    if (result.success) records.push(result.data)
    else
      issues.push(
        ...result.error.issues.map((i) => ({ path: [index, ...i.path], message: i.message })),
      )
  }
  if (issues.length > 0) throw new AgentBlogContentError(file, issues)

  return records
}

/**
 * Parse one file into a `Post`, or return `null` when it is invalid and
 * `onInvalid` is `'warn'`.
 */
function parsePost(
  source: { name: string; file: string; raw: string },
  authors: ReadonlyMap<string, Author>,
  categories: ReadonlyMap<string, Category>,
  paths: ResolvedPaths,
): Post | null {
  const { data, content } = matter(source.raw)
  const normalized = normalizeFrontmatter(data, content, source.name, authors, categories, paths)

  const result = PostSchema.safeParse(normalized.value)
  const issues: ContentIssue[] = [
    ...normalized.issues,
    ...(result.success
      ? []
      : result.error.issues.map((i) => ({ path: i.path, message: i.message }))),
  ]

  if (issues.length === 0 && result.success) return result.data

  const error = new AgentBlogContentError(source.file, issues)
  if (paths.onInvalid === 'warn') {
    console.warn(`${error.message}\n  Skipping this post because onInvalid is "warn".`)
    return null
  }
  throw error
}

/* -------------------------------------------------------------------------- */
/*  Frontmatter normalisation                                                 */
/* -------------------------------------------------------------------------- */

/**
 * Apply the authoring conveniences, then hand a plain object to `PostSchema`.
 *
 * Frontmatter is written by humans and by agents, so a small amount of tolerance
 * here buys a lot. Each convenience below is deliberate and bounded; anything not
 * listed is the schema's business, not this function's.
 */
function normalizeFrontmatter(
  raw: Record<string, unknown>,
  body: string,
  fileName: string,
  authors: ReadonlyMap<string, Author>,
  categories: ReadonlyMap<string, Category>,
  paths: ResolvedPaths,
): { value: Record<string, unknown>; issues: ContentIssue[] } {
  const issues: ContentIssue[] = []
  const value: Record<string, unknown> = { ...raw, body }

  // Convenience 1: the slug defaults to the filename. `my-post.mdx` is already a
  // statement of intent, and repeating it in frontmatter is one more thing to get
  // out of sync with the URL.
  if (value['slug'] === undefined) value['slug'] = fileName.replace(POST_FILE, '')

  // Convenience 2: a bare `YYYY-MM-DD` is widened to midnight UTC. `IsoDateTime`
  // requires an offset because Google reinterprets an offset-less date in
  // Googlebot's own timezone, but demanding `2026-08-06T00:00:00Z` in handwritten
  // frontmatter is hostile. Widening also covers the YAML parser turning an
  // unquoted date into a JavaScript `Date`, which would otherwise fail every time.
  value['datePublished'] = widenDate(value['datePublished'])
  value['dateModified'] = widenDate(value['dateModified'])
  if (Array.isArray(value['citations'])) {
    value['citations'] = value['citations'].map((citation) =>
      isRecord(citation) && 'datePublished' in citation
        ? { ...citation, datePublished: widenDate(citation['datePublished']) }
        : citation,
    )
  }

  // Convenience 3: an omitted `author` falls back to the configured default.
  // This runs before hydration below, so the default goes through the same
  // "does that slug exist" check as an author written in frontmatter and a
  // typo in the config is reported against the post that relied on it.
  if (value['author'] === undefined && paths.defaultAuthor !== undefined) {
    value['author'] = paths.defaultAuthor
  }

  // Convenience 4: `author` and `category` may be a slug string. Frontmatter
  // references a record; the adapter hydrates it. Inline objects still work, for
  // the one-off post whose author is not in the roster.
  if (typeof value['author'] === 'string') {
    const author = authors.get(value['author'])
    if (author) value['author'] = author
    else issues.push(unknownRecord('author', value['author'], paths.authorsFile))
  }
  if (typeof value['category'] === 'string') {
    const category = categories.get(value['category'])
    if (category) value['category'] = category
    else issues.push(unknownRecord('category', value['category'], paths.categoriesFile))
  }

  return { value, issues }
}

const BARE_DATE = /^\d{4}-\d{2}-\d{2}$/

function widenDate(value: unknown): unknown {
  if (value instanceof Date) return value.toISOString()
  if (typeof value === 'string' && BARE_DATE.test(value)) return `${value}T00:00:00Z`
  return value
}

function unknownRecord(field: 'author' | 'category', slug: string, file: string): ContentIssue {
  return {
    path: [field],
    message: `unknown ${field} slug "${slug}". Add a record with that slug to ${file}, or inline the ${field} object in this post's frontmatter.`,
  }
}

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === 'object' && value !== null && !Array.isArray(value)
}

/* -------------------------------------------------------------------------- */
/*  Selection, ordering, and related posts                                    */
/* -------------------------------------------------------------------------- */

function isPublished(post: Post): post is PublishedPost {
  return post.draft === false
}

/**
 * Apply the draft gate.
 *
 * The cast on the `includeDrafts` branch is the one place this file lies to the
 * type system, and it is the interface's own wart: the list methods return
 * `PublishedPost[]` so that `generateStaticParams` is structurally unable to emit
 * a draft slug, which leaves no honest type for the opted-in preview read.
 */
function selectPosts(posts: readonly Post[], query: PostQuery | undefined): PublishedPost[] {
  if (query?.includeDrafts === true) return [...posts] as PublishedPost[]
  return posts.filter(isPublished)
}

/**
 * Newest first, then slug ascending as a tiebreak.
 *
 * The tiebreak is not cosmetic. Two posts published on the same day would
 * otherwise sort by whatever order the filesystem handed them back, and an
 * ordering that changes between machines makes `generateStaticParams` churn.
 */
function byRecencyThenSlug(a: Post, b: Post): number {
  const delta = Date.parse(b.datePublished) - Date.parse(a.datePublished)
  return delta !== 0 ? delta : compareStrings(String(a.slug), String(b.slug))
}

/** Code-unit comparison, not `localeCompare`, so the order does not vary by locale. */
function compareStrings(a: string, b: string): number {
  return a < b ? -1 : a > b ? 1 : 0
}

/**
 * Editorial picks first, in the order the author wrote them, then computed fills.
 *
 * Precedence is the whole point: `post.relatedPosts` is a human decision and beats
 * anything this function can infer. Computed matches only fill the remaining slots,
 * ranked by shared category, then shared tag count, then recency.
 */
function pickRelated(post: Post, pool: readonly PublishedPost[], limit: number): PublishedPost[] {
  const bySlug = new Map(pool.map((candidate) => [String(candidate.slug), candidate]))
  const picked: PublishedPost[] = []
  const taken = new Set<string>()

  for (const slug of post.relatedPosts) {
    if (picked.length >= limit) break
    const match = bySlug.get(String(slug))
    if (!match || taken.has(String(slug))) continue
    picked.push(match)
    taken.add(String(slug))
  }

  if (picked.length >= limit) return picked

  const tags = new Set(post.tags.map((tag) => tag.toLowerCase()))
  const sharedTags = (candidate: Post): number =>
    candidate.tags.filter((tag) => tags.has(tag.toLowerCase())).length
  const sameCategory = (candidate: Post): number =>
    candidate.category.slug === post.category.slug ? 1 : 0

  const computed = pool
    .filter((candidate) => !taken.has(String(candidate.slug)))
    .sort(
      (a, b) =>
        sameCategory(b) - sameCategory(a) ||
        sharedTags(b) - sharedTags(a) ||
        byRecencyThenSlug(a, b),
    )

  return [...picked, ...computed.slice(0, limit - picked.length)]
}

/**
 * Stamp each category with the publish date of its newest published post.
 * Categories with no published posts are returned unchanged, so `sitemap.ts` can
 * tell "no posts yet" from "posts, none recent".
 */
function withLatestPostDates(categories: readonly Category[], posts: readonly Post[]): Category[] {
  const latest = new Map<string, string>()
  for (const post of posts) {
    if (post.draft) continue
    const slug = String(post.category.slug)
    const current = latest.get(slug)
    if (!current || Date.parse(post.datePublished) > Date.parse(current)) {
      latest.set(slug, post.datePublished)
    }
  }
  return categories.map((category) => {
    const date = latest.get(String(category.slug))
    return date ? { ...category, latestPostDate: date as Category['latestPostDate'] } : category
  })
}

/**
 * The only place MDX is compiled in AgentBlog. Keep it that way.
 *
 * ---------------------------------------------------------------------------
 * WHY `next-mdx-remote-client/rsc` AND NOT `@next/mdx`
 * ---------------------------------------------------------------------------
 * The decisive reason is the two plugin files sitting next to this one.
 *
 * `@next/mdx` compiles MDX through the Turbopack loader, and Turbopack is the
 * default bundler in Next 16. Turbopack's compiler is Rust, so it cannot accept
 * a JavaScript function as a plugin. Its documented workaround is to name
 * plugins as strings that Turbopack resolves by module name:
 * `remarkPlugins: ['remark-gfm', ['remark-toc', { heading: 'The Table' }]]`.
 *
 * `remarkExtractToc` and `remarkAnswerCapsule` are local functions in your
 * repository. Under `@next/mdx` plus Turbopack they would have to be published
 * to npm and resolvable by package name before they could run at all. That is
 * an absurd price for two files you can read in a minute.
 *
 * `next-mdx-remote-client` compiles MDX at runtime, inside this function, using
 * `@mdx-js/mdx` directly. It never touches the Turbopack loader, so neither the
 * string-name restriction nor the serializable-options restriction applies.
 * Local plugin functions work normally.
 *
 * The secondary reason: this same code path renders a `.mdx` file read off disk
 * and an MDX string read out of a database, because it takes a string and
 * returns a React element. `@next/mdx`'s pattern is a dynamic import of
 * `@/content/<slug>.mdx`, which ties content to the filesystem and cannot
 * satisfy the `ContentSource` interface in `lib/types.ts` for any adapter that
 * is not files on disk.
 *
 * ---------------------------------------------------------------------------
 * STABILITY IS THE DESIGN GOAL
 * ---------------------------------------------------------------------------
 * Every consumer of MDX in this blog calls `renderMdx(source, origin)` and
 * receives `{ content, toc }`. Nothing else in the codebase imports an MDX library,
 * knows what a vfile is, or names a plugin. If `next-mdx-remote-client` ever
 * stalls, swapping to `@mdx-js/mdx`'s own `compile` and `evaluate` is a change
 * to this one file with zero consumer impact. Adding a plugin is also a change
 * to this one file. Resist the urge to compile MDX anywhere else.
 *
 * Verified API of `next-mdx-remote-client@2.1.11`: the `/rsc` entrypoint
 * exports `MDXRemote` and `evaluate`. `evaluate({ source, options, components })`
 * resolves to `{ content, mod, frontmatter, scope, error? }`. It does not throw
 * on a compile failure, it returns the error on the result, which is why the
 * throw below is explicit.
 *
 * ===========================================================================
 * THE TRUST MODEL. READ THIS BEFORE POINTING A NEW CONTENT SOURCE AT IT
 * ===========================================================================
 * MDX is a programming language, not a markup format. `renderMdx` compiles the
 * body you give it and runs the result inside your server process, with that
 * process's `process`, `process.env`, and `process.getBuiltinModule` reachable
 * from any expression in the document.
 *
 * **A post body is code. Only pass this content from an author you would give a
 * shell account on the same machine.**
 *
 * Concretely, with no `import` statement anywhere in the file:
 *
 *     {(() => { const cp = process.getBuiltinModule('child_process')
 *               return cp.execSync('id -un').toString() })()}
 *
 * executes on the server and renders its output. `Secret is {process.env.X}`
 * prints an environment variable into public HTML. Both were reproduced against
 * this file.
 *
 * `disableImports` and `disableExports` are set below and they are worth having,
 * but be clear about what they do and do not buy:
 *
 *   they DO strip `import` and `export` statements from the document, so a body
 *   cannot pull in a module or hang a value off the evaluated module's exports;
 *
 *   they DO NOT stop expression evaluation. `{anything}` still runs. There is no
 *   option on `@mdx-js/mdx` or on `next-mdx-remote-client` that turns MDX into a
 *   sandbox, because evaluating expressions is what MDX is for.
 *
 * That is fine for `mdxSource`, where posts are files in your repository and
 * arrive through code review and a deploy. It is not fine the moment the content
 * source is something anyone but a trusted author can write to. If you are
 * writing a `supabaseSource`, a `sanitySource`, or anything with a non-admin
 * write path, the body from that source must not reach this function: render it
 * through a CommonMark pipeline with raw HTML disabled, or gate MDX behind an
 * author role you control. See the same warning on `ContentSource` in
 * `lib/types.ts`.
 *
 * @see https://docs.agentblog.dev/reference/files
 */
import 'server-only'

import { evaluate } from 'next-mdx-remote-client/rsc'
import type { ReactElement } from 'react'
import rehypeAutolinkHeadings from 'rehype-autolink-headings'
import rehypePrettyCode from 'rehype-pretty-code'
import type { Options as PrettyCodeOptions } from 'rehype-pretty-code'
import rehypeSlug from 'rehype-slug'
import remarkGfm from 'remark-gfm'

import { mdxComponents } from '@/components/mdx'
import { FAQ_HEADING, FAQ_HEADING_ID } from '@/components/mdx/faq'
import { remarkAnswerCapsule } from '@/lib/mdx-plugins/remark-answer-capsule'
import { remarkExtractToc } from '@/lib/mdx-plugins/remark-extract-toc'
import type { TocEntry } from '@/lib/toc'

export interface RenderedMdx {
  readonly content: ReactElement
  readonly toc: TocEntry[]
  /**
   * The `id` to put on the FAQ section heading a route renders after the body,
   * and on the table-of-contents entry that points at it.
   *
   * It is issued here rather than chosen by the route because the only slugger
   * that can allocate a NON-COLLIDING id is the one that already issued every
   * body heading id. A post whose body contains an H2 titled "Frequently asked
   * questions" gets `frequently-asked-questions-1` back, which is exactly what
   * `rehype-slug` would have produced had the heading been in the body. See
   * `trailingHeadings` in `lib/mdx-plugins/remark-extract-toc.ts` for the bug
   * that made this necessary.
   *
   * Always a string. A route with no FAQ to render simply ignores it.
   */
  readonly faqHeadingId: string
}

/**
 * Two themes, resolved at compile time into `--shiki-light` and `--shiki-dark`
 * custom properties on every token. `styles/agentblog.css` picks between them
 * under `.dark`. `keepBackground: false` drops Shiki's own surface colour so
 * the code block sits on `bg-muted` like everything else.
 */
const prettyCode: PrettyCodeOptions = {
  theme: { light: 'github-light', dark: 'github-dark' },
  keepBackground: false,
}

/**
 * The anchor `rehype-autolink-headings` appends inside each heading. It is a
 * real `<a href="#id">` in the HTML, which is what gives search and retrieval
 * systems clean section boundaries. `styles/agentblog.css` keeps it faint until
 * the heading is hovered or the link is focused.
 */
const autolink = {
  behavior: 'append',
  properties: { className: ['agentblog-anchor'], ariaLabel: 'Link to this section' },
  content: { type: 'element', tagName: 'span', properties: {}, children: [] },
} as const

/** `3:12` or `3:1-3:11` anywhere in a message, which is how MDX writes positions. */
const POSITION_IN_MESSAGE = /\b(\d+):(\d+)(?:-\d+:\d+)?\b/

/**
 * Position of a compile failure, as `:line:column`, or `''` when the compiler
 * did not report one.
 *
 * Two sources, because there are two shapes this error arrives in.
 *
 * A raw `@mdx-js` syntax failure is a `VFileMessage` carrying `line` and
 * `column` as own properties. `VFileMessage` is not exported as a class anywhere
 * we can import from, and an `instanceof` against a transitive dependency's
 * class breaks the moment two copies of `vfile` are installed, so the two fields
 * are read structurally.
 *
 * `next-mdx-remote-client` does not hand that object over. `createFormattedMDXError`
 * in its `dist/lib/util.js` rebuilds a plain `Error` whose message is the
 * original text plus a code frame, so the structured fields are gone by the time
 * we see it and the only position left is the one inside the message. Hence the
 * regex fallback. Some MDX messages carry no position at all, which is why this
 * returns `''` rather than guessing: the code frame the library appends is the
 * better answer in that case, and the file name from `origin` is the part that
 * was actually missing.
 */
function positionOf(error: unknown): string {
  if (typeof error !== 'object' || error === null) return ''

  const { line, column, message } = error as {
    line?: unknown
    column?: unknown
    message?: unknown
  }
  if (typeof line === 'number') {
    return typeof column === 'number' ? `:${line}:${column}` : `:${line}`
  }

  const match = typeof message === 'string' ? POSITION_IN_MESSAGE.exec(message) : null
  return match ? `:${match[1]}:${match[2]}` : ''
}

/**
 * Turn a compile failure into an error that names the post.
 *
 * The compiler never sees a filename, because `renderMdx` is handed a body
 * string by design (see the header). Without this wrapper the thrown error reads
 * `Unexpected character '/' (U+002F) before local name` with a stack pointing at
 * this file, so a red build tells you MDX broke and not which of two hundred
 * posts broke it. The compiler's own message and stack are preserved on `cause`.
 */
function locateFailure(error: unknown, origin: string | undefined): Error {
  const where = origin ?? 'an MDX source with no origin passed to renderMdx()'
  const reason = error instanceof Error ? error.message : String(error)
  return new Error(`Failed to compile MDX in ${where}${positionOf(error)}: ${reason}`, {
    cause: error,
  })
}

/**
 * Compile an MDX string into a server-rendered element plus its table of
 * contents. The source may come from a file, a database row, or an API
 * response. This function does not care and must never start caring.
 *
 * `origin` names where the source came from and is used for one thing: the error
 * message when compilation fails. Pass the file path when the caller has one and
 * the post slug otherwise. It is optional so that a caller with genuinely no
 * origin (a scratch preview, a test) is not forced to invent one, and every
 * caller that has an origin should pass it.
 */
export async function renderMdx(source: string, origin?: string): Promise<RenderedMdx> {
  const { content, scope, error } = await evaluate<
    Record<string, never>,
    { toc: TocEntry[]; trailingToc: TocEntry[] }
  >({
    source,
    components: mdxComponents,
    options: {
      // Frontmatter is parsed by the content source before it ever reaches
      // here, so `Post.body` is already the body alone.
      parseFrontmatter: false,
      /*
       * Strip `import` and `export` from the document. Both default to `false`
       * in `next-mdx-remote-client@2.1.11`, which means nothing is stripped
       * unless you say so: `composeCompileOptions` in `dist/lib/compile.js`
       * passes `[disableExports && 'export', disableImports && 'import']` to
       * `remark-mdx-remove-esm`, and two `undefined`s remove nothing.
       *
       * `import` previously failed only by accident, because `mdxOptions.baseUrl`
       * is unset and `@mdx-js` throws when it has no base to resolve against.
       * An accident is not a control. `export` did not fail at all: a body could
       * write `export const stolen = process.env.SECRET` and read it back off
       * `mod`.
       *
       * Read the trust model in the header for what this does not fix. Stripping
       * ESM is a reduction in surface, not a sandbox, and the seed posts are
       * unaffected because MDX components arrive through the `components` map
       * rather than through an `import`.
       */
      disableExports: true,
      disableImports: true,
      // Lifts both fields written by `remarkExtractToc` onto the result.
      vfileDataIntoScope: ['toc', 'trailingToc'],
      scope: { toc: [], trailingToc: [] },
      mdxOptions: {
        /*
         * `trailingHeadings` is what makes the FAQ heading id come out of the
         * same slugger run as the body ids instead of being guessed by the post
         * route. Read the option's own doc comment before changing this array:
         * the id it reserves is the one thing standing between a post with a
         * body H2 titled "FAQ" and two elements sharing an id.
         */
        remarkPlugins: [
          remarkGfm,
          [remarkExtractToc, { trailingHeadings: [FAQ_HEADING] }],
          remarkAnswerCapsule,
        ],
        rehypePlugins: [
          rehypeSlug,
          [rehypeAutolinkHeadings, autolink],
          [rehypePrettyCode, prettyCode],
        ],
      },
    },
  })

  // `evaluate` swallows compile errors and returns an empty div. Deploying a
  // post whose body silently vanished is worse than a red build, so rethrow,
  // wrapped so the message names the post rather than only this file.
  if (error) throw locateFailure(error, origin)

  /*
   * `trailingToc[0]` is the FAQ entry, because `trailingHeadings` above has one
   * element. The fallback is the unallocated spelling from `components/mdx/faq`
   * and exists so the return type is a plain `string`; it can only be reached if
   * somebody empties that array, in which case the route renders the FAQ with
   * the same id it always had and nothing regresses silently.
   */
  return {
    content,
    toc: scope.toc,
    faqHeadingId: scope.trailingToc[0]?.id ?? FAQ_HEADING_ID,
  }
}

/**
 * `remarkExtractToc` - build the table of contents on the server, once.
 *
 * ---------------------------------------------------------------------------
 * WHAT THIS IS
 * ---------------------------------------------------------------------------
 * A remark (mdast) plugin that walks the parsed document, collects every H2 and
 * H3, and writes the result to `file.data.toc`. `lib/render-mdx.tsx` lifts that
 * value out of the vfile and returns it alongside the rendered content, so a
 * post route gets its TOC from the same compile that produced the article.
 *
 * ---------------------------------------------------------------------------
 * WHY IT IS A COMPILE-TIME PLUGIN AND NOT CLIENT CODE
 * ---------------------------------------------------------------------------
 * The obvious alternative is a `useEffect` that queries `document` for headings
 * after mount. That fails the one requirement this project exists to meet: AI
 * crawlers do not execute JavaScript, so a client-built TOC is invisible to
 * them, and the in-page anchor links that give retrieval systems clean section
 * boundaries never appear in the HTML at all. Building it here means the list
 * ships in the first response byte.
 *
 * ---------------------------------------------------------------------------
 * THE ID PARITY REQUIREMENT (the thing that breaks if you edit carelessly)
 * ---------------------------------------------------------------------------
 * The `id` computed here must be byte-identical to the `id` that `rehype-slug`
 * writes onto the rendered heading, or every TOC link is a dead anchor. Two
 * rules keep that true:
 *
 *   1. We use `github-slugger`, which is the exact module `rehype-slug` itself
 *      uses. It is already in the tree as a transitive dependency of
 *      `rehype-slug`, but we declare it explicitly because relying on another
 *      package's private tree is unsound under strict package managers and
 *      would break the day `rehype-slug` swaps implementations.
 *   2. We call `slugger.slug()` for headings of *every* depth, not only the
 *      H2s and H3s we keep. The slugger's duplicate counter is stateful: two
 *      headings named "Setup" produce `setup` and `setup-1`. Skipping an H4
 *      named "Setup" here while `rehype-slug` counts it would silently shift
 *      every later duplicate suffix by one.
 *
 * A third, subtler point: `rehype-slug` reads heading text from hast, after MDX
 * compilation. `headingText` below mirrors hast's behaviour rather than
 * mdast's default, which is why images contribute nothing (an `<img>` element
 * has no text) and raw HTML is skipped.
 *
 * @see https://docs.agentblog.dev/reference/files
 * @see https://docs.agentblog.dev/concepts/geo-playbook*/
import GithubSlugger from 'github-slugger'
import type { Heading, Nodes, Root } from 'mdast'
import type { VFile } from 'vfile'

import type { TocEntry } from '@/lib/toc'

declare module 'vfile' {
  interface DataMap {
    /** Written by `remarkExtractToc`. Read by `renderMdx` in `lib/render-mdx.tsx`. */
    toc: TocEntry[]
    /**
     * One entry per `trailingHeadings` option, in the order given. Written by
     * `remarkExtractToc`, read by `renderMdx` in `lib/render-mdx.tsx`.
     */
    trailingToc: TocEntry[]
  }
}

/** Depths that reach the reader. H1 comes from frontmatter; H4 and below are too fine to list. */
const TOC_DEPTHS = new Set<number>([2, 3])

export interface ExtractTocOptions {
  /**
   * Headings a route renders AFTER the compiled body, named by their visible
   * text. The FAQ section is the one this exists for: `<Faq>` is rendered by
   * `app/blog/[slug]/page.tsx` from frontmatter, outside the MDX, so no walk of
   * this tree can ever discover it.
   *
   * ---------------------------------------------------------------------------
   * WHY THE ROUTE CANNOT JUST PICK AN ID
   * ---------------------------------------------------------------------------
   * It did, and it shipped a bug. The FAQ heading id was the hardcoded string
   * `faq`, which the slugger below had never issued and therefore never
   * reserved. A post with a body H2 titled "FAQ" slugged to `faq` through
   * `rehype-slug` and the page went out with two `<h2 id="faq">`, two
   * `href="#faq"` entries in the table of contents, and a "Frequently asked
   * questions" link that scrolled to the body heading, because a duplicate id
   * resolves to the first match in the document.
   *
   * Passing the text here instead puts the id through the SAME slugger
   * instance, after the whole body has been walked, so it participates in the
   * duplicate counter exactly as a body heading would. A post that already used
   * the name gets the usual `-1` suffix and the two elements stay distinct.
   *
   * The reservation is unconditional and costs nothing when the route decides
   * not to render the section: it happens after the last body heading, so no
   * body id can shift because of it, and `rehype-slug` runs its own slugger
   * over the compiled body alone and never sees this call at all.
   */
  readonly trailingHeadings?: readonly string[] | undefined
}

export function remarkExtractToc(options: ExtractTocOptions = {}) {
  return (tree: Root, file: VFile): undefined => {
    const slugger = new GithubSlugger()
    const toc: TocEntry[] = []

    forEachHeading(tree, (heading) => {
      const text = headingText(heading)
      // Called for every depth on purpose. See "THE ID PARITY REQUIREMENT".
      const id = slugger.slug(text)
      if (TOC_DEPTHS.has(heading.depth) && text.length > 0) {
        toc.push({ id, text, depth: heading.depth as 2 | 3 })
      }
    })

    file.data.toc = toc

    // After the body walk, never before it. See `trailingHeadings`.
    file.data.trailingToc = (options.trailingHeadings ?? []).map((text) => ({
      id: slugger.slug(text),
      text,
      depth: 2 as const,
    }))
  }
}

/**
 * Visit every heading in document order.
 *
 * Written inline rather than pulled from `unist-util-visit` to keep this
 * plugin's npm surface at exactly one package. Headings nested inside a
 * blockquote or a list item are reached too, because `rehype-slug` reaches them
 * and the duplicate counter has to stay in step.
 */
function forEachHeading(node: Nodes, visit: (heading: Heading) => void): undefined {
  if (node.type === 'heading') visit(node)
  if ('children' in node) {
    for (const child of node.children) forEachHeading(child, visit)
  }
}

/**
 * The text a reader sees in a heading.
 *
 * Mirrors `hast-util-to-string`, which is what `rehype-slug` uses: text and
 * inline code contribute their value, everything else contributes only through
 * its children. Notably an image contributes nothing, because after compilation
 * it is an `<img>` element with no text content.
 */
function headingText(node: Nodes): string {
  if (node.type === 'text' || node.type === 'inlineCode') return node.value
  if ('children' in node) {
    let out = ''
    for (const child of node.children) out += headingText(child)
    return out
  }
  return ''
}

/**
 * `remarkAnswerCapsule` - a lint, not a transform.
 *
 * ---------------------------------------------------------------------------
 * WHAT AN ANSWER CAPSULE IS AND WHY IT GETS ITS OWN PLUGIN
 * ---------------------------------------------------------------------------
 * The answer capsule is the 40 to 60 word direct answer that opens a post. It
 * is the passage an AI engine lifts verbatim when it cites you, which makes it
 * the single highest-value paragraph on the page: state the complete answer
 * first, no preamble, no links inside it. A capsule appeared in 72.4% of
 * ChatGPT-cited pages in Indig's corpus (industry study, directional).
 *
 * Under 40 words the passage rarely answers the question completely, so an
 * engine has to stitch it together with surrounding text and the citation goes
 * to whoever wrote a self-contained one. Over 60 words it gets truncated and
 * the sentence that survives is no longer the answer.
 *
 * The H1 is not in the MDX body: it comes from frontmatter and is rendered by
 * the post route. So the capsule is simply the first paragraph of the document,
 * or the first paragraph after an H1 if an author wrote one anyway.
 *
 * ---------------------------------------------------------------------------
 * WHY THIS WARNS AND NEVER FAILS
 * ---------------------------------------------------------------------------
 * `agentblog audit` is where this rule is enforced properly: it reads the whole
 * corpus, reports every post at once, and exits non-zero in CI. This plugin
 * exists only to surface the same thing in the terminal while you are writing,
 * at the moment the post compiles. A build that refuses to render a post
 * because its opening paragraph is eight words long would be a worse tool than
 * one that renders it and tells you.
 *
 * ---------------------------------------------------------------------------
 * WHY IT ALSO WRITES TO THE CONSOLE
 * ---------------------------------------------------------------------------
 * `file.message()` is the correct unified-ecosystem way to report this, and any
 * tooling that inspects the vfile will find it. But `evaluate()` from
 * `next-mdx-remote-client` returns content, frontmatter, and scope, and does
 * not hand the vfile back to its caller, so a message alone would go nowhere.
 * We mirror it to `console.warn` in development for that reason. Production
 * builds stay quiet: by then `agentblog audit` has already had its say.
 *
 * @see https://docs.agentblog.dev/reference/files
 * @see https://docs.agentblog.dev/concepts/geo-playbook*/
import type { Nodes, Paragraph, Root } from 'mdast'
import type { VFile } from 'vfile'

const MIN_WORDS = 40
const MAX_WORDS = 60
const RULE_ID = 'answer-capsule'

export function remarkAnswerCapsule() {
  return (tree: Root, file: VFile): undefined => {
    const opening = openingParagraph(tree)

    if (!opening) {
      warn(file, `post has no opening paragraph, so it has no answer capsule`, undefined)
      return
    }

    const words = countWords(paragraphText(opening))
    if (words < MIN_WORDS || words > MAX_WORDS) {
      warn(
        file,
        `answer capsule is ${String(words)} words, target is ${String(MIN_WORDS)} to ${String(
          MAX_WORDS,
        )}. This paragraph is what an AI engine quotes, so it has to answer the question on its own.`,
        opening,
      )
    }
  }
}

/**
 * The first paragraph of the body.
 *
 * Only top-level children are considered. A paragraph nested in a blockquote or
 * a list is not an opening answer, and treating it as one would produce a
 * warning the author cannot act on.
 */
function openingParagraph(tree: Root): Paragraph | undefined {
  for (const node of tree.children) {
    if (node.type === 'paragraph') return node
    // An H1 in the body is redundant with the frontmatter title, but authors
    // write one sometimes. Step over it rather than reporting the wrong node.
    if (node.type === 'heading' && node.depth === 1) continue
    if (node.type === 'thematicBreak') continue
    // Anything else first (a table, a code block, a component) means the post
    // opens with something other than prose, which is its own problem.
    return undefined
  }
  return undefined
}

function paragraphText(node: Nodes): string {
  if (node.type === 'text' || node.type === 'inlineCode') return node.value
  if ('children' in node) {
    let out = ''
    for (const child of node.children) out += paragraphText(child)
    return out
  }
  return ''
}

function countWords(text: string): number {
  const trimmed = text.trim()
  return trimmed.length === 0 ? 0 : trimmed.split(/\s+/).length
}

function warn(file: VFile, reason: string, place: Nodes | undefined): undefined {
  const message = file.message(reason)
  message.ruleId = RULE_ID
  message.source = 'agentblog'
  // `fatal: false` is the vfile default and is the whole point here: this is a
  // warning, and nothing downstream is allowed to promote it.
  message.fatal = false
  if (place?.position) message.place = place.position

  if (process.env.NODE_ENV !== 'production') {
    console.warn(`[agentblog:${RULE_ID}] ${reason}`)
  }
}

[
  {
    "//": "KEEP THE SLUG, REPLACE THE VALUES. Both seed posts carry `author: editorial` in their frontmatter, and `defaultAuthor` in agentblog.config.ts points here too. Deleting this record, or renaming its slug, fails the next build with: unknown author slug \"editorial\". Edit name, bio, jobTitle, knowsAbout, and sameAs freely. If you want a different slug, change it here, in `defaultAuthor`, and in the `author:` line of every post in content/blog, in one commit. Keys starting with // are ignored by the schema.",
    "slug": "editorial",
    "name": "The Editorial Team",
    "bio": "Replace this with two or three sentences about who writes here and why they are credible on these topics. This bio is a real ranking surface, not filler: the Search Quality Rater Guidelines direct raters to look authors up, and an answer engine resolving \"who wrote this\" reads this field together with the sameAs links below.",
    "jobTitle": "Editorial",
    "knowsAbout": ["Replace with a topic this author is genuinely credible on", "And another"],
    "sameAs": [
      "https://example.com/replace-with-a-real-linkedin-profile",
      "https://example.com/replace-with-a-real-github-profile"
    ]
  },
  {
    "//": "SAFE TO DELETE. No seed post names this slug, so removing this record breaks nothing. It exists only to show the shape of a roster with more than one person in it.",
    "slug": "guest-author",
    "name": "Guest Author",
    "bio": "Replace this or delete the record. Posts reference an author by slug, so deleting a record is safe exactly when no post and no config field names that slug.",
    "knowsAbout": [],
    "sameAs": []
  }
]

[
  {
    "//": "KEEP THE SLUG WHILE THE SEED POST IS HERE. content/blog/how-ai-search-engines-read-your-blog.mdx carries `category: ai-search`. Renaming or deleting this record before you edit or delete that post fails the next build with: unknown category slug \"ai-search\". Keys starting with // are ignored by the schema.",
    "slug": "ai-search",
    "name": "AI Search",
    "description": "How generative engines such as ChatGPT search, Perplexity, Claude, and Google AI Mode find, read, and cite web pages, and what that means for a blog that wants to be one of the cited sources."
  },
  {
    "//": "SAFE TO DELETE. No seed post names this slug.",
    "slug": "technical-seo",
    "name": "Technical SEO",
    "description": "Rendering, crawlability, structured data, sitemaps, and the parts of a site that decide whether a crawler can read your content at all before anyone argues about the writing."
  },
  {
    "//": "KEEP THE SLUG WHILE THE SEED POST IS HERE. content/blog/what-makes-an-ai-engine-cite-your-post.mdx carries `category: content-strategy`.",
    "slug": "content-strategy",
    "name": "Content Strategy",
    "description": "What to write, how to structure it, and how to keep it accurate over time. Covers answer capsules, question headings, internal linking, and the refresh cadence that keeps a post citable."
  },
  {
    "//": "SAFE TO DELETE. No seed post names this slug. This taxonomy is a starting point, not a recommendation: these four are AgentBlog's own topics, and yours will be different. Five to ten fixed categories is the working range, because each one is an indexable hub page and a hub with two posts on it is a crawl liability. Every category needs a real description for the same reason.",
    "slug": "engineering",
    "name": "Engineering",
    "description": "How the blog itself is built and operated: the rendering path, the publish pipeline, the schema graph, and the checks that catch a regression before a crawler does."
  }
]

---
title: How AI Search Engines Read Your Blog
description: GPTBot, ClaudeBot, and PerplexityBot read the raw HTML your server returns and never run JavaScript. Here is what they get, and how to check it yourself.
answerCapsule: >-
  AI search engines fetch your page once, read the raw HTML your server returns,
  and never execute JavaScript. Anything your blog renders in the browser is
  invisible to them. Prerendered HTML with the article body and the metadata
  already in place is the only format every AI crawler can read.
datePublished: 2026-07-14T09:00:00Z
dateModified: 2026-08-04T16:20:00Z
author: editorial
category: ai-search
tags:
  - AI crawlers
  - GPTBot
  - server-side rendering
  - Next.js
relatedPosts:
  - what-makes-an-ai-engine-cite-your-post
citations:
  - name: The rise of the AI crawler
    url: https://vercel.com/blog/the-rise-of-the-ai-crawler
    author: Vercel and MERJ
    datePublished: 2024-12-17
    kind: industry
  - name: 'Next.js documentation: generateMetadata'
    url: https://nextjs.org/docs/app/api-reference/functions/generate-metadata
    author: Vercel
    kind: official-docs
  - name: AI features and your website
    url: https://developers.google.com/search/docs/appearance/ai-features
    author: Google Search Central
    kind: official-docs
faq:
  - question: Do AI crawlers execute JavaScript?
    answer: >-
      No. Vercel and MERJ analysed roughly 1.3 billion AI crawler fetches across
      their network and reported that none of the major AI crawlers render
      JavaScript. GPTBot and ClaudeBot download .js files but do not run them,
      so client-rendered content never reaches them.
  - question: Does my blog need a separate version for AI crawlers?
    answer: >-
      No. Google Search Central states that you do not need to create new machine
      readable files, AI text files, or markup to appear in AI features. Serve one
      well-formed HTML document to every visitor and every crawler.
  - question: How do I check what an AI crawler sees on my page?
    answer: >-
      Run curl with the crawler's user agent and search the raw response for a
      sentence from your article. If the sentence is missing from the response
      body, the crawler cannot see it either. Use view-source, never the DevTools
      Elements panel, which shows the DOM after JavaScript has run.
  - question: Is a page that ranks first on Google always visible to ChatGPT?
    answer: >-
      No. Googlebot executes JavaScript and the major AI crawlers do not, so a
      client-rendered page can rank in Google Search and still arrive at ChatGPT,
      Claude, and Perplexity as an empty shell.
---

## What does an AI crawler actually receive?

An AI crawler receives one HTTP response and nothing else. It sends a GET request, reads the bytes your server returns, and moves on. No browser starts, no scripts run, no second request fires for data. Whatever text sits in that first response body is the entire article as far as the crawler is concerned.

That single fact reorganises what matters on a blog. Bundle size, hydration cost, and route transitions are human concerns, and they are real ones, but an AI crawler never experiences any of them. It experiences your HTML document exactly once.

The practical test is simple. Open the page, view source rather than the DevTools Elements panel, and look for the opening sentence of your article. The Elements panel shows the DOM after JavaScript has run, which is a different document from the one your server sent. View source shows the response body, which is what GPTBot and ClaudeBot parse.

Most blogs pass this test without knowing it, because static site generators and server-rendered frameworks produce complete documents by default. Blogs fail it when the article body is fetched client side, when content mounts on interaction, or when the page is a single-page application shell that fills itself in after load.

## Do AI crawlers execute JavaScript?

No. Vercel and MERJ instrumented AI crawler traffic across Vercel's network over a month and reported plainly that none of the major AI crawlers render JavaScript. The crawlers do request script files, which is easy to mistake for execution, but requesting a file and running it are different operations.

<Stat
  value="23.84%"
  label="of Claude crawler requests targeted .js files, which the crawler downloaded but never executed"
  source="Vercel and MERJ, The rise of the AI crawler, December 2024"
/>

The same study put the equivalent figure for OpenAI's ChatGPT crawler at 11.50% of requests. Both numbers describe fetching, not rendering. As the authors put it, ChatGPT and Claude crawlers do fetch JavaScript files but they do not execute them, so they cannot read client-side rendered content.

Googlebot is the exception worth naming, because it does execute JavaScript, and Google AI Overviews and AI Mode are grounded in the Google Search index that Googlebot builds. That asymmetry produces the failure mode that surprises people most: a page can rank in Google Search on the strength of JavaScript-rendered content and simultaneously be blank to ChatGPT search, Claude, and Perplexity. Ranking well is not evidence that AI crawlers can read you.

<Callout type="warning">
  Fetching a script is not running it. Server logs showing `.js` requests from GPTBot tell you
  nothing about whether GPTBot saw your article.
</Callout>

## Which AI crawlers visit a blog, and what does each one do?

Six user agents account for most AI crawler traffic to a typical blog, and they do three different jobs: collecting training data, building a citation index, and fetching a page live because a user asked about it. Only the second and third can ever produce a citation, and neither of them runs JavaScript.

| Crawler          | Operator   | What it does                                            | Runs JavaScript |
| ---------------- | ---------- | ------------------------------------------------------- | --------------- |
| GPTBot           | OpenAI     | Collects pages for model training                       | No              |
| OAI-SearchBot    | OpenAI     | Builds the index ChatGPT search cites from              | No              |
| ClaudeBot        | Anthropic  | Collects pages for model training                       | No              |
| Claude-SearchBot | Anthropic  | Builds Claude's search index                            | No              |
| PerplexityBot    | Perplexity | Builds Perplexity's citation index                      | No              |
| Googlebot        | Google     | Builds the Search index behind AI Overviews and AI Mode | Yes             |

Two consequences follow from the table. First, blocking training crawlers in `robots.txt` does not block the search crawlers that generate citations, so the two decisions are separable. Second, since every AI crawler in the list except Googlebot reads raw HTML only, one correctly prerendered document satisfies all of them at once. There is no per-crawler variant to maintain.

Verify crawler identity by IP range rather than by user agent string. OpenAI, Anthropic, and Perplexity each publish their ranges as JSON, and a user agent header is trivially spoofed. Any measurement of AI crawler traffic that trusts the header is partly counting impostors.

## Why does client-side rendering lose in AI search?

Client-side rendering loses because the article does not exist yet when the crawler stops reading. The server sends a shell, the browser runs a script, the script requests the content, and the content arrives in a second round trip that an AI crawler never makes. The crawler sees the shell.

| Rendering approach                         | What the first response body contains | What a non-JavaScript crawler gets |
| ------------------------------------------ | ------------------------------------- | ---------------------------------- |
| Prerendered at build time                  | The complete article                  | The complete article               |
| Server rendered per request                | The complete article                  | The complete article               |
| Fetched client side after load             | An empty container                    | Nothing                            |
| Mounted on click, as in a tab or accordion | Markup absent until interaction       | Nothing                            |

The fourth row catches people who did everything else right. Content hidden with CSS is present in the HTML and is read normally. Content that a JavaScript handler mounts on click is genuinely absent from the response, so an FAQ inside an interactive accordion can be invisible while an FAQ inside a `details` element is fully readable. The distinction is whether the markup exists before interaction, not whether it is visible.

Infinite scroll fails for the same reason, and so does any pagination built on click handlers instead of real anchor elements. If a crawler cannot reach a URL by following an `href`, that URL is not in its crawl frontier.

Google Search Central is direct about the remedy, and the remedy is not a new file format. Its AI features guidance states that "you don't need to create new machine readable files, AI text files, or markup to appear in these features." One well-formed HTML document, served to everyone, is the whole requirement. What you put inside that document is a separate question, covered in [what makes an AI engine cite your post](/blog/what-makes-an-ai-engine-cite-your-post).

## Where does Next.js put your title tag?

Next.js streams metadata for pages that render dynamically, which moves the title and canonical tags out of `<head>` and into `<body>` for any crawler the framework does not recognise as HTML-limited. The Next.js documentation describes the behaviour without ambiguity:

> When `generateMetadata` resolves, the resulting metadata tags are appended to the `<body>` tag.
>
> Next.js documentation, generateMetadata

Next.js exempts a list of HTML-limited bots from streaming and blocks rendering for them so their metadata lands in `<head>` instead. That default list is built around social preview crawlers and traditional search engines. AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot are not on it, which means a dynamically rendered post can serve them a canonical link inside the document body.

There are two fixes and they are not alternatives. The structural fix is to prerender posts, because the same documentation notes that when a page can be prerendered and `generateMetadata` introduces no dynamic behaviour, the resulting metadata is included in the page's initial HTML. The belt-and-braces fix is to set `htmlLimitedBots` in `next.config.ts` to a regular expression that includes the AI crawlers. Note that this config value replaces the default list rather than extending it, so the replacement has to repeat every default entry or you will quietly remove Googlebot and Bingbot from HTML-limited treatment.

## How do you verify what a crawler sees?

Verify with `curl`, one command per crawler, checking for a distinctive sentence from your article in the raw response. This takes under a minute and is the only check that proves anything, because every other tool in the chain shows you a rendered DOM rather than the response body.

```bash
# Does the article body reach a crawler that cannot run JavaScript?
curl -s -A "GPTBot" https://yourdomain.com/blog/your-post \
  | grep -c "a distinctive sentence from your article"

# Does the title tag land in <head> rather than in <body>?
curl -s -A "ClaudeBot" https://yourdomain.com/blog/your-post \
  | sed -n '/<\/head>/q;p' | grep -i "<title>"
```

A count of zero from the first command means the crawler receives a page without your article in it. A missing title from the second means metadata is streaming into the body for that user agent.

Run both against production rather than against a local dev server, because the failure this catches most often is not in your code at all. A CDN or WAF sitting in front of the site can rate-limit or block AI crawlers while your `robots.txt` cheerfully allows them, and the result looks identical to having no content: a crawler that never gets a readable response. Testing from outside your own network with each user agent in turn is what separates the two causes.

<KeyTakeaways
  items={[
    'AI crawlers read one HTTP response and never execute JavaScript, so the first response body is the entire article to them.',
    'Prerendered or server-rendered HTML satisfies every AI crawler at once. There is no per-crawler variant to maintain.',
    'Content mounted on click is absent from the response. Content hidden with CSS is present and readable.',
    'In Next.js, streamed metadata can append the title and canonical tags to the body for AI crawlers. Prerender posts and set htmlLimitedBots.',
    'Verify with curl against production using each crawler user agent. A rendered DOM in DevTools proves nothing.',
  ]}
/>

---
title: What Makes an AI Engine Cite Your Post
description: Answer capsules, question headings, real tables, statistics with sources, and repeated entity names. The formatting that measurably increases AI citations.
answerCapsule: >-
  An AI engine cites a post when a single chunk of it answers the question
  completely on its own. Open with a 40 to 60 word direct answer, use
  question-format headings, put comparative data in a real table, cite named
  sources, and repeat entity names instead of pronouns.
datePublished: 2026-07-28T11:30:00Z
dateModified: 2026-08-05T10:05:00Z
author: editorial
category: content-strategy
tags:
  - generative engine optimization
  - answer capsule
  - content structure
  - citations
relatedPosts:
  - how-ai-search-engines-read-your-blog
citations:
  - name: 'GEO: Generative Engine Optimization'
    url: https://arxiv.org/abs/2311.09735
    author: Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan, and Deshpande
    kind: peer-reviewed
  - name: AI features and your website
    url: https://developers.google.com/search/docs/appearance/ai-features
    author: Google Search Central
    kind: official-docs
  - name: The rise of the AI crawler
    url: https://vercel.com/blog/the-rise-of-the-ai-crawler
    author: Vercel and MERJ
    datePublished: 2024-12-17
    kind: industry
faq:
  - question: How long should an answer capsule be?
    answer: >-
      Forty to sixty words. That is long enough to answer a question completely
      and short enough to be quoted whole. Put the capsule directly under the
      heading it answers, state the answer first with no preamble, and keep
      hyperlinks out of it.
  - question: Do statistics and quotations really increase AI citations?
    answer: >-
      The peer-reviewed GEO paper (ACM KDD 2024) reports that its top-performing
      methods, Cite Sources, Quotation Addition, and Statistics Addition,
      achieved a relative improvement of 30-40% on the Position-Adjusted Word
      Count metric. Those measurements come from 2023 era models, so treat the
      direction as reliable and the exact figures as dated.
  - question: Should I add FAQ schema to every post?
    answer: >-
      Only when the questions and answers are visible on the page. Marking up
      FAQs that a reader cannot see violates Google's structured data policy, and
      FAQ markup is the most common way blogs trip it.
  - question: Does keyword density still matter?
    answer: >-
      No. The GEO paper found keyword stuffing among the methods offering little
      to no improvement in generative engine responses. Repeat entity names for
      retrieval clarity, not keywords for density.
---

## What is an answer capsule, and where does it go?

An answer capsule is a 40 to 60 word paragraph that answers a question completely, placed directly under the heading that asks it. Every post gets one under the H1, and every H2 section gets its own. The capsule states the answer first, with no preamble and no hyperlinks inside it.

The reason is mechanical rather than stylistic. Retrieval systems split a page into chunks, embed each chunk as a vector, and match a reader's question against those vectors. A chunk whose first sentence is the answer survives truncation and produces a clean embedding. A chunk that opens with context, history, or a warm-up paragraph buries the answer in the middle, where it competes with everything around it.

Two rules make an answer capsule work. Answer first, then elaborate: the inverted pyramid that newspapers use for human skimmers happens to be the same shape a retrieval system rewards. And keep links out of the capsule itself, because a hyperlink in the middle of an answer suggests the answer lives somewhere else.

Write every answer capsule as if it will be quoted alone, without the paragraph before it and without the heading above it, because that is exactly how it gets used.

## Which content edits actually increase AI citations?

The strongest evidence comes from the GEO paper presented at ACM KDD 2024, which tested nine content edits against generative engine responses and measured which ones increased a source's visibility. Citing sources, quoting named sources, and adding real statistics led the results. Keyword stuffing did not.

> Our top-performing methods, Cite Sources, Quotation Addition, and Statistics Addition, achieved a relative improvement of 30-40% on the Position-Adjusted Word Count metric and 15-30% on the Subjective Impression metric.
>
> Aggarwal et al., "GEO: Generative Engine Optimization", ACM KDD 2024

<Stat
  value="30 to 40%"
  label="relative improvement in Position-Adjusted Word Count from citing sources, adding quotations, and adding statistics"
  source="Aggarwal et al., GEO: Generative Engine Optimization, ACM KDD 2024"
/>

| Content edit        | What it means in practice                                   | Reported effect                                              |
| ------------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
| Cite Sources        | Link the primary source for every factual claim             | Top-performing group, 30-40% on Position-Adjusted Word Count |
| Quotation Addition  | Quote a named person or organisation, with attribution      | Top-performing group, 30-40% on Position-Adjusted Word Count |
| Statistics Addition | Replace vague quantifiers with a real number and its source | Top-performing group, 30-40% on Position-Adjusted Word Count |
| Keyword Stuffing    | Repeat the target phrase to raise its density               | Reported as offering little to no improvement                |

Two caveats belong next to those numbers. The GEO paper's measurements were taken on 2023 era models, so the ranking of the methods is more durable than the percentages attached to them. And the paper's abstract frames the overall result as a boost of up to 40% in generative engine responses, which is a ceiling rather than an expectation.

None of this matters if the crawler cannot read the page at all. See [how AI search engines read your blog](/blog/how-ai-search-engines-read-your-blog) for the rendering side of the problem, which sits upstream of every writing decision on this page.

## Why do question-format headings work?

A question-format heading matches the shape of the query it should answer. When a reader asks an assistant "do AI crawlers execute JavaScript", a section headed with that exact question produces a chunk whose heading, embedding, and opening sentence all point at the same thing. A heading like "Crawler capabilities" points at nothing in particular.

Mine the phrasing rather than inventing it. Google's People Also Ask boxes, the query report in Search Console, and the questions people actually type on Reddit are three free sources of real wording. The value is in matching how the question gets asked, not in guessing at a keyword.

Give every H2 and H3 a stable `id` so the heading is addressable. Stable anchors let a table of contents link into the page, let Google offer jump-to links, and give an extraction pipeline clean section boundaries to cut on. Changing those ids later breaks every inbound deep link, so derive them from the heading text once and leave them alone.

Nest headings without skipping levels. An H2 followed directly by an H4 is a broken outline, and the outline is one of the few structural signals a parser can read without any ambiguity at all.

## How long should a section be?

Aim for 150 to 300 words per H2 section, and make each section answer its own heading completely without depending on the sections around it. That range is short enough to fit inside a typical retrieval chunk and long enough to say something substantive.

Self-containment is the harder discipline. A section that refers back to "the table above" or "as we saw earlier" produces a chunk that is incomprehensible on its own, because the chunk reaches the model without the table and without the earlier section. Repeat the small amount of context a section needs, even when the repetition feels redundant to a reader going top to bottom.

One idea per paragraph, and the highest-value sentence first in each one. When a chunk gets truncated, the truncation happens at the end, so anything saved for a closing flourish is the first thing lost.

Total length follows from coverage rather than from a target. Longer posts do get cited more often, but length is a proxy for completeness, not a cause of citation. A 900 word post that fully answers a narrow question beats a padded 2,500 word post, and the padding degrades the chunk quality of every section it touches.

## Why do tables beat prose for extraction?

A markdown table encodes relationships that prose only implies. Three columns and five rows give an extraction pipeline fifteen labelled cells, each one carrying its row key and its column key. The same information in a paragraph gives it one block of text from which those relationships have to be reconstructed, imperfectly.

| How you present comparative data | What a parser recovers                      | What a reader recovers                   |
| -------------------------------- | ------------------------------------------- | ---------------------------------------- |
| Markdown or HTML table           | Every cell with its row and column labels   | Fast scanning and comparison             |
| Prose paragraph                  | Unstructured text, relationships implied    | Slow comparison, easy to lose the thread |
| Bulleted list                    | Items, but no shared dimensions across them | Fine for lists, poor for comparisons     |
| Screenshot of a table            | Nothing, without OCR                        | Fine, unless the image fails to load     |

That last row is the expensive mistake. A chart or table that exists only as an image is invisible to a crawler that does not run OCR, and it is invisible to a screen reader too. Original data of your own is one of the strongest assets a blog can publish, so pair every chart image with the underlying numbers in a real table beside it.

Use tables where the data genuinely has shared dimensions. Forcing prose into a two-column table to look structured produces a table with one meaningful column, which helps nobody and costs the reader a scan.

## Why repeat entity names instead of pronouns?

Because a chunk is retrieved without the sentences that came before it. Write "Next.js Cache Components" rather than "it", and "PerplexityBot" rather than "the crawler". When the paragraph holding the pronoun is retrieved on its own, the referent is gone and the chunk describes an unnamed thing.

Entity repetition is not keyword stuffing, though the two are easy to confuse. Keyword stuffing repeats a phrase to raise its density whether or not the repetition means anything, and the GEO paper found keyword stuffing among the methods offering little to no improvement. Entity repetition names the specific thing a sentence is about so that the sentence still parses in isolation. One is padding, the other is precision.

The same principle applies across a whole site rather than only inside one post. Naming products, people, and organisations consistently helps an engine resolve them to a single entity, and the `sameAs` links in your Person and Organization structured data are what connect that name to a real profile somewhere else on the web.

Google Search Central is clear that none of this requires special markup: "you don't need to create new machine readable files, AI text files, or markup to appear in these features." Ordinary, well-structured HTML with accurate structured data is the entire surface.

## What actively hurts?

Four things reliably work against citation, and all four are common. Keyword stuffing measurably underperforms. FAQ markup with no visible FAQs on the page violates Google's structured data policy. Content mounted on click never enters the crawler's copy at all. And a `dateModified` bumped without a real content change teaches engines to distrust your freshness signals.

<Callout type="warning">
  Never invent a statistic to fill a "statistics increase citations" slot. A number without a
  verifiable source is worse than no number, because the first reader who checks it learns that your
  citations cannot be trusted. If a figure cannot be sourced, state the claim qualitatively instead.
</Callout>

Freshness deserves its own note. Updating strong posts on a schedule is genuinely useful, and changing the modified date only when the content actually changed is the discipline that makes the signal mean anything. A refresh routine that bumps dates unconditionally manufactures exactly the distrust it was meant to avoid.

The rendering failures are worth re-checking on the same schedule, since a post that was readable at launch can become unreadable after a redesign. The `curl` checks in [how AI search engines read your blog](/blog/how-ai-search-engines-read-your-blog) take under a minute per post and catch that regression before a crawler does.

<KeyTakeaways
  items={[
    'Open every post and every H2 section with a 40 to 60 word direct answer, with no links inside it.',
    'Use question-format H2s taken from real query phrasing, each with a stable id.',
    'Keep sections to 150 to 300 words and make each one comprehensible on its own.',
    'Put comparative data in a real table. A table that exists only as an image is invisible to a crawler.',
    'Cite named sources and use real statistics. Never fabricate a number to fill the slot.',
    'Repeat entity names instead of pronouns, because a retrieved chunk arrives without its context.',
  ]}
/>