title: Build From Scratch: Mintlify-like AI-driven Documentation Generator CLI - Part 038 description: Mendesain code example execution dan verification untuk documentation generator: extracting runnable examples, sandbox policy, language runners, dependency setup, expected outputs, snapshot tests, API mocks, deterministic verification, safety gates, and CI integration. series: learn-mintlify-like-ai-docs-cli seriesTitle: Build From Scratch: Mintlify-like AI-driven Documentation Generator CLI order: 38 partTitle: Code Example Execution and Verification tags:

• documentation
• ai
• cli
• code-examples
• testing
• sandbox
• quality-gates
• developer-tools date: 2026-07-03

Part 038 — Code Example Execution and Verification

Code examples adalah bagian dokumentasi yang paling sering dicopy.

Jika code example salah, user langsung kehilangan trust.

Examples yang salah bisa muncul dari:

• code lama,
• API berubah,
• SDK berubah,
• CLI flag berubah,
• dependency berubah,
• environment berubah,
• AI hallucination,
• manual typo,
• syntax error,
• missing import,
• command destructive,
• secret hardcoded,
• dan code block yang bukan intended runnable example.

Karena itu, docs generator production-grade harus punya sistem:

code example execution and verification

Tujuannya bukan mengeksekusi semua code block secara membabi buta.

Tujuannya:

• ekstrak example yang memang runnable,
• klasifikasikan jenisnya,
• jalankan dalam sandbox/isolated environment,
• validasi output/exit code,
• gunakan mock untuk API calls,
• cek syntax untuk bahasa yang tidak dieksekusi penuh,
• scan safety,
• dan jadikan hasilnya quality gate.

1. Mental model: examples are tests

Treat documentation examples like tests.

Not every code block is executable. But every executable example should be verifiable.

2. Example categories

export type CodeExampleKind =
  | "shellCommand"
  | "cliCommand"
  | "httpRequest"
  | "apiSample"
  | "sdkSample"
  | "sourceSnippet"
  | "configSnippet"
  | "outputSnippet"
  | "pseudoCode"
  | "nonRunnable";

Examples:

Verification strategy depends on kind.

3. Verification levels

export type VerificationLevel =
  | "none"
  | "parse"
  | "lint"
  | "compile"
  | "execute"
  | "executeWithMock"
  | "snapshot";

4. Example metadata

Use code fence metadata or MDX props.

```bash title="Build docs" docforge verify
docforge build --strict

Runner:

export class CliCommandExampleRunner implements CodeExampleRunner {
  id = "cli-command";

  supports(example: ExtractedCodeExample): boolean {
    return example.kind === "cliCommand";
  }

  async plan(example: ExtractedCodeExample, ctx: VerificationContext): Promise<ExampleExecutionPlan> {
    const command = parseShellCommand(example.code);

    return {
      exampleId: example.id,
      runnerId: this.id,
      sandbox: ctx.sandboxPolicy,
      steps: [{
        type: "runCommand",
        command: command.command,
        args: command.args,
        cwd: resolveExampleCwd(example, ctx),
        env: minimalEnv(),
      }],
      expected: {
        exitCode: example.metadata.expectedExitCode ?? 0,
      },
    };
  }
}

Need command allowlist.

15. Shell parsing

Do not execute shell string directly if possible.

Parse simple command:

export type ParsedCommand = {
  command: string;
  args: string[];
};

For multi-line shell commands, use shell only in stricter sandbox or parse line by line.

Initial safe approach:

• support single command,
• support line continuation \,
• reject pipes/redirection unless allowed,
• reject command substitution,
• reject && chains unless explicitly allowed.

Diagnostic:

warning example.shell.unsupportedSyntax
Shell example uses pipes or command chaining and cannot be executed safely.

16. JavaScript/TypeScript verification

Levels:

• parse using parser,
• typecheck in fixture tsconfig,
• execute with Node if standalone,
• mock API calls.

Generated fetch examples may require environment.

Simpler initial:

• syntax parse,
• optional execute if metadata says.

Plan:

1. write example.ts,
2. create minimal package.json,
3. install? avoid network,
4. use project dependencies if available,
5. run node/tsx only if tool exists.

Better for generated HTTP examples:

• verify code generation snapshot,
• do not execute network calls by default.

17. Python verification

Levels:

• ast.parse for syntax,
• execute in temp dir if no external dependencies,
• skip if imports unavailable,
• use mock for requests if configured.

Python requests is not standard library. If unavailable, syntax parse still useful.

Runner can detect imports:

export function extractPythonImports(code: string): string[] {
  // simple parser or regex fallback
  return [];
}

If imports include unavailable packages, skip compile/execute with diagnostic.

18. Java verification

Levels:

• syntax/compile via javac,
• execute if full class with main,
• SDK samples require dependencies.

Java snippets often are not full files. Metadata can provide wrapper.

```java docforge:example kind="sdkSample" verify="compile" wrapper="java-main"
AcmeClient client = ...

But imports/classes must exist. For SDK, need dependency setup.

Initial: syntax-ish by wrapping only when dependencies available; otherwise skip.

19. Go verification

Go examples can be compiled if full file.

For snippet:

• wrap in package main,
• add imports if generator knows them.

Use gofmt/go test if Go installed.

Plan:

write main.go
go test ./...

No network by default, so dependencies must be local/module cached.

20. SDK sample verification

SDK samples need package availability.

Options:

1. verify against local SDK source in monorepo,
2. verify against installed package,
3. syntax only,
4. mock SDK types,
5. skip.

Best for build-from-scratch:

• if SDK package exists in repo, use workspace.
• if generated docs for external SDK, require verification fixture.

SDK verification config:

{
  "examples": {
    "sdk": {
      "typescript": {
        "packageRoot": "packages/sdk-js",
        "verify": "typecheck"
      },
      "java": {
        "modulePath": "sdk/java",
        "verify": "compile"
      }
    }
  }
}

21. API sample verification with mock server

HTTP examples should not call real production APIs in CI.

Use mock server generated from OpenAPI examples.

Mock verifies:

• method,
• path,
• query required,
• body shape,
• auth placeholder maybe,
• content type.

No real network.

22. Mock server model

export type MockServer = {
  start(registry: OpenApiRegistry): Promise<MockServerHandle>;
};

export type MockServerHandle = {
  baseUrl: string;
  stop(): Promise<void>;
  requests(): MockRequestRecord[];
};

For example cURL:

curl -X POST "$MOCK_API_URL/users" ...

During verification, replace server URL with mock server URL.

Request builder/code samples should support verification base URL override.

23. HTTP request verification

For cURL sample:

1. sanitize/replace base URL,
2. execute cURL if available,
3. mock server records request,
4. validate request matched operation,
5. validate exit code,
6. validate response status.

If cURL unavailable, parse sample and verify generated request model.

24. Avoid real network by default

Config:

{
  "examples": {
    "execution": {
      "allowNetwork": false
    }
  }
}

If example requires network:

```bash docforge:example requiresNetwork verify="none"
curl https://api.example.com/status
```

Quality gate can warn:

warning example.network.skipped
Example requires network and was not executed.

25. Expected output verification

Example metadata:

```bash docforge:example verify="execute" expectedOutput="Build completed"
docforge build
```

Model:

export type ExpectedOutput =
  | { type: "contains"; text: string }
  | { type: "regex"; pattern: string }
  | { type: "snapshot"; snapshotId: string };

Snapshot file:

__snapshots__/quickstart-build.stdout

Be careful: output can include paths/timestamps. Normalize.

26. Output normalization

export function normalizeExampleOutput(output: string): string {
  return output
    .replace(/\r\n/g, "\n")
    .replace(/\/tmp\/docforge-[A-Za-z0-9_-]+/g, "<TMP>")
    .replace(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+Z/g, "<TIMESTAMP>")
    .trim();
}

Snapshots should be stable.

27. Environment variables

Examples may use env vars:

API_TOKEN=... docforge ...

Do not require real secret.

Use placeholder env:

export function buildExampleEnv(example: ExtractedCodeExample): Record<string, string> {
  return {
    ...minimalEnv(),
    API_TOKEN: "test-token",
    DOCFORGE_TEST_MODE: "1",
  };
}

If env var is secret-like, use fake value. Never pull from real environment unless configured.

28. Dependency setup

Some examples require dependencies.

Metadata:

```ts docforge:example verify="execute" setup="pnpm install --offline"
...
```

But setup commands can be dangerous/slow.

Prefer prebuilt fixtures.

Config:

{
  "examples": {
    "fixtures": {
      "quickstart": "fixtures/examples/quickstart"
    }
  }
}

Example metadata:

```bash docforge:example fixture="quickstart" verify="execute"
docforge build

Runner copies fixture to temp dir, runs command.

export async function prepareFixtureCwd(
  fixturePath: string,
  ctx: VerificationContext
): Promise<string> {
  const cwd = await createTempDir(ctx.tempRoot);
  await copyDirectory(fixturePath, cwd);
  return cwd;
}

This makes examples reproducible.

30. Generated example verification

Generated code samples from Part 026 can be verified at generation time.

export type GeneratedCodeSample = {
  ...
  verification?: ExampleVerificationResult;
};

Flow:

generate sample -> verify sample -> attach status -> render docs

If sample fails, do not render or mark with diagnostic.

31. Example provenance

Verification result should be stored with provenance.

export type ExampleVerificationRecord = {
  exampleId: string;
  pageId: PageId;
  blockId: string;
  codeHash: string;
  verificationLevel: VerificationLevel;
  status: "passed" | "failed" | "skipped" | "blocked";
  runnerId: string;
  runnerVersion: string;
  environmentHash: string;
  verifiedAt: string;
  diagnostics: Diagnostic[];
};

If code changes, verification stale.

32. Environment hash

Verification depends on environment.

export type VerificationEnvironment = {
  os?: string;
  nodeVersion?: string;
  pythonVersion?: string;
  javaVersion?: string;
  goVersion?: string;
  packageLockHash?: string;
  runnerVersion: string;
};

Hash:

export function hashVerificationEnvironment(env: VerificationEnvironment): string {
  return sha256(stableJson(env));
}

If environment changes significantly, verification may need rerun.

33. Verification cache

Cache by:

• example code hash,
• metadata hash,
• runner version,
• fixture hash,
• environment hash,
• relevant source hashes.

export type ExampleVerificationCacheKey = {
  exampleId: string;
  codeHash: string;
  metadataHash: string;
  runnerId: string;
  runnerVersion: string;
  fixtureHash?: string;
  environmentHash: string;
};

If same, reuse result.

34. Code example quality gate

Quality gate:

export async function runCodeExampleGate(input: QualityGateInput): Promise<QualityGateResult> {
  const examples = input.compiledPages.flatMap(extractCodeExamples);
  const results = await verifyExamples(examples, input);

  const diagnostics = results.flatMap((result) => result.diagnostics);

  return {
    gateId: "code.examples",
    category: "codeSamples",
    status: diagnostics.some((d) => d.severity === "error") ? "fail" : "pass",
    diagnostics,
    stats: {
      examples: examples.length,
      passed: results.filter((r) => r.status === "passed").length,
      failed: results.filter((r) => r.status === "failed").length,
      skipped: results.filter((r) => r.status === "skipped").length,
      blocked: results.filter((r) => r.status === "blocked").length,
    },
  };
}

Policy:

• generated code sample failure: error,
• manual runnable example failure: warning/error by config,
• unsafe blocked example: error,
• non-runnable skipped: okay.

35. Verification policy config

{
  "examples": {
    "verify": {
      "enabled": true,
      "defaultLevel": "parse",
      "generatedSamples": "executeWithMock",
      "manualSamples": "parse",
      "failOnGeneratedFailure": true,
      "failOnManualFailure": false,
      "failOnUnsafe": true,
      "allowNetwork": false,
      "timeoutMs": 10000
    }
  }
}

36. Skip policy

Allow explicit skip with reason.

```bash docforge:example verify="none" reason="Requires a live production API token"
curl https://api.example.com/users
```

But skipped generated sample should warn unless justified.

Diagnostic:

info example.verification.skipped
Example verification skipped: Requires a live production API token.

If no reason:

warning example.verification.skippedWithoutReason
Runnable example skipped verification without reason.

37. Non-runnable examples

Some code blocks are illustrative.

Mark:

```ts docforge:example kind="pseudoCode" verify="none"
function algorithmSketch() {
  ...
}
```

Pseudo-code should not be tested.

But generated formal API samples should not be pseudo-code.

38. Example expected failure

Some docs show failing command.

docforge build
# expected to fail because config is invalid

Metadata:

```bash docforge:example verify="execute" expectedExitCode="3"
docforge build

This lets troubleshooting docs be tested.

39. Multi-step examples

Some examples require sequence.

```bash docforge:example-group="quickstart" step="1"
docforge init
```

```bash docforge:example-group="quickstart" step="2"
docforge dev
```

Group model:

export type ExampleGroup = {
  id: string;
  examples: ExtractedCodeExample[];
  sharedCwd: boolean;
  setup?: ExecutionStep[];
  cleanup?: ExecutionStep[];
};

Verify group in order.

Be careful with long-running commands like dev.

For dev server, use --once or skip.

40. Long-running commands

docforge dev does not exit.

Verification needs mode:

docforge dev --once
docforge dev --check

If command lacks test mode, do not execute directly.

Metadata:

```bash docforge:example verify="parse"
docforge dev