LLM Lab

LLM Lab grows several token sequences from the same prompt. Each walker carries one complete generated prefix. A step extends those prefixes, scores the new tokens, and embeds the resulting text. Wave or Graph then uses reward and embedding distance to choose which branches supply the next continuations. Cloning carries the donor’s complete prefix, score, and ancestry together. The next request supplies that selected prefix as the assistant text to continue, including every earlier chunk in the donor’s sequence.

Open the /llm/ application alongside Continuous-control laboratory and Optimization Lab. Generation contains the model and search settings, run controls, and leading traces. Analysis lets you follow branches, replay recorded iterations, and inspect the decisions behind cloning. Benchmark compares full answers, distributions, and computational work across sampling methods. Evaluation, the fourth tab, contains optional model grading, judge score charts, the trace browser, and pinned answer comparisons. Benchmark and Evaluation share the selected source, trials, and saved report, including a standalone current Fractal recording. Inspect the text, token probabilities, and clone history before interpreting the search’s best score. A high likelihood measures the selected model’s preference for the generated text; it does not establish that the answer is correct or useful.

Start a small run

From the repository root, launch the lab with the command below. The default port is 8080; open http://localhost:8080/llm/.

make llm-lab

Enter an OpenRouter API key and a prompt. The key stays in browser memory for the session and is excluded from recordings. Generation and embedding requests go to OpenRouter; the prompt and whichever text you select for embeddings are sent with those requests. Optional Mean XED scoring also sends the question and generated answers to Together AI, using a separate session key. Negative mean Xent and beam-style scoring need no additional model requests.

The initial generation model is qwen/qwen3.5-35b-a3b, with Alibaba (alibaba) preferred during route discovery. The embedding model is openai/text-embedding-3-small. Model identifiers are editable; DeepSeek remains selectable. Every selected route must supply token log probabilities and pass the continuation checks at the start of each run. The lab then pins the provider and requests the required parameters explicitly. A model appearing in a catalog does not establish that its endpoint implements the needed continuation behavior. See OpenRouter’s assistant-prefill reference.

Before generating the population, the lab tries up to four endpoints that advertise token log probabilities. A candidate must pass an exact-copy prose probe at temperature zero, using three requests of up to eight tokens each. The first must return the beginning of the expected prose. The second must match the next part exactly when given the first chunk as its assistant prefix. The third receives both preceding chunks and must match the next text after that complete prefix. The first two responses must end at their token limits; a model stop is never continued. Restarting, repeating the prefix, or changing the expected text rejects the route for that run. Rejected routes and their reasons are recorded.

The first successful route is pinned for the run, with no provider fallback during population generation. One embedding probe follows the successful generation checks. These checks use your API key and consume provider usage; their probe data and usage are recorded separately from the search population. Probe metadata stores the expected text and the check identifier prose_suffix_v1, so a saved check can be inspected directly.

Earlier live Wave and Graph runs on 2026-09-09 used DeepSeek Flash through StreamLake (streamlake/fp8) and openai/text-embedding-3-small. Their checks validated recording structure, clone ancestry, and .fgllm export/import, but the original continuation probe failed to detect restarted answers. Inspecting saved chunks showed correct donor prefixes followed by newly returned text that began the answer again. Those runs therefore did not establish correct continuation. Supplying DeepSeek’s explicit prefix flag also proved insufficient: StreamLake received it but still intermittently restarted the probe. A later list probe gave false confidence on Qwen through Venice, which continued numbered lines but restarted prose. The three-chunk prose probe supersedes both weaker checks.

On 2026-09-09, Qwen through Alibaba passed the three-chunk prose check twice with its required partial-message flag. Live Wave and Graph runs on the sky-color prompt used 4 initial walkers, 32 tokens per chunk, a 256-token sequence cap, a 24-iteration limit, concurrency 2, a Graph population cap of 32, temperature 0.6, and seed 7. Wave saved four EOS completions after five iterations and four clone events, using 562 of its 1,024 generated-token allowance. Graph ended with token_budget after 19 committed iterations and 29 clone events, using exactly 1,024 tokens. It had no EOS completions; its deepest trace contained 96 tokens, and unfinished traces remained partial. Graph therefore exercised the budget ending without claiming completion-target success or marking branches capped prematurely.

Inspection of both runs confirmed that outgoing continuation requests contained complete donor prefixes and returned chunks continued the inherited prose coherently. Wave’s completed answers and Graph’s partial traces were preserved. This is evidence for those tested requests; each new run still revalidates the selected route.

Start with Wave and press Step. Inspect the generated tokens, then step again to check continuation from the same prefix. Press Run to advance repeatedly. Pause finishes the current iteration before pausing. Stop cancels pending work and retains the last completed population. Reset starts a fresh run with the selected configuration. Changes to the prompt, models, scoring, or search settings require a new run. Once the run reaches one of its normal stopping conditions, Run and Step stay disabled until Reset.

Control	Initial value	What it controls
Algorithm	Wave	Fixed population or Graph’s growing search
Walkers	8	Initial population size and saved EOS completion target
Tokens per step	32	Maximum new tokens requested per continuation
Sequence cap	256	Maximum generated tokens in a branch
Concurrency	4	Simultaneous generation requests
Iteration limit	32	Maximum search iterations
Temperature	1	Generation sampling parameter
Reward coefficient	1	Reward contribution to cloning fitness
Distance coefficient	1	Diversity contribution to cloning fitness
Graph population cap	256	Maximum Graph population
Objective	Beam-style length normalization	Full-sequence score used to guide cloning and rank answers
Beam α	0.6	Length exponent, adjustable from 0 to 2 when beam scoring is selected
XED direction	Maximize	Whether larger or smaller mean XED guides selection
XED scoring model	Qwen/Qwen3.5-9B	Together model used for both supplied-text evaluations
Embedding input	Generated sequence only	Text represented by the observation vector
Distance	Cosine	Observation comparison used for diversity

Graph’s Freeze shared Graph prefix after N nodes setting defaults to 0 (disabled). A positive threshold archives the path from the active root up to the first branching point once N unbranched nodes lie between them. The archived path is excluded from the active population cap and drawn green in Analysis. Its sequence ancestry remains in exported recordings, so any surviving answer can still be traced back to the prompt.

The seed controls the lab’s own sampling choices. Remote generation need not reproduce identical tokens from the same seed, even when a provider accepts a seed parameter. Keep the recorded responses when you need to inspect exactly what happened.

A request can return fewer tokens than the chunk limit. A model stop records an end-of-sequence (EOS) completion; reaching the sequence cap marks a branch capped. Reaching only the per-step token limit leaves the branch available for continuation. The trace records the actual generated-token count and finish reason separately.

Keep the answer when a branch finishes

Think of a walker slot as a place to do more work, and its saved trace as the work already done. When the provider returns finish_reason: "stop", that branch has finished. The lab keeps its text, token probabilities, score, and ancestry in an immutable EOS node. It never asks the model to continue that node. Cloning can then assign the available slot to an unfinished branch, carrying that branch’s actual prefix into the next request. This reuse does not erase the answer that freed the slot.

The target is N saved EOS completions, where N is the initial walker count. These accumulate over the run; the current walkers need not all finish in the same iteration. Each distinct saved EOS node counts once. Replaying a cached transition or copying a finished trace does not add a completion, but two independent generations of identical text count separately. A stop with zero new tokens still records a completion. If its complete answer is empty, it counts toward the target but is ineligible for Best.

Reaching the target stops queued requests from starting. Requests already in flight finish and are preserved, so the saved total can exceed N. The progress display shows saved EOS completions against the target. A capped branch also keeps its trace, but it does not count as an EOS completion. If no unfinished branches remain, the run ends with the evidence it has; it does not restart the prompt to manufacture more candidates.

There is also a shared allowance of N × sequence cap newly generated scored tokens. Suppose you start eight walkers with a 256-token sequence cap. The run may generate at most 2,048 such tokens, even if Graph grows beyond eight slots. A ten-token prefix copied into four slots still costs ten generated tokens. Inherited prefixes and cached transition replays consume no additional allowance. Probes and embeddings are outside this counter; tokens generated on a branch that selection later discards still count.

Concurrent requests reserve their token allowances before starting. Each request fits both the branch’s remaining sequence cap and the unreserved run allowance. A shorter response releases unused reservations. The final request may therefore ask for fewer tokens than the configured chunk size. Reaching the shared allowance preserves unfinished traces as partial; it does not turn them into EOS completions or claim that they reached their own sequence caps.

After committing an iteration, the lab checks the reasons below in order. The first applicable reason becomes the recorded ending, shown alongside generated-token usage and EOS progress. Only eos_target means the requested number of completed answers was obtained. The other endings still preserve the run for inspection and export. Reset before starting another run.

Priority	Recorded reason	What ended the run
1	eos_target	At least N distinct EOS nodes were saved
2	token_budget	The N × sequence-cap generated-token allowance was used
3	no_active_branches	No unfinished branches remain available for continuation
4	iteration_limit	The configured number of search iterations was reached

Read likelihood with its context

Every generation request includes the original prompt. After the first chunk, the complete generated prefix is supplied as the assistant prefill. After cloning, this is the donor’s full sequence, including its inherited text. The provider must also recognize that this final assistant message is unfinished. For Qwen, the lab sets partial: true on that message, as required by Alibaba’s partial-mode API. For deepseek/* models, it sets prefix: true. These flags construct explicit continuation requests; the observed DeepSeek restarts show why sending a flag alone is insufficient evidence that a route honors it. There is no extra user instruction between chunks.

A new token’s probability uses both the prompt and the preceding generated text. Only newly returned tokens add to the recorded score; sending the prefix again does not score it again. The prose-suffix preflight checks that the selected route honors continuation on its test text. Inspect the actual chunk boundaries as well: correct ancestry alone does not establish that the provider continued the answer as requested.

Definition 7 (LLM Lab objectives)

For a generated sequence of \(n>0\) scored tokens, let

\[ L_n=\sum_{t=1}^{n}\log p(x_t\mid\text{prompt},x_{<t}). \]

Beam-style length normalization, the default for new generation runs, has score \(S=L_n/n^\alpha\), with \(0\leq\alpha\leq2\) and default \(\alpha=0.6\). At \(\alpha=0\) it equals total log likelihood; at \(\alpha=1\) it equals negative mean Xent, \(S=L_n/n\), measured in nats per generated token. Legacy Total likelihood uses \(S=L_n\), in nats.

For Mean XED, let a fixed scoring model \(J\) tokenize the complete answer into \(m>0\) answer tokens \(y_1,\ldots,y_m\). Let \(c(q)\) be the recorded assistant scoring format containing the original question \(q\), and let \(c(\varnothing)\) be the identical format with an empty question. Define

\[ L_{J,\mathrm{conditional}} =\sum_{t=1}^{m}\log p_J(y_t\mid c(q),y_{<t}),\qquad L_{J,\mathrm{baseline}} =\sum_{t=1}^{m}\log p_J(y_t\mid c(\varnothing),y_{<t}), \]

\[ S_{\mathrm{XED}} =\frac{L_{J,\mathrm{conditional}}-L_{J,\mathrm{baseline}}}{m}. \]

Mean XED is measured in nats per scorer token. Both evaluations must use identical answer token IDs. Prompt and formatting tokens do not enter either sum, and scorer tokens need not agree with generator tokens.

The engine maximizes a cumulative utility \(U\). For negative mean Xent, beam scoring, and total likelihood, \(U=S\). For XED, \(U=S_{\mathrm{XED}}\) when maximizing and \(U=-S_{\mathrm{XED}}\) when minimizing. An environment transition earns \(U(\text{child})-U(\text{parent})\). The empty sequence has internal utility zero and is excluded from Best-answer selection.

Think of Xent as the model’s surprise bill for the words it sees. A familiar word in a familiar context adds a small charge; an unlikely word adds a larger one. Every extra token adds another nonnegative charge, so comparing whole bills automatically disadvantages longer answers. Negative mean Xent compares the charge per token and changes its sign so that larger values are preferred. The beam exponent lets you adjust how strongly length enters this comparison. It is a decoding preference, not a test of factual accuracy.

Now XED asks a different question: how much did supplying this question change the scorer’s surprise about this answer? If an answer has conditional log likelihood −12 and baseline log likelihood −24 across six scorer tokens, its mean XED is 2 nats per token. Maximizing favors that increase in predictability; minimizing favors the reverse. The displayed XED remains 2 in either mode. Only the internal utility changes sign. The baseline still contains the recorded assistant format, so this is a comparison with an empty question in that format, not an unconditional probability over arbitrary text.

Rewards compare complete prefixes. Adding the means of separate chunks would give chunk size an unintended role in selection. Cloning therefore carries the donor’s complete text and cumulative utility; the next transition compares its child’s utility with that donor’s value.

For likelihood and answer-XED objectives, Best selects among nonempty finished or capped traces using the configured utility. Until one exists, the displayed best partial trace is selected from the greatest reached token depth and is labelled partial. In Xent game mode, Best instead considers every successfully scored nonempty prefix, regardless of depth or termination. Empty roots and empty completed answers are ineligible. Wave’s historical elite reinjection is disabled for this lab. Terminal branches are also ineligible for native best-walker protection, so Graph can recycle their slots. Best still selects from the immutable recording archive, which keeps previously observed candidates after recycling.

Generation scores use the provider’s returned token log probabilities under the chosen model and request settings. Responses must contain finite, valid probabilities aligned with the generated text. Missing or placeholder values cause an error. The lab never substitutes zero for a missing probability and never invents an unavailable end-of-sequence probability. Termination remains a separate recorded fact.

Score supplied answers with Together

Choose Mean XED to reveal its direction, scoring model, and Together API-key controls. Both XED evaluations use the same model and a fixed non-thinking assistant format. The initial scorer is Qwen/Qwen3.5-9B. Together’s completions API documents echo and logprobs for supplied-text scoring, but that interface alone does not establish that a particular hosted model supports the required fields. The lab probes the selected scorer before starting XED generation. It checks answer boundaries, Unicode text, coverage of supplied-token probabilities, and finite values. An unsupported scorer stops preparation; it does not cause a silent switch to another objective. Live support requires a successful model-specific probe; the model name is not evidence of a verified route.

Live checks on 2026-09-09 verified supplied-text scoring for Together’s Qwen/Qwen3.5-9B. The endpoint rejected max_tokens: 0; the one-token fallback worked. Three exact supplied-text probes, including Unicode, passed, and both XED terms were scored successfully for a short answer. This establishes the scoring behavior of those requests. It does not establish answer quality or support for another model.

The endpoint returned echoed probabilities under choices, omitted token IDs, and displayed some split Unicode tokens with replacement characters. Those displayed fragments cannot reconstruct the original token bytes. For this model, the lab therefore uses a bundled Qwen tokenizer to reconstruct exact IDs and bytes locally, checking every prompt token and the prompt-token count against the response before accepting its probabilities. The tokenizer runtime is @huggingface/tokenizers version 0.2.0; the Qwen assets are pinned to revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. Approximately 12.8 MB of tokenizer data loads only when an XED response needs this reconstruction. This local work makes no additional model requests. A mismatch fails scoring.

Every distinct new nonempty prefix is scored before it enters the engine, so XED influences which walkers clone as well as which answer leads the final ranking. This requires two supplied-text evaluations, including the complete answer each time. Scoring asks for zero new tokens when supported; otherwise it requests one and ignores that output when computing XED. Its usage is still recorded. The lab limits scoring concurrency and caches results by model, format, question, and complete answer. Unchanged text and cloned prefixes reuse cached scores. Independent EOS nodes remain distinct completions even when their identical text shares a scoring result.

Scoring requests, latency, provider usage, and errors are separate from generation and embedding measurements. They do not consume the search’s newly generated-token allowance, but they do incur provider work and cost. If scoring fails after generation succeeds, the accepted generation and any available scoring evidence survive alongside the last committed population. The run reports an error rather than admitting an unscored prefix into the search. XED is optional; the default beam objective and optional negative mean Xent objective use the generation probabilities already returned.

Generate a game with a fixed target

Choose Xent game · fixed target in Generation, write a game brief, and select Generate game. The generation model produces a title, background, and target passage. Inspect the preview and creation token usage before sampling, then choose the game mode. The target stays fixed: the walkers write additional context that changes how predictable that passage is to the selected Together scoring model. The existing sampling settings and scorer remain the defaults. Game creation uses OpenRouter; target scoring requires the Together session key. Export game settings saves JSON accepted by the existing command-line benchmark runner. Import game lets you reuse a saved game in the browser.

For example, imagine a target that describes snow falling on a beach. Context about a winter storm might make that passage easier to predict. Context about a hot summer afternoon might make it harder. The probabilities decide whether either attempt succeeds. Make it unsurprising rewards a decrease in target surprise; Make it surprising rewards an increase. Both games expose the same background and target to the player. The scoring template contains the background and candidate context, without the player’s mode instruction.

Every nonempty context within the configured sequence cap is legal, including copying the target or giving direct instructions. Consequently, the winning context is the best sampled intervention under this score. The score alone does not establish that the text is an original explanation or a useful answer to a separate question.

Definition 8 (Fixed-target Xent game score)

Fix a scoring model \(J\), background \(b\), target tokens \(y_1,\ldots,y_m\) with \(m>0\), and scoring template \(c(b,x)\) containing additional context \(x\). Define target surprise and its empty-context baseline by

\[ H(x)=-\frac{1}{m}\sum_{t=1}^{m} \log p_J(y_t\mid c(b,x),y_{<t}), \qquad H_0=H(\varnothing). \]

The two utilities to maximize are

\[ U_{\mathrm{unsurprising}}(x)=H_0-H(x), \qquad U_{\mathrm{surprising}}(x)=H(x)-H_0. \]

All three quantities have units of nats per target token. Each evaluation uses the same target bytes and token identities; background and context tokens do not enter the sum. The empty root has utility zero and is ineligible to win. An extension from \(x\) to \(x'\) receives incremental reward \(U(x')-U(x)\).

Suppose the baseline surprise is 3 nats per target token and a candidate lowers it to 2. Its unsurprising score is \(+1\) and its surprising score is \(-1\). The two directions therefore ask opposite questions about the same measurement. Existing Mean XED evaluates the generated answer with and without its question. A Xent game instead evaluates one fixed target while the generated context changes.

The lab evaluates each complete candidate prefix before Wave or Graph can use it in cloning. The baseline is shared, and repeated identical contexts reuse their scoring result. Every successfully scored nonempty prefix can win, including unfinished prefixes and branches that later disappear from the population. EOS is unnecessary for a game candidate to be eligible. Identical contexts count once in candidate distributions. A scoring failure retains the generation evidence but cannot admit that unscored candidate to the population.

Compare both directions

Select Benchmark both modes in Generation or Benchmark to run two linked benchmarks sequentially. They share the frozen game, judge, baseline, generation route, settings, and trial seeds. Each compares the selected Wave or Graph algorithm with independent sampling matched to its actual generated-token usage and a temperature-zero reference. Compare each method’s best context, target surprise, empty-context baseline, signed utility, and best-score-so-far curve against generated tokens. The best score means the best sampled score; it does not certify a global optimum.

The pair preserves two benchmark archives, accessible through the saved benchmark dropdown. Retrying the pair resumes both modes automatically while retaining completed methods and partial attempts. Pause and stop controls also preserve collected evidence. Game creation, preparation, generation, embeddings, and target scoring have distinct costs. Matching generated tokens matches that sampling allowance; total provider work can still differ because the methods submit different contexts for scoring.

Saved evidence includes the game brief and creation provenance, fixed game, scoring template and judge, baseline probabilities, and candidate target probabilities. These measurements reproduce the game scores offline. Exported game configurations also work with the existing command-line benchmark runner using objective: "xent_game". A run freezes its game and settings; regenerating creates another instance. If game generation fails, the previous valid instance remains available for inspection and an explicit retry.

Choose what distance sees

Generated sequence only embeds the complete generated prefix. Prompt plus generated sequence embeds the original prompt followed by two newlines and that prefix. This switch changes the observation used for diversity. Both choices still condition generation and token probabilities on the original prompt.

Cosine distance compares vector directions, with values from 0 to 2 after numerical clamping. L2 compares their Euclidean separation and can also respond to vector magnitudes. Embedding similarity is a property of the selected model; it does not certify that two answers have the same meaning or factual content. Use the text inspector to check what a distance difference represents in your run.

The shared core defaults to L2 for existing consumers. LLM Lab explicitly selects Cosine, and its distance selector also permits L2. This setting is shared by Wave and Graph’s observation-diversity calculations; it does not redefine the Euclidean gas’s physical geometry. Saved configurations retain the selected metric.

The adapter batches repeated embedding inputs, deduplicates exact matches, and caches results by model and input text. It checks vector dimensions and finite, nonzero values. Empty initial prefixes use a sentinel observation until text exists. Context-limit checks use conservative UTF-8 byte bounds rather than a provider tokenizer, so they can reject some text that the model would accept. Oversized inputs produce an error instead of silent truncation, so the distance never quietly switches to representing only part of a branch. See OpenRouter’s embedding API for the request and response format.

Inspect and save the evidence

Select a trace in Generation, or a node in Analysis, to read its complete text and token details. Compare the selected objective, total and mean log likelihood, token count, and termination status. XED displays its raw score and selected direction; internal utility is labelled separately. The population table describes the current search; the archive also retains branches that were discarded. A reused walker slot is not a new identity for an old branch: immutable generation nodes and clone events preserve lineage independently of slots.

The token inspector preserves the complete Unicode text. A provider token can contain only part of a UTF-8 character, so a colored text span need not correspond to one token. Use the token details to inspect exact token records, bytes, and log probabilities; do not count visible characters to infer the token count. The full-sequence score counts each recorded token once even when several token records contribute to one displayed character.

Use comparison to keep two selected traces side by side. Read their shared prefix, the point of divergence, and the remaining text alongside their lengths and scores. A difference in total likelihood includes the effect of different lengths. The comparison does not rescore text or ask the provider to evaluate one branch under the other branch’s context.

Recordings retain generated token text and bytes, log probabilities, embeddings, parents, companion choices, cloning decisions, population snapshots, model/provider settings, request identifiers, usage, timing, and errors. XED recordings additionally retain scorer identity, format version, conditional and baseline answer-token probabilities, scorer-token count, direction, and scoring usage. Those saved measurements reproduce XED offline. These records make it possible to follow both the provider’s output and the algorithm’s subsequent selection. Temporary provider failures use bounded retries. Invalid responses or exhausted retries stop the run without turning a failed request into a scored continuation.

Use Export to save a versioned .fgllm recording and Import to inspect one offline. New recordings use version 3 and retain run progress through completion_target, eos_node_ids, token_budget, generated_tokens, and stop_reason. The stop reason is null until a normal stopping condition is reached. These counters describe the entire run, including saved completions whose walker slots have since been reused. Imported version 3 recordings show the saved stopping reason alongside EOS and generated-token progress.

Versions 1 and 2 remain readable without an API key. They retain their original information without applying the new run limits retroactively. Legacy total and mean configurations keep their original rankings and rewards; an imported configuration missing its objective retains the historical total default. New configurations default to objective: "beam" with beam_alpha: 0.6. Beam and XED settings are additive version 3 fields, including beam_alpha, xed_direction, and scoring_model; importing does not request missing scores. Version 2 adds measured decision data from the cloning stage; version 1 shows absent decision metrics as Not recorded. Importing replays recorded data and does not resume paid generation. Both provider API keys are excluded, but the prompt and generated text remain part of the recording. The 64 MiB recording limit bounds stored history; the run stops cleanly before exceeding it. Export a useful result before resetting the session. The Analysis PNG export saves an image of the view; retain the .fgllm file as well when you need the underlying data.

Compare sampling in Benchmark

Suppose branching finds an interesting answer. How much of that result came from sharing promising prefixes, and how much came from simply asking the model several times? The benchmark records both experiments and puts their answers side by side. Each trial runs the selected Wave or Graph search, the selected independent baseline or baselines, and one answer at temperature zero. Every method receives the same prompt, generation model, pinned provider, continuation chunk size, and sequence cap. Independent sampling uses the configured temperature. Its trajectories continue their own prefixes without cloning, donor selection, or sharing sampled responses, even when two prefixes happen to contain identical text.

The comparison mode determines what is held equal. Same population starts one independent answer per initial Fractal walker and continues each until model completion or the sequence cap. Same generated-token budget first measures the tokens actually generated by Fractal, then spends that allowance on independent trajectories. Completed independent trajectories are replaced with fresh ones while budget remains. Both collects both baselines using the same Fractal run and temperature-zero answer for that trial.

The Fractal method uses the same EOS target, token reservations, and stopping rules as interactive generation. N is fixed by the initial walker count, even when Graph grows. All four normal stopping reasons finish the benchmark method with a completed status, while only eos_target establishes completion-target success. Read the saved reason to distinguish obtaining N EOS answers from ending at a budget, branch-exhaustion, or iteration limit. Errors and user cancellation retain their separate statuses, accumulated run progress, and preserved partial work.

Keep the token counter in mind. If four walkers inherit the same ten-token prefix, cloning has copied access to that prefix; it has not generated forty new tokens. The matching budget counts each realized continuation once. It excludes inherited prefixes, cached transition replays, preflight probes, embeddings, XED scoring, and retries that produced no accepted output. Concurrent requests reserve their allowances before starting, and the final request can use a shorter chunk. A trajectory cut off by the shared budget remains partial in the saved data. For independent baselines, three consecutive rounds without generated tokens end the method with an explicit incomplete status.

This budget matches generated-token counts. Requests also resend prompt and prefix text, and Fractal uses embeddings. XED adds supplied-text scoring requests, so equal generated-token counts do not imply equal provider cost or elapsed time. The archive retains request usage and timing so those quantities can be examined separately. An unavailable provider measurement is recorded as unavailable, rather than zero. Token likelihood still measures the model’s preference under the request settings. In particular, the temperature-zero answer’s returned likelihood is not a rescore at the independent sampler’s temperature. Optional model grading is a separate operation, with its own measurements and cost.

Run and save in the browser

Open the full-width Benchmark tab after Generation and Analysis. Select the matching mode and the number of repetitions; the defaults are token matching and one trial. Start captures the current lab configuration. Fractal runs first within each trial, followed by the baselines in the recorded fixed order. Methods run sequentially, while requests within a method use the configured concurrency. Preflight validates the common provider route, including generation at temperature zero, and stores probe usage separately. Each trial records its algorithm seed derived from the configured seed and trial index. These seeds do not guarantee repeatable remote model responses. The Fractal method’s status displays the same EOS progress, generated-token allowance, and recorded stopping reason as Generation.

Pause waits at a completed generation boundary; Continue continues the active attempt. Stop cancels pending work and preserves what has already been saved. IndexedDB stores request outcomes and completed boundaries incrementally. The saved-benchmark selector retrieves earlier data, while Export and Import transfer a portable .fgllmbench archive. Export before clearing browser storage or moving to another browser profile.

After an interruption or reload, completed methods remain complete. Explicitly retry an unfinished method to start a new attempt of that method from its beginning; the partial attempt remains recorded separately. Continuing a saved benchmark revalidates its pinned route and skips completed methods. A storage failure stops further scheduling, leaving saved data available for export.

Choose the answers being compared

The source selector starts at Current Fractal run. Its metrics and figures work immediately, including for an imported recording; a baseline is not required. Starting or importing a benchmark selects that experiment. Results update as generation boundaries are saved. Select trials, methods, and termination states to choose the evidence shown. Generation settings remain in Generation, accessible from the configuration summary.

Archived answers and Retained population answer different questions. The archive counts each generated endpoint once, including answers subsequently discarded by Fractal. Separate independent generations remain separate observations even if their text is identical. The retained view keeps the last Wave population or Graph frontier slots, including clone multiplicity; Graph interior nodes and unused slots are excluded. For independent sampling it uses the latest endpoint of each trajectory. Both views appear together by default.

Full-answer comparisons initially include nonempty EOS and sequence-capped traces, with their counts shown separately. Use EOS-only or partial filters when appropriate. If the current run has only partial traces, the tab shows a labelled partial preview. Failed and interrupted attempts are excluded from the main comparison; the attempt inspector exposes their preserved evidence. An absent baseline or missing measurement stays unavailable rather than appearing as zero.

Read distributions and inspect their traces

Start with the summary table, then inspect the distribution behind a mean. Reward and length plots share bins and axes across methods. The default score is mean token log likelihood; the configured full-trace objective, total likelihood, and NLL are also available. Full-trace reward is the utility at the endpoint relative to the empty root. Under XED minimization, its sign is therefore opposite to the raw XED score. It is separate from a single chunk’s objective increment.

Diversity distributions use endpoint embeddings within each method, trial, and population. They include pairwise cosine or L2 distances, nearest-neighbor distance, and exact-text duplication. Retained copies keep their population weight. One temperature-zero answer has no within-run pairwise diversity; distances from other answers to that answer are a different comparison. Groups use all pairs up to 50,000 and a deterministic uniform pair sample beyond that, with sampled and total counts displayed. The shared PCA picture is only a two-dimensional projection: reported distances use the original embeddings. Cosine mode normalizes vectors before projection. Incompatible or missing embeddings cannot support a shared distance comparison.

Completion and compute views show EOS, capped, partial, and empty outcomes, along with stopping reasons, generated-token work, provider usage, requests, cost, and timing. Generation, embedding, XED scoring, and grading measurements remain separate. Benchmark descriptions retain the objective, beam exponent, XED direction, and scorer identity so results with different scoring settings can be distinguished. XED curves use saved scored-node boundaries; they do not pretend that scorer tokens align with generator tokens. Progress curves place each answer at the boundary where it was recorded; later answers cannot improve an earlier budget’s result. Fractal structure views show cloning, distinct retained endpoints, concentration, and shared ancestry. With repeated trials, summaries give each trial equal weight. Differences use paired trials, and bootstrap intervals resample whole trials rather than cloned answers or distance pairs. A single trial has no such uncertainty interval.

Select a histogram range, scatter point, or table row to open the corresponding traces in Evaluation. Pin two answers from any methods to compare complete text, token bytes and probabilities, chunk boundaries, cumulative reward, termination, embedding distance, and available grades. Recorded shared ancestry differs from text that merely matches. Sort the trace table, copy an answer, export displayed numeric data, or download a chart image. Keep the underlying recording or comparison report when you need to reproduce the figure.

Evaluate answers with Gemini Flash or another model

Likelihood and embedding distance cannot tell you whether an explanation is correct. Open Evaluation after Benchmark to request an optional judge assessment. You can evaluate the current Fractal recording without running a benchmark. The tab also contains the trace browser, pinned comparisons, saved grading sessions, judge-score charts, and judge-specific summary columns. Dataset and trial selections stay shared with Benchmark when you switch tabs.

Press Evaluate with Gemini Flash to prepare the route and start grading. The default requested model is ~google/gemini-flash-latest. The lab resolves that alias through OpenRouter’s catalog to the newest stable standard Gemini Flash release before looking up its endpoints. Lite, image, audio/TTS, batch, and preview variants are excluded. The concrete model and a compatible provider are pinned for the grading session; the requested alias is recorded separately. This matters when an alias has no endpoint listing of its own. Older alias-based grading sessions remain available for inspection.

The OpenRouter session-key field is accessible in Evaluation; credentials are not saved in reports or browser storage. Advanced settings let you select a different judge independently from the generation model, edit guidance for correctness, relevance, completeness, and clarity, and supply a reference answer if useful. Each criterion receives a score from 0 to 4 or an unavailable result. An overall score from 0 to 100 is calculated with equal weights only when all four criteria were assessed.

The judge sees the prompt, candidate answer, rubric, and optional reference. It does not receive sampling-method names, likelihoods, temperatures, or clone history. Candidate text is material to evaluate, not instructions to follow. For the selected trials, identical prompt/answer pairs share one assessment within a grading configuration. Grading uses the session key, a separately pinned compatible endpoint, temperature zero, at most two concurrent requests, and at most 1,024 output tokens per response. Structured results are validated before becoming grades.

The distinct-answer count, request limit, and available cost estimate appear automatically without paid grading requests. There is no required preview step: the evaluation button starts missing or failed assessments within the displayed limit. Pause, continue, stop, and retry controls preserve valid assessments. Missing credentials, unavailable routes, rejected requests, truncated responses, and invalid grades appear beside the controls so you can address the cause and retry; a failed assessment never becomes a valid score. Changing the concrete judge model, provider, rubric, or reference creates a separate grading session. Raw responses, short explanations, errors, usage, timing, and grading coverage stay with that session. Score distributions and grade-versus-likelihood or diversity plots describe the judge’s assessments; inspect the explanations and coverage when interpreting them. Opening saved comparisons never starts grading requests.

Rank answers with paired judge comparisons

Choose Pairwise ranking in Evaluation when the useful question is which of two answers the judge prefers. Absolute grades remains available for the original scoring workflow. A pairwise session freezes the current source, selected trials, rubric, judge and provider, sampling seed, and request budget. Its candidates are nonempty EOS or sequence-capped answers from completed methods. Every pair addresses the same prompt; scores from separate prompts do not form a common leaderboard.

The judge assesses correctness, relevance, completeness, clarity, and overall quality separately. Each verdict is A better, B better, tie, or cannot assess, with a short explanation. A tie is evidence of equivalence under the rubric. An unassessable result supplies no preference evidence for that category. Overall quality has its own editable rubric, initially prioritizing correctness and relevance before completeness and clarity. It is not an average of the four category ratings.

Each selected pair is shown in both presentation orders. The judge sees the prompt, answers, rubric, and optional reference, without method names, likelihoods, temperatures, or ancestry. Both verdicts remain inspectable. If swapping the answers changes the preference, the record shows that disagreement; it does not turn it into an explicit tie. The two orders form one paired evidence unit, rather than two independent judges.

Start with Run pairwise evaluation after inspecting the allocation and available cost estimate. The default judge uses the resolved Gemini Flash route, temperature zero, concurrency two, and a 2,048-token response cap. Structured responses are validated locally. A session has a default ceiling of 600 provider POST attempts, including reversed presentations and retries. That ceiling never grows automatically. Pause, continue, stop, and explicit retry preserve completed judgments; an extension records a new budget linked to the earlier session without rewriting its conclusions.

Spend comparisons on three different questions

Imagine drawing a line between two answers whenever the judge compares them. A connected network lets shared opponents tell us something about a pair that never met. A network with two disconnected islands cannot establish which island contains the better answers. Even within one connected network, a single thin connection can leave substantial uncertainty.

The default allocation reserves 50% of planned comparisons for fitting the ranking, 40% for a randomized method audit, and 10% for held-out ranking validation. With Fractal alone, the allocation is 80% ranking and 20% validation. The ranking cohort is a reproducible sample balanced across methods and trials, capped at 200 distinct answers and approximately one answer per three available ranking pairs. The interface reports its coverage; answers outside that cohort remain unranked.

Ranking first builds connected coverage, targeting four distinct opponents per answer. It then selects batches of ten pairs: eight seek to reduce rating uncertainty, while two explore randomly. These adaptive choices answer “which comparison would teach the ranking model most?” They do not provide a representative estimate of how often one sampling method beats another.

The method audit asks that second question by sampling occurrence pairs uniformly without replacement within each trial, population, and Fractal-versus-baseline contrast. A third set, the held-out validation pairs, is sampled before fitting and asks whether the fitted model predicts new verdicts. Audit and validation identities are kept out of adaptive training. The fit is frozen before their outcomes are revealed, and those outcomes do not refit the reported validated ranking. When a small dataset cannot support disjoint connected training and validation sets, ranking coverage takes priority and validation is explicitly unavailable.

Read ratings as estimates of judge preference

The ranking model gives each answer a strength for each category. The regularized Davidson model turns differences in strength into probabilities of a win, tie, or loss; it also estimates a presentation-position effect. All training results are fitted together, so the rating does not depend on the order in which requests finished. Strengths have zero mean and are shown on an Elo scale centered at 1500. The numerical origin is arbitrary: rating differences carry the information.

Regularization keeps a sparsely compared or undefeated answer from acquiring an extreme strength on scant evidence. Versioned Gaussian priors use standard deviations 2 for answer strength, 1.5 for the log-tie parameter, and 1 for the position effect. Each presentation has half weight; an unassessable category contributes none. Connectivity is checked separately by category, because a network connected for clarity might have too little assessable correctness evidence. Disconnected components cannot support cross-component rankings.

The tables show rating intervals, rank intervals, top-five membership probabilities, and distinct opponents. Pair predictions distinguish observed judgments from inferred win/tie/loss probabilities. Approximate posterior intervals come from a Laplace approximation with 2,000 seeded draws. Inspect the convergence and prior-sensitivity diagnostics: a narrow-looking interval from an unstable or prior-dominated fit is not strong evidence.

Held-out log loss, Brier score, and calibration compare predictions with unseen judgments and with a constant-outcome predictor. Position disagreement, missing assessments, and contradictory cycles are also reported. A scalar rating cannot exactly represent every cycle of preferences. These checks help reveal failures of that approximation; they do not certify that the judge knows the correct answer. Statistical precision concerns this judge, rubric, and evidence, not factual truth.

Compare methods without counting clones as new evidence

An identical prompt and answer share one ranking candidate even if generation produced that text several times. Those occurrences retain their original population weights. Ten retained copies can matter greatly to the question “what answer would a random retained slot give me?” They do not amount to ten independent opinions about the quality of that text.

The primary method statistic is Fractal preference share: a Fractal win contributes 1, an explicit tie contributes one half, and a loss contributes 0. The interface also reports win, tie, loss, and assessment coverage separately. Archived answers and retained populations remain separate comparisons. Each trial is evaluated first, and trials receive equal weight in the summary. Unassessable results remain missing; worst-case bounds show what happens if their unknown scores are all 0 or all 1.

The randomized finite-source audit uses fixed-sample bounds for bounded outcomes, with Bonferroni control over the reported method contrasts, populations, and criteria. These bounds are conditional on the saved answers and judge protocol. They do not say how a fresh generation trial will behave. Trial-level variation and approximate trial-cluster bootstrap intervals address that separate question when repeated trials exist; a single trial has no generalization interval.

Start with method comparisons, then inspect the rating table, prediction matrix, coverage, and validation diagnostics. Open an answer or pair in the existing trace browser to read the complete text and both judge presentations. An optional, separately budgeted second-model or blinded human audit samples 20 completed pairs. Its judgments and agreement remain separate from primary ratings, so disagreement is visible instead of being hidden by pooling judges.

Run from the command line

The command-line runner uses the same benchmark generation and validation code. Save a JSON configuration such as the following as config.json. The nested config object accepts the lab’s existing settings; omitted settings use their usual defaults. Here two small trials each collect both independent baselines. Supply the API key through OPENROUTER_API_KEY in the shell environment; keep credentials out of configuration files and archives. For XED, also set TOGETHER_API_KEY, select "objective": "xed", and choose "xed_direction": "maximize" or "minimize". The default scoring_model is "Qwen/Qwen3.5-9B". Beam scoring uses "objective": "beam" and a "beam_alpha" between 0 and 2, defaulting to 0.6. Omitting the objective in a new run chooses this beam-style objective.

{
  "config": {
    "prompt": "Explain why the sky is blue in three sentences.",
    "algorithm": "wave",
    "walkers": 8,
    "chunk_tokens": 8,
    "sequence_tokens": 24,
    "iterations": 3,
    "temperature": 1,
    "seed": 7
  },
  "comparison": "both",
  "repetitions": 2
}

Run from the repository root. The output directory contains an append-only journal and an atomically updated manifest. Resuming preserves completed methods; --retry-incomplete explicitly authorizes a fresh attempt of an interrupted method. Only one writer may own a benchmark at a time.

npm --prefix fractal-gas-web run benchmark:llm -- run --config config.json --output outputs/my-benchmark
npm --prefix fractal-gas-web run benchmark:llm -- resume --output outputs/my-benchmark --retry-incomplete

Process the saved records

A .fgllmbench file is a versioned JSONL archive: a manifest followed by incremental records using the same schema in the browser and command-line runner. It retains resolved settings, provider metadata, trial and attempt identities, stop reasons, requests and retries, generated prefixes and parents, token text and bytes, log probabilities, embeddings, Fractal population snapshots, run progress, and clone decisions. XED scorer configuration, measurements, and separate usage survive journaling and portable export. The journal and portable export preserve the Fractal completion target, saved EOS node identities, token allowance, actual generated-token count, and stopping reason. Accepted responses are retained even if an interruption prevents them from entering a committed population. This lets a later reader distinguish generated evidence from the algorithm’s subsequent selection.

Processing validates the archive and writes deterministic JSONL tables for runs, trajectories, nodes, tokens, requests, snapshots, and clone decisions. Stable identifiers link the tables. Derived lengths, likelihoods, negative log likelihoods, and usage totals come from the saved records; processing requires neither an API key nor network access. Use an exported archive as the input:

npm --prefix fractal-gas-web run benchmark:llm -- process --input archive.fgllmbench --output outputs/processed

Export a .fgllmcompare report to keep an immutable source snapshot together with grading sessions and view settings. Its source can be a standalone Fractal recording or a benchmark archive. Browser storage retains the report, and importing it reproduces the comparison offline. Existing .fgllm and .fgllmbench exports remain available. The same process command accepts a comparison report and exports metrics, evaluations, and grades alongside the underlying data tables. Processing reads saved grades without requesting new ones; paid grading is a browser action.

Version 2 comparison reports additionally preserve pairwise ranking sessions: candidate and occurrence identities, frozen sampling plans and partitions, request attempts, both presentation orders, validated judgments, model settings, and fit provenance. Browser storage journals ranking work incrementally and prevents concurrent writers. A storage failure stops new requests while leaving preserved evidence exportable. Reopening a session does not silently repeat a completed judgment.

The same offline process command exports ranking candidates, pairs, judgments, ratings, method comparisons, validation, and audit tables. Derived ranking results are recomputed from saved judgments using the shared numerical implementation; no judge request is needed to reproduce them. Version 1 comparison reports remain readable, as do the existing recording and benchmark formats. Ranking request costs remain separate from absolute grading, generation, embeddings, and scoring. Credentials are excluded from exports; the saved prompts, answers, and judge explanations remain part of the report.

npm --prefix fractal-gas-web run benchmark:llm -- process --input report.fgllmcompare --output outputs/comparison

Fractal attempts can also be extracted as ordinary .fgllm recordings for the existing lab recording tools. New version 3 extractions keep their run progress; version 1 and version 2 .fgllm compatibility is unchanged. The existing 64 MiB bound applies to each Fractal recording; benchmark storage accumulates trials incrementally instead of imposing that bound on the entire experiment. Archives exclude credentials but include the prompt and generated text. Keep the archive when exact responses, partial attempts, and provider measurements matter to a later comparison.

Follow branches in Analysis

Start with one recorded iteration and select a branch you recognize from the text inspector. Wave displays the immutable generation tree: each node is a saved sequence prefix, and an edge follows a real continuation from its parent. Cloning can make several population slots share the same prefix. That does not create additional generated text, so slot multiplicity and clone events are displayed separately from generation ancestry.

Graph displays the actual recorded graph slots and their parent-slot links at the selected iteration. Several slots may refer to the same sequence record; they remain distinct graph slots. Changing iterations can change this graph. Use the recorded slot identity when reading Graph’s structure and the immutable sequence identity when reading the generated text. The two answer different questions about the same search.

Selecting a Graph decision opens the generation tree so you can inspect its pre-cloning participants, including prefixes that have left the stored graph. The stored Graph view and its parent-slot links remain accessible separately.

Move through the view and the recording

Choose the cumulative-token layout to place a prefix according to how many tokens it contains. Choose the iteration layout to see when it entered the recording. A long chunk and a short chunk can occur in the same iteration, so these axes need not place nodes in the same order or at the same separation. The other axis arranges branches for readability; separation on the canvas is not an embedding distance.

Pan and zoom to inspect a crowded branch, use the minimap to move across the view, and collapse subtrees to keep their surrounding ancestry readable. Filters restrict the displayed candidates; text search helps find a phrase or branch of interest. These operations change the view, not the saved population, the generated text, or the cloning algorithm.

Scrub the recorded-iteration timeline to inspect an earlier population. Playback advances through saved iterations without generating tokens. Follow latest returns to the newest recorded iteration and follows new snapshots from a live run. Looking at an older iteration does not restore an earlier engine state. Use Generation’s Pause or Stop when you want to control ongoing API work; Analysis playback and generation are separate controls.

Read the metric before reading the color

The initial color metric is Selected objective. Read its formula and legend before comparing node colors; XED also shows the selected optimization direction. A beam score is not a log probability of an event and must not be exponentiated into a purported answer probability. Likelihood, incremental reward, distance, and fitness measure different things. In particular, a high cloning fitness need not identify the most likely sequence: fitness also incorporates the configured diversity contribution.

Quantity	What it measures
Token log probability	The provider’s natural logarithm of the probability of one generated token in its prompt and prefix context
Token probability	The exponential of that token’s log probability; a value between 0 and 1
Total log likelihood / logp	The sum of token log probabilities along the complete generated prefix
Mean token log likelihood	Total log likelihood divided by the number of generated tokens
Total / mean NLL	The negative of the corresponding log likelihood; smaller values are better for the selected likelihood objective
Selected objective	Negative mean Xent, beam-normalized likelihood, raw mean XED, or signed fixed-target game utility under the saved configuration
Internal utility	The value maximized by the engine; the negative of raw XED only when XED minimization is selected
Chunk objective increment	The child’s utility minus its parent’s utility, after the actual returned chunk
Distance	The recorded observation distance to the sampled distance companion at the decision stage
Fitness	The selection value computed by the algorithm at that decision stage, after its normalization and configured coefficients

The chunk objective increment uses the tokens actually returned. In total mode it equals that chunk’s sum of log probabilities. In mean mode it is the change in the whole prefix’s mean, which can be positive even though every individual token log probability is nonpositive. Beam scoring uses the same full-prefix comparison with its selected exponent. XED uses the two complete scorer evaluations and the chosen utility sign. Xent games use the change in signed target-surprise utility defined in Definition 8. No objective divides by the requested chunk size or assigns reward to tokens that were never returned.

Decision inspection identifies the evaluated prefix, its distance companion, its proposed cloning donor, and the resulting prefix. Distance companions and cloning donors are separate samples. The measured distance, normalized values, fitness, clone score, random draw, and protection flags belong to the recorded pre-cloning decision. A clone proposal and an applied clone can differ when the algorithm protects a row or rejects a donor. Read the final clone outcome alongside those flags.

The resulting population may contain different prefixes from those evaluated before cloning. Assigning its old fitness to a newly generated prefix would tell the wrong story. Analysis keeps these stages separate. Missing legacy decision metrics are marked Not recorded. Unmeasured at this step means that the prefix has no recorded evaluation at or before the selected playback step. Neither case substitutes zero or reconstructs a measurement from a later population. Sequence scores and chunk increments that follow directly from saved token records remain available in all supported recording versions. XED requires its saved scorer measurements; older recordings do not acquire them by importing.

How the token environment reuses the core

Wave and Graph are the shared native Fractal implementations. The LlmEnvironment adapter supplies the snapshot-based BatchEnv contract: state restoration, sampled continuation actions, observations, incremental rewards, terminal flags, and actual transition durations. Here duration means new generated tokens. A state references an immutable sequence record, so cloning restores the complete prefix and its score history together. JavaScript calculates the selected cumulative utility and passes it explicitly through the transition bridge and native snapshots. Native rewards use the difference between child and source utility. Undispatched rows keep their source utility, zero reward, and zero generated-token duration.

OpenRouter requests run in the browser worker. The standalone WebAssembly build uses Asyncify to wait for asynchronous generation, embedding, and optional XED scoring without copying the algorithms into JavaScript. Only one native engine operation can be pending at a time; network concurrency occurs inside that operation. Embedding-model discovery uses OpenRouter’s public /api/v1/models?output_modalities=embeddings catalog because the separate /api/v1/embeddings/models route does not permit browser CORS access.

Fresh action identifiers request independent continuations. Once a transition has been realized, its source state, action, duration, and configuration identify the saved result, allowing replay without another API call. This is the environment contract needed by future FMC and Jump Wave integration; the first lab exposes Wave and Graph. A live acceptance check must verify actual log probabilities and prefill continuation over at least two chunks on the selected route before treating that route as supported.

Run the focused native and adapter checks from the repository root with the command below. Automated checks exercise the environment and recorded response handling; live provider compatibility still requires a session key and actual continuation responses.

make llm-test