Guide

QuasiLeak-Bench evaluates inferential disclosure, not merely whether a sensitive word appears. The system must recover complete sets of individually weak cues that jointly reveal an attribute the sender chose to hide.

Message x

The sender's natural-language message, including dispersed clues and irrelevant details.

Profile p

Eight true attribute values. These are reference values, not evidence that may justify a reported group.

Policy π

A context-specific map assigning KEEP, HIDE or NEUTRAL to each attribute.

Each predicted object couples a set of evidence spans with one attribute. The benchmark evaluates the decision to flag a disclosure, so an otherwise revealing group must not be reported when its attribute is KEEP or NEUTRAL.

A gold quasi-evidence group G = {e₁, …, eₖ} comprises two to four non-overlapping contiguous spans of x. Each span is weak by itself; their combination passes the benchmark's fixed inferability test. The annotation is a complete reference group, not necessarily the mathematically smallest revealing subset.

Scoring definitions follow Appendix D.1. Interactive examples illustrate the formulas; recorded leaderboard results come from the saved Python evaluation outputs.

QLB-Full spans 20 contexts with 10 messages per profile. The supervised split uses 3,500 training, 500 development, and 1,000 Test documents.

Training and test partitions

Difficulty slices, not new benchmark versions

Pair / Trio / Quad, attribute, cue distance (NEAR / MEDIUM / FAR), and surfaced-attribute count are best implemented as filters within a fixed dataset and scorer. Treating them as separate competitions would obscure what is being held constant.

Paper policy ablation

500 messages, each evaluated under original and reassigned policies (500 × 2). The paper's controlled ablation keeps the message, profile, prompt and decoding fixed, changing only the policy. Summarize suppression and surfacing behavior together with paired score changes.

P-GF1 is not independent-span F1. Its scorer first establishes the structural correspondence between predicted and gold cues, then aligns entire groups one-to-one.

A single predicted span can credit at most one gold cue. Duplicating the same words does not multiply recall, and reusing cues across different proposed groups cannot circumvent the one-to-one group assignment.

A prediction can be structurally close yet semantically wrong

A returned cue can be accurately localized, but belong to the wrong group. Likewise, a correctly assembled group can name the wrong attribute, or the right attribute under the wrong policy. The next four factors separate these mistakes.

The scorer assigns graded credit to an already matched predicted–gold group pair. Missing evidence, unrelated spans, cross-group mixing, and attribute errors affect distinct factors.

EL and BG decompose one within-group purity quantity. One penalizes irrelevant text; the other penalizes genuine but wrongly assembled evidence. The split is diagnostically useful even when the product is identical.

Attribute binding is categorical, not approximate

Only the primary label earns full credit. A secondary label must be explicitly annotated for that gold group; arbitrary semantically related attributes do not receive half credit.

For each document, QuasiLeak-Bench aligns predicted groups to gold groups one-to-one using only structural evidence. This prevents attribute or policy relabeling from changing which evidence groups get matched.

Only positive-weight edges participate. Matching first maximizes total structural weight, then uses the paper's prescribed deterministic tie breaks: number of aligned pairs, followed by prediction and gold indices.

Three levels of group credit

The scorer pools credits and denominators over the entire evaluation set. It does not average each document's F1 equally, nor does it grant full credit merely because a model named the correct attribute.

With λ_K = 1, a zero-credit prediction involving KEEP information consumes its normal prediction slot plus one additional policy-precision penalty. A NEUTRAL false alert consumes the ordinary prediction denominator only.

What differs across the three F1 stages?

Evidence and binding stages share the ordinary prediction denominator. The policy stage adds λ_KN_K, which is why a false alert against deliberately permitted KEEP evidence receives additional penalty.

Worked example · Why incomplete groups lose credit

Take a single primary-HIDE gold quad. A model returns one correctly attributed group containing only two of the four annotated cues, with no extraneous spans or KEEP contamination.

Illustrative one-group arithmetic only. Real scores require global matching and corpus-wide aggregation. A predicted revealing subset is not a full reconstruction of the annotated evidence.

Our paper reports diagnostic quantities at evidence, binding and policy levels so that two systems with similar P-GF1 can be distinguished by where they make mistakes.

Pre-policy evidence recall (ER)

For detector-transfer comparisons, the paper extracts raw located spans before attribute mapping, grouping, or policy filtering, deduplicates offsets, and matches them one-to-one to HIDE gold cues at θ = 0.75. ER is recovered HIDE cues divided by all HIDE gold cues (9,074 for QLB-Full in the manuscript). It measures whether the detector made the evidence available to later reasoning—not whether it assembled a correct violation report.

Distractor selection rate (DSR)

Lower DSR is better. It tests whether systems mistake non-revealing spans for relevant evidence. It is not a privacy-leakage probability.

How to interpret the failure pattern

Every system receives the same message, complete reference profile and disclosure policy. The expected output is a JSON object containing evidence groups, not a yes/no classification or a plain list of PII spans.

{
  "evidence": [
    {
      "attribute": "health",
      "spans": ["first verbatim cue", "second verbatim cue"],
      "rationale": "Each cue is weak alone; together they reveal..."
    }
  ]
}

Public prediction artifacts use offsets

The quoted-span response above describes the model-facing prompt protocol. Public scorer submissions instead use qlb.solution.v1, with sample_id, a predictions array, one attribute per group, and half-open Unicode-code-point start/end spans. Empty predictions are valid; the public wrapper rejects missing, duplicate, extra, or malformed document rows rather than silently dropping them.

The benchmark's canonical Python scorer remains the authority. Submission guidance →

Dataset IDs, preprocessing, model access, prompt regime, decoding, output parsing and scorer version must be recorded as part of an evaluation result.

Reproducible submissions should identify

What the policy reassignment experiment measures

For a frozen 500-document sample, the paper evaluates original and reassigned disclosure policies while preserving each message, profile, model prompt and decoding setting. Suppress measures the share of groups previously found that disappear when their attribute becomes KEEP or NEUTRAL; Surface measures the share of newly HIDE-assigned groups that the model emits. Both should be reported next to the paired change in P-GF1, rather than replacing the main retrieval metric.