Frozen benchmark · exact evidence gate
Regression evidence with its limits intact.
30 exact Visual Studio Marketplace artifacts are frozen and hash-pinned. A public result appears only when its exact Deep Scan can be reopened with the matching artifact, scanner build, ruleset, and canonical report.
Reproducible reports30/30Artifacts30 VSIX files
Correct interpretation
Exact report evidence, not a detached score.
The previous run remains an internal engineering regression record. It is not displayed as a current public scanner result.
Every published row must reopen the same stored report that produced its decision. A newer “latest” scan can never replace the evidence link.
Frozen corpus
Exact artifact boundaryIdentity remains visible while results are rerun
Corpus
30 artifacts
Extension, version, and SHA-256 pinned
Published
30 reports
Only exact benchmark-purpose scans
Withheld
0 results
No evidence substitution
Publication gate
All checks requiredEvery visible result passes the same checks
Artifact match
Exact
Identity, version, and SHA-256 agree
Coverage
100%
Required analyzers completed
Report identity
Immutable
Scanner build, ruleset, and scan ID recorded
Artifact evidence
Exact SHA-256 shownInspect all 30 frozen identities
Rows awaiting a current benchmark-purpose scan show identity only. They do not inherit the previous result or a newer registry scan.
Exact artifactCohortScan stateSeverityScores
amazonwebservices.aws-toolkit-vscode
credential sensitivefresh artifact holdoutreview100% coverage · scan f018bcc7MEDIUMrules 2026.07.19M0 · R61diagnostic indexes@4.10.0 · 9197f84348510b68aquasecurityofficial.trivy-vulnerability-scanner
security tool controlfresh artifact holdoutreview100% coverage · scan d80619abHIGHrules 2026.07.19M0 · R73diagnostic indexes@1.8.11 · 421289db1a46d354Continue.continue
ai agenticfresh artifact holdoutreview100% coverage · scan a2243508MEDIUMrules 2026.07.19M0 · R61diagnostic indexes@2.1.0 · 28e1c4691f090080dbaeumer.vscode-eslint
legitimate privilegedprior exposurereview100% coverage · scan 0d2022b6MEDIUMrules 2026.07.19M0 · R51diagnostic indexes@3.0.33 · a2ed5477988332d5eamodio.gitlens
legitimate privilegedprior exposurereview100% coverage · scan 318f75faMEDIUMrules 2026.07.19M0 · R61diagnostic indexes@2026.7.160544 · 20a80baa19fbeb69esbenp.prettier-vscode
benign controlfresh artifact holdoutallow100% coverage · scan ff059f7cINFORMATIONALrules 2026.07.19M0 · R0diagnostic indexes@12.4.0 · 46d22a567b35ebe5formulahendry.code-runner
legitimate privilegedfresh artifact holdoutreview100% coverage · scan 51ba6cf5HIGHrules 2026.07.19M0 · R72diagnostic indexes@0.12.2 · 4c8e4aea7dd07c9cGitHub.copilot
ai agenticfresh artifact holdoutreview100% coverage · scan 89c0f352MEDIUMrules 2026.07.19M0 · R61diagnostic indexes@1.388.0 · ed18caf3e3cd23eaGitHub.copilot-chat
ai agenticprior exposurereview100% coverage · scan 6d69433bHIGHrules 2026.07.19M0 · R80diagnostic indexes@0.48.1 · 7852df60c171be0bGitHub.vscode-pull-request-github
credential sensitivefresh artifact holdoutreview100% coverage · scan 3bd81a9bMEDIUMrules 2026.07.19M0 · R58diagnostic indexes@0.159.2026071604 · 5e40bb78c3af7ad2golang.go
legitimate privilegedfresh artifact holdoutreview100% coverage · scan eab5e402MEDIUMrules 2026.07.19M0 · R61diagnostic indexes@0.56.0 · 9f5959fb17ba0a8dhumao.rest-client
credential sensitivefresh artifact holdoutreview100% coverage · scan 95a2daccMEDIUMrules 2026.07.19M0 · R61diagnostic indexes@0.25.1 · 0d2cd9f70181d085mhutchie.git-graph
legitimate privilegedprior exposurereview100% coverage · scan 1ec6408eMEDIUMrules 2026.07.19M0 · R57diagnostic indexes@1.30.0 · b0779a30caf98664ms-azuretools.vscode-docker
legitimate privilegedprior exposureallow100% coverage · scan 44113defINFORMATIONALrules 2026.07.19M0 · R0diagnostic indexes@2.0.0 · 631cac7a4a7d9c2fms-kubernetes-tools.vscode-kubernetes-tools
credential sensitivefresh artifact holdoutreview100% coverage · scan 6cdab772MEDIUMrules 2026.07.19M0 · R54diagnostic indexes@1.4.0 · b7ae0e17eaf14e98ms-python.python
legitimate privilegedprior exposurereview100% coverage · scan 5e5fec2cMEDIUMrules 2026.07.19M0 · R58diagnostic indexes@2026.5.2026070801 · 7edf45e8e93fd155ms-vscode-remote.remote-containers
legitimate privilegedprior exposurereview100% coverage · scan 6043c614MEDIUMrules 2026.07.19M0 · R56diagnostic indexes@0.467.0 · b3bd40702da5dd7dms-vscode-remote.remote-ssh
credential sensitivefresh artifact holdoutreview100% coverage · scan 08cd2a9bMEDIUMrules 2026.07.19M0 · R56diagnostic indexes@0.125.2026062315 · 862aaf35b7154013ms-vscode.azure-account
credential sensitivefresh artifact holdoutreview100% coverage · scan a7e2a65fHIGHrules 2026.07.19M0 · R80diagnostic indexes@0.13.0 · c7fb78223cf1b0e1ms-vscode.cpptools
native packedfresh artifact holdoutreview100% coverage · scan 2b12bf0dMEDIUMrules 2026.07.19M0 · R63diagnostic indexes@1.33.4 · e05cf0f982318849PKief.material-icon-theme
benign controlprior exposureallow100% coverage · scan fa9ef78cINFORMATIONALrules 2026.07.19M0 · R0diagnostic indexes@5.36.1 · d72c538c8b328fcaredhat.java
legitimate privilegedfresh artifact holdoutreview100% coverage · scan bcc4d98dMEDIUMrules 2026.07.19M0 · R55diagnostic indexes@1.56.2026071508 · 1f6099292a4e811aritwickdey.LiveServer
legitimate privilegedprior exposurereview100% coverage · scan 76edde37MEDIUMrules 2026.07.19M0 · R61diagnostic indexes@5.7.10 · 0f8ef819d4c2007fRooVeterinaryInc.roo-cline
ai agenticfresh artifact holdoutreview100% coverage · scan 3b33a937HIGHrules 2026.07.19M0 · R80diagnostic indexes@3.54.0 · caf96d596d69cb34rust-lang.rust-analyzer
native packedfresh artifact holdoutreview100% coverage · scan 165b587eMEDIUMrules 2026.07.19M0 · R41diagnostic indexes@0.4.2976 · 5ff02c55c7d5a732saoudrizwan.claude-dev
ai agenticfresh artifact holdoutreview100% coverage · scan a8bcdd01MEDIUMrules 2026.07.19M0 · R57diagnostic indexes@4.0.8 · a92973db05d0c293semgrep.semgrep
security tool controlfresh artifact holdoutreview100% coverage · scan 891257cfMEDIUMrules 2026.07.19M0 · R55diagnostic indexes@1.17.0 · a06ce8efc8630256snyk-security.snyk-vulnerability-scanner
security tool controlfresh artifact holdoutreview100% coverage · scan 8fb2c5a9HIGHrules 2026.07.19M0 · R75diagnostic indexes@2.31.0 · 5e6ea92b390f7552SonarSource.sonarlint-vscode
security tool controlfresh artifact holdoutreview100% coverage · scan d36b847cMEDIUMrules 2026.07.19M0 · R58diagnostic indexes@5.5.0 · d755728fcd9d87b1usernamehw.errorlens
benign controlfresh artifact holdoutallow100% coverage · scan 4609574eINFORMATIONALrules 2026.07.19M0 · R0diagnostic indexes@3.28.0 · edebbbcbd211d6ecPublication boundary
What the benchmark refuses to substitute.
A result remains withheld until the current scanner produces a complete report for the exact frozen artifact.
Required
Required
Required
Evidence boundary
Re-run first. Publish second.
The benchmark keeps its original visual structure while withholding every result that cannot reopen its exact canonical Deep Scan.