Evaluation

This page collects the numbers I publish about Nightshade, with the denominator, version and limits next to each one. All of them come from one proof run of the released 0.1.1 CLI, installed from the release tarball on 2026-10-24.

Nightshade 0.1.1, proof run on 2026-10-24. None of these numbers is a precision or recall estimate.

Showcase examples#

Each of the 10 examples selects one rule. The positive fired in 10 of 10, the corrected version was quiet in 10 of 10 and the lookalike was quiet in 9 of 9 (one example has no lookalike). Every variant passed Compact 0.34.0 compiler validation and complete analysis. Exit codes are from the same runs.

Synthetic corpus#

The repository's test corpus pairs each rule with positive cases that should match and corrected, lookalike or other negative cases that should stay quiet. I ran every case declared in the five corpus registries with the installed CLI.

The run made 915 distinct CLI scans covering 1161 declarations across 93 rules. 915 scans passed compiler validation, 7 were incomplete and 0 had process failures. A check counts only when the compiler accepted the file and analysis completed. Registries overlap, so the last row counts each (scan, rule, role) once.

Misses and control hits. 3 positive checks were missed, all for identity-signature-key. 17 corrected or lookalike controls still matched; the corpus labels all of them as expected matches (0 were unexpected), and they get no quiet credit. 7 checks were incomplete and count as failures, not as quiet. 21 synthetic triplets have no mapped target rule; they were scanned but not scored.

Corpus scans used budgets of 100,000 steps, 256 paths and 100,000 abstract values, with no suppressions, severity filters or baselines. The run took 1105 seconds on 4 CPU cores.

Real contracts#

I reran the installed 0.1.1 CLI on a fixed set of 300 contract roots from 144 repositories. The assembled inputs are not redistributed, so only aggregate counts are published: no project names, paths or findings. All 300 roots stay in every denominator, including the 27 the compiler rejected, which were not scanned.

The 15 accepted roots without complete baseline analysis are coverage gaps, not clean results. One root did not get the semantic profile it requested and is counted as incomplete. The 315 finding instances are static observations under the default audit selection; I did not review them for accuracy here, so they are not confirmed vulnerabilities.

Compiler accept and reject decisions came from the authenticated frozen manifest of the original run; compilers were not rerun. Source hashes for all 300 roots and 66 dependency trees were checked before and after the run and did not change. The earlier pre-release 0.1.0 measurement on the same inputs recorded the same completion counts (258/300 and 237/300) and 315 findings.

What these numbers do not show#

• There is no independently established precision or recall estimate for the release. Synthetic tests check defined rule behavior, not unknown vulnerabilities or ecosystem-wide recall.
• A quiet corrected or lookalike case shows that the selected rule stopped firing. It does not establish that a contract is secure.
• Finding counts on real contracts are static observations, not confirmed vulnerabilities, and completion counts are not accuracy.
• Incomplete analysis and compiler rejection are never counted as clean results.
• Compiler validation used --skip-zk, so it skips proof generation and does not establish runtime behavior.
• Results apply to version 0.1.1 and the compiler profiles shown. Other versions can differ.

Versions and identities#

The showcase source fixtures match the files under tests/corpus in the repository at v0.1.1 by SHA-256; showcase files that were trimmed say so. The corpus also contains no public audit reproductions in this release, so no result here is a claim about a third-party project.

Reproduce#

One example. Install the release as in getting started, download an example's files from the examples page, and run from that folder:

./.nightshade/node_modules/.bin/nightshade vulnerable.compact --config config.json
./.nightshade/node_modules/.bin/nightshade corrected.compact --config config.json

The recorded runs also passed --compiler with the path to a Compact 0.34.0 compactc to validate each file with the compiler. Expect exit code 1 and one finding for the flagged file, and exit code 0 with no findings for the corrected one.

The corpus. The fixtures and the five registries are in the repository under tests/corpus at v0.1.1. The proof pack holds the input snapshot, raw outputs, scripts and checksums of this run. To check it without rescanning:

tar -xzf nightshade-proof-0.1.1.tar.gz
cd nightshade-proof-0.1.1
sha256sum --check --quiet SHA256SUMS
mkdir -p release
curl -fL -o release/nightshade-compact-0.1.1.tgz https://github.com/DicksonWu654/nightshade/releases/download/v0.1.1/nightshade-compact-0.1.1.tgz
python3 scripts/validate.py

validate.py recomputes every count on this page from the recorded outputs and checks their hashes. To rescan the synthetic corpus, follow REPORT.md in the pack: install the release into the pack, then run reproduce.py --stage synthetic with --toolchains pointing at Compact 0.31.1 and 0.34.0 compilers and --oz-root at the OpenZeppelin checkout above. The real-contract inputs cannot be redistributed, so only the synthetic results are publicly reproducible.

Pack SHA-256: 9d95e631dc318006610882911c0d58b7c25fa7754c39b49205c309d2e2e57b3b (1.7 MB).