The central issue is not missing methodology — the current profiler does not measure the reported gas savings
I reviewed the public repository behind the final report, including the v2.0.0 tag that predates this June report.
The local development environment, templates, exports, dashboard and Docker workflow may all be useful deliverables. However, the central performance claim has a deeper problem than the absence of raw benchmark files:
The current profiling route does not deploy and execute the paired contracts to measure their gas use. It reproduces a configured Rust-versus-Solidity estimate table and then presents the derived result as a contract profile.
That distinction changes the status of the reported 32.6% execution saving and 26.7% TCO saving.
1. What the visible workflow suggests
The final report says that the gas profiling system:
- deploys Rust and Solidity contracts side by side;
- supports local, Sepolia and mainnet environments;
- produces verified gas savings;
- tracks those results historically in the dashboard.
The README is even more explicit:
“side-by-side Rust vs Solidity comparison with exact numbers, not estimates”
That creates a clear measurement expectation:
paired source contracts
→ successful compilation
→ deployment on a defined network state
→ equivalent operations and inputs
→ actual execution
→ transaction receipts / traces
→ measured gas comparison
→ reproducible KPI
2. What the current code actually does
The public v2.0.0 route is different:
profile command
→ compile Rust and Solidity
→ create GasProfiler
→ query network name and block number
→ always enter estimation mode
→ load fixed language-level operation values
→ compare those constants
→ save/export/display the result
The decisive line is in:
src/profiler/gas-profiler.ts
// Always use estimation mode (no actual deployment needed)
After the RPC/network check, profileContract() does not deploy either contract and does not execute its functions. It calls:
estimateDeploymentGas(compilation)
estimateFunctionGas(compilation)
The _testCases parameter is not used. The ABI, actual functions, input values, storage state and resulting state transitions do not determine the function-level result.
Instead, estimateFunctionGas() reads this table from:
src/config/constants.ts
Rust:
read 5,000
write 12,000
compute 8,000
oracle 75,000
Solidity:
read 6,000
write 20,000
compute 15,000
oracle 103,000
These are exactly the figures displayed in the README and repeated in the final report:
- reads:
16.67%;
- writes:
40%;
- compute:
46.67%;
- oracle:
27.18%.
The reported 32.63% is then produced by arithmetically averaging those four percentage differences in:
src/profiler/comparator.ts
The TCO result is also derived from the configured estimates. The comparator assumes 100 executions for each generic operation, adds formula-based deployment estimates and calculates the percentage difference. It is not reconstructing an observed workload.
So the central number is not the output of a benchmark that independently confirmed the KPI. It is the output of the assumptions already encoded in the tool.
3. Why the problem is easy to miss
The network connection succeeds, and the profile can retain a real network name and block number.
That creates an observer conflict:
visible profile:
contract + network + block + timestamp + precise gas table
actual evidence path:
network metadata + static per-language constants
The RPC connection provides context, but it does not turn an estimate into an execution measurement.
A dashboard can then store these results, display trend lines, calculate historical averages and export polished JSON/CSV/HTML reports. But historical repetition of the same configured model is not independent empirical confirmation.
Precision of presentation is not provenance of measurement.
4. A failed Rust build can still produce a successful profile
There is a second boundary problem in:
src/commands/profile.ts
If Rust compilation fails, the command creates a synthetic successful compilation result using an assumed ~20 KB WASM artifact:
success: true
bytecode: '00'.repeat(estimatedWasmSize)
warnings: ['Static estimation mode — Rust compilation unavailable']
The synthetic result is then passed into the same comparison route and the command can finish with:
Gas profiling complete!
An estimate mode may legitimately provide a rough reference when a toolchain is unavailable. But it cannot share the same success semantics as a measured benchmark.
For a measured result, the invariant must be:
No successful build → no deployable artifact → no measured profile.
5. There is no implemented statistical benchmark layer
The repository contains a benchmark command, but the command currently states that the feature is still in development. Multiple iterations, statistical analysis, performance metrics and advanced visualization are listed as planned features:
src/commands/benchmark.ts
That means the repository does not currently show the repeated execution layer needed to support claims about consistency, variance or network-condition sensitivity.
This matters because a defensible Rust-versus-Solidity comparison must control at least:
- semantic equivalence of the paired operations;
- input vectors;
- pre-state and post-state;
- cold versus warm storage/access conditions;
- calldata size;
- contract activation and deployment treatment;
- compiler and toolchain versions;
- node/Nitro/Stylus version;
- chain or fork state;
- repetitions and dispersion;
- failed and reverted executions.
6. The current result is contract-insensitive at the function layer
Under the present route, two entirely different Rust contracts receive the same generic:
read / write / compute / oracle
table.
Two entirely different Solidity contracts receive the same corresponding Solidity table.
Changing the business logic, storage layout, branching, host calls, calldata or number of operations does not change those four function estimates. Only the language selects the table.
That is the hidden architectural break:
The output is presented as a profile of the selected contract, while the function-level values are properties of the configured language label.
The selected contract is therefore not causally bound to the primary KPI.
7. The correct fix is to separate two legitimate products
The estimation functionality does not need to be deleted. It needs a separate identity and evidence class.
A. ESTIMATE_REFERENCE
Purpose:
- rough planning;
- offline comparison;
- illustrative defaults;
- early developer guidance.
Requirements:
- explicit
measurement_kind: estimate;
- identify the model/config version;
- expose every assumption;
- do not label the output “verified” or “exact measured gas”;
- do not use it to close a benchmark KPI;
- do not allow network metadata to imply that execution occurred.
B. MEASURED_BENCHMARK
Purpose:
- empirical Rust-versus-Solidity comparison;
- contract-specific profiling;
- grant/KPI verification;
- historical performance tracking.
Required route:
exact Rust source + exact Solidity source
→ semantic-operation mapping
→ successful reproducible builds
→ artifact hashes
→ deploy both artifacts to the same frozen environment
→ establish equivalent pre-state
→ execute the same input vectors
→ collect receipts, traces and Stylus-specific execution data
→ verify equivalent post-state
→ repeat runs
→ report distributions and provenance
The two modes must never be merged into one trend series or one average.
8. Minimum evidence object for every measured comparison
A measured result should retain at least:
benchmark_id
measurement_kind
source commit(s)
Rust source hash
Solidity source hash
compiler/toolchain versions
node/Nitro/Stylus versions
chain ID
fork or block reference
deployed addresses
deployment transaction hashes
artifact hashes
semantic operation ID
input vector
pre-state hash
transaction hash
receipt gasUsed
trace / execution evidence
post-state hash
equivalence assertion
run index
failure/revert status
Without this binding, the result cannot answer the most important question:
Did these numbers come from these exact contracts performing the same operation under the same conditions?
9. Concrete code-level changes
I would make the correction at the architecture boundary rather than patching the displayed percentages.
- Add an explicit mode to
profileContract():
estimate | measured
- In
measured mode, require:
- successful compilation;
- real artifacts;
- a funded authorized signer;
- deployment receipts;
- declared semantic test cases;
- actual execution receipts;
- state-equivalence assertions.
-
Remove the synthetic successful Rust build from the measured path. A failed build must fail the measured run.
-
Keep GAS_FUNCTION_ESTIMATES only as a versioned reference model. Do not use it in the default contract-profiling route.
-
Implement the currently missing benchmark layer with repeated trials and statistical outputs.
-
Add provenance to every saved profile and export.
-
Make the dashboard visibly separate:
REFERENCE ESTIMATES
MEASURED EXECUTIONS
-
Prevent KPI, historical-average and TCO calculations from mixing those classes.
-
Rename existing historical outputs as estimates unless transaction-level evidence exists.
10. Acceptance tests that would close the issue
A corrected release should pass at least these tests:
-
Contract sensitivity
Two contracts in the same language with materially different logic must not automatically receive the same measured function table.
-
No-network boundary
An unavailable node may produce an estimate, but never a measured or verified result.
-
Build boundary
Failed Rust compilation must make a measured benchmark fail.
-
Execution provenance
Every measured function result must resolve to a transaction receipt or equivalent execution evidence.
-
Semantic equivalence
Rust and Solidity operations must produce equivalent declared post-state before their gas values are compared.
-
Mutation test
A meaningful code mutation must either alter the measured result or be accompanied by evidence explaining why execution remained equivalent.
-
Repeated-run statistics
Results should include sample count, median, min/max and dispersion rather than a single configured value.
-
Mode isolation
Estimate and measured records must never be aggregated into the same KPI or trend.
-
Reproducible environment
Another reviewer must be able to recreate the exact benchmark from the saved package without relying on undocumented local state.
Conclusion
This is not primarily a request for more documentation.
It is a measurement-identity correction.
The toolkit may still be a useful local Stylus development environment. But based on the public v2.0.0 execution route, the central gas-savings KPI should currently be classified as:
model-derived estimate, not verified contract execution measurement
until a transaction-bound benchmark package demonstrates otherwise.
The important next milestone is therefore not to defend the existing 32.6% figure. It is to turn the profiler from a configured benchmark-table renderer into an evidence-producing measurement system.
Once that boundary is corrected, the dashboard, exports and historical tracking become genuinely valuable because every number will be bound to the contract, operation, state and execution that produced it.