@wyszomirski, your latest replies materially clarify the state of the work.
The three think-aloud sessions are useful formative evidence, and the scheduling interpretation raised by one delegate is not a side observation. It points to a second legitimate construct that should now be separated formally from the individual per-vote instrument.
I rechecked the current public implementation at source level, including:
- the grant release on
main;
feature/dfi-per-delegate-research;- the ecosystem and per-event fatigue calculations;
- the Snapshot and Tally vote-history clients;
- the API and persistence layer;
fatigue_config.yaml;- the current per-event tests.
I have not executed the live service or inspected the private NASA-TLX records, so the findings below concern the public implementation and the measurement architecture.
After reviewing both branches, I would not describe the issue simply as “the DFI is ecosystem-level.”
The project now contains two genuinely different instruments:
-
the grant-release calculation on main, which measures shared ecosystem governance burden and produces the same result for every supplied address;
-
the research-branch per-event calculation, which estimates workload for one specific delegate vote using proposal-specific reading/novelty terms and contextual terms derived from that delegate’s observed vote history.
Both may be useful.
The problem is that they currently share too much identity, terminology, persistence structure, configuration, and status language despite answering different questions.
The clean separation should be:
ECOSYSTEM GOVERNANCE LOAD
→ when is the governance environment congested?
DELEGATE EVENT WORKLOAD
→ how much observable workload surrounded one delegate’s specific vote?
NASA-TLX RESPONSE
→ how much workload did that participant report for that task?
SCHEDULING OUTCOME
→ did the timing and overlap of proposals affect engagement?
Those four objects should not be collapsed into one DFI narrative.
1. Give the two instruments separate identities
The original endpoint:
GET /delegates/{address}/fatigue
is explicitly ecosystem-level. The address does not participate in the calculation.
That calculation is better represented as an object such as:
GOVERNANCE_LOAD_INDEX
with an endpoint such as:
GET /governance/load
Its identity should be bound to:
- DAO or governance space;
- observation-window boundaries;
- proposal-corpus manifest;
- source completeness;
- formula version;
- configuration version;
- computation time.
It should not be persisted separately under arbitrary delegate addresses.
The research endpoint:
GET /delegates/{address}/per-event-fatigue
is a different object. It is specific to:
- one delegate;
- one vote event;
- one proposal stage;
- one vote timestamp;
- one historical evidence boundary.
That object should have a distinct name and schema, for example:
DELEGATE_EVENT_WORKLOAD_ESTIMATE
The scheduling interpretation from the think-aloud session fits the ecosystem-level object. The NASA-TLX comparison fits the per-event object.
This is not a reason to discard either calculation. It is a reason to stop asking one object to support two different construct claims.
The current LOW / MODERATE / HIGH / CRITICAL bands should also remain workload bands during validation.
The configuration currently associates the highest band with “burnout risk.” NASA-TLX measures task workload; it does not by itself establish burnout. A burnout claim would require a separate longitudinal outcome and measurement contract.
2. Close the historical look-ahead boundary
There is a concrete temporal-validity issue in the current per-event path.
The endpoint correctly:
- fetches the delegate’s vote history;
- selects a target vote;
- sets
as_of to that target’s voted_at timestamp.
However, it then passes the complete currently retrieved history into compute_per_event().
Inside _compute_metrics(), the 7-day and 30-day windows are calculated from:
proposal.start
rather than:
vote.voted_at
There is no prior filter requiring every context vote to satisfy:
context_vote.voted_at <= target_vote.voted_at
This permits look-ahead contamination.
For example:
- proposal B starts before target vote A;
- the delegate votes on A;
- the delegate votes on B several days later;
- the API is later asked to reconstruct workload at vote A;
- B can enter A’s historical context because B’s proposal start was before A, even though the delegate had not yet cast the B vote at the time of A.
The resulting historical DFI can therefore depend on information that was not yet present at the declared as_of boundary.
The per-event evidence set must first be frozen as:
context_votes = votes where voted_at <= target_vote.voted_at
Then the volume windows should be computed from voted_at if the intended metric is “votes per week/month.”
Proposal start and end should be used only for explicitly proposal-time concepts such as:
- whether the proposal was available;
- whether it was active;
- whether multiple decisions overlapped.
They should not silently substitute for the delegate’s vote-event timestamp.
This requires a regression test in which a later vote concerns a proposal that began before the target vote. Adding that later vote must not alter the already frozen historical measurement.
The invariant should be:
same target event
- same evidence available at target time
- same instrument version
= same measurement forever
3. Separate available governance load from revealed voting activity
The source comments correctly acknowledge that voting activity is endogenous.
That boundary deserves its own data model.
A delegate may face ten active proposals and vote on only two.
The two observed votes measure revealed engagement. They do not prove that the other eight proposals imposed no workload.
The reverse problem is also important: if heavy workload causes abstention or delay, using only completed votes as the context can reduce the measured workload precisely when actual burden is highest.
The per-event model should therefore preserve two separate context channels:
Available governance load
- all proposals active or decision-relevant at the target time;
- overlap and timing;
- text volume;
- proposal stage;
- eligibility or expected responsibility where known.
Revealed delegate engagement
- votes actually cast;
- vote timestamps;
- proposal stages encountered;
- repeated or amended votes;
- prior participation history.
The current calculation primarily uses the second channel but sometimes interprets it as the first.
Both are useful, but their absence and meaning are different.
A scheduling index should be driven mainly by available governance load.
A delegate-specific behavioral context may use revealed engagement, but it should be labelled as such and should not be treated as complete exposure.
4. Make Snapshot and Tally capability states explicit
The current research endpoint combines Snapshot and Tally histories, but the two source adapters do not currently produce equivalent evidence.
The Tally proposal query retrieves:
- proposal ID;
- title;
- description;
- start timestamp.
It does not retrieve the proposal end timestamp.
The transient Tally Proposal therefore has no end.
The concurrency calculation requires:
start <= as_of <= end
and substitutes zero when end is absent.
Consequently, Tally proposals cannot contribute to concurrent_active under ordinary positive timestamps, even when they were active at the target vote time.
That creates source-dependent scoring:
- Snapshot decisions may contribute to concurrency;
- equivalent Tally decisions may not.
Either Tally end-state evidence must be acquired and frozen, or concurrency for that source must be marked:
CONCURRENCY_UNAVAILABLE
It must not silently become zero.
There is a second source-boundary issue.
The clients currently return an empty list for several different conditions:
- the delegate had no votes;
- the API key is missing;
- the provider failed;
- GraphQL returned an error;
- the request timed out;
- the selected page contained no usable records.
These states are not equivalent.
A measurement should carry a capability record such as:
Snapshot:
- COMPLETE
- COMPLETE_TO_DECLARED_LIMIT
- PARTIAL
- UNAVAILABLE
- NO_ACTIVITY_OBSERVED
Tally:
- COMPLETE
- COMPLETE_TO_DECLARED_LIMIT
- PARTIAL
- UNAVAILABLE
- NO_ACTIVITY_OBSERVED
If Tally is unavailable and Snapshot succeeds, the system may still produce a Snapshot-bounded result, but the result must state that its evidence plane is partial.
If both sources are unavailable, the result is not “no votes found.” It is:
MEASUREMENT_BLOCKED_SOURCE_UNAVAILABLE
The current fixed retrieval limits also need a completeness state. A list containing exactly 200 records does not prove that 200 is the complete history.
5. Define cross-source governance-event identity
The implementation currently treats Snapshot and Tally records as a direct union because their proposal IDs differ.
That prevents literal ID collision, but it does not resolve semantic identity.
One governance initiative may have:
- a forum discussion;
- a Snapshot temperature check;
- an off-chain election or poll;
- an on-chain Tally execution proposal.
Those may represent separate cognitive tasks, or different stages of the same governance decision.
The system needs to record that relationship explicitly.
A useful model would contain:
governance_decision_id;proposal_stage_id;source;source_proposal_id;stage_type;parent_or_predecessor_stage;vote_event_id;voted_at;semantic_grouping_status.
Then the instrument can apply an explicit policy:
- count each stage separately;
- group stages into one decision;
- treat the relationship as unresolved.
The current rule — different source IDs imply different workload events — is not sufficient as a semantic rule.
Snapshot vote records are also deduplicated by proposal ID. That means a changed or repeated vote can collapse into one proposal-level event.
The study should decide whether its task unit is:
- one proposal encountered;
- one final vote state;
- every distinct vote submission;
- one complete governance-decision lifecycle.
That decision must be frozen before validation.
6. Persist a real per-event measurement receipt
The per-event API response contains useful fields such as:
- mode;
- target proposal ID;
- target title;
as_of.
However, the persisted FatigueSnapshot uses the same table as the ecosystem calculation and does not preserve:
- measurement mode;
- target vote ID;
- target proposal ID;
- source;
- proposal stage;
- exact vote timestamp;
- exact context-vote set;
- source capability states;
- pagination state;
- formula hash;
- configuration hash;
- implementation commit;
- input-corpus manifest.
The stored component values and aggregate metrics are enough to display the previous result.
They are not enough to reconstruct or audit the exact evidence population that produced it.
The persistence model should create an immutable per-event receipt containing at least:
- measurement ID;
- instrument type;
- instrument version;
- delegate address or research participant pseudonym;
- target vote-event ID;
- source and proposal-stage identity;
- target vote timestamp;
- proposal text hash;
- exact context-event IDs and timestamps;
- available-load corpus hash;
- Snapshot capability state;
- Tally capability state;
- configuration SHA-256;
- implementation commit or artifact hash;
- component inputs;
- component outputs;
- aggregate output;
- creation time;
- supersession state.
A deterministic measurement identity can be derived from:
instrument version
- target vote-event identity
- as-of boundary
- context manifest
- configuration hash
- implementation hash
The same frozen inputs should resolve to the same measurement identity rather than create another scientific observation.
7. Stop using GET requests to create measurement history
Both fatigue GET endpoints currently insert a new database row during an ordinary read.
Therefore:
- refreshing the page creates another snapshot;
- repeated API calls duplicate the same measurement;
- historical row count reflects client traffic as well as governance history;
- arbitrary address queries create address-bound ecosystem records;
- the same per-event result can be stored repeatedly without a new vote or new evidence state.
A public read request should retrieve a measurement, not manufacture its scientific history.
A cleaner boundary is:
- an explicit measurement command, scheduled computation, or controlled study action creates an immutable receipt;
GET retrieves that receipt;- repeated retrievals are idempotent;
- a genuinely changed source/configuration/instrument creates a new version that references the previous receipt.
If an on-demand calculation is required, it can remain computationally on demand while persistence uses an idempotent key. Repeating the same request must not create a new historical event.
8. Align metric names with what is actually measured
Several current metric labels and implementations describe different quantities.
Volume
The per-event configuration describes:
The engine counts:
- proposals whose
start lies in the 7-day window;
- proposals whose
start lies in the 30-day window.
Those are not the same metric.
Choose and name one explicitly:
delegate_votes_cast_7d;delegate_votes_cast_30d;
or:
relevant_proposals_started_7d;relevant_proposals_started_30d.
Do not call proposal starts delegate votes.
Burstiness
The current weekly baseline is:
proposals_30d / 4.33
The current 7-day numerator is already contained inside that 30-day denominator.
This is a contemporaneous ratio, not a clean comparison against a prior four-week baseline. It mechanically dampens the spike it is intended to measure.
If the construct is surprise relative to prior activity, use non-overlapping windows, for example:
current window:
[t - 7d, t]
baseline window:
[t - 35d, t - 7d)
Otherwise rename the component to describe the actual ratio.
Reading time
Word count is a defensible text-length or reading-load proxy.
It is not observed reading time.
Until actual reading behavior or a fixed reading-rate transformation is part of the instrument, the component should be named:
TEXT_LENGTH_LOAD
or:
ESTIMATED_READING_LOAD
Novelty
The per-event novelty component is currently a global keyword classifier.
It returns one only when:
- at least one novel keyword is present;
- no routine keyword is present.
Therefore, one routine keyword vetoes every novel signal. A proposal containing both “emergency security” and “report” can be assigned zero novelty.
More importantly, this measures a proposal’s keyword category, not novelty relative to a particular delegate.
A subject can be novel to one delegate and routine to another.
The fact that novelty was zero for all three sessions may indicate:
- the three proposals were routine;
- the vocabulary did not match;
- a routine term vetoed a novel term;
- proposal novelty is not delegate novelty.
Those possibilities are currently indistinguishable.
If the intended construct is personalized cognitive novelty, it needs:
- a versioned domain taxonomy;
- the delegate’s prior domain exposure;
- relevant role or specialization evidence;
- an explicit unavailable state.
Where prior exposure is unknown, the result should be:
DELEGATE_RELATIVE_NOVELTY_UNRESOLVED
The current keyword flag can remain as an ecosystem proposal-type feature, but it should not be interpreted as personalized novelty.
9. Freeze configuration and computation identity before N=50
The implementation states that weights come from YAML, but the human-readable FORMULA string remains hardcoded to:
40 / 25 / 20 / 10 / 5
If the YAML weights change, the returned calculation may use the new weights while the API still reports the old formula.
The formula shown in a measurement must be generated from the exact loaded configuration.
For research operation, configuration loading should also fail closed:
- missing configuration must not silently fall back to defaults;
- weights that do not sum correctly must not produce only a warning;
- incomplete per-event references must not inherit another instrument’s values without an explicit version transition.
The configuration remains version 1.0.0 even though per-event reference values and a second instrument have been added.
At minimum, each study measurement should bind the exact configuration hash rather than relying only on the semantic version string.
The current Arbitrum medians are useful reference scaling.
They are not construct validation, which you have already stated correctly.
10. Separate formative work from confirmatory validation
The first three think-aloud sessions are valuable, but their proper role is formative:
- test comprehension;
- identify missing concepts;
- detect wording and interface problems;
- refine the instrument;
- expose alternative uses such as scheduling.
They should not yet be treated as evidence that the numerical instrument is calibrated.
Before the N=50 phase, freeze or preregister:
- instrument version;
- component definitions;
- weights;
- reference values;
- status thresholds;
- source-capability requirements;
- primary NASA-TLX outcome;
- scoring method;
- treatment of subscales;
- exclusion and missing-data rules;
- planned statistical comparison;
- criteria for support, revision, or rejection.
If the instrument changes after participant 1, every response must remain bound to its exact version. Results from materially different versions should not be pooled as though they came from one instrument.
The anti-anchoring order already described is correct:
NASA-TLX first
→ DFI revealed afterward
That should remain a hard study invariant.
11. Clarify what N=50 can establish
If each of 50 delegates rates one vote, the study can estimate a cross-sectional relationship between:
- one observed event-level index;
- one reported task-workload result.
It cannot yet establish that the instrument tracks changes within the same delegate over time.
To make a longitudinal “per-delegate” claim, the design needs repeated observations:
- multiple votes per delegate;
- across different proposal types;
- across different ecosystem-load conditions.
That allows separation of:
- stable delegate differences;
- proposal-specific effects;
- temporal ecosystem effects;
- measurement error.
If repeated measures are not feasible, the claim should remain:
cross-sectional per-event workload association
rather than:
validated individual fatigue tracking
Participant reachability also forms a selection boundary. Delegates who accept a 45-minute session may differ from delegates who are most overloaded and do not respond. That limitation should be preserved in the study result rather than treated as a minor recruitment detail.
12. Use the correct calibration method for the stated target
You previously mentioned empirical re-weighting through PCA.
PCA can be useful for exploring covariance and redundancy among the five DFI components.
It does not calibrate the instrument against NASA-TLX because PCA does not use the NASA-TLX outcome.
If the objective is:
determine which component weights best correspond to the external workload criterion
then the outcome must enter the calibration procedure.
With a small N, that should remain conservative:
- preregistered rank correlation or regression;
- regularization if weights are estimated;
- participant-level cross-validation where repeated observations exist;
- bootstrap confidence intervals;
- clear separation between model development and final evaluation.
Do not search across weights, thresholds, NASA-TLX totals, and all subscales and then report the strongest association without a frozen primary endpoint.
PCA may remain an exploratory structural analysis. It should not be described as criterion calibration.
13. Treat the scheduling interpretation as a separate testable hypothesis
The delegate’s scheduling interpretation may become one of the strongest practical outputs of the project.
But:
two heavy proposals in the same week reduce engagement
is currently a plausible hypothesis raised during formative work, not yet an established result.
It can be tested directly using the ecosystem-load object.
For example:
ecosystem load at proposal launch
→ turnout
→ time to first vote
→ completion rate
→ abstention
→ quorum attainment
→ delegate coverage
That study should use all relevant proposals and fixed historical windows.
It is different from:
per-event delegate workload
→ NASA-TLX reported workload
Keeping the two validation paths separate would make both stronger:
Study A — Scheduling and governance congestion
Instrument:
GOVERNANCE_LOAD_INDEX
Outcome:
observable participation and timing behavior.
Study B — Individual event workload
Instrument:
DELEGATE_EVENT_WORKLOAD_ESTIMATE
Outcome:
task-specific NASA-TLX.
A scheduling result should not be used to validate individual fatigue, and an individual NASA-TLX correlation should not automatically validate proposal-timing recommendations.
14. Add boundary tests before freezing the study instrument
The per-event acceptance suite should include at least:
-
A vote cast after the target event cannot change the target’s historical score.
-
A proposal that started before the target but was voted on afterward is excluded from the target vote-history context.
-
Snapshot and Tally representations of one governance lifecycle follow the declared grouping policy.
-
A Tally proposal without an end timestamp produces CONCURRENCY_UNAVAILABLE, not zero concurrency.
-
A missing Tally key is distinguishable from zero Tally votes.
-
A Snapshot timeout is distinguishable from zero Snapshot votes.
-
A retrieval result reaching the page limit is marked potentially truncated unless completion is proven.
-
Repeated GET requests do not create duplicate measurement history.
-
Ecosystem and per-event measurements cannot enter the same persistence row without an explicit instrument type.
-
A persisted measurement can be reproduced from its frozen context manifest.
-
Changing the YAML weights changes both the computation and displayed formula identity.
-
A later configuration cannot silently reinterpret an earlier NASA-TLX record.
-
Mixed novel and routine keywords follow an explicit precedence rule.
-
The same frozen target event produces the same receipt after service restart.
-
The original ecosystem-level calculation remains unchanged by per-event corrections.
Recommended closure order
I would close the current work in this sequence:
-
Give the ecosystem and per-event instruments separate canonical identities.
-
Fix historical filtering so every context event respects the target vote’s as_of boundary.
-
Separate vote timestamps from proposal start/end semantics.
-
Add explicit Snapshot and Tally capability/completeness states.
-
Resolve Tally concurrency and cross-source lifecycle identity.
-
Create an immutable per-event measurement receipt.
-
Make measurement creation idempotent and keep GET read-only.
-
Correct the volume, burstiness, reading-load, and novelty semantics.
-
Freeze the configuration, implementation identity, and validation protocol.
-
Run the N=50 study as a version-bound criterion test.
-
Test the scheduling hypothesis separately against observable governance outcomes.
The highest-value next artifact is not another dashboard component.
It is one frozen, independently reproducible per-event measurement containing:
- one exact target vote;
- one exact historical evidence boundary;
- complete source-capability states;
- one context manifest;
- one configuration hash;
- one implementation hash;
- one score receipt;
- one NASA-TLX record collected before score disclosure.
Once that object can be reproduced exactly, the empirical study can determine whether the proposed instrument works.
Without that object, disagreement between DFI and a delegate cannot be localized: it may come from the theory, the weights, future-event leakage, incomplete source coverage, source asymmetry, or an unreproducible historical input.
The project is close enough that these boundaries can be closed without discarding the grant work.
The important next move is to freeze the identity of what is being measured before increasing the sample size.