We would like to thank @SEEDGov for their efforts on this initiative. We recognize the significant challenges involved in designing a system that accommodates various complexities and stakeholders. While both proposed systems have their trade-offs, we believe each is an improvement over the current one. However, we believe that further iterations are still necessary to enhance their effectiveness.
We believe that v1.1 falls short of achieving the goal of driving quality contributions compared to v1.5. Our primary concern with v1.5 is the subjectivity in scoring. For instance, if someone receives a Level 2 (acceptable) score for clarity, what differentiates a score of 35% from 45%? This ambiguity can lead to discrepancies in individual grading. We recommend implementing a more objective scoring system to reduce variability. Instead of using percentage grades, we suggest assigning 1 point for each level, with a max of 4 points for each criteria and a maximum of 20 points total for the first 5 criteria (initial score).
Additionally, we believe that the criteria for Presence in discussions should carry more weight in evaluating contributions. We need to consider what constitutes a healthy level of presence in discussions. We would argue that a target closer to 50% is more appropriate, therefore anything less than that should be a reducing factor.
Here’s an example of our proposed changes to the scoring to demonstrate what we mean.
Proposed multiplier change:
Less than 25% presence in discussions: 0.9x multiplier
25-50% presence in discussions: 1x multiplier
50-75% presence in discussions: 1.1x
75%+ presence in discussions: 1.2x
To visualize (using the same example provided but with the revised scoring):
- Relevance: Level 3 - Scoring achieved = 3
- Depth of Analysis: Level 2 - Scoring achieved = 2
- Timing: Level 4 - Scoring achieved = 4
- Clarity and Communication: Level 2 - Scoring achieved = 2
- Impact on Decision-Making: Level 3 - Scoring achieved = 3
Initial Score/Average: 70% or 14/20 or 2.8/4
- Participation in Discussions: Level 3 - Multiplier assigned: 1.1x
Final Score: 70% x 1.1 = 77% or 23.1/30 Delegates’ Feedback points.
Lastly, it would be helpful to demonstrate how the current system would assess three delegates using the rubric and scoring methodology. This would provide delegates with a clearer understanding of the expected level of participation and effort under the new system.