OpenBlock Labs STIP Efficacy + Sybil Analysis (2/24)

[paulsengh]

OpenBlock Labs - Update for Arbitrum STIP

We’re thrilled to update you on OpenBlock’s progress in STIP efficacy analysis, and the performance of protocols across a variety of KPIs. Additionally, we’ll provide insights from a Sybil analysis in this update.

Sybil Results

Updated as of February 24, 2024 07:00 UTC.

The results below are wallets identified by either phase:

Project	Attacker Count	Sybil Count	ARB Amount Claimed by Sybil	Claimer Count	ARB Amount Claimed by All	Sybil Ratio of Claimers	Sybil Ratio of ARB Amount Claimed
RabbitHole	2,842	50,107	500,306	67,096	564,602	74.680%	88.612%
Gains Network	271	4,243	295,615	6,024	2,213,036	70.435%	13.358%
WOOFi	1,563	2,251	20,291	11,648	322,807	19.325%	6.286%
Vertex	9	49	19,742	1,596	2,462,552	3.070%	0.802%
Perennial	44	48	19,231	1,104	567,287	4.348%	3.390%
Galxe	178	485	17,572	3,878	169,019	12.506%	10.397%
MUX Protocol	203	311	11,539	1,551	4,269,236	20.052%	0.270%
Pendle	6	15	10,180	1,358	1,037,932	1.105%	0.981%
GMX	29	973	5,051	18,227	7,740,727	5.338%	0.065%
Trader Joe	6	13	4,570	1,263	1,186,285	1.029%	0.385%
Gamma	10	34	4,040	2,740	1,208,886	1.241%	0.334%
Thales	36	50	2,207	2,565	115,621	1.949%	1.909%
Camelot	11	36	1,529	3,842	1,806,248	0.937%	0.085%
Tide	3	6	320	138	6,000	4.348%	5.333%
Notional	1	1	223	259	186,580	0.386%	0.119%
Silo	2	7	137	1,880	759,946	0.372%	0.018%
Timeswap	1	1	64	439	9,320	0.228%	0.686%
Lodestar	4	4	33	2,234	583,220	0.179%	0.006%
OpenOcean	1	1	9	85	2,096	1.176%	0.407%
JonesDAO	1	1	1	714	1,679,576	0.140%	0.000%
Radiant	1	1	1	414	155,482	0.242%	0.000%
KyberSwap	1	1	1	627	54,201	0.159%	0.001%
Vela	1	1	0	1,533	144,337	0.065%	0.000%

The wallets detected on both the first and second phases are given in a table below.

Project	Attacker Count	Sybil Count	ARB Amount Claimed by Sybil	Claimer Count	ARB Amount Claimed by All	Sybil Ratio of Claimers	Sybil Ratio of ARB Amount Claimed
RabbitHole	12	40,280	371,663	67,096	564,602	60.033%	65.827%
Gains Network	23	1,695	109,862	6,024	2,213,036	28.137%	4.964%
Perennial	1	2	11,724	1,104	567,287	0.181%	2.067%
WOOFi	2	348	4,045	11,648	322,807	2.988%	1.253%
Galxe	1	12	1,111	3,878	169,019	0.309%	0.657%
Thales	1	1	461	2,565	115,621	0.039%	0.399%
GMX	1	1	2	18,227	7,740,727	0.005%	0.000%
Vela	1	1	0	1,533	144,337	0.065%	0.000%

Sybil Methodology

Some reward distribution mechanisms are not sybil resistant, meaning that users get better rewards splitting their activity to multiple addresses. Certain actors exploit such mechanisms, by creating sybil addresses and interacting with the protocols with those addresses, either manually or with automated means.

The method we apply to identify such wallets is a two-phase approach. The first phase consists of the following steps:

1. Identify every address that claimed ARB reward as part of Arbitrum STIP.
2. For every address in step 1, find the first funding of the address with ETH. This funding can be a direct ETH transfer or through contract call (with traces).
3. For every first funding from step 2, consider the EOA that sent it. Exclude if EOA that sent the transaction (i.e. tx.origin) or address that ended up sending the value (i.e. msg.sender) is a known CEX or bridge address. This condition decreases the recall, but increases the precision of the method. The sybil activity that funds the addresses that would then interact with the protocols through a CEX withdrawal or bridge transfer should be studied separately using other means than funding patterns.
4. For each EOA from step 3, count the number of claimer addresses first funded each month (funded_by_same_funder_in_month). Consider corresponding first fundings from step 3, count the number of claimer addresses that are first funded in that particular transaction (funded_by_same_tx).
5. For each claimer from step 3, label as sybil if one of the following is true: funded_by_same_funder_in_month >= 60 or funded_by_same_tx >= 10

The second phase consists of the following steps:

1. Create an Asset Transfer Graph (ATG) consisting of all wallets that completed a claim transaction for ARB and the wallets that initially funded those with ETH (since a wallet can not transact on the network until an initial deposit of ETH is provided, the initial funding wallet typically emphasizes an especially strong relationship between wallets).
2. Use the Louvain community detection algorithm (Blondel et al (2008)), implemented in Python (T. Aynaud. python-louvain 0.16) on the resulting ATG, to partition the graph into clusters of connected wallets.
3. Analyze the clusters to identify common Sybil structures such as Branching (Tree-Structured) and Chaining actions within the ATG.
4. Chaining and Branching structures are highly likely to be Sybil attacks, especially the deeper the chain or the larger the amount of descendants a node has, respectively. For these structures, if the depth of the chain or the number of descendants was larger than 10, the structure was classified as Sybil.

---

Protocol Rankings

Ranking by growth in TVL per claimed ARB:

Ranking by growth in sequencer fees per claimed ARB:

Ranking by growth in volume per claimed ARB:

Ranking by nominal growth in TVL:

Ranking by nominal growth in volume:

Ranking by nominal growth in sequencer fees:

Conclusion

We invite the community to kickstart a dialogue on the effectiveness of incentives and contribute further insights to complement OpenBlock’s quantitative methods. Recognizing that data is just one aspect of grant allocations, we anticipate engaging with the community to harness our insights for the development of more robust campaigns in the future. Stay tuned for more updates!

Twitter: @openblocklabs

Website: www.openblocklabs.com

Email: team@openblocklabs.com

[Jonezee]

Appreciate your work on this; very interesting data!

[bflynn]

Hi @paulsengh, we appreciate you sharing this analysis with the DAO and believe it represents a meaningful discussion surrounding the goals of incentives and the behaviors they attract.

Upon reviewing the data you’ve provided, we’d like to offer some insights into Boost Protocol’s (previously known as RabbitHole) impact during STIP.

---

Context on Boost

For wider context, Boost is a protocol to deploy token incentive offers to do onchain actions. While the protocol is permissionless and can be used by anyone for different strategies, Boost Studios uses the protocol as a grant recipient to maximize Arbitrum DAO sequencer fees for the cheapest possible cost.

For users, a boost is made up of two transactions: the target action that is being incentivized, and the reward claim itself. In both transactions, network & sequencer fees are being generated for the Arbitrum DAO, in addition to growing third-party protocols on Arbitrum. One of the main benefits of Boost is that all of the data is onchain, and can be verified by any third-party. In fact, you can see all the data here about our STIP grant in our STIP Dune dashboard.

---

Context on Boost’s STIP grant

Our primary objective with our STIP grant has been to maximize sequencer revenue at the cheapest possible cost. In doing so, we’ve been able to minimize the net cost to the network (STIP spend) and maximize the transaction fees it’s produced to the extent we’re net profitable in our incentive spending. As of this writing, we’ve generated over $1.16M in network revenue, with only $1.08M in rewards distributed.

(Source: Dune)

Boost is a flexible targeting tool for different strategies

In light of OpenBlock Labs’ detailed Sybil analysis, we’d like to take this opportunity to clarify Boost Protocol’s stance on Sybil behavior. While we have developed infrastructure to mitigate Sybil activities on the protocol with allowlisting, we’ve developed a strategy for this grant to maximize Arbitrum DAO’s sequencer fees at the cheapest possible cost.

Our 3+ years of experience working with sybil solutions has taught us that tracking and blocking sybil is a cat and mouse game. As a result, we’ve designed incentives that naturally discourage malicious behavior by aligning rewards with genuine network usage. To achieve this, we reward Boost incentives at or near the completion cost, effectively rebating the cost of the users’ transactions with network ownership (ARB). In most cases, Boost completions are near break-even (References: Spend/network fees by Boost deployed, all Boost completions & associated fees, daily Arbitrum network profit). Theoretically, most ‘attackers’ would be better suited to swap ETH for ARB on any exchange than to ‘attack’ Boost protocol. However, as there are indications of ‘attacks’ on Boost, our dilemma becomes whether it’s beneficial for us to halt this behavior, as completions under break-even is profit to the DAO.

As previously mentioned, Boost protocol has developed the infrastructure to enable anybody to deploy boosts that exclude sybil behavior; we’ve been reluctant to apply these measures for a few reasons.

1. The notion of sybil as a potential ‘Attack’ at the network level is debatable.
   Although sybil behavior can affect outcomes in scenarios that assume each human is a unique actor, such as for identity protocols, user-centric airdrops, or quadratic funding on Gitcoin, we don’t see it as inherently malicious in the context of driving network activity.

2. We believe incentives at the network level should prioritize maximizing sequencer revenue for the cheapest possible cost.
   Our experience shows that a strategy focused on profitable incentive distribution maximizes value returned directly to the DAO, which could theoretically make incentives sustainable. Applying Open Block’s sybil methodology has revealed that only 3/22 participants engaging in what might be considered ‘sybil attacks’ have actually profited from such activities (with a combined profit under $2K). This brings into question whether the network fees generated by automated transactions are inherently less valuable than those from other sources.

Should the DAO prioritize sybil or profit?

If there’s a widespread desire within the DAO to curb sybil activities, we’re open to exploring allowlisted approaches at the expense of higher costs and less profitability for the DAO. We believe it’s crucial for DAO members to engage in an open dialogue surrounding incentive priorities—whether maximizing revenue and transactions is a viable objective or whether all grantees should maximally prioritize discouraging ‘sybil attacks’ on principle.

[DisruptionJoe]

First, to the work @paulsengh is sharing - well done. As someone with a history of working directly on sybil detection, I can say this is a high quality analysis. I especially appreciate the consideration to how they preference avoiding false positives in a sensitivity vs specificity tradeoff.

Now to the point of “What does this mean?”

I became bearish on proof of individual humanity during my time working on the problem. We look at sybil as in input metric that - if treated - can lead to better outcomes. @bflynn points out that there are specific situations where this is important.

@bflynn

The notion of sybil as a potential ‘Attack’ at the network level is debatable.
Although sybil behavior can affect outcomes in scenarios that assume each human is a unique actor , such as for identity protocols, user-centric airdrops, or quadratic funding on Gitcoin, we don’t see it as inherently malicious in the context of driving network activity.

This is almost accurate. imho it does not go far enough.

A sybil-free round is a requirement of running a mathematically optimal quadratic funding round. It is NOT a requirement of that QF round resulting in a better society. Nor is it the only assumption the mechanism must contend with. What we are actually trying to do is optimally allocate capitol to encourage Arbitrum to continue spearheading the evolution of decentralized technology and governance.

@bflynn

Should the DAO prioritize sybil or profit?

If there’s a widespread desire within the DAO to curb sybil activities, we’re open to exploring allowlisted approaches at the expense of higher costs and less profitability for the DAO. We believe it’s crucial for DAO members to engage in an open dialogue surrounding incentive priorities—whether maximizing revenue and transactions is a viable objective or whether all grantees should maximally prioritize discouraging ‘sybil attacks’ on principle.

I don’t fully agree with the premise of this question. It is not a binary. I agree that sybil accounts paying enough to cover their acquisition is an intriguing concept.

What we don’t have here is the understanding of RETENTION based on a shared understanding of what qualifies as an engaged user at both the protocol and network levels. Luckily, our grants to Open Source Observer and Helika Gaming are just starting to give us insights here. (PSST LTIPP!)

So what does this mean?

@bflynn is taking the approach that we will likely learn and Boost is in a strong position to iteratively improve. The opposite angle would be to argue that the opportunity cost isn’t worth it.

To understand better, imagine the extreme of case: All of Boost was 1 sybil actor (this isn’t close to true - just a thought experiment)

In that case, out of $1 million spent on incentives by the DAO, 1 person was acquired by the network - this person likely wont be an engaged user without further incentives. However, it cost that person about $1 million in transaction fees to do this. We just moved $1 million from the treasury to the treasury.

That example is little facecious, but it shows how important it is to think about the acquisition cost of sybils vs the acquisition of real - potentially engaged - users.

We need the data to understand engagement - It’s been indexed and is now ready to analzye

Maybe we should drop an analysis into the Open Data Community Permisionless suggestion box grants to look into this?

[cerv1]

This is great analysis and discussion!
I just wanted to add that we recently completed our first analysis on onchain + off-chain impact metrics for 300 projects in Arbitrum. My write-up is here but more importantly all of the data (and infra) is open for others to replicate – or better yet – fork our analysis and come up with their own impact metrics. `
What I would like to see, which I think @disruptionjoe gets at in his response, is that we need good comps to determine an appropriate level of farmed to retained users for a given incentive program. We should also recognize that sybil attack vectors for an onboarding protocol like Boost / Rabbithole may be different (and more clustered) than a DEX’s.

[stB3n]

Great to have that level of data and analysis!

[longshortleftight]

great stuff and very detailed. may i know how the ‘funded_by_same_funder_in_month >= 60 or funded_by_same_tx >= 10’ parameters thresholds were chosen - is there some kind of distribution or historical true positives that support these thresholds?