Analysing impact and performance of previous ARB incentive programs (STIP, Backfund, LTIPP) - LTIPP Research bounty - PYOR

[PYOR]

Our Proposal
We are proposing to analyse the impact of the three incentive programs run by the Arbitrum DAO and measure their impact on the eco-system and ROI. This will be achieved by analysing the pre and post-incentive performance of Grantees. Key areas that will be addressed are:

• Analysing growth in the user base and cross-usage of protocols amongst users.

• Analysing changes in user behaviour and frequency: Avg no of transactions per user, fee per user

• Analysing the cost of acquiring users through CAC and the payback period.

• Calculate the lifetime value (LTV) of users and LTV / CAC ratio.

• Analysing the quality of users through retention cohorts and repeat ratios.

• Build and analyse the behaviour of various Grantee segments (By scale, size and business category). Segments will be made by dividing Grantees into groups with similar characteristics or behaviours.

• Benchmarking of ARB eco-system metrics with comparable L2s across metrics and protocol activity. This will enable us to outline areas where ARB is outperforming other L2s and where it is lacking.

Why our work is important
Capital efficiency is very important for achieving the best ROI. Incentive spend constitutes a significant portion of the foundation’s spend and allocating it for the right projects is most important for the long-term growth of the ecosystem. Our work will answer the following questions amongst several others:

1. What is the ideal set of protocols for DAO to fund and support through Grants based on their Sector, Scale and traction? Understand if DAO should double down on established protocols or if DAO should make bets on newer projects. Or DAO should fund a DEX or a lending protocol etc.
2. What is the optimal size of the grant to generate maximum impact?
3. What is the ROI of various segments and who were the best-performing and worst-performing segments/grantees?
4. For a sample set of sectors analyse why they underperformed or overperformed through analysis and discussions with Grantees/community members.
5. Create a leaderboard of performers based on varied sets of metrics e.g. which segments created the most sticky users (best retention cohorts), which segments are driving Daily active user growth, and which segments drive maximum TVL.
6. What sectors DAO should attract and fund more based upon the sector performance on Arbitrum vs sector performance on another ecosystem? Identifying market demand for certain protocols, mechanisms, and profit-driven actions
7. What were the unique incentivization strategies implemented by the Protocols to attract new and recurring users? and what other protocols can be learned from it.

Methodology
We will segment Grantees based on different parameters and will conduct Pre and Post analysis of those segments for all 3 programs(STIP/ Backfund STIP and LTIPP).

Segments for Grantees

1. Based on sectors
   Protocols will be aggregated based on the services they offer.
   Major categories will be

• Dexes
• Perps
• Lending / Borrowing
• Options
• CDPs
• Yield

2. Based on the size of the Grant

• According to the predefined quantitative methodology, protocols will be divided into ‘High’, ‘Medium’ and ‘Low’ categories.
• The methodology for the categorization will be documented and presented separately.

3. Based on the protocol Size/scale/recency (when was the protocol launched)

• According to the predefined quantitative methodology, protocols will be divided into ‘Large’, ‘Medium’ and ‘Small’ categories.
• The methodology for the categorization will be documented and presented separately.

Metrics for performance analysis for protocols and segments

1. Cumulative users:

• Calculate the total users who have used the protocol and ARB since inception
• Monthly breakdown of new vs existing users showing how many new users were acquired in each week

2. Daily and monthly active users:

• Daily Active Users (DAU)
• Monthly Active Users (MAU)
• Ratio of DAU / MAU

3. Breakdown of DAU, MAU & DAU/MAU by new vs existing users:

• New to the respective protocol
• New to Arbitrium
• Existing users of Arbitrium.

4. Breakdown of DAU, MAU & DAU/MAU by new vs existing users:

• Retention cohorts for each protocol and overall for the ARB ecosystem show retention at 1W, 2W, 4W, and 8W.
• Frequency: Analyse the Average number of transactions per week per user for Protocol and Arbitrium.
• Economics: Analyse trends in Average fee per transaction for protocol and overall Arbitrium.

5. Analyse trends in user behaviour and activity by segmenting users based on :

• Existing vs New wallets

• Age of Wallet

• Based on the ARB balance in the wallet

• Did they hold ARB before the program

• Nature/business of dapps interacted with e.g. Dexes.

• Nature of activity mercenary miners, institutional investors, traders etc

  (Segments for this section will be defined quantitatively and documented appropriately with the reasoning behind the decisions.)

6. Analyse the overlap between the Protocols that received grants in STIP and LTIPP:

• Segment users by how many protocols they interacted with e.g. 1, 2, 3 etc
• Analyse the difference in the business performance of wallets which interacted with different numbers of Protocols (Retention, behaviour, metrics). E.g. Do wallets which interact with more than one Protocol have better retention compared to users who interacted with only 1 Protocol or do they have a higher average frequency of interactions with Arbitrium?

7. Customer acquisition cost and LTV:

• LTV - Lifetime Term Value - The Gas contribution of a wallet is taken as the LTV provided by a wallet.
• Calculate the LTV of users at the overall Arbitrium level
• Calculate the customer acquisition cost (CAC)
• Analyse CAC vs LTV ratio
• Calculate the break-even LTV vs. CAC ratio for Arbitrium. At what level of activity and retention will ARB recover grant money in fees?
• Calculate payback period

8. Analyse Unit economics :

• Avg Fees per Transaction
• Avg Fees per Wallet
• Avg No of transactions per user (Weekly and monthly)

9. Analyse Gas contribution and payback period:

• Calculate the Gas fees generated by each Protocol pre and post-program.
• Calculate the incremental gas generated and payback period for the grant.

How we will present this information to the DAO

1. A public Dashboard for the metrics we will analyze and a related document of methodologies and definitions.
2. A report containing Insights and conclusions of our research and analysis.

Team Background

PYOR is a crypto data analytics company backed by Castle Island and Coinbase Ventures. We specialize in measuring the business performance of protocols/chains. We work with large institutional investors and protocols in the web3 space. We’ve worked with Ribbit Capital, M31 Capital, Tezos, Compound, ICP, Swell, QuickSwap, etc.

Past work

• We work extensively with large institutional customers supporting their Crypto analytics and research needs. Our customers include Ribbit Capital and M31 Capital amongst others.

• We extensively work with Protocols building custom solutions tailored to their needs. Some of the Protocols/chains we have worked with include Compound, Tezos, ICP, Swell, QuickSwap, and Osmosis among others.

• We have built an interface X-ray for institutional investors to give them an in-depth look into the business and financial metrics. Our platform includes metrics around retention, LTV, user activity, stablecoin volumes etc.

The following contains a few examples of user activity analysis for Arbitrum that we’ve done previously -

Figure 1: Daily and Weekly active users

Figure 2: Monthly transactions count per user and Monthly Fees per user

Figure 3: User retention cohort

• We regularly publish in-depth and quantitative reports and have partnered with the Ethereum Enterprise Foundation to publish research reports.

Requested Budget & Cost breakdown

• We request a budget of ARB 100k for this project.
• Budget breakdown:

	Cost
On-chain data Data collection	10,000 ARB
Infra cost for data modelling and analysis	24,000 ARB
Team cost (3 Research analysts and 1 data scientist for 3 months)	54,000 ARB
Engineering efforts for dashboard - Front end and back end	12,000 ARB

[WintermuteGovernance]

Hi @PYOR,

Thanks for putting forward your Research Proposal! The scope of your proposals seems to focus heavily on identifying areas of overspending/underspending and identifying gaps in Arbitrum’s ecosystem for future funding initiatives which is great. Something that we would like to see added is that once you have identified category leaders across certain metrics, it would be beneficial for the DAO to understand the differences between competitors with an accompanying explanation as to why you might think that is the case (e.g., is it the incentive mechanism, existing user base, asset selection, age & security of the protocol).

It would also be great if you could provide a bit more insight into the cost structure as other applicants seem to have notably lower data collection and infra costs. $34k (11.3k/month) for data collection and infra seems a bit high.

[PYOR]

hey @WintermuteGovernance,

Thank you for your feedback.

Regarding your suggestion to delve deeper into identifying category leaders and their competitive advantages, we’ll be sure to incorporate a comprehensive “Best Practices from the Leaderboard” section in our report. This segment will meticulously analyze the factors contributing to the exceptional performance of these leaders compared to their competitors. We’ll dissect aspects such as incentive mechanisms, protocol maturity, and user base dynamics to distil actionable insights for the DAO’s consideration.

Regarding the cost breakdown for infrastructure, our proposed analysis demands intricate segmentation of on-chain data at both wallet and protocol levels. To achieve this, we need to process extensive transaction-level data and logs, totalling over 3TB in size. This necessitates dedicated infrastructure to execute complex data models effectively. Here are some of the examples of data models required:

1. Filtering and decoding transaction data from all grantee protocols on Arbitrum.
2. identifying wallet birthdates and ecosystem interactions.
3. Segmenting databases by protocol, user activity, etc.
4. Analyzing cross-protocol usage and overlaps.
5. Benchmarking against comparable chains.

Given the scale and complexity of these tasks, the outlined infrastructure costs are essential to ensure accurate and insightful analysis.

here is the breakdown for infra cost

Activity	Cost
Total storage cost for 3 months	279
On-chain data sourcing cost (infra consumed by Node + infra consumed by AWS to pull information for 3 months)	6,000
Data processing Infra (Glue/Snowflake) for compute (estimated based on 2.5 hours of compute for each of 142 protocols funded by STIP/Backfund and LTIPP and 70 hours for consolidated analysis)	27,720
Total	33,999

Let us know if there are any more questions.

Thanks

[404DAO]

Thank you for your proposal @PYOR!

Overall, the application is sound. We appreciate the segmentation component in the methodology as it likely brings in much needed context for measuring protocols against their comps. The proposal addresses a lot of data points which will be pulled from different market conditions (STIP vs LTIPP). How do you plan to gauge these results when comparing across the programs? It could be beneficial to see how certain strategies are more/less effective in today’s market conditions versus during STIP’s.

[PYOR]

Hey @404DAO,

Thank you for your query. We fully acknowledge the critical role market conditions play in the growth of protocols. Indeed, we recognize the variance in market dynamics between the timeframe of STIP and back fund STIP (Nov-March, with an extended deadline) and the upcoming period from June to August, when incentive distribution for funded protocols in LTIPP is proposed.

Understanding that there is no one-size-fits-all approach, we will perform an analysis of each program’s performance individually. Subsequently, we will undertake a comparative assessment, considering the differing market contexts, and normalize the results accordingly.

A few exemplary strategies are:

1. To construct a benchmark for growth rates across various metrics for the different timeframes of STIP and LTIPP.
2. Compare the growth of a particular protocol against the overall growth of all the participants put together in the particular program.
3. Choose control groups outside of grantees to quantify sector growth, and benchmark against them.

Once the data is ready for processing, initial EDA would help us come up with more fine-tuned strategies to quantify cross-program comparisons.

Let us know if you have further questions regarding this.

Thanks!

[Saurabh]

PYOR’s Proposition

PYOR’s extensive experience collaborating with protocols like Tezos, Compound, ICP, Swell, QuickSwap, Osmosis, and others highlights their adaptability and understanding of various ecosystems, relevant to Arbitrum.

The proposal looks very detailed and has clear KPIs. It would be great if you could answer some of the questions about it.

• Regarding the segmentation of the grantees OpenBlocks did a similar research during STIP. How would it be different from that?
• How will you ensure the accuracy and consistency of the methodologies used for segmenting protocols by sector, size, and scale? Could you provide more detailed documentation on the predefined quantitative methodologies you mentioned for these segmentations?
• Can you elaborate on the approach and data sources you will use for benchmarking ARB ecosystem metrics against other L2s? Specifically, how will you handle data normalization to ensure fair comparisons across different ecosystems?

[PYOR]

Hi @Saurabh,

Thank you for raising these questions.

About the first question:

Firstly, we want to acknowledge the valuable work done by Openblock. While acknowledging their significant contributions, it’s important to highlight the distinctions in our approach.

In our view, The Openblocks project was focused on capturing metrics relevant to incentive efficacy and presenting “What” happened after the grants to defined metrics. While the base remains the same our proposal takes one step ahead to answer “Why it happened” and goes much deeper in analysis to metrics not covered in the earlier project e.g. wallet level, digging metrics and trends from overlaps, wallet economics, cost of acquisition and retention.

About the second question:

When Interpreting results from large and varied data sets it is important to break the data set into smaller sets. This helps achieve the following objectives:

• Removes bias/impact of scale: Since subjects which have the biggest numbers overshadow the relatively smaller subjects, analysing at an overall level more often than not results in these being unnoticed due to materiality.
• Like-to-Like comparison & Relative analysis (create groups with similar characteristics or behaviours): In a varied data set where characteristics of constituents vary it is important to analyse them separately. E.g. if we combine Meme coins and L1’s in our analysis for price movement (e.g. WIF and ETH) in the last 4 months, it may lead to an interpretation that ETH is an underperforming subpar asset. However, their fundamentals vary on account of scale (2Bn MCAP vs 200Bn Mcap) and use case.
• Summarise insights for a group of subjects rather than interpret each subject individually. This makes the insights each consume, relatable to a business activity and more actionable.

Here are the quantitative methodologies for segmentation:

• The objectives of our segmentation analysis are geared to address the above objectives. The process for categorising grantees into various segments will be:

  • Step 1: Research the grantees and create a list of their key attributes by answering the questions relevant to their business model. Below are a few examples of the questions we answer in this step. It is an inviting process where at times answers lead to subsequent questions:
    • What is the model of the protocol i.e. is it a lending protocol, DEX etc?
    • Are there sub-segments in the business model and do different sub-segments vary significantly in their business model e.g. 1) Is the Dex AMM or order book based? This could impact how they attract liquidity and answer whether it warrants a sub-categorization. 2) Is it a Perps exchange
    • When was the protocol launched first and what is the timeline for its deployment across various ecosystems? E.g. A particular protocol which had already achieved significant scale in a different ecosystem will have a different growth trajectory compared to a protocol that was launched for the first time within the ARB eco-system. Although both may be new deployments on ARB they will require a separate classification.
    • Relevant to the business model of the protocol which KPIs accurately capture the performance and scale of a protocol. E.g. For a lending protocol, the value of deposits & borrowings will be relevant where whereas for a DEX trading volume and TVL may be relevant.
    • How well funded is the protocol - overall funding received and ARB grant?
  • Step 2: Analysing the insights created in step 1 and categorizing them into like-to-like buckets. This involves
    • Understanding the distribution across the attributes and creating segments/ranges that best reflect the dispersion.
    • This process requires significant judgement and to avoid any bias and to ensure we draw correct comparables we undertake the following process:
      • The data captured in step 1 and classification in step 2 go through a three-step review process. In this step, three different team members review the data and basis of classification. Any anomalies are discussed and resolved.
      • During the analysis phase, we investigate any trends which are materially different from other constituents of the group. Upon investigation, it may be that there genuine reasons that explain the trend or it could be that the underlying attributes of the constituent vary significantly with the group. In scenario 2 we reclassify the constituents to the appropriate group.

Overall, the segments are determined by the qualities of the data sets and their constituents. The examples mentioned above are standard examples which are generally true. However, often more segments are noted during bottom-up and top-down analysis of the data.

About the last question:

We will use on-chain data sets and metrics curated by us and 3rd parties to create this comparison.

This section intended to create an outside comparison of performance between ARB and other L2s across key metrics. In addition, we are proposing to compare the composition of on-chain activity across sectors e.g. Dexes to show how this varies between ARB and other eco-systems.

The normalization in this section will be limited as doing a detailed normalisation will require doing a bottom-up analysis of other L2s as well which will expand the scope 3-4X and will not be feasible.

Having said that we have already done work on analyzing 50+ protocols and chains in depth. We will use this tribal knowledge to adjust for differences where possible.

we hope this answers your questions. Let us know if there are any further questions.

Thank you!

[alicecorsini]

Thank you for bringing this proposal forward. I agree incentive spend is a significant portion of the overall costs, hence identifying how these incentives are impacting ecosystem growth and retention is key.

However, a number of analysis have already been published. Examples include Blockwork’s STIP analysis, ChaosLabs’ STIP analysis 1 and STIP analysis 2, OpenBlock’s STIP incentive efficacy analysis and STIP efficacy + Sybil analysis, StableLab’s analysing STIP efficacy and LTIPP comparison.

Can you please explain how your research would be different?

[karelvuong]

Thanks for submitting your application as well as your detailed responses back to comments, @PYOR!

With a max budget of 200k ARB to be allocated towards the LTIPP research bounties, PYOR ranks near/at the top of my list of recommendations.

What I like:

• Prior reports published with other institutional customers and protocols serve as a strong demonstration of PYOR’s capabilities.
• Strong methodology and baseline set re: segments and defined metrics provided.
• Proposed budget at 100k is on the higher end to take up half of the research budget but feel it has been well justified through the proposal and responses provided.

Any concerns have already been raised and responded to.

[PYOR]

Thanks @karelvuong for your comments, we appreciate your time.

[PYOR]

Hi @alicecorsini,

Thank you for your question.

We greatly value the significant contributions made by the teams you mentioned in their analysis of STIP and LTIPP. Their work has been an inspiration to us.

There are a couple of key distinctions that set our approach apart:

Instead of focusing solely on individual protocols or aggregating all grantees together, we believe in a nuanced segmentation approach. This allows us to extract the most insightful findings without sacrificing granularity or breadth.

Furthermore, our methodology diverges from OpenBlock’s STIP analysis by emphasising a unique rationale tailored to uncover deeper insights.

Our primary objective remains unchanged from the outset: to equip DAOs with actionable insights derived from STIP, backfund, and LTIPP data. This will enable DAOs to construct a robust framework for future incentive programs.

We hope this answers your query. Let us know if there are any further questions.

Thanks!