<  Blog

Kin Integrates Digital Ecosystems with Cryptocurrency

September 27, 2022
in
Case Studies

Introduction to Kin 

There are millions of active users today benefiting from being part of the Kin Ecosystem. The reason behind why the $KIN token has gotten so popular today is because it allows creators to directly earn rewards from the value their contributions generate to the community. In the Kin Ecosystem, users can purchase digital goods and services across dozens of apps with their token $KIN, which they can also earn as rewards for being an active user or creator. That means you get to earn money and have a better user experience for your product when building with Kin. Every user, content creator or developers in this Web3 ecosystem benefits directly from its growth. You automatically increase the value of your app and utility of Kin when building with it. And because it’s a cryptocurrency with an incredible array of built-in functionality and programmability so your app can be as flexible as possible. 

“We went from having no source control and resource management to having not only that but also a CI/CD pipeline, being able to rapidly deploy the infrastructure, run the ETL and redeploy our data warehouse as well as our data marts” - Tyler Mason, Lead Data Engineer at Kin

The problem:  

They reach out to us with a very specific problem. There was no good way in their current solution for them to gather the data from all $KIN transactions dating back to the beginning of 2021. 

So the challenge here was to get all current and past information about Kin from the Solana foundation historical archives and process it. 

How did they process it previously? 

Their previous method for gathering the Kin transactions data was extracting whatever they could from their application RPC calls and from multiple RPC endpoints. The problem with that approach is that many transactions were lost and there’s no way to keep track of even how many of them were missing.  

But having the information about all Kin transactions would lead them to be more recognized by the Solana Foundation, receiving extra bonuses to help them expand. 

They also have a built-in incentive model called the Kin Rewards Engine that performs the community payouts. Using their legacy code this service would take around 20 minutes to run just one day of transactions. That was definitely not effective as the ecosystem grows to support more people and to run at scale.  

Before RPC calls missing some data many lines unorganized lines going to legacy solution. 

KIN data architecture before shakudo
KIN data architecture before shakudo

The Solution:  

KIN data architecture after shakudo
KIN data architecture after shakudo

*historical solana data just needed to be extracted once. Now they use shakudo and dask jobs to distribute kin rewards instead of old legacy code. Parallelization 120x faster. 

It would take several days to collect all the blockchain data needed in traditional ways and download everything to a database. 

Instead, our team managed to do the whole process of gathering and processing all the around 100TB of data in just one day.  

We could only do that because the Shakudo platform allowed us to have a pipeline of over 4 thousand jobs running simultaneously on the platform to extract all the raw data. In the next step, this data was sent to be processed through a second pipeline running distributed Dask jobs and extracting information from the data within minutes.  

This whole process that would usually take several days to run, now only takes a few hours in an extremely easy way to set it up. Now that they have access to all historical data they will soon also be able to get real-time data also from inside the platform, switching their RPC endpoints to the ones provided by Shakudo to take advantage of an optimized and reliable option inside their current application environment.  After they switched to Shakudo, they also realized the platform could help them not only get all the data they need but also solve their infrastructure scalability issues when dealing with the Kin Rewards Engine.  Using Shakudo alongside Snowflake, the process that used to take over 20 minutes to run one day of transactions can now run every transaction from today to their ingestion date in 2021 in just one hour. Doing some quick math, this is the equivalent of now taking 10 seconds to process one day of transactions or 120 times faster. 

Not only that, but their resource management was also improved. When comparing the cost savings with their previous solution we get an estimate of over 85% reduction already calculated counting all Shakudo fees inside. 

Being able to streamline your infrastructure and not having to manage anything about it, empowered Kin’s Data Engineers to be able to do what would take a whole DevOps team weeks to achieve in just a few days. 

In their words, with the Shakudo Platform “You just need one line of Python and a SQL Script to get a lot done” 

KIN Metrics:  

Figure A-1_ Kin Transaction Ecosystems Stats by period
Figure A-1_ Kin Transaction Ecosystems Stats by period
Figure A-2_ Combo diagram of wallets created with SOL
Figure A-2_ Combo diagram of wallets created with SOL
Figure A-3_ Daily Ecosystem Creates with Distinct Apps Making Creates
Figure A-3_ Daily Ecosystem Creates with Distinct Apps Making Creates
Figure 4-A_ Daily Transactions by App
Figure 4-A_ Daily Transactions by App

Shakudo Team

Shakudo unites all of the data tools and services into a single platform, allowing your team to develop and deploy solutions with ease.

Ready to get started?