Open Source software plays a critical role in our everyday life here at °41North. We use countless frameworks and plugins and snippets and so on in every one of our projects. It’s quite frankly impossible nowadays to build anything without relying upon the huge amount of effort that thousands of developers have put into Open Source software.
For these reasons we feel it’s important to give back to the Open Source community, and this article introduces our first contribution as a company.
But before we get into that let me share some background on why we ended up creating Exflo in the first place.
Down the rabbit hole
Over the past several years I have been getting increasingly more involved within the Crypto / Blockchain community, with my main focus being Ethereum.
This began with some contributions to the client for Ubiq, a stable fork of Ethereum. Following that I ended up creating a basic clone of Infura called Shokku. After participating in the ”Open-Source Blockchain Explorers Now” initiative led by Griff Green I eventually ended up working with MyEtherWallet on a blockchain explorer.
Thus began the slow descent down the rabbit hole that is ’introspecting the Ethereum blockchain‘.
Anyone who has ever tried this usually starts out with the Web3 API and eventually ends up forking one of the main clients and customising it. We were no different.
Devcon IV
Whilst attending Devcon IV in Prague, I met some developers working on a new Java based client for Ethereum called Pantheon.
At that time we were using EthereumJ but we weren’t particularly happy with it and looking for an alternative, ideally JVM based.
Pantheon looked promising and we gave it a trial run but in the end it just wasn’t ready.
Hyperledger Besu
Fast forward a couple of years and Pantheon, now a part of the Hyperledger Foundation, has been renamed Besu. An awful lot of work has gone into improving and expanding it during that time.
So after setting up our new company and deciding we weren’t quite finished yet with Ethereum, we set out to explore approaches for more easily extracting chain data, and Besu seemed like the logical starting point.
- It has a modern codebase with good design patterns and documentation.
- There is a plugin system allowing for customization and integration without the need to maintain a fork.
- It’s written in Java, which allows us to work within the JVM and use our preferred language for processing, Kotlin.
- Their community is open and welcoming.
Introducing Exflo
We set out with Exflo to try and simplify the approach of extracting chain data from the Ethereum blockchain with one guiding principle:
Extract or compute as much information as possible from within the node itself
The information we wanted to extract included:
- Block Headers.
- Transactions.
- Receipts.
- Transaction Traces.
- Contract creations and self-destructs.
- Log events for standards-compliant ERC20, ERC721, ERC777, and ERC1155 tokens.
- A detailed breakdown of Ether movements, e.g. block rewards, transaction fees, simple ether transfers, internal transactions and so on.
- Per-block account state changes.
After a few iterations and a lot of testing, Exflo was born.
Designed as a plugin for Hyperledger Besu, it takes advantage of their plugin API and a few additional workarounds, to extract the information above into either a Postgres database or a Kafka topic, with more data stores to be integrated later.
Written in Kotlin, it uses a mix of what we think are pretty cool frameworks and libraries such as:
- Jooq for automatic entity generation from Postgres.
- RxJava for reactive concurrency patterns.
- FlatBuffers a compact binary format (used by the Kafka data store).
- KotlinPoet for auto-generation of Kotlin code.
Be warned, it is currently in alpha, with some known gaps and issues we want to address in the near future. Most of our testing has been with a private testnet and Ropsten, so there may still be some dragons lurking about.
Join and contribute!
Our hope is to continue to improve upon this initial version as our resourcing permits, adding more tests, data stores and improving performance.
That being said we are also more than happy to accept help, so feel free to visit the official repository and be sure to checkout our contribution guidelines before creating your first PR!
Get in touch
If you think Exflo would be useful for your use case and want to talk more about it you can reach out to us via our contact form or by sending an email to [email protected]. We try to respond within 48 hours and look forward to hearing from you.
