Onchain Data Processing

At OP Labs we deal with data from more than 20 blockchains, so we needed a low cost, scalable and maintainable way to manage data pipelines and build data models for all of them.

Blockbatch is a data ingestion and processing pattern developed at OP Labs to address these challenges. Let’s cover the pattern in detail.

• Design Principles
  • Batch Size Configuration
  • Data Artifacts and Storage Location
  • Date Partitioning
  • Partitioning at Ingestion
• Marker Metadata Layer
  • Why are markers needed?
  • Monitoring
  • Job Planning
• Ingestion
  • Raw Data Audits
  • Raw Data Schemas
• Blockbatch Models
  • Inexpensive Joins
  • DuckDB Compute Engine
  • Interface
  • Model Definitions in Python
  • Prototyping
  • Integration Testing