OP Labs Data Platform

This guide takes you through the OP Labs Data Platform. We go over the platform architecture and common patterns used by the platform to ingest and process data.

• Architecture
  • Ingestion
  • Storage
  • ClickHouse Data Warehouse
  • BigQuery Data Warehouse
  • Processing and Orchestration
  • Insights
• Onchain Data Processing
  • Design Principles
    • Batch Size Configuration
    • Data Artifacts and Storage Location
    • Date Partitioning
    • Partitioning at Ingestion
  • Marker Metadata Layer
    • Why are markers needed?
    • Monitoring
    • Job Planning
  • Ingestion
    • Raw Data Audits
    • Raw Data Schemas
  • Blockbatch Models
    • Inexpensive Joins
    • DuckDB Compute Engine
    • Interface
    • Model Definitions in Python
    • Prototyping
    • Integration Testing
• Third-Party Data Ingestion
  • Overview
    • DataAccess Pattern
    • Directory Structure
  • DailyDataset (GCS)
    • Historical Note
    • Subclassing the DailyDataset class
    • Discoverability
    • Writing Data
    • Root Paths
    • Other built-in functionality
    • Execution and Scheduling
    • Prototyping and Debugging
    • Monitoring
    • Advanced use cases
    • When to use ClickHouseData instead
  • ClickHouseDataset
    • Subclassing the ClickHouseDataset class.
    • CREATE TABLE
    • Writing Data
    • Execution
    • Scheduling
    • Monitoring
• Data Warehouse
  • Onchain Data Pipelines: Blockbatch Load
    • Ad-hoc Blockbatch Queries
    • Data Loading Jobs
      • Loading by Blockbatch
      • Loading by Date and Chain
    • Load Job Specification
      • Location
      • Naming convention
      • Job Specification
    • Data Readiness, Markers, and Job Idempotency
    • Building Data Pipelines
      • Execution
      • Prototyping and Backfilling
      • Scheduling
      • Monitoring
  • General Purpose Data Pipelines: Transforms
    • Transform Groups
    • Transform Specification
      • Directory Structure and Naming Convention
    • Transform Execution Model
    • Building Data Pipelines
      • Execution
      • Prototyping and Backfilling
      • Scheduling
      • Markers
      • Monitoring
  • Comparison to dbt and SQLMesh
    • What dbt and SQLMesh offer
    • Why we didn’t chose dbt or SQLMesh
      • Lineage
      • Testability
      • SQL First (no indirection)
    • When dbt and SQLMesh might be a better fit
• OP Labs Public BigQuery Data
  • Data Readiness
  • Loading Function
  • Datasets
    • Raw Onchain Data (listing URL)
    • Account Abstraction (listing URL)