Third-Party Data Ingestion

At OP Labs we ingest data from many different third-party data sources. To help implement data ingestion we have developed some common patterns which we cover in this section.

• Overview
  • DataAccess Pattern
  • Directory Structure
• DailyDataset (GCS)
  • Historical Note
  • Subclassing the DailyDataset class
  • Discoverability
  • Writing Data
  • Root Paths
  • Other built-in functionality
  • Execution and Scheduling
  • Prototyping and Debugging
  • Monitoring
  • Advanced use cases
  • When to use ClickHouseData instead
• ClickHouseDataset
  • Subclassing the ClickHouseDataset class.
  • CREATE TABLE
  • Writing Data
  • Execution
  • Scheduling
  • Monitoring