Development Guide#

In this section we have short guides for getting your development environment up and running.

Getting familiar with the repo#

The op-analytics repo has been undergoing a migration from our legacy codebase to a new python codebase for our data platform. The new codebase is all defined under src/op-analytics so that should be the main entry point to work on new code or find the code for some functionality in our data pipeline.

Contents Overview#

Here is an overview of some important directories and files in the repo.

  • .circleci: Configuration for CircleCI jobs.

  • .github: Configuration for CODEOWNERS.

  • .venv: Virtual environment managed by uv.

  • .vscode: Shared VS Code settings. Make sure to install all the recommended extensions.

  • dashboards: Streamlit dashboards.

  • ddl: CREATE TABLE statements for tables that have been created manually in our databases.

  • demos: Archive for demo materials.

  • docs: Directory where HTML for renderd documentation is written out. Shouldn’t be edited manually.

  • helm/dagster: Helm chart configuration for our Dagster deployment. Has information about how Dagster was set up and holds the values.yaml file that configures all of Dagster (including the user deployments image version).

  • k8s: Kubernetes manifests. Before moving to Dagster we used to run cronjobs on Kubernetes. We don’t do that anymore but we still use Kubenetes for lengthy backfill jobs.

  • notebooks: IPython notebooks. For testing ideas, prototyping pipelines and backfilling.

  • scripts: Useful shell scripts for adhoc purposes.

  • sphinx: The source files for the documentation site.

  • src/op_analytics: Source code for the data platform.

  • tests/op_analytics: Unit tests for the data platform.

  • Makefile: Make commands to do things like build docker images or build the documentation site.

  • Dockerfile: Definition of the op-analytics docker image that gets deployed to Kubernetes.

  • Dockerfile.dagster: Definition of the op-analytics docker image that gets deployed to Dagster.

The following directories/files are part of our legacy data platform and are still in use:

  • helper_functions

  • op_chains_tracking

  • op_collective_economics

  • op_governance_data

  • other_chains_tracking

  • reference_data

  • rpgf

  • value_locked_flows

  • Pipfile

Virtual Environment#

We use uv to manage the virtual environment and project dependencies. After installing uv you can create your development virtualenv by running:

$ uv sync

Google Cloud Credentials#

Some functionality on this repo requires you to be authenticated with google cloud. You should install the gcloud cli and then run:

gcloud auth application-default login
gcloud config set project oplabs-tools-data

Note, if you see the following error then either you are not authenticated or you do not have access to the required resource:

DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials

Secrets Management#

Whenever we need to make use of a secrete value in our codebase we use the env_get() function part of the vault.py module. This function can load up secrets from multiple locations and expose them to our code uniformly.

When running on Kubernetes/Dagster secrets are provided via a SecretProviderClass kubernetes resource (see helm/dagster/secret-provider.yaml and k8s/secret-provider.yaml). The provider manifest configures usage to a specific version, which is controlled manually using the Secrets Manager UI in Google Cloud Platform.

When running locally you can provide secretes using the .env file in the repo. This file should have a single variable called OP_ANALYTICS_VAULT which is an encoded string that contains the secret information. The value for this can be found in the shared team vault.

Kubernetes + Helm#

To work witho our kubernetes cluster and the Dagster helm deployment that runs on it you will need the kubectl and helm command-line tools.

Setting up Docker#

You will need to install docker on your laptop.

To be able to publish docker images you will need to create and configure a Github Personal Access Token (the classic version). Go here for more instructions.

On your personal github settings page (https://github.com/settings/tokens) you need to “Configure SSO” for the token and grant it access to the ethereum-optimism organization. This will give you permissions to docker push images on the ethereum-optimism container registry.

Once you have your personal access token and you have granted it the right permissions you should use the docker login command to authenticate to the Github Container Registry (ghcr.io). Follow this link for more instructions on how to pass the token to the docker login command.

Here are the package links for the docker images that we publish:

Updating Dagster#

To deploy a new version of the code to Dagster you need to first manually bump the Dagster image version in two places in the codebase:

  • Makefile

  • helm/dagster/values.yaml

You can search and replace for the version and replace it in both places at once.

After updatatng the version then run the make docker-dagster command to build the image, tag it with the updated version and push it to the Github Container Registry.

Finally update the Dagster helm chart, telling it about the new image by running make helm-dagster.

You can do both of the above at once with make docker-dagster && make helm-dagster.

Updating the Docs#

The source files for the documentation can be found in the sphinx/ directory. To iterate locally on the docs before deploying you can open up a terminal and run make sphinx-serve. This will let you browse the docs over in localhost:8000. You can leave the server open and run make html on a different terminal to update the output after editing markdown files.