datarepo - Neuralink's platform for complex data

Today we’re announcing datarepo, Neuralink’s platform for managing multimodal data. 

We’re open sourcing the first component of the datarepo ecosystem - the datarepo catalog. This Python library allows you to:

• Declaratively define a catalog as code, federating disparate data sources
• Use a uniform query interface for all data sources
• Generate a web catalog for easy browsing
• Generate a scalable, stateless API by exporting a ROAPI configuration

See the datarepo project on GitHub.

Try out the demo.

See our talk on the platform (previously called Neuralake) at Databricks’ Data + AI Summit.

We're building a generalized brain interface to help people with unmet medical needs and to expand human capability. 

Our operations generate complex multimodal data, including: 

• Real-time neural signals streaming from the brain
• Billion-pixel histopathology images for assessing biocompatibility
• Real-time manufacturing data 
• Surgery video and telemetry
• 3D volumetric brain scans

This is more than just tables and log stores—we're managing a diverse set of multimodal data. This data is stored across object stores, relational databases, and even spreadsheets.

Complex scientific investigations rarely end where they begin—one question inevitably branches into many. Answering deeply interdisciplinary questions requires access across different data sources. Our device designs and surgical process are incredibly nuanced, and we can only improve them with data-driven decisions.

We’ve built a uniform interface for accessing that data – a simple data catalog that can be defined in code, generating a web UI and a stateless API server.

Regardless of where data is stored, it can be cataloged with datarepo and queried using the uniform interface.

A datarepo data catalog is defined in code using a declarative Python syntax. A data table can be a Delta Lake table, Parquet table, or a virtual table backed by a Python function. 

from datarepo.core import (
    ParquetTable,
    Filter,
    Partition,
    PartitioningScheme,
)

import polars as pl

partsupp = ParquetTable(
    name="partsupp",
    uri="s3://my-bucket/tpc-h/partsupp",
    partitioning=[
        Partition(column="ps_partkey", col_type=pl.Int64),
        Partition(column="ps_suppkey", col_type=pl.Int64),
    ],
    partitioning_scheme=PartitioningScheme.HIVE,
    docs_filters=[
        Filter("ps_partkey", "=", 1),
        Filter("ps_suppkey", "=", 1),
    ],
    description="""
    Part supplier information from the TPC-H benchmark.
    Contains details about parts supplied by suppliers.
    """,
    table_metadata_args={
        "data_input": "Supplier inventory and pricing data",
        "latency_info": "Real-time updates from supplier",
        "example_notebook": "https://example.com/notebooks/partsupp.ipynb",
    },
)

The datarepo catalog can be serialized to JSON, which can then be used to generate a web UI for easy browsing. 

The web catalog can be generated with a single Python statement.

from datarepo.export.web import export_and_generate_site
from tpc_catalog import TPCCatalog

export_and_generate_site(
    catalogs=[("tpc", TPCCatalog)], output_dir=str(output_dir)
)

Unlike other data catalog systems, the datarepo web catalog is deployed as a static site – it loads instantly and doesn’t need any running services or databases. 

You can try out a demo catalog here.

A datarepo catalog can be queried through a uniform Python interface. All tables return a polars LazyFrame, allowing for lazy evaluation, predicate pushdown, and fluent syntax for dataframe manipulation.

No matter where your data lives, you can query it with the same syntax.

>>> from tpc_catalog import TPCCatalog
>>> from datarepo.core import Filter
>>> 
>>> # Get part and supplier information
>>> part_data = TPCCatalog.db("tpc-h").table(
...     "part",
...     (
...         Filter('p_partkey', 'in', [1, 2, 3, 4]),
...         Filter('p_brand', 'in', ['Brand#1', 'Brand#2', 'Brand#3']),
...     ),
... )
>>> 
>>> supplier_data = TPCCatalog.db("tpc-h").table("supplier")
>>> 
>>> # Join part and supplier data and select specific columns
>>> joined_data = part_data.join(
...     supplier_data,
...     left_on="p_partkey",
...     right_on="s_suppkey",
... ).select(["p_name", "p_brand", "s_name"]).collect()
>>> 
>>> print(joined_data)

A datarepo catalog can be serialized into a ROAPI config file. ROAPI uses Apache Arrow and Datafusion to spin up a stateless API server. Users can query the ROAPI server with SQL, REST APIs, GraphQL, and the Postgres wire protocol.

ROAPI can be configured as a Grafana data source, allowing for self-serviced low-code visualizations and alerting on data cataloged with datarepo.

With datarepo, your catalog definition lives in one place, allowing for web browsing, a uniform query interface, and stateless API generation.

We use the datarepo catalog in production here at Neuralink. By making our tooling free and open source, we hope to help accelerate data engineering and biotechnology communities around the world. We look forward to seeing what the open source community can build with these tools. 

This is just the beginning – the datarepo catalog is a core component of the larger datarepo ecosystem, which includes real-time ingestion pipelines and an LLM-based interface for natural language queries.

We plan to open source other components of the datarepo ecosystem, so stay tuned!

The datarepo platform is designed to be extendable. If you have specific use cases that aren't supported, please make a Pull Request, file an issue ticket, or start a discussion! We welcome code contributions that add support for new data sources, provide optimizations, or make other improvements. 

NOTE: This is an open source tool for users to manage their own data. This is not a release of any Neuralink data.