A Python library for managing datasets with lineage tracking in data science projects.

• Automatic Lineage Tracking: Track data provenance through all operations automatically
• Dataset Management: Integration with datasets.yaml for organized dataset registration
• Pandas-Compatible API: Familiar pandas-like interface via from sunstone import pandas as pd (CSV, Excel, JSON)
• Geospatial Support: Lineage-tracking GeoJSON/TopoJSON read/write via from sunstone import geopandas as gpd (requires the [geo] extra)
• Plugin System: Extensible architecture for custom auth providers, URL handlers, and format handlers via entry points
• Strict/Relaxed Modes: Control whether operations can modify datasets.yaml
• Validation Tools: Check notebooks and scripts for correct import usage
• Full Type Hints: Complete type hint support for better IDE integration

# Using uv (recommended)
uv add sunstone-py

# Using pip
pip install sunstone-py

To use the latest commit from github:

dependencies = [
    "sunstone-py @ git+https://github.com/sunstoneinstitute/sunstone-py.git",
]

If you are making changes to a local checkout of sunstone-py and want to test them from your project, add a [tool.uv.sources] override to your project's pyproject.toml:

[tool.uv.sources]
sunstone-py = { path = "../path/to/sunstone-py", editable = true }

The path is relative to your project's pyproject.toml. Leave the regular PyPI dependency in [project.dependencies] unchanged — the sources override takes precedence locally. Remember to remove the [tool.uv.sources] block before committing.

git clone https://github.com/sunstoneinstitute/sunstone-py.git
cd sunstone-py
uv venv
uv sync
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Create a datasets.yaml file in your project directory:

inputs:
  - name: School Data
    slug: school-data
    location: data/schools.csv
    source:
      name: Ministry of Education
      location:
        data: https://example.com/schools.csv
      attributedTo: Ministry of Education
      acquiredAt: 2025-01-15
      acquisitionMethod: manual-download
      license: CC-BY-4.0
    fields:
      - name: school_id
        type: string
      - name: enrollment
        type: integer

outputs: []

from sunstone import pandas as pd
from pathlib import Path

# Set project path (where datasets.yaml lives)
PROJECT_PATH = Path.cwd()

# Read data - lineage automatically tracked
df = pd.read_csv('data/schools.csv', project_path=PROJECT_PATH)

# Transform using familiar pandas operations
result = df[df['enrollment'] > 100].groupby('district').sum()

# Save with automatic lineage tracking and dataset registration
result.to_csv(
    'outputs/summary.csv',
    slug='school-summary',
    name='School Enrollment Summary',
    index=False
)

# View lineage information
print(result.lineage.sources)      # Source datasets
print(result.lineage.operations)   # Operations performed
print(result.lineage.get_licenses())  # All source licenses

sunstone-py provides a drop-in replacement for pandas that adds lineage tracking:

from sunstone import pandas as pd

# Works like pandas, but tracks lineage
df = pd.read_csv('input.csv', project_path='/path/to/project')
df2 = pd.read_csv('input2.csv', project_path='/path/to/project')

# All pandas operations work
filtered = df[df['value'] > 100]
grouped = df.groupby('category').sum()

# Merge/join operations combine lineage from both sources
merged = pd.merge(df, df2, on='key')
concatenated = pd.concat([df, df2])

With the [geo] extra installed, sunstone.geopandas mirrors the pandas facade for GeoJSON and TopoJSON vector data. Read functions resolve a slug/path against datasets.yaml and return a GeoDataFrame wrapping a geopandas.GeoDataFrame with metadata and lineage:

from sunstone import geopandas as gpd

# Read by registered slug or path (auto-detect format with read_file)
gdf = gpd.read_geojson('regions.geojson')
topo = gpd.read_topojson('regions.topojson')
auto = gpd.read_file('regions')

# Access the underlying geopandas.GeoDataFrame
print(gdf.data.crs)

# Write with lineage tracking (slug + name required for new outputs)
gdf.to_geojson('output.geojson', slug='regions-out', name='Regions Output')
gdf.to_topojson('output.topojson', slug='regions-topo', name='Regions TopoJSON')

Relaxed Mode (default):

• Writing to new outputs auto-registers them in datasets.yaml
• More flexible for exploratory work

Strict Mode:

• All reads and writes must be pre-registered in datasets.yaml
• Ensures complete documentation of data operations
• Enable via strict=True parameter or SUNSTONE_DATAFRAME_STRICT=1 environment variable

# Enable strict mode
df = pd.read_csv('data.csv', project_path=PROJECT_PATH, strict=True)

# Or globally
import os
os.environ['SUNSTONE_DATAFRAME_STRICT'] = '1'

Check notebooks for correct import usage:

import sunstone

# Check a single notebook
result = sunstone.check_notebook_imports('analysis.ipynb')
print(result.summary())

# Check all notebooks in project
results = sunstone.validate_project_notebooks('/path/to/project')
for path, result in results.items():
    if not result.is_valid:
        print(f"\n{path}:")
        print(result.summary())

sunstone-py uses a plugin architecture for reading, writing, and fetching data. Built-in handlers cover common formats (CSV, JSON, Excel, Parquet, TSV) and HTTP/HTTPS, local file, GCS, and S3/R2 URLs.

Plugins implement one or more of these protocols:

• AuthProvider: Injects authentication headers into HTTP requests
• URLHandler: Opens URLs for reading/writing, returning file-like streams (BinaryIO/TextIO)
• FormatHandler: Reads and writes data formats not built into sunstone

pip install sunstone-py          # Core + HTTP + local file handling
pip install sunstone-py[gcs]     # Adds GCS (gs://) support
pip install sunstone-py[s3]      # Adds S3 (s3://) and R2 (r2://) support
pip install sunstone-py[geo]     # Adds GeoJSON/TopoJSON vector support
pip install sunstone-py[gcs,s3]  # Both

Plugins are discovered via Python entry points:

[project.entry-points."sunstone.plugins"]
my-plugin = "my_package:MyPlugin"

Plugin config uses cascading precedence (later sources override earlier):

1. datasets.yaml — plugins.<name> section
2. pyproject.toml — [tool.sunstone.plugins.<name>] table
3. Environment variables — SUNSTONE_PLUGIN_<NAME>_<KEY>

For more control, use the DataFrame class directly:

from sunstone import DataFrame

# Read with explicit parameters
df = DataFrame.read_csv(
    'data.csv',
    project_path='/path/to/project',
    strict=True
)

# Access underlying pandas DataFrame
pandas_df = df.data

Set metadata on DataFrames that flows through to datasets.yaml on write:

from sunstone import pandas as pd

df = pd.read_csv('input.csv', project_path=PROJECT_PATH)
result = df[df['value'] > 100]

# Set output identity and description
result.metadata.slug = "filtered-data"
result.metadata.name = "Filtered Data"
result.metadata.description = "Values above threshold"

# Set RDF metadata
result.metadata.rdf_prefixes = {"schema": "https://schema.org/"}
result.metadata.custom_properties = {"schema:about": "Analysis"}

# Annotate columns
result.set_field_metadata("value", description="Measured value", unit="kg")

# Write — slug/name come from metadata
result.to_csv('outputs/filtered.csv', index=False)

Available metadata:

• df.metadata.slug: Dataset slug (used at write time)
• df.metadata.name: Dataset name (used at write time)
• df.metadata.description: Dataset description
• df.metadata.rdf_prefixes: RDF namespace prefixes
• df.metadata.custom_properties: Custom properties (RDF-style)
• df.set_field_metadata(column, *, description, unit, source, type, constraints): Annotate a column

from sunstone import DatasetsManager, FieldSchema

manager = DatasetsManager('/path/to/project')

# Find datasets
dataset = manager.find_dataset_by_slug('school-data')
dataset = manager.find_dataset_by_location('data/schools.csv')

# Add new output dataset
manager.add_output_dataset(
    name='Analysis Results',
    slug='analysis-results',
    location='outputs/results.csv',
    fields=[
        FieldSchema(name='category', type='string'),
        FieldSchema(name='count', type='integer'),
        FieldSchema(name='avg_value', type='number')
    ],
    publish=True
)

• Contributing Guide
• Changelog
• API Reference (below)

Drop-in replacement for pandas with lineage tracking:

• read_csv(filepath, project_path, strict=False, **kwargs): Read CSV with lineage
• read_excel(filepath, project_path, strict=False, **kwargs): Read Excel (.xlsx/.xls) with lineage
• read_json(filepath, project_path, strict=False, **kwargs): Read JSON with lineage
• merge(left, right, **kwargs): Merge DataFrames with combined lineage
• concat(dfs, **kwargs): Concatenate DataFrames with combined lineage

Lineage-tracking facade for vector data (requires the [geo] extra):

• read_geojson(slug_or_path, project_path=None): Read GeoJSON into a GeoDataFrame
• read_topojson(slug_or_path, project_path=None): Read TopoJSON into a GeoDataFrame
• read_file(slug_or_path, project_path=None): Read with format auto-detected from the dataset
• GeoDataFrame.to_geojson(path, slug, name): Write GeoJSON and register
• GeoDataFrame.to_topojson(path, slug, name): Write TopoJSON and register
• GeoDataFrame.data: Access the underlying geopandas.GeoDataFrame
• GeoDataFrame.metadata: Access the unified metadata container

Main class for working with data:

• read_csv(filepath, project_path, strict=False, **kwargs): Read CSV with lineage tracking
• read_excel(filepath, project_path, strict=False, **kwargs): Read Excel with lineage tracking
• to_csv(path, slug, name, publish=False, **kwargs): Write CSV and register
• merge(right, **kwargs): Merge with another DataFrame
• join(other, **kwargs): Join with another DataFrame
• concat(others, **kwargs): Concatenate DataFrames
• set_field_metadata(column, **kwargs): Annotate column metadata
• .data: Access underlying pandas DataFrame
• .metadata: Access unified metadata container
• .lineage: Access lineage metadata (deprecated — use .metadata.lineage)

Manage datasets.yaml files:

• find_dataset_by_location(location, dataset_type='input'): Find by file path
• find_dataset_by_slug(slug, dataset_type='input'): Find by slug
• get_all_inputs(): Get all input datasets
• get_all_outputs(): Get all output datasets
• add_output_dataset(...): Register new output
• update_output_dataset(...): Update existing output

• check_notebook_imports(notebook_path): Validate a single notebook
• validate_project_notebooks(project_path): Validate all notebooks in project

• AuthProvider: Implement authenticate(url, headers, dataset) -> headers to inject auth
• URLHandler: Implement can_handle(url) -> bool and open(url, mode) -> BinaryIO | TextIO
• FormatHandler: Implement can_read(path, format), read(stream, **kwargs), can_write(path, format), write(df, stream, **kwargs)

Singleton that discovers and manages plugins:

• PluginRegistry.get(): Get the singleton registry instance
• get_auth_providers(): Return all registered auth providers
• get_url_handlers(): Return all registered URL handlers
• get_format_handlers(): Return all registered format handlers
• find_url_handler(url): Find first handler that can handle a URL
• find_format_reader(path, format): Find first handler that can read a file
• find_format_writer(path, format): Find first handler that can write a file
• fetch(url, dest): Convenience — download URL to local file via open()

• SunstoneError: Base exception
• DatasetNotFoundError: Dataset not found in datasets.yaml
• StrictModeError: Operation blocked in strict mode
• DatasetValidationError: Validation failed
• LineageError: Lineage tracking error

• SUNSTONE_DATAFRAME_STRICT: Set to "1" or "true" to enable strict mode globally
• SUNSTONE_PLUGIN_<NAME>_<KEY>: Override plugin configuration (highest precedence)

See CONTRIBUTING.md for development setup and guidelines.

uv run pytest

uv run mypy

uv run ruff check
uv run ruff format

Sunstone Institute is a philanthropy-funded organization using data and AI to show the world as it really is, and inspire action everywhere.

MIT License - see LICENSE file for details.

• Issues: GitHub Issues

Made with ❤️ by Sunstone Institute