datashard Documentation

Welcome to the official documentation for datashard, a Python implementation of Apache Iceberg concepts for safe concurrent data operations in ML/AI workloads.

Getting Started

• Installation
  • Prerequisites
  • Installing from PyPI
  • Installing from Source
  • Development Installation
  • Verification
• Quickstart Guide
  • Creating Your First Table
  • Adding Data with Transactions
  • Reading Data and Snapshots
  • Time Travel Example
  • Advanced Example: Complex Data Operations
  • Reading Data with Filters
  • Streaming Large Tables
  • Cleaning Up
  • What’s Next?

User Guide

• Core Concepts
  • What is datashard?
  • Key Concepts
  • How it Relates to Apache Iceberg
  • Benefits of datashard
• Transactions and ACID Properties
  • Understanding Transactions
  • The ACID Properties
  • Using Transactions
  • Transaction Operations
  • Concurrency Control
  • Transaction Isolation Levels
  • Error Handling
  • Best Practices
  • Example: Complete Transaction Workflow
• Schemas and Data Types
  • Schema Overview
  • Creating Schemas
  • Data Types
  • Working with Schemas
  • Schema Evolution
  • Best Practices
  • Example: Complete Schema Workflow
• Time Travel and Historical Data Access
  • Understanding Time Travel
  • Snapshot Basics
  • Accessing Snapshots
  • Time Travel Operations
  • Managing Snapshots
  • Use Cases for Time Travel
  • Advanced Time Travel
  • Complete Time Travel Workflow
• Concurrency and Safe Multi-Process Access
  • Concurrency Challenges
  • Optimistic Concurrency Control (OCC)
  • Multiple Process Scenarios
  • Best Practices for Concurrency
  • Monitoring Concurrency
  • Advanced Concurrency Patterns
• S3-Compatible Storage
  • Overview
  • Benefits
  • Configuration
  • Usage Examples
  • Storage Structure
  • Architecture
  • Performance
  • Troubleshooting
  • Best Practices
  • Security
  • See Also

API Reference

• iceberg Module API
  • Main Functions
  • Classes
• transaction Module API
  • Classes
• data_structures Module API
  • Enums
  • Classes
• metadata_manager Module API
  • Exceptions
  • Classes
• snapshot_manager Module API
  • Classes

Advanced Topics

• Performance Optimization
  • File Format Selection
  • Partitioning Strategies
  • Transaction Optimization
  • Memory Management
  • Concurrency Optimization
  • Monitoring and Profiling
  • Query Optimization (NEW in v0.3.3)
  • Hardware Considerations
• Troubleshooting
  • Common Issues and Solutions
  • Debugging Techniques
  • Common Configuration Issues
  • Environment-Specific Issues
  • Support and Community

Community

• Contributing to datashard
• Changelog

About datashard

datashard provides ACID transactions, time travel, and safe concurrent access to data in a pure Python implementation of Apache Iceberg concepts. It’s designed specifically for ML/AI workloads where data integrity and concurrent access are critical.

Key Features:

• ACID Transactions: Atomic, Consistent, Isolated, Durable operations

• Time Travel: Query data as it existed at any point in time

• Safe Concurrency: Multiple processes can read/write without corruption

• S3-Compatible Storage: AWS S3, MinIO, DigitalOcean Spaces support (v0.2.2+)

• Distributed Workflows: Run workers across different machines with shared S3 storage

• Schema Evolution: Add/remove columns while maintaining history

• Multiple File Formats: Parquet, Avro, ORC support

• Data Integrity: Built-in validation and error handling

Indices and tables

• Index

• Module Index

• Search Page