Kore — Killer Optimized Record Exchange

A high-performance, columnar file format for analytics with cloud storage connectors.

Kore is a Rust-based columnar file format designed for efficient storage and analysis of structured data. It provides zero external dependencies in the base library with optional cloud connectors for AWS S3, Azure Blob Storage, and Google Cloud Storage.

pip install kore-fileformat==1.1.4

from kore_fileformat import compress_csv

# Compress a CSV file (that's it!)
original, compressed, ratio = compress_csv("data.csv", "data.kore")
print(f"✅ Compressed {ratio:.1%}")

from kore_fileformat import get_kore_info

# Get file information
info = get_kore_info("data.kore")
print(f"Records: {info['total_records']:,}")

👉 New users: Start with docs/INSTALLATION.md → docs/USER_GUIDE.md

• ✅ Zero External Dependencies: Lightweight base crate
• ✅ Columnar Format: Optimized for analytical queries
• ✅ Compression: Built-in data compression
• ✅ Multi-Platform: Windows, macOS, Linux, web

• ✅ AWS S3: Full implementation in v1.0.0
• ⏳ Azure Blob Storage: Full implementation coming in v1.1.0
• ⏳ Google Cloud Storage: Full implementation coming in v1.1.0

• ✅ Python: PyO3 wheel for Python 3.9-3.12
• ✅ Java: JNI bindings and Maven package
• ✅ JavaScript: NAPI module for Node.js
• ⏳ Go: Coming in v1.2.0

• ✅ GitHub Actions: 10 automated jobs for testing and publishing
• ✅ Docker Support: Integration tests with emulators
• ✅ Multi-Registry Publishing: crates.io, PyPI, Maven Central, npm

Process large datasets efficiently with columnar storage:

import kore_fileformat
# Store analytics data in columnar format for fast queries

Store data directly in S3, Azure, or GCS:

from kore_fileformat import S3Reader
reader = S3Reader(region='us-east-1')
data = reader.read_file('my-bucket', 'path/to/data.kore')

Use Kore from Python, Java, or JavaScript in the same project:

# Python: import kore_fileformat
# Java: import com.kore.cloud.S3Reader;
# JS: const kore = require('kore-fileformat');

kore_fileformat/
├── core/           # Base library (zero dependencies)
├── cloud/          # Cloud connectors (optional)
│   ├── s3/        # AWS S3 (working)
│   ├── azure/     # Azure Blob (v1.1+)
│   └── gcs/       # Google Cloud (v1.1+)
└── bindings/       # Language bindings
    ├── python/    # PyO3 wheel
    ├── java/      # JNI library
    └── napi/      # Node.js addon

# Base: zero external dependencies
kore_fileformat = "1.0.0"

# With S3
kore_fileformat = { version = "1.0.0", features = ["s3"] }

# With all cloud (v1.1.0+)
kore_fileformat = { version = "1.0.0", features = ["s3", "azure", "gcs"] }

# With Python bindings
# Use: pip install kore-fileformat

Kore is designed for analytics workloads:

• Compression: 5-10x reduction on typical datasets
• Query Speed: Columnar format enables fast aggregations
• Storage: 10-50 MB files with millions of rows
• Cloud: Direct S3/Azure/GCS integration (no intermediate files)

• Rust: 1.70+ (for building from source)
• Python: 3.9-3.12 (for Python wheel)
• Java: 17+ (for Java bindings)
• Node.js: 14+ (for JavaScript bindings)
• Docker: 20.10+ (for testing with emulators)

pip install kore-fileformat

cargo add kore_fileformat --features s3

git clone https://github.com/arunkatherashala/Kore.git
cd Kore
cargo build --release --features s3

cargo test

# Start emulators (LocalStack, Azurite, GCS)
docker-compose up -d

# Run tests
cargo test --features s3,azure,gcs --test integration_tests -- --nocapture

# Stop emulators
docker-compose down

See DOCKER_EMULATORS_GUIDE.md for detailed setup.

from kore_fileformat import S3Reader

reader = S3Reader(region='us-east-1')
data = reader.read_file('bucket', 'object.kore')
reader.write_file('bucket', 'object.kore', data)

from kore_fileformat import AzureBlobReader

reader = AzureBlobReader('account', 'key')
data = reader.read_file('container', 'blob.kore')

from kore_fileformat import GcsReader

reader = GcsReader('project-id')
data = reader.read_file('bucket', 'object.kore')

We welcome contributions! Here's how:

1. Report Issues: GitHub Issues
2. Discuss Ideas: GitHub Discussions
3. Submit PRs: Fork, branch, code, and create a pull request

See V1_1_ROADMAP.md for planned features and how to help.

• S3 connector with full API
• Python, Java, JavaScript bindings
• Integration tests with emulators
• Complete documentation

• Azure Blob Storage full implementation
• Google Cloud Storage full implementation
• Performance optimizations
• Streaming support

• Go language bindings
• Multi-region support
• Caching layer
• Advanced compression

See V1_1_ROADMAP.md for detailed phases and milestones.

• Zero external dependencies in base library
• Optional SDKs are version-pinned and updated regularly
• Integration tests verify cloud connectivity
• GitHub Actions security scanning

Please email: arunkatherashala@gmail.com

• Documentation: DOCUMENTATION_INDEX.md
• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: arunkatherashala@gmail.com

• GitHub: Star the repository
• Releases: Watch for v1.1.0 announcement
• Email: Subscribe to release notifications

Kore is licensed under the Apache License 2.0.

Copyright 2024-2026 Sai Arun Kumar Ktherashala

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Sai Arun Kumar Ktherashala

• Email: arunkatherashala@gmail.com
• GitHub: @arunkatherashala
• LinkedIn: Sai Arun Kumar

1. Read: PYTHON_USER_GUIDE.md or DOCKER_EMULATORS_GUIDE.md
2. Install: pip install kore-fileformat
3. Explore: Check DOCUMENTATION_INDEX.md for your role

1. Review: V1_1_ROADMAP.md for v1.1.0 features
2. Clone: git clone https://github.com/arunkatherashala/Kore.git
3. Setup: Follow DOCKER_EMULATORS_GUIDE.md
4. Code: Create feature branch and submit PR

1. Setup: CI_CD_SECRETS_SETUP.md for automated publishing
2. Monitor: GitHub Actions workflows on each push
3. Release: Tag v1.0.1 or v1.1.0 to trigger publishing

Thank you for choosing Kore! We're excited to see what you build.

Questions? Open an issue or discussion on GitHub.
Want to help? Check V1_1_ROADMAP.md for features to implement.
Found a bug? Report it on GitHub Issues.

Latest Release: v1.0.0
Last Updated: May 14, 2026
Status: Production Ready ✅

🚀 Let's build amazing data infrastructure together!