BREAKING CHANGE - Enhance Python bindings plus streaming decryption support, improve chunk handling, add comprehensive tests

[dirvine]

🚨 Breaking Changes

This PR includes several commits that packages each change in a more reviewable chunk of changes. Please go commit by commit to follow this one. The large issues are clean up of API, removing unused and confusing parts, remove old streaming functionality and introduce a newer and small decrypt streaming facility with the ability to retrieve chunks is parallel. There may Abe further small tweaks to confirm data.

Python API Changes:
Removed 'py_' prefix from Python binding function names for a cleaner API.
Changed Python module structure to use _self_encryption as the internal module name.

✨ Features

Python Binding Enhancements
Comprehensive Bindings:
Implemented Python bindings for all core functionalities.
Restructured Python package to follow best practices.
Fixed naming conflicts between Rust and Python modules.
Improved package structure and dependency management.
Streaming Decryption Support:
Added streaming decryption functionality with parallel chunk retrieval.
Implemented streaming decryption in Python bindings.
Added examples and documentation for streaming operations.
Type Hints and Documentation:
Added proper type hints and docstrings to all Python functions and classes.
Improved function documentation and type annotations.
Updated README with comprehensive Python usage examples.
Chunk Handling Improvements
Enhanced encrypt_from_file:
Modified to store all chunks, including parent chunks.
Improved Decryption Functions:
Updated decrypt functions to handle parent chunks correctly using efficient HashMap lookups.
Fixed decrypt_full_set to use efficient chunk retrieval.
DataMap Enhancements:
Improved DataMap functionality with better child handling.
Added serialization/deserialization functions to the public API.

🧪 Testing Enhancements

Comprehensive Test Suite:
Added tests covering all combinations of encryption and decryption methods:
In-memory encryption → All decryption methods.
File-based encryption → All decryption methods.
Streaming decryption with parallel chunk retrieval.
Added tests for large file handling and data map shrinking.
Verified cross-platform compatibility.
Fixed Test Failures:
Resolved issues in StreamSelfDecryptor tests.
Ensured decrypted content is available at the expected output path.
Improved Error Handling in Tests:
Added detailed debug output for better diagnostics.
Enhanced error handling tests to match actual behavior.

🔨 Refactoring

Stream Module:
Moved stream-related methods to a separate src/stream.rs file.
Extracted stream structs and implementations for better readability.
Utility Functions:
Moved non-public utility functions from lib.rs to utils.rs.
Added pub(crate) visibility to utility functions.
API Simplification:
Removed unused seek_info related code to simplify the public API.
Renamed and restricted visibility of internal functions for proper encapsulation.
Renamed decrypt to decrypt_sorted_set.
Changed visibility from pub to pub(crate).

🗃️ Build System

Python Packaging:
Updated pyproject.toml for better package management.
Fixed maturin configuration for proper module naming.
Added proper dependencies in setup.py.

📚 Documentation

User Guides and Examples:
Updated README with comprehensive Python usage examples.
Added detailed API documentation for Python bindings.
Developer Documentation:
Improved implementation details section.
Added examples for streaming operations.

🛠️ General Improvements

Error Handling:
Improved error messages and handling across the codebase.
Used std::io::Error with appropriate ErrorKind in StreamSelfDecryptor.
Code Cleanup:
Resolved linter warnings and performed code cleanup.
Ensured consistent use of file paths and error handling.
Organized code logically and added helpful comments.

📝 Impact

These changes provide a more robust, well-tested, and user-friendly interface for both Rust and Python users. The enhancements enable efficient handling of large files through streaming operations and improve the reliability of chunk handling. Refactoring efforts have resulted in cleaner code, better maintainability, and enforced proper encapsulation, reducing maintenance burden and simplifying the API surface.

✅ Checklist

[x] Added tests that cover all major use cases and edge conditions.
[x] Ensured backward compatibility where possible.
[x] Updated documentation accordingly.
[x] Verified cross-platform compatibility.

Short description

Enhance Python bindings with streaming decryption support, improve chunk handling, add comprehensive tests, and refactor codebase for better maintainability.
Footnote
Please review the breaking changes and updated APIs before integrating this PR.