fix : ZstdError handling for Python <3.14 & >=3.14 compatibility#767
Conversation
- Import ZstdError from compression.zstd for Python >= 3.14 - Import ZstdError from zstd for older Python versions - Add test to verify ZstdError is properly caught for corrupted data Fixes Lightning-AI#766
bhimrazy
requested review from
justusschock,
lantiga and
tchaton
as code owners
There was a problem hiding this comment.
Pull request overview
This PR fixes ZstdError import handling to ensure compatibility across Python versions, specifically addressing the breaking change where Python 3.14+ includes zstd as a built-in module under compression.zstd while older versions use the third-party zstd package. The fix properly handles the exception class name differences (ZstdError in Python 3.14+ vs Error in older versions) to prevent import errors and ensure corrupted data is properly caught.
Key Changes:
- Updated exception handling in
_load_index_fileand_save_index_fileto import and use the correct ZstdError class based on Python version - Added test coverage to verify ZstdError is properly caught when decompressing corrupted data
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/litdata/raw/indexer.py |
Added version-specific imports for ZstdError and updated exception handlers to use the imported exception class instead of zstd.ZstdError |
tests/raw/test_indexer.py |
Added test to verify corrupted zstd data raises ZstdError and is properly handled by the indexer |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #767 +/- ##
===================================
Coverage 80% 80%
===================================
Files 52 52
Lines 7343 7347 +4
===================================
+ Hits 5883 5890 +7
+ Misses 1460 1457 -3 🚀 New features to boost your workflow:
|
bhimrazy
changed the title
ZstdError handling for Python <3.14 & >=3.14 compatibility
What does this PR do?
Fixes #766
Changes
PR review
Anyone in the community is free to review the PR once the tests have passed.
Did you have fun?
Make sure you had fun coding ��