Add lightweight PDB (Protein Data Bank) file support#7926
Open
behroozazarkhalili wants to merge 4 commits intohuggingface:mainfrom
Open
Add lightweight PDB (Protein Data Bank) file support#7926behroozazarkhalili wants to merge 4 commits intohuggingface:mainfrom
behroozazarkhalili wants to merge 4 commits intohuggingface:mainfrom
Conversation
- Add zero-dependency pure Python parser for PDB format - Support ATOM and HETATM record types with configurable filtering - Handle fixed-width column parsing per official PDB specification - Support gzip, bzip2, and xz compression via magic bytes detection - Support .pdb and .ent file extensions - Add comprehensive test suite with 24 tests - Add documentation to loading.mdx Columns include: atom_serial, atom_name, residue_name, chain_id, residue_seq, x, y, z, occupancy, temp_factor, element, and more. Part of the bioinformatics file format support series.
Based on reviewer feedback from @lhoestq, refactor the PDB loader to follow the ImageFolder pattern where each row contains one complete protein structure file, rather than one row per atom. Changes: - Add ProteinStructure feature type for lazy loading structure files - Refactor PdbFolder to use FolderBasedBuilder with folder/metadata support - Support automatic label inference from directory names - Support metadata files (CSV/JSONL) for additional annotations - Simplify from ~400 lines atom-parser to ~55 lines folder-based builder This approach matches how researchers typically work with protein structures: - One structure = one data point - Supports downstream tools (BioPython, PyMOL, MDAnalysis) - Consistent with ImageFolder/AudioFolder/VideoFolder patterns Test: 43 tests passing (15 feature + 28 builder tests)
Fix bug in FolderBasedBuilder._generate_examples where drop_metadata=True would fail with IndexError when metadata files were included in the files list but skipped due to extension filtering. Root cause: enumerate(files) created gaps in shard_idx when files were skipped, causing builder.py to fail when indexing original_shard_lengths. Solution: Use separate valid_shard_idx counter that only increments when samples are actually yielded, ensuring contiguous shard IDs.
- Sort imports alphabetically in features.py - Fix line length in protein_structure.py error messages - Format function call in test_pdb.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for loading PDB (Protein Data Bank) files with
load_dataset(), following the ImageFolder pattern where one row = one structure.Based on feedback from @lhoestq in #7930, this approach makes datasets more practical for ML workflows:
Architecture
Uses
FolderBasedBuilderpattern (likeImageFolder,AudioFolder):New
ProteinStructureFeature TypeSupported Extensions
.pdb,.entUsage
Test Results
All 28 PDB tests + 15 ProteinStructure feature tests pass.
Related PRs
cc @lhoestq @georgia-hf