Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix small bugs with async map #7445

Merged
merged 5 commits into from
Mar 13, 2025
Merged

Fix small bugs with async map #7445

merged 5 commits into from
Mar 13, 2025

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Mar 11, 2025

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

  • fix with_indices
  • fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
  • remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()

lhoestq added 2 commits March 10, 2025 19:06
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq added 3 commits March 11, 2025 19:44
@lhoestq lhoestq merged commit f09db01 into main Mar 13, 2025
12 of 15 checks passed
@lhoestq lhoestq deleted the fix-async-map-resuming branch March 13, 2025 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants