Support ndjson data files

[albertvillanova]

Support ndjson (Newline Delimited JSON) data files.

Fix #7153.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[severo]

Good idea to support it.

But note that it's a dead project; its author recommends using jsonlines: ndjson/ndjson-spec#35 (comment). Maybe something to add as a comment in the code

[albertvillanova]

Thanks for your review, @severo.

Yes, I was aware of this. From internal conversation:

Please note that although NDJSON was planned to be submitted as an RFC standard spec, it is no longer maintained:

• See note from the author: jsonlines ndjson/ndjson-spec#35 (comment)
• See that their official website domain has expired: https://ndjson.org/

The purpose of this PR is just supporting datasets with ndjson data files (e.g. Wikimedia Enterprise data files), but it should not imply any recommendation or endorsement of this format from our part.