less script docs

[lhoestq]

• mark as legacy in some parts of the docs since we'll not build new features for script datasets

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

Thanks.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005810 / 0.011353 (-0.005543)	0.003984 / 0.011008 (-0.007024)	0.064347 / 0.038508 (0.025839)	0.031943 / 0.023109 (0.008834)	0.252596 / 0.275898 (-0.023302)	0.274032 / 0.323480 (-0.049448)	0.003494 / 0.007986 (-0.004492)	0.002817 / 0.004328 (-0.001511)	0.050132 / 0.004250 (0.045881)	0.048008 / 0.037052 (0.010955)	0.249037 / 0.258489 (-0.009452)	0.288526 / 0.293841 (-0.005315)	0.031038 / 0.128546 (-0.097509)	0.012542 / 0.075646 (-0.063104)	0.205682 / 0.419271 (-0.213590)	0.038022 / 0.043533 (-0.005511)	0.259001 / 0.255139 (0.003862)	0.267455 / 0.283200 (-0.015744)	0.021980 / 0.141683 (-0.119703)	1.123996 / 1.452155 (-0.328159)	1.173801 / 1.492716 (-0.318915)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.102827 / 0.018006 (0.084821)	0.317210 / 0.000490 (0.316720)	0.000222 / 0.000200 (0.000022)	0.000052 / 0.000054 (-0.000002)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019483 / 0.037411 (-0.017928)	0.064098 / 0.014526 (0.049572)	0.076219 / 0.176557 (-0.100337)	0.122898 / 0.737135 (-0.614237)	0.080657 / 0.296338 (-0.215681)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.278378 / 0.215209 (0.063169)	2.792314 / 2.077655 (0.714659)	1.516439 / 1.504120 (0.012319)	1.374052 / 1.541195 (-0.167143)	1.370848 / 1.468490 (-0.097642)	0.756002 / 4.584777 (-3.828775)	2.349581 / 3.745712 (-1.396131)	2.994094 / 5.269862 (-2.275768)	1.904242 / 4.565676 (-2.661435)	0.078769 / 0.424275 (-0.345506)	0.005103 / 0.007607 (-0.002505)	0.336331 / 0.226044 (0.110287)	3.329502 / 2.268929 (1.060574)	1.863545 / 55.444624 (-53.581079)	1.554690 / 6.876477 (-5.321787)	1.588448 / 2.142072 (-0.553624)	0.787322 / 4.805227 (-4.017905)	0.138345 / 6.500664 (-6.362320)	0.042228 / 0.075469 (-0.033241)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.968607 / 1.841788 (-0.873181)	11.972076 / 8.074308 (3.897768)	9.927608 / 10.191392 (-0.263784)	0.141666 / 0.680424 (-0.538758)	0.014591 / 0.534201 (-0.519610)	0.301995 / 0.579283 (-0.277288)	0.274360 / 0.434364 (-0.160004)	0.338396 / 0.540337 (-0.201941)	0.431081 / 1.386936 (-0.955855)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006122 / 0.011353 (-0.005231)	0.004201 / 0.011008 (-0.006807)	0.050204 / 0.038508 (0.011695)	0.033222 / 0.023109 (0.010113)	0.274357 / 0.275898 (-0.001542)	0.296238 / 0.323480 (-0.027242)	0.004542 / 0.007986 (-0.003444)	0.002880 / 0.004328 (-0.001449)	0.049103 / 0.004250 (0.044852)	0.042294 / 0.037052 (0.005242)	0.286459 / 0.258489 (0.027970)	0.324988 / 0.293841 (0.031147)	0.032084 / 0.128546 (-0.096462)	0.012329 / 0.075646 (-0.063318)	0.060261 / 0.419271 (-0.359010)	0.034130 / 0.043533 (-0.009403)	0.271432 / 0.255139 (0.016293)	0.306251 / 0.283200 (0.023051)	0.019744 / 0.141683 (-0.121939)	1.153483 / 1.452155 (-0.298672)	1.209126 / 1.492716 (-0.283591)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.004737 / 0.018006 (-0.013270)	0.313458 / 0.000490 (0.312968)	0.000216 / 0.000200 (0.000017)	0.000053 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022472 / 0.037411 (-0.014939)	0.076725 / 0.014526 (0.062199)	0.091356 / 0.176557 (-0.085201)	0.132427 / 0.737135 (-0.604708)	0.091072 / 0.296338 (-0.205266)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.294414 / 0.215209 (0.079205)	2.913695 / 2.077655 (0.836040)	1.567309 / 1.504120 (0.063189)	1.448664 / 1.541195 (-0.092531)	1.466386 / 1.468490 (-0.002105)	0.718605 / 4.584777 (-3.866172)	0.951963 / 3.745712 (-2.793749)	2.812565 / 5.269862 (-2.457297)	1.886483 / 4.565676 (-2.679193)	0.077912 / 0.424275 (-0.346363)	0.005371 / 0.007607 (-0.002236)	0.349528 / 0.226044 (0.123484)	3.431049 / 2.268929 (1.162121)	1.920210 / 55.444624 (-53.524414)	1.637927 / 6.876477 (-5.238549)	1.767502 / 2.142072 (-0.374570)	0.808672 / 4.805227 (-3.996555)	0.134261 / 6.500664 (-6.366403)	0.041295 / 0.075469 (-0.034174)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.023454 / 1.841788 (-0.818334)	12.433731 / 8.074308 (4.359423)	10.413191 / 10.191392 (0.221799)	0.156813 / 0.680424 (-0.523611)	0.015446 / 0.534201 (-0.518755)	0.301935 / 0.579283 (-0.277348)	0.133655 / 0.434364 (-0.300709)	0.340296 / 0.540337 (-0.200041)	0.466314 / 1.386936 (-0.920622)

[mayorblock]

Hi @lhoestq,

I was confused by legacy prefix added to the image data loading script section. I have a custom image dataset and have looked through the documentation to find something similar but can't find a good alternative What is now the recommend way to create a custom image dataset then? I want the HF format but will not host it on the hub.

Apologies in advance if this is the wrong place to ask such questions...

[lhoestq]

We stopped making new features for datasets with scripts for obvious security reasons, that's why they are marked as "legacy". What is blocking you from hosting on HF ?

[mayorblock]

Hi, thanks for the prompt answer :) I am working on proprietary datasets for the company where I am employed. We want to keep the data in-house but would like to investigate the use of the HF ecosystem.

[lhoestq]

I see ! Note that it's possible to have private repos on HF (+ dataset viewer) and you can even choose the storage region, if it can help