feat: add support for pyarrow arrays as input

[notjedi]

closes #1415

[notjedi]

yet to add support for pyarrow.LargeString

[notjedi]

pulling arrow from git because the current stable version v51.0 links against v0.20 of pyo3 while the bindings link against v0.21. there is already a merged pull request(apache/arrow-rs#5566) which migrates from v0.20 to v0.21 of pyo3 which i think will be included in the next release of v52.0.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

I think we can leave the dep as optional wdyt?

[notjedi]

I think we can leave the dep as optional wdyt?

makes sense, will get it changed @ArthurZucker

[notjedi]

i've added pyarrow as a feature in pyproject.toml. not sure if that is what we want here. let me know if i need to fix anything else. @ArthurZucker

[ArthurZucker]

I think we want to avoid having to use git for the repo 😢
ALso do you have any bench / examples of it being use and fast? 🚀

[notjedi]

yeah will update that once there is a stable release of arrow (v52.0). not really, i don't have any benchmarks and the only example i have is in the tests. i don't need this personally, just wanted to close an open issue and contribute.

[strategy155]

Good afternoon everyone. How do you think the optimal benchmark should look like? Native pyarrow vs numpy conversion for string arrays?

[ArthurZucker]

something like that yeah, I have no idea in what context pyarrow is used as I have not used it, but in optimal context usage