Enable Compartment Joins for Unified Parquet Output

[d33bs]

Description

The changes in this PR seek to enable compartment data joins as a single parquet file. Along the way, many other related changes were added. Thank you in advance for your comments and suggestions!

Outline of changes:

• A focus was placed on CellProfiler CSV and SQLite data source conversion (seeking to cover Convert CellProfiler CSV files to parquet #4 , Convert CellProfiler SQLite files to parquet #5)
• Presets were added for common configuration patterns (with a focus, for now, on CellProfiler source data)
• Testing datasets have been directly added to this repo for the purposes of validating results under tests/data.
• Initial testing for cytominer-database is included with related changes (beginning to address Add or verify cytominer-database SQLite data source input compatibility #18 )
• A CITATION.cff file has been added to help document testing dataset sources and other contributions.
• DuckDB was added as a dependency to assist with both SQLite data reads and SQL-based join work.
• Documentation covering technical and data architecture (in addition to other areas) were added to help share knowledge associated with the work in this repo and other areas. (seeks to cover parts of Document system and workflow architecture options #17)
• Prefect logging controls were added to minimize logging output with default settings. (Provide optional logging suppression control #21)
• Some modules were reorganized for consistency (for example, exceptions.py) or increasing readability through line-length reduction (record-based tasks moved from convert.py to records.py) (abiding pylint too-many-lines / C0302).

Design considerations:

SQL-based DuckDB joins:

Initial work towards this PR included PyArrow.Table.join() and related configuration presets for controlling join behavior with a Python dictionary (I'll reference this as a "join dictionary"). The data structure of this join dictionary mimicked that of SQL-based options (for ex LEFT JOIN table ON x=y as {"left":"table","join_keys":["x"]}, etc). Given this, these points urged me to change the code before this PR.

• Using the join dictionary involved a complex for-loop to join the entirety of the data where otherwise SQL allows for multiple joins to be stated in a single statement.
• I couldn't justify reinventing SQL join syntax from a human understandability standpoint; using SQL I feel will help ensure the code is better understood and able to be maintained over time. It also offers a cross-language way to manage this data which doesn't rely on Pythonic syntax or data (should a move from Python to another be desired).
• Using DuckDB enabled for SQLite-based data ingest in addition to enabling the join work to take place (where earlier in these changes I had relied on connector-x for SQLite data ingest due to its high-performance capabilities).
• There may be additional benefits to using DuckDB and/or SQL in this way which have not yet been implemented (record batching, performance, etc).

Citation references:

I wanted to make sure we acknowledged the great work of those who generated datasets used for testing within this repo. Using a CITATION.cff file for this felt right but I was unsure of the best way to both link to existing CITATION.cff files and also intermix references to individuals not found within a CITATION.cff file. I'd welcome any thoughts on the best or most preferred way to do this so we make sure those involved are credited.

What is the nature of your change?

• Bug fix (fixes an issue).
• Enhancement (adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).
• This change requires a documentation update.

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

• I have read the CONTRIBUTING.md guidelines.
• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• New and existing unit tests pass locally with my changes.
• I have added tests that prove my fix is effective or that my feature works.
• I have deleted all non-relevant text in this pull request template.

[falquaddoomi]

Nice work! I made a few comments. Generally, I'd say that if you end up excluding some functions from the public interface, it's ok to not document them from the perspective of an end user, but if they are intended to be used by end users I'd definitely include descriptions of what the end user should pass for parameters. For example, It'd be good to see a sample input dataset in the documentation, an identification of all the files in the dataset, and how you'd use the library to process it, e.g., what specific file paths in the dataset are given to which methods.

[d33bs]

Thank you again @gwaybio and @falquaddoomi for your reviews! I've addressed or created new issues for the comments you provided.