data to long when using s3 to gcs

[Lee2532]

Description

If a large amount of s3 data is moved using s3 to gcs, data too long for column error occurs when storing related values in the last xcom

Use case/motivation

I think we can prevent this error by adding a logic that compresses the list to show only some of the data

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[raphaelauv]

XCOM is not for data , it's for metadata ( xcom backend is just a work around , in all case Airlfow should not see your data )

Airflow is a scheduling tool not an ETL tool

Airlfow operators are most of them simple python helpers not data-transfert efficient tools