Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fall back to TableRow representation for typed BQ writes containing a LocalDateTime field #5623

Merged
merged 2 commits into from
Mar 4, 2025

Conversation

clairemcginty
Copy link
Contributor

typed FILE_LOADS writes containing a LocalDateTime field fail with this error:

  "status": {
    "errorResult": {
      "location": "query",
      "message": "Field month has incompatible types. Configured schema: datetime; Avro file: string.",
      "reason": "invalidQuery"
    },
    "errors": [
      {
        "location": "query",
        "message": "Field month has incompatible types. Configured schema: datetime; Avro file: string.",
        "reason": "invalidQuery"
      }
    ],
    "state": "DONE"
  },

as it turns out, Beam's TableRow-to-Avro converter writes LocalDateTimes as a datetime string, which is supported by the Storage Writes API. Unfortunately, FILE_LOADS expects a micro timestamp as a Long.

… LocalDateTime
@clairemcginty clairemcginty marked this pull request as ready for review March 4, 2025 13:51
Copy link
Contributor

@jto jto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Hopefully this gets fixed in Beam and we can just remove the workaround.

Copy link

codecov bot commented Mar 4, 2025

Codecov Report

Attention: Patch coverage is 30.43478% with 16 lines in your changes missing coverage. Please review.

Project coverage is 61.30%. Comparing base (93a20de) to head (71db71c).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...otify/scio/bigquery/syntax/SCollectionSyntax.scala 26.31% 14 Missing ⚠️
...-core/src/main/scala/com/spotify/scio/io/Tap.scala 0.00% 1 Missing ⚠️
...scala/com/spotify/scio/bigquery/BigQueryUtil.scala 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5623      +/-   ##
==========================================
- Coverage   61.33%   61.30%   -0.04%     
==========================================
  Files         314      314              
  Lines       11257    11274      +17     
  Branches      782      814      +32     
==========================================
+ Hits         6905     6912       +7     
- Misses       4352     4362      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@clairemcginty clairemcginty merged commit d0a572d into main Mar 4, 2025
12 checks passed
@clairemcginty clairemcginty deleted the fallback-datetime-types branch March 4, 2025 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants