Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race in sparkey write #4937

Merged
merged 11 commits into from
Jul 28, 2023
Merged

Fix race in sparkey write #4937

merged 11 commits into from
Jul 28, 2023

Conversation

kellen
Copy link
Contributor

@kellen kellen commented Jul 26, 2023

Intended to fix #4890

Introduces a write to a temporary location before a finalization step moving temp files to their final destination.

Unifies the various SparkeyUri subclasses for simpler handling, removes kryo coder, moves provision of RemoteFileUtil to transform steps.

@kellen kellen changed the base branch from v0.14.x to main July 26, 2023 01:13
@codecov
Copy link

codecov bot commented Jul 26, 2023

Codecov Report

Merging #4937 (2116c3c) into v0.14.x (308f332) will increase coverage by 0.37%.
Report is 7 commits behind head on v0.14.x.
The diff coverage is 90.00%.

❗ Current head 2116c3c differs from pull request most recent head 1751cb4. Consider uploading reports for the commit 1751cb4 to get more accurate results

@@             Coverage Diff             @@
##           v0.14.x    #4937      +/-   ##
===========================================
+ Coverage    62.98%   63.36%   +0.37%     
===========================================
  Files          282      284       +2     
  Lines        10648    10698      +50     
  Branches       789      781       -8     
===========================================
+ Hits          6707     6779      +72     
+ Misses        3941     3919      -22     
Files Changed Coverage Δ
...src/main/scala/com/spotify/scio/avro/package.scala 100.00% <ø> (ø)
...fy/scio/avro/schemas/instances/AvroInstances.scala 0.00% <ø> (ø)
.../scala/com/spotify/scio/coders/AvroBytesUtil.scala 100.00% <ø> (ø)
.../com/spotify/scio/protobuf/util/ProtobufUtil.scala 100.00% <ø> (ø)
...src/main/scala/com/spotify/scio/coders/Coder.scala 93.47% <ø> (ø)
...ify/scio/io/dynamic/syntax/SCollectionSyntax.scala 88.88% <ø> (-1.50%) ⬇️
...c/main/scala/com/spotify/scio/schemas/Schema.scala 68.18% <ø> (ø)
.../main/scala/com/spotify/scio/pubsub/PubsubIO.scala 15.11% <0.00%> (ø)
...y/scio/extra/sparkey/instances/SparkeyWriter.scala 58.33% <58.33%> (ø)
...ro/src/main/scala/com/spotify/scio/avro/taps.scala 31.03% <75.00%> (ø)
... and 12 more

... and 2 files with indirect coverage changes

@kellen kellen marked this pull request as ready for review July 26, 2023 16:25
Copy link
Contributor

@RustedBones RustedBones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this target 0.14 as this is breaking mima ?

@kellen kellen changed the base branch from main to v0.14.x July 27, 2023 15:30
@kellen kellen merged commit f55831b into v0.14.x Jul 28, 2023
@kellen kellen deleted the kellen/sparkeyrace branch July 28, 2023 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Race in sparkey write
3 participants