Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Create more detailed error responses when submitting batch jobs (#2940)
* Enables airflow operator level retry. (#2894) * Update docker stuff for latest airflow 2.7.0 * Use AirflowException instead of AirflowFailException to allow for retries Signed-off-by: Rich Scott <richscott@sent.com> * Remove codecov workflows (#2902) Signed-off-by: Rich Scott <richscott@sent.com> * Upgrade Pulsar Client to v0.11 (#2896) * update * update pulsar client * Fix bug causing server spinning * Abstract out the retry until success logic for testing (#2901) * Respond to review --------- Co-authored-by: Chris Martin <chris@cmartinit.co.uk> Co-authored-by: Daniel Rastelli <rastellidani@gmail.com> Signed-off-by: Rich Scott <richscott@sent.com> * Sync quickstart/index.md with gh-pages/quickstart.md (#2891) Signed-off-by: Rich Scott <richscott@sent.com> * Log Call Site (#2909) * allow logger to report caller * allow logger to report caller * lint --------- Co-authored-by: Chris Martin <chris@cmartinit.co.uk> Signed-off-by: Rich Scott <richscott@sent.com> * Add cleaner test output for mage with os/exec.Command (#2907) Signed-off-by: Rich Scott <richscott@sent.com> * feat: Update Semver from version 6.3.0 to 6.3.1 (#2686) Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * fix: upgrade @typescript-eslint/parser from 5.52.0 to 5.61.0 (#2743) Snyk has created this PR to upgrade @typescript-eslint/parser from 5.52.0 to 5.61.0. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * fix: upgrade @types/react from 16.14.32 to 16.14.43 (#2747) Snyk has created this PR to upgrade @types/react from 16.14.32 to 16.14.43. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Bump github.com/go-openapi/jsonreference from 0.20.0 to 0.20.2 (#2316) Bumps [github.com/go-openapi/jsonreference](https://github.com/go-openapi/jsonreference) from 0.20.0 to 0.20.2. - [Release notes](https://github.com/go-openapi/jsonreference/releases) - [Commits](go-openapi/jsonreference@v0.20.0...v0.20.2) --- updated-dependencies: - dependency-name: github.com/go-openapi/jsonreference dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Order leased jobs by serial (#2912) This will ensure the job leased first, gets send to the cluster first Currently we just order by postgres default sorting - which often picks the most recently leased - causing the first lease jobs to get stuck - This only occurs when scheduling is faster than leasing Signed-off-by: Rich Scott <richscott@sent.com> * Bump webpack from 5.75.0 to 5.77.0 in /internal/lookout/ui (#2302) Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.77.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.75.0...v5.77.0) --- updated-dependencies: - dependency-name: webpack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Bump word-wrap from 1.2.3 to 1.2.5 in /internal/lookout/ui (#2806) Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.5. - [Release notes](https://github.com/jonschlinkert/word-wrap/releases) - [Commits](jonschlinkert/word-wrap@1.2.3...1.2.5) --- updated-dependencies: - dependency-name: word-wrap dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * resolve flaky (#2914) Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * fix: upgrade @typescript-eslint/eslint-plugin from 5.52.0 to 5.61.0 (#2744) Snyk has created this PR to upgrade @typescript-eslint/eslint-plugin from 5.52.0 to 5.61.0. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * fix: upgrade react-router-dom from 6.9.0 to 6.14.1 (#2746) Snyk has created this PR to upgrade react-router-dom from 6.9.0 to 6.14.1. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Bump semver from 6.3.0 to 6.3.1 in /internal/lookout/ui (#2661) Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md) - [Commits](npm/node-semver@v6.3.0...v6.3.1) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Run CodeQL once daily on a schedule (#2918) Signed-off-by: Rich Scott <richscott@sent.com> * Helm chart update: executor (#2917) * Helm chart update: executor At the moment the helm chart for the executor doesn't include priorityClass even though one is created in the chart. This means that the executor deployment is unable to set the priorityClass. Signed-off-by: Rich Scott <richscott@sent.com> * Patch/dependencies (#2923) * Bump github.com/go-openapi/strfmt from 0.21.3 to 0.21.7 Bumps [github.com/go-openapi/strfmt](https://github.com/go-openapi/strfmt) from 0.21.3 to 0.21.7. - [Release notes](https://github.com/go-openapi/strfmt/releases) - [Commits](go-openapi/strfmt@v0.21.3...v0.21.7) --- updated-dependencies: - dependency-name: github.com/go-openapi/strfmt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump github.com/go-openapi/runtime from 0.24.2 to 0.26.0 Bumps [github.com/go-openapi/runtime](https://github.com/go-openapi/runtime) from 0.24.2 to 0.26.0. - [Release notes](https://github.com/go-openapi/runtime/releases) - [Commits](go-openapi/runtime@v0.24.2...v0.26.0) --- updated-dependencies: - dependency-name: github.com/go-openapi/runtime dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump github.com/goreleaser/nfpm/v2 from 2.25.1 to 2.29.0 Bumps [github.com/goreleaser/nfpm/v2](https://github.com/goreleaser/nfpm) from 2.25.1 to 2.29.0. - [Release notes](https://github.com/goreleaser/nfpm/releases) - [Changelog](https://github.com/goreleaser/nfpm/blob/main/.goreleaser.yml) - [Commits](goreleaser/nfpm@v2.25.1...v2.29.0) --- updated-dependencies: - dependency-name: github.com/goreleaser/nfpm/v2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump github.com/go-playground/validator/v10 from 10.11.1 to 10.14.1 Bumps [github.com/go-playground/validator/v10](https://github.com/go-playground/validator) from 10.11.1 to 10.14.1. - [Release notes](https://github.com/go-playground/validator/releases) - [Commits](go-playground/validator@v10.11.1...v10.14.1) --- updated-dependencies: - dependency-name: github.com/go-playground/validator/v10 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump Grpc.Net.Client in /client/DotNet/ArmadaProject.Io.Client Bumps [Grpc.Net.Client](https://github.com/grpc/grpc-dotnet) from 2.47.0 to 2.52.0. - [Release notes](https://github.com/grpc/grpc-dotnet/releases) - [Changelog](https://github.com/grpc/grpc-dotnet/blob/master/doc/release_process.md) - [Commits](grpc/grpc-dotnet@v2.47.0...v2.52.0) --- updated-dependencies: - dependency-name: Grpc.Net.Client dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * fix: upgrade @mui/material from 5.10.17 to 5.13.6 Snyk has created this PR to upgrade @mui/material from 5.10.17 to 5.13.6. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr * fix: upgrade prettier from 2.7.1 to 2.8.8 Snyk has created this PR to upgrade prettier from 2.7.1 to 2.8.8. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr * fix: upgrade @mui/icons-material from 5.10.16 to 5.14.3 Snyk has created this PR to upgrade @mui/icons-material from 5.10.16 to 5.14.3. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr * fix: upgrade eslint-plugin-import from 2.26.0 to 2.28.0 Snyk has created this PR to upgrade eslint-plugin-import from 2.26.0 to 2.28.0. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr * fix: upgrade eslint-config-prettier from 8.5.0 to 8.10.0 Snyk has created this PR to upgrade eslint-config-prettier from 8.5.0 to 8.10.0. See this package in npm: See this project in Snyk: https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr * Trying to update klog * go mod fix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Fix bug causing GetJobSetEvents to get stuck (#2903) * Add error message of final job run to JobFailedMessage When we hit the maximum retry limit, the JobFailedMessage just says something along the lines of "Job has been retried too many times, giving up" Now we include the final run error in that message - to make it easier to work out the cause of retries * Fix bug causing GetJobSetEvents to get stuck GetJobSetEvents only increments its fromId variable on sending new messages However now all redis events produce api events that will be sent downstream The issue here is if we get 500 redis events in a row that don't produce api events, then the fromId never gets updated - Meaning the watching gets stuck here To fix this, ReadEvents now returns a lastMessageId. So if there are no messages to process, the fromId should be updated using the lastMessageId * Formatting Signed-off-by: Rich Scott <richscott@sent.com> * Bump @adobe/css-tools from 4.0.1 to 4.3.1 in /internal/lookout/ui (#2931) Bumps [@adobe/css-tools](https://github.com/adobe/css-tools) from 4.0.1 to 4.3.1. - [Changelog](https://github.com/adobe/css-tools/blob/main/History.md) - [Commits](https://github.com/adobe/css-tools/commits) --- updated-dependencies: - dependency-name: "@adobe/css-tools" dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Rich Scott <richscott@sent.com> * Update CreateJobs Signed-off-by: Rich Scott <richscott@sent.com> * Update CreateJobs to return a SubmitResponse with error details Signed-off-by: Rich Scott <richscott@sent.com> * Update createJobs sub functions to check jobs individually * Update function usage to count errored jobs Signed-off-by: Rich Scott <richscott@sent.com> * Fix grammar Signed-off-by: Rich Scott <richscott@sent.com> * Added updated test cases Signed-off-by: Rich Scott <richscott@sent.com> * Update gang job testing Signed-off-by: Rich Scott <richscott@sent.com> * Merge branch 'master' into feat/create_job_error Signed-off-by: Rich Scott <richscott@sent.com> * Lint fix Signed-off-by: Rich Scott <richscott@sent.com> * Rework gRPC to send JobSubmitResponse over status.details * Add better nil checking Signed-off-by: Rich Scott <richscott@sent.com> * Typo == instead of != Signed-off-by: Rich Scott <richscott@sent.com> * Wrap gRPC SubmitJob function Signed-off-by: Rich Scott <richscott@sent.com> * Create new client function instead of sharing Signed-off-by: Rich Scott <richscott@sent.com> * Change import order Signed-off-by: Rich Scott <richscott@sent.com> * Add a space between imports Signed-off-by: Rich Scott <richscott@sent.com> * Avoid nil pointer deference Signed-off-by: Rich Scott <richscott@sent.com> * Improved etcd protection (#2925) * Initial commit * Delete unused code * Export metrics collection delay metrics * Add mutex to InMemoryJobRepository * Add tests * Lint * Update internal/executor/configuration/types.go * Lint --------- Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com> Signed-off-by: Rich Scott <richscott@sent.com> * Stop executor requesting more jobs when it still has leased jobs (#2932) * Stop executor requesting more jobs when it still has leased jobs Currently we "queue" jobs to be submitted on the executor - which sit the leased state until they are submitted to kubernetes However this causes 2 issues with our current setup: - It prevents back-pressure from working well on the scheduler side. As it sees all these "Leased" jobs as active, so just keep scheduling more - In the case we are slowing submission due to etcd going over its limit. We "queue" lots of jobs, and as soon as etcd goes under its limit we hit it with potentially thousands of jobs This flow needs further work and thought - however for now this is the minimal fix to prevent bad behaviour Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * WIP Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Fix scheduler side tests Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Implement number of requested jobs on executor side Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Remove unused config Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Fixing panic on startup when etcd health monitor not registered Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Enhance logging Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> * Set more sensible default for maxLeasedJobs Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> --------- Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> Signed-off-by: Rich Scott <richscott@sent.com> * Fix race in etcd protections (#2937) * Initial commit * Fix MultiHealthMonitor race Signed-off-by: Rich Scott <richscott@sent.com> * Fix etcd health metric naming conflict (#2939) * Fix metric naming conflict * Fix metric names * Fix metrix prefix * Fix label Signed-off-by: Rich Scott <richscott@sent.com> * lint fix Signed-off-by: Rich Scott <richscott@sent.com> * Return clearer errors for multiple-jobs validation. Signed-off-by: Rich Scott <richscott@sent.com> * Return more detailed submission/validation errors. Generate and return more detailed submission and/or validation errors. If there are numerous jobs with errors, just give the number of failed jobs (and the total number originally submitted), and truncate the list of failed jobs errors to just the first 5 (this is defined in a single constant variable, if neededed to change later). Signed-off-by: Rich Scott <richscott@sent.com> --------- Signed-off-by: Rich Scott <richscott@sent.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com> Co-authored-by: Clif Houck <me@clifhouck.com> Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com> Co-authored-by: Chris Martin <council_tax@hotmail.com> Co-authored-by: Chris Martin <chris@cmartinit.co.uk> Co-authored-by: Daniel Rastelli <rastellidani@gmail.com> Co-authored-by: Kanu Mike Chibundu <michotall95@gmail.com> Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com> Co-authored-by: Dave Gantenbein <dave@gr-oss.io> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com> Co-authored-by: Sarthak Negi <122533767+sarthaksarthak9@users.noreply.github.com> Co-authored-by: owenthomas17 <owen@owen-thomas.co.uk> Co-authored-by: Raajheer1 <raaj.patel229@gmail.com> Co-authored-by: Raaj Patel <41763998+Raajheer1@users.noreply.github.com> Co-authored-by: Albin Severinson <albin@severinson.org>
- Loading branch information
17 people
committed
Dec 12, 2023
1 parent
d255ffc
commit 356a5f0