Skip to content

Commit

Permalink
Testsuite improvements (#3065)
Browse files Browse the repository at this point in the history
* Sync out testsuite changes (#19)

* Update simulator

* Replace Output with C

* Typo

* Restore pkg proto

* Restore files

* Fixing simulator changes (#6)

* Fixing simulator changes

* Changed to less than or equal

Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>

* Simulator Changes (#9)

* Add config and dependency injection to scheduler metrics (#2892)

* Replace metrics singleton with an injection pattern.

* fix

* add configuration structures to metrics

* add configuration

* rename elements

* Maker Pulsar ReceiverQueueSize Configurable (#2895)

* wip

* wip

* set receiverQueueSize to 100

* remove old PulsarReceiverQueueSize

* revert

* subscriptionin api

---------

Co-authored-by: Chris Martin <chris@cmartinit.co.uk>

* Add poll_interval (#2805)

* Add poll_interval

* Add poll_interval

* Added poll_interval

* update by running tox-e docs

---------

Co-authored-by: Kevin Hannon <kannon1992@gmail.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>

* Seperate python script for armada v1 and v2 system diagrams (#2758)

* Seperate python script for armada v1 system diagram

* removed generate.py so it can be replaced with two seperate files for Armada V1 and Armada V2

* Python script to generate Armada V2 system diagram

* generate_v1.py Update #1

* generate_v1.py Update Number:2

* generate.py runs generate_v1.py as well as generate_v2.py and it is consistent with our instructions as 'docs/design/diagrams/relationships'

* generate_v1.py Update No:3

* Armada V1 and Armada V2 diagrams

* updated relationships_diagram.md to include armada v1 and v2 diagrams

---------

Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>

* Add config to use autoupdater on tagged branches (#2905)

* #2904 add autoupdate config

* #2904 add label config and other options

* docs: create README.md for plugins directory (#2897)

* Create README.md for plugins directory

* Update README.md

* Update plugins/README.md

Co-authored-by: Kevin Hannon <kehannon@redhat.com>

* Update README.md

---------

Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>

* Enables airflow operator level retry. (#2894)

* Update docker stuff for latest airflow 2.7.0

* Use AirflowException instead of AirflowFailException to allow for retries

* Remove codecov workflows (#2902)

* Upgrade Pulsar Client to v0.11 (#2896)

* update

* update pulsar client

* Fix bug causing server spinning

* Abstract out the retry until success logic for testing (#2901)

* Respond to review

---------

Co-authored-by: Chris Martin <chris@cmartinit.co.uk>
Co-authored-by: Daniel Rastelli <rastellidani@gmail.com>

* Sync quickstart/index.md with gh-pages/quickstart.md (#2891)

* Log Call Site (#2909)

* allow logger to report caller

* allow logger to report caller

* lint

---------

Co-authored-by: Chris Martin <chris@cmartinit.co.uk>

* Add cleaner test output for mage with os/exec.Command (#2907)

* feat: Update Semver from version 6.3.0 to 6.3.1 (#2686)

Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>

* fix: upgrade @typescript-eslint/parser from 5.52.0 to 5.61.0 (#2743)

Snyk has created this PR to upgrade @typescript-eslint/parser from 5.52.0 to 5.61.0.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* fix: upgrade @types/react from 16.14.32 to 16.14.43 (#2747)

Snyk has created this PR to upgrade @types/react from 16.14.32 to 16.14.43.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Bump github.com/go-openapi/jsonreference from 0.20.0 to 0.20.2 (#2316)

Bumps [github.com/go-openapi/jsonreference](https://github.com/go-openapi/jsonreference) from 0.20.0 to 0.20.2.
- [Release notes](https://github.com/go-openapi/jsonreference/releases)
- [Commits](go-openapi/jsonreference@v0.20.0...v0.20.2)

---
updated-dependencies:
- dependency-name: github.com/go-openapi/jsonreference
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Order leased jobs by serial (#2912)

This will ensure the job leased first, gets send to the cluster first

Currently we just order by postgres default sorting - which often picks the most recently leased - causing the first lease jobs to get stuck
 - This only occurs when scheduling is faster than leasing

* Bump webpack from 5.75.0 to 5.77.0 in /internal/lookout/ui (#2302)

Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.77.0.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](webpack/webpack@v5.75.0...v5.77.0)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Bump word-wrap from 1.2.3 to 1.2.5 in /internal/lookout/ui (#2806)

Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.5.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.5)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* resolve flaky (#2914)

Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>

* fix: upgrade @typescript-eslint/eslint-plugin from 5.52.0 to 5.61.0 (#2744)

Snyk has created this PR to upgrade @typescript-eslint/eslint-plugin from 5.52.0 to 5.61.0.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* fix: upgrade react-router-dom from 6.9.0 to 6.14.1 (#2746)

Snyk has created this PR to upgrade react-router-dom from 6.9.0 to 6.14.1.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Bump semver from 6.3.0 to 6.3.1 in /internal/lookout/ui (#2661)

Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md)
- [Commits](npm/node-semver@v6.3.0...v6.3.1)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Run CodeQL once daily on a schedule (#2918)

* Helm chart update: executor  (#2917)

* Helm chart update: executor

At the moment the helm chart for the executor doesn't include priorityClass even though one is created in the chart. This means that the executor deployment is unable to set the priorityClass.

* Patch/dependencies (#2923)

* Bump github.com/go-openapi/strfmt from 0.21.3 to 0.21.7

Bumps [github.com/go-openapi/strfmt](https://github.com/go-openapi/strfmt) from 0.21.3 to 0.21.7.
- [Release notes](https://github.com/go-openapi/strfmt/releases)
- [Commits](go-openapi/strfmt@v0.21.3...v0.21.7)

---
updated-dependencies:
- dependency-name: github.com/go-openapi/strfmt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/go-openapi/runtime from 0.24.2 to 0.26.0

Bumps [github.com/go-openapi/runtime](https://github.com/go-openapi/runtime) from 0.24.2 to 0.26.0.
- [Release notes](https://github.com/go-openapi/runtime/releases)
- [Commits](go-openapi/runtime@v0.24.2...v0.26.0)

---
updated-dependencies:
- dependency-name: github.com/go-openapi/runtime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/goreleaser/nfpm/v2 from 2.25.1 to 2.29.0

Bumps [github.com/goreleaser/nfpm/v2](https://github.com/goreleaser/nfpm) from 2.25.1 to 2.29.0.
- [Release notes](https://github.com/goreleaser/nfpm/releases)
- [Changelog](https://github.com/goreleaser/nfpm/blob/main/.goreleaser.yml)
- [Commits](goreleaser/nfpm@v2.25.1...v2.29.0)

---
updated-dependencies:
- dependency-name: github.com/goreleaser/nfpm/v2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/go-playground/validator/v10 from 10.11.1 to 10.14.1

Bumps [github.com/go-playground/validator/v10](https://github.com/go-playground/validator) from 10.11.1 to 10.14.1.
- [Release notes](https://github.com/go-playground/validator/releases)
- [Commits](go-playground/validator@v10.11.1...v10.14.1)

---
updated-dependencies:
- dependency-name: github.com/go-playground/validator/v10
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump Grpc.Net.Client in /client/DotNet/ArmadaProject.Io.Client

Bumps [Grpc.Net.Client](https://github.com/grpc/grpc-dotnet) from 2.47.0 to 2.52.0.
- [Release notes](https://github.com/grpc/grpc-dotnet/releases)
- [Changelog](https://github.com/grpc/grpc-dotnet/blob/master/doc/release_process.md)
- [Commits](grpc/grpc-dotnet@v2.47.0...v2.52.0)

---
updated-dependencies:
- dependency-name: Grpc.Net.Client
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: upgrade @mui/material from 5.10.17 to 5.13.6

Snyk has created this PR to upgrade @mui/material from 5.10.17 to 5.13.6.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

* fix: upgrade prettier from 2.7.1 to 2.8.8

Snyk has created this PR to upgrade prettier from 2.7.1 to 2.8.8.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

* fix: upgrade @mui/icons-material from 5.10.16 to 5.14.3

Snyk has created this PR to upgrade @mui/icons-material from 5.10.16 to 5.14.3.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

* fix: upgrade eslint-plugin-import from 2.26.0 to 2.28.0

Snyk has created this PR to upgrade eslint-plugin-import from 2.26.0 to 2.28.0.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

* fix: upgrade eslint-config-prettier from 8.5.0 to 8.10.0

Snyk has created this PR to upgrade eslint-config-prettier from 8.5.0 to 8.10.0.

See this package in npm:


See this project in Snyk:
https://app.snyk.io/org/dave-gantenbein/project/5064983e-fa14-4803-8fc2-cfd6f1fa81b6?utm_source=github&utm_medium=referral&page=upgrade-pr

* Trying to update klog

* go mod fix

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Fix bug causing GetJobSetEvents to get stuck (#2903)

* Add error message of final job run to JobFailedMessage

When we hit the maximum retry limit, the JobFailedMessage just says something along the lines of
"Job has been retried too many times, giving up"

Now we include the final run error in that message - to make it easier to work out the cause of retries

* Fix bug causing GetJobSetEvents to get stuck

GetJobSetEvents only increments its fromId variable on sending new messages

However now all redis events produce api events that will be sent downstream

The issue here is if we get 500 redis events in a row that don't produce api events, then the fromId never gets updated
 - Meaning the watching gets stuck here

To fix this, ReadEvents now returns a lastMessageId. So if there are no messages to process, the fromId should be updated using the lastMessageId

* Formatting

* Bump @adobe/css-tools from 4.0.1 to 4.3.1 in /internal/lookout/ui (#2931)

Bumps [@adobe/css-tools](https://github.com/adobe/css-tools) from 4.0.1 to 4.3.1.
- [Changelog](https://github.com/adobe/css-tools/blob/main/History.md)
- [Commits](https://github.com/adobe/css-tools/commits)

---
updated-dependencies:
- dependency-name: "@adobe/css-tools"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Improved etcd protection (#2925)

* Initial commit

* Delete unused code

* Export metrics collection delay metrics

* Add mutex to InMemoryJobRepository

* Add tests

* Lint

* Update internal/executor/configuration/types.go

* Lint

---------

Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com>

* Stop executor requesting more jobs when it still has leased jobs (#2932)

* Stop executor requesting more jobs when it still has leased jobs

Currently we "queue" jobs to be submitted on the executor - which sit the leased state until they are submitted to kubernetes

However this causes 2 issues with our current setup:
 - It prevents back-pressure from working well on the scheduler side. As it sees all these "Leased" jobs as active, so just keep scheduling more
 - In the case we are slowing submission due to etcd going over its limit. We "queue" lots of jobs, and as soon as etcd goes under its limit we hit it with potentially thousands of jobs

This flow needs further work and thought - however for now this is the minimal fix to prevent bad behaviour

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* WIP

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Fix scheduler side tests

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Implement number of requested jobs on executor side

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Remove unused config

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Fixing panic on startup when etcd health monitor not registered

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Enhance logging

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Set more sensible default for maxLeasedJobs

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

---------

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

* Fix race in etcd protections (#2937)

* Initial commit

* Fix MultiHealthMonitor race

* Fix etcd health metric naming conflict (#2939)

* Fix metric naming conflict

* Fix metric names

* Fix metrix prefix

* Fix label

* Bump golang.org/x/sync from 0.1.0 to 0.3.0 (#2946)

Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.1.0 to 0.3.0.
- [Commits](golang/sync@v0.1.0...v0.3.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add more scheduler metrics (#2906)

* Add jobs considered and refactor to counters

* Add fair share metrics

* Add reset for gauge metrics

* format

* cycle imports

* modify cycle return struct

* verbose logging

---------

Co-authored-by: Albin Severinson <albin@severinson.org>

* Update config.yaml (#2953)

* Remove gang job cardinality submit check. Add placeholder for min gang size

* Add msumner91 and mustafai to magic list of trusted people (#2956)

* Add msumner91 to magic list of trusted people

* Update .mergify.yml

* Airflow: always set credentials from args in channel ctor (#2952)

In the GrpcChannelArguments constructor, always set the
credentials_callback_args member from what is given. Add a test to
verify serialization round-tripping is complete, and a __eq__
implementation for GrpcChannelArguments.

Signed-off-by: Rich Scott <richscott@sent.com>

* Removed Makefile from repo (#2915)

Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>

* Add per-queue scheduling rate-limiting (#2938)

* Initial commit

* Add rate limiters

* go mod tidy

* Updates

* Add tests

* Update default config

* Update default scheduler config

* Whitespace

* Cleanup

* Docstring improvements

* Remove limiter nil checks

* Add Cardinality() function on gctx

* Fix test

* Fix test

* Add note about signed commits to Contributor documentation (#2960)

* Add note about signed commits to Contributor documentation

Signed-off-by: Aviral Singh <itsaviral.2609@gmail.com>

* Add note about signed commits to Contributor documentation

---------

Signed-off-by: Aviral Singh <itsaviral.2609@gmail.com>

* ArmadaContext that includes a logger (#2934)

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* compilation!

* rename package

* more compilation

* rename to Context

* embed

* compilation

* compilation

* fix test

* remove old ctxloggers

* revert design doc

* revert developer doc

* formatting

* wip

* tests

* don't gen

* don't gen

* merged master

---------

Co-authored-by: Chris Martin <chris@cmartinit.co.uk>
Co-authored-by: Albin Severinson <albin@severinson.org>

* Bump armada airflow operator to version 0.5.4 (#2961)

* Bump armada airflow operator to version 0.5.4

Signed-off-by: Rich Scott <richscott@sent.com>

* Regenerate Airflow Operator Markdown doc.

Signed-off-by: Rich Scott <richscott@sent.com>

* Fix regenerated Airflow doc error.

Signed-off-by: Rich Scott <richscott@sent.com>

* Pin versions of all modules, especially around docs generation.

Signed-off-by: Rich Scott <richscott@sent.com>

* Regenerate Airflow docs using Python 3.10

Signed-off-by: Rich Scott <richscott@sent.com>

---------

Signed-off-by: Rich Scott <richscott@sent.com>

* Simulator Changes

Made a number of changes to the simulator and simulator tests, most notably:
 - Fixed implementation of minSubmitTime setting for workload
   specifications
 - Added tests for SchedulingConfigsFromPattern,
   ClusterSpecsFromPattern, WorkloadFromPattern
 - Added sample workloads, clusters and scheduling configs
 - Added tests which simulate per-pool and per-executorGroup scheduling
 - Implemented further metrics for use in simulator tests, such as a
   cluster's aggregate resources, number of preemptions and schedules
   for a given test run
 - Added optimisation to speed up simulator, whereby the scheduler skips
   the current schedule event if no eventSequences have been received
   since the previous schedule.

* Simplified TestClusterSpecsFromPattern and TestWorkloadFromPattern tests

* Removed unused test

* Fixed malformed yaml

* Improved metrics for simulations. Improved simulator tests with errorgroups.

* Removed all simulator test data except basic data necessary for testing

* Implementing CLI

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>
Signed-off-by: Rich Scott <richscott@sent.com>
Signed-off-by: Aviral Singh <itsaviral.2609@gmail.com>
Co-authored-by: Daniel Rastelli <rastellidani@gmail.com>
Co-authored-by: Chris Martin <council_tax@hotmail.com>
Co-authored-by: Chris Martin <chris@cmartinit.co.uk>
Co-authored-by: Sarthak Negi <122533767+sarthaksarthak9@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kannon1992@gmail.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Pradeep Kurapati <113408145+Pradeep-Kurapati@users.noreply.github.com>
Co-authored-by: Dave Gantenbein <dave@gr-oss.io>
Co-authored-by: Shivang Shandilya <101946115+ShivangShandilya@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Co-authored-by: Clif Houck <me@clifhouck.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>
Co-authored-by: Kanu Mike Chibundu <michotall95@gmail.com>
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com>
Co-authored-by: owenthomas17 <owen@owen-thomas.co.uk>
Co-authored-by: Albin Severinson <albin@severinson.org>
Co-authored-by: Mark Sumner <m.sumner91@hotmail.co.uk>
Co-authored-by: Rich Scott <rich@gr-oss.io>
Co-authored-by: MeenuyD <116630390+MeenuyD@users.noreply.github.com>
Co-authored-by: Aviral Singh <itsaviral.2609@gmail.com>
Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>

* Adding verbose flag to simulator CLI, changing logging context in simulator

* Improved simulator CLI output, removed redundant features, implemented parallel simulations by addressing mutability of structures inputted into the simulator

* Removed unknown logging library

* Changing threadSafeLogger Info call to Print. Adding separation back between simulation results

* Implemented stochastic runtime for jobs using a shifted exponential distribution (#13)

* Implemented stochastic runtime for jobs using a shifted exponential distribution

* Implemented min submit time from dependency completion (#14)

Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>

* Fixed tests

* Fixed implementation of shifted exponential distribution

* Using FP unrounded parameters to sample from distribution

* Modified stochastic runtime definition

* Adding logging to simulator

Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>
Signed-off-by: Rich Scott <richscott@sent.com>
Signed-off-by: Aviral Singh <itsaviral.2609@gmail.com>
Co-authored-by: Albin Severinson <larsalbins@uberit.net>
Co-authored-by: Mustafa Ilyas <Mustafa.Ilyas@gresearch.co.uk>
Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>
Co-authored-by: Daniel Rastelli <rastellidani@gmail.com>
Co-authored-by: Chris Martin <council_tax@hotmail.com>
Co-authored-by: Chris Martin <chris@cmartinit.co.uk>
Co-authored-by: Sarthak Negi <122533767+sarthaksarthak9@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kannon1992@gmail.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Pradeep Kurapati <113408145+Pradeep-Kurapati@users.noreply.github.com>
Co-authored-by: Dave Gantenbein <dave@gr-oss.io>
Co-authored-by: Shivang Shandilya <101946115+ShivangShandilya@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Co-authored-by: Clif Houck <me@clifhouck.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>
Co-authored-by: Kanu Mike Chibundu <michotall95@gmail.com>
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com>
Co-authored-by: owenthomas17 <owen@owen-thomas.co.uk>
Co-authored-by: Albin Severinson <albin@severinson.org>
Co-authored-by: Mark Sumner <m.sumner91@hotmail.co.uk>
Co-authored-by: Rich Scott <rich@gr-oss.io>
Co-authored-by: MeenuyD <116630390+MeenuyD@users.noreply.github.com>
Co-authored-by: Aviral Singh <itsaviral.2609@gmail.com>

* Add missing brace

* Lint

* Lint

* Lint

* Cleanup

* Testsuite improvements

* Lint

* Tidying

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>
Signed-off-by: Rich Scott <richscott@sent.com>
Signed-off-by: Aviral Singh <itsaviral.2609@gmail.com>
Co-authored-by: Albin Severinson <Albin.Severinson@gresearch.co.uk>
Co-authored-by: Albin Severinson <larsalbins@uberit.net>
Co-authored-by: Mustafa Ilyas <Mustafa.Ilyas@gresearch.co.uk>
Co-authored-by: Mustafa Ilyas <mustafai@uberit.net>
Co-authored-by: Daniel Rastelli <rastellidani@gmail.com>
Co-authored-by: Chris Martin <council_tax@hotmail.com>
Co-authored-by: Chris Martin <chris@cmartinit.co.uk>
Co-authored-by: Sarthak Negi <122533767+sarthaksarthak9@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kannon1992@gmail.com>
Co-authored-by: Adam McArthur <46480158+Sharpz7@users.noreply.github.com>
Co-authored-by: Pradeep Kurapati <113408145+Pradeep-Kurapati@users.noreply.github.com>
Co-authored-by: Dave Gantenbein <dave@gr-oss.io>
Co-authored-by: Shivang Shandilya <101946115+ShivangShandilya@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Co-authored-by: Clif Houck <me@clifhouck.com>
Co-authored-by: Mohamed Abdelfatah <39927413+Mo-Fatah@users.noreply.github.com>
Co-authored-by: Kanu Mike Chibundu <michotall95@gmail.com>
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JamesMurkin <jamesmurkin@hotmail.com>
Co-authored-by: owenthomas17 <owen@owen-thomas.co.uk>
Co-authored-by: Mark Sumner <m.sumner91@hotmail.co.uk>
Co-authored-by: Rich Scott <rich@gr-oss.io>
Co-authored-by: MeenuyD <116630390+MeenuyD@users.noreply.github.com>
Co-authored-by: Aviral Singh <itsaviral.2609@gmail.com>
  • Loading branch information
1 parent f395cc7 commit 1ef0c22
Show file tree
Hide file tree
Showing 25 changed files with 2,326 additions and 814 deletions.
150 changes: 150 additions & 0 deletions cmd/simulator/cmd/root.go
@@ -0,0 +1,150 @@
package cmd

import (
"github.com/spf13/cobra"
"golang.org/x/exp/maps"

"github.com/armadaproject/armada/internal/common/armadacontext"
"github.com/armadaproject/armada/internal/common/util"
"github.com/armadaproject/armada/internal/scheduler/simulator"
"github.com/armadaproject/armada/pkg/armadaevents"
)

func RootCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "Simulate",
Short: "Simulate running jobs on Armada.",
RunE: runSimulations,
}
// cmd.Flags().BoolP("verbose", "v", false, "Log detailed output to console.")
cmd.Flags().String("clusters", "", "Glob pattern specifying cluster configurations to simulate.")
cmd.Flags().String("workloads", "", "Glob pattern specifying workloads to simulate.")
cmd.Flags().String("configs", "", "Glob pattern specifying scheduler configurations to simulate.")
cmd.Flags().Bool("showSchedulerLogs", false, "Show scheduler logs.")
cmd.Flags().Int("logInterval", 0, "Log summary statistics every this many events. Disabled if 0.")
return cmd
}

func runSimulations(cmd *cobra.Command, args []string) error {
// Get command-line arguments.
clusterPattern, err := cmd.Flags().GetString("clusters")
if err != nil {
return err
}
workloadPattern, err := cmd.Flags().GetString("workloads")
if err != nil {
return err
}
configPattern, err := cmd.Flags().GetString("configs")
if err != nil {
return err
}
showSchedulerLogs, err := cmd.Flags().GetBool("showSchedulerLogs")
if err != nil {
return err
}
logInterval, err := cmd.Flags().GetInt("logInterval")
if err != nil {
return err
}

// Load test specs. and config.
clusterSpecs, err := simulator.ClusterSpecsFromPattern(clusterPattern)
if err != nil {
return err
}
workloadSpecs, err := simulator.WorkloadsFromPattern(workloadPattern)
if err != nil {
return err
}
schedulingConfigsByFilePath, err := simulator.SchedulingConfigsByFilePathFromPattern(configPattern)
if err != nil {
return err
}

ctx := armadacontext.Background()
ctx.Info("Armada simulator")
ctx.Infof("ClusterSpecs: %v", util.Map(clusterSpecs, func(clusperSpec *simulator.ClusterSpec) string { return clusperSpec.Name }))
ctx.Infof("WorkloadSpecs: %v", util.Map(workloadSpecs, func(workloadSpec *simulator.WorkloadSpec) string { return workloadSpec.Name }))
ctx.Infof("SchedulingConfigs: %v", maps.Keys(schedulingConfigsByFilePath))

// Setup a simulator for each combination of (clusterSpec, workloadSpec, schedulingConfig).
simulators := make([]*simulator.Simulator, 0)
metricsCollectors := make([]*simulator.MetricsCollector, 0)
eventSequenceChannels := make([]<-chan *armadaevents.EventSequence, 0)
schedulingConfigPaths := make([]string, 0)
for _, clusterSpec := range clusterSpecs {
for _, workloadSpec := range workloadSpecs {
for schedulingConfigPath, schedulingConfig := range schedulingConfigsByFilePath {
if s, err := simulator.NewSimulator(clusterSpec, workloadSpec, schedulingConfig); err != nil {
return err
} else {
if !showSchedulerLogs {
s.SuppressSchedulerLogs = true
} else {
ctx.Info("Showing scheduler logs")
}
simulators = append(simulators, s)
mc := simulator.NewMetricsCollector(s.Output())
mc.LogSummaryInterval = logInterval
metricsCollectors = append(metricsCollectors, mc)
eventSequenceChannels = append(eventSequenceChannels, s.Output())
schedulingConfigPaths = append(schedulingConfigPaths, schedulingConfigPath)
}
}
}
}

// Run simulators.
g, ctx := armadacontext.ErrGroup(ctx)
for _, s := range simulators {
s := s
g.Go(func() error {
return s.Run(ctx)
})
}

// Log events to stdout.
for _, c := range eventSequenceChannels {
c := c
g.Go(func() error {
for {
select {
case <-ctx.Done():
return ctx.Err()
case eventSequence, ok := <-c:
if !ok {
return nil
}
ctx.Debug(*eventSequence.Events[0].Created, simulator.EventSequenceSummary(eventSequence))
}
}
})
}

// Run metric collectors.
for _, mc := range metricsCollectors {
mc := mc
g.Go(func() error {
return mc.Run(ctx)
})
}

// Wait for simulations to complete.
if err := g.Wait(); err != nil {
return err
}

// Log overall statistics.
for i, mc := range metricsCollectors {
s := simulators[i]
schedulingConfigPath := schedulingConfigPaths[i]
ctx.Infof("Simulation result")
ctx.Infof("ClusterSpec: %s", s.ClusterSpec.Name)
ctx.Infof("WorkloadSpec: %s", s.WorkloadSpec.Name)
ctx.Infof("SchedulingConfig: %s", schedulingConfigPath)
ctx.Info(mc.String())
}

return nil
}
13 changes: 13 additions & 0 deletions cmd/simulator/main.go
@@ -0,0 +1,13 @@
package main

import (
"os"

"github.com/armadaproject/armada/cmd/simulator/cmd"
)

func main() {
if err := cmd.RootCmd().Execute(); err != nil {
os.Exit(1)
}
}
2 changes: 1 addition & 1 deletion cmd/testsuite/cmd/test.go
Expand Up @@ -79,7 +79,7 @@ func testCmdRunE(app *testsuite.App) func(cmd *cobra.Command, args []string) err
app.Params.PrometheusPushGatewayJobName = prometheusPushgatewayJobName

// Create a context that is cancelled on SIGINT/SIGTERM.
// Ensures test jobs are cancelled on ctrl-C.
// Ensures test jobs are cancelled on ctrl-c.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
stopSignal := make(chan os.Signal, 1)
Expand Down
5 changes: 2 additions & 3 deletions internal/scheduler/scheduler.go
Expand Up @@ -17,7 +17,6 @@ import (
"github.com/armadaproject/armada/internal/common/logging"
"github.com/armadaproject/armada/internal/common/stringinterner"
"github.com/armadaproject/armada/internal/scheduler/database"
"github.com/armadaproject/armada/internal/scheduler/interfaces"
"github.com/armadaproject/armada/internal/scheduler/jobdb"
"github.com/armadaproject/armada/internal/scheduler/kubernetesobjects/affinity"
"github.com/armadaproject/armada/internal/scheduler/schedulerobjects"
Expand Down Expand Up @@ -407,7 +406,7 @@ func EventsFromSchedulerResult(result *SchedulerResult, time time.Time) ([]*arma
if err != nil {
return nil, err
}
eventSequences, err = AppendEventSequencesFromUnschedulableJobs(eventSequences, result.FailedJobs, time)
eventSequences, err = AppendEventSequencesFromUnschedulableJobs(eventSequences, FailedJobsFromSchedulerResult[*jobdb.Job](result), time)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -510,7 +509,7 @@ func AppendEventSequencesFromScheduledJobs(eventSequences []*armadaevents.EventS
return eventSequences, nil
}

func AppendEventSequencesFromUnschedulableJobs(eventSequences []*armadaevents.EventSequence, jobs []interfaces.LegacySchedulerJob, time time.Time) ([]*armadaevents.EventSequence, error) {
func AppendEventSequencesFromUnschedulableJobs(eventSequences []*armadaevents.EventSequence, jobs []*jobdb.Job, time time.Time) ([]*armadaevents.EventSequence, error) {
for _, job := range jobs {
jobId, err := armadaevents.ProtoUuidFromUlidString(job.GetId())
if err != nil {
Expand Down
53 changes: 53 additions & 0 deletions internal/scheduler/simulator/events.go
@@ -0,0 +1,53 @@
package simulator

import "time"

// Event is a simulator-internal event.
type Event struct {
// Time at which the event was submitted.
time time.Time
// Each event is assigned a sequence number.
// Events with equal time are ordered by their sequence number.
sequenceNumber int
// Either armadaevents.EventSequence or scheduleEvent.
eventSequenceOrScheduleEvent any
// Maintained by the heap.Interface methods.
index int
}

// scheduleEvent is an event indicating the scheduler should be run.
type scheduleEvent struct{}

type EventLog []Event

func (el EventLog) Len() int { return len(el) }

func (el EventLog) Less(i, j int) bool {
if el[i].time == el[j].time {
return el[i].sequenceNumber < el[j].sequenceNumber
}
return el[j].time.After(el[i].time)
}

func (el EventLog) Swap(i, j int) {
el[i], el[j] = el[j], el[i]
el[i].index = i
el[j].index = j
}

func (el *EventLog) Push(x any) {
n := len(*el)
item := x.(Event)
item.index = n
*el = append(*el, item)
}

func (el *EventLog) Pop() any {
old := *el
n := len(old)
item := old[n-1]
old[n-1] = Event{} // avoid memory leak
item.index = -1 // for safety
*el = old[0 : n-1]
return item
}
106 changes: 106 additions & 0 deletions internal/scheduler/simulator/metrics.go
@@ -0,0 +1,106 @@
package simulator

import (
"fmt"
"strings"
"time"

"golang.org/x/exp/maps"
"golang.org/x/exp/slices"

"github.com/armadaproject/armada/internal/common/armadacontext"
"github.com/armadaproject/armada/pkg/armadaevents"
)

type MetricsCollector struct {
c <-chan *armadaevents.EventSequence
OverallMetrics MetricsVector
MetricsByQueue map[string]MetricsVector
// If non-zero, log a summary every this many events.
LogSummaryInterval int
}

type MetricsVector struct {
TimeOfMostRecentJobSucceededEvent time.Duration
NumEvents int
NumSubmitEvents int
NumLeasedEvents int
NumPreemptedEvents int
NumJobSucceededEvents int
}

func NewMetricsCollector(c <-chan *armadaevents.EventSequence) *MetricsCollector {
return &MetricsCollector{
c: c,
MetricsByQueue: make(map[string]MetricsVector),
}
}

func (mc *MetricsCollector) String() string {
var sb strings.Builder
sb.WriteString("{")
sb.WriteString(fmt.Sprintf("Overall metrics: %s, Per-queue metrics: {", mc.OverallMetrics))
i := 0
queues := maps.Keys(mc.MetricsByQueue)
slices.Sort(queues)
for _, queue := range queues {
metrics := mc.MetricsByQueue[queue]
sb.WriteString(fmt.Sprintf("%s: %s", queue, metrics))
i++
if i != len(mc.MetricsByQueue) {
sb.WriteString(", ")
}
}
sb.WriteString("}}")
return sb.String()
}

func (m MetricsVector) String() string {
return fmt.Sprintf(
"{FractionLeasedSucceeded: %f, TimeOfMostRecentJobSucceededEvent: %s, NumEvents: %d, NumPreemptedEvents: %d, NumLeasedEvents: %d, NumJobSucceededEvents: %d}",
float64(m.NumJobSucceededEvents)/float64(m.NumLeasedEvents), m.TimeOfMostRecentJobSucceededEvent, m.NumEvents, m.NumPreemptedEvents, m.NumLeasedEvents, m.NumJobSucceededEvents,
)
}

func (mc *MetricsCollector) Run(ctx *armadacontext.Context) error {
for {
select {
case <-ctx.Done():
return ctx.Err()
case eventSequence, ok := <-mc.c:
if !ok {
return nil
}
mc.addEventSequence(eventSequence)
if mc.LogSummaryInterval != 0 && mc.OverallMetrics.NumEvents%mc.LogSummaryInterval == 0 {
ctx.Info(mc.String())
}
}
}
}

func (mc *MetricsCollector) addEventSequence(eventSequence *armadaevents.EventSequence) {
queue := eventSequence.Queue
mc.OverallMetrics.NumEvents += 1
perQueueMetrics := mc.MetricsByQueue[queue]
perQueueMetrics.NumEvents += 1
for _, event := range eventSequence.Events {
switch event.GetEvent().(type) {
case *armadaevents.EventSequence_Event_SubmitJob:
mc.OverallMetrics.NumSubmitEvents += 1
perQueueMetrics.NumSubmitEvents += 1
case *armadaevents.EventSequence_Event_JobRunLeased:
mc.OverallMetrics.NumLeasedEvents += 1
perQueueMetrics.NumLeasedEvents += 1
case *armadaevents.EventSequence_Event_JobRunPreempted:
mc.OverallMetrics.NumPreemptedEvents += 1
perQueueMetrics.NumPreemptedEvents += 1
case *armadaevents.EventSequence_Event_JobSucceeded:
mc.OverallMetrics.TimeOfMostRecentJobSucceededEvent = event.Created.Sub(time.Time{})
perQueueMetrics.TimeOfMostRecentJobSucceededEvent = event.Created.Sub(time.Time{})
mc.OverallMetrics.NumJobSucceededEvents += 1
perQueueMetrics.NumJobSucceededEvents += 1
}
}
mc.MetricsByQueue[queue] = perQueueMetrics
}

0 comments on commit 1ef0c22

Please sign in to comment.