-
-
Notifications
You must be signed in to change notification settings - Fork 547
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add parallelism to major chunk of hooks. Check Parallelism
section in README
#620
Merged
+292
−41
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As non-zero codes now can popup only in subshells, there is no more reason to keep that option in main shell
This reverts commit b6017f1.
MaxymVlasov
added
feature
New feature or request
hook/terraform_tflint
Bash hook
hook/terraform_fmt
Bash hook
hook/terraform_docs
Bash hook
estimate/2h
Need 2 hours to be done
hook/terraform_checkov
Bash hook
hook/terraform_validate
Bash hook
hook/terraform_tfsec
Bash hook
hook/terragrunt_validate
Bash hook
hook/terraform_providers_lock
Bash hook
hook/terrascan
Bash hook
hook/terragrunt_fmt
Bash hook
hook/tfupdate
Bash hook
labels
Feb 9, 2024
Co-authored-by: George L. Yermulnik <yz@yz.kiev.ua>
MaxymVlasov
commented
Feb 13, 2024
yermulnik
approved these changes
Feb 13, 2024
MaxymVlasov
requested review from
antonbabenko and
robinbowes
and removed request for
robinbowes
February 13, 2024 13:33
MaxymVlasov
commented
Feb 14, 2024
MaxymVlasov
commented
Feb 14, 2024
Loading status checks…
…623) So, I found that `nproc` always shows how many CPUs available is. K8s "limits" and docker `--cpus` are throttling mechanisms, which do not hide the visibility of all cores. There are a few workarounds, but IMO, it is better to implement checks for that than do them >Workaround for docker - set `--cpuset-cpus` >Workaraund for K8s - somehow deal with [kubelet static CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#cpu-management-policies), as [recommend in Reddit](https://news.ycombinator.com/item?id=25224714) * Send all "colorify" logs through stderr, as make able to add user-facing-logs in functions that also need to return same value to the function-caller. Needed for `common::get_cpu_num` err_msg show up --------- Co-authored-by: George L. Yermulnik <yz@yz.kiev.ua>
MaxymVlasov
commented
Feb 15, 2024
Loading status checks…
MaxymVlasov
commented
Feb 15, 2024
Details: #620 (review)
yermulnik
reviewed
Feb 15, 2024
Co-authored-by: George L. Yermulnik <yz@yz.kiev.ua>
antonbabenko
approved these changes
Feb 17, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OMG! It seems to work as expected
Yeah, that's a bit of magic by Max 😄 |
antonbabenko
pushed a commit
that referenced
this pull request
Feb 17, 2024
# [1.87.0](v1.86.1...v1.87.0) (2024-02-17) ### Features * Add parallelism to major chunk of hooks. Check `Parallelism` section in README ([#620](#620)) ([6c6eca4](6c6eca4))
This PR is included in version 1.87.0 🎉 |
github-actions bot
pushed a commit
to MaxymVlasov/pre-commit-terraform-712
that referenced
this pull request
Jan 24, 2025
# 1.0.0 (2025-01-24) ### Bug Fixes * `grep: warning: stray \ before /` which pop-up in `grep 3.8` ([antonbabenko#625](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/625)) ([e1a93b2](e1a93b2)) * **`terraform_docs`:** Fix issue and prioritize `output.file` setting from `.terraform-docs.yml` config over `--hook-config=--path-to-file=` ([antonbabenko#698](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/698)) ([9d6a22b](9d6a22b)) * **`terraform_docs`:** Fix issue with processing multiply files without `terraform-docs` markers. Issue introduced in v1.95.0 ([antonbabenko#720](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/720)) ([2b1aec8](2b1aec8)), closes [antonbabenko#717](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/717) [/github.com/antonbabenko/pre-commit-terraform/blob/869a106a4c8c48f34f58318a830436142e31e10a/hooks/terraform_docs.sh#L216](https://github.com//github.com/antonbabenko/pre-commit-terraform/blob/869a106a4c8c48f34f58318a830436142e31e10a/hooks/terraform_docs.sh/issues/L216) * **`terraform_docs`:** Fix non-GNU `sed` issues, introduced in v1.93.0 ([antonbabenko#704](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/704)) ([3c8734d](3c8734d)) * **`terraform_docs`:** Fix non-GNU sed issues, introduced in v1.93.0, as previous fix doesn't work correctly ([antonbabenko#708](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/708)) ([c986c5e](c986c5e)) * **`terraform_docs`:** Restore `--hook-config=--add-to-existing-file` default behavior. Regression from 1.94.0. ([antonbabenko#716](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/716)) ([315342e](315342e)) * **`terraform_docs`:** Restore multiply `--hook-config` args support. Regression from v1.95.0 ([antonbabenko#731](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/731)) ([87143fb](87143fb)) * **`terraform_docs`:** Suppress "terraform command not found" error message in case binary does not exist ([antonbabenko#693](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/693)) ([6ff3572](6ff3572)) * **`terraform_docs`:** Suppress redundant warnings pop-ups introduced in v1.92.2 ([antonbabenko#700](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/700)) ([59b2454](59b2454)) * **`terraform_providers_lock`:** Require `terraform init` (and `terraform_validate` hook) run when only lockfile changed ([antonbabenko#649](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/649)) ([02c1935](02c1935)) * **`terraform_validate`:** Run `terraform init` on "Missing required provider" error ([antonbabenko#586](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/586)) ([6e2bb2e](6e2bb2e)) * Add `--env-vars`, deprecate `--envs` ([antonbabenko#410](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/410)) ([2b35cad](2b35cad)) * Add `--tf-init-args`, deprecate `--init-args` ([antonbabenko#407](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/407)) ([c4f8251](c4f8251)) * analyse all folders with tflint and don't stop on first execution ([antonbabenko#289](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/289)) ([7c6ad7c](7c6ad7c)) * Change terraform_validate hook functionality for subdirectories with terraform files ([antonbabenko#100](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/100)) ([7694fb9](7694fb9)) * Check all directories with changes and pass all args in terrascan hook ([antonbabenko#305](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/305)) ([66401d9](66401d9)) * command not found ([antonbabenko#251](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/251)) ([e33c654](e33c654)) * Correct deprecated parameter to terraform-docs ([antonbabenko#156](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/156)) ([3a07570](3a07570)) * Correctly handle arrays in terraform_docs.sh ([antonbabenko#141](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/141)) ([f2cab31](f2cab31)) * Describe migration instructions from `terraform_docs_replace` ([antonbabenko#451](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/451)) ([a8bcaa7](a8bcaa7)) * **docker:** Checkov installation silently fails on `docker build` in arm64. Workaround till issue will be fixed in `checkov` itself ([antonbabenko#635](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/635)) ([f255b05](f255b05)) * Dockerfile if INSTALL_ALL is not defined ([antonbabenko#233](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/233)) ([3bdcf51](3bdcf51)) * Dockerized pre-commit-terraform ([antonbabenko#219](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/219)) ([ce02f94](ce02f94)) * **docker:** Prevent all possible "silent errors" during `docker build` ([antonbabenko#644](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/644)) ([0340c8d](0340c8d)) * execute tflint once in no errors ([antonbabenko#250](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/250)) ([390a264](390a264)) * Extend `terraform_validate` `--retry-once-with-cleanup` errors list ([antonbabenko#566](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/566)) ([19188e5](19188e5)) * Fix `terraform_providers_lock` hook broken in v1.79.0 ([antonbabenko#521](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/521)) ([6bfc5bf](6bfc5bf)) * Fix and pin versions in Dockerfile ([antonbabenko#193](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/193)) ([d27074b](d27074b)) * Fix regex considering terraform-docs v0.10.0 old ([antonbabenko#151](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/151)) ([d773f4a](d773f4a)) * Fix terraform_wrapper_module_for_each for when resource name contains 'variable' ([antonbabenko#573](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/573)) ([941177e](941177e)) * Fix terraform_wrapper_module_for_each hook heredoc vars defaults ([antonbabenko#554](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/554)) ([6fd4263](6fd4263)) * Fix the terraform_wrapper_module_for_each hook for modules without outputs or variables ([antonbabenko#552](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/552)) ([f24b3fa](f24b3fa)) * Fixed `terraform_fmt` with `tfenv`, when `terraform` default version is not specified ([antonbabenko#389](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/389)) ([1b9476a](1b9476a)) * Fixed `tfupdate` to work in all cases, not only `pre-commit run --all` ([antonbabenko#375](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/375)) ([297cc75](297cc75)) * Fixed 1.54.0 where `terraform_docs` was broken ([antonbabenko#272](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/272)) ([4c50943](4c50943)) * Fixed args expand in terraform_docs ([antonbabenko#260](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/260)) ([01a6170](01a6170)) * Fixed docker build ([antonbabenko#288](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/288)) ([4543f10](4543f10)) * Fixed git fatal error in Dockerfile ([antonbabenko#372](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/372)) ([c3f8dd4](c3f8dd4)) * Fixed ordering issue in terraform_wrapper_module_for_each hook ([antonbabenko#565](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/565)) ([dc12be1](dc12be1)) * Fixed spacing in `terraform_wrapper_module_for_each` hook ([antonbabenko#503](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/503)) ([ddc0d81](ddc0d81)) * Fixed url for wrappers in generated README (terraform_wrapper_module_for_each) ([antonbabenko#429](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/429)) ([fe29c6c](fe29c6c)) * Improve `tflint --init` command execution ([antonbabenko#361](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/361)) ([d31cb69](d31cb69)) * Improve README and drop quotes from hook env vars ([antonbabenko#651](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/651)) ([daec682](daec682)) * label auto-adding after label rename ([antonbabenko#226](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/226)) ([4faee7b](4faee7b)) * Make hooks bash 3.2 compatible ([antonbabenko#339](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/339)) ([4ad825d](4ad825d)) * make terraform_docs Windows compatible ([antonbabenko#129](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/129)) ([81770aa](81770aa)) * make terraform_tfsec.sh executable ([antonbabenko#140](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/140)) ([077c423](077c423)) * **non-linux:** Bash environment variables in arguments not expanded + Add `trace` log level ([antonbabenko#645](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/645)) ([a2a2990](a2a2990)) * Pass args and env vars to terraform validate ([antonbabenko#125](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/125)) ([774c63e](774c63e)) * Pass command line arguments to tflint init ([antonbabenko#487](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/487)) ([29a8c00](29a8c00)) * Passed scenario in `terraform_docs` hook now works as expected ([7ac2736](7ac2736)) * pre-build docker image ([antonbabenko#292](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/292)) ([01d262c](01d262c)) * Pre-commit-terraform terraform_validate hook ([antonbabenko#401](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/401)) ([d9f482c](d9f482c)) * Properly exclude .terraform directory with checkov hook ([antonbabenko#306](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/306)) ([b431a43](b431a43)) * remove dead code from terraform-docs script ([antonbabenko#229](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/229)) ([ff54bb4](ff54bb4)) * remove sed postprocessing from the terraform_docs_replace hook to fix compatibility with terraform-docs 0.11.0+ ([antonbabenko#176](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/176)) ([90d4521](90d4521)) * Replace `mapfile` to support Bash 3.2.57 pre-installed in macOS ([antonbabenko#628](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/628)) ([01ab3f0](01ab3f0)) * Run `terraform_tfsec` only on terraform code changes ([antonbabenko#571](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/571)) ([4253162](4253162)) * Speed up x2 TFLint hook execution in dirs with violations ([antonbabenko#514](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/514)) ([49974ab](49974ab)) * Speedup `terrascan` hook up to x3 times in big repos ([antonbabenko#307](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/307)) ([2e8dcf9](2e8dcf9)) * Squash terraform_docs bug ([antonbabenko#138](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/138)) ([6c77a6c](6c77a6c)) * Support custom TF paths which contains spaces ([antonbabenko#714](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/714)) ([2bca410](2bca410)) * Suppress duplicate error messages in `terraform_validate` ([antonbabenko#577](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/577)) ([4ea6b14](4ea6b14)) * Terraform validate for submodules ([antonbabenko#172](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/172)) ([827af52](827af52)) * terraform_tflint hook executes in a serial way to run less often ([antonbabenko#211](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/211)) ([3404eed](3404eed)) * **terraform_tflint:** Restore current working directory behavior ([antonbabenko#302](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/302)) ([93029dc](93029dc)) * terraform-docs version 0.10 removed with-aggregate-type-defaults ([antonbabenko#150](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/150)) ([6f3b125](6f3b125)) * terrafrom_tflint ERROR output for files located in repo root ([antonbabenko#243](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/243)) ([3f66432](3f66432)) * TFSec outputs the same results multiple times ([antonbabenko#237](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/237)) ([71f7c34](71f7c34)) * trigger terraform-docs on changes in lock files ([antonbabenko#228](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/228)) ([b67dbd9](b67dbd9)) * typo in arg name for terraform-docs ([antonbabenko#283](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/283)) ([feabecc](feabecc)) * Updated formatting in README (closes [antonbabenko#113](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/113)) ([27e6369](27e6369)) * Updates all dependencies used in Dockerfile and fix Docker image ([antonbabenko#507](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/507)) ([dc177fe](dc177fe)) * **WSL:** Make parallelism work appropriately ([antonbabenko#728](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/728)) ([e87ee43](e87ee43)) ### Features * **`terraform_docs`:** Add `terraform-docs` default markers support and describe how to migrate to them ([antonbabenko#609](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/609)) ([4a0e1fe](4a0e1fe)) * **`terraform_docs`:** Add support for custom markers to better support other formats than Markdown ([antonbabenko#752](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/752)) ([cd090b6](cd090b6)) * **`terraform_docs`:** Drop support for `terraform-docs` <0.12.0 ([antonbabenko#717](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/717)) ([81e4572](81e4572)) * **`terraform_docs`:** Start seamless migration to `terraform-docs` markers ([antonbabenko#701](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/701)) ([d03f44f](d03f44f)) * **`terraform_providers_lock`:** Add `--mode` option and deprecate previous workflow ([antonbabenko#528](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/528)) ([2426b52](2426b52)) * **`terraform-docs`:** Add support for `replace` mode for TF 0.12+; Use native saving to file for TF 0.12+. Both requires `terraform-docs` v0.12.0+ which released in 2021. ([antonbabenko#705](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/705)) ([1a1b4a3](1a1b4a3)) * Add __GIT_WORKING_DIR__ to terraform_checkov ([antonbabenko#399](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/399)) ([ae88ed7](ae88ed7)) * add __GIT_WORKING_DIR__ to tfsec ([antonbabenko#255](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/255)) ([2973f85](2973f85)) * Add `--retry-once-with-cleanup` to `terraform_validate` ([antonbabenko#441](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/441)) ([96fe3ef](96fe3ef)) * Add `terraform_docs` hook settings ([antonbabenko#245](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/245)) ([7b11401](7b11401)) * Add `terragrunt_providers_lock` hook ([antonbabenko#632](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/632)) ([77940fd](77940fd)) * Add `terragrunt_validate_inputs` hook to check unused and undefined inputs ([antonbabenko#677](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/677)) ([a139b71](a139b71)) * Add checkov support ([antonbabenko#143](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/143)) ([293b64c](293b64c)) * Add GH checks and templates ([antonbabenko#222](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/222)) ([53a866e](53a866e)) * Add infracost_breakdown hook ([antonbabenko#252](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/252)) ([cff42e6](cff42e6)) * Add mixed line ending check to prevent possible errors ([antonbabenko#221](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/221)) ([c920368](c920368)) * Add new hook for `terraform providers lock` operation ([antonbabenko#173](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/173)) ([d69e86d](d69e86d)) * Add parallelism to major chunk of hooks. Check `Parallelism` section in README ([antonbabenko#620](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/620)) ([6c6eca4](6c6eca4)) * Add PATH outputs when TFLint found any problem ([antonbabenko#234](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/234)) ([ce02cd1](ce02cd1)) * Add possibility to share tflint config file for subdirs ([antonbabenko#149](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/149)) ([cf07b5e](cf07b5e)) * Add support for `pre-commit/pre-commit-hooks` in Docker image ([antonbabenko#374](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/374)) ([017da74](017da74)) * Add support for quoted values in `infracost_breakdown` `--hook-config` ([antonbabenko#269](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/269)) ([e2604ea](e2604ea)) * Add support for set env vars inside hook runtime ([antonbabenko#408](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/408)) ([d490231](d490231)) * Add support for specify terraform-docs config file ([antonbabenko#244](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/244)) ([25cddd9](25cddd9)) * Add support for version constraints in `tfupdate` ([antonbabenko#437](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/437)) ([a446642](a446642)) * add terragrunt validate hook ([antonbabenko#134](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/134)) ([f6caf21](f6caf21)) * Added `terraform_checkov` (run per folder), deprecated `checkov` hook ([antonbabenko#290](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/290)) ([e3a9834](e3a9834)) * Added semantic release ([antonbabenko#296](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/296)) ([1bcca44](1bcca44)) * Added support for `tfupdate` to update version constraints in Terraform configurations ([antonbabenko#342](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/342)) ([ef7a0f2](ef7a0f2)) * Added terraform_wrapper_module_for_each hook ([antonbabenko#376](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/376)) ([e4e9a73](e4e9a73)) * Added Terramate as sponsor ([antonbabenko#676](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/676)) ([dae1a48](dae1a48)) * Adding init to terraform_tflint hook ([antonbabenko#352](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/352)) ([1aff30f](1aff30f)) * Adds support for Terrascan ([antonbabenko#195](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/195)) ([fee2387](fee2387)) * Allow `terraform_providers_lock` specify terraform init args ([antonbabenko#406](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/406)) ([32b232f](32b232f)) * Allow env vars expansion in `--args` section for all hooks ([antonbabenko#363](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/363)) ([caa01c3](caa01c3)) * Allow passing of args to terraform_fmt ([antonbabenko#147](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/147)) ([de2f624](de2f624)) * Allow running container as non-root UID/GID for ownership issues (docker) ([antonbabenko#433](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/433)) ([abc2570](abc2570)) * **ci:** Build multi-arch Docker images (`amd64`, `arm64`) ([antonbabenko#496](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/496)) ([923c2c6](923c2c6)) * **deps:** Bump Python version in docker image from 3.11.5 to v3.12.0 ([antonbabenko#597](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/597)) ([28e3cde](28e3cde)) * **docker:** Add ssh-client to Docker image to access private modules via ssh ([antonbabenko#553](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/553)) ([1d76157](1d76157)) * Expand environment variables in `--args=` which contains lowercase symbols, like `${TF_VAR_lowercase}` ([antonbabenko#719](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/719)) ([bf156b4](bf156b4)) * have option for terraform_tfsec hook to only run in relevant modified directories ([antonbabenko#135](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/135)) ([108c75f](108c75f)) * Hook terraform_wrapper_module_for_each should use versions.tf from the module if it exists ([antonbabenko#657](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/657)) ([b127601](b127601)) * Improve performance during `pre-commit --all (-a)` run ([antonbabenko#327](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/327)) ([7e7c916](7e7c916)) * Improved speed of `pre-commit run -a` for multiple hooks ([antonbabenko#338](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/338)) ([579dc45](579dc45)) * Make terraform_validate to run init if necessary ([antonbabenko#158](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/158)) ([d303bff](d303bff)) * Pass custom arguments to terraform init in `terraform_validate` hook ([antonbabenko#293](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/293)) ([45575c3](45575c3)) * Removed `coreutils` (realpath) from dependencies for MacOS ([antonbabenko#368](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/368)) ([944a2e5](944a2e5)) * Set up PR reviewers automatically ([antonbabenko#258](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/258)) ([cc59119](cc59119)) * Skip legacy modules (with provider block) in terraform_wrapper_module_for_each hook ([antonbabenko#560](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/560)) ([456cc76](456cc76)) * Speedup `terraform_validate` - firstly try run validate without checking is `.terraform/` is valid ([antonbabenko#524](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/524)) ([d0d08ac](d0d08ac)) * Support for TFSec ([antonbabenko#103](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/103)) ([2be8fe5](2be8fe5)) * Support set custom TF/OpenTofu binary. | If you use a custom Docker image build, please note that `TERRAFORM_VERSION` now must be provided ([antonbabenko#670](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/670)) ([c7011c0](c7011c0)) * Suppress color for all hooks if `PRE_COMMIT_COLOR=never` set ([antonbabenko#409](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/409)) ([b12f0c6](b12f0c6)) * TFLint: Add `--hook-config=--delegate-chdir` to use `tflint -chdir` ([antonbabenko#512](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/512)) ([1e9debc](1e9debc)) * **trivy:** Add `terraform_trivy` hook and deprecate `terraform_tfsec` ([antonbabenko#606](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/606)) ([f3c819a](f3c819a)) * Updated Docker image from Ubuntu to Alpine ([antonbabenko#278](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/278)) ([71302a9](71302a9)) * When a config file is given, do not specify formatter on cli (terraform_docs) ([antonbabenko#386](https://github.com/MaxymVlasov/pre-commit-terraform-712/issues/386)) ([962054b](962054b))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
estimate/2h
Need 2 hours to be done
feature
New feature or request
hook/terraform_checkov
Bash hook
hook/terraform_docs
Bash hook
hook/terraform_fmt
Bash hook
hook/terraform_providers_lock
Bash hook
hook/terraform_tflint
Bash hook
hook/terraform_tfsec
Bash hook
hook/terraform_trivy
Bash hook
hook/terraform_validate
Bash hook
hook/terragrunt_fmt
Bash hook
hook/terragrunt_validate
Bash hook
hook/terrascan
Bash hook
hook/tfupdate
Bash hook
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Put an
x
into the box if that apply:Reasoning
We have a GH workflow that runs lockflies updates every week (implementation and reasoning here).
It usually takes from 2h 30min to 3h 15min. That was fine for us, till we found that our GH runners, based on AWS EC2s, started silently failing after 30min "without recent logs", and that was fixed by crutch which sends a dummy log every 10min.
However, during the debugging, I spent some time describing why hooks were not utilizing all the provided resources.
And that means a waste of time and money, not only for that corner case but for every huge commit, which can cause opting out by
git commit -n
of using hooks locally for changes that affect many directories.Description of your changes
--parallelism-limit
setting to--hook-config
. Defaults tonumber of logical CPUs - 1
Main implementation usesflock
, which not pre-installed in macOS and some ancient BSD/Linux systems.To maintain backward compatibility, for cases where noflock
- fallback to "bash-native locking mechanism" was added.t init
in parallel.t init
directory.In quick tests, I can say that on big changes:
Withflock
, there 300-500% speed increase for most of the hooks.Withoutflock
, there 250-450% speed increase for most of the hooks.terraform_validate
, and up to 500% - for other affected hooks.--parallelism-limit=1
I observed an insignificant increase in time (about 5-10%) compared to v1.86.0 which has no parallelism at all. This may be the cost of maintaining parallelism or the result of external factors since the tests were not conducted in a vacuum.For small changes, improvements are less significant.
How can we test changes
There is no easy way to test it, but that could be used as a testing template