-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
addrs: limit source component length to 16 #12962
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of suggestions. If a user drops a binary in the plugins directory with more then 16 directories do we prevent it from loading?
Is there a way to validate the address of the discovered plugin so that we don't need to track the depth of the filepath.Walk?
That's a good call for plugins that are completely manually installed, we don't track the depth for filepath.Walk, but iirc there's something built-in for limiting how deep we visit the filesystem? |
63830fe
to
cecdb79
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Functionality wise this looks good and handles both cases of installing or lading a plugin that is more the 16 paths deep. The error messaging doesn't display well so we need to improve that.
When specifying/installing plugins, a source URI is required for Packer to be able to locate or install a plugin to the local plugin hierarchy. The plugin hierarchy is based on the plugin source, where each component in this hierarchy will become a directory. In order to avoid sources with too many levels of nesting, causing a lot of mkdirs, we limit the number of sources to 16 in this commit, this should be long enough for most of our users.
ee7a526
to
4f6299e
Compare
The updated error for this particular case looks good. But we need to clean up the other error message by adding a newline as they can be a bit long, maybe even repetative since the name of the plugin is already displayed within the "invalid source string ..." message.
|
ec80b8f
to
344618b
Compare
@nywilken I pushed a change to the error messages, trying to make lines shorter and removing mentions of the source string since it'll be in the error message, let me know what you think |
Beautiful
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like you might of rebase with plugin install docs. I would just spot check for consistency between your branches.
I have indeed; will cleanup now |
The ParsePluginSource function can be invoked from either a HCL2 context (when parsing a required_plugins block), or from the command-line itself. While in the first context a hcl.Diagnostics is coherent, in case the source to parse is a command-line argument, for example when installing or removing a plugin, the error message cannot have an HCL context, leading to errors that are incorrectly prefixed by a <nil> string dure to the lack of a reference to attach the diagnostic to. Therefore, in order to fix this behaviour, the logic that parses plugin sources now returns an error, and attaching the error to an HCL subject is done independently, if needed.
344618b
to
5d399fc
Compare
[](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [hashicorp/packer](https://togithub.com/hashicorp/packer) | minor | `"1.10.0"` -> `1.11.0` | --- ### Release Notes <details> <summary>hashicorp/packer (hashicorp/packer)</summary> ### [`v1.11.0`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1110-2024-05-31) [Compare Source](https://togithub.com/hashicorp/packer/compare/v1.10.3...v1.11.0-beta) ##### NOTES: - A LICENSE.txt file has been added to the Packer release artifacts. [GH-12931](https://togithub.com/hashicorp/packer/pull/12931) [GH-12940](https://togithub.com/hashicorp/packer/pull/12940) - **Breaking Change**: Support for loading single-component plugins has been removed from Packer. [GH-12785](https://togithub.com/hashicorp/packer/pull/12785) - **Breaking Change**: Support for loading plugin binaries following the naming convention of packer-plugin-name has been dropped. Packer will now only load plugins stored under the [Packer plugin directory](https://developer.hashicorp.com/packer/docs/configure#packer-s-plugin-directory) using the expected namespaced directory and CHECKSUM files. This change drops support for loading plugin binaries in Packer's executable directory or a template's current working directory. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828) ```shell /Users/dev/.packer.d/plugins └── github.com └── hashicorp └── happycloud ├── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64 └── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64_SHA256SUM ``` ##### FEATURES: - core: Add `-ignore-prerelease-plugins` flag to disable the use of development plugin binaries for the `build` and `validate` commands development plugin binaries. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828) [GH-12882](https://togithub.com/hashicorp/packer/pull/12882) - Packer users can now track Packer version and plugin versions used for each build artifact in HCP Packer. [GH-12866](https://togithub.com/hashicorp/packer/pull/12866) - hcl2: add textencodebase64 and textdecodebase64 funcs For feature parity with Terraform, and since having access to strings encoded in something that is not UTF-8 is required in some cases, we add both the textencodebase64 and textdecodebase64 functions to HCL2 templates. Please note these functions return base64 encoded byte slices because of how cty/hcl defines strings (NFC normalised, UTF-8 encoded). [GH-12997](https://togithub.com/hashicorp/packer/pull/12997) ##### SECURITY: - Bump github.com/go-jose/go-jose/v3 to address GHSA-c5q2-7r4c-mv6g. [GH-12880](https://togithub.com/hashicorp/packer/pull/12880) - Bump golang.org/x/net to v0.24.0 to address GO-2024-2687. [GH-12924](https://togithub.com/hashicorp/packer/pull/12924) ##### IMPROVEMENTS: - core: Bump github.com/hashicorp/hcp-sdk-go from 0.90.0 to 0.96.0. [GH-12935](https://togithub.com/hashicorp/packer/pull/12935) [GH-12942](https://togithub.com/hashicorp/packer/pull/12942) [GH-12960](https://togithub.com/hashicorp/packer/pull/12960) [GH-12979](https://togithub.com/hashicorp/packer/pull/12979) - core: Bump github.com/hashicorp/packer-plugin-sdk from 0.5.2 to 0.5.3 [GH-12932](https://togithub.com/hashicorp/packer/pull/12932) - core: Bump go-getter/v2 from 2.2.1 to v2.2.2 [GH-12988](https://togithub.com/hashicorp/packer/pull/12988) - datasource/http: don't error on 2xx code Previous versions of Packer only supported 200 as a success case for the http datasource. This change makes any status code from 200 to 299 successful. [GH-12989](https://togithub.com/hashicorp/packer/pull/12989) - core: Move to predictable plugin loading schema - Packer will now only load plugins stored under the [Packer plugin directory](https://developer.hashicorp.com/packer/docs/configure#packer-s-plugin-directory) using the expected namespaced directory and CHECKSUM files. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828) - core: Remove support loading single-component plugins. [GH-12785](https://togithub.com/hashicorp/packer/pull/12785) - core: Rename internal `packer plugin` command to `packer execute` to avoid user confusion with the plugins subcommand. [GH-12865](https://togithub.com/hashicorp/packer/pull/12865) - core: Packer now considers development binaries when evaluating plugin version constraints. This work allows users to use binaries with versions reported as "x.y.z-dev" to be used with the Packer `required_plugins` block. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828) - core: Packer now supports local paths to plugins for the `packer plugins remove` command. This addition makes it possible to pipe commands like `packer plugins installed` with it for speedy cleanup of installed plugins. [GH-12886](https://togithub.com/hashicorp/packer/pull/12886) - core: Relax Packer source address URIs within the `required_plugins` block to support the installation of local plugin binary using a custom or internal source address (e.g. mycompany.com/plugins/happyorg/happycloud). Remote installation using `packer init` or `packer plugins install` does not support non-GitHub source URIs. Users using alternative hosts must install plugins manually using `packer plugins install --path`. [GH-12911](https://togithub.com/hashicorp/packer/pull/12911), \[[GH-12962](https://togithub.com/hashicorp/packer/issues/12962)] ([https://github.com/hashicorp/packer/pull/12962](https://togithub.com/hashicorp/packer/pull/12962)) - core: Remote plugins installed containing an internal version number that differs from the version number within the binary name can lead to confusion when tracking Packer plugin version information. To help track such discrepancies in the plugin version, `packer init` and `packer plugin install` have been updated to reject installation of such plugins. 1.0.0-dev). Users are encouraged to notify plugin maintainers of any version mismatches. [GH-12915](https://togithub.com/hashicorp/packer/pull/12915), [GH-12953](https://togithub.com/hashicorp/packer/pull/12953), [GH-12972](https://togithub.com/hashicorp/packer/pull/12972) - core: don't load plugins with metadata in name To avoid confusion with multiple plugins that report the same effective version, plugins installed need to have no metadata in their name. When installed through Packer commands, the metadata is scrubbed from the name of the installed plugin binary, but manually it may still be possible, so we enforce that scrubbing at load-time as well. [GH-12980](https://togithub.com/hashicorp/packer/pull/12980) - core: Error when multiple paths are specified for PACKER_PLUGIN_PATH Since Packer 1.11 removed the capability for PACKER_PLUGIN_PATH to specify multiple directories separated by `:` or `;` (depending on the platform), we are explicitly erroring when this is discovered, with suggestions as to how to fix the problem. [GH-12967](https://togithub.com/hashicorp/packer/pull/12967) - core: Version metadata support for plugins. Plugins may now formally have metadata in their versions, Packer supports it, and applies the semver recommendations on them, i.e. they are ignored for comparison/sorting purposes, but allowed for adding extra information about a plugin. [GH-12888](https://togithub.com/hashicorp/packer/pull/12888) Given the specified version constraint only versions greater than or equal to 1.1.0 will be considered. ```hcl amazon = { source = "github.com/hashicorp/amazon" version = ">= 1.1.0" } ``` If a development binary is installed, Packer will use it if: 1. It is the highest compatible version installed. 2. There is no final plugin version with the same version number installed alongside it. ```shell /Users/dev/.packer.d/plugins └─ github.com └─ hashicorp └── amazon ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64 ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64_SHA256SUM ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64 └── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM ``` Version 1.1.1-dev of the Amazon plugin will match the specified version constraint and be used for executing the Packer build. If, however, a 1.1.1 release version of the plugin is available, it will have precedence over the development binary. ```shell /Users/dev/.packer.d/plugins └─ github.com └─ hashicorp └── amazon ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64 ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM ├── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64 └── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64_SHA256SUM ``` ##### BUG FIXES: - core: fix plugin version ordering to not be lexicographic. This fixes an issue with how plugins are discovered by Packer, and ensures proper version ordering. This means that with this change, versions that are semantically greater, but lexicographically inferior will be loaded. Ex: 1.0.9 vs. 1.0.10; `1.0.9 > 1.0.10` lexicographically, but semantically `1.0.10 > 1.0.9` - core/hcp: fix potential race condition when storing plugin details to the HCP Packer metadata storage map. [GH-12936](https://togithub.com/hashicorp/packer/pull/12936) - core: fix plugin listing on Windows This fix addresses bugs present in the alpha releases of 1.11, where the discovery of Windows binaries were not matching against the filename extension (.exe). [GH-12981](https://togithub.com/hashicorp/packer/pull/12981) ### [`v1.10.3`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1103-April-22-2024) [Compare Source](https://togithub.com/hashicorp/packer/compare/v1.10.2...v1.10.3) ##### NOTES - A LICENSE.txt file has been added to the Packer release artifacts. [GH-12931](https://togithub.com/hashicorp/packer/pull/12931) ##### FEATURES - Packer users can now track Packer version and plugin versions used for each build artifact in HCP Packer. [GH-12891](https://togithub.com/hashicorp/packer/pull/12891) [GH-12910](https://togithub.com/hashicorp/packer/pull/12910) [GH-12923](https://togithub.com/hashicorp/packer/pull/12923) [GH-12926](https://togithub.com/hashicorp/packer/pull/12926) ##### SECURITY - Bump github.com/go-jose/go-jose/v3 to address GHSA-c5q2-7r4c-mv6g. [GH-12928](https://togithub.com/hashicorp/packer/pull/12928) - Bump golang.org/x/net to v0.24.0 to address GO-2024-2687. [GH-12924](https://togithub.com/hashicorp/packer/pull/12924) ##### BUG FIXES - cmd/fmt: Display information error when Packer `fmt` fails due to HCL2 parsing error. [GH-12870](https://togithub.com/hashicorp/packer/pull/12870) ### [`v1.10.2`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1102-March-6-2024) [Compare Source](https://togithub.com/hashicorp/packer/compare/v1.10.1...v1.10.2) ##### NOTES: - Continuing the work in in Packer v1.10.0 we introduced the ability to install a locally sourced plugin using packer `plugins install --path`, this release extends support to development plugin binaries - binaries that report "dev" as part of their plugin version. Instead of manually placing a downloaded binary into the executable or current working directory we encourage you to run the command `packer plugins install –path <path-to- downloaded-extracted-binary> github.com/hashicorp/happycloud` to install the binary into a Packer compatible path. [GH-12855](https://togithub.com/hashicorp/packer/pull/12855) ##### IMPROVEMENTS: - cmd/plugins: Add support for installing local development binaries to `packer plugins install`. [GH-12855](https://togithub.com/hashicorp/packer/pull/12855) - core: Validate bucket name when using `hcp_packer_registry` block. [GH-12820](https://togithub.com/hashicorp/packer/pull/12820) - core: Update github.com/hashicorp/hcp-sdk-go from 0.83.0 to 0.85.0. [GH-12850](https://togithub.com/hashicorp/packer/pull/12850) [GH-12827](https://togithub.com/hashicorp/packer/pull/12827) ##### BUG FIXES: - core/hcp: HCP Packer build failures properly distinguish between incompatible plugins and general publishing errors. [GH-12854](https://togithub.com/hashicorp/packer/pull/12854) [GH-12835](https://togithub.com/hashicorp/packer/pull/12835) ### [`v1.10.1`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1101-January-30-2024) [Compare Source](https://togithub.com/hashicorp/packer/compare/v1.10.0...v1.10.1) ##### NOTES: - **HCP Packer** has been updated to follow an artifact-agnostic approach for publishing and consuming built artifacts. To best support the new approach, HCP Packer components within Packer have been updated to use the v2 nomenclature when publishing build metadata to HCP Packer. These changes include data source components for consuming HCP Packer stored artifacts. The data source components [hcp-packer-version](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-version) and [hcp-packer-artifact](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-artifact) respectively replace, the now deprecated, [hcp-packer-iteration](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-iteration) and [hcp-packer-image](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-image) component data sources. The added changes are compatible with all existing HCP Packer workflows and do not warrant any immediate template changes by the user. However, we encourage all HCP Packer users to visit the updated HCP Packer [CHANGELOG](https://developer.hashicorp.com/hcp/docs/changelog) to familiarize themselves with the nomenclature changes. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) [GH-12799](https://togithub.com/hashicorp/packer/pull/12799) - The `hcp-packer-image` data source has been deprecated, please use HCP Packer Artifact data source instead. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) - The `hcp-packer-iteration` data source has been deprecated, please use HCP Packer Version data source instead. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) - The contextual variable `packer.iterationID` has been deprecated, please use the new contextual variable `packer.versionFingerprint`. The iterationID represents the HCP Packer iteration that build metadata has been publish to, with this release an Iteration is now referred to as a Version and the ID has been replaced by the Version Fingerprint. [GH-12803](https://togithub.com/hashicorp/packer/pull/12803) ##### FEATURES: - **New Data source** `hcp-packer-artifact` The HCP Packer Artifact Data Source retrieves information about an artifact from the HCP Packer Registry. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) - **New Data source** `hcp-packer-version` The HCP Packer Version Data Source retrieves information about HCP Packer Version from the HCP Packer. Registry. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) ##### SECURITY: - core: Bump Go dependencies to address various CVEs. [GH-12777](https://togithub.com/hashicorp/packer/pull/12777) - core: Bump github.com/cloudflare/circl to address GHSA-9763-4f94-gfch. [GH-12781](https://togithub.com/hashicorp/packer/pull/12781) ##### IMPROVEMENTS: - core: Bump github.com/hashicorp/hcp-sdk-go from 0.79.0 to 0.81.0 [GH-12792](https://togithub.com/hashicorp/packer/pull/12792) [GH-12764](https://togithub.com/hashicorp/packer/pull/12764) - core: Update error messaging for HCP incompatible plugin builds. [GH-12800](https://togithub.com/hashicorp/packer/pull/12800) - core: Update HCP Packer integration to use HCP Packer v2 nomenclature, with this change HCP Packer iterations are now referred to as Versions, and Images are now referred to as Artifacts. The data sources for interacting with HCP Packer have been respectively renamed to align with the terminology of Version and Artifact. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794) [GH-12799](https://togithub.com/hashicorp/packer/pull/12799) - core: Update spacing for flags within Help text. [GH-12742](https://togithub.com/hashicorp/packer/pull/12742) [GH-12743](https://togithub.com/hashicorp/packer/pull/12743) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/hetznercloud/packer-plugin-hcloud). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM3Ny44IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
When specifying/installing plugins, a source URI is required for Packer to be able to locate or install a plugin to the local plugin hierarchy.
The plugin hierarchy is based on the plugin source, where each component in this hierarchy will become a directory.
In order to avoid sources with too many levels of nesting, causing a lot of mkdirs, we limit the number of sources to 16 in this commit, this should be long enough for most of our users.