Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

addrs: limit source component length to 16 #12962

Merged
merged 2 commits into from
May 14, 2024

Conversation

lbajolet-hashicorp
Copy link
Contributor

When specifying/installing plugins, a source URI is required for Packer to be able to locate or install a plugin to the local plugin hierarchy.

The plugin hierarchy is based on the plugin source, where each component in this hierarchy will become a directory.

In order to avoid sources with too many levels of nesting, causing a lot of mkdirs, we limit the number of sources to 16 in this commit, this should be long enough for most of our users.

@lbajolet-hashicorp lbajolet-hashicorp added the tech-debt Issues and pull requests related to addressing technical debt or improving the codebase label May 9, 2024
@lbajolet-hashicorp lbajolet-hashicorp requested a review from a team as a code owner May 9, 2024 15:34
Copy link
Contributor

@nywilken nywilken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of suggestions. If a user drops a binary in the plugins directory with more then 16 directories do we prevent it from loading?

Is there a way to validate the address of the discovered plugin so that we don't need to track the depth of the filepath.Walk?

@lbajolet-hashicorp
Copy link
Contributor Author

That's a good call for plugins that are completely manually installed, we don't track the depth for filepath.Walk, but iirc there's something built-in for limiting how deep we visit the filesystem?
I'll see what I can do, alternatively we can derive the source depth from the path of the binary

@lbajolet-hashicorp lbajolet-hashicorp force-pushed the cap_length_source_16 branch 3 times, most recently from 63830fe to cecdb79 Compare May 9, 2024 21:14
Copy link
Contributor

@nywilken nywilken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionality wise this looks good and handles both cases of installing or lading a plugin that is more the 16 paths deep. The error messaging doesn't display well so we need to improve that.

When specifying/installing plugins, a source URI is required for Packer
to be able to locate or install a plugin to the local plugin hierarchy.

The plugin hierarchy is based on the plugin source, where each component
in this hierarchy will become a directory.

In order to avoid sources with too many levels of nesting, causing a lot
of mkdirs, we limit the number of sources to 16 in this commit, this
should be long enough for most of our users.
@lbajolet-hashicorp lbajolet-hashicorp force-pushed the cap_length_source_16 branch 2 times, most recently from ee7a526 to 4f6299e Compare May 10, 2024 20:53
@nywilken
Copy link
Contributor

The updated error for this particular case looks good. But we need to clean up the other error message by adding a newline as they can be a bit long, maybe even repetative since the name of the plugin is already displayed within the "invalid source string ..." message.

~>  packer plugins install --path packer-plugin-amazon https://github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups
Invalid source string "https://github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups": The provided source URL "https://github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups" is invalid. The following errors have been discovered:
* A source URL must not contain a scheme (e.g. https://).

A valid source looks like "github.com/hashicorp/happycloud"

[go-1.21.8] [0] dev@dev-GQPG9GPY75 in ~/Development/packer-plugin-amazon/ on main
~>  packer plugins install --path packer-plugin-amazon github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups
Invalid source string "github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups": The source URL must have at most 16 components, and the one provided has 17.
This is unsupported by Packer, please consider using a source that has less components to it.
If this is a blocking issue for you, please open an issue to ask for supporting more components to the source URI.

@lbajolet-hashicorp
Copy link
Contributor Author

@nywilken I pushed a change to the error messages, trying to make lines shorter and removing mentions of the source string since it'll be in the error message, let me know what you think

@nywilken
Copy link
Contributor

Beautiful

~>  packer plugins install --path packer-plugin-amazon https://github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups
Invalid source string "https://github.com/hashicorp/azr/a/b/c/d/e/f/g/h/i/j/k/l/m/hashicups": The provided source URL is invalid.
The following errors have been discovered:
* A source URL must not contain a scheme (e.g. https://).

A valid source looks like "github.com/hashicorp/happycloud"

Copy link
Contributor

@nywilken nywilken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you might of rebase with plugin install docs. I would just spot check for consistency between your branches.

@lbajolet-hashicorp
Copy link
Contributor Author

I have indeed; will cleanup now

The ParsePluginSource function can be invoked from either a HCL2 context
(when parsing a required_plugins block), or from the command-line
itself.

While in the first context a hcl.Diagnostics is coherent, in case the
source to parse is a command-line argument, for example when installing
or removing a plugin, the error message cannot have an HCL context,
leading to errors that are incorrectly prefixed by a <nil> string dure
to the lack of a reference to attach the diagnostic to.

Therefore, in order to fix this behaviour, the logic that parses plugin
sources now returns an error, and attaching the error to an HCL subject
is done independently, if needed.
@lbajolet-hashicorp lbajolet-hashicorp merged commit c7f2508 into main May 14, 2024
11 checks passed
@lbajolet-hashicorp lbajolet-hashicorp deleted the cap_length_source_16 branch May 14, 2024 18:47
jooola referenced this pull request in hetznercloud/packer-plugin-hcloud Jun 3, 2024

Verified

This commit was signed with the committer’s verified signature.
philip-peterson Philip Peterson
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [hashicorp/packer](https://togithub.com/hashicorp/packer) | minor |
`"1.10.0"` -> `1.11.0` |

---

### Release Notes

<details>
<summary>hashicorp/packer (hashicorp/packer)</summary>

###
[`v1.11.0`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1110-2024-05-31)

[Compare
Source](https://togithub.com/hashicorp/packer/compare/v1.10.3...v1.11.0-beta)

##### NOTES:

-   A LICENSE.txt file has been added to the Packer release artifacts.
    [GH-12931](https://togithub.com/hashicorp/packer/pull/12931)
    [GH-12940](https://togithub.com/hashicorp/packer/pull/12940)

- **Breaking Change**: Support for loading single-component plugins has
been removed from Packer.
[GH-12785](https://togithub.com/hashicorp/packer/pull/12785)

-   **Breaking Change**: Support for loading plugin binaries following
the naming convention of packer-plugin-name has been dropped. Packer
will now only load
plugins stored under the [Packer plugin
directory](https://developer.hashicorp.com/packer/docs/configure#packer-s-plugin-directory)
using the expected namespaced
directory and CHECKSUM files. This change drops support for loading
plugin
binaries in Packer's executable directory or a template's current
working
directory. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828)

```shell
/Users/dev/.packer.d/plugins
└── github.com
    └── hashicorp
        └── happycloud
            ├── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64
            └── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64_SHA256SUM
```

##### FEATURES:

- core: Add `-ignore-prerelease-plugins` flag to disable the use of
development
plugin binaries for the `build` and `validate` commands development
plugin
binaries. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828)
    [GH-12882](https://togithub.com/hashicorp/packer/pull/12882)
- Packer users can now track Packer version and plugin versions used for
each
    build artifact in HCP Packer.
    [GH-12866](https://togithub.com/hashicorp/packer/pull/12866)
-   hcl2: add textencodebase64 and textdecodebase64 funcs
For feature parity with Terraform, and since having access to strings
encoded in something that is not UTF-8 is required in some cases, we add
both the textencodebase64 and textdecodebase64 functions to HCL2
templates.
Please note these functions return base64 encoded byte slices because of
how
    cty/hcl defines strings (NFC normalised, UTF-8 encoded).
    [GH-12997](https://togithub.com/hashicorp/packer/pull/12997)

##### SECURITY:

-   Bump github.com/go-jose/go-jose/v3 to address GHSA-c5q2-7r4c-mv6g.
    [GH-12880](https://togithub.com/hashicorp/packer/pull/12880)
-   Bump golang.org/x/net to v0.24.0 to address GO-2024-2687.
    [GH-12924](https://togithub.com/hashicorp/packer/pull/12924)

##### IMPROVEMENTS:

-   core: Bump github.com/hashicorp/hcp-sdk-go from 0.90.0 to 0.96.0.
    [GH-12935](https://togithub.com/hashicorp/packer/pull/12935)
    [GH-12942](https://togithub.com/hashicorp/packer/pull/12942)
    [GH-12960](https://togithub.com/hashicorp/packer/pull/12960)
    [GH-12979](https://togithub.com/hashicorp/packer/pull/12979)
- core: Bump github.com/hashicorp/packer-plugin-sdk from 0.5.2 to 0.5.3
    [GH-12932](https://togithub.com/hashicorp/packer/pull/12932)
-   core: Bump go-getter/v2 from 2.2.1 to v2.2.2
    [GH-12988](https://togithub.com/hashicorp/packer/pull/12988)
-   datasource/http: don't error on 2xx code
Previous versions of Packer only supported 200 as a success case for the
http
datasource. This change makes any status code from 200 to 299
successful.
    [GH-12989](https://togithub.com/hashicorp/packer/pull/12989)
- core: Move to predictable plugin loading schema - Packer will now only
load
plugins stored under the [Packer plugin
directory](https://developer.hashicorp.com/packer/docs/configure#packer-s-plugin-directory)
using the expected namespaced
    directory and CHECKSUM files.
    [GH-12828](https://togithub.com/hashicorp/packer/pull/12828)
-   core: Remove support loading single-component plugins.
    [GH-12785](https://togithub.com/hashicorp/packer/pull/12785)
- core: Rename internal `packer plugin` command to `packer execute` to
avoid user confusion with
    the plugins subcommand.
    [GH-12865](https://togithub.com/hashicorp/packer/pull/12865)
- core: Packer now considers development binaries when evaluating plugin
version constraints. This work allows users to use binaries with
versions
reported as "x.y.z-dev" to be used with the Packer `required_plugins`
    block. [GH-12828](https://togithub.com/hashicorp/packer/pull/12828)
- core: Packer now supports local paths to plugins for the `packer
plugins remove`
    command. This addition makes it possible to pipe commands like
`packer plugins installed` with it for speedy cleanup of installed
plugins.
    [GH-12886](https://togithub.com/hashicorp/packer/pull/12886)
- core: Relax Packer source address URIs within the `required_plugins`
block to
support the installation of local plugin binary using a custom or
internal
source address (e.g. mycompany.com/plugins/happyorg/happycloud). Remote
installation using `packer init` or `packer plugins install` does not
    support non-GitHub source URIs. Users using alternative hosts must
    install plugins manually using `packer plugins install --path`.
[GH-12911](https://togithub.com/hashicorp/packer/pull/12911),
\[[GH-12962](https://togithub.com/hashicorp/packer/issues/12962)]
([https://github.com/hashicorp/packer/pull/12962](https://togithub.com/hashicorp/packer/pull/12962))
- core: Remote plugins installed containing an internal version number
that
    differs from the version number within the binary name can lead to
confusion when tracking Packer plugin version information. To help track
such discrepancies in the plugin version, `packer init` and `packer
plugin
install` have been updated to reject installation of such plugins.
    1.0.0-dev). Users are encouraged to notify plugin maintainers of any
    version mismatches.
[GH-12915](https://togithub.com/hashicorp/packer/pull/12915),
[GH-12953](https://togithub.com/hashicorp/packer/pull/12953),
[GH-12972](https://togithub.com/hashicorp/packer/pull/12972)
-   core: don't load plugins with metadata in name
To avoid confusion with multiple plugins that report the same effective
version,
    plugins installed need to have no metadata in their name.
When installed through Packer commands, the metadata is scrubbed from
the name of the
installed plugin binary, but manually it may still be possible, so we
enforce
    that scrubbing at load-time as well.
    [GH-12980](https://togithub.com/hashicorp/packer/pull/12980)
-   core: Error when multiple paths are specified for PACKER_PLUGIN_PATH
Since Packer 1.11 removed the capability for PACKER_PLUGIN_PATH to
specify
multiple directories separated by `:` or `;` (depending on the
platform), we
are explicitly erroring when this is discovered, with suggestions as to
how
    to fix the problem.
    [GH-12967](https://togithub.com/hashicorp/packer/pull/12967)
- core: Version metadata support for plugins. Plugins may now formally
have metadata
in their versions, Packer supports it, and applies the semver
recommendations on
them, i.e. they are ignored for comparison/sorting purposes, but allowed
for
    adding extra information about a plugin.
    [GH-12888](https://togithub.com/hashicorp/packer/pull/12888)

Given the specified version constraint only versions greater than or
equal to 1.1.0 will be considered.

```hcl
amazon = {
  source = "github.com/hashicorp/amazon"
  version = ">= 1.1.0"
}
```

If a development binary is installed, Packer will use it if:

1.  It is the highest compatible version installed.
2. There is no final plugin version with the same version number
installed alongside it.

```shell
/Users/dev/.packer.d/plugins
└─ github.com
   └─ hashicorp
    	└── amazon
          ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64
          ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64_SHA256SUM
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64
          └── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM
```

Version 1.1.1-dev of the Amazon plugin will match the specified version
constraint and be used for executing the Packer build.

If, however, a 1.1.1 release version of the plugin is available, it will
have precedence over the development binary.

```shell
/Users/dev/.packer.d/plugins
└─ github.com
   └─ hashicorp
    	└── amazon
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM
          ├── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64
          └── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64_SHA256SUM
```

##### BUG FIXES:

- core: fix plugin version ordering to not be lexicographic. This fixes
an issue
with how plugins are discovered by Packer, and ensures proper version
ordering.
This means that with this change, versions that are semantically
greater,
    but lexicographically inferior will be loaded.
Ex: 1.0.9 vs. 1.0.10; `1.0.9 > 1.0.10` lexicographically, but
semantically
    `1.0.10 > 1.0.9`
- core/hcp: fix potential race condition when storing plugin details to
the HCP
    Packer metadata storage map.
    [GH-12936](https://togithub.com/hashicorp/packer/pull/12936)
-   core: fix plugin listing on Windows
    This fix addresses bugs present in the alpha releases of 1.11, where
    the discovery of Windows binaries were not matching against the
    filename extension (.exe).
    [GH-12981](https://togithub.com/hashicorp/packer/pull/12981)

###
[`v1.10.3`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1103-April-22-2024)

[Compare
Source](https://togithub.com/hashicorp/packer/compare/v1.10.2...v1.10.3)

##### NOTES

-   A LICENSE.txt file has been added to the Packer release artifacts.
    [GH-12931](https://togithub.com/hashicorp/packer/pull/12931)

##### FEATURES

- Packer users can now track Packer version and plugin versions used for
each
    build artifact in HCP Packer.
    [GH-12891](https://togithub.com/hashicorp/packer/pull/12891)
    [GH-12910](https://togithub.com/hashicorp/packer/pull/12910)
    [GH-12923](https://togithub.com/hashicorp/packer/pull/12923)
    [GH-12926](https://togithub.com/hashicorp/packer/pull/12926)

##### SECURITY

-   Bump github.com/go-jose/go-jose/v3 to address GHSA-c5q2-7r4c-mv6g.
    [GH-12928](https://togithub.com/hashicorp/packer/pull/12928)
-   Bump golang.org/x/net to v0.24.0 to address GO-2024-2687.
    [GH-12924](https://togithub.com/hashicorp/packer/pull/12924)

##### BUG FIXES

- cmd/fmt: Display information error when Packer `fmt` fails due to HCL2
parsing error.
[GH-12870](https://togithub.com/hashicorp/packer/pull/12870)

###
[`v1.10.2`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1102-March-6-2024)

[Compare
Source](https://togithub.com/hashicorp/packer/compare/v1.10.1...v1.10.2)

##### NOTES:

- Continuing the work in in Packer v1.10.0 we introduced the ability to
install
    a locally sourced plugin using packer `plugins install --path`, this
release extends support to development plugin binaries - binaries that
report "dev" as part of their plugin version. Instead of manually
placing a
downloaded binary into the executable or current working directory we
encourage you to run the command `packer plugins install –path <path-to-
downloaded-extracted-binary> github.com/hashicorp/happycloud` to install
    the binary into a Packer compatible path.
    [GH-12855](https://togithub.com/hashicorp/packer/pull/12855)

##### IMPROVEMENTS:

- cmd/plugins: Add support for installing local development binaries to
`packer plugins install`.
    [GH-12855](https://togithub.com/hashicorp/packer/pull/12855)
-   core: Validate bucket name when using `hcp_packer_registry` block.
    [GH-12820](https://togithub.com/hashicorp/packer/pull/12820)
-   core: Update github.com/hashicorp/hcp-sdk-go from 0.83.0 to 0.85.0.
    [GH-12850](https://togithub.com/hashicorp/packer/pull/12850)
    [GH-12827](https://togithub.com/hashicorp/packer/pull/12827)

##### BUG FIXES:

- core/hcp: HCP Packer build failures properly distinguish between
incompatible
    plugins and general publishing errors.
    [GH-12854](https://togithub.com/hashicorp/packer/pull/12854)
    [GH-12835](https://togithub.com/hashicorp/packer/pull/12835)

###
[`v1.10.1`](https://togithub.com/hashicorp/packer/blob/HEAD/CHANGELOG.md#1101-January-30-2024)

[Compare
Source](https://togithub.com/hashicorp/packer/compare/v1.10.0...v1.10.1)

##### NOTES:

- **HCP Packer** has been updated to follow an artifact-agnostic
approach for
publishing and consuming built artifacts. To best support the new
approach, HCP Packer components within Packer have
been updated to use the v2 nomenclature when publishing build metadata
to
HCP Packer. These changes include data source components for consuming
HCP
Packer stored artifacts. The data source components
[hcp-packer-version](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-version)
and
[hcp-packer-artifact](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-artifact)
respectively replace, the now deprecated,

[hcp-packer-iteration](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-iteration)
and
[hcp-packer-image](https://developer.hashicorp.com/packer/docs/datasources/hcp/hcp-packer-image)
component data sources. The added changes are compatible with all
existing HCP Packer workflows and do not warrant any
immediate template changes by the user. However, we encourage all HCP
Packer users to visit the updated
HCP Packer
[CHANGELOG](https://developer.hashicorp.com/hcp/docs/changelog) to
familiarize themselves with the nomenclature changes.
[GH-12794](https://togithub.com/hashicorp/packer/pull/12794)
[GH-12799](https://togithub.com/hashicorp/packer/pull/12799)
- The `hcp-packer-image` data source has been deprecated, please use HCP
Packer
    Artifact data source instead.
    [GH-12794](https://togithub.com/hashicorp/packer/pull/12794)
- The `hcp-packer-iteration` data source has been deprecated, please use
HCP
    Packer Version data source instead.
    [GH-12794](https://togithub.com/hashicorp/packer/pull/12794)
- The contextual variable `packer.iterationID` has been deprecated,
please use
the new contextual variable `packer.versionFingerprint`. The iterationID
represents the HCP Packer iteration that build metadata has been publish
to, with this release an Iteration is now referred to as a Version and
the
    ID has been replaced by the Version Fingerprint.
    [GH-12803](https://togithub.com/hashicorp/packer/pull/12803)

##### FEATURES:

- **New Data source** `hcp-packer-artifact` The HCP Packer Artifact Data
Source
retrieves information about an artifact from the HCP Packer Registry.
    [GH-12794](https://togithub.com/hashicorp/packer/pull/12794)
- **New Data source** `hcp-packer-version` The HCP Packer Version Data
Source
    retrieves information about HCP Packer Version from the HCP Packer.
Registry. [GH-12794](https://togithub.com/hashicorp/packer/pull/12794)

##### SECURITY:

- core: Bump Go dependencies to address various CVEs.
[GH-12777](https://togithub.com/hashicorp/packer/pull/12777)
- core: Bump github.com/cloudflare/circl to address GHSA-9763-4f94-gfch.
[GH-12781](https://togithub.com/hashicorp/packer/pull/12781)

##### IMPROVEMENTS:

-   core: Bump github.com/hashicorp/hcp-sdk-go from 0.79.0 to 0.81.0
    [GH-12792](https://togithub.com/hashicorp/packer/pull/12792)
    [GH-12764](https://togithub.com/hashicorp/packer/pull/12764)
-   core: Update error messaging for HCP incompatible plugin builds.
    [GH-12800](https://togithub.com/hashicorp/packer/pull/12800)
- core: Update HCP Packer integration to use HCP Packer v2 nomenclature,
with
this change HCP Packer iterations are now referred to as Versions, and
Images are now referred to as Artifacts. The data sources for
interacting
with HCP Packer have been respectively renamed to align with the
terminology
    of Version and Artifact.
    [GH-12794](https://togithub.com/hashicorp/packer/pull/12794)
    [GH-12799](https://togithub.com/hashicorp/packer/pull/12799)
-   core: Update spacing for flags within Help text.
    [GH-12742](https://togithub.com/hashicorp/packer/pull/12742)
    [GH-12743](https://togithub.com/hashicorp/packer/pull/12743)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/hetznercloud/packer-plugin-hcloud).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM3Ny44IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
tech-debt Issues and pull requests related to addressing technical debt or improving the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants