Support versioned dependencies

[sersut]

🚧

Currently it's a pain if one wants to have different versions of softwares in different projects when using omnibus. This approach introduces a new DSL attribute on projects called software_version_pinning (looking for a better name) which can be used to pin the version of a software component at the project level.

Example usage of it would be to update chef.rb and chef-dk.rb in omnibus-chef repo as below:

# chef.rb in omnibus-chef
software_version_pinning = {
   "ruby" => "1.9.3-p484"
}

# chef-dk.rb in omnibus-chef
software_version_pinning = {
   "ruby" => "2.1.0-p121"
}

We've considered achieving this goal by introducing different software configurations with different names but it took us to dependency hell because of inter-software dependencies.

This PR aims to vet this approach and make sure we agree before moving further into it.

Note that this will likely be a breaking change because it changes the keys in the software_map from file names to the names specified in the software definitions. We will most likely couple this with another breaking change coming from @danielsdeleo about producing dmg files by default on Mac.

@lamont-granquist / @schisamo thoughts?

One question for you @schisamo: How would this impact the build speed improvements that we're bringing in?

[danielsdeleo]

Pinning on the exact version is gonna be a pain, I think. You'll have to update multiple places when you bump the patch version of ruby. It should be possible to get what we want by using some other piece of information than the version number. I'm not sure what the name would be, maybe "alternative", "collection", "tag", or something like that. Then in your project, you'd do:

# projects/chef-dk.rb

use_alternative("ruby", "2.1")

# other stuff

and in the software definition for ruby 2.1:

# software/ruby_2_1.rb

version "2.1.0p123"
alternative "2.1"

In omnibus itself, you'd do:

dependency.select! {|a| a.alternative == project.alternatives[dependency_name] }

Other than that, the approach makes sense to me.

[sersut]

@danielsdeleo I like the alternative better then software_version_pinning.

However we will still need to change the software_map works because it is based on the file names rather than the name specified in the software config.

Also do you think is there a better way of getting the name and version out of software DSL then the approach of using SoftwareDefinition sub-dsl that I've just pushed.

[sersut]

@schisamo This might be a good compromise IMO. LMK what you think?

[danielsdeleo]

@sersut I think some of the software configs mutate global state, like setting the version number of the omnibus project. Should make sure it's okay to evaluate all the config/software/*.rb files, and see what we need to do to fix it, if there are such cases.

[sersut]

Only place that needed change was the version-manifest.rb here:

chef/omnibus-software@be64ef3

[sdelano]

Just checking in here so that I get updates when you guys leave comments. I haven't yet read through it.

[sethvargo]

@sdelano did you know know that GitHub added a subscribe button now? It's hidden by the labels and stuff, but you can subscribe and unsubscribe at leisure 😄

[schisamo]

I've had some conversation with a number of y'all about this change. I'll try to summarize my findings to the group:

1. Being forced to eat the whole omnibus-software bowl in order to get an update for a single software definition is less than ideal.
2. It would be nice to have the ability for 1..n versions of a software definition to exist concurrently. A concrete example of this would be software definitions for Ruby 1.9.2 and Ruby 2.1. An individual Omnibus project should be able to peg/lock to a single version for all items in it's dependency tree (including transitive deps). Bonus points if this locking understood MAJOR.MINOR.PATCH semantics.
3. Sometimes a software definition has associated configuration code (including things like service definitions). This configuration code currently exists as part of the project repo and is executed with PROJECT-ctl reconfigure. It would be nice for multiple Omnibus projects to leverage the same configuration code in a composable way.

So while keeping points 1-3 in mind, my questions/comments to the group are as follows:

• Do we feel introducing alternative or software_version_pinning to the DSL (and thus public API) is the correct course of action?
• Would we be OK with the workaround of copying software definitions into individual Omnibus projects? This is a little hacky but at lease the hack is isolated to an individual Omnibus project as opposed to the framework.
• We should also begin defining a solution that addresses 1-3. This should be a real project with a requirements document and feedback process.

[danielsdeleo]

I don't agree with the bonus points section of requirement 2. The use case we have right now is that we want to build chef-client with ruby 1.9.3pWHATEVER and chef-dk with 2.1.XpWHATEVER. The problem with implementing the switching in terms of versioning is:

• We don't want to specify exact versions, because then we have to change multiple things just to get a patch release in.
• Version sorting might not be reliable, depending on the upstream's versioning policy. Ruby changed to semver from weird-4-version-number-semi-semver. OpenSSL is weird. Other projects are probably also weird.
• There might be other reasons to change software definitions, like building optional components or not. We can get this extra generality with no extra code cost if we use some kind of tag instead of the software version to decide which software definition to use.

[lamont-granquist]

@schisamo #1 is actually a bit orthogonal to this. I obviously want to be able to do that, but it doesn't help with this problem since we want to share chefdk and chef config in the same project repository (since chefdk is mostly a superset of chef). We're going to have to pull in a single omnibus-software repo for that. We want to launch with chedk on high revs of ruby and stuff so that means forking config in omnibus-software, which is also bad -- I went down that road and it was a lot of copypasta, and it got somewhat ugly based on software definitions that suck in rubygems and ruby themselves and all need to be changed to point at rubygems-2.2 and ruby-2.1 and need to be forked. Copying software definitions into omnibus-chef doesn't really get away from this it just contains the damage away from the server (but also forks things like the libyaml patch that we can immediately reuse today and we'd have to copy it into omnibus-chef in the future).

[lamont-granquist]

@danielsdeleo I'm not wild about having config/software/{ruby_1.9.rb,ruby_2.1.rb}, etc all with mostly identical config.

[danielsdeleo]

@lamont-granquist I think you either need to eat the duplication between software definition alternatives, or else invent some other sharing mechanism, e.g. (total SWAG here),

inherit_definition("ruby")
version "2.1.0p2342"

patch "ruby_210_only_patch"
# etc.

While it does sound nice to have the project be able to reach in and just flip the version, in practice there are things like patches that don't apply to newer versions (this happened to me in https://github.com/danielsdeleo/omnibus-rubies/blob/master/config/software/ruby-200.rb#L77-96 and also in the patch I recently made to ncurses, which includes the version number in the source tree).

[sersut]

Just chatted with @schisamo and @lamont-granquist on this. A few things we want to make sure are:

• We should come up with a solution that will work for our needs (chef-dk packaging and other projects' needs) without a large cost.
• We should not introduce DSL changes to omnibus that we might regret later on.
• We don't want to copy software definition files around.

With these in mind I think the idea @lamont-granquist and I came up with is going to work in the short term for us:

The idea is to be able to inject a version to a software definition at the project level. We can continue to use the same software definition for ruby but it will be able to build more than one version of itself. Eg:

# in omnibus-chef/config/projects/chef-dk.rb
override("ruby", { "version" => "2.1.0" } )

# in omnibus-software/config/software/ruby.rb
name "ruby"
version "1.9.3-p484"

package_signatures = {
  "1.9.3-p484" => '8ac0dee72fe12d75c8b2d0ef5d0c2968',
  "2.1.0" => 'xxyyaazanaalhsdflajhdf'
}

source :url => "http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-#{version}.tar.gz",
       :md5 => package_signatures[version]

This way I think we can pin software versions differently at the project level. Also bumping a version of a software will not mean bumping the version of the software in all projects. A few things about the approach:

• We can make this override functionality only applicable to version or generic so that any attribute can be overridden.
• Maybe we can use the existing @overrides functionality for this purpose. On first look I don't think we can because it is initiated with a separate config file dropped out of the band of the main repos.
• Maybe we can change the @overrides functionality for this purpose. I don't think there is anyone using this.

Thoughts on this? @sdelano I think with this and tagging the project definitions with the version being released we can achieve reproducible builds.

[lamont-granquist]

Yeah what i'd like would be what @sersut suggested with conditional patching like:

patch "ruby_19_patch" if version.to_f == 1.9

It looks like the project is passed into Omnibus::Software.new() so we should be able to patch that class to override the version of the software based on inspecting the project, then we should be able to use that...

Heh, actually and there's already an @override_version in the software definition:

https://github.com/opscode/omnibus-ruby/blob/master/lib/omnibus/software.rb#L150-L156

That is almost exactly what I want, except that I think the overrides are global for a repo, whereas we want to have one omnibus-chef repo with different versions for the chefdk and chef projects in it. Just fix that so that you can suck override_version out of the project DSL as well.

[schisamo]

@sersut I like the approach! Maybe we should turn the software version DSL attribute into a block and make it additive? This would create easy to read code like:

# in omnibus-software/config/software/ruby.rb
name 'ruby'
default_version '1.9.3' 

version '1.9.3' do
  source 'http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p484.tar.gz'
  md5 '8ac0dee72fe12d75c8b2d0ef5d0c2968'
end

version '2.1.0' do
  source 'http://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.0.tar.gz'
  md5 '9e6386d53f5200a3e7069107405b93f7'
end

# in omnibus-chef/config/projects/chef-dk.rb
override 'ruby', version: '2.1.0'
override 'foo', version: 'bar'

I also agree with @lamont-granquist that it would be nice to make the project DSL override work in conjunction with the omnibus.overrides file. We could even make the file project specific: PROJECT_NAME.overrides.

[lamont-granquist]

I'm not super wild with adding project-specific stuff to the overrides or having a project-specific override file. I think it should probably go into the project DSL itself to keep it all in one place. Otherwise you've got a bit of a spooky-action-at-a-distance where you're doing per-project stuff but you go look at the project file and its not there, and you pull your hair out until someone mentions the project-specific override...

[sersut]

I like making version a block @schisamo. Since md5 and source is pretty standard across I think it would make sense. I'll get it done.

Also I'm on the @lamont-granquist's camp about the overrides file. I'm not sure if having an override file outside the git repo is a good practice. Leaning towards not changing the @overrides since I'm not sure what the best way is and also to decrease the scope of the change.

LMK if this plan works.

[schisamo]

@sersut :thumsbup: up. Looking forward to reviewing the next round of code!

I think the overrides file stuff should mostly keep working as the overrides are applied after project definitions have been evaluated:
https://github.com/opscode/omnibus-ruby/blob/master/lib/omnibus.rb#L222-L224

[sersut]

@lamont-granquist will be posting a new PR with the design we've agreed on above.

Nice discussion folks... Gracias :)