Iterative dependency resolver #971

Menduist · 2021-12-21T11:27:21Z

This PR switch nimble from a recursive dependency solver to an iterative one. This allows the algorithm to be a lot smarter, since it has a global view of dependency tree. Ex:

Example with already installed dependencies

[~/nim-libp2p] $ nimble install -d
   Checking nimcrypto required by: user (>= 0.4.1)
   Checking https://github.com/ba0f3/dnsclient.nim required by: user (0.1.0)
   Checking bearssl required by: user (>= 0.1.4)
   Checking chronicles required by: user (>= 0.10.2)
   Checking chronos required by: user (>= 3.0.6)
   Checking metrics required by: user (any version)
   Checking secp256k1 required by: user (any version)
   Checking stew required by: user (#head), chronos (any version), secp256k1 (any version)
   Checking websock required by: user (any version)
   Checking unittest2 required by: bearssl (any version), chronos (#head)
   Checking testutils required by: chronicles (any version)
   Checking json_serialization required by: chronicles (any version)
   Checking httputils required by: chronos (any version), websock (>= 0.2.0)
   Checking asynctest required by: websock (>= 0.3.0 & < 0.4.0)
   Checking zlib required by: websock (any version)
   Checking unittest2 required by: bearssl (any version), chronos (#head), testutils (#head)
   Checking serialization required by: json_serialization (any version)
   Checking faststreams required by: serialization (any version)
     Info:  Dependency on unittest2@0.0.3 already satisfied
     Info:  Dependency on dnsclient@0.1.0 already satisfied
     Info:  Dependency on stew@0.1.0 already satisfied
     Info:  Dependency on testutils@0.4.2 already satisfied
     Info:  Dependency on nimcrypto@0.5.4 already satisfied
     Info:  Dependency on bearssl@0.1.5 already satisfied
     Info:  Dependency on asynctest@0.3.0 already satisfied
     Info:  Dependency on zlib@0.1.0 already satisfied
     Info:  Dependency on httputils@0.3.0 already satisfied
     Info:  Dependency on secp256k1@0.5.1 already satisfied
     Info:  Dependency on chronos@3.0.10 already satisfied
     Info:  Dependency on faststreams@0.3.0 already satisfied
     Info:  Dependency on metrics@0.0.1 already satisfied
     Info:  Dependency on serialization@0.1.0 already satisfied
     Info:  Dependency on json_serialization@0.1.0 already satisfied
     Info:  Dependency on chronicles@0.10.2 already satisfied
     Info:  Dependency on websock@0.1.0 already satisfied

Example with no dependency installed

$ nimble install libp2p
   Checking libp2p required by: user (any version)
Downloading https://github.com/status-im/nim-libp2p using git
   Checking nimcrypto required by: libp2p (>= 0.4.1)
Downloading https://github.com/cheatfate/nimcrypto using git
   Checking https://github.com/ba0f3/dnsclient.nim required by: libp2p (0.1.0)
Downloading https://github.com/ba0f3/dnsclient.nim using git
   Checking bearssl required by: libp2p (>= 0.1.4)
Downloading https://github.com/status-im/nim-bearssl using git
   Checking chronicles required by: libp2p (>= 0.10.2)
Downloading https://github.com/status-im/nim-chronicles using git
   Checking chronos required by: libp2p (>= 3.0.6)
Downloading https://github.com/status-im/nim-chronos using git
   Checking metrics required by: libp2p (any version)
Downloading https://github.com/status-im/nim-metrics using git
   Checking secp256k1 required by: libp2p (any version)
Downloading https://github.com/status-im/nim-secp256k1 using git
   Checking stew required by: libp2p (#head), chronos (any version), secp256k1 (any version)
Downloading https://github.com/status-im/nim-stew using git
   Checking websock required by: libp2p (any version)
Downloading https://github.com/status-im/nim-websock using git
   Checking unittest2 required by: bearssl (any version)
Downloading https://github.com/status-im/nim-unittest2 using git
   Checking testutils required by: chronicles (any version)
Downloading https://github.com/status-im/nim-testutils using git
   Checking json_serialization required by: chronicles (any version)
Downloading https://github.com/status-im/nim-json-serialization using git
   Checking httputils required by: chronos (any version), websock (>= 0.2.0)
Downloading https://github.com/status-im/nim-http-utils using git
   Checking https://github.com/status-im/nim-unittest2 required by: chronos (#head), testutils (#head)
Downloading https://github.com/status-im/nim-unittest2 using git
   Checking asynctest required by: websock (>= 0.3.0 & < 0.4.0)
Downloading https://github.com/markspanbroek/asynctest using git
   Checking zlib required by: websock (any version)
Downloading https://github.com/status-im/nim-zlib using git
   Checking serialization required by: json_serialization (any version)
Downloading https://github.com/status-im/nim-serialization using git
   Checking unittest2 required by: bearssl (any version), chronos (#head), testutils (#head), serialization (any version)
Downloading https://github.com/status-im/nim-unittest2 using git
   Checking faststreams required by: serialization (any version)
Downloading https://github.com/status-im/nim-faststreams using git
 Installing unittest2@0.0.3
  Success:  unittest2 installed successfully.
 Installing dnsclient@0.1.0
   Building dnsclient/dnsclient using c backend
  Success:  dnsclient installed successfully.
 Installing stew@0.1.0
  Success:  stew installed successfully.
 Installing testutils@0.4.2
   Building testutils/ntu using c backend
  Success:  testutils installed successfully.
 Installing nimcrypto@0.5.4
  Success:  nimcrypto installed successfully.
 Installing bearssl@0.1.5
  Success:  bearssl installed successfully.
 Installing asynctest@0.3.0
  Success:  asynctest installed successfully.
 Installing zlib@0.1.0
  Success:  zlib installed successfully.
 Installing httputils@0.3.0
  Success:  httputils installed successfully.
 Installing secp256k1@0.5.1
  Success:  secp256k1 installed successfully.
 Installing chronos@3.0.10
  Success:  chronos installed successfully.
 Installing faststreams@0.3.0
  Success:  faststreams installed successfully.
 Installing metrics@0.0.1
  Success:  metrics installed successfully.
 Installing serialization@0.1.0
  Success:  serialization installed successfully.
 Installing json_serialization@0.1.0
  Success:  json_serialization installed successfully.
 Installing chronicles@0.10.2
  Success:  chronicles installed successfully.
 Installing websock@0.1.0
  Success:  websock installed successfully.
 Installing libp2p@0.0.2
  Success:  libp2p installed successfully.

As a reminder, current nimble output to install libp2p

  Verifying dependencies for libp2p@0.0.2
 Installing nimcrypto@>= 0.4.1
Downloading https://github.com/cheatfate/nimcrypto using git
  Verifying dependencies for nimcrypto@0.5.4
 Installing nimcrypto@0.5.4
   Success: nimcrypto installed successfully.
 Installing https://github.com/ba0f3/dnsclient.nim@0.1.0
Downloading https://github.com/ba0f3/dnsclient.nim using git
  Verifying dependencies for dnsclient@0.1.0
 Installing dnsclient@0.1.0
   Building dnsclient/dnsclient using c backend
   Success: dnsclient installed successfully.
 Installing bearssl@>= 0.1.4
Downloading https://github.com/status-im/nim-bearssl using git
  Verifying dependencies for bearssl@0.1.5
 Installing unittest2@any version
Downloading https://github.com/status-im/nim-unittest2 using git
  Verifying dependencies for unittest2@0.0.3
 Installing unittest2@0.0.3
   Success: unittest2 installed successfully.
 Installing bearssl@0.1.5
   Success: bearssl installed successfully.
 Installing chronicles@>= 0.10.2
Downloading https://github.com/status-im/nim-chronicles using git
  Verifying dependencies for chronicles@0.10.2
 Installing testutils@any version
Downloading https://github.com/status-im/nim-testutils using git
  Verifying dependencies for testutils@0.4.2
 Installing https://github.com/status-im/nim-unittest2.git@#head
Downloading https://github.com/status-im/nim-unittest2.git using git
  Verifying dependencies for unittest2@#head
 Installing unittest2@#head
   Success: unittest2 installed successfully.
 Installing testutils@0.4.2
   Building testutils/ntu using c backend
   Success: testutils installed successfully.
 Installing json_serialization@any version
Downloading https://github.com/status-im/nim-json-serialization using git
  Verifying dependencies for json_serialization@0.1.0
 Installing serialization@any version
Downloading https://github.com/status-im/nim-serialization using git
  Verifying dependencies for serialization@0.1.0
 Installing faststreams@any version
Downloading https://github.com/status-im/nim-faststreams using git
  Verifying dependencies for faststreams@0.3.0
 Installing stew@any version
Downloading https://github.com/status-im/nim-stew using git
  Verifying dependencies for stew@0.1.0
 Installing stew@0.1.0
   Success: stew installed successfully.
      Info: Dependency on testutils@any version already satisfied
  Verifying dependencies for testutils@0.4.2
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
 Installing chronos@any version
Downloading https://github.com/status-im/nim-chronos using git
  Verifying dependencies for chronos@3.0.10
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
 Installing httputils@any version
Downloading https://github.com/status-im/nim-http-utils using git
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
 Installing httputils@0.3.0
   Success: httputils installed successfully.
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
 Installing chronos@3.0.10
   Success: chronos installed successfully.
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
 Installing faststreams@0.3.0
   Success: faststreams installed successfully.
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
 Installing serialization@0.1.0
   Success: serialization installed successfully.
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
 Installing json_serialization@0.1.0
   Success: json_serialization installed successfully.
 Installing chronicles@0.10.2
   Success: chronicles installed successfully.
      Info: Dependency on chronos@>= 3.0.6 already satisfied
  Verifying dependencies for chronos@3.0.10
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on httputils@any version already satisfied
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
 Installing metrics@any version
Downloading https://github.com/status-im/nim-metrics using git
  Verifying dependencies for metrics@0.0.1
      Info: Dependency on chronos@>= 2.6.0 already satisfied
  Verifying dependencies for chronos@3.0.10
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on httputils@any version already satisfied
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
 Installing metrics@0.0.1
   Success: metrics installed successfully.
 Installing secp256k1@any version
Downloading https://github.com/status-im/nim-secp256k1 using git
  Verifying dependencies for secp256k1@0.5.1
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on nimcrypto@any version already satisfied
  Verifying dependencies for nimcrypto@0.5.4
 Installing secp256k1@0.5.1
   Success: secp256k1 installed successfully.
 Installing stew@#head
Downloading https://github.com/status-im/nim-stew using git
  Verifying dependencies for stew@#head
 Installing stew@#head
   Success: stew installed successfully.
 Installing websock@any version
Downloading https://github.com/status-im/nim-websock using git
  Verifying dependencies for websock@0.1.0
      Info: Dependency on chronos@>= 3.0.0 already satisfied
  Verifying dependencies for chronos@3.0.10
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on httputils@any version already satisfied
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
      Info: Dependency on httputils@>= 0.2.0 already satisfied
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on chronicles@>= 0.10.2 already satisfied
  Verifying dependencies for chronicles@0.10.2
      Info: Dependency on testutils@any version already satisfied
  Verifying dependencies for testutils@0.4.2
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
      Info: Dependency on json_serialization@any version already satisfied
  Verifying dependencies for json_serialization@0.1.0
      Info: Dependency on serialization@any version already satisfied
  Verifying dependencies for serialization@0.1.0
      Info: Dependency on faststreams@any version already satisfied
  Verifying dependencies for faststreams@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on testutils@any version already satisfied
  Verifying dependencies for testutils@0.4.2
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
      Info: Dependency on chronos@any version already satisfied
  Verifying dependencies for chronos@3.0.10
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on httputils@any version already satisfied
  Verifying dependencies for httputils@0.3.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on https://github.com/status-im/nim-unittest2.git@#head already satisfied
  Verifying dependencies for unittest2@#head
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on stew@any version already satisfied
  Verifying dependencies for stew@0.1.0
      Info: Dependency on stew@>= 0.1.0 already satisfied
  Verifying dependencies for stew@0.1.0
 Installing asynctest@>= 0.3.0 & < 0.4.0
Downloading https://github.com/markspanbroek/asynctest using git
  Verifying dependencies for asynctest@0.3.0
 Installing asynctest@0.3.0
   Success: asynctest installed successfully.
      Info: Dependency on nimcrypto@any version already satisfied
  Verifying dependencies for nimcrypto@0.5.4
      Info: Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.1.5
      Info: Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.0.3
 Installing zlib@any version
Downloading https://github.com/status-im/nim-zlib using git
  Verifying dependencies for zlib@0.1.0
      Info: Dependency on stew@>= 0.1.0 already satisfied
  Verifying dependencies for stew@0.1.0
 Installing zlib@0.1.0
   Success: zlib installed successfully.
 Installing websock@0.1.0
   Success: websock installed successfully.
 Installing libp2p@0.0.2
   Success: libp2p installed successfully.

The current iterative algorithm is fairly simple, but effective. We start by building the dependency graph:

We build a list of things to install, and for each thing to install, we create a list of constraints (eg stew required by: user (#head)
We'll then process them one at a time, adding it's dependencies as constraints where required. If a package is missing, we'll download it (but not install it yet) to check it's dependencies. (eg, when processing chronos, we notice that it also requires stew, so we add it to stew constaints stew required by: user (#head), chronos (any version))
If we processed a package, but a later package adds an constraint incompatible with the version we chose during the processing, we simply add the incompatible version back to the list of things to process.
If two packages have incompatible constraints, we'll fail:

$ ../nimble/nimble install stew@#head stew@0.0.0
   Checking stew required by: user (#head), user (0.0.0)
Downloading https://github.com/status-im/nim-stew using git
       Tip: 3 messages have been suppressed, use --verbose to show them.
/home/tavon/Status/nimble/src/nimble.nim(1938) nimble
/home/tavon/Status/nimble/src/nimble.nim(1857) doAction
/home/tavon/Status/nimble/src/nimble.nim(696) install
/home/tavon/Status/nimble/src/nimble.nim(576) installIteration

    Error:  Cannot satisfy the dependencies on stew

When we've finished building the dep graph, we'll install every missing package (after their deps).

This is still WiP. I originally based my work on 0.13.1, and when porting it to master a lot of things broke. Currently fixing remaining issues.

The major thing missing from this algorithm is what python calls "backtracking ".
eg, if two have:

A@4.0: requires C@1.0
B@3.0: requires C@1.0
B@4.0: requires C@2.0

nimble install A@>2.0 B@>2.0 will not work, because the latest version of A & B are not compatible. But we could find out that using B@3.0 would solve the issue.
With the current nimble file format, that would imply downloading a lot of versions to find a compatible one, like python does, which is not ideal.
If a user stumble onto this, he can "manually" backtrack by asking for an older version of B manually.

Menduist · 2021-12-22T13:31:41Z

The only standing CI failure is:

    D:\a\nimble\nimble\src\nimble.exe --nimbleDir:D:\a\nimble\nimble\tests\nimbleDir lock -y
         Info:  Generating the lock file...
   Checking dep1 required by: user (> 0.1.0)
       Tip: 5 messages have been suppressed, use --verbose to show them.
D:\a\nimble\nimble\src\nimble.nim(1964) nimble
D:\a\nimble\nimble\src\nimble.nim(1925) doAction
D:\a\nimble\nimble\src\nimble.nim(1594) lock
D:\a\nimble\nimble\src\nimble.nim(622) installIteration

    Error:  Cannot satisfy the dependencies on dep1

    D:\a\nimble\nimble\tests\tlockfile.nim(267, 42): Check failed: output.processOutput.inLines(invalidDevelopDependenciesVersionsMsg(errors))
  [FAILED] cannot lock because develop dependency is out of range

It seems that the current master only tries develop versions installed, whereas this branch will try to find a compatible version from the package_list even if there is a develop file (and fails because no version > 0.1.0 exists)
The new behavior doesn't seem wrong to me, but maybe I'm missing something?

Apart from that, CI is green, should be ready for review

Menduist · 2021-12-22T14:01:49Z

src/nimble.nim

-  let dependencies = pkgInfo.processFreeDependencies(options).map(
-    pkg => pkg.toFullInfo(options)).toSeq
+  let
+    packageList = getInstalledPkgsMin(options.getPkgsDir(), options)
+    developDeps = processDevelopDependencies(pkgInfo, options)
+    installedInfo = installIteration(concat(packageList, developDeps), pkgInfo.requires, options)
+    dependencies =
+      block:
+        # To avoid warning on `map`
+        var res = toSeq(installedInfo.installed.values)
+        res.apply(x => x.toFullInfo(options))
+        res
+    dependenciesGraph = dependencies.buildLockFileDeps(installedInfo.dependencies, options)
+
  pkgInfo.validateDevelopDependenciesVersionRanges(dependencies, options)
-  var dependencyGraph = buildDependencyGraph(dependencies, options)
-
-  if currentDir.lockFileExists:
-    # If we already have a lock file, merge its data with the newly generated
-    # one.
-    #
-    # IMPORTANT TODO:
-    # To do this properly, an SMT solver is needed, but anyway, it seems that
-    # currently Nimble does not check properly for `require` clauses
-    # satisfaction between all packages, but just greedily picks the best
-    # matching version of dependencies for the currently processed package.
-
-    dependencyGraph = mergeLockedDependencies(pkgInfo, dependencyGraph, options)
-
-  let (topologicalOrder, _) = topologicalSort(dependencyGraph)
-  writeLockFile(dependencyGraph, topologicalOrder)
+  writeLockFile(dependenciesGraph, toSeq(dependenciesGraph.keys()).sorted)


I modified this bit because it was based on the PackageInfo.requires, but the requires is less useful than the dependency graph returned by installIteration.
The returned dependency graph will be fully resolved (eg, in require there could be https://github.com/status-im/nim-unittest2.git which will become unittest2 in the dependency graph)

I'm also not sure why the merge system was there? Instead this version will generate a predictable lockfile (since everything is sorted), so if one dependency change, only the lines of the dependency will be impacted.

Menduist · 2021-12-22T14:05:21Z

src/nimble.nim

+  let
+    depsGraph = toSeq(pkgInfo.lockedDeps.pairs).map(pair => (pair[0], pair[1].dependencies))
+    (order, _) = topologicalSort(depsGraph.toOrderedTable())
+    developModeDeps = getDevelopDependencies(pkgInfo, options)


The current master version relies on json object key ordering to install locked deps in the right order, but json doesn't give this guarantee afaik
I added a second sort topological sort here, to be safe. See also comment on lock(

Menduist added 18 commits December 20, 2021 17:56

Iterative install

a2b9a4e

More work

c20d0a1

Correctly add rev dep

acc6c14

Fix some develop issues

ebf4e4a

Add already statisfied message

cb32d7f

Greener CI

65e1bf6

Greaner ci

c7b0a8c

Fix develop revdep

b348ce7

use topology sorter

1a439e0

replace lockfile merging with a sort

ec084c7

remove resolveAlias

ce9c3aa

better lockfile handling

52a9ca1

Greener CI

0e471b7

fix test

4930f01

fix warnings

fa84a6f

fix locked dep install

c9ad0c8

fix comments

6434c20

sort lockfile deps

1c36dc3

Menduist marked this pull request as ready for review December 22, 2021 13:32

Menduist commented Dec 22, 2021

View reviewed changes

revert some useless changes

3231010

Menduist changed the title ~~[WiP] Iterative dependency resolver~~ Iterative dependency resolver Dec 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iterative dependency resolver #971

Iterative dependency resolver #971

Menduist commented Dec 21, 2021 •

edited

Menduist commented Dec 22, 2021 •

edited

Menduist Dec 22, 2021

Menduist Dec 22, 2021 •

edited

Iterative dependency resolver #971

Are you sure you want to change the base?

Iterative dependency resolver #971

Conversation

Menduist commented Dec 21, 2021 • edited

Menduist commented Dec 22, 2021 • edited

Menduist Dec 22, 2021

Choose a reason for hiding this comment

Menduist Dec 22, 2021 • edited

Choose a reason for hiding this comment

Menduist commented Dec 21, 2021 •

edited

Menduist commented Dec 22, 2021 •

edited

Menduist Dec 22, 2021 •

edited