bootstrapping VMs and Linux containers

[slivingston]

closes #33

The changes here are under the contrib/ directory, so committing directly to master branch may have been welcome. Nonetheless I opened this pull request to create a place for discussion and, possibly, edits before merging changes.

I decided not to create scripts that launch instances on AWS or GCP because doing so would require API keys for the respective service. Getting API keys is not much more difficult than launching a virtual machine instance and uploading vm-bootstrap.sh to it. Because vm-bootstrap.sh is intended to run without interaction and is agnostic about the cloud provider, I argue that it is a good solution to simply tell users to create a VM however they prefer and to run vm-bootstrap.sh from it.

Both vm-bootstrap.sh and tulip-in-a-box.py have been demonstrated on

• Google Cloud Platform (https://cloud.google.com/) GCE instance running Debian 9 (stretch) GNU/Linux, and
• AWS (https://aws.amazon.com) EC2 instance type t2.small running Ubuntu 16.04 LTS (xenial) (in particular AMI ID ami-cd0f5cb6).

Notes about trying vm-bootstrap.sh yourself

The final step of vm-bootstrap.sh is to run tulip-in-a-box.py from master branch. Therefore, it will not succeed until this PR is merged. Meanwhile, you can manually copy tulip-in-a-box.py into the VM that you are using and proceed from there.

Notes about trying the Dockerfile yourself

If you have a local installation of Docker, it should suffice to checkout the vm branch and

cd contrib
docker build -t tulip:latest .

Beware that the Docker image debian:stable will be fetched if you do not already have it; the download size might be too large for very low bandwidth connections.

As described in the documentation in Dockerfile, you can use the resulting image through an interactive shell, e.g.,

docker run -i -t tulip:latest
source tulip-control/PYt/bin/activate
python -c 'import tulip; print(tulip.__version__)'

Related work

Vagrantfile

This pull request obsoletes contrib/Vagrantfile because vm-bootstrap.sh and tulip-in-a-box.py achieve the same ends as a shell script that was embedded within contrib/Vagrantfile. If someone wants to use Vagrant to install TuLiP from source code, then she can create a basic Vagrantfile using vagrant init and then modify it to read and run vm-bootstrap.sh.

nessainstall

The script and instructions in contrib/nessainstall/ address the use-case of building dependencies and installing when sudo capabilities are not available. It is complementary to the work from this pull request.

[slivingston]

Note that Docker is not the only Linux container based environment where tulip-in-a-box.py can be used. E.g., I have used it on an Ubuntu 14.04 image managed by LXD.

Depending on how you configure the LXD daemon, the tar command from extern/get-lily.sh might fail because it cannot preserve owner permissions. One workaround is to add the switch --no-same-owner in extern/get-lily.sh

diff --git a/extern/get-lily.sh b/extern/get-lily.sh
index 6bfcecc..df8fccb 100755
--- a/extern/get-lily.sh
+++ b/extern/get-lily.sh
@@ -1,4 +1,4 @@
 curl -ksO https://www.iaik.tugraz.at/content/research/opensource/lily/lily-2008-06-19.tar.gz -o lily-200
 echo '2442e15223e116a4ebf6fbab42646ee64111f286458b4393f18ac7b1a9e19c952bba91f548e0e9c323d50f29e639a5089b
 shasum -a 512 -c -
-tar xzf lily-2008-06-19.tar.gz
+tar --no-same-owner -xzf lily-2008-06-19.tar.gz

[johnyf]

Running the image with docker lands me within ~/tulip-control, i.e.,

docker run -i -t tulip:latest
source PYt/bin/activate
python -c 'import tulip; print(tulip.__version__)'

where tulip-control has been removed compared to the OP.

[johnyf]

I created an image from the Dockerfile and run it from an pre-existing Docker installation, with no effort dedicated to this PR. This witnesses the simplicity of the proposed approach. I suggest the following:

1. To move the files contrib/Dockerfile, contrib/tulip-in-a-box.py and contrib/vm-bootstrap.sh to a dedicated directory, for example named virtual_machine (thus not restricting its scope to Docker images, or docker if deemed more appropriate, or container--perhaps less obvious a name).
   Motivation: structure contrib/ to group related files together.
   The file nonroot-vagrant-instructions.md could be moved there, though I am unsure whether it belongs there.

2. Add a minimal contrib/virtual_machine/README.md, the commands listed in the OP (and the Dockerfile) seem sufficient. Perhaps a link or two to more information about Docker, and a note that the image file will be stored somewhere else, so that users won't wonder why nothing changed in the current directory after running docker build.

3. Mention in the README that docker build can take an hour or more.

4. Mention in the README that PyYAML is required (I couldn't find any mention, so I added on in the script's docstring), or point to a requirements.txt file. Using a dedicated directory allows for creating requirements files for the script and other utilities that we may add to the image (e.g., IPython as remarked below).

5. Is the nonroot-vargant-instructions.md still needed?

6. Could consider publishing the resulting image to Docker Hub.

7. May be worth installing ipython in the virtualenv, though that diverges from the core purpose of installing tulip, and so is ill-defined (i.e., if we start installing extra things that aren't dependencies, then where should we stop?). Also, it diverges from the design of tulip-in-a-box.py, which is based on .travis.yml. But I mention this possibility as an idea.

8. I think that installing CPython from source would provide a good example practice for users, though this complicates the process, so it could be a future modification.

I recommend not force pushing to this PR, but rather follow CPython development practice and reserve rebasing for the merging stage. If we decide to rebase (given that new code is introduced), we may want to squash, in order to preserve commit message information.

[slivingston]

@johnyf thanks for your review! I am going to rebase this branch and read the proposed changes carefully again to make sure it has not suffered bit-rot after 2 years.

[johnyf]

I intend to study my notes above, because I think it may be possible to simplify them. For example, placing an image in an external service could introduce a dependency to a website interface (if configured to be automatically uploaded). Also:

• I considered the possibility of installing CPython from source. The installation of CPython from source is educationally (and in everyday practice) useful, nonetheless:

  • an example of this installation is available in https://github.com/tulip-control/tulip-control/blob/main/contrib/nessainstall/install.sh
  • the image file provides an environment for trying to use the software, with dependencies installed. The full details of how to build each dependency from source seems to me now to be a next step of study, after first running the collection of software that is available in the image.
  • using software available from the package manager of an operating system simplifies the building of the image, and also provides an example installation that involves fewer steps, for a user to reproduce in their own development environment.

• environments for interacting with Python, e.g., ipython, are useful and can be installed with pip install, so they could be installed by the user, depending on their use case.