Merge pull request #10 from grycap/manpages

man pages and updating documentation

Author: dealfonso

README.md

@@ -1,29 +1,35 @@
-# MiniDock - minimization of Docker containers
+# minicon - minimization containers
 
-When you run Docker containers, you usually run a system that has a whole Operating System and your specific application. The result is that the footprint of the container is bigger than needed.
+When you run containers (e.g. in Docker), you usually run a system that has a whole Operating System, documentation, extra packages, etc. and your specific application. The result is that the footprint of the container is bigger than needed.
 
-**minidock** aims at reducing the footprint of the Docker containers, by just including in the container those files that are needed. That means that the other files in the original container are removed.
+**minicon** aims at reducing the footprint of the Docker containers, by just including in the container those files that are needed. That means that the other files in the original container are removed.
 
-The purpose of **minidock** is better understood with the use cases explained in depth in the section "[Examples](#4-examples)": the size of an Apache server is reduced from 216MB. to 50.4MB., and the size of a Perl application in a Docker container is reduced from 206MB to 50.4MB.
+**minicon** is a general tool to analyze applications and executions of these applications to obtain a filesystem that contains all the dependencies that have been detected. In particular, it can be used to reduce Docker containers. The **minicon** package includes **minidock**
+which will help to reduce Docker containers by hiding the underlying complexity of running **minicon** inside a Docker container.
 
+The purpose of **minicon** and **minidock** is better understood with the use cases explained in depth in the section "[Examples](#4-examples)": the size of a basic UI that contains bash, ip, wget, ssh, etc. commands is _reduced from 211MB to 10.9MB_; the size of a NodeJS application along with the server is _reduced from 686 MB (using the official node image) to 45.6MB_; the size of an Apache server is _reduced from 216MB to 50.4MB_, and the size of a Perl application in a Docker container is _reduced from 206MB to 43.3MB_.
 
-> [**minidock**](#minidock---minimization-of-docker-containers) is based on [**minicon**](doc/minicon.md), [**importcon**](doc/importcon.md) and [**mergecon**](doc/mergecon.doc), and hides the complexity of creating a container, mapping minicon, guessing parameters such as the entrypoint or the default command, creating the proper commandline, etc.
+> [**minidock**](doc/minidock.md) is based on [**minicon**](doc/minicon.md), [**importcon**](doc/importcon.md) and [**mergecon**](doc/mergecon.md), and hides the complexity of creating a container, mapping minicon, guessing parameters such as the entrypoint or the default command, creating the proper commandline, etc.
 
-## 1. Why **minidock**?
+## 1. Why **minicon** and **minidock**?
 
-Reducing the footprint of one container is of special interest, to redistribute the container images and saving the storage space in your premises. There are also security reasons to minimize the unneeded application or environment available in one container image (e.g. if the container does not need a compiler, why should it be there? maybe it would enable to compile a rootkit). 
+Reducing the footprint of one container is of special interest, to redistribute the container images.
 
-In this sense, the publication of the NIST "[Application Container Security Guide](https://doi.org/10.6028/NIST.SP.800-190)" suggests that "_An image should only include the executables and libraries required by the app itself; all other OS functionality is provided by the OS kernel within the underlying host OS_".
+It is of special interest in cases such as [SCAR](https://github.com/grycap/scar), that executes containers out of Docker images in AWS Lambda. In that case, the use cases are limited by the size of the container (the ephemeral storage space is limited to 512 Mb., and SCAR needs to pull the image from Docker Hub into the ephemeral storage and then uncompress it; so the maximum size for the container is even more restricted).
 
-**minicon** is a tool that enables a fine grain minimization for any type of container (it is even interesting for non containerized boxes). Using it for Docker images consist in a simple pipeline:
+But there are also security reasons to minimize the unneeded application or environment available in one container image. In the case that the application fails, not having other applications reduces the impact of an intrusion (e.g. if the container does not need a compiler, why should it be there? maybe it would enable to compile a rootkit). 
+
+In this sense, the recent publication of the NIST "[Application Container Security Guide](https://doi.org/10.6028/NIST.SP.800-190)" suggests that "_An image should only include the executables and libraries required by the app itself; all other OS functionality is provided by the OS kernel within the underlying host OS_".
+
+**minicon** is a tool that enables a fine grain minimization for any type of filesystem, but it is possible to use it to reduce Docker images following the next pipeline:
 
 1. Preparing a Docker container with the dependencies of **minicon**
 1. Guessing the entrypoint and the default command for the container.
 1. Running **minicon** for these commands (maping the proper folders to get the resulting tar file).
 1. Using **importcon** to import the resulting file to copy the entrypoint and other settings.
 1. etc.
 
-**minidock** is a one-liner for that procedure whose aim is just to convert a
+**minidock** is a one-liner that automates that procedure to make that reducing a container consist in just to convert a
 
 ```bash
 $ docker run --rm -it myimage myapp
@@ -35,8 +41,6 @@ into
 $ minicon -i myimage -t myimage:minicon -- myapp
 ``` 
 
-To obtain the minimized Docker image, and hiding the internal procedure.
-
 ## 2. Installation
 
 ### 2.1 From packages
@@ -63,13 +67,13 @@ $ yum install ./minicon-1.2-1.noarch.rpm
 
 ### 2.2 From sources
 
-**minidock** is a bash script that runs the other applications in the _minicon_ package, to analyze the docker containers. So you just simply need to have a working linux with bash and the other dependencies installed and get the code:
+**minicon** are a set of bash scripts. So you just simply need to have a working linux with bash and the other dependencies installed and get the code:
 
 ```bash
 $ git clone https://github.com/grycap/minicon
 ```
 
-In that folder you'll have the **minidock** application. Then the commands in the _minicon_ distribution must be in the PATH. So I would suggest to put it in the _/opt_ folder and set the proper PATH var. Otherwise leave it in a folder of your choice and set the PATH variable:
+In that folder you'll have the **minicon**, **minidock** and other applications. The commands in the _minicon_ distribution must be in the PATH. So I would suggest to put it in the _/opt_ folder and set the proper PATH var. Otherwise leave it in a folder of your choice and set the PATH variable:
 
 ```bash
 $ mv minicon /opt
@@ -78,51 +82,37 @@ $ export PATH=$PATH:/opt/minicon
 
 #### 2.2.1 Dependencies
 
-**minidock** depends on the commands _minicon_, _importcon_ and _mergecon_, and the packages _jq_, _tar_ and _docker_. So, you need to install the proper packages in your system.
+**minidock** depends on the commands _minicon_, _importcon_ and _mergecon_, and the packages _jq_, _tar_ and _docker_. **minicon** and the others have dependencies on other packages. So, you need to install the proper packages in your system:
 
 **Ubuntu**
 
 ```bash
-$ apt-get install jq tar 
+$ apt-get install jq tar libc-bin tar file strace rsync
 ```
 
 **CentOS**
 ```bash
-$ yum install tar jq which
+$ yum install tar jq glibc-common file strace rsync which
 ```
+
 ## 3. Usage
 
-**minidock** has a lot of options. You are advised to run ```./minidock --help``` to get the latest information about the usage of the application.
+The **minidock suite** enables to prepare filesystems for running containers. The suite consists in the next commands: 
 
-The basic syntax is
+1. **minidock** ([doc](doc/minidock.md)) analyzes one existing Docker image, reduces its footprint and leaves the new version in the local Docker registry. It makes use of the other tools in the _minicon_ package.
 
-```bash
-$ ./minidock <options> <options for minicon> [ --docker-opts <options for docker> ] -- <run for the container>
-```
-
-Some of the options are:
-- \<run for the container\>: Is the whole commandline to be analised in the run. These are the same parameters that you would pass to "docker run ... <image> <run for the container>". 
-> * the aim is that you run "minidock" as if you used a "docker run" for your container.
-- **\<options for docker\>**: If you need them, you can include some options that will be raw-passed to the docker run command used during the analysis. (i.e. minidock will executedocker run <options generated> <options for docker> ...). Some examples are mapping volumes (i.e. **-v** Docker flag)
-- **\<options for minicon\>**: If you need to, you can add some minicon-specific options. The supported options are --include --exclude --plugin: --exclude will exclude some path, --include will include specific files or folder, and --plugin can be used to configure the _minicon_ plugins.
-- **--image \<image\>**: Name of the existing Docker image to minimize.
-- **--tag \<tag\>**: Tag for the resulting image (random if not provided).
-- **--mode \<mode\>**: Sets the mode to include the used files in the filesystem. There are 4 modes available _skinny_ (default), _slim_, _regular_ and _loose_.
-  * skinny only includes those files that have been used during the simulation of the commands.
-  * slim also includes some whole folders that have been opened.
-  * regular also includes any whole folder that have been checked to exist (e.g. stat).
-  * loose also includes the whole folders in which are stored the files that have been opened during the simulation.
-- **--default-cmd**: Analyze the default command for the containers in the original image.
-- **--apt**: Install the dependencies from minicon using apt-get commands (in the container used for the simulation).
-- **--yum**: Install the dependencies from minicon using yum commands (in the container used for the simulation).
-- **--execution \<full commandline execution\>**: Commandline to analyze when minimizing the container (i.e. that commandline should be able to be executed in the resulting container so the files, libraries, etc. needed should be included). 
-- **--run \<full commandline run\>**: Similar to _--execution_, but in this case, the Entrypoint is prepended to the commandline (docker exec vs docker run).
-- **-2 \<image\>**: If needed, you can merge the resulting minimized image with other. This is very specific for the "mergecon" tool. It is useful for (e.g.) adding a minimal Alpine distro (with _ash_ and so on) to the minimized filesystem.
-- **--verbose | -v**: Gives more information about the procedure.
-- **--debug**: Gives a lot more information about the procedure.
+1. **minicon** ([doc](doc/minicon.md)) aims at reducing the footprint of the filesystem for the container, just adding those files that are needed. That means that the other files in the original container are removed.
+
+1. **importcon** ([doc](doc/importcon.md)) importcon is a tool that imports the contents from a tarball to create a filesystem image using the "docker import" command. But it takes as reference an existing docker image to get parameters such as ENV, USER, WORKDIR, etc. to set them for the new imported image.
+
+1. **mergecon** ([doc](doc/mergecon.md)) is a tool that merges the filesystems of two different container images. It creates a new container image that is built from the combination of the layers of the filesystems of the input containers.
+
+Please refer to the documentation of each command to get help about them.
 
 ## 4. Examples
 
+In this section we are including examples on using **minidock**, that makes use of all the other commands. Please refer to the documentation of each command to get examples about each individual command.
+
 ### 4.1 Basic Ubuntu UI in less than 11 Mb.
 
 In this example we will create a basic user interface, from ubuntu, that include commands like `wget`, `ssh`, `cat`, etc.
@@ -413,17 +403,4 @@ minicon             uc6                 7c85b5a104f5        5 seconds ago
 minicon             uc6fat              1c8179d3ba94        4 hours ago         206MB
 ```
 
-In this case, the size has been reduced from 206MB to about 43.3MB.
-
-# 5. Flexible Manipulation of Container Filesystems
-
-The **minidock suite** enables to prepare filesystems for running containers. The suite consists in **minidock**, [**minicon**](doc/minicon.md), [**mergecon**](doc/mergecon.md) and [**importcon**](doc/importcon.md):
-
-1. **minidock** ([direct link](minidock---minimization-of-docker-containers)) analyzes one existing Docker image, reduces its footprint and leaves the new version in the local Docker registry. It makes use of the other tools in the _minicon_ package.
-
-1. **minicon** ([direct link](doc/minicon.md)) aims at reducing the footprint of the filesystem for the container, just adding those files that are needed. That means that the other files in the original container are removed.
-
-1. **importcon** ([direct link](doc/importcon.md)) importcon is a tool that imports the contents from a tarball to create a filesystem image using the "docker import" command. But it takes as reference an existing docker image to get parameters such as ENV, USER, WORKDIR, etc. to set them for the new imported image.
-
-1. **mergecon** ([direct link](doc/mergecon.md)) is a tool that merges the filesystems of two different container images. It creates a new container image that is built from the combination of the layers of the filesystems of the input containers.
-
+In this case, the size has been reduced from 206MB to about 43.3MB.

doc/man/importcon

@@ -0,0 +1,128 @@
+.\" Manpage for importcon.
+.\" Contact [email protected] to correct errors or typos.
+.TH man 1 "01 Mar 2018" "1.2-1" "importcon man page"
+.SH NAME
+importcon - IMPORT CONtainer copying features
+.SH SYNOPSIS
+importcon <options> <container filesystem in tar file> 
+.SH DESCRIPTION
+When a container is running, it is possible to export its filesystem to a tar file, using the command 
+.I docker export <mycontainer>. 
+Later, it is possible to import that filesystem into Docker to be used as a Docker image, using a command like 
+.I docker import <mytarfile>. 
+The problem is that the new container has lost all the parameters from the original image (i.e. ENTRYPOINT, USER, CMD, etc.).
+
+.B importcon
+is a script that enables to import a filesystem exported using 
+.I docker export
+into Docker, and to copy the parameters from the original image (i.e. ENTRYPOINT, USER, CMD, VOLUME, etc.)
+
+
+.SH OPTIONS
+.B --image | -i <image>
+  Name of the existing image to copy the parameters.
+
+.B --tag | -t <tag>
+  Tag for the image that will be created from the tarfile (random if not provided)
+
+.B --env | -E
+  Copy ENV settings
+
+.B --entrypoint | -e
+  Copy ENTRYPOINT settings
+
+.B --expose | -x
+  Copy EXPOSE settings
+
+.B --onbuild | -o
+  Copy ONBUILD settings
+
+.B --user | -u
+  Copy USER settings
+
+.B --volume | -V
+  Copy VOLUME settings
+
+.B --workdir | -w
+  Copy WORKDIR settins
+
+.B --cmd | -c
+  Copy CMD settings
+
+.B --all | -A
+  Copy all the previous settings: ENV, ENTRYPOINT, EXPOSE, ONBUILD, USER, VOLUME, WORKDIR and CMD.
+
+.B --version
+  Displays the version of the package and exits.
+
+.B --help | -h
+  Shows this help and exits.
+
+.SH EXAMPLES
+
+If we take the next Dockerfile
+
+  FROM ubuntu
+  RUN apt-get update && apt-get install -y --force-yes apache2
+  EXPOSE 80 443
+  VOLUME ["/var/www", "/var/log/apache2", "/etc/apache2"]
+  ENTRYPOINT ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]
+
+we can build an image using the command
+
+.I docker build . -t tests:apache
+
+Then we can run a container
+
+.I docker run --rm -id -p 10000:80 tests:apache
+
+And export its filesystem to a file:
+
+.I docker export hungry_payne -o myapache.tar
+
+If we simply import the file back to a Docker image
+
+.I docker import myapache.tar tests:myapache
+
+we can check the differences between the configuration of each image for the new containers:
+
+.I docker inspect tests:apache | jq '.[0].Config'
+  {
+  ...
+    "Entrypoint": [
+      "/usr/sbin/apache2ctl",
+      "-D",
+      "FOREGROUND"
+    ],
+  ...
+  }
+
+.I docker inspect tests:myapache | jq '.[0].Config'
+  {
+  ...
+    "Entrypoint": null,
+  ...
+  }
+
+
+We can see that our new image has all the setting set to empty. We can use 
+.B importcon
+to import the container, taking the original image as reference:
+
+.I importcon -t tests:apacheimportcon -i tests:apache myapache.tar -A
+.I docker inspect tests:apacheimportcon | jq '.[0].Config'
+  {
+  ...
+    "Entrypoint": [
+      "/usr/sbin/apache2ctl",
+      "-D",
+      "FOREGROUND"
+    ],
+  ...
+  }
+
+.SH SEE ALSO
+minicon(1), minidock(1), jq(1)
+
+.SH AUTHOR
+Carlos de Alfonso ([email protected])

doc/man/mergecon

@@ -0,0 +1,54 @@
+.\" Manpage for mergecon.
+.\" Contact [email protected] to correct errors or typos.
+.TH man 1 "01 Mar 2018" "1.2-1" "mergecon man page"
+.SH NAME
+mergecon - merge Docker containers
+.SH SYNOPSIS
+mergecon <options>
+
+.SH DESCRIPTION
+Docker containers are built from different layers, and 
+.B mergecon
+is a tool that merges the filesystems of two different container images. It creates a new container image that is built from the combination of the layers of the filesystems of the input containers.
+
+.SH WHY MERGECON?
+
+If you create a minimal application filesystem (i.e. using minicon), you will be able to run your application, but you will not have any other application available for (e.g.) debugging your application.
+
+Using mergecon, you will be able to overlay the files of your container image to other existing container image. In this way, you can overlay your minicon resulting application over a whole ubuntu:latest container. The effect is that you will have the support of a whole operating environment over the minimal container.
+
+On the other side, if you have an application that needs to be compiled and installed, you can create a one-liner installation, and combine that layer with other container. That means that you will be able to compile such image in (e.g.) ubuntu, and create a final container with other flavor (e.g. CentOS).
+
+.SH OPTIONS
+.B --first | -1 <image>
+ Name of the first container image (will use docker save to dump it)
+
+.B --second | -2 <image>
+ Name of the second container image (will use docker save to dump it) the default behaviour gives more priority to the second image. I.e. in case of overlapping files in both input images, the files in the second image will be exposed to the final image.
+
+.B --tag | -t <name>
+ Name of the resulting container image. If not provided, the resulting name will be the concatenation of the names of the two input images: <image1>:<image2> (the : in the input image names is removed).
+
+.B --working | -w <folder>
+ Working folder. If not provided will create a temporary folder in /tmp
+
+.B --list | -l
+ Lists the layers in the images (useful for filesystem composition).
+
+.B --file | -f <file>
+ tar file where the resulting image will be created. If not provided, the image will be loaded into docker.
+
+.B --keeptemporary | -k
+ Keeps the temporary folder. Otherwise, the folder is removed (if it is created by mergecon).
+
+.B --verbose | -v
+ Gives more information about the procedure.
+
+.B --debug
+ Gives a lot more information about the procedure.
+
+.SH SEE ALSO
+minicon(1)
+
+.SH AUTHOR
+Carlos de Alfonso ([email protected])