Skip to content

Commit d72420a

Browse files
committed
Added content re the docker-archive boostrap agent - including pushing images to the cloud library ...
1 parent 3357c0b commit d72420a

File tree

4 files changed

+165
-5
lines changed

4 files changed

+165
-5
lines changed

create_project.png

29.9 KB
Loading

lolcow_lib_listing.png

72.4 KB
Loading

push_prototype.png

69.3 KB
Loading

singularity_and_docker.rst

+165-5
Original file line numberDiff line numberDiff line change
@@ -514,8 +514,8 @@ During the build process, progress can be monitored in the Sylabs Cloud portal o
514514

515515
.. _sec:mandatory_headers_docker_locally_boostrapped_cli:
516516

517-
Locally Cached Images
518-
---------------------
517+
Locally Available Images: Cached by Docker
518+
------------------------------------------
519519

520520
Singularity containers can be built at the command line from images cached locally by Docker. Suppose, for example:
521521

@@ -564,10 +564,120 @@ results in ``lolcow_from_docker_cache.sif`` for native use by Singularity. There
564564

565565
.. note::
566566

567-
The Sylabs Cloud Remote Builder *does not* interoperate with local Docker daemons; therefore, images cached locally by Docker, *cannot* be used to bootstrap creation of SIF files via the Remote Builder service.
567+
The Sylabs Cloud Remote Builder *does not* interoperate with local Docker daemons; therefore, images cached locally by Docker, *cannot* be used to bootstrap creation of SIF files via the Remote Builder service. Of course, a SIF file could be created locally as detailed above. Then, in a separate, manual step, :ref:`pushed to the Sylabs Cloud Singularity Library <sec:pushing_locally_available_images_to_library>`.
568+
569+
570+
571+
.. _sec:mandatory_headers_docker_locally_stored_bootstrap_cli:
572+
573+
Locally Available Images: Stored Archives
574+
------------------------------------------
575+
576+
Singularity containers can also be built at the command line from Docker images stored locally as ``tar`` files. Suppose, for example, ``lolcow.tar`` is locally stored archive in the *current* working directory; then the contents of this archive can be revealed as follows:
577+
578+
.. code-block:: none
579+
580+
$ tar tvf lolcow.tar
581+
drwxr-xr-x 0/0 0 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/
582+
-rw-r--r-- 0/0 3 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/VERSION
583+
-rw-r--r-- 0/0 1417 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/json
584+
-rw-r--r-- 0/0 122219008 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/layer.tar
585+
drwxr-xr-x 0/0 0 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/
586+
-rw-r--r-- 0/0 3 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/VERSION
587+
-rw-r--r-- 0/0 482 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/json
588+
-rw-r--r-- 0/0 14848 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/layer.tar
589+
-rw-r--r-- 0/0 4432 2017-09-21 19:37 577c1fe8e6d84360932b51767b65567550141af0801ff6d24ad10963e40472c5.json
590+
drwxr-xr-x 0/0 0 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/
591+
-rw-r--r-- 0/0 3 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/VERSION
592+
-rw-r--r-- 0/0 482 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/json
593+
-rw-r--r-- 0/0 3072 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/layer.tar
594+
drwxr-xr-x 0/0 0 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/
595+
-rw-r--r-- 0/0 3 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/VERSION
596+
-rw-r--r-- 0/0 406 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/json
597+
-rw-r--r-- 0/0 125649920 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/layer.tar
598+
drwxr-xr-x 0/0 0 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/
599+
-rw-r--r-- 0/0 3 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/VERSION
600+
-rw-r--r-- 0/0 482 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/json
601+
-rw-r--r-- 0/0 15872 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/layer.tar
602+
drwxr-xr-x 0/0 0 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/
603+
-rw-r--r-- 0/0 3 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/VERSION
604+
-rw-r--r-- 0/0 482 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/json
605+
-rw-r--r-- 0/0 5632 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/layer.tar
606+
-rw-r--r-- 0/0 574 1970-01-01 01:00 manifest.json
607+
608+
In other words, it is evident that this 'tarball' is a Docker-format image comprised of multiple layers along with metadata in a JSON manifest.
609+
610+
.. TODO check the JSON manifest stmt above
611+
612+
Through use of the ``docker-archive`` bootstrap agent, a SIF file (``lolcow_tar.sif``) for use by Singularity can be created via the following ``build`` command:
613+
614+
.. code-block:: none
615+
616+
$ singularity build lolcow_tar.sif docker-archive://lolcow.tar
617+
INFO: Starting build...
618+
Getting image source signatures
619+
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
620+
119.83 MiB / 119.83 MiB [==================================================] 6s
621+
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
622+
15.50 KiB / 15.50 KiB [====================================================] 0s
623+
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
624+
14.50 KiB / 14.50 KiB [====================================================] 0s
625+
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
626+
5.50 KiB / 5.50 KiB [======================================================] 0s
627+
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
628+
3.00 KiB / 3.00 KiB [======================================================] 0s
629+
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
630+
116.56 MiB / 116.56 MiB [==================================================] 6s
631+
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
632+
3.33 KiB / 3.33 KiB [======================================================] 0s
633+
Writing manifest to image destination
634+
Storing signatures
635+
INFO: Creating SIF file...
636+
INFO: Build complete: lolcow_tar.sif
637+
638+
There are two important differences in syntax evident in the above ``build`` command:
639+
640+
1. The ``docker`` part of the URI has been appended by ``archive``. This ensures Singularity seek a Docker-format image stored locally as ``lolcow.tar`` to boostrap the conversion process to SIF, as opposed to attempting to retrieve an image remnotely hosted via the Docker Hub.
641+
642+
2. ``sudo`` is *not* prepended to the ``build`` command for Singularity. This is *not* required if the executing user has the appropriate access privileges to the stored file.
643+
644+
.. note::
645+
646+
The Sylabs Cloud Remote Builder *does not* interoperate with locally stored Docker-format images; therefore, images cached locally by Docker, *cannot* be used to bootstrap creation of SIF files via the Remote Builder service. Of course, a SIF file could be created locally as detailed above. Then, in a separate, manual step, :ref:`pushed to the Sylabs Cloud Singularity Library <sec:pushing_locally_available_images_to_library>`.
647+
648+
649+
.. _sec:pushing_locally_available_images_to_library:
650+
651+
Pushing Locally Available Images to a Library
652+
---------------------------------------------
653+
654+
The outcome of bootstrapping from an image cached locally by Docker, or one stored locally as an archive, is of course a locally stored SIF file. As noted above, this is the *only* option available, as the Sylabs Cloud Remote Builder does not interoperate with the Docker daemon or locally stored archives in the Docker image format. Once produced, however, it may be desirable to make the resulting SIF file available through the Sylabs Cloud Singularity Library; therefore, the procedure to ``push`` a locally available SIF file to the Library is detailed here.
655+
656+
From the `Sylabs Cloud Singularity Library <https://cloud.sylabs.io/library>`_, select ``Create a new Project``. In this first of two steps, the publicly accessible project is created as illustrated below:
657+
658+
.. image:: create_project.png
659+
660+
Because an access token for the cloud service already exists, attention can be focused on the ``push`` command prototyped towards the bottom of the following screenshot:
661+
662+
.. image:: push_prototype.png
568663

569-
Of course, a SIF file could be created locally as detailed above. Then, in a separate, manual step, uploaded to the Sylabs Cloud Singularity Library.
664+
In fact, by simply replacing ``image.sif`` with ``lolcow_tar.sif``, the following upload is executed:
570665

666+
.. code-block:: none
667+
668+
$ singularity push lolcow_tar.sif library://ilumb/default/lolcow_tar
669+
INFO: Now uploading lolcow_tar.sif to the library
670+
87.94 MiB / 87.94 MiB [=============================================================================] 100.00% 1.25 MiB/s 1m10s
671+
INFO: Setting tag latest
672+
673+
674+
Finally, from the perspective of the Library, the **hosted** version of the SIF file appears as illustrated below. Directions on how to ``pull`` this file are included.
675+
676+
.. image:: lolcow_lib_listing.png
677+
678+
.. note::
679+
680+
The hosted version of the SIF file in the Sylabs Cloud Singularity Library is maintainable. In other words, if the image is updated lcoally, the update can be pushed to the Library and tagged appropriately.
571681

572682

573683
Working with Definition Files
@@ -737,7 +847,45 @@ Then,
737847
738848
In other words, this is the definition-file counterpart to :ref:`the command-line invocation provided above <sec:mandatory_headers_docker_locally_boostrapped_cli>`.
739849

740-
.. TODO remote builder content note - exclusion
850+
.. TODO remote builder content note - exclusion above
851+
852+
Alternatively when ``docker-archive`` is the bootstrap agent in a Singularity definition file, SIF containers can be created from images stored locally by Docker. Suppose the definition file ``lolcow-da.def`` has contents:
853+
854+
.. code-block:: singularity
855+
856+
Bootstrap: docker-archive
857+
From: lolcow.tar
858+
859+
Then,
860+
861+
.. code-block:: none
862+
863+
$ sudo singularity build lolcow_tar_def.sif lolcow-da.def
864+
WARNING: Authentication token file not found : Only pulls of public images will succeed
865+
INFO: Starting build...
866+
Getting image source signatures
867+
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
868+
119.83 MiB / 119.83 MiB [==================================================] 6s
869+
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
870+
15.50 KiB / 15.50 KiB [====================================================] 0s
871+
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
872+
14.50 KiB / 14.50 KiB [====================================================] 0s
873+
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
874+
5.50 KiB / 5.50 KiB [======================================================] 0s
875+
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
876+
3.00 KiB / 3.00 KiB [======================================================] 0s
877+
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
878+
116.56 MiB / 116.56 MiB [==================================================] 6s
879+
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
880+
3.33 KiB / 3.33 KiB [======================================================] 0s
881+
Writing manifest to image destination
882+
Storing signatures
883+
INFO: Creating SIF file...
884+
INFO: Build complete: lolcow_tar_def.sif
885+
886+
through ``build`` results in the SIF file ``lolcow_tar_def.sif``. In other words, this is the definition-file counterpart to :ref:`the command-line invocation provided above <sec:mandatory_headers_docker_locally_stored_bootstrap_cli>` .
887+
888+
.. TODO RB Test
741889
742890
743891
Optional Headers
@@ -905,9 +1053,21 @@ In this case, the ``lolcow`` image was hosted remotely via user ``godlovedc`` at
9051053
.. code-block:: none
9061054
9071055
$ singularity inspect --deffile lolcow_from_docker_cache.sif
1056+
9081057
from: godlovedc/lolcow:latest
9091058
bootstrap: docker-daemon
9101059
1060+
On the other hand, if a locally stored copy of a Docker image was used to bootstrap creation of a container for Singularity, then ``inspect`` reveals existence of ``docker-archive`` in the definition file that was used as follows:
1061+
1062+
.. code-block:: none
1063+
1064+
$ singularity inspect --deffile lolcow_tar.sif
1065+
1066+
bootstrap: docker-archive
1067+
from: lolcow.tar
1068+
1069+
The name of the archive used, ``lolcow.tar`` in this case, appears in the ``from`` header of the definition file retrieved by ``inspect``.
1070+
9111071
Of particular relevance to :ref:`execution precedence <sec:def_files_execution_SUB_execution_precedence>`, however, is the ``--runscript`` option for ``inspect``. For example, using the definition file above, the runscript is unsurprisingly:
9121072

9131073
.. code-block:: none

0 commit comments

Comments
 (0)