stimela pull -s error on ILIFU

[o-smirnov]

Anyone seen this?

# 2020/05/14 17:01:48  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:01:48  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/var/lib/apt/lists/auxfiles} destination filesystem does not support xat
trs, further warnings will be suppressed                                              
# 2020/05/14 17:01:50  info unpack layer: sha256:d972307d9fa8968bb7b272efbb88bdbf55ca0c31c4e580d2e626e946edf95ac0
# 2020/05/14 17:01:51  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:01:51  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/etc/gshadow} destination filesystem does not support xattrs, further wa
rnings will be suppressed                                                             
# 2020/05/14 17:04:20  info unpack layer: sha256:b428246021ce6e0d64dfe248fb2510810ae75eeedc28159a9d306f4bdb605218
# 2020/05/14 17:04:21  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:04:21  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/etc/gshadow} destination filesystem does not support xattrs, further wa
rnings will be suppressed                                                             
# FATAL:   While performing build: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: usr/lib/python3.7/distutils/README: link: unpriv.li
nk: unpriv.wrap linkname: link /scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/usr/lib/python3.6/distutils/README /scratch/users/oms/singularity_tmp/root
fs-099f2131-95f3-11ea-aab7-fa163e13d453/usr/lib/python3.7/distutils/README: operation not permitted
Traceback (most recent call last):                                                    
  File "/users/oms/.venv/caracal/bin/stimela", line 7, in <module>                    
    exec(compile(f.read(), __file__, 'exec'))                                         
  File "/users/oms/projects/Stimela/bin/stimela", line 14, in <module>                
    main.main([a for a in sys.argv[1:]])                                              
  File "/users/oms/projects/Stimela/stimela/main.py", line 592, in main               
    _cmd(argv)                                                                        
  File "/users/oms/projects/Stimela/stimela/main.py", line 309, in pull               
    image, simage, directory=pull_folder, force=args.force)                           
  File "/users/oms/projects/Stimela/stimela/singularity.py", line 43, in pull         
    os.path.join(directory,store_path), fp])                                            
  File "/users/oms/projects/Stimela/stimela/utils/xrun_poll.py", line 189, in xrun    
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: singularity returns error code 255                                         

My folders are set properly:

$  set | grep SINGU
SINGULARITY_CACHEDIR=/scratch/users/oms/singularity_cache
SINGULARITY_PULLFOLDER=/scratch/users/oms/singularity_images
SINGULARITY_TMPDIR=/scratch/users/oms/singularity_tmp

[KshitijT]

@o-smirnov , which image is this for?
I see the weird messages too, but my stimela pull (albeit the one by running CARACal) seems to have gone through. E.g. for owlcat:

# 2020/05/13 22:14:20  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} destination filesyste
m does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:14:25  info unpack layer: sha256:cc429581b6e11b942ddbe730b10902f5bcd06da5b3a6112f08fa05f7f1657e3b
# 2020/05/13 22:14:25  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} ignoring ENOTSUP on c
learxattrs
# 2020/05/13 22:14:25  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} destination filesyste
m does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:15:19  info unpack layer: sha256:0bf0e60a13745d325c320c845f0d37ab72e5d1de832b94644240743d15a21aec
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/usr} ignoring ENOTSUP on cl
earxattrs
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/usr} destination filesystem
 does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:15:19  info unpack layer: sha256:95dc7348703240c1de73f765b098b559c62b49169c3b9fd7eac9bab1aeca3a7f
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/etc} ignoring ENOTSUP on cl
earxattrs
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/etc} destination filesystem
 does not support xattrs, further warnings will be suppressed
# INFO:    Creating SIF file...

[o-smirnov]

For all of them... but maybe it depends on filesystem, let me check...

[HaoChen-ast]

Another error in pull Stimela image on ilifu.
But as we have Stimela_image on ilifu now at /idia/software/containers/STIMELA_IMAGES caracal-pipeline/caracal#1125, we can use that STIMELA_IMAGES directly (no need to pull again).

# 2020/05/20 09:12:43  info unpack layer: sha256:5a424948411eaa63e57cc6f817865b444b32a5f130c57b5bd7bccdc2e3e03b61
# INFO:    Creating SIF file...
# FATAL:   While performing build: while creating squashfs: create command failed: exit status 1:
# Write failed because No space left on device
# 
# FATAL ERROR:Failed to write to output filesystem
Traceback (most recent call last):
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/bin/stimela", line 14, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/main.py", line 593, in main
    _cmd(argv)
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/main.py", line 310, in pull
    image, simage, directory=pull_folder, force=args.force)
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/singularity.py", line 49, in pull
    image_path, fp])
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/utils/xrun_poll.py", line 189, in xrun
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: singularity returns error code 255

[KshitijT]

@HaoChen-ast , this is a slightly different error.

Could you please set the following (as per Jordan's suggestion) in your bashrc file:

export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache
export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

Should do the trick.

[HaoChen-ast]

export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache
export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

There is no directory of /software/astro/caracal/ on ilifu.

/idia/software/containers/STIMELA_IMAGES/ is enough for us.
No need to pull by ourselves again.
Please remember to keep it updating on time.
Thanks, @KshitijT!

[SpheMakh]

Has this been resolved?

[ulricharmel]

I am getting this error on tina when I run stimela pull -s
I don't know if it is related to this issue

Cleaning up...
ERROR: Failed squashing image, left template directory at: /tmp/.singularity-build.XrN9KQ
ERROR: pulling container failed!
Traceback (most recent call last):
File "/home/ulrich/stimela_venv/stimela3/bin/stimela", line 14, in
main.main([a for a in sys.argv[1:]])
File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/main.py", line 602, in main
_cmd(argv)
File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/main.py", line 319, in pull
image, simage, directory=pull_folder, force=args.force)
File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/singularity.py", line 46, in pull
name, fp])
File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/utils/xrun_poll.py", line 189, in xrun
raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 1

[INgauriS]

export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache
export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

I did updated my bashrc for above line with uptodate stimela path: /idia/software/containers/STIMELA_IMAGES
"export SINGULARITY_CACHEDIR=/idia/software/containers/STIMELA_IMAGE/.singularity_cache
export SINGULARITY_TMPDIR=/idia/software/containers/STIMELA_IMAGES/.singularity_tmp"

I still have problem in pulling stimela (after 3 years) :(, How to resolve this.
Copying the log file line below:

2023-04-13 11:06:55 CARACal.Stimela.obsconf INFO: Adding cab '/idia/software/containers/STIMELA_IMAGES/stimela_casa_1.7.1.sif' (5.8.0) to recipe, container name 'listobs-ms0'
2023-04-13 11:06:55 CARACal.Stimela.obsconf INFO: Adding cab '/idia/software/containers/STIMELA_IMAGES/stimela_msutils_1.4.6.sif' (1.0.1) to recipe, container name 'summary_json-ms0'

running cd /idia/software/containers/STIMELA_IMAGES/ && singularity pull --name stimela_owlcat_1.6.6.sif docker://quay.io/stimela/owlcat:1.6.6

WARNING: Cache disabled - cache location /idia/software/containers is not writable.

INFO: Converting OCI blobs to SIF format

FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: unable to create new build: failed to create build parent dir: stat /idia/software/containers/STIMELA_IMAGES/.singularity_tmp: no such file or directory

2023-04-13 11:06:58 CARACal ERROR: cd /idia/software/containers/STIMELA_IMAGES/ && singularity returns error code 255 [StimelaCabRuntimeError]
2023-04-13 11:06:58 CARACal INFO: More information can be found in the logfile at output/logs-20230413-110655/log-caracal.txt
2023-04-13 11:06:58 CARACal INFO: You are running version 1.0.6
2023-04-13 11:06:58 CARACal ERROR: Traceback (most recent call last):
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/main.py", line 187, in __run
2023-04-13 11:06:58 CARACal ERROR: pipeline.run_workers()
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers
2023-04-13 11:06:58 CARACal ERROR: worker.worker(self, recipe, config)
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/workers/obsconf_worker.py", line 95, in worker
2023-04-13 11:06:58 CARACal ERROR: recipe.add("cab/owlcat_plotelev", step, {
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/recipe.py", line 612, in add
2023-04-13 11:06:58 CARACal ERROR: job.setup_job(image=image, config=config,
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/recipe.py", line 298, in setup_job
2023-04-13 11:06:58 CARACal ERROR: singularity.pull(image_url,
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/singularity.py", line 44, in pull
2023-04-13 11:06:58 CARACal ERROR: utils.xrun(f"cd {directory} && singularity", ["pull",
2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2023-04-13 11:06:58 CARACal ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2023-04-13 11:06:58 CARACal ERROR: stimela.utils.StimelaCabRuntimeError: cd /idia/software/containers/STIMELA_IMAGES/ && singularity returns error code 255
2023-04-13 11:06:58 CARACal INFO: exiting with error code 1

[INgauriS]

Problem resolved!

Required:

Stable of version of Caracal, on date 13/04/2023: caracal version 1.0.6
along with this
/software/astro/caracal/STIMELA_IMAGES_1.7.5

Note: if working on ILIFU then do not use /idia/software/containers/STIMELA_IMAGES/ , instead go with /software/astro/caracal/STIMELA_IMAGES_x.x.x

[PeterKamphuis]

So I'm getting Oleg's original error on our GlowHPC so it would be great to know if there is a solution for this.
This is with singularity 3.4.2-1:

 Copying config sha256:ba3e353764d42bc1cd3d23ca9f3b7baa423baaac0eece17e426e80a108a009d5
# 
#  0 B / 6.33 KiB [--------------------------------------------------------------]
#  6.33 KiB / 6.33 KiB [======================================================] 0s
# Writing manifest to image destination
# Storing signatures
# 2023/07/04 16:12:40  info unpack layer: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
# FATAL:   While making image from oci registry: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: bin: apply hdr metadata: clear xattr metadata: /home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/sbuild-171818620/fs/bin: unpriv.lclearxattrs: unpriv.lremovexattr: operation not supported
Traceback (most recent call last):
  File "/home/pkamphuis/venv/caracal_latest_venv/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/main.py", line 314, in pull
    singularity.pull(
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/singularity.py", line 44, in pull
    utils.xrun(f"cd {directory} && singularity", ["pull", 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 255

[PeterKamphuis]

Updating to singularity-ce version 3.9.9 changes the error slightly.

 2023/07/05 09:34:37  info unpack layer: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
# 2023/07/05 09:34:37  warn xattr{/home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs/bin} ignoring ENOTSUP on clearxattrs
# 2023/07/05 09:34:37  warn xattr{/home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs/bin} destination filesystem does not support xattrs, further warnings will be suppressed
# FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: bin: restore parent directory: restore xattr metadata: /home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs: unpriv.lsetxattr: invalid argument

Can this be related to the file system being Ceph. Installing on a normal ubuntu system with singularity version 3.8.6 seems to go through correctly.