GCZ: client build sometimes fails

[martenson]

likely running out of memory, example below

workaround is to galaxyctl stop before the build, which clears enough memory for this to finish reliably

we should increase the memory of the machine or decrease the consumption of the client build step

TASK [galaxyproject.galaxy : Build client] *********************************************************************************************************************************************************************
fatal: [usegalaxy.cz]: FAILED! => changed=false
  cmd: /usr/bin/gmake client-production-maps
  msg: |-
    warning Resolution field "chokidar@3.5.3" is incompatible with requested version "chokidar@^2.0.0"
    Browserslist:********@latest
      Why you should do it regularly: https://github.com/browserslist/update-db#readme
    error Command failed with signal "SIGKILL".
    error Command failed with exit code 1.
    gmake: *** [Makefile:226: client-production-maps] Error 1
  rc: 2
  stderr: |-
    warning Resolution field "chokidar@3.5.3" is incompatible with requested version "chokidar@^2.0.0"
    Browserslist: caniuse-lite is outdated. Please run:
      npx update-browserslist-db@latest
      Why you should do it regularly: https://github.com/browserslist/update-db#readme
    error Command failed with signal "SIGKILL".
    error Command failed with exit code 1.
    gmake: *** [Makefile:226: client-production-maps] Error 1
  stderr_lines: <omitted>
  stdout: |-
    if [ -f ".venv/bin/activate" ]; then . ".venv/bin/activate"; fi; cd client && yarn install --network-timeout 300000 --check-files
    yarn install v1.22.22
    [1/4] Resolving packages...
    success Already up-to-date.
    Done in 10.22s.
    if [ -f ".venv/bin/activate" ]; then . ".venv/bin/activate"; fi; cd client && env NODE_OPTIONS=--max-old-space-size=3072 GALAXY_PLUGIN_BUILD_FAIL_ON_ERROR=0     yarn run build-production-maps
    yarn run v1.22.22
    $ NODE_ENV=production gulp && yarn run webpack-production-maps && yarn run stage-build
    [10:58:54] Using gulpfile /srv/galaxy/server/client/gulpfile.js
    [10:58:54] Starting 'default'...
    [10:58:54] Starting 'fonts'...
    [10:58:54] Starting 'stageLibs'...
    [10:58:54] Starting 'icons'...
    [10:58:54] Starting 'buildPlugins'...
    [10:58:54] Finished 'stageLibs' after 17 ms
    No changes detected for annotate_image
    No changes detected for chiraviz
    No changes detected for drawrna
    No changes detected for editor
    No changes detected for fits_graph_viewer
    No changes detected for fits_image_viewer
    No changes detected for h5web
    No changes detected for heatmap_default
    No changes detected for hyphyvision
    No changes detected for jqplot_bar
    No changes detected for media_player
    No changes detected for mvpapp
    No changes detected for nora
    No changes detected for nvd3_bar
    No changes detected for openseadragon
    No changes detected for PCA_3Dplot
    No changes detected for phylocanvas
    No changes detected for pv
    No changes detected for scatterplot
    No changes detected for tiffviewer
    No changes detected for ts_visjs
    [10:58:54] Finished 'buildPlugins' after 540 ms
    [10:58:54] Starting 'installPlugins'...
    [10:58:56] Finished 'fonts' after 2.53 s
    Building icon pack "galaxy"
     - loading icon "gxd galaxyLogo" from file "galaxyLogo.duotone.svg"
     - loading icon "gxd textLarger" from file "textLarger.duotone.svg"
     - loading icon "gxd textSmaller" from file "textSmaller.duotone.svg"
    Saving generated Icon Pack to: "./src/assets/icons.json"
    Icons built successfully
    [10:58:56] Finished 'icons' after 2.55 s
    Installed package @galaxyproject/cytoscape@0.0.3 in ../config/plugins/visualizations/cytoscape
    Installed package @galaxyproject/ngl@0.0.11 in ../config/plugins/visualizations/ngl
    Installed package @galaxyproject/msa@0.2.1 in ../config/plugins/visualizations/msa
    Installed package @galaxyproject/openlayers@0.0.0 in ../config/plugins/visualizations/openlayers
    Installed package @galaxyproject/venn@0.0.3 in ../config/plugins/visualizations/venn
    Installed package @galaxyproject/vizarr@0.1.6 in ../config/plugins/visualizations/vizarr
    [10:59:19] Finished 'installPlugins' after 24 s
    [10:59:19] Starting 'cleanPlugins'...
    [10:59:19] Finished 'cleanPlugins' after 84 ms
    [10:59:19] Starting 'stagePlugins'...
    [10:59:20] Finished 'stagePlugins' after 1.08 s
    [10:59:20] Finished 'default' after 26 s
    $ GXY_BUILD_SOURCEMAPS=1 NODE_ENV=production webpack
    info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
  stdout_lines: <omitted>

[martenson]

I tested various configurations for max heap using:

GXY_BUILD_SOURCEMAPS=1 NODE_ENV=production NODE_OPTIONS="--max_old_space_size=2048" yarn run webpack

on qa-1 and as far as I can tell there is no heap size that will reliably finish. I think we need to automate stopping the service before we build the client on the smaller qa machines

for production machines this is not a good solution, since it increases the downtime

[martenson]

the qa machines were addressed by this: #152