Skip to content

assets:precomile fails, but deploy succeeds #719

New issue
New issue
Open
Open
assets:precomile fails, but deploy succeeds#719
@PeterVenhuizen

Description

@PeterVenhuizen
PeterVenhuizen
opened on Mar 2, 2023

The invoke :'rails:assets_precompile' step in my deploy block fails. I know why it fails, because some of the config files are formatted incorrectly. This issue is that there is a rake aborted! message + the SyntaxError, but instead of the GitLab CI deploy job stopping there, mina just moves on to the next task.

The output below is the full output of the deploy job, with certain pieces of information [REDACTED].

How does mina listen for errors when running tasks and why wasn't the error picked up here?

Hope someone can help out with this, since this forms quite a risk for the stability of our production environment.

Thanks!

Running with gitlab-runner 15.8.2 (4d1ca121)
  on [REDACTED]
Preparing the "docker" executor 00:03
Using Docker executor with image [REDACTED] ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image [REDACTED] ...
Using docker image [REDACTED] for [REDACTED] with digest [REDACTED][REDACTED] ...
Preparing environment 00:01
Running on [REDACTED] via [REDACTED]...
Getting source from Git repository 00:01
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Executing "step_script" stage of the job script 01:52
Using docker image [REDACTED] for [REDACTED] with digest [REDACTED][REDACTED] ...
$ cd /app
$ . bin/keytab_auth
Generate keytab ticket...
$ mina -f config/deploy.rb $STAGE deploy
! !  Unable to check latest version  ! !
Warning: Permanently added '[REDACTED]' (ECDSA) to the list of known hosts.
-----> Creating a temporary build path
-----> Reading master key
-----> Using git commit "18b60e30"
Cloning into '.'...
remote: Enumerating objects: 9137, done.        
remote: Counting objects: 100% (380/380), done.        
remote: Compressing objects: 100% (19/19), done.        
remote: Total 9137 (delta 369), reused 362 (delta 359), pack-reused 8757        
Receiving objects: 100% (9137/9137), 1.83 MiB | 0 bytes/s, done.
Resolving deltas: 100% (6742/6742), done.
Switched to a new branch 'current_release'
-----> Updating submodules
-----> Using this git commit
Peter Venhuizen (18b60e3):
> chore: test change init_command
-----> Symlinking shared paths
-----> Generate config files from vault
-----> Successfully created /app/config/auth.yml
-----> Successfully created /app/config/database.yml
-----> Successfully created /app/config/mw.yml
-----> Successfully created /app/config/nr.yml
-----> Successfully created /app/config/nr3.yml
-----> Successfully created /app/config/service_availability_database.yml
-----> Successfully created /app/config/sidekiq_redis.yml
-----> Installing gem dependencies using Bundler
[REDACTED]
  -----> DB migrations unchanged; skipping DB migration
-----> Precompiling asset files
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
rake aborted!
Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 1 column 1
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/psych-3.3.4/lib/psych.rb:457:in `parse'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/psych-3.3.4/lib/psych.rb:457:in `parse_stream'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/psych-3.3.4/lib/psych.rb:391:in `parse'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/psych-3.3.4/lib/psych.rb:351:in `safe_load'
/[REDACTED]/config/initializers/service_availability_database.rb:5:in `<top (required)>'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:667:in `load'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:667:in `block in load_config_initializer'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4.2/lib/active_support/notifications.rb:208:in `instrument'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:666:in `load_config_initializer'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:620:in `block (2 levels) in <class:Engine>'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:619:in `each'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/engine.rb:619:in `block in <class:Engine>'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:32:in `instance_exec'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:32:in `run'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:61:in `block in run_initializers'
/data/ruby/installed/3.1.0/tsort.rb:228:in `block in tsort_each'
/data/ruby/installed/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/data/ruby/installed/3.1.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/data/ruby/installed/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from'
/data/ruby/installed/3.1.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:50:in `each'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:50:in `tsort_each_child'
/data/ruby/installed/3.1.0/tsort.rb:415:in `call'
/data/ruby/installed/3.1.0/tsort.rb:415:in `each_strongly_connected_component_from'
/data/ruby/installed/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component'
/data/ruby/installed/3.1.0/tsort.rb:347:in `each'
/data/ruby/installed/3.1.0/tsort.rb:347:in `call'
/data/ruby/installed/3.1.0/tsort.rb:347:in `each_strongly_connected_component'
/data/ruby/installed/3.1.0/tsort.rb:226:in `tsort_each'
/data/ruby/installed/3.1.0/tsort.rb:205:in `tsort_each'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/initializable.rb:60:in `run_initializers'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/application.rb:372:in `initialize!'
/[REDACTED]/config/environment.rb:48:in `<top (required)>'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in `require'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in `require'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/application.rb:348:in `require_environment!'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4.2/lib/rails/application.rb:511:in `block in run_tasks_blocks'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:24:in `execute'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/task.rb:61:in `block (2 levels) in define'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:24:in `execute'
/[REDACTED]/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `load'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:23:in `run'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli.rb:484:in `exec'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli.rb:31:in `dispatch'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/cli.rb:25:in `start'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/exe/bundle:48:in `block in <top (required)>'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/data/ruby/installed/gems/3.1.0/gems/bundler-2.3.7/exe/bundle:36:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => environment
(See full trace by running task with --trace)
-----> Cleaning up the auto deployment dependancies
-----> Deleting the dockerfiles directory
-----> Deploy finished
-----> Building
-----> Moving build to [REDACTED]
-----> Build finished
-----> Launching
-----> Updating the [REDACTED] symlink
-----> Restarting background process: [REDACTED]
Reinitializing monit daemon
[REDACTED]
-----> Done. Deployed version 106
Connection to [REDACTED] closed.
       Elapsed time: 91.16 seconds
-----> install sentry if not installed
       /usr/bin/sentry-cli
       
       changed 13 packages, and audited 14 packages in 6s
       
       found 0 vulnerabilities
-----> make root the owner
-----> propose sentry release version
-----> make sentry release
       Created release [REDACTED]
       Success! Set commits for release [REDACTED]
       Finalized release [REDACTED]
       Elapsed time: 18.96 seconds
Cleaning up project directory and file based variables 00:02
Job succeeded

Activity

reneweteling

reneweteling commented on Apr 4, 2023

@reneweteling
reneweteling
on Apr 4, 2023

Hi @PeterVenhuizen,

So as you know I've worked on this issue, for some reason exit codes get ignored when you write multiline commands.

# When writing it like this, it does gets executed but starts ignoring error codes, so the deploy will continue even if something fails
command %(
  if [ -d #{fetch(:deploy_to)}/fake_current ]; then
    echo "-----> Deleting dummy app structure"
    rm -rf #{fetch(:deploy_to)}/fake_current
  fi
  if [ -f ~/.netrc ]; then
    echo "-----> Cleaning up the auto deployment dependancies"
    sed -i.bk '/machine nms-git/d' ~/.netrc
    if [ -f ~/.netrc.bk ]; then
      rm -fr ~/.netrc.bk
    fi
  fi
  if [ -d #{fetch(:current_path)}/dockerfiles ]; then
    echo "-----> Deleting the dockerfiles directory"
rm -fr #{fetch(:current_path)}/dockerfiles
  fi
)
# This respects the error codes, 2 things ive noticed. 
# 1 - make sure you end each line with a ;
# 2 - one if per command, nested seem te be ok :S
command %(
  if [ -d #{fetch(:deploy_to)}/fake_current ]; then
    echo "-----> Deleting dummy app structure";
    rm -rf #{fetch(:deploy_to)}/fake_current;
  fi
)
command %(
  if [ -f ~/.netrc ]; then
    echo "-----> Cleaning up the auto deployment dependancies";
    sed -i.bk '/machine nms-git/d' ~/.netrc;
    if [ -f ~/.netrc.bk ]; then
      rm -fr ~/.netrc.bk;
    fi
  fi
)
command %(
  if [ -d #{fetch(:current_path)}/dockerfiles ]; then
    echo "-----> Deleting the dockerfiles directory";
    rm -fr #{fetch(:current_path)}/dockerfiles;
  fi
)

So I've mittegated this by putting 1 command in the command method. This seems to work. Why and How... no clueue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @reneweteling@PeterVenhuizen

        Issue actions
          assets:precomile fails, but deploy succeeds · Issue #719 · mina-deploy/mina