Testing of the uploading process

[jhsu802701]

This can only be done by someone who is authorized. I'm not authorized, so I cannot do this.

The ultimate goal is to be able to use GitHub Workflows to automatically generate the MX Linux ISOs and upload them. I'm currently working on the former part.

Something to do in the interim is to allow GitHub Workflows to upload small test files to SourceForge and place the badge in the README.md file.

The things to do:

1. Create a new project in SourceForge, perhaps mx-linux-test. This is for all automated uploads from GitHub Workflows.
2. Add "tmp*" to the .gitignore file.
3. Add a script (scripts/upload-setup) consisting of the following code (and replace YOUR_USERNAME):

#!/bin/bash

# NOTE: set -o pipefail is needed to ensure that any error or failure causes the whole pipeline to fail.
# Without this specification, the CI status will provide a false sense of security by showing builds
# as succeeding in spite of errors or failures.
set -eo pipefail

# Creating the test file to upload
mkdir -p tmp0
TIME_STAMP=`date -u +%Y%m%d-%H%M%S`
echo "$TIME_STAMP" > tmp0/timestamp.txt

TYPE='ed25519'
FILENAME="id_$TYPE"
PATHNAME_PRIVATE="$HOME/.ssh/$FILENAME"
PATHNAME_PUBLIC="$PATHNAME_PRIVATE.pub"
PATHNAME_KNOWN_HOSTS="$HOME/.ssh/known_hosts"

# Use default path name for SSH key files.
# Use no password for the SSH key files.
# Piping in the newline character means automatically pressing enter.
echo 'STEP 1: generating the SSH key'
ssh-keygen -t "$TYPE" -N '' -f "$PATHNAME_PRIVATE" -C '[email protected]' <<<$'\n'

# No password needed to log in.
echo 'STEP 2: ssh-keyscan'  
ssh-keyscan -H frs.sourceforge.net >> "$HOME/.ssh/known_hosts"

echo 'STEP 3: Go to https://sourceforge.net/auth/shell_services .'
echo "Copy the contents of $PATHNAME_PUBLIC to SSH Public Keys and"
echo 'click on "Save".'
echo 'Press Enter when finished.'
read

echo ''
echo 'STEP 4: uploading tmp0/timestamp.txt'
rsync -avPz -e ssh "tmp0/timestamp.txt" [email protected]:/home/frs/p/swiftlinux/test-upload-manual/$MX_VERSION/

echo ''
echo "If all went well, the last step (uploading tmp0/timestamp.txt)"
echo 'proceeded without any input from you.'
echo ''
echo 'STEP 5: Configuring GitHub.'
echo '*  Go to this repository in GitHub.'
echo '   Go to Settings -> Secrets -> Actions'
echo '* Copy and paste the contents from your SSH files into the repository secrets.'
echo '   *  Add any required repository secrets that are not already present.'
echo '   *  For the repository secrets that are already present, you must update them.'
echo '   *  Use the chart below as a reference.'
echo ''
echo 'Repository Secret | File Contents'
echo '------------------|----------------------'
echo "SSH_KNOWN_HOSTS   | $PATHNAME_KNOWN_HOSTS"
echo '-----------------------------------------'
echo "SSH_PRIVATE_KEY   | $PATHNAME_PRIVATE"
echo '-----------------------------------------'
echo "SSH_PUBLIC_KEY    | $PATHNAME_PUBLIC"
echo '-----------------------------------------'
echo ''
echo 'STEP 6: Run the GitHub Workflow.  If all goes well, the initial test upload works,'
echo 'which means that you have completed the SSH configuration.'
echo ''

4. Run the above script. If you don't want to replace your SSH keys, just follow all the steps in the above script manually (except 1 through 3, of course).
5. Add another GitHub Workflows process to regularly make small test uploads to SourceForge. I'll get into that in a comment below.

[AdrianTM]

Let's take a step back, let's discuss what we want to achieve and whether we really want to do that in the first place.

Personally, I don't think building ISOs every time there's a script update and upload them automatically to SourceForge (or any other place) is desirable. Do I understand that correctly? Would this be triggered automatically by any script update, or we'll have control when the build is triggered? Also, our official builds require manual interaction and custom .debs that are not placed in Github repo, what would be the purpose to create unpolished ISOs and push them to SourceForge, just for testing?

One more question, since I'm familiar at all with CI backend, do we get that kind of power, networks use and storage for free? It's a lot of data and processing power can you please provide some info about the availability and any potential costs of such setup? I don't want to have our account suspended... (maybe it's just a silly worry).

@dolphinoracle any opinion about this subject?

[dolphinoracle]

at the moment the workflows are triggered on any update of the git repo, I think, plus a scheduled run.

I'm leery of connecting github to the sourceforge account. if the isos are built but not uploaded, where do they go?

I'm also leery of "nigthlies". I don't want to support them, and in our dev model, they aren't necessary.

I'm conservative by nature of our build system, an attitude that has served us well over the last several years. I am interested in the workflows concept, but I'm wondering if its more appropriate for building software packages rather than the whole iso.

[jhsu802701]

Some ideas on the frequency of uploading ISO files:

• Configure the build.yml file to upload ONLY in the master branch. One of the steps in the build process is already configured to run ONLY in the master branch.
• At the moment, the build.yml file is configured (with the cron job) to run the master branch once per week. This could be reduced to once per month or just a few days per year.
• Another option is to run the build script daily but configure the process of uploading the ISO file to run ONLY when a certain condition is met, such as a certain day of the year or only when a certain secret environment variable (such as UPLOAD_ISO) is set.
• GitHub Workflows does NOT store the ISO file unless you use the caching feature.
• Files uploaded to SourceForge can be deleted.

[jhsu802701]

Even if you prefer manual uploads from your local machine, there are still good reasons to use continuous integration, which is why I've been working on this. These reasons are:

1. If something in the process breaks (whether due to a code change or something upstream), using continuous integration limits the time lag between the problem cropping up and the team finding out about it.
2. Continuous integration provides a good common reference setup that addresses the infamous "but it works on my machine" problem. If the process works in the continuous integration setup but not in your local setup, then you know that the latter is the problem.
3. Continuous integration is good for onboarding developers new to the project by providing an example of how the code is supposed to be run.