automatically update vendored dependencies

[pohly]

In our various Kubernetes-CSI repos we should regularly update dependencies to ensure that we have included all upstream bug fixes.

This includes:

• running dep ensure -update
• running git subtree pull for release-tools (see https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating)
• creating a new PR for changes (if none is pending currently) or updating an existing PR by force-pushing to the branch

Then once the PR is pending, CI testing will cover the initial round of testing. Even if that fails, having a PR pending is a reminder to the maintainer that the component is not up-to-date.

The steps above could be automated with a script, like update-dependencies.sh in this repo. The hub tool might be useful for managing the PRs. Bonus points for making it possible to update release branches.

Running this script for one or more component is probably best done by the same person at a regular cadence. If multiple people do it, we end up with multiple competing PRs. We could also try to move it into a Prow job, but that would be a second step.

@msau42 does that sound useful?

[msau42]

Yes I think that sounds valuable. It would be great if there was also a wrapper script that can update and create prs across all the branches

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[pohly]

Quick-and-dirty script that I just used to update csi-relase-tools:

#!/bin/sh -x

die () {
    echo >&2 "\nERROR: $@\n"
    exit 1
}

while read repo branches; do
    if [ "$repo" != "#" ]; then
    (
        cd $repo || die "$repo: does not exit"
        git fetch origin || die "$repo: git fetch"
        for i in $branches; do
            git checkout -B prow-update-$i origin/$i || die "$repo:$i checkout"
            git subtree pull --prefix=release-tools -m "release-tools: update" https://github.com/kubernetes-csi/csi-release-tools.git master || die "$repo:$i update release tools"
            changes=$(git log --no-merges --oneline origin/$i..prow-update-$i)
            if [ "$changes" ]; then
                release-tools/update-vendor.sh || die "update vendor"
                git add vendor
                git commit --amend -F - <<EOF
release-tools: update

Commit summary:
$changes
EOF
                make test || die "$repo:$i make test"
                git push pohly prow-update-$i || die "pohly:prow-update-$i push failed - probably there is already an unmerged branch and pending PR"
                hub pull-request -F - -b kubernetes-csi/$repo:$i -h pohly/$repo:prow-update-$i <<EOF
$i: update release-tools

Commit summary:
$changes

\`\`\`release-note
NONE
\`\`\`
EOF
                if [ "$?" -ne 0 ]; then
                    die "$repo:$i pull-request"
                fi
            fi
        done
    ) || die "failed"
    fi
done <<EOF
csi-driver-host-path master
csi-driver-iscsi master
csi-driver-nfs master
csi-lib-utils master
csi-proxy master
csi-test master
external-attacher master
external-health-monitor master
external-provisioner master
external-resizer master
external-snapshotter master
livenessprobe master
node-driver-registrar master
EOF

[msau42]

/lifecycle frozen

[pohly]

In kubernetes-csi/csi-driver-smb#28 we were discussing how to use squashing instead of including the commits as done so far. We came up with a revised script, too. However, when I just tried to do the Go 1.15 update (see PR #97) I found that importing fails if (and only if) --squash was used. It could be because I am pulling from a different repo than usual, but I doubt it. I'll investigate once that PR is merged and in the meantime, continue with importing all commits.

/cc @animeshk08

csi-driver-host-path$ git subtree pull --prefix=release-tools https://github.com/pohly/csi-release-tools.git go-1.15 --squash
From https://github.com/pohly/csi-release-tools
 * branch              go-1.15    -> FETCH_HEAD
Auto-merging release-tools/travis.yml
CONFLICT (content): Merge conflict in release-tools/travis.yml
Auto-merging release-tools/SIDECAR_RELEASE_PROCESS.md
CONFLICT (add/add): Merge conflict in release-tools/SIDECAR_RELEASE_PROCESS.md
Recorded preimage for 'release-tools/SIDECAR_RELEASE_PROCESS.md'
Recorded preimage for 'release-tools/travis.yml'
Automatic merge failed; fix conflicts and then commit the result.

csi-driver-host-path$ git reset --hard origin/master
HEAD is now at 55f7ce61 Merge pull request #193 from msau42/changelog

csi-driver-host-path$ git subtree pull --prefix=release-tools https://github.com/pohly/csi-release-tools.git go-1.15 
From https://github.com/pohly/csi-release-tools
 * branch              go-1.15    -> FETCH_HEAD
hint: Waiting for your editor to close the file... Waiting for Emacs...
Merge made by the 'recursive' strategy.
 release-tools/SIDECAR_RELEASE_PROCESS.md | 6 ++++++
 release-tools/travis.yml                 | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

[Jiawei0227]

Thanks for putting up this script!

I am able to run the script #7 (comment) with some small changes.
The prerequisite for running the scripts is installing hub command. And have all the repos you want to update in place.

One thing people need to notice is that it depends on how their local git remote setup is. My remote setup is origin -> my fork and upstream -> the upstream repo. Which is not working in this script so I have to manually change some of the stuff there. If your setup is origin->upstream and <your github name>->your fork. Then the script above can be easily adopted.

One thing to investigate is the squash thing mentioned #7 (comment).