Skip to content
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5001e06
build debian packages on gh actions
DaAwesomeP Jan 29, 2023
95b9d2f
debian fix python package names, build deps, and paths
DaAwesomeP Jan 29, 2023
ce470f3
ignore codespell line for debian gh actions build
DaAwesomeP Jan 29, 2023
5f7a379
debian gh action add test
DaAwesomeP Jan 29, 2023
1516247
gh actions debian build allow test fail
DaAwesomeP Feb 17, 2023
e48696c
Reorder build command line in autopkgtest; Closes: Debian #913704.
yoe Nov 14, 2018
e06c239
gh actions debian fix missing python tools, proper git, remove allow …
DaAwesomeP Feb 21, 2023
e251bab
gh actions debian allow some builds to fail to allow successful build…
DaAwesomeP Feb 21, 2023
c374bc8
github actions debian parallel build
DaAwesomeP Feb 21, 2023
74a73e7
debian attempt revert python install to dist-packages
DaAwesomeP Feb 21, 2023
de1febd
More updates for python3
yoe Nov 18, 2020
478c16c
Merge remote-tracking branch 'upstream/0.10' into DaAwesomeP-GitHubAc…
DaAwesomeP Feb 26, 2023
21d4321
gh actions debian remove python3 lint tools
DaAwesomeP Feb 26, 2023
7d2fc11
gh actions debian test only python3
DaAwesomeP Feb 26, 2023
98420b6
gh actions debian upload test output
DaAwesomeP Feb 26, 2023
77b73c0
gh actions debian fix codespell
DaAwesomeP Feb 26, 2023
7028dc9
gh actions debian fix test command3 change directory
DaAwesomeP Feb 26, 2023
cdbacd8
gh actions debian always upload build artifacts
DaAwesomeP Feb 26, 2023
c5dbb4a
gh actions debian correct path for rdm tests
DaAwesomeP Feb 26, 2023
9d8fd10
gh actions debian fix ola-rdm-tests dependency ola-python version
DaAwesomeP Feb 26, 2023
c293c89
gh actions debian attempt fix codespell lint again
DaAwesomeP Feb 26, 2023
f435c41
gh actions debian rdm tests test deps
DaAwesomeP Feb 26, 2023
82906ac
gh actions debian rdm tests no python2
DaAwesomeP Feb 27, 2023
69f2914
gh actions debian pybuild env
DaAwesomeP Feb 27, 2023
7bebcac
bump debhelper version
DaAwesomeP Feb 27, 2023
49841fe
bump debhelper version 13
DaAwesomeP Feb 27, 2023
f1fc4cf
bump debhelper version 13 less strict
DaAwesomeP Feb 27, 2023
cfe0772
Merge remote-tracking branch 'upstream/0.10' into DaAwesomeP-GitHubAc…
DaAwesomeP Mar 11, 2023
a44c1de
gh actions debian is it inferring python2 from the shebangs?
DaAwesomeP Mar 11, 2023
586258e
Revert "gh actions debian is it inferring python2 from the shebangs?"
DaAwesomeP Mar 12, 2023
7bb6cb8
gh actions debian try to rewrite python shebangs to python3
DaAwesomeP Mar 12, 2023
b194a1b
gh actions debian try to rewrite python shebangs to python3 tab chara…
DaAwesomeP Mar 12, 2023
4bc3826
gh actions debian add python3 shebang patch to changelog
DaAwesomeP Mar 12, 2023
bacbcc6
Update debian/changelog proper new release build
DaAwesomeP Mar 20, 2023
0b1d815
gh actions debian ola-rdm-tests revert to binary ola-python dep
DaAwesomeP Mar 20, 2023
1fa474b
gh actions debian try without override_dh_clean manual python clean
DaAwesomeP Mar 20, 2023
ae3e9ef
gh actions debian matrix arch
DaAwesomeP Mar 21, 2023
4385e6e
gh actions debian matrix arch test too
DaAwesomeP Mar 21, 2023
0157b0b
gh actions debian rerun
DaAwesomeP Mar 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .codespellignorelines
Original file line number Diff line number Diff line change
Expand Up @@ -195,3 +195,5 @@ import java.nio.ByteOrder;
self.assertEqual('caf\\xe9', StringEscape(u'caf\xe9'))
self.assertEqual('caf\\xe9', ("%s" % StringEscape(u'caf\xe9')))
"forin": true,
dcmd mv ../*ges built/
run: autopkgtest --output-dir=test-output built/*ges -- null
90 changes: 90 additions & 0 deletions .github/workflows/debian.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# This action originally modeled off of the Debian build action:
# https://salsa.debian.org/wouter/ola/-/blob/a38a396f6994b2b1af8efec9e208aee4e67f77aa/.gitlab-ci.yml

name: debian
on: [push, pull_request]
jobs:
debian-build:
name: 'Debian Build ${{ matrix.image_tag }} amd64'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
image_tag: [bullseye, bookworm, sid]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm no Debian naming expert, I guess we want to use static names so people know what they want, although if we used dynamic ones we'd always be testing the bleeding edge in testing.

Copy link
Member Author

@DaAwesomeP DaAwesomeP Mar 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we want to use the static names because every little while everything shifts (i.e. testing becomes stable). We probably want to manually account for those shifts. It is also favorable to name APT repo dists with the OS version name and not stable, testing, etc. because then when the version shifts you are not suddenly receiving broken packages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could cheat it and either build testing too, or ideally check if testing was an alias for one of those other names. Or just remember to check/add occasionally which is probably easier!

Actually presumably the oldest (bullseye?) will fail to build when they shuffle them all on, as its repo may go away?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repo won't disappear right away (it becomes "oldstable" I believe). I'm mainly thinking ahead to possibly having a nightly APT repo where we wouldn't want to break packages. It also makes it easier to go back in time if you are unfortunately stuck with managing an old machine (i.e. you can still find the last build of an old release even if not built currently anymore).

I think we can just do our best to add new releases and remove the old ones when they shuffle down. I can possibly add a check for it in a later PR (just have a Python script read the CI yaml). Personally I install Debian about once every other month so I am happy to submit update PRs for the time being.

container: debian:${{ matrix.image_tag }}
steps:
- name: Get number of CPU cores
run: echo "NUM_CPU_CORES=$(grep -c processor /proc/cpuinfo)" >> $GITHUB_OUTPUT
id: num-cpu-cores
- name: Update package database
run: apt-get update -y
# See comments beginning at
# https://github.com/actions/runner/issues/763#issuecomment-1435474884
# Without Git, actions/checkout@v3 will resort to REST and will not
# create a .git folder or .git.config. The Problem Matcher looks for
# .git/config to find where the root of the repo is, so it must be
# present.
- name: Install Git
run: apt-get -y install git
- uses: actions/checkout@v3
- name: Install build tools
run: apt-get -y install devscripts adduser fakeroot sudo
- name: Install build dependencies
run: mk-build-deps -t "apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends" -i -r
- name: Set up build user
run: |
adduser --disabled-password --gecos "" builduser
chown -R builduser:builduser .
chown builduser:builduser ..
- name: Build
run: sudo -u builduser dpkg-buildpackage -b -rfakeroot -j${{ steps.num-cpu-cores.outputs.NUM_CPU_CORES }}
- name: Move built files
if: always()
run: |
mkdir built
dcmd mv ../*ges built/
- name: Display structure of built files
if: always()
run: ls -alR
working-directory: built
- name: SHA256 built files
if: always()
shell: bash
run: find . -type f -exec sha256sum {} \;
working-directory: built
- uses: actions/upload-artifact@v3
if: always()
with:
name: ola-built-debian-${{ matrix.image_tag }}-amd64
path: ./built
debian-test:
name: 'Debian Test ${{ matrix.image_tag }} amd64'
needs: debian-build
if: "always()" # Run if some builds fail but ensure they all complete first
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if you've changed this, but on a previous test say bookworm had failed, but it still tried to run all the tests. We can't easily make them run as three two step actions, rather than three actions linked to another three actions can we?

Can we if the previous matrix step status?

Possibly one for a later PR...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't easily make them run as three two step actions, rather than three actions linked to another three actions can we?

No, the real limitation is that GitHub Actions has no way to properly define matrix dependencies, only whole-job dependencies. What we can do is probably a better check than always() so that is properly shows those matrix iterations as skipped instead of failed. I will look into it for a future PR.

runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
image_tag: [bullseye, bookworm, sid]
container: debian:${{ matrix.image_tag }}
steps:
- uses: actions/checkout@master
- name: Download build artifact
uses: actions/download-artifact@v3
with:
name: ola-built-debian-${{ matrix.image_tag }}-amd64
path: built
- name: Display structure of artifact files
run: ls -alR
working-directory: built
- name: Update package database
run: apt-get update -y
- name: Install test tools
run: apt-get -y install autopkgtest
- name: Test
run: autopkgtest --output-dir=test-output built/*ges -- null
- uses: actions/upload-artifact@v3
if: always() # Always upload the test output, even on failed tests
with:
name: ola-test-output-debian-${{ matrix.image_tag }}-amd64
path: test-output
2 changes: 1 addition & 1 deletion AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Contributors:
Masaki Muranaka, various patches
Nicolas, for the win32 port of libartnet
Nils Van Zuijlen, typos and python lib small update
Perry Naseck, EPoll delay fix, various GitHub Actions CI workflows
Perry Naseck, EPoll delay fix, GitHub Actions CI workflows, Debian pkg fixes
Peter Newman, MilInst Plugin, Scanlime Fadecandy support and numerous fixes
Ravindra Nath Kakarla, RDM Test Server (Google Summer of Code 2012)
Rowan Maclachlan (hippy) for various changes
Expand Down
10 changes: 10 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
ola (0.10.10-1) UNRELEASED; urgency=medium

* Fix ola-rdm-tests Debian package by patching python shebangs to python3
with dh_python3.
* Reorder build command line in autopkgtest; Closes: Debian #913704.
* This change pulled from Debian by Perry Naseck <[email protected]>
https://salsa.debian.org/wouter/ola/-/commit/223f9ddaf9d4001dfc908734f0641315edbf9ea2

-- Wouter Verhelst <[email protected]> Wed, 14 Nov 2018 23:44:42 +0100

ola (0.10.9-1) unstable; urgency=low

* New upstream release
Expand Down
6 changes: 3 additions & 3 deletions debian/control
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ Source: ola
Priority: optional
Maintainer: Wouter Verhelst <[email protected]>
Uploaders: RenZO <[email protected]>
Build-Depends: debhelper (>= 9), autotools-dev, dh-autoreconf, bash-completion, libcppunit-dev, bison, flex, pkg-config, uuid-dev, python, python-protobuf, libprotobuf-dev, protobuf-compiler, libprotoc-dev, libusb-1.0-0-dev, libftdi1-dev, liblo-dev, libmicrohttpd-dev, libncurses5-dev, libavahi-client-dev, python-numpy
Build-Depends: debhelper (>= 13), autotools-dev, dh-autoreconf, dh-python, bash-completion, libcppunit-dev, bison, flex, pkg-config, uuid-dev, python3, python3-protobuf, libprotobuf-dev, protobuf-compiler, libprotoc-dev, libusb-1.0-0-dev, libftdi1-dev, liblo-dev, libmicrohttpd-dev, libncurses5-dev, libavahi-client-dev, python3-numpy
Standards-Version: 3.9.8
Section: libs
Vcs-Git: https://github.com/OpenLightingProject/ola.git
Expand All @@ -28,7 +28,7 @@ Description: Open Lighting Architecture - development libraries
Package: ola-python
Section: libs
Architecture: all
Depends: ola (>= ${source:Version}), ola (<< ${source:Version}.~), ${python:Depends}, ${misc:Depends}, python-protobuf
Depends: ola (>= ${source:Version}), ola (<< ${source:Version}.~), ${python3:Depends}, ${misc:Depends}, python3-protobuf
Description: Open Lighting Architecture - Python Classes
The DMX512 standard for Digital MultipleX is used for digital
communication networks commonly used to control stage lighting and
Expand All @@ -42,7 +42,7 @@ Description: Open Lighting Architecture - Python Classes
Package: ola-rdm-tests
Section: libs
Architecture: all
Depends: ola (>= ${source:Version}), ola (<< ${source:Version}.~), ola-python (= ${binary:Version}), ${python:Depends}, ${misc:Depends}, python-numpy, libjs-jquery, libjs-jquery-ui, lsb-base
Depends: ola (>= ${source:Version}), ola (<< ${source:Version}.~), ola-python (= ${source:Version}), ${python3:Depends}, ${misc:Depends}, python3-numpy, libjs-jquery, libjs-jquery-ui, lsb-base
Suggests: bash-completion
Description: Open Lighting Architecture - RDM Responder Tests
The DMX512 standard for Digital MultipleX is used for digital
Expand Down
4 changes: 2 additions & 2 deletions debian/ola-python.install
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
usr/lib/python*/dist-packages/ola/*.py
usr/lib/python*/dist-packages/ola/rpc/*.py
usr/lib/python*/*-packages/ola/*.py
usr/lib/python*/*-packages/ola/rpc/*.py
4 changes: 2 additions & 2 deletions debian/ola-rdm-tests.install
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
usr/bin/*.py
usr/lib/python*/dist-packages/ola/testing/*.py
usr/lib/python*/dist-packages/ola/testing/rdm/*.py
usr/lib/python*/*-packages/ola/testing/*.py
usr/lib/python*/*-packages/ola/testing/rdm/*.py
usr/share/man/man1/rdm_*
usr/share/ola/rdm-server/*
usr/share/ola/rdm-server/images/*
11 changes: 9 additions & 2 deletions debian/rules
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,21 @@
export VERBOSE=1

%:
dh $@ --parallel --with autotools_dev,autoreconf,bash_completion,python2
dh $@ --parallel --with autotools_dev,autoreconf,bash_completion,python3

override_dh_auto_configure:
dh_auto_configure -- --enable-python-libs --enable-rdm-tests
dh_auto_configure -- --enable-python-libs --enable-rdm-tests CXXFLAGS='-Wno-error=deprecated-declarations -Wno-error=unused-parameter' pythondir='/usr/lib/python3/dist-packages'

override_dh_installinit:
dh_installinit -p ola --name=olad
dh_installinit -p ola-rdm-tests --name=rdm_test_server

override_dh_makeshlibs:
dh_makeshlibs -V

override_dh_clean:
find . -type d -name '__pycache__' -print0 | xargs -0 rm -rf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought there might be a better clean tool and it seems it's this:
https://manpages.ubuntu.com/manpages/bionic/man1/py3clean.1.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also check what other packages use?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got this from downstream: https://salsa.debian.org/wouter/ola/-/blob/4a0ec01039b06f8ff2d350520a1d031b86e7920b/debian/rules#L27-29

Not many results (although this is not a definitive search) of using py3clean with an override: https://www.google.com/search?q=%22override_dh_clean%22+py3clean

It seems that dh_python3 may run py3clean at some point (prerm-py3clean step) but given that this was deemed necessary downstream I sort of trust that it is deleting something that isn't getting removed otherwise? I suppose this should be easy to test out.

Copy link
Member Author

@DaAwesomeP DaAwesomeP Mar 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a change, I'll check the build for any rouge __pycache__ folders when it completes. https://github.com/DaAwesomeP/ola/actions/runs/4471323397

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I checked all of the *.deb files in the bullseye build and there are not any __pycache__ folders, *.pyc files or *.pyo files!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoe is there some technical reason we're missing why this needs to be included? Maybe because we build and compile files for ola-rdm-tests, but you don't currently clean them correctly as you don't currently install/package ola-rdm-tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoe is there some technical reason we're missing why this needs to be included? Maybe because we build and compile files for ola-rdm-tests, but you don't currently clean them correctly as you don't currently install/package ola-rdm-tests?

tbh, I'm not sure why I added that line and not the py3clean one. For sure the cache and pyc and pyo files are required to not be shipped but be generated on Debian systems, so the fact that we remove them before packaging is not something we can get rid of, but how we do it is very much open for discussion...

dh_clean -a

override_dh_python3:
dh_python3 --shebang=/usr/bin/python3
8 changes: 4 additions & 4 deletions debian/tests/control
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ Depends: ola
Test-Command: ola_rdm_get --list-pids
Depends: ola

Test-Command: rdm_responder_test.py --help
Test-Command: set -e ; for py in $(py3versions -s 2>/dev/null) ; do echo "Testing with $py:" ; $py $(which rdm_responder_test.py) --help ; done
Depends: ola-rdm-tests

Test-Command: set -e ; for py in $(pyversions -s 2>/dev/null) ; do cd "$ADTTMP" ; echo "Testing with $py:" ; $py -c "from ola.ClientWrapper import ClientWrapper; print(ClientWrapper)" ; done
Depends: python-all, ola, ola-python
Test-Command: set -e ; for py in $(py3versions -s 2>/dev/null) ; do cd "$ADTTMP" ; echo "Testing with $py:" ; $py -c "from ola.ClientWrapper import ClientWrapper; print(ClientWrapper)" ; done
Depends: python3-all, ola, ola-python

Test-Command: pkg-config --cflags --libs libola
Depends: libola-dev, pkg-config

Test-Command: set -e; g++ -o linktest $(pkg-config --cflags --libs libola) debian/tests/hw.cc; ./linktest
Test-Command: set -e; g++ -o linktest debian/tests/hw.cc $(pkg-config --cflags --libs libola); ./linktest
Depends: libola-dev, g++, pkg-config
Restrictions: rw-build-tree