FreeBSD packages can disappear from one day to the next

[grahamperrin]

-CURRENT

https://matrix.to/#/!EKNFbsWSwXpDOGLRex:matrix.org/$MdfA8Gvk7rhjk5BBC6tVlsbny7FPH01NapdBXtRdi28?via=matrix.org&via=fosdem.org&via=t2bot.io (2021-01-15):

3 packages disappeared in 13-CURRENT packages...

pkg-static: No packages available to install matching 'virtualbox-ose-additions' have been found in the repositories
pkg-static: No packages available to install matching 'utouch-kmod' have been found in the repositories
pkg-static: No packages available to install matching 'drm-legacy-kmod' have been found in the repositories

Retrospective

Disappearance of drm-legacy-kmod was proper; https://www.freshports.org/graphics/drm-legacy-kmod/#history shows removal.

Looking ahead

1. At any time, seek main-amd64 at https://pkg-status.freebsd.org/builds?type=package
2. then for example, https://pkg-status.freebsd.org/builds/default:default:main-amd64:p565091_sa375ec52a7:beefy18#new_failed New Failed
3. and if you like, compare to the previous build.

[grahamperrin]

End of life (EOL) FreeBSD 12.1-RELEASE

release_2

ISO/build.sh

Line 175 in 4173696

sed -i -e 's|quarterly|release_2|g' "${uzip}/etc/pkg/FreeBSD.conf"

#1 (comment)

… not using this quarter's packages but last quarter's ones, because …

FreeBSD 12.2-RELEASE-p3

quarterly and default

… quarterly can have missing packages any day...

1. At any time, seek 122amd64 at pkg-status.freebsd.org/builds?type=package
2. then for example, http://beefy2.nyi.freebsd.org/jail.html?mastername=122amd64-quarterly for quarterly – or http://beefy6.nyi.freebsd.org/jail.html?mastername=122amd64-default

When a port was updated, it can take some time until the binary package gets updated. How much time? I don't know.

Exceptionally (for an initial build of as much as possible): poudriere might take around four days. See for example:

• http://beefy2.nyi.freebsd.org/build.html?mastername=122amd64-quarterly&build=563981 95:37:36
• http://beefy6.nyi.freebsd.org/build.html?mastername=122amd64-default&build=563982 96:41:58

Normally: poudriere can complete its work in far less time. See for example:

• http://beefy2.nyi.freebsd.org/build.html?mastername=122amd64-quarterly&build=564755 23:59:00
• http://beefy2.nyi.freebsd.org/build.html?mastername=122amd64-quarterly&build=564922 02:57:47
• http://beefy6.nyi.freebsd.org/build.html?mastername=122amd64-default&build=564752 52:42:55

[grahamperrin]

#1 (comment)

… this quarter's packages can disappear from one day to the next, making it impossible to build the ISO).

We still need to find a better way for this. …

#135 (comment)

… cannot use quarterly packages because packages can disappear there from one day to the next, preventing us from doing ISO builds. …

The key to avoiding disappointment will be knowledge of fallout of required packages.

Fallout

We see https://lists.freebsd.org/mailman/listinfo/freebsd-pkg-fallout – freebsd-pkg-fallout -- Fallout logs from package building – however the last post was in 2019.

I do receive fallout-related e-mails. Thirteen hours ago, for example:

[Bug 253484] [exp-run] Update KDE Frameworks to 5.79.0

Re: https://bugs.freebsd.org/bugzilla/show_activity.cgi?id=253484 I'm not a cc recipient of such things, so I need to remind myself of the method through which I chose to receive notifications. Something in Bugzilla, I guess.

[probonopd]

The key to avoiding disappointment will be knowledge of fallout of required packages.

The actual key will be to get to some sort of setup that prevents Fallout from happening. After all, we do not want to know that we cannot build, we want to avoid the situation that we cannot build...

I certainly don't want to keep track of which packages are missing today, and need to be able to build at any time. Not being able to build for a couple of days would be a showstopper for me.

I can't remember that I had ever track down missing packages in e.g., Debian Unstable...

[grahamperrin]

Fair enough,

… Not being able to build for a couple of days would be a showstopper for me. …

Maybe use poudriere to maintain, for that build purpose, a minimal repo.

Will you have a separate computer for builds, or are you limited to 12.1?

Maybe some overlap with #51

[probonopd]

Will you have a separate computer for builds, or are you limited to 12.1?

In fact, my "computer" that does all the building is https://cirrus-ci.com/ and they do a fantastic job at it. Much better than I could ever do.

[grahamperrin]

Is disk space a concern?

[probonopd]

Well, it's a continuous build pipeline without persistent storage.

[probonopd]

Looks like we are not the only ones frustrated by this. @pkgdemon pointed out on IRC:

but the reason i dropped furybsd (...) is things like drm-kmod stay broken in quarterlies for each point release for an enitre quarter. on top of that desktop packages, and even browsers were randomly broken. it lead me to conclude that the only way forward was to invest in a build infrstrucutre to be able to build and test my own
packages and push when they were ready. as much as i hoped stock would work. therefore i joined ghostbsd and rolled my work into that since they already had the donations to support a good build mirror/infrastructure. i'm slowly fixing a lot of the openrc bugs there and things are becoming a lot more polished and on the way to acting more like stock behavior.

[grahamperrin]

NB also the comments on latest. Me:

Interesting observations on quarterly. I very rarely notice breakages with latest.

@pkgdemon replied:

Yeah I’m starting to think also latest is better kept

I’m also using latest now

[crees]

The fallout really falls on the ports committer who committed broken code... though it would be nice for the 'working' previous package to not disappear. I'll find out some more about exactly why this happens.

By the way, you should just remove drm-legacy-kmod as it's not relevant or necessary, but my latest pull request deals with that anyway. #165

[grahamperrin]

#181 (comment) emulators/virtualbox-ose-additions is available in latest and quarterly for FreeBSD:13:amd64

[probonopd]

@igalic suggested to mirror the needed packages on a private server:

we could probably get away with: freebsd/poudriere#798 and freebsd/poudriere#797

Does anyone have a web server (that can be expected to stay around for a prolonged area of time) with good bandwidth and a few leftover GB of storage?

[probonopd]

Today 13.0 based ISO builds are broken because:

pkg: No packages available to install matching 'py37-xdg' have been found in the repositories

https://cirrus-ci.com/task/5147859336036352?logs=Environment#L51

[probonopd]

pkg: No packages available to install matching 'sha' have been found in the repositories

Super annoying!

These moments when essential packages in FreeBSD disappear from one day to the next... should be fixed at their root cause. Why not keep the last build, and only replace it with a new build once there is one? Packages NEVER should go missing.

No builds today.

[probonopd]

Turns out the binaries from the sha packages are now in base and hence the package was removed.

[probonopd]

Worse:
https://www.freshports.org/commit.php?message_id=97cf09b1df1318292899840cfc300d5072a1d670

Apparently FreeBSD has a policy that requires software to be constantly updated in order to remain in ports and packages.

This makes it impossible to have "complete" software (that has reached its final, as-perfect-as-it-gets state and is no longer constantly changed) in FreeBSD ports and packages?

I happen to use a lot of such software (and in some ways even prefer it to software that is an ever-changing "moving target"). Which is the least problematic (to me) because it doesn't change anymore and either has no bugs or only known bugs.

Point in case: ssvnc. https://www.freshports.org/net/ssvnc/

net/ssvnc: Deprecate and set expiration date to 2022-03-31

Abandonware, last release in 2011 and broken functionality
reported upstream

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260716

it has been serving us well for 11 years without updates. Even if it has some "broken functionality" I did not run into it, and it did not affect my use of it.

I actually started to code a whole new appliction around net/ssvnc only 1.5 years ago, assuming net/ssvnc would no longer change (a good thing if you want to develop something around something).

Looks like we need to scan for such messages somehow. Can it be automated @rene0? I am looking for a command that would scan my entire system for packages which might be candidates for removal. Ideally I would like to know this a lot of time before they actually get removed. Having just one quarter "early warning" is not sufficient!

Edit: cd /usr/ports ; grep -r "EXPIRATION_DATE=" . | cut -d ":" -f 1 | cut -d / -f 2-3 can do it if you have ports installed. I am looking for something that works without the ports tree being installed, and that gives me this information for the packages on my system.

Would it be possible to have an "abandonware" section in FreeBSD ports where such software might remain until it no longer can be built? This way the software would not disappear from packages from one day to the next, but it would still be clear to everyone that the software is no longer updated upstream.

[probonopd]

So, all of those are at the risk of disappearing soon...

FreeBSD% grep -r "EXPIRATION_DATE=" . | cut -d ":" -f 1  | cut -d / -f 2-3
dns/bind916
dns/bind9-devel
dns/bind918
games/pouetchess
games/cultivation
games/critterding
Tools/scripts
audio/tuxguitar
audio/exaile
textproc/php-mecab
ports-mgmt/portfmt
multimedia/ccextractor
multimedia/php-pHash
multimedia/schroedinger
archivers/p7zip
science/gramps
graphics/magnum-extras
graphics/py-pyglet150
graphics/magnum-examples
graphics/magnum
graphics/magnum-plugins
security/libgringotts
security/ruby-bitwarden
mail/rubygem-tmail
math/py-cdecimal
math/pecl-stats2
deskutils/osmo
www/typo3-10
www/py-rackspace-monitoring
www/py-hyper
www/py-django-babel
www/phalcon4
www/py-gandi.cli
www/py-django-sudo
www/py-django-bootstrap-form
www/shttpd
www/py-django22
www/py-jsonfield2
www/py-django-bulk-update
www/py-django-happenings-update
www/py-django-compat
www/py-django-common-helpers
www/py-django
www/py-django-cron
www/ilias6
emulators/kcemu
net-im/jabber.el
net-im/py-slackclient
net-im/pecl-stomp2
ftp/php-fastdfs
net-mgmt/zabbix6-frontend
net/gogoc
net/pecl-zmq
net/pecl-amqp
net/pecl-mosquitto
databases/php-xapian
databases/pgpool-II-37
databases/pecl-cassandra
databases/pecl-couchbase
databases/py-bsddb3
databases/db5
databases/pgpool-II-36
lang/perl5.32
lang/gcc6-aux
lang/gcc10-devel
lang/python2
lang/python27
lang/php74-extensions
lang/perl5-devel
lang/perl5.34
lang/gcc9-devel
lang/php74
lang/perl5.30
devel/pecl-APCu_bc
devel/ioncube
devel/py-meliae
devel/py-apns2
devel/pecl-jsmin
devel/pecl-jsond
devel/pecl-hrtime
devel/avalon-framework
devel/itext
devel/pecl-eio
devel/qbs
devel/pecl-propro2
devel/py-aiorpcX-legacy
sysutils/cfengine317
sysutils/cfengine-masterfiles316
sysutils/ansible2
sysutils/ganglia-webfrontend
sysutils/cfengine-masterfiles317
sysutils/cfengine316
sysutils/mgeupsd
print/pecl-pdflib

Which is a shame. Just because Python 2 will not change anymore it does not meen it should be eradicated. There is still software that uses it which will never be rewritten... just as an example.

[crees]

The Python issue has been discussed at length, and I've not really an opinion on it other than the fact that it's now abandoned, but I don't think that's something that will change. Debian has also removed it.

I know I've been less than active recently, but if there's anything you need rescuing from removal, I'm happy to work on those-- a lot of the time it just needs to have someone willing to answer questions/fix bugs on it, which a dead website is usually indicative of an absence of it. I've not touched security/cfs in a while or sysutils/runit, but because they have a person who will usually respond to emails about it, they are able to stay. It's no work for me being maintainer of either, because they are both extremely well written-- I adore Runit and it's the first thing I install on all of my systems after sudo and vim.

By the way, you want grep -R [a-z]* rather than ., as Tools/scripts is unlikely to be removed :)

[probonopd]

As of today, qt5-location seems gone:

pkg-static: http://pkgmir.geo.freebsd.org/FreeBSD:13:amd64/quarterly/All/qt5-location-5.15.2p6.pkg: Not Found

As a result, we cannot build ISOs today. Was working yesterday.

Could the error message at least point one to the URL that would contain more information about what happened to a package in those cases? Will qt5-location re-appear anytime soon?

[probonopd]

Today is not the beginning of a new quarter yet the so-called "quarterly" package set is broken, having missing packages, rendering me unable to produce a helloSystem ISO. This frustrates the hell out of me!

A simple measure to this that FreeBSD could take would be to freeze quarterly packages at the beginning of a quarter. After all, if I wanted latest packages, I'd use those...

Even better would be to have, in addition to latest and quarterly, things like 1Q2022, 2Q2022,... etc. This way I would now be using 2Q2022 which would be reflecting the state of quarterly at the end of Q2. Not changing anymore. With no risk of packages going missing. This would not cause much additional work to the FreeBSD project at all, just require some storage space.

[probonopd]

As of today, the hplip package is missing for FreeBSD:14:amd64.

https://www.freshports.org/print/hplip says:

What does this mean?
Can one find out what is blocking it from being there?

(I would not be surprised if installing the FreeBSD 13 package for hplip would also work on FreeBSD 14...)

[grahamperrin]

"quarterly" package set is broken,

No, it's not.

The FreeBSD Ports Management Team portmgr@ manages things for "thousands, if not tens of thousands, of user installations that track the ports tree on a daily basis".

It may help to visualise https://github.com/helloSystem/ as just one of the countless consumers of FreeBSD:13:amd64 quarterly. Consumers with diverse wishes, diverse requirements, some of which are naturally in conflict.

It is, simply, impossible for a single repository to completely fulfil everyone's requirements 24/7/365. I like to think of most repositories for Tier-1 platforms as being quality assured to fulfil the wishes of as many users as possible.

[probonopd]

With my name prominent as author of this issue, I'll prefer the originally given title.

I certainly respect your wishes.

I am always trying to find concise titles that summarize what the issue is, in other words: What needs to be fixed so that the ticket can be closed.

Example:
Instead of "Color of the error message" I would write something like "Error messages must be red" or "Error messages are sometimes green" so that one immediately recognizes what is the bug/issue/problem.

[probonopd]

On FreeBSD 14, ISO generation is still broken with pkg-static: No packages available to install matching 'hplip' have been found in the repositories.

Like a week ago, #141 (comment).

So, doing crude trickery for now:
93cb8e3

[grahamperrin]

hplip

[probonopd]

Thanks @grahamperrin. How did you find that page, and what does it tell us?

It would be enormously helpful if that page was linked to from the empty boxes in

https://www.freshports.org/print/hplip:

Opened a feature request upstream:

• Link to the reason if a package is not available for an arch FreshPorts/freshports#384

[grahamperrin]

How did you find that page, and what does it tell us?

Q&A, helloSystem/hello#382

[probonopd]

Today we can't build the FreeBSD 14 based ISO because pkg-static: No packages available to install matching 'virtualbox-ose-additions' have been found in the repositories.

Let's see if I can figure it out:

• Go to https://pkg-status.freebsd.org/?type=package&all=1
• Search main-amd64 (which is FreeBSD 14)
• It says beefy18, which is the machine on which FreeBSD 14 packages get built for amd64
• Go to its latest build, which is http://beefy18.nyi.freebsd.org/build.html?mastername=main-amd64-default&build=pff8f3a1b75aa_s1b602f641a
• Search for virtualbox-ose-additions in all search boxes on that page; it is found under "Queued ports"
• What does this tell us now? Why is this package missing today?

[probonopd]

• How to find the reason for missing packages? hello#384

[grahamperrin]

The key to avoiding disappointment will be knowledge of fallout of required packages.

See

• How to find the reason for missing packages? hello#384 (comment)

[probonopd]

As of today, we still cannot build FreeBSD 14 based ISOs due to

pkg-static: No packages available to install matching 'virtualbox-ose-additions' have been found in the repositories

[probonopd]

Thanks @grahamperrin for having submitted a bug for the missing 'virtualbox-ose-additions' upstream:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267079

[grahamperrin]

Spinning off:

• How to find the reason for missing packages? hello#384 (reply in thread)

[probonopd]

Unfortunately, I can't gather any insight as to why gimp-gmic-plugin suddenly disappeared.

[probonopd]

Today we can't build the ISO because the hw-probe is missing in quarterly packages (Q12024) and there is no explanation on https://www.freshports.org/sysutils/hw-probe/ that I could understand...

This, plus the fact that GPU kernel modules are not released in sync with base, are the main showstoppers for new helloSystem releases at the moment.