Add support for LZ4 compressor #4289
Conversation
| PKG_LICENSE="BSD" | ||
| PKG_SITE="http://www.lz4.org/" | ||
| PKG_URL="https://github.com/Cyan4973/lz4/archive/$PKG_VERSION.tar.gz" | ||
| PKG_DEPENDS_TARGET="toolchain" |
There was a problem hiding this comment.
PKG_DEPENDS_HOST="" and remove PKG_DEPENDS_TARGET
|
For older kernels without LZ4 support, you need to add backported patches. See the LZ4 and squashfs related patches here: https://github.com/arco/linux-amlogic The squashfs options I use for my Amlogic S805 based stick is: Which gets OpenELEC booted in around 17 seconds. |
Compared with how long without? What's the benefit of adding this package? Nothing so far in this PR says why this is needed or why it's a good idea. Edit: I see you mention some benefit for LZ4 in #4288 - some hard timing numbers on any actual benefit would seem like a minimum before making this kind of change. Just my humble opinion though. |
|
For my device, Amlogic S805 based, with using gzip the boot is around 25 seconds, so there's a clear improvement by using LZ4. The benefits are improved compression/decompression speed, with the latter as being most important in OpenELEC's case. There's some comparison numbers here: https://github.com/Cyan4973/lz4 |
|
OK thanks, that's the kind of information that would have been useful in the first post. :) And you're proposing "LZ4 HC" as the compressor, if I'm reading the second commit correctly ( |
|
I'll update it with some more info. :) Yeah, -Xhc gives the best compression ratio, while keeping the good decompression speed. |
How are you determining the boot duration - is this the time to start kodi? eg. time of first log entry, or "potato time" when the GUI appears? It would be useful to know what your reference is when trying to compare against other systems - for instance, with existing squashfs defaults, an RPi2 will start kodi.bin in under 8.5 seconds (first kodi.log entry): (You can add monotonic time logging to kodi with this patch) |
|
Trying to build with this PR and getting an almost immediate build failure (build log) while building In |
|
http://askubuntu.com/questions/470796/fatal-error-sys-cdefs-h-no-such-file-or-directory Btw, I time it from the u-boot logo appears on the TV, until Kodi GUI appears. With a stop-watch, so yeah, it's kinda "potato-time" and not very scientifically. :) 8.5 seconds for kodi.bin to start? That can't be correct. For me the log says: 02:53:06 10.872539 T:3040669696 NOTICE: special://profile/ is mapped to: special://masterprofile/ But when I use stop-watch to clock it from Kodi logo appears until GUI appears, it takes about 3 seconds. If I study the log, and extract the difference between the following two lines, that seems more right. Not sure why it starts with the number 10 though. 02:53:06 10.872773 T:3040669696 NOTICE: Starting Kodi (15.1 Git:f4dda26). Platform: Linux ARM 32-bit |
|
On "limited" hardware it may be beneficial to run squashfs with no compression. It results in double the file-size on disk but you keep the one-file packaging benefits and if you gzip the installer img there's only a ~10% increase in the file users are downloading. The additional overhead from the compression is (was) enough to cause all manner of problems with the old AppleTV build. The benefit is probably less on multi-core systems with more RAM available but other low power hardware may benefit. In all cases it would serve as a reference for how much penalty compression adds. |
LZ4 is a lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems. Benchmark comparison and more info at: https://github.com/Cyan4973/lz4
Strange that this is the only package with this issue. Or maybe other packages solve this same issue without needing additional packages installed on the build system. It would be great if this package could build on existing build systems.
Yep, it's correct - just timed it with a stopwatch, and cross referenced against entries in journalctl to ensure it is correct. The monotonic time recorded in
Because on your system it takes 10.872539 seconds to get the system to a point where it can start Remember the system has to boot the kernel and start all the systemd services before it can start Kodi - the monotonic time in |
|
That's odd about the build failure. No issues here on Linux Mint 17.2, based on Ubuntu 14.04. And in OpenELEC build folder, which I guess is what it's actually using: And yep, you're right about the timing. |
In |
What on Earth is your kernel doing for 5.5 seconds? :)
I've uploaded a dmesg from the RPi2: http://sprunge.us/ASVS - maybe you can spot where the delays are occurring in your kernel startup. Maybe this kind of discussion should move to the forum. |
|
Could be because I run kernel 3.10, and that yours 4.1.6 has some major improvements. Or it could be the systemd-modules-load.service which takes almost 4 seconds. If it's counted with the kernel timing that is. I'm not sure. |
|
Priority now would be to resolve the build issue... |
|
By the way, the options you say you're using are already the defaults on all projects so any difference is presumably just down to |
|
Yeah, too easy to get carried away. :) Just finish that off with saying it looks by the kernel log that there's a lot of nand discard/recovering going on, that seems to take time. Maybe the nand chip is in not so good shape. About the build issue, I'll try to see if I can find something. Clearly there must be more package sources that includes the features.h file... |
|
Honestly I see no harm in having this added, however, I did test all this before and realized it was useless because it doesn't save space or time compared to other methods. |
|
@MilhouseVH Yeah, those options doesn't exist in legacy kernels, like 3.10 which I'm stuck with, so some stuff had to be backported. I guess those are also responsible for some of the improved speed i'm seeing. Regarding the build error, I'm pretty sure you need to install the package providing /usr/include/sys/cdefs.h, which in Ubuntu is libc6-dev-i386. This again should be a dependency of gcc-multilib. For example, in the case of Android, it's a build requirement to have it installed. |
Hmmm... I dunno, this just feels wrong... installing 32-bit libraries to solve a build problem on a 64-bit system doesn't seem like the right solution. I'm just really quite loathe to use this sledgehammer solution right now, which may be masking the correct solution, besides I'm out of time for testing - maybe later in the week (and hopefully someone can suggest a less dramatic build solution). |
|
/usr/include/sys/cdefs.h is actually just a symlink to ../x86_64-linux-gnu/sys/cdefs.h (does this exist on your system?), which is part of the libc6-dev:amd64 package. If i delete cdefs.h from /usr/include/sys/, it still builds fine here, as long as the file in /usr/include/x86_64-linux-gnu/sys/ exists. |
|
|
Well, I'm baffled. It's amazing that anything at all compiles for you, when taking into the fact that cdefs.h is included from features.h, which in turn is included from stdlib.h, which in turn is included in many many sources. I removed libc6-dev-i386 completely, with all its dependencies, and lz4 sources still compiled fine here. My gcc configuration, which you can see contains the path to the x86_64 include files. |
No description provided.