Static packages in woof-CE

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Static packages in woof-CE

iguleder
This post was updated on .
Yo!

Sometimes when my sleep is disturbed I wake up with a new idea in mind. Yesterday I haven't slept very well and I remember I woke up at 3-4 AM with this idea in mind :)

I have a big bunch of static packages (ksh, gzip, lbzip2, dhcpcd, Dropbear, htop, ircii, iw, less, libarchive, lynx, mandoc, mpg123, nano, ncdu, rfkill, Vile, wget, wpa_supplicant, xz, ytree, zlib and even graphical stuff) for x86_64.

I was thinking, maybe we can put a big bunch static binaries usable by multiple Puppies inside woof-CE (or in a "common"-like repo). Gradually, we can add more and more, until we reach a state woof-CE can build a barebones Puppy which has nothing but a kernel and static binaries.

I guess you wonder - Why do that?
- These static binaries can be used to magically update any older Puppy, because they're static. I mean, you can rebuild Lucid with today's woof-CE and newer packages - it will "just work".
- This allows the *user* to choose the "compatible distro". If Puppy comes without the compatible distro's glibc, the user can use the packages of any distro (but without the ability to mix packages from different distros). I think this is pretty cool. I'm working on a suckless awk/bsdtar-based package manager which supports packages from multiple distros - so far, it's able to install Slackware packages on a 100% static Puppy and they actually work.
- This way, all Puppies have a common base so they share bugs and each bug fixed benefits everyone. Also, we can easily update that base - i.e update Bash to the latest version in all Puppies.

Some time ago, I did an experiment. I took Precise and Slacko, extracted them, replaced many binaries with static ones and re-packaged the whole thing. They worked flawlessly, with reduced RAM usage.

What do you think?

Dima
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

jamesbond3142
I like the idea.

I have a similar idea some time ago; though my list of static binaries is not as extensive as yours, and I didn't follow it to the logical conclusion like you do.

Why not even extend it this way: put all of these static binaries to initrd, together with Puppy scripts?
Then keep initrd as part of the stackable filesystem (the lowest layer, so it can be overriden by basesfs/puppy.sfs).
Puppy.sfs only needs to contain programs and libraries not included in initrd; or may be to override scripts/static binaries in initrd with the "fuller version" of the same.

In the absence of basesfs, kernel+initrd can be run as a "superbare-bones" Puppy.

We can even merge initrd + kernel into one single file, so we have one "vmlinuz" that is virtually fail-proof - whatever happens to the puppy.sfs, you can always boot (to graphical desktop if need be).

Then equip this initrd with your package manager and the ability to pull and build the basesfs itself from any supported distro (the instructions on how to build it e.g. DISTRO_SPECS etc can be pulled out from the web as well). We'll have a puppy that is resilient enough to fix itself when its basesfs is missing or corrupted. The same infrastructure would easily support upgrades and/or switchover to any future puppies (within reason).

Individual flavours of Puppy will still continue - the basesfs can override *everything* that comes from initrd if need be, so nothing is lost on that front. But like you say, it's good to have a unified base for all flavours.

cheers!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

iguleder
The single-file Puppy vision sounds nice, but it's problematic.

The FAT boot image of hybrid BIOS/UEFI ISO images must be <= 32 MB. As far as I remember, it's an ISO9660 limitation. When I experimented with a static 64-bit distro contained in the initramfs, I learnt that the hard way.

However, we still can have two main SFSs, one with static binaries (shared between various flavors) and another, with the compatible distro stuff (unique to each flavor).
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

mavrothal
It is a nice idea but the devil is in the details. Particularly in a distro.
We are obviously are talking about a big base (>32MB) with plenty of packages (+200?) that need to be constantly maintained and updated as needed. That is a major long term task that needs some people to step up the plate and maintain it.
What do you propose woof should have? The static binaries or the build scripts? Scripts is the reasonable proposal but then you also need a well defined _common_ and fail-proof build environment. How do we go about that?
Having a mixed environment of static + flavor specific you may run into lib/app version incompatibility and you may need to update everything. In this case  the static part looks like a waste. Thus a minimal static system may make more sense economy-wise. Fewer packages to build and maintain, less waste if you need to "update" any or all. Tinycore Linux has shown that a 5MB initrd is plenty to boot to a GUI desktop. Isn't it possible to have something similar under 32MB using static packages?
Might be a good idea to start with a list of the static packages that need to be in the base so we have a better idea of how this may materialize.

My $0.02 ;)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

iguleder
There is a build system. A couple of hours ago, I built a 32 bit flavor of
this collection in a hour or so, automatically.

And yes, X in 32 MB is perfectly possible. But there's very little room if
all kernel modules are included - don't forget how limited Tiny Core's
hardware support is. My 64 bit kernel is 13 MB compressed, X is 1-5 MB I
guess and other stuff like init is less than 1 MB.

EDIT: forgot to mention - I can package all these static packages as PETs. I took a look at the packages in "common" and I think it's perfectly possible to rebuild all of them statically against musl and tinyxlib, but there are some problematic packages, like align (which is written in Perl). I think it won't be that hard to strip the problematic packages (about five packages - not too many). For example, look at truncate, which can be easily dropped in favor of head and tail.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

jamesbond3142
On Mon, 21 Apr 2014 02:21:27 -0700 (PDT)
"iguleder [via woof-CE]" <[hidden email]> wrote:
>
> The FAT boot image of hybrid BIOS/UEFI ISO images must be <= 32 MB. As far
> as I remember, it's an ISO9660 limitation. When I experimented with a static
> 64-bit distro contained in the initramfs, I learnt that the hard way.

Ah thanks, I wasn't aware of that. I've now confirmed that 32MB limit, thanks to El-Torito boot specification that only provides 16-bit integer for sector count. :( (to be fair, el-torito was conceived in 1995, and 32MB for a boot loader *was* big at that time. What I don't understand is why UEFI chooses to use a specification from 1995 ...).

But all is not lost - if you can't get the kernel/initrd combo to be less than 32MB, then only the bootloader needs to be in that UEFI image, the initrd/sfs can be outside.

>
> However, we still can have two main SFSs, one with static binaries (shared
> between various flavors) and another, with the compatible distro stuff
> (unique to each flavor).

This is another possible solution. The static sfs will be very small. We just need modify initrd to look for static.sfs if the standard basesfs can't be found (or, alternatively, provide another pfix= code).

The rest of the response assumes we're talking about kernel+static initrd combo, not the separate static.sfs

On Mon, 21 Apr 2014 06:53:37 -0700 (PDT)
"mavrothal [via woof-CE]" <[hidden email]> wrote:

> It is a nice idea but the devil is in the details. Particularly in a distro.
> We are obviously are talking about a big base (>32MB) with plenty of
> packages (+200?) that need to be constantly maintained and updated as
> needed. That is a major long term task that needs some people to step up the
> plate and maintain it.

No, I don't mean we put all of Puppy into one single large file. That would be unworkable.
Just the minimal part of it - kernel+initrd (combined into one). I think we should be able to bring it to less than 32 MB.
The basesfs stays *outside* so it is easily replaceable.

> What do you propose woof should have? The static binaries or the build
> scripts?

Woof already provides static binaries anyway, in initrd-tree0 or something (=e.g. the static busybox).
Just need to add a few more. And I don't think it needs to be updated very often; just like we don't update that initrd static busybox very often.

> Scripts is the reasonable proposal but then you also need a well
> defined _common_ and fail-proof build environment.

No, there should only be *one* build environment - the one to build initrd with the fail-proof built-in.
As well as the rest of Puppy is concerned, they can continue to be unchanged - unless, of course, you want to take advantage of some static binaries already provided in initrd.

> How do we go about that?
> Having a mixed environment of static + flavor specific you may run into
> lib/app version incompatibility and you may need to update everything.

Unlikely, but possible. The point of having static binaries is that you don't depend on anny *libs* at all.

But yes, it is possible that some binaries are required (contrived hypothetical example: abiword requires the real "/usr/bin/ls" instead of the static version; in this case we just include the real /usr/bin/ls inside puppy.sfs which will override whatever is in initrd (initrd is in the lowest layer).

> In
> this case  the static part looks like a waste.

Well, it depends. The very basic Puppy.sfs may not come with Xorg at all, and instead it relies on Xvesa or Xfbdev already in initrd. This will obvious shave a few megs of puppy.sfs. Those that needs high-powered Xorg can download Xorg. It's similar in concept to Xorg/Xorg_High split, except we push it to the extreme.

Of course, for Puppy builds that include Xorg; yes I agree the static bins is a waste. But if memory is a constraint, one can always delete the content of initrd-copy in memory (e.g if Xorg is installed, delete the copy of Xfbdev/Xvesa in memory, for example).

> Thus a minimal static system
> may make more sense economy-wise. Fewer packages to build and maintain, less
> waste if you need to "update" any or all.

The static systems is built once and for all. The point is that it doesn't get updated often. Unless, of course, you want to have a complete source-based build system - like lazyux and others.

On Mon, 21 Apr 2014 07:28:23 -0700 (PDT)
"iguleder [via woof-CE]" <[hidden email]> wrote:

>
> And yes, X in 32 MB is perfectly possible. But there's very little room if
> all kernel modules are included - don't forget how limited Tiny Core's
> hardware support is. My 64 bit kernel is 13 MB compressed, X is 1-5 MB I
> guess and other stuff like init is less than 1 MB.

You don't have to include all drivers, just the minimal ones. Maybe keyboard, mouse, some network and wireless. I don't know what you've got there in 13MB, but it sounds big. The rest stays in basesfs. Fatdog64's kernel is about 4.5MB, that includes basic filesystems (fat/ext4), usb, keyboards, mice. No network/wireless, though.

cheers!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

iguleder
As I'm typing this, I'm building a fresh and updated 64-bit repository of static packages. The small package manager I'm working on is in good shape - I'm going to upload a test build configured to use this repository, so stay tuned.

It should be pretty cool - it's able to install packages in a directory, so theoretically, I can start working on a debootstrap-like thingy on top of it.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

jamesbond3142
On Mon, 5 May 2014 07:10:45 -0700 (PDT)
"iguleder [via woof-CE]" <[hidden email]> wrote:

>
>
> As I'm typing this, I'm building a fresh and updated 64-bit repository of
> static packages. The small package manager I'm working on is in good shape -
> I'm going to upload a test build configured to use this repository, so stay
> tuned.
>
> It should be pretty cool - it's able to install packages in a directory, so
> theoretically, I can start working on a debootstrap-like thingy on top of
> it.
>

Sounds interesting. Looking forward to it.

cheers!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Static packages in woof-CE

iguleder
Loading...