EasyOS version 2.3.2, June 22, 2020

[lp-dolittle]

@ Barry,

maybe, the following observation is of interest to you after your success in starting a Xorg GUI in initrd:

Running totally in RAM EasyOs 8.0.1 or Quirky Xerus64-8.6, I noticed some interesting differences, as for the keyboard layout settings.

(1) In EasyOS, keyboard layout settings via the setxkbmap command in 'Sakura' always need to be refreshed after every restart of the X sever.
In contrast, running Quirky Xerus64-8.6, the same settings mostly seem to survive X server restarts but not on every machine. Comparing the hardware-info of such laptops and pc's, I so far was not able to make out which hardware component might cause these different reactions

(2) When I plug in a usb-keyboard into laptops whose (internal) keyboard layout either needs or does not need a refresh after X server restarts, sakura responds equally to both input devices as long as the external keyboard is already attached during the setting.
The situation is different if I first set the wanted keyboard layout via setxkbmap and plug in the usb-keyboard, thereafter. On laptops whose keyboard layout settings need to be refreshed after X server restarts, I now have the option to toggle in characters which belong to different layouts (in my case 'Swiss German' according to the laptop keyboard or 'US American' according to the usb-keyboard) into one and the same sakura window!! On laptops whose keyboard layout survives X server restarts (running Quirky Xerus) the internal and the external keyboard commands always are responded equally by sakura even if the usb-keyboard is plugged in only after the specific setxkbmap command.

kind regards

[rufwoof]

Any chance of having lz4 support being added to the default EasyOS kernel? So a easy sfs compressed using lz4 is mountable.

I have my frugal HDD installed version using lzo level 1 compression (... sfs/easyos/oe/pyro/easy_1.0_amd64.sfs) and its real quick. lz4 can further improve upon that.

I've set it not to copy the easy sfs to ram on bootup as well, so starts up quicker. I prefer it that way as given a slower boot (easy sfs loaded into ram), faster first time run of programs, versus faster boot/slower first time run of a program (thereafter often stays cached so second time runs are comparable in speed to having had easy sfs copied to ram), the latter uses less ram in total as the former loads everything into ram whereas the latter only loads what is actually used (for instance why load gparted into ram if you don't do any disk partitioning/changes during a session ... wasteful).

[rufwoof]

Some weeks back, someone asked about EasyOS and cgroup's so that limits could be set for a Easy Container for swap, memory ...etc.

I've been toying around in EasyOS with cgroups and so far these are my notes/tests that I've run so far, with a few of adding cgroup controls into my own frugal EasyOS

Code: Select all

# Check if cgroups are active

ls /sys/fs/cgroup/

# If (likely) no return is seen, then cgroup needs to be created/mounted ...

# Prepare the cgroups
mount -t tmpfs none /sys/fs/cgroup
mkdir /sys/fs/cgroup/memory
mount -t cgroup none /sys/fs/cgroup/memory -o memory

# Make a new 'easy' cgroup
mkdir /sys/fs/cgroup/memory/easy

# now we should be able to ls /sys/fs/cgroup/memory/easy and see ....

ls /sys/fs/cgroup/memory/easy
blkio.io_merged
blkio.io_merged_recursive
blkio.io_queued
blkio.io_queued_recursive
blkio.io_service_bytes
blkio.io_service_bytes_recursive
blkio.io_serviced
blkio.io_serviced_recursive
blkio.io_service_time
blkio.io_service_time_recursive
blkio.io_wait_time
blkio.io_wait_time_recursive
blkio.leaf_weight
blkio.leaf_weight_device
blkio.reset_stats
blkio.sectors
blkio.sectors_recursive
blkio.time
blkio.time_recursive
blkio.weight
blkio.weight_device
cgroup.clone_children
cgroup.event_control
cgroup.procs
cgroup.sane_behavior
cpuacct.stat
cpuacct.usage
cpuacct.usage_all
cpuacct.usage_percpu
cpuacct.usage_percpu_sys
cpuacct.usage_percpu_user
cpuacct.usage_sys
cpuacct.usage_user
cpu.shares
devices.allow
devices.deny
devices.list
memory.failcnt
memory.force_empty
memory.kmem.failcnt
memory.kmem.limit_in_bytes
memory.kmem.max_usage_in_bytes
memory.kmem.slabinfo
memory.kmem.tcp.failcnt
memory.kmem.tcp.limit_in_bytes
memory.kmem.tcp.max_usage_in_bytes
memory.kmem.tcp.usage_in_bytes
memory.kmem.usage_in_bytes
memory.limit_in_bytes
memory.max_usage_in_bytes
memory.move_charge_at_immigrate
memory.oom_control
memory.pressure_level
memory.soft_limit_in_bytes
memory.stat
memory.swappiness
memory.usage_in_bytes
memory.use_hierarchy
notify_on_release
release_agent
tasks

# Set the amount of memory which tasks in the easy cgroup can use :
echo 100000000 > /sys/fs/cgroup/memory/easy/memory.limit_in_bytes
# and set the cgroup to also have swap turned off ... 
# See what currently is allocated swap

cat /sys/fs/cgroup/memory/easy/memory.swappiness
# mine returns 60

# and set it to zero
echo "0" > /sys/fs/cgroup/memory/easy/memory.swappiness
# ... or whatever limits/things you want to apply to the cgroup task(s)

# start a new shell, so we can later exit out of it 
sh

# Move the current (new) shell process into the easy cgroup
# so that all other processes started in this shell are automatically also
# moved into the easy cgroup:

echo $$ > /sys/fs/cgroup/memory/easy/tasks

### THIS IS WHERE WE WOULD START THE EASY CONTAINER I.E. 
### AS A MEMBER OF THE EASY CGROUP (THAT HAS WHATEVER
### LIMITS WE SET BEING APPLIED).

# and see what we're permitted for swap now
cat /sys/fs/cgroup/memory/easy/memory.swappiness
# should return 0

# We set a 100,000,000 byte memory limit above, so create a python script
# that eats memory (I called mine eat-mem) ...

f = open("/dev/urandom", "r")
data = ""

i=0
while True:
    data += f.read(10000000) # 10mb
    i += 1
    print "%dmb" % (i*10,)

# no need to make it executable as
# we run that with python ./eat-mem
# ... and you should see ...
10mb
20mb
30mb
40mb
50mb
60mb
70mb
80mb
Killed

# cgroups can’t be removed until every processes in the tasks file has exited or
# been reassigned to another group. 
# Exit the shell to exit the easy cgroup
exit

# now back at the original shell level
cat /sys/fs/cgroup/memory.swappiness
#should see 60 (or whatever swap was previously available
# also if you now run the eat-mem script, it should get a lot further
# (I ctrl-c'd out of it once it had clearly passed 100MB).

# DON'T rm -rf the /sys/fs/cgroup/memory/easy FOLDER
# ... just use rmdir instead.

Note that in the above I disabled swap, as otherwise even with a memory limit it could start using swap to appear to have access to more than 100MB of memory.

Of course, you can also set other things such as attaching the easy containers cgroup to a particular core(s)
cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-3

cpuset.mems="0"
cpuset.cpus="0-1"

... etc

[rufwoof]

This is what my ec-chroot-easy now looks like (restricts easy container maximum memory to 500MB (approx), and disables the easy container access to swap

Code: Select all

#!/bin/sh

# if not already set up a easy cgroup then ...
TST=`mount | grep "/sys/fs/cgroup/memory"`
if [ -z "$TST" ]; then

	# enable cgroup's
	mount -t tmpfs none /sys/fs/cgroup
	mkdir /sys/fs/cgroup/memory
	mount -t cgroup none /sys/fs/cgroup/memory -o memory

	# create a easy cgroup
	mkdir /sys/fs/cgroup/memory/easy

	# limit memory and disable swap to easy cgroup
	echo 500000000 > /sys/fs/cgroup/memory/easy/memory.limit_in_bytes
	echo "0" > /sys/fs/cgroup/memory/easy/memory.swappiness

fi

# attach this pid to the easy cgroup before running the easy container
echo $$ > /sys/fs/cgroup/memory/easy/tasks
empty -f ec-chroot easy

Running a eat-mem script within the container now gets to 440MB shown, stalls for a bit, then shows 'Killed'. So now a runaway program or hack attempt inside the easy container doesn't lockup the main system.

EDIT: After some playing around and setting cgroup limits etc. I decided to leave the code in place (for reference), but commented out. The easy container all too easily became 'jerky' in actual use - most likely due to my (poor) choices of tweaks/settings, whilst the standard (commented out code) ... is just sweet/great, works (very) well.

[rufwoof]

I created a Debian bootstrap ... so minimal debian (cli). 120MB .gz filesize. Official Debian (main repo's only).

dir2sfs'd that folder to form a EasyPack (sfs), ensuring that the content was all in a sub-directory (otherwise when you load it, it fails (system crashes). Booting with that sfs loaded and I've set it so it appears as /debian-stretch.

Adding a Xephyr :1 (resizeable window) I'm chrooting into that using pflask and it works really well. I've added (Debian apt-get) xorg, mpv, firefox-esr and it all runs as well as the main Easy system, no lag apparent etc. ... at least when using cwm for the window manager (if I use jwm it does seem to conflict in some areas with the main EasyOS's jwm).

Currently I've set the chroot to have all capabilities. But I guess when I run firefox-esr within that I should first be dropping the chroot capability as part of firefox-esr invocation.

Great how EasyOS acts as a hypervisor - such as being able to drag/drop files to/from EasyOS into the debian chroot. I find that having the debian running on desktop 2 serves as a visual reminder of which OS your actually using. Also nice having access to fully patched up choices (run debians apt-get update;apt-get upgrade), along with the full debian repo.

[zygo]

Barry,

Could you compile Hiawatha with xslt support. This would allow it to produce the directory index in HTML. Then it would be compatible with sfsget.

Edit: Sorry: HTML directory index works but no sfs found by sfsget.

Thanks

[rufwoof]

I dropped unsquashfs into the initrd (used the main sfs version and copied over the ldd's for that into initrd's /lib folder - expands initrd to around 8.3MB (non compressed)).

Testing main session snapshots using lzo level 1, lz4 and lz4 high compression ... had respective compressed file sizes of 376, 435, 368 MB (noted purely for relative comparison).

I've tweaked the /sbin/rollback script within initrd so that when doing a rollback of the main session, instead of rm -rf of the existing .session it just mv's that to a .delme version and kicks off a rm -rf of that .delme folder in the background ... so the boot is less held up wating to wade through 1GB or whatever of deleting existing .session folder content

I've also replaced the mount and cp existing method of roll back (.session) with a unsquashfs.

Overall considering time to create a snapshot (choice of compression) and time to rollback a session within the initrd ... I've settled for a choice of -comp lzo -Xcompression-level 1 for snapshots created within easy-version-control. Takes around 30 seconds with the unsquashfs progress bar being shown when initrd rolls back that 376MB snapshot sfs on my (11 year old HDD frugal install) setup.

That's for a debian bootstrapped save folder content, with firefox-esr/cwm installed. I've noticed that with use (firefox cache) the save folder (.session) can grow to 750MB+ compressed (lzo level 1) size quite easily (and potentially considerably more if other Debian packages were installed/used (Openshot, Libre, Audacity, Gimp ..etc.), so the standard/as-is EasyOS method of rm -rf and then mount/cp the .session when using considerably slower gzip ... could/will lead to considerably slow reboots when rolling back for some. lzo level 1 based compression/decompression seems to me to be the better choice for reducing down those delays, at least for HDD frugal based EasyOS setups. Especially if Iike me you rollback often (under Puppy things changes are stored in (finite) ram and you can set it to only (optionally) save at the end of a session, so if you have a clean version you might just shutdown without saving, so the next reboot is also clean. With EasyOS the approach is different (changes are stored on disk - so only restricted by available disk space), set things up, and create a snapshot (sfs) of that clean version, use it, then roll back (unsquashfs) the clean version back into the .session (save) area. So you end up tending to rollback at each/every reboot in order to achieve repeated clean boots (as opposed to wading around in muddy boots ))

[BarryK]

Guys, lots of posts for me to read back on sometime. For now, getting the work done so far out as a release, version 1.0.11 has arrived:

http://bkhome.org/news/201903/easyos-x8 ... eased.html

This is a very interesting release. Not much has changed after bootup, all the new stuff happens at early bootup.

[rufwoof]

I'm still on 1.0. ec-chroot within that doesn't cater for already having another Xephyr process running (i.e. if using Xephyr for something else). That may have been fixed in a later release however ???

I'm running both the easy container (on :1) and a pure debian (on :2), so a workaround is to just ensure I start the easy container first.

A nice 'feature' is that provided you leave one window open in each of the main, easy and debian displays, then alt-tab flips you between them.

Screenshot is with debian firefox-esr running on :2, easy container with firefox quantum running on :1 and the main session as per the screenshot, running top. Both the separate firefox sessions are playing youtubes ... but the sound is only heard from one (whichever grabs /dev/snd first).

[Sage]

1.0.11: Works well. Some duplication setting locale lang+k/b at boot up and again in setup panel? Regret the logic of direct frugal install to HD doesn't help in respect of .iso on optical media when using for diagnosis, rectification, demonstration, etc.

[BarryK]

Already a couple of bugs found in 1.0.11:

http://bkhome.org/news/201903/bugs-in-easyos-1011.html

...note item-2, upgrading an existing installation, requires a manual fix.

[blgs]

manual fix fix for easyOS 1.0.11 wasn' the correct solution for me. Switched back to the previous version (1.0.,

Waiting for the permanent fix before updating again.

[Billtoo]

New install to a 64GB Sandisk Cruzer flash drive.
Installed firefox and devx sfs files,added Vlc and Gnome MPV with
petget.
Gnome MPV and Vlc work well.

Enjoying new version so far,
Thanks.

EDIT: the flash drive is a Sandisk Ultra,

[BarryK]

Already a couple of bugs found in 1.0.11:

http://bkhome.org/news/201903/bugs-in-easyos-1011.html

...note item-2, upgrading an existing installation, requires a manual fix.

I fixed those two bugs, and have re-uploaded 1.0.11.

The "salt" problem with encryption is fixed, I think. Now using a fixed string for the salt, and for an existing installation, if the fixed string fails, it falls back to using the $WKG_DISKID as the salt, as before.

Notice to everyone, if you want to do an install to hard drive, grab this latest upload. From now on, the salt problem shall be no more.

[BarryK]

New install to a 64GB Sandisk Cruzer flash drive.
Installed firefox and devx sfs files,added Vlc and Gnome MPV with
petget.
Gnome MPV and Vlc work well.

Enjoying new version so far,
Thanks.

EDIT: the flash drive is a Sandisk Ultra,

Good, thanks for confirming that the media-player PETs work.

[BarryK]

manual fix fix for easyOS 1.0.11 wasn' the correct solution for me. Switched back to the previous version (1.0.,

Waiting for the permanent fix before updating again.

Grab the re-uploaded 1.0.11 image file. To upgrade an existing installation, when you click on 'initrd', don't choose the automatic fix. You will need to manually edit 'BOOT_SPECS' to have the correct values for the BOOT_UUID and WKG_UUID.

For future upgrades, automatic fixing of BOOT_SPECS should work, as long as I don't shift the gaol-posts again.

[blgs]

Downloaded easy-1.0.11-amd64.img.gz 2019-Mar-08 16:08:47. (latest update?)

Running easyOS 1.0.8. Did open the new image file (easy-1.0.11) by clicking two times. Copied the 3 files to the boot partition.
Manualy edit the initird file wit the correct UUID's for boot and the working partition.

Rebooted the computer. During bootup the boot and working partition are recognized. The screens regarding the language/keyboard and pasword setiing appears.

Then it stops....

I have attached a jpg file with the error.

Currently I'm switching back to easyos 1.0.8.

[BarryK]

Downloaded easy-1.0.11-amd64.img.gz 2019-Mar-08 16:08:47. (latest update?)

Running easyOS 1.0.8. Did open the new image file (easy-1.0.11) by clicking two times. Copied the 3 files to the boot partition.
Manualy edit the initird file wit the correct UUID's for boot and the working partition.

Rebooted the computer. During bootup the boot and working partition are recognized. The screens regarding the language/keyboard and pasword setiing appears.

Then it stops....

I have attached a jpg file with the error.

Currently I'm switching back to easyos 1.0.8.

The fact that you get the GUI asking for locale/keyboard-layout shows that the BOOT_SPECS file is wrong.

You only get that GUI at first bootup, not when doing an upgrade.

The reason for this is probably that you did not fix WKG_DIR. For example, if you had previously set it as WKG_DIR='easy/', then you will have to manually do so to the new BOOT_SPECS.

I have now fixed the script that handles this, /usr/sbin/edit-initramfs, which will be in Easy 1.0.12, and have updated the docs online:

https://easyos.org/user/easy-version-up ... grade.html

But, you can test the script. Download, remove the fake ".gz", make sure it is executable, and replace the script in /usr/sbin

I hope that you have success this time!

[rufwoof]

I do like the fundamental flipping of the EasyOS design.

Puppy pulls in other systems (Debian ... etc.) programs to use as its own, within its own environment; EasyOS core permits other systems to be run within controlled environment, set up a debian bootstrap folder for instance and use a combination of Xephyr, unshare, capabilities dropping and chroot ... to run the native (kernel/binary compatible) version of that ... Easy OS in effect becomes a hypervisor that can limit (or not) what system resources are made available.

Puppy saves changes to ram, which is finite. EasyOS saves changes to disk and via snapshots/rollbacks caters for whether changes are persistent, or not.

Of the two, EasyOS's design feels to be distinctly better IMO.

I've just added a debian bootstrap folder into /usr/local/debian-stretch for instance, which expanded the easy.sfs by around 50MB. Fire up a terminal and chroot that and I'm in a pure debian (main repo only) terminal session, into which I can use apt to build whatever debian system I desire within that, and EasyOS caters for what resources are made available, or not, or whether those changes are saved, or not. I can even create different versions, saving each to separate snapshots ... such that any one of those snapshots can be restored at any time, without having to reboot.

Multiple desktops are nice, as that way you can better mentally separate what systems might be running at the same time. Leave desktop 1 for EasyOS main, Easy container on desktop 2, Debian on desktop 3 ...etc. Yet despite such multiple systems running concurrently, operationally they're as good as running on bare metal (because fundamentally they are). Works incredibly well, even on my 11 year old (2GB ram) hardware. And software choice galore. Want to run Audacity? Then pick which OS you'd rather run that under, Puppy, Debian, Ubuntu ...etc. and just install/run it within that environment/desktop. Neat! Thanks Barry.

[rufwoof]

1.0.11 for my kit (BIOS, frugal HDD installed) the framebuffer isn't activated until very late. Predominately a text boot, with the kernel activating the framebuffer just before X is loaded.

So the current init in 1.0.11 throws out a error message on first run (/dev/fb0 exists but isn't activated, so when the logo for instance is run ... ERROR>> messages are generated).

The way I had X and network connection (and everything else) working inside init before was to actually chroot the main systems sfs sbin/init, and tagged on a exit (back into initrd's init) at the end of /etc/profile. Along with a flagging setup so that on subsequent loads it didn't just exit after re-running /etc/profile.

EDIT:

Ran checks and if prior to using the framebuffer in initrd/init such as loading the logo, running cp /dev/fb0 /dev/null and checking $? (return code) is one way to ensure the framebuffer is active before writing to it. The copy does however take a second or so to do when it is available (booting with video=640x480 i.e. text mode flashes through that (fails) pretty much instantly), so rather than copying the whole framebuffer, just coping 10 bytes or so is quicker. Line 254 or so in initrd being changed to ...

Code: Select all

#190307 display logo, -f = reduces to fit screen, keeps proportions
[ -e /dev/fb0 ] && head -c 10 /dev/fb0 > /dev/null && idump -f logo1920x1440.png

There are a couple of other /dev/fb0 usage in init, but they're subject to ext4 and I'm booting frugally to a non ext4 partition so I haven't tested those (no password being set, so no need for ask keyboard/password etc.).