Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 16 Jul 2019, 06:31
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
How does Puppy do 'white out' files?
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 3 [33 Posts]   Goto page: Previous 1, 2, 3 Next
Author Message
mikeb


Joined: 23 Nov 2006
Posts: 11264

PostPosted: Fri 12 Jul 2019, 04:43    Post subject:  

hmm i used touch to recreate whiteouts for my save sfs though the contents are then copied back into the ram tmpfs layer at boot. Cannot recall if whiteouts worked in the sfs itself. Perhaps they only work on the topmost layer eg pup_rw. Indeed maybe it needs to be a read/write layer...

Its been a while lol

mike
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3243

PostPosted: Fri 12 Jul 2019, 04:44    Post subject:  

I believe the kernel code only looks for .whiteout files in the upper read/write copyonwrite (cow/save) layer and when found it hides the associated file/folder. adrv is a read only mounted filesystem and as such can't be the upper read/write 'save' area so any .wh files within that (or the main sfs) are considered as being standard files. To get the adrv .wh files to work as intended they'd have to be copied out of adrv to the topmost save/read-write layer manually.
_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3243

PostPosted: Fri 12 Jul 2019, 05:03    Post subject:  

greengeek wrote:
What is the ACTUAL difference between a save file and an adrv? A save file contains a "file system" doesn't it? But an sfs does not i think.

Is it possible to build an sfs (xdrv) that also contains a filesystem like a savefile?

Is the layering system treating them differently because it is capable of recognising filesystems as a complete unit.

What does a savefile look like inside?

A sfs is a squashed filesystem. i.e. compressed. The kernel when designed/configured to do so, can mount/use sfs as per any other filesystem, but being compressed its read only.

Yes a sfs can contain another filesystem, but again being within a compressed filesystem that would be mounted a read-only filesystem.

Visualise it as any other compressed file. With gzip for instance you can compress multiple files into a single file that takes up less space, once compressed however you can't change a file contained within that single compressed file without first decompressing and then applying the change and compressing again (recreating the compressed file). For the typical sfs that full extract/change/recompress task is a relatively slow/intensive process.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
mikeb


Joined: 23 Nov 2006
Posts: 11264

PostPosted: Fri 12 Jul 2019, 05:07    Post subject:  

See rufwoof for the elegant version of my post Laughing

Onions have layers...
ps iirc I used find and touch to create the whiteout copies.

mike
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3243

PostPosted: Fri 12 Jul 2019, 05:50    Post subject:  

mikeb wrote:
See rufwoof for the elegant version of my post Laughing

Posted seconds apart. Would have said great minds think alike but this dumb-ass didn't want to insult you Mike Smile

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3243

PostPosted: Fri 12 Jul 2019, 07:04    Post subject:  

You can create a file filesystem, a single file that contains a complete filesystem within that, that can be mounted and the contents changed. Those changes remain persistent i.e. umount it and remount it again and the prior changes are evident.

A squashed filesystem however cannot have the changes made persistent, they need to be stored/applied by other means as in effect the whole filesystem was compressed as though a single file.

But what about if you used something like upx to individually compress every file inside a file filesystem. Well in that case you also have a form of sfs, but one that can be modified. A issue however is that compressing each/every file isn't as compressible as compressing all the contents (files) using a single pass, so the 'updatable sfs' would tend to be bigger, especially if lots of small files were being contained within that filesystem. upx also just compresses binary files, not text files. And you'd also have to factor in decompression of every file when being accessed/run (upx overhead). Yet another issue is that a file filesystem is set to be a certain size, you might create it initially with say 128MB of space, whereas a sfs file system is more dynamic (yes file filesystem size can be modified, but that takes time/effort).

Another option/variant might be to use a file filesystem initialised with zeros that is then compressed. Creating a file filesystem
Code:
dd if=/dev/zero of=ffs bs=1M count=128

and then formatting that file filesystem as ext2
Code:
mkfs.ext2 ffs

results in a 128MB (134,217,728 bytes) filesize (128 x 1024 x 1024).

gzip that and it reduced down to 135,547 bytes (132KB)

In order to load/use that however you need to first decompress it (back to 128MB space) and then mount/use it
Code:
gzip -d ffs.gz
mkdir -p /mnt/ffs-mountpoint
mount ffs /mnt/ffs-mountpoint

before umounting and recompressing for storage
Code:
umount /mnt/ffs-mountpoint
gzip ffs

With sfs + save (changes) you can end up storing slight differences in files - twice, the version inside the sfs and the modified version in the save area. A variation I've used in the past is to have a completely empty sfs and everything stored in the save area ... and that works well. Tends to avoid the .wh (whiteout) bugs that are still evident to the present date. What I did with Debian was to make the save area a partition type save (instead of a file or folder save area choice), and I installed the standard debian to that partition (so the save area was a complete OS installation). That way I could boot either using the debian standard full install (partition), or the layered Puppy style - where optionally saves or not saving could be used. A nice feature with that is you can boot as though a standard debian, apply updates or installation of programs etc. and then reboot back into puppy style where you boot/use/shutdown without saving. Fundamentally that's a more stable way to run things, as whiteout files and the bugs/issues with that ... are less evident.

Equally however, rather than a full install (all files in 'save' area) to a partition, you could use a file filesystem instead of that partition. But again, that doesn't involve compression, so the file filesystem is large, typically around twice as large as a sfs.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2953
Location: Cradle of Humankind

PostPosted: Fri 12 Jul 2019, 21:17    Post subject:  

rufwoof wrote:
I believe the kernel code only looks for .whiteout files in the upper read/write copyonwrite (cow/save) layer and when found it hides the associated file/folder. adrv is a read only mounted filesystem and as such can't be the upper read/write 'save' area so any .wh files within that (or the main sfs) are considered as being standard files. To get the adrv .wh files to work as intended they'd have to be copied out of adrv to the topmost save/read-write layer manually.


Yes, must be very top layer pup_rw it seems. Interesting though - I've had a look at the snapmergepuppy script. In the section that deals with white out files, there is code to disregard any white out files that may be present in a lower layer than pup_rw which suggest to me that the kernel handling of white out files "evolved" over time and is now restricted to the very top layer only.
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2953
Location: Cradle of Humankind

PostPosted: Sat 13 Jul 2019, 00:30    Post subject:  

greengeek wrote:
The save file must contain whiteout files - is that correct? ie: there is no other way to "delete" a file from the base sfs other than for the savefile to contain a whiteout file blanking that file.

So - the save file must receive a copy of the pup-rw whiteout file.

I could not manually copy the whiteout files - so how did BK manually copy them into the savefile??

Can't be a function of the layering system surely - as the layering system knows nothing of what a savefile (or savefolder) actually is. Does it?

@nic007 - did you succeed in getting the whiteout files included in the adrv? Are they actually there??

Yes, you can copy the specific white out entry but it does not work. Handling of white out files only works with very top layer pup_rw.
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 3986
Location: holland

PostPosted: Sat 13 Jul 2019, 07:33    Post subject:  

It's possible to load an sfs with whiteout files inside, to make it work the same as save-file or folder, the trick is to use "rr+wh" (see at code below)
I tested the following on dpup-stretch with "my.sfs" containing whiteouts and it works as expected :

Code:
# edit below path to directory and path to .sfs containing whiteout files
NEWMOUNT=/mnt/sda2/newmount
MYSFS=/mnt/sda2/my.sfs

mkdir -p $NEWMOUNT
mount -o loop $MYSFS $NEWMOUNT
mount -t aufs -n -o remount,add:1:$NEWMOUNT=rr+wh unionfs /


Probably the init script inside initrd.gz needs editing to make loading of e.g. zdrv work similar way at boot.

Fred

_________________
Dog Linux website
Tinylinux blog by wiak
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5582
Location: Republic of Novo Zelande

PostPosted: Sat 13 Jul 2019, 16:15    Post subject:  

What technique are you guys using to create the whiteout files? My system just won't allow me to create them manually or copy them.
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 3986
Location: holland

PostPosted: Sat 13 Jul 2019, 16:48    Post subject:  

greengeek wrote:
What technique are you guys using to create the whiteout files? My system just won't allow me to create them manually or copy them.


Creating a whiteout file on (or copy to) the virtual filesystem (e.g. /root) doesn't work for me, but on any place outside it (e.g. /mnt/home or some partition, e.g. /mnt/sdb1) it does work.
Also some file-managers refuse to copy these, so then better use "cp" command.

Fred

_________________
Dog Linux website
Tinylinux blog by wiak
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5582
Location: Republic of Novo Zelande

PostPosted: Sat 13 Jul 2019, 17:11    Post subject:  

fredx181 wrote:
Creating a whiteout file on (or copy to) the virtual filesystem (e.g. /root) doesn't work for me, but on any place outside it (e.g. /mnt/home or some partition, e.g. /mnt/sdb1) it does work.
Thanks fred - yes that technique works.

Do you have any idea if your method of forcing the sfs to load whiteout files at top layer could be used in some way to get a pet to load whiteout files? Since a pet is also supposed to act at top layer?
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 3986
Location: holland

PostPosted: Sat 13 Jul 2019, 17:29    Post subject:  

greengeek wrote:
Do you have any idea if your method of forcing the sfs to load whiteout files at top layer could be used in some way to get a pet to load whiteout files? Since a pet is also supposed to act at top layer?


No, don't think so, a pet is a tar archive to extract in the filesystem, very different from an sfs that can be mounted.
EDIT: Well, never tried, but maybe it is possible to mount a .tar archive with some special tools:
https://www.google.com/search?q=linux+mount+tar.gz

_________________
Dog Linux website
Tinylinux blog by wiak
Back to top
View user's profile Send private message 
mikeb


Joined: 23 Nov 2006
Posts: 11264

PostPosted: Yesterday, at 13:30    Post subject:  

Quote:
mount -t aufs -n -o remount,add:1:$NEWMOUNT=rr+wh unionfs /


fred you never cease to amaze me Smile

cheers

mike
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 3986
Location: holland

PostPosted: Yesterday, at 15:35    Post subject:  

mikeb wrote:
Quote:
mount -t aufs -n -o remount,add:1:$NEWMOUNT=rr+wh unionfs /


fred you never cease to amaze me Smile

cheers

mike


Smile

Ok we can say: WOW !! It's possible Laughing , but... is it useful somehow ? Question
(perhaps for someone it can be, but to be honest, I can't imagine myself using sfs this way)

Fred

_________________
Dog Linux website
Tinylinux blog by wiak
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 3 [33 Posts]   Goto page: Previous 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1473s ][ Queries: 13 (0.0287s) ][ GZIP on ]