Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 13 Dec 2019, 03:32
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
How does Puppy do 'white out' files? (Solved)
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 3 [42 Posts]   Goto page: 1, 2, 3 Next
Author Message
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Wed 13 Feb 2019, 05:24    Post subject:  How does Puppy do 'white out' files? (Solved)  

Which script/scripts directs how Puppy deals with white out files? In other words - there must be a script that looks for white out files (probably in the savefile/folder) and then "mask" (hides) the files from view if "deleted" from the base sfs.
Last edited by nic007 on Wed 17 Jul 2019, 04:00; edited 1 time in total
Back to top
View user's profile Send private message 
mfb

Joined: 22 Mar 2016
Posts: 91

PostPosted: Wed 13 Feb 2019, 08:17    Post subject:  

Hi nic007,

Late last year I tested your adrv code which worked superbly well and unbelievably efficiently, except, as you yourself wrote, it did not cater for deletions.

Hopefully, a useful answer to your post immediately above, would enable you to fix this show-stopping defect to your (otherwise) better than sliced-bread improvement.

I made very minor adjustments to your code (resized @426B) where it took 6 seconds to back-up by renaming and make a new <30M adrv (whereas my normal save sfs is 200M with some 90M used).

I also used a save sfs of 1M which never needed amending or updating ( though I don't remember if or why I needed to do that) and I also made some minor tweaks in my /root/Startup directory to fix the odd tray icon hitch.
image.jpg
 Description   
 Filesize   102.32 KB
 Viewed   417 Time(s)

image.jpg

Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Wed 13 Feb 2019, 08:46    Post subject:  

Exactly the reason I'm asking this. Wink
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3675

PostPosted: Wed 13 Feb 2019, 15:35    Post subject:  

system code ... union/aufs mounting basically merges two folders into one, typically via allocating two folders by name, one ro, the other rw that are to be merged along with a third folder that reflects the image of the merging of those two folders. If a file somefile.txt in the ro layer is to be 'deleted' then the system creates a .wh.somefile.txt file in the rw layer to indicate that the top layer doesn't 'show' that somefile.txt file.

In short, handled by the OS, not by a script.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Wed 13 Feb 2019, 22:33    Post subject:  

rufwoof wrote:
system code ... union/aufs mounting basically merges two folders into one, typically via allocating two folders by name, one ro, the other rw that are to be merged along with a third folder that reflects the image of the merging of those two folders. If a file somefile.txt in the ro layer is to be 'deleted' then the system creates a .wh.somefile.txt file in the rw layer to indicate that the top layer doesn't 'show' that somefile.txt file.

In short, handled by the OS, not by a script.


You are describing in short how the layered fs works. However, this is not my problem. When you have a frugal install on HDD any changes to the system are saved to the savefile/folder which loads on top of the base sfs as you mentioned . White out files are very clearly marked in the savefile. However, I do not work with a savefile but save my system changes to an adrv which also loads on top of the base sfs like a savefile.
In this case however, white out files are not dealt with, why? So in short - we have a situation where white out files in the savefile (loaded as /initrd/pup_rw) are dealt with correctly but not when those white out files are indicated in the adrv (loaded as /initrd/pup_a). My guess is that it has something to do with the init script in initrd.gz and I want to isolate and edit that part so that it includes the adrv as well as other sfs's that may have preference to the base sfs at load time.
Back to top
View user's profile Send private message 
ITSMERSH

Joined: 02 May 2018
Posts: 911

PostPosted: Wed 13 Feb 2019, 23:11    Post subject:  

I'm not sure about this, though I doubt it is in the init script. I think this happens after performing the switch-root when /etc/profile, rc.sysinit and such stuff is executed.
_________________
RSH

Beware of the Dog ähem nic007! Wink
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5624
Location: Republic of Novo Zelande

PostPosted: Tue 09 Jul 2019, 05:05    Post subject:  

I can't explain it very well but i had similar questions when I tried to make a .pet that contained the contents of pup-rw

My plan was to save that .pet and load it on top of the base sfs next boot. It didn't seem to be handling whiteouts properly. Sorry i never found an answer - but I will be watching this thread with interest.

I kind of had the feeling that there was an inbuilt "permissions" error that stopped the whiteouts being copied. I didnt finish my researching it tho'
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Wed 10 Jul 2019, 01:20    Post subject:  

greengeek wrote:
I can't explain it very well but i had similar questions when I tried to make a .pet that contained the contents of pup-rw

My plan was to save that .pet and load it on top of the base sfs next boot. It didn't seem to be handling whiteouts properly. Sorry i never found an answer - but I will be watching this thread with interest.

I kind of had the feeling that there was an inbuilt "permissions" error that stopped the whiteouts being copied. I didnt finish my researching it tho'


I think there is a section in the snapmergepuppy and/or shutdown scripts that deals with white out files. We need to do a test - copy the contents of a savefile (which has white-out files) and load it as an adrv. If the white out files are then dealt with correctly, one can take it from there...
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5624
Location: Republic of Novo Zelande

PostPosted: Wed 10 Jul 2019, 01:52    Post subject:  

No idea what value this might be - but /etc/rc.d/rc.shutdown contains the following references to .wh files:

#120522 precise puppy with aufs, have two dirs .wh..wh.orph, .wh..wh.plnk, in /initrd/pup_rw, filter out.

#120522 testing precise puppy with aufs, have two dirs here .wh..wh.orph, .wh..wh.plnk, filter out...
RDIRS="`find /initrd/pup_rw/ -mindepth 1 -maxdepth 1 -mount -type d | grep -v '/\.wh\.' | grep -v -E '/$|/mnt|/tmp|/proc|/sys|/dev|/lost' | tr '\n' ' '`"

Just posted it for reference. Why would such files only be filtered out in Precise? And why not mentioned for other pups?

Also, the file /usr/sbin/snapmergepuppy contains the following references to ".wh":

#w003 bug fix for .wh.__dir_opaque whiteout file.

#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq.

#Handle Whiteouts...
find . -mount \( -regex '.*/\.wh\.[^/]*' -type f \) | sed -e 's/\.\///;s/\.wh\.//' |
while read N
do
BN="`basename "$N"`"
DN="`dirname "$N"`"
[ "$BN" = ".wh.aufs" ] && continue #w003 aufs has file .wh..wh.aufs in /initrd/pup_rw.
[ "$DN" = "." ] && continue
#110212 unionfs and early aufs: '.wh.__dir_opaque' marks ignore all contents in lower layers...
if [ "$BN" = "__dir_opaque" ];then #w003
#'.wh.__dir_opaque' marks ignore all contents in lower layers...
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer. 110212 delete entire dir.
mkdir -p "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug? 111229 rerwin: need -p, may have to create parent dir.
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh.__dir_opaque" 2>/dev/null
rm -f "$SNAP/$DN/.wh.__dir_opaque" #should force aufs layer "reval".
continue
fi
#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq ...
if [ "$BN" = ".wh..opq" ] ; then
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer.
mkdir -p "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug? 111229 rerwin: need -p, may have to create parent dir.
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh..wh..opq" 2>/dev/null
rm -f "$SNAP/$DN/.wh..wh..opq" #should force aufs layer "reval".
continue
fi
#comes in here with the '.wh.' prefix stripped off, leaving actual filename...
rm -rf "$BASE/$N"
#if file exists on a lower layer, have to save the whiteout file...
#110206 Dougal: speedup and refine the search...
for P in $SFSPoints
do
if [ -e "$P/$N" ] ; then
[ ! -d "${BASE}/${DN}" ] && mkdir -p "${BASE}/${DN}"
touch "${BASE}/${DN}/.wh.${BN}"
break
fi
done #110206 End Dougal.
rm -f "$SNAP/$DN/.wh.$BN" #remove whiteout file. should force aufs layer "reval".
done

#Directories... v409 remove '^var'. w003 remove aufs .wh. dirs.
#w003 /dev/.udev also needs to be screened out... 100820 added var/tmp #110222 shinobar: remove all /dev
#find . -mount -type d | tail +2 | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^root/tmp|^\.wh\.|/\.wh\.|^dev/|^var/tmp' |
#110224 BK revert, leave save of /dev in for now, just take out some subdirs... 110503 added dev/snd
find . -mount -type d | tail +2 | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^root/tmp|^\.wh\.|/\.wh\.|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^dev/snd|^var/tmp' |
while read N
do
#v4.01 graceful exit if shutdown X (see /usr/X11R7/bin/restartwm,wmreboot,wmpoweroff)...
[ "$XRUNNING" = "yes" ] && [ -f /tmp/wmexitmode.txt ] && exit
mkdir -p "$BASE/$N"
#i think nathan advised this, to handle non-root user:
chmod "$BASE/$N" --reference="$N"
OWNER="`stat --format=%U "$N"`"
chown $OWNER "$BASE/$N"
GRP="`stat --format=%G "$N"`"
chgrp $GRP "$BASE/$N"
touch "$BASE/$N" --reference="$N"
done
#100222 a quick hack: BOOTCONFIG written to in init, before timezone set, can cause trouble...
touch /etc/rc.d/BOOTCONFIG
FREEBASEK=`df -k | grep -w "$BASE"| head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah #110222 shinobar
rm -f /tmp/snapmergepuppy-nospace #110222 shinobar
#Copy Files... v409 remove '^var'. w003 screen out some /dev files. 100222 shinobar: more exclusions. 100422 added ^root/ftpd. 100429 modify 'trash' exclusion. 100820 added var/tmp #110222 shinobar: remove all /dev
#find . -mount -not \( -regex '.*/\.wh\.[^/]*' -type f \) -not -type d | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^pup_|^zdrv_|^root/tmp|_zdrv_|^dev/|^\.wh\.|^var/run|^root/ftpd|^var/tmp' | grep -v -E -i '\.thumbnails|\.trash|trash/|\.part$' |
#110224 BK: revert, leave save of /dev in for now... 120103 rerwin: add .XLOADED
find . -mount -not \( -regex '.*/\.wh\.[^/]*' -type f \) -not -type d | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^pup_|^zdrv_|^root/tmp|_zdrv_|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^\.wh\.|^var/run|^root/ftpd|^var/tmp|\.XLOADED$' | grep -v -E -i '\.thumbnails|\.trash|trash/|\.part$' |
while read N
do

...


BN="`basename "$N"`" #111229 rerwin: bugfix for jemimah code (110212).
DN="`dirname "$N"`" #111229 "
[ -e "$BASE/$DN/.wh.${BN}" ] && rm "$BASE/$DN/.wh.${BN}" #110212 jemimah bugfix - I/O errors if you don't do this

My head hurts just looking at it.

(these references relevant to Slacko 5.6)
.

Last edited by greengeek on Wed 10 Jul 2019, 02:19; edited 1 time in total
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5624
Location: Republic of Novo Zelande

PostPosted: Wed 10 Jul 2019, 02:17    Post subject:  

I don't know if this is relevant - but I just repeated the tests I did a while ago trying to create a test pet to understand how to handle .wh files.

(using Slacko 5.6)
First (just for the sake of a learning example) i find the file /root/Web-Server/hiawatha.htm and delete it. This creates a file called /initrd/pup_rw/root/Web-Server/.wh.hiawatha.htm

Look back at the hiawatha.htm file in /root and see that it is now gone. Get rid of the .wh file (or rename it as something else) and hiawatha.htm reappears

Then I make a pet in /root by creating a "whiteout_testpet" directory including the directory /initrd/pup_rw/root/Web-Server/ and in that directory I try to copy the file /initrd/pup_rw/root/Web-Server/.wh.hiawatha.htm from the running filesystem - however the system will not let me do it (see pic 1 below)

No matter - I decide to create the wh file manually (in /root/whiteout_testpet/initrd/pup_rw/root/Web-Server/ - however it will not let me create it manually either!
(see pic 2 below)

That's as far as I ever got.
CopyWhFilePermissionDenied.jpg
 Description   
 Filesize   35.87 KB
 Viewed   147 Time(s)

CopyWhFilePermissionDenied.jpg

CreateWhFilePermissionDenied.jpg
 Description   
 Filesize   15.51 KB
 Viewed   148 Time(s)

CreateWhFilePermissionDenied.jpg

Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Thu 11 Jul 2019, 21:45    Post subject:  

There must be an init script that deals with whiteout files at bootup and ONLY when included in a savefile/folder (the very top of the layered system). Copying and including a specific white out entry to an adrv for instance, does not work.
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 14542
Location: Gatineau (Qc), Canada

PostPosted: Thu 11 Jul 2019, 22:41    Post subject:  

Hi guys.

As rufwoof mentioned in his February post above, this is automatically handled by
unionfs / aufs.

Use a script to go counter that OS behavior and you'll mess everything up.
(And need aspirin, too!) Smile

BFN.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3200
Location: Cradle of Humankind

PostPosted: Fri 12 Jul 2019, 00:48    Post subject:  

musher0 wrote:
Hi guys.

As rufwoof mentioned in his February post above, this is automatically handled by
unionfs / aufs.

Use a script to go counter that OS behavior and you'll mess everything up.
(And need aspirin, too!) Smile

BFN.

But this does not explain why it does not work for an adrv as mentioned because an adrv is booted/mounted to a layer on top of (preferred to) the base sfs like a savefile.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5624
Location: Republic of Novo Zelande

PostPosted: Fri 12 Jul 2019, 02:56    Post subject:  

The save file must contain whiteout files - is that correct? ie: there is no other way to "delete" a file from the base sfs other than for the savefile to contain a whiteout file blanking that file.

So - the save file must receive a copy of the pup-rw whiteout file.

I could not manually copy the whiteout files - so how did BK manually copy them into the savefile??

Can't be a function of the layering system surely - as the layering system knows nothing of what a savefile (or savefolder) actually is. Does it?

@nic007 - did you succeed in getting the whiteout files included in the adrv? Are they actually there??
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5624
Location: Republic of Novo Zelande

PostPosted: Fri 12 Jul 2019, 03:03    Post subject:  

What is the ACTUAL difference between a save file and an adrv? A save file contains a "file system" doesn't it? But an sfs does not i think.

Is it possible to build an sfs (xdrv) that also contains a filesystem like a savefile?

Is the layering system treating them differently because it is capable of recognising filesystems as a complete unit.

What does a savefile look like inside?
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 3 [42 Posts]   Goto page: 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0680s ][ Queries: 13 (0.0121s) ][ GZIP on ]