slacko-6.0 beta 2

A home for all kinds of Puppy related projects
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#361 Post by gyro »

SFR wrote:Hmm, problem with disappearing whiteout files again?
Yes, I tracked it down to /usr/sbin/sfs_load removing whiteout files. My proof of concept was much simpler, I simply commented out the line in /usr/sbin/sfs_load that calls "cleanwhite".

This is bit of a problem, because if you add an extra sfs, all whiteout files are removed every time, sfs_load loads the sfs, which is every boot.
If you delete a file that requires a whiteout file, then re-boot, the file comes back again. Not good.
When extra sfs's were loaded in init, at least the "cleanup" only mucked things up when the extra list was changed, not every boot.

It also confuses "Boot Manager"->"Startup apps", because you end up with both a ".desktop.bak" file and the old ".desktop" file in /root/.config/autostart.

I'm intend to stick with no whiteout file cleaning, for a while.

I wonder if we would be better off if sfs_load did not do any whiteout file cleaning, and we had a utility that would remove any whiteout files that correspond to files in the specified sfs?

Or, sfs_load becomes even more sophisticated:
1) Only does a whiteout file cleanup when the list of sfs's changes, or ideally only the first time an sfs is loaded.
2) Only removes whiteout files that correspond to files that exist in the sfs being loaded.

Puppy should not make it impossible to "delete" a file that exists in an sfs.

gyro

User avatar
SFR
Posts: 1800
Joined: Wed 26 Oct 2011, 21:52

#362 Post by SFR »

@Gyro: Yeah, I wish sfs_load did nothing to other layers than pup_rw.
The idea to backup and restore .wh files is perhaps a bit unusual, but plain and simple, no need to manually process all the whiteouts and remove them thereafter.
But like I said elsewhere, I don't know sfs_load enough; the most qualified person in this matter is Shinobar himself. 8)
Billtoo wrote:After just over 1 hour the warning screen popped up.
Did it happen when FIrefox was up and running, like it was the previous time?
Have you been actively browsing at the same time when saving took place?
And finally, I assume it was again just a false alarm?

Hmm, could you please try the attached, modified version of snapmergepuppy (copy it to /usr/sbin/ and save the session)?
The idea behind the modification is that if copying a file failed, it tries once more and only then reports an error.

Not sure if this will work, because I have suspicions that only re-evaluating of all the layers (what takes place after copying) fixes anomalies responsible for those false errors and makes that the next save is error-free.
On the other hand it might be just a matter of bad timing (an app is doing something with a file that is being processed by snapmerge at the same moment), so it's worth a try, I believe.

You're quite "lucky", btw. :wink: It happens to me no more than 1-2 times a month...and I'm saving a lot!

Does anyone else have this problem too?

Greetings!
Attachments
snapmergepuppy_billtoo.tar.gz
Unpack and copy to /usr/sbin
(4.24 KiB) Downloaded 173 times
[color=red][size=75][O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource[/size][/color]
[b][color=green]Omnia mea mecum porto.[/color][/b]

User avatar
Billtoo
Posts: 3720
Joined: Tue 07 Apr 2009, 13:47
Location: Ontario Canada

#363 Post by Billtoo »

SFR wrote:
Billtoo wrote:After just over 1 hour the warning screen popped up.
Did it happen when FIrefox was up and running, like it was the previous time?
Have you been actively browsing at the same time when saving took place?
And finally, I assume it was again just a false alarm?

Hmm, could you please try the attached, modified version of snapmergepuppy (copy it to /usr/sbin/ and save the session)?
The idea behind the modification is that if copying a file failed, it tries once more and only then reports an error.

Greetings!
I forget whether or not firefox was active, I usually have it going but I think I was playing a game and listening to music.

I didn't check the error file at the time, I shut down for an hour or so and just booted up a few minutes ago, the /tmp/snapmergepuppy-error file is empty now.

I'll try your modified version and report back if the popup continues.

Thanks.

User avatar
shinobar
Posts: 2672
Joined: Thu 28 May 2009, 09:26
Location: Japan
Contact:

whiteout

#364 Post by shinobar »

sfs_load lone 52-57 has some options, it'll be useful for testing:

Code: Select all

#some options the puplet builder can choose
WIPEWHITEONLOAD="true"	# true/false
WIPEWHITEUNLOAD="true"	# recommend 'true'
WIPEMASKONLOAD="false" # recommend 'false'
DISSOSIATE="true"	# excute'losetup -d', some 3.2.x kernel may hung up
WIDESEARCH="false"   #v2.0: search sfs other than /mnt/home and /mnt/home/PSUBDIR
But WIPEWHITEONLOAD/WIPEWHITEUNLOAD works only when the extra sfs is load/unload by sfs_load. I forgot to implement dealing with the whiteout when extra sfs are handled by the bootmanager.

Removing the extra sfs function from the bootmanager is one option we can take.

But an issue remained with zdrv/adrv/ydrv. Both sfs_load and bootmanager cannot handle zdrv/adrv/ydrv. Only the way is to add/remove using a filer by hand with RAM mode... :(
Downloads for Puppy Linux [url]http://shino.pos.to/linux/downloads.html[/url]

User avatar
shinobar
Posts: 2672
Joined: Thu 28 May 2009, 09:26
Location: Japan
Contact:

whiteout patch

#365 Post by shinobar »

Add one line against sfs_load-2.0.11:

Code: Select all

--- sfs_load-11	2014-07-03 04:23:32.826584625 +0900
+++ sfs_load	2014-07-04 05:53:02.526655485 +0900
@@ -2063,6 +2063,7 @@
   #v1.9.2: skip fixmenus, hope done before
   #v2.0: see the LASTUNIONRECORD backup
   if [ "$LASTUNIONRECORD" != "$PREVUNIONRECORD" ]; then
+   [ "$WIPEWHITEONLOAD"  = "true" -o "$WIPEWHITEUNLOAD"  = "true" ] && cleanwhite && touch $MYTMPDIR/CLEANWHITE
    [ "$NEED_FIXMENUS" ] || SKIP_FIXMENUS="y"
    if [ "SKIP_HAS_FIXMENUS" ]; then
      #not checked, force all on
Downloads for Puppy Linux [url]http://shino.pos.to/linux/downloads.html[/url]

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: whiteout

#366 Post by gyro »

shinobar wrote:sfs_load lone 52-57 has some options, it'll be useful for testing:
Thanks for this information.
I've currently set:

Code: Select all

WIPEWHITEONLOAD="false"	# true/false
WIPEWHITEUNLOAD="false"	# recommend 'true'
and since no whiteout file processing is being done by sfs_load, all is well with /root/.config/autostart.
I also have not encountered any problems caused by unwanted whiteout files.
gyro

User avatar
SFR
Posts: 1800
Joined: Wed 26 Oct 2011, 21:52

#367 Post by SFR »

shinobar wrote:

Code: Select all

WIPEWHITEONLOAD="true"   # true/false 
WIPEWHITEUNLOAD="true"   # recommend 'true' 
Thanks! Setting the above to 'false' seems to fix also my old problem in PUPMODE=13 & SAVEINTERVAL=0, without the need of 'tar' workaround.
So, why it is recommended to have them set to 'true'? What are possible drawbacks of 'false' state?

Greetings!
[color=red][size=75][O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource[/size][/color]
[b][color=green]Omnia mea mecum porto.[/color][/b]

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: whiteout patch

#368 Post by gyro »

shinobar wrote:Add one line against sfs_load-2.0.11:
I edited sfs_load with your patch, set the whiteout options back to "true".
Loaded the devx sfs using sfs_load.
Rebooted and the whiteout files in /root/.config/autostart had been removed. (expected)
I fixed the files in /root/.config/autostart.
Rebooted and the whiteout files in /root/.config/autostart had been removed. (not expected)
I unloaded the devx sfs using sfs_load.
I fixed the files in /root/.config/autostart.
Rebooted and the whiteout files in /root/.config/autostart had been removed. (expected)

So it seems that the patch makes no difference to the problem, whereas setting the whiteout options to "false" does.

gyro

User avatar
shinobar
Posts: 2672
Joined: Thu 28 May 2009, 09:26
Location: Japan
Contact:

wh

#369 Post by shinobar »

How. About ONLOAD false/UNLOAD true?

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: wh

#370 Post by gyro »

shinobar wrote:How. About ONLOAD false/UNLOAD true?
That's probably a reasonable compromise with the current sfs_load.
Edit: Did that, works as expected, but it's a compromise, I now have whiteout cleanup, but I still get annoyed on the next boot after interactively unloading an sfs, because it removes ALL whiteout files.

So, I'm suggesting 3 changes to sfs_load:
1) On load, only remove whiteout files that correspond to files that exist in the sfs being loaded.
2) Don't ever run cleanwhite if called during startup or shutdown.
3) On unload, only remove whiteout files corresponding to files in the sfs being removed.

I want any whiteout files that correspond to files that exist in an sfs to be removed the first time I load it, so that all it's files get exposed by aufs, none get hidden by some ancient whiteout file. But I then want to be able to "delete" files in this sfs and not have them restored at the next boot.

Then change BootManager to use sfs_load to handle all extra sfs processing.

Note: I will attempt to create some bash code that removes whiteout files that correspond to files in a specified sfs. And report back.

gyro

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#371 Post by 01micko »

I've made a few changes to PUI.

I think I will go with shinobar's dotpup and also include the frugal-installer - both patched for new woof icons. Only a few lines. PUI is covered by shinobar's dotpup.

PUI itself:

Changes (a brief list, see the diff attached)
  • deleted the ispupfunc() - didn't work
  • added the hunks of shinobar's patch for ext4 extlinux bug
  • added a new function linux_func() which attempts to identify puppy installs, linux installs and windows.
  • added a test for other linux versions loosely based on shinobar's patch (actually that is used as a fallback) based on the existence of /etc/os-release, tested working with fedora and slackware. Will work with recent ubuntu/debian also.
  • changed many gui to gtkdialog and cleaned up the appearance of some of the existing gtkdialog.
  • fixed numerous bugs mostly dated with a recent date so searching for '#140' (yymmdd) should reveal them.
  • bold big warning before you commit to install (full installs)
  • completed the partial removal of ZIP/LS120 disk support for installation. This has been crippled for over 3 years anyway.
  • all icons calls are to new woof icons
  • disabled installing to windows partitions from PUI. Support for installing to windows partitions by other methods are unaffected by this change
  • hard coded version to 0.0 so rc.update runs in full installs. It did anyway but since the test was broken there were 'integer expected' errors
  • partial code cleanup so in most cases you don't need a nine foot wide monitor.
  • made full installs (all sfs) a loop through the various sfs files which may exist.
  • added 64 bit support, tested working.
I limited my wording changes to probably 10 words (in GUI) so hopefully translations are not too broken. This is why I didn't do a full rewrite. Possibly deletions will have a bad effect on translations. IDK. I'll release the code for testing soon but in the meantime, those interested go over the diff (it should apply cleanly to most recent woof versions of PUI) and pick it apart. There is always room for improvement. :)
Attachments
Screenshot(1).jpg
(86.67 KiB) Downloaded 408 times
puppyinstaller-wce-140704.diff.gz
(17.49 KiB) Downloaded 154 times
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#372 Post by gyro »

I've done a crude edit to /usr/sbin/sfs_load so it only removes whiteout files that correspond to files in the sfs file being loaded, and it dosen't run 'cleanwhite' when called during startup.
And it works, well at least in my very limited test.
In the function 'append_sfs', I replaced

Code: Select all

[ "$WIPEWHITEONLOAD"  = "true" ] && [ ! -f $MYTMPDIR/CLEANWHITE ] && cleanwhite && touch $MYTMPDIR/CLEANWHITE
with

Code: Select all

[ "$WIPEWHITEONLOAD"  = "true" ] && [ "$STARTSCRIPT" = "" ] && [ ! -f $MYTMPDIR/CLEANWHITE ] && cleanwhite $MNTPNT && touch $MYTMPDIR/CLEANWHITE
In the function 'make_cleanup_script', I replaced

Code: Select all

S=$(ls /initrd/pup_ro{?,??}/"$D/$B" /initrd/pup_[az]/"$D/$B" 2>/dev/null| grep -vw "^$PREM"| head -n 1)
[ "$S" ] || rm -f "$BASE$W"
with

Code: Select all

S=$(ls "$PREM/$D/$B"  2>/dev/null)
[ "$S" ] && rm -f "$BASE$W"
I certainly do not suggest that this is an appropriate patch for sfs_load, but just a proof of concept.

Very limited test in pupmode=12:
1) Used interactive sfs_load to load the devx sfs.
2) In ROX-filer I deleted /usr/dietlibc/, a directory that exists only in the devx sfs.
3) After reboot, /usr/dietlibc/ is still deleted, whiteout file /usr/.wh.dietlibc was not removed from pup_rw.
4) Used interactive sfs_load to unload the devx sfs.
5) The file /usr/.wh.dietlibc is now removed from pup_rw.
The whiteout files in /root/.config/autostart remained in tact throughout the whole test.

Note: In pupmode=13 probably need to remove whiteout files from pup_ro1 as well.

gyro

User avatar
SFR
Posts: 1800
Joined: Wed 26 Oct 2011, 21:52

#373 Post by SFR »

01micko wrote:I've made a few changes to PUI.
Looks pretty nice. :)

But I can't get the same results as on your screenshot - I have Win7_x64 + Slacko-5.7.0 (Noryb's installer) on sda2.
I'm getting "Unknown Linux" instead.
The log:

Code: Select all

# ./puppyinstaller 
/mnt/sda1
no pups found
no distro
no distro
OLINUX=
no distro
no distro
/mnt/sda1
is windows? 
/initrd/mnt/dev_save
found 1 pups
name is /initrd/mnt/dev_save/Puppy_Slacko_5.7.0/puppy_slacko_5.7.0.sfs
continuing search
no distro
no distro
no distro
OLINUX=
no distro
distro is Unknown Linux
/mnt/sda3
no pups found
no distro
no distro
OLINUX=
no distro
no distro
/mnt/sda3
is windows? 
./puppyinstaller: line 742:  9656 Killed                  gtkdialog-splash -bg yellow -close box -text "$SPLSHMSG"
# 
So, I set DISTRO variable (linux_func) to null right after checking for vmlinuz and indeed I got the expected "sda2: ntfs, size 111.5 GiB, Windows 64 bit and Puppy Slacko 5.7.0 installed".

Btw, I tried to apply the patch against the puppyinstaller from woof, but there were 9 rejects, so I did it against the one from Slacko-5.7 and ended up with only 2 (applied them manually).

Greetings!
[color=red][size=75][O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource[/size][/color]
[b][color=green]Omnia mea mecum porto.[/color][/b]

Jades
Posts: 466
Joined: Sat 07 Aug 2010, 22:07
Location: Somewhere in Blighty.
Contact:

#374 Post by Jades »

01micko wrote:[*]disabled installing to windows partitions from PUI. Support for installing to windows partitions by other methods are unaffected by this change.
To clarify, does this mean that PUI is no longer able to automatically create a new Frugal on any NTFS partition or just one where it finds an actual Windows install? Would I be correct in assuming that "other methods" includes creating a directory and manually copying the various bits and bobs from the CD to it?
Zhaan - AMD K6 2 500, 512MB RAM, ATI Rage 128 VR. Full install Wary 5.5 [url=http://tinyurl.com/dy66kh8]HardInfo Report[/url]
Merlin - Core i5-4590, 8GB RAM, Radeon R9 270X. Slacko 5.7.0

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#375 Post by 01micko »

Jades wrote:
01micko wrote:[*]disabled installing to windows partitions from PUI. Support for installing to windows partitions by other methods are unaffected by this change.
To clarify, does this mean that PUI is no longer able to automatically create a new Frugal on any NTFS partition or just one where it finds an actual Windows install? Would I be correct in assuming that "other methods" includes creating a directory and manually copying the various bits and bobs from the CD to it?
Any from PUI.

You can do it manually, or with NoryB's installer but 'officially' from PUI I say it is unsupported. Still doesn't stop save from CD. That's fine.

I am dead against installing to NTFS on a hard drive. I know nooby would not have been too happy, but there is a reason. You can completely screw up a hibernated windows install. Windows 8+ hibernates by default at "shutdown" (which really is not shut down) and thus windows can be rendered unbootable and warranty voided. It must be discouraged to install to NTFS in my opinion.

That said, it is still doable. I should add a bold warning somewhere about windows 8.

I may reconsider allowing installation (from PUI) to an NTFS partition that does not contain windows OS.. but is there any point in that? Most noobs are going to have 1 NTFS partiton (plus all the hidden garbage for EFI if on a new computer).

----
SFR wrote:But I can't get the same results as on your screenshot - I have Win7_x64 + Slacko-5.7.0 (Noryb's installer) on sda2.
I'm getting "Unknown Linux" instead.
My bad for using a 'dummy' install. Anyway, I added a search for a frugal in that block in case this happens.

Code: Select all

if [ ! "$DISTRO" ];then
	  KERNELS=$(find -L "$ISPUPMNTPT" -maxdepth 2 -type f -iname 'vmlinuz*' 2>/dev/null) #shinobar, but reduced depth to 2
	  [ "$KERNELS" ] && DISTRO="Unknown Linux"
	  # could be a frugal pup though
	  FRUGPUP=`find "$ISPUPMNTPT" -type f -name '*up*[0-9]*.sfs' -maxdepth 2 2>/dev/null`
	  [ "$FRUGPUP" ] && DISTRO=''
	  #echo 5 ##debug
	  [ "$DISTRO" ] && echo "distro is $DISTRO" || echo "no distro"
	fi
I'll attach the whole script, also with mods to the windows search, tested with XP, Fista, 7 and server 2008R2 installed (bare metal). I'm not too worried if the versions are reported wrong (they are only a generic guess anyway) but there existence must be confirmed.

EDIT: updated windows test in script
Attachments
puppyinstaller.gz
(25.69 KiB) Downloaded 182 times
Last edited by 01micko on Fri 04 Jul 2014, 13:58, edited 1 time in total.
Puppy Linux Blog - contact me for access

User avatar
SFR
Posts: 1800
Joined: Wed 26 Oct 2011, 21:52

#376 Post by SFR »

Better...actually, more than better - have a look at the screenshot. :lol:

Ok, sda1 is that small boot partition, so I guess there's no point in installing Puppy there anyway:

Code: Select all

# find /mnt/sda1 -type f -maxdepth 2 -name BCD
/mnt/sda1/Boot/BCD
# 
but sda3 contains only the 'recovery' crap:

Code: Select all

# find /mnt/sda3 -type d -maxdepth 3 -name EFI
/mnt/sda3/HDDRecovery/ODDFiles/EFI
# 
Perhaps some test for 'recovery' string ( echo $RESULT | grep -i 'recovery' && ignore ), however other companies (than Toshiba) might have named it differently.
Or additional check for exstance of 'Windows' directory ( find "$ISPUPMNTPT" -type d -maxdepth 1 -iname 'windows' || ignore ), which presence in root dir is, I think, common for all Win versions..?

Greetings!
Attachments
Screenshot.jpg
(51.4 KiB) Downloaded 610 times
[color=red][size=75][O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource[/size][/color]
[b][color=green]Omnia mea mecum porto.[/color][/b]

User avatar
zigbert
Posts: 6621
Joined: Wed 29 Mar 2006, 18:13
Location: Valåmoen, Norway
Contact:

#377 Post by zigbert »

I can Woofify the main gui (at least). No time today, but weekend is free.

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#378 Post by 01micko »

:lol: @SFR (and edited attachment previous page with test for /windows, case insensitive)

Yeah will need some help. I have never bought a new PC with windows installed! So I don't have too much idea about all the recovery crap. All I know is that with UEFI there are small vfat partitions and other stuffs.

I suppose /Windows always exists? I'll modify so that is the first test, and if it doesn't then no more tests are needed, actually the vars doen't even need to be set (which wastes time).

What do you think about my stance on installing to NTFS?
Last edited by 01micko on Fri 04 Jul 2014, 14:00, edited 1 time in total.
Puppy Linux Blog - contact me for access

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#379 Post by 01micko »

zigbert wrote:I can Woofify the main gui (at least). No time today, but weekend is free.
I don't think it's worth it for now. PUI is so complex and spaghetti like that I think I've mangled it enough! :lol:
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#380 Post by gyro »

01micko wrote:What do you think about my stance on installing to NTFS?
I agree, particularly as Windows now has the capability to shrink the partition of it's active C: drive.
gyro

Post Reply