A SAVE-session to directory option added for PUPs [REOPENED]

A home for all kinds of Puppy related projects
Message
Author
User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#21 Post by rufwoof »

jamesbond wrote:@rufwoof - because it happens too frequent that people setup a small savefile and before you know it, it gets full. Save-to-directory "saves" you from this problem but it doesn't consume a whole partition like save-to-partition. And you can still have a full backup easily by copying the entire save directory, if you wish. But save-to-directory only makes sense for others who are already running Linux since it requires a Linux-compatible partition (not only ext2/3/4, you can have it with f2fs, xfs, jfs, reiserfs, btrfs, zfs, nfs, whatever-fs you can think of which is an Unix-compatible partition --- ntfs, vfat and cifs are specifically *excluded*).
Thanks JB.

I started using Puppy back in early March in anticipation of the April XP deadline. Initially I thought I'd just create a small Puppy LiveCD for online banking purposes - a sort of browser linux, and use XP for all other stuff. I haven't however even booted XP for over a month now. I'm finding that Puppy caters for all of my needs.

Initially I did try a savefile and experienced what you said - mozilla cache was filling it up. Played around with moving .mozilla outside of savefile, creating backups etc, but by then I'd more or less remastered a liveCD that had everything I needed and as I liked, so decided to just ram boot and sym link any changes I did make to outside of the savefile space. That's worked well for me and I've stayed with that since.

I don't want to full install as I'm more comfortable booting with the same pristine fresh image all of the time (I realise that could be achieved with frugal or full installs - but I'm ok with using a CD - especially as once booted the CD can be removed - leaving everything just running in RAM (no HDD's, on CD)).

I've a small Slacko based liveCD, 80MB (uses extreme compression so squeezes more into less space i.e. -b 1024K compression dictionary size) that boots quite quickly. The rest I've set up as a few clickable icons that each load multiple sfs's/pet's (word processing, audio/video editing etc). I just drag the EXTRA folder (the name I've given to the folder containing all of the sfs's) to / and then load them from there (in memory).

I store all images, music, docs etc on the HDD's, together with a script that sym links in the various selective persistent saves that I want - primarily program configuration (under /root) files.

Even though this ancient single core runs at 100% CPU when heavily loaded, Puppy manages the timeslicing/loading well and actual interaction is very snappy/quick. So snappy in fact that I've been unwilling to boot sluggish XP.

I've tried loads of other distro's, none work as well as Puppy as a LiveCD IMO. The developers/team are to be congratulated.

A big thanks to all concerned.
Attachments
snap.jpg
(78.01 KiB) Downloaded 626 times

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#22 Post by jamesbond »

greengeek wrote:
jamesbond wrote: Save folder is implemented in Fatdog, since 620beta1 ...... For this to work, the filesystem of the partition must be a compatible Linux filesystem.
What is the reason for this limitation? If puppy can read say, FAT32, easily, what is the reason it can not store system files there? Is it the concern that such files could be modified/damaged by other operating systems using that partition?

Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?
There are deep technical and usability problems around it, but I'll save you all that and be direct: you'll get kernel panics otherwise :wink: If you need to keep the foreign filesystem, stick to savefile.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
saintless
Posts: 3862
Joined: Sat 11 Jun 2011, 13:43
Location: Bulgaria

#23 Post by saintless »

greengeek wrote:Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?
Not possible for full persistent because permissions and symlinking will be broken on NTFS and Fat.
But already included as option in Porteus for saving some type of files only with the strong advice to use ext save file instead NTFS or Fat folder:
http://www.porteus.org/component/conten ... lders.html

Toni

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#24 Post by gyro »

Assuming a frugal install using a "psubdir", why not have a save folder with the same filename as a save file but with a ".dir" extension instead of ".2fs", ".3fs" or ".4fs". And store it in the "psubdir" along with the system ".sfs" file.
In "init", if the pupsavefile has ".dir" extension then do a "mount --bind" insted of a "losetup" and a "mount".
At first shutdown, if "/mnt/dev_save" is a linux partition, create the save folder, and don't ask any questions.

gyro

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#25 Post by greengeek »

saintless wrote:But already included as option in Porteus for saving some type of files only with the strong advice to use ext save file instead NTFS or Fat folder:
http://www.porteus.org/component/conten ... lders.html
Those Magic Folders sound interesting in that they can use separate savefiles for administrator and for guest (user) - I would really love to be able to save two separate "personalities" without having to save the massive overhead of two different 512MB savefiles:

One "personality" could have a NewZealand timezone and language setting, and the other would have US timezone and US language (to cater for the two usergroups in my house). No need for totally separate savefiles or usernames - we are happy to share/combine our data, but just a preference for different "personality" overlays that suit a particular user.

I don't know if that perspective would be better served by a save folder or compressed save file but just thought I'd chuck it in the ring...

EDIT :Sort of like this but maybe with an icon on the desktop that could "pull in" a save folder and overlay it. (maybe two icons on desktop - one a NZ flag and one a US flag)
So - not really like a full user environment - which puppy is not designed for - but just a 'partial overlay' that creates a different environment based on a save folder. (as usual just spitballin' here...)

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#26 Post by mikeb »

@ greengeek
did you not try the folder save (and sfs) and the multiuser option from the pup you had from me? ..a place I put my mouth :D
Then you can try out all this theory in practice.

As for multiple saves... well the choice of save file name boogie on any pup...but changing the read write layer is not really a feasible option. On the other hand some was playing with wiping and reloading the contents of tmpfs from an sfs ...crude but seemed to work.

Using a save folder requires a posix filesystem ... symlinks and permissions are 2 reasons that come to mind.

@giro
your are being too logical... you must leave :D

mike

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#27 Post by rufwoof »

Using a save folder requires a posix filesystem ... symlinks and permissions are 2 reasons that come to mind.
That's subjective to how much you want to have saved/persist-across-reboots. In my case I just save program config data i.e. /root/.notecase, /root/.openshot ...etc. Most if not all of those config files are relatively simple files - neither sym-linked nor permissions dependent.

Even for some system configuration changes sym's/permissions aren't critical, /root/.jwmrc-tray for instance.

When you extend up to wanting to save more changes, libs, /etc/rc etc then permissions/sym's become more important. But at that level shouldn't you really be thinking of doing a full install anyway.

If permissions/syms can't be maintained on the likes of ntfs and instead a container (sfs) has to be used, then how about a dynamic sfs, a pointer record included with the fixed part, that points to the end of (sfs) file and a size record immediately after EOF that identifies how much additional space beyond the 'end of file' that has been 'attached' (reserved) for read/write purposes. 1MB sfs file might have 100K additional filespace (1.1MB total filesize) where that last 100K is read/write. That additional read/write space could easily be dyamic (need to expand from 1.1MB to 1.2MB then just add the extra 100K onto the end and update a pointer/record (new sfs+additional filesize 1.2MB).

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#28 Post by mikeb »

are we talking about 2/3/4fs image files or squashfs archives?..not following that last bit. A sfs save simply grows according to the amount of data in the r/w layer if thats what you mean.

Subject in hand was using a save folder on a posix partition as a save layer.

Ok add a library by installing a pet .... if the r/w layer is not posix it breaks...end of story. Its linux...its built to run from a posix filesystem. The layered filesystem has to reproduce a posix filesystem for it to work...that requires posix layers.
Just like the initrd behaves as a posix filesystem.

Access to fat32 or ntfs is a convenience feature.... I can access ext2/3 partitions from windows with a convenience driver... windows itself still wants to exist on ntfs (not sure if fat32 ok now for vista/7/8)

File naming restrictions vary too I believe.

I wonder whats for dinner... :)

mike

gcmartin

#29 Post by gcmartin »

There is great implementations floating around as this discussion progresses. I think members here get the picture that as we look at this Linux distro, there exist some measures that would allow a simple method of system management in addition to or in leu of a compressed save file or 2fs/3fs/4fs ...

The practice would work no matter if it was done on HDD or USB or DVD or SD or Blu-ray or ...

I have alway saved Linux stuff to linux compatibles. For all other stuff, mainly data and other content, the other filesystem options can and are used. But, again, NOT for linux needs where its symbolic linkages will preserve.

In offering a save folder solution for expansions primarily due to Package Management processes or required local changes in the RAM filesystem, when the options is given to the user, the shutdown subsystem will already know (thru its testings) whether the filesystem options are posix or not and ONLY allow a selection to the user to elect where on the posix to place the save-session information. This selection process, of course, ONLY happens once in the life of any reboot of the PUPs bearing the technology. The save-session would still continue to work the same (at a high level) as it does today; namely "at first shutdown, where so save ..."

It seems that this review is a good one and it seems that plausible solutions are being presented.

I want to weigh-in on the size issue: SIZE does NOT measure performance! We, users, see performance and as RAM pricing, new, use, or given away goes, its reasonable plentiful, today. The save-session processing that we are discussion is a one-time process that occurs on first-shutdown where a user is given options. We are NOT discussing FULL-INSTALLs, as there is no need for such in those systems. If someone/anyone changes a full-install to operate with a save-session, it no longer is a full-install.

For frugal and Live needs where PUP operates out of RAM, this thread has a place and is pertinence.

Hope this is helpful

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#30 Post by mavrothal »

gyro wrote:Assuming a frugal install using a "psubdir", why not have a save folder with the same filename as a save file but with a ".dir" extension instead of ".2fs", ".3fs" or ".4fs". And store it in the "psubdir" along with the system ".sfs" file.
In "init", if the pupsavefile has ".dir" extension then do a "mount --bind" insted of a "losetup" and a "mount".
At first shutdown, if "/mnt/dev_save" is a linux partition, create the save folder, and don't ask any questions.

gyro
I wish was that simple.
pupsave is hardcoded to be a file not a directory.
If you eliminate this, then search through folders and subfolders for puppy files fails. So the logic must be revised.
This can be easily done if you sacrifice other install options but if you want to keep all the preexisting and add savefolders it becomes less "simple" (at least for me. I'm sure that sooner or latter someone will have code to offer)
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

gcmartin

#31 Post by gcmartin »

Hello @Gyro

2fs/3fs/4fs are files that get created, then written into. The shutdown processing subsystem uses Linux utilities to create a file that looks like a posix partition, then writes the system's changes into that posix look-a-like. Once this is done, that file, namely 2fs/3fs/4fs can be place "undisturbed" on ANY known partiton (ext/NTFS/vFAT/FAT32/...etc) which a booting PUP can capably search on boot-up.

What is discussed here is an addition to shutdown-restart processing (persistence) change needs, where the changes (save-session) is a "real" folder for linux use, not one of these special files. The advantage and has been mention earlier, is that the primary system needs are NOT restricted to the size of the 2fs/3fs/4fs sizes....ever!

Hope this helps

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#32 Post by greengeek »

mikeb wrote:@ greengeek
did you not try the folder save (and sfs) and the multiuser option from the pup you had from me? ..a place I put my mouth :D
Then you can try out all this theory in practice.
No, i didnt get that far. I booted it live and had a look around but really it was just a cursory look with the intention of coming back to it (a bad habit of mine unfortunately :oops: ). I shall go back and do an install. Thanks for the reminder

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#33 Post by mikeb »

No, i didnt get that far. I booted it live and had a look around but really it was just a cursory look with the intention of coming back to it (a bad habit of mine unfortunately Embarassed ). I shall go back and do an install. Thanks for the reminder
ok well the save folder and archive is simply added to the shutdown options.
For multiuser rename /etc/slim.conf.bak to slim.conf and that triggers xwin to give a login screen for root or user. Use the create user wizard from the menu. Suggest test one thing at a time :D

mike

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#34 Post by rufwoof »

gcmartin wrote:The advantage and has been mention earlier, is that the primary system needs are NOT restricted to the size of the 2fs/3fs/4fs sizes....ever!
This resize2fs manual page suggests that resizing might be possible whilst still being mounted (if ext3/4)

The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3 and ext4.).

Might then it be viable to monitor/dynamically (live) expand the savefile by 25% once it was >75% filled .... to equal effect (assuming ext3/4).

gcmartin

#35 Post by gcmartin »

rufwoof wrote:... Might then it be viable to monitor/dynamically (live) expand the savefile by 25% once it was >75% filled .... to equal effect (assuming ext3/4).
That intelligence would need to added to the running system somewhere; say, when attempting to add subsystem functionality via PPM.

But, if user had selected the option to use a folder, this intelligence would not be necessary. Every user, Apple/Microsoft/unix/Linux understands "drive level space management" solutions. ... everyone. Not so, when understanding of a "file" which is a filesystem which has run out of space and any problems which surface related to such. This is why the discussion is being pondered. In reality, a folder approach would reduce the number of questions that surface on the forum about issues related to 2fs/3fs/4fs.

Folders offer a clearer and clean solution. And certainly make for a universally understandable item to address. As is mentioned earlier about FATDOG, that approach attempts to address much of this. As does the other case where "under certain conditions" one is given the option of partition selection. But, this thread ponders the option of folder selection at shutdown to easy understandable save-session/persistence management for the RAM approach to Puppy operation. Primary, again, for both Live and Frugal experienced or non-experienced users.

Hope this is helpful.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#36 Post by gyro »

After checking the "init" from Dpup Wheezy, I've changed my suggestion to no longer require a ".dir" extension.

If we are looking on a linux partition, and a save folder exists, use that,
else, look for a save file.

The following is some example code for "init":

replace

Code: Select all

     [ -d /mnt/data${PSUBDIR} ] && FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type f -iname ${DISTRO_FILE_PREFIX}save*.[234]fs | grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`"
with

Code: Select all

     if [ -d /mnt/data${PSUBDIR} ]; then
      case $ONEFS in
       ext2|ext3|ext4|reiserfs|minix|f2fs) FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type d -iname ${DISTRO_FILE_PREFIX}save'*'| grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`" ;;
      esac    
	  [ "$FND_PUPSAVES" = "" ] && FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type f -iname ${DISTRO_FILE_PREFIX}save*.[234]fs | grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`"
     fi
replace

Code: Select all

  else #pupsave not encrypted.
with

Code: Select all

  elif [ -d /mnt/dev_save$PUPSAVEFILE ]; then
    mount -o bind /mnt/dev_save${PUPSAVEFILE} $CREATEPUPSAVE2FS
  else #pupsave not encrypted.
Note1: I have not got around to testing this yet.

Note2: Unfortunately a couple of the lines in the code are quite long, and get wrapped in this forum.

gyro
Last edited by gyro on Mon 05 May 2014, 12:15, edited 1 time in total.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#37 Post by mikeb »

looking good gyro.... my way of thinking is why not work with a save system located where the pup sfs is rather than trying to search next doors garden for something....only exception would be the live cd for those who have an aversion to grub4dos.
If a save folder is found or better still defined then let the save be in there = a dead simple test.

Trying to cover absolutely any combination regardless of how unlikely its needed just produces diffficult to work code...
saying that the puppy 2 code manages most of that in a much simpler way.... ls /mnt/data/*/pup_{$PUPPYVERSION}.sfs also does a targetted subfolder search... ..alternatively /* could be a variable set by psubdir... so rather than all that subfolder searching just target a specified folder or search the drives roots. Like a nit nurse....

mike

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#38 Post by rufwoof »

gcmartin wrote:But, if user had selected the option to use a folder, this intelligence would not be necessary. Every user, Apple/Microsoft/unix/Linux understands "drive level space management" solutions. ... everyone. Not so, when understanding of a "file" which is a filesystem which has run out of space and any problems which surface related to such.
Seems to my noob minded view that it would just be a slow transition to a (almost) full install, excepting unchanging bin's/lib's.... Wouldn't therefore an alternative be to create a partial full install option but with the core (unchanging) bin's/lib's (busybox.. etc) being retained inside Puppy? Personally I can't see much of a benefit of that compared to a full install - after all the cost per MB of HDD storage is (very) inexpensive (in Puppy's small size terms).

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#39 Post by rufwoof »

Would a kernel change/extension approach be viable where :

An external file that contained a master list of all filenames, location of file, permissions, size, state - which could even be textual/standard and as such storable on any partition type (ntfs ...etc).

When puppy needs to access a file it looks at that table of files and follows that route. The state flag/field would indicate whether that file was active or deleted (unavailable). The state flag might also indicate whether writes could actually be made (or not). i.e. when puppy changed a file and the current location was read only (inside puppy) but the user had permission to write to that file, then store the changed/new file outside of puppy and update the master file table to point to that.

Where that might get interesting is that the location could even be a wget to a remote (cloud) location. Personally my internet connection speed is approaching my LAN speed (on a old 100T LAN with 50Mb - 75Mb internet speed), I'm old enough to remember sub 1200 baud speeds and accordingly can see future potential speeds being significantly higher than current levels. A Puppy based on purely pointer driven file locations could be very small in size when the internet connection speed was adequately fast enough, or larger in size (files all local) for no/slow internet speeds.

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#40 Post by mavrothal »

gyro wrote: The following is some example code for "init":

replace

Code: Select all

     [ -d /mnt/data${PSUBDIR} ] && FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type f -iname ${DISTRO_FILE_PREFIX}save*.[234]fs | grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`"
with

Code: Select all

     if [ -d /mnt/data${PSUBDIR} ]; then
      case $ONEFS in
       ext2|ext3|ext4|reiserfs|minix|f2fs) FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type d -iname ${DISTRO_FILE_PREFIX}save'*'| grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`" ;;
      esac    
	  [ "$FND_PUPSAVES" = "" ] && FND_PUPSAVES="`find /mnt/data${PSUBDIR} -maxdepth 1 -xdev -type f -iname ${DISTRO_FILE_PREFIX}save*.[234]fs | grep -v ' ' | sed -e 's%^/mnt/data%%' | tr '\n' ' '`"
     fi
replace

Code: Select all

  else #pupsave not encrypted.
with

Code: Select all

  elif [ -d /mnt/dev_save$PUPSAVEFILE ]; then
    mount --bind /mnt/dev_save${PUPSAVEFILE} $CREATEPUPSAVE2FS
  else #pupsave not encrypted.
Note1: I have not got around to testing this yet.

Note2: Unfortunately a couple of the lines in the code are quite long, and get wrapped in this forum.

gyro
I'm afraid is not looking very good.
As I eluted to earlier the indicated changes a) will fail to find a savefile if it is also present, more important b) will result in a kernel panic.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

Post Reply