data loss by improper unmount of savefile

[MU]

The problem is old, and seems to be system immanent using frugal installations.
So I am somewhat clueless.

Potential ideas to work around these issues I will write in bold.

http://murga-linux.com/puppy/viewtopic.php?t=26594
http://www.murga-linux.com/puppy/viewtopic.php?t=17987
http://murga-linux.com/puppy/viewtopic.php?t=27108

In the last days we also had several posts about lost data.
Sometimes after an improper shutdown, sometimes "just like this".
I just lost my firefox history after Puppy freezed when I used an erratic driver.
Stefan (Minisys) reported, that the minisys database easily gets corrupted, so we must run minisys in a external additional savefile without unionfs.

In Muppy 008.3, I will force an "e2fsck -y" in initrd.
I hope this avoids an accumulation of errors over weeks.

Maybe it also would be better, to go back to ext3 again for the save-file?
The journal might help to revover data.

If I understood the other threads correct, the problem is caused like this:
In init, Puppy mounts some files, and merges them using unionfs.
After that, the "pivot-root" or "switch-root" is done, setting "/" to a new point in the current filesystem hierarchy.
This switch-root will not allow at shutdown to properly remove the unionfs-layers, so that the mounted files cannot be unmounted properly.
For this reason at every shutdown deleted or invalid inodes are added to the filesystem.

My current savefile, that I use since some weeks, even seems to confuse e2fsck in pass1, you see many numbers floating over the screen, too fast to see what is wrong.
I just remember this from kernel panics.
I think the reason is, that I did not run e2fsck for weeks on my savefile, so that so many errors occured, that they cannot be completely repaired.

Apart from that, it still is usable, and I will continue to use it for 2 weeks, until next Muppy is released.

If there are ideas or suggestions concerning my bold written plans for Muppy, I'd be glad to hear them
Mark

[jcoder24]

I'm all for Option 2.

With respect to Option 1 , I think the current puppy3 does this by default. That is why you see all of those periods "." when it is loading the save file.

I would prefer if we could somehow get rid of the errors all together instead of allowing them to be created then fix them.

[jonyo]

I'm all for Option 2.

With respect to Option 1 , I think the current puppy3 does this by default. That is why you see all of those periods "." when it is loading the save file.

Was wondering what that was about...

[Béèm]

I don't know when it appeared in alpha *, but in alpha 7 at boot the pup_save is checked with e2fsck.

So this option is a good idea.

BTW, didn't have an error indication up to know.

[kirk]

I've been meaning to modify the init and shutdown scripts for ext3, but laziness has been preventing me. I've posted this before, this is the output of running fsck on my ext3 partition:

# e2fsck /dev/sda4
e2fsck 1.38 (30-Jun-2005)
/dev/sda4 has been mounted 321 times without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda4: 89516/1938272 files (3.9% non-contiguous), 2322368/3899392 blocks
#

That's after crashing it countless times, battery going dead, video game locking up, just deleting my pup_save file and hitting the power switch, you name it. And after all that, over a period of about 6-8 months, fsck has nothing to do!

Also, you have to wait for the fsck to run at every boot. Very quick on an empty pup_save, longer the more files stored in your pup_save. It's been my experience that on power failure with ext2, with a fsck on reboot, some recent data is lost. With ext3 all seems to still be there.

[BarryK]

Yes, as Béèm commented, Dingo does a e2fsck on the pup_save at every boot. As kirk commented, that causes a bit of a delay at bootup.

For what it's worth, for Puppy 4.01 I plan to revisit the whole shutdown thing, try and find some way to shutdown cleanly. I'll work with the very latest Unionfs which may help -- they have recently improved handling of adding and removing layers, well more recently than the Unionfs version that we currently have in the 2.6.21.7 kernel anyway. I subscribe to their mail list and have seen some announcements about that.

[MU]

ok, thanks.
Barry, if you really can solve that, I think we will move to Dingo for Muppy 009.

I made some modifications to Muppy 008 (for tests), now it creates ext3 and runs fsck.ext3 at startup.
I still get errors about deleted inodes.
I vaguely remember, that small partitions/savefiles can hold no journal, so I now try larger ones.
I just tested with a 64 MB savefile.

I will go to northern germany to meet Stefan for 2 weeks on tuesday, we then will discuss how to workaround the problems for Muppy 008.3.

In any case, we will use the filesystem check at startup.
Not shure yet, but I think it makes sense, to use ext3 as filesystem, also according to kirk.

Using a external savefile for Minisys is something I am really not glad of, but at moment it seems to be inevitable, as we have critical data in the database it uses. If you just lose a bookmark in seamonkey - no problem.
But if you lose bills of customers or break an inventory system of a producer, this would be a catastrophe.
In such a special case, we could live with an additional setup if it cannot be avoided.

On the other hand we would like to keep Minisys easy to use (no extra-setup), as it also will be used for example as a grafical Samba file-sharing utility, so that you can setup Puppy as a comfortable Samba-Server in a Windows network.

If it can be solved by a newer Kernel and a rewrite of the init, this would be a really great improvement.

Thanks for all replies,
Mark

[kirk]

Mark,

You shouldn't need the fsck if you move to ext3. I think the journal is used for recovery at mount.

You could test this by booting muppy with your ext3 save file and then reboot using another save file. As you know, the ext3 save file you used at first has some inodes that need to be fixed. Mount that save file ( from command line, make sure to mount it as ext3) then unmount it and run a fsck on it. There should be no errors.

[Dingo]

O.K. Everything will be solved then Puppy 4 (codename Dingo), but if I want to continue to use 3 puppy? There will be no solution? for the 3.xx series?

[kirk]

Tried it. Ext3 does recover it's journal at mount time and fixes the bad inodes. Here's what I did:

* Edit init script, replaced all "mount -t ext2" with "mount -t ext3" (ya, I know, ugly)

* Made a blank pup_save file with an ext3 partition.

* Booted Puppy with modified init and ext3 pup_save.

* Shutdown and rebooted with normal init.

* Mounted and unmounted the ext3 pup_save, then ran a fsck.

# losetup /dev/loop6 ./pup_save-ext3.3fs
# mount -t ext3 /dev/loop6 /mnt/msdos
# umount /mnt/msdos
# losetup -d /dev/loop6
losetup: : No such device or address
# losetup /dev/loop6 ./pup_save-ext3.3fs
# e2fsck /dev/loop6
e2fsck 1.40.2 (12-Jul-2007)
/dev/loop6: clean, 407/16000 files, 12769/64000 blocks
#

Dingo,

Don't go off the cliff Mark is being extra paranoid because of the way he intends to use it. I'd probably be too. But for my own use, ext2 is fine, just don't like to wait for the fsck. Also, Barry has been back porting things to Puppy 3.

[urban soul]

My current system runs on Puppy 3.02alpha1 (official release) and fsck only takes 2 seconds at each boot. This is fine for me. One of the keys is the shutdown script and it has been greatly improved since Puppy 3.xx came out.

One of the geeks said ext3 does not make any sense in a squashfile - but I never understood exactly why. Does anyone remember this subject ?

Dont be paranoid - just wait for the union people. Lost inodes from deleted files don't do any harm. And of course such a major improvement will be backported.

@Mark
Maybe you can rewrite the shutdown script. This should prevent data lost. If you can reproduce how you lost data in minisys - can you provide additional information about it ? (Eg. kernel log, output of e2fsck) ? Which processes do you have running when shutting down ?

Urban

[BarryK]

O.K. Everything will be solved then Puppy 4 (codename Dingo), but if I want to continue to use 3 puppy? There will be no solution? for the 3.xx series?

Everything I do for Puppy4 can be backported to Puppy3. That's what I did for the 3.02alpha1. It has got out of sync now, but sometime I'll resync my latest Dingo with what Tronkel has been doing with 3.02.
Regarding the 2.6.21.7 kernel, we can recompile that with the latest unionfs -- fortunately the developers are creating patches for older kernels, back to 2.6.18.8.

[floborg]

Seems to be more than a bookmarks problem (and there are multiple threads about the bookmarks issue):

http://www.murga-linux.com/puppy/viewtopic.php?t=27178

http://www.murga-linux.com/puppy/viewtopic.php?t=25238

[MU]

we did not have loss in the database yet.
Stefan reported, that sometimes symlinks disappear after reboot.
Also Minisys recompiles some modules, what indicates a change in the files. It just recompiles itself, if something unexpected happened, this is a protection mechanism.

With the forced check of the filesystem at startup in my new pup_save.2fs I currently just get entries about wrong dates for deleted inodes, so that would be ok. This has to be observed over a longer period.

I will do more tests in northern germany, and let Stefan use my latest testversion of Muppy.
He then can see, if Minisys still recompiles itself, what would be suspicious, so that we had to go for the extra save.2fs.

I certainly will not spread panic.
In my normal use I did not encounter data-loss.
Except with firefox, but that might have other reasons, as floborg just indicated.
Lets not forget, that the mozilla-team fixed over 900 bugs in Firefox 3 beta, so FF2 seems to be full of them.
Stefan also reported, that Firefox sometimes "forgets" things in Suse-Linux.

Except when I plugged out a cable by accident, or the power got lost. Then data were lost, that were not synced yet. But that is a different issue.


Mark

[MU]

We had other strange symptoms with Opera on another computer.
If we used Opera, Muppy often started slower afterwards.
This was extreme, if pup_save.2fs was saved on a reiserfs partition.
Puppy then needs half an hour to start.

I then moved the file to a vfat, what seemed to have cured it.
That was some weeks ago, I now was told, there is a delay again.
I will have a closer look on wednesday.

Opera and Firefox create many temporary files, so I think the slow start might be explained by the wrong inode entries. But this is just a gues, I must verify that with my new initrd.gz.
Mark

[Barburo]

My current savefile, that I use since some weeks, even seems to confuse e2fsck in pass1, you see many numbers floating over the screen, too fast to see what is wrong.

Hi Mark,
I also got this symptom on my notebook when I extended the size of the Muppy save file, and made a mistake.
I had restarted and the wrong puppy (Lighthouse 3.01) was beginning to boot.
I did a hard shutdown and rebooted into Muppy but the file never increased in size. I got the "many numbers" symptom for both save files (both 3.01 derivatives) although they both continued to boot OK. Must have corrupted both somehow.
The good news: the problem corrected itself after I re-requested an extension of the save fie (for both). And no data loss that I could see.

[urban soul]

Also Minisys recompiles some modules, what indicates a change in the files. It just recompiles itself, if something unexpected happened, this is a protection mechanism.

I like the idea of recompiling automatically for absolute security - but my first thought was to refine the algorithm that dedects if files were changed on disk. Let's say only the filesize is checked... how about that ?

For very important data (this is all user generated data) I do not rely on unionfs. E.g. /root/.mozilla is a symlink to a place outside the savefile on my system. Other experienced Puppy users do it the same way. This has a couple of advantages: You can reuse your data in various places, the savefile does not need any repairing and firefox (1.5 + 2.0.0.11) never forgets something.

Urban

ps.
@Mark:
see also: Microsoft requires Komoku
http://www.heise.de/newsticker/Microsof ... from/rss09
http://blogs.technet.com/forefront/arch ... omoku.aspx
--> just four fun.

[floborg]

For very important data (this is all user generated data) I do not rely on unionfs. E.g. /root/.mozilla is a symlink to a place outside the savefile on my system. Other experienced Puppy users do it the same way. This has a couple of advantages: You can reuse your data in various places, the savefile does not need any repairing and firefox (1.5 + 2.0.0.11) never forgets something.

I love it! Just put it into practice on a fat32 partition. Keeping my fingers crossed. Now, I'll be able to determine who is at fault for the missing cookie problem, Puppy or Seamonkey.

[MU]

We had other strange symptoms with Opera on another computer.
If we used Opera, Muppy often started slower afterwards.
This was extreme, if pup_save.2fs was saved on a reiserfs partition.
Puppy then needs half an hour to start.

I then moved the file to a vfat, what seemed to have cured it.
That was some weeks ago, I now was told, there is a delay again.
I will have a closer look on wednesday.

Opera and Firefox create many temporary files, so I think the slow start might be explained by the wrong inode entries. But this is just a gues, I must verify that with my new initrd.gz.
Mark

Interesting, had a look now.
The windowmanager (icewm) did not start.
I then exited X, and ran fixmenus, as I thought that it might block the start.
That was right.
fixmenus was creating a Kernel-panic, with messages somewhat like:
"unionfs error, recursive branches cannot be resolved".
I then ran e2fsck on this savefile from another Puppy, and it started again.

Some minutes ago I received a EMail about a corrupted save-file of Puppy 2.17.
Maybe the fscheck was removed at that version already?

This test made clear, that it is a good decision, to run fsck at startup.

Mark

[Béèm]

This test made clear, that it is a good decision, to run fsck at startup.Mark

As is done in alpha 7 (and before already)