GtkDialog - Make Image / Save / SFS files utility.

[mikeb]

Parallel developement eh... c'est la vie

Ok forgot to mention my puppyfied slax rolled in at around 105MB iso which includes their considerably larger kernel.

Thing is if there is a lack of in house built software then its natural for people to look elsewhere for sources. As I mentioned before custom building for puppy can be a way of keeping light and fluffy, compatible and avoid having to include the Out house sink....to me much preferred...something I do much more often than I used to. Perfect example is you are asking me how many packages there are available with slackware.

In answer to that you can browse with gftp thier repository plus for multimedia and similar there is slacky.eu

My experience is I use debs or pets/sfs (??? yes there are well built packages around puppy and often the neatest solutions) and the rest of the time I build. I have a saying ...'It's all linux'
I do it semi manually. slaptget never worked for me... don't think it works too well for others. The structure and core might be solid but they have a 'slack' approach to packages... often broken , so I avoid when I can... as you say 'go figure'
My tests of slax 7 showed that the situation had not changed..some major breakages. So resorting to the big boys is not always the perfect solution it may seem. Do note Tomas spends most of the build time bug fixing the slackware he uses for slax.

Rough estimate... kernel is puppy, boot wrappers and system wrappers ..puppy. Configs..puppy. Core binaries...mixed bag..often older varieties odd in house built. Rest of user space -the distro its 'based' on with exceptions such as gparted.. Puppy 2 was not claimed to be any particular distro but its core binaries appeared to be from slackware...puppy 4... hard to say..perhaps the same.

There was a 'what is puppy' thread started...curious question of how its defined officially and unofficially. My initial finger pointed to the wrappers that give it its structure. I suppose that would come from the slax experience as that pretty much sums it up ... a few kernel adjustments to suit the live linux/union but the rest is slackware.

This distro was officially a free for all, it now tries to be structured.... there is some identity crises going on... this forum is not strictly speaking a puppy forum but stuff happens. The botany bay of the Linux world. There is an old saying...'don't ask why, it just is'..ah ha... perhaps that is the definition of puppy.

A plan?...God likes is when we plan...it makes him laugh...
I am full of these this morning

adieu

mike

[amigo]

1. kernel is patched for aufs and possibly others. Kernel configuration is puppy specific -mostly done by heuristics where users say "I am missing such and such module" and so it gets added. Several kernel config options are key to the way(s) that puppy is able to boot.

2. As with any LiveCD distro, much of the magic that makes that possible is done within the intird -which is just a small system which runs before the real root is mounted.

3. Puppy uses its' own init system which is simply a very messy shell script. I say messy because most of the mess should really be in the initrd. Once /sbin/init is run, the boot process(init script) for a LiveCD should be just like a normal hard-drive or other boot process.

4. core libraries and programs mostly come from whatever poor distro is being hacked, with the exception that utilities from busybox are used to replace many utilities which are normally in the core-utils and linux-utils packages.

5. Puppy requires a few program which are not found in other distros, are not normally installed by other distros, or puupy usage requires the program to be configured differently at compile-time.

6. Many programs are pre-configured to run under puppy by placing files under (eeeWW) /root.

7. Most of the activity during late-boot is done by puppy-specific scripts and small binaries -things like auto-hardware detection, etc.

The problem with using native upstream repositories arises where normal packages would intefere with the way puppy does things. for instance, installing sysvinit (the *real* init program) would overwrite Puppy's super(sic) /sbin/init. Installing or upgrading core-utils would overwrite the busybox utilities. Installing upstream kernels would also not work as theyare not patched and configured to boot the puppy way.

So, a lot of puppy-specific things need to be protected from being overwritten by using package blacklisting with apt-get or whatever upstream is.

As you may infer, puppyisms pervade every step of the boot/init/desktop sequence. But, as far as disk-space and numbers, upstream packages are a much larger part of the total system.

The problem with all these approaches to using SFS'S or pets or whatever, is that you don't have control of either upstream or even Puppy development. That means that you can't intervene in whatever is already happening -you can only do something afterward.

Most puppy users are impressed with how 'flexible' puppy is to boot. However, the boot/init process which puppy uses actually reduces flexibility and makes it more complex to achieve any gaol other than the goals foreseen by BK. A good example is the boot-to-root versus multi-user support. On nearly any normal distro, booting straight to desktop with autologin as root takes one extra small program and two changed lines in two files. On puppy, in order to restore multi-user support you'd need to modify many dozens of puppy-specific scripts(nearly all of them), add a few extra small programs and replace a few busybox tools with their real counterparts.

Puppy uses some really weird methods for stuff, like, since busybox mount doesn't support loop-mounting, the FULL mount is also included in Puppy, a the 'mount' program is replaced with a script which calls either the busybox mount or the real FULL one, if the loop option is being used. I mean, why not just use the real thing and eliminate the extra wrapper completely?

In fact, the use of busybox utilities throughout poses a great handicap for puppy when trying to use many normal applications which often depend on full-featured utilities and fail when used with cut-down busybox versions. So, this is another thing to think about.

As I've said elsewhere, it really is no problem to treat a filesystem image (SFS, e2fs, etc) like an archive and extract it somewhere instead of mounting it. and, it's no problem to treat an archive like a n FS image and mount it somewhere, instead of extracting it. So, all the fuss about SFS vs. pet/deb is really irrelevant.

The idea of making software available dynamically simply means that you need to make the software visible and available for use, but without really 'installing' anything. A private, one-time union mount is the only way to achieve this in every case. Using wrappers which pass PATH's to the application will only work for some applications. The other way is to use temporary links which are placed in the normal PATHs, but which point to your preferred items which are in an isolated location. This last method is the messyiest of all, most prone to creating problems with the main system, hardest to understand -and I say, if you are going to create objects (links) in the real PATHs, why not just install the program?

I do not recommend trying to mess with puppy's underlying aufs layers at all. It's already doing more than it should be trying... A system using unionfs-fuse and fakechroot would be completely transparent to the user, could be run sand-boxed, any number of 'modules' could be used at any time, the method would always work.

The approach of using an SFS embedded inside an AppDir is quite workable -but only if rox-filer is being used. Still, if the AppRun script for an AppDir is linked into a normal PATH, then the AppDir will also work when clicked on from another file manager. Since the /opt dir is made for doing things in an unorthodox dir structure, it makes the perfect place to put such AppDirs and have them link the AppRun into /opt/bin -which is also legal there. AppImage-style applictaions (like from portable-apps.org) work like this, but without the need to be linked-to at all.

You'd need to have both unionfs-fuse and fakechroot installed for the system to work -or include them in every 'deliverable'.

[mikeb]

Amigo is my PA..he fills in the blanks and saves my memory and typing from too much strain

Actually busybox mount appears to handle loop mounts ..in some ways better than mount..it automatically releases the loop afterwards and seem to be the one used in puppy for sfs insertion. The mount script makes a discrimination for ntfs and uses 3g to give read write mounting. It also has lots of junk for the desktop icons and is much much simpler once they are removed. indeed on slax some form of wrapper is needed for ntfs though the system mounts all at boot as it happens perhaps to circumvent this.

But generally in slax for example the main system init is basically slackware...only the initrd is doing the live linux related stuff...it does give a nice separation and the initrd side is elegantly simple.
The result from a users point of view is you are dealing with standard slackware et al....and yes things like multiuser come as standard issue.

Ok your chrooted method...could you make a working example to help get a handle on this .... something simple like RSH did with sfs handling just for a quick test?

mike

[amigo]

sunburnt got me working on this for him once -with a quite challenging one: google-chrome.

I put together a script which creates an AppDir of it. It does not use an sfs but could be made to do so to reduce the size of it.

The script is called prepare-chrome-AppDir. Download the attached archive which includes the script, icon and AppInfo.xml. The script will dowload a deb package of chrome and fix it up as an AppDir. The AppRun script will then use unionfs-fuse and fakechroot to create and run the chrooted union. You will need to have unionfs-fuse, fakechroot and pkill for it to work. The AppRun could of course be modified to mount an sfs instead of having the chrome files in an open dir inside the AppDir.

chrome does still throw a couple of errors about dbus connections, but still runs. For most applications such an approach would suffice to run them without problems. The dbus errors can be fixed, but I had other things to do...

[mikeb]

Is this something I could test in Lucid.... that may be a dumb question but had to ask.

I still have slax 7 around but then there's no roxapp support in there but perhaps I could run the script directly.

Never used chromium...is it self contained like firefox? ie could this be used for say pidgin?

mike

[musher0]

Hi, sunburnt and all.

http://www.murga-linux.com/puppy/viewto ... ost#709032

This is the "Simple Pupsave Create" utility in GUI that I was talking about. It was initiated by
dejan and then improved upon by ASRI and RSH. So that's one base covered, I think.

Also, let's remember that 2fs files and sfs files are two different animals.

BFN.

musher0

[musher0]

Hi, guys.

Glad I caught you guys before another message was posted: is it just me or is the
utility I mentioned above useless?...

I created two pupsaves for raring 3992 with it and got the message:

raringsave-other.3fs:
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
#
e2fsck: Bad magic number in super-block while trying to open raringsave-other.3fs

for each of them. Then, boot process halts for 60 seconds, it continues to desktop,
but you can't save anything to those pupsaves. Like Teflon, nothing sticks.

Let's test some more (someone? Please?) and be sure before we break the news
to that team? Skilled people too. Strange...

Gotta go. BFN.

musher0.

~~~~~~~~~~~~~
PS @ sunburnt: You're right, there's lots still to be fixed in Puppy. The above incident
brought up the info that the blkid utility used in raring 3992 (recent stuff, this raringPup)
dates back to 2003... Yikes!

[sunburnt]

musher; As I`ve said, I use to write a diddy in V.B. and show it to my neighbor and it`d crash his PC.
Coding is hazardous at best. That`s why I opt for simple languages, Bash, BaCon, etc.

# I don`t think there`s a problem with my image file maker, but I haven`t tried every possibility.

Mike; You`ve given us the best description of Puppy`s guts yet. As I thought, Puppy`s a bag of bones.
Slackware sounds a little incomplete as far as the PM goes anyway. Pkg. management`s a tough job.
Deb and Ubu, and other offshoots seem to have the best handle on it. Debian is the Linux base...

# I can make a small AppDir type package for you to look at, a calculator is small enough to post here.
OR... Download one of the Virtual AppPkgs I posted, but they`re for Precise.


amigo; You`ve always confirmed that Puppy is a square wheel ( I still LOL at that one ).
I`m looking at Deb-Live project and trying to get Saintless`s Wheezy to work with my limited knowledge.
Saintless has the right idea, use Debian and reverse engineer a Puppy clone ( or better...).
He`s simply starting with a Deb-Live Wheezy and removing pkgs. from it, so it`s a good base to start from.
# I`ve got high hopes for one or the other of these approaches to work. It`s the big fix.
.

[mikeb]

I liked featherweightlinux... he made a nice custom job of featherweight which was a stipped down knoppix...so one removed and the other added.

Debian ... I do not recall ever getting a bad build from them...just not keen on the way it bloated out at sid as linux did generally which is why I seem to build now when I want something shiny and new.
udeb and others have always been popular for their 'solidness'.

Its not just about user space binaries.
Puppies wrappers got too heath robinson for my liking... amazing how similar functionality can be obtained with a fraction of the code. The puppy simple and fast feel needs a revisit. This is when I found Lucid to be slower than XP I questioned the status quo...by the way I got that one to boot twice as fast in the end.
I don't do the recent firefox boogie for the same reasons...feels like its lost the plot and better renderers are not enough to tempt me. Unfortunately if devs work on octal core machines with 8GB of ram (more a firefox dig but might apply to here...I notice a 'who cares how big any heavy this stuff is attitude creeping in over the years' ) they might tend to lose a little 'real world' objectivity. I have known software developers to deliberately use old machines and systems on the basis if it works on that it will be happy on anything. Good web designers work on the same basis ...see what happens when they do not.

As for testing...I want something that normally is scattered around the file tree but in this 'ere chroot enviroment really...then I can play with adding squashiness (if needed)

mike

[mikeb]

Sorry for a double post but if something is built from something else then to me its that something else.

My nimblex/slax thingy to me is still a derivative of slax/nimblex regardless of how many puppyisms get added.

My oddball puppies are still puippies...that's how they started life. (they are more puppy 2 than anything but the recent declaration of what is puppy means they are not...I will just ignore that one)

If you see another distro as a better choice then I suggest embracing that.

mike

[sunburnt]

Yes Debian is bloated, but many offshoots slim it down nicely. I tried Feather Linux and I liked it a lot.
Software is layers, so we must always be cautious about each of them, and in particular the lower ones.
# I`m making up a Calculator AppPkg using a union for you to look at...

### I realized last night that I didn`t include a Swap file in my app. It`s an image file also.
.

[musher0]

musher; As I`ve said, I use to write a diddy in V.B. and show it to my neighbor and it`d crash his PC.
Coding is hazardous at best. That`s why I opt for simple languages, Bash, BaCon, etc.

# I don`t think there`s a problem with my image file maker, but I haven`t tried every possibility. (...)
.

Hi, sunburnt.

My bad... I re-read your initial post and realized that you've designed your utility
for the new Devian-live Puppy-compatible distro.

Forget what I said. Sorry for any confusion. I've got to get myself some new
eyeglasses...

BFN.

musher0

[sunburnt]

Not really, it has a "Puppy extension" checkbox now. So for many distros.

And I think I`ll add a swap file to it also, a swap file is an image file too.

[mikeb]

@amigo
ok finally got to test in Lucid pup. Not the latest but does give a broader view of the feasibility.

Firstly added the missing bits... unionfs-fuse and fakechroot.
ran the installer but lacked ar...unpacked the deb manually to app.
fakechroot from ubuntu wants full getopt... and -- needed adding.
chrome bin path was wrong (debian sid packages would not unpack???)
Ok this is all minor stuff plus it does not need to be rox/appdir centric to work so I basically did everything by hand following the script.
But files would need adding and a suitable system for configuring. I would say a gui to do it locally to encompass a choice of sources...hard drive, sfs and so on...its flexible in that respect.

In doing so I did create the union.
In my layman's terms it builds a union using the normal puppy file system as the bottom layer and a folder of choice for the read write...the app itself is sandwiched in between. The app itself could be a mounted sfs/image file or a folder anywhere.

It effectively sandboxes the app and does indeed leave the main file system untouched apart from where the user chooses to put it.

Ok onto running chrome.
Well a that was a non starter since it wants newer libstdc++ than lucid can run. (also wanted mozilla libs and at 150MB installed its quite a behemoth..not sure what the attraction is..firefox all is forgiven.. but I digress) It also seemed a poor test since chrome like firefox can run from anywhere so the only 'benefit' would be the relocation of its profile. I then extracted a package of vlc to the app folder . It did appear in the fake union ...but when I ran vlc it said it could not find its plugins...they did exist in the same fake union ..message was
cannot read (null)/plugins/plugins-04041e-28.dat

So all looked correct yet had a problem finding itself. I also noticed ldd did not work either. Am I missing something in running it perhaps or does this mean each app would need specially configuring to work this way?

Is union-fuse used as we are mounting on an already existing kernel unionfs? Does the fuse version have much impact on performance?...I did not really get a chance to test that aspect.

Advantage I can see... apps can be run independently so conflicting libs do not matter.... A neater, if it can be implemented simply, way than using LD..variables plus sfs can be taken advantage of.

It could be cited it leaves the system clean after use but that equally applies to kernel inserted sfs. If unioned outside the puppy filesystem it would be kept out of remasters. Might be a good choice for such as a dev toolchain too.

Not sure if configs and such being written elsewhere is an advantage but it would group together these items to give the option of a total clean up after usage....eg union to /tmp . Good for the paranoid

There is a warning that dbus connection will be killed...does that mean of the whole system?

Performance.... hmm fuse union layer on unionlayer inserting possible a loop mounted device... . a system in a system

Ok that's it for now.... interesting stuff

mike

[musher0]

Hi, sunburnt.

I was finally able to create a good usable pupsave with your script, when I added the
-m 0 -F -F parameters, like so, to the mkfs command.

Code: Select all

echo y | mkfs.$FS -m 0 -F -F $P/$F

Maybe a single -F is be enough, but I did read on the mkfs man page that doubling it
may be useful, solving problems at times.

I suppose the other types of files don't need the -F parameter? A swap file, probably
not, it being necessarily a raw, unformatted file in which bytes come and go. Debian
live-rw savefiles and such, I don't know if they need their file system formatted. (Can
someone answer that please? It would be helpful, I think. Thanks in advance.)

What I am trying to say, sunburnt, is that perhaps the mkfs command you use in your
script needs to be further differentiated per savefile type? Just a thought.

In any case... the command

Code: Select all

echo y | mkfs.$FS -m 0 -F -F $P/$F

produces a valid pupsave file, into which you can really save stuff, it's not a Teflon
savefile.

However, on first use of this pupsave file, in the Puppy desktop, the icons appeared as
danger road signs, and the ROX pinboard had no background. To get a responsive
ROX pinboard, I had to use the command

Code: Select all

gdk-pixbuf-query-loaders --update-cache

suggested by watchdog here:
http://www.murga-linux.com/puppy/viewto ... ost#722565

I know this last paragraph is outside the subject of this thread, but I thought I'd mention
it anyway, in case somebody got stuck with that bug.

Back on the dot, pupsave files created with the above command can be opened and
minimally populated. For example, in the file that I created with the edited sunburnt
script, I was able to create a /root/Startup structure and copy the gdk command in that
folder. It didn't execute on first boot, as a script in /root/Startup normally does on
subsequent boots, but it was there and usable. So I was able to correct my pix-buffers
problem immediately simply by clicking on it.

Another thing which I noticed is that the pupsave file created with the "-m 0 -F -F"
parameters had a "Lost and Found" folder, which the usual pupsave files do not have.

I hope my little test report was clear enough. Any questions, please ask.

BFN.

musher0

~~~~~~~~~~~~~~;
PS. My test Puppy was pemasu's raringPup 3992.

[amigo]

mikeb, glad you had some success -chrome is really a bad example because it needs some extra steps mount-binding of /tmp for one thing -which may fix the dbus errors.

The main little seed I wanted to sow was the mechanism for creating the union/chroot which is private and self-destructive once the program exits.

unionfs-fuse is completely independent of aufs -although it would not surprise me if there *is* a way that they get mixed-up.

About using such AppDirs with non-rox filers, all you need is a link from the AppRun script into your normal path(name the link after the program). This way, clicking on that link will do the same as clicking on the AppDir under rox.

If you want to play with that method, then you'll find it works without extra stuff (mount ---bind) for lots of stuff -no LD_LIBRARY_PATH mods needed and accesses to /etc /usr/share work as expected.

All chroots may suffer from strangeness similar to what chrome is doing particularly regarding underlying system mounts. mount --binding them in the union fixes it.

The process does look a bit complicated, but it's less messy than loads of links, and inside the chroot everything is transparent, while being sandboxed.

[mikeb]

yes I have one of those handles on this.. its often clearer when doing rather than when reading about it on a forum. I have a mental picture of the process and that's usually a good sign.

Only outstanding question is can you shed any light on why vlc could not find its plugin folder?.
I will test out more apps this way but if you have any pointers on the subject.

I also did not try out a mounted image or sfs yet so will play with that too...at least I have a working method now.

Note to general...this mounts the app in whatever form it takes on top of the puppy filesystem like good bunnies need to

mike

[sunburnt]

Mike & amigo; Yes, Chrome is a bad example.

unionfs-fuse is really touchy as to what options are used.
I found mhddfs works without any fuss, but as amigo said: "it`s not made for the purpose".

Like aufs, each running union instance is a cpu + ram load, but added layers not so much.
User space unionfs-fuse is slow compared to aufs, no doubt about it.
This is my reasoning for worrying about using many of these = many instances.
A few to augment a normal app install system or lots of apps in one would be no problem.

# Mike; Perhaps you could test the speed / sys. load effects of a few of these.?
.

[mikeb]

Hmm as mentioned it may particularly be useful for dev tool chains or sandboxing for the nervous.... at least it might save the reboot cycle some seem to go through.

One I will play with ..... just visiting lucid shutdown loops and its installer in general....qemu fun which is a little slow....and a general tidy round.

I think chrome should be called lead.... more appropriate metal.

mike

[sunburnt]

Chrome is actually fairly fast.
Firefox needed nothing, it runs from anywhere as-is. Very well written app.
The newer Chrome needed only one link because they put it in /lib ( Duh...).
But this is a no-brainer, to add and remove a link is far better than any union.


### MIke & amigo; A Q if I may...

I`ve got Rox installed on Debian Wheeezy and I have the Rox desktop line in:

$HOME/user/.bash_aliases

Line = rox -p /(path)/PinBd &

### This should start the Rox desktop at boot, but it doesn`t.
But if a Vterm is started then the Rox desktop starts, close the term. and Rox is gone.
# I figure this behavior is the key to getting it to start properly.

Any ideas that may help guys.?
.