Scratch File Tool

Filemanagers, partitioning tools, etc.
Message
Author
User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

Scratch File Tool

#1 Post by sc0ttman »

Scratch File Tool

Create, mount and unmount a scratch file - an ext2 linux filesystem inside a file

This is provided as is, although I will happily add any fixes or improvements suggested to me.
I don't think I know enough to make this any better, but will do so with help.

The mount point is leftover after unmounting, but other than that, this seems to work fine.

All scratch files are mounted to /mnt/scratch_mnt_point

A lot of credit should go to the contributions here

There is also a tip on the thread above which tells you how to mount the scratch file at startup, giving you more space automatically, when you boot :)
Attachments
scratch.png
(43.42 KiB) Downloaded 2862 times
scratch-file-tool.pet
(2.69 KiB) Downloaded 1360 times
Last edited by sc0ttman on Mon 30 Aug 2010, 13:18, edited 3 times in total.
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#2 Post by sc0ttman »

Any feedback on this?

I'd like to make it work 100%, but can't do it by myself :(
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
abushcrafter
Posts: 1418
Joined: Fri 30 Oct 2009, 16:57
Location: England
Contact:

#3 Post by abushcrafter »

What file systems does it support?

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#4 Post by sc0ttman »

ext2, its in the screenshot :)
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
abushcrafter
Posts: 1418
Joined: Fri 30 Oct 2009, 16:57
Location: England
Contact:

#5 Post by abushcrafter »

Thanks.

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#6 Post by sc0ttman »

Anyone used this? Any suggestions? I may have another go at making this again, and would be happy to include any requests, if anyone is interested..
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
abushcrafter
Posts: 1418
Joined: Fri 30 Oct 2009, 16:57
Location: England
Contact:

#7 Post by abushcrafter »

Haven't used it yet. As for features to add, all I can think of is more filesystem choices.
[url=http://www.adobe.com/flashplatform/]adobe flash is rubbish![/url]
My Quote:"Humans are stupid, though some are clever but stupid." http://www.dependent.de/media/audio/mp3/System_Syn_Heres_to_You.zip http://www.systemsyn.com/

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#8 Post by sc0ttman »

abushcrafter wrote:Haven't used it yet. As for features to add, all I can think of is more filesystem choices.
That's all I can think of, too.. Aside from cleaning up the code a bit.. :oops:
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

muggins
Posts: 6724
Joined: Fri 20 Jan 2006, 10:44
Location: hobart

#9 Post by muggins »

Was it from this thread?

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#10 Post by sc0ttman »

muggins wrote:Was it from this thread?
Yes it was! I couldn't find the thread again, to post a thanks/credits..
So yes, all credit for this goes to the guy who done the thing on that thread! :D
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

Any response on scratch files?

#11 Post by emil »

Hello sc0ttman,

did you have any response on this? I resently built a puppy derivate "sagelive" which contains the mathematic software sage (rather big). I also used noryb009's method to create a Windows Installer exe, so it is possible to install it frugaly into an Windows Filesystem from inside windows with one click.

I thought this is a cool method and major imporvement, especially because sage does not compile on windows, and even there is no Cygwin port available. This is a pitty because this cuts the potential userbase of the project enourmosly (especially in education).

However my approach was taken very sceptical from the sage community. There were fears to break something in the boot process, and also serious concern to protect the integrity of the NTFS filesystem. It is also understandable, because they want to do math and not primarily play around with bootloaders, partitions or fix broken systems of students.

Maybe I could counter this fears with the possibility to create a reasonable sized linux file on the NTFS partition which could be mounted at startup by default. It would be possible then to create all necessary files and diskspace once (safe file and scratch file), so later use would be independent of any NTFS - write operations.

However you mention that the scratch file is small in the beginning and grows later with use. Does this mean there is access to the NTFS system during its use to resize it?

Kind regards
emil

aarf

#12 Post by aarf »

Emil: Post a link to the sage discussion and we'll send over some heavys and give'em a whoopin'

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

Re: Any response on scratch files?

#13 Post by sc0ttman »

Hi emil...

In answer to your question, actually, although I was told the scratch file would grow in size, I am pretty sure it doesnt - if you create a 100mb scratch file, it will be 100mb right away... and doesnt get bigger.

You can auto mount the scratch file with a script in /root/Startup.. IT could also be put in rc.local, or .xinitrc (I guess)...

The command to mount the scratch file is:

mount -t ext2 -o loop /mnt/home/ext2_scratch.img /mnt/ext2_scratch/

(or similar)
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#14 Post by amigo »

Here's a couple of links that show one way of creating empty files under windows:
http://www.webtlk.com/2010/10/31/how-to ... g-systems/

http://www.softwaretipspalace.com/how-t ... _size.html

I used to maintain a distro called amigoXP which was designed to be installed from a running windows 98 or XP system. I used a program called bigfile.exe to create an empty file on the windows partition. The installer (a .bat file) would then create an entry in the boot.ini file which would let you boot the amigoXP system. Booting amigoXP would format the empty file with a linux filesystem, loop-mount it and then populate it with the main part of the distro. It was quite similar to what you will recognize as a frugal installation -excpet that everything was on the hard drive. The kernel and initrd were located in the root of the C: drive. The initrd contained some parts of the system, the rest were inside the loop-mounted partition image (what was the empty file).

The above two links show how to create the empty file using standard windows tools which can be scripted with a *.bat file or accessed using other methods (noryb009's method).

As for files that 'grow' as needed, it is possible to create a 'sparse' file which is large or small depending on which program you examine it with. See here:
http://murga-linux.com/puppy/viewtopic. ... 39&t=54800

and here:
http://www.murga-linux.com/puppy/viewto ... 5ec6e195fa

As mentioned in those posts, results will vary on windows. The thing is that these 'sparse' files are not really sparse -they only seem to be smaller than they really are because they are filled with zeros. Using the du '--apparent-size' option shows you the real size of them.

Oops, I just noticed that one of those threads is what is linked to above -still, the 'sparse' is completely separate and is only done with the 'skip' option to dd. Filling the file with zeros(without the skip option) will just show the file as having it's real size.

If you have no qualms about using ntfs-3g to access and write to ntfs drives, then you could always create and/or resize the files at will from linux -if the filesystem supports resizing -like ext2/3. The advantage of using ext2 is that less space is consumed by the filesystem formatting itself. Using ext3/4, reiserfs or other journalling filesystem will consume lots of space even before anything is written to the filesystem.

I run a system with a patched kernel which includes support for compressed ext2 filesystem, so that any files you copy to such a partition (or loop-mounted partition image) are automatically compressed whe they are copied. When reading the files they are transparently de-compressed. Using this feature displays similar properties as the sparse file -using du with and without the --apparent-size option gives you two different figures. But, in this case, you are able to put twice as many files in the same space. In other words, you can put 100MB of files into a 50MB partition or partition image without any extra tricks. By using upx and then ext2-comp, you xan put what would normally be 100MB of files into 25MB! kewl... but hard to maintain since it means maintaining a rather large and invasive kernel patch.

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#15 Post by sc0ttman »

amigo, the second link you gave is the one on which this tool is based.. listed earlier... I have not found the file that is created by this code to be a 'sparse' file..

I'm sure it's meant to, but that's just not my experience...

When i choose 100mb, its 100mb straight away... The file created by the code in the 2nd link you provided has always worked for me like an additional save file or something (stays the same size) ...
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

scratchfiles

#16 Post by emil »

Hello all, scottman, amigo!

thank you all for for the great and informative replies!
A few questions come to my mind:

1) What is the main advantage to have different scratchfiles? If I need diskspace of 5G is it not sufficient to blow up the already existing puppy-savefile?

2) Is it possible to lock down the NTFS partition (i.e is it possible to have the scratchfile and savefile working, but the rest locked)?

I did some research on this topic, chmod does not work on NTFS drives in Puppy. It is probably because the basic support for file permissions is not included by default in the ntfs-3g drivers. I read about it here:
http://b.andre.pagesperso-orange.fr/permissions.html

Maybe there is another possibility or workaround I am not aware of, but if not its not a big problem.

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

Re: scratchfiles

#17 Post by sc0ttman »

emil wrote:1) What is the main advantage to have different scratchfiles? If I need diskspace of 5G is it not sufficient to blow up the already existing puppy-savefile?
Well, a very large pupsave file will do a similar job, but here is the benefit, for me at least...
With a scratch file, I was able to have a ext3 filesystem/partition sitting on any drive I wanted..

This was much needed, until I dug out an old 4gb ATA disk from the loft.

I was using the scratch file to store a lot of files and pets I was making (vlc-gtk, others), that required file permissions to be set correctly, so could not be stored on FAT or NTFS..

(as you mention, chmod doesn't work on ntfs, but it works inside a scratch file, on an ntfs drive...)

Also, I was able to mount this scratch file in any puppy I was using, as it was not tied to any particular install..
That was the main advantage of using a scratch file and not my save file.

Essentially, if someone needs an linux (ext2) working area that can be accessed whenever, they can either format a whole drive, or create a scratch file.
emil wrote:2) Is it possible to lock down the NTFS partition (i.e is it possible to have the scratchfile and savefile working, but the rest locked)?
You can mount any partition as read-only, so you could mount your NTFS partition as read-only once booted into Puppy...

This should still allow you to save to your save file, even if it is on the read-only NTFS drive..

Not sure how scratch files will work in this case, haven't tested it - I need to read and write to my drives...
But there's only 1 way to find out! :)

Sorry I can't give you any specific technical info about this!
I'm no expert myself!
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

thanks

#18 Post by emil »

Hi scott,
I'm no expert myself!
well, it's all relative ...

Helped me a lot. At least I think those scratchfiles can be very usefull, especially on windows only computers.

About the NTFS - have to try that, it its still possible to save into the savefile while whole partition is mounted read-only?! sounds like voodoo.

:cool: big *thumbs up* emil

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#19 Post by amigo »

I'm not sure how ntfs-3g will act if you try to write to a 'scratch' file while the partition itself is mounted read-only -sounds ike it won't work. But, I think this will:
mount the partition read-write, then use losetup to assign the scratch file to a device. Then use the '-o remount ro' option to mount to remount the partition as read-only (or maybe even unmount it) -the device created with losetup will still exist and be usable.

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

hmm...

#20 Post by emil »

Hi amigo,

thank you. That might be an important piece in my puzzle :D.

I try a short summary of the paradox: How to run puppy from a NTFS disk, "but don't touch it"

1) There is the windows installer from noryb
http://www.murga-linux.com/puppy/viewto ... 4&start=30
it could be modified, so that there is an option to use "normal" or "protected" mode ("paranoid" mode would sound good too).

"normal" mode would be like now,

"protected mode" would do a defragmentation, chkdsk and then 2 blank files would be created from windows , e.g cmd like so

Code: Select all

fsutil file createnew C:\installdir\savefile 1024000000 
fsutil file createnew C;\installdir\scratchfile.img 5G
then at first startup of puppy look if we are sitting in an NTFS partition,
If using "protected" mode, then format the 2 blank files to ext2 or ext3 like so

Code: Select all

mke2fs -g -m 0 -F /mnt/sda1/installdir/savefile
then use the losetup like so:

Code: Select all

losetup /dev/loop0 scratchfile.img
mount /dev/loop0 /home/you/dir
then unmount the NTFS partitions.

Have to try that (when time alows).
emil

Post Reply