Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 27 Jan 2020, 03:29
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
rw sfs (read write squashfs)
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [19 Posts]   Goto page: 1, 2 Next
Author Message
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 12:42    Post subject:  rw sfs (read write squashfs)  

Latest script as attached

===================

Initial (original) design/concept.

Outline

A sfs file with the equivalent of a 'changes' folder integrally contained within that single sfs file. So you don't have to re-form (build) the sfs to incorporate changes (otherwise, to make changes to a standard sfs you have to extract it using unsquashfs, apply the desired changes, and then mksquashfs that extracted content to form a new sfs containing those changes). With this design you can open the sfs, modify/add/delete the content, close the sfs and for small changes the sfs will be updated and closed very quickly (changes made persistent).

Where you (script) runs the 'open' command and provides a sfs filename (or 'click it') and it opens up that sfs showing the top folder, and where you can modify files within that as desired. And a close option with optional save (or not) changes. Along with another option that caters for a 'defrag the sfs' action (reform with all changes moved into the sfs and a empty changes folder content).

Concept

Easy conversion of existing sfs's to rw from ro
Simple single file (.sfs) management and operation

Design

File structure

Existing or new sfs, appended with changes folder tarball and a 8 byte pointer - that points to the offset from the start of file to where the start of the changes folder tarball starts

i.e. the sfs file is modified to being a single file that is comprised of the concatenation of ..

sfs (generally fixed size)
changes tarball (compressed - variable size)
8 byte pointer (offset from start of file to the start of the changes tarball)


Notes

If you append data to the end of a existing .sfs file, then it can still be mounted/loaded ...etc as usual. What then if we simply append a tarball of the changes folder to the end of a .sfs along with a pointer that indicates where the start of that tarball occurs within the (single) file. Whilst the changes folder content could be stored in a sfs that was appended to the main sfs, in some environments such as in busybox we may be able to mount sfs but not have the unsquashfs command available - but where the tar command is (more often) available, so tar is the generally better choice for storing the changes folder content over using a sfs to store the changes folder content.

The main sfs is a fixed filesize, whilst changes folder (tarball) is variable size, so the changes tarball is better placed when appended to the end of the file, after the (fixed size) main sfs rather than the other way around.

Last 8 bytes of file is the offset pointer from start of file to where the changes folder tarball starts. 8 byte pointer caters for massive filesizes. Could be smaller, but what the heck, we're only saving bytes by making it smaller.

For the tarball, I'm opting for lzop -1 compression as lzop is commonly available and its fast to compress/decompress. But isn't the tightest of compression methods. i.e. here I'm going for speed over size/space. lz4 would be a better choice, but lz4 isn't, at least currently, commonly available (lzop for instance is typically available within busybox, lz4 isn't).

For choice of mounting, overlay is pretty much default kernel present nowadays (but not in Puppy's !!), aufs requires kernel mods, so overlayfs seems to be the current better general choice.

Pseudo Code

The script/program to 'open' the sfs, in a (inefficient) pseudo code like manner ...

#read last 8 bytes .. pointer to start of changes tarball
tail -c 8 > tarball.dat
X=`cat tarball.dat`
rm tarball.dat
# not sure about this ... blocksize may have restrictive limits ??
# but for now ... to strip out changes folder tarball ..
dd bs=$X skip=1 if=filename.sfs of=changes.tar.lzo
# remove last 8 bytes (pointer)
truncate -s -8 changes.tar.xz
# Extract the tarball
lzop -d -c changes.tar.lzo | tar xvf -
# We now have the main sfs (can mount filename.sfs as-is)
# and changes folder content (extracted tarball)
# ... so we can overlay (or could be aufs) mount those using standard
# methods and read/update/remove files/folders etc. as desired
mkdir s w t
# (note folder c here = changes folder which would already have been
# created and loaded (extracted tarball content) earlier/above)
mount filename.sfs s
mount -t overlay overlay -o lowerdir=s,upperdir=c,workdir=w t

folder c then has all the changes stored in it, and t is the top view folder (merged view of changes and main sfs). Where we can add/remove/change files/folders in t and those changes are recorded in c

... when done we can either just release all of that, and no changes will be preserved, or we can reform the main file, again with the unchanged main sfs content, appended with a tarball of the changes folder (c in the above), appended with a pointer to the start of that changes folder tarball.

Truncating the original file is a quick operation, just involving the truncate command (to the size of the sfs), pretty much a immediate return. Then we tar up the changes folder (which I've used 'changes' for here rather than the earlier 'c' folder name, for greater clarity)

tar cvf - changes | lzop -1 >>filename.sfs

and append that to the truncated sfs file

and also append the 8 byte offset to the start of the changes tarball
cat $SFS_SIZE >>filename.sfs


Periodically we might want to 'defrag' that file, i.e. merge all of the changes stored in the tarball to inside the main sfs ... and then reset (zero) the changes tarball. Again that is a relatively simple straightforward task, that just involves opening the file, and making a sfs of the 'top' view (combined sfs + changes view), and then reform the file again but using that new sfs instead of the original one and with a empty changes tarball content (along with the 8 byte pointer to that at the end of the file).
rwsfs.gz
Description 
gz

 Download 
Filename  rwsfs.gz 
Filesize  2.49 KB 
Downloaded  65 Time(s) 

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh

Last edited by rufwoof on Sat 19 Oct 2019, 19:59; edited 6 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 13:25    Post subject:  

This code is for checking a existing sfs file is a actually a sfs file (by checking it magic number) and that it isn't already a rw sfs, and if so it appends a empty tarball of a changes folder along with the reference. I've opted to extend the original design 8 byte binary assumed offset to a combined 20 byte reference where the first 5 bytes of that are a "rwsfs" magic code indicator and the remainder 15 bytes are a ascii numeric indicator of the offset into the file to the start of the tarball (that runs up to the end of file excepting the 20 byte reference that is at the absolute end of file)

Update: with a few test scripts created it became apparent that using no compression for tar was better speed wise

script name : rwsfs-convert
Code:
#!/bin/bash

# This script adds a reference to a sfs that is a pure sfs,
# not already a rw sfs

# $1 is sfs filename

# reference is the last 20 bytes that is appended after the tarball
# has been appended to the sfs
# reference format is ...
#   5 bytes magic "rwsfs" indicator
#   15 bytes for offset from start of file to just after the sfs
#   i.e. start of .changes.tar (which runs up to the start of
#   the reference i.e. last 20 bytes)
#   offset is stored as a 15 byte string of filesize with leading zeros

FILENAME=${1}

# Validations
if [ -z ${1} ] || [ ! -f ${1} ]; then
   echo "Usage ${0} sfs-filename"
   echo
   echo "This script takes an existing standard sfs file"
   echo "and turns it into a rw sfs, where changes made"
   echo "can optionally be saved when the sfs is closed"
   echo
   exit 1
fi

# Check file is actually a sfs file using sfs magic code
V=`head -c 30 ${FILENAME} | hexdump -C | head -1 | grep "00000000  68 73 71 73"`
if [ -z "$V" ]; then
    echo ${FILENAME} is not a sfs
    exit 1
fi

# check if already a rw sfs (reference already exists)
# (Weird tail of bin file errors shown unless do it this way)
O=`mktemp`
tail -c 20 $FILENAME >$O
REF=`cat $O | grep rwsfs`
rm $O
MAGIC=`echo $REF | head -c 5`
if [ "$MAGIC" = "rwsfs" ]; then
   echo "Already a rw sfs"
   exit 1
fi

# Add reference
FILESIZE=`du -b "${FILENAME}" | cut -f1`
STARTOFTARBALL=`expr $FILESIZE + 1`
STARTOFTARBALL=`printf "%015d" $STARTOFTARBALL` # leading zeros

# First we append the .changes.tar file (initially a empty version)
CD=`pwd`
cd /tmp
TEMPDIR=`mktemp -d`
cd ${TEMPDIR}
mkdir .${FILENAME}-changes
tar cvf .${FILENAME}.tar .${FILENAME}-changes >/dev/null 2>&1
mv .${FILENAME}.tar ${CD}/.
rmdir .${FILENAME}-changes
cd /tmp
rm -rf $TEMPDIR
cd $CD
cat .${FILENAME}.tar >> ${CD}/${FILENAME}
rm .${FILENAME}.tar

# Then we append the reference
echo -n "rwsfs" >>${FILENAME}
echo -n $STARTOFTARBALL >>${FILENAME}

echo Done
exit 0

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh

Last edited by rufwoof on Thu 17 Oct 2019, 11:55; edited 9 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 16:49    Post subject:  

Script to open (mount) a rw-sfs for viewing/editing the content

Script : rwsfs-open
Code:
#!/bin/bash

# This script opens and mount a rw sfs
# sfs should be stored/run from a standard linux format partition
# (not from within a tmpfs)

# $1 is the rw sfs filename
# $2 is the mount point (empty folder) where the rw layer should be mounted to
# $3 .. optional. If exists then rox isn't opened

FILENAME=${1}
MOUNTPOINT=${2}

# For validations ...
# Set the variable for bash behaviour
# nullglob If set, bash allows patterns which match no files to expand to
# a null string, rather than themselves.
# dotglob – If set, bash includes filenames beginning with a . in the results
# of pathname expansion.
shopt -s nullglob
shopt -s dotglob

# Validations
chk_files=(${2}/*) # create a array of files in $2 - i.e. should be empty
if [ -z ${2} ] || [ ! -f ${1} ] || (( ${#chk_files[*]} )); then
   echo "Usage ${0} sfs-filename mount-point"
   echo
   echo "   Opens (mounts) a rw sfs file"
   echo
   exit 1
fi

# Check file is actually a sfs file using sfs magic code
V=`head -c 30 ${FILENAME} | hexdump -C | head -1 | grep "00000000  68 73 71 73"`
if [ -z "$V" ]; then
    echo ${FILENAME} is not a sfs
    exit 1
fi

# check if already a rw sfs (reference already exists)
REF=`tail -c 20 ${FILENAME}`
MAGIC=`echo $REF | head -c 5`
if [ "$MAGIC" != "rwsfs" ]; then
   echo ${1} is not a rw sfs
   exit 1
fi

mkdir ${MOUNTPOINT}
if [ $? -ne 0 ]; then
   echo "error creating mount point ${MOUTPOINT}"
   exit 1
fi

if [ -d .${FILENAME}-s ] || [ -d .${FILENAME}-w ] ; then
   echo "Already open ?? (intended work folders .${FILENAME}-s and/or .${FILENAME}-w already exist)"
   echo "unable to continue"
   rmdir ${MOUNTPOINT} >/dev/null 2>&1
   exit 1
fi

# locate offset from start of file to start of tarball
OFFSET=`tail -c 15 ${FILENAME}`
OFFSET=`expr $OFFSET + 0`           # strips of leading zeros
SFSSIZE=`expr $OFFSET - 1`
echo "opening ..."
# Extract out the tarball + reference (we un-tar with that intact - doesn't seem to hurt)
dd bs=$SFSSIZE status=none skip=1 if=${FILENAME} | \
   tar xf - --one-top-level=.${FILENAME}-changes --strip-components 1 >/dev/null 2>&1

mkdir .${FILENAME}-s .${FILENAME}-w # prepare sfs and work folders
mount ${FILENAME} .${FILENAME}-s    # mount the sfs
# Mount (overlay) to the specified mount point $2
mount -t overlay overlay -o lowerdir=.${FILENAME}-s,upperdir=.${FILENAME}-changes,workdir=.${FILENAME}-w ${MOUNTPOINT}
# Open rox showing the top layer folder
if [ -z ${3} ]; then
   rox ${MOUNTPOINT}
fi
exit 0

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh

Last edited by rufwoof on Sat 19 Oct 2019, 18:13; edited 6 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 19:08    Post subject:  

close rw sfs

If 'save' isn't specified it just releases the mounts/closes the overlay
If 'save' is specified then changes made to the opened sfs file/folder content are preserved.

Script : rwsfs-close
Code:
#!/bin/bash

# This script closes a rw sfs

# $1 is sfs filename
# $2 is optional [save] if included changes are saved,
#                      otherwise closes without saving

FILENAME=${1}

# Validations
if [ -z ${FILENAME} ] || [ ! -f ${FILENAME} ]; then
   echo "Usage ${0} sfs-filename [save]"
   echo
   echo "     if 'save' is not specified the sfs is closed"
   echo "     without any changes being preserved"
   echo
   exit 1
fi

# Check file is actually a sfs file using sfs magic code
V=`head -c 30 ${FILENAME} | hexdump -C | head -1 | grep "00000000  68 73 71 73"`
if [ -z "$V" ]; then
    echo ${FILENAME} is not a sfs
    exit 1
fi

# check if already a rw sfs (reference already exists)
# (Weird tail of bin file errors shown unless do it this way)
O=`mktemp`
tail -c 20 $FILENAME >$O
REF=`cat $O | grep rwsfs`
rm $O
MAGIC=`echo $REF | head -c 5`
if [ "$MAGIC" != "rwsfs" ]; then
   echo "Not a rw sfs"
   exit 1
fi

# where its mounted
MOUNTPOINT=`mount | grep ${FILENAME} | grep overlay | awk '{print $3}'`

if [ -z "$MOUNTPOINT" ]; then
   echo Not found
   exit
fi

if [ "${2}" != "save" ]; then
   # just close
   umount ${MOUNTPOINT}
   umount .${FILENAME}-s
   sync
   rm -rf ${MOUNTPOINT} .${FILENAME}-s .${FILENAME}-w .${FILENAME}-changes
   echo "${FILENAME} closed (any changes lost)"
else
   umount ${MOUNTPOINT}
   umount .${FILENAME}-s
   rm -rf ${MOUNTPOINT} .${FILENAME}-s .${FILENAME}-w
   
   # create tarball of changes folder content (only if exists)
   if [ -d .${FILENAME}-changes ]; then
      echo "saving ..."
      OFFSET=`tail -c 15 ${FILENAME}`
      FULLOFFSET=$OFFSET
      OFFSET=`expr $OFFSET + 0`  # strips of leading zeros
      SFSSIZE=`expr $OFFSET - 1`
      # truncate the file down to just the sfs part
      truncate -s $SFSSIZE ${FILENAME}
      # Append the tarball
      tar cf - .${FILENAME}-changes >> ${FILENAME}       
      # Append the reference
      echo -n "rwsfs" >>${FILENAME}
      echo -n $FULLOFFSET >>${FILENAME}
      rm -rf .${FILENAME}-changes &
      echo "${FILENAME} closed (changes saved)"
   fi
fi
exit 0

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh

Last edited by rufwoof on Sat 19 Oct 2019, 18:14; edited 8 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 19:09    Post subject:  

defrag rw sfs initial draft

i.e. open the rw sfs, make a new sfs of the top view (combined sfs + changes view), and append a empty changes tarball (as now all changes are in the main sfs) + append a new reference (last 20 bytes of file).
Code:
#!/bin/bash

# open the rw sfs, make a new sfs of the top view (combined sfs + changes view),
# and append a empty changes tarball (as now all changes are in the main
# sfs) + append a new reference (last 20 bytes of file).

T=".a.b.c"
./rwsfs-open $1 $T norox
if [ $? -ne 0 ]; then
   echo error opening rw sfs
   exit 1
fi
mksquashfs $T newsfs.sfs -comp lz4
if [ $? -ne 0 ]; then
   echo error creating sfs
   exit 1
fi
./rwsfs-convert newsfs.sfs
if [ $? -ne 0 ]; then
   echo error setting newsfs.sfs as a rw sfs
   exit 1
fi
./rwsfs-close $1
rm $1
mv newsfs.sfs $1

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh

Last edited by rufwoof on Sat 19 Oct 2019, 16:02; edited 2 times in total
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 14724
Location: Gatineau (Qc), Canada

PostPosted: Wed 16 Oct 2019, 20:32    Post subject:  

Isn't this the job of aufs?
_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Wed 16 Oct 2019, 21:09    Post subject:  

It's a higher layer Musher.

Yes aufs could be used, but my code is currently using overlayfs. Changing the code from overlayfs to aufs is a relatively small change, so a later version might include the option to choose (or sense) one or the other.

From my earlier code as-is, converting a ro sfs to a rw sfs is adding around a 200 bytes overhead. Generally a relatively small increase in filesize for the benefit of ease of use. Click to open it, make changes to files/folders within that, click close-with-save and it quickly closes with those changes recorded in the single sfs file. No separate ro content (sfs) and changes (rw) files/folders - or having to reform the entire sfs (slow for larger content) in order to incorporate changes.

If for instance I send you a sfs containing pictures, you click to open it, view the images, perhaps opt to remove some, add others, click close-with-save and the sfs is quickly updated to include those changes. Still the same single sfs file, but dynamic in size according to the changes made.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Thu 17 Oct 2019, 02:57    Post subject:  

The convert ro sfs to a rw sfs script, together with the open and close rw sfs scripts are starting to take shape. The defrag script has yet to be coded.

Those scripts functionality has deviated from the original (first post) design/concept but remain pretty much in line with that original conceptualisation.

Tests using a large sfs and adding in large additional data indicate reasonable performance, whilst being considerably quicker than having to re-form the entire sfs (as is required a traditional sfs's content is being modified) to incorporate few/small changes.

For clarity I should mention that changes can only be made persistent when the rw sfs is opened (overlay mounted) i.e. using the scripts. If the sfs is directly mounted (without layering) then changes cannot be made/preserved as that just mounts the ro sfs element alone.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Thu 17 Oct 2019, 10:43    Post subject:  

Let's say you have a bunch of music files, perhaps that individually are only lightly compressed (higher quality sound) all stored in /mnt/sda1/MUSIC

You create a sfs of that folder

mksquash MUSIC/ music.sfs

.. so they're all together in a single file (music.sfs). Being a large number and individually large size files producing that sfs can take a considerable amount of time to create. If you want to add (or remove) a file from that, you have to again go through the same (slow) process in order to 'update' that sfs (create a new sfs containing the changes).

With the rw-sfs approach however you can open the sfs, add or delete files within that, close (with the save option) the sfs ... and that will close much quicker with the changes incorporated into your music.sfs file.

A downside however is that added files aren't compressed, its as though they were just tagged onto the end in uncompressed form (but if that/those files were already compressed then they remain in that compressed form). Also the original file that they might replace still remains in the sfs, as though it were just flagged for deletion, not actually deleted. So the rw-sfs format is less space efficient than totally rebuilding a new sfs with the changes incorporated. However the defrag script (that has yet to be coded) mitigates that - as that in effect re-forms the sfs - similar to completely rebuilding a traditional sfs with the incorporated changes.

Fundamentally, the rw sfs is quicker to use than the traditional sfs when making small changes to the content of the sfs, single/few files added or changes to lots of small files. But after a while as the number of changes accumulate you'll want to defrag the rw sfs.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Thu 17 Oct 2019, 15:59    Post subject:  

Not being able to mount the rw-sfs and see the changes is a disadvantage.

Using just standard sfs alone we might for instance layer the main sfs with a changes folder content, that automatically stores/updates changes in that changes folder as the top layer is accessed/used.

If we have a master copy of the sfs ... master.sfs, and we create a copy of that ... copy.sfs, and changes are being stored in folder 'changes', then we can

mkquashfs changes copy.sfs -keep-as-directory

to insert that changes folder into the sfs, which is a relatively quick process, especially if using a fast compression method such as lzo or lz4. At the next boot we might mount the sfs, copy out the changes folder content (cp -a changes .) and then aufs (or overlay) layer the master.sfs with that changes folder content. Repeating that copy master.sfs to copy.sfs, insert changes folder content ... for each reboot/iteration (storing the latest changes inside the sfs).

Formed that way and the changes are visible when the sfs is simply mounted, unlike the method outlined in this thread where the changes are only visible when the sfs is actually loaded (overlaid).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1862
Location: not Bulgaria

PostPosted: Fri 18 Oct 2019, 00:06    Post subject:  

The concept of main sfs but with changes as tar component seems good to me; then you occasionally update the sfs with the changes content and start new tar changes. However, what is the purpose/advantage of concatenating the changes onto the main sfs? i.e. what is wrong with keeping sfs as a normal sfs and the changes as a separate tar? Seems to me that would save needing to calculate and append the offset to changes information?

wiak

_________________
New Puppy/Dog forum: https://puppylinux.rockedge.org
Tiny Linux Blog: https://www.tinylinux.info/
Check Firmware: http://murga-linux.com/puppy/viewtopic.php?p=1022797
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 3102

PostPosted: Fri 18 Oct 2019, 02:42    Post subject:  

That can be interesting!
However, circumvents one basic SFS idea, that they are immutable.
If this is to be implemented should

a) load and save the tarball ONLY with explicit user direction (in both cases)
b) provide a new (tar) version number every time the tarball is changed that will be displayed upon loading the tarball so seemingly identical SFS can be identified
c) provide and check the tarball hash to check if the file has been tempered
d) make the tarball layer easily identifiable and user reviewable (ie see what's in there)
e) preferably have the script integrated in the immutable part of the SFS and ignore calls from elsewhere.

I guess what I'm thinking is if "my_super_os-v123.SFS" is not the same file and the differences can not be easily and safely identified by everyone, can generate both a debug havoc and a a security problem.

(didn't go through the code Embarassed so apologies if any of the above is already implemented)

_________________
== Here is how to solve your Linux problems fast ==
Back to top
View user's profile Send private message 
stemsee

Joined: 27 Jun 2013
Posts: 2572
Location: In The Way

PostPosted: Fri 18 Oct 2019, 04:58    Post subject:  

wiak wrote:
The concept of main sfs but with changes as tar component seems good to me; then you occasionally update the sfs with the changes content and start new tar changes. However, what is the purpose/advantage of concatenating the changes onto the main sfs? i.e. what is wrong with keeping sfs as a normal sfs and the changes as a separate tar? Seems to me that would save needing to calculate and append the offset to changes information?

wiak


One possible explanation is that a particular tar goes with a particular sfs....no possibility to mix'n'match!!

Could this not also be done in the same way with a regular savefile? (assuming the changes are for the base sfs)

Does this replace the need for a savefile or do they work together, as the changes are appended to each user sfs? If so multiple sfs files with appended tarred changes get handled....?

stemsee
Back to top
View user's profile Send private message MSN Messenger 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Fri 18 Oct 2019, 10:26    Post subject:  

mavrothal wrote:
However, circumvents one basic SFS idea, that they are immutable.
.
.
c) provide and check the tarball hash to check if the file has been tempered

SFS's can be tampered. The super-block (first 100 bytes or so) details the layout and things are stored sequentially (super-block also obviously declares/records the compression method selected for that sfs). Nothing to stop me from reading that compression method, and the directory table for a particular file within the sfs and identifying that files inode that indicates how many bytes from the start of the sfs file to where the compressed data for that individual file starts (sfs's only require one inode as the compressed data all then follows sequentially) ... and then change those bytes. So if say the sfs was using gzip compression, I could replace the gzip'd content for a bin file with alternative gzip'd data and provided that injected programs compressed binary size was less than the compressed binary it was replacing (i.e. fitted within the available space), then that alternative code would be run whenever the 'intended' program was launched. Or indeed a subset of that - for instance replace part of busybox so that ls was redefined to do something different to intended (or seem to run as usual and do something else in the background).

Personally for such reason I checksum my main sfs as part of bootup if it is being stored on hdd (more usually I keep it on my boot usb that is unplugged once the system has booted) as a intrusion detection check. However that is less likely a target than is the MBR, bootloader, kernel if a crack has gained access to the system.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3717

PostPosted: Fri 18 Oct 2019, 11:08    Post subject:  

stemsee wrote:
wiak wrote:
The concept of main sfs but with changes as tar component seems good to me; then you occasionally update the sfs with the changes content and start new tar changes. However, what is the purpose/advantage of concatenating the changes onto the main sfs? i.e. what is wrong with keeping sfs as a normal sfs and the changes as a separate tar? Seems to me that would save needing to calculate and append the offset to changes information?

wiak

One possible explanation is that a particular tar goes with a particular sfs....no possibility to mix'n'match!!

Could this not also be done in the same way with a regular savefile? (assuming the changes are for the base sfs)

Does this replace the need for a savefile or do they work together, as the changes are appended to each user sfs? If so multiple sfs files with appended tarred changes get handled....?

stemsee

I had no real particular application in mind, just liked the idea of keeping it altogether rather than having to associate a sfs to a particular record (file) of changes. Also just coded it up in order to gain a feel of how well (or not) it actually ran in practice.

Overall however the 'better' choice could be (???) to just use the sfs alone. Using just standard sfs alone you could replace the need for a 'permanent' save file/folder ... of sorts, by making it a temporary file/folder, where the changes/save content was read out/written back to the sfs at bootup/shutdown (whenever). Mount the sfs that also contains a copy of changes/save folder content and extract out the save content and then aufs layering the sfs with that extracted folder content. Could even create a temp file filesystem specifically for that (ext file filesystem located on a fat/ntfs format partition and unsquashfs the changes folder content to that). But with speed in mind that is best matched with having a second 'clean/master' version of the sfs as copying that and inserting just the additional 'changes' (save) is generally much quicker than reforming the entire sfs (mksquashfs).

cp mastercopy.sfs newcopy.sfs
# add the changes/save content to that (relatively quick provided not too big)
mksquashfs /mnt/sda1/changes newcopy.sfs -keep-as-directory
rm currentcopy.sfs
mv newcopy.sfs currentcopy.sfs

Compared to the sfs/tarball approach ...

truncate sfs down to just the sfs size alone (very fast operation)
tarball append the changes/save content to the end of that (moderately fast)
Append the rw-sfs's 'reference' (last few bytes of file that indicates the offset from start of file to where the tarball starts, which if the actual real sfs filesize remains unchanged (more often the case) is the exact same content/value as before).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [19 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0926s ][ Queries: 12 (0.0093s) ][ GZIP on ]