How to put a progress bar in dir2sfs? (Solved)

[rg66]

I just have one question: you wrote this above in an earlier message:

Code: Select all

This also works for unsquashfs as well  Very Happy 

Am I misunderstanding? I dropped one of the newly created .sfs files on the script to test the unsquashfs capabilitiy, and the script does nothing.

Was/Is the script meant to do unsquashing too just by dropping a .sfs on it? (I don't see unsquashfs commands in the script, so was trying to figure out what you meant when you wrote "this also works for unsquashfs as well...very happy".



P.S. Just found something weird...in Micko's latest Slacko64 (700-r3), on another machine at my desk here, the script does nothing. It won't launch at all, even though it is set up the same way with the same permissions as in X-Slacko-4.4. Weird, as it should easily run in 64-bit slacko because it's only a script after all, right?

Sorry belham2, I meant the code will also work with the unsquashfs command.

I've updated pac-tools with the new progress bar (attached below, just remove fake .gz and make sure it's executable) put it in /usr/bin. You can select as many sfs's as you want (as long as /tmp has enough space to extract them) and right click > repack-extract packages > extract. You can also put the new dir2sfs in /sbin and right click multiple folders > Convert to sfs. The right click options are in X-series and Carolina.

Slacko64 (700-r3) probably doesn't have yad so the script will fail. Also check if it has the script command, if not, download slackware util-linux from the repo.

Edit: pac-tools had a few typo's do to a sed accident , I've uploaded a new one, see below.

[fredx181]

Hi Belham, here's sfs2dir, basically the same as the last dir2sfs from rg66, but the other way around.
Unsquashes to directory name, same as the name of .sfs (but without the .sfs extension)

Code: Select all

#!/bin/bash

[ ! -f "$1" ] && exit
DIR="${1%.*}"

if [ "$(echo "$1" | wc -w)" -gt 1 ]; then
  yad --window-icon="application-x-squashfs-image" --title="Rename File" --text="Remove blank spaces from\n<b>$(basename "$1")</b> \
and try again." --text-align="center" --borders="10" --width="220" --no-buttons --timeout="3"
exit
fi

if [ -d "$DIR" ];then
  yad --window-icon="application-x-squashfs-image" --title="Overwrite" --text="<b>$(basename $DIR)</b> already exists.\n Would you like to overwrite it?\n" \
  --text-align="center" --borders="10" --buttons-layout="center" --button="gtk-yes:0" --button="gtk-no:1"
  case $? in
    0) rm -rf "$DIR" ;;
    *) exit ;;
  esac
fi

(script -q -c "stty rows 40 cols 100; unsquashfs -d "$DIR" "$1"" | while read -n 50 LINE ; do
echo $LINE | busybox strings | egrep '[0-9]\%' | awk '{print $NF}'; done
) | Xdialog --title "Extracting Squashfile " --gauge "Extracting $(basename $1)\n\nPlease Wait..." 7 50 0
if [ $? -ne 0 ]; then
  killall unsquashfs
  rm -f typescript
  exit
fi

rm -f typescript
sync 

Fred

[technosaurus]

goingnuts and I did a progress bar implementation for tar here ... could probably be adapted

[slavvo67]

So if you don't have the script command, I guess you're out of luck?

[musher0]

Hi all.

I still think you guys are foillowing the principle: "Why do simple when you can do
complicated!" A couple of lines of poetry are in order, I think:

"As I see you guys go,
My love for console grows."

Really!

As a reminder:
http://murga-linux.com/puppy/viewtopic. ... ost#969545

Where is the world heading?!

Merrry Christmas to all.

[phat7]

So if you don't have the script command, I guess you're out of luck?

No, you are lucky. You can ignore all this.
musher0 has a point.

[stemsee]

Hi all

This was just what I was looking for!

Up until this point the other versions didn't allow me to have several working at the same time, not sure why.

I have made some modifications and the yad version is my preferred version.

stemsee

[stemsee]

The only problem I have is that with --auto-close we don't see the progress bar reach the end. Without --auto-close we have to manually close the yad gui. either by getting focus and pressing Esc key, or using the Stop button.

It would be great to get the 100 exported so some monitoring loop could kill the yad when 100 has been detected, but I couldn't do it.

stemsee

[rockedge]

@stemsee

I tried out the script on WeeDog (Void Linux) and it works perfectly.

[stemsee]

Thanks for confirming report @rockedge

[stemsee]

Now I made the script give defaults for overwrite and compression options. And in case of using Stop to interrupt, the partially created sfs is removed.

Code: Select all

if [ -f "$1.sfs" ];then
  DO=$(yad --window-icon="application-x-squashfs-image" --title="Overwrite" --text="<b>$(basename $1).sfs</b> already exists.\n Would you like to overwrite or rename it?\n" \
  --text-align="center" --borders="10" --buttons-layout="center" --button="gtk-yes:1" --button="gtk-no:0" --center --timeout 2)

  case $DO in
    0|252) exit ;;
    70|1) rm -f "$1.sfs" "$1.sfs-md5.txt" ;;
  esac
fi

Code: Select all

--button="GZIP:0" --button="XZ:1" --button="XZ-HC:2" --center --timeout 2
[[ $? == "" ]] && $?="1"
case $? in
  0) COMP="-comp gzip" ;;
  1) COMP="-comp xz" ;;
  2) COMP="-comp xz -b 1024k -Xbcj x86" ;;
  *) exit ;;
esac

Code: Select all

if [ $? -ne 0 ]; then
  rm -f "$1.sfs"
  killall mksquashfs
  kill $PID
  rm -f typescript
  exit
fi

However killall mksquashfs is not ideal in case you have more than one mksquashfs job running! We need to target the specific mksquashfs process id! Or by getting the time each process started we can use ps -e | grep mksquashfs, time running to determine which process to kill.

EDIT: Maybe we can assign the pid for each mksquashfs process.

Code: Select all

echo "$((RANDOM))" >/tmp/$1

Code: Select all

cat  /tmp/$1 > /proc/sys/kernel/ns_last_pid; mksquashfs

Code: Select all

kill $(cat /tmp/$1)

[rufwoof]

I'd suggest including lz4 compression also. Very fast (but relatively poor compression). lz4 -Xhc tends to be more like gzip for compression, but is slower (relatively speaking), so not really worth including that IMO when gzip is available (unless you want very fast decompression also), just the standard lz4 ... for its speed. lzo -Xcompression-level 1 is a alternative, not as quick, but relatively quick ... and more commonly available (lzop is in busybox for instance).

Re killing. I know you can get a process name for a pid using the comm= parameter

# ps -p 2523 -o comm=
geany

maybe something along those lines might be used??

[stemsee]

Hi rufwoof

can you provide a scripted example of your ps * comm= command? I could not use it as is.

Added lz4.

I found a way to catch the mksquashfs pid by time, but I cannot now get the script to kill it.

Code: Select all

time=$(date | awk '{print $4}')  # unique time label

( sleep 1; ps -e | grep mksquashfs | tail -1 | awk '{print $1}' >/tmp/$time; exit ) & # delay and catch the last mksquashfs pid and store in file

(script -q -c "stty rows 40 cols 100; mksquashfs "$1" "$1.sfs" $COMP" | \
while read -n 100 LINE; do echo $LINE | busybox strings | egrep '[0-9]\%' | \
awk '{print $NF}'; done ) | yad --progress --center --undecorated --on-top \
--skip-taskbar --progress-text="Building $1.sfs"  --borders="10" \
--text-align="center" --button="gtk-stop:0" \
--buttons-layout="center" --auto-close

if [[ $? -eq 0 ]]; then
  pd=$(cat /tmp/$time) # read file for pid
  kill $pd
  rm -f "$1.sfs"
  rm -f "$1.sfs-md5.txt"
  rm -f typescript
  exit
fi

[rufwoof]

Maybe its different under Fatdog (proper/full Linux) ??

For instance if I run galculator and put it into the background, it shows pid 4550 was used. If I then run ps -p 4550 -o comm= ... then it returns the name i.e. galculator.

# galculator &
[1] 4550
# ps -p 4550 -o comm=
galculator
#

Does that do similar for you?

If you use args= instead, then it shows both the command and arguments. Starting geany /mnt/sdb1/menu.lst for instance

# geany /mnt/sdb1/menu.lst &
[1] 4899
# ps -p 4899 -o comm=
geany
# ps -p 4899 -o args=
geany /mnt/sdb1/menu.lst
#

Another approach might be to bring the relevant backgrounded task to the foreground and kill that (i.e. normal fg, bg, jobs commands)

# jobs
[1]+ Running geany /mnt/sdb1/menu.lst &
#
# fg %1
... and geany becomes the active window/task

or kill %1 ... to kill it.

With mksquashfs it does use multi-cores, but running mksquashfs /usr usr.sfs & and then I can run jobs and see its [2] and fg %2 brings that to the front and I can ctrl-c end it and all cores (child) tasks also closed. When doing that interactively however you get the mksquashfs progress bars ticking away between running the commands unless you direct that to /dev/null.

You can select jobs by the first few characters of the string

galculator &
fg %gal

or via a string within the command line, so if you were mksquashfs /usr ... and mksquashfs /lib64 ... then you could foreground (or kill) the mksquashfs /lib one by using fg %?lib ... i.e. ?lib matches the substring lib contained within the command line.

[stemsee]

Cracked it!

I got it working with specific mksquashfs process IDs. And lz4 option.

Probably there is a better way.

stemsee

[technosaurus]

I'd suggest including lz4 compression also. Very fast (but relatively poor compression). lz4 -Xhc tends to be more like gzip for compression, but is slower (relatively speaking), so not really worth including that IMO when gzip is available (unless you want very fast decompression also), just the standard lz4 ... for its speed. lzo -Xcompression-level 1 is a alternative, not as quick, but relatively quick ... and more commonly available (lzop is in busybox for instance).

For mksquashfs slow compression isn't that big of a deal since it only compresses once but gets decompressed thousands of times over. Using lz4 with high compression can give comparable compression to gzip but decompresses 10s of times faster - kind of important for read only filesystems especially during boot. zstd is another one that got added in recent years that is better than deflate (gzip) in compression speed, compression ratio and decompression speed - with the possibility of approaching xz levels of compression with minimal effects on speed. There was another one added for small data (4096 bytes) to better compress swap memory called zBeWalgo or 842 - looks like it would be horrible for squashfs

[stemsee]

ZSTD and LZ4 -Xhc options available.

also tidied up ending. md5 was not created!

[step]

Cracked it!

I got it working with specific mksquashfs process IDs. And lz4 option.

Probably there is a better way.

stemsee

Code: Select all

# original
( sleep 0.3; ps -e | grep mksquashfs | tail -1 | awk '{print $1}' >/tmp/$time ) &

# replace tail -1 with awk:
# when awk enters END $0 is the last input line
( sleep 0.3; ps -e | grep mksquashfs | awk 'END {print $1}' >/tmp/$time ) &

A shorter way to get the pid of the lastly-executed mksquashfs command is pgrep -n mksquashfs

Code: Select all

( sleep 0.3; pgrep -n mksquashfs >/tmp/$time ) &

(pgrep must be installed)

Concerning the initialization of variable time, it could be simplified this way
time=$(date +%T). You could replace %T with %s, too.

SIDE NOTE

A bigger change (but worth the learning exercise, and the improvement it would make to script robustness) is to replace the whole idea of writing a temporary script and attaching it to the Quit button, is to set an exit trap.

Code: Select all

trap 'my commands here' EXIT

Then, when the script is about to exit it runs 'my commands here'. Your commands could be the stuff that currently goes into the temporary script.

Two gotchas concerning traps:

1/
You need to set the trap in the subshell that is supposed to run it.

Code: Select all

# main script
trap 'echo BYE' EXIT

prints BYE when main script exits.

Code: Select all

# main script
trap 'echo BYE' EXIT
# subshell
(trap 'echo HI' EXIT)

prints HI --from the subshell-- then BYE --from main script.

Any command placed to the right of a pipe character runs in a subshell.

2/
trap "echo "$VAR"" substitutes $VAR when the *trap* command is executed, while trap 'echo "$VAR"' substitutes $VAR when echo is executed (on EXIT). In the latter case $VAR must be defined in the process that *executes* echo.

In the latter case, if the trap runs in a subshell, the main script needs to export VAR otherwise the subshell will find $VAR == "".

[stemsee]

Excellent intro to trap, thanks step. It would be appreciated if you implemented your suggestions.

At first I was using a function. But the behaviour of yad buttons and the syntax kept changing....

Code: Select all

--button="Stop:1"

Code: Select all

--button="Stop:bash -c 'endit'"

Code: Select all

--button="Stop":"bash -c 'endit'"

Code: Select all

--button="Stop" "bash -c 'endit'"

Code: Select all

--button="Stop" "echo 1"

Code: Select all

--button="Stop":"echo 1"

Code: Select all

--button="Stop!gtk-stop!Stop operation:1"

Can anyone offer guidance on the various behaviours??

[stemsee]

This version with some of step's optimisations. I left the exit script for now.