Basic Shell (Console) operation for beginners

[Shep]

. duplicate post deleted

[Shep]

(2) converts tabs to four spaces
(3) then removes trailing spaces

Code: Select all

mv $1 $1-BAK
< $1-BAK sed 's/\t*/    /g' | sed 's/ *$//' > $1

On line two, will you please show how to eliminate the re-piping to sed?

Sure, happy to.

You have it almost right. (I've changed it back to quote your first posting, as it was closer to being correct.)

The pattern \t* matches "zero or more occurrences" of the TAB. But if you think about it, you really need "one or more occurrences" of the TAB to be replaced by spaces. So change it to \t\t* which reads as "one TAB followed by zero or more TABs". The difference is hugely important, as you doubtless discovered, for one fundamental reason---in their wisdom the designers of sed declared that alongside every character in a stream is a match for "zero occurrences" of anything.

Similarly, to match your block of trailing spaces, specify "one space followed by zero or more extra spaces" by coding it as two spaces followed by an asterisk. While your original code for this step will provide the same result, it will subject the CPU to greater toil because sed finds a match alongside every character in the input stream and laboriously substitutes that nothing with ... a different nothing!

So the solution is:

sed -e 's/\t\t*/ /g' -e 's/ *$//'

To make sed code easier to write and understand, most implementations allow extensions to the basic commands. In addition to the * in pattern matching, we have \+ which matches "one or more occurrences of" the character that precedes it.

This means your code could also be expressed as:

sed -e 's/\t\+/ /g' -e 's/ \+$//'


HTH

[Bruce B]

Shep,

I made a script quick and dirty and it works perfect, thanks.

Code: Select all

<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//'
echo -n Ctrl+C to quit ; read a
<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//' > $2

Bruce

The cat -s removes extra empty lines > 2

I shoulda cat at the end and will.

The net result is a nicely cleaned file

~

PS the explanations were most helpful.

~

[Bruce B]

Shep,

One reason I'm weak on sed is because I was spoiled earlier by two
excellent line modification utilities for DOS and Windows

FU & LMOD - between the two they are better and easier than sed,
maybe combined they have more functionality.

When I came to Linux, rather than learn sed, I wrote various pipes in C.
The pipes not very flexible, yet work perfect. Due to lack of flexibility, I
have various pipes.

Then little by little learning sed. I understand Linux is a Unix clone, but it
wouldn't hurt to add a few novelties.

The author of FU I can't find. I'd like to get the source code.

The author of LMOD is still around and active.

Horst Schaeffer's Software Pages

By the powerful nature of LMOD and its very small size I'd think it is
written in assembly.

On the other hand Schaeffer has a Pure Basic section. Maybe that's how it
is coded.

I wonder if you know how hard or easy it would be to port DOS Pure Basic
or Assembly Code to Linux?

Most everything with LMOD is standard I/O, it seems.

Bruce

~

[Bruce B]

Programming Style

This script example is intended to be an actual practical script for making
pupsave backups.

The primary reason I may not have a current backup is because of
convenience factors. If I make backups very convenient or even
automatic, I will have current backups.

I often don't follow Linux conventions in scripting. Part this is C
conventions which I learned to like. In C, variables are conventionally
lower case and structure is function based. The way I write a program is
by making small functions. (It is easier for me and easier to test and
debug). Also, I like to use sufficient white space to make things
more readable.

The script below has been tested, I'll attach a zip file soon. I kept
everything to a character length of less than 80. To do this I used a \
which tells Bash to continue on to the next line as part of the command.

It is extremely important there be no white space following the \
otherwise, the script won't even work. Also, one reason for the scripts
posted above by me and wonderfully polished by Shep, is to remove
trailing white space.

Note the function names tend to serve as a comment giving a hint about
what the function is for.

Here is my function based script.

Code: Select all

#!/bin/bash

main() {

    variables
    sanity
    already_bked_up
    clear_pupsave
    make_bkup
    exit 0

}

variables() {

    datestamp=`date "+_%Y_%m_%d"`

    mode=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
    | cut -d = -f 2`

    pupdir=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
    | sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`

    pupfile=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
    | sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`

    script=`basename $0`

    xchck=`pidof  X`

    devsave="/initrd/mnt/dev_save"

    for i in `echo $pupfile | tr "." " "` ; do
        ext=$i
    done

    pupbasefn=`basename $pupfile .$ext`

}

sanity() {

    [ ! -d /initrd/pup_rw ] && echo "This is not a Puppy Frugal install, \
    goodbye" && exit

    [ "${#xchck}" -gt "1" ] && echo "X is running, goodbye" && exit

    if [ "$mode" != "12" ] ; then
        echo ; echo "$script has only been tested with Puppy's PUPMODE 12,"
        echo "due to lack of testing on other PUPMODES,this opeartion"
        echo "is terminated"
        echo ; echo "Recommendation,do your own testing and refining of"
        echo "$script for other PUPMODES" ; echo
        exit
    fi

}

already_bked_up() {

    if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

        a=N

        echo "You already have a backup for today"
        echo -n "Do you want another backup (y,n)? "
        read -n 1 a

        if [ "$a" = "y" ] ; then

            for i in {a..z} ; do

                if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ]
                then
                    datestamp="${datestamp}_$i"
                    break
                fi

            done

        else

            echo ;  exit 0
        
        fi

    fi

    echo

}

clear_pupsave() {

    echo "Clearing free space in $pupfile, please wait..."
    dd if=/dev/zero of=/initrd/pup_rw/zeros.tmp bs=1M
    rm /initrd/pup_rw/zeros.tmp

}

make_bkup() {

    echo "Making $devsave/$pupdir/$pupbasefn$datestamp.zip"
    zip $devsave/$pupdir/$pupbasefn$datestamp.zip $devsave/$pupdir/$pupfile
    ls -l $devsave/$pupdir/$pupbasefn$datestamp.zip
}

main

Here is the identical script without functions, with uppercase variables and
no extra white space.

Code: Select all

#!/bin/bash
DATESTAMP=`date "+_%Y_%m_%d"`
MODE=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
| cut -d = -f 2`
PUPDIR=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`
PUPFILE=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`
SCRIPT=`basename $0`
XCHCK=`pidof  X`
DEVSAVE="/initrd/mnt/dev_save"
for I in `echo $PUPFILE | tr "." " "` ; do
    EXT=$I
done
PUPBASEFN=`basename $PUPFILE .$EXT`
[ ! -d /initrd/pup_rw ] && echo "This is not a Puppy Frugal install, \
goodbye" && exit
[ "${#XCHCK}" -gt "1" ] && echo "X is running, goodbye" && exit
if [ "$MODE" != "12" ] ; then
    echo ; echo "$SCRIPT has only been tested with Puppy's PUPMODE 12,"
    echo "due to lack of testing on other PUPMODES,this opeartion"
    echo "is terminated"
    echo ; echo "Recommendation,do your own testing and refining of"
    echo "$SCRIPT for other PUPMODES" ; echo
    exit
fi
if [ -f $DEVSAVE/$PUPDIR/${PUPBASEFN}$DATESTAMP.zip ] ; then
A=N
echo "You already have a backup for today"
echo -n "Do you want another backup (y,n)? "
read -n 1 A
if [ "$A" = "y" ] ; then
    for I in {a..z} ; do
        if [ ! -f $DEVSAVE/$PUPDIR/${PUPBASEFN}${DATESTAMP}_$I.zip ] ; then
            DATESTAMP="${DATESTAMP}_$I"
            break
        fi
    done
else
    echo ; exit 0
fi
echo
echo "Clearing free space in $PUPFILE, please wait..."
dd if=/dev/zero of=/initrd/pup_rw/zeros.tmp bs=1M
rm /initrd/pup_rw/zeros.tmp
echo "Making $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip"
zip $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip $DEVSAVE/$PUPDIR/$PUPFILE
ls -l $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip
exit 0

These scripts are much easier to work with using a text editor with syntax
highlighting.

Bruce

Any comments or questions?

~

[Bruce B]

Indenting

Indenting when we use what is most commonly called statements.

Below are two examples from the post above.

Example Indented

Code: Select all

if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

    a=N

    echo "You already have a backup for today"
    echo -n "Do you want another backup (y,n)? "
    read -n 1 a

    if [ "$a" = "y" ] ; then

        for i in {a..z} ; do

            if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ] ; then
             
                datestamp="${datestamp}_$i"
                break

            fi

         done

    else

        echo ;  exit 0
       
    fi

fi

The commands are indented inside each controlling statement.

When statements are nested, each statement is indented.

In order to follow what is going on, we look at the start of the statement.
Then look straight down the column for the end instruction of the
particular statement. It could be 'done' on a 'for loop', or 'esac' for a 'case'
statement or 'fi' for an 'if' statement.

~~~

Example Not Indented (harder to follow)

Code: Select all

if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

a=N

echo "You already have a backup for today"
echo -n "Do you want another backup (y,n)? "
read -n 1 a

if [ "$a" = "y" ] ; then

for i in {a..z} ; do

if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ] ; then
             
datestamp="${datestamp}_$i"
break

fi

done

else

echo ;  exit 0
       
fi

fi

~

[Shep]

Code: Select all

<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//'

The cat -s removes extra empty lines > 2

I shoulda cat at the end and will.

The sed script here does not create or delete any line/s, so with your cat -s filter at the input, there will be no need for an identical cat -s filter at the output. It would have nothing to do.

Code: Select all

echo -n Ctrl+C to quit ; read a

Small point, but you can combine the prompt and the read:

read -p 'Ctrl+C to quit, or <RET> to continue:' -n 1 a

Though neater to test the response and not rely on interrupt:

read -p 'Enter q to quit, or <RET> to continue:' -n 1 a

[Shep]

FU & LMOD - between the two they are better and easier than sed,
maybe combined they have more functionality.

Sorry, I can't help with either. I doubt that if they were easier to learn, they could be more powerful than sed. The power of sed comes, in part, from its extended regular expression handling. Yes, it does take practise to master some of the more esoteric commands in sed, but you can still achieve much just with its common commands.

But even if you were to find a non-standard utility, the readers with a standard install would not be able to benefit from all the work you put into designing scripts with it!

[SimpleWater]

"Customizing the Shell Prompt"

that sounds really intresting

[Bruce B]

The sed script here does not create or delete any line/s, so with
your cat -s filter at the input, there will be no need for an identical cat -s filter
at the output. It would have nothing to do.

True, but my after thought was, if there was any trailing white space in a
line, maybe cat wouldn't consider it an empty line. In order to qualify as an
empty line the only character would be \n (0x0a), the actual behavior would
depend on how the cat authors think. And as usual, man pages don't say
everything.

As for leading white space, the script doesn't deal with that, because I want
to preserve it.

I could test, that's for sure.

And many thanks for all the helpful information, I would never have intuited it.

~

[Shep]

mode=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
| cut -d = -f 2`

pupdir=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`

pupfile=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`

Bruce, you have put a lot of thought into this script. As far as I'm concerned, if the script works and is understandable enough to be easily modified, then it's fine.

There are one or two things I can point out. The code cat file | grep string can almost always be written as grep string file thereby saving one process and a pipe.

On big linux installs, tr is a small fast binary and tr -d deletes characters from a stream more effortlessly than does sed.

In the above code snippet, it would be better to save an intermediate processed value from the first `...` and just put it through one more cut to extract the value for the second and third parameters. You three times cat the same file and pass it through almost identical cut filters when there is almost certainly no need to do it more than once.

However, the block of code above is a perfect case where builtin shell commands can do the same job with a fraction of the effort. In particular, for string manipulation consider Shell Parameter Expansion techniques. ] EDIT: oops! While I provided that reference, and strongly recommend it, I actually didn't use that method for this task. In the end it was easy to do solely using bash's extended set command.

a=`grep PUPSAVE /initrd/pup_rw/etc/rc.d/PUPSTATE`
echo $a
IFS="[ ,/]"
set -- $a
mode=$2
pupdir=$3
pupfile=$5
echo $mode $pupdir $pupfile

IFS is the field separator (can be a single char or a group of chars) used to split the string. One thing to watch if you employ the set command is to make sure you have beforehand saved the positional parameters $1 $2 etc if they passed values (e.g., filenames) into the script when it was called. Any initial values are overwritten when you use set to reassign shell positional parameters $1 $2 etc. within your program as I illustrate above. As a technique to be applied to other puppies, this will only work if you can be sure the contents of file PUPSTATE will adhere to identical syntax for all puplets. Can you be sure of this? Otherwise, a more robust algorithm will be needed.

EDIT: mulling over bash extensions, I believe a one-liner is all that is needed for the job:

IFS="[ ,/']"; read x mode pupdir pupfile <<< `grep PUPSAVE /etc/rc.d/PUPSTATE`
echo $mode $pupdir $pupfile

To variable x I've assigned the field we aren't interested in when the line containing PUPSAVE is read in. The three arrows is a way to feed an input string to a command which expects it. IFS breaks the line into fields according to the 3 chars in the square brackets.

This discussion may be of interest to others who are following along with your explorations here.

HTH

[Bruce B]

My standard prompt is what I call the OS2 prompt. Because that's how OS2
did it.

PS1='[\w] ' gives [~/bin]

If the path gets too long I have an alias to change it to PS1='[\W] '

It shows not the full path, just the current directory, like this:

[bin]

But don't stop there, because the sky is the limit, when it comes to custom
prompts.

~

[Bruce B]

Shep,

That's why I didn't attach it. It works fine. But it need more work, it's not really
ready to deliver. It needs review and refinement. Thanks for the input. I'll study
it.

I also want to add a function, which I think I'll let the user fill in. The purpose of
the function is to delete browser cache. There is no reason to back that stuff
up.

I could find .mozilla SeaMonkey and Firefox cache with a script, but these days
people are using browsers, I'm not going to even bother with learning.

Bruce

[Bruce B]

read -p 'Enter q to quit, or <RET> to continue:' -n 1 a

Maybe have I ran into a sequence or quote type problem. I'll try it your way.

Code: Select all

read n1 a -p "Enter choice "

I think something didn't do right, so I decided to echo -n

~

[Shep]

...but my after thought was, if there was any trailing white space in a line, maybe cat wouldn't consider it an empty line.

A perfectly logical precaution, too. However, there will be no trailing spaces because you have given sed the code to delete all trailing spaces. And if you modify it to delete trailing TABS, can there be any other trailing whitespace characters?

Just modify 's/ \+$//' to become 's/[ \t]\+$//'

[Shep]

Code: Select all

read n1 a -p "Enter choice "

I think something didn't do right, so I decided to echo -n

I'd say that n1 could be the problem. I reckon -n 1 might work.

[Bruce B]

As a technique to be applied to other puppies, this will only work if you can
be sure the contents of file PUPSTATE will adhere to identical syntax for all
puplets. Can you be sure of this? Otherwise, a more robust algorithm will be
needed

Shep,

After about seven years working with Puppy, I can be confident in saying,
'Barry changes things.' I'm certain I don't know what he will do in the future.

In this case, I might, (probably will), force an exit, with a printed comment,
for all versions I haven't personally tested.

It would also be a good exercise for this audience to test and rework existing
code, as needed.

Thanks,

Bruce

~

[Bruce B]

Boot sector backups

Sometimes we have data damaging problems. I'd like the reader to read the
linked article below, from our regular users section.

Download the script and run it.

If anything ever happens to a boot sector, especially a Microsoft boot sector,
you can be good to go in a matter of a few minutes. Boot sectors can be
damaged by the operator, malware and maybe even some shareware trying to
make sure you don't reinstall.

Can't mount NTFS partition in Puppy 5.1

~

[Shep]

After about seven years working with Puppy, I can be confident in saying,
'Barry changes things.' I'm certain I don't know what he will do in the future.

It would also be a good exercise for this audience to test and rework existing code, as needed.

I have further polished the code I suggested for setting pupdir & pupfile a few posts back. Scroll back to see. It's now down to one line, so I probably can't reduce it much further.

Still pondering a one-line method for incrementing the suffix on a filename, to increment from file_a to file_b. Will post if/when I solve it.

[Bruce B]

Numbering Lines

I decided to try posting scripts with line numbers for easy reference.
It is entirely feasible the reader will not understand some aspects of
of a script I post. By referencing line numbers I can explain easily,
things I think should be explained.

-) This is a script to number lines.

4) The variables don't know about the command line parameters.
$@ is how I pass the parameters to variables. Commonly you will see
"$@" as the way to do it. In this case, I'm only passing a filename. All
my file names follow Linux conventions, so there is no need to quote.

4-8) Are function calls from main. Somewhat like a goto command, but
goto is not necessary. The name of the function tells Bash to go to that
function and execute the commands within the function, then return to main
and execute the next line if there is one.

15-16) I exit with a error code of 1, if the tests are true.

22) grep can count lines by using the -c switch. What all lines have in
common is a beginning of the line. The ^ says to grep the beginning of
the line. This combination makes counting lines easy. The variable will be
used later in the script.

26) function get_format. I want to format my lines right justified.
I set a variable to help me do that, when later, I use the printf command.

By formatting the line numbering this way, it makes the numbered file
easier to read.

41) Will make a numbered file to be used later.

46) The printf command prints formatted and does many other things. Printf
probably means 'print formatted'

53) Introducing the external 'paste' command. It pastes two files side by
side, making the output joined columns. You can study the command in the
script. It uses the most basic command of paste.

58) Strips the path from my script and sets a variable called script,
which contains only the name of the script. I don't know how to pass $0 to
a function, so I did it outside a function. The variable called 'script'
can be used inside any function.

59) Main is the command to go to the function main and start executing the
commands inside the function.

The program flow in Bash is: execute from top to bottom. The same flow
exists when we use functions. But we can give the appearance of a
different flow with functions. And even gain flow control. But the execution
Bash uses is still top to bottom.

A function is not a command, Bash reads the file from top to bottom and
doesn't do anything until it runs into a command. The command 'main' tells
Bash to go to the function main. Main calls functions. When the last line in a
function is complete, Bash returns to execute the next command from where
the function was called.

Main has the command 'exit'. When we use the command exit anywhere, Bash
does just that. This means that if there were a command after the main
command, it would be of null effect, because the function main exited the
script after its commands were finished.

Code: Select all

 0  #!/bin/bash
 1
 2  main() {
 3
 4      sanity $@
 5      count_infile_lines $@
 6      get_format $@
 7      make_numbers
 8      paste_files $@
 9      exit 0
10
11  }
12
13  sanity() {
14
15      [ ! $1 ] && echo "Enter filename to number" && exit 1
16      [ ! -f $1 ] && echo "File $2 doesn't exist" && exit 1
17
18  }
19
20  count_infile_lines() {
21
22      lines=`grep -c ^ $1`
23
24  }
25
26  get_format() {
27
28      if [ "$lines" -le "9" ] ; then
29          format=1
30      elif [ "$lines" -le "99" ] ; then
31          format=2
32      elif [ "$lines" -le "999" ] ; then
33          format=3
34      else
35          echo "Error, exiting"
36          exit 1
37      fi
38
39  }
40
41  make_numbers() {
42
43      [ -f /tmp/${script}.tmp ] && rm /tmp/${script}.tmp
44
45      for ((i;i<=${lines};i++)) ; do
46          printf "%${format}d \n" $i >> /tmp/${script}.tmp
47      done
48
49  }
50
51  paste_files() {
52
53      paste /tmp/${script}.tmp $1 > $1.nbr
54      echo "Numbered file saved as \"$1.nbr\""
55
56  }
57
58  script=`basename $0`
59  main $@
60

~