Basic Shell (Console) operation for beginners

[SimpleWater]

That is a good tip. I have been reading a lot tutorials, and i find warnings to be very helpful. I am sure there are many more ways that a script can cause corruption, it would be nice to see a list of recommendations, and also inflicted corruption possibilities.

Code: Select all

Puppy has its DNA. One of the things nice about having forum members
with lot of years on the forum, is, they remember.

A former significant contributor, GuestToo, contributed the script. The time
stamp says: Apr 27, 2005

I'm not sure on that date Bash was a default in Puppy. 

I see. Good to know

[RetroTechGuy]

As you can see, you have a blank page with the text "#!/bin/sh" as the first line. All shell scripts must begin with this line

Bruce B teaches with #!/bin/bash, care to elaborate a little more? Newbies need to know

Happy to. I'm here to serve, and learn.

The first line explicitly says what interpreter to use.

#!/bin/bash

This instructs Linux to use the bash interpreter in the /bin directory.

Let me expand a little further.

There is also nothing to prevent the user from installed a variety of shells, each with its own peculiar "language".

For example, my old scripts written under "tcsh" won't run under "bash", so rather than allow the computer to decide which shell to run under, I would specify "!/bin/tcsh/" to tell it to use that shell, instead of "bash". Of course, this assumes that tcsh has been installed (and has been installed in /bin/ instead of /usr/bin/ or whatever -- otherwise I would need to correct the path as well).

This is not dissimilar to specifying a compiler -- you wouldn't feed Pascal or Fortran code to the C compiler, and wouldn't feed C code to the Basic compiler (assuming one exists).

Script language is a language, just as are C, python, Fortran, Pascal, Basic, Forth, etc...

How would I install "tcsh", I suspect I could just install the primary package from here (I _may_ need other libraries, but would try it without first)

http://packages.debian.org/squeeze/tcsh

Or csh: http://packages.debian.org/squeeze/csh

Or ksh: http://packages.debian.org/squeeze/ksh

Or zsh: http://packages.debian.org/squeeze/zsh

Or...well, you get the idea...

Shells, on Wiki: http://en.wikipedia.org/wiki/Unix_shell

"Comparison of Shells": http://www.vias.org/linux-knowhow/lnag_05_05_02.html

You might find page following the link at the bottom of "Comparison" useful "Customizing the Shell Prompt" (you can do things like have the command prompt display the current working directory)

And there is a lot of information in the pages following...

[Shep]

Here is what I think is a fair and realistic batch programming challenge.

We have files named like this:

Mr. Kool - Can't dance (1970).mp3
Susan Crazy - Won't sing, will dance (LP-1962).mp3

I want a script to rename these type of files to names like below.
Meaning no spaces and characters bash needs escaped.

Mr_Kool_-_Cant_dance_1970.mp3
Susan_Crazy_-_Wont_sing_will_dance_LP-1962.mp3

Here's a late entry. I used the opportunity to brush up on sed

ls *.mp3 | sed "h; s/['"'()]//g; :a; s/[ \.]\(.*\.\)/_\1/; ta; x; G; s/\(.*\)\n/mv "\1" /' | sh

This generates the required shell commands, then pipes the stream directly into sh.

Explanation: program preserves one copy of the filename in the hold space, while editing the copy in the pattern space, finally it joins them and prepends with "mv".

s/[ \.]\(.*\.\)/_\1/ - replaces the left-most space or dot with an underscore
\1 - in the replacement represents what is matched by the pattern between \( and \)
:a ..... ta - execution loops through statements this pair encloses until there is only one dot remaining, this will be the dot in ".mp3"
h - pushes the pattern space onto the hold space
G - appends the hold space to the pattern space inserting a newline separator \n

As for accommodating malformed filenames, this won't accept a double-quote in a filename, and I wouldn't care to predict precisely what it will do with a filename containing a newline, or worse. There can be hazards associated with indiscriminently piping directly into sh. (Though double-quote handling is easy enough to arrange, if needed.) The only major problem I can foresee is that multiple filenames may map to a single rename, e.g., "file.1.mp3", "file 1.mp3", "file'1.mp3" all become file_1.mp3

I think best practice would be to generate just the pairs of filenames without the "mv" and pipe these into xargs -L 1 mv instead of sh.

[Bruce B]

Shep,

Will you please help with this one? It's purpose is to do a modest clean up on scripts.

(1) makes a backup of original file
(2) converts tabs to four spaces
(3) then removes trailing spaces

Code: Select all

mv $1 $1-BAK
< $1-BAK sed 's/[\t*]/    /g' | sed 's/[ *]$//' > $1

On line two, will you please show how to eliminate the re-piping to sed?
Also, improve it, if can be.

TIA

Bruce

~

[Bruce B]

I haven't tested this, but I think if profiles's SHELL= was set to SHELL=/bin/sh,
then Linux would think sh is the intepreter and not parse the bash configuration
files.

Which is not desirable for those wanting .bashrc read in.

~

[Shep]

. duplicate post deleted

[Shep]

(2) converts tabs to four spaces
(3) then removes trailing spaces

Code: Select all

mv $1 $1-BAK
< $1-BAK sed 's/\t*/    /g' | sed 's/ *$//' > $1

On line two, will you please show how to eliminate the re-piping to sed?

Sure, happy to.

You have it almost right. (I've changed it back to quote your first posting, as it was closer to being correct.)

The pattern \t* matches "zero or more occurrences" of the TAB. But if you think about it, you really need "one or more occurrences" of the TAB to be replaced by spaces. So change it to \t\t* which reads as "one TAB followed by zero or more TABs". The difference is hugely important, as you doubtless discovered, for one fundamental reason---in their wisdom the designers of sed declared that alongside every character in a stream is a match for "zero occurrences" of anything.

Similarly, to match your block of trailing spaces, specify "one space followed by zero or more extra spaces" by coding it as two spaces followed by an asterisk. While your original code for this step will provide the same result, it will subject the CPU to greater toil because sed finds a match alongside every character in the input stream and laboriously substitutes that nothing with ... a different nothing!

So the solution is:

sed -e 's/\t\t*/ /g' -e 's/ *$//'

To make sed code easier to write and understand, most implementations allow extensions to the basic commands. In addition to the * in pattern matching, we have \+ which matches "one or more occurrences of" the character that precedes it.

This means your code could also be expressed as:

sed -e 's/\t\+/ /g' -e 's/ \+$//'


HTH

[Bruce B]

Shep,

I made a script quick and dirty and it works perfect, thanks.

Code: Select all

<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//'
echo -n Ctrl+C to quit ; read a
<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//' > $2

Bruce

The cat -s removes extra empty lines > 2

I shoulda cat at the end and will.

The net result is a nicely cleaned file

~

PS the explanations were most helpful.

~

[Bruce B]

Shep,

One reason I'm weak on sed is because I was spoiled earlier by two
excellent line modification utilities for DOS and Windows

FU & LMOD - between the two they are better and easier than sed,
maybe combined they have more functionality.

When I came to Linux, rather than learn sed, I wrote various pipes in C.
The pipes not very flexible, yet work perfect. Due to lack of flexibility, I
have various pipes.

Then little by little learning sed. I understand Linux is a Unix clone, but it
wouldn't hurt to add a few novelties.

The author of FU I can't find. I'd like to get the source code.

The author of LMOD is still around and active.

Horst Schaeffer's Software Pages

By the powerful nature of LMOD and its very small size I'd think it is
written in assembly.

On the other hand Schaeffer has a Pure Basic section. Maybe that's how it
is coded.

I wonder if you know how hard or easy it would be to port DOS Pure Basic
or Assembly Code to Linux?

Most everything with LMOD is standard I/O, it seems.

Bruce

~

[Bruce B]

Programming Style

This script example is intended to be an actual practical script for making
pupsave backups.

The primary reason I may not have a current backup is because of
convenience factors. If I make backups very convenient or even
automatic, I will have current backups.

I often don't follow Linux conventions in scripting. Part this is C
conventions which I learned to like. In C, variables are conventionally
lower case and structure is function based. The way I write a program is
by making small functions. (It is easier for me and easier to test and
debug). Also, I like to use sufficient white space to make things
more readable.

The script below has been tested, I'll attach a zip file soon. I kept
everything to a character length of less than 80. To do this I used a \
which tells Bash to continue on to the next line as part of the command.

It is extremely important there be no white space following the \
otherwise, the script won't even work. Also, one reason for the scripts
posted above by me and wonderfully polished by Shep, is to remove
trailing white space.

Note the function names tend to serve as a comment giving a hint about
what the function is for.

Here is my function based script.

Code: Select all

#!/bin/bash

main() {

    variables
    sanity
    already_bked_up
    clear_pupsave
    make_bkup
    exit 0

}

variables() {

    datestamp=`date "+_%Y_%m_%d"`

    mode=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
    | cut -d = -f 2`

    pupdir=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
    | sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`

    pupfile=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
    | sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`

    script=`basename $0`

    xchck=`pidof  X`

    devsave="/initrd/mnt/dev_save"

    for i in `echo $pupfile | tr "." " "` ; do
        ext=$i
    done

    pupbasefn=`basename $pupfile .$ext`

}

sanity() {

    [ ! -d /initrd/pup_rw ] && echo "This is not a Puppy Frugal install, \
    goodbye" && exit

    [ "${#xchck}" -gt "1" ] && echo "X is running, goodbye" && exit

    if [ "$mode" != "12" ] ; then
        echo ; echo "$script has only been tested with Puppy's PUPMODE 12,"
        echo "due to lack of testing on other PUPMODES,this opeartion"
        echo "is terminated"
        echo ; echo "Recommendation,do your own testing and refining of"
        echo "$script for other PUPMODES" ; echo
        exit
    fi

}

already_bked_up() {

    if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

        a=N

        echo "You already have a backup for today"
        echo -n "Do you want another backup (y,n)? "
        read -n 1 a

        if [ "$a" = "y" ] ; then

            for i in {a..z} ; do

                if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ]
                then
                    datestamp="${datestamp}_$i"
                    break
                fi

            done

        else

            echo ;  exit 0
        
        fi

    fi

    echo

}

clear_pupsave() {

    echo "Clearing free space in $pupfile, please wait..."
    dd if=/dev/zero of=/initrd/pup_rw/zeros.tmp bs=1M
    rm /initrd/pup_rw/zeros.tmp

}

make_bkup() {

    echo "Making $devsave/$pupdir/$pupbasefn$datestamp.zip"
    zip $devsave/$pupdir/$pupbasefn$datestamp.zip $devsave/$pupdir/$pupfile
    ls -l $devsave/$pupdir/$pupbasefn$datestamp.zip
}

main

Here is the identical script without functions, with uppercase variables and
no extra white space.

Code: Select all

#!/bin/bash
DATESTAMP=`date "+_%Y_%m_%d"`
MODE=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
| cut -d = -f 2`
PUPDIR=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`
PUPFILE=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`
SCRIPT=`basename $0`
XCHCK=`pidof  X`
DEVSAVE="/initrd/mnt/dev_save"
for I in `echo $PUPFILE | tr "." " "` ; do
    EXT=$I
done
PUPBASEFN=`basename $PUPFILE .$EXT`
[ ! -d /initrd/pup_rw ] && echo "This is not a Puppy Frugal install, \
goodbye" && exit
[ "${#XCHCK}" -gt "1" ] && echo "X is running, goodbye" && exit
if [ "$MODE" != "12" ] ; then
    echo ; echo "$SCRIPT has only been tested with Puppy's PUPMODE 12,"
    echo "due to lack of testing on other PUPMODES,this opeartion"
    echo "is terminated"
    echo ; echo "Recommendation,do your own testing and refining of"
    echo "$SCRIPT for other PUPMODES" ; echo
    exit
fi
if [ -f $DEVSAVE/$PUPDIR/${PUPBASEFN}$DATESTAMP.zip ] ; then
A=N
echo "You already have a backup for today"
echo -n "Do you want another backup (y,n)? "
read -n 1 A
if [ "$A" = "y" ] ; then
    for I in {a..z} ; do
        if [ ! -f $DEVSAVE/$PUPDIR/${PUPBASEFN}${DATESTAMP}_$I.zip ] ; then
            DATESTAMP="${DATESTAMP}_$I"
            break
        fi
    done
else
    echo ; exit 0
fi
echo
echo "Clearing free space in $PUPFILE, please wait..."
dd if=/dev/zero of=/initrd/pup_rw/zeros.tmp bs=1M
rm /initrd/pup_rw/zeros.tmp
echo "Making $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip"
zip $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip $DEVSAVE/$PUPDIR/$PUPFILE
ls -l $DEVSAVE/$PUPDIR/$PUPBASEFN$DATESTAMP.zip
exit 0

These scripts are much easier to work with using a text editor with syntax
highlighting.

Bruce

Any comments or questions?

~

[Bruce B]

Indenting

Indenting when we use what is most commonly called statements.

Below are two examples from the post above.

Example Indented

Code: Select all

if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

    a=N

    echo "You already have a backup for today"
    echo -n "Do you want another backup (y,n)? "
    read -n 1 a

    if [ "$a" = "y" ] ; then

        for i in {a..z} ; do

            if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ] ; then
             
                datestamp="${datestamp}_$i"
                break

            fi

         done

    else

        echo ;  exit 0
       
    fi

fi

The commands are indented inside each controlling statement.

When statements are nested, each statement is indented.

In order to follow what is going on, we look at the start of the statement.
Then look straight down the column for the end instruction of the
particular statement. It could be 'done' on a 'for loop', or 'esac' for a 'case'
statement or 'fi' for an 'if' statement.

~~~

Example Not Indented (harder to follow)

Code: Select all

if [ -f $devsave/$pupdir/${pupbasefn}$datestamp.zip ] ; then

a=N

echo "You already have a backup for today"
echo -n "Do you want another backup (y,n)? "
read -n 1 a

if [ "$a" = "y" ] ; then

for i in {a..z} ; do

if [ ! -f $devsave/$pupdir/${pupbasefn}${datestamp}_$i.zip ] ; then
             
datestamp="${datestamp}_$i"
break

fi

done

else

echo ;  exit 0
       
fi

fi

~

[Shep]

Code: Select all

<$1 cat -s | sed -e 's/\t\+/ /g' -e 's/ \+$//'

The cat -s removes extra empty lines > 2

I shoulda cat at the end and will.

The sed script here does not create or delete any line/s, so with your cat -s filter at the input, there will be no need for an identical cat -s filter at the output. It would have nothing to do.

Code: Select all

echo -n Ctrl+C to quit ; read a

Small point, but you can combine the prompt and the read:

read -p 'Ctrl+C to quit, or <RET> to continue:' -n 1 a

Though neater to test the response and not rely on interrupt:

read -p 'Enter q to quit, or <RET> to continue:' -n 1 a

[Shep]

FU & LMOD - between the two they are better and easier than sed,
maybe combined they have more functionality.

Sorry, I can't help with either. I doubt that if they were easier to learn, they could be more powerful than sed. The power of sed comes, in part, from its extended regular expression handling. Yes, it does take practise to master some of the more esoteric commands in sed, but you can still achieve much just with its common commands.

But even if you were to find a non-standard utility, the readers with a standard install would not be able to benefit from all the work you put into designing scripts with it!

[SimpleWater]

"Customizing the Shell Prompt"

that sounds really intresting

[Bruce B]

The sed script here does not create or delete any line/s, so with
your cat -s filter at the input, there will be no need for an identical cat -s filter
at the output. It would have nothing to do.

True, but my after thought was, if there was any trailing white space in a
line, maybe cat wouldn't consider it an empty line. In order to qualify as an
empty line the only character would be \n (0x0a), the actual behavior would
depend on how the cat authors think. And as usual, man pages don't say
everything.

As for leading white space, the script doesn't deal with that, because I want
to preserve it.

I could test, that's for sure.

And many thanks for all the helpful information, I would never have intuited it.

~

[Shep]

mode=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPMODE \
| cut -d = -f 2`

pupdir=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 2`

pupfile=`cat /initrd/pup_rw/etc/rc.d/PUPSTATE | grep PUPSAVE \
| sed "s/'//g" | tr "," " " | cut -d " " -f 3 | cut -d "/" -f 3`

Bruce, you have put a lot of thought into this script. As far as I'm concerned, if the script works and is understandable enough to be easily modified, then it's fine.

There are one or two things I can point out. The code cat file | grep string can almost always be written as grep string file thereby saving one process and a pipe.

On big linux installs, tr is a small fast binary and tr -d deletes characters from a stream more effortlessly than does sed.

In the above code snippet, it would be better to save an intermediate processed value from the first `...` and just put it through one more cut to extract the value for the second and third parameters. You three times cat the same file and pass it through almost identical cut filters when there is almost certainly no need to do it more than once.

However, the block of code above is a perfect case where builtin shell commands can do the same job with a fraction of the effort. In particular, for string manipulation consider Shell Parameter Expansion techniques. ] EDIT: oops! While I provided that reference, and strongly recommend it, I actually didn't use that method for this task. In the end it was easy to do solely using bash's extended set command.

a=`grep PUPSAVE /initrd/pup_rw/etc/rc.d/PUPSTATE`
echo $a
IFS="[ ,/]"
set -- $a
mode=$2
pupdir=$3
pupfile=$5
echo $mode $pupdir $pupfile

IFS is the field separator (can be a single char or a group of chars) used to split the string. One thing to watch if you employ the set command is to make sure you have beforehand saved the positional parameters $1 $2 etc if they passed values (e.g., filenames) into the script when it was called. Any initial values are overwritten when you use set to reassign shell positional parameters $1 $2 etc. within your program as I illustrate above. As a technique to be applied to other puppies, this will only work if you can be sure the contents of file PUPSTATE will adhere to identical syntax for all puplets. Can you be sure of this? Otherwise, a more robust algorithm will be needed.

EDIT: mulling over bash extensions, I believe a one-liner is all that is needed for the job:

IFS="[ ,/']"; read x mode pupdir pupfile <<< `grep PUPSAVE /etc/rc.d/PUPSTATE`
echo $mode $pupdir $pupfile

To variable x I've assigned the field we aren't interested in when the line containing PUPSAVE is read in. The three arrows is a way to feed an input string to a command which expects it. IFS breaks the line into fields according to the 3 chars in the square brackets.

This discussion may be of interest to others who are following along with your explorations here.

HTH

[Bruce B]

My standard prompt is what I call the OS2 prompt. Because that's how OS2
did it.

PS1='[\w] ' gives [~/bin]

If the path gets too long I have an alias to change it to PS1='[\W] '

It shows not the full path, just the current directory, like this:

[bin]

But don't stop there, because the sky is the limit, when it comes to custom
prompts.

~

[Bruce B]

Shep,

That's why I didn't attach it. It works fine. But it need more work, it's not really
ready to deliver. It needs review and refinement. Thanks for the input. I'll study
it.

I also want to add a function, which I think I'll let the user fill in. The purpose of
the function is to delete browser cache. There is no reason to back that stuff
up.

I could find .mozilla SeaMonkey and Firefox cache with a script, but these days
people are using browsers, I'm not going to even bother with learning.

Bruce

[Bruce B]

read -p 'Enter q to quit, or <RET> to continue:' -n 1 a

Maybe have I ran into a sequence or quote type problem. I'll try it your way.

Code: Select all

read n1 a -p "Enter choice "

I think something didn't do right, so I decided to echo -n

~

[Shep]

...but my after thought was, if there was any trailing white space in a line, maybe cat wouldn't consider it an empty line.

A perfectly logical precaution, too. However, there will be no trailing spaces because you have given sed the code to delete all trailing spaces. And if you modify it to delete trailing TABS, can there be any other trailing whitespace characters?

Just modify 's/ \+$//' to become 's/[ \t]\+$//'