The Text-User Interface

Using applications, configuring, problems
Message
Author
Bruce B

The Text-User Interface

#1 Post by Bruce B »

Introduction

Notice - I removed the new version of bash. I had some odd problems
with it running Midnight Commander and never resolved the issue, and I
tried. I'll leave the post intact minus the attachment. Sorry.

Some users are interested in learning Text User Interface (TUI). I enjoy learning, teaching and helping. I particularly enjoy the TUI and all that can be done with it. By knowing it well you can add a lot of extensibility to your Linux system.

The emulator and shell

If a person opens up rxvt, Puppy's default terminal emulator, you will be working with the bash shell. You will see Puppy's default prompt, the #.

The need to learn

There is nothing and I mean nothing intuitive about the bash shell with a # sign. For this reason, the TUI has to be learned. Once a certain undefinable point in learning curve has been achieved, the user has enough know how and confidence to take it from there and work on his own.

The curriculum

I'll try and present easy chapters with a balance of theory and practical application. If you're like me, you'll be tempted to not follow instructions to the letter, to personalize things. I ask you not to do this under certain circumstances. I'll explain why.

With computers there are various ways to accomplish the same end. For the purpose of these chapters, I'll state which is mandatory. Items will be mandatory on the basis that I expect to use them later, and in my future writing I expect things to be in certain places.

The conclusion

When the chapters reach a logical conclusion, I'll conclude. At the conclusion, you'll be able to put everything back just how it was if you want. You then will also be able to set things up in very customized fashion.

Optional Exercise

This is strictly optional. The idea is to update bash. Attached to this post is:

GNU bash, version 4.0.10(1)-release (i686-pc-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>


I've used this bash for a while now with Puppy version 4.00 and it works swell.

Let's consider a few things. (1) if this version is not newer than your existing version, it won't be an update. (2) if it doesn't work right, you will want to restore your old version.

Basically, what I really want to do is have you type in some commands and learn some things, and make a modification to your system.

Open your terminal emulator - rxvt

Find out your bash version

bash --version

Check if it's lower than 4.0.10

Where is bash? The which command will tell you.

which bash

Download the attached file to that directory

cd to that directory, if /bin then

cd /bin

Check if bash and the download is there with the ls command

ls bash*

If you don't see bash and bash.bz2, STOP something isn't right

Make a backup copy of bash

cp bash bash.bak

We are using bash to replace bash, but the bash we are using should be in memory.

Unpack bash.bz2

bzip2 -d -f bash.bz2

-d for decompress -f for force overwrite

Close rxvt

exit

Open rxvt and check the bash version

-----------------

Edit attachment temporarily removed for changes in instructions.
Last edited by Bruce B on Mon 10 Jan 2011, 16:34, edited 4 times in total.

Bruce B

profile.local

#2 Post by Bruce B »

profile.local

The easy life - hopefully soon you will see the command line can be an easy way to do things, even mundane tasks.

Here is one example:

rox /usr/bin

If you open rxvt and type the above command, you should note everything happened much faster and easier, than if you clicked to open ROX-Filer and navigated up and down to /usr/bin.

In this chapter we are going to work on a file called 'profile.local'. The main purpose is preparing to make our life on the TUI easier. The file itself probably doesn't exist, but it will by the end of this chapter.

This will be the contents:

export PATH="$PATH:/root/bin"
export PS1="[\w] "
alias cp='cp --interactive'
alias mv='mv --interactive'
alias rm='rm --interactive'
alias bin='cd /root/bin'
alias path='echo $PATH'


Explained somewhat

PATH (uppercase, always upper case) is a list of all the directories Linux will use to search for executable files. I included an alias called 'path', which you can use to see the PATH displayed on screen. Linux will look for executable files in all these directories and only these directories. The search is left to right, the first match is the one Linux uses.

/root/bin will become our working directory. We will make our scripts and other files in this directory. Doing it this way, our scripts and other related files will never be mixed in with files in other directories.

If you make a script, you give it a name, good practice is make sure the name doesn't already exist in the PATH statement. Simply type in the intended name before assigning it to a script.

Never make a script called 'test' because test is an actual command.

Also note we put /root/bin at the end of the search chain. Later, if you are crazy like me you can put it first, but not in the course of this study.

PS1 is how we define main prompt. In this case, it will show your full working directory. Having the prompt display your full working directory is more meaningful than the # sign. Also, there are entire web pages about various ways to display the prompt. Although this exercise is mandatory - change the PS1 as you please

cp, mv, rm - these are the commands to copy, move and remove files. The default Puppy doesn't make them an "Are you sure?" They obey at the enter key.

The aliases above have redefined these commands to get your permission on certain command line actions. ( we added a safety factor )

The alias 'bin' makes it easy to change directories to our work directory /root/bin

The mandatory assignment

1) make a text file /etc/profile.local - copy and paste the commands in darkred from above to that file, and of course save the changes.

The contents of this file will probably show effect the next time you open rxvt.

2) make a directory /root/bin, this might be tricky. Historically there never was such a directory in Puppy. The command below would be all you need:

mkdir /root/bin

I noticed in newer Puppy versions we have a symlink like this

/root/bin -> /root/my-applications/bin

I trust you can solve the riddle and make the directory with the tools at hand.

~

Chapter 1 - profile.local

Bruce B

#3 Post by Bruce B »

The Source Command

In Chapter 1, we made a file - profile.local

* it is not a shell script - no #/bin/sh
* it is not in the PATH
* it doesn't have an executable bit set, but somehow;
* it is a text file that runs commands

Absent all the qualities we normally associate with an executable file, every command is executed. I'll explain.

The source command is what executes it. Also, in Linux we often have short commands and long commands to do the same thing.

Two examples:

source is the long command : . is the short command
test is the long command : [ is the short command

How /etc/profile.local works in Puppy

Considering that /etc/profile might change with Puppy versions, I otherwise expect you will find a line like the one below, which I will explain.

[ -r /etc/profile.local ] && . /etc/profile.local

There it is - our shorthand, I'll show what it would look like in long hand

test -r /etc/profile.local && source /etc/profile.local

English is the harder of the languages. In English it is saying

[ ( means test )
-r ( defines our test - if the file exists and is readable )
/etc/profile.local ( the name of the test file )
&& ( means if test is true, do next command otherwise do nothing )

Before we installed /etc/profile.local, the test would have been false and nothing would have happened. After making the file, the outcome of the test is true and profile.local gets sourced, meaning all the commands it contains get executed, using this command:

. /etc/profile.local

Shells, shells and more shells

The human eye will not discern this, but when we run a bash script, the shell we are looking at and possibly interacting with is not the same shell that ran the script.

Perhaps well to conceptualize a parent shell spawning a child shell. The parent shell passes most of its environment to the child shell. The child, however does not return with the environment it set. ( as a rule )

Here is an example which should be easy to follow:

The parent shell has this variable EDITOR=mp

The child sets EDITOR=nano

nano will be the value for the variable EDITOR during the execution of the script, but when the script finishes, the value of EDITOR is mp, as it was before the script started.

Sourcing, as opposed to running a shell script is more the equivalent of typing commands into the existing shell. A child process is not spawned when sourcing.

When we sourced /etc/profile.local, we changed the environment in the same shell which sourced it. And that shell was before X ran, and before the terminal emulator hosted another shell.

The environment get passed up fairly easily.

For a related exercise for this chapter, open rxvt and run this command:

env

It will show a lot of your Linux environment. It is also a command which can be helpful in writing some scripts and troubleshooting.

~

Chapter 2 - The Source Command

Bruce B

#4 Post by Bruce B »

Enhancing our TUI

This chapter is for the purpose of improving your Text User Interface, making it an easier, more pleasant and powerful working environment.

We will install two applications from Puppy's official repository

Add the terminal emulator mrxvt

mrxvt-0.5.2.pet

The reason why is, because its better than rxvt, in a number of areas. Of particular importance to me is the handling of F keys and the eol (end of line) and bol (beginning of line).

Puppy includes a great TUI text editor 'mp' ( Minimum Profit ), but rxvt doesn't handle the bol and eol as it should for some reason, thus making working with mp quite difficult.

Add Midnight Commander

mc-4.6.1.pet
If mc doesn't run - maybe you need gpm-1.20.1.pet

Run Midnight Commander by typing mc in terminal emulator. It is a very powerful, extensible and time honored file manager for the TUI.

Link: ftp.ibiblio.org/pub/linux/distributions/puppylinux

Look for these packages in pet_packages 3 and 4

Please consider this an essential assignment.

~

Chapter 3 - Enhancing our TUI

Bruce B

#5 Post by Bruce B »

First script

We will make a script for making scripts.

Making a script might take two or three steps. If we can do it in one step it's easier.

This is the top line of our script. It tells the Linux that it's a Bourne shell script text executable

#!/bin/bash

We want to make sure the user enters an argument, so we make a test. Testing was already touched on in Chapter 2. Remember '[' is shorthand for test. We signal the end of the test with ]

The explanation point ! means NOT

$1 is the first argument.

[ ! $1 ] && echo "usage: newsh <filename> && exit

Explained in English: If there is no argument, show the user how to use our utility and exit.

We are using two &&, this means that two conditions must be true in order for our line to fully execute. If the first && is false nothing at all happens.

Using && moves left to right and if any conditions are not true the remaining && commands do run.

In order for an execution, it must be true - there is no argument.

#!/bin/bash
[ ! $1 ] && echo "usage: newsh <filename>" && exit


------------------------

Bash executes in a top to bottom manner. If we get past the conditional test in line two, we can execute other commands.

Here I introduce an if/else statement combined with a test. If the 'text' in our argument $1 matches an existing file, we will do one thing, otherwise we will do something else.

a) if [ -f $1 ] ; then
b) leafpad $1 &
c) else
d) echo "#!/bin/bash">$1
e) chmod 755 $1
f) leafpad $1 &
g) fi

Explained

a) if argument $1 is -f (a existing file) ; then

b) open it with leafpad in the background. a single & means background. the practical effect of using the single & is our command prompt will return, without the & our command prompt hangs until leafpad is closed

c) means means otherwise (else) do the next commands. (d),(e) and (f) only get executed if (a) is false, which means there is no file matching the name in $1

d) is the echo command, meaning display the text entered to standard output, our monitor typically.

in this case we use the greater than symbol > to redirect the output to a non existent file of the name in $1 - which actually makes the file by the redirection.

e) chmod is an external command assigning attributes to the new file. As far as what the numbers mean, I think it would be better at a later time to simply refer you to a web page explaining it.

For now 755 is what we will use to make our files executable

f) same as (b)

g) the end of the if statement, note 'if' spelled backward is 'fi'

Let's put it together and have a look at it.

Code: Select all

#!/bin/bash
[ ! $1 ] && echo "usage: newsh <filename>" && exit

if [ -f $1 ] ; then
	leafpad $1 &
else
	echo "#!/bin/bash">$1
	chmod 755 $1 
	leafpad $1 &
fi
How to make the first script

Open the terminal emulator

bin # you did fill out /etc/profile.local
leafpad newsh

Select the code from the forum post and copy to leafpad, close leafpad

chmod 755 newsh

If all went according to plan, we have a new utility we didn't have before

~

Chapter 4 - First script

Bruce B

#6 Post by Bruce B »

Words

I want to keep our work separate from the other Puppy directories and files. At the end of the training, we will know right where everything is. You can clean up and start over, keep your work - do as you wish.

This is required for future lessons

Make a directory called /root/doc
Make a directory called /root/doc/words

mkdir /root/doc
mkdir /root/doc/words

Download this file: mword10.zip from Project Gutenberg, Moby Words II by Grady Ward

The file is 3mb and the contents are 10mb - make sure you have the space.

I want this file because we can use its contents in a variety of ways. And, you will end up with some really nice utilities, which otherwise don't even exist. I'd rather us build meaningful utilities, when possible, than merely demonstrate various bash features.

I hope I'm also demonstrating that using the TUI need not be substitute for the GUI, rather a way of enhancing your operating system, making it more fun and powerful.

Maybe, you've noted by now, the text in darkblue is reserved for commands you will actually type in. We accomplish some learning simply by doing and repetition. After enough repetition, the commands become second nature, and it doesn't take all that much either.

Suppose we download the zip file to /tmp, here's how to extract it to the directory we need the extracted files to be in;

cd /root/doc/words
unzip /tmp/mword10.zip

That's it for this chapter, we will use these files in later lessons.

~

Chapter 5 - Words

Bruce B

Enhancing mrxvt

#7 Post by Bruce B »

Enhancing mrxvt

The exercises in this chapter are optional, but likely desirable.

You installed mrxvt. I noted the rc file; /root/.mrxvtrc is very empty of the commands available. I've included some commands to add to the rc file and mrxvt documentation to put in your doc directory.

Step 1)

Open the rc file

leafpad /root/.mrxvtrc

Hint: You can copy the command from the forum post and paste it to the command line. To do so, highlight the command, activate mrxvt and push the mouse wheel. (The TUI doesn't necessarily involve a lot of typing, sometimes very little.)

Copy and paste this text into leafpad /root/.mrxvtrc

mrxvt*geometry: 80x25
mrxvt*bottomTabbar: true
mrxvt*mouseWheelScrollPage: true
mrxvt*tabTitle: bash
mrxvt*scrollbarRight: true
mrxvt*foreground: white
mrxvt*background: black
mrxvt*font: 10x20
!mrxvt*autohideTabbar: true
!mrxvt*scrollBar: false


Among other things, these rc instructions will give you a larger font.

Note how you can change colors, I set it for black background and white text. Change it to your preferences.

Step 2)

I put together two text files. One containing the short commands for mrxvt and another containing the long commands.

Code: Select all

Archive:  mrxvt-text.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
     5504  07-01-09 12:04   mrxvt-long.txt
     7060  07-01-09 12:02   mrxvt-short.txt
 --------                   -------
    12564                   2 files
Download mrxvt-text.zip and extract the contents to /root/doc/

Step 3)

Open your favorite browser, navigate to /root/doc and make a bookmark for the directory. With the bookmark, you will be able to easily find any documentation pertaining to this course and view it in the browser.

~

Chapter 6 - Enhancing mrxvt
Attachments
mrxvt-text.zip
(3.8 KiB) Downloaded 2619 times

Bruce B

A Modest Introduction to Variables

#8 Post by Bruce B »

A Modest Introduction to Variables

A variable is a way of storing data in a name. The data is accessed by referencing the name.

The construct of a variable is like this

VARIABLE_NAME="this text is the data stored in the variable VARIABLE_NAME"

The left side of the construct is the name of the variable
The = defines the end of the name and begins the data portion
The right side is the data the variable holds

We set a variable exactly as shown above. No spaces allowed before or after the = sign.

We access the data by prepending the variable name with a $ sign

When we refer to the variable with the $ sign we are referencing the data.

Command line example and exercise;

myname="first name last name"
echo $myname


Variables have many, many uses. As we begin building our scripts, the uses, as well as the necessity of variables will become more apparent.

------------------------

Earlier we made a shell script - newsh. In the script we used leafpad as our GUI editor. Suppose we build 50 scripts using leafpad and then decide we'd rather use geany. We would have to go back and change every instance of leafpad to geany.

We can easily use one variable and effect the change in all 50 scripts, merely by changing the data in one variable

We can revisit profile.local and add this variable:

guiedit="leafpad"

Our first script originally said;

leafpad $1 &

With the variable guiedit the script would read like this:

$guiedit $1 &

It would do exactly the same thing, except if we changed the data in the variable to something like 'geany' or 'medit' the same script will open a different editor.

For the purpose of exercise and learning, let's make the change. At this point, you should be able to do this without instruction, nevertheless I'll provide the commands to run.

(remember about how to copy and paste from forum to the command line and to the text editor)

leafpad /etc/profile.local
add this text guiedit="leafpad"

leafpad /root/bin/newsh
change the two leafpad lines to read like this:
$guiedit $1 &

~

Chapter 7 - A Modest Introduction to Variables

Bruce B

Aliases

#9 Post by Bruce B »

Aliases

If you notice, we've already opened /etc/profile.local a few times. When you discover you are performing long commands on the command line repetitively, it's a good indicator you might want to make an alias for the command.

We want to make it easy to add aliases - and we will.

We can edit /etc/profile.local with a short command - an alias. An alias is constructed like this:

alias profilel='$guiedit /etc/profile.local ; source /etc/profile.local'

( note the variable, the source command and the command separator ';' )

We will add the alias above and three more;

alias bashrc='$guiedit /root/.bashrc ; source /root/.bashrc'
alias rclocal='$guiedit /etc/rc.d/rc.local'
alias mrxvtrc='$guiedit /root/.mrxvtrc'


rc.local

Note we didn't source rc.local, we simply edited it. rc.local is a file we place non X commands we want to run only one time per session.

This file will contain things like mounting devices we want mounted, such as;

mount -t ext2 /dev/sda3 /mnt/sda3
modprobe usbcore


We only want to do these actions one time, rc.local runs only one time, so it is the appropriate file for such commands.

.bashrc

Is for commands we 'source' over and over. /root/.bashrc gets sourced each time we open a terminal instance of bash. This is a default.

profile.local

We can control if it runs each time an instance of bash runs.

Conventions

rc.local and .bashrc are conventional Linux files. profile.local isn't particularly conventional with Linux distributions. Rather a 'feature' included with Puppy.

An important consideration for all scripting

Do not refer to any variable before it has been defined. Our files read from top to bottom. We defined a variable guiedit=leafpad

Now we are about to add some aliases which use the variable. They must be inserted below the definition of variable, otherwise we get errors.

Assignment (required)

Open

leafpad /etc/profile.local

Add these lines;

alias bashrc='$guiedit /root/.bashrc ; source /root/.bashrc'
alias profilel='$guiedit /etc/profile.local ; source /etc/profile.local'
alias rclocal='$guiedit /etc/rc.d/rc.local'
alias mrxvtrc='$guiedit /root/.mrxvtrc'



---------------------------------------------

You should be able to remember the names of your own made aliases. We also have the alias command, which if typed with no arguments will display your aliases.

If for some reason you don't want an alias for a bash session you can unalias it. Maybe you are removing a lot of files with the 'rm' command and you don't want the interactive portion.

Merely type: unalias rm

If you want to bypass an alias for just a few times, but not for your entire bash session, then type the following;

\rm <arguments>

The \ will cause the rm command to behave as if there was no 'rm' alias.

------------------------------------------------

Hereafter you will simply type the alias name to open the files we made editing aliases for. (thus making life easier)

~

Chapter 8 - Aliases

Bruce B

#10 Post by Bruce B »

grep

grep is search filter utility and commonly used in scripting.

In this chapter we will make two simple scripts using grep.

------------

We can run most any application on the CLI, including those in our Puppy menu. But the name in the menu doesn't say what the name of the executable file is.

Suppose we want to know the file name for the menu item Wizard Wizard. We can make a simple script which will make it easy to discover the executable name for Wizard Wizard, as well as any other menu item.

First script

bin ; newsh grepmenu

[ ! $1 ] && echo "Missing argument" && exit
</root/.jwmrc grep --ignore-case "$1"

Let's make another script

bin ; newsh grepalias

[ ! $1 ] && echo "Missing argument" && exit
alias | grep --ignore-case "$1"

That's it! Not so hard.

But I do need to explain the scripts. The first lines have been covered. I'll explain the second lines of each script.

------------

</root/.jwmrc grep --ignore-case "$1"

The < and > are our redirection symbols. The < directs the content of the file /root/.jwmrc to the command grep. As you can see, grep is instructed to ignore case. The menu item says Wizard Wizard. The "$1" says to use argument one.

Our command line can say:

grepmenu wizard
or with quotes
grepmenu "wizard wizard"

-------------

alias | grep --ignore-case "$1"

The alias command is standard output to the monitor, we use this symbol: '|' to pipe the output to grep, which will filter the output to match our search criteria.

Pipes and redirection are very important in constructing scripts. You've been shown some examples, if you don't fully understand, just hang in. < | > will be repeated over and over many times.

Our example scripts show a very simple usage for grep. Grep is a very powerful utility and capable of doing complex chores. Included with this chapter is the man page for grep in text format. I recommend downloading it and unpacking it to your /root/doc directory. When you are writing scripts you can open it in your browser and use it for reference.

If you want to get an overview of the many ways grep is used, you can click on the forum search link at the bottom of the post and enter the search criteria grep.

Chapter 9 - grep
Attachments
grep.zip
grep man page - text format
(10.54 KiB) Downloaded 2285 times

Bruce B

#11 Post by Bruce B »

Comments, for loop, read command, string comparisons and more

Our utility lower renames files to the lower case. Linux convention is lowercase names without spaces. The word list files you downloaded and unzipped are all uppercase. I wrote a utility to make easy work of changing the filename case to lower case.

Code: Select all

# lower converts files names to lower case
# without overwriting conflicting names

# known bug doesn't rename files with spaces
# feature: doesn't rename directories

# feature: preview changes
# feature: interactive

for f in *
do
    if [ -f $f ] ; then
        g=`echo -n "$f" | tr '[A-Z]' '[a-z]'`
        if [ "$f" !=  "$g" ] ; then
           echo mv "$f" "$g"
        fi
    fi
done

# get user permission

read -n 1 -p "Continue with conversion (y,n) " ans

# if yes make changes

if [ "$ans" = "y" -o "$ans" = "y" ] ; then
    for f in *
    do
        if [ -f $f ] ; then
        g=`echo -n "$f" | tr '[A-Z]' '[a-z]'`
            if [ "$f" !=  "$g" ] ; then
               mv --verbose --interactive "$f" "$g"
            fi
        fi
    done
fi
echo
ls
Explanations

# comments

You will hear the term comment the line or comment out. Most computer languages have a way of adding comments to the code - a way of signaling the processor or compiler to ignore the commented text. Bash uses #. If someone tells you to comment out some lines in bash, they mean to say put a # sign in front of the code.

We can use a comment to communicate our thoughts in the code and to block code without deleting it. The idea being everything on that line after the # is ignored.

# for loop

The for loop will be used over and over in our scripting. If you don't understand it now, you will soon enough.

Here is the construct:

for myvar in args
do
my_commands $myvar
done


In English it says, for my variable (myvar), process all my arguments according to my commands. When the last one is processed, finish.

If / else / conditional tests

In the second set we have three if / else constructs

get user input

read -n 1 -p "Continue with conversion (y,n) " ans

Construct 1

Test user input, if true run the commands

if [ "$ans" = "y" -o "$ans" = "y" ] ; then

Construct 2

If our variable represents a file then do the work, otherwise don't, meaning we aren't renaming directories

if [ -f $f ] ; then

Construct3

We process the variable f converting any uppercase letters to lower case and store the information in a variable called g

The use of the backtick ` is necessary to move the output into the variable, you will learn when to use the `

g=`echo -n "$f" | tr '[A-Z]' '[a-z]'`

Here we make a comparison to see if they do not match, not matching makes for a true condition, which we need for execution. Meaning any files in the directory which are not already lower case will be converted

if [ "$f" != "$g" ] ; then
mv --verbose --interactive "$f" "$g"

The mv command is also the rename command. We rename the files which match our criteria, with verbosity, meaning the users sees what goes on. Interactive means if a conflict occurs the user is notified and asked what to do.

More

This is an example of an interactive shell script. An important aspect of the interactivity is using the read command to get an answer from the user. We want to user to give an informed answer, so we display the results of the proposed changes in the top routines.

Please note the indentations - they are very helpful in reading the file. And are standard good coding practice. You will want to learn when and where to indent.

This may be the last time I show you how to make the script, using our existing system, like this;

bin ; newsh lower

(copy and paste the text, save and close the editor and the script is done)

Then run the script

cd /root/doc/words
lower

Optional exercise - make a script called upper which converts file names to upper case

bin ; cp lower upper ; newsh upper

Then figure out what to modify, test the script in /root/doc/words

~

Chapter 10 - Comments, for loop, read command, string comparisons and more

Bruce B

Line endings, file, dos2unix, hexdump

#12 Post by Bruce B »

Line endings, file, dos2unix, hexdump

Line endings

With our text files we don't see the line with in most editors. (non visible printing characters)

Different operating systems use different line endings, probably for the purpose of being different.

Mac : uses carriage return
MS-DOS : uses carriage return plus line feed
Unix : uses linefeed

Puppy gives us a utility called file, it's purpose is to help us learn the characteristics and type of file.
The way to use file is:file<filespec>

I ran 'file' on our word lists and this is the output:

  • aareadme.txt: ASCII English text, with CRLF line terminators
    acronyms.txt: \012- Sendmail frozen configuration \012- - version 4GL\015
    common.txt: ASCII English text, with CRLF line terminators
    compound.txt: Non-ISO extended-ASCII English text, with CRLF line terminators
    crosswd.txt: ASCII C program text, with CRLF line terminators
    crswd-d.txt: ASCII text, with CRLF line terminators
    fiction.txt: ASCII English text, with CRLF line terminators
    freq-int.txt: ASCII English text, with CRLF line terminators
    freq.txt: ASCII English text, with CRLF line terminators
    kjvfreq.txt: ASCII English text, with CRLF line terminators
    names-f.txt: ASCII text, with CRLF line terminators
    names-m.txt: ASCII text, with CRLF line terminators
    names.txt: Non-ISO extended-ASCII English text, with CRLF line terminators
    oftenmis.txt: ASCII text, with CRLF line terminators
    places.txt: ASCII text, with CRLF line terminators
    single.txt: \012- Sendmail frozen configuration \012- - version \015
    usaconst.txt: ISO-8859 English text, with CRLF line terminators
I wasn't sure what acronyms.txt and single.txt are in terms of line endings, I order to get a better look, I ran hexdump -C single.txt.

I saw plenty of the hexadecimal 0x0D and 0x0A - which are the messy dos line endings.

I then opened it with geany and it looked like a long list of single words. It looked fine, I felt safe in fixing the line endings on these two supposed Sendmail files.

The tool we use is fix the messy line endings is dos2unix, also included with Puppy. It accepts wildcards, so there won't be a need to make a script.

After running dos2unix my output from the file utility is:
  • aareadme.txt: ASCII English text
    acronyms.txt: \012- Sendmail frozen configuration \012- - version 4H
    common.txt: ASCII English text
    compound.txt: Non-ISO extended-ASCII English text
    crosswd.txt: ASCII C program text
    crswd-d.txt: ASCII text
    fiction.txt: ASCII English text
    freq-int.txt: ASCII English text
    freq.txt: ASCII English text
    kjvfreq.txt: ASCII English text
    names-f.txt: ASCII text
    names-m.txt: ASCII text
    names.txt: Non-ISO extended-ASCII English text
    oftenmis.txt: ASCII text
    places.txt: ASCII text
    single.txt: \012- Sendmail frozen configuration \012- - version
    usaconst.txt: ISO-8859 English text
Notice everyone of the with CRLF line terminators are gone. We now have genuine Unix files, conforming to the proper standards and conventions for our operating system.

For your exercise, please run the commands in blue.

cd /root/doc/words

lower (if you haven't already ran it)

file *.txt

hexdump -C single.txt and view the output especially looking for Od Oa - the messy dos
line terminators - Ctrl + C to break

dos2unix *.txt

hexdump -C single.txt and note there are no more 0d characters

file *.txt and note there are no reports of CRLF in the files

------------------------

Summary, you ran your new utility 'lower', learned a little about other utilities; hexdump, dos2unix and file

The skill learned is how to test and clean up text files to make them truly Linux files.

~

Chapter 11 - Line endings, file, dos2unix, hexdump

Bruce B

Look

#13 Post by Bruce B »

Look

This chapter is very short. I finished the script for using the various word lists. The file name is called look

Usage:

look arg1
or
look arg1 arg2

Where the arguments represent a portion of text in a word you want to lookup.

To search the beginning of the text, use the ^ sign

look ^arg1

For example: to look up Bruce and other words which start with br, I would use this syntax

look ^br

The $ sign represents the end of the word, I could enter arguments like this

look ^b e$

The output would show every word that starts with b and ends with e in the list I'm searching.

------------------------

Other word lists from different languages could also be added to /root/doc/words. The program is very extensible.

I won't try and explain the program in this post, because it will be much easier if I wait a couple days, until you've had a chance to interface with it, inspect it and see how it works. This way the explanation will have more meaning.

-------------------------

Installation instructions, unzip look from the package look.zip in the /root/bin directory

~

Chapter 12 - look
Attachments
look.zip
'look' script file for searching word lists
(511 Bytes) Downloaded 2679 times

Bruce B

#14 Post by Bruce B »

Midnight Commander

I made change to Puppy's official Midnight Commander's mc.ext file. It will now open both .pup and .pet packages in a virtual filesystem.

Simply highlight the .pup or .pet with the arrow keys and hit the enter key to open the archive.

This makes it very easy to inspect contents and/or perform manual installs.

Installation instructions: Download the attached zip package and extract it in /usr/local/share/mc

OR;

In order to demonstrate just how easy mc makes things, you can do like this;

* download mc.ext.zip
* open mc
* navigate to where you downloaded mc.ext.zip
* highlight the file
* tap the enter key to open it
* in the other panel, navigate to /usr/local/share/mc
* copy mc.ext from one panel to the other

It can even open iso files with the tap of an enter key

- Enjoy

~

Chapter 13 - Midnight Commander
Attachments
mc.ext.zip
modified to open .pet and .pup packages
(4.84 KiB) Downloaded 2640 times

Bruce B

#15 Post by Bruce B »

updatedb

I wrote three scripts and packaged them as 'findutils.zip'. There are binary packages available which do about the same thing and share similar names as these scripts. If you happen to have these already installed, I'll leave it to you which to use.

In this chapter, I present updatedb

The style and program flow I use here, is not conventional for bash scripting, in the following ways;
  • * everything is done in functions

    * the program flow is the function main

    * the variables are in lowercase
I've never actually seen this type of programming and in your other courses, you probably won't either. But you might like it because;
  • * its very easy to write complex programs by breaking things down to small and simple functions

    * it's easier for me to read lowercase text variables

    * the program flow is very easy to follow, we have only one command, main

    * main controls other commands

    * if we look at main we can see the entire flow

    * after each function, I return to main for the next command;

    * no jumping around between functions

    * if you are having problems, it's very easy to comment out commands in main and debug the troublesome function

    * the style is quite similar to how C is written, it's from C I got the idea to do bash in a similar way
Ultimately, you'll develop your style. I suppose it's good to be exposed to different ways of doing things. Don't think my style is the right style, it's just a style.

Code: Select all

#!/bin/bash

main() {

    var
    filesdb
    dirsdb
    showfiles

}

var() {

    file="/var/log/updatedbf"
    dirf="/var/log/updatedbd"
    dirs="archive bin etc lib opt root sbin usr var"

}

filesdb() {

    [ -f $file ] && > $file

    for i in $dirs ; do
        echo "Processing files in /$i . . ."
        find /$i -noleaf -mount -type f >>$file
    done

}

dirsdb() {

    [ -f $dirf ] && > $dirf

    for i in $dirs ; do
        echo "Processing directories in /$i . . ."
        find /$i -noleaf -mount -type d >>$dirf
    done

}

showfiles() {

    echo
    ls -l --color=always $dirf
    ls -l --color=always $file
    echo

}

main
( don't copy and paste this code, I have a package prepared for download, it will be attached in the next post)

Explaining updatedb

Compared to searching all of Puppy for a file, it's lightening fast to simply search a database. We build two databases, one for directories and one for files.

var: we set our variables, which include the directories we plan to catalog, this can be modified

filesdb: the instructions for making the files database and echoing the progress on screen

dirsdb: instructions for making the directories database and echoing the progress on screen

showfiles: we simply display the two databases

---------------

In the next chapter I'll show a find utility and upload the three combined files

~

Chapter 14 - updatedb

Bruce B

#16 Post by Bruce B »

Find scripts

You've already been introduced to; grep, redirection, variables.

Now, explaining command line variables. They are numbered from left to right as you type them. If you have three arguments, then you have variables $1 $2 $3

$@ means all command line arguments
$# means the number of arguments

In this example we use the case statement to determine which commands to run. And this is your first viewing of case in this study

The information we give case is $# which is the total count of our arguments. Valid matches, (true conditions), are 1 to 3. If we don't give case a number between 1 to 3 arguments, then * is true and it echos an error and does nothing.

Have a look and see what you can learn and derive.

Code: Select all

#!/bin/bash

main() {

    var
    dofind "$@"

}


var() {

    file="/var/log/updatedbd"

}

dofind() {

    case $# in

        1)
        <$file grep --ignore-case "$1"
        ;;

        2)
        <$file grep --ignore-case "$1" \
        | grep --ignore-case "$2"
        ;;

        3)
        <$file grep --ignore-case "$1" \
        | grep --ignore-case "$2" \
        | grep --ignore-case "$3"
        ;;

        *)
        echo "Incorrect number of arguments"
        ;;

    esac

}    

main "$@"
To install these scripts, download the package, use Midnight Commander to open the zip package in a virtual filesystem and copy the three scripts to /root/bin

findf is for finding files
findd is for finding directories
updatedb is for building the databases

findf means find files
findd means find directories
First run updatedb to build your database, then run findd and see if you can locate the directories containing pixmaps and icons.

Remembering, we don't want to type too much, if you want the output of a line, just double-click it to select it and shift+insert or push mouse button to paste the text on your command line or into an editor. Useful when writing scripts to make sure you don't make a typo.

Suppose you want to change to a directory in your output, type cd (space) in the command line, then paste the directory to the command line and hit enter.

There are many ways to keep from typing too much when using the command line, in time we will learn more and more

~

Chapter 15 - Find scripts
Attachments
findutils.zip
findutils kit : contents : three scripts
(1.03 KiB) Downloaded 2027 times

Bruce B

#17 Post by Bruce B »

tar and gzip

This chapter is short, sweet and important. I'll tell you why it's important. In the thousands of posts I've read on the forum, I seen many, many posts were it's obvious to me the problem is somewhere in the directory /etc

But I can't tell remotely exactly what the problem is.

/etc is not a static directory like /bin. /etc changes some every boot. Usually what brings on the users problem is a bad shutdown. Then, they can't recover.

Let's avoid problems by maintaining a backup of /etc

Code: Select all

# filename bkup-etc
tar cvzf /var/etc.tar.gz /etc
echo Done!
ls -l --color=always /var/etc.tar.gz
What's introduced in this script is tar and gzip.

I'll talk a little about tar. We only run one utility here, tar, but tar works in conjunction with gzip (or bzip2). So we are running two utilities, although it doesn't look like it.

A .tar.gz file is very, very common in Linux. You will also find you can make and extract tar.gz archives very easily with Midnight Commander, simply hit the F2 (the menu) key to intuit how.

Tar makes the archive, but it's uncompressed, gzip is the utility that compresses the tar archive. This is why the two extensions, tar says a tar archive and gz says compressed by gzip. But sometimes it's displayed as .tgz. When you encounter a .tgz or tar.gz file, just know it is the same thing, described differently.

The switches used are cvzf

c - create
v - verbose output
z - says compress it with gzip
f - the filename

f - should be the last in the chain, just proceeding the filename

If we wanted to extract the package we'd say

tar xvzf etc.tar.gz

(the x says extract)
(the c says create)

Not so hard. You already have the tools and know how to easily make the script - bkup-etc, so do it.

/root
contains your user settings and maybe important files. If you have the space to backup /root - I recommend doing it - use bkup-etc as your template for making the script.

The etc.tar.gz will be about 1/2 megabyte

If /root is too big for pup_save, you could do something like this:

tar cvzf /mnt/home/root.tar.gz /root


I'll leave it to you as an extra-curricular kind of thing to figure out the logistics.

~

Chapter 16 - tar and gzip

~
Last edited by Bruce B on Sat 04 Jul 2009, 13:59, edited 1 time in total.

Bruce B

#18 Post by Bruce B »

Change Directories - the cd command

The cd command is what we use to change directories. I'll start by offering some examples

cd : changes to the $HOME directory, for us it would be: /root

cd .. : takes us up one level

cd ../.. : takes us up two levels

cd /bin : takes us to /bin

cd bin : takes us to a child directory bin

cd /usr/bin : takes us to /usr/bin

bin : takes us to /root/bin - the reason why is, we made an alias called bin

cd - : returns us to our last directory

Related commands pushd and popd

pushd /usr/local/bin changes to /usr/local/bin and saves the directory we were in when we ran pushd

popd : returns to the the directory saved by pushd, 1x only

Adding extra commands

The uu command saves our current directory for the duration of the session or until we run uu again

The u command returns us to the directory saved by uu

The ad command displays the first level child directories, if any, of our current working directory.

The cdd command behaves like cd and shows all the first level subdirectories, if any.

The normal cd command used this way: cd /usr will change directories to /usr but not show the subdirectories in /usr

cdd /usr will change to /usr and show all the subdirectories in /usr/ This makes it easy to dig deeper with tab completion or copy and paste the output.

Adding the new commands;

Code: Select all

function cdd {
	[ $1 ] && cd $1
	ad
}
alias uu='echo cd $PWD>/tmp/uu~'
alias u='source /tmp/uu~'
alias ....='cd ../../../.. ; ad'
alias ...='cd ../.. ; ad'
alias ...='cd ../../.. ; ad'
alias ..='cd .. ; ad'
alias rf='source /root/.bashrc ; echo "bashrc refreshed"'
To install these new aliases and function;

bashrc
followed by the old copy, paste and save routine

ad is a shell script. Download ad.zip. Add the contents 'ad' to our /root/bin directory using Midnight Commander.

After you've used ad a bit, I'll explain it. Part of what I'm doing in these earlier chapters is helping make your TUI environment as user friendly as possible. Soon, quite soon, changing directories on the TUI will be easier and faster than with the GUI, but it doesn't start off that way.

~

Chapter 17 - Change Directories - the cd command
Attachments
ad.zip
ad.zip : all directories : a script to show all subdirectories
(411 Bytes) Downloaded 1986 times

Bruce B

Why?

#19 Post by Bruce B »

Why?

In our last chapter, we made aliases, a function and installed a shell script. In this chapter I'll present theory for the whys of our whats.

Changing directories in a shell script

Remembering that our shell script is a child process, if we change directories in a script, we will change directories in the script. When the script is finished we are in the same directory we were in when when ran the script.

This script called usrbin;

#/bin/bash
cd /usr/bin


will actually do what it says, but when it finishes we are left in the same directory we ran it from.

Changing directories with an alias

This alias of the same name: usrbin

alias usrbin='cd /usr/bin'


Actually changes our current directory to /usr/bin, just as if we typed the commands. This alias doesn't launch a child process, thus giving the desired results.

What about the function?

We also added a function to .bashrc , I'll explain the rational for this now;

function cdd {
[ $1 ] && cd $1
ad
}


The function as an alias would look like this:

alias cdd='[$1] && cd $1 ; ad'


The problem with using an alias; is I've found aliases horrible for accepting command line arguments in the middle of the structure.

But they do great at the end of the structure, here's a working example;

alias mount='mount-FULL'

With this alias, when I use the command mount, on the command line, the actual mount command will be used, which in Puppy, and probably only Puppy is, mount-FULL

aliases accept appended arguments very well. Our problem is inserting them somewhere other than the end.

mount -t ext /dev/hda1 /mnt/hda1 , without an alias runs a Puppy shell script called mount which in turn will run mount-FULL and return to the parent shell when finished

With the alias

mount -t ext /dev/hda1 /mnt/hda1 runs the mount-FULL in the current shell

Why the shell script ad?

It works swell as a shell script, it lends itself well to shell scripting.

There nothing wrong with shell scripting. We will be doing a lot of shell scripting.

Major point: Sometimes there is a better approach to shell scripting, especially when trying to change directories in our current shell.

~

Chapter 18 - Why?

Bruce B

Chapter 19

#20 Post by Bruce B »

Troubles with whitespace

For our purposes; whitespace could be defined as characters which don't display.

Our most common text editor whitespace characters would be line endings, space and tab characters.

In bash we have the backslash \ character, an instructoion for the bash interpreter.

For this post I'll define \ as the 'do something different for the next character character'

In bash, the end of line (eol) character is the signal for end of command.

Sometimes I post code and have you insert the code, other times I don't post the code and have you download it.

Here is an excerpt from the file 'ad' in chapter 17

Code: Select all

    find  -maxdepth 1 -type d | cut -b 3-80 \
    | sort -g | mkfiles
Note the use of \, it says do something different about the next character, which is the eol character, (0x09). What it does different is; bash doesn't treat the whitespace, the eol, as a signal that the command is complete.

It continues the command execution to the line below. Without the \ character the command would need to look like this:

Code: Select all

    find  -maxdepth 1 -type d | cut -b 3-80 | sort -g | mkfiles
Sometimes we make very long commands, longer than our text editor can display and longer than a forum post will safely display.

The forum will use the space character as a delimiter for line wrapping. We don't know our reader's font size or screen resolution. Because of this, we can get unwanted line wrapping with long code excerpts.

The long code might wrap at the wrong place. If the user copies and pastes code which the forum has wrapped (at the wrong place for code), the code will likely be very defective.

With the code block below, it is imperative that no white space follow the \ except for the end of line character. By using the \ we have to be sure the character we want to treat differently is actually - the eol character - and not a space character.

I didn't post the code, because I don't want it copied from the forum and have someone inadvertently insert a space character. To be safe, download it, because if I'm doing my part right, there are no boo-boos in the in the code and it will run as expected.

Code: Select all

    find  -maxdepth 1 -type d | cut -b 3-80 \
    | sort -g | mkfiles
~

Chapter 19 - Troubles with whitespace

Post Reply