Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 22 Oct 2014, 10:30
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Odd Bash script inconsistancy...
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Author Message
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Fri 22 Jan 2010, 15:06    Post subject:  Odd Bash script inconsistancy...  

I wrote a little script to add up the size of files selected by wildcard argument.
The lines of code work fine run individually in Xterm.
But in the script it only gets the first file, not the whole wildcard list.
Code:
#!/bin/sh
#####   Get total size of files by wildcard argument.

if [ -z "$1" ];then exit ;fi
FILES=`du $1`
#FILES=`ls -1l $1 | awk '{print $5, $8}'`
echo '###   FILES'
echo "$FILES"
SIZES=`echo "$FILES" | awk '{print $1}'`
echo '###   SIZES'
echo "$SIZES"
ADD=`echo $SIZES |  sed 's/ / + /g'`
echo '###   $ADD'
echo "$ADD"
echo '============================================='
echo "$FILES"
echo '================================'
echo `expr $ADD`' Total of File Sizes.'

The bad lines are "ls" or "du", and both give only the first line in the script.
If you run either line in Xterm it gives all the matching files and sizes.

Last edited by sunburnt on Fri 22 Jan 2010, 16:10; edited 1 time in total
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2258

PostPosted: Fri 22 Jan 2010, 15:34    Post subject:  

Instead of calling ls, awk(twice!), sed, du and expr:
Code:
#!/bin/bash
for FILE in * ; do
   [[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
done
echo TOTAL_SIZE=$TOTAL_SIZE
Back to top
View user's profile Send private message 
seaside

Joined: 11 Apr 2007
Posts: 887

PostPosted: Fri 22 Jan 2010, 20:47    Post subject:  

Also, how about this -

Code:
ls -l | awk  '{total+=$5} END{print total}'
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sat 23 Jan 2010, 16:05    Post subject:  

Thanks guys! But...

amigo; Your code also only returns the first file! ( run in a Bash script )
I changed the * to $1 so the wild card path would work properly.
Code:
for FILE in $1 ; do
   [[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
   echo $FILE
done
echo TOTAL_SIZE=$TOTAL_SIZE ;exit

It returns: TOTAL_SIZE=1607264396 ( the size of the first file )
There are 2 files in the dir. that start with C*
The line "echo $FILE" shows this to be true.

seaside; I`m not sure how your code is to be used...
You use $5 to get the file sizes from "ls", but "ls" is the problem !!!
Again... All of this works in a console, but it doesn`t in a Bash script.

### My hope was to display the files and their sizes, and a total.
That`s why the many lines in my code to get all the display items.
The other lines are to test each step to show where the failure is.

This is just another example of the incongruity of Bash and the other commands that support it.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2258

PostPosted: Sat 23 Jan 2010, 16:27    Post subject:  

I originally had this:
for FILE in $(ls -1) ; do
but any files with spces in the names would not work there.
The '*' works fine here and should anywhere -very strange if it is not working for you.
Back to top
View user's profile Send private message 
seaside

Joined: 11 Apr 2007
Posts: 887

PostPosted: Sat 23 Jan 2010, 17:24    Post subject: Re: Odd Bash script inconsistancy...  

sunburnt wrote:
I wrote a little script to add up the size of files selected by wildcard argument.


Sunburnt,

I thought you might just use "ls" with wildcard selection like this (not in a loop).

Code:
ls -l *wildcard* | awk  '{total+=$5} END{print total}'


s.
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sat 23 Jan 2010, 21:04    Post subject:  

amigo; Try it like this so a wildcard argument can be fed to it:
Code:
#!/bin/sh
#####   Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
for FILE in $1 ; do
  echo $FILE
done

Let me know if it works for you and gives the full wildcard list.

seaside; That is what my script does...
My code line:
Code:
FILES=`ls -1l $1 | awk '{print $5, $8}'`

Your code line:
Code:
ls -l *wildcard* | awk  '{total+=$5} END{print total}'

Have you tried using "ls" in a script to see if it will list all the wildcard files?
Both "ls" and "du" seem to work at the command line but not in a script.
All I get from them and amigo`s "for loop" is the first wildcard match, not the list.
Back to top
View user's profile Send private message 
seaside

Joined: 11 Apr 2007
Posts: 887

PostPosted: Sun 24 Jan 2010, 00:12    Post subject:  

sunburnt,

This works in a script.
Code:
#!/bin/sh
wild=s*
total=`ls -l $wild | awk  '{total+=$5} END{print total}'`
echo $total
files=`ls -l *.txt`
echo $files
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sun 24 Jan 2010, 13:19    Post subject:  

seaside; I don`t know if it`s the fact I need to use $1 for the wildcard argument.
Again... Your code only pulls the first file and it`s size! ( using $1 )
Code:
#!/bin/sh
if [ -z "$1" ];then exit ;fi
wild=$1
total=`ls -l $wild | awk  '{total+=$5} END{print total}'`
echo $total
files=`ls -l $1`
echo $files

Using "ls", "du", and "stat" just doesn`t seem to work in a script...
Try the simplest form:
Code:
#!/bin/sh
if [ -z "$1" ];then exit ;fi
ls -l1 $1
du $1

All methods ( first "ls" then "du" ) give this:
Code:
# filesize /mnt/sda6/v/0/C*
-rw-r--r-- 1 root root 1607264396 2010-01-07 06:32 /mnt/sda6/v/0/Children-1.jpg
1607264396 /mnt/sda6/v/0/Children-1.jpg

There are the pix: /mnt/sda6/0/Children-1.jpg and /mnt/sda6/0/Children-2.jpg
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2258

PostPosted: Sun 24 Jan 2010, 14:06    Post subject:  

ls and du both can cause problems because they use tabs to format the output -maybe depending on whether run in a script. du is particularly troublesome.

The script fragment I posted is *bash* not POSIX shell, so /bin/sh may not work for the shebang -depending on what /bin/sh is linked to.

There is no way that something like this:
wild=$1
total=`ls -l $wild
is gonna return more than one item, unless you do something like echo * | name-of-script

This also works:
for FILE in $(echo *) ; do
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sun 24 Jan 2010, 14:35    Post subject:  

amigo; I tried it with "#!/bin/bash", no go... I didn`t think it`d work.
The "for" loop only iterates just once using "$1".
However... Putting the argument in place of "$1" works!
So... Using the "$1" argument variable seem to mess it up!!!
Try this:
Code:
#!/bin/sh
echo "$1"

It gives the first file... Not the wildcard argument.!!!
It seems that the wildcard is being interpreted by being the "$1" argument!
So feeding wildcards to a script just doesn`t work! Period!
This works with: /mnt/sda6/0/C
### Sort of...
Code:
#!/bin/sh
echo "$1"*

You have to assume the "*" at the end, it can`t be placed in the argument...
But it keeps the argument from being interpreted, so it works.
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sun 24 Jan 2010, 20:48    Post subject:  

So "ls", "du". or any other command isn`t the problem, it`s that
the wildcard argument is interpreted and it can`t be stopped.

Here`s what I came up with, it`s not what I wanted ( as usual...).
The script adds the "*" on the end of the argument:
Code:
#!/bin/sh
#####   Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
FILES=`du "$1"*`
SIZES=`echo "$FILES" | awk '{print $1}'`
ADD=`echo $SIZES |  sed 's/ / + /g'`
echo '==============================================='
echo "$FILES"
echo '============================'
echo `expr $ADD`' Total of File Sizes.'

So you use "/mnt/sda6/0/C" instead of "/mnt/sda6/0/C*"
As before, if you add the "*" at the end it`ll only give the first file.
Back to top
View user's profile Send private message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Mon 25 Jan 2010, 02:58    Post subject:  

maybe more useful:
Code:

FILES=`du -S  "$1"*`
Back to top
View user's profile Send private message 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Mon 25 Jan 2010, 13:03    Post subject:  

I used this in one of my scripts

Code:
#find total size
filesizes=`du -ch | grep total`
echo $filesizes




here was the original script it creates an index
I wrote it to make the index for my packages

when you dragNdrop a folder on it

it works fine if you dont have folders in subdirectories
I still need to figure out how to get that working correctly
if you amigo or anyone else gets the subdirectories working
it would make for a nice file browser too Very Happy
Code:


#!/bin/sh
# last updated 2-3-2010
# script  made by Joe arose first template 
# this makes the index.html for a simple package repo
# it is an all in one script so I can easily keep the package list updated
# I built this out of need and it needed to be easily updated
# I did'nt want web page developement to take all my time
# Im too busy doing other things


-------------------

#
# you dragN drop a folder on this script
# or set up the right click in ROX

# the rxvt terminal opens there
# you could change rxvt to xterm if you prefer
# Joe Arose ...big_bass

DIR="$1"
 
if [ -d "$DIR" ]
then
   echo "$DIR directory  exists!"
   cd "$1"
  ##  rxvt >/dev/null 2>&1
else
   echo "$DIR directory not found!"
    exec xmessage -bg "yellow" -center -title "ERROR" "folders /directories only "
fi


-----------------


echo >/root/html-tagset.html  #blank it  will be renamed to index.php-new later 
cd $DIR 

#find total folder size
filesizes=`du -ch | grep total`






# starts the head of the html  edit the names its a big echo
echo '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /slaxer_pup 4.12 </title>
 </head></font>
 <body>
<BODY BGCOLOR="#000000" BACKGROUND="stars_background.gif">
 <pre><img src="slxminilogo.gif" alt="Icon "> <a href="?C=N;O=D"></a>   
 <pre><img src="newlogo.png" alt="Icon "> <a href="?C=N;O=D"></a><br />
<font size="+2" color="#FFFFFF"> </a>
<br />'>>/root/html-tagset.html

#end of the big echo and the end the html head

# here below  is a simple sample format of what the output will look like without the  package size and date

#<img src=compressed.png alt=[ ]> <a href=geany-0.17-i486-slxr.pet><font size="5" color="#FFFF00">geany-0.17-i486-slxr.pet</font></a>


# body of the  html code is generated by bash
# this was tricky to get the readable values
#in the format for package name  date and size
#to get everything read at once for each package 


for package in `ls -p `; do
name=`basename "$package"`
datemodified=`ls -lh "$package" | awk '{ print $6 }'`
size=`ls -lh "$package" | awk '{ print $5 }'`


# edit this with care it was tedious to format it. Save a copy first 
echo "<img src="compressed.png" alt="[  ]">" "<"a href="$name"">""<"font size="5" color="#FFFF00"">"$name"<"/font">""<"/a">" "<"font size="5" color="#FFFFFF"">"  $datemodified "<"font size="5" color="#FFFF00"">"   $size >>/root/html-tagset.html       

done

# the last echo to end the index
echo "<hr></pre>$filesizes

</body></html>">>/root/html-tagset.html   

mv /root/html-tagset.html  /root/index.php-new
rm /root/html-tagset.html


if you copy
index.php-new that file into the folder you draNdropped it will work as an indexer


updated again
I fixed it to do the subdirectories
for package in `ls -p `; do
Joe

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though

Last edited by big_bass on Thu 04 Feb 2010, 16:58; edited 4 times in total
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2258

PostPosted: Mon 25 Jan 2010, 14:56    Post subject:  

I seem to have mis-read the original intention of the script.
First, you are wanting to pass arguments to this script to specify one or more locations and/or types of files?
And you want to be able to use wildcards in those arguments?

Answering in order:
if you want to pass multiple args to the script, then using '$1' is not right. You need to use $@ so that all the arguments get interpreted (by a loop):
Code:
#!/bin/sh
for ITEM in $@ ; do
 echo $ITEM
done


I assume you may be wanting to process directories instead of just files, so this will work for dirs as well:
Code:
#!/bin/sh
for ITEM in $@ ; do
   SIZE=$(du -sk $ITEM)
   SIZE=$(echo $SIZE |cut -f1 -d' ')
   TOTAL_SIZE=$(( $TOTAL_SIZE + $SIZE ))
   # the next line here is for debugging
   echo $ITEM = $SIZE
done
echo TOTAL_SIZE = $TOTAL_SIZE KB


Now about wildcards. They are going to be expanded by the shell *before* passing them to the script. for instance, if you save th above script as 'tryme' and place it in a directory where there are several files with a '.txt' suffix, if you run:
sh tryme *.txt
it will work correctly in that directory. It will also work like this:
sh tryme *.txt test/*.txt /path/to/another/plain/dir
(To show how the shell expands these first, just do:
echo *.txt test/*.txt
or whatever paths with wildcards you are using)

Here you have to use 'du' to get the sizes for directories as stat will not do that. You need the '-s' option to output just the summary total for the item, and the '-k' option so you know hat the unit is. If you use the '-h', then you'll get mixed output with some items as ??K, some as ??M and whatever.

You have to do this:
SIZE=$(echo $SIZE |cut -f1 -d' ')
so that echo reduces the whitespaces in the output from du. When you echo something unquoted, any mutiple spaces get converted to single spaces, tabs get converted to single spaces and any leading spaces are eliminated. The output from du is formatted with tabs or maybe even a mixture of tabs and spaces, so if you tried this:
SIZE=$(du -sk $ITEM|cut -f1 -d' ')
you'll get an error because of the garbage it returns(as fas as doing any mat is concerned)

This will work:
SIZE=$(echo $(du -sk $ITEM)|cut -f1 -d' ')
but this will not:
SIZE=$(echo $(du -sk $ITEM|cut -f1 -d' '))
I do them separately in the example for better dependability and readability. Since 'echo' is a shell builtin it doesn't cost any extra *external* process to do it.
awk will handle the tabs/spaces problem handily enough, but I still think it is overkill for this task. And, you could use ls instead of 'du', but you'll have the same problem with formatted output, and if you wanto to return totals from content inside directories, then you'd have to make an extra loop which can descend(mutiple levels) into each dir/subdir and add up each item inside there. 'du' already does that for you, and recursing routines with the sehll get messy.

Of course, when you are done, you can rename the script to whatever, make it executable and drop it in your PATH to be able to use it from wherever.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1091s ][ Queries: 11 (0.0032s) ][ GZIP on ]