Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 18 Jun 2013, 21:32
All times are UTC - 4

 Forum index » Off-Topic Area » Programming
sed problem with a table file <solved> bypass sed for awk
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Author Message
8-bit


Joined: 03 Apr 2007
Posts: 3033
Location: Oregon
PostPosted: Sat 10 Sep 2011, 16:52    Post_subject:  sed problem with a table file <solved> bypass sed for awk
Sub_title: how to change a line with no preceding space character		  
I am again playing with trying to fill a table from a created file that already displays in columns when opened as text.
I have used sed, most likely not correctly, to remove spaces done to one globally and then replace the space with a pipe.
But some lines do not have a preceding space and it messes up the file and the table.
Any suggestions as to mods to get the last lines in the tab2test.txt file formatted correctly for reading into a table?

This is what I have so far:
Code:

#!/bin/sh

GTKDIALOG=gtkdialog
ps -A > /tmp/tabtest.txt
cat /tmp/tabtest.txt | sed 's/    / /g' |sed 's/   / /g' | sed 's/  / /g' | sed 's/ /\|/2g'> /tmp/tab2test.txt
export MAIN_DIALOG='
<vbox>
  <table>
    <width>500</width><height>200</height>
    <variable>TABLE</variable>
    <label>   PID      | TTY |     TIME          | CMD</label>
         <input>'"cat /tmp/tab2test.txt"'</input>
    <action>echo $TABLE</action>
  </table>
  <hbox>
    <button ok></button>
    <button cancel></button>
  </hbox>
</vbox>
'

$GTKDIALOG --program=MAIN_DIALOG

Edited_time_total
Back to top
View user's profile Send_private_message 
seaside

Joined: 11 Apr 2007
Posts: 841
PostPosted: Sat 10 Sep 2011, 18:51    Post_subject: Re: sed problem with a table file
Sub_title: how to change a line with no preceding space character		  
8-bit wrote:

But some lines do not have a preceding space and it messes up the file and the table.


8-bit,

I like Awk for formatting ps output -

Code:

 ps -a | awk '{print $1"\|"$2"\|"$3"\|"$4"\|"}'


Regards,
s
Back to top
View user's profile Send_private_message 
8-bit


Joined: 03 Apr 2007
Posts: 3033
Location: Oregon
PostPosted: Sat 10 Sep 2011, 19:37    Post_subject:  
Thank you seaside!
Even though this was just a learning test for me, it showed me things I did not know.

It sure beats trying to format the file with a long sed sequence.
It also shows you that I have a lot to learn.

Also, when I ran it from a terminal the first time, I got a message of

awk: warning: escape sequence `\|' treated as plain `|'

So I went back and edited out the escape characters and it worked fine without the warning coming up.

Revised code:
Code:

#!/bin/sh

GTKDIALOG=gtkdialog
ps -A | awk '{print $1"|"$2"|"$3"|"$4"|"}' > /tmp/tabtest.txt
export MAIN_DIALOG='
<vbox>
  <table>
    <width>500</width><height>200</height>
    <variable>TABLE</variable>
    <label>   PID      | TTY |     TIME          | CMD</label>
         <input>'"cat /tmp/tabtest.txt"'</input>
    <action>echo $TABLE</action>
  </table>
  <hbox>
    <button ok></button>
    <button cancel></button>
  </hbox>
</vbox>
'

$GTKDIALOG --program=MAIN_DIALOG
Back to top
View user's profile Send_private_message 
seaside

Joined: 11 Apr 2007
Posts: 841
PostPosted: Sat 10 Sep 2011, 20:31    Post_subject:  
8-bit,

Yes, awk is great at what I call "field awareness". It takes a human approach to realizing that no matter how much or exactly what kind of white space is separating items, it knows where the fields are.

Also, you could eliminate the file i/o by doing this -
Code:


#!/bin/sh

GTKDIALOG=gtkdialog3

psTAB() {
ps -a | awk '{print $1"|"$2"|"$3"|"$4"|"}'
}
export -f psTAB

export MAIN_DIALOG='
<vbox>
  <table>
    <width>500</width><height>200</height>
    <variable>TABLE</variable>
    <label>   PID      | TTY |     TIME          | CMD</label>
         <input>psTAB</input>
    <action>echo $TABLE</action>
  </table>
  <hbox>
    <button ok></button>
    <button cancel></button>
  </hbox>
</vbox>
'

$GTKDIALOG --program=MAIN_DIALOG



Regards,
s
Back to top
View user's profile Send_private_message 
01micko


Joined: 11 Oct 2008
Posts: 7037
Location: qld
PostPosted: Sat 10 Sep 2011, 21:38    Post_subject:  
Hi 8-bit

sed can be a tricky animal

Is this what you are after? (sed can do it)
Code:
ps -A  | sed -e 's/[[:space:]]*[[:space:]]/\|/g' \
                           -e 's/^|//g' > /tmp/tab2test.txt

There is probably a better way.. some guru (shep?) will know how to do it. The problem for me was the leading single space, my code only covers 2 or more, so I end up with a leading pipe which then needs dealing with.
_________________
keep the faith Cool ..
Back to top
View user's profile Send_private_message Visit_website 
big_bass


Joined: 13 Aug 2007
Posts: 1736
PostPosted: Sat 10 Sep 2011, 22:33    Post_subject:  
Hey 01micko



Code:
ps -A  | tr -s ' ' '|*' | sed  's/^|//g' > /tmp/tab6test.txt


this | tr -s ' ' '|*'

does a compression of all the spaces
and does 99% of all the work

the sed 's/^|//g' takes care of just the first pipe removal

I did this a long time ago when I wrote a filename repair tool call sana_auto

P.S nice job with only using sed

Joe
_________________
slackware 14
Back to top
View user's profile Send_private_message 
01micko


Joined: 11 Oct 2008
Posts: 7037
Location: qld
PostPosted: Sat 10 Sep 2011, 23:07    Post_subject:  
Thanks Joe

I guess you realised that I did deal with the leading pipe char delimiter, didn't explain it properly, that's why I used the -e option instead of making 2 sed calls, it's nearly twice as fast as sed only has to scan once.

Mick
_________________
keep the faith Cool ..
Back to top
View user's profile Send_private_message Visit_website 
8-bit


Joined: 03 Apr 2007
Posts: 3033
Location: Oregon
PostPosted: Sat 10 Sep 2011, 23:29    Post_subject:  
Wow, a few solutions.
01micko,
In my first example, the number before 'g' says to skip the first space and not change it.
The first space is actually there for alignment of the PID numbers in the first column.
The problem I had with my first script was that later in the file, the number column did not have any space preceding the number.
And my first attempt was basically removing spaces till one space remained between each entry on the line. The last sed I piped to said to skip the first space and change all following spaces in each line to pipe characters. And I used an escape character before the pipe character so it would be interpreted correctly.

Your sed solution is cleaner, but also seaside's suggestion to use awk is really good also.

I guess it shows just a few of the ways the output of "ps -A" can be read into a table.
If I really had wanted to clean things up, I could have used sed to remove the first line as it is a duplication of the column names.

Also, remember that I had never heard of any of these string manipulation commands before I was introduced to Puppy and programming.

My programming attempt that I made into a PET and uploaded, Floppy_Formatter-1.7.PET was actually a real mess as programming goes. And it started using a gtkdialog template from technosaurus.
If one looks at the script, you can see that.
But it went over well and I did a lot of testing before I ever uploaded the final version.
Back to top
View user's profile Send_private_message 
big_bass


Joined: 13 Aug 2007
Posts: 1736
PostPosted: Sat 10 Sep 2011, 23:34    Post_subject:  
.
_________________
slackware 14
Back to top
View user's profile Send_private_message 
8-bit


Joined: 03 Apr 2007
Posts: 3033
Location: Oregon
PostPosted: Sun 11 Sep 2011, 00:14    Post_subject:  
big_bass,
Sorry to leave you out!
Your solution is really short and it works too. Now if I only understood the part of your code that appears before the pipe to sed, I would be learning even more.

I have also never got around to trying your TXZ Pup. I am going to have to make some space on my hard drives available first as I have saved every PET, ISO, tar.gz, zip, post snips, etc to my drives.
I will just have to bite the bullet and copy a bunch of them to DVD data disk.

With Puppy versions and derivatives flying out for testing, I have given up trying to try them all.
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 3845
PostPosted: Sun 11 Sep 2011, 03:30    Post_subject:  
Quote:
ps -A > /tmp/tabtest.txt
cat /tmp/tabtest.txt | sed 's/ / /g' |sed 's/ / /g' | sed 's/ / /g' | sed 's/ /\|/2g'> /tmp/tab2test.txt


if you are looking to speed it up... and using only shell (well except for ps which is the whole purpose)

Code:
#!/bin/ash
dummyfunc(){
   busybox ps -A| while read LINE; do
      B=`echo $LINE`
      echo ${B// /|}
   done
}

A=`dummyfunc`
echo "${A}"
...


btw you should almost never need to cat into sed (its faster to let sed open the file) and you can combine sed statements like this
sed 's/ / /g ; s/ / /g ; ... ' (in this case tr -s " " would have done the trick too)
_________________
Puppy Web Desktop Now with pet packages - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
Bruce B


Joined: 18 May 2005
Posts: 10823
Location: The Peoples Republic of California
PostPosted: Sun 11 Sep 2011, 03:51    Post_subject:  
See explanation at bottom of post

ps -A | sed -e 's/^ *//' -e 's/ */ /g' -e 's/ /|/g'
outputs

PID|TTY|TIME|CMD
1|?|00:00:00|busybox
2|?|00:00:00|kthreadd
3|?|00:00:00|migration/0
4|?|00:00:03|ksoftirqd/0
5|?|00:00:01|events/0
6|?|00:00:00|khelper
7|?|00:00:00|async/mgr
8|?|00:00:00|sync_supers
9|?|00:00:00|bdi-default
10|?|00:00:00|kblockd/0
11|?|00:00:00|kacpid
12|?|00:00:00|kacpi_notify
13|?|00:00:00|kacpi_hotplug
14|?|00:00:00|ata/0
15|?|00:00:00|ata_aux
16|?|00:00:00|kseriod
18|?|00:00:01|kswapd0
19|?|00:00:00|aio/0
20|?|00:00:00|crypto/0
22|?|00:00:00|scsi_eh_0
23|?|00:00:00|scsi_eh_1
26|?|00:00:00|kpsmoused
87|?|00:00:00|ksuspend_usbd
88|?|00:00:00|khubd
153|?|00:00:00|usbhid_resumer
157|?|00:00:00|aufsd/0
161|?|00:00:00|scsi_eh_2
162|?|00:00:00|usb-storage
164|?|00:00:00|scsi_eh_3
165|?|00:00:00|usb-storage
167|?|00:00:00|scsi_eh_4
168|?|00:00:00|usb-storage
908|?|00:00:00|flush-8:0
1147|?|00:00:00|kjournald
1156|?|00:00:05|loop1
1226|?|00:00:00|kjournald
1297|?|00:00:00|loop0
1405|?|00:00:00|loop4
1412|?|00:00:00|loop5
1576|?|00:00:00|pup_event_backe
5685|?|00:00:00|pup_event_backe
5686|?|00:00:00|pup_event_backe
5715|?|00:00:00|dbus-daemon
5725|tty1|00:00:00|sh
5726|tty2|00:00:00|getty
5729|tty3|00:00:00|getty
5734|tty4|00:00:00|getty
5739|tty5|00:00:00|getty
5755|?|00:00:00|hiawatha
5757|tty1|00:00:00|xwin
5808|tty1|00:00:00|xinit
5809|tty6|00:08:38|X
5814|tty1|00:00:06|jwm
6020|tty1|00:00:00|dbus-launch
6021|?|00:00:00|dbus-daemon
6086|tty1|00:00:02|urxvt
6087|pts/0|00:00:00|bash
6097|?|00:00:00|kjournald
6252|?|00:00:00|loop2
6253|?|00:00:00|kjournald
6750|?|00:00:00|flush-7:1
7041|pts/0|00:00:00|urxvt
7042|pts/1|00:00:00|todo.sh
7043|pts/1|00:00:00|cat
7052|tty1|00:00:00|firefox
7056|tty1|00:00:00|run-mozilla.sh
7060|tty1|00:15:52|firefox-bin
7063|?|00:00:00|gconfd-2
7095|tty1|00:14:42|plugin-containe
7153|pts/0|00:00:00|dbus-launch
7154|?|00:00:00|dbus-daemon
11819|?|00:00:00|kjournald
12695|?|00:00:01|flush-1:6
12696|tty1|00:00:00|jwm|<defunct>
12697|tty1|00:00:00|urxvt
12698|pts/2|00:00:00|bash

ps -A | sed -e 's/^ *//' -e 's/^/ /' -e 's/ */ /g' -e 's/ /|/g'
outputs

|PID|TTY|TIME|CMD
|1|?|00:00:00|busybox
|2|?|00:00:00|kthreadd
|3|?|00:00:00|migration/0
|4|?|00:00:03|ksoftirqd/0
|5|?|00:00:01|events/0
|6|?|00:00:00|khelper
|7|?|00:00:00|async/mgr
|8|?|00:00:00|sync_supers
|9|?|00:00:00|bdi-default
|10|?|00:00:00|kblockd/0
|11|?|00:00:00|kacpid
|12|?|00:00:00|kacpi_notify
|13|?|00:00:00|kacpi_hotplug
|14|?|00:00:00|ata/0
|15|?|00:00:00|ata_aux
|16|?|00:00:00|kseriod
|18|?|00:00:01|kswapd0
|19|?|00:00:00|aio/0
|20|?|00:00:00|crypto/0
|22|?|00:00:00|scsi_eh_0
|23|?|00:00:00|scsi_eh_1
|26|?|00:00:00|kpsmoused
|87|?|00:00:00|ksuspend_usbd
|88|?|00:00:00|khubd
|153|?|00:00:00|usbhid_resumer
|157|?|00:00:00|aufsd/0
|161|?|00:00:00|scsi_eh_2
|162|?|00:00:00|usb-storage
|164|?|00:00:00|scsi_eh_3
|165|?|00:00:00|usb-storage
|167|?|00:00:00|scsi_eh_4
|168|?|00:00:00|usb-storage
|908|?|00:00:00|flush-8:0
|1147|?|00:00:00|kjournald
|1156|?|00:00:05|loop1
|1226|?|00:00:00|kjournald
|1297|?|00:00:00|loop0
|1405|?|00:00:00|loop4
|1412|?|00:00:00|loop5
|1576|?|00:00:00|pup_event_backe
|5685|?|00:00:00|pup_event_backe
|5686|?|00:00:00|pup_event_backe
|5715|?|00:00:00|dbus-daemon
|5725|tty1|00:00:00|sh
|5726|tty2|00:00:00|getty
|5729|tty3|00:00:00|getty
|5734|tty4|00:00:00|getty
|5739|tty5|00:00:00|getty
|5755|?|00:00:00|hiawatha
|5757|tty1|00:00:00|xwin
|5808|tty1|00:00:00|xinit
|5809|tty6|00:08:40|X
|5814|tty1|00:00:06|jwm
|6020|tty1|00:00:00|dbus-launch
|6021|?|00:00:00|dbus-daemon
|6086|tty1|00:00:02|urxvt
|6087|pts/0|00:00:00|bash
|6097|?|00:00:00|kjournald
|6252|?|00:00:00|loop2
|6253|?|00:00:00|kjournald
|6750|?|00:00:00|flush-7:1
|7041|pts/0|00:00:00|urxvt
|7042|pts/1|00:00:00|todo.sh
|7043|pts/1|00:00:00|cat
|7052|tty1|00:00:00|firefox
|7056|tty1|00:00:00|run-mozilla.sh
|7060|tty1|00:16:08|firefox-bin
|7063|?|00:00:00|gconfd-2
|7095|tty1|00:14:57|plugin-containe
|7153|pts/0|00:00:00|dbus-launch
|7154|?|00:00:00|dbus-daemon
|11819|?|00:00:00|kjournald
|12695|?|00:00:01|flush-1:6
|12696|tty1|00:00:00|jwm|<defunct>
|12697|tty1|00:00:00|urxvt


The logic explained below
ps -A | sed -e 's/^ *//' -e 's/^/ /' -e 's/ */ /g' -e 's/ /|/g'


ps -A | sed -e 's/^ *//'
removes leading spaces

-e 's/^/ /'
adds one leading space

-e 's/ */ /g'
changes two or more spaces to one space globably, but the forum posting doesn't show the two spaces, however they are in the command.

-e 's/ /|/g'
changes space to pipe globably
_________________
New! Puppy Linux Links Page
Back to top
View user's profile Send_private_message 
8-bit


Joined: 03 Apr 2007
Posts: 3033
Location: Oregon
PostPosted: Sun 11 Sep 2011, 04:18    Post_subject:  
Bruce B,
Thank you for the example solution with the explanation of the sed commands. One I was looking for to add a space to the beginning of a line I could not find in the sed.html instructions I have.
I have to admit this is and has been a genuine learning experience.
Back to top
View user's profile Send_private_message 
Bruce B


Joined: 18 May 2005
Posts: 10823
Location: The Peoples Republic of California
PostPosted: Sun 11 Sep 2011, 05:59    Post_subject:  
Thank you 8-bit

When people explain, it makes it so easy. Even when one is good a reading code, comments can really help.

~
_________________
New! Puppy Linux Links Page
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 3845
PostPosted: Sun 11 Sep 2011, 21:12    Post_subject:  
this got me thinking that we do this type of formatting a lot so I wrote a function that will do it for us for any command:

Code:
formatdb(){
if [ $1 ];then
   if [ "$1" = "-f" ]; then #reads the file(s) & pipe it through formatter
      shift
      while read LINE; do
         B=`echo $LINE`
         echo ${B// /|}
      done < $@
   else #run passed params as command & pipe it through formatter
      $@|while read LINE; do
         B=`echo $LINE` #echo removes excess white spaces
         echo ${B// /|} #replace each space as |
      done
   fi
else #this will handle piped data
   while read LINE; do
      B=`echo $LINE`
      echo ${B// /|}
   done
fi
}

usage:

formatdb <command>
or
<command> |formatdb |<command>
or
formatdb -f /path/to/file


Code:

#some test cases ... make sure to have the above function in the script
formatdb ps -a >/tmp/tmpps-a

echo using a temp file

cat /tmp/tmpps-a

A=`echo "a  b  c  d
e  f  g  h  i
jkl mno      p" |formatdb`


ps -A >/tmp/tmpps-A

formatdb -f /tmp/tmpps-A

echo "using a variable

${A}

"

_________________
Puppy Web Desktop Now with pet packages - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Off-Topic Area » Programming
Jump to:  


Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.0922s ][ Queries: 12 (0.0087s) ][ GZIP on ]