Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 22 Nov 2017, 05:44
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
speeding up scripts
Post new topic   Reply to topic View previous topic :: View next topic
Page 4 of 4 [55 Posts]   Goto page: Previous 1, 2, 3, 4
Author Message
wiak

Joined: 11 Dec 2007
Posts: 372
Location: not Bulgaria

PostPosted: Thu 07 Sep 2017, 16:20    Post subject:  

Yes, sed can produce some amazingly efficient fast results. So much code with multiple while read loops and pipes via cut and grep which a single sed command can do (via multiple commands to the single called sed instance). There are also many instances of multiple sed calls being then re-piped into sed again and sometimes again, when a single sed call (with multiple sed command instructions can do).

Part of the problem is that many just know simple basics of sed (mainly just sed 's\\\;'. It can do a lot more than that and all on the one line... but a bit of a learning curve understanding all the things it can do and how to do them altogether in one sed call instance.

wiak
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2548
Location: UK

PostPosted: Thu 07 Sep 2017, 17:30    Post subject:  

i only know what you guys have shared, but this seems legit:

David Butcher: Speeding Up Your UNIX Shell Scripts
http://www.los-gatos.ca.us/davidbu/faster_sh.html

And i am guilty of piping sed to sed, among a thousand other shell-related sins..

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 372
Location: not Bulgaria

PostPosted: Thu 07 Sep 2017, 23:54    Post subject:  

sc0ttman wrote:

And i am guilty of piping sed to sed, among a thousand other shell-related sins..


Yes, me too, but doesn't matter anyway as long as not in some long loop type situation - sometimes/often it's more important the code is easy to read. Nice link about shell code speed up by the way.

I'm pretty sure jlst is constantly tuning woof-CE code to gradually speed it up, but there is a lot of code to work though - plenty of speed up possible I'm sure.

wiak
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2548
Location: UK

PostPosted: Fri 08 Sep 2017, 07:52    Post subject:  

Yeah, that link it quite good.. Some stuff I never even considered..

I like the idea of replacing

Code:
[ "$var" = 'foo' -a "$var2" = 'bar' ] && echo blah


with

Code:
case "$var1$var2" in
  foobar) echo blah ;;
esac


Although readability is not great (imho)..

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2548
Location: UK

PostPosted: Fri 08 Sep 2017, 07:54    Post subject:  

Is this

Code:
if "$var" = 'foo'; then



faster or (different at all) than this:

Code:
if [ "$var" = 'foo']; then


?

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
MochiMoppel


Joined: 26 Jan 2011
Posts: 1392
Location: Japan

PostPosted: Fri 08 Sep 2017, 08:23    Post subject:  

sc0ttman wrote:
Is this
Code:
if "$var" = 'foo'; then

faster or (different at all) than this:
Code:
if [ "$var" = 'foo']; then
?
Both are incorrect. You mean which of the error messages you will receive is faster?
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2548
Location: UK

PostPosted: Fri 08 Sep 2017, 09:03    Post subject:  

MochiMoppel wrote:
sc0ttman wrote:
Is this
Code:
if "$var" = 'foo'; then

faster or (different at all) than this:
Code:
if [ "$var" = 'foo']; then
?
Both are incorrect. You mean which of the error messages you will receive is faster?

lol yeah fine, add the fixes mentally Rolling Eyes I think u know what I mean..

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
MochiMoppel


Joined: 26 Jan 2011
Posts: 1392
Location: Japan

PostPosted: Fri 08 Sep 2017, 10:50    Post subject:  

if test "$var" = 'foo' ?
if [[ "$var" = 'foo' ]] ?
Wait ... where is my crystal ball?
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 11170
Location: Gatineau (Qc), Canada

PostPosted: Fri 08 Sep 2017, 12:45    Post subject:  

Hi scotman.

In the 1st issue of the Puppy Linux Newsletter, January of this year, I have
explained a number of tricks I have used to very rapidly (+/- 1 second)
create a +/- 16Kb wmx menu.

It boils down to :
-- use ash instead of bash if appropriate (ash does not have all of bash's
string manipulation capacities; bash may be faster than ash in such cases.)

-- use internal bash or ash commands as much as possible

-- take advantage of bash's fantastic string manipulation capacities: they
are generally faster than using awk for the same result


-- sort lists and items that need sorting before processing them.
Remember that somewhere in a computer, the alphabetical and numerical
orders are still built-in. It is not just an historical thing. For example, if you
don't respect alphabetical-order processing when you can, expect to lose
precious time.

-- use the case...esac conditions structure whenever possible

-- use LC_ALL=C before any important non-linguistic processing and
know where and when to release it with LC_ALL="" -- at the proper place
in the processing
. Otherwise, you'll get junk results.

(By "non-linguistic processing", I mean any processing not based on
human language.)

With LC_ALL=C, the LANG variable remains untouched. You're only
suspending it for the time being.

LC_ALL=C suspends utf-8 and makes the utf-8 bites available for general
processing, not just for language. So this multiplies the speed of your script
by a factor of 2 to 4. "Your script gets the whole boulevard to itself," in a
manner of speaking.

This will make a bash script approach C processing speed. But you have to
know when to release it, especially if you want to integrate human
languages other than bare-bones English. (No special characters
allowed.)

-- time the loops for speed. A < while read line;do > may be faster or slower
than a < for i bla ble bli;do > loop, depending on the material.

-- avoid writing to disk as much as possible. Prefer string manipulation. If
you need to write to disk, write in the same directory as the script. You'll
save only a millisecond, but they add up when using a loop.

-- use the most efficient logic for the problem at hand. You may gain speed
by changing the order of the "processing steps". This you learn by what I
call "living with the problem" for a little -- or a long! -- while.

Finally, please note that the above are "working notes" derived from my
personal experience. I know they work, but I'm just a "ground-hog" Wink :
some "eagle" with general perspective will have to provide the theory of it.

IHTH.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4756
Location: Kingwood, TX

PostPosted: Fri 15 Sep 2017, 00:39    Post subject:  

MochiMoppel wrote:
Even when reading smaller files sed tends to be faster.

Not if you are running sed from inside a script, using a non-bash shell with LANG=C
If memory serves me, you have made several localization contributions, so I am guessing you bothered to set $LANG and probably have /bin/sh as a link to bash since it minimizes problems with all the bashisms that riddle Puppy scripts.

I did a bunch of testing to figure out when to use read loops vs. sed, grep and awk and IIRC on average it came out to just short of 100 lines on average (depending on the shell)

FWIW, Those timing values are quite suspect because of all the echoes, considering I can process all the desktop files in /usr/share/applications/ to generate my jwmrc file in about the same amount of time... but then again, I build a large string and print it once to a file instead of doing it one echo at a time to stdout ... I guess that's another tip ... printing to console/terminal is slow, so batch them up if possible

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 4 of 4 [55 Posts]   Goto page: Previous 1, 2, 3, 4
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0827s ][ Queries: 12 (0.0077s) ][ GZIP on ]