Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 24 Oct 2014, 19:56
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
system optimization techniques
Moderators: Flash, Ian, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Author Message
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 18:03    Post_subject:  system optimization techniques  

Wondering whether or not to recompile a program in your puplet to eliminate large shared libs that are only used once or twice

Code:
for x in ${PATH//://* }/* ; do [ -x "$x" ] && objdump -x $x 2>/dev/null & done|grep NEEDED|sort |uniq -c |sort -n >binaudit


this will tell you how many times each library is needed (if it isn't on the list, it can likely just be removed --- not always though, need to remember dlopen and/or plugins may use it too ... in "module" mode)

now open binaudit in your text editor and get to work, starting with the ones that are needed only once ... you can build them in statically

The ones that are used the most would benefit the most tools by improving their compiles (I will discuss that later)

I'll go over optimization techniques soon.

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:30    Post_subject:  

Reserved compile techniques.

In case you ever wondered whether or not to set certain flags, it is helpful to know which ones already are set. This can be done by:

Code:
export CFLAGS="your cflags"
for x in x; do
gcc -Q --help=optimizers
gcc -Q --help=target
gcc -Q --help=warnings
gcc -Q --help=undocumented
done >gccflags


CFLAGS
-Os (optimize for speed unless it will increase size)
-finline-small-functions (inline if it is smaller to inline than call it)
-finline-functions-called-once -fearly-inline (requires gcc update)
-ffunction-sections -fdata-sections with -Wl,--gc-sections LDFLAG
pdf explanation by Denys Vlasenko (busybox maintainer)
(allows compiler to remove functions and data that aren't actually used even if they are in the same object file as one that is)
-fno-unwind-tables -fno-asynchronous-unwind-tables (don't include unnecessary unwind info)
Rich Felker wrote:
By default, modern GCC generates DWARF2 debug/unwind tables in the .eh_frame section of the object files/binaries. This adds significant bloat (as much as 15%) to the size of the busybox binary, including the portion mapped/loaded into memory at runtime (possibly a big issue for NOMMU targets), and the section is not strippable with the strip command due to being part of the loaded program text. I've since done some further checking - both testing and asking the GCC developers about it - and it seems the solution is to add to the CFLAGS -fno-unwind-tables and -fno-asynchronous-unwind-tables. If debugging is disabled, this will prevent GCC from outputting DWARF2 tables entirely. But since busybox builds with -g by default, the interesting case is what happens then. I originally thought these options would break debugging, but they don't; instead, they tell GCC to output the DWARF2 tables in the .debug_frame section instead of the newish .eh_frame section (used for exception handling). With these options added, busybox_unstripped is still fully debuggable, and the final busybox binary loses the 15% bloat factor from the DWARF2 tables.



LDFLAGS
--gc-sections (see above)
--as-needed (don't explicity link all libraries passed on the command line - try to see if any are useless ... all the locations of functions accessed indirectly via direct dependencies get stored in the binary when they can be loaded at runtime much more effectively - assuming the libraries get updated periodically, the locations will all be wrong and take much much longer to load)
NOTE: gcc should have a --warn-as-needed to inform developers which libraries are unneeded when their crappy build scripts are broken

_________________
Web Programming - Pet Packaging 100 & 101

Edited_times_total
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:31    Post_subject:  

Shrinking files.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:32    Post_subject:  

Boot speed.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:41    Post_subject:  

X
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:42    Post_subject:  

Alternative apps/libs
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:43    Post_subject:  

Kernel configs
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 16 Mar 2012, 19:44    Post_subject:  

Portability
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Thu 10 May 2012, 17:35    Post_subject:  

Hi technosaurus; I wondered if this could do a statistical analysis of lib. usage in common apps.
I did a sudo analysis myself that showed a small percentage of libs. are actually in common use.

This is all about rules for which libs. to include in static builds.
Once the libs. are identified, then scripts for statically building them.
I don`t know if a build script "scrap" can be used many times like this.
Back to top
View user's profile Send_private_message 
scsijon

Joined: 23 May 2007
Posts: 1047
Location: the australian mallee

PostPosted: Thu 23 Aug 2012, 22:27    Post_subject:  

Technosaurus, I wonder if a common lib could be built, something on the Busybox basis?

It should shrink the overall size a bit!

_________________
Puppy T290 - BarryK's Racy but with T2 Version 9.0 is under development.
http://www.murga-linux.com/puppy/viewtopic.php?t=94575
Back to top
View user's profile Send_private_message Visit_website 
disciple

Joined: 20 May 2006
Posts: 6449
Location: Auckland, New Zealand

PostPosted: Fri 24 Aug 2012, 02:01    Post_subject:  

FWIW there was a project to build GTK and X and stuff all into one binary... but it seems to be dead.
_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Fri 24 Aug 2012, 11:10    Post_subject:  

Goingnuts and I have done several x11, xaw and gtk1 apps as multicall binaries.... Gtk2 apps tend to use plugins - thus requiring significant patches... However there is an in between method where you mix static and dynamic libs based on analysis of dependencies. It gets complicated when you have to build pic versions of static libs so they can be statically linked into plugins (like png for imlib) and further complicated if you have another app that links to those libs directly (such as jwm with png support) requires changing the build to link to the imlib plugins for jwm. This is fine if you are doing a 1 time build of an embedded system with limited resources, but a royal pain to maintain for a distro. Thus the reason we hand picked a few compatible apps (sometimes even specific versions) and limited them to the minimum functionality needed to use/build/rebuild a system either from scratch using any distro's packages or so that if your whole shared library system gets corrupted, you can still recover comfortably with an intact desktop environment and still do basic everyday tasks like editing text, browsing the web or your file system, answering email and of course wasting time even if your libc gets corrupted.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sat 25 Aug 2012, 16:32    Post_subject:  

I see no point in BusyBox in Squash file based O.S.s.
It just ends up being double compressed, usually with little improvement.

And here at Puppy we know BusyBox commands just don`t measure up.
Thought it could be compiled with full spec. commands...

BusyBox is a good idea for loose file systems, but not Squash file ones.
Back to top
View user's profile Send_private_message 
disciple

Joined: 20 May 2006
Posts: 6449
Location: Auckland, New Zealand

PostPosted: Sat 25 Aug 2012, 17:53    Post_subject:  

Quote:
It just ends up being double compressed, usually with little improvement.

What do you mean? Busybox isn't about compression, is it?

_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send_private_message 
technosaurus


Joined: 18 May 2008
Posts: 4353

PostPosted: Sat 25 Aug 2012, 20:21    Post_subject:  

The advantage of statically compiling a multicall binary is that unused functions get compiled out. Busybox also has the capacity to "prefer applets" and call them as a function or by forking itself (both are much faster than running a separate binary... even moreso if that binary has to find and load shared libs) the only compression is the help text. Really it wouldn't hurt much to put a full busybox in the initramfs and copy it over. The only reason busybox applets sometimes fail is because the scripts weren't specifically written for it (tinycore scripts all work because they were) But I did notice that upx slows busybox down and the extra compression is negligible.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0791s ][ Queries: 12 (0.0080s) ][ GZIP on ]