Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 23 Feb 2018, 12:46
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
The Problem with Building Apps in 64-bit “Ubuntus”
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [11 Posts]  
Author Message
mikeslr


Joined: 16 Jun 2008
Posts: 2199
Location: 500 seconds from Sol

PostPosted: Sun 16 Jul 2017, 12:00    Post subject:  The Problem with Building Apps in 64-bit “Ubuntus”
Subject description: Suggested Solutions -- Experts needed
 

As you may know, from time to time I build applications for Tahrpup64 and XenialPup64. See, for example, http://www.murga-linux.com/puppy/viewtopic.php?p=940751#940751

I’ve also built apps for their 32-bit siblings. My technique for 32-bits is pretty straight-forward:

1. Install LazyPuppy’s PaDS plus unrpm-all from this thread: http://www.murga-linux.com/puppy/viewtopic.php?p=940751#940751
1. Use PPM to download (but not install) the application’s .debs and dependencies.
2. Move the downloaded .debs to a folder bearing the name and description of the application to be built, e.g. MyApp_xen64-1.2.3 identifying it as “MyApp” for XenialPup64 (app version) 1.2.3.
3. Right-Click the folder and select Combine to SFS –a component of PaDS-- which will create an SFS in root; in the above example MyApp_xen64-1.2.3.
4. (Optional: to create Pet) Create a folder with the same name. Mount the SFS. Drag its files into the newly created folder. Open a terminal in the folder’s parent. Type ‘dir2pet NAME_OF_FOLDER.

For 32-bit Applications, the procedure works reasonably well. Sometimes a dependency is missed. But then –unless the application depends on python-- ldd will identify the missing lib sending me on a “lib chase”, adding it to the named folder and using “Combine to SFS” to rebuild the SFS.

But for 64-bit “Ubuntus*”, but NOT Slackos, the procedure is not that simple. As far as I can tell, in developing the 64-bit Trusty Tahr (maybe earlier) Ubuntu* altered the operating system’s structure to enable it to handle both 32-bit and 64-bit libraries (sharing the same name). The new structure includes three folders: /lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu and /usr/local/lib/x86_64-linux-gnu. Applications/(debs) created to run under Ubuntus locate 64-bit libs to those folders and Ubuntu OSes find and make use of those libs. Puppies built from Ubuntu binaries can’t find the libs located in those folders.

* I don’t know whether Ubuntu independently developed that structural change or copied it from debian. I don’t currently have a 64-bit Puppy based on debian, nor know of one. But my examination of 64-bit “DebianDogs” reveal the same folders which, I note, doesn’t pose a problem to them. Nor does their existence under 64-bit XenialDogs.

Slackware apparently took a slightly different approach to managing multi-architecture OSes. I recently built xfce4_panel for both Slacko64 and XenialPup64. PPM for both missed xfce4-session, which I guess didn’t qualify as a dependency. Xfce4_panel ran without its presence. But any configurations you made to the panel would be lost on re-boot. That it wasn’t a dependency was understandable. The creators of xfce4_panel expected it to be used with xfce as window-manager, and for xfce xfce-session is a dependency.

Currently, my work-around has been to use the optional 4th Step above, then a lot of grunt work: open two windows into the WorkFolder and manually move the libs in /lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu and /usr/local/lib/x86_64-linux-gnu to /lib, /usr/lib and /usr/local/lib, respectively.

Edit: I've just notice that when folders are moved from the x86_64-linux-gnu folders, symlinks may be broken. So an inspection is necessary, and the broken symlinks replaced with new links to the intended files/folders.

I can think of four better solutions to this problem, none of which I have the expertise to implement.

The best solution might be for those working on Woof to actually examine and determine how the “real” debians and Ubuntus and the “DebianDogs” implement multi-architexture and revise Woof accordingly. One of the principal motivations for creating Woof in the first place was to enable Puppies to make use of applications created by other distros.

Alternatively, as the current structure generated by Woof does work when building a Slacko from Slackware, perhaps only two changes would suffice when Ubuntu/debian provides the source: (1) code within woof which would automatically move libs from the problem folders to wherever they would be located in the “Slackware” structure; and (2) [analogous to how Puppy Package Manager used to work, separating out Dev and Doc files] modifying PPM in “Ubuntu/debians” so that when debs are obtained the files in the problem folders would automatically be moved to folders that Puppies can use.

However, both of the above methods may involve more work than the few Devs developing Woof are currently able to undertake. The third “fix” would be a modification of how PaDS works. Either if a deb is detected, or a check-box filled in when the user knows debs are involved, rather than just unpacking the debs and copying their contents to the corresponding folders in PaDS’s WorkFolder, the contents of problem folders would be copied to folders that Puppies can use.

The simplest solution would be a script/pet that would do the ‘grunt work’: Mount the ‘Interim’ SFS, and copy its files into a WorkFolder, moving those files from the problem folders to folders that Puppies can use, then packaging the WorkFolder as an SFS or Pet.

mikesLr

Last edited by mikeslr on Tue 19 Sep 2017, 19:41; edited 2 times in total
Back to top
View user's profile Send private message 
DPUP5520

Joined: 16 Feb 2011
Posts: 813

PostPosted: Sun 16 Jul 2017, 19:09    Post subject:  

Why aren't you jus compiling from source? It's just as easy if not easier and a lot of times (if not almost always) the programs in the actual buntu repos are older versions than what are currently available.
_________________
PupRescue 2.5
Puppy Crypt 528
Back to top
View user's profile Send private message 
mikeslr


Joined: 16 Jun 2008
Posts: 2199
Location: 500 seconds from Sol

PostPosted: Sun 16 Jul 2017, 20:43    Post subject:  

Hi DPUP5520,

DPUP5520 wrote:
Why aren't you just compiling from source? It's just as easy if not easier and a lot of times (if not almost always) the programs in the actual buntu repos are older versions than what are currently available.


Compiling from source is a skill set I don't have and frankly am intimidated by. It may be easy to compile simple applications, or those which "are tried and true" by merely following in the instructions found with the source package. To a large extent, my impression was that those applications are provided by the Devs who create Puppies and by others who make their end products available.

Beyond that I thought to compile from source you actually had to know in advance where you wanted files to actually end up in the finished application. In other words, you have to know how all the bits and pieces which make up a Linux OS tie together, something I also don't know.

And at what point do you start compiling from source. 56 debs were used to build the xfce4_panel for Xenialpup64. It was my impression that it was each deb (and maybe --not sure-- their dependent libraries) which is the end product of a separate compile. I may be mistaken but I have the impression that during the compile process you can provide the compiler with instructions where each set of libraries is to be located in the finished product. xfce4_panel uses over 100 libraries. I don't know which deb used included which libraries. I'd have to know that before initiating each of the 56 compiles.

And while you are correct in that the Ubuntu/debian repos often do not have the latest versions of application, the latest debs published by an application's creator are compiled to run under Ubuntu/debian OSes and consequently include libraries at those locations where Ubuntu/debian OSes would expect to find them. [On occasion, I've built applications from the debs available via web-browser download from the original publishers; resorting to Ubuntu/debian or other repos only for "missing libs"].

Lastly, compiling from source should be a method of last resort -- for that application you and only a few others would want which isn't otherwise available or needs to be specially constructed to function under your hardware. There's just nothing efficient about many people having to compile from source an application which many people would want especially when compiled binaries are easily available. Compiling from source may well be a relic of a time when internet speed limitations took longer to download the text file instructions (the source) as it takes today to download the deb.

As I said in my original post, I built xfce4_panel for both Slacko64 and Xenialpup64. With Slacko64, It took about 6 minutes to download the 44 txz's and about 2 minutes for PaDS to produce a finished product. Add another 10 minutes to discover that xfce-session (not identified as a dependency, but needed if xfce4-panel was to preserve configurations when Xfce isn't the Window-manager) was necessary, download it and re-run PaDS. How is compiling 44 binaries easier than that?

If you're going to spend the time to build a complex application which many people may want, why not make the finished product available to them. I intend to make both versions available through my FREE account at mediafire. But frankly, for Slacko64 that isn't really necessary. Anyone who wants it can just spend 8 minutes with PPM and PaDS. With XenialPup64 on the other hand, add about 15 minutes of grunt work ONCE YOU KNOW WHY PPM + PaDS DIDN'T PRODUCE A WORKING APPLICATION. Letting people know why was one of the reasons for my post.

mikesLr
Back to top
View user's profile Send private message 
DPUP5520

Joined: 16 Feb 2011
Posts: 813

PostPosted: Mon 17 Jul 2017, 08:06    Post subject:  

All fair points, and I didn't realize nearly quite how large the entire package for the xfce4_panel is, that would be quite a large undertaking.
_________________
PupRescue 2.5
Puppy Crypt 528
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 3420
Location: King's Lynn, UK.

PostPosted: Mon 17 Jul 2017, 08:10    Post subject:  

@DPUP5520:-

I suspect Mike's more like me; happier manually 'digging around' in the file system than he is doing stuff the 'pure' geek way; via the terminal.

I'm the same; I've been trying for 4 years now, but I just cannot get my head round CLI-stuff. But manually moving files around to where I want them, creating/deleting stuff as and where necessary, making/breaking sym-links, setting up permissions, etc.....I can whizz round in a fraction of the time and achieve what I want to do, far quicker than trying to suss out where I've missed (or added) an extra space, or where I've mis-spelt/mis-typed something slightly wrong in the terminal.

We're not all gifted with the same abilities; I'm more of a 'mechanic' than a brain surgeon.....and I suspect Mike is, too..! Very Happy

-----------------------------------------------------

@Mikeslr:-

Back on topic. I agree with you about all this blasted 'multi-architecture' stuff. I like the way that Phil has sym-linked lib64 into lib, and /usr/lib64/into /usr/lib in both Tahr and Xenial. Makes things quite a bit easier. Mick doesn't do that in Slacko, since I guess it's not part of the Slackware 'structure'.

I don't consider myself to be much of a 'packager', TBH; that's why I tend to stick with the easier stuff, where I know the capacity for cock-ups is much reduced! Laughing

And, too (like yourself), I initially make stuff for myself. If it works as expected, I'm more than happy to share.


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------


Last edited by Mike Walsh on Mon 17 Jul 2017, 12:27; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
mikeslr


Joined: 16 Jun 2008
Posts: 2199
Location: 500 seconds from Sol

PostPosted: Mon 17 Jul 2017, 12:25    Post subject:  

Hi DPUP5520, Hi Mike,

I think Mike has identified something I know, but always forget. Everyone's mind works somewhat differently. What some people find easy, I find difficult.

I have a really poor memory for details. Working with the command line actually involves remembering all the details of a formula and, if necessary, changing them.

My memory is very visually oriented. While in college, over fifty years ago, I wrote a paper on a particular subject in which I referenced information in a book from my college's library. I don't remember what was written, the name of the book, or who wrote it. But I do remember on which floor, at which location, on which self of the stack, and roughly in which part of the book the information was found.

When I come across some code someone posted on the forum which I think may be useful, I'll cut and paste it into geany or LibreWriter and save it to /mnt/home/my_stuff/Notes/xxx_SUBJECT. I won't remember the code. I'll remember where I put it. For me, that works fine for filling an occasional need. It would be torture if I had to constantly stop and reference my notes in order to get something done.

So I write about structures, how I visualized them, and how I manipulated them, picturing as I write, how something was moved from here to there >motion pictures.

When, in my prior post, I wrote "many people having to compile from source" I had visualized something not unlike that of a sweat-shop, filled with people, all doing the same tasks, striving to produce identical items. I must admit, however, that now that I've thought about it, the same picture exists if the "many people" were "manipulating" rather than compiling. My evaluation was biased by my inability to do what others may find easy.

So DPUP5520, you're probably right. If you have a decent memory for details, compiling from source --even if it isn't as efficient as stealing someone else's work Smile -- may be easier and certainly would be more gratifying.

mikesLr
Back to top
View user's profile Send private message 
LateAdopter

Joined: 27 May 2011
Posts: 285
Location: Reading UK

PostPosted: Mon 17 Jul 2017, 13:35    Post subject:  

Hello Mike

I'm glad I'm not the only one who doesn't understand how multiarch is meant to work. The documentation tells packagers what to do but doesn't explain what the result is.
From the debian multiarch howto:
Quote:
In general you can have libraries of more than one architecture installed together and applications from one architecture or another installed as alternatives. Note that it does not enable multiple architecture versions of applications to be installed simultaneously.

I think this means that there is no need to keep 32 and 64 bit libs in a separate directory, its more about resolving dependencies.

I have not found any mention of the backwards symlink solution, but it must originate from debian/ubuntu because dpkg needs to know that it should follow the symlink to the parent directory and not convert the symlink to a directory. With this solution 32bit and 64bit libs end up in the same directory and don't conflict, I think/hope.

The multiarch documentation says the the triplet directories (their jargon) need to be on the path for the libs to be found.

So there seem to be two options that can't coexist:
a) a backwards symlink that puts the libs directly in the /lib directories when you install from a .deb. But you have to manually move the files up a level when making an SFS

b) have real multiarch directories and put them on the path. I don't know whether you can get an SFS to add the multiarch directories to the path, but that would eliminate the need to move the files manually.

It is a one way change but wouldn't cause problems for libs already in the /lib directory except that you might have duplicates.
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 3420
Location: King's Lynn, UK.

PostPosted: Mon 17 Jul 2017, 14:13    Post subject:  

Hi, LateAdopter.

Ah, I'm glad somebody else agrees with me!

I guess its summat we're gonna be stuck with from now on.....especially with the general move to 64-bit only computing. (That's why I'm glad Puppy still supports - and produces new versions of - 32-bit OSs. Keeps my 15-yr old Dell Inspiron P4 lappie alive and kicking.....and far more productive than could ever be expected with MyCrudSoft's offerings..!)

It's also why I tend to produce all my packages to sit in /opt, wherever I can get away with it. It's a nice 'catch-all' directory, that neatly sidesteps & avoids all the aggravations with this multi-arch stuff. I know /usr/lib is traditionally where a lot of stuff is supposed to go.....but I've never been much of a traditionalist..!

Nice to see another UK 'Puppian'. We Brits need to stick together! Laughing


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
mikeslr


Joined: 16 Jun 2008
Posts: 2199
Location: 500 seconds from Sol

PostPosted: Mon 17 Jul 2017, 14:59    Post subject:  

Hi LateAdopter,

Thanks for getting this thread back on track. I wasn't aware of the documentation you cited. Here's what I do know from experience. When tracking down Ubuntu libs via https://packages.ubuntu.com/ often you'll get to a point at which you're offered the choice between 32-bit and 64-bit packages (and others) or sometime the package is for "all" architectures. As far as I know, the package for the 32-bit bears the same name as the package for the 64-bit, except that the former will include i386 (or 4 or 586) while the latter will include amd64 in their respective labels. I haven't pursued that far enough to know whether the files within each deb bore distinct names. If they do, both files will happily exist within the same folder. If they don't, the last file installed will overwrite its predecessor.

When a file with the wrong architecture employed, the application will not run and ldd will report that the wrong "ELF class" was attempted to be used. [I would think it rare that you would ever need both the 32 and 64 bit version of a file.] The most likely cause is "human error" --in searching you followed the wrong path; less likely, PPM made a mistake. But sometimes --especially when trying to build using the latest --not a Ubuntu repo-- version of an application, the necessary lib isn't in a Ubuntu repo at all. So, you've expanded your search to pkg.org and others, used what appeared to be the most likely package --albeit built for Opensuse or whatever-- and ended up with an ELF error. At the point at which a package isn't available in the repo of the distro to which your Puppy has 'binary compatibility', the ability to compile just the package you need would really be a useful.

My inability to do that surfaced a couple of days ago. Rather impressed by how well Slacko64 worked, I set out to see whether the applications I run as external files and AppImages could be run under it. Many worked, but a couple required Qt5. Maybe I missed them, but I couldn't find any Qt5 packages for Slackware or its spinoffs.

But, getting back on point, "have real multiarch directories and put them on the path." Emphasis supplied. So, in addition to the 'solutions' I mentioned, that would be another. But as with those I previously mentioned, I don't have the ability to accomplish that either.

mikesLr

p.s. Just read Mike Walsh's post above. That's also been my experience. But I don't know why. When --except for a symlink on the path to the application's binary-- an entire application including the folders containing its libraries-- is located in /opt or /mnt/home/SOMEPLACE 'Ubuntu' Puppies don't have any difficulty finding the libs in, for example, /opt/my-application/usr/lib/x86_64-linux-gnu. So, another possible solution. But this one also involves moving stuff, albeit perhaps easier: the entire structure to /opt with one move. But is, for example, /opt/my-application/etc or /opt/my-application/root a configuration Puppies would know how to handle?

p.p.s. Might not be as simple as that. Just tried this: Still having the original debs available in the folder I used to create the preliminary SFS, I used PaDS to create a new SFS. Then I created a blank folder which included the term OPT in it so that I could distinguish it from the working version. Mounted the new preliminary SFS and copied its files into a folder named opt within a folder named xfce4_panel. Adjacent to opt, I created a folder named root, within that one named Startup. xfce4_Panel has no menu entries, but to run before it xfce4-session and then it must be started. So in /root/Startup I placed a script which would call both from their locations in /opt. dir2sfs the xfce4...opt folder creating a xfce4_panel..opt.sfs. Unloaded my working SFS and loaded the opt.sfs. Restarted X. Nothing happened. Typed xfce4-session in terminal and received the message that a library was missing. So type ldd followed by the full path (opt/etc.) and name of xfce4-session and received the message that libraries I know are in /opt/...etc/usr/lib were missing.

But I may have done something wrong. Still worth exploring by others.
Back to top
View user's profile Send private message 
LateAdopter

Joined: 27 May 2011
Posts: 285
Location: Reading UK

PostPosted: Tue 18 Jul 2017, 03:56    Post subject:  

Hello mikeslr

You can edit the /etc/ld.so.conf file to add the x86_64 paths and then run ldconfig each time you load an SFS.
666philb already put the i386 paths there and says that you need to run ldconfig when installing the 32bit libs SFS.
That may be why you have better results with 32bit than 64bit

EDIT
From http://www.debian.org/doc/debian-policy/ch-sharedlibs.html
---
8.1.1. ldconfig

Any package installing shared libraries in one of the default library directories of the dynamic linker (which are currently /usr/lib and /lib) or a directory that is listed in /etc/ld.so.conf [60] must use ldconfig to update the shared library system.

The package maintainer scripts must only call ldconfig under these circumstances:

When the postinst script is run with a first argument of configure, the script must call ldconfig, and may optionally invoke ldconfig at other times.

When the postrm script is run with a first argument of remove, the script should call ldconfig.
----

There must be a way to get AUFS to execute ldconfig, but I can't find any instructions for this. You can put ldconfig in rc.local so that it is always run during boot-up, but it will slow boot a bit.

EDIT 2

It looks as though rc.update runs ldconfig anyway, during boot, so maybe adding the paths to ld.so.conf is all that is necessary. However I don't know whether that always happens.
Back to top
View user's profile Send private message 
mikeslr


Joined: 16 Jun 2008
Posts: 2199
Location: 500 seconds from Sol

PostPosted: Tue 18 Jul 2017, 12:03    Post subject: MultiArch -- editing /etc/ld.so.conf  

Hi LateAdopter,

Very nice info regarding id.so.conf. But perhaps --hopefully-- you've bought into a problem before it exists. Maybe all that would be necessary is to "inform" 'Ubuntu' Puppies that the directories it couldn't find do exist, and where, by editing id.so.conf to specify them. AFAIK, id.so.conf is part of the system, read-in and established on boot-up. So it may not be necessary to re-run it each time an installed pet is opened or an SFS is loaded and/or opened.

We'll see. I can test. Attached is a pet which adds the problem directories to id.so.conf. After re-building xfce4_panel from the original debs again, I'll install the pet and load xfce4-panel into 're-virginized' Xenialpup64; then see if the panel will run, and if ldd continues to report libs 'not found'. I won't structure the panel to run from /opt [which would add a layer of complexity] -- just leave it as the Ubuntu devs structured it to run under Xenial Xerus.

But before I take the time to do that I really want to publish the xfce4_panels that I've had sitting around for over a week. It's taken much longer to provide guidance than to create them.

Edit: Just re-read your post which now has Edit 2. Seems we've developed the same hypothesis, 'though --as to be expected-- you're reasoning provides more details.


mikesLr
Ubuntu_MultiArch-0.1.pet
Description  Hopefully make problem folders usable
pet

 Download 
Filename  Ubuntu_MultiArch-0.1.pet 
Filesize  401 Bytes 
Downloaded  22 Time(s) 
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [11 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 1.9946s ][ Queries: 13 (1.7919s) ][ GZIP on ]