Puppy for parallel computation using openMPI

A home for all kinds of Puppy related projects
Message
Author
Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#16 Post by Everitt »

I'm going to stick to 2.16 while I get a few things set up properly. I had tried 2.17, but nobody seems to be able to make the humungous initrd.gz boot propely.
As for existing code, I don't really know. OpenMPI has support for C[++] and fortran. Java should be possible, there are open-source projects avaliable, but the trouble is that to use java you need the java runtime enviroment. According to the Java Parallel Processing Framework FAQ you'd need a good 512MB of RAM. Not very puppy huh?
For comparision, a standard netboot of 2.16 needs somewhere between 128MB and 192MB (192 works, 128 doesn't). I plan on stripping a lot out for the nodes, so that should fall a bit too.

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#17 Post by Everitt »

I spent most of today putting together the node version. I've got it working, but by the look of it there is a reasonable amount in the root filesystem that I don't need/want, so cleaning that out is going to take a while. :(
Anyway, free reports a total of 63568KB used (62MB)

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#18 Post by cb88 »

that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

User avatar
bobwrit
Posts: 283
Joined: Mon 12 Mar 2007, 23:33
Contact:

#19 Post by bobwrit »

This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. :wink:
I need help with my forum. [b][u]LINK:[/u][/b][url]http://www.programers.co.nr/[/url]
[url]http://www.freewebs.com/programm/iframe.html[/url] is my gateway page...

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#20 Post by Everitt »

cb88 wrote:that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out
Yep, at the moment it uses just over 62MB of RAM. Hopefully I should be able to lower that some by cleaning out things I'm not using in the root filesystem package, and by removing some modules from zdrv. (No point having graphics of sound drivers for example.) I'll certainly take a look at those varients, thanks for the tips. :)
bobwrit wrote:This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. :wink:
What, pay for a computer? Never! :P I'm in the UK anyway.
I plan to post something usable at some point, but I warn you, it probably won't be soon and it will me rather restriced in what it can do. If you want to try something like this I'd recoment parallelknoppix. That at least has something you can download!

Success! Just as I was typing this I got a positive result, a node booting and running a job without any setup on my part. That leaves me a single test away from declaring it usable and starting work (again) on the server. (aka. the hard bit.)

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#21 Post by cb88 »

could you compile the octave pankage for puppy that is in parallelknopix with matrixpup it seems to be a powerful matematical application and puppy doesn't have one yet i think it needs python which i don't have it will ./configure and make but fails because there is not enough dependacies on my pc a recompile of gcc would be nice too that would even be advantagious to barry sice he could compile faster in theory

probaby best to do theses as sfs modules

i realize that it might take a while to get all this stuff probably what you planed anyway

BTW my laptop has 64mb ram but no net card LOL might need to get a router and a card
Last edited by cb88 on Thu 06 Sep 2007, 19:29, edited 1 time in total.
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#22 Post by Everitt »

As far as I can see octave needs to be present on ever node, and so needs python on every node too. I'd rather not include that.
sfs files are a little impractical, they'd need to be on over nodes HDD. I could share them over NFS or SMB, but I'm trying to avoid that to reduce network traffic (so it runs better)

I'm not sure if gcc can be run in parallel, but there is a version of make that can (dmake) I'll certainlly look into that, but I won't include it at the moment.

What I'm planning at the moment could well be the solution to all of this. I plan to include the unleashed system, with a few basic scripts to make things easier, so you can create a initrd.gz on the fly, containing what you think you'll need. That way you aren't tied to what I have time to do, or my vision for the project. Of course, it'll mean you need a reasonable amount of ext3 space, but that's not so hard to come by.

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#23 Post by cb88 »

could an sfs be uploaded to every node to the correct place on the hard drive then have the PC reboot? are local sfs files detected by net booting puppies? boot>>load sfs files>>reboot with the sfs files better than remastering and would only be a burden on the net during boot

that would be a good feature since it is likely that most pc that are used with this will be dedicated nodes and not used for other stuff(in my case that is what they will be)

i am gonna try geting python and octave working on my pc then i may wat to set up a small home cluster with some old'puters probably in the P3-4 256-512mb range.

guess gigabit ethernet would be good too wouldn't it? good idea making unleashed that way there is less chance of the project fizzleing out.
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#24 Post by Everitt »

It would certainly possible to move files onto nodes. I'm already making use of SSH, so there is no reason why you couldn't use the scp command. sfs files do load on a netbooted puppy, as long as you have a save file on that machine, just like from the CD. This does of course require nodes to have disks, and involves copying the file to each indervidual node one at a time (or with a script, still in a linear, therefore slow, manner.) It's good idea though, if you plan to use the same nodes time and time again. I'll add a script to do that to my ever growing todo list.

I'm planning to add the ability to poweroff or reboot node using MPI. If you try to run the commands directly then they never return 0 (how could they?), so you have to wait for it to time out on the server. A couple of simple binaries should fix that though. Yet another thing on my todo list. :)

Gigabit makes transfers faster, but actually has a slightly higher latency, so the benifits actually depend on the code you run. I believe that overall it is a big improvement, but then again it's still nastily expensive. :(

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#25 Post by cb88 »

it may be costly as an pci board but many computers even old ones (p4 cerca 2002) have gigabit. is it not possible to have multiple links between computers perhaps have both 100mb and gigabit(that really would be expensive unless you found some for cheap)? still haven't gotten octave i suspect that it may be the old gcc puppy 2.16 uses 2.20 will have 4.x so maybe i will try again with it.

also 2.20 may come in a new kernel and old kernel versions so you could probably use most of the same stuff in 2.20 with 2.18x kernel

i really have no experience with net booting at all.

i did notice that python was very big when i compiled it the .pet ended up being 5.3mb
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#26 Post by Everitt »

The switchs are also expensive, which is what put me off. I guess I'm just cheap. :P
Multiple NIC's would be somewhat complicated to set up, and even harder to get an app to use them seperately for different data loads. I'd suggest just using what you have, chances are you wouldn't really notice a diference anyway. I don't know, I don't have any real experiance.

You shouldn't really need much. Just boot the CD in on computer, then on the nodes all you need to do is ensure they're set to boot from the network. If there isn't an option in the bios (pre 2001 machines) you can get boot floppies.

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#27 Post by cb88 »

sounds easy enough

if you are in the us this might be a good buy even though everitt must already have a switch

http://www.newegg.com/Product/Product.a ... 6833180028
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#28 Post by Everitt »

Well, I can do it, so yeah, it's got to be pretty damned easy. :)

Progress report: Work on the server has reached a point where it is now possible to create a working CD. I hope. It works booting the server itself over PXE. with a little fiddeling for reasons of filesize. No reason why a CD shouldn't work. I'm calling this a milestone anyway. Now it's a matter of tidying up a few things, rewriting the unleashed scripts, and intagrating that. First up though I'm going to try and write a few scripts/binaries I've been thinking of to make using it a bit easier.

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#29 Post by cb88 »

like i said i tried compiling octave again and it said that a sanity check failed i tried to compile a program i had already packaged and it did the same so i guess my devx is messed up i'll fix it tomorrow i guess

also there is a version of octave for mpi or is it a patch it seems that octave is a near exact clone of command line matlab but i am not sure

my calculus professor said he would give the windows version a look see "happily" he still uses win 98 and 2000 and there is hope he will never use vista!


here is octavempi if you haven't seen it already: https://sourceforge.net/projects/octavempi/
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#30 Post by Everitt »

cb88 wrote:here is octavempi if you haven't seen it already: https://sourceforge.net/projects/octavempi/
There is also MPITB. Seems more mature to me, or at least to have better documentatiopn; and itseems to have more suitable requirements. (No NFS.)
Who knows? It's 4AM, I've spent most of the night scripting. :P

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

Getting there!

#31 Post by Everitt »

Last night I burn the first disk of MatrixPuppy. I scrapped it after one boot, but then I burnt a slightly different one, and it almost worked. Just.

The server boots fine, and the nodes load up ok. The (Very) basic tests I've run have shown that code is being executed on all the nodes. I've written some VERY basic documentation, found in /readme.text (a typo, but I like it :) )
Things I know: They two binaries I wrote to poweroff and reboot the nodes are broken. I think I know why, but I haven't had any time to look into it.

If anyone is interested in trying it then I have a 142MB ISO that I'd be more than happy to upload for you, but I have no idea where I could host a file like that. I tried puppyiso@ftp.servage.net but it timed out at 99%, twice, so I guess that's not going to work.

msumner
Posts: 205
Joined: Fri 05 Jan 2007, 01:10
Location: Lincolnshire, England.

#32 Post by msumner »

Hi Everitt, I only just noticed this thread. I have fancied trying out something like this with a selection of old scrappers. I find people giving me some quite decent boxes, the latest two have 850 and 950 mhz durons with 384 and 512 mb ram. I would love to try puppy on these!

User avatar
HairyWill
Posts: 2928
Joined: Fri 26 May 2006, 23:29
Location: Southampton, UK

#33 Post by HairyWill »

Will
contribute: [url=http://www.puppylinux.org]community website[/url], [url=http://tinyurl.com/6c3nm6]screenshots[/url], [url=http://tinyurl.com/6j2gbz]puplets[/url], [url=http://tinyurl.com/57gykn]wiki[/url], [url=http://tinyurl.com/5dgr83]rss[/url]

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#34 Post by Everitt »

msumner wrote:Hi Everitt, I only just noticed this thread. I have fancied trying out something like this with a selection of old scrappers. I find people giving me some quite decent boxes, the latest two have 850 and 950 mhz durons with 384 and 512 mb ram. I would love to try puppy on these!
Those machines are way out of my league. I'm testing this on PII's (350 + 450 oc'd to 510)
I got the memory useage for the nodes down to about 48MB earlier. Unfortunately I also noticed that I'd made a slight mistake, so you could only actually use parellel code. Add that to a few lesser bugs / updates I've been playing with and I don't see much point actually releasing alpha1. I'll try to get a fixed version out soon, but right now I'm getting odd errors. I know recompiling will fix it, but I think that might break something else. I need to do some reading around this.

Thanks HairyWill, that looks like it should do nicely. :D

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#35 Post by Everitt »

Right, lets try this again.
Getting alpha2 ready has taken longer than I expected as I found a couple of pretty majour bugs that I had to work around. Hopefully though this econd alpha will last ong enough to actually get uploaded. :)
Changes include:
Rewrite of my mpi launcher script to include a lot more options, and to fix a couple of bugs.
Modification of library path varaibles to include the mpi libs (:D)

I'm going to update the readme, look into another possible bug, then try to upload. Fingers crossed. :)

Post Reply