Putting Unleashed under a Revisioning System

[raffy]

Is there a simple explanation for the simple cructacean mind?

From http://www.yolinux.com/TUTORIALS/Subversion.html

Subversion is a software source code "Change Management" (CM) system for collaborative development. It maintains a history of file and directory versions. The files and directories are checked out of the repository and into your local project work area. This [is] called your "working directory". Changes are made to files in your "working directory". After changes are made to create the next working version, the files are checked into the Subversion CM repository.

It's almost like a wiki, except that you play with the codes in your work area and test that it works before "checking [it] into" the server. Perhaps the new terms are "check out" instead of download and "check into" instead of upload.

(Am still feeling the ropes, so please correct me if am wrong. )

[Lobster]

Thanks Raffy,

So the ability to check in again and upload any changes is essential?
Running a local unleashed tree will mean I need a bigger HD (14 Gig seemed fine a few months ago)

[technosaurus]

for egroupware you can see an example at http://www.egroupware.org/demo

for the script ... could either use a bash script + diff or just compare md5sums (the benefit is that it could also be used to create the MD5 sums) if the diff is the same or md5 matches just replace one with a symlink - ideally (for simplicity) the checked files should all be in one directory with only symlinks in the viewable tree. When a new file is uploaded to the tree it could be md5 checked and either moved to the main section and replaced with a symlink or if the md5 matches an existing file replace it with a symlink to the existing file...maybe keep its name?

I have no idea how to do this automatically when a file is uploaded so unless someone has an idea how to do that the server would need to be set up to run the script periodically.

also if someone verifies that a .pet works for a different puppy version all they would have to do is upload the md5 to the tree and it would automatically generate the symlink on the next run of the script.

[lwill]

I then got 'the fear of geek' and need to know if there is an SVN front end?

What about http://trac.edgewall.org/ ?
I played with it a bit on my local FC machine with SVN to track changes on a OSCommerce site I was playing with, I kept loosing track of all the mods I was doing, and SVN command line was more than I wanted to learn, but I did want to get my feet wet with a versioning system. I have seen several large projects using it. Example in use - http://svn.mythtv.org/trac/

[HairyWill]

esvn is a svn client gui
it is in the puppy 3 repository
be warned it will install qt as well

[Pizzasgood]

Did some stress testing tonight. Wrote a script to choose files at random, make changes (either prepending/appending random amount of random data, or using sed to replace random letters with other letters), and commit. Played around with different combinations of how much data to add and how many files to edit in each commit and made around 230 commits in the end. The size increases were reasonable - large amounts of changes (~50 commits of 4-20 modified files per commit including maybe 20kB new data per commit) tended to make the remote repo grow by 2-12 MB or so. When I adjusted things to make a lot of changes and add a lot of new data, the local repo increased by about 200 MB, while the remote repo only grew by about 130 MB. I haven't noticed any performance difference yet, other than sometimes 'svn update' takes a little while, probably just due to the number of directories involved. It was like that before though. Time-wise, running update in the packages/ directory feels similar to running find there. In fact, that's probably what happens. It runs fast if you do the update in a single package's directory, by the way, so if you are only worried about updating one package at the moment you can just do that and it's nice and fast.

I did all of that testing in a branch, so it didn't dirty the trunk.


Probably the next thing I'll look at are web-frontends.

[HairyWill]

Code: Select all

svn log -v  svn://pizzasgood.no-ip.org/puppy/branches/stress_test/packages

returned over 43000 lines to the other side of the pond in less than 30 seconds.

[Pizzasgood]

Neato.

[Pizzasgood]

I made an error. I noticed today that the svn repo didn't have a single library file in it. To be more specific, any file that ended in .so, .o, or one of those with numbers, was missing.

Turns out SVN tries to be smarter than it is when you make an import, and it assumes you want to ignore library and object files (probably due to how often it is used for C/C++ projects). So when I made the initial import, it ignored about half of Puppy's guts...

Easily corrected. Just have to use the --no-ignore option when making the import. So I wiped out the old repo and redid it with that option (simpler than writing a script to locate and add every missing file).

So that means the actual base-size of the repository on the remote machine is 416MB, and the size of the initial trunk on the local machine is 2.3GB, and checking out the trunk takes about 441MB of bandwidth.

[cb88]

is it possible to compress the connection? with gzip or lzma?

[Pizzasgood]

I think it does that automatically, at least when using the svn:// protocol like I am. I just tested it by creating a big (~78MB) file consisting of nothing but zeros and committing it. It only used about 94kB of bandwidth.

[Nathan F]

PIzzasgood, I applaud your initiative. Long and load. With gusto and extra cheese.

I've had a number of packages hosted @ google code for @ six months now. MOstly stuff I was working on localizing. It was working very well for that purpose before I went awol. I think this is the way to go for unleashed, but obviously we're going to need a lot of bandwidth and a server with some horsepower if it's to be official and see heavy use by a lot of devs, plus usually a larger number of anonymous checkouts.

Someone earlier mentioned custom versions. I know it's not the main thread, but I will say I modified the createpuppy script a while back to use profiles. So I had a packages.txt.opera, packages.txt.seamonkey, etc. Problem is that the version I worked from is now excessively old - Puppy-2.0 actually. But using subversion, the unleashed scripts themselves could be easily hacked at as well as kept up to date by the community. So basically my point is that it isn't a difficult thing to implement, and subversion would make it easier to maintain.

Nathan

[polymath]

i've modded the createpuppy script to allow the addition of a patch directory as a seperate package right before the build of the SFS files. I've also redirected the SFS destination so I can make my puplet boot off of a GRUB CD.

I just think that there should be more functionality within createpuppy, and it doesn't seem to take advantage of all the power it could have, which might be because the unleashed core hasn't changed much beyond kernel revisions since puppy 2.

[Pizzasgood]

Okay, I've been fiddling around with web interfaces. I got websvn working, and it's nice enough, but it simply cannot handle something as big as this. Works fine for small projects, but for Unleashed it is horrendously slow. And it also winds up demanding obscene amounts of ram (I increased its limit to 64 MB and still got occasional problems )

So that's out. Which is unfortunate, because that was a very simple one to set up. No fancy dependencies or anything. Oh well.


So I tried ViewVC instead, which worked out. I tried the 1.1.0-beta1 package first, but that has similar issues with large projects. But the stable version, 1.0.7, works fine. Took a while to get running, as it requires SWIG and the SWIG SVN bindings, which took forever to compile on my old clunker. Also requires Python, but I had that installed already due to the devx_xxx.sfs file. I also installed Highlight to add syntax highlighting, and I tweaked the 'lib/viewvc.py' file of ViewVN so that it wouldn't override Highlight's internal filetype detection. The ViewVN method relied on extension, but we deal with a lot of extensionless scripts so that would be pretty ineffective. The built-in detection of Highlight does that, but also supports the shebang (as in, #!/bin/sh), so it works on the majority of the scripts in Puppy (exceptions being scripts which are included rather than executed, like /etc/profile, which can't really be helped).

Try it out here:
http://pizzasgood.no-ip.org/viewvc/Puppy/
(There aren't any files that have been modified more than twice though, and even those are few. trunk/puppy-unleashed/createpackages is one of them.)

And we do of course have the ability to modify the theme and what-not. Actually, I already made a small change to that. I added an hr tag above the "Download GNU tarball" link so that it doesn't blur into the files so much.

[raffy]

Here's a screen capture (from the other side of the Pacific):

[Pizzasgood]

Over the weekend I realized that I overlooked something obvious: SVN sticks a bunch of .svn/ directories everywhere. I knew that, but it hadn't clicked in my head that that means we can't simply use the working directory as a working unleashed tree. Not without tweaking.

I see three options:

A: Don't use the working directory to build "real" Puppies (when just testing stuff it doesn't matter). Instead, export the repo to create a new clean tree.

B: Modify the createpuppy and createpets scripts to exclude any directories named '.svn'. In the process, the copying from the packages/ directory into rootfs-complete/ could be replaced with rsync to speed it up a little if you've already built a puppy there, since only changed files would be copied.

C: Trade SVN for Git, which doesn't litter the tree with a bunch of hidden directories.


I'm leaning toward B out of inertia, but this does make me much more tempted to set up a repository in Git running alongside the SVN one. Maybe I'll play with it this weekend. Before I realized this, SVN seemed more suited out-of-the-box for our purposes, but maybe the effort to get Git working would be worth it, to avoid having to do stuff like exclude all directories with a specific name...


Any thoughts?

[Dougal]

C: Trade SVN for Git, which doesn't litter the tree with a bunch of hidden directories.

What about .gitignore files?

[Pizzasgood]

I didn't know about those. I haven't used Git very much yet. I don't think those pop up automatically though, right? And from what I read, they can be placed in a lower level of the tree and still apply to the files deeper, so we could still use them as long as they aren't placed any deeper than puppy-unleashed/packages/, puppy-unleashed/boot/, etc.

But if we have things deeper in the tree that we don't actually want revisioned, then we'd still have to do the equivalent of an export to get a clean tree for building Puppy, unless those files all follow some type of pattern so that the createpuppy script can ignore them, or unless it could parse the .gitignore files.

[HairyWill]

I did wonder about the .svn problem as well.
Whatever VCS we use it will have to have metadata somewhere. I don't fully understand git.
When doing a build from a local copy you could union the copy read-only with a rw layer that includes whiteout files for all the .svn, but that seems a bit convoluted as well.

[Nathan F]

Modify the createpuppy and createpets scripts to exclude any directories named '.svn'.

Reasonable. Adds a bit of cpu time but it shouldn't be a big problem.

In the process, the copying from the packages/ directory into rootfs-complete/ could be replaced with rsync to speed it up a little if you've already built a puppy there, since only changed files would be copied.

And with that stroke you just saved more cycles than you lost. I like it.

I personally think there are a lot more people who are comfortable with svn (a very small number) than are comfortable with git (almost nobody). Could be the difference between a dozen or so developers actually working on the builds vs. 2-3 people. Seriously, that's my cynical thinking at present.

Nathan