Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 02 Sep 2014, 05:08
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects » Next Puppy Development » 4.x
Pre Release Testing Methodologies for Puppy Linux
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 4 [54 Posts]   Goto page: 1, 2, 3, 4 Next
Author Message
ecomoney


Joined: 25 Nov 2005
Posts: 2183
Location: Lincolnshire, England

PostPosted: Sun 05 Apr 2009, 22:54    Post subject:  Pre Release Testing Methodologies for Puppy Linux
Subject description: Boring but necessary
 

A lot of good ideas have come out of this thread...To make sure they dont get missed in amongst the discussion Im going to summerize them at the top here, with the origional question below.

Conclusions

- Pre-release ticklist - Ecomoney

- Whole Forum "sandbox" section for bugs, with "bug per thread" approach, rather than one thread for all bugs -Ttuxxx

- Recruit "Package use experts" (not the compilers) and ask them to take part in testing before development begins - Patriot

- Have a "Testing Manager" to keep track of the testing process, only release when theyre happy too do so...no rush! - Ecomoney

- Make it easy to collect bug reports (default homepage) - James C

- Keep changes/package updates to only the ones needed by Users - Ecomoney


Origional Post

Ok, The puppy linux team Have just released the first official version without BarryK. Lots of new features and upgrades were added, and its received glowing praise from distro critics and even a brief #2 Ranking at Distrowatch.

Although we had a long and ardious testing process, with plenty of pre-releases, Betas and Alphas, 4.2 was released with still quite a few bugs present, and also one particularly nasty one thats been very well discussed and eventually conclusions drawn from. I dont want to go back into any of that here, this is looking forward rather than back.

What can we learn from the successes and failures of 4.2? Here are some things that have already been mentioned.

Quote:

1. Using a full release of Puppy as a test bed for individual pieces of software is a mistake.

2. Each package should be thoroughly tested using some systematic procedure before it ever gets near the overall build.

3. Introducing a new package or attempting to upgrade a package in the middle of the development process is asking for trouble.

4. Third-party criticism is an important part of the process.


from rcrsn51

There was also some discussion that a separate group/team of "test pilots" be created at the inception of each release. These should generally be separate from the developers, which would free their time for compiling etc. and need not necessarily have/need any coding skill whatsoever.

The "test should have a single person as a central co-ordinator who the developer informs of changes/upgrades/new features and packages they have implemented that need testing, and also what other areas this migh effect (say in the case of an upgraded library) That person can then assign the job of testing to an individual with experience of that package, and keep track of what has been tested. Only when that person is happy that everything is ticked off the list will the Puppy Build be released.

If we are discussing events/mistakes in the past, please try not to assign blame. Constructive comments only please.

_________________
Puppy Linux's Mission

Sorry, my server is down atm!

Last edited by ecomoney on Tue 07 Apr 2009, 09:40; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15117
Location: Paradox Realm

PostPosted: Sun 05 Apr 2009, 23:08    Post subject:  

4.2 Bugs and fixes
http://www.murga-linux.com/puppy/viewtopic.php?p=288849#288849

Patches and updates
http://www.murga-linux.com/puppy/viewtopic.php?p=289717#289717

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
Dougal


Joined: 19 Oct 2005
Posts: 2505
Location: Hell more grotesque than any medieval woodcut

PostPosted: Mon 06 Apr 2009, 05:59    Post subject:  

I think before all these fancy rules about how/when packages should be added, there is something much more basic: to actually see that they run.
This isn't the first release where some menu items ended up just plainly not working. I honestly don't understand what the beta-testers are doing if such a state exists -- I would expect the first thing a beta-tester would do is go over the menu and try and start every item.

_________________
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Back to top
View user's profile Send private message 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15117
Location: Paradox Realm

PostPosted: Mon 06 Apr 2009, 07:36    Post subject:  

Quote:
actually see that they run.


Good plan Very Happy
I am planning to try and make some sense of Git
- hopefully that will be helpful too Cool

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
WhoDo


Joined: 11 Jul 2006
Posts: 4441
Location: Lake Macquarie NSW Australia

PostPosted: Mon 06 Apr 2009, 07:46    Post subject:  

Dougal wrote:
This isn't the first release where some menu items ended up just plainly not working..

Which ones, Dougal? In building each release that was one of the things I tried to do - click on each menu item, especially if it was an upgraded or new package, to see that it actually worked. Due to the time constraints, however, I didn't click on menu items where the underlying program was unchanged. I haven't yet found any menu item that doesn't work in Puppy-4.2, BUT I haven't tried them all ... so knowing which one(s) aren't working for you would help.

Cheers

_________________
Actions speak louder than words ... and they usually work when words don't!
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com
Back to top
View user's profile Send private message 
ecomoney


Joined: 25 Nov 2005
Posts: 2183
Location: Lincolnshire, England

PostPosted: Mon 06 Apr 2009, 10:21    Post subject:  

Ok, test each menu entry...individually

That can be one thing on the "tick list"

_________________
Puppy Linux's Mission

Sorry, my server is down atm!
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
Patriot


Joined: 15 Jan 2009
Posts: 734

PostPosted: Mon 06 Apr 2009, 10:30    Post subject: Re: Pre Release Testing Methodologies for Puppy Linux
Subject description: Boring but necessary
 

Hmmm ...

Just my thoughts no matter how lame ...

ecomoney wrote:
.....Although we had a long and ardious testing process, with plenty of pre-releases, Betas and Alphas, 4.2 was released .....

IMHO, its not long enough (but thats just me) but I can understand the reasoning behind it and with puppy's recent history of changes, Whodo did what was necessary.

Quote:
1. Using a full release of Puppy as a test bed for individual pieces of software is a mistake.

Depends on the intended outcome. Some say yes, some say no.

Quote:
2. Each package should be thoroughly tested using some systematic procedure before it ever gets near the overall build.

It should be but everything's sort of informal here ... Anyway, this should have been raised much much much earlier ...

Quote:
3. Introducing a new package or attempting to upgrade a package in the middle of the development process is asking for trouble.

Huh? I must have done things wrong in my own development work then ? Again, I believe this depends on the intended outcome ...

Quote:
4. Third-party criticism is an important part of the process.

Yeah, constructive criticism that is ..... yet, I found out the hard way years ago that most have no tact when delivering their "constructive" criticism. I see most shoot first and ask questions later ... how is that being constructive ?


Quote:

from rcrsn51

There was also some discussion that a separate group/team of "test pilots" be created at the inception of each release. These should generally be separate from the developers, which would free their time for compiling etc. and need not necessarily have/need any coding skill whatsoever. .....

For one, the only real developers that I've seen here are the distro developer(s) and the puppy specific apps developers. I've yet to see the actual applications software developers for Gnumeric or Abiword commenting here. I've seen and read mtpaint's developer (wtjaguar) comments and it's good to have him here.

I don't know if its feasible to have a pool of testers making their own compiled binaries. Compiling takes time that varies between packages and even between testers machines. It's fine if one tester does one or two packages but all the packages ?

What we have here is a group of people working on packaging the various pieces of software into one distro. There's no real formal structure to it. BarryK made it into a sort of not-so-formal thing. That's one reason I was attracted to puppy. I already have enough time pressures on my back and I sure ain't looking forward to adding more from FOSS ...

I can understand the need for testing methodologies. But do we have sufficient man power to handle it ? The way I see it, I believe we need package maintainers that knows the ins-and-outs of the packages they handled. They will have to sit down and learn as much as possible about the package that they maintain. Keep track of what works and what breaks. They should handle the bug reports by looking into possible causes/fixes or to file the bug reports to the actual developer if necessary ... These package maintainers should be consulted into what should go into a distro release and what should be left as options. Being a package maintainer, they'll have to keep their guards up and be up-to-date with changes, bugfixes, tweaking, hacks, some inner-workings, whatever, etc etc ... Life can be hell for the maintainers ... Wink

We already have a lot of users offering pet packages. Why not ask them if they're willing to become package maintainers? Or at least get them into informal groups that works towards a common goal?

"See a need, fill a need" - Bigweld (Robots)

I did CUPS package for a reason. After I got it to work, I saw the need to get fuss-free printing working in puppy. I focused on it when I can and it was a slow process. If not for wanting some functionality in newer CUPS, I wouldn't have bothered. There was no big pool of testers for CUPS. I crashed and burned with the first 1.4b2 pet ... But it was an experience that helped me understood the linux printing system better ... I would have just taken it for granted if I was on Ubuntu ...

Actually, now I'd be much much more than happy to let anyone take this off my back ... It's too much for one person to do it alone ... It would be great if there's a team in puppyland that will maintain all the different printing (not just cups) related packages.

Getting back to testing methodologies, I believe we could start some sort of an informal QA/QC team that makes a checklist of all packages that may be included into a future distro. A fresh start now will help cut down on future pre-release testing time. Having a group of selected package maintainers will also reduce the workload/time pressures/havoc. I believe that when the next distro coordinator have been appointed, he/she will find that the QA/QC checklist will be of a great help.

I know my active time here is only a few days old but again I reiterate, one of the reason I was attracted to puppy is because its a bit more relaxing here. Too much red tape hinders creativity and I've banged my head against it many times in the real world. It will be great to have some QA/QC methods but don't let it spoil the fun please ...

Rgds
Back to top
View user's profile Send private message 
rcrsn51


Joined: 05 Sep 2006
Posts: 9066
Location: Stratford, Ontario

PostPosted: Mon 06 Apr 2009, 14:36    Post subject:  

Just for the sake of clarification, that quote above is not from me. It has been cut-and-pasted incorrectly from another post by ecomoney.
Back to top
View user's profile Send private message 
James C


Joined: 26 Mar 2009
Posts: 5726
Location: Kentucky

PostPosted: Mon 06 Apr 2009, 14:59    Post subject:  

Just as a point of comparison, I am envolved in the Windows 7 beta testing program. Microsoft began with a limit of about 2 million downloads, then later removed that limit. I believe each download gets 3 different installation keys.....anyway,there is a large pool of testers. Crash info is automatically sent to Microsoft so it's not necessary to file bug reports.
The point is, even with all of this testing, Windows still crashes and burns. It is impossible to catch every bug, but hopefully the "showstoppers" will be caught.And I won't be purchasing 7 after the test, too big and bloated.I'm trying to get away from Windows and to Linux full time.
The puppy community is smaller, the release schedule quicker and there are way fewer developers.I don't know the solution but this thread is a beginning.
Back to top
View user's profile Send private message 
ecomoney


Joined: 25 Nov 2005
Posts: 2183
Location: Lincolnshire, England

PostPosted: Mon 06 Apr 2009, 20:44    Post subject:  

When I logged on I was really heartened to so much constructive feedback and so many good ideas have been posted on this thread. Thank you everyone Very Happy

Puppy is a really important project , and its important that people take responsibility to ensure that it is as good as it can be. I agree that it does need to fun, and done in a pleasant, blame and flame free environment for our programmers/packager/compilers to produce their best work. Lets face it no one is here for the money, but even Micro$oft recognise this. Im by no means saying I havnt been guilty of this myself, but I am doing my best to learn from my mistakes.

Comparing open source with commercial development (Ive been commercial developer) there is always a pressure to get something "working" as quickly as possible and get the code (and the invoice) out. Development time is a "cost" that eats into "profit", so there is an incentive to make ugly hacks and patches that build up problems and inefficiencies for the future. This is one reason why Windows i think has become so bad. If you put programmers under pressure then they make mistakes...its just not productive to do it. Open source has no commercial pressures and doesnt need to suffer with set release deadlines....thats why its so much BETTER!

Puppy has always had a breakneck development speed, Barryk used to release versions every couple of months, and sometimes they had some quite big bugs and a new version released (wireless was broken in one version around 2.1x?). Generally speaking he did set aside a reasonable proportion of each "cycle" to testing though, and each change was small and incremental on the last.

Looking back on the events of puppy 4.2 development, Whodo chose to issue lots of small incremental versions which worked very well I thought. First released were quite a few Alphas with different packages in to try. Ok they were buggy, but they gave people the chance to try different potential packages (i.e. firepup) before a final package list was defined...great! There were then several betas to give the maintainers of those packages the time to refine their work and improve the look and the feel (pWidgets got a lot better over this time for example, and the themes were worked on). There were still a lot of issues being identified at this stage, which is why i got a bit worried when Whodo announced Release Candidate 1. I was worried there wasnt going to be enough time to finish testing, but whodo assured us it was merely a title to get more people to test it...fine. I did get the impression though whodo by this stage you were very keen to get 4.2 released and signed off? I did actually think of suggesting a pre-release checklist then and I really wish I had sorry. Embarassed

@ Patriot - good points, none of them "lame". I assume because you produce the JWM window manager (which has been given the opportunity to really shine in 4.2...its so much better when its not coloured grey!) youve got a lot of experience in taking part in many different development efforts from many different open source projects. Thank you for taking the time to pass on your experience of them.

From what Ive seen of different packages, whether something is "ready" to be included is merely semantics. Im using aMSN 0.98 beta in a big cybercafe install, because Ive tested it and it seems pretty stable, as well as having the extra features. 0.97.x was stable too, 0.96 was very buggy and we had to revert back to 0.95 which we used for a long time. On the other hand you would think that Abiword 2.6.6 final would be relatively bugless, but it wasnt. The thing to learn from the future here is that we shouldnt assume/consider ANYTHING bugless or "bugless enough" until weve actually tested it ourselves. Part of the job of producing a "Distro" is to make sure we test and distribute only the best applications and the best quality version of that application? Thats the point.

Many hands make light work, and different people have different skills, and different packages they use. Assigning "Experts" to different packages (a maintainer/compiler, and a tester), who know the ins and outs of that package and can research it thoroughly is an excellent idea Splitting up the distro on the basis of its component applications is an excellent way of doing it.

The puppy community is growing as puppy becomes more popular. When I joined it, it was at number 20 in Distrowatch, now its at number 8! Ive oticed about people that when they come over to puppy at first they are very grateful they have been given something, and are more than happy to do a little bit back and be part of the project.

A first draft of a potential test checklist...

Quote:
Puppy version 4.2.1

App. Ver Packager compiled yn? Tester Tested (Y/N) Submt(Y/N)

Abiword 2.6.5 Fred Yes Frank no no
Blinky 0.9.8 Boris Yes Charles yes yes

etc etc

Perhaps a column in there as well to state if the package was an upgrade on a previous version (and therefore needed testing) would be handy...but you get the idea.

How can this be kept up to date?

@James

Theres no reason we cant learn from Windows as well...they are for some reason quite successful, at least for the moment. You are saying with the windows 7 testing program, they have the facilities for collecting bug reports actually built into the program itself? How could this concept be implemented (better) in Puppy?

How about in future, making the default home page in Seamonkey refer to a page in the Puppy Wiki. This would change depending on the release (alpha/beta/RC)...

The alpha could have a page detailing the programs being tried out

The betas could have a lits of known bugs, and what needs to be tested

The Release candidates would have the checklist above to keep track of all of the final bugs, and ensure everything was checked thoroughly before release.


As I say, this is just one general idea and open to a lot of refinement.

Thank you for your continued reading if youve made it this far...Wink

_________________
Puppy Linux's Mission

Sorry, my server is down atm!
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
James C


Joined: 26 Mar 2009
Posts: 5726
Location: Kentucky

PostPosted: Mon 06 Apr 2009, 21:18    Post subject:  

@Ecomoney

Most of my experience is with XP. In XP if an application,say Firefox or Adobe reader, freezes up or crashes the user has the option to upload an error report to Microsoft. To the best of my knowledge, in Windows 7 Beta the option is taken away and the error report is automatically sent.Sort of the opposite of "automatic updates".I personally would not want that info
sent without my permission on an official release, but I don't mind on a beta release.
Anything to help get accurate bug reports back to the developers in a timely fashion would probably help.Keeping the bug reporting threads on track would probably help as well.
Back to top
View user's profile Send private message 
raffy

Joined: 25 May 2005
Posts: 4765
Location: Manila

PostPosted: Tue 07 Apr 2009, 04:32    Post subject: Git and groups  

The Git repository has just been updated by Pizzasgood for 4.2. This is a good beginning. People can focus working on their chosen aplication/s and see improvements immediately.

Someone can start tallying people's names working on apps and probably come up with a list of individuals or groups. What this will do is help testers or developers get directed to the proper communication channels. However, the usual channel will still be the forum.

What's more, the developers can put links/directions in their forum signatures, the way MU, Zigbert et al are doing.

_________________
Puppy user since Oct 2004. Want FreeOffice? Get the sfs (English only).
Back to top
View user's profile Send private message 
WhoDo


Joined: 11 Jul 2006
Posts: 4441
Location: Lake Macquarie NSW Australia

PostPosted: Tue 07 Apr 2009, 06:18    Post subject: Re: Pre Release Testing Methodologies for Puppy Linux
Subject description: Boring but necessary
 

Patriot wrote:
I know my active time here is only a few days old but again I reiterate, one of the reason I was attracted to puppy is because its a bit more relaxing here. Too much red tape hinders creativity and I've banged my head against it many times in the real world. It will be great to have some QA/QC methods but don't let it spoil the fun please ...

Hallelujah ... and pass the mustard! Razz

@All - raffy is correct. Git is the opportunity to stabilise package development for Puppy. It's getting close to ready and when it is I will be using it to build Puppy-4.2.1, or perhaps 4.3, for release in a few weeks to a month.

How to proceed? Read the Git instruction manual, available online when the server is ready for general access.

Like everything else in Puppy, if you want to help in the development effort then go for it. No-one is going to tell anyone here what to do or what not to do; maybe that's a bit chaotic, sure, but highly productive in a creativity sense. OTOH I've been known to ask for things occasionally, but never with any expectation that a dev will comply. No-one can say "no" until they are asked. Wink

Above all, HAVE FUN. Forget ego. Forget ownership. Forget personalities. Be "Nike" about the whole thing (IOW, "Just do it"!), and don't get all bent out of shape if what you do doesn't find its way into the main release. I firmly believe that no sincere effort born in generosity is ever wasted, even by exposure to ingratitude.

@Patriot - I hope for all our sakes that it just gets easier for you to stay, have fun and continue contributing great stuff the way you have so far in your relatively short time with us. If you are finding yourself under too much pressure to deliver in any area, please don't hesitate to discuss that with me via a PM before voting with your feet, ok? A big part of my job here, at least for the next release of 4.x, is to ensure that no volunteered resources - developer, packager, tester, maintainer, etc - are either abused or wasted, ok? Cool

_________________
Actions speak louder than words ... and they usually work when words don't!
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com
Back to top
View user's profile Send private message 
ecomoney


Joined: 25 Nov 2005
Posts: 2183
Location: Lincolnshire, England

PostPosted: Tue 07 Apr 2009, 06:23    Post subject:  

Quote:
Someone can start tallying people's names working on apps and probably come up with a list of individuals or groups. What this will do is help testers or developers get directed to the proper communication channels. However, the usual channel will still be the forum.


Why not get the users to give us this information instead of going and chasing them?

I noticed a thread on here asking people how many computers they had and what OS was on them...it got very popular. What about a tempory sticky in the Users ( For the regulars ) asking "What are your favourite applications?". its been my observation that regular puppy users love posting on threads like this. It would then just be a matter of going through the thread and sending pm's to the relevant people asking them if they would be prepared to help test their favourite application in the next development cycle. This could be the job of the "testing tzar" to assemble a testing team.

It would also help us on the "Puppy should have all the applications necessary for daily use" part of the mission statement.

I think its important to have a team of people with the right skills assembled before each development cycle, with their responsibilities defined. My name was included in the release notes for 4.2 under the "testing" section Shocked ...not because I had been asked to be a tester, but because I had submitted some bug reports of things I had come across. When I pointed out about the Abiword bug I got "Well you were on the testing team...why didint you report it!". I dont even use Abiword!

_________________
Puppy Linux's Mission

Sorry, my server is down atm!
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
ecomoney


Joined: 25 Nov 2005
Posts: 2183
Location: Lincolnshire, England

PostPosted: Tue 07 Apr 2009, 06:46    Post subject:  

You beat me to the post button again whodo!

Im not sure about this GIT repository but Im looking forward to reading about it. I hope popular apps like aMSN, Openoffice and firefox are included...look at which apps are popular in ubuntu for example. Where can i find out more?

The problem with the "nike - just do it" approach is that is someone "just doesnt do it" then faults get missed i.e. Abiword. Im not sure what background you have is software development, but testing is an important part, and should always be done methodically...not just "hope for the best". If this type of work isnt to your taste, then pass it this way.

Please dont think Im ungrateful for your efforts Whodo because I think its important to look at what didnt go so well as what did, and how we might do even better next time. Puppy 4.2 is a vast improvement on previous puppies with just a few rough edges...you should take responsibility for it all.

Not wishing to curb your enthusiasm for moving on to even bigger and better things, puppy has always had .1 and .2 releases. Currently on our main puppylinux.org download page, users are being served with a distro with a "semi-broken" wordprocessor, as well as a few other faults. Shouldnt this be put right as soon as possible before even more new features are introduced?

_________________
Puppy Linux's Mission

Sorry, my server is down atm!
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
Display posts from previous:   Sort by:   
Page 1 of 4 [54 Posts]   Goto page: 1, 2, 3, 4 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects » Next Puppy Development » 4.x
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1464s ][ Queries: 12 (0.0141s) ][ GZIP on ]