The Testing is half empty
 

or half finished, depending on whether you are a pessimist or an optimist, i suppose. Looking at the wiki and other sources of discussion on the Extras-Testing validation process, there are a few things that stand out as missing and in need of implementation.

Before i cover those, though, let's review the purpose of -testing:

• provide a quarantine time for serious problems with an app to be discovered
• provide Quality Assurance by requiring testing of basic essential criteria for apps destined for Extras
• act as a "demilitarized zone" for power users to use and test apps

In short, it should be a way for apps to get thoroughly tested for basic requirements by others before allowing an app in Extras. Here are the needed changes to accomplish this in a way that is less onerous to developers and testers alike:

First, a way to have two tiers of testers and criteria. The description was of a pool of "official" testers required to give some percentage of the total thumbs-up. The importance of this is twofold: it allows for thorough testing of all QA criteria, and it allows for "casual" or fan testers, with a smaller mandate, to do the majority of the tests, thus expediting testing. All that is needed for this is a process for admitting "official" testers, a mechanism in the promoter script to recognize and identify those in the test reporting page, and a revised list of QA criteria for the informal testers. The official testers could use the existing comment mechanism to report what they had checked in-depth to verify all points of the QA checklist were covered.

Second, a reduced burden of testing and quarantine for updates. The present requirement for ten karma and ten days does not encourage "release early, release often".

Note that deputizing ordinary users as testers via the mechanism in the first point would help alleviate this update testing problem, but ten days and ten full testing karma is too much. The current testing QA list is too confusing for ordinary users who want to help, which leaves them frustrated at the delay in releases and with no way to help.

Re: The Testing is half empty
 

Yeap, the testing queue is stuck. The queue is now a popularity contest as I was afraid in fall. It is hard to say what people have actually tested, and I think only few have used QA checklist. Plus the website needs some work. Also I think that testing queue shouldn't be a place where software is brought to find out what bugs there is, software should be release quality and testing queue is for acceptance testing only. If a feature does not work, disable it and release when it is ready.

As I understand the medicine is to:
1) Retain discussion history for a package
2) For new versions there should be changelog
3) Automatically test as many things as possible on the checklist
2) Those not automatically tested should be tested atleast by one tester
4) Functionality should be tested by more than one person. I would still keep 10 people as acceptance level for now
5) Thumb down should be accompanied by a bug report

When this system would work, then would be a time to fine-tune:
- How to make bugfix releases go through the system faster?
- How to clean up the queue from stuff which is not going through?
- How many testers is right amount to make certain that functionality is ok?

Re: The Testing is half empty
 

You should only make promotion form Extras-Testing into Extras harder if there are any real problems with Extras packages. As there are no big problems with Extras packages that I know of at the moment, making promotion harder will not bring any advantage. It will simply prevent packages from reaching Extras, spoiling it for everybody.

Re: The Testing is half empty
 

I agree with VRe. Also, I think more exposure is a good thing. Few people know this is possible or that it helps or know where to vote and what to test.

I'm thinking mandatory exposure to community, say, reserved forum section/threads? Complete with links?

Also, a checklist template could be used to tag votes, maybe we can see later what was tested and wasn't? Maybe by whom and how so we can find and ask later how it worked?

Re: The Testing is half empty
 

Hang on a moment guys. I don't really care if extra features are added but i think that is Brainstorm material.

All i am requesting up front is that two things already planned to be part of the process--official testers and reduced requirements for updates--are finished. The system as it exists in a half-finished state is not accomplishing its purpose and at the same time getting in the way.

If i knew better where to make this request, i would, but posting a plea in community seemed like the best bet.

Re: The Testing is half empty
 

We have got at least 3 problems that I recall in this relatively short period: wallpapers with Nintendo copyrights, Radio FM messing with speakers affecting phone calls and Load applet advertising screencasts that actually won't work, confusing several users.

Actually these three problems were not difficult to find, and they were even commented during the QA process. Perhaps what happens is that the goal is difficult to achieve in practice (one person going through 10 blocker criteria, with the skills it implies). Which leads to even the more riguroous evaluators being soft here and there.

I think the key feature that would make testing more accessible to power users and even fun is to split ratings by blocker criteria, as discussed some time ago. Some users will be happy checking whether the features advertized are in place. Others will follow the steps to check whether the app is optified. If at the end the remaining items are system performance and power managent, normal users will be able to assess at least if they noticed serious/noticeable problems. Of course much better if someone can run the right tools and do proper tests, but in their absence happy testers have chances to lead to happy users.

Re: The Testing is half empty
 

Honestly, my experience from Diablo is that I can leave a package in Extras-devel for 3 months and get almost no feedback, and then I promote it to Extras and I get three bug reports in a day.

I suspect the same thing is happening in Fremantle. I can't be sure, my apps are stuck in -testing with sufficient karma but I guess not enough quarantine time?

The ten days quarantine thing is the part I don't understand. What exactly are they waiting for? If you get the votes, your package should move onwards.

Re: The Testing is half empty
 

I guess we could go for this if at the same time it is agreed that packages can be pulled from Extras back to Extras-testing if problems arise later on.

That is another problem. Packages were pulled only when there were serious problems floating and after contacting personally the maintainers. The Load applet for instance is still there, even if the bug is evident. Still, the responsibility of pull it off (or even ask for it) is not defined and there you have still the app getting more downloads and confusing more users.

And don't get me wrong, I don't blame the developers! Bugs happen, problems happen. This is not about being harder in order to try to prevent problem, but about being more flexible so the problems can be dealt easily when they come.

Re: The Testing is half empty
 

Testing attributes need to be built into the Maemo infrastructure IMO. Apps should have the right metadata wrapped around them throughout their development/deployment lifecycle, and there needs to be a Maemo-managed web-based system in place to interact with that metadata. Once that's in place, making sure testing efforts are meaningful becomes simply a matter of exposing and updating the proper App attributes at various points along the lifecycle. Right now there does not appear to be enough detail.

related to this Brainstorm: http://talk.maemo.org/showthread.php?t=38014

(sorry, Flandry, I see this was outside your intended scope)

Re: The Testing is half empty
 

I think the quarantine is useful, for instances like the load applet, where the thorough testers can shout: "hey I realize this has enough votes but issue X needs to be resolved in some way before this goes to extras"

Since current voting system seems to lend itself more towards a popularity contest as others have said, perhaps the quarantine period could be used as final acceptance testing by 'official' testers?

I agree that a well defined process for removing a package from extras would be beneficial, but I'm not sure that alone is enough to reduce the quarantine time.