The Testing is half empty
 

or half finished, depending on whether you are a pessimist or an optimist, i suppose. Looking at the wiki and other sources of discussion on the Extras-Testing validation process, there are a few things that stand out as missing and in need of implementation.

Before i cover those, though, let's review the purpose of -testing:

• provide a quarantine time for serious problems with an app to be discovered
• provide Quality Assurance by requiring testing of basic essential criteria for apps destined for Extras
• act as a "demilitarized zone" for power users to use and test apps

In short, it should be a way for apps to get thoroughly tested for basic requirements by others before allowing an app in Extras. Here are the needed changes to accomplish this in a way that is less onerous to developers and testers alike:

First, a way to have two tiers of testers and criteria. The description was of a pool of "official" testers required to give some percentage of the total thumbs-up. The importance of this is twofold: it allows for thorough testing of all QA criteria, and it allows for "casual" or fan testers, with a smaller mandate, to do the majority of the tests, thus expediting testing. All that is needed for this is a process for admitting "official" testers, a mechanism in the promoter script to recognize and identify those in the test reporting page, and a revised list of QA criteria for the informal testers. The official testers could use the existing comment mechanism to report what they had checked in-depth to verify all points of the QA checklist were covered.

Second, a reduced burden of testing and quarantine for updates. The present requirement for ten karma and ten days does not encourage "release early, release often".

Note that deputizing ordinary users as testers via the mechanism in the first point would help alleviate this update testing problem, but ten days and ten full testing karma is too much. The current testing QA list is too confusing for ordinary users who want to help, which leaves them frustrated at the delay in releases and with no way to help.

Re: The Testing is half empty
 

Yeap, the testing queue is stuck. The queue is now a popularity contest as I was afraid in fall. It is hard to say what people have actually tested, and I think only few have used QA checklist. Plus the website needs some work. Also I think that testing queue shouldn't be a place where software is brought to find out what bugs there is, software should be release quality and testing queue is for acceptance testing only. If a feature does not work, disable it and release when it is ready.

As I understand the medicine is to:
1) Retain discussion history for a package
2) For new versions there should be changelog
3) Automatically test as many things as possible on the checklist
2) Those not automatically tested should be tested atleast by one tester
4) Functionality should be tested by more than one person. I would still keep 10 people as acceptance level for now
5) Thumb down should be accompanied by a bug report

When this system would work, then would be a time to fine-tune:
- How to make bugfix releases go through the system faster?
- How to clean up the queue from stuff which is not going through?
- How many testers is right amount to make certain that functionality is ok?

Re: The Testing is half empty
 

You should only make promotion form Extras-Testing into Extras harder if there are any real problems with Extras packages. As there are no big problems with Extras packages that I know of at the moment, making promotion harder will not bring any advantage. It will simply prevent packages from reaching Extras, spoiling it for everybody.

Re: The Testing is half empty
 

I agree with VRe. Also, I think more exposure is a good thing. Few people know this is possible or that it helps or know where to vote and what to test.

I'm thinking mandatory exposure to community, say, reserved forum section/threads? Complete with links?

Also, a checklist template could be used to tag votes, maybe we can see later what was tested and wasn't? Maybe by whom and how so we can find and ask later how it worked?

Re: The Testing is half empty
 

Hang on a moment guys. I don't really care if extra features are added but i think that is Brainstorm material.

All i am requesting up front is that two things already planned to be part of the process--official testers and reduced requirements for updates--are finished. The system as it exists in a half-finished state is not accomplishing its purpose and at the same time getting in the way.

If i knew better where to make this request, i would, but posting a plea in community seemed like the best bet.

Re: The Testing is half empty
 

We have got at least 3 problems that I recall in this relatively short period: wallpapers with Nintendo copyrights, Radio FM messing with speakers affecting phone calls and Load applet advertising screencasts that actually won't work, confusing several users.

Actually these three problems were not difficult to find, and they were even commented during the QA process. Perhaps what happens is that the goal is difficult to achieve in practice (one person going through 10 blocker criteria, with the skills it implies). Which leads to even the more riguroous evaluators being soft here and there.

I think the key feature that would make testing more accessible to power users and even fun is to split ratings by blocker criteria, as discussed some time ago. Some users will be happy checking whether the features advertized are in place. Others will follow the steps to check whether the app is optified. If at the end the remaining items are system performance and power managent, normal users will be able to assess at least if they noticed serious/noticeable problems. Of course much better if someone can run the right tools and do proper tests, but in their absence happy testers have chances to lead to happy users.

Re: The Testing is half empty
 

Honestly, my experience from Diablo is that I can leave a package in Extras-devel for 3 months and get almost no feedback, and then I promote it to Extras and I get three bug reports in a day.

I suspect the same thing is happening in Fremantle. I can't be sure, my apps are stuck in -testing with sufficient karma but I guess not enough quarantine time?

The ten days quarantine thing is the part I don't understand. What exactly are they waiting for? If you get the votes, your package should move onwards.

Re: The Testing is half empty
 

I guess we could go for this if at the same time it is agreed that packages can be pulled from Extras back to Extras-testing if problems arise later on.

That is another problem. Packages were pulled only when there were serious problems floating and after contacting personally the maintainers. The Load applet for instance is still there, even if the bug is evident. Still, the responsibility of pull it off (or even ask for it) is not defined and there you have still the app getting more downloads and confusing more users.

And don't get me wrong, I don't blame the developers! Bugs happen, problems happen. This is not about being harder in order to try to prevent problem, but about being more flexible so the problems can be dealt easily when they come.

Re: The Testing is half empty
 

Testing attributes need to be built into the Maemo infrastructure IMO. Apps should have the right metadata wrapped around them throughout their development/deployment lifecycle, and there needs to be a Maemo-managed web-based system in place to interact with that metadata. Once that's in place, making sure testing efforts are meaningful becomes simply a matter of exposing and updating the proper App attributes at various points along the lifecycle. Right now there does not appear to be enough detail.

related to this Brainstorm: http://talk.maemo.org/showthread.php?t=38014

(sorry, Flandry, I see this was outside your intended scope)

Re: The Testing is half empty
 

I think the quarantine is useful, for instances like the load applet, where the thorough testers can shout: "hey I realize this has enough votes but issue X needs to be resolved in some way before this goes to extras"

Since current voting system seems to lend itself more towards a popularity contest as others have said, perhaps the quarantine period could be used as final acceptance testing by 'official' testers?

I agree that a well defined process for removing a package from extras would be beneficial, but I'm not sure that alone is enough to reduce the quarantine time.

Re: The Testing is half empty
 

Well, honestly, I would forgive the Load applet, no screencasts is no big deal. The Nintendo wallpaper thing has been obvious from the first day, I have no idea why nobody at Nokia noticed it. The wallpapers were technically ok though, so they passed through the approval process. This leaves Radio FM messup. Not much in terms of problems.

I do not think there is anything you can do to increase the number of people testing stuff. "Power users" generally do not test and they do not vote. They just use, even if it means enabling potentially unsafe repositories. Making every voter go through a checklist just reduces the number of people who vote. People who do go through checklists appear to be a very special kind of people, more like career Wikipedia editors. These are rare and they are weird.

So, as you might have guessed, I am very skeptical on any "improvements" anyone is going to bring to the Extras vetting process. If anything, we need less enforcement, not more.

Re: The Testing is half empty
 

Who is in charge of this and what is necessary to get changes made?

Let me point it out that at no point in my initial post am i advocating a system less secure than what is in place lest the impression be given that it was a request that needs to be debated. Again, it's simply a request that the process be completely implemented as planned.

Since we're kinda off in la-la hypothetical land with the discussion, let me throw at you a scenario: it is discovered that a popular app in Extras has a timebomb built into it that will do something really heinous in one more day. What do we do?

First of all, we do need a demote button and it does need to be accessible to official testers. They need to be able to demote something from Extras found to be in violation of QA criteria, and able to throw testing versions back to -devel when blockers are discovered (to avoid wasting tester's time).

Second of all, the system needs a built-in facility to inject a stub package into Extras that displays a customized message when installed or executed. Bonus points if app manager recognizes it as an "urgent update" and gives unusual notice of its existence.

Re: The Testing is half empty
 

Tell Tero Kojo, Niels Breet, or Jeremiah Foster, who remove the application. Why are you asking? Has this ever happened?

There are no "official testers" right now.

No. No single person, except the dedicated repository adminstrator, should have the right to single-handedly demote or promote packages.

Why does the system need that?

Re: The Testing is half empty
 

Because there are hours when none of them are around. This is a la-la land hypothetical situation with equally hypothetical and general solutions that rely on no one person to work. The relying on individuals doesn't seem to be getting us far, so maybe it's not a good strategy?

Yes, exactly. ;) Can we get back to why not and when?

Re: The Testing is half empty
 

There's nothing wrong with my wallpapers. They are fair use and legal and I have not been told otherwise. They were taken down because of a misunderstanding and I'm not bothering to put them back up because, quite frankly, I'm tired of hearing this kind of crap.

It's a big problem when I use a small portion of a bigger work for my wallpapers (fair use), but it's perfectly okay for certain other members of the community to grab 100% of a copyright image and use it in their package. Awesome. I'm so done with all this.

Re: The Testing is half empty
 

Well, I have provided a real solution to your hypothetical situation. If needed, there are more Nokia employees and contractors who can take care of the situation.

Whatever mechanism you are suggesting to implement, it will still rely on individuals. I would rather have it rely on technical Nokia people whom we all know to generally do the right thing, than on a bunch of random people I do not know.

In general, I would strongly suggest against implementing "solutions" against hypothetical situations. Nothing good has ever come from it. Remember this next time you fly somewhere.

Re: The Testing is half empty
 

Good, so we are agreed that this doesn't need more debate and just needs to be finished as planned. Wonderful.

BTW there's no need to rely on individuals for every bit of infrastructure. That's the whole point of open source. Would linux exist without Linus? Not per se. Would it have evolved into what it is today without him? Probably.

I'm sure i don't speak for only myself when i say there are volunteers to help get this implemented as planned, i didn't come stir things up without willingness to participate where possible.

Re: The Testing is half empty
 

Oh, you know perfectly well that it has long stopped being about legality of certain fair use practices. Nowadays, it all comes down to the size of the entity you piss off. :( Nintendo is pretty high on the list and they absolutely hate people using even a few pixels of their "intellectual property".

Re: The Testing is half empty
 

I agree there is value in being able to demote an app from extras to testing.

But I fear providing facilities to demote from testing->devel will just end up causing more of the 'Power User' demographic to go enable the devel repository.

Re: The Testing is half empty
 

I'm responsible for the coordination of these improvements, but unfortunately Neils(X-Fade) that will implement the improvements is busy with things that have more priority at the moment. I think he will accept help if someone want to jump in and contribute some code.

Re: The Testing is half empty
 

Sorry, but what plans are you talking about? I have not been agreeing to any plans so far.

Call me stupid, but I do not quite understand how Linux and open source are related to your plans for Extras control mechanisms. Would you be so kind to elaborate?

Oh yes, haven't doubted this for a second...

Re: The Testing is half empty
 

Could you give a short summary of the plans Flandry is talking about? What are they, who made them, and how many people agreed to them?

Re: The Testing is half empty
 

Actually, i think that's the problem: there are no individuals relied upon. Users are in fact discouraged from testing by 'there-be-dragons' warnings. No wonder when i peek around the package list and look at comments, i only see a few familiar names/faces
I have a proposal: make -devel the snake lair. Regular users will be discouraged from using it, but those with the know will be urged to try it more, to compensate. The main objective would be to make sure the app doesn't kill the device or make it go crazy.
Once the app is declared sane, it can go to -testing. Since there are (thanks to the above paragraph) now a lot more users, things get tested quicker and more thoroughly (law of large numbers says faults come out better on a larger testing base).
And the normal extras gets the high-quality apps that should be showcased. That's not to say they can't have any bugs, but those should be minor and more like annoyances
Also, +1 for official testers

Re: The Testing is half empty
 

Ok would you please point out where this is at? Gitorious?

Re: The Testing is half empty
 

Re: The quarantine issue:

I think that the 10 day quarantine just slows down the whole system. It certaintly slows down me.

I promoted a package to testing that had a rather serious bug in it: one of the dialogs in the settings interface would not open. It was a rather small issue, because I'm not sure that hat many people would have a need to open that dialog, but for those that did try, it would be very confusing.

After 3 or 4 days into the testing process someone raised it to my attention and it was a simple bug, one line needed changing.

I did this and uploaded a fixed version to -devel. Problem was, this was during a builder outage and the builder would not build it. Finally three or four days later I got the package built and went to promote it to -testing. I saw the version with the bug in it that had been sitting in -testing had 11 karma. The 10 day quarantine was going to be up soon, but should I promote it to Extras and then queue the new version into -testing? With the bug?

I decided against it and threw away 9 days of quarantine and 11 positive votes for a one line change. This fixed version got the required karma in a short amount of time, but am still waiting for the quarantine to be up... (which I think is completely useless).

It's bad enough you have to get 10 positive votes even for a 1 line bug fix, why delay it any further?

Additionally, it says on the wiki:

http://wiki.maemo.org/Help_testing_software

So, how are any bugs going to be found in +10 karma apps if people aren't even testing them?

Re: The Testing is half empty
 

Well, that bit of wiki was my own edit from a while back when trying to streamline the process for new testers. The quarantine does still allow time for serious bugs to come out, but i think this is a good case for official testers covering the full spectrum of tests and user testers doing casual testing via usage during the rest of the process.

Re: The Testing is half empty
 

Amen. If those are the worst problems experienced lately after the dire warnings made over and over, I would say the current process is basically trouble-free. Except that programs that should be released to users haven't been. Since there is overprotection (a useful chess term!), some people have basically ignored the warnings. This is the same thing that happens when cigarette taxes get too high -- smuggling starts.

Whatever enhances the collaborative process between users and developers should be encouraged. Whatever discourages the collaborative process should be abandoned. Regulations should be minimized, and carefully chosen.The ultimate aim is NOT protection. The ultimate aim is fun or useful programs.

Re: The Testing is half empty
 

I think you missed the first page of the thread. You came in about the time i started throwing out pie-in-the-sky hypothetical suggestionsl because that's what was coming back at me. This is a fairly basic problem with fairly basic needs, as you and geneven and others have pointed out in response to my bait: we need to improve th testing process so it's not a barrier.

I started the thread to request that portions of the process that were discussed to be part of it actually be added. We don't really need ten testers; we need a thorough test of the app and 10 seemed like an arbitrarily good way to ensure that. We do't need 10 days of quarantine... " " "

So, let's deputize the really active testers as "official", give them pet criteria to cover if they want, make the new requirement 3 or 2 or 1 official tester who will verify all points as assigned, and leave the other testing to power users during the quarantine period.

And let's not make new versions go through the same process as first-time apps.

Edit: as for the individual vs community example, it's the difference between relying on one person (xfade in this case) vs a team or community to get things done. He's "busy with other, more important things" which says to me right there that there is too much hanging over that one person's head.

Re: The Testing is half empty
 

And that was the very first post at the very first page of the thread.

Judging form your initial post (first post at the first page of this thread) you actually wanted to add more barriers, in order to protect against some hypothetical security breaches.

Well it has long been agreed that 5 votes is usually sufficient. In fact, it was agreed in an IRC meeting months ago.

...and the only people willing to be "official testers" are going to be above-mentioned career wikipedia editors, then ones who vote package down "because it has no bugtracker URL" (kinda nonsensical, as each package has maintainer's name and email address).

This, again, has been agreed months ago.

Well, if you would like to take some job off XFade, creating more policies for him to implement is not likely to be the right way to do it :)

Re: The Testing is half empty
 

The weird people who test or edit wikipedia (much like the weird people who spend time arguing about computer platforms on an Internet forum) are exactly what we need in that role, and empowering them to follow the letter of a QA checklist is a great way to assure QA. That you don't agree with one or more items on the QA checklist is a different issue that ought to be pursued in a discussion on the topic of what constitutes accepable quality in an Extras app. The issue at hand is getting the established guidelines for Extras QA verified in a more expeditious manner. That was the thesis of my first post (but not the one you replied to, which you still seem to think was the OP) and is still my goal with this thread: rattle cages or squeak wheels until the means of getting the actual process in agreement with the agreed process is clear.

Re: The Testing is half empty
 

my personal frustrations with the process come because i promote to extras tesing to get broader feedback since most people are told not to go near devel.
but that doesn't mean i'm proposing that version to promote up.
i found that the first couple of times I promoted to extras-testing people gave thums up others gave feedback. i worked on andifxed issues, made more improvements, promote the next version...lose all vote history, each time i go round the cycle the program gets better but it recieves fewer votes/interest. there is no distinction between 'try and give feedback' versus 'i think it's ready for extras please vote'

Re: The Testing is half empty
 

I guess it makes no sense to argue beyond this point. No way you can persuade me that giving red button to a petty maniac is good.

Re: The Testing is half empty
 

Don't know if this changed during the server move(I've to ask), but should be here: http://trac.midgard-project.org/brow...maemo.packages

Re: The Testing is half empty
 

Don't know about his plans, but the ones that I'm talking about were agreed through several discussions and after that we had a meeting at the IRC(far far away :)), you were there AFAIR :) . Of course we also need Neils input on that.

Re: The Testing is half empty
 

i said nothing about making the thumbs down red. It is unfortunate that you can't give up the misconceptions you got from jumping in the thread late, but forunately for you we're actually both interested in the same outcome.

It does bring the thread (finally) back to the initial question, but now it appears there is even stranger discrepancies between the agreed process and the actual one. Why hasn't something as simple and useful as reducing the promotion karma to 5 been done? Changing one number in the code...

Re: The Testing is half empty
 

Yes, I remember the meeting. But I do not seem to remember any of the points proposed by Flandry agreed at that meeting. This is kinda troubling, too.

Re: The Testing is half empty
 

Well can't do anything about this, but he can (and should) give his opinion and propose improvements.

Just fixed the link above, now points directly to the package interface code.

Re: The Testing is half empty
 

It probably predates that meeting, and is marked "not final": http://wiki.maemo.org/Extras-testing...OT_FINAL_FIXME

If no longer actually part of the testing plan as discussed at said IRC meeting (which transcripts would be nice to see), that section should be removed by someone who is privvy to the information.

It seems the most sensible way to make sure every app is given at least a good testing (as opposed to the present system where it can be resolved entirely by popular vote), but that would mean giving those creepy weird careful tester people some specific tester status.

Nobody has given a sensible counter to that including you, who just seem to be bothered by those who will apply literally a set of guidelines.

Re: The Testing is half empty
 

Here is a summary of the possible improvements: http://wiki.maemo.org/Extras-testing...A_Improvements

This was not discussed yet with the maemo.org tema, and is far from final, feel free to suggest some improvements.

Re: The Testing is half empty
 

Can we look at the Debian project to get some ideas?

In Maemo, there is one thing that bothers me sometimes: the overreaction of people when someone points to a link in extras-testing or extras-devel. The you-will-be-eaten-by-dragons stuff.

I use Debian testing as my stable desktop. Of course there are hip-cups sometimes, but that's part of the game that I'm willing to play.

What if instead of all that overreaction, people could be asked some simple question: "Do you think you can troubleshoot, re-flash your device and restore your backup if anything goes wrong?"

1) "Yes"->extras-devel
2) "Maybe, if I follow a re-flash guide"->extra-testing
3) "I don't know what you are talking about" -> stay on extras only

maybe more people would show up, but they would be warned in a positive and constructive way.