First of all, I'd like to emphasize how important this task is and how much I appreciate Valerio's (et al) efforts. This is just not to misunderstand what I'm saying below, you have my full support and I'll try to be a good member of the bugsquad when the time comes. Let me elaborate:

If you want to test just one app, you should be able to do that, without any hassle, and be a tester. If tester here means something else (as in a tester who is a more devoted tester than an ordinary tester), it should be made clear. We don't want to scare people away with special procedures or by making them think they should do something special to be a 'tester' (i.e. not to loose people to nomenclature, like seen from RevdKathy's post).

The point is that IMHO it's the admin who *decides* what's a blocker based on the testing guidelines (and the actions above are just consequences of that decision). One of the major reasons to have an admin is exactly to filter the feedback so packages would not get invalid thumbs-downs because of misunderstandings.

[ plug ]Not really related, but I use AppWatch with great success for this. [ /plug ]

And this is my main beef. The testing squad should certainly help stuck apps, but should also stay clear of replacing users. If the person in charge of an app does not understand what an app is supposed to do (or interested in it), it's not realistic to expect a good QA process. Organized testings also have the problem that the different testing squad members may be in different time zones, different schedules, etc (for example most of my free time is weekends - that's more than 5 days, and I will hardly be able to do anything during workdays so anything that should be done by Friday is an automatic fail for me).

Stuff that is missed out from my QAmaster proposal altogether (just mentioning it so we are in the clear if this is intentional): the cooperation of tester admins with actual developers. Testing, if issues found, should ALWAYS result in a bug report(s) and, preferably a pointer for the developer how to resolve it. Remember, the goal is not to clean out extras-testing (that's just a consequence), but to help streamline promotion to Extras with a special focus on quality.

Another thing which is probably very hard to implement, so I'll just list it as an opinion. Per user thumbs down IMHO is a bad metric and is shows very little about what has (not) been checked and how much overlap there is among the reports. Not only should we have categories, but THAT is our score/karma. I should not care if there are a 100 thumbs-ups and just one thumb down, if that thumb down is really a thumb down (blocker). So the 'karma' of the app is, in fact, not the number of people who checked it, but the number of categories it has been checked on. This works the other way round, too - it's pointless to have 10 thumbs downs because of a common issue. It should be one thumb down (in the summary), which multiple users can confirm, with a bug report, end of story.

Now, if there were folks who read through this, happy to hear comments. No pressure, no matter what solution we end up with, it can't really be worse than the current situation