[extras-testing QA] giving thumbs up in testing without following QA
 

Just recognized apps getting "thumbs up" from "testers" which brakes with QA list!
What should be done about this?

Serious issue as if you get enough assigned testers doing so you can promote the app from testing to extras.

Example:
http://maemo.org/packages/package_in...egweled/0.9-7/

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

And the other way ... if you have a "niche" apps with less than 10 users your apps will never reach extras ...

I think each app in extras should require 5000 votes ... and any negative vote block promotion. So only perfect applications will reach extras ... and we will not have any problems anymore ... as we will not have any apps :)

EDIT : And everyone didn't agree with some rules setted in QA ... there is community votes ... nothing.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

in that case it doesn't matter. even when there are 10k thumbs up, it won't be promoted because of the lack of bucktracker.

and now you need 3 super tester votes, 10 ordinary isn't enough. that should be enough in order to get every criteria filled before promotion.

btw try searching sio2interactive thread here, misuse has already happened.

ps khertan set your own repo and you can be the one who decides about QA by yourself....

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

10 ordinary votes are enough, but you can also promote
the packages if there are less than 10, but at least 3 super-tester votes.
And this package got 3 super-tester votes. It is unlocked!
Even without fulfiling the QA-criteria.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

QA list should be shrunk. The number of required votes should be reduced from 10 to 5. Because even now, most people have switched from Extras to Extras-Testing or even Extras-Devel. Your golden goose is dead, starved by extensive restrictions on its diet. Get used to that.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

http://khertan.net/khertan_repository

But of course ... this mean that current version in extras are outdated, and buggy ;)

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

It’s about finding the right balance. It is inevitable that some people will not vote right and miss (to seasoned testing veterans) obvious blockers - that is why we have multiple votes and comments so we could figure out what’s going on. I personally hope that apps like kisstester will make it easier for people to leave feedback and with some automated tests, it will make it less error-prone than the current process.

Now, since this is a crowdsourcing effort, there is no guarantee, just a best effort that very broken or incompliant does not slip through. If it does, we poke the maintainer just as we would on any other fatal bug that might not be apparent when the app was pushed. That’s why super-tester votes are important when an app has only a few votes - if we just lowered vote-requirement, that, low number could be reached "accidentally". This actually works, we have practically halved our locked package numbers since super-tester activity started - but we DO need a lot more user activity so super-tester votes become a last resort method, not a de facto way of getting to unlock.

Khertan, there IS a community vote. I didn’t set these rules in place, some of them I agree with, some I don’t, but what we have is a very general compromise on what is necessary to provide a safe experience and provide us with a "enabled by default" state from Nokia. I very well understand the pain of not having 10 people to vote for you, I myself have such packages. Feel free to lobby for alternate solutions, and if the proposal gains traction, I will be more than happy to hand the new requirements over to Niels.

That said, Battle Gweled is a somewhat special case - when that package was submitted, IIRC the bugtracker rule was not yet perfectly known, optification was still in it’s infancy, etc (note - even Jaffa did a thumbs up on that one ! :) ). I don’t know if mikkov abandoned it or not, but I did not feel the package be in such a bad shape to require a manual removal.

In any case, we WILL have to come up with a mechanism to deal with orphaned packages (i.e. what to do if a package gets unlocked and the maintaner does not publish it for whatever reason). Thoughts welcome.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

One tiny remark to this. The goal of extras-testing is not to protect devel stage software from people willing to try it out, but to protect people from broken software who DON’T want to try it out. Extras-devel will ALWAYS be cooler than extras (or extras-testing). And as for actual statistics, I’ll try and see whether we can get a more detailed split from downloads so this "most people" thing can be put into perspective.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I just want to stress attila's point: testing and devel are easily available for those who want to use those applications. this leaves extras for one group: for those who really, absolutly only want to use well-tested software.... or those who don't know or care about technical details (and the additional repos). both groups are best served by a strict testing-procedure; i don't see anyone gaining anything anything from relaxed QA criteria.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I would go further and say its useful for everyone. I imagine I am more risk averse than most people on the forum but Im happy to use useful apps in devel and testing if I know that the risks are acceptable so I generally read through the relevent threads before installing such an application.

I trust the extras repository though and just install anything that looks interesting as I see it as being inherently low risk. I appreciate the quality control and would not like the borders between extras and testing/devel to become blurred.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Thanks for reminding us all of the original intent of having extras-testing. Unfortunately, that intent has now been replaced with "to have a repository where developers can upload their work for everyone to use", i.e. the original goal of Extras. This resulted from the non-functional promotion process and passive refusal by maemo.org admins to change it.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

just wait a while. there has been major leaps after kisstester came out....

I think the problem is lack of tools, not some criteria which all make sense.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I will repeat myself once more. "to have a repository where developers can upload their work for everyone to use" is extras-devel. Devs upload, and whoever wants to, uses, no guarantees or strings attached. If it burns your house down, too bad. Extras is "to have a repository where end-users can download reasonably functional/safe software from. It doesn’t mean of course that it is foolprof (even commercial QA isn’t that), but it means at least someone bothered to install it and see what it does.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Actually, no. Quoting from http://wiki.maemo.org/Extras-devel :

So, Extras-devel is not for everyone to use. It is only for people who really know what they are doing. As it stands now though, Extras-testing and Extras-devel are being used to just about everyone, because the actual repository destined for that use (Extras) has been rendered useless by obstructive promotion policies (i.e. QA guidelines and the number of required approvers) and mechanisms (i.e. the hideous web site UI).

We can argue about this topic forever, but you only need to relate to the definitions in the Wiki to see what the original purpose of each repository has been. Please, do so.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Okay, without making this a terminology debate, let's get a little more factual. First, I know first hand plenty of people not using it so it would be prudent to have at least some numbers before blanket statements to the contrary are made. Second, the super-tester effort is exactly reducing the number of required approvers (based on their previous input). Third, you don't need to use that hideous web site UI, I personally encourage you to use (and contribute to) kisstester.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

yes and no. yes you can add it easily to your repository list and it works just like any repository. don't get distracted by the warnings, those are there for the people who don't bother to do any research.

just tell me then how you would deal with repos? please keep in mind that in order extras to be enabled by default, the contents has to be tested and controlled.

I know devs don't want to write stuff that makes ones device to freeze but one guy can never test an app such thorougly unless he/she has dozen devices wirh different apps etc installed....

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

After a bit of pondering - this will be very hard to analyse. Comparing direct numbers is not really possible, as the download numbers do not reflect the ratio of users that have extras-devel enabled on a regular basis (e.g. if you install a version that is present in Extras, that will count as extras even if you have extras-devel enabled, but error is possible the othey way, too - just because someone turned extras-devel on for a single package doesn't mean he keeps it turned on constantly).

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Rather than ponder about it so extensively, why not compare the total numbers of downloads from Extras, Extras-Testing and Extras-Devel? That will give you an overall figure of how popular each repository is.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

sounds like people are thumbing up things that work for them rather than things that pass all specific testing criteria.


roll on KISSTester!
http://maemo.org/packages/view/kisstester/

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

As I said, unless you're able to link the number of downloads to number of users, it's pointless. As they get updates more often and will try out more packages, people tracking extras-devel are bound to have more downloads 'per person'. As we don't know HOW much more (2:1 ? 10:1 ? 100:1 ?), comparing number of downloads mean little with regard to comparing number of users (which would define 'repository popularity').

But all that is besides the point - as long as there ARE people who require such a safe-zone (and there are, even in this thread), if we can, we should provide them with such thing. If these people are not your target audience, why push software onto them by force ? Note that Harmattan will be even more susceptible to this - my bet would be that the vast majority of users of that device will be completely clueless when it comes to extras-devel type of stuff.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

To me, the number of downloads is a sufficient measure of repo popularity. You can, of course, produce all kinds of arguments to the contrary, but if Extras-Testing or Extras-Devel had several times more downloads than Extras, then the only conclusion is that they are way more popular than Extras.

Yes, I completely agree with your point. So, in order to serve these people, I propose to do the following:

1. Rename Extras to Extras-Safe
2. Rename Extras-Testing to Extras
3. Leave Extras-Devel as it is

Do you agree with my modest proposal?

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

What, exactly, do you imagine that would achieve though? If that were to happen, Nokia would have to switch the default enabled repo to your new "extras-safe" repo, and stuff in "extras" would still need to include warnings to end-users and would require testing from people who bother doing so.
End result, no change whatsoever from the current situation except the repos have different names.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I already mailed Niels to provide us with the numbers if possible so we'll see - I'm just highlight aspects so that people don't jump to conclusions, either way. I also asked if maybe we could get the number of downloads of the Packages file of the different repositories, that would be a far closer metric to what we are curious about here.

My question is who would this nomenclature change appeal to (as in, would it matter all that much, especially considering the potential confusion of swapping repositories) ? Or are you suggesting that we suggest Nokia to enable (what is now called) Extras-testing by default on the next firmware ?

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I am suggesting to name the repositories according to their actual function. If Extras-Testing is no longer being used for testing, but rather for general distribution of software (as it appears to be used now), then the word -Testing must be removed. On the other hand, if Extras is actually being used by people who want to be safe from untested software, as you suggested, then it makes sense to rename it to Extras-Safe to signify its new function. As you see, my proposal simply follows your original statement about the status of Extras.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

what problem would be solved by doing this?

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I would like to see PPA style repositories
so that people can subscribe to individual developers upgrades and improvements without having to muck around with general upgrades to things they have from stable repositories

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

lcuk : this is in progress with the Community Meego OBS ...

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

The problem of Extras being pretty much dead in the water, due to excessively restrictive promotion policies. Of course there is an easier solution to this problem: just relax the promotion policies. But a few "super-testers" in the crowd are obviously against this solution.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Extras is not "dead in the water"; it's in fact more successful than the ovi store where developers get both Nokias support and the chance to make money.

But then, your proposal wouldn't actually change anything:

Extras-safe (now extras) would be the one repo activated by default in Nokias firmwares. There's no way they'd ever activate a repository that contains applications that are not well tested.

Extras (now extras-testing) would not be activated by default. There would be a procedure for applications to be promoted from extras to extras safe. This procedure would be the same as the one we have now. also, as a consequence, there would be a warning for users that this new "extras" contains untested applications and they shouldn't install unless they really know what they're doing.

So what exactly are we talking about?

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

That is where things get really pitiful: when you are forced to compare it to the Ovi Store to show how "successful" it is :)

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

I don't think changing the names will help anything.

I personally would like to know whether the developer thinks a given program is ready for popular consumption or not. An actual comment by a developer might even be handy. Like "this might have a few bugs but is probably not dangerous".

I think that the hysterical warnings quoted above are less than helpful. They have the effect of warning fearful users not to try out programs that are perfectly safe in general.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Not really, since everything has to go through Extras-Devel and Extras-Testing - but not everything from Extras-Devel and Extras-Testing makes it to Extras. If I push 3 in-development versions to -devel, and it is installed by 10 people but only the last one works, that's 20 downloads of broken software and 10 downloads of working software.

When that version gets through to Extras and is used by 10 people, it gets ten downloads of working software.

As Maemo pushes updates for your whatever repo you have enabled, you could potentially see more "users of -devel" in this (not uncommon) scenario, than actually there are.

I think there are, however, useful stats we can use to gauge the success of the QA criteria and the -testing process. We could also measure these to see the effect of various changes and strive for the numbers to be as good as possible:

1. A comparison between the number of Section: user/* packages in Extras-Devel, Extras-Testing and Extras.
2. The average number of downloads per day for the current version of each user-facing package in each repo.
3. The average time it takes for a version of software which gets through to Extras to get through.
4. The number of "user/" packages older than 10 days in Extras-Testing.
5. The number of separate testers.
6. The average number of packages rated by each tester each month.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

The only problem I see is this (from Super-testers):
That is, testing karma of 100 is not any prof at all if the tester knows anything about what he's doing!

Cheers

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Agreed. If you take a look at my proposal you'll see that 100 karma is just a *prerequisite*, not a number that will automatically grant you that status (yes, the wiki excerpt misses that part of the proposal). In addition to the karma you need to be nominated/supported by a Council member or an existing tester based on your past activities - and those can be seen on the comments list. I hope this addresses your concerns.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Quick question; with reference to the following

'If you don't feel ready to join the testing Squad or don't have enough skills, you can still help by testing the applications from a user point of view (functionalities, description, etc...). For more information read article Help testing software.'

I am wanting to test/assist along these lines (and then maybe, when i have enough experience apply as a tester) and have done so for a few apps. Do i need to do so as per the QA checklist or, as I have been doing so far, from a general end-user perspective of basic looks, functionality, integration, navigation, completeness of app, etc?

I really would like to help but I dont want to damn any apps to the netherworlds as a result of not following the required QA process.

Any advice or suggestions would be welcome.

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

You can always leave feedback for the packages (even without voting), and it is most welcome from the aspect of both testers and package maintainers. Generally you should *not* be thumbing up packages just because they are able to start and look cool. OTOH if you DO run into problems that cause breakage, feel free to thumb down (with adequate explanation).

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Thanks for the feedback!

Re: [extras-testing QA] giving thumbs up in testing without following QA
 

Sorry for the thread bump, but I think I made one of those niche apps. The problem I run into is that votes are at 0 for every new version.

So let's say my app's got 7 votes after a long while, and I (or one of the testers) notice a bug that really needs to be fixed. I upload a new version, promote it to testing, and all those precious votes are gone :(
I don't want to harrass the testers to look into it again, of course..

I do understand that a version 0.3 can be safe, and that a version 0.3.1 might murder kittens, even though it looks like a bugfix to 0.3.

But this way, niche apps will indeed never reach Extras, not when updates are released on a regular basis (while updating and fixing should be encouraged).

Any thoughts on this?