Time to start pushing packages to extras-testing?
 

Let me say BIG THANK YOU to Jeremiah, Niels and whoever else has been involved in the development of the maemo.org packages interface - http://communitizer.blogspot.com/200...-packages.html

Now... let's have a look to the queue of apps that have been pushed from extras-devel to extras-testing:

http://maemo.org/packages/repository...ng_free_armel/

Zero! OK, makes sense. How do we get started, though?

People testing Maemo 5 inside Nokia are configuring extras-devel and trying out apps available there. However, the % of apps with dependency problems or other basic issues is relatively high. Enough to feel lazy about filing bugs at that stage.

So what about starting the process of developers pushing the packages that they think are ready to extras-testing? This will be more human testers (Nokia developers with units, SDK users...) concentrated in less but more solid applications.

Re: Time to start pushing packages to extras-testing?
 

I've read that dependencies in maemo 5 are a very big headache... I think we'll have lot of problems if something doesn't change :(

Re: Time to start pushing packages to extras-testing?
 

A final release announcement would give developers a reason to start polishing their apps for Fremantle.

Developing against a moving target like Fremantle that is hard to test in the current situation isn't fun:

• No task switcher in SDK (developer can't test integration of their apps)
• On-device software more recent than SDK contents (at least with beta1 this was the case for several weeks)
• Slow test cycles: upload, ask testers, wait for reply, ...
• On-device situations that are unreproducible in the SDK
• On-device features that are not emulated by the SDK
• Uncertain release date of final/frozen version

Having some kind of feature freeze or release announcement would help developers have a clear picture of what to expect in the end product and when to expect it. FLOSS developers write software in their free time.

Again, developers would have good reasons to get their apps running in Fremantle if they know the SDK contents are frozen (feature complete; only bugfixes) and they can expect a release in e.g. one month or whatever the timeframe is.

Re: Time to start pushing packages to extras-testing?
 

I resolved my dependency problem tonight, anyone with a real device and a Roku Soundbridge want to give Tabletbridge a try?

thanks,
Frank

Re: Time to start pushing packages to extras-testing?
 

Given the high developer relevance of this, I think maemo-developers should also be involved (if not the primary place of this discussion).

Otherwise, I'm echoing thp's thoughts entirely. The one bit of software I've published for Fremantle apparently has bugs, but I don't want more testing: I want to get the information necessary to fix those bugs first. Even once fixed, anyone using the accelerometers is going to have to have an awful lot of self-confidence to push something for wider testing.

Some of the "dependency problems" could also be being caused by the buggy implementation of the (not-quite-sure-if-it's-a-good-idea) third party package policy.

Re: Time to start pushing packages to extras-testing?
 

I don't know whether this is a topic for maemo-developers, here or a sprint task. I was just trying to kick off the discussion.

With all the problems you point out...

I still think it's worth to open the maemo-testing door. First because we need to test and probably fine tune the own promotion from devel to testing. Better to do it now that things are relatively calm than when a final release is out and everybody is in a rush.

Actually it is good to note that we haven't agreed on the criteria and process to promote a package from -devel to -extras (or have we?). That would be a good start. :) There was this maemo-developers thread and there is a wiki page, but no conclusions AFAIR.

In principle the promotion is meant to be automatic: the maintainer of a package thinks it meets the -testing criteria and pushes a botton. Then the automatic process runs some automatic checks and either promotes it or keeps it in -devel. So we need to agree on the criteria, see what can be automated now and put the trust on the developers for the rest.

My initial proposal for -testing quality criteria was:

We should add "The developers thing it's ready for testing". :)

And that's it. The fine-tuning and polishing belongs more to the promotion from extras-testing to extras. I agree with you that it is difficult for most hobbyist developers to go through this task without a final SDK, a product announced and an idea about the sales start date.

Clearly the current situation doesn't help getting applications polished. We can't give any dates for further SDK releases, but we consider this Beta 2 quite stable. Fremantle is in pure bugfix mode targeting the final release. The plan is to have a pre-final SDK release that will become the final either without changes or through a minor update synchronised with the Maemo 5 final release.

About the origin of the package dependency problems... The fact is with 120 apps around it's difficult to keep track of the broken dependencies created by the Package Policy, the ones caused by dependencies on deprecated components, the Python related problems... You don't know whether the developer is aware or not, whether it's in the top priority or anyway he is working on other stuff first, whether he actually wants to push that app to end users using Extras or is just playing around.

If the promotion to extras-testing is open then we can focus on the feedback of the developers willing to push their packages. And we can focus the human testing on the apps making it there.

Re: Time to start pushing packages to extras-testing?
 

This requirement doesn't make sense until there is Fremantle Testing category in downloads.

Applications which are in testing have to be clearly identified in downloads. Otherwise whole extras-testing -> extras stuff doesn't have any purpose because everyone will be using testing stuff unknowingly.

Re: Time to start pushing packages to extras-testing?
 

fwiw currently app entries in Downloads do have a label to state their quality:

Stable: http://maemo.org/downloads/product/O...personal-menu/

Beta: http://maemo.org/downloads/product/OS2008/aarddict/

Alpha: http://maemo.org/downloads/product/OS2008/gnumeric/

Should this be simply mapped to Extras, -testing and -devel?

Re: Time to start pushing packages to extras-testing?
 

I am really impressed by Niel's hard work on this project. He has done a lot of complicated coding to make the QA process easier for developers and brought a smart way to promote packages. His dedication to this community should be evident to everyone.

Re: Time to start pushing packages to extras-testing?
 

Let me elaborate a bit more:

- A developer is just playing with some packages in extras-devel: no Downloads page.

- A developer thinks the app is in a shape good enough to be promoted to -testing: Page created with Alpha label.

- An app makes it to extras-testing, ideally the page would be automatically updated to Beta within a week. Otherwise the change would be made manually.

- An apps makes it to Extras, ideally geta automatically updated to Stable quality. Otherwise the change is done manually.

Re: Time to start pushing packages to extras-testing?
 

I've just blogged about the maemo-select-menu-location issue. It seems that quite a lot of packages seem to have dependencies on this and this prevents promotion.


PS. It is Fremantle and Niels :)

Re: Time to start pushing packages to extras-testing?
 

I think we should not show any app which is not in Extras (or equivalent) in Downloads. Downloads should in my point of view be for end-user level users. Having packages from various repositories is the road to pain.

Keep in mind that Downloads clicks automatically get an .install and thus can enable these unstable repositories.

The reason why I haven't enabled Downloads for Fremantle yet, is because there are no packages in Fremantle Extras yet.

We should market the extras-testing queue for community testers and developers, but really keep this separate from what end-users see. The testing queue has commenting and favouriting, so even if there is no central bugtracker, people can leave their comments about the package there.

Re: Time to start pushing packages to extras-testing?
 

True, true, true!

That was proposed when someone like me could not dream of pages like http://maemo.org/packages/view/bluemaemo/ :)

Let's drop then the criteria of having a Downloads page. Good package description visible in the Application Manager is enough.

We will still need to find a way to check that an app willing to make it to Extras will have a proper Downloads page in place, but this belong to the discussion about promoting apps from -testing to Extras.

Re: Time to start pushing packages to extras-testing?
 

Just as a side note, this package won't work in Fremantle, this is the normal BlueMaemo version for Diablo, I think Jeremiah built and sent it to Fremantle extras-devel by mistake, and also a lot of the BlueMaemo dependencies aren't satisfied yet in Fremantle :( (python-efl), seems that the builder don't check for the dependencies in the right place, don't know if this is already fixed.

Re: Time to start pushing packages to extras-testing?
 

trying to promote package but cannot do it because it depends on a user package that should be promoted ... but I have promoted it ...

packages gnokii-gconf, libgnokii4-dev, libgnokii4

Re: Time to start pushing packages to extras-testing?
 

This is because I need to run the first few promotions manually, so I can see if everything goes the way it is supposed to go. The promotion and dependency checking code I wrote is quite large and complex, problems will probably only pop-up in real life testing. So please bear with me :)

Oh, btw. Please don't put libraries as section(category) user/*, gnokii seems to do that. End-users should not have to see the lib in their AM.

Re: Time to start pushing packages to extras-testing?
 

Apart from the earlier comments about difficulties in getting any app stabilised enough, I think there are three things necessary:

1. Clearer communication on what's required from app authors (there's a process doc which is the output of the earlier task. Not directly suitable for package maintainers.
2. Technology in place (maemian etc.) to do some basic automated checks (big warnings which have to be specifically overruled for any lib* package in section: user/..., for example)
3. Instructions for how to promote when you want to.

The last point is particularly important: I just went to my package and couldn't see a "Promote" button anywhere; despite looking.

Of course, with a bit of thought, it's obvious I have to promote an individual version - but then I have to promote the armel & x86 versions separately: a step back from the earlier extras-devel -> extras promotion interface, IMHO.

I think it's likely a developer will want to promote all the architectures, and OS versions, for the latest version of a package. Perhaps some kind of checkbox & promote button on a package; or a "Promote latest" button would help?

Ultimately, this is a question for Niels - he knows whether the software's stable and ready for more testing yet.

Re: Time to start pushing packages to extras-testing?
 

Agreed, I could really use some help on this part. I spent and am still spending a lot of hours on getting all related scripts/services into shape, so I didn't have time to write up any documentation yet. Or even start a discusssion...

When promotion is requested for the armel version for instance, we will promote the related source and i386 packages at the same time. This obviously needs to be documented. When I remove the manual step of promotion, you will see that the other package will be updated at the same time.

The promotion link actually only shows when all conditions for a package instance have been satisfied. This means that currently all dependencies must be available and you are maintainer of the package (and when in testing, karma value must be > 5). This way we prevent accidental promotions and also prevent packages with missing dependencies from entering the repository.

Re: Time to start pushing packages to extras-testing?
 

No problemo, thanks for the good work
I thought I'll try to promote just to help test the process so here you are :)

You're right : I changed it

Re: Time to start pushing packages to extras-testing?
 

I've added a check + warning to prevent this now:
http://maemo.org/packages/package_in...6.28cvs-maemo2

I will remove the first promotion from the queue manually, so you can promote the proper one.

Re: Time to start pushing packages to extras-testing?
 

Thanks I could then promote libgnokii4, gnokii and gnokii-gconf
but then there is no promote link on the phonelink page ...
I suppose I'm being too impatient ?

Edit : As I am now a maintainer of my package, the link is there
But I can't promote because this depends on gnokii-gconf which, I suppose, has not be internally processed yet ...

Fred

Re: Time to start pushing packages to extras-testing?
 

There are some issues with the tool doing the promotion inside the repository, Jeremiah needs to look at this first. I hope this issue will be fixed soon. I can't continue until it is, sorry to keep you waiting.

Re: Time to start pushing packages to extras-testing?
 

No Problem I'll keep on checking around here on and off till I see a sign ...

Re: Time to start pushing packages to extras-testing?
 

After waiting for a long time, I took matters into my own hands and created scripts for the backend of extras-testing.

Extras-testing is now available for Fremantle. All promotion requests have been processed and the package interface now has these packages in the QA queue.

When you have access to a Fremantle device, please test these applications and use favouriting to give the package karma. On the version details page of a package, you can see who has voted for your package and what the vote was.

Please use the comments system to report all issues you noticed for a certain package.

I'll keep an eye on the promotion part and will see if problems arise. Please ping me when something doesn't work as expected.

Keep in mind that I still run some parts of the promotion process by hand, so I can test for bugs. So there can be quite a time delay.

Re: Time to start pushing packages to extras-testing?
 

Good!

I have disabled extras-devel from my application manager and I will concentrate watching and helping testing apps reaching extras-testing.

First impressions:

- What is the canonical checklist testers should look up to give their thumb up or down?

- We are carrying again that "bug" from Midgard: forcing a title in the comments when actually not title is much better. :)

The rest feels good!

Re: Time to start pushing packages to extras-testing?
 

Who can promote packages to extras-testing ?

A outdated version of my app (BlueMaemo) was promoted, and the associated dependencies don't exist in extras-testing yet[1], and isn't the only one there it that issue, according to the package interface.
Also my app wasn't successfully tested under extras-devel yet :(

[1] - https://maemo.org/packages/package_i...luemaemo/0.3-5

Re: Time to start pushing packages to extras-testing?
 

I have started drafting http://wiki.maemo.org/Extras-testing

Feel free adding / improving.

Re: Time to start pushing packages to extras-testing?
 

When thumbing down, it should be mandatory to leave a comment. Comment could describe the problem or just link to bugzilla or somewhere.

Here's a thumb down, but I have no idea about the reason:
https://maemo.org/packages/package_i...0.5.3-1maemo2/

Re: Time to start pushing packages to extras-testing?
 

I did leave a comment, and not short! Will try again later...

Re: Time to start pushing packages to extras-testing?
 

Hmm, there seems to be a bug then.

Crimson Fields has a bugzilla too
https://bugs.maemo.org/enter_bug.cgi...imson%20Fields

Re: Time to start pushing packages to extras-testing?
 

I also posted a comment in the BlueMaemo page but doesn't show up either.

Actually I just posted a simple test comment again at http://maemo.org/packages/package_in...0.5.3-1maemo2/ but it doesn't show up.

Can someone else try, please?

Re: Time to start pushing packages to extras-testing?
 

Please don't use the BlueMaemo version on extras-testing as reference, as I said it's outdated and wasn't promoted by me.
Version 0.3.6 on extras-devel is the correct one :)

Thanks !!!

Ps -> My comment doesn't show up either

Re: Time to start pushing packages to extras-testing?
 

There seems to be a rights problem for adding comments. I'm investigating this now.

Re: Time to start pushing packages to extras-testing?
 

Comment issue should be fixed now. *bangs head against table*

I'll work on package removal from the QA queue, as some packages obviously shouldn't be there. (They were promoted before some checks were in place)

Re: Time to start pushing packages to extras-testing?
 

I have started adding quality criteria at http://wiki.maemo.org/Extras-testing#Quality_criteria based on various previous discussions. Have a look and comment, please.

The criteria need to be as discrete as possible, so it's easy for testers to give their thumbs up or down. They also need to be easy to evaluate, of course.

For instance, let's take the example of Security. Everybody agrees that an app causing security risks shouldn't make it to Extras. Fine. But how a tester can detect that? I guess the answer is that those testers should check that there are not known security bugs filed in bmo, or perhaps even security reports against dependent packages upstream. But in no way they are going to do the security checks themselves. Another problem is "how insecure" an app can be.

Similar examples can be found with missing features, system performance affected, etc.

Another question that comes to mind is the content: would maemo.org distribute apps dealing with pornography, racism...?

And then something that is not clear (to me at least) is what happens really when pressing thumbs up/down. Right, thumbs up means that to me this looks good to Extras. But does a thumb down mean that it should go back to extras-devel? Or just stay in extras-testing? How do we tell "still not ready for Extras but don't send it back to extras-devel"?

And finally, what are the criteria that needs to be met to push an app to extras? What about this:

- Minimum 11 days quarantine.
- Minimum 10 testers + 1 admin giving thumbs up.
- No thumbs down.

Re: Time to start pushing packages to extras-testing?
 

I think we have 3 possible outcomes:

Minimum age and minimum thumbs up reached:
Automatic promotion.

Minimum age reached, not enough votes:
Admin should evaluate app and promote/reject.

Minimum age reached, karma above value, but with some thumbs down:
Admin should evaluate app and promote/reject.

I don't think we should make a difference between admins and power users when counting karma. Voting an application down should not be a direct rejection, I'm leaning towards what we do in social news. A vote down there is a penalty of -4 votes.

Karma > 10, can mean 15 votes up and one vote down.

Re: Time to start pushing packages to extras-testing?
 

Thanks!

I don't know that there are any acceptable ones, maybe change it to "verified"?

There's a flip side to this: security fixes for packages that were already in extras should probably get a karma bump and prioritised for promotion, even if they come with some more "minor" faults (eg missing features or no screenshot).

Sounds good in principle, though the list of dependencies can be long and cumbersome to check in some cases.

But: what about security issues in libraries that could affect several (otherwise secure) apps? Should the insecure library packages be demoted thus making the apps that depend on them non-installable? Should those apps also be demoted until the dependencies are fixed? Maybe, maybe not, but the severity is multiplied in a case like this. Perhaps we need some sort of "emergency response" team for security issues in general, but that's starting to get off-topic.

That should be grounds for removal from all repositories IMHO.

Hm, a can of worms that one. I'd prefer we never saw those kinds of apps personally, but I would like arbitrary censorship even less. I guess ultimately Nokia would be legally responsible for whatever is published, so we should go with whatever Finnish law allows.

(I assume that content refers to what is present in the packages themselves, rather than content that could potentially be accessed by the application, so it wouldn't apply for example to an email client that could at some point receive a v!@gra spam containing dubious images).

Re: Time to start pushing packages to extras-testing?
 

IMHO, I think there should be a nice documented way to allow the non-usual group of testers to pull the apps they want from the extras-testing repository then disable it (so not to mess with other parts of the system or applications they do not want to test).This may help obscurer applications.

Re: Time to start pushing packages to extras-testing?
 

Ok, but... If the entry barrier to become a extras-testing evaluator is as low as just registering to maemo.org then it would be quite easy to abuse the system. Not getting a news piece in the maemo.org frontpage because of 1-2 thumbs down of unclear reasoning is not that bad, but having an app stuck in extras-testing for a similar reason because of 1-2 "unknown" votes might be more problematic.

Would a minimum karma requirement (say 100) make sense here?

Re: Time to start pushing packages to extras-testing?
 

A list of voters is shown in the details page, so you can see who voted. This makes it easier to spot if somebody is creating a lot of accounts.

How about this:

The application needs to have > 10 karma, of which at least 3 positive votes of (people with karma > 100 OR people in specially created community testers group)

We could create a community testers group where we add people who have a proven track record within the maemo.org community?