Re: More testers needed to push apps out of extras-testing
 

I wrote about the process on the developer list in 1st of November. One of the issues was a missing manual QA/testing checklist and that everyone wouldn't need to test everything. Automated testing is then a bonus on top of this (and remove at least #1 and #6). As the testing interface is still the same, there is a queue, and it is not clear who has tested what - I made one text based checklist:

Testing checklist:
1. [ ] Bug database exist.
2. [ ] Licensing ok.
3. [ ] Working provided features.
4. [ ] No missing announced features.
5. [ ] Optification ok.
6. [ ] No performance problems.
7. [ ] No power management issues.
8. [ ] No illegal/dubious content.
9. [ ] No known security risks.

* Copy&paste this to the comment box.
* Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
* Vote up if there were no FAILs, if there was even one FAIL vote down.
* UI usability issues cannot be used as reason for vote down.
* Always test functionality - that is, run the program and try if it works as it should.

imaginary example:
1. [x] Bug database exist.
2. [ ] Licensing ok.
3. [x] Working provided features.
FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123).
FAIL: When exporting file the program crashes (see bug: http://url/456)
4. [ ] No missing announced features.
5. [x] Optification ok.
FAIL: the package uses 1512kb from root.
6. [ ] No performance problems.
7. [ ] No power management issues.
8. [ ] No illegal/dubious content.
9. [ ] No known security risks.

Re: More testers needed to push apps out of extras-testing
 

I added the list to the end of http://wiki.maemo.org/Extras-testing/QA_checklist - maybe I'll be then actually used.. :)

Re: More testers needed to push apps out of extras-testing
 

Thanks, that's a nice concise checklist for -testing and a nice addition to that wiki page. However, he was asking about tests that have passed before the app shows up in -testing, ie the automated tests. I'm not sure what they are at present.

Anyone who is going to start working on the wiki, please note my previous post so you don't reinvent the wheel.

One of the biggest problems we face is that people just don't read before they ask questions or even add new wiki pages. That means we end up with a lot of wasted effort and redundant content. Case in point: asking what optified means after i described it clearly a couple pages up. That innocent but already-answered question then took a few more posts that needn't have been made...

Re: More testers needed to push apps out of extras-testing
 

I know I am going to regret sticking my head above the parapet like this but....

Is there anyone or group overseeing this section of the wiki at present? I know that there has been a lot of effort gone into UX and documentation recently so if anyone knows then please shout out. I'm happy to put some leg work into collating things and try to put some words together but don't want to duplicate effort or step on anyone's toes.

Regarding sections in the wiki and pages can we gather suggestions here or should we start a new thread for that?

Re: More testers needed to push apps out of extras-testing
 

Hmm... a list of what the Automated testing covers will be good for n00bs such as myself, I agree.

Anyway, once my situation is clearer (job hunting right now so it is a bit hard to focus on all this) you can count me in, also (this sort of stuff has been my job for awhile so I'll do whatever I can to help :) ).

Re: More testers needed to push apps out of extras-testing
 

Would it be a possibility to get a group of people who's main testing responsibility is to check for optification.
When a new package is released to testing they check it, and post a comment saying if it's optified or not. This would save other people from having to do it.

Or at least we should encourage the first person testing to post a comment saying if the package is optified or not

Re: More testers needed to push apps out of extras-testing
 

From the wiki: https://wiki.maemo.org/Extras-devel#Promotion_checks

Short summary in user terms: It checks that the package can be installed without requiring adding additional repos, plus some copyright and naming stuff. Very basic QA right now, in other words.

Re: More testers needed to push apps out of extras-testing
 

Why not force optification...

Put everything in opt and symlink stuff into / if it is REALLY necessary (.

What about /home/opt/maemo/{bin,libs,share} is libtool run on these folders after installing a package. If so, all stuff could go here and optification could be forced.

I just check ${PATH} and /home/opt/bin is not in it.

if /home/opt/maemo/${libs,shared} is not handled by libtool, we might should LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/home/opt/maemo/libs"

Just some thoughts...

A maemo filesystem guide for developers would be greate. Where to put stuff..

Cheers
Bjoern

Re: More testers needed to push apps out of extras-testing
 

would it be possible to have an app tester application?

for example you can tell it to take a preinstall system snapshot eg memory and cpu usage, directories. could it also have a hook for the application installer to log all the install activity eg file locations. a task manager which can filter the application process like process monitor for windows giving you all the cpu and memory usage. having it log the data. all this data would make more sense to the people who develop the apps then the average user who wants to help. command line stuff is all well and good but people like simple with the ability to go advanced.

One thing that i think is important for app testing is the app enviorment details. what other apps are running app conflicts. Having some app to records this information and compress it to a nice zip file to upload would make bug tracking alot easier. could even write a parser for the bug tracking data you receive to check for consistant data like opt and rootfs usage.

sorry i'm not great at explaining whats in my head.

Re: More testers needed to push apps out of extras-testing
 

@smegheadz, this is a great idea and has been used successfully on windows software projects in my experience. depending on what is in your snapshot (registry, config files, installed files/locations/versions) it can be a very intensive and large output to then diff. I am speaking in windows terms, i know, as i'm still learning *nix and how its all wired.

I would envision something that that would check the above and the other environment variables (existing installed apps/versions/etc, memory footprint, CPU, etc) and then log it to a txt or xml file. then you can run the tool before and after a test app install or use and then copy it off to a desktop for analysis/diffing.

I'd be happy to help test such an app or work with any devs that are trying to plan and implement such.

However i would assume that Nokia/Maemo would already have something similar already in-house. Anybody on the 'inside' care to comment or share on what the process is for how the various quality gates are met prior to code making it to the Ovi store or any other 'official' repository?

Re: More testers needed to push apps out of extras-testing
 

The process of promoting an app from Extras-devel to Extras-testing should FAIL if the app is NOT optified.

That way all apps in Extra-testing would be OPTIFIED.

If necessary, demote all existing apps from Extras-testing back to Extras-devel and only let the OPTIFIED ones be re-promoted.

That way, the only repository that may contain a non optified app is Extras-devel.

Re: More testers needed to push apps out of extras-testing
 

Uh... that's great except there's currently no automated test for this. That's why it's on the list of things for testers to check...

Many of the things in the QA list could be automated, but it's easier to get people to test than to get them to write a testing app and test it. I'm sure that such code would be welcome, though.

Re: More testers needed to push apps out of extras-testing
 

that was the point for help testing software page:
being as simple instruction collection as possible.

then extras-testing can include all dev -stuff etc and new people to sw testing don't get frightened and run away..

Re: More testers needed to push apps out of extras-testing
 

Everything that could be automated should be as soon as possible. This should be the priority now! As the user base for Maemo devices increases, there will be more developers for it too. This will mean more applications to test and this backlog will become only worse if something is not done to make the testing faster/easier.

The problem is that many of these new users have a very limited experience with Linux (like me) or no previous experience at all. Therefore something as "simple" as checking optification might already discourage potential beta testers, who might be more than qualified to check the functionality of the application and provide valuable feedback.

Re: More testers needed to push apps out of extras-testing
 

My point is that given being OPTIFIED is virtually a critical requirement of any app, I think whatever needs to be done to AUTOMATE that check during the promotion from Extras-devel to Extras-testing should be done ASAP.

I realise almost any test *could* be automated if enough resources were dedicated to developing an appropriate testing app, but surely given how important being OPTIFIED is, we could at least get that automated ??

I'm surprised this wasn't one of the first things done once the problem was discovered and /OPT was chosen as the solution.

Seems crazy to allow more non-optified apps into the Eco system, even if there are warning everywhere about enabling Extras-testing.

Re: More testers needed to push apps out of extras-testing
 

From my experience this is what is considered part of a Build Acceptance Test and comes before any kind of integration is done with the larger code base. Terminology may differ depending on your environment, but the idea is that there are a class of bugs that are so critical that it is a waste of a Tester's time to install them if they fail. Sure we could have 10-50-500 people install an app while being helpful testers and then report the bug, but if we just filled up their memory and bricked the device, I'm sure they all would have preferred that a simple go-nogo test was done before they were put into a compromised situation, and will be Much less likely to help out with Testing in the future.

This has to be a problem already solved, and likely automated by Nokia/Maemo. If I'm just dreaming, will someone who has some info please let me know.

If Nokia/Maemo hasn't got the tooling or is able to share with the Community, then we Testers need to understand better what happens between -devel and -testing and how the promotion testing occurs and how it could be improved upon.

Re: More testers needed to push apps out of extras-testing
 

what happened to the alpha and beta naming and v0.4 etc of apps?
it's universal. everyone generally understands that terminology.

My suggestions for making the testing of apps better are:

have a dummy hello world type app for noobs and first time testers to help which covers some of the basic functions apps could have. have a guide what to check and they can learn etc.

have an application that takes system snapshots. (see my previous post page 5) maybe have it as a bug tracker app, allowing quick feedback to maemo.org

allow people to tag an app as testing it so a user can see who and how many people have downloaded it to test and they can message each other about tests they have run etc (already implamented)

Make it easier to do things, 1 login rather then 3 for the site. it just gets annoying. people will get lazy and just not bother because they have to make another account.

when i get my n900 i'd like to help in testing apps and making it so people who just want a cool phone are able to take part without being a nix hacker, majority of people are from a windows background and haven't really scratched dos or scripting. Theres more user's then developers in the world so making it easy for them to test safely is ideal, gives people who can develop more time to do the more advanced testing and developing

Re: More testers needed to push apps out of extras-testing
 

I am used to the Gentoo installation system.

One thing that comes in very handy is the "fake root filesystem".

Every ebuild installs itself in a fake root, replicating the directory structure there. If everything was okay during the install, this directory just gets (in simple words) copied to the live file system.

This could make testing a bit easier cause you can check if everything is optified without affecting the live filesystem. This done, copy those files to the live system and start testing the app.

Is there a similar mechanism in apt?

Cheers
Bjoern

Re: More testers needed to push apps out of extras-testing
 

Xournal for a while had three different versions in each repository.
There was package 13 in Extras, one in Extras-Testing and one in Extras-Devel (package 21).

I promoted package 21 to Testing a month ago and that's probably what you got when enabling Testing.

Yesterday, though, I promoted it to Extras, but you've got it already installed. So no need to update there.

Aniello

Re: More testers needed to push apps out of extras-testing
 

i think this should be stickied to make more people aware that they can help with bringing more apps to their n900

Re: More testers needed to push apps out of extras-testing
 

Then what's the point of the two repositories. I thought an app in the developer repo had some sort of peer review before being promoted to the testing repo. :confused:

Re: More testers needed to push apps out of extras-testing
 

I assumed -devel is for "release early, release often" for those comfortable with having to flash their N900s if, or more likely when, these go wrong. Once the developer is satisfied that his package is reasonably stable then it is moved to testing and then those who don't want bleading edge but are willing to have a little instability can find any major shop-stoppers before the rest of the world gets it.

Basically this is a variant of Debian, but leaving it up to the delevoper to promote to -testing and using voting to get from -testing, unlike Debian that uses severe bugs to handle migration from unstable to testing.

Re: More testers needed to push apps out of extras-testing
 

ok i have no problems in testing stuff out.. unless it bricks my phone

is it easy to flash it back and what risks are involved in flashing my n900

Re: More testers needed to push apps out of extras-testing
 

Apps in Extras-testing should be safe to install without any risks of bricking.
Flashing is easy and is done under 5 minutes.

Re: More testers needed to push apps out of extras-testing
 

http://maemo.org/packages/package_in.../0.6.7-0maemo/

this is required for Ur-Quan Masters game, which has enough votes already afaict. Works fine here. Please test.

Re: More testers needed to push apps out of extras-testing
 

Note by: this thread belongs to the "community" section.

There is *still* a major lack of active voters for extras-testing applications.

Extras-testing voting instructions are available here:

http://wiki.maemo.org/Extras-testing/QA_Checklist

If even a fraction of the active forum-goers would bother once a week to check through interesting applications in the QA testing list linked from the above page, apps would not have problems passing in 10 days if ready for extras.