Re: Seriously! Extras-testing procedures stink
 

Yet, if detailed manually filled change-log would be required to transfer the votes, it might give a rather good indication what needs to be tested again. I agree that whenever and update is released, testing is needed. But how much, depends on the amount and type of changes. Also, it should never mean that it's a completely new app, because that it's not... in most of the cases. ;)

Re: Seriously! Extras-testing procedures stink
 

That might work: if the developer believes that the app shouldn't need full re-testing, s/he could apply to have previous votes ported, with a full change-log. Alternatively, s/he would have the choice to simply re-upload the app and start the testing process again.

That would reduce the workload on the 'testing-masters' to only those situations (such as adding a help file) where the dev is very confident that the changes don't affect stability.

Re: Seriously! Extras-testing procedures stink
 

Which is raised in every single other discussion about Extras-testing. Seriously, this is fragmenting the discussion and we'll just end up with different sub-groups making the same points over and over again - or, worse, different people thinking different solutions are going to be enacted.

It's also worth mentioning that talk.maemo.org has yet to cover anything which hasn't already been discussed on maemo-developers. I suggest anyone who wants to really get involved and make this better starts reading there, and then the other threads which try to take a step back here.

I move this thread is closed, or merged into one of the many existing ones.

Re: Seriously! Extras-testing procedures stink
 

PS. anyone who thinks that any repackage or change to the source code can't introduce a large and horrible regression has never done any serious programming :-)

Re: Seriously! Extras-testing procedures stink
 

I always take direct questions seriously. I do not take warnings seriously. Any warning I issue is issued probably for good reason.. but will likely never make any sense :D.

ETA: I'm with Jaffa and Tex ^^.. should be merged.

Re: Seriously! Extras-testing procedures stink
 

Should have been merged three pages ago. ;)

Re: Seriously! Extras-testing procedures stink
 

Humph. Had someone used "Report This" and suggested a specific, existing thread for the merge, it may have come sooner.

EDIT: Okay, apparently Texrat did do exactly that. (See below.) I haven't received the resulting email message though.

Anyway, the thread "Seriously! Extras-testing procedures stink" with twenty-six posts has been merged into this thread.

Re: Seriously! Extras-testing procedures stink
 

I did. ;)

Re: Seriously! Extras-testing procedures stink
 

Yeah, sorry for causing unnecessary work by starting a new thread. :o

Re: Extras-testing QA Checklist and Quarantine Period
 

is this now to be merged with http://talk.maemo.org/showthread.php?t=41179
?? just received on for the stinks thread to be merged over to it... whole thread or from which post on?

Re: Seriously! Extras-testing procedures stink
 

I have not looked in detail into the approval process for apps so apologies if i say something totally off. From what I understand and app needs a certain amount of positive votes in order to get out of testing.

What I wonder is:

o are there defined test cases for a project/applications
- unit tests for especially scripted applications (python ;)
=> I would expect the answer to be yes, as most projects get ported and hopefully have the tests executed that many bring along.
- functional tests for well - functionality.
=> I saw these in the wiki, however not very well organized at the time I checked
- integration tests - to check that components that can interact actually do that as expected ;)
=> also saw these in the wiki, just not too much of them (i was loking for n800 testing, it might have improved in the meantime)


o are results from running these test tracked

This is the more interesting item from my point of view. If I understand it right then only the 'summery' from the 'testers' gets collected and once the 'PASSED' level reaches a certain point, the application is considerd tested successfully.
The problem i see there - and that is actually also the main reason for me answering on that topic at all - is that this process can be easily cheated, which in fact already happens. Another maemo developer whom i know private asked me to please give him the thumbs up so his package finally makes it into extras ...
I guess this is not the only occasion of this happening.

I can't say what the easiest technical approach there would be but know that there exists test tracking software for manual test cases that is open source. There is even a test extension for bugzilla itself called 'testopia' - it's pretty bloated and can be quite slow - so I'd look for something simpler but I really don;t care how the testing gets tracked in the end as long as it's tracked.

On the topic of rerunning all tests on a version/release upgrade - when the test cases are sorted in several dimensions, one can start selecting tests specifically for the area changes in the upgrade happened and run this subset instead of all tests.


o is it easy to add test cases?

In my opinion there have not to be that many testcases, just something more than the big "thumbs up / thumbs down".

More important is that anyone involved is able to easy add tests he thinks might cover functionality he needs and that this gets propagated. These most likely will be of varying quality but if you have some test case staging area and some regular reviews then I would expect that coverage especially in the area of functional tests will rise fast.

Also, having such a system, Nokia could add their own tests for new features as well as upcoming bugfixes and give dedicted beta testers a better guide on what to check.

Maybe all these things got covered in the meantime, when i was interested in n800 testing a year ago, the major problem in my pov
was the missing tool to track test results.



Once you have some defined tests, it becomes possible to better determine the actual quality of the application and also interested users that would like to use it, but are not sure if something from extras-testing might screw their device, can see a more detailed overview on what they have to expect when package is installed

Examples of basic tests I personally would expect having to at least pass to get stuff even into testing (i would expect this is already written down somewhere ;)

o package integraty
- goes to /opt
- no errors from install scripts in the package
- ...

o basic function
- appears in the menu
- can be started (very importan for python apps ;)
- does not exceed defined cpu usage per time if not used
- does not exceed defined disk space usage per time
- runs as user
- can be terminated from gui without remains like processes, temporal files and alike


o normal function (these now of course depend on the application, so I list the some I'd like to see for an image viewer)
- opens an image
- can rotate
- can zoom
- can handle X amount of images in defined time
- ...


When these succeed, put the package in testing and let the real features and advanced functionality of an app be tested, like changing options, attaching tags, and so on.

I myself install randomly software from testing and devel as I generally know what I'm doing and have no problem with reflashing the device. So I would love to give feedback on what actually works, and not only the bugs i see and that feedback could help users as well as developers.

Tell me where to report results - and please don't send me to a wiki page ;)

Re: Extras-testing QA Checklist and Quarantine Period
 

Let's leave this one the master and merge the other into it.

Re: Extras-testing QA Checklist and Quarantine Period
 

What is the actual fact -- isn't it impossible to make bug-free software? Is't that a byproduct of the Godel principle that you can't construct a complete mathematics, or something?

I agree with the Texrat idea "a single trusted gatekeeper with full control' till things are straightened out. Of course, the identity of the gatekeeper is important. I nominate Texrat.

Endless delay at this critical juncture is not a good solution -- it's what we seem to have now. The N900 is not going to be a new device forever.

Ever read the book Blink, which suggests that a flash impression is often better than a carefully considered one?

Re: Seriously! Extras-testing procedures stink
 

this is actually an old Unix Sys Admin "joke", as quite often you might type "rm -rf ./*" (i.e. remove everything below the current directory), but if you miss the "." it wipes everything off the filesystem.

Hence, experienced admins developed methods of avoiding situations of having to type that type of thing due to having at least once accidentally done it themselves.
The most common reason for having to type this, was to remove some software.

Re: Extras-testing QA Checklist and Quarantine Period
 

I actually wouldn't mind the work but I don't think I'm qualified yet.

Re: Extras-testing QA Checklist and Quarantine Period
 

I'm sure you would do a much better job than the current system. ;)

Re: Extras-testing QA Checklist and Quarantine Period
 

Heh. First I need my replacement N900...

Re: Extras-testing QA Checklist and Quarantine Period
 

There we have it! You already have tested vigorously the USB port, so surely you can test apps too... ;) :D

Re: Extras-testing QA Checklist and Quarantine Period
 

Oh now you're scaring me. No more usb port testing!

Re: Extras-testing QA Checklist and Quarantine Period
 

Please stay on topic.

Re: Extras-testing QA Checklist and Quarantine Period
 

A minor deviation now and then doesn't kill the discussion, chemist. In fact it can help recharge one that stalls.

Re: Extras-testing QA Checklist and Quarantine Period
 

I've been thinking about this single Gatekeeper thing. I don't think one is enough. The amount of apps/updates coming out already now is fairly considerable and it will only increase. In my opinion, this would be a too big load for just one Gatekeeper.

What I propose instead, would be several Gatekeepers who would have the keys to the Extras kingdom. Each of them would have the power to allow or reject the app/update. The testers would only make recommendations based on their testing. This would eliminate the need for rigid quarantine time.

Also a Gatekeeper could communicate with the devs with the app comments. Something like this:

Then the dev could inform the testers via the app comment what he has done, so the tester would know what they should test.

Re: Extras-testing QA Checklist and Quarantine Period
 

A bit late to the party but... we're rehashing ideas now, we should get to an action phase. I have myself suggested over a month ago a similar idea (just called it QAmaster instead of Gatekeeper), and even started knocking on some doors with the idea, to a mixed response. Here's a snipped of a further letter of mine on the topic:

In the end, it's important to have a solid support on this from the powers that be - qamastering (or gatekeeping, doesn't matter how you call it) is not a trivial task, and, as essentially a position of power, has the potential to rub some people the wrong way. If we (as in community) can reach a conclusion to accept such member position(s), the exact workflow, number of people, etc, can be worked out on sprints. If there is no support for a position of such role and authority, all the workflow discussions are not worth much.

Re: Extras-testing QA Checklist and Quarantine Period
 

I think the council is behind getting this done.

It is pretty clear to me that everyone is clamoring for more apps on the N900, and if every change in a program requires at least 10 days to implement (and much much longer if a bug is found during the -testing phase), it can take months to get apps into Extras.

I've been told that the problem is that the people who actually do this stuff (the paid staff) are totally swamped with infrastructure problems at the moment. There's no spare capacity for making these kinds of changes. But it is definitely something to put on next month's sprint.

Re: Extras-testing QA Checklist and Quarantine Period
 

Well, can't we have a simple poll? QAmaster/Gatekeeprer: yes or no?

Re: Extras-testing QA Checklist and Quarantine Period
 

And as long as people talk to X-Fade about how to implement it, there's no reason it has to be done by a paid contributor. See the source-code link here:

http://talk.maemo.org/showpost.php?p...6&postcount=16

Re: Extras-testing QA Checklist and Quarantine Period
 

That sounds like a poor idea: polls are self-selecting, people aren't well communicated to, they're easy to manipulate, and it suggests that the majority are as wise as the experts.

We have representatives and stakeholders for a reason :-)

Re: Extras-testing QA Checklist and Quarantine Period
 

Well, could my representative, whoever he/she is, make a move and vote for this change then? :p

I understand that there are more pressing issues currently. But a decision about this issue could be made and then inform about the outcome to us poorly informed and easily manipulated peasants of the maemo.org realm. This way we could go on ploughing and sowing code for new apps, while having this inner feeling of tranquillity resulting from knowing that our benevolent feudal lords are aware of our blight and have taken steps to alleviate our suffering. :rolleyes:

Re: Extras-testing QA Checklist and Quarantine Period
 

Our proposal is a team of testers(Testing squad :) ), no only one Gatekeeper, some of the members of the team will have special powers, that should be only used in very special situations.

More on that very soon, I only have to check a few more things before proceed.

Re: Extras-testing QA Checklist and Quarantine Period
 

Just couple of short questions. Where is it advised that developers should/could make "[Announce]" topic to talk>application area? Could not find from wiki or have I just failed in reading. Actually it could be quite hard to get 10 votes without making announce to talk area.

That just came to my mind after noticing new version of Battery-Eye in extras-testing and going garage page and just thinking that why would I use that forum there to talk to developer? Feature requests and bugs yeah but so called general talking. Yes i know that there is forum also but for me it just feels bit weird. Too many places..makes me dizzy.

.edit
Oh maybe this was wrong topic?

Re: Extras-testing QA Checklist and Quarantine Period
 

That kind of announcements at talk are always welcome, even if you don't need voters, more people will be aware of your apps, they will try it and post some feedback there.

You can ignore the garage forum, is what I do :D

Re: Extras-testing QA Checklist and Quarantine Period
 

VDVsx,
Thanks, but my questions were basically to admins and people who manage extras-testing process or who wrote those rules because I think that there should be notation in WIKI about announce thing. Right now it seems that it's just thing that some developers do and some do not and we might not be sure if it is just lack of information or just that developers just doesn't want to announce even thought he/she wants votes to get extras.

Why would anyone promote to extras-testing if he/she would not want to eventually go to extras? You can stay in development and just fool around there.

Re: Extras-testing QA Checklist and Quarantine Period
 

I was wondering about that, after a discussion at IRC I realized that some devs want their apps in -testing because is more safe for users than -devel, but they are not willing to promote them/or the apps still have some issues, so announcements are advisable but not a requirement.

Re: Extras-testing QA Checklist and Quarantine Period
 

I also ran across a developer recently who stated that not following the QA rules is more important to him personally than seeing his package in Extras. - So there's all kinds of people out there, and their priorities and personal preferences are mixed and sometimes unpredictable. In the end you can only accept it.

Re: Extras-testing QA Checklist and Quarantine Period
 

Why should we accept someone who flaunts the process just to feed their ego? Someone with that mindset isn't exactly a true contributor. Let them manage everything on their own, including repositories... we don't need that attitude.

Re: Extras-testing QA Checklist and Quarantine Period
 

But please, use the same thread for each new release, or something. Unless there is a hugely active thread about an app, there's no need for a separate one for every iteration!

It is confusing for people, though. Actually, garage in general is.I still find myself wondering if i should be putting apps into garage rather than just -devel. Is the general perception now that garage is deprecated?

/me builds a bridge and a spur and reroutes the thread onto its track.

Is it "very soon" yet?

Re: Extras-testing QA Checklist and Quarantine Period
 

No, garage is up&running, you can use the resources there as you like, personally only use garage SVN/GIT and the bugtracker for a small project.

Really don't know, I don't have time nor knowledge to implement these changes, so depending on others, but I guess php coders are welcome.

Re: Extras-testing QA Checklist and Quarantine Period
 

I had a dream. In my dream, we were not thinking in terms of garage, git, svn, t.m.o., bugtracker links, screenshots, icons, but we had Maemo Groups. A group could be centered around anything, but most often these were applications. Users could see pointers to all the resources related to that group, regardless of where they're actually kept. No traversing t.m.o. to see if there is a thread about it, no hunting for bugtrackers, source repositories, mailing lists, reviews, screenshots, cauldron lists, testing status, whatever.To reitate, it's not about integrating everything into a single site, it's about having a single place where people can find out everything about a project, *regardless* of where it stores it's resources (google code for source, b.m.o. for bugs, launchpad for tasks, t.m.o. for feedback = 4 different places supporting one project ? fine !). The same place could be used to administer maemo specific aspects of projects ((re)set bugtracker links, screenshots, icons, etc).

Re: Extras-testing QA Checklist and Quarantine Period
 

To attila77's point: this is why I think the governing web site needs to be a portal rather than a typical site. Hopefully MeeGo will become one.

Re: Extras-testing QA Checklist and Quarantine Period
 

That's what i'd like to see, too. It was sort of the idea behind my "official thread" request for devs in the Games forum (e.g. http://talk.maemo.org/showthread.php?t=40903), but to work right it really has to be a first-class citizen: it doesn't matter if it's implemented as a wiki, forum, or custom web app, as long as it is the first place people end up when looking for app-specific information, and has delegable edit rights (something a forum post doesn't have...but at least a thread does via replies).

While that would require some work up front, it actually reduces the load for the web developers and maintainers overall as the community developers don't have to spend time integrating some of the resources and can defer the linking of the relevant resources to the dev/maintainer, who can in turn save time by not trying to follow disparate streams and being able to delegate to fans in the case of really busy/popular apps.