Extras-testing marathon
 

As suggested in the community mailing list, and in order to get more software into Extras, I'm thinking in organize a Extras-testing marathon, this can be a starting point for future similar initiatives.

My plans are:

- Get a lot of N900 owners and developers with applications in the extras-testing repository in a IRC channel.
- Run through the list of packages, testing all of them, leaving comments and reporting bugs. Always encouraging the discussion with the developer(s) attending the session.

More fun things can be done. Suggestions ?

Re: Extras-testing marathon
 

It feels almost stupid that this has to be organized at all because this should ALREADY BE HAPPENING ANYWAYS. 300 people got N900s at Summit and apparently aren't interested in helping at all. Incredible.

Add another option... "Anytime", please.

Re: Extras-testing marathon
 

hum ...

I didn't have time to test things made by other ... as my primary goal is to made my own application available for testing ...

And as at the maemo summit there was many developpers, i think most of them are trying to port their application, instead of testing the other available.

But i ll try to found some time to do it.

Re: Extras-testing marathon
 

That's fine, but you can't tell me you didn't install ANY apps from extras-devel or extras-testing. And if you have, it just takes a second to thumb up or thumb down.

We're just asking you to take a moment of your time if you've tried out the apps there, that's all. We're not expecting extensive testing.

Re: Extras-testing marathon
 

I think one issue is the package interface is a separate flow from using the apps.

One idea is something like red pill mode, if there was a testers mode where we could report from the app manager and be requested to report on upgrade and uninstall, I think there would be a lot more feedback.

Re: Extras-testing marathon
 

It'd be a great project to really get things moving on the hildon-application-manager development collaboration.

Re: Extras-testing marathon
 

I suspect most people using apps from extras-testing don't even know how to get to the page where they can register their thumbs-up or thumbs-down.

A link to the extras-testing page should be spammed everywhere, on the front of maemo.org downloads, in the mailing lists, here on talk...

This is the Extras Testing QA link! Spread it far and wide!

(I keep forgetting myself :( )

Re: Extras-testing marathon
 

spamming the link is good, but it still does not get the correct eyes infront of the correct app tickbox.

apps installed from extras-testing should/could be added to a special list on the device.
then using the same "Updates available" ui, flash and solicit feedback directly on the device sometime after installation, or when the user goes back into the app manager "installed application" list.

"Its been 24hours since you installed uber widget, what do you think?"

Re: Extras-testing marathon
 

khertan,

the idea is collaboration.
Its not just a chance for you to test other peoples apps.
its a chance for other people to test your apps.

Re: Extras-testing marathon
 

btw, the sooner we decide on a day, the more chance people will have to prepare their apps for that day.
i've chosen the saturday partly because everyone else has, and partly because then it gives a couple of extra valuable days for developers test/debug things themselves so we get as many through as possible.

daniel, will the servers cope with a burst of activity? :D

Re: Extras-testing marathon
 

Small comment: I know this is aimed at those who got devices at the Summit. But if you wanted a wider audience, you might want to delay until after the due despatch date in most areas.

Re: Extras-testing marathon
 

No need, we can do more sessions later, depending on the number of applications in testing.

Re: Extras-testing marathon
 

Come on, Jay, that's a huge overstatement. I've already seen *some* activity arising from the 300 device handout.

But in my case, I wanted to use it at least a week before entering bugs or blogging about the experience. I've already discovered that several bugs aren't really bugs per se but rather possible enhancement requests or me misunderstanding a purpose. So I see some value in taking stock of the devices THEN taking some big plunges. Like this one.

Re: Extras-testing marathon
 

(Yes, some activity. I'm not talking about development work at all here in this thread.)

I don't think it's that much of a huge overstatement at all, actually. I've only seen, at most, maybe four or five new names appear on http://www.maemo.org/packages/ that weren't there already before Summit. No packages have seen massive jumps in thumbs up like we were expecting.

There's nothing wrong with wanting to savour the taste of the N900 before diving in with filing bugs. That takes time to do properly. It takes more information and work and reproduction steps and all that jazz. That's understandable.

Clicking a thumbs up or thumbs down to say "works for me", which takes all of 10 seconds to do, should not somehow get in the way of you taking your time with your device before filing bugs. In fact, it's the PERFECT time to be doing it because you're going to be trying out all sorts of packages to play with on your fun new toy.

It's just very discouraging when 300 people got to go to Summit, get new devices and then don't bother to take the time to vote on a package or two in return. I didn't expect everyone to do it... but at least a somewhat decent amount. 50 out of 300 people voting on at least one package shouldn't be too much to ask, but apparently it was. And I can totally understand that if these were normal end users who didn't want to bother or don't know how or whatever... but these are developers and power users.

I wasn't there, but maybe Nokia didn't push the idea of voting on packages to the 300. I don't know. And now I'm just rambling because I'm just frustrated.

Re: Extras-testing marathon
 

I agree with Texrat - I have many enhancement requests, based upon how I feel things could be improved with the basic OS.

I have only installed 3 applications from Extras-Testing (Dataplan Monitor, WordPy and Mauku), and again, no bugs encountered. I guess I should then give thumbs up where appropriate - I did not even know the existence of this page until qole put it out there. Thanks!

Re: Extras-testing marathon
 

The point is to have as much applications in Extras (stable and working ones, of course) available before the devices are shipped. Of course, with Nokia choosing to ship them, as always, with Extras disabled, not sure what the rush is...

Re: Extras-testing marathon
 

zerojay - you've hit one frustration point that I have with the loaned devices. We were told that we had them for evaluation, but we were not told how to provide any feedback! At least this information was not provided publicly at the Summit.

Re: Extras-testing marathon
 

I guess it also doesn't help that the interface is somewhat confusing. Be sure you're clicking on the particular version of the package you are testing so that you can give it a thumbs up from this page: http://maemo.org/packages/repository...xtras-testing/

Re: Extras-testing marathon
 

Maybe they knew all along they were going to ship with Extras disabled, so they just didn't bring it up to you guys. *sigh*

Re: Extras-testing marathon
 

Is it really disabled? (source)

edit: or is it really enabled? (source) (I have no idea)

Re: Extras-testing marathon
 

Well, you've flashed the latest firmware image, you saw its disabled state, right? :)

And: https://bugs.maemo.org/show_bug.cgi?id=3279#c5

Re: Extras-testing marathon
 

Not really because somebody flashed it for me and he forced me to make a backup :) but thanks for the source!

Re: Extras-testing marathon
 

Last word I had from Quim was that it was going to ship disabled but they might change that in PR 1.1. . . .

Re: Extras-testing marathon
 

Priorities. Right now finding, reporting and triaging official platform/apps bugs is much more important, because once the thing is released it'll be months before we see an update. Extras on the other hand are much more flexible.

Re: Extras-testing marathon
 

Your response makes sense until you realize it just takes seconds to vote on packages anyway. Also, end users don't want to pick up their new phone and see nothing but opensshd being the only app they can install.

Believe me, I fully understand the importance of testing and filing bugs - look at my maemo.org profile - and if it took any longer than a few seconds to vote, I'd fully agree.

Re: Extras-testing marathon
 

As long as some one like me doesn't have to resubmit something like LIBICU42. A almost 2 hour build time for the Armel port. ;-) At least the x86 version is only about 30 minutes. But 2.5 hours for a single package to be built would kill some serious productivity on the "party" day.

Nathan.

Re: Extras-testing marathon
 

Yes, I've just been thinking that it would be wonderfully helpful if the Extras-testing armel version of a package was more prominent than the others. That's the one most people are expected to rate, but it's lumped (in apparently random order) in with three other very similar links.

I'm fairly certain I mistakenly rated the i386 version of Hermes the first time I ventured into the system, and I know very well the difference between i386 and ARMEL and where each is used. It's just far too easy to follow the wrong link.

Re: Extras-testing marathon
 

Please don't do that - either test properly (which takes time) and then vote, or not at all.

With my "end user" hat on, I prefer a stable working system first, even if it means waiting another couple of weeks for extras. But that's just me.

Re: Extras-testing marathon
 

Yes, at first it's a bit confusing, but after some time there you learn to avoid the noise ;-)

My use of it now, it's go here: http://maemo.org/packages/repository...xtras-testing/

And press the version number, it will go to the right place where you should vote and comment.

Re: Extras-testing marathon
 

Some experience about testing apps in wxtras testing.

It takes at least for me quite a long time see if app is really working properly. I will not give app thumb up only for installing correctly and running. I see that there should be entry for app in bugzilla etc. It should also be possible to remove app without any problems and big files left behind. I have given few thumb ups only on few applications and filled some bugs for some of them. This all have taken already hours to do for few apps.

For example as simple application as password safe had some bugs and usability issues so I haven't yet given it thumb up but as it's working I'm bit hesitant to give it thumb down as some other testers might see it already good candidate for extras.

Many apps in the extras testing needs in my opinnion some love before they are ready for extras even if there are no any bugs on them. I would like that extras is for apps that are really good and not just quick ports.

Re: Extras-testing marathon
 

What mve said.

I spoke too simply previously. It's not just bugs I've been wrong about, but app usability/enjoyability/etc. So, yeah, I like to play with things a while. But committing to a marathon on my day off from work should do the trick... :D

Re: Extras-testing marathon
 

Uh... I didn't once say "don't test properly", so please, don't put words into my mouth. I'm talking about voting on the apps you're already using anyway, not clicking thumbs up just because they install cleanly.

And by the way, users much prefer having OS bugs than not having apps available. Look at how many people bought and keep using their iPhones even though iPhones seem to drop 10-30% of their calls (depending on who you ask). That's the most major function on a phone with a really major problem, but yet the iPhone is still flying off the shelves. Why? Because it has the apps that users want.

As soon as the user sees that a phone doesn't have any software, they almost always just end up returning it and buying something else instead. That's why testing these apps and actually trying to help get packages the extra love or thumbs up/down is so important.

Re: Extras-testing marathon
 

I plan to work today on a basic checklist both for developers and betatesters. That list will probably contain evident tests e.g. it installs / deinstalls and others less evident e.g. power management abuse.

I believe that once we have a more mature community testing workflow it won't be needed that all betatesters go through all items. Some might spend few mnutes to check that descrptiond and bugzilla component are in place, others might look at legal content and overall functionality in place. Other familiar with the right tools will check power management and system performance. Others might concentrate on optification, and so on.

There are two things I'm missing after using the system a lot:

- Clear differentiation between new uploads and updates. If an app is already in Extras and what comes is a minor update with a documented changelog and perhaps even a link to a diff then testing might be quite strightforward in many cases.

- Clear differentiation between items I have already rated and the rest. Already in the generic list. It would be also great to see the general history of evaluations of previous versions. And my own ratings of previous versions of an app. Testing a minor update of an app that has gone through 5 updates all with very positive feedback and always thumbs up from me is a different case than an app that got rejected a couple of times and actually got thumbs down from me in the past.

Yes, I know all this is plenty of work. I was wondering if with so many new people we wouldn't find more web developers around?

Re: Extras-testing marathon
 

Maybe looking into the Bugzilla addon Testopia (http://www.mozilla.org/projects/testopia/) would be a good idea. Programmers/project managers can create full test plans which basically work exactly like a checklist, split up into parts. Perhaps user A will run through the Save/Load checklist and user B will run through installation procedures. Once a test fails, a bug is automatically created with most of the important points and steps already filled in.

Even if it's deemed that Testopia is too hardcore for our users, it can be used as a checklist management tool anyways, hidden away.

Re: Extras-testing marathon
 

Sweet.

Uhm, I happen to know of a web developer. Me. ;-) Why?

Nathan.

Re: Extras-testing marathon
 

Showing that the app is already in Extras is pretty easy to add.

Changelog is on my TODO, but this is a bigger item. This requires parsing of the actual debs.

You can now see on which app you have voted already in the QA list.

History should be relatively easy to add. I'm not actually sure if this is a good thing. Aren't you more likely to vote thumb up and test less when you have voted on the app in the past?

As said before, a simple fix in a package can make it not behave or crash. Every version of a package should be tested, no matter what.

Re: Extras-testing marathon
 

I have some code to do that. I can dump the deb contents somewhere if you like?

I think so. But it would be nice to see how others voted.

The list of older versions is already kinda long and bound to get longer so I think limiting history at least might be worthwhile.

Re: Extras-testing marathon
 

I don't agree.

a) I am sure you are using some of these programs already (are you really not using gainroot/openssh/... etc?) So you can at least leave a comment/rating for those applications you are already using. I doubt that their number really is zero

b) If all those 300 people were developers adding their programs right now, we'd be seeing much more new software coming into extras-devel

Re: Extras-testing marathon
 

I have had the device for a month now, but I also got a daily work and a girlfriend to whom I want to spend my evenings with.
It's already HARD to find time to work my own application (and that's REALLY FUN!).
I decided to give the two hours I had yesterday evening to testing instead of playing with my code and I voted a few apps.

I will try to do it again, but it's already hard to dedicate a full (two hours!) evening to Xournal let alone testing (properly) also. Even though I use some of those apps already (and, in fact, I voted for them).

So.. please, it's not easy for us to find time to do testing.
But we'll do it anyway.
But understand us :)

Re: Extras-testing marathon
 

this video at the beginning (also at the end of part1, easily found) is about the app manager. the nokia employee shows how to enable the by default disabled extras repository.