Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Were you answering me? Does this mean the answer is NO?

I assume so, but it was not really clear to me?

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I at least have a "rm" project link in https://garage.maemo.org/my/ . But of course, I only mean totally empty projects.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

The "rm" link wants me to pass admin on the project to someone else. Oh well.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

We can make that a "hot potato" project and just pass it around. :D

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Its true people do game the system. I"m currently working on an app to dim the screen when cp cycles used nears 100%. If it wasn't for karma. I'd just keep it for myself. Trivial apps do matter.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Quim's answer is more or less accurate. If your software is linked at http://maemo.org/downloads/, you get karma when it's rated & downloaded. tablet-encode is not in Extras, but gets me karma. Garage has nothing to do with download karma.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Sorry I thought it was clear, I said the files are posted on penguinbait.com and tablethacker.com. I asked IF there was a way to submit my download numbers to receive karma. So again "yes here is how" or "NO you cannot" is what I was looking for.

Not a reiteration of how it works, that was clear.

So having a repo or website of your own and providing all the software in the world won't allow you to gain any karma. :confused:

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

No, you cannot submit your own download numbers. a) the interface and format are too hideous and complicated to think about; b) although no-one's attempted to really game the system, that'd be far too trivial.

However, if an application is downloaded through http://maemo.org/downloads/, it's count is increased even if the ultimate destination is on another machine (IIRC).

Please don't ask questions and then make statements based on an assumed answer.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I had to assume what the answer is because initially neither you nor Qium could clearly articulate the answer, although I think you thought you did.

So did I misspeak?, was my assumed answer not correct?

As long as we are asking nicely, Please don't tell me what to do.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Hm, is that what we want to count for downloads karma? I expect most downloads to happen from h-a-m in which case the repository access logs should be used instead. Maybe someone with access to both sets of stats could run a comparison?

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

If a developer is not using maemo.org Extras neither maemo.org Downloads then it makes sense that he doesn get maemo.org karma, isn't it?

Re: Brainstorm: Developers should get karma based on the relevance of their softwarea
 

The downloads from the repository are also counted in maemo.org downloads. They are synchronized every X time. Where X is a value that X-Fade knows ;-)

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Download counts are updated by clicks on the install icon(live) and are corrected once a week by the actual download stats we receive from the caching network. So in the end we only count actual downloads from the repository.

We're looking into going from weekly to daily corrections, but that requires some changes and bureaucracy, so that might take a while still.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Another thing we could use is the app karma feature (In Downloads). App karma is basically going to show which apps are relevant. (Relative downloads stats, comments and ratings are being used there)

This app karma can be used as a modifier for the actual karma the developer gets for that app. App karma is a float value between 0 and 1.
The most relevant apps with have an app karma value of very close to 1.

Calculation could be then be:

Example:

This would also mean that once your app gets less relevant, your karma for it will go down. It will be more dynamic and your karma will fluctuate a bit.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

A random thought on download karma... Merging versions with downloads is a bit tricky, right ? I mean, when you push something to extras, you will certainly see a surge in downloads from people who are updating in addition to those who are installing for the first time. This means projects updating more often will certainly see a higher number of downloads even with the same userbase...

... and now, the kicker...

What if we use these spikes as an additional metrics in some form ? After all, the volume of the spike is the approximate numbers who still have it installed. So it is possible to measure uninstalls, even if in an indirect form. Yes, I know, some people never update, but the point is that the metric will be equally wrong on all projects, so you do get a realistic comparison of userbases. Note that this wont work if you update software TOO often as the update dispersal can take a while.

See this graph from maemo mapper - you see a steady stream of downloads, but the spikes containing the updates are those which show the real userbase increase.

EDIT: I'm not saying this should replace download count, but could be something like the t.m.o. posts vs thanks karma. Download karma (=posts) is cheap, update karma (=thanks) is value.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Not really, there are several people who work on projects that thousands of people use and attract people to the platform that never get put in the garage. Do these people not deserve some sort of Karma for their efforts? Perhaps its impossible to quantify perhaps its not. You can see here http://logs.penguinbait.com/index.html. I have dedicated my home website to mainly Nokia Tablet related traffic. Additionally my numbers from tablethacker.com which is also dedicated to Nokia Internet Tablets are even greater. So I am just saying should there be a way for people who have continuously supported the platform, drawn many people to the platform, published many packages but never in the garage to receive some sort of karma for their efforts.

I understand the fact that you want things to be put in the Garage and in maemo extras. I feel as though there should be some sort of QA approval process if something is going to be placed into maemo/extra's so users can expect some level of consistency. While I think that "hackers" or "hacks" providing beta software at best should never be placed into maemo/extra's I still think they play an important role in the community.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

It does make sense that maemo.or karma is measured against factors maemo.org can register. But if you have additional ideas please submit them in the Brainstorm proposal we are discussing.

Note that I'm not mentioning Garage at all. Host your code wherever you prefer.

We are talking about distribution channels. You have extras-devel, extras-testing and Extras for your software in unstable, testing or stable quality. If you need something else then file an enhancement request or an idea in the Brainstorm proposal that started this thread.

And we are talking about maemo.org/downloads that, true, if you are distributing "hackish" software you better don't use.

You can use bugs.maemo.org and that activity will bring you karma.

Yo can talk here about your software, bed thanks and get karma for it.

You can blog about your software and get thumbs up getting karma as well.

The system is probably not so bad when you have 377 karma http://maemo.org/profile/view/penguinbait/

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

atila77, your reasoning about downloads is very good! Maybe we need to use those peaks for karma instead of full downloads, indeed. Could you please submit the idea? It seems that we can keep the ratio 100 downloads = 1 karma for these peaks? If v1.0 gets a peak 1000 downloads you get 100 karma points. If v1.1 gets a peak of 1500 you get 150 instead of 100. But if v1.1 gets a peak of only 200 then you get 20 instead of 100. Makes sense.

XFade's point about app karma is also good. We should use it somehow to add karma, but perhaps not to diminish karma. We have talked many times about users diminishing karma by 'being older' if they don't add new karma, to make old disappeared contributors go down the ranking while new contributors very active go up up. It lools that this way would be enough to get minus karma. Just like we do with news, they push your karma up but won't push it down.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

If karma app is a value between 0 and 1 then waht about this:

karma points = max app karma * 100

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I think nobody has really explained how karma is currently calculated for appliactions (Products in profile page).

You get couple of points (<10) for having an application in maemo.org/downloads, then you get additional points for the rating. Maximum karma for one application is 45 or so. I couldn't find the formula so this is from memory. There is no karma for downloads.

Garage karma is called Groups in profile page. If you have single project where you are the maintainer you get 9 points ( I think it is 3 for having an account, 3 for being member of a project, 3 for being maintainer of a project)

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

OK, let's just clarify a few things beforehand, as there is no edit.

Not sure if it's just terminology, but I strongly suggest peak volume instead of peak value. It would be a shame to loose karma because your peak was at night (split across two days), influenced by day-of-week, holidays, etc.

BTW, you say 100 : 1 but the ratio is 10 : 1 karma in the examples :)

Can someone clarify the X axis relation in the graphs (Niels) ? I can see the data points are weekly, but I don't know if it means the value is SUM of dowloads that week, just a snapshot for a single day or daily AVG for that week which might influence karma multiplier choice... Or, if it is possible, could I get daily resolution data for a project just to see clearer ?

About updates being too close - this is actually not that big of a problem as I initially thought. It is on the current graphs as you can't differentiate between versions, but since these ARE separate files, they might just as well be statted separately for karma purposes.

Here's another graph, this time for OMWeather - it has updates more often than mapper so you can see the principle even better:

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Ok some observations:

1. There is no way to edit your submitted solution in Brainstorm (Bug submitted). There is no 'Brainstorm' component in bugzilla either.

2. The discussion of this brainstorm topic at the Brainstorm page itself has only about 13 replies, here at Talk has reached 7 pages already. I propose using Talk to discuss brainstorm ideas, or at least provide a way to link a brainstorm idea to a Talk thread.

3. I propose prompting for a reason why you are voting 'no' on a proposed solution. I think it is good to gather the 'cons' of a proposed solution to give way to a new proposed solution, or to clarify misunderstandings. Also, it might be good to provide a way to change your vote for a proposed solution from yes to no or no to yes.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Just to comment on peaks. This can be abused as well. A version 1.0 can get a high peak, and v1.0001 can also get a high peak for fixing a bug. v1.0002 can get a high peak again for fixing another bug. A developer can get more karma by spreading bug fixes across different versions.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Err, the way I understood it peak karma is not cumulative, you always get the value for the last peak.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Why are we even discussing changing this metric now, so close to a new device release?

(...and before anyone jumps down my throat I am just playing the devils advocate here.)

To a casual observer it appears that someone doesn't like or think that a member or members who have high karma now, deserve any of the future perks that may come from that metric.

If this is the case, what's going to happen the next time if high karma earners from this change fall out of favor?


What I'm saying is that if a future perk, developers device, or all expense paid trip to Las Vegas (release rumors early and update often :D ) or something is going to be determined by karma, changing the equation of how karma is calculated now might seem a bit tacky and disingenuous.

Some may discount my post based on my relatively low karma but I am an observer of the process. :)

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I'm proposing to discuss this now in order to have the machinery totally fine with a post-N900 device comes. Yes, we are that slow... :)

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Is having some kind of routine to keep track of installed software built into application manager that can report back to maemo.org what is currently installed when it performs an update. Perhaps this is going to far, to much like big brother?

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

http://popcon.debian.org/

This has been done and is available for quite some time.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Here are the formulas in use: http://wiki.maemo.org/Karma

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I've proposed that there be more links to get to that and similar pages. See bug 4959 (vote and comment please!)

https://bugs.maemo.org/show_bug.cgi?id=4959

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

In that case, that might be more dangerous then. There is a chance that the developer might stop improving an app if the peak is already high, despite a lot of requests to add more features. The result: stale apps.

Another thing that might happen is the developer(s) might release that same new app with new features added and appending a 'pro' or 'ultimate' to the app name. The developers get to preserve the high peak on the old app, and get new karma on the 'pro' and 'ultimate' versions.

I believe in rewarding the developers who continuously improve their apps. If they get a million karma because they keep improving their apps, then good. They deserve it.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

I'm not sure I follow... Someone is afraid to post a new version of his software because he fears more users might have uninstalled than installed it in the meantime ? That almost sounds like a psychological problem...

Now, if someone wants to abuse the karma system, he will do so, you can't stop karma wh*ring, no matter what system you come up with, especially if the karma system is public and the users are anonymous. The whole thread started off because now we can't tell important projects from unimportant ones. Not to mention there are a LOT simpler ways to fake karma than such constructs.

But this is the problem, what's the non-karma-abusable measure of app improvement, if it exists at all ?

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

How about project karma has an expiration. After a set time development karma would expire maybe a year. this would stop people from sitting on a project and not updating it because their karma would go away. Equally karma from blogging and helpful posting could also expire. If a user used to supply a great deal of information but after a new device or os upgrade comes out becomes very quiet the high karma currently does not reflect the users activity. Questions directed toward a quiet user based on current karma may not get answered with the same speed or quality of response.

I don't think that karma should be subtracted just expired hopefully this would avert negative karma.

If there is negative karma this to should expire. Maybe after a longer time than positive karma expires. people sometimes do bad things but should have the ability to re-hab and not have bad karma follow forever.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

If a developer's primary motivation is to accumulate karma points then their apps will get stale no matter what the formula used. I don't think there's much risk of that though, as most reasonable developers would value their time much higher than a potential device discount or weekend attending talks in Amsterdam :-)

I would like to see some more quality (as opposed to popularity) metrics used however. User ratings certainly, and also things like ratio of open/fixed bugs, code checkins and so on. Yes, all of this can be "gamed" to an extent, but let's please assume that someone who bothers to climb the steep learning curve, negotiate the autobuilder etc to publish an app does so because of a genuine desire to scratch an itch and contribute. There are far easier ways to get karma points after all.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Here's the thing. I'm suggesting to bring in the end-user to reward the developer(s) so apps keep improving. If you provide a solution that somehow stops the development of the app, even how small, then that's a potential problem.

I like how the site versiontracker works. Users provide feedback for each version, and each version gets to be rated (e.g. this). You can immediately see that the system works. The only problem is how to incorporate karma to it.

Is there a successful system that rely on peaks?

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

There was a scientific paper of this phenomenon in the latest Open Source Systems conference (OSS 2009). Here is the reference:

Wiggins, A., Howison, J. & Crowston, K. 2009. Heartbeat: Measuring Active user Base and Potentieal User Interest in FLOSS Projects. In Boldyreff, C., Crowston, K., Lundell, B. & Wasserman, A.I. (eds.): Open Source Ecosystems: Diverse Communities Interacting, Proceedings of 5th IFIP WG 2.13 International Conference on Open SOurce Systems, OSS 2009, Skövde, Sweden, June 2009.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

Agreed, and maybe the best way to keep karma current is to constantly calculate it within a moving window, say 1 year. If people cease to be active, their karma will naturally decline... and this also ensures that karma reflects currency.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

The ratio approach makes the most sense to me and should be used where ever possible and practical. So maybe a download metric incorporates percentage of views converted to downloads? Or maybe the first released version of an app is solely a benchmark, and karma is applied to ratio (and/or number) of updates after initial download? Surely loyalty to a developer or app is an excellent indicator of quality... and addresses the uninstall question to an extent.

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

May I just throw in a number of criteria that may be useful or not. I'm not very good at doing the maths in terms of karma=(a*b/17)^2+(e)*f, and I'm also not sure if i got the idea of what "developer karma" should be... anyway:

Port vs. native development
I once ported a command line application and used it on my device without publishing it. When packaging and extras became an issue (developers told us here that the whole process is broken and oh so difficult), I tried if I could bring it to extras without prior knowledge of debian packaging. I could. An I went the whole way and included it in /downloads/OS2008/. When I asked on the mailing list who I should name as the author of this package, I basically got "whoever you want to get karma for it". Now I wanted to be named as a contact person, I wanted receive karma,... but still: Is it OK to apply the very same developer-karma calculations for someone who'd only re-package an existing tool without even touching the source code? Maybe using two fields in the downloads page ("author" vs. "contact") could be a starting point to change this.

Dependencies
What about packages that are listed as dependencies of other packages? Say I port either a library (or an interpreter for a new language...) to Maemo, and 359 applications start using this library (or interpreter). Most probably nobody would ever download it from /downloads/OS2008/ or comment on it, but without it, 359 high-rated apps wouldn't have been possible. So: Count dependencies in packages.

Rating vs. downloads vs. comments
Relevance is an underdefined term. Whats relevant? Is a game relevant because it has high download rates? Is mplayer relevant because of its download rates... or is it relevant because it pushes the boundaries of the platform? I don't know. - I do believe, though, that counting only the average rating or the number of downloads or the number of comments given for this application will probably give a wrong result.
Downloads can be wrong for a number of reasons already discussed in this thread
Rating is a most irrelevant figure as long as it's the average rating. I give my own application 5 stars and have a higher multiplier than a really popular app that users find minor bugs in and that therefore ends up with 4 stars.
Number of comments could be nice to throw in. How many people think your application is worth writing about? (These could be, of course, all bad reviews like "What's that? It doesn't even install!", so it needs to be just one factor of many)

The penguinbait-effect
There must be a way to gain karma by doing things that don't quite fit the "install a deb from extras" scheme. Many interesting concepts that show how versatile the whole ecosystem is will not, cannot or should not be deployed to end users this way. Still, I'd personally consider them at least as (if not more) relevant than a game that gets downloaded 2000 times a week.
We used to have thumbs up/down on the maemo.org profile pages. This never worked for karma, IIRC, and is now disabled. How about adding a way for developers to add specific projects to their profiles (name of project, link to project homepage, short description, no screenshot) and allow others to give karma via thumbs up?

Fire exit
Karma calculations are one thing and will always be "wrong" for some people, no matter how good the system actually is.
The developer device program is another issue. It can be, but needn't be exclusively tied to karma. You could, say, make it public that a list of x developers will receive a discount based on their karma. Afterwards, those who're not part of this list should be given a chance to be elected by popular outrage ;) ... you know, any developer should be able to stand up and say "hey! community! i think i do deserve it for my projects, but the karma system doesn't work for me. you know me. you love my projects. vote for me at...". we're missing the "at ..." for such a community-ticket.

non-developers
During the N810 device program, I read:
Is this still valid or will we fall back to developers only again? Just confused because this thread is about developers only. (And no, I'm not asking for myself... I live in a country without a Nokia online store, so I wouldn't be able to use any discount code, anyway.)

Re: Brainstorm: Developers should get karma based on the relevance of their software
 

You realize that if you go through with this chain of thought it means development should NOT relate to karma ? If you have karma, you have competitiveness, and if we really want to hair-split that in itself can result in scenarios that stop development (isn't this the crux of the penguinbait conundrum is this very thread ?). As Texrat pointed out, there is no 100% scoring system, the best we can do is come up with something that works in as close to 100% as possible while covering as many scenarios as we can.

No idea, it was just a random thought (which apparently popped up in other places, too). This changes the question into is there a system that relies on peaks and failed because of it ?