Future of Maemo 5 Fremantle
 

EDIT:
These scripts for creating backup of maemo packages are now obsolated (but still can be used to verify openess of packages).

I created patch for apt-mirror which can download packages also from downloads.maemo.nokia.com, so apt-mirror is good way to backup apt rpositories. See wiki page: http://wiki.maemo.org/Apt-mirror

We know that Nokia abandoned Maemo5 and some day Nokia will stop Maemo5 repositories (SDK, apps, extras...). So I'm started creating shell scripts which will backup maemo 5 fremanle repositories (downloads.maemo.nokia.com and repository.maemo.org).

Scripts can also create package index (from repositories) and separate open and closed source. Some info about closed packages wrote Carsten Munk on his page - but it was for PR1.1. I asked him for application which generated his page, but he refuse publish it. So I'm created my own (scripts + wiki page).

I published all my scripts attached in this post/thread. It may contains bugs and download script is very very slow - but it works.

Size:
downloads.maemo.nokia.com mr0+apps 280MB
repository.maemo.org sdk+tools+nokia-binaries 5.8GB

Backed repositories are fully compatible with apt, so it may be used for n900 as mirror server.

How to use scripts:
download.sh - backup packages (source + binary) from repositories specified in sources.list and use authorisation informations from file auth (both files are in n900 in /etc/apt)

other scripts needs downloaded indexes files from repositories

installed-packages.sh need dpkg status file (copy from n900 after reflash /var/lib/dpkg/status)

Now I can say, that in default PR1.3 image is 355 closed packages and 364 open. But a lot of closed packages are now open (on meego.gitorious.org or maemo.gitorious.org or in CSSU) and a lot of are marketing/third-party (skype, flash, installers) and some maybe be replaced (ofono, icd2, ...)

I created wiki page where is table of all closed packages + info:
http://wiki.maemo.org/Fremantle_closed_packages

I wrote some informations about packages (if exists alternative, if is now open, ...) - but not for all packages. Maybe this will be good start point how to clean/rewrite Maemo 5. Anybody who is interesting in this is welcome.

Re: Future of Maemo 5 Fremantle
 

Also, it seems like someone from one of those countries where copyright is ignored to go ahead and make a large torrent image with *everything*(all closed packages, open packages, firmware images, etc.) just in case...

Re: Future of Maemo 5 Fremantle
 

maemo5 may well have the kind of support that diablo has, but I don't see them closing down the repositories any time soon.
But then again, can we trust them not to do just that?

Re: Future of Maemo 5 Fremantle
 

great work, thank you so much

Re: Future of Maemo 5 Fremantle
 

This is good news if i ever seen on here and i hope pali is not hindered by anyone, i did note from his post that Carsten once again refuses to cooperate and that is still sad news but never the less good luck with this because i can tell you many people want this to happen !.

Re: Future of Maemo 5 Fremantle
 

Out of interest how large are the repos currently (extras, testing + dev, plus the others, Nokia etc)?

Can't see them closing them down any time soon, but would be a good idea to back everything up and I'd be happy to help (with server space etc or anything else) :D

Re: Future of Maemo 5 Fremantle
 

pali: Please see my email in regards to downloads.maemo.nokia.com deb files, amongst that link that I provided, there are other bits and pieces of things that I have found and noted which maybe can be of use to you :)

(edit) I'm currently on IRC on the same #maemo channel if you need to contact me :) (/edit)

Re: Future of Maemo 5 Fremantle
 

extras, extras-testing and extras-devel are very very big. I will not backup it (sorry, but with my scripts can other users do it).

downloads.maemo.nokia.com has 280MB and SDK+tools+nokia-binaries has now on my disk 5.8GB (and still downloading)

Re: Future of Maemo 5 Fremantle
 

Well I don't mind backing up the lot (plus updating as new stuff is added - so if they do shut it down we have a complete backup), just need to know how big it is first, to ensure I have enough space. Can anyone give me an estimate, or is there anyway to work it out :)

Re: Future of Maemo 5 Fremantle
 

My old mirrored downloads.maemo.org amongst a few other repositories totalled around half a terabyte. This doesn't include downloads.maemo.nokia.com (which requires a slightly different method to download) and it includes repositories from other 3rd party developers. Also it doesn't contain any harmattan stuff.

If I were to run that mirror script now it would probably tell me there would be another 4+GB worth of downloads to grab minus downloads.maemo.nokia.com and harmattan-dev repositories.

Re: Future of Maemo 5 Fremantle
 

shell scripts published, see first post.

Re: Future of Maemo 5 Fremantle
 

Thanks, should be able to put aside 4-5 TB to back things up, which I guess will easily be enough.

@pali - Haven't got time to look at the scripts today, which repos do they backup?

Re: Future of Maemo 5 Fremantle
 

I'm not creating public backup (and I do not have servers for that). I created backup for my own usage (develop without internet)

But now everyone can use my script to backup anything.

Re: Future of Maemo 5 Fremantle
 

wow. thanks!

Re: Future of Maemo 5 Fremantle
 

ok I lied, I just ran du -h --max-depth=1 on that apt-mirror made backup and here is the output:
162G ./repository.maemo.org
244K ./maemo.o-hand.com
166M ./maemo-hackers.org
7.5G ./qole.org
162M ./www.cobb.uk.net
124M ./catalogue.tableteer.nokia.com
5.7M ./www.math.ucla.edu
7.9M ./www.mulliner.org
193M ./mg.pov.lt
2.1M ./b-man.xceleo.org
7.8M ./downloads.kernelconcepts.de
33M ./my.arava.co.il
3.5M ./al.robotfuzz.com
15M ./packages.geexbox.org
13M ./www.amsn-project.net
644K ./armin-warda.de
852K ./my-maemo.com
32M ./my.svgalib.org
11M ./marceloeduardo.com
170G .
170G total

Re: Future of Maemo 5 Fremantle
 

I think we have to find a way to merge with debian armel repository at least
for the free and not nokia modified stuff (apart from optification) this will ensure us:
1) updates and bug fixes
2) security updates
and offload the need to keep terabytes of backed up repositories.
A way to achieve optification on the fly must be developed in this case.
Non free stuff could be slowly phased out as free alternatives will be available
and tested. Obviously the help of more experienced and technically savvy
users is needed to have a minimal chance of success.

Re: Future of Maemo 5 Fremantle
 

All closed packages can be found on:
http://wiki.maemo.org/Fremantle_closed_packages

You can try to find alternatives or update info about packages.

Re: Future of Maemo 5 Fremantle
 

at a first glance I would suggest:

droid fonts as alternative for ui-fonts
fennec as alternative for the browser stuff

osso-sounds-game-chess and osso-sounds-game-mahjong
should be easily replaced by some of our music skilled forum members.

BTW: "cellular service" should be more correct than "celular service"

Re: Future of Maemo 5 Fremantle
 

feel free to edit wiki page

Re: Future of Maemo 5 Fremantle
 

Are there any plans for a community based/supported file server/repo, or are there any in place already?

Would something like this be useful to the community?

EDIT: Which could be synced with debian repos etc space isn't the issue really....depending on how much it is :D

Re: Future of Maemo 5 Fremantle
 

OK, I will do. I will add (?) to my suggestions as they are untested.

Re: Future of Maemo 5 Fremantle
 

Well thats not a lot 170 GB? is that it?

Re: Future of Maemo 5 Fremantle
 

BTW: debian arm/armel is only 49 GB.... not sure what other catagories we would need (if any) but the whole thing is only 610 GB :D

Re: Future of Maemo 5 Fremantle
 

I would suggest to purge some not critical packages from the list
as all the 3rd party installers, the games stuff as they are not
needed for basic operation. Is there a way to grey them out in the
wiki table?

Re: Future of Maemo 5 Fremantle
 

I do not know, I'm new to wiki. But please do not remove any package from list. You can mark them as 3rd or do not mark Needed for N900

Re: Future of Maemo 5 Fremantle
 

Ok, will let them untouched for now.

Re: Future of Maemo 5 Fremantle
 

Carsten Munk wrote me:
I refuse to publish because 1) they depend on a dataset from internal nokia that i can't distribute and 2) the scripts are so hacky and ugly noone would want to use them

Re: Future of Maemo 5 Fremantle
 

Mind you this is from a very old mirror that I did awhile back. It was the size for what it would have used from almost 3 months ago (the last update was sometime around April). Also the usual other rules apply such as the fact it does not contain any downloads.maemo.nokia.com repository, the harmattan-dev repository and any other repositories that I may have missed (which includes repositories like opera, some of the repositories which simply cannot be mirrored via the same tool) and repository like catalogue.tableteer.nokia.com is unusually small.

Still I estimate after one does a full mirror it may only require at most 250GB, and that should in theory include everything, including some of the sources that cannot be obtained easily (those of you whom mirrored some repositories that contained sources which were later on pulled down/out would know what I mean).

Mind you, community developments for N900 is still ongoing so fremantle mirror may end up becoming a fairly large space eater versus the rest of the maemo tablets :)

Re: Future of Maemo 5 Fremantle
 

You could also add free OVI applications:
https://downloads.maemo.nokia.com/fremantle1.2/ovi/

But to be able to download the packages, the patched apt binary is needed. By the way, I think it should be included in CSSU as it fixes a "bug".

Re: Future of Maemo 5 Fremantle
 

sure, report CSSU bug/send patch.

Re: Future of Maemo 5 Fremantle
 

Well I have been reading around and the active debian mirror is 610 GB, which basically means the all data we will ever need, harmattan-dev etc included, isn't going to be any more than that! :D even with the debian dependancy packages synced.

So one mirror for everything (plus some extra for more dev catalogs and other sources) isn't going to be more than 1 TB (1.5 for safety).

Is it worth setting something like this up?

Re: Future of Maemo 5 Fremantle
 

The bug was already addressed to in CSSU, no need to point it out that needs to be fixed.

Re: Future of Maemo 5 Fremantle
 

I personally think it would be worth even if it will cause some breakage in the test phase while trying to sort out what packages could be updated through
debian armel repos. Sadly I have not the bandwith to provide a mirror
but I volunteer for testing If the project starts.

Re: Future of Maemo 5 Fremantle
 

Well thats what I was thinking, maybe worth doing this now and get it working well before the repos are shut down (if they ever do get shut down).

Does anyone know what sort of bandwidth is needed for a decent mirror?

I have the spare hardware + time, but will need a hand setting it up.

Re: Future of Maemo 5 Fremantle
 

you probably want to have a top speed home connection with as many Mbps upload speed as possible.
considering that most Maemo packages are reasonably small (the former Firefox the largest one i came across so far is about 15MBs, but you obviously don't have to mirror it) you probably may want to throttle individual bandwidth to such a level that the average package download (when that bandwidth is available) doesn't exceed a few seconds.

also worth checking is your provider's "fair use policy" which may include monthly (or daily) caps. considering that a lot of consumers are using torrents, the number of connections may (or may not) be okay.

EDIT: the home connection may still be a problem if your IP address changes.
this usually doesn't happen if your server remains on-line and reboots quickly (before the IP address is allocated to another user) but ISP terms & conditions usually stipulate that the IP address is not guaranteed.
you may also want to spent some time configuring a firewall.

EDIT2: Pali, i hope you don't mind me mentioning this here, but...
that is one of the problems that was at least alluded to on the council thread; i.e. organizing & financing hosting. home hosting is a possibility, but it obviously comes with the cost of having to manage the server; like: RAID or @ least good backup & ready to replace a HD on a moments notice, et cetera, et cetera, et cetera...

Re: Future of Maemo 5 Fremantle
 

davedickson: Apologies for the late reply, yes I think 1TB should be plentiful to cover maemo/meego related repositories, including all the firmwares and what not such as:
- Developer environments (scratchbox)
- All the minor projects that has either been completed and/or is incomplete (and abandoned or ongoing) such as cordia, Mer and what not.

There's a few more other things that may need to be covered such as:
- maemo/meego wiki pages (unless X-Fade is willing to negotiate it'll be hard to mirror them without hogging excessive space).
- t.m.o. threads (talk.maemo.org), there's lots of useful information that has not yet been `wikified'.
- m.x.r. (maemo cross-reference site), very handy for any interested developers. Again, most of the site is dynamically generated and without negotiating to whomever owns it to mirror it properly will hog excessive space if one tries to wget it).
- bug site, handy to know what was fixed, etc.
- garage site: home of many 3rd party developers' apps. Again very handy and may include things that were not added into the main repositories due to various reasons.
- Qt stuff, needless to say that its used extensively on fremantle.

Hopefully that should cover everything that's related to maemo and its rather extensive history of things. All in all I really do not have a clue on how much will the final size be for everything aforemented here.

misterc: Dynamic IP address can be easily addressed with Dynamic DNS providers. They are mostly free/cheap and is generally fuss free.

Last but not least, two more things worth noting:
1) archive.org can be a really handy place for maemo.org's final resting place should the various alternative mirrors decides to move on. Better yet would be to try and somehow negotiate with them on making their lives a little easier by not mirroring maemo.org site constantly and instead of keeping all the chronological mirrors of maemo.org is to have one big mirrored version from whomever decides to pull the plug on their mirrored hosting.
2) It would also be somewhat a wise idea to have maybe a torrent and/or rsync server set up on alternative mirrors so that any other potential mirror can keep up to date with the alternative mirror (to keep it in sync in other words).

Re: Future of Maemo 5 Fremantle
 

For the Dynamic IP address using dyndns.org could be the solution.

BTW: I have a SS4000E home Nas lying around with debian installed,
with 4 250gb disks, but performance, number of concurrent connections
and my upload bandwidth will be a problematic. Also from a security
point of view I'm stuck with an old kernel version or I have to forward port
a bunch of kernel patches (and my spare time is very limited).
Nonetheless if it could be somehow useful i will set it up with
the help of some real admin.

Re: Future of Maemo 5 Fremantle
 

for a beginning, that sounds like a terabyte for the grab :)
certainly for setting up & configuring the repository properly.
and as long as the official repositories are still available (knock knock), bandwidth isn't an issue per se, is it?

i'm an openSUSE (& a little bit CentOS) centric user, so debs & debian are a little out of my league, i'm afraid ¦:-}

i'm going to contact an Unix User group and inquire what their conditions are for hosting a repository.
having an up & running server they can mirror is certainly going to be a requirement. but most likely a name and possibly a registered foundation of some sort (domain name?) will be too.

i'll be travelling coming week, so won't be able to do much, but i'll get back on this beginning of august.

Re: Future of Maemo 5 Fremantle
 

recess - 1st part, HW?
{if replying thanks to cut from this line down to the "end of quote tag" except for specific parts you'd like to refer to ¦:-}

just came across a post showing a Maemo tombstone and came to wonder... what future do we want for Maemo Fremantle?
or, to put it more down to earth, from ;)

personally i would like to keep using my N900s as long as possible.
so far, this community has done a great job keeping the N900 up to today's standard.
thanks for all the dedicated work! :o

but let's be realistic; even though there are still quite a few N8x0s and even 770s around, hardware is the limiting factor here.
don't misunderstand me, i still have my 7650 and even though the keys are spooky (don't react 1st or send several signals at once...), the 9 & 1 still do it properly and browsing the call history or contacts still works flawlessly. but obviously, (independently of a problem with the headset connector which loads static very quickly) i wouldn't be able to put even only one MP3 album on it.

same for the N900; with 64GB of storage, MP3s shouldn't be a problem, for the time being :rolleyes: but 256MB of RAM could :(
FireFox is known for being a memory hog (to put it mildly) thus we have to wonder how long it will be until Fennec outgrows 256MB.
then, obviously, we would need to replace our beloved N900 or compromise on newer (web?) features.

this brings me to my 1st question which could also be of relevance to getting mirroring sites host a Maemo Fremantle (for lack of an other name for now) repository:
would it be possible to port Maemo Fremantle (...) to other hardware? N950 might be an easy one, but obviously is not going to move masses :rolleyes:

Symbian and Wm or Wp devices are out as the firmware can not be flashed without OVI suite (Symbian) or at all (Wx?) i believe.

what about Android devices?
if it is possible to run NITDroid on the N900, it should also be possible to port Maemo Fremantle (...) to Android devices, no?

Re: Future of Maemo 5 Fremantle
 

recess - 2nd part, SW?
{if replying thanks to cut from this line down to the "end of quote tag" except for specific parts you'd like to refer to ¦:-}

of course, porting Maemo Fremantle (for lack of an other name for now) to non NOKIA hardware can be considered a software aspect too but with software i'm actually thinking broader.

namely, to put it bluntly, in how far does it make sense to keep maintaining Maemo Fremantle (...) if we can (eventually) get a comparable environment with MeeGo 1.x + Cordia?

is Maemo Fremantle's future limited to a potential time between the moment NOKIA may end its support and there is a valid alternative to Maemo Fremantle or the N900s become so obsolete that there is no alternative but to move on?

or can Maemo Fremantle outgrow the N900 and become the 1st truly open source smart phone operating system?

with regard to that, will it be possible to compete with MeeGo on this?
and we shouldn't only think NOKIA here but also consider other Android device manufacturers (LG, Samsung, ???) that may decide to use MeeGo as an alternative OS for smart phones