maemo.org - Talk - wiki.maemo.org : current archive and future

maemo.org - Talk (https://talk.maemo.org/index.php)

- Community (https://talk.maemo.org/forumdisplay.php?f=16)

- - wiki.maemo.org : current archive and future (https://talk.maemo.org/showthread.php?t=87808)

www.rzr.online.fr

2012-11-10 12:44

wiki.maemo.org : current archive and future

Hi,

Tell me if I am wrong but I feel a bit scared of the future of maemo.org and meego.com ...

let me explain, http://wiki.meego.com went down and is still offline for a week , and yesterday http://harmattan-dev.nokia.com is b0rken ...

So I feel it is safe to coordinate some backups .... this thread is not for that , just check :

https://wiki.merproject.org/wiki/Mee...frastructure_:

Let's focus *ONLY* on the wiki contents :

Before we start hammering the webserver ... do you know tools to backup the wiki ? I planned to use wget or httrack . but something specific to mediawiki may exists already ...

Then If I have no suggestions or solutions I will try to crawl and dump contents in a couple of days ... ie : 2012-11-13 (but I can delay this)...

The question is now would it be possible to publish an archive of the backup, it seems not :

http://maemo.org/legal/terms_of_use/

Tell me if I am doing it wrong ...

Related thread :

*
http://talk.maemo.org/showthread.php?t=87815# How to be efficient on this survival plan ? #MaemO #MeeGo #HarmattaN !n950club #n900 #n9 #mer
* https://wiki.merproject.org/wiki/MeeGo
* http://talk.maemo.org/showthread.php?t=69681
* http://talk.maemo.org/showthread.php...76#post1292676
* http://talk.maemo.org/showthread.php...79#post1292679
* http://maemo-archive.wedrop.it/

Update :

wiki is backuped in XML form ask DocScrutinizer05 if needed ...

misiak

2012-11-10 12:52

Re: wiki.maemo.org : current archive and future

I wonder if the evopedia dumper (available at http://dumpathome.evopedia.info/contribute ) could be modified and used?

titilambert

2012-11-10 16:32

Re: wiki.maemo.org : current archive and future

GRrrr ... I just receive my n950 and I can't develop.... cause a I can't download the SDK ...

pali	2012-11-10 16:33

Re: wiki.maemo.org : current archive and future

License of Maemo wiki is CC-BY-SA. So to push backup is legal.

Dave999

2012-11-10 16:49

Re: wiki.maemo.org : current archive and future

Great move rzrfreefr. You seems to be the only one that acts on the countdown. Not sure what I can do to help you...

jackburton

2012-11-10 16:55

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by titilambert (Post 1292742)

GRrrr ... I just receive my n950 and I can't develop.... cause a I can't download the SDK ...

same here. I hope there is good reason or some transition to the community or Jolla going on.

misterc

2012-11-10 17:25

Re: wiki.maemo.org : current archive and future

Manual:Backing up a wiki
already been backing up most of the extras-* repositories with w3mir since a few months & making a refresh is a matter of a few minutes.
i'm out for the next 24 hrs or so, but will look into mediawiki after that...

MINKIN2

2012-11-10 18:37

Re: wiki.maemo.org : current archive and future

How much space would be needed to back these up?

shawnjefferson

2012-11-10 18:41

Re: wiki.maemo.org : current archive and future

This is one of the things that the Hildon Foundation BoD will be handling as far as hand-over and transition goes isn't it? They may be able to get backups right from the source.

Still, not a bad idea to grab whatever backups we can get.

Dave999

2012-11-10 19:05

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by shawnjefferson (Post 1292775)

I haven't seen a single action from the directors yet. Are they elected are they looking in to the issues? what's plan or do the working with that plan at the moment? The clock is ticking. It's a countdown in place...

suzughia

2012-11-10 21:09

Re: wiki.maemo.org : current archive and future

I've got a backup of harmattan-dev.nokia.com/ free, non-free and nokia-binaries on an ext hdd created with apt-mirror. Also I've downloaded the documentation for harmattan from nokia site and I've also the backup of harmattan-sdk that would be downloaded by the harmattan python sdk script that you will run on your linux machine.

It's a 32gb archive, anyone have any ideas for putting online a mirror, I'm avaible for populating it with what I have.

titilambert

2012-11-10 21:14

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by suzughia (Post 1292818)

Hello !
I have one server where I can put a mirror if you want !

www.rzr.online.fr

2012-11-10 21:16

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by suzughia (Post 1292818)

I've got a backup of harmattan-dev.nokia.com/ free, non-free and nokia-binaries on an ext hdd created with apt-mirror.

thank you for your preservation efforts ...

Do you know how we can legally release an archive of those snapshots ?

How did you setup apt-mirror ? I have a mirror of diablo apt too ...

Does anyone have a copy of meego wiki too ?

Do you think it is useful to list existing resources that need to be backuped at

https://wiki.merproject.org/wiki/MeeGo

suzughia

2012-11-10 21:24

Re: wiki.maemo.org : current archive and future

I've also have a backup of downloads.maemo.nokia.com too.

This is my apt-mirror config - picked up from this wiki if I remember right -

Code:

# su - apt-mirror -c apt-mirror



### System configuration (this can be changed) ###

set base_path             /var/spool/apt-mirror

set nthreads              20



### Repositories configuration ###

set defaultarch           armel

set auth_no_challenge     1

set no-check-certificate  1



### Official Nokia/Maemo repositories ###



# Nokia Applications

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/apps/ ./



# Nokia Tools

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/tools/ ./



# Nokia System Software Updates

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/001/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/003/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/005/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/006/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/007/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/009/ ./

deb https://qa9recEP:Pat2UGuP@downloads.maemo.nokia.com/harmattan/rm680/ ./



# SDK repository

deb http://harmattan-dev.nokia.com/ harmattan/sdk free non-free

deb-i386 http://harmattan-dev.nokia.com/ harmattan/sdk free non-free

deb-src http://harmattan-dev.nokia.com/ harmattan/sdk free



# Nokia-binaries

deb http://harmattan-dev.nokia.com/ harmattan/41667a5bd857be02f487c2ce806fbf85 nokia-binaries

deb-i386 http://harmattan-dev.nokia.com/ harmattan/41667a5bd857be02f487c2ce806fbf85 nokia-binaries



# Apps for MeeGo

deb http://repo.apps.formeego.org/harmattan/apps/ ./



# Apps for MeeGo (testing)

deb http://repo.apps.formeego.org/harmattan/apps-testing/ ./



### Other 3rd repositories ###



# MohammadAG's repo

deb http://mohammadag.xceleo.org/n9inceptedrepo/ harmattan-incepted free non-free

deb-src http://mohammadag.xceleo.org/n9inceptedrepo/ harmattan-incepted free non-free

tortoisedoc

2012-11-10 21:45

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by titilambert (Post 1292820)

Hello !
I have one server where I can put a mirror if you want !

A mirror of the harmattan + maemo would be awesome!

suzughia

2012-11-10 21:55

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by titilambert (Post 1292820)

Hello !
I have one server where I can put a mirror if you want !

Oh, this would be great! if you can provide me space and a way to upload - an ftp or whatever you want - I can put all the things up.

I've done a dirtree of the directory that I've got on the external hdd, you can find it here

http://pastebin.ca/Lv7v6oSJ
Password: N9_tree

This is all what I've got for the Nokia N9 - plus some other docs/apps sparse on the desktop and others app/docs also on the notebook.

sophia:~ suzughia$ du -h /Volumes/WD1TB/Nokia/N9/
41G /Volumes/WD1TB/Nokia/N9/
sophia:~ suzughia$

Cheers.

The firmware files are not needed on the backup I think ;) we can save 15GB.

pali	2012-11-10 21:58

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by www.rzr.online.fr (Post 1292821)

See wiki package about apt-mirror and *all* maemo and harmattan repositories: https://wiki.maemo.org/Apt-mirror

tortoisedoc

2012-11-10 22:01

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by www.rzr.online.fr (Post 1292821)

Is the merproject going down too?

MartinK

2012-11-10 22:09

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by tortoisedoc (Post 1292829)

Is the merproject going down too?

No, as it is not directly related to Nokia or Linux foundation. The only thing that could be problematic for Mer would be the shutdown of the COBS infrastructure - which looks like is not coming soon, thankfully.

tortoisedoc

2012-11-10 22:11

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by MartinK (Post 1292833)

No, as it is not directly related to Nokia or Linux foundation. The only thing that could endanger Mer would be the shutdown of the COBS infrastructure - which looks like is not coming soon, thankfully.

COBS infrastructure?

MartinK

2012-11-10 22:19

Re: wiki.maemo.org : current archive and future

The Community Open Build Service OBS instance.

joerg_rw

2012-11-11 01:28

Re: wiki.maemo.org : current archive and future

so good so far:
http://maemo-archive.wedrop.it/

Thanks Sebastian Lauwers aka Crashanddie aka teotwaki.

Much needed: a way to create a semi-decent backup of wiki.maemo.org. Mere wget will fail since there are too many pages/links like "edit", "history", etc. I seem to recall somebody talking about a script that uses http://wiki.maemo.org/Special:AllPages and http://wiki.maemo.org/Special:Export.

/j

[edit]
[2012-11-11 03:32:09] <Skry> I did wget --convert-links for pages in AllPages, then, cat ./* | grep -o '<a .*href=.*>' | sed -e 's/<a/\n<a/g' | sed -e 's/<a .*href=['"'"'"]//' -e 's/["'"'"'].*$//' -e '/^$/ d' | grep http\:\/\/wiki.maemo.org\/ | sed 's/^http\:\/\/wiki.maemo.org\///' | sed '/index.php/d' | sed -n 'G; s/\n/&&/; /^$[ -~]*\n$.*\n\1/d; s/\n//; h; P' | sort | xclip and pasted the result in export page

[2012-11-11 04:10:05] <Skry> merlin1991: https://dl.dropbox.com/u/49989728/maemo_wiki_main.txt
[2012-11-11 04:16:22] <Skry> https://dl.dropbox.com/u/49989728/maemo_wiki_file.txt
[2012-11-11 04:19:17] <Skry> https://dl.dropbox.com/u/49989728/maemo_wiki_user.txt
[2012-11-11 04:21:45] <Skry> https://dl.dropbox.com/u/49989728/maemo_wiki_talk.txt
[2012-11-11 04:25:08] <merlin1991> curl -X POST -d catname= -d curonly=1 -d wpDownload=1 --data-urlencode pages@$PAGESFILE -o $OUTFILE http://wiki.maemo.org/index.php?titl...&action=submit

merlin1991

2012-11-11 05:08

Re: wiki.maemo.org : current archive and future

If one wants to grab the media files aswell, this snipped worked for me (all the media files are in the xml that comes from maemo_wiki_file.txt):

Code:

egrep "<title>File:[^<]+</title>" $XMLFILEFROMFILETXT | cut -d ':' -f 2 | cut -d '<' -f 1 > $SOMEFILE



while read line; do curl -GL -o "$line" "http

://wiki.maemo.org/index.php?title=Special:FilePath&file=$line"; done < $SOMEFILE

panjgoori

2012-11-11 08:35

Re: wiki.maemo.org : current archive and future

im willing to help to create backup but i don't know how to create backup. if anyone can guide me then im more than happy to help you people.

www.rzr.online.fr

2012-11-11 12:03

Re: wiki.maemo.org : current archive and future

well there are several options yet ...

how big a such archive would be ?

may we release it for the end of the world day , around 20th december 2012 ? or let's say for jan 2013 ?

this snapshot archive could be mirrored then ....

pali	2012-11-11 12:05

Re: wiki.maemo.org : current archive and future

Need to backup also:
http://repository.maemo.org/stable/ (there is scratchbox instalaltion)
http://tablets-dev.nokia.com/MADDE.php (there is madde installation + n900 virtual qemu image)

www.rzr.online.fr

2012-11-11 12:51

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by pali (Post 1292945)

Need to backup also:
http://repository.maemo.org/stable/ (there is scratchbox instalaltion)
http://tablets-dev.nokia.com/MADDE.php (there is madde installation + n900 virtual qemu image)

Lets keep this thread for the wiki and use that one for other actions :

http://talk.maemo.org/showthread.php?t=87815

ivgalvez

2012-11-11 13:29

Re: wiki.maemo.org : current archive and future

Please don't panic.

We haven't received any notification about these systems going down, so it's more probably that it's simply problems of availability.

Maemo's wiki and Garage problems during this week happened while X-Fade was travelling, now he's looking into it. However, I'm not so sure about Meego's wiki.

If everyone starts doing their own backups, it's probably that we overload servers.

During last Council, it was decided to form a group of system administrators to take care of the infrastructure once it gets migrated, so it would be better if the backups where made by them in a coordinated and organised way.

www.rzr.online.fr

2012-11-11 13:33

Re: wiki.maemo.org : current archive and future

Quote:

Originally Posted by ivgalvez (Post 1292976)

Please don't panic.

If everyone starts doing their own backups, it's probably that we overload servers.

During last Council, it was decided to form a group of system administrators to take care of the infrastructure once it gets migrated, so it would be better if the backups where made by them in a coordinated and organised way.

That's the reason of my threads

http://talk.maemo.org/showthread.php?t=87815# How to be efficient on this survival plan ? #MaemO #MeeGo #HarmattaN !n950club #n900 #n9 #mer

www.rzr.online.fr

2012-11-11 13:51

Re: wiki.maemo.org : current archive and future

we're improving ;

http://talk.maemo.org/showthread.php?t=87815# How to be efficient on this survival plan ? #MaemO #MeeGo #HarmattaN !n950club #n900 #n9 #mer

razcampagne

2012-11-11 14:03

Re: wiki.maemo.org : current archive and future

If you want, I have space and bandwidth available for mirror / backups.

Even if it is early to panic, I am a bit pissed off by harmattan-dev being down the same day I decided to start developing for the n9.

joerg_rw

2012-11-11 14:08

Re: wiki.maemo.org : current archive and future

Please note that above I pointed you to a pretty comprehensive backup of *all* available repos, and we almost completed a backup of wiki. So please chill! :-D

/j

www.rzr.online.fr

2012-11-11 14:59

Re: wiki.maemo.org : current archive and future

note, it has been said that there is a wiki backup in XML form , then we can try to release an archive file, but is there a need for it ?

Ie let's make an app for that :) something where you can sync the wiki parts to your mobile to read later :)

anyway let me connect this url :

http://talk.maemo.org/showthread.php?p=1292955# backup of maemo resources - a community effort , keep zen no panic ...

tortoisedoc

2012-11-12 20:18

Re: wiki.maemo.org : current archive and future

DAMN THIS DOWNTIME >_<!

I can't believe how dependent I got on Nokia's (crappy but helpfull) Harmattan docs!

www.rzr.online.fr

2012-11-13 20:50

Re: wiki.maemo.org : current archive and future

those docs are back online same for meego wiki ,
please keep this thread open only for maemo wiki to be used as offline archive ...

For harmattan-dev check :

http://identi.ca/tag/harmattandev

All times are GMT. The time now is 17:46.

vBulletin® Version 3.8.8