Here's the promised complete post mortem of everything that happened.

What was planned:
Do a backup of our VMs and a software upgrade to all machines on a Saturday. I estimated about 2 hours for that, xes corrected me into 6.

Turns out it went into 10 days of frantic firefighting colliding with day jobs, family and giving talks at conferences.

What happend:

Saturday, 19.11.2016
10:00 - start updates and backups on blade-a
14:30 - backups and updates complete on blade-a, reboot confirmed successful
14:31 - uptime induced filesystem check after 1347 days
15:00 - start of backups on blade-b
17:12 - filesystem check complete, blade-a up and running
17:30 - first systems on blade-a confirmed up and working
18:30 - software upgrade on stage and mail complete
20:15 - backups of blade-b finished and copied onto blade-a backup space
20:16 - start of updates on blade-b
21:00 - updates on blade-b complete, reboot
21:01 - blade-b stuck in boot with corrupt bios image in flash
23:30 - all available remote recovery options tried, none working
23:40 - decision to go for Plan B, boot talk.maemo.org on blade-a, redirect everything else to talk.m.o
23:45 - blade-b turned off through IPMI
23:53 - talk.m.o available again

Monday, 21.11.2016
16:00 - Datacenter visit, trying to boot blade-b with attached USB key for BIOS recovery
18:00 - No recovery possible, Board hangs with "A9" after attaching USB devices, decision to swap board. Unable to swap board directly because Hardware has to be powered off and removed from the rack, there are APC powerstrips in the way.

Tuesday, 22.11.2016
22:30 - Starting backup of stage.m.o

Wednesday, 23.11.2016
21:30 - stage.m.o available again

Saturday, 26.11.2016
16:00 - Again, Datacenter visit, Security Guard doesn't show up until 16:45
16:45 - Swapping CPU and Memory to a spare board in the chassis
17:30 - Powering up spare board, removing udev.d rules, thereby accidentally swapping network interfaces of blade-b
19:30 - Upgrade of both blades to latest kernel and XEN version with security patches
21:40 - Correction of interface udev rules to match interface names to physical interfaces
22:20 - Every thing considered to be fully operational, applying finishing touches

Sunday, 27.11.2016
00:20 - maemo.org infrastructure declared operational

Monday, 28.11.2016
05:22 - blade-b kernel: NETDEV WATCHDOG: eth1 (igb): transmit queue 3 timed out
Network interfaces of blade-b started to reset due to bogon emissions
Affected systems: www, wiki, garage, builder, vcs
19:20 - reload of igb kernel modules fixed the condition, everything working again.

Tuesday, 29.11.2016
05:22 - blade-b has the same error as on monday
15:20 - fixed by reload of igb kernel module
15:40 - reboot of blade-b, disabled APSM

Total time spent staring at screens and in the colocation: 60+ hours

Time without serious hiccups: 1234 days (which was the uptime of both blade-a and blade-b)

So if you want to thank your admins, here are our wishlists:

xes - (prefers giftcards from amazon.it to xes.maemo (at) gmail.com) https://www.amazon.it/dp/B005VG4G3U
falk - https://www.amazon.de/gp/registry/wishlist/168D3W6163KG

Best,

xes & falk

Thanks.

Do you have any special order of wishes?
Or is there a possibility to pay only part of a high-prized item so few ants can make an elefant?

Furthermore I propose that council/board should think about spending some funds to those guys.

Thanks!
This is a great example of what you guys are doing behind-the-scene.
Hope your Xmas wishes comes true

Yes that would be a nice idea. I don't like amazon and have deleted my account there some time ago. I'm not willing to reopen it for you two guys. But of course i'm grateful for your efforts, and would make a separate donation for you, if I can just use the bank account of maemo e.V..

Thanx guys!

I'd be very happy about the bamboo tablet. Sadly there is no ants way, but this is only a wishlist

Best,

Falk

BUMP! bump.

Done

Done
Edit: didn't see that about the tablet, and I picked a safe (cheap) choice.
Will consider top-up, but no promises here.

Thanks a lot guys.

@reinob
Thanks, generous as ever!

What I do not get is the lack of supporters here in this thread. For some devs we easily crossed the 1k border (and never heard again). So I do not get why people reluct to say thanks with a little gift.
Those guys are working since years behind the scenes in the background but without their support we would have only 404.
Of course they are all doing it on a voluntary base but this example shows how much effort/time they put into it!


@all for those who cannot understand italian, you can use your own amazon site (just follow gift card / via e-mail) please follow below link (please do not use own amazon site as this will clutter the gifted money to different domains)
https://www.amazon.de/gp/switch-lang...&language=enGB
on this site you may choose appropriate language

@fstern: would make sense you create/post an e-mail address, too?
You can enter any value into the gift card ...


--
I would like to thank here also our long-time, unfortunately not anymore, repo maintainer merlin1991 and the CSSU/FPTF devs. ( as well as Maemo e.V. board )

@reinob
You are in the Community council and the question still stands: Any way to use the Maemo e.V. account for saying thank you?

It is a bit troublesome for tax reasons to do it directly. You can donate to Maemo Community and give directions how you would like to have your donation used.
I urge you however to make donations directly to @xes & @warfare.