tracking issues and events with maemo.org infra migration#2
 

As the thread title says, in this thread I'll update all users about news and pending issues with our migration to the IPHH located community server.

Our iron got moved to IPHH and Falk and xes managed to install base OS with XEN and already migrated some preliminary VM images. Work continues tomorrow and on Monday.

I will report in loose schedule here about our progress. So far the impact of migration#2 on actual operation of *.maemo.org should be minimal to non-existent.

This is a continuation of thread at http://talk.maemo.org/showthread.php?t=88659

cheers
jOERG

Re: migration#2
 

This thread is NOT meant for general discussion! Users reporting technical issues in here is however much appreciated - and only that. Techstaff is supposed to have an abo on this thread.
parts of this thread been moving to http://talk.maemo.org/showthread.php?t=93451

Re: migration#2
 

I'm proud that my working place is just in the neighborhood of a new maemo.org server at iphh! just a few hundred meters away! :)

Re: migration#2
 

In Deutschland/Germany? Cool :)

Re: migration#2
 

for you geeks out there:
echo 213.128.137.20 maemo.org www.maemo.org >>/etc/hosts and have a first look (hope you know how to clean your etc/hosts after that, otherwise don't do this!)

Also http://213.128.137.6/mrtg/
You see we're still syncing stuff

cheers
jOERG

Re: migration#2
 

@joerg_rw,

Just a quick THANKS A LOT! Unfortunately I haven't followed the whole migration/drama, but I'm happy that it's all getting sorted out (largely thanks to you).

Although I'm 800km separated from IPHH, it feels nice to be close Maemo, so to speak :)

Re: migration#2
 

today we completed rsyncs and basic setup.
All services are up and "running" - of course we migrated all bugs in "old" infra to new one as well.
Whatever, let's call it our new infra's birthday :-D

Please do not use new infra for any production purposes yet. You're free to use above mentioned hosts hack to have a thorough look at the baby though ;-)

cheers
jOERG

icing on top: http://213.128.137.28/showthread.php?p=1325297

Re: migration#2
 

Today we checked some configuration issues that forbid decent access speed to new infra due to firewall internal fights about the master status. Everything back to "lightning fast" since Falk touched it with his magic hands.

Thedead1440 got an account on new scratchbox VM to migrate it to this new infra, together with Jussi. Many thanks!

General notion among tech staff is we need more storage. Alas the deal between HiFo and Nokia regarding this moves slowly, so we consider getting a single interim stop-gap 2TB drive to have some space for snapshots etc. We're still pondering who might pay for it.

DNS transfer to HiFo seems also a slow process like everything with a big company like Nokia is. We hope our interim solution hidden-primary though will happen RSN, so we could switch to new baby.

cheers
jOERG

Re: migration#2
 

Joerg,

I haven't donated anything yet because I refuse to use paypal. If there's a way I could transfer some money directly to you (or Falk, or IPHH) I would gladly pay for (part?) of that 2TB disk.

No idea what it costs (I guess it's not an average consumer disk), but something around EUR 100-200 would be fine.

Re: migration#2
 

Much appreciated. Look http://talk.maemo.org/showpost.php?p...&postcount=587, this goes to me and i'll forward it to Falk and put on top what's missing.

cheers
jOERG

[edit] oops, i'm a bit tired it seems. You said "no paypal", right. I'll post you a German bank account soon.

Re: migration#2
 

Erm Joerg,

he does not want to use "paypal".
There was already discussion about different method, but ...

I resigned and used paypal ;)

Thanks for that offer, reinob!!!

Re: migration#2
 

@Joerg,

Thanks. Send me a PM with the bank details. Sorry about the Paypal thing, but I used to have an account, which I never used (and deleted). Now whenever I think "heck, I'll just do it this one time" I always get an error "Sorry — your last action could not be completed".

Probably some blocked/outdated cookies, combined with the never ending problem of being in Germany but connecting (work) through a proxy in The Netherlands. Geolocation, my a##.

Add.: Thanks for the message Jörg. The money is on its way!

Re: migration#2
 

Falk ordered 2 pcs 2TB


> 2x Western Digital WD20NPVT Green 2TB
> 2,5'' / SATA II / I.Power / 8MB / bulk
> (interne 2,5'' SATA II Festplatte, Kapazität: 2 TB, Cache-Speicher: 8 MB, Intelli Power)
>
> sofort lieferbar EUR 149,00
>
>
>
> Zwischensumme (inkl. MwSt.): EUR 298,00
> Versandkosten: EUR 7,99
>
> Rechnungsbetrag (inkl. MwSt.): EUR 305,99

Reinob, many thanks for your outstanding support! It enabled the above, together with some donations left over from server shipping "issue" 2 weeks ago and me adding the missing rest on top.

[edit] while this is a great start, we still hope for Nokia eventually shipping their hw addon they promised to us, which still is needed to get autobuilder and repo under full load work nicely together, as well as for storing decent long term backups.

cheers
jOERG

Re: migration#2
 

last few days we shanghaied Jacekowski for autobuilder maintainer ;-). Welcome Jacekowsky!
He did a hell of a job analyzing what's state of things with autobuilder, and together with Falk and Xes (and me watching others working while I asked stupid questions ;-D) they were able to track down root culprit of hashsum problem which is obviously file corruption caused by stale NFS. NFS been borked by a bug in XEN, so we fixed XEN (http://lists.xen.org/archives/html/x.../msg00404.html), fixed NFS, and fixed hashsum, and possibly even builder - the latter still under investigation and for sure there are dragons sleeping there.

Anyway:Preferred method: patch your /etc/hosts. Please report back to us about your results if you do so. And keep in mind this isn't meant to be production grade stability and availability yet, so can see downtimes and inadverted reboots any time.


Woody will continue on his voting infra project and fixing karma. Which is becoming a pretty important functionality, since
in 3 weeks elections for new council term will start!
Please consider to run for maemo councilor!


The 2 pcs 2TB HDD should be shipping right now.

We're still waiting for DNS control transferred to hidden-primary, or whole domain getting handed to HiFo board. Until that happens, your only real option is above mentioned /etc/hosts hack


cheers
jOERG

Re: migration#2
 

Re: migration#2
 

What are the files in the link for? English with technique as topic is not easy for me to understand, so I didn't understand the whole text.

Re: migration#2
 

They are, I believe, basically an indication that the autobuilder in the new infra is working, as they they are newly built packages.

Re: migration#2
 

Just to make it clear, put:

on its own line.

It's not going to be perfect for now, but it works.

Re: migration#2
 

[2013-03-11 14:37:34] <warfare> I'll put in the disks tomorrow.

Thanks to all who donated, much appreciated, particularly since I don't have inbound traffic on my account right now, except from that.

Also we suffered a *very* strange load runaway with HDDs going r/o, which needed a blade reboot :-O Investigations ongoing.

Starting tomorrow (Tue) ~1900UTC we'll possibly put (some) services into read-only mode, to do the pre-final sync. Absolutely final sync will happen when DNS got switched to new server IPs (scheduled for Thu 1700UTC) and we can shut down services on old infra
Services on *old* that go r/o will stay like this until that point in time. And services on *new* will go r/o or even down for a short while (max a few hours) after DNS switch happened


cheers
jOERG
(admin manager)

Re: migration#2
 

Mar 12 16:32:20 blade-a kernel: ata3: exception Emask 0x10 SAct 0x0 SErr 0x4000000 action 0xe frozen
Mar 12 16:32:20 blade-a kernel: ata3: irq_stat 0x00000040, connection status changed
Mar 12 16:32:20 blade-a kernel: ata3: SError: { DevExch }
Mar 12 16:32:20 blade-a kernel: ata3: hard resetting link
Mar 12 16:32:25 blade-a kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 16:32:26 blade-a kernel: ata3.00: ATA-8: WDC WD20NPVT-00Z2TT0, 01.01A01, max UDMA/133
Mar 12 16:32:26 blade-a kernel: ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
Mar 12 16:32:26 blade-a kernel: ata3.00: configured for UDMA/133
Mar 12 16:32:26 blade-a kernel: ata3: EH complete
Mar 12 16:32:26 blade-a kernel: scsi 2:0:0:0: Direct-Access ATA WDC WD20NPVT-00Z 01.0 PQ: 0 ANSI: 5
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: Attached scsi generic sg2 type 0
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: [sdc] 4096-byte physical blocks
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: [sdc] Write Protect is off
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 12 16:32:26 blade-a kernel: sdc:
Mar 12 16:32:26 blade-a kernel: sd 2:0:0:0: [sdc] Attached SCSI disk
Mar 12 16:32:27 blade-a kernel: ata4: exception Emask 0x10 SAct 0x0 SErr 0x4050000 action 0xe frozen
Mar 12 16:32:27 blade-a kernel: ata4: irq_stat 0x00400040, connection status changed
Mar 12 16:32:27 blade-a kernel: ata4: SError: { PHYRdyChg CommWake DevExch }
Mar 12 16:32:27 blade-a kernel: ata4: hard resetting link
Mar 12 16:32:32 blade-a kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 16:32:33 blade-a kernel: ata4.00: ATA-8: WDC WD20NPVT-00Z2TT0, 01.01A01, max UDMA/133
Mar 12 16:32:33 blade-a kernel: ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
Mar 12 16:32:33 blade-a kernel: ata4.00: configured for UDMA/133
Mar 12 16:32:33 blade-a kernel: ata4: EH complete
Mar 12 16:32:33 blade-a kernel: scsi 3:0:0:0: Direct-Access ATA WDC WD20NPVT-00Z 01.0 PQ: 0 ANSI: 5
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: [sdd] 4096-byte physical blocks
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: [sdd] Write Protect is off
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 12 16:32:33 blade-a kernel: sdd: unknown partition table
Mar 12 16:32:33 blade-a kernel: sd 3:0:0:0: [sdd] Attached SCSI disk

that's how hotplugging two HDD 2TB looks like
:-)
/j

Re: tracking issues and events with maemo.org infra migration#2
 

Any news when the autobuilder will work with dput?

Re: tracking issues and events with maemo.org infra migration#2
 

Autobuilder is working now. You have to upload to NEW drop.

Re: tracking issues and events with maemo.org infra migration#2
 

most of you probably didn't even notice, but you're reading this served from our *new* tmo server already. Our brilliant sysops and admins did a nifty little hack and thus ported tmo to *new* without waiting for DNS switching. Alas tmo notification mail broke temporarily during this, not sure if it's back to operational for anybody/everybody yet.

Anyway, the scheduled time for
DNS switchover is today, Thursday 14th, 1700UTC
we don't really expect it to happen without any negative side effects, so prepare yourself for a few nuisances.
if anything goes terribly awry, please visit mwkn.net for latest update on workarounds

Thank you for riding with maemo.org
your maemo.org - Team

/j

Re: tracking issues and events with maemo.org infra migration#2
 

tmo mail should work again

Re: tracking issues and events with maemo.org infra migration#2
 

Yay DNS changes seem to be propagating!

Finally a fully Community-led infra. Congrats everybody!

Re: tracking issues and events with maemo.org infra migration#2
 

> Do 14. Mär 18:06:59 CET 2013
37c37
< Host mail.maemo.org not found: 3(NXDOMAIN)
---
> mail.maemo.org has address 213.128.137.23
137c137
< Host monitor.maemo.org not found: 3(NXDOMAIN)
---
> monitor.maemo.org has address 213.128.137.6
143c143
< Host blade-a.maemo.org not found: 3(NXDOMAIN)
---
> blade-a.maemo.org has address 213.128.137.4
149c149
< Host blade-b.maemo.org not found: 3(NXDOMAIN)
---
> blade-b.maemo.org has address 213.128.137.5
1c1
< Do 14. Mär 18:06:59 CET 2013
---
> Do 14. Mär 18:12:07 CET 2013
19c19
< repository.maemo.org has address 188.117.59.205
---
> repository.maemo.org has address 213.128.137.22
44c44
< lists.maemo.org mail is handled by 1 smtp01.wmfi.net.
---
> ;; connection timed out; no servers could be reached
82,83c82,83
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
90,91c90,91
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
98,99c98,99
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
107,108c107,108
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
116,117c116,117
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
124,125c124,125
< maemo.org has address 188.117.59.200
< maemo.org mail is handled by 1 lists.maemo.org.
---
> maemo.org has address 213.128.137.20
> maemo.org mail is handled by 10 mail.maemo.org\032.
155c155
< Host firewall-a.maemo.org not found: 3(NXDOMAIN)
---
> firewall-a.maemo.org has address 213.128.137.2
161c161
< Host firewall-b.maemo.org not found: 3(NXDOMAIN)
---
> firewall-b.maemo.org has address 213.128.137.3


and now clean

Re: tracking issues and events with maemo.org infra migration#2
 

And what's the new drop?

Re: tracking issues and events with maemo.org infra migration#2
 

the new drop is down right now, for syncing

Re: tracking issues and events with maemo.org infra migration#2
 

and what will be its address when it's back again?

Re: tracking issues and events with maemo.org infra migration#2
 

seems we're basically up and running on *new*

Re: tracking issues and events with maemo.org infra migration#2
 

drop.maemo.org ?

Re: tracking issues and events with maemo.org infra migration#2
 

Aaah, so nothing changes. I thought that it's another url :)

Re: tracking issues and events with maemo.org infra migration#2
 

Congratulations to the successful migration!

My phone just picked up new packages :)

Re: tracking issues and events with maemo.org infra migration#2
 

We are facing XEN VirtualInterFace bugs on repository VM
this causes repo going offline and we can't get it online before we reboot complete blade-a. Which in turn causes temporary downtimes on the mail,monitor, scratchbox VMs

sorry for the inconvenience, we're working on it

cheers
jOERG

Re: tracking issues and events with maemo.org infra migration#2
 

Hey Jörg (and of course all other),

I still cannot login to maemo.org.
I knew it was mixed up, and then forgot about it. Just today I decided to try again (let behind: voting for some packages). But to no avail.

I followed all that migration stuff and know it was discussed, but a solution?

Thanks in advance

Re: migration#2
 

is it ok to remove this line with leafpad by hand ?
already did but was wondering if there ismore to it ...

Re: migration#2
 

Should be. I haven't yet, but will do now.

Re: tracking issues and events with maemo.org infra migration#2
 

Hi Peterleinchen,
sorry, nobody looked into it yet. I will try to find somebody in techstaff to allocate to it.

Hi j0zeph,
yes, it is ok to remove the line manually.
another way would be
cheers
jOERG

Re: tracking issues and events with maemo.org infra migration#2
 

completly lost track here....

Some how HAM can't reach all repo's..?
Getting a "gzip returned an error code (1)"

i've changed the hosts file, and back again.

known problem at the moment?

It used to work great last couple of weeks...

(edit)
looks like all app's are still in the list, and HAM downloads and installs them...

Re: tracking issues and events with maemo.org infra migration#2
 

signed service contract by IPHH sent to HiFo as of today.