maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Community (https://talk.maemo.org/forumdisplay.php?f=16)
-   -   Forum outages (https://talk.maemo.org/showthread.php?t=64319)

Reggie 2010-10-25 14:33
Forum outages
 
Due to the PR1.3 announcement today, expect TMO outages due to high server load.

Sorry for the inconvenience.

If you want, you can follow me on Twitter for updates: @rsuplido

plaban 2010-10-25 14:35
Re: Server outages
 
I thought that there is a problem with my internet connection.Thanks for the confirmation.

noobmonkey 2010-10-25 15:05
Re: Server outages
 
Have to admit reggie, the server has done us proud! Last time it died in the build-up to the update release! so for it only to go today, thats pretty good! :)

NvyUs 2010-10-25 15:57
Re: Server outages
 
more servers needed its not good that a firmware announcement should kill the site for a day and not only that it kills meego forums too.
rather separate them to different servers or get more b/c imo its unacceptable that it impacts other sites

danramos 2010-10-25 16:16
Re: Server outages
 
Quote:
Originally Posted by plaban (Post 850410)
I thought that there is a problem with my internet connection.Thanks for the confirmation.
Next time you wonder whether a site is down for everyone or just you, try http://downforeveryoneorjustme.com/

Dave999 2010-10-25 16:22
Re: Server outages
 
we need more servers or less updates...

bonapart 2010-10-25 16:30
Re: Server outages
 
or something like bittorent downloading updates

danramos 2010-10-25 16:34
Re: Server outages
 
Quote:
Originally Posted by bonapart (Post 850512)
or something like bittorent downloading updates
I can't believe that torrent isn't already the norm, for crying out loud. I can still remember when everybody was bawling and bi*ching to get N800/N810 images and ended up torrenting the files for the same reasons. It's FAST, RELIABLE, OPEN-SOURCE and best of all it maintains INTEGRITY (hash codes!) and can even be interrupted safely. Ugh. Whatever.

sjgadsby 2010-10-25 16:41
Re: Server outages
 
Clearly it was a mistake to integrate the high-speed networking hardware directly into the forum server. If the forums had continued to simply tether via Bluetooth to a separate, dedicated, high-speed networking server that could be upgraded independently on the user's choice of ISP, we wouldn't have this mess.

Or maybe it's all the orange. I don't remember anymore.

wmarone 2010-10-25 16:41
Re: Server outages
 
Quote:
Originally Posted by danramos (Post 850524)
I can't believe that torrent isn't already the norm, for crying out loud. I can still remember when everybody was bawling and bi*ching to get N800/N810 images and ended up torrenting the files for the same reasons. It's FAST, RELIABLE, OPEN-SOURCE and best of all it maintains INTEGRITY (hash codes!) and can even be interrupted safely. Ugh. Whatever.
Well, feel free to code up a bittorrent based version of APT ;)

danramos 2010-10-25 17:18
Re: Server outages
 
Quote:
Originally Posted by wmarone (Post 850535)
Well, feel free to code up a bittorrent based version of APT ;)
Well, GOOD NEWS! It's already been done. :rolleyes:
http://sianka.free.fr/

slender 2010-10-25 18:18
Re: Server outages
 
Why on earth you people do not have proper way to communcate on server issues.

I have tried to follow updates from official source about TMO sitsuation:
http://maemo.org/news/
http://maemo.org/
http://maemo.org/news/announcements/

And total silence. Why?

Itīs nice that reggie is updating his twitter account (found it after i was able to read this thread :|) but oh boy :| Where is official maemo.org twitter account or facebook account etc. Shouldnīt council at least be interested in these kinds of matter?

NvyUs 2010-10-25 18:25
Re: Server outages
 
get ready for another outage, every time its approached 2400 viewers today its gone down

danramos 2010-10-25 18:54
Re: Server outages
 
Quote:
Originally Posted by slender (Post 850579)
Why on earth you people do not have proper way to communcate on server issues.

I have tried to follow updates from official source about TMO sitsuation:
http://maemo.org/news/
http://maemo.org/
http://maemo.org/news/announcements/

And total silence. Why?

Itīs nice that reggie is updating his twitter account (found it after i was able to read this thread :|) but oh boy :| Where is official maemo.org twitter account or facebook account etc. Shouldnīt council at least be interested in these kinds of matter?
What do you mean by "you people?" :)

All kidding aside, I entirely agree. I still stand by my earlier suggestion to just do the smartest thing and use a bittorent method of upgrades and/or apt. It would absolutely cut down on all the unnecessary waste of bandwidth and server resources for Maemo/Nokia/whatever.

storkus 2010-10-25 19:41
Re: Server outages
 
Funny (or not) thing is that TMO goes down for me all the time. Even before today I'll be reading and it will go down for hours. Now I don't know where the servers are and I'm in Phoenix usually accessing it from work during swing shift, so maybe Reggie is just asleep when it happens and can't fix it until morning, but it definitely happens way more than just today (when it is entirely justified--/. effect).

danramos 2010-10-25 19:46
Re: Server outages
 
Quote:
Originally Posted by storkus (Post 850639)
Funny (or not) thing is that TMO goes down for me all the time. Even before today I'll be reading and it will go down for hours. Now I don't know where the servers are and I'm in Phoenix usually accessing it from work during swing shift, so maybe Reggie is just asleep when it happens and can't fix it until morning, but it definitely happens way more than just today (when it is entirely justified--/. effect).
Actually, I've noticed that too. Usually seems to happen around 5-6am Eastern Daylight Savings Time.

Reggie 2010-10-25 20:28
Re: Server outages
 
Daily backup happens at 1AM (EDT) for about an hour or so, so during that time, you might not be able to access the server.

slender 2010-10-25 20:30
Re: Server outages
 
Quote:
Originally Posted by danramos (Post 850613)
What do you mean by "you people?" :)

All kidding aside, I entirely agree. I still stand by my earlier suggestion to just do the smartest thing and use a bittorent method of upgrades and/or apt. It would absolutely cut down on all the unnecessary waste of bandwidth and server resources for Maemo/Nokia/whatever.
By you people I mean all gnomes and goblins behind maemo.org servers. I do not know what kind of reporting policy or workflow these people have for reporting problems but I have to say that the time what I have been here it sometimes feel totally fragmented. Sometimes someone will update something to some page somewhere (here in maemo.org) or actually in internet.

Maybe I should start lurking maemo irc channel but IMO thatīs just not so obvious way of reaching all people in todays web 2.0.

And about technology behind updates and server. I have seen some forums that change to text only mode when load getīs too high. That probably is not rocket science either.

danramos 2010-10-25 20:34
Re: Server outages
 
Quote:
Originally Posted by Reggie (Post 850693)
Daily backup happens at 1AM (EDT) for about an hour or so, so during that time, you might not be able to access the server.
Old-school backup system where you have to take down the service for a while? I thought everyone did backups on the fly these days (i.e. RAID, Tivoli, etc.). More curious than anything, old-school backup system is a pain but I'm sure it's cheaper, too.

nidO 2010-10-25 21:22
Re: Server outages
 
Quote:
Originally Posted by danramos (Post 850613)
All kidding aside, I entirely agree. I still stand by my earlier suggestion to just do the smartest thing and use a bittorent method of upgrades and/or apt. It would absolutely cut down on all the unnecessary waste of bandwidth and server resources for Maemo/Nokia/whatever.
I'm not fully sure what you imagine this alternative implementation of delivering updates would achieve.
You should be aware that the update repos are delivered via Akamai and have essentially unlimited resources in terms of Nokia's needs to push maemo updates, TMO however is hosted by a single server, entirely separate from Akamai. The reason TMO goes down is not because "zomg so many people are downloading teh updatez" as they come from a totally different source, it goes down simply because on update days there are THOUSANDS of concurrent visitors looking at the forum.
Changing the method of delivering updates doesnt in any way whatsoever affect the ability of the forum to cope with the number of people looking at the forum on the day a patch happens.

Reggie 2010-10-25 21:58
Re: Server outages
 
Quote:
Originally Posted by danramos (Post 850703)
Old-school backup system where you have to take down the service for a while? I thought everyone did backups on the fly these days (i.e. RAID, Tivoli, etc.). More curious than anything, old-school backup system is a pain but I'm sure it's cheaper, too.
Yes, the backup is online and set to incremental. For some reason though, MySQL locks some of the major tables during the backup making the the site seem unaccessible. The major tables are quite large as well.

I'm checking some more items that might be causing the problem now. It looks like the server is responding well now, despite the not so low server load.

Just to clarify, the forum server is separate from the other parts of maemo.org. As nidO has mentioned, it's the spike on the number of visitors in the forum that causes the forum server outage.

YoDude 2010-10-25 22:22
Re: Server outages
 
Quote:
Originally Posted by NvyUs (Post 850467)
more servers needed its not good that a firmware announcement should kill the site for a day and not only that it kills meego forums too.
rather separate them to different servers or get more b/c imo its unacceptable that it impacts other sites
Simple fix:
Increase dues...

storkus 2010-10-25 22:27
Re: Server outages
 
Hehe, nidO, I thought that was obvious, but I guess not.

danramos 2010-10-26 07:59
Re: Server outages
 
Quote:
Originally Posted by Reggie (Post 850778)
Yes, the backup is online and set to incremental. For some reason though, MySQL locks some of the major tables during the backup making the the site seem unaccessible. The major tables are quite large as well.

I'm checking some more items that might be causing the problem now. It looks like the server is responding well now, despite the not so low server load.

Just to clarify, the forum server is separate from the other parts of maemo.org. As nidO has mentioned, it's the spike on the number of visitors in the forum that causes the forum server outage.
Ah okay! The context and the way it was being explained made it sound like the same servers--if it's purely refreshes and genuine forum traffic, then there's only the MySQL problem (although I'm still not sure I understand why backups have to occur on a set schedule as opposed to a live syncing backup or something more efficient with no downtime.. but then I don't know the rest of the story either) and bandwidth (which I suppose could be throttled at the server or, better, the load balancer/router/whatever you're using.

*Sonic* 2010-10-26 18:17
Re: Server outages
 
Its been off most of the evening for me today, and all last night and meego is still down

Is this a private server or provided / funded by Nokia ?

None of the other Maemo / N900 sites have been down, so why this one

Is it a case of it just needs more money thrown at it ?

Not a rant serious questions as I visit a few high visitor forums and its incredibly rare they ever go down and they have thousands of users pretty much all of the time and they are vbulletin too

Dave999 2010-10-26 19:04
Re: Server outages
 
more users here now than when pr 1.2 was released. Dont remember any longer server issues back then.

???

Reggie 2010-10-26 19:14
Re: Server outages
 
Quote:
Originally Posted by *Sonic* (Post 852095)
Its been off most of the evening for me today, and all last night and meego is still down
The site was actually up the most of the night last night as I was monitoring it. It only went down a few minutes at 1 AM (EDT) for backup.

Yes, today is another busy day and I am indeed monitoring the server, tweaking and even restarted it a few minutes ago.

More patience guys. I'm doing the best I can. It will pass.

gazza_d 2010-10-26 19:19
Re: Server outages
 
don't seem to have seen many other replies on here thanking Reggie and his team (does he have a team) for the hard work and effort pugt in keeping this particular show on the road. i frequently found the site timing out over the last 24 hours, and realised why.

*Sonic* 2010-10-26 19:44
Re: Server outages
 
I have no idea on how the site is run, whether it is a hobby or a full time job so cant make comments on how hard the team work, if it is a hobby site then kudos to the people that run it, as I know how hard it is to run a forum and keep it going, if its a full time paid job then that is what Reggie is being paid to do, as I say I have no idea though

Reggie everytime I tried to get on the site last night it either took about 10 minutes to load the page, or just didnt even load and at one point I got an instant page cannot be found

Meego forum is still down for me and this evening it has timed out several times, although it is better than last night

I am on virgin 20mb broadband in the UK if that makes any difference as to why the site never worked at all last night

danramos 2010-10-26 23:36
Re: Server outages
 
I'd thank Reggie but the server keeps timing out whenever I try and it's easier to re-try posts. :P

Reggie 2010-10-29 22:23
Re: Server outages
 
FYI, the forum server might reboot several times today to fix some memory issues.

Sorry for the inconvenience.

scribbles 2010-10-29 22:42
Re: Server outages
 
Quote:
Originally Posted by Reggie (Post 850693)
Daily backup happens at 1AM (EDT) for about an hour or so, so during that time, you might not be able to access the server.
I should have read this thread 2 nights ago. I've been getting pissed the last 2 nights here on the west coast. I start having problems around 10:20pm or so when I can't even access the page until close to midnight :mad:

Reggie 2010-10-29 23:01
Re: Server outages
 
Server back up, and memory fixed with the help of the host.

I will keep monitoring the server though.

Thanks for your patience.

slender 2010-10-30 05:41
Re: Server outages
 
And pretty please could you report about these issues & maintenance in maemo.org announcements/news. Itīs somewhat weird to report here in single thread which gets burried quite quickly and moreover will be offline during downtime.

Thanks for the all hard work :)

.edit
Combo
Facebook+twitter+announcements(maemo.org)+IRC would be nice.

Reggie 2010-10-31 05:17
Re: Server outages
 
Quote:
Originally Posted by danramos (Post 850703)
Old-school backup system where you have to take down the service for a while? I thought everyone did backups on the fly these days (i.e. RAID, Tivoli, etc.). More curious than anything, old-school backup system is a pain but I'm sure it's cheaper, too.
Finally found the problem. Everything should be good now and backup should now work on the fly (posting this while the backup is running).

Again, sorry for the inconvenience and thanks for the patience and understanding.

Radicalz38 2010-10-31 12:12
Re: Server outages
 
does this issue have anything to do about the repo being locked out from being navigated through browser? Yesterday I could still browse and download some updated apps from repo by downloading it manually through pc but today I can't even access the repo's folder view I can just download if I get to have the hotlink.

Reggie 2010-10-31 14:11
Re: Server outages
 
No sorry. The repos run on an entirely different server which I don't have access to. You might want to start a new thread about it.

I've changed the thread title to lessen the confusion.

Thanks.


All times are GMT. The time now is 05:14.

vBulletin® Version 3.8.8