MEGA FLUSH at Mika's proxy ?

[wrickard]

Maybe the stats server was hacked too?
Worth a look?

 

[TwoFace]

wrickard, it would be if the teamanandtech.com domain had anything to do with the anandtech.com domain, but it doesn't ... teamanandtech.com is hosted off our own boxes with completely different IPs and all that compared to anandtech.com Still not bad thinking tho' might be some punk decided to make us all have one "§%!§"!% day :|

With love and respect your fellow TA member

Two-Face
My stats:
RC5
OGR-25
Seti
Gamma Flux

 

[Adul]

So what is going on here?

 

[dkappos]

I am on page 2 again on week stats .. this means .... FLUSH EVERYTHING

 

[Fandu]

http://www.teamanandtech.com/dnet/rc5/weekstats/byhost.asp

8,633,562 RC5 blocks and growing.
1.46 P(penta?) OGR nodes and growing

 

[MereMortal]

Jiggle the handle on that proxy---it'll stop flushing!

 

[Mika]

As best as we can tell, for some reason, core (the mail pproxy) started flushing blocks to edge, which in turn flushed them back to core, which in . . . well you get the idea. We are not sure what caused the problem at this point, but will hopeully come up with some sort of answer .

Brad is currently writing an app to parse the rc5log and ogrlog files to remove any dupes in the files. This should give us the approximate log files as they would have been without the loop. For the time being, we have stopped running the stats, as they were taking way too much time.

We hope to have everything sorted out and back up and running in a few hours.

Sorry for the trouble and confusion.

Mika

 

[imported_CyberWire]

thx fer the update milka, hope things get worked out soon

 

[Polo]

Thanks for all the hard work Brad and Mika !

Now, the question is : Are we able to leave without stats ?

 

[Dougal]

Thanks for the upgrade guys. Mika, Braad, good luck with this one.

 

[bphantom]

Phew! Ok, everything is fixed and back to normal. Not exactly the most pleasant thing to do, but at least it is back up. The stats pages are updating once again. Enjoy!

For pure insanity... The console log file is 855MB and the rc5key log file was 303MB uncompressed (on Core and Edge). Fortunately compression shrunk it down far enough to allow download and remote examination. Fun stuff I tell you!

Brad..

 

[Jator]

Thanks for fixing it guys. You show the value of keeping you 2 around.

***EDIT***

Wait, you mean I didn't do over 30,000 blocks today?!?

 

[Mika]

Thank you for the great work Brad!

Brad's ingenuity and programming skills made it possible to repair the log files, and replace the corrupt log files without taking the pproxy offline!

 

[Poof]

Thanks much guys!!!

(/me slinks back to page 5 again... )

 

[mindless]

Thanks Mika and bphantom

 

[bphantom]

Sorry Jay! Hate to disappoint you and everyone else. If you thought that ~8.4 million daily block count was high, well please note ppstats stopped working 2 1/2 hours after the problem started. The problem was active for seven hours! Since I know everyone likes stats: I let my box crank on the log file I grabbed and there were a total of 24,481,141 2^28 blocks that passed from Edge. That is a bit insane to say the least. hehe

As for not taking the proxy offline... That was a work of art.

Edit: BTW Jay, your original totals as of 14:19 UTC were 66,525 blocks.

Brad..

 

[Jator]

Dammit Brad, don't tease me like that. I could only wish for a 60,000 block day. Last time I checked, there was no BORG associated with my name, so I guess I am SOL.

 

[jaygee]

Guess we all had our fifteen minutes (or thereabouts) of fame.

 

[ViRGE]

Here's to another act of unselfishness by Brad and Mika. Not only did Brad get out of bed early(I know how to get to his pager), and give up some time on a short vaction to fix this, but he did a fantasic job. And Mika, he deserves something for just not pulling all of his hair out. Once again, thank you guys!

PS I hope to release more details on what happened, later

 

[BGod]

My hats off to you guys!!!

I was wondering how I went from #22 to #2 in just 6 hours. Wish I could have stayed there.

 

[Adul]

Wow, so that is what happened. To bad it wasnt real blocks. The DPC wouldhave freaked out at such a dump.

 

[ViRGE]

Well, I have some info for you guys. After speaking with Mika, it looks like the most likely cause(though we're not even close to being 100% sure that this is the case) is that the Core couldn't find the Dnet pproxy, and picked the Edge instead for some weird reason(most likely because it's ont he same subdomain), hence what wee got. Mika instituted a fix for it now though, so it shouldn't happen again. Thanks for understanding guys.