more armies in limbo...? urgent action needed |
Post Reply |
Page <1 3456> |
| Author | |||
GM Stormcrow
Moderator Group GM Joined: 23 Feb 2010 Location: Illyria Status: Offline Points: 3820 |
Post Options
Thanks(0)
Quote Reply Posted: 26 Feb 2014 at 10:33 |
||
^^ qft, and yes - exactly one (or more) of those things.
And producing a replication (outside of a live environment under stress) isn't happening as yet.
But we have, at least, found the precise moment that a thread deadlock/collision occurred whilst one or more threads were under pressure.
However, stepping through without dumps in a hyperthreaded dual hexacore (24-core total) environment isn't a walk in the park :)
Still working on it - will keep you posted.
SC
|
|||
![]() |
|||
Le Roux
Wordsmith Joined: 30 May 2012 Status: Offline Points: 151 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 19:10 |
||
|
. . . and ?
|
|||
|
|||
![]() |
|||
darpansah
New Poster Joined: 19 Jan 2014 Status: Offline Points: 11 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 19:17 |
||
|
Made some attcks on red cities report came in saying mission aborted cant attack friendly city...lol....is this a glitch or ??????
Edited by darpansah - 04 Mar 2014 at 19:18 |
|||
![]() |
|||
Rill
Postmaster General Player Council - Geographer Joined: 17 Jun 2011 Location: California Status: Offline Points: 6903 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 19:24 |
||
|
darpansah, there might be reinforcements in the city from an alliance with which you are NAP'd or confed. Have your alliance leader check your diplomacy.
|
|||
![]() |
|||
GM Stormcrow
Moderator Group GM Joined: 23 Feb 2010 Location: Illyria Status: Offline Points: 3820 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 23:03 |
||
EDIT: ty Le Roux for nudging me to give an answer. It's still not a complete answer, but I don't communicate enough atm.
Step through is complete, and it appears to have been caused by a broad combination of multiple things happening at the same time:
All of these things put pressure on the database. However, the database is built so that it should work through these things linearly (and it's been under a lot more pressure than this in the past).
However, basically, a deadlock that should have been broadly impossible (before I get jumped on, I do understand there's no such word in code) in a nolock read /rowlock write environment produced a NULL in a (db-constrained) NOT NULL column.
And other not-directly related things had fits thereafter, based on reading a single NULL data entry that really shouldn't have been there if there was fair and just code deity.
This behaviour was unexpected enough for us to have raised a ticket with our db vendors - with whom we have a fairly close relationship - and we've been rapidly through the "are you patched up qs" and it's been escalated into realms beyond our understanding, but will hopefully descend again in a SQL update patch in the near future. We'll keep on applying pressure for a satisfactory resolution, but there is only so much we can do.
Somehow we manage to find these issues that have everyone shaking their heads. On a different issue recently we found a (reproducible) bug in .NET that has entailed us actually running our very own, unique version of the core .NET framework to enable us to continue development. I do sometimes wonder if we're not a bit too far on the "extreme" side, but I gather a "fix for all" will make it into the next update.
On a more practical level, I can't say it can't and won't happen again, but the areas that exhibited obvious symptoms (
such as herb respawning and some armies getting stuck)
have been instructed how to bypass such an event so they now understand how to handle NULLs, even in columns that are apparently constrained to be un-NULLable.
We've also written some monitoring procedures to actively look for brief glimpses of these creatures that live at the bottom of the Mandelbrot set, and to alert us immediately should they happen in the future. I can't guarantee we've got them all or are looking in all the right places, but it's a stab at it anyway.
In terms of a longer term fix, beyond the hopefully forthcoming vendor patch, there are various options we're exploring in terms of re-architecting various core components.
Many thanks for your patience, and I apologise if it got a bit technical in some areas above - I know some of you are techies, and many aren't, so I tried to balance decent answers with explanatory ones.
SC
Edited by GM Stormcrow - 04 Mar 2014 at 23:05 |
|||
![]() |
|||
Angrim
Postmaster General Joined: 02 Nov 2011 Location: Laoshin Status: Offline Points: 1173 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 23:12 |
||
|
so what i'm reading is: sieging ELECTROK broke the database.
getting a NULL into a NOT NULL column sounds medal-worthy. Edited by Angrim - 04 Mar 2014 at 23:13 |
|||
![]() |
|||
Rill
Postmaster General Player Council - Geographer Joined: 17 Jun 2011 Location: California Status: Offline Points: 6903 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 23:13 |
||
|
Stormcrow, have you ever considered repositioning Illyriad Ltd. as a bug-finding enterprise? Where you create value by testing the limits of software and hardware architecture and identifying bugs for organizations like Google?
You could put us players on salaries and pay us to break stuff!
|
|||
![]() |
|||
Merlinus
Greenhorn Joined: 15 Feb 2014 Location: Tennessee USA Status: Offline Points: 81 |
Post Options
Thanks(0)
Quote Reply Posted: 04 Mar 2014 at 23:39 |
||
|
I used to write qa code, but I'm glad I'm not writing THIS code. :)
Just a question: Has anybody looked into the .round function--32 v. 64 bit for time lag crashes? Just wondering. |
|||
|
In Peace we reign. In War we RULE!
Long live the Royal House of Merlinus! |
|||
![]() |
|||
Miklabjarnir
Greenhorn Joined: 07 Mar 2012 Status: Offline Points: 73 |
Post Options
Thanks(0)
Quote Reply Posted: 05 Mar 2014 at 01:03 |
||
Rather not, Rill. I have been a professional breaker of code (AKA QA manager) long enough, and am currently retired
|
|||
![]() |
|||
GM Stormcrow
Moderator Group GM Joined: 23 Feb 2010 Location: Illyria Status: Offline Points: 3820 |
Post Options
Thanks(0)
Quote Reply Posted: 05 Mar 2014 at 19:37 |
||
We think so! Not sure our database vendor agrees :)
One of the most perfect technical descriptions of the current illyriad corp I've ever read
tbh
Merlinus,
that question is probably above my pay grade. I'll ask TC.
Best,
SC
|
|||
![]() |
|||
Post Reply |
Page <1 3456> |
|
Tweet
|
| Forum Jump | Forum Permissions
You
cannot post new topics in this forum
You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |