I'd like to give you an update on why we performed an emergency shutdown today at 16:10 UTC.
Before I begin, I want to stress that there was NO data breach and we were able to resolve this issue.
Around 15:44 UTC, our system automatically alerted us that thousands of objects had suddenly been deleted at the same time. To give you a visual example, the bar graph below shows in-game object deletions that happen every 30 seconds. The red bar graph shows when the mass deletion occurred.
After confirming that a large number of objects were deleted and receiving several reports of players unable to log in to their characters, I made the call to perform an emergency shutdown. A similar event of this type has occurred once or twice before, most notably on July 23rd, 2021 when approximately 7,000 houses had vanished from Talus without rhyme or reason. All of these previous occurrences were rectified.
Once the server was shutdown, we immediately began reverting our database to a moment in time just slightly before all the objects were deleted. As a result, our database was rolled back by approximately 6 minutes.
While the roll back was occurring, Development began investigating the root cause for the mass deletion, trying to determine whether this was an unknown exploit from a malicious actor or if this was some process within the game that had gone haywire. We thoroughly examined any areas of the code where large deletions can occur as well as scrutinizing the game logs to find any clues.
Development was able to spot a discrepancy within the logs. After reviewing the code function that produces that log, Development was able to reproduce the discrepancy on our test server. We can now confirm that the mass deletions were caused by an exploit from someone in-game with malicious intent.
Development has fixed this exploit and confirmed that it can no longer be reproduced. This fix is live on Omega and the server came back online at 18:21 UTC without any issues.
Our CSR team has stepped in to investigate and deal with the accounts that were flagged for performing this exploit.
In addition, we have shared this exploit fix with other NGE SWG servers as well as SWGSource in good faith.
Ultimately, this exploit was used to harm SWG:Legends. But they have failed, and we have gotten stronger. During this time, Development has been able to make improvements to many of our backend systems, including:
- Reduced time for a database rollback from hours to minutes.
- Improved our logging practices significantly, with anomaly alerts within minutes for suspicious activity.
- These have also been used to stop credit and item duplications multiple times over the last 3 months.
- Verified our backup and recovery procedures, in the event of mission-critical data loss.
- Added additional access controls to in-game commands.
- ...and many more internal development improvements.
We apologize for the inconvenience this downtime may have caused. We appreciate all your continued support as we work to make this game even better.
May the Force be with You,
- SWG:Legends Staff