Looking back at Solana outages, why are they frequent?

Oct 06, 2022

Solana has now released Mainnet-v1.10.40 and v1.13.2, which fix the issue that caused the network outage last week. Last week, we have learned, Solana outages is due to the error of a validator configuration, running the repeated validator instances, this means that when it’s turn to produce block and each instance will be produce a block, for the other validators, some see is the instance of A block, some see is the instance of the block B, They couldn’t tell which block was correct, eventually causing the network to go down.

Solana’s “downtime” problem is now often criticized by the community, in addition to last week’s more than 10 hours of downtime, Solana has experienced several downtime or network jamming in the past, WuBlockchain organized as follows：

1. Around October 1, 2022, the network is down due to a node configuration error.

2. Around August 3, 2022, there was a mass theft of tokens from Solana wallets, which was eventually found to be due to a vulnerability caused by the centralized Sentry server.

3. Around June 1, 2022, due to the Durable Nonce function vulnerability in the transaction, the network restarted, and the interruption time was about 4.5 hours.

4. Around May 1, 2022, Solana network nodes lost consensus due to the emergence of a large number of bot transactions due to the casting of a new NFT project, after which block production was suspended for seven hours.

5. Around January 21, 2022, due to the high volatility of the market, the network was flooded with a large number of trades submitted by arbitrage bots, resulting in a heavy load on the network and an outage of up to 30 hours. However, it was officially characterized as Degraded Performance at the time, and the Solana community subsequently updated the main network to version 1.8.14 to try to improve the network state.

6. Around September 14, 2021, due to the hot IDO activity of Grape Protocol on Raydium platform, many users sent a large number of transactions through machine scripts written by them, which caused “memory overflow”, crashed validator node, and finally the whole network could not produce blocks. The outage lasted 17 hours.

7. Around September 3, 2021, the network is unstable, and the performance degrades for about one hour.

8. Around May 4, 2021, network performance degraded, resulting in a large number of transactions not being completed.

Looking back at historical network events, we will find that the emergence of a large number of transactions is the main reason for historical network interruption, which may be related to Solana’s mechanism. According to Hu Zhiwei, president of BIANJIE.AI, since Solana also transmits consensus messages between validator nodes as a special transaction message, A large number of messages are blocked, resulting in the failure of consensus messages to be delivered normally, and consensus cannot be carried out normally. At the same time, some features of Solana were exploited to cause network downtime. For example, the write-lock for concurrent transactions is locked on many important addresses, which makes the transactions become sequential rather than concurrent, greatly affecting the processing ability of messages. To handle forks, nodes keep possible fork information, resulting in memory overflow.

Solana co-founder Anatoly Yakovenko acknowledged the problem in the face of common spam transactions leading to network performance degradation and even downtime, it said it would introduce “actual flow control” in version 1.9 of the Solana mainnet beta to address this issue. As for network outages caused by transaction nonce and node configuration errors, Solana officials quickly released fixes for node upgrades. However, the frequency and duration of this high frequency of downtime are inevitably criticized by the community. Solana and other new public chains are still in the early stage. With the adoption of the network and the diversification of the ecology, it is bound to usher in more challenges in the future.

Reference:

https://mp.weixin.qq.com/s/JwPpmJOxIIsR7MzY_CtDog

https://twitter.com/SolanaStatus

https://twitter.com/laine_sa_

Wu Blockchain

Discussion about this post