Coinbase Explains What Caused Trading Blackout on June 1
The impact of the outage that ensued on Coinbase platform on June 1 affected Coinbase.com, pro.coinbase.com, as well as their mobile applications.
On June 6, Coinbase released a post mortem regarding the outage that happened just as Bitcoin attained $10k price level. The exchange noted that trading through the API which accounts for the majority of trading volume, remained functional throughout this time. They were quick to discover what caused it and remediated the issue.
In the blog post, the US-based cryptocurrency exchange said there was an increase (five times) in traffic within 4 minutes, prompted by the sudden increase in the price of Bitcoin, thereby creating an autoscaling issue that saturated web servers.
As a result of the web servers’ saturation, incoming requests were more than the listening processes, causing the requests to either be queued and timeout, or fail immediately. Request error surged by 50 percent and customers encountered errors while using coinbase.com and the exchange’s mobile apps.
“In an effort to mitigate the saturation, we redeployed the API at 16:20 PDT to increase the machines serving the traffic. Once this deploy completed, the previous deploy’s instances were taken out of rotation, leading to another 2 minute outage due to instances saturating and being marked unhealthy. This was handled automatically by our autoscaling.”
Coinbase said they are working on several improvements to prevent such issues in the future. They were able to fix the health endpoint towards ensuring that saturated instances are not taken out of rotation.
Likewise, they are taking measures to reduce the effect of price-related traffic surges through pre-scaling and caching. On long-term basis, they are working on the improvement of their deployment process towards mitigating some of the autoscaling issues encountered.
The price of BTC declined from about $10k to $8k within some hours prior to the Bitcoin halving event and Coinbase users encountered 502 error messages while interacting with the exchange’s platform, sharing the screenshot with the hashtag #Coinbase on Twitter.