My Thoughts

We're fully aware of the timeout issue that has been plaguing the site over the past week and it has been getting increasingly worse.

As per the post I wrote a while back regarding a large server order, all of those servers have been received and installed and put in to production. This issue we are facing today is as a result of the *NEW* massive increase in demand due to the servers we put in to place.

This is the ongoing issue with deviantART. The growth curve is very fast, demand on the servers increases exponentially and hardware requirements are also exponential.

In other words, 1 server = X traffic. 2 servers does NOT equal 2X traffic. 100 servers = X traffic, 200 servers = 1.4X traffic or so. The bigger the site gets, the more it costs to serve each individual user.

Our mission is to remain a privately held company, we feel that taking a financing round under typical terms would seriously impair our ability to properly serve the community and would put a heavy emphasis on "where ART meets corporation."

On the other hand, the price we pay is timeouts sometimes as the site grows very rapidly.

There are financing options which allow us to retain full control, we are actively seeking those but they take a bit longer.

All is not even remotely lost, this issue will be resolved and our ability to scale once again is hours away.


Official Server Status Update

Two machines are particularly holding us up. The issue is in fact rather flattering to the demand for deviantART: An unforeseen bottleneck with a reverse proxy and load balancer has exceeded its physical limitation to serve traffic.

This happened as a result of new hardware installations which enabled the serving of approximately 8 million additional page views per day which generated approximately 173,913,043 overall additional "server architecture hits" per day.

We have a 3 prong approach to solving the issue.

throwing hardware at it. 1 hour from now we should know the results of this test. (2:34pm) it may resolve the issue all together.

installing apache2 across the board. this would take until late tonight.

installing squid and configuring, this would take one more day.

So this issue will be fixed in either 1 hour, 8 hours, or 24 hours from now. The solution will be something we can count on and scale on for a good amount of time.

Thanks for your patience we won't sleep until this is resolved.

-- A