CrystalCommerce

[status] Resolved: This incident has been resolved. stspg.io/x0zf97bgczby?u…

[status] Investigating: Our engineers are aware of the issues with the POS and Admin and are working to resolve it as soon as possible. stspg.io/x0zf97bgczby?u…

[status] Identified: File Exports are working again. stspg.io/s2j0t5nbj89b?u…

[status] Identified: We are seeing spikes in response times on websites that are leading to "Website is under Heavy Load" errors. We are seeing unusual activity hitting the sites that we are working on resolving. Our File Exports are … stspg.io/s2j0t5nbj89b?u…

[status] Monitoring: So far we've been seeing a big improvement after the deploy today. We will keep monitoring things to make sure they are stable, but the "brownouts" seem to have ceased. Please let us know if you do continue to see… stspg.io/dndbjrmcl7c8?u…

[status] Identified: We have been continuing to work on the 503 errors affecting the frontend websites. We will be deploying an update in the next hour or so that we are expecting to help with the site performance. We also have some … stspg.io/dndbjrmcl7c8?u…

[status] Investigating: We’ve identified that system brownouts were being caused by intensive database queries. To address this, we’ve implemented a performance solution that has already resulted in a 90% improvement, and will provide… stspg.io/dndbjrmcl7c8?u…

[status] Resolved: We got the Resque Scheduler back online, and the backed up queues have cleared now. stspg.io/bmp7dr7t2bk1?u…

[status] Identified: We've found the root cause of the background workers issue this morning, and are implementing a fix. It may take a while for all of the stalled jobs to catch up, as there are a lot of them in the queue. We are wor… stspg.io/bmp7dr7t2bk1?u…

[status] Investigating: We are currently working on resolving an issue with Resque Scheduler, which controls all of the background jobs. That means right now all background jobs are down. This includes batch updates, mass create, orde… stspg.io/bmp7dr7t2bk1?u…

[status] Resolved: We've deployed a fix, and line items are saving again. stspg.io/530vbgycj1cq?u…

[status] Investigating: We are currently seeing an issue with line items not saving properly. If you are using the POS, please be aware that this means inventory changes will not be saved for items you add to the POS cart, and the ord… stspg.io/530vbgycj1cq?u…

[status] Resolved: This is fixed, and POS line items are saving properly again. stspg.io/sqyvjldd3vqk?u…

[status] Investigating: The Point of Sale is not currently saving line items. If you add an item to the POS cart, it will not adjust the inventory, and will not show up on the invoice for that order. We are working on fixing this ASAP. stspg.io/sqyvjldd3vqk?u…

[status] Monitoring: The issue has been resolved. It may still need some time for the DNS to propagate., but we are seeing things come back online now. If it still isn't for you, it should be soon. We are continuing to monitor the sit… stspg.io/0qkyyzdn37rp?u…

[status] Identified: We were able to get the nameservers reverted just now, which means now we need to wait for the DNS to propagate. After that, we should start seeing everything come back online. Currently, we are seeing errors, but… stspg.io/0qkyyzdn37rp?u…

[status] Investigating: We are currently experiencing downtime across our system. Our developers are investigating, and working on getting the system up and running as soon as possible. stspg.io/0qkyyzdn37rp?u…