SWITCHengines

All systems are working as expected. We have formally closed the incident. Sorry for the problems this has caused. We are investigating why the switch started to act up. ^jcf

We have been able to fix the root cause (a switch was blackholing internal traffic in our compute and storage clusters). This has been fixed for an hour now and we have seen most services recover. Our engineers are monitoring the situation. ^jcf

We are experiencing problems with various parts of SWITCHengines that lead to various problems with services running on it. We are assembling a team of engineers now to diagnose and work on it. ^jcf

Data, data everywhere...come help us keep it safe recruitingapp-2563.umantis.com/Vacancies/417/…

The SWITCHengines dashboard is currently unavailable. VMs are running normally but it is not possible to make changes to VMs or to provision new projects or VMs via the web. API/command line access is still available. We are investigating. ^AHa

Communication via the message queuing system has been restored. All functions are reporting normal operation in Lausanne again. ^AHa

Control plane operations such as starting/stopping/resizing VMs are currently unavailable in Lausanne. Running VMs are not impacted. Underlying connectivity is verified, and we continue to investigate. ^AHa

Performance on control plane in ZH degraded again this afternoon. We have increased the resources for key components which has relieved the performance issues. We are continuing to monitor closely while debugging root cause. ^AHa

Performance stabilised in ZH from 13:00 following troubleshooting. It is now possible to provision or deprovision VMs with reasonable response times. Running VMs were not impacted. ^AHa

Since c 11:30 see degraded performance on SWITCHengines control plane Zürich. Running VMs are not impacted but creating, deleting or changing VMs is impacted and some APIs are not responding. We are investigating and may need to restart some systems to restore service. ^AHa

Supporting new services in 2021 - after a successful pilot we're looking for expressions of interest in Office365 backup with your Data@SWITCH @SWITCH_ch ICT Focus.

Operational improvements 2021 - more OpenStack upgrades, new S3 cluster, ISMS maturity @SWITCH_ch ICT Focus

Our strategic focus areas 2021 - Data@ SWITCH, supporting data management and data science use cases for higher education & research in CH @SWITCH_ch ICT Focus

We're part of the basic infrastructure roots in the swissuniversities Open Science vision @SWITCH_ch ICT Focus

BlueBrain Nexus on k8s running on SWITCHengines presented at @SWITCH_ch ICT Focus 2020

@widodh @larsmb @danvanderster @Ceph Thanks to @simonl for pointing out that we have an older cluster: $ ceph osd dump | head -n 3 epoch 2228763 fsid 2a246621-98fc-40dd-829d-8e885452627e created 2014-06-26 19:26:34.724491

@widodh @larsmb @danvanderster @Ceph Oh damn - we are the youngsters on the block:

Pretty cool to see that this @Ceph cluster was installed more than 7 years ago. It start with the Argonaut (v0.48) release and has been upgraded to Nautilus (v14.2) while safely storing all the data. It saw the introduction of BlueStore, the Mgr and many other features. Nice!

Connectivity in Lausanne has been restored. Cause was a subset of IPv4 routes being blackholed unexpectedly. ^AHa

We are experiencing network problems in our LS region. This causes VM IPv4 connections interrupted. We are investigating the problem. ^HR

All assignments of IPv4 addresses have been restored. If you still experience problems, please file a ticket with engines-support@switch.ch