ASU Web Services

@olesovhcom It shouldn't take this long to remove the bad router, and restart services to the rest of the world!

Ces derniers jours, l’intensité des attaques DDoS a beaucoup augmenté. Nous avons décidé d’augmenter notre capacité de traitement de DDoS en ajoutant de nouvelles infrastructures dans notre DC VH (US-EST). Une mauvaise configuration du routeur a provoqué la panne du network.

Suite à une erreur humaine durant la reconfiguration du network sur notre DC à VH (US-EST), nous avons un souci sur la toute la backbone. Nous allons isoler le DC VH puis fixer la conf.

@olesovhcom @olesovhcom - How long until services restored? Why is there no backup/roll back plan? And why does it take nearly an hour to fix a config issue!

@VergeresP It means the status is normal. It will have a normal status if it's in normal boot or rescue64-ovh but has ping and no faults logged.

@ASUCloud Hello by having this on our OVH manager, does this mean our server is OK and safe after the fire? Can you please confirm this?

@ASUCloud any support concerning a contact change (admin or technical)in my OVH manager? Trying to replace a person who left my company. But never ending story 😡😡😡and no help from OVH support 😡😡😡. We are owning and paying and we are stucked as we cannot change 😡😡

@VergeresP Have you tried to change boot option, reselect Rescue64-pro and enter a different email address, and then reboot? This should work

@ASUCloud I hope this is the problem because I still cannot access the server who moved after the fire....

@FactoryOyster @olesovhcom No problem at all. Hope you're back up and running soon and your business recovers.

@ASUCloud @olesovhcom Thankyou, very helpful again really appreciate your time 🙂

We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend to activate your Disaster Recovery Plan.

@FactoryOyster @olesovhcom Yes. Like i said ours was like this for 5 or more days. If you can't reboot or launch IPMI due to intervention (you'll get that error in the manager) then you'll have to wait until they finish what they're doing. SSH will connect because the network is connected.

@ASUCloud @olesovhcom my tech guy says everything looks good to go just email needed, pinging for 2 days now, we did spk to OVH and they said soon as rack completed we get email and we imaginged them needing to reboot after which happened this morning and email said completed too?

@FactoryOyster @olesovhcom Yes. That means OVH are still working on it. They have the process much quicker now. Ours was pinging for 5 days or so before we got it back but i know they have been streamlining the process

@ASUCloud @olesovhcom Ok so my tech guy said we did try that this morning but Problem is we can’t reboot the server due to the intervention? if that makes any sence to you (sorry im not good on these things) 😶

@FactoryOyster @olesovhcom Have you tried to change the boot option again, reselect Rescue64-pro and enter an email, then reboot it. See if you get the email this time?

@ASUCloud @olesovhcom Looks like it was rebooted tbis morning and we spoke to OVH day before yesterday and they said they we just finishing the rack then everyone in rack will get email, we did get email to say server check and then another to say completed too?

@FactoryOyster @olesovhcom Has your server previously been in rescue64-ovh (internal)? If it's already been in internal Rescue then you should have got an email with password. If it hasn't then they will likely put it in internal rescue, then you'll get an email when they boot it back in to Rescue64-pro.

@ASUCloud @olesovhcom hi, yes we are in PRO and server pings all looks good to go but still no password to access it, any ideas? or would you know how long before we are likly to get the email 🙄 Thanks in advance for any help

@FactoryOyster @olesovhcom Are you in Rescue64-pro or rescue64-ovh?

@ASUCloud @olesovhcom Hi, so we are still waiting on password, we are in pro and server pings fine but just cant get in, how long did it take for your password to come? 🤔

@leviBKL @olesovhcom Did you check the disks and raid from Rescue64-pro? If not reboot into rescue and then check they're intact. You can also try netboot with the ovh kernel incase the ondisk kernel is the issue

@ASUCloud @olesovhcom I think it can’t see the raid

@leviBKL @olesovhcom Does it boot into Rescue64-pro ok? It only has issue in normal boot?

@ASUCloud @olesovhcom Yes, but my root password is not recognised, I am stuck at the login screen and can’t login. Server not responding to ping. OVH automatic check says Software configuration to be corrected by customer , But I didn’t changed any software on it since the incident.

@leviBKL @olesovhcom Have you tried checking IPMI for any issues?

@olesovhcom Hi @olesovhcom I can’t access my server after reboot (internal hd boot), can you please ask someone to take a look ? i sent a ticket but no answer, i called support but still no answer.

@ProchSerj @IamAgonia @olesovhcom No you won't get the password until they boot it into Rescue64-pro. They are still working on it

@ASUCloud @IamAgonia @olesovhcom But I still can't log into the server, because don't know the password. If they recycled (removed) my data, I'll shoot myself 😪

@ProchSerj @IamAgonia @olesovhcom Not sure what that one is I'm afraid. Haven't seen that before

@ASUCloud @IamAgonia @olesovhcom my server in "rescue64-recycling (Rescue recycling 64bits)" boot mode. what is it? ping is ok, but I can not login. I do not know password. I am afraid they gonna to recycle my data (deleting them). 🙁

@IamAgonia @ProchSerj @olesovhcom That was a generic email sent to all customers in SBG1 who weren't destroyed.

@ProchSerj @olesovhcom Wtf. Got an email that my service will be up before 23 April. But I got all files. If I replace my service with my substitute server. Will I get it immedently.

@FactoryOyster @olesovhcom When your server is up you'll get an email with the credentials to login to the rescue system. It's up to you what you do within the rescue system. I'd recommend doing a disk check and back up before normal boot. There should be no issues but a backup is advisable.

@olesovhcom do you use IPMI module to access our server or is public network up? We can see on the backend our network interface is up but as it’s in rescue mode attempted to access via IPMI &it just says it couldn’t conect due to the intervention? can someone help me please 🥴

@FactoryOyster @olesovhcom You'll get an email when it's ready and in Rescue64-pro. It's in rescue64-ovh at the moment i assume? Feel free to DM if you need any assistance

@ASUCloud @olesovhcom Hi did u use IPMI module to acess yor server, was your public network up? We can see on te backend our network interface is up but as it’s in rescue mode atempted to acess via IPMI and it just says it couldn’t conect due to the intervention was just wondering if u saw same issue

@JMarcBlanchard @olesovhcom Yes, any Linux utility will work in rescue mode. Rsync / NFS / scp etc

@ASUCloud @olesovhcom Hi, how did you manage to do a backup in rescue mode ? Is rsync working ?