Announcement

Collapse
No announcement yet.

mmmm, so what happened?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • mmmm, so what happened?

    Looks like you guys had a busy 12 hours and the biggest outage I have ever seen. What happened?

  • #2
    I got an email saying:

    There has been a major outage in our US data center where the whole building went in emergency mode and the electricity was completely shut down for hours. The power outage in the US data center is now being resolved and everything comes back to normal.
    But how does that explain that my sites hosted in the Finland datacenter was down?

    I was fooled to think that the idea with 4 dns:es was to create redundancy for these kind of situations. I think I have read somewhere that the nameservers are spread out on different data centers. Aren't they?

    Comment


    • #3
      All my servers are in the UK and whilst the servers stayed up, all DNS failed. (I have a monitor running which pings the IP and also tests the http connection)

      I was under that impression too, that ns1 and ns2 are in the USA, but ns3 is in UK DC (Pulsant, Maidenhead) and ns4 is in Ficolo

      Comment


      • #4
        Sorry Clive, pulled an all nighter. Just have risen from the grave...

        Тhe power outage in the US data center was caused by an Emergency Power Off (EPO) triggered during a routine on-premise safety system maintenance.

        The whole power supply was cut, which rendered the UPS systems incapable of taking over.

        The sudden power cut resulted in a network equipment failure on our end, which took our admins 5 hours to fix.

        They needed a few more hours to take all basic services to full recovery afterwards.

        As a result, most shared hosting servers have been restored back to normal.

        Some of you may still experience issues with your email communication due to the large message queues that have been formed on some of the mail servers.

        We are still working on restoring all services on SupremePanel26.com, as well as on some Hepsia-based VPS

        Comment


        • #5
          Thanks for the update yav0r,

          Can you please explain why this affected the clients on the European DCs?

          Aren't the four nameservers supposed to give redundancy for things like this?

          Comment


          • #6
            Hey Maxa, they are indeed supposed to, that is why we had dns3 and dns4 pointed to US/AU datacenters. Having said that the complete power outage killed the master DNS which was supposed to switch over but because of the sudden and complete outage did not had time to realize the dns1/dns2 were failing so they never kicked in. As a result all 4 dns were down completely until the matter was resolved. The fact of the matter is first the UK/AU/BG/FI datacenters came up since the master DNS went online first, and the rest of the US remained dark. We will be conducting a thorough investigation and work on several possible workarounds in order to ensure this is a one time "offer" and never happens again. That much I can promise (for all its wroth).

            Comment


            • #7
              Thanks for the explanation and all your hard work

              Comment


              • #8
                thanks for the email the other day explaining what happened. i use the free reseller program. did our hosting customers that have purchased hosting through our storefront also receive an email explaining what had occurred regarding the outage? thanks

                Comment


                • #9
                  Hello, we're now preparing a detailed explanatory message that will be posted in customers' Control Panels later today. Thanks for your patience and understanding in this cumbersome matter!

                  Comment

                  Working...
                  X