Current System Status

Jan

N/A

Dec

N/A

Nov

N/A

Oct 16

Unexpected Gateway Disconnection for UME

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Oct 18, 17:07 UTC+8

Identified - The issue has been identified that OTA database reached its connection limit, causing a gateway disconnection problem at 11:48. The problem is resolved after the OTA database automatically restarted at 11:55 on Oct 16.
Oct 17, 10:24 UTC+8

Investigating - The customer reported at 16:40 that multiple gateways were disconnected at 10:00. We are investigating the root cause and will provide another update.
Oct 16, 16:40 UTC+8

Sep 18

Cassandra Database Access Permission Issue

Recovery - data supplement - we have written back the data that was lost during the problem period by 22:11 UTC+8 on Sep 18.
Sep 18, 22:11 UTC+8

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Sep 18, 13:58 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Sep 18.
Sep 18, 13:35 UTC+8

Investigating - We have identified a problem with Cassandra database access permission issue, which prevented us from reading and writing data from database. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Sep 18.
Sep 18, 13:33 UTC+8

Aug

N/A

Jul 17

Abnormal Gateway Connection Status

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Jul 17, 18:29 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Jul 17.
Jul 17, 16:10 UTC+8

Investigating - We have identified a problem with abnormal gateway connection status. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Nov 7.
Jul 17, 15:10 UTC+8

Jun

N/A

May

N/A

Apr 18

Unexpected Websocket Disconnection and Abnormal Database Writing

Recovery - data supplement - we have written back the data that was lost during the problem period by 15:07 UTC+8 on Apr 22.
Apr 22, 15:07 UTC+8

Monitoring - A fix has been implemented and is rolled out to production by Apr 18 - we are currently monitoring and verifying with impacted customers.
Apr 18, 09:50 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Apr 18.
Apr 18, 08:29 UTC+8

Investigating - We have identified a problem with unexpected websocket disconnection between gateway to IOEP and abnormal database writting of S3. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Apr 18.
Apr 18, 08:12 UTC+8

Mar 31

Unexpected Gateway Disconnection for UME

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Mar 31, 14:45 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Mar 31.
Mar 31, 13:00 UTC+8

Investigating - We have identified a problem with Gateway connection, which cannot communicate with the cloud IOEP server. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Mar 31.
Mar 31, 12:42 UTC+8

Feb

N/A

Jan 30

Unexpected Extra Network Traffic of Iwatani Single Tenant

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Jan 30, 18:28 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Jan 30, 19:00 UTC+8.
Jan 30, 18:00 UTC+8

Investigating - We have identified a problem with OTA server, which make customer’s gateway keep downloading firmware, causing an increase in network traffic. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Jan 30.
Jan 30, 17:42 UTC+8

Dec

N/A

Nov 7

Unexpected Gateway Disconnection

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Nov 9, 13:07 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Nov 9.
Nov 7, 18:00 UTC+8

Investigating - We have identified a problem with Gateway connection, which cannot communicate with the cloud IOEP server. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Nov 7.
Nov 7, 13:45 UTC+8

Oct 5

Unexpected Extra Downtime While Upgrading EKS Version From v1.23 to v1.24

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Oct 6, 04:30 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by 03:30 UTC+8.
Oct 6, 03:30 UTC+8

Investigating - We have identified a problem with Istio ingress controller pods, which make EKS cannot correctly receive data after upgrading. We are investigating the root cause and will provide another update before 8:00 AM UTC+8 on Oct 6.
Oct 5, 23:30 UTC+8

Sep

N/A

Aug

N/A