Current System Status

Sep

N/A

Aug

2025-08-14 EneSense 資料缺值

2025-08-13 所有 SaaS 無法登入（IoE Suite/Portal、EMS APP、EneSense）

Jul

N/A

Jun

N/A

May

2025-05-28 iOS EMS APP 關聯 EDGE 過程中設定Wi-Fi時App會閃退

Apr

2025-04-21 [EG2] Redis 清空 導致 API 呼叫誤判 Gateway 斷線

Mar

Unexpected Gateway Disconnection for UME

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Oct 18, 17:07 UTC+8

Identified - The issue has been identified that OTA database reached its connection limit, causing a gateway disconnection problem at 11:48. The problem is resolved after the OTA database automatically restarted at 11:55 on Oct 16.
Oct 17, 10:24 UTC+8

Investigating - The customer reported at 16:40 that multiple gateways were disconnected at 10:00. We are investigating the root cause and will provide another update.
Oct 16, 16:40 UTC+8

2025-03-05 coreDNS 被全數收回，導致 PROD 服務中止

Feb 23

2025-02-23 CPO 付款狀態錯誤，導致無法進行下一個充電交易

Feb 21

2025-02-21 WL APP 強更 Redirect 到錯的 Store 頁面

Feb 05

2025-02-05 IWATANI fargate更新後回復太慢

Feb 04

Intermittent CoreDNS Failure Impacting CPO Service

Monitoring - CoreDNS resource configuration has been adjusted. We are monitoring system stability and preparing to roll out further mitigation in production.
Feb 4, 16:37 UTC+8

Identified - The issue was identified as intermittent CoreDNS failure, which caused CPO service to return 5xx errors. Root cause is insufficient CoreDNS resource allocation and lack of redundancy.
Feb 4, 16:17 UTC+8

Investigating - Customer reported recurring errors at multiple timestamps.

Engineers confirmed service unavailability correlated with CoreDNS instability.
Feb 4, 12:05 UTC+8

2025-02-04 CPO反應response 5xx

Jan 16

Unexpected Login Failure for IoE Suite

Monitoring - The issue has been mitigated by rolling sre-pgbouncer-readonly to on-demand instances. The service is currently stable, and we are monitoring to ensure no further customer impact.
Sep 23, 15:49 UTC+8

Identified - The root cause was identified as a compatibility issue after upgrading Karpenter, which triggered EC2 instances to Amazon Linux 2023. Due to instance type differences between QA and Prod, m7g.xlarge required longer initialization, causing sre-pgbouncer-readonly Pod eviction and unavailability on spot instances.
Sep 23, 15:45 UTC+8

Investigating - Customers reported login failures and 504 server errors at 15:39. We are investigating the root cause and will provide another update.
Sep 23, 15:39 UTC+8

2025-01-16 部分客戶用的Website/App無法登入

Dec

N/A

Nov

N/A

Oct 16

Unexpected Gateway Disconnection for UME

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Oct 18, 17:07 UTC+8

Identified - The issue has been identified that OTA database reached its connection limit, causing a gateway disconnection problem at 11:48. The problem is resolved after the OTA database automatically restarted at 11:55 on Oct 16.
Oct 17, 10:24 UTC+8

Investigating - The customer reported at 16:40 that multiple gateways were disconnected at 10:00. We are investigating the root cause and will provide another update.
Oct 16, 16:40 UTC+8

Sep 18

Cassandra Database Access Permission Issue

Recovery - data supplement - we have written back the data that was lost during the problem period by 22:11 UTC+8 on Sep 18.
Sep 18, 22:11 UTC+8

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Sep 18, 13:58 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Sep 18.
Sep 18, 13:35 UTC+8

Investigating - We have identified a problem with Cassandra database access permission issue, which prevented us from reading and writing data from database. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Sep 18.
Sep 18, 13:33 UTC+8

Aug

N/A

Jul 17

Abnormal Gateway Connection Status

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Jul 17, 18:29 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Jul 17.
Jul 17, 16:10 UTC+8

Investigating - We have identified a problem with abnormal gateway connection status. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Nov 7.
Jul 17, 15:10 UTC+8

Jun

N/A

May

N/A

Apr 18

Unexpected Websocket Disconnection and Abnormal Database Writing

Recovery - data supplement - we have written back the data that was lost during the problem period by 15:07 UTC+8 on Apr 22.
Apr 22, 15:07 UTC+8

Monitoring - A fix has been implemented and is rolled out to production by Apr 18 - we are currently monitoring and verifying with impacted customers.
Apr 18, 09:50 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Apr 18.
Apr 18, 08:29 UTC+8

Investigating - We have identified a problem with unexpected websocket disconnection between gateway to IOEP and abnormal database writting of S3. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Apr 18.
Apr 18, 08:12 UTC+8

Mar 31

Unexpected Gateway Disconnection for UME

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Mar 31, 14:45 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Mar 31.
Mar 31, 13:00 UTC+8

Investigating - We have identified a problem with Gateway connection, which cannot communicate with the cloud IOEP server. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Mar 31.
Mar 31, 12:42 UTC+8

Feb

N/A

Jan 30

Unexpected Extra Network Traffic of Iwatani Single Tenant

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Jan 30, 18:28 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Jan 30, 19:00 UTC+8.
Jan 30, 18:00 UTC+8

Investigating - We have identified a problem with OTA server, which make customer’s gateway keep downloading firmware, causing an increase in network traffic. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Jan 30.
Jan 30, 17:42 UTC+8

Dec

N/A

Nov 7

Unexpected Gateway Disconnection

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Nov 9, 13:07 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by Nov 9.
Nov 7, 18:00 UTC+8

Investigating - We have identified a problem with Gateway connection, which cannot communicate with the cloud IOEP server. We are investigating the root cause and will provide another update before 18:00 UTC+8 on Nov 7.
Nov 7, 13:45 UTC+8

Oct 5

Unexpected Extra Downtime While Upgrading EKS Version From v1.23 to v1.24

Monitoring - A fix has been implemented and is rolled out to production - we are currently monitoring and verifying with impacted customers.
Oct 6, 04:30 UTC+8

Identified - The issue has been identified and a fix is ongoing. This fix is currently soaking and will be fully deployed to production by 03:30 UTC+8.
Oct 6, 03:30 UTC+8

Investigating - We have identified a problem with Istio ingress controller pods, which make EKS cannot correctly receive data after upgrading. We are investigating the root cause and will provide another update before 8:00 AM UTC+8 on Oct 6.
Oct 5, 23:30 UTC+8

Sep

N/A

Aug

N/A