Updated March 4, 2026

Description of Issue

A configuration distribution issue within our ingress system caused certificate and routing disruptions across the us-central1 region. This resulted in intermittent 502 responses, TLS/certificate errors, and temporary unavailability of the Gigalixir console.

Scope of the Issue

The issue affected applications running in the us-central1 GCP region.

During the remediation process, each ingress system required a rolling restart, which caused a brief but broader period of disruption. For the affected applications, this downtime lasted 9 minutes on average.

Prevention Measures

We are implementing changes to further isolate ingress configurations across our infrastructure. These measures include, but are not limited to:

• improved isolation of ingress configuration distribution

• additional validation of configuration changes before application

• enhanced monitoring and alerting for ingress health

Customer Recommendations

There are no customer recommendations for this incident. The incident was entirely internal to the infrastructure of Gigalixir.

Incident Timeline

3 March - 22:15 UTC / 16:15 CST

We began to see intermittent disruptions affecting a small number of applications on one of our shared ingress systems. The impact was limited and sporadic at this stage.

3 March - 22:35 UTC / 16:35 CST

Our monitoring systems raised alerts indicating elevated error rates. Our engineering team began investigating the issue as a possible concern for more than just the one application we had detected at 16:15 CST. Our first customer report came in at 22:40 UTC.

3 March - 23:13 UTC / 17:13 CST

We determined the issue was not isolated to individual applications but was a system-level ingress configuration problem. We declared an incident and began working on determining the root cause.

3 March - 23:32 UTC / 17:32 CST

The root cause was identified. Invalid configurations were being applied to the ingress systems in an intermittently destructive manner. We began to investigate the best solution to resolve the problem with the least amount of additional disruption.

3 March - 23:46 UTC / 17:46 CST

It was determined the best fix would require each of our ingress systems to be forcibly restarted. Although this would result in guaranteed downtime for applications, it was deemed the safest way to recover fully. 

We began applying the fix to the ingress system. This required a rolling restart of the ingress infrastructure, which temporarily caused a broader disruption. During this window, most applications experienced intermittent SSL errors and 503 responses. The average downtime per application during this phase was approximately 9 minutes.

4 March - 00:04 UTC / 18:04 CST

The fix was fully applied and all services were restored. We continued to monitor the system to ensure stability.

After resolution

We have also added additional verifications in our configuration distribution to prevent application of errant configurations.

We are implementing additional isolation measures for our ingress configuration distribution to prevent similar issues in the future.

Applications in the US-Central-1 region experienced intermittent 502 responses, TLS/certificate errors, and temporary unavailability of the Gigalixir console.

The root cause was a configuration distribution issue within our ingress system that caused certificate and routing disruptions across the region. Our engineering team identified and resolved the issue, and all services have been restored.

We are implementing changes to further isolate ingress configurations and prevent similar issues in the future. We will continue monitoring to ensure stability.

The earlier issues affecting applications in the US-Central-1 region, including intermittent 502 errors, TLS/certificate errors, and temporary unavailability of the Gigalixir console (https://console.gigalixir.com), appear to be resolved.

Services have recovered and applications should be operating normally. We are continuing to monitor the system to ensure stability.

Investigating: Some applications in the US-Central-1 region are experiencing intermittent failures.

This includes elevated 502 responses, occasional TLS/certificate errors, and the Gigalixir console (https://console.gigalixir.com) currently being unavailable.

Our engineering team is actively investigating the issue and working on mitigation. Updates will be posted here as we learn more.

We are continuing to work on a fix for this incident.

This incident has been resolved. We'll continue to monitor, please reach out if you have questions.

We continue to monitor, please reach out to support@gigalixir.com if you are affected and need assistance.

We are continuing to monitor GitHub's status for their expected timeline to fix their incident.