Gateway Service
Gateway Service disruption
- New cluster verification - completed.
- Server import/restoration - completed.
- Monitoring performance - active.
iMonnit.com and Monnit.com have returned to normal operation. Device communication is actively being restored. Please note: while devices are reconnecting, it may take additional time for historical data to fully appear in the portal. We’re continuing to monitor system performance to ensure full recovery.
Issue Summary:
The Hyper-V cluster experienced VLAN corruption, resulting in inconsistent server-to-server communication across hosts. Initial remediation efforts, which involved relocating services to specific hosts, unintentionally propagated the corruption. This led to widespread connectivity failures between servers, the Internet, and backend databases, ultimately impacting client-facing services.
Current Status:
- A new Hyper-V cluster is being built from scratch to ensure a clean and stable environment.
- The rebuild is in the verification phase, with infrastructure integrity and connectivity being actively tested.
- Server imports and service restoration will begin once the new cluster is validated.
Next Steps: - Complete verification of the new cluster.
- Begin phased import of servers and restoration of services.
- Monitor performance and stability post-restoration.
Estimated Time to Resolution: - No definitive timeline available yet. Updates will be provided as progress continues.
Systems offline running validation tests.
Services offline while we’re working on them.
Communication restored, ongoing investigations into gateway service stability continue
Working with the networking admins to resolve communication errors in the network
We are working on getting gateway services stabilized.