Our team continued to monitor Pulse Monitoring through the weekend, and we see that the systems are now performing as expected and there have been no false alerts since the last incident on April 28, 5:35am UTC. This incident has been resolved.
Posted May 08, 2023 - 11:16 PDT
Monitoring
A fix was implemented by our team and we are not observing any false heartbeat alerts since April 28, 5:35am UTC. We are going to continue monitoring the performance.
Posted May 03, 2023 - 08:30 PDT
Identified
There were some false heartbeat alerts sent again today April 26, 2023 at 5:35 a.m UTC. Our team has identified one possible contention point that could be causing the issue and is working to remedy it.
Posted Apr 26, 2023 - 15:09 PDT
Update
At this point, investigations point out to networking issues that could have caused the data to be missed causing the false alerts. We are continuing to investigate the issue.
Posted Apr 25, 2023 - 05:04 PDT
Investigating
We have discovered false Heartbeat alerts for several deployments starting April 23, 11:35 am UTC. Our team is investigating the issue.
Our team has separate monitoring that is set up for all deployments for which they get alerted on if a node or deployment is unavailable. At this time, we checked all alerts triggered via Pulse and have confirmed that the deployments for which these alerts were triggered are all in a healthy state.
Posted Apr 24, 2023 - 18:42 PDT
This incident affected: SearchStax Cloud (Pulse Monitoring).