Send alert when Sitecore running jobs get stuck

Recently we faced an issue in production where a Sitecore job hung. It blocked other jobs, and all our search index rebuild jobs were stuck in the queue. End users were not getting a good search experience, and our latest published content was not showing on the live site. It doesn’t happen often: twice in the last six months. After a lot of searching, we found that recycling the application pool is the only fix. Sitecore Support told us there is currently no way to kill individual jobs, and they are still analysing dumps and logs to find the root cause. That leaves an open problem: how do we know when jobs get stuck? We can check running jobs on the “Jobs.aspx” page, but how do we know when to check? If we don’t respond quickly, it could crash the site.

We decided that, at the very least, we should get an alert when jobs get stuck. We need a scheduled task that runs at a set interval and checks for long-running jobs, then uses Sitecore EXM to send an email. At first, we planned to read the recipient list from a config file. Then we considered using the Sitecore List Manager: add the recipient emails there, read the list from the back end and send the emails. In the end, we realised we could simply create an email campaign in Sitecore EXM and add the List Manager list as its recipients.

First, create a list in the Sitecore List Manager and add the recipients to it:

Next, create an email, add the list in the recipients section and make a note of the item ID:

Then create a Command in Sitecore Tasks and enter the method to invoke in the type field:

Then create a schedule and set the interval. In the Items field, enter the email campaign ID from the previous step:

Now that the schedule and email are ready, we need to send the email from the back end based on how long jobs have been running. We will need “SendingManagerFactory” and “ExmCampaignService”.

 var sendingManagerFactory = ServiceLocator.ServiceProvider.GetService<ISendingManagerFactory>();
            var campaignService = ServiceLocator.ServiceProvider.GetService<IExmCampaignService>();

Then we read the email campaign using the item parameter passed in from the schedule.

 MessageItem emailCampaign = campaignService.GetMessageItem(itemArray.Length > 0 ? itemArray[0].ID.ToGuid() : new Guid());

Finally, we get the jobs from the “JobManager”, check whether any have been running for more than 20 minutes, and if so, send the email.

 var jobs = Sitecore.Jobs.JobManager.GetJobs();

            foreach (var job in jobs)
            {
                if (job.Status.State == Sitecore.Jobs.JobState.Running)
                {
                    var timeDiff = System.DateTime.UtcNow - job.QueueTime;
                    if (timeDiff.Minutes > 20)
                    {
                        ISendingManager sendingManager = sendingManagerFactory.GetAsyncSendingManager(emailCampaign);
                        sendingManager.SendMessage();

                        break;
                    }
                }
            }

Thanks
MH