In the IBM TRIRIGA Administrator Console, under Performance Monitor > Monitor a Single Value, several monitors are no longer listed.
The following changes were made in the Administrator Console, under Performance Monitor > Monitor a Single Value:
- CRYSTAL_QUEUE_AGENT_RUNNING is renamed to REPORT_QUEUE_AGENT.
- CLEANUP_AGENT_RUNNING is renamed to PLATFORM_MAINTENANCE_SCHEDULER.
- KEYWORD_PARSER_AGENT_RUNNING and WAREHOUSE_AGENT_RUNNING were deprecated in a previous release and are now removed from the Performance Monitor.
Since the CRYSTAL_QUEUE_AGENT_RUNNING and CLEANUP_AGENT_RUNNING monitors are used for integration, only the performance monitor names are changed, not the values or URLs.
[Admin: This post is related to the 07.14.15 post about renaming CleanupAgent (Cleanup Agent) to PlatformMaintenanceScheduler (Platform Maintenance Scheduler). To see other related posts, use the Monitoring tag.]
We have quite a lot of rows in the extended formula queue that started to back up since Aug 29. At one point, we had over 900K in the queue and the process server is processing it pretty slowly. As of Sep 8, the system is still processing the queue. We are adding daily on average over 125K rows to the EF_QUEUE table.
We tried starting and stopping the Extended Formula Agent and Formula Recalc Agent and tried rebooting the process servers. We swapped the agents to run on different servers. We also ran the segment shrink on the EF_QUEUE table and gathered statistics. It is still extremely slow. We still have over 500K in the queue.
In parallel, we did open the platform logging on the extended formula activity and cache building, and we are working to identify the processes that can be fixed to reduce the multiple calls to the Extended Formula Agent queue. Do you have any recommendations to speed up the process to clear the extended formula queue?
Review the following two APARs. They may need to be applied to your environment to address this issue: (1) Platform: APAR IV87030. (2) Application: APAR IV87104.
[Admin: This post is related to the 07.28.16 post and 05.04.16 post about the Extended Formula (EF) Agent not processing and backing up the queue. To see other related posts, use the Extended Formula tag.]
In a clean environment, when creating the second or third facility assessment analysis with a large number of years in the analysis (e.g. 25), the component renewal generation begins to spin infinitely, generating too many component renewals and jamming the workflow queue.
We needed to improve the performance of the facility assessment analysis by making the process asynchronous, as follows…
[Admin: The same article is also posted in the IBM Support Portal as a technote.]
A maximum threads error followed by a server crash can happen in some cases, where there is a large number of asynchronous workflows having trigger tasks in the WF_EVENT queue. This problem was first seen with Reserve, with the server crashing soon after starting due to triStartWakeUp and triEndWakeUp tasks. The problem described in this APAR is not specific to Reserve.
[Admin: This post is related to the 07.29.16 post about the recommended maximum threads, and the 08.24.15 post about using a thread “lock” in the workflow.]
The Extended Formula (EF) Agent stopped processing the queue.
We need to limit the rows in EF_QUEUE (to reset AGENT_ID to -1) to only rows that have an existing valid AGENT_ID value during server startup. The AGENT_ID only gets populated with the ID of the EF Agent that picked it for processing. For whatever reason, the row may not have completed its process, but the server has already shutdown, thus having rows flagged with a non-existing EF Agent. In order for the EF Agent to continue processing these rows, their AGENT_ID values need to be reset to -1, so that the next time the agent awakens, they will be picked up for processing. Refactor the existing code, so it can be more JUnit tested, and is using SQL code, instead of a direct database connection.
Moving forward, the application server will now start up, even if there is a large number of rows in the Extended Formula queue.