You can configure TRIRIGA to use Tivoli Directory Integrator as its ETL runtime engine to run ETLJobItems from within TRIRIGA.
Before you begin
Install Tivoli Directory Integrator, if not already installed, on all the TRIRIGA systems that could run a TDI ETL Job Item. During the TDI install:
- Make note of the installation directory you enter on the Destination panel. You will enter this value later in TRIRIGAWEB.properties.
- Select either installation type. TRIRIGA requires only the TDI Server component.
- When prompted for the location of the Solution Directory, you can select any option. TRIRIGA specifies its own solution directory at runtime. However selecting the option “Use Install Directory” may simplify troubleshooting.
- Make note of the value you enter in the Server Port field on the Server Port Values Panel. You will enter this value later in TRIRIGAWEB.properties.
- Clear the “Start the Configuration Editor” check box on the Install Complete panel.
- Note: This step is very important for TDI/TRIRIGA integration to work. After you have installed Tivoli Directory Integrator, update it with the recommended fix packs (per TRIRIGA support matrix). TDI must be at least at FP04 (126.96.36.199) or it will not automatically start the TririgiaETLDispatch.xml assembly line which will result in ETL job items failing to run successfully.
- Edit TRIRIGAWEB.properties file to enable TRIRIGA to manage TDI server. Set the following properties…
- Install a JDBC driver library so that Tivoli Directory Integrator can use it to access TRIRIGA database…
- Edit TDI global.properties file to allow TRIRIGA to check and stop the TDI server from localhost without requiring authentication and authorization certificates. Set the api.remote.ssl.on property to false to tell TDI to trust requests from localhost…
- Start Tivoli Directory Integrator Agent from TRIRIGA Admin Console and verify that it starts successfully…
[Admin: This post is related to the 08.03.16 post about installing, upgrading, or uninstalling TRIRIGA TDI, and the 05.01.16 post about documentation on developing TDI with TRIRIGA. To see other related posts, use the TDI tag.]
When trying to use Tivoli Directory Integrator (TDI) with TRIRIGA, an error message is being thrown to the system.out file that says that you don’t have a valid license to start TDI, and that the agent requires a TRIRIGA Metrics Technology License. The documentation states that the license includes TDI, but the system thinks otherwise.
The documentation implies that TDI works with some licenses, but there are only certain licenses that TDI actually works with. Moving forward, the documentation for TDI will be updated with the exact licenses that provide access to the functionality.
[Admin: This post is related to the 08.03.16 post about installing, upgrading, and uninstalling TDI.]
IBM TRIRIGA platform version 3.4.1 and later provides integration with Tivoli Directory Integrator (TDI). TRIRIGA TDI integration offers support for running TDI-based ETL Job Items from within TRIRIGA. This wiki describes how to install the TDI feature of TRIRIGA. It also provides information about upgrading TRIRIGA when the TDI feature is installed and about uninstalling TRIRIGA TDI.
Installing TRIRIGA TDI
TDI is installed and configured by the TRIRIGA installer. The version of TDI installed is as follows…
Check the following install logs to verify that TDI installed successfully and to troubleshoot problems with TDI installation…
Upgrading TRIRIGA with TDI Installed
Before beginning a TRIRIGA upgrade: 1. Uninstall TDI (see Uninstalling TDI below)…
During a TRIRIGA upgrade, uninstall TDI before you rename the directory in which TRIRIGA is installed…
[Admin: This post is related to the 05.01.16 post about documentation on developing TDI with TRIRIGA. To see other related posts, use the TDI tag.]
After redeploying the TRIRIGA application by using ./ant install-was, our customer can no longer process an ETL job item with Tivoli Directory Integrator (TDI). In the log, we have found the following warning:
2016-06-29 16:30:16,035 WARN [com.tririga.platform.workflow.runtime.taskhandler.WFTaskHandlerBase](WFA:127701296 - 127598391 triProcessManual:309624523 IE=309624523) Workflow task handler error while Calling custom task class: com.tririga.platform.workflow.runtime.taskhandler.ETL.RunETL. TaskStep: Custom Task(39) WFTID=19192848.7 TSID=204231 Label='Call Kettle Transformation API' FormulaRecalc='Recalculate as Needed' EventAction='' StepInstance: WFIID=375605519425576 SID=204231 UserEvent=' ' SO=127598391 Results=1 Sum=0 Status='ACTIVE'. Cause: com.tririga.platform.util.locator.LocatorException: Could not find bean of type: class com.tririga.platform.workflow.runtime.taskhandler.ETL.TDIRequest[MID-3376369984]...
The bean “class com.tririga.platform.workflow.runtime.taskhandler.ETL.TDIRequest” (which is used by the workflow custom task) has not been loaded correctly. I’m thinking maybe some classpath settings are missing after “/ant install-was”. Have you seen this kind of issue?
Our customer wants to install multiple TRIRIGA environments on the same server. If I’ve understood correctly, we should not share the same Tivoli Directory Integrator (TDI) instance between different TRIRIGA environments. So we have to install one TDI instance (program and solution directory) per TRIRIGA environment… Is that correct? What’s the best practice for our requirement? Any suggestion will be appreciated!
Each TRIRIGA server that can run ETL Job Items must install TDI which will create a TDI_IE directory and TDISolDir directory under TRIRIGA Home (as you have indicated in your diagram)… Yes, you can install multiple TDI instances on the same physical server. We have this setup in our lab. We have multiple TRIRIGA installations, each with TDI installed, that are running on the same box. In this type of setup, make sure that the ports you specify in the “IBM Tivoli Directory Integrator Ports” installation panel are unique and not in use.
[Admin: This post is related to the 06.13.16 post and 02.16.16 post about multiple TRIRIGA instances on the same physical server, and the 05.01.16 post about developing Tivoli Directory Integrator (TDI) with TRIRIGA.]
IBM TRIRIGA uses either the Tivoli Directory Integrator (TDI) ETL development environment Configuration Editor, or the Pentaho ETL development environment Spoon to generate transform XML files. These transforms, when run through the API, move data from source to destination tables. In IBM TRIRIGA, ETL scripts are used to periodically take snapshots of the transactional data. These snapshots are time stamped and stored in metric fact tables. Metric queries use the information in the metric fact tables to visually display the information in metric charts.
I’m currently working on an interface which will load the users and organizations into TRIRIGA from an external system. The data loading work will be based on flat files, since the source system can’t provide the file with a format served by TRIRIGA. I plan to use TDI to consume the incoming file, transform it, and then write the transformed data into staging tables in TRIRIGA (I’m using DataConnect).
Since the TDI script will be invoked by TRIRIGA, there must be some point where TDI development will be different from a classic TDI development, such as how we should get the JDBC connection with the parameters (JDBC URL, username, password, etc.) passed by TRIRIGA. In the IBM Knowledge Center, most of the articles on TDI are aimed to calculate the metric table. Is there any documentation that introduces the TDI development under TRIRIGA in a general way? Thanks in advance for your help!
How to write TDI-based ETLs for TRIRIGA is documented in the Application Building for Performance Framework [PDF] user guide. Also, in the IBM Knowledge Center, check out Performance Framework > Data structures > ETL integration > Defining and Maintaining ETL transforms > Using ETLs with IBM Tivoli Directory Integrator Configuration Editor.