Collaborating on Talend Open Studio Routines: Part 3 - Source Code

This post describes integrating Java code from an externally-prepared build tool into Talend Open Studio as a Routine.

In previous posts, I described using the Create Routine function in Talend Open Studio (TOS) to add a custom class of static methods for use in TOS jobs.  I recommended an alternative process for developing routines in a team setting.  I advocated for extracting the Java code that would typically make up a Routine into a source code-managed location that is built with Maven into a JAR.  The JAR is then imported into TOS.

Separate Java Code from Talend Job

The recommended alternative Routine development process separates Java code from the Talend job. This separation allows me to manage the external Java file in a configuration management system such as Git.  Git lets me track the changes and resolve conflicts among a whole team working on the Routine.

In the following screenshot, a Java file BRules.java is modified to add a new method.   BRules.java is a Talend routine that contains static methods like "pad" which will pad a string with a certain number of padding characters.  Because BRules.java is managed in source control -- and not embedded in a favorite TOS project -- it's easily shared among others and there is good automated support for reconciling differences.

Git Managing Talend Routine Java Files

Maven

To separate the code from TOS, we need a build tool.  When the code is kept in Talend Open Studio (Create Routine), the code is compiled into a .class file by TOS.  Maven is a tool that can do a similar compilation and a lot more.  Maven can also package the code into a JAR, run unit tests, version the software, and manage dependencies.

Dependency management supports the creation of more valuable routines by providing a mechanism to add in third-party libraries.  Some build environments (Ant for example) will require the developer to round up compilation or runtime dependencies.

What's especially nice about Maven is the reliance on convention over configuration.  This means that I don't have to maintain trivial code.  Where something can be expressed as a default value, don't bother to specify that in the Maven build definition file, pom.xml.  Take the case of compilation.  Simply store your code in the src/main/java subfolder of a project and your code will be compiled.

BRules.java is in the src/main/java Subfolder which is the Default Compilation Location

The above screenshot shows the brules project which contains a class BRules.java.  BRules.java will be compiled into a .class file that can be used in TOS when Maven is invoked using a "mvn install" or other command.  The result is more than the compilation, it will also run the unit test stored in the default location of src/main/test, and package the class files in a JAR file based on the name of the project and the version used in the pom.xml.

All of this is done with a minimum of build-related code.

Once the Java code for a Routine is separated from TOS, a configuration management system like Git can manage the file.  However, TOS is no longer responsible for compiling the file, so an external build tool like Maven is required.  But this extra work is justified because the proliferation of different versions from different TOS projects is stemmed and the repeated building of the code can be automated so that each participating developer can have their work checked after they have added code.