Comparing 2 File Lists in Talend Open Studio

This post uses the add() and remove() Java Collections calls to the set difference of two lists of files.  Files from 2 directories are listed and one directory's files are removed from the other.  The result is a difference used in later processing.

java.util.List.add() and List.remove() can gather the results of a tFileList for later processing.  Pulling in a library like Commons Collections, these lists can be easily searched, sorted, transformed, and operated on with a variety of set-based operations (intersection, subtraction, etc.).

tFileList produces an iteration that can be converted to a flow using the tIterateToFlow.  The flow can then drive a tJavaRow.  The tIterateToFlow has the benefit of conveniently defining a flow schema.  The schema in this example is a single field (the full file path), but could be extended with additional columns for the other variables produced by tFileList.

Job Comparing 2 Directories' Files

tSetGlobalVar

The job starts with a tSetGlobalVar, listed as "Init Builder List".  You may have seen examples of setting variables of simple types like "FileName" set to "C:\data\file1.txt".  This job sets a variable that is a Java object which will be unpacked in the other components.

Use the following as the Key and Value of the tSetGlobalVar component.  There are no quotes around the 'new' statement.

"builderList" / new ArrayList<String>() 

tFileList

Both tFileLists -- "List csv files" and "List txt files" -- are standard.  They are searching for files of pattern "*.csv" and "*.txt".  (In a production job, these components should use context variables that parameterize the directory. )  

tIterateToFlow

The tIterateToFlow components connect the tFileList with custom Java code (tJavaRow).  The tIterateToFlow defined a single-field schema.  If additional file list information were needed, the schema could be expanded to include other tFileList variables.

tFlowToIterate Configuration

 Both tIterateToFlow components are similar except for the tFileList source.

tJavaRow

Two tJavaRows work with the list using the following algorithm.

1. Strip the suffix from a CSV file
2. Add the CSV file (minus the .csv) to the builderList
3. Strip the suffix from a TXT file
4. Remove the file name from builderList
5. The result is a list of suffix-less filenames 

Add to builder list

This component uses the following tJavaRow code

List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.add( row1.filename.substring(0, row1.filename.length() -4) );

In the Advanced settings tab, java.util.List is imported.  (There is a line that can be uncommented.)

Remove from builder list

The second tJavaRow component uses

List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.remove(row2.filename.substring(0, row2.filename.length()-4));
 
Remaining Work

The job ends with the execution of a tJava.  This tJava can deal with the build-up lists or could be replaced with another set of iteration components.

You can always integrate large blocks of Java code into Talend for some functionality that's missing off-the-shelf.  However, it's usually more maintainable to keep the jobs as standard as possible.  If you do choose to write Java code, try packaging the Java code in a routine or component rather than embedding chunks throughout the job.