Replacing tWriteJSONField with a Routine to Fix Threading Problem

Talend's tWriteJSONField component uses Java Threads, presumably to improve performance.  However, this can throw a Main Flow into havoc when using globalMap.  It's easy to form JSON, so I am replacing the standard component with a Routine and a tMap.

tWriteJSONField is using a Java Thread as shown in the screenshot from TOS 5.6.1.

A Component that Kicks Off a Thread

The problem with using a Thread is that I sometimes count on a Main Flow to be a single loop.  For example, I may use a tSetGlobalVar to record an identifier early in a Main Flow, work with components like tRESTClient during the Main Flow, then unpack the global variable in a later component like a tMSSqlOutput.  This scheme works fine when there is one thread of control, but introducing another Java Thread means that I don't have assurance of what I'm unpacking as a global variable downstream.

So, because it's easy to form JSON, I create a Talend Routine to accept several fields to form my JSON String.  Here is such a Routine.

package routines;

public class FieldsToJSON {

public static String fieldsToJSON(String field1, String field2, String field3) {
  String s = "{";

  s += "\"field1\": \"" + field1 + "\", ";
  s += "\"field2\": \"" + field2 + "\", ";
  s += "\"field3\": \"" + field3 + "\"";

  s += "}";

  return s;
 }
}
 field1, field2, and field3 correspond to a schema.  While it's not ideal to step out of the standard schema mechanism of Talend (a later schema update won't propagate to this Java code), it's easy to test this code.  The fieldsToJSON method can be written generally if you know how to use Java Reflection.

This Talend Routine fits cleanly into a tMap.  To see this in practice, here is a job that applies the Routine in a tMap.

Job that Applies a Custom JSON Routine to an Input

The tFixedFlowInput generates records for a 3-field schema.  The fields are "field1", "field2", and "field3".



Input Data and Schema from a tFixedFlowInput

The tMap then carries through the input schema and adds a JSON derived field which is produced from the Talend Routine described earlier.

tMap Applying a JSON Serializer Routine

The results, echoed by a tLogRow, show the generated fields plus the derived jsonBody field produced by the Talend Routine.

A JSON Field Added to the Input Schema

In addition to the threading problem I found using tWriteJSONField, there is a schema mismatch I was getting when entering a tWriteJSONField with components like tSetGlobalVar or tGroovy.  tWriteJSONField seemed to reference the Struct of the tSetGlobalVar which was based on the preceding schema (leading to a "Type mismatch").  If you encounter either of these problems, try a Talend Routine.