Transform
With great power comes great responsibility
Last updated
Was this helpful?
With great power comes great responsibility
Last updated
Was this helpful?
As flow receives events from it's input, you can use the Transform section to transform whichever events you choose before they are loaded to the output.
There are several scenarios where transforming events on the fly with the can be useful:
Cleansing values that are represented differently in a database; for example changing values of 0 or 1 to false or true, or rounding floats to integers.
Enriching data with the results of look-ups from other sources, such as Geo-IP resolution or currency conversion.
Discarding undesired events.
Splitting up a single event into multiple events across different database tables, in order to make the data more modular and easily query-able.
Generating notifications.
Gathering metrics on incoming data.
Scrubbing PII (Personally Identifiable Information).
When you create a new flow, the Publish Transformation checkbox is checked by default, which will cause all events to skip the transformation. If you choose to do some transformation, make sure to uncheck the Publish Transformation checkbox while creating a new flow
When you create a flow with "Publish Transformation" unchecked, the Transform stage of the flow will be grayed out as shown below in the flow Dashboard view. Click on the Transform box to perform the transformation and publish for the events to flow thru'
Use the embedded code editor in the browser to write some Groovy script or standard Java code. The predefined variable data hold the input event data and the variable metadata hold the metadata of the current event.
You can test your transformation for any syntax issue by clicking on the Test button. The sample message displayed will be used to evaluate your transformation script and the results will be display on the bottom right section as shown in the above screenshot. Once you are comfortable with your transformations, you can publish them to take effect. The published transformations will be applied on all new events coming from that point onward.
If you create the follow with Publish Transformation unchecked, then all the events that are pending transformation will be transformed soon after you publish for the first time.
To conditionally discard an event, just set the attribute data.$exclude to Y or true (Boolean or String "true"). All events that are discarded in transform will be moved to io-ignored topic.
Transform will be performed on all events from io-input topic and will be published into io-transformed topic after successful transformation. Any event discarded during transform will be published into io-ignored topic. All the events that are erred out during transformation will be published into io-error topic with errorType metadata attribute populated with Transform
