Transform
With great power comes great responsibility
As flow receives events from it's input, you can use the Transform section to transform whichever events you choose before they are loaded to the output.
There are several scenarios where transforming events on the fly with the can be useful:
Cleansing values that are represented differently in a database; for example changing values of 0 or 1 to false or true, or rounding floats to integers.
Enriching data with the results of look-ups from other sources, such as Geo-IP resolution or currency conversion.
Discarding undesired events.
Splitting up a single event into multiple events across different database tables, in order to make the data more modular and easily query-able.
Generating notifications.
Gathering metrics on incoming data.
Scrubbing PII (Personally Identifiable Information).
Publish Transformation
When you create a new flow, the Publish Transformation checkbox is checked by default, which will cause all events to skip the transformation. If you choose to do some transformation, make sure to uncheck the Publish Transformation checkbox while creating a new flow
When you create a flow with "Publish Transformation" unchecked, the Transform stage of the flow will be grayed out as shown below in the flow Dashboard view. Click on the Transform box to perform the transformation and publish for the events to flow thru'
Using the Code Editor
Use the embedded code editor in the browser to write some Groovy script or standard Java code. The predefined variable data hold the input event data and the variable metadata hold the metadata of the current event.
Testing Your Transformation
You can test your transformation for any syntax issue by clicking on the Test button. The sample message displayed will be used to evaluate your transformation script and the results will be display on the bottom right section as shown in the above screenshot. Once you are comfortable with your transformations, you can publish them to take effect. The published transformations will be applied on all new events coming from that point onward.
Discarding Events
To conditionally discard an event, just set the attribute data.$exclude to Y or true (Boolean or String "true"). All events that are discarded in transform will be moved to io-ignored topic.
Topics
Transform will be performed on all events from io-input topic and will be published into io-transformed topic after successful transformation. Any event discarded during transform will be published into io-ignored topic. All the events that are erred out during transformation will be published into io-error topic with errorType metadata attribute populated with Transform
Last updated
Was this helpful?