Search

New Processors - Decision / JSON Parser / SortBy / Empty Dataset


A number of new Processors have been added this week to Fire. They include:

  • JSON Parser Processor

  • Decision Processor

  • SortBy Processor

  • Empty Dataset Processor

JSON Parser Processor

The JSON Parser Processor allows parsing the JSON content in a column of the DataFrame.

Suppose we read in a file which has JSON string as the content of one of the columns. We would want to parse the JSON string into new columns for further processing.



Decision Processor

The Decision Processor allows specifying a condition. Based on the condition, processing stops or proceeds further along the workflow.

An example is if there are no records in the Dataset, do not proceed with further processing. The next stage of processing might be a HIVE QL node which fires a query. And we do not want that query to be fired if there are no incoming records in the Dataset.


SortBy Processor

The SortBy Processor allows sorting the DataFrame by various columns. Users can specify a number of columns for sorting, and for each column they can specify the sorting order too.



Empty Dataset Processor

There are times when we need an empty dataset. One scenario we ran into was that the user needed to run a HIVE QL. The HIVE QL Processor needs input from the previous Processor and it passes it along as it to the next Processor after executing the user provided HIVE QL.

Since the HIVE QL Processor has to receive input from a previous Processor, Empty Dataset Processor comes handy in this scenario.



48 views

RESOURCES

SOCIAL

  • facebook
  • linkedin
  • twitter
  • angellist
© 2020 Sparkflows, Inc. All rights reserved. 

Terms and Conditions