Fire Insights now has Pipelines

Overview

Fire Insights now provides building and executing Pipelines. Pipelines consist of workflows connected to each other.

The framework is flexible and would support more nodes types to be part of the Pipeline in the coming weeks.

Pipelines in Fire Insights is currently integrated with Apache Airflow. For execution, the pipeline is submitted to Airflow.

More Details of Pipelines in Fire Insights is available here:

• http://docs.sparkflows.io/en/latest/installation-upgrading/configuring-pipeline.html

• http://docs.sparkflows.io/en/latest/user-guide/pipelines.html

Configuring Pipelines

Fire uses Apache Airflow for executing Pipelines. Hence Airflow has to be installed on the same machine as Fire.

Below are the configurations needed in Fire.

Creating a Pipeline

Each Application now allows creating Pipelines.

Below is an example Pipeline with 3 Workflows.

Executing a Pipeline

Pipelines can be executed like workflows. When a Pipeline is executed, its execution is submitted to Airflow.

The Pipeline tab displays the list of Pipelines for the current Application.

Clicking on the Execute Action icon opens the Pipeline Execute Page.

Pipeline Execution

Once a Pipeline is fired, its details are visible in Pipeline Executions.

Workflow Executions

The Pipeline Details page displays the list of workflow executions associated with the Pipeline.