ETL Transformers

Transformer components are executed in a pipeline. They work with the received input and return output.

Before the execution, the $input variable is always assigned, so you can get at run-time and use if needed.

Available Transformers

CSV Deprecated

DEPRECATED : since version 2.1.4 please use the new csv extractor instead Converts a String in a Document parsing it as CSV.

Component description.

• Component name: csv
• Supported inputs types: [String]
• Output: ODocument

Syntax

Example

Transforms a row in CSV (as ODocument), using comma as separator, considering "NULL" as null value and skipping the rows 2-4:

{ "csv": { "separator": ",", "nullValue": "NULL",
           "skipFrom": 1, "skipTo": 3 } }

FIELD

Execute a SQL transformation against a field.

Component description.

• Component name: vertex
• Supported inputs types: [ODocument]
• Output: ODocument

Syntax

Examples

Transform the field 'class' into the ODocument's class by prefixing it with '_':

{ "field": { "fieldName": "@class", "expression": "class.prefix('_')"} }

Apply the class name based on the value of another field:

{ "field": { "fieldName": "@class", "expression": "if( ( fileCount >= 0 ), 'D', 'F')"} }

Assign the last part of a path to the "name" field:

{ "field": { "fieldName": "name",
      "expression": "path.substring( eval( '$current.path.lastIndexOf(\"/\") + 1') )" }

Assign a fixed value:

{ "field": { "fieldName": "counter", "value": 0} }

Rename a field from 'salary' to 'remuneration':

{ "field": { "fieldName": "remuneration", "expression": "salary"} },
{ "field": { "fieldName": "salary", "operation": "remove"} }

Rename multiple fields in one call (As of v2.1):

{ "field": { "fieldNames": ["remuneration","salary"], "operation": "remove"} }

MERGE

Merges input ODocument with another one, loaded by a lookup. Lookup can be a lookup against an index or a SELECT query.

Component description.

• Component name: merge
• Supported inputs types: [ODocument, OrientVertex]
• Output: ODocument

Syntax

Example

Merge current record against the record returned by the lookup on index "V.URI" with the value contained in the field "URI" of the input's document:

{ "merge": { "joinFieldName":"URI", "lookup":"V.URI" } }

VERTEX

Transform a ODocument in a OrientVertex.

Component description.

• Component name: vertex
• Supported inputs types: [ODocument, OrientVertex]
• Output: OrientVertex

Syntax

Example

Transform the ODocument in a Vertex setting as class the value of "$className" variable:

{ "vertex": { "class": "$className", "skipDuplicates": true } }

EDGE

Transform a JOIN value in one or more EDGEs between current vertex and all the vertices returned by the lookup. Lookup can be a lookup against an index or a SELECT query.

Component description.

• Component name: EDGE
• Supported inputs types: [ODocument, OrientVertex]
• Output: OrientVertex

Syntax

Example 1

Create an EDGE from the current vertex, with class "Parent", to all the vertices returned by the lookup on "D.inode" index with the value contained in the field "inode_parent" of the input's vertex:

{ "edge": { "class": "Parent", "joinFieldName": "inode_parent",
            "lookup":"D.inode", "unresolvedLinkAction":"CREATE"} }

Example 2: single line CSV contains both vertices and edge

{
    "source": {
        "content": {
            "value": "id,name,surname,friendSince,friendId,friendName,friendSurname\n0,Jay,Miner,1996,1,Luca,Garulli"
        }
    },
    "extractor": {
        "row": {}
    },
    "transformers": [
        {
            "csv": {}
        },
        {
            "vertex": {
                "class": "V1"
            }
        },
        {
            "edge": {
                "unresolvedLinkAction": "CREATE",
                "class": "Friend",
                "joinFieldName": "friendId",
                "lookup": "V2.fid",
                "targetVertexFields": {
                    "name": "${input.friendName}",
                    "surname": "${input.friendSurname}"
                },
                "edgeFields": {
                    "since": "${input.friendSince}"
                }
            }
        },
        {
            "field": {
                "fieldNames": [
                    "friendSince",
                    "friendId",
                    "friendName",
                    "friendSurname"
                ],
                "operation": "remove"
            }
        }
    ],
    "loader": {
        "orientdb": {
            "dbURL": "memory:ETLBaseTest",
            "dbType": "graph",
            "useLightweightEdges": false
        }
    }
}

FLOW

Control the pipeline flow. Supported operations are "skip" and "halt". This transformer usually operates with the "if" attribute.

Component description.

• Component name: flow
• Supported inputs types: Any
• Output: same type as input

Syntax

Example

Skip the current record if name is null:

{ "flow": { "if": "name is null", "operation" : "skip" } }

CODE

Executes a snippet of code in any of the JVM supported languages. Default is Javascript. Last object in the code is returned as output. In the execution context, the following variables are bound:

• input with the input object received
• record with the record extracted from input object, when is possible. In case the input object is a Vertex/Edge, the underlying ODocument is assigned to the variable

Component description.

• Component name: code
• Supported inputs types: [Object]
• Output: Object

Syntax

Example

Displays current record and returns the parent.

{ "code": { "language": "Javascript",
            "code": "print('Current record: ' + record); record.field('parent');"}
}

LINK

Transform a JOIN value into a LINK in the current record, with the result of the lookup. Lookup can be a lookup against an index or a SELECT query.

Component description.

• Component name: link
• Supported inputs types: [ODocument, OrientVertex]
• Output: ODocument

Syntax

Example

Transform a JOIN value into a LINK in the current record (set as "parent" of type LINK) with the result of the lookup on index "D.inode" with the value contained in the field "inode_parent" of the input's document:

{ "link": { "linkFieldName": "parent", "linkFieldType": "LINK",
            "joinFieldName": "inode_parent", "lookup":"D.inode", "unresolvedLinkAction":"CREATE"} }

LOG

Logs the input object to System.out.

Component description.

• Component name: log
• Supported inputs types: Any
• Output: Any

Syntax

Example

Simply log current value:

{ "log": {} }

Log current value with "-> " as prefix:

{ "log": { "prefix" : "-> "} }

Block

Executes a Block as a transformation step.

Component description.

• Component name: block
• Supported inputs types: [Any]
• Output: Any

Syntax

Example

Simply log current value:

{ "block": {
    "let": {
      "name": "id",
      "value": "={eval('$input.amount * 2')}"
    }
  }
}

Command

Executes a command.

Component description.

• Component name: command
• Supported inputs types: [ODocument]
• Output: ODocument

Syntax

Example

{
  "command" : {
    "command" : "select from E where id = ${edgeid}",
    "output" : "edge"
  }
}