Customize

Summary

This plugin provides users the ability to create/delete columns and extract data from certain raw layer tables. The columns created with this plugin must be start with the prefix x_

NOTE: All columns created by this plugin are of the datatype VARCHAR(255)

Sample Request

To extract data, switch to Advanced Mode on the first step of creating a Blueprint and paste a JSON config as the following:

The example below demonstrates how to extract status name from the table _raw_jira_api_issues and assign it to the x_test column of the table issues. We leverage the package https://github.com/tidwall/gjson to extract value from the JSON. For the extraction syntax, please refer to this docs

• table: domain layer table name
• rawDataTable: raw layer table, from which we extract values by json path
• rawDataParams: the filter to select records from the raw layer table (The value should be a string not an object)
• mapping the extraction rule; the key is the extension field name; the value is json path

[
  [
    {
      "plugin":"customize",
      "options":{
        "transformationRules":[
          {
            "table":"issues", 
            "rawDataTable":"_raw_jira_api_issues", 
            "rawDataParams":"{\"ConnectionId\":1,\"BoardId\":8}", 
            "mapping":{
              "x_test":"fields.status.name" 
            }
          }
        ]
      }
    }
  ]
]

You can also trigger data extraction by making a POST request to /pipelines.

curl 'http://localhost:8080/pipelines' \
--header 'Content-Type: application/json' \
--data-raw '
{
    "name": "extract fields",
    "plan": [
        [
            {
                "plugin": "customize",
                "options": {
                    "transformationRules": [
                        {
                            "table": "issues",
                            "rawDataTable": "_raw_jira_api_issues",
                            "rawDataParams": "{\"ConnectionId\":1,\"BoardId\":8}",
                            "mapping": {
                                "x_test": "fields.status.name"
                            }
                        }
                    ]
                }
            }
        ]
    ]
}
'

Get all extension columns(start with x_) of the table issues

GET /plugins/customize/issues/fields

response

[
    {
        "columnName": "x_test",
        "columnType": "VARCHAR(255)"
    }
]

Create extension column x_test for the table issues

POST /plugins/customize/issues/fields

{
    "name": "x_test"
}

Drop the column x_text for the table issues

DELETE /plugins/customize/issues/fields/x_test