Source: https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/
Documentation version: v5.3.2

# Data import configurations

This topic introduces the configuration file for data import in NebulaGraph Exchange.

## Configuration file structure

The structure of the NebulaGraph Exchange configuration file is as follows:

````
{
    taskType: <task_type>

    nebula: {}  

    error: {}

    checkPoint: {}

    sourcesCommon: {}

    writeCommon: {}

    sources: []
}

````

For different data sources, the structure of the NebulaGraph Exchange configuration file is the same. The main differences are in the sections `sourcesCommon` and `sources`. Configuration file templates for different data sources are provided in the directory `nebula-graph-exchange/template/` of the NebulaGraph Exchange package.

As you can see, the configuration file consists of the following sections:

- `taskType`: Specify the task type. For more information, see [Task type configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#task_type_configurations).
- `nebula`: Specify the connection configurations for NebulaGraph Database Developer Edition. For more information, see [NebulaGraph Database Developer Edition configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#database_configurations).
- `error`: Specify how to handle dirty records and failed records during import. For more information, see [Error configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#error_configurations).
- `checkPoint`: Specify the configurations for checkpointing. For more information, see [Checkpoint file configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#checkpoint_file_configurations).
- `sourcesCommon`: Specify the common configurations for the data source. For more information, see [Common source configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#common_source_configurations).
- `writeCommon`: Specify the common configurations for the write operation. For more information, see [Common write configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#common_write_configurations).
- `sources`: Specify the connection information and configurations specific to the data source, how to preprocess the source data, and how to map the source data to graph elements in NebulaGraph Database Developer Edition. For more information, see [Source configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/).

## Task type configurations

 | Option | Description | Type | Required | Default value
 | `taskType` | The task type. Valid values:

- `import`: Import data.
- `check`: Check the validity of the configuration file but do not import data. Check results are recorded in the log file `logs/import.log` in the `nebula-graph-exchange/` directory or in the directory specified by `-log`.
- `gqlTemplate`: Import data to NebulaGraph Database Developer Edition by using custom GQL templates. For more information, see [Import data with GQL templates](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/import-data/template-import/).
- `dryRun`: Perform a dry run of the import process. During a dry run, NebulaGraph Exchange reads the source data and counts the records to be imported, but does not import data into NebulaGraph Database Developer Edition.
- `reload`: Reload the GQL statements that failed to execute. Each time a data import task is completed, a unique ID named `app id` is returned. You can specify `app id` to reload statements that failed in the previous run. You can also check failed data in the path specified by [`error.path`](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#error_configurations).
- `gql`: Generate a file containing GQL statements based on up to 200 source records. The statements are generated based on [Common write configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/#common_write_configurations) and stored in `/tmp/gql/`, where each statement represents one write request.

If the option `-check`, `-dry`, or `-reload` is specified in the [data import command](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/import-data/#step_3_import_data) and conflicts with the task type configured here, the option in the command takes precedence. For example, if the import command specifies `-dry` while `taskType` is set to `check`, a dry run is performed. | String | Yes | `import`

## NebulaGraph Database Developer Edition configurations

This section describes how to configure NebulaGraph Database Developer Edition in NebulaGraph Exchange.

### Sample configuration

Replace `<current-password>` with the current password for the specified account. If the password has changed, use the new password.

````
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    enableEncryptPasswd: false
    passwd: "<current-password>"
    schema: "/default_schema"
    timezone: "Asia/Shanghai"
    dateFormat: "%Y-%m-%d"
    localDatetimeFormat: "%Y-%m-%dT%H:%M:%S"
    zonedDatetimeFormat: "%Y-%m-%dT%H:%M:%S %z"
    localTimeFormat: "%H:%M:%S"
    zonedTimeFormat: "%H:%M:%S %z"
    connectTimeoutMs: 6000
    requestTimeoutMs: 6000
    enableDataCheck: false
    retryTimes: 3
    retryIntervalMs: 0    
    tls: {
        enableTls: false
        disableVerifyServerCert: false
        ca: "/tmp/ca.pem"
        cert: "/tmp/cert.crt"
        key: "/tmp/key"
    }    
    preHooks:[
        """ CREATE GRAPH TYPE IF NOT EXISTS movie_type AS {
                NODE Actor (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}),
                NODE Director (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}),
                NODE `User` (LABEL Person {id INT64 PRIMARY KEY}),
                NODE Movie (LABEL Movie {id INT64 PRIMARY KEY, name STRING}),
                NODE Genre (LABEL Genre {id INT64 PRIMARY KEY, name STRING}),
                EDGE Act (Actor)-[:Act]->(Movie),
                EDGE Direct (Director)-[:Direct]->(Movie),
                EDGE Watch (`User`)-[:Watch {rate FLOAT}]->(Movie),
                EDGE WithGenre (Movie)-[:WithGenre]->(Genre)
        }""",
        "CREATE GRAPH IF NOT EXISTS movie TYPED movie_type"
        ]
  }

````

### Configuration options

 | Option | Description | Type | Required | Default value
 | `graphAddr` | The IPs and ports of Graph services. Separate multiple addresses with commas. | String | Yes | N/A
 | `graphName` | The name of the graph to which data is written. | String | Yes | N/A
 | `user` | The username of the Graph service. | String | No | `"root"`
 | `enableEncryptPasswd` | Whether to use the encrypted password. | Bool | No | N/A
 | `passwd` | The current password for the account used to connect. If `enableEncryptPasswd` is set to `true`, `passwd` must be set to the encrypted password. For more information, see [Encrypt passwords](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/encrypt/). | String | Yes | N/A
 | `schema` | The schema to which data is written. | String | No | `null`, which indicates the default schema in the path `/default_schema` is used.
 | `timezone` | The time zone to be set in the session of the Graph service. | String | No | `null`, which indicates the time zone on the server that runs the Graph service is used.
 | `dateFormat` | The format of the date to be set in the session of the Graph service. | String | No | `null`, which indicates the default format `%Y-%m-%d` is used.
 | `localDatetimeFormat` | The format of the local datetime to be set in the session of the Graph service. | String | No | `null`, which indicates the default format `%Y-%m-%dT%H:%M:%S` is used.
 | `zonedDatetimeFormat` | The format of the zoned datetime to be set in the session of the Graph service. | String | No | `null`, which indicates the default format `%Y-%m-%dT%H:%M:%S %z` is used.
 | `localTimeFormat` | The format of the local time to be set in the session of the Graph service. | String | No | `null`, which indicates the default format `%H:%M:%S` is used.
 | `zonedTimeFormat` | The format of the zoned time to be set in the session of the Graph service. | String | No | `null`, which indicates the default format `%H:%M:%S %z` is used.
 | `connectTimeoutMs` | The timeout in milliseconds for establishing a connection to the Graph service. | Long | No | `3600000`
 | `requestTimeoutMs` | The timeout in milliseconds for sending a write request to the Graph service. | Long | No | `3600000`
 | `enableDataCheck` | Whether to check the validity of data in NebulaGraph Exchange. Enable this option if the source data contains dirty data. | Boolean | No | `false`
 | `retryTimes` | The number of retry attempts when a write operation fails. | Integer | No | `3`
 | `retryIntervalMs` | The time interval between retries when a write operation fails. Unit: milliseconds. | Integer | No | `0`
 | `enableTls` | Whether to enable TLS. | Boolean | No | `false`
 | `disableVerifyServerCert` | Whether to disable server certificate verification. | Boolean | No | `false`
 | `ca` | The path to the CA (Certificate Authority) certificate file, which is used to verify the server's certificate. Required if `enableTls` is set to `true` and `disableVerifyServerCert` is set to `false`. | String | No | N/A
 | `cert` | The path to the certificate file of the client. Required if `tls_client_verify` is set to `true` in the configuration file of NebulaGraph Database Developer Edition's Graph service. | String | No | N/A
 | `key` | The path to the private key file of the client. Required if `tls_client_verify` is set to `true` in the configuration file of NebulaGraph Database Developer Edition's Graph service. | String | No | N/A
 | `preHooks` | The list of GQL statements to be executed sequentially before importing data. Use `"""` to enclose each GQL statement if it contains newlines, brackets (`[]`), or double quotation marks(`"`). Otherwise, use `"` to enclose it. | List | No | N/A

## Error configurations

This section describes the error configurations in NebulaGraph Exchange.

### Sample configuration

````
  error: {
    path: "file:///tmp/errors"
    maxRecords: 10000
  }

````

### Configuration options

 | Option | Description | Type | Required | Default value
 | `path` | The path for storing dirty and failed records in the form of GQL statements. You can specify a path from one of the following sources: 

- Local: Use an absolute path, a relative path, or the prefix `file://` followed by an absolute path in the local file system.
- HDFS: Use the prefix `hdfs://` followed by the path to the source data in HDFS. 
- Amazon S3: Use the prefix `s3a://` followed by the path to the source data in Amazon S3. 
- Tencent Cloud: Use the prefix `cosn://` followed by the path to the source data in Tencent cloud. 

 | String | No | `"file:///tmp/errors/"`
 | `maxRecords` | The maximum total of dirty and failed records that are allowed before the import task is suspended. If set to a negative number, the import task will run to completion regardless of the number of dirty and failed records. | Int | No | `Int.MAX_VALUE`, which equals `2,147,483,647`.

## Checkpoint file configurations

This section describes how to configure checkpoint files in NebulaGraph Exchange to enable checkpointing.

Note

Configurations for checkpoint files apply to all types of data sources except for Kafka. To enable resumable transfer for Kafka, set configuration option `startingOffsets` to `latest` to read data from the latest offset. For more information, see [Kafka configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/kafka/#data_source_configurations).

### Sample configuration

````
  checkPoint: {
    path: "file:///tmp/checkpoint"
    resume: true
  } 

````

### Configuration options

 | Option | Description | Type | Required | Default value
 | `path` | The relative or absolute path of the checkpoint file that records the data transfer progress. When importing different data batches, specify a unique path for each batch. You can specify a path from one of the following sources: 

- Local: Use an absolute path, a relative path, or the prefix `file://` followed by an absolute path in the local file system.
- HDFS: Use the prefix `hdfs://` followed by the path to the source data in HDFS. 
- Amazon S3: Use the prefix `s3a://` followed by the path to the source data in Amazon S3. 
- Tencent Cloud: Use the prefix `cosn://` followed by the path to the source data in Tencent cloud. 

 | String | No | `"file:///tmp/checkpoint"`
 | `resume` | Whether to enable checkpointing. | Bool | No | `false`

## Common source configurations

The common source configurations specify global settings for data sources and differ depending on the data source.

These configurations are optional and can also be specified for each data source in the [source configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/). When specified, the configurations on specific data sources override the global settings.

For the detailed common configuration options for each data source, see the corresponding source configuration topic as follows:

- [Local source configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/local/#data_source_configurations)
- [HDFS configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/hdfs/#data_source_configurations)
- [Hive configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/hive/#data_source_configurations)
- [JDBC configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/jdbc/#data_source_configurations)
- [Kafka configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/kafka/#data_source_configurations)
- [MaxCompute configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/maxcompute/#data_source_configurations)
- [Neo4j configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/neo4j/#data_source_configurations)
- [NebulaGraph 3.x configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/nebula3/#data_source_configurations)
- [Amazon S3 configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/s3/#data_source_configurations)
- [Tencent Cloud configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/tencent/#data_source_configurations)

## Common write configurations

The common write configurations specify global settings for writing data to NebulaGraph Database Developer Edition 5.x.

These configurations are optional and can also be specified for each node type or edge type in the [source configurations](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/config/sources/). When specified, the configurations on specific node types or edge types override the global settings.

### Sample configuration

````
  writeCommon:{
    enableRealTimeQPSLog: true
    batchSize:20
    mode: "INSERT IGNORE"
    ignoreHangingEdge: false
  }

````

### Configuration options

 | Option | Description | Type | Required | Default value
 | `enableRealTimeQPSLog` | Whether to enable real-time logging for write requests. | Bool | No | `false`
 | `batchSize` | The number of records (that is, nodes and edges combined) that are processed in each write request. | Int | No | `1000`
 | `mode` | The data manipulation mode when writing data to NebulaGraph Database Developer Edition. 
Valid values are `"INSERT"`, `"INSERT IGNORE"`, `"INSERT REPLACE"`, `"INSERT UPDATE"`, `"UPDATE"`, `"DELETE"`, and `"DETACH DELETE"`. For more information, see [Data manipulation](https://dev-docs.nebula-graph.io/v5.3.2/database-gql-reference/dml/). | String | No | `"INSERT IGNORE"`
 | `ignoreHangingEdge` | Whether to ignore edges that do not connect to nodes on either or both sides, known as dangling edges. Options:
`true`: No error is reported. The write operation for a dangling edge is ignored, but it is still counted as a successful write. 
`false`: An error is reported. The write operation for a dangling edge fails. | Bool | No | `false`

Note

The common write configurations above apply only to the standard import workflow (`taskType: import`). For custom write logic, see [Import data with GQL templates](https://dev-docs.nebula-graph.io/v5.3.2/utilities/exchange/import-data/template-import/).