Data import configurations¶

This topic introduces the configuration file for data import in NebulaGraph Exchange.

Configuration file structure¶

The structure of the NebulaGraph Exchange configuration file is as follows:

{
    taskType: <task_type>

    nebula: {}  

    error: {}

    checkPoint: {}

    sourcesCommon: {}

    writeCommon: {}

    sources: []
}

For different data sources, the structure of the NebulaGraph Exchange configuration file is the same. The main differences are in the sections sourcesCommon and sources. Configuration file templates for different data sources are provided in the directory nebula-graph-exchange/template/ of the NebulaGraph Exchange package.

As you can see, the configuration file consists of the following sections:

taskType: Specify the task type. For more information, see Task type configurations.
nebula: Specify the connection configurations for NebulaGraph Database Developer Edition. For more information, see NebulaGraph Database Developer Edition configurations.
error: Specify how to handle dirty records and failed records during import. For more information, see Error configurations.
checkPoint: Specify the configurations for checkpointing. For more information, see Checkpoint file configurations.
sourcesCommon: Specify the common configurations for the data source. For more information, see Common source configurations.
writeCommon: Specify the common configurations for the write operation. For more information, see Common write configurations.
sources: Specify the connection information and configurations specific to the data source, how to preprocess the source data, and how to map the source data to graph elements in NebulaGraph Database Developer Edition. For more information, see Source configurations.

Task type configurations¶

Option	Description	Type	Required	Default value
`taskType`	The task type. Valid values: `import`: Import data. `check`: Check the validity of the configuration file but do not import data. Check results are recorded in the log file `logs/import.log` in the `nebula-graph-exchange/` directory or in the directory specified by `-log`. `gqlTemplate`: Import data to NebulaGraph Database Developer Edition by using custom GQL templates. For more information, see Import data with GQL templates. `dryRun`: Perform a dry run of the import process. During a dry run, NebulaGraph Exchange reads the source data and counts the records to be imported, but does not import data into NebulaGraph Database Developer Edition. `reload`: Reload the GQL statements that failed to execute. Each time a data import task is completed, a unique ID named `app id` is returned. You can specify `app id` to reload statements that failed in the previous run. You can also check failed data in the path specified by `error.path`. `gql`: Generate a file containing GQL statements based on up to 200 source records. The statements are generated based on Common write configurations and stored in `/tmp/gql/`, where each statement represents one write request. If the option `-check`, `-dry`, or `-reload` is specified in the data import command and conflicts with the task type configured here, the option in the command takes precedence. For example, if the import command specifies `-dry` while `taskType` is set to `check`, a dry run is performed.	String	Yes	`import`

NebulaGraph Database Developer Edition configurations¶

This section describes how to configure NebulaGraph Database Developer Edition in NebulaGraph Exchange.

Sample configuration¶

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    enableEncryptPasswd: false
    passwd: "<current-password>"
    schema: "/default_schema"
    timezone: "Asia/Shanghai"
    dateFormat: "%Y-%m-%d"
    localDatetimeFormat: "%Y-%m-%dT%H:%M:%S"
    zonedDatetimeFormat: "%Y-%m-%dT%H:%M:%S %z"
    localTimeFormat: "%H:%M:%S"
    zonedTimeFormat: "%H:%M:%S %z"
    connectTimeoutMs: 6000
    requestTimeoutMs: 6000
    enableDataCheck: false
    retryTimes: 3
    retryIntervalMs: 0    
    tls: {
        enableTls: false
        disableVerifyServerCert: false
        ca: "/tmp/ca.pem"
        cert: "/tmp/cert.crt"
        key: "/tmp/key"
    }    
    preHooks:[
        """ CREATE GRAPH TYPE IF NOT EXISTS movie_type AS {
                NODE Actor (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}),
                NODE Director (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}),
                NODE `User` (LABEL Person {id INT64 PRIMARY KEY}),
                NODE Movie (LABEL Movie {id INT64 PRIMARY KEY, name STRING}),
                NODE Genre (LABEL Genre {id INT64 PRIMARY KEY, name STRING}),
                EDGE Act (Actor)-[:Act]->(Movie),
                EDGE Direct (Director)-[:Direct]->(Movie),
                EDGE Watch (`User`)-[:Watch {rate FLOAT}]->(Movie),
                EDGE WithGenre (Movie)-[:WithGenre]->(Genre)
        }""",
        "CREATE GRAPH IF NOT EXISTS movie TYPED movie_type"
        ]
  }

Configuration options¶

Option	Description	Type	Required	Default value
`graphAddr`	The IPs and ports of Graph services. Separate multiple addresses with commas.	String	Yes	N/A
`graphName`	The name of the graph to which data is written.	String	Yes	N/A
`user`	The username of the Graph service.	String	No	`"root"`
`enableEncryptPasswd`	Whether to use the encrypted password.	Bool	No	N/A
`passwd`	The current password for the account used to connect. If `enableEncryptPasswd` is set to `true`, `passwd` must be set to the encrypted password. For more information, see Encrypt passwords.	String	Yes	N/A
`schema`	The schema to which data is written.	String	No	`null`, which indicates the default schema in the path `/default_schema` is used.
`timezone`	The time zone to be set in the session of the Graph service.	String	No	`null`, which indicates the time zone on the server that runs the Graph service is used.
`dateFormat`	The format of the date to be set in the session of the Graph service.	String	No	`null`, which indicates the default format `%Y-%m-%d` is used.
`localDatetimeFormat`	The format of the local datetime to be set in the session of the Graph service.	String	No	`null`, which indicates the default format `%Y-%m-%dT%H:%M:%S` is used.
`zonedDatetimeFormat`	The format of the zoned datetime to be set in the session of the Graph service.	String	No	`null`, which indicates the default format `%Y-%m-%dT%H:%M:%S %z` is used.
`localTimeFormat`	The format of the local time to be set in the session of the Graph service.	String	No	`null`, which indicates the default format `%H:%M:%S` is used.
`zonedTimeFormat`	The format of the zoned time to be set in the session of the Graph service.	String	No	`null`, which indicates the default format `%H:%M:%S %z` is used.
`connectTimeoutMs`	The timeout in milliseconds for establishing a connection to the Graph service.	Long	No	`3600000`
`requestTimeoutMs`	The timeout in milliseconds for sending a write request to the Graph service.	Long	No	`3600000`
`enableDataCheck`	Whether to check the validity of data in NebulaGraph Exchange. Enable this option if the source data contains dirty data.	Boolean	No	`false`
`retryTimes`	The number of retry attempts when a write operation fails.	Integer	No	`3`
`retryIntervalMs`	The time interval between retries when a write operation fails. Unit: milliseconds.	Integer	No	`0`
`enableTls`	Whether to enable TLS.	Boolean	No	`false`
`disableVerifyServerCert`	Whether to disable server certificate verification.	Boolean	No	`false`
`ca`	The path to the CA (Certificate Authority) certificate file, which is used to verify the server's certificate. Required if `enableTls` is set to `true` and `disableVerifyServerCert` is set to `false`.	String	No	N/A
`cert`	The path to the certificate file of the client. Required if `tls_client_verify` is set to `true` in the configuration file of NebulaGraph Database Developer Edition's Graph service.	String	No	N/A
`key`	The path to the private key file of the client. Required if `tls_client_verify` is set to `true` in the configuration file of NebulaGraph Database Developer Edition's Graph service.	String	No	N/A
`preHooks`	The list of GQL statements to be executed sequentially before importing data. Use `"""` to enclose each GQL statement if it contains newlines, brackets (`[]`), or double quotation marks(`"`). Otherwise, use `"` to enclose it.	List	No	N/A

Error configurations¶

This section describes the error configurations in NebulaGraph Exchange.

Sample configuration¶

  error: {
    path: "file:///tmp/errors"
    maxRecords: 10000
  }

Configuration options¶

Option	Description	Type	Required	Default value
`path`	The path for storing dirty and failed records in the form of GQL statements. You can specify a path from one of the following sources: Local: Use an absolute path, a relative path, or the prefix `file://` followed by an absolute path in the local file system. HDFS: Use the prefix `hdfs://` followed by the path to the source data in HDFS. Amazon S3: Use the prefix `s3a://` followed by the path to the source data in Amazon S3. Tencent Cloud: Use the prefix `cosn://` followed by the path to the source data in Tencent cloud.	String	No	`"file:///tmp/errors/"`
`maxRecords`	The maximum total of dirty and failed records that are allowed before the import task is suspended. If set to a negative number, the import task will run to completion regardless of the number of dirty and failed records.	Int	No	`Int.MAX_VALUE`, which equals `2,147,483,647`.

Checkpoint file configurations¶

This section describes how to configure checkpoint files in NebulaGraph Exchange to enable checkpointing.

Note

Configurations for checkpoint files apply to all types of data sources except for Kafka. To enable resumable transfer for Kafka, set configuration option startingOffsets to latest to read data from the latest offset. For more information, see Kafka configurations.

Sample configuration¶

  checkPoint: {
    path: "file:///tmp/checkpoint"
    resume: true
  }

Configuration options¶

Option	Description	Type	Required	Default value
`path`	The relative or absolute path of the checkpoint file that records the data transfer progress. When importing different data batches, specify a unique path for each batch. You can specify a path from one of the following sources: Local: Use an absolute path, a relative path, or the prefix `file://` followed by an absolute path in the local file system. HDFS: Use the prefix `hdfs://` followed by the path to the source data in HDFS. Amazon S3: Use the prefix `s3a://` followed by the path to the source data in Amazon S3. Tencent Cloud: Use the prefix `cosn://` followed by the path to the source data in Tencent cloud.	String	No	`"file:///tmp/checkpoint"`
`resume`	Whether to enable checkpointing.	Bool	No	`false`

Common source configurations¶

The common source configurations specify global settings for data sources and differ depending on the data source.

These configurations are optional and can also be specified for each data source in the source configurations. When specified, the configurations on specific data sources override the global settings.

For the detailed common configuration options for each data source, see the corresponding source configuration topic as follows:

Common write configurations¶

The common write configurations specify global settings for writing data to NebulaGraph Database Developer Edition 5.x.

These configurations are optional and can also be specified for each node type or edge type in the source configurations. When specified, the configurations on specific node types or edge types override the global settings.

Sample configuration¶

  writeCommon:{
    enableRealTimeQPSLog: true
    batchSize:20
    mode: "INSERT IGNORE"
    ignoreHangingEdge: false
  }

Configuration options¶

Option	Description	Type	Required	Default value
`enableRealTimeQPSLog`	Whether to enable real-time logging for write requests.	Bool	No	`false`
`batchSize`	The number of records (that is, nodes and edges combined) that are processed in each write request.	Int	No	`1000`
`mode`	The data manipulation mode when writing data to NebulaGraph Database Developer Edition. Valid values are `"INSERT"`, `"INSERT IGNORE"`, `"INSERT REPLACE"`, `"INSERT UPDATE"`, `"UPDATE"`, `"DELETE"`, and `"DETACH DELETE"`. For more information, see Data manipulation.	String	No	`"INSERT IGNORE"`
`ignoreHangingEdge`	Whether to ignore edges that do not connect to nodes on either or both sides, known as dangling edges. Options: `true`: No error is reported. The write operation for a dangling edge is ignored, but it is still counted as a successful write. `false`: An error is reported. The write operation for a dangling edge fails.	Bool	No	`false`

Note

The common write configurations above apply only to the standard import workflow (taskType: import). For custom write logic, see Import data with GQL templates.