Import data with NebulaGraph Exchange¶

This topic describes how to import data into NebulaGraph Database Developer Edition using NebulaGraph Exchange.

Note

This topic describes the standard import workflow, which writes data to NebulaGraph Database Developer Edition using predefined write modes (INSERT, INSERT IGNORE, etc.). To customize the write logic, see Import data with GQL templates.

Prerequisites¶

NebulaGraph Exchange is deployed. For more information, see Deploy NebulaGraph Exchange.
A NebulaGraph Database Developer Edition service group is running. For more information, see Getting started.

Step 1: Prepare the configuration file¶

The configuration file must be in the HOCON (Human-Optimized Config Object Notation) format and contain the following information:

Connection configurations of the Graph service in a NebulaGraph Database Developer Edition service group.
Error configurations, such as the path to store errors and the limits of dirty and failed records.
Checkpoint file configurations, if applicable.
Common configurations, including common source and write configurations.
Data source configurations, including:
- The connection information and configurations specific to the data source. For more information, see the following topics:
- Data preprocessing configurations. For more information, see Preprocessing configurations.
- Mapping configurations between the source data and graph elements in NebulaGraph Database Developer Edition. For more information, see Mapping configurations.

For more information about the structure of the configuration file, see Data import configurations.

In this example, we import data from local CSV files that are part of the example dataset. The complete configuration file is as follows:

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

Local source configuration example

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"local"
    format: "csv"
    header:true
    separator:","
  }

  writeCommon:{
    batchSize: 2000 
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      path: "/tmp/import_movie/sub_actor.csv"

      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      path: "/tmp/import_movie/sub_movie.csv"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      path: "/tmp/import_movie/sub_actor_act_movie.csv"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}

(Optional) Step 2: Create a graph in NebulaGraph Database Developer Edition¶

Skip this step if you have configured preHooks in NebulaGraph Database Developer Edition configurations.

If you do not configure preHooks, you must create a graph in NebulaGraph Database Developer Edition to store the data to be imported. In NebulaGraph Database Developer Edition, graphs are created based on graph types, which define the data structure of the graphs. To ensure that your data is imported correctly, the graph type must match the data structure of the source data.

For example, assume that your source data contains the movie dataset, you need to create a graph type and a graph as follows after connecting to a NebulaGraph Database Developer Edition service group:

'''
// Create a graph type named `movie_type`
CREATE GRAPH TYPE IF NOT EXISTS movie_type AS { 
    NODE Actor (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}) , 
    NODE Director (LABEL Person {id INT64, name STRING, birthDate DATE, PRIMARY KEY (id)}), 
    NODE `User` (LABEL Person {id INT64 PRIMARY KEY}), 
    NODE Movie (LABEL Movie {id INT64 PRIMARY KEY, name STRING}), 
    NODE Genre (LABEL Genre {id INT64 PRIMARY KEY, name STRING}), 
    EDGE Act (Actor)-[:Act]->(Movie), 
    EDGE Direct (Director)-[:Direct]->(Movie), 
    EDGE Watch (`User`)-[:Watch {rate FLOAT}]->(Movie), 
    EDGE WithGenre (Movie)-[:WithGenre]->(Genre) 
    }
'''

'''
// Create a graph named `movie` based on graph type
CREATE GRAPH IF NOT EXISTS movie TYPED movie_type
'''

For more information about creating graph types and graphs, see CREATE GRAPH TYPE and CREATE GRAPH.

Step 3: Import data¶

In the directory nebula-graph-exchange/, use the JAR package cli-5.3.0.jar to import data as follows:

java -jar cli-5.3.0.jar -f <path_of_config_file> -n 5 -p 2

The following table lists the command-line options:

Option	Alias	Description	Required	Default value
`-file`	`-f`	The path to the configuration file. You can specify a path from one of the following sources: Local: Use an absolute path, a relative path, or the prefix `file://` followed by an absolute path in the local file system. HDFS: Use the prefix `hdfs://` followed by the path to the source data in HDFS. Amazon S3: Use the prefix `s3a://` followed by the path to the source data in Amazon S3. Tencent Cloud: Use the prefix `cosn://` followed by the path to the source data in Tencent cloud.	Yes	N/A
`-jar`	`-j`	The path to the NebulaGraph Exchange JAR package.	No	`nebula-exchange-5.3.0.jar`
`-log`	-	The custom subdirectory in the `nebula-graph-exchange/logs` folder used to store logs. If this option is NOT specified, logs are stored in the `nebula-graph-exchange/logs` folder by default.	No	N/A
`-driverMemory`	`-d`	The maximum memory that can be used by the driver. The driver manages data import tasks on worker nodes.	No	`1G`
`-memory`	`-m`	The maximum memory that can be used by each job. A job includes multiple tasks to complete a data import process.	No	`1G`
`-num`	`-n`	The number of jobs for importing data.	No	`20`
`-parallel`	`-p`	The number of cores that run in parallel for each job.	No	`5`
`-ip`	-	The local IP of the Spark server. If not specified, the IP is automatically obtained.	No	N/A
`-port`	-	The port of the Spark server.	No	`1077`
`-check`	-	Whether to check the validity of the configuration file. If this option is specified, NebulaGraph Exchange checks the validity of the configuration file and does not import data. The check results are recorded in the log file `logs/check.log` in the `nebula-graph-exchange/` directory or the directory specified by `-log`. If this option is NOT specified, NebulaGraph Exchange attempts to import data based on the configuration file. The import results are recorded in the log file `logs/import.log` in the `nebula-graph-exchange/` directory or the directory specified by `-log`.	No	N/A
`-dry`	-	Whether to perform a dry run. If this option is specified, NebulaGraph Exchange performs a dry run of the import process, during which it reads the source data, counts the number of records to be imported, but does not import the data to NebulaGraph Database Developer Edition.	No	N/A
`-h`	-	Whether to print help information.	No	N/A
`-reload`	-	Whether to reload the GQL statements that failed to execute. Each time the data import task is completed, a unique ID named `app id` is returned. You can specify `app id` to reload the GQL statements that failed to execute during the previous run. Besides, you can check the files in the path specified in the `error.path` to view failed data.	No	N/A
`-spark`	`-s`	The path to your Spark home directory. Enable it as needed if you have your own Spark.	No	N/A
`-mode`	`-o`	The master mode of your Spark. Required if `-spark` is specified.	No	N/A
`-conf`	-	The advanced configuration items. Separate multiple configuration items by space, such as `-conf a -conf b`. To access the HDFS data using Kerberos authentication, specify this option as follows: `-conf "spark.driver.extraJavaOptions=-Djava.security.krb5.conf=./krb5.conf" -conf "spark.executor.extraJavaOptions=-Djava.security.krb5.conf=./krb5.conf"`.	No	N/A
`-files`	-	The path to resource files to be distributed with the job. For example, to access the HDFS data using Kerberos authentication, specify this option as follows: `-files "/local/path/to/hdfs.keytab,/local/path/to/krb5.conf"`.	No	N/A

Note: If the product of n and p is larger than that of i and c, which are set when you deploy NebulaGraph Exchange, it is automatically adjusted to the product of i and c.

If the data import command is submitted successfully, the following example messages are displayed:

...
starting the server...

starting to import...

exchange app id: <your app id>
import finished!

After the import is complete, you can check the import results in the following files:
- nebula-graph-exchange/<app_date_number_result>.json: the import results include the following fields:
  - costSeconds: the total number of seconds taken to complete the entire data import process
  - qps: queries per second, showing the processing speed of the job
  - totalSourceRecords: the total number of source records
  - totalSuccessRecords: the total number of successfully processed records
  - successNodes: the number of successfully processed node records
  - successEdges: the number of successfully processed edge records
  - totalFailureRecords: the total number of records that failed to process
  - FailureNodes: the number of node records that failed to process
  - FailureEdges: the number of edge records that failed to process
  - totalAffectedRecords: the total number of records affected by operations like updates or deletions
  - affectedNodes: the number of node records affected by operations like updates or deletions
  - affectedEdges: the number of edge records affected by operations like updates or deletions
- nebula-graph-exchange/logs/import.log: the real-time progress of the import task
- nebula-graph-exchange/logs/<stderr._number>.log: the runtime results and specific error messages for each job
- Files in error.path: dirty records and failed records