JDBC configurations

This topic introduces the configuration options for importing data from a JDBC data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the JDBC data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "jdbc".	String	Yes	N/A
url	The JDBC URL of the data source.	String	Yes	N/A
driver	The JDBC driver class name. For MySQL, set the value to "com.mysql.cj.jdbc.Driver". For PostgreSQL, set the value to "org.postgresql.Driver". For other databases, set the value according to the JDBC driver version.	String	Yes	N/A
user	The username of the data source.	String	Yes	N/A
password	The password of the user.	String	Yes	N/A
partitionColumn	The column used to partition the data for parallel reading. The column must be of a numeric type.	String	No	N/A
lowerBound	The lower bound value of the partition column.	Int	No	N/A
upperBound	The upper bound value of the partition column.	Int	No	N/A
fetchSize	The number of rows that are fetched in each read request.	Int	No	1000
readParallel	The number of concurrent read requests that are allowed.	Int	No	32
statement	The SQL statement used to retrieve data from the data source. This option cannot be used together with table.	String	No	N/A
table	The name of the table from which data is retrieved. This option cannot be used together with statement.	String	No	N/A
custom_options	Custom Spark JDBC data source options, corresponding to options set via Spark DataFrameReader.option() or .options() methods. The value is a map of key-value pairs. For example, custom_options:{isolationLevel:"NONE"}. For the available keys, see Spark JDBC Data Source options. If a key in custom_options overlaps with a dedicated JDBC option on this page (such as fetchSize or partitionColumn), the value in custom_options takes precedence.	Map	No	N/A

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• url
• driver
• user
• password
• readParallel

Note that the configurations on specific data sources override the global settings.

Data source configuration example

JDBC source configurations

sources: [
  {
      type:"jdbc"
      url:"jdbc:mysql://192.168.8.149:3306/movie"
      driver:"com.mysql.cj.jdbc.Driver"
      user:"nebula"
      password:"Nebula@123"
      partitionColumn: birthDate
      lowerBound:15
      upperBound:60
      fetchSize:1000
      readParallel:20
      #table:"actor"
      statement:"select people_id,name,birth_date from movie.actor"
      custom_options:{
        isolationLevel:"NONE"
      }
//      nodetypes: []
//      edgetypes: []
  }
]

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"jdbc"
    url:"jdbc:mysql://192.168.8.149:3306/movie"
    driver:"com.mysql.cj.jdbc.Driver"
    user:"nebula"
    password:"Nebula@123"
  }

  writeCommon:{
    batchSize: 2000
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      statement:"select people_id,name,birth_date from movie.actor"
      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      statement:"select movie_id,name from movie.movie"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      statement:"select people_id,movie_id from movie.act"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}