Local source configurations

This topic introduces the configuration options for importing data from a local data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the local data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "local".	String	Yes	N/A
format	The file format of the source data. Valid values are: "csv", "json", "orc",and "parquet"	String	No	"csv"
header	Whether the CSV file contains a header. This option takes effect only when format is set to "csv".	Bool	No	false
separator	The separator used in the CSV file. This option takes effect only when format is set to "csv".	String	No	","
path	The path to the source data in a local file system. You can specify the path in the following ways: Use the prefix file:// with an absolute path, such as "file:///tmp/person.csv". Use an absolute path, such as "/tmp/person.csv". Use a relative path that is relative to the NebulaGraph Exchange JAR file directory, such as "tmp/person.csv".	String	Yes	N/A

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• format
• header
• separator

Note that the configurations on specific data sources override the global settings.

Data source configuration example

Local file system source configurations

sources: [
  {
    type:"local"
    format:"csv"
    header:false
    separator:","
    path: "file:///tmp/movie.csv"
//    nodetypes: []
//    edgetypes: []
  }
]  

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"local"
    format: "csv"
    header:true
    separator:","
  }

  writeCommon:{
    batchSize: 2000
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      path: "/tmp/import_movie/sub_actor.csv"

      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      path: "/tmp/import_movie/sub_movie.csv"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      path: "/tmp/import_movie/sub_actor_act_movie.csv"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}