HDFS configurations

This topic introduces the configuration options for importing data from an HDFS data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the HDFS data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "hdfs".	String	Yes	N/A
format	The file format of the source data. Valid values are: "csv", "json", "orc",and "parquet"	String	No	"csv"
header	Whether the CSV file contains a header. This option takes effect only when format is set to "csv". If the CSV file has no header, use the format _cX to refer to columns when you configure the mapping between HDFS data and graph elements. For example, _c0 represents the first column, _c1 represents the second column, and so on.	Bool	No	false
separator	The separator used in the CSV file. This option takes effect only when format is set to "csv".	String	No	","
path	The path to the source data in HDFS. The path can be specified as a directory or a file using the prefix hdfs:// followed by the source data path.	String	Yes	N/A

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• format
• header
• separator

Note that the configurations on specific data sources override the global settings.

Data source configuration example

HDFS source configurations

sources: [
  {
    type:"hdfs"
    format:"csv"
    header:false
    separator:","
    path: "hdfs://127.0.0.1:9000/import_movie/sub_actor.csv"
//    nodetypes: []
//    edgetypes: []
  }
]  

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"hdfs"
    format: "csv"
    header:true
    separator:","
  }

  writeCommon:{
    batchSize: 2000
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      path: "hdfs://127.0.0.1:9000/import_movie/sub_actor.csv"

      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      path: "hdfs://127.0.0.1:9000/import_movie/sub_movie.csv"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      path: "hdfs://127.0.0.1:9000/import_movie/sub_actor_act_movie.csv"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}