Hive configurations

This topic introduces the configuration options for importing data from a Hive data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the Hive data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "hive".	String	Yes	N/A
uris	The thrift URIs for the remote metastore services. You can obtain the URIs from the property hive.metastore.uris in the file hive-site.xml. Optional if your Spark is integrated with Hive.	String	No	N/A
defaultFS	The URI of the default file system where Hive data is located. Optional if your Spark is integrated with Hive.	String	No	N/A
statement	The HiveQL statement used to retrieve data from Hive.	String	Yes	N/A

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• uris
• defaultFS

Note that the configurations on specific data sources override the global settings.

Data source configuration example

Hive source configurations

sources: [
  {
      type:"hive"
      uris:"thrift://192.168.8.149:9083"
      defaultFS:"hdfs://192.168.8.149:9000"
      statement:"select people_id,name,birth_date from movie.actor"
//      nodetypes: []
//      edgetypes: []
  }
]    

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"hive"
    uris:"thrift://192.168.8.149:9083"
    defaultFS:"hdfs://192.168.8.149:9000"
  }

  writeCommon:{
    batchSize: 2000 
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      statement:"select people_id,name,birth_date from movie.actor"
      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      statement:"select movie_id,name from movie.movie"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      statement:"select people_id,movie_id from movie.act"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}