MaxCompute configurations

This topic introduces the configuration options for importing data from a MaxCompute data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the MaxCompute data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "maxcompute".	String	Yes	N/A
table	The name of the MaxCompute table from which data is retrieved.	String	Yes	N/A
project	The name of the MaxCompute project.	String	Yes	N/A
odpsUrl	The URL for accessing MaxCompute.	String	Yes	N/A
tunnelUrl	The URL for accessing the MaxCompute tunnel service.	String	Yes	N/A
accessKeyId	The AccessKey ID for accessing MaxCompute.	String	Yes	N/A
accessKeySecret	The AccessKey Secret for accessing MaxCompute.	String	Yes	N/A
partitionSpec	The partitions of the MaxCompute table.	String	Yes	N/A
readParallel	The number of concurrent read requests that are allowed.	Int	No	32

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• project
• odpsUrl
• tunnelUrl
• accessKeyId
• accessKeySecret
• readParallel

Note that the configurations on specific data sources override the global settings.

Data source configuration example

MaxCompute source configurations

sources: [
  {
      type: "maxcompute"
      table: movie
      project: project
      odpsUrl: "http://service.cn-hangzhou.maxcompute.aliyun.com/api"
      tunnelUrl: "http://dt.cn-hangzhou.maxcompute.aliyun.com"
      accessKeyId: xxx
      accessKeySecret: xxx
      partitionSpec: "dt='partition1',dd='partition2'"
      readParallel:32
//      nodetypes: []
//      edgetypes: []
  }
]  

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 10000
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  }  

  sourcesCommon:{
    type:"maxcompute"
    project: movie
    odpsUrl: "http://service.cn-hangzhou.maxcompute.aliyun.com/api"
    tunnelUrl: "http://dt.cn-hangzhou.maxcompute.aliyun.com"
    accessKeyId: xxx
    accessKeySecret: xxx
    readParallel:32
  }

  writeCommon:{
    batchSize: 2000
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  }

  sources: [
    # node type Actor
    {
      table: actor
      partitionSpec: "dt='partition1'"
      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      table: movie
      partitionSpec: "dt='partition1'"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      table: act
      partitionSpec: "dt='partition1'"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          nebulaSrcKeys: [id]
          targetKeys: [movie_id]
          nebulaDstKeys: [id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}