Amazon S3 configurations

This topic introduces the configuration options for importing data from an Amazon S3 data source. For information about the complete configuration file and options, see Data import configurations.

Data source configurations

The following table lists the configuration options specific to the Amazon S3 data source.

Option	Description	Type	Required	Default value
type	The type of the data source. In this case, set the value to "s3".	String	Yes	N/A
format	The file format of the source data. Valid values are: "csv", "json", "orc",and "parquet"	String	No	"csv"
path	The path to the source data in Amazon S3. The path can be specified as a directory or a file using the prefix s3a:// with an absolute path.	String	Yes	N/A
endpoint	The service endpoint of the Amazon S3 server in the format of s3.<region-code>.amazonaws.com. Replace <region-code> with the region code of your AWS account. For more information, see Region codes.	String	Yes	N/A
accessKey	The access key for accessing the Amazon S3 bucket.	String	Yes	N/A
secretKey	The secret key for accessing the Amazon S3 bucket.	String	Yes	N/A
header	Whether the CSV file contains a header. This option takes effect only when format is set to "csv". If the CSV file has no header, use the format _cX to refer to columns when you configure the mapping between Amazon S3 data and graph elements. For example, _c0 represents the first column, _c1 represents the second column, and so on.	Bool	No	false
separator	The separator used in the CSV file. This option takes effect only when format is set to "csv".	String	No	","

Common source configurations

The following configuration options can also be specified as global settings in the section sourcesCommon of the configuration file:

• type
• format
• header
• separator
• endpoint
• accessKey
• secretKey

Note that the configurations on specific data sources override the global settings.

Data source configuration example

Amazon S3 source configurations

sources: [
{
      type:"s3"
      format: "csv"
      path: "s3a://bucket/path/player.csv"
      endpoint: "s3.us-west-2.amazonaws.com"
      accessKey: "xxx"
      secretKey: "xxxx"
      header: true
      separator: ","
//      nodetypes: []
//      edgetypes: []
    }
]  

Complete configuration example

Replace <current-password> with the current password for the specified account. If the password has changed, use the new password.

{
  taskType: import
  nebula: {
    graphAddr: "127.0.0.1:9669"
    graphName: "movie"
    user: "root"
    passwd: "<current-password>"
  }

  error: {
    path: "/tmp/errors"
    maxRecords: 100
  }

  checkPoint: {
    path: "/tmp/checkpoint"
    resume: false
  } 

  sourcesCommon:{
    type:"s3"
    format: "csv"
    endpoint: "s3.us-west-2.amazonaws.com"
    accessKey: "xxx"
    secretKey: "xxxx"
    header: true
    separator: ","
  }

   writeCommon:{
    batchSize: 2000
    enableRealTimeQPSLog: true
    mode: "INSERT IGNORE"
  } 

  sources: [
    # node type Actor
    {
      path: "s3a://nebula-data-trans-test-us-west-2/import_movie/sub_actor.csv"

      pre_processes: {
          filter: ["birth_date!='\\N'"]
          concatConstant: [
            {
              field: "name"
              prefixValue: "test_"
              suffixValue: ""
            }
          ]
      }

      nodetypes: [
        {
          name: "Actor"
          sourceFields: [people_id,name,birth_date]
          nebulaFields: [id,name,birthDate]
        }
      ]
    }

    # node type Movie
    {
      path: "s3a://nebula-data-trans-test-us-west-2/import_movie/sub_movie.csv"
      nodetypes: [
        {
          name: "Movie"
          sourceFields: [movie_id,name]
          nebulaFields: [id,name]
        }
      ]
    }

   # edge type Act
    {
      path: "s3a://nebula-data-trans-test-us-west-2/import_movie/sub_actor_act_movie.csv"
      edgetypes: [
        {
          name: "Act"
          sourceKeys: [people_id]
          targetKeys: [movie_id]
          sourceFields: []
          nebulaFields: []
        }
      ]
    }

  ]
}