CREATE VECTOR INDEX statement

The CREATE VECTOR INDEX statement creates a vector index on a specific graph.

To specify the graph, use either of the following methods:

• Add a USE clause before this statement to set the working graph.
• Use SESSION SET to set the graph for the current session in advance.

If both methods are used, the one specified in the USE clause takes precedence. If neither is used, an error occurs.

You can create a vector index based on a single property of a node type or edge type. Vector indexes speed up approximate nearest neighbor (ANN) queries that search for similar vectors based on the indexed properties.

To create other types of indexes, see CREATE NORMAL INDEX, CREATE SPATIAL INDEX, and CREATE FULLTEXT INDEX.

How vector indexes are implemented

After a vector index is created, the following data is generated:

• Metadata that describes the vector index.
• Key-value pairs that are persisted in RocksDB and map the indexed vectors to their corresponding nodes or edges.
• An ANN index, which resides in memory to speed up ANN searches.

Note

If multiple Storage services are running, multiple ANN indexes are created and maintained on each Storage service.

• The availability of the ANN indexes is ensured by the Raft consensus algorithm. Even if a leader change occurs, the ANN indexes on the new leader are guaranteed to be consistent with the ones on the previous leader.
• The maximum memory usage of all ANN indexes in each Storage service is controlled by the configuration option ann_index_memory_quota_in_kb. If this quota is exceeded, insertion operations related to vector indexes will lead to an error.

To guarantee data integrity for the ANN index, Write-ahead logging (WAL) is implemented, which ensures changes to the index are written only after they have been logged, that is, after the WAL records are persisted to disk.

Each ANN index corresponds to two files that are persisted to disk: a WAL file (.annwal) and a full index file (.annidx). Changes to the ANN index are written only after they have been logged in the WAL file. When the threshold set by the configuration option ann_write_idx_file_threshold_of_log_num is exceeded, records in the WAL file are summarized and written to the full index file. The overlapping records in the WAL file are then cleared.

The WAL file takes up more disk space than the full index file, but outperforms the latter in terms of I/O efficiency. It is recommended that you set the threshold to a value that balances the trade-off between storage usage and I/O efficiency.

The WAL file and full index file are stored in the path specified by the configuration option data_path of the Storage service. Both WAL and full index files are required to recover the ANN index in case of a system crash.

Note

An operation involving a non-default schema or an object in it requires access to the schema, either implicitly or through an explicit USAGE grant, as well as the privilege required for the operation. The schema access check and the privilege check for the requested operation are independent; satisfying one does not satisfy the other. Schema access control does not apply to temporary graphs or built-in procedures and functions. For more information, see Schema access control.

Prerequisites

• A graph is created. For more information, see CREATE GRAPH.

• To create a vector index, you must either be the owner of the graph, be granted to perform the action INDEX on the graph, or be granted to perform the action CREATE INDEX on the used schema. For more information, see Roles and privileges.

Limitations

After vector indexes are created, you cannot back up vector indexes. You can still back up graph data, but vector indexes are not included in backup files.

Syntax

diagramsource

create_vector_index_statement

ann_index_option

create_vector_index_statement ::=
    'CREATE' 'VECTOR' 'INDEX' ('IF NOT EXISTS')? index_name 'ON' ( ( ('NODE' | 'VERTEX') 'TYPE'? node_type_name ) | ( ('EDGE'|'RELATIONSHIP') 'TYPE'? edge_type_name ) ) '::' property_name 'OPTIONS' '{' ann_index_option (',' ann_index_option)* '}'  

ann_index_option ::= 
    'DIM' ':' int_value
  | 'METRIC' ':' ( 'L2' | 'IP' )
  | 'TYPE' ':' ( 'IVF' | 'HNSW' )
  | 'NLIST' ':' int_value
  | 'TRAINSIZE' ':' int_value
  | 'MAXDEGREE' ':' int_value
  | 'EFCONSTRUCTION' ':' int_value
  | 'CAPACITY' ':' int_value

Parameters

IF NOT EXISTS

This clause checks if an index with the same name already exists. If it already exists, the index is not created and no exception is thrown.

If you do not specify this clause, an exception is thrown if an index with the same name already exists.

index_name

The name of the vector index to be created. The name is a regular identifier, and must be unique among all types of indexes within the same graph.

For more information about identifiers, see Rules for naming identifiers.

node_type_name

The name of the node type on which the vector index is created. The node type must have at least one property whose value type is VECTOR.

edge_type_name

The name of the edge type on which the vector index is created. The edge type must have at least one property whose value type is VECTOR.

property_name

The name of the property of the node type or edge type on which the vector index is created. The value type of the property must be VECTOR. For more information, see Vector types.

ann_index_option

The options for creating the vector index, among which the option DIM is required while the others are optional.

DIM

The number of dimensions of the vector index to be created, which must be the same as that of the property on which the index is created, whose value type must be VECTOR.

METRIC

The metric used to calculate the similarity between vectors in the vector index. Valid values are:

• L2: The Euclidean distance metric, where the similarity between vectors is calculated based on the Euclidean distance between them.
• IP: The inner product metric, where the similarity between vectors is calculated based on the inner product between them.

The similarity metric specified here determines the function used to calculate the similarity between vectors when you perform ANN searches. For more information, see Nearest neighbor query.

If you do not specify this option, the default value L2 applies.

TYPE

The algorithm used to create the vector index. Valid values are:

• IVF: The index is created using the IVF (Inverted File) algorithm.
• HNSW: The index is created using the HNSW (Hierarchical Navigable Small World) algorithm.

If you do not specify this option, the default value IVF applies.

NLIST

The number of clusters created during the IVF index construction. Each cluster corresponds to an inverted list that stores references to vectors assigned to that cluster. This option takes effect only when the option TYPE is set to IVF.

This option determines the granularity of the IVF index. A larger value reduces the average number of vectors in each cluster, which speeds up the search performance but decreases the recall.

If you do not specify this option while TYPE is set to IVF, the default value 8 applies.

TRAINSIZE

The number of vectors used to train the IVF index during construction. This option takes effect only when the option TYPE is set to IVF.

This options determines clustering quality of the IVF index. The larger the value, more vectors are scanned to find the best cluster for each vector, which improves the recall but increases the training time.

If you do not specify this option while TYPE is set to IVF, the default value 3 applies.

MAXDEGREE

The maximum number of edges that each node can have in the HNSW graph. This option takes effect only when the option TYPE is set to HNSW.

This option controls the sparsity of the HNSW graph to balance search efficiency and recall. The larger the value, the more accurate the search results, but the slower the search speed and the more memory consumed.

If you do not specify this option while TYPE is set to HNSW, the default value 8 applies.

EFCONSTRUCTION

The size of the dynamic candidate list used during HNSW graph construction, which determines how many neighbors are evaluated when inserting a new node into the graph. This option takes effect only when the option TYPE is set to HNSW.

This option controls the quality of the HNSW graph. A larger value allows more thorough exploration of potential neighbors, which improves the recall but increases the construction time.

If you do not specify this option while TYPE is set to HNSW, the default value 16 applies.

CAPACITY

The maximum number of elements that the HNSW index can store. This option takes effect only when the option TYPE is set to HNSW.

This option dictates the memory usage of the HNSW index. If the capacity is exceeded, the HNSW index is automatically resized, which brings additional overhead when building the HNSW index.

If you do not specify this option while TYPE is set to HNSW, the default value 10000 applies.

Examples

The following examples are based on the graph type ann_test created in CREATE GRAPH TYPE.

Create an IVF vector index on a node type

'''
CREATE GRAPH IF NOT EXISTS ann_test_graph TYPED ann_test    // Create a graph
'''

'''
USE ann_test_graph CREATE VECTOR INDEX IF NOT EXISTS idx_node_vec1 ON NODE N1::vec1 OPTIONS { DIM: 3, METRIC: L2, TYPE: IVF, NLIST: 10, TRAINSIZE: 4 }
'''

The preceding example creates an IVF vector index named idx_node_vec1 on the node type N1 based on the property vec1.

Create an HNSW vector index on an edge type

'''
CREATE GRAPH IF NOT EXISTS ann_test_graph TYPED ann_test    // Create a graph
'''

'''
USE ann_test_graph CREATE VECTOR INDEX IF NOT EXISTS idx_edge_vec2 ON EDGE E1::vec2 OPTIONS { DIM: 7, METRIC: IP, TYPE: HNSW, MAXDEGREE: 3, EFCONSTRUCTION: 6, CAPACITY: 500 }
'''

The preceding example creates an HNSW vector index named idx_edge_vec2 on the edge type E1 based on the property vec2.