
== Import and Export to Cypher

Refers to the issue https://github.com/neo4j-contrib/neo4j-apoc-procedures/issues/439/[#439] we documented some performance tests of export and import of a big graph into Cypher format file.
For the test we used a server with this characteristics:

* 6 cores

* Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz

* 128 GB of RAM


.Neo4j configuration

* dbms.memory.heap.initial_size=8192m

* dbms.memory.heap.max_size=8192m

* dbms.memory.pagecache.size=4g

They have not be noticed significative difference with 4 GB of heap memory.

.Graph info

* total *nodes* 3.158.994

* total *relationships* 16.800.936

Download here https://dl.dropboxusercontent.com/u/14493611/ldbc_sf001_p006.tgz/[LDBC SF1]

== Script to execute all the tests

We created a script that execute all the tests explained below, you can run it like in this example:

[source,bash,subs=attributes]
----
./performanceCypherTest.sh 'neo4jHome' 'userName' 'password' 'address'
----

the `address` parameter  is optional, the default address is : bolt://localhost:7687
If you use the LDBC SF1 graph, or another big one is better to change the `open files allowed` from the default 1024 at last to 40.000.

Download link:{script}/performanceCypherTest.sh[performanceCypherTest.sh]

== Export all

=== Batch size

With the use of the config param `batchSize` we done some tests with different batch size.
The default value is 20000.

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword
"call apoc.export.cypher.all('yourPath/exportAll', {format:'neo4j-shell', batchSize: 10000})"
----

==== Results

.Default 20.000

----
real 1m52.744s
user 0m0.936s
sys 0m0.064s
----

.Batch size 10.000

----
real    1m50.715s
user    0m0.932s
sys     0m0.076s
----

.Batch size 1.000

----
real    1m49.577s
user    0m0.888s
sys     0m0.120s
----

.Batch size 100

----
real    1m51.297s
user    0m0.928s
sys     0m0.088s
----

=== Different output formats

We try the different output formats, changing the `config` parameter `format`.

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword  
"call apoc.export.cypher.all('yourPath/exportData.cypher', {format:'neo4j-shell'})"
----

==== Results

.neo4j-shell

----
real    1m49.268s
user    0m0.904s
sys     0m0.072s
----

.cypher-shell

----
real    1m55.089s
user    0m0.892s
sys     0m0.092s
----

.plain

----
real    1m54.490s
user    0m0.932s
sys     0m0.076s
----

=== Many files (separateFiles config)

With the param `separateFiles` (default false) we can export our graph or part of it, in different files.
In the example below we name the exported file `exportAll.cypher` so our export will be:

 * exportAll.cleanup.cypher
 * exportAll.nodes.cypher
 * exportAll.relationships.cypher
 * exportAll.schema.cypher

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword  
"call apoc.export.cypher.all('yourPath/exportAll.cypher', {format:'neo4j-shell',separateFiles:true})"
----

==== Result

----
real 1m55.229s
user 0m0.960s
sys  0m0.084s
----


== Export from query

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword
"call apoc.export.cypher.query('MATCH (n) OPTIONAL MATCH p = (n)-[r]-(m) RETURN n,r,m',
'yourPath/exportQuery.cypher', {format:'neo4j-shell', batchSize: 10000})"
----

=== Result

----
real    3m34.924s
user    0m0.992s
sys     0m0.068s
----

== Export from given nodes and rels

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword  "Match (n:Person)-[r:LIKES_COMMENT]->(c:Comment)
with collect(n) as colN, collect(c) as colC, collect(r) as colR
CALL apoc.export.cypher.data(colN+colC,colR, 'yourPath/exportData.cypher',{format:'plain'}) YIELD nodes, relationships
RETURN nodes, relationships"
----

=== Result

----
real    2m30.576s
user    0m6.264s
sys     0m0.372s
----

== Export from graph object

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword "CALL apoc.graph.fromDB('test',{})
yield graph CALL apoc.export.cypher.graph(graph, 'yourPath/exportGraph.cypher',null)
YIELD nodes, relationships
RETURN nodes, relationships"
----

=== Result

.no config options
----
real    4m50.006s
user    17m26.149s
sys     0m13.145s
----

.cypher shell

----
real    5m6.467s
user    19m14.328s
sys     0m11.821s
----

.batch size 1.000

----
real    4m57.598s
user    17m26.557s
sys     0m13.465s
----

== Import Schema file

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword
"call apoc.cypher.runSchemaFile('yourPath/exportPlain.schema.cypher')"
----

=== Result

----
real    0m0.683s
user    0m0.896s
sys     0m0.092s
----

== Import from file

=== runFile

The `runFile` procedure takes much time to import from files like the ones we've generated before.
These file has more than 19.000.000 rows.
It was created an issue about this https://github.com/neo4j-contrib/neo4j-apoc-procedures/issues/500[#500].

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword
"call apoc.cypher.runFile('yourPath/import/exportPlain.cypher')"
----

With this command we import not more than 10/15 nodes per second.

=== Import cypher-shell

[source,bash,subs=attributes]
----
time ./cypher-shell -u yourUsername -p yourPassword
< 'yourPath/import/exportCypherShell.cypher'
> 'yourPath/cypherShellOutput'
----

.Result

----
real    890m38.003s
user    43m34.935s
sys     23m10.951s
----

* imported nodes 3.158.994
* imported relationships 16.800.936

=== Import neo4j-shell

[source,bash,subs=attributes]
----
time ./neo4j-shell -u yourUsername -p yourPassword -file
< 'yourPath/import/exportNeo4jShell.cypher'
> 'yourPath/neo4jShellOutput'
----

We tried to import the DB via neo4j-shell, but after 24 hours it was still uploading.
We tried with a subset of the graph :

* 130.000 nodes
* 500.000 relationships

The result is that neo4j-shell is 7 times slower than the cypher-shell.

.Result

Cypher-shell

----
real    14m43.923s
user    1m1.448s
sys     0m48.556s
----

Neo4j-shell

----
real    98m54.617s
user    21m5.140s
sys     37m35.852s
----
