= Job management and periodic execution

== Introduction asynchronous transactional execution

[NOTE]
this document is work in progress

Cypher is great for querying graphs and importing and updating graph structures.
While during imports you can use `PERIODIC COMMIT` to control transaction sizes in memory, 
for other graph refactorings it's not that easy to commit transactions regularly to free memory for new update state.

Also sometimes you want to schedule execution of Cypher statements to run regularly in the background or asynchronously ("fire & forget").

The `apoc.periodic.*` procedures provide such capabilities.

Many procedures run in the background or asynchronously. This setting overrides the default thread pool size (processors*2).

`apoc.jobs.default.num_threads=10`

Many periodic procedures rely on a scheduled executor that has a pool of threads with a default fixed size (processors/4, at least 1). You can configure the pool size using the following configuration property:

`apoc.jobs.scheduled.num_threads=10`

== apoc.periodic.iterate

With `apoc.periodic.iterate` you provide 2 statements, the *first* outer statement is providing a stream of values to be processed.
The *second*, inner statement processes *one* element at a time or with `iterateList:true` the whole batch at a time.

The results of the outer statement are passed into the inner statement as parameters, they are automatically made available with their names.

.configuration options
[options=header]
|===
| param | default | description
| batchSize | 1000 | that many inner statements are run within a single tx params: {_count, _batch}
| parallel | false | run inner statement in parallel, note that statements might deadlock
| retries | 0 | if the inner statement fails with an error, sleep 100ms and retry until retries-count is reached, param {_retry}
| iterateList | false | the inner statement is only executed once but the whole batchSize list is passed in as parameter {_batch}
| params | {} | externally passed in map of params
|===

NOTE: We plan to make `iterateList:true` the default in upcoming releases, due to the automatic UNWINDing and providing of nested results as variables,
most queries should continue work.

So if you were to add an `:Actor` label to several million `:Person` nodes, you would run:

[source,cypher]
----
CALL apoc.periodic.iterate(
"MATCH (p:Person) WHERE (p)-[:ACTED_IN]->() RETURN p",
"SET p:Actor", {batchSize:10000, parallel:true})
----

Which would take 10k people from the stream and update them in a single transaction, executing the *second* statement for each person.

Those executions can happen in parallel as updating node-labels or properties doesn't conflict.

If you do more complex operations like updating or removing relationships, either *don't use parallel* OR make sure that you batch the work in a way that each subgraph of data is updated in one operation, e.g. by transferring the root objects.
If you attempt complex operations, try to use e.g. `retries:3` to retry failed operations.

[source,cypher]
----
CALL apoc.periodic.iterate(
"MATCH (o:Order) WHERE o.date > '2016-10-13' RETURN o",
"MATCH (o)-[:HAS_ITEM]->(i) WITH o, sum(i.value) as value SET o.value = value", {batchSize:100, parallel:true})
----

.iterating over the whole batch (more efficient)
[source,cypher]
----
CALL apoc.periodic.iterate(
"MATCH (o:Order) WHERE o.date > '2016-10-13' RETURN o",
"MATCH (o)-[:HAS_ITEM]->(i) WITH o, sum(i.value) as value SET o.value = value", {batchSize:100, iterateList:true, parallel:true})
----

The stream of other data can also come from another source, like a different database, CSV or JSON file.


== apoc.periodic.commit

Especially for graph processing it is useful to run a query repeatedly in separate transactions until it doesn't process and generates any results anymore.
So you can iterate in batches over elements that don't fulfill a condition and update them so that they do afterwards.

The query is executed repatedly in separate transactions until it returns 0.

[source,cypher]
----
call apoc.periodic.commit("
match (user:User) WHERE exists( user.city )
with user limit {limit}
MERGE (city:City {name:user.city})
MERGE (user)-[:LIVES_IN]->(city)
REMOVE user.city
RETURN count(*)
",{limit:10000})
----

----
+=======+==========+
|updates|executions|
+=======+==========+
|2000000|200       |
+-------+----------+
----

== apoc.periodic.countdown
Repeats a statement until the termination is reached. The statement must return a numeric value and it should decrement (like a monotonically decreasing function). When the return value reaches 0 than the iteration stops.
For example, define a counter with a numeric property:

[source,cypher]
----
CREATE (counter:Counter) SET counter.c = 10
----

and decrement this property by 1 each second:

[source,cypher]
----
CALL apoc.periodic.countdown('decrement',"MATCH (counter:Counter) SET counter.c = counter.c - 1 RETURN counter.c as count", 1)
----


== Further Functions

// include::overview.adoc[tags=periodic]

[separator=¦,opts=header,cols="1,1m,5"]
|===
include::../build/generated-documentation/apoc.periodic.csv[]
|===

* there are also static methods `Jobs.submit`, and `Jobs.schedule` to be used from other procedures
* jobs list is checked / cleared every 10s for finished jobs

////

.copies over the `name` property of each person to `lastname`
[source,cypher]
----
CALL apoc.periodic.iterate('
match (p:Person) return p', 
'SET p.lastname=p.name', {batchSize:20000,parallel:true, iterateList:true})
----
////
