You can track data of your run from many processes, in particular running on different machines.

Neptune is fully compatible with distributed computing frameworks like e.g. Apache Spark, so you can use them to synchronize your processes. On top of that, Neptune provides some synchronization methods that will help you handle more sophisticated workflows. In particular, you might want to use wait() and sync().

Like in parallel computing, it is advised not to modify one variable from many processes to avoid potential race condition issues.

Local representation

Neptune is optimized for rapid metadata tracking and does not perform API calls to Neptune servers if not needed. That’s why Neptune stores its own local representation of the run structure and assumes that no other process is modifying the run at the same time in a conflicting way, like:

• using the same variable name with a different variable type e.g. output: File vs output: StringSeries;

• removing/renaming variables that are being used by another process.

In case of conflict, tracking methods will throw an exception and some of the tracked data may not be stored on Neptune servers.

In particular, you should respect a variable’s type even within a simple script:

run["learning_rate"] = 0.5
run["learning_rate"] = "let’s try a string"
# error, Float variable cannot be assigned with a str.
​
run["accuracy"].log(0.95)
run["accuracy"] = 0.99
# error, FloatSeries does not support = operator. Use log() instead.