Distributed computing

You can track data of your run from many processes, in particular running on different machines.
Neptune is fully compatible with distributed computing frameworks like e.g. Apache Spark, so you can use them to synchronize your processes. On top of that, Neptune provides some synchronization methods that will help you handle more sophisticated workflows. In particular, you might want to use wait() and sync().
Like in parallel computing, it is advised not to modify one variable from many processes to avoid potential race condition issues.
Local representation
Neptune is optimized for rapid metadata tracking and does not perform API calls to Neptune servers if not needed. That’s why Neptune stores its own local representation of the run structure and assumes that no other process is modifying the run at the same time in a conflicting way, like:
using the same variable name with a different variable type e.g. output: File vs output: StringSeries;
removing/renaming variables that are being used by another process.
In case of conflict, tracking methods will throw an exception and some of the tracked data may not be stored on Neptune servers.
In particular, you should respect a variable’s type even within a simple script:
1
run["learning_rate"] = 0.5
2
run["learning_rate"] = "let’s try a string"
3
# error, Float variable cannot be assigned with a str.
4
​
5
run["accuracy"].log(0.95)
6
run["accuracy"] = 0.99
7
# error, FloatSeries does not support = operator. Use log() instead.
Copied!
When working in the asynchronous (default) mode remember that metadata you track may not be available immediately to fetch from the server even if it appears in the local representation. In order to prevent that you can use .wait(). 
As a rule of thumb if you are fetching metadata that you logged from within the same script always use .wait():
1
run["learning_rate"] = 0.5
2
​
3
# Always use wait() when fetching metada you tracked in the same script
4
run.wait()
5
learning_rate = run["learning_rate"].fetch()
Copied!

Log to multiple runs in one script

Parallel computing

Last modified 13d ago

Export as PDF

Copy link

Contents

Local representation