Using Neptune in distributed computing#
You can track run metadata from several processes, running on the same or different machines.
Neptune is fully compatible with distributed computing frameworks, such as Apache Spark. Neptune also provides some synchronization methods that will help you handle more sophisticated workflows:
wait()– wait for all the metadata tracking calls to finish before executing the call. See below for an example.
sync()– synchronize the local representation of the run or other object with the Neptune servers.
Neptune is optimized for rapid metadata tracking and does not perform API calls to Neptune servers if not needed. As such, Neptune stores its own local representation of the run structure and assumes that no other process is modifying the run at the same time in a conflicting way.
Examples of such conflicting cases would be:
- Using the same field name with a different field type, such as
- Removing or renaming variables or fields that are being used by another process.
In the case of a conflict, tracking methods will throw an exception and some of the tracked data may not be stored on Neptune servers.
In particular, you should respect the type of a field even within a simple script:
Like in parallel computing, to avoid potential race condition issues, we advise against modifying a variable from multiple processes.
- To learn more about field types, see Field types reference.
Fetching logged metadata from within the same script#
When working in asynchronous mode (default), the metadata you track is periodically synchronized with the servers in the background. Because of this, the data may not be immediately available to fetch from the server, even if it appears in the local representation.
To work around that, you can use the
wait() method mentioned above.
Rule of thumb
If you fetch metadata that you logged from within the same script, use
wait() on the object before querying: