Jupyter Notebook Access

Requirements

Setting Up

During simulation runtime, libyt will activate libyt Jupyter kernel (libyt kernel). We need another process to start Jupyter Notebook/JupyterLab and connect to libyt kernel.

Starting Jupyter Notebook/JupyterLab

The process of starting Jupyter Notebook/JupyterLab and running simulation are separate and independent.

  1. Get jupyter_libyt and jupyter-client.

  2. After installing libyt, add <libyt-install-prefix>/share/jupyter to JUPYTER_PATH.

    export JUPYTER_PATH=<libyt-install-prefix>/share/jupyter:$JUPYTER_PATH

    Check if libyt_kernel is listed:

    jupyter kernelspec list

  3. Export environment variable LIBYT_KERNEL_INFO_DIR to where the simulation executable directory is.

    export LIBYT_KERNEL_INFO_DIR=<path-to-simulation-dir>

    Jupyter Notebook / JupyterLab will grab the connection information in LIBYT_KERNEL_INFO_DIR environment variable.

  4. Launch Jupyter Notebook/JupyterLab

    jupyter notebook # launch Jupyter Notebook
jupyter lab      # launch JupyterLab

  5. Click Libyt to connect to libyt kernel once the simulation is running and libyt kernel is activated.

Running Simulation

  1. Compile libyt in jupyter kernel mode (-DJUPYTER_KERNEL=ON).

  2. Call libyt API yt_run_JupyterKernel. When flag file is detected, libyt will activate libyt kernel.

  3. libyt API yt_run_JupyterKernel returns YT_SUCCESS after libyt kernel shuts down (see How to Exit).

  4. Simulation can continue its process.

Example

Using Jupyter Notebook/JupyterLab

Basics

Python Prompt and libyt Defined Commands

The cell takes Python statements and libyt Defined Commands. It can also import other Python modules.

Put Python statements and libyt defined commands in separate cells.

How does it execute Python statements and libyt defined commands?

  1. Takes user inputs in the cell.

  2. MPI root process broadcasts the inputs to other processes.

  3. Every MPI process executes the same piece of input synchronously.

  4. Get outputs from each process and print feedbacks on the screen.

Changes is kept and maintained in user’s inline script’s namespace, and it will be brought to the following round of in situ analysis.

Auto-Completion

Use TAB key to do auto-completion.

jedi must be installed to use this feature. Generally, if you have IPython installed, you probably already have jedi.

User Interface

This is the same as traditional Jupyter Notebook UI. Be careful when sending kernel related instructions (ex: shutdown/interrupt/restart a kernel), since the kernel is managed by libyt API.

If the notebook is connected to libyt kernel, restarting only exits the kernel, and libyt API yt_run_JupyterKernel returns. It won’t restart another kernel by itself.

How to Exit

Go to “Running Terminals and Kernels” “Shutdown Kernel”.

“Interrupt the kernel” button () won’t have any effect.

Reconnecting to libyt Jupyter Kernel

After exiting the kernel, the simulation process may continue its job. While waiting for a new libyt kernel being launched, we don’t need to close the entire notebook to reconnect to it. We can press “Restart” button () to reconnect to the new libyt kernel.

Make sure you already disconnect libyt kernel before reconnecting. Otherwise, libyt kernel will simply shut down the kernel, and your simulation will continue. There will be a pop-up window asking “if you want to restart” if you are already connected to a libyt kernel.

Known Limitations

  • The functionality is limited to taking Python inputs and printing outputs from all the MPI processes only. libyt hasn’t done implementing Jupyter’s full feature, like data streaming, supporting ipwidgets yet.

  • See Limitations in MPI Related Python Tasks.