Skip to main content

Storage Volumes

How to set up a persistent volume with Kubeflow​

  1. Open the Kubeflow Central Dashboard and select a namespace to work on.

  2. Click Volumes in the side panel.

  3. Click + New Volume

  4. In the New Volume settings, select the following parameters:

    • Name: enter any name
    • Volume size in Gi: 10 (or as needed)
    • Storage Class: standard
    • AccessMode: ReadWriteMany

Kubeflow volume settings New volume settings

  1. Click Create.

How to set up a shared persistent volume​

Use a shared persistent volume if multiple team members are running separate notebooks in Kubeflow and using the same files in a dataset.

First steps​

Before starting to set up the volume, contact the AI Platform team and ask them to provision the necessary resources and configure them in the Azure subscription. You must supply them with the following information:

  • Name of the project. For example, Deli.
  • List of the full usernames or Azure Active Directory Group that should have access to the storage. For example: haij@equinor.com, clbi@equinor.com, etim@equinor.com or Aurora Developers.
  • If you own an AI Platform cluster, please supply the name of the cluster or public endpoint. For example: deli12, or deli.dev.aurora.equinor.com. If you do not have a working cluster yet, one will be provisioned for you.
info
  • These steps only need to be carried out once. All new AI Platform clusters for this project will use the same configuration.
  • For each user present in the provided list or Azure Active Directory Group, a Kubeflow profile will be created in the AI Platform cluster (See the page on multi-user isolation for more information about namespaces/profiles).

Best practices​

  1. Teams should setup "shared-data-volume" (with "standard" storage type and "ReadWriteMany" access) under their shared project profile. This volume can be then mounted with multiple notebook servers by multiple people so that stored data can be shared among the team.
  2. On the shared data volume, create two folders: code and data.
    1. Under the code folder, create folders with each team member's Equinor short name. This is where each member can clone their GitHub repos and work on their own code once they mount the volume with their own notebook servers.
    2. Shared team data can be stored under data folder. The directory structure can be determined by the team based on their needs.

How to attach a shared volume to your notebook​

To attach a shared volume to your notebook, follow these steps:

  1. Open the AI Platform site.
  2. Manually create a notebook server.
    note

    For details on how to manually create a notebook server, go to Browser-Based Notebooks (JupyterLab, VS Code, RStudio).

  3. In the Data Volumes section, click + Add exiting volume and set the following:
  • a.) Type: choose the Existing option
  • b.) Name: use the <PROJECT_NAME>-<SHORT_USERNAME>-azurefile name structure, for example deli-clbi-azurefile. Project and usernames must be in lowercase letters, the username does not include email extension, i.e., @equinor.com.
  • c.) Mount Point: can be any value but it should be a path agreed upon by your team.

Data volumes settings Data volumes settings

  1. Click LAUNCH. When the notebook server provisioning is complete, you should see an entry for your server on the Notebook Servers page.
  2. Click CONNECT to start the notebook server.
  3. After connecting to the notebook server, you should see the Jupyter dashboard interface and shared storage should be available under the Mount Point provided above.

How to set up a volume with AzureMount​

AzureMount is a custom component that has been developed in house for Equinor.

This component is used to create a volume that is linked to either an AzureFile or AzureBlob store.

note

Microsoft documentation:

To create a link to your file share, do the following:

  1. Get the Azure storage account's details.

    1. Open the Azure Portal.
    2. Go to Storage account > Security + networking > Access keys.
    3. Copy key and storage account name for use in the following steps.

    AccessKey Azure storage account access keys

  2. In the Kubeflow UI main menu, go to the AzureMount component.

  3. Fill in the details with the following information:

    1. Storage type: choose either Azure Blob or Azure File
    2. Share / Container name: container or share name within the storage account
    3. Name: Azure storage account name
    4. Key: access key obtained previously

    AzureMount AzureMount component in Kubeflow