Preparing a Kubernetes cluster for Lumerical jobs

This guide discusses steps for enabling Lumerical job submissions on an existing Kubernetes cluster. Familiarity with kubectl and a working cluster are assumed.

Lumerical submits simulations to Kubernetes as Kubeflow `MPIJob` resources. To accept those jobs, your cluster needs four things:

1. The Kubeflow MPI Operator
2. A namespace with a small RBAC role
3. A ReadWriteMany PersistentVolumeClaim for simulation files
4. A container image of the Lumerical solver in a registry the cluster can pull from

When those are in place, hand a kubeconfig and a few values to your Lumerical user and they can submit jobs from the CAD UI.

1. Install the MPI Operator

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.5.0/deploy/v2beta1/mpi-operator.yaml

kubectl wait --for=condition=available \
  deployment/mpi-operator -n mpi-operator --timeout=120s

If the operator is already installed at `v0.5.0` or later, skip this step.

2. Create a namespace and grant access

Create the namespace:

kubectl create namespace lumerical-jobs

Apply this role (`lumerical-role.yaml`):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: lumerical-client
  namespace: lumerical-jobs
rules:
  - apiGroups: ["kubeflow.org"]
    resources: ["mpijobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "pods/log",
                "persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "create", "delete"]

kubectl apply -f lumerical-role.yaml

Bind the role to the user, group, or ServiceAccount your Lumerical user will authenticate as. For example, with a ServiceAccount:

kubectl create serviceaccount lumerical-client -n lumerical-jobs
kubectl create rolebinding lumerical-client \
  --role=lumerical-client \
  --serviceaccount=lumerical-jobs:lumerical-client \
  -n lumerical-jobs

3. Provision shared storage

Lumerical needs a PersistentVolumeClaim named lumerical-simfiles in the lumerical-jobs namespace, with ReadWriteMany access mode. The launcher pod, worker pods, and a small file-staging pod mount it concurrently at Lumerical.

The yaml script to set this is seen below.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lumerical-simfiles
  namespace: lumerical-jobs
spec:
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 100Gi
  storageClassName: <your-rwx-storage-class>

Bash commands in addition to the yaml script are as follows

kubectl apply -f lumerical-pvc.yaml
kubectl get pvc lumerical-simfiles -n lumerical-jobs   # must show Bound

Size the PVC for the largest expected simulation's input plus results.

4. Build and publish the Lumerical container image

Lumerical provides two DockerFiles in the product distribution under installer/linux/docker/:

• Dockerfile-rocky-9-openmpi — base image (Rocky 9 + OpenMPI + the Lumerical RPM)
• Dockerfile-rocky-9-openmpi-ssh — adds the `sshd` the MPI Operator requires

Build both and push the `-ssh` image to a registry your cluster can pull from:

cd <lumerical-product>/installer/linux/docker
cp -r <path-to-Lumerical-rpm>/rpm_install_files .

docker buildx build -t <registry>/lumerical-openmpi:<tag> \
  -f Dockerfile-rocky-9-openmpi .

docker buildx build -t <registry>/lumerical-openmpi-ssh:<tag> \
  -f Dockerfile-rocky-9-openmpi-ssh \
  --build-arg BASE_IMAGE=<registry>/lumerical-openmpi:<tag> .

docker push <registry>/lumerical-openmpi-ssh:<tag>

If your registry requires authentication, configure pull credentials on the namespace's default ServiceAccount:

kubectl create secret docker-registry regcred \
  --docker-server=<registry> --docker-username=<u> --docker-password=<p> \
  -n lumerical-jobs
kubectl patch serviceaccount default -n lumerical-jobs \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

5. Hand off the configuration

Give your Lumerical user:

• A kubeconfig file that authenticates as the identity bound in step 2
• The values for these `job_scheduler_input.json` keys:

The user then follows Lumerical job scheduler integration (Kubernetes) to apply these on their workstation.

Verification

To verify your cluster is ready for job submission, run the following as the bound identity:

kubectl auth can-i create mpijobs.kubeflow.org -n lumerical-jobs   # yes
kubectl auth can-i create pods/exec             -n lumerical-jobs   # yes
kubectl get pvc lumerical-simfiles -n lumerical-jobs                # Bound
kubectl get crd mpijobs.kubeflow.org                                # exists

If all four pass, the cluster is ready.