Troubleshooting

Logs are an essential first step in the troubleshooting process. To access the logs, run the following:

$ kubectl logs -n kube-system csi-provisioner-0 --since=10m
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, hs-csi-plugin-controller
I0820 14:59:44.061877 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
I0820 15:01:51.086764 1 reflector.go:536] sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:845: Watch close - *v1.PersistentVolume total 6 items received...

Review the Hammerspace plugin container specifically to find out more information.

$ kubectl logs -n kube-system csi-provisioner-0 -c hs-csi-plugin-controller --since=20m
{
"level": "error",
"msg": "Failed to set tag. Error - %!v(MISSING)exit status 1",
"time": "2024-08-20 14:50:28"
}
{
"level": "warning",
"msg": "failed to set additional metadata on share exit status 1",
"time": "2024-08-20 14:50:28"
}
{
"level": "info",
"msg": "Total time taken for create volume 13.905738725s",
"time": "2024-08-20 14:50:28"

Information can be correlated from these logs against a support bundle that can be sent to Hammerspace for review as needed. If there appears to be trouble mounting the share, ssh into the worker node the pod was scheduled on (if possible) and try to mount the share to that host. Mounted shares can be found in /var/lib/kubelet/<pod-id>/volumes/.

For a deeper dive, utilize k9s, a terminal based UI that aims to simplify the navigation, observation, and management of deployed applications. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with observed resources.