How to create/commit a new image from the running Docker? I want to keep those changes on the Docker.

If you have made those Conda, Python and CUDA Toolkit changes on the nvidia/cuda Docker, you should follow this tutorial to create/commit a new custom docker image.

1. Verify the running nvidia/cuda Docker.

fyicenter# docker ps 

CONTAINER ID   IMAGE                   COMMAND            STATUS        NAMES
e4395c83ac54   nvidia/cuda:11.0-base   "sleep 36000000"   Up 20 hours   my_cuda

2. Commit/Create a new Docker image, fyi_cuda, from the running Docker.

fyicenter# docker commit my_cuda fyi_cuda
  sha256:...

fyicenter# docker images
  REPOSITORY    TAG         IMAGE ID       CREATED         SIZE
  fyi_cuda      latest      1a4cb45485ee   4 minutes ago   3.91GB
  nvidia/cuda   11.0-base   2ec708416bb8   12 months ago   122MB

3. Rename the new Docker image to fyi/cuda:1.

fyicenter# docker image tag 1a4cb45485ee fyi/cuda:1

fyicenter# docker images
  REPOSITORY    TAG         IMAGE ID       CREATED         SIZE
  fyi_cuda      latest      1a4cb45485ee   4 minutes ago   3.91GB
  fyi/cuda      1           1a4cb45485ee   10 minutes ago  3.91GB
  nvidia/cuda   11.0-base   2ec708416bb8   12 months ago   122MB

fyicenter# docker image remove fyi_cuda

fyicenter# docker images
  REPOSITORY    TAG         IMAGE ID       CREATED         SIZE
  fyi/cuda      1           1a4cb45485ee   10 minutes ago  3.91GB
  nvidia/cuda   11.0-base   2ec708416bb8   12 months ago   122MB

3. Start, fyi/cuda:1 as a new Docker container.

fyicenter# docker run --name my_cuda_2 --detach fyi/cuda:1  sleep 36000000

fyicenter# docker ps 
CONTAINER ID   IMAGE                   COMMAND            STATUS         NAMES
2239e7c983a4   fyi/cuda:1              "sleep 36000000"   Up 9 seconds   my_cuda_2
e4395c83ac54   nvidia/cuda:11.0-base   "sleep 36000000"   Up 20 hours    my_cuda

4. Verify the new Docker container.

fyicenter# docker exec -it my_cuda_2 /bin/bash 

(base) root@2239e7c983a4:/# conda activate py3.9

(py3.9) root@2239e7c983a4:/# python --version 
  Python 3.9.6

5. Stop and remove the old Docker container to free up resource.

fyicenter# docker stop my_cuda
  my_cuda

fyicenter# docker ps --all
CONTAINER ID   IMAGE                   COMMAND           STATUS         NAMES
2239e7c983a4   my/cuda:1               "sleep 36000000"  Up 5 minutes   my_cuda_2
e4395c83ac54   nvidia/cuda:11.0-base   "sleep 36000000"  Exited (137)   my_cuda

fyicenter# docker container rm my_cuda
my_cuda

CONTAINER ID   IMAGE       COMMAND            STATUS         NAMES
2239e7c983a4   my/cuda:1   "sleep 36000000"   Up 9 minutes   my_cuda_2

⇒ Export and Import Docker Image Files

⇐ Install CUDA Toolkit in Conda on nvidia/cuda Docker

⇑ NVIDIA GPU Server Docker Image

⇑⇑ Docker Container Platform - Tutorials