Photon OS on vCloud Director 8.10 with Docker volume driver for vSphere

Today my colleague and I had a discussion if it would be possible to build a Docker cluster at our service provider running vCloud Director 8.10. We gave it a try and were successful. So how could this be achieved.

Photon OS Installation on vCloud Director 8.10

First thing you need to do is to download the OVA File for Photon OS. The OVA File (Photon OS, Version 1.0 — OVA with virtual hardware v11) could be obtained here : https://vmware.github.io/photon/

Please be sure to run vCloud Director 8.10 to use hardware version 11 otherwise download hardware version 10.

Afterwards login to your vCloud Director Instance and use the “Add vApp from OVF” function under “My Cloud”.

Follow the dialog and choose your VDC, your storage policy and a network with DHCP or a vCloud Director backed static IP-Pool network. Choose 2 vCPUs because this is the minimum requirement for Photon OS and for our test 2 GB RAM should also be enough. The disk size does not matter for our test so you could choose any size here.

Wait until the import was successful.  Now start the vApp.

Under Virtual Machines find the allocated IP-Address and connect to it via any SSH client. Username is root and initial password is “changeme”. First thing that needs to be done is to change the password. Please take care that no dictionary word is being used.

Docker is not initially started and enabled. To do this we have to run the following commands:

systemctl start docker
systemctl enable docker

The detailed procedure could be found here:
https://github.com/vmware/photon/wiki/Running-Photon-OS-on-vSphere

Docker Volume Driver for vSphere/vCloud Installation

The Docker Volume Driver consists of two parts. The driver inside the Photon OS Instance and the VIB driver on the vCloud Director ESXi Hosts. Only when both components are installed we will be able to provide persistent storage for our docker containers.

To install the docker driver in the Photon OS machine login to the machine via ssh and just run the following command:

rpm -Uvh https://github.com/vmware/docker-volume-vsphere/releases/download/0.7/docker-volume-vsphere-0.7-1.x86_64.rpm

The output should look like this:

If you run the docker volume ls the output should be empty. With this we installed the volume driver on the Photon OS.

The second part is to install the driver for the vCloud Director ESXi Hosts. This part could be a bit tricky if you depend on a provider like we do. But after a short discussion and explanation the provider was ready to help us out with this. The installation is quite easy.

First you need to download the VIB File. The file could be obtained here:
https://github.com/vmware/docker-volume-vsphere/releases

Second step is to upload the VIB file to your ESXi Host TMP directory.
scp vmware-esx-vmdkops-0.7.vib root@esxi01.local:/tmp
You can use SCP or any other method to copy the file to you ESXi host.

Afterwards you need to install the VIB to the host. No reboot is required for this operation. Connect via SSH to your ESXi host and run the following command:
esxcli software vib install --no-sig-check -v /tmp/vmware-esx-vmdkops-0.7.vib

The output should look like this:

Now we are able to see which disks have already been created via docker volume driver. Just run this command:
/usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls

In the beginning the output will be empty. But if we have created some docker volumes the output should look similar to this:

The installation could also be done via “Offline Depot” or “vSphere Update Manager”. Detailed information about the process could be found here:
https://vmware.github.io/docker-volume-vsphere/user-guide/install/

Now that we prepared the infrastructure we should be able to create shared docker volumes.

Docker Volume creation

So let us create our first vSphere/vCloud backed Docker volume.

docker volume create --driver=vmdk --name=torben.hamburg.volume -o size=10gb

A lot more parameters are possible here like: policy, diskformat, access, fstype you will find a lot of examples on this page:
https://vmware.github.io/docker-volume-vsphere/user-guide/docker-volume-cli/

With docker volume ls we are now able to see our newly created volume.

On the vSphere / vCloud site the VMDK will be created on the same datastore the Photon OS is running. You could also adjust this by setting the policy parameter when creating the disk. The VMDK is stored underneath the folder /dockvols

We created our first volume which we would like to attach to a docker instance.

Docker Volume usage

Now let us start a docker instance and mount the newly created volume into it. We can simply doing this by running the following command:

docker run --rm -it -v torben.hamburg.volume:/mnt/myvol busybox

This will start a busybox docker image and mount the volume. Afterwards we are able to consume the volume we created. The output for the docker run command should look like this:

The magic that happens in the background is that the Photon OS VM is mounting the VMDK to its virtual machine.

If you are able to logon with SSH to the ESXi Host and run the previously described vmdkops admin python script you are also able to see the mount status on the ESXi Host.

Conclusion

With those techniques described above you are now be able to setup a highly available docker cluster based on vSphere/vCloud. For vCloud you need to be able to install the VIB to your hosts. Our provider was very helpful with this. I will dive deeper into the setup of the cluster in my next article.