Neither Conservancy nor Christoph takes this action lightly nor without exhausting every other possible alternative first. This lawsuit is the outgrowth of years of effort to convince VMware to comply with GPL.
In October 2011, Conservancy received a GPL violation report on BusyBox for VMware's ESXi products. Conservancy opened the matter in its usual, friendly, and non-confrontational way. Nevertheless, VMware immediately referred Conservancy to VMware's outside legal counsel in the USA, and Conservancy negotiated with VMware's legal counsel throughout late 2011, 2012 and 2013. We exchanged and reviewed CCS candidates, and admittedly, VMware made substantial and good efforts toward compliance on BusyBox. However, VMware still refused to fix a few minor and one major compliance problem that we discovered during the process. Namely, there was a major violation regarding Linux itself that ultimately became Christoph's key complaint in this lawsuit.
Meanwhile, when Conservancy realized in late 2012 there might be a major Linux violation still present in VMware's ESXi products, Conservancy representatives sought every industry contact we had for assistance, including those from trade associations, companies (both competitors and collaborators with VMware), and everyone else we could think of who might be able to help us proceed with friendly negotiations that would achieve compliance. While we cannot name publicly the people we asked for help to convince VMware to comply, they include some of the most notable executives, diplomats, and engineering managers in the Linux community. No one was able to assist Conservancy in convincing VMware to comply with the GPL. Then, in early 2014, VMware's outside legal counsel in the USA finally took a clear and hard line with Conservancy stating that they would not comply with the GPL on Linux and argued (in our view, incorrectly) that they were already in compliance.
Conservancy in parallel informed Christoph fully of the details of the Linux violation on Christoph's copyrights, and based on Conservancy's findings, Christoph began his own investigation and confirmed Conservancy's compliance conclusions. Christoph then began his own enforcement effort with legal representation from Till Jaeger. Christoph has been unable to achieve compliance, either, through his negotiations in 2014. VMware's last offer was a proposal for a settlement agreement that VMware would only provide if Christoph signed an NDA, and Christoph chose (quite reasonably) not to sign an NDA merely to look at the settlement offer.
Thus, this lawsuit comes after years of negotiations by Conservancy to achieve compliance — negotiations that ended in an outright refusal by VMware's lawyers to comply. Those events were then followed by a year of work by Christoph and Till to achieve compliance in a separate action.
Simply put, Conservancy and Christoph fully exhausted every possible non-litigation strategy and tactic to convince VMware to do the right thing before filing this litigation.
Conservancy prepared this diagram to show the technical situation as we understand it. The diagram compares the technical architecture of a full, running Linux kernel with a full, running VMware kernel:
![[Diagram of Linux and VMware running kernels]](/linux-compliance/linux-vs-vmkernel_en.png)
If you want to download the diagram, it's available in SVG (English), PNG (English), SVG (German), and PNG (German).
The GPL violation at issue involves VMware's ESXi product. Conservancy independently reviewed ESXi 5.5 and its incomplete CCS release as part of our GPL enforcement efforts described above.
Conservancy's preliminary investigation indicated that the operating system kernel of VMware ESXi product consists of three key components:
- the proprietary component “vmkernel”, which is released in binary form only,
- the kernel module “vmklinux”, which contains modified Linux Code, and for which (at least some) source code for which is provided.
- other kernel modules with device drivers, most of which are modified Linux drivers, and for which (at least some) source code is provided.
Conservancy examined the incomplete CCS alongside the binary “vmkernel” component. Such examination indicates that function in “vmkernel” do make function calls to Linux's kernel code in the usual way for a single program written in C.
Many in the media have talked about the possibility that VMware might use some so-called “shim layer” between Linux code and VMware's proprietary code. While, for decades, there has been much talk of various mechanisms of GPL obligation avoidance, Conservancy believes that merely modifying technical details of a combination's construction does not typically influence the legal analysis in a combined or derivative work scenario.
Furthermore, the technical details of VMware's alleged GPL violation do not even mirror the typical scenarios that have usually been called “shim layers”. Conservancy's analysis of VMware's ESXi product, in fact, indicates that VMware rather flagrantly combined Linux code in their own kernel, and evidence seems to indicate the work as a whole was developed by modifying Linux code in tandem with modifications to “vmkernel” in a tightly coupled manner.
There are numerous examples available that show this. The details of alleged infringement specifically relating to Hellwig's contributions to Linux are of course the main matter of the allegations in the litigation, and Conservancy released the diagram above to exemplify that issue. Conservancy continues to hope VMware will agree to make public all court documents as a matter of public good, since the court documents discuss the specifics of alleged infringement on Hellwig's copyrights.
However, Conservancy examined VMware's ESXi 5.5 product in detail even before Hellwig's enforcement action began. Below is one example among many where VMware's CCS was incomplete per GPLv2§2(c) and GPLv2§3(a). (One can verify these results by downloading and installing the binary and source packages for VMware's ESXi 5.5 Update 2.) Note that this example below is not necessarily regarding Hellwig's copyrights; VMware incorporated Linux code copyrighted by many others as well into their kernel.
Example of “vmkernel”'s combination with Linux code
As one example, examine the file
called vmkdrivers/src_92/vmklinux_92/vmware/linux_pci.c,
which can be found in the “Open Source” release for
ESXi 5.5.0 Update 2 (5.5U2). A small excerpt from that file, found in the
function LinuxPCIDeviceRemoved(), reads as follows:
if (unlikely(
/* NOTE: vmk_PCIGetDeviceName is defined in vmvisor64-vmkernel */
vmk_PCIGetDeviceName(vmkDev, vmkDevName, sizeof(vmkDevName)-1) != VMK_OK))
{
vmkDevName[0] = 0;
}
[...]
/* VMKAPI_MODULE_CALL_VOID is a macro calling driver's remove() here */
VMKAPI_MODULE_CALL_VOID(pciDevExt->moduleID,
linuxDev->driver->remove,
linuxDev);
The function, vmk_PCIGetDeviceName() must be defined, with an
implementation, for this code above to work, or even compile.
Inside BLD/build/HEADERS/vmkapi-current-all-public/vmkernel64/release/device/vmkapi_pci_incompat.h,
found in the vmkdrivers package of ESXi 5.5U2, shows a
function header definition for vmk_PCIGetDeviceName().
However, the source of its implementation is not provided there or
anywhere in the source release.
Further evidence that the implementation of this function occurs elsewhere
can by found by running objdump -x on the un-vmtar'ed
vmklinux_9 module. Note the following output in the “SYMBOL
TABLE” section:
0000000000000000 *UND* 0000000000000000 vmk_PCIGetDeviceName…and the following lines found in the “RELOCATION RECORDS FOR [.text]” section:
00000000000327ff R_X86_64_PC32 vmk_PCIGetDeviceName+0xfffffffffffffffc 0000000000035318 R_X86_64_PC32 vmk_PCIGetDeviceName+0xfffffffffffffffc 00000000000387e1 R_X86_64_PC32 vmk_PCIGetDeviceName+0xfffffffffffffffc 000000000003cf40 R_X86_64_PC32 vmk_PCIGetDeviceName+0xfffffffffffffffc
The above two properties both suggest that the vmklinux_9
module requires: (a) a definition of the vmk_PCIGetDeviceName()
function to operate, but (b) that function is not defined
inside vmklinux_9 itself.
The definition can however be found in binary-only software provided in
ESXi 5.5U2 — specifically, inside a file named k.b00,
which is located in partition 5 on a disk where ESXi has been installed (or
in the ESXi 5.5U2 installer ISO image). Running file
after gunzip on this file yields “ELF 64-bit LSB shared
object”. Meanwhile, file k.b00 reports “gzip
compressed data, was ‘vmvisor64-vmkernel.stripped’”.
These findings strongly suggests this is an image of the
“vmkernel” component. An objdump -x yields this
“SYMBOL TABLE” section:
000041800036a408 g F .text 0000000000000137 vmk_PCIGetDeviceName… which indicated these binary file contains the function body for
vmk_PCIGetDeviceName.
Finally, after detailed searching, Conservancy found no evidence that any
other code (other than modified Linux code) makes calls
to vmk_PCIGetDeviceName. This provides a strong indication
that this function's primary purpose is to combine Linux code with
“vmkernel”. Conservancy also found other functions where similar analysis
yields similar results as above.
Given this evidence and related contextual clues, the only logical conclusions are:
vmklinux_9, as a binary object, dynamically links withk.b00, another binary object, to form a single running binary.- That binary contains code licensed under the GPLv2, and can be distributed in binary form only under permissions provided under GPLv2 — in particular GPLv2§2 and GPLv2§3.
- GPLv2§3(a–b) requires that
complete corresponding machine-readable source code
must accompany binary distributions such as these. GPLv2§3 further states thatfor an executable work, complete source code means all the source code for all modules it contains
. - The binary work in question contains modules from
k.b00andvmlinux_9. - VMware did not provide source code for any modules found in
k.b00. - Therefore, VMware failed to comply with the GPLv2, as such
compliance requires source code (or an offer therefor) for the material
in
k.b00.
The above is but one piece of evidence among many, but hopefully it helps to explain the types of “combined work” violations found in VMware's ESXi product.
The binary and source packages mentioned above are available
on VMware's website. These packages contain the
previously-mentioned linux_pci.c,
vmkapi_pci_incompat.h, and k.b00 files, as well as
vmklinux_9 and the source code that builds the latter.
To obtain the source components, follow these steps (no login is required):
- Visit https://my.vmware.com/web/vmware/details?downloadGroup=ESXI55U2_OSS&productId=353.
- Click the “Download” button beside the text that reads “Open Source Code for VMware vSphere ESXi 5.5 Update 2”.
- Confirm that the SHA-1 hash matches the published one (d121634668a137ec808b63679fd941cef9a59715), found under “Read More” on that web page.
- Mount (or otherwise open) the
downloaded
VMware-ESX-550U2-ODP.iso. - Extract
vmkdrivers/src_92/vmklinux_92/vmware/linux_pci.candBLD/build/HEADERS/vmkapi-current-all-public/vmkernel64/release/device/vmkapi_pci_incompat.hfromvmkdrivers-gpl/vmkdrivers-gpl.tgzwith tar and gzip. - Generate
vmklinux_9by following the steps invmkdrivers-gpl/BUILD.txtin the ISO. (Note:vmklinux_9is also available pre-built on a running ESXi system; see below for instructions on how to access it). - You may need the “Supporting Toolchain packages for VMware
vSphere ESXi 5.5.0 Update 2” file from the above download page to
complete the build — upon downloading you will find it is named
VMware-TOOLCHAIN-550u2-ODP.isoand has a SHA-1 hash of f679e81ffb2f92729917bbc64c2d541cf75b5b94.
To obtain the binary components, follow these steps (a login is required):
- Register for an account at https://my.vmware.com/web/vmware/registration.
- Click the “Activate Now” link in the follow-up email. Enter the password used at registration time. Click “Continue”.
- Visit https://my.vmware.com/web/vmware/evalcenter?p=free-esxi5.
- Click “Register” (under the text that reads “You have not registered for this product”).
- Enter the number of servers you plan to install on (e.g., 1). Click “Continue”.
- If the “VMware vSphere Hypervisor 5.5 Update 2 – Binaries” section is not expanded, click the plus sign next to it.
- Click the “Manually Download” link that's beside “ESXi 5.5 Update 2 ISO image (Includes VMware Tools)”.
- Confirm that the SHA-1 hash matches the published one (9475938b51cafc86c8b17d09f2493cb6b4fae927).
- Mount (or open via some other means) the
downloaded
VMware-VMvisor-Installer-5.5.0.update02-2068190.x86_64.iso. - Find the
k.b00file in the root directory. Extract it usingzcat k.b00 > vmvisor64-vmkernel(or a similar command). Repeat the steps described above usingobjdump -x vmvisor64-vmkernel. - To retrieve
vmklinux_9you will need to install ESXi on your system by booting the ISO and following the instructions. Once booted, you can then enable SSH access using “Customize System/View Logs -> Troubleshooting Options -> Enable SSH”. Login to the system with SSH and then runfind /vmfs -name misc_dri.v00 -print. On the resulting file, runzcat misc_dri.v00 > misc_dri.vmtarthenvmtar -x misc_dri.vmtar -o misc_dri.tar. You can then extractmisc_dri.tarusing the usualtarto extractusr/lib/vmware/vmkmod/vmklinux_9. Themisc_dri.v00file is also available next tok.b00in the root directory of the ISO (mentioned above), but thevmtarcommand itself is only available when logged into an ESXi system.vmtarcan be found atbin/vmtarinsidesb.v00on the ISO, but one needsvmtarto opensb.v00, similar tomisc_dri.v00above.
Note that VMware may present you with EULAs and ToS when you download software from VMware's website. Conservancy strongly suggests that you review these terms in great detail with the assistance of your own legal counsel before downloading the software and/or engaging in the process that Conservancy discusses above.