Virtualization

Threads can be understood as virtual CPUs.

• Abstraction:
  • a thread looks like a CPU executing a sequence of instructions.
• Physical resource:
  • an actual core or hardware thread.
• Mechanism:
  • the operating system scheduler maps many threads onto fewer physical cores.

This allows:

• more threads than physical cores;
• a thread to run on different physical cores over its lifetime;
• time-multiplexing of CPU resources.

Virtual memory is another canonical example.

• Abstraction:
  • byte-addressable memory.
• Physical resource:
  • real semiconductor memory.

By hiding the real physical memory location, the OS can:

• make a virtual page not physically present at all;
  • this enables paging to disk;
• transparently change the physical page backing a virtual address;
  • this enables mechanisms such as copy-on-write.

Operating systems already contain many virtualized abstractions:

• sockets and pipes as virtual communication links;
• RAID volumes as virtual disks;
• virtual file systems;
• virtual networks.

Other examples include:

• database virtualization;
• VPNs;
• VLANs;
• Java Virtual Machine;
• Android emulators.

Multiplexing takes one physical resource and exposes it as multiple virtual entities.

Each virtual entity appears to have the full resource.

Multiplexing can happen:

• in time;
  • example: CPU scheduling;
• in space;
  • example: virtual memory address spaces;
• in both time and space.

Hardware support is often needed for efficient multiplexing.

Examples:

• virtual memory uses the MMU;
• CPU multiplexing needs timer interrupts and trap mechanisms;
• registers must be saved and restored across traps/context switches.

Aggregation takes multiple physical resources and combines them into one virtual resource.

The virtual resource often has better properties than a single physical resource.

Examples of improved properties:

• more capacity;
• better availability;
• redundancy;
• sometimes better performance.

The canonical example is RAID.

Several physical disks are combined to present one virtual disk. Depending on the RAID level, the virtual disk can provide higher capacity, fault tolerance, or faster I/O.

Emulation uses software to make one resource appear as a different kind of resource.

This is especially useful for:

• backward compatibility;
• development and debugging;
• replacing old hardware without changing old software.

Examples:

• RAM disk:
  • physical resource: memory;
  • virtual resource: disk;
  • very fast, but volatile;
  • useful for experiments where disk I/O should not be the bottleneck.
• Virtual tape:
  • physical resource: disk;
  • virtual resource: old tape-drive interface;
  • useful when old backup software expects a tape drive.
• Java Virtual Machine:
  • physical resource: x86, ARM, or another real CPU;
  • virtual resource: Java bytecode processor;
  • allows Java bytecode to run on many real architectures.
• Android emulator:
  • lets developers run/debug Android applications on a laptop or desktop without needing the physical device.

The VM should behave like a real machine.

The guest OS should be able to run as if it were installed directly on physical hardware.

Multiple VMs should execute without interfering with one another.

A crash or bug inside one VM should not crash other VMs or the host.

Isolation is also important for security, especially in cloud systems where different customers may share the same physical machine.

The VM should run close to real hardware speed.

If every operation were hundreds of times slower, VMs would be impractical.

Therefore, high-performance virtualization tries to execute ordinary code directly on the CPU whenever possible.

VMs allow one physical machine to run:

• multiple copies of the same OS;
• different versions of the same OS;
• completely different operating systems.

This is useful for testing, compatibility, and consolidation.

Because the VM abstracts away the real hardware, the system image is less tied to a specific physical machine.

This enables:

• prototyping new hardware;
• running old/legacy applications;
• moving a system image to another machine.

A VM can isolate:

• OS crashes;
• buggy applications;
• malicious software;
• workloads belonging to different users or organizations.

A crash inside one VM should normally only affect that VM.

The VMM can allocate resources to VMs and potentially provide performance guarantees.

For example, a cloud provider can give a VM a certain amount of CPU time, memory, or I/O capacity.

Multiple virtual servers can run on the same physical server.

This is a major idea behind cloud computing.

Instead of dedicating one physical machine to one service, many isolated virtual machines can share a large physical server.

A VM can be moved from one physical machine to another.

This is useful for:

• hardware maintenance;
• load balancing;
• redistributing resources;
• avoiding downtime when physical hardware needs repair.

VMs can be snapshotted and copied.

This allows:

• restoring to a known safe state;
• testing software changes;
• debugging failures;
• quickly cloning a preconfigured system.

VMs are very useful for OS development.

If a kernel crashes inside a VM, it does not crash the physical machine.

The lecturer also mentioned older systems such as DragonFly BSD, where some kernel-related code could be tested in user space, making debugging easier.

VMs and containers later became the more common tools for this kind of safe experimentation.

A Type I VMM runs directly on the bare metal.

It is itself the host operating system.

It is also called a native hypervisor.

Examples:

• Xen;
• VMware ESXi / vSphere;
• Microsoft Hyper-V.

Type I VMMs are common on servers, where the main purpose of the machine is to run VMs.

Advantages:

• often faster;
• direct control over hardware;
• no separate general-purpose host OS in the way.

Disadvantages:

• more complex;
• must include its own device drivers;
• must implement many OS-like functions itself.

A Type II VMM runs on top of an existing host OS.

It leverages the host OS for services such as:

• file systems;
• networking;
• device drivers;
• process management.

Examples:

• KVM on Linux;
• VMware Workstation;
• VMware Fusion;
• Parallels.

KVM blurs the distinction because it is a Linux kernel module that turns Linux itself into a hypervisor-like system, but for this lecture it is treated as Type II.

Advantages:

• easier to implement;
• can reuse host OS functionality;
• convenient on desktops and laptops.

Disadvantages:

• more overhead;
• the host OS may receive interrupts first and then forward events to the VMM;
• performance depends on the host OS and workload.

The virtual hardware must be sufficiently equivalent to the real hardware.

The same software that runs on the real machine should be able to run in the VM.

In full virtualization, this means an unmodified guest OS can run inside the VM.

The VM must be isolated from:

• other VMs;
• the VMM itself.

The guest should behave as if it has its own dedicated hardware, even though the physical machine is shared.

The overhead must be low enough that the VM is usable like a real machine.

Virtualization must not make ordinary execution unacceptably slow.

The CPU has a mode bit indicating whether the current code runs in:

• kernel mode;
• user mode.

Kernel mode is privileged and reserved for the OS.

User mode is for applications.

Virtual memory adds a level of indirection between:

• virtual addresses used by software;
• physical memory addresses.

This protects OS memory from applications.

Without virtual memory protection, a user program could overwrite kernel memory.

The CPU/OS must support traps and returns from traps.

A trap switches control from user mode to kernel mode.

The OS handles the event and later returns to user mode.

System calls, exceptions, and page faults all rely on this basic idea.

An interpreter simulates the target CPU instruction by instruction.

For each guest instruction, the interpreter:

1. reads the instruction;
2. decodes what it means;
3. simulates its effects in software;
4. advances the guest program counter;
5. repeats.

Examples mentioned:

• Bochs;
• JavaScript as an interpreted execution model.

Interpretation is:

• simple conceptually;
• flexible;
• entirely implemented in software;
• easy to stop, inspect, or debug at arbitrary points.

Interpretation is very slow.

The lecture gives a rough slowdown of:

• \(10\times\) to \(1000\times\).

Reasons:

• every guest instruction requires many host instructions;
• the interpreter must repeatedly decode instructions;
• the CPU pipeline is used inefficiently;
• every instruction must be manually implemented;
• bugs in the interpreter become another source of incorrect behavior.

Dynamic binary translation translates guest instructions into host instructions.

Example:

• guest: ARM;
• host: Intel x86.

The translation does not have to be one guest instruction to one host instruction.

Often, a block of guest instructions is translated into a block of host instructions that preserves the same effect.

Translation is usually done at basic block granularity.

A basic block is a straight-line sequence of instructions with no internal branches.

This makes it easier to preserve control flow.

Generated blocks can be connected together to preserve the program’s branch structure.

On the first encounter with a block, the system translates it.

On later encounters, the existing translation is reused.

Therefore dynamic binary translation works best when code is executed repeatedly.

If a block is translated only once and never reused, the translation overhead may not be worth it.

Over time, many translated blocks are generated.

These translated blocks consume memory.

Therefore the system may need to garbage-collect old translations.

The lecture mentions QEMU as a well-known example.

It also mentions JSLinux as a fun example of how far virtualization can be pushed:

• Chrome on OS X;
• running a full x86 PC inside the browser;
• running Windows 2000 inside that virtual PC;
• running Firefox inside Windows 2000.

The point is not that this is highly useful, but that the abstraction can be pushed very far.

Dynamic binary translation:

• works for many architectures;
• does not require special hardware support;
• is much faster than pure interpretation;
• can also translate same-architecture instructions into safer instruction sequences.

Dynamic binary translation has:

• high overhead;
• high implementation complexity;
• difficult correctness issues;
• nontrivial optimization problems.

Correctness is the first goal. Optimization only matters after the translated code preserves the correct semantics.

Modern CPUs distinguish between:

• unprivileged instructions;
• privileged instructions.

Unprivileged instructions include ordinary operations such as:

• add;
• subtract;
• divide.

These can run directly in user mode.

Privileged instructions include operations that modify hardware or privileged state, for example:

• changing interrupt behavior;
• loading interrupt descriptor tables, such as lidt on x86.

These should only run in kernel mode.

For performance, applications execute unprivileged instructions directly on the CPU.

They do not ask the OS to perform every arithmetic operation.

This direct execution is a form of layer bypass.

Without it, even adding two numbers would require a system call, which would be far too slow.

With a VMM, there is an extra layer below the guest OS.

The VMM must be the truly privileged layer.

The guest OS thinks it is the kernel, but it cannot be allowed to directly control the real hardware.

Thus:

• unprivileged guest instructions can run directly;
• privileged guest instructions need special treatment.

This is called limited direct execution.

The execution is “direct” for ordinary instructions, but “limited” because privileged instructions cannot simply run unchecked.

The VMM can scan the guest instruction stream.

When it sees a problematic privileged instruction, it replaces it with a safe sequence.

For example, the replacement sequence may call into the VMM.

The guest OS believes it executed its original instruction, but actually the VMM executed a safe emulation of the effect.

Advantages:

• no special hardware support required;
• can handle architectures where trap-and-emulate is not enough.

Disadvantages:

• slow;
• complex;
• requires careful instruction-stream rewriting.

The VMM can run guest kernel code in a less privileged mode.

When the guest tries to execute a privileged instruction, the instruction traps.

The trap goes to the VMM.

The VMM then checks:

• did this trap come from a VM?
  • if yes, emulate the effect for that VM;
  • if no, let the normal host handler handle it.

The VMM then returns control to the guest.

This makes the guest believe it controls the hardware, while the VMM actually controls the real machine.

A privileged instruction is an instruction that:

• can run only in kernel mode;
• traps if executed outside kernel mode.

A sensitive instruction is one that can affect or reveal privileged machine state.

The lecture describes two kinds.

Control-sensitive instructions:

• change machine resources or configuration.

Behavior-sensitive instructions:

• produce different results depending on the current mode or configuration.

The Popek-Goldberg theorem says that trap-and-emulate virtualization works for an architecture iff:

\[ \{\text{sensitive instructions}\} \subseteq \{\text{privileged instructions}\}. \]

In words:

Every sensitive instruction must be privileged.

Why?

Because if every sensitive instruction traps, then the VMM can intercept it and emulate it safely.

If a sensitive instruction does not trap, the guest can observe or modify something important without the VMM getting control.

Important nuance:

• the theorem is about VMMs based on direct execution and trap-and-emulate;
• a VMM may still be possible using dynamic binary translation.

32-bit x86 has four protection rings:

• ring 0: kernel;
• ring 3: user space;
• rings 1 and 2: mostly unused by modern OSes.

Although x86 was designed after the Popek-Goldberg theorem, classic 32-bit x86 still violated the theorem.

It has sensitive instructions that are not privileged.

The lecture says there are 17 such sensitive but unprivileged instructions.

The code segment register contains the current privilege level.

If the guest OS reads it, it may discover that it is not really running at ring 0.

This leaks the fact that the OS is virtualized and breaks the illusion of full equivalence.

PUSHF saves the flags register onto the stack.

This includes the interrupt flag.

If it exposes the real interrupt flag controlled by the VMM, the guest can observe privileged host state.

POPF loads flags from the stack into the flags register.

In kernel mode, it can modify privileged flags.

In user mode, changes to privileged flags may be silently ignored.

This is dangerous for virtualization:

• the guest OS may believe it disabled interrupts;
• the instruction does not trap;
• the VMM does not learn what happened;
• interrupts may still occur;
• the guest’s view of the machine becomes inconsistent.

Paravirtualization relaxes the requirement that the VM exactly mirror real hardware.

The guest OS is made aware that it is virtualized.

Therefore the guest OS cooperates with the hypervisor.

Instead of directly executing problematic privileged instructions, the guest explicitly calls into the hypervisor.

These calls are called hypercalls.

They are analogous to system calls, but instead of calling from an application into the OS, the guest OS calls into the hypervisor.

Paravirtualization requires modifying the guest OS or installing special drivers/tools.

Example:

• Xen is a classic paravirtualization system.

The lecturer also mentions guest tools in systems like VMware.

Guest tools make the guest aware of the virtualization environment and help with features such as:

• file sharing between guest and host;
• display integration;
• graphics support;
• networking;
• drag-and-drop;
• better performance.

Instead of changing the instruction set or rewriting guest code, the CPU can add explicit virtualization support.

A typical approach is to add an extra protection level below kernel mode.

Then:

• the VMM runs at the most privileged virtualization level;
• the guest OS can still believe it has kernel-like privilege;
• the hardware can distinguish whether a trap came from the host or a guest.

Hardware support may also include a second copy of privileged state.

This avoids some traps because guest-visible privileged state can be stored separately from real host privileged state.

Modern 64-bit processors provide hardware support for virtualization.

Examples:

• Intel VT-x;
• AMD-V.

These technologies became available around 2005.

They effectively duplicate the protection machinery and make modern CPUs satisfy the conditions needed for efficient virtualization.

Today, essentially all major virtualization systems rely on hardware support.

This is not enough.

It tells us only the guest physical address, not the real host physical address.

The VMM would still need to walk GPA -> HPA in software.

This would make the TLB much less useful.

This uses the TLB for the second-level translation.

However, the VMM would need to emulate the guest page-table walk for VA -> GPA.

That is also expensive.

This is the desired mapping.

It directly maps the guest virtual address to the real host physical address.

This allows hardware to use the TLB effectively and avoids software walking on every memory access.

The problem is that the system must somehow compose:

\[ VA \to GPA \]

and

\[ GPA \to HPA \]

into:

\[ VA \to HPA. \]

In the paravirtualized approach:

• the guest maintains VA -> GPA in its own guest page table;
• the VMM maintains GPA -> HPA in a separate memory map;
• the guest explicitly informs the VMM when it changes page tables.

The guest uses hypercalls to request page-table updates.

The VMM then performs the necessary work and installs the final VA -> HPA translation used by the CPU.

Suppose:

• guest page table says:
  • 0x1000 -> 0x0000;
• VMM memory table says:
  • 0x0000 -> 0x5000.

Then the final translation is:

• 0x1000 -> 0x5000.

The plus sign in the slide means composition, not numeric addition.

The page offset is preserved during translation.

To create a new mapping:

1. the guest inserts a VA -> GPA entry in its guest page table;
2. the guest invokes the VMM through a hypercall;
3. the VMM stores or updates the GPA -> HPA mapping;
4. the VMM computes the composed VA -> HPA mapping;
5. the VMM inserts the final mapping into the CPU-visible page table/TLB path.

Paravirtualized memory is:

• relatively simple;
• efficient for memory access;
• able to leverage TLB caching well.

It changes the hardware/software interface.

The guest OS must know that it is virtualized.

The guest must call the VMM whenever it changes relevant memory mappings.

Therefore it cannot run a completely unmodified OS.

Shadow page tables support unmodified guest OSes.

The guest does not know it is virtualized.

The guest still maintains its own page tables.

However, the CPU does not actually use the guest page tables directly.

Instead, the VMM maintains a hidden shadow page table.

The shadow page table contains VA -> HPA mappings.

The CPU uses the shadow page table.

The guest does not see it.

On x86, CR3 is the page-table base register.

With shadow page tables, CR3 points to the VMM-maintained shadow page table, not to the guest’s real page table.

The hard part is keeping the shadow page table synchronized with the guest page table.

The guest may modify its page table without explicitly telling the VMM.

Therefore the VMM must detect such modifications.

One technique is:

1. mark the guest page-table pages as read-only in the shadow page table;
2. if the guest OS tries to write to its page table, a page fault occurs;
3. the page fault traps to the VMM;
4. the VMM decodes what the guest was trying to do;
5. the VMM updates the shadow page table accordingly.

This is transparent to the guest because the guest does not see the shadow page table.

The lecture also describes a lazy optimization.

For new mappings, the VMM may avoid eagerly intercepting every update.

Instead:

1. the guest adds a new VA -> GPA mapping;
2. the VMM does not immediately create the shadow entry;
3. on the first access, there is no backing VA -> HPA mapping;
4. the access causes a page fault;
5. the VMM handles the fault;
6. the VMM allocates/maps a host physical page;
7. the VMM inserts the VA -> HPA entry into the shadow page table;
8. execution restarts and succeeds.

This is useful for additions.

For changed or removed mappings, the VMM still needs to be careful.

The lecture explains that guest TLB invalidations help here:

• when an OS changes or removes an in-use mapping, it must invalidate stale TLB entries;
• TLB invalidation instructions are privileged;
• therefore they trap to the VMM;
• the VMM can use that trap to update or remove the corresponding shadow mappings.

Whenever the VMM is invoked, it must determine:

• which VM caused the trap;
• which guest process was running;
• what instruction was executing;
• whether the guest was modifying page tables;
• whether the shadow table must add, remove, or change a mapping.

The lecturer describes this as a case-by-case pattern-matching problem.

It works, but it is complex.

Shadow page tables:

• allow the guest OS to run unmodified;
• do not require special hardware support;
• are fast once the shadow mappings are established.

Shadow page tables have:

• trap/page-fault overhead;
• complexity in keeping the shadow table synchronized;
• subtle correctness issues around updates, removals, and TLB invalidations.

Nested page tables add hardware support for the two levels of translation.

The CPU supports two page tables:

• guest page table:
  • VA -> GPA;
• host/hypervisor page table:
  • GPA -> HPA.

There are separate base address registers.

The hardware walks both tables.

Technologies:

• Intel: Extended Page Tables, or EPT;
• AMD: Rapid Virtualization Indexing, or RVI
  • also commonly called Nested Page Tables, or NPT.

For each memory access:

1. the CPU uses the guest page table to translate VA -> GPA;
2. the CPU uses the host page table to translate GPA -> HPA.

The final VA -> HPA table does not need to exist explicitly anywhere.

Because the hardware understands the two-level translation, the guest can modify its own page tables without trapping on every change.

If both page tables had only one level, nested translation would roughly double the page-walk cost:

• first resolve VA -> GPA;
• then resolve GPA -> HPA.

This would not be too bad.

Real page tables are hierarchical.

For example, x86-64 commonly uses four-level page tables, and modern systems can have up to five levels.

The problem is that page tables themselves are stored in memory pages.

Therefore, before the CPU can read a guest page-table entry, it may need to translate the address of the page-table page itself.

With two nested hierarchical page tables, the cost becomes quadratic.

The lecture gives the important example:

• two nested four-level page tables can require 24 memory accesses.

This is a huge performance hit.

The cost can be mitigated by:

• TLB caching;
  • a larger TLB gives more hits;
• parallelizing parts of the page-table walk;
• out-of-order execution hiding some memory latency;
• amortization:
  • once a page is translated, the 4KB page may be accessed many times.

However, pathological workloads can still be bad.

For example, if a program touches only one byte from each page, it may pay many page-walk costs with little reuse.

Device emulation uses trap-and-emulate.

The guest OS runs an unmodified device driver.

The guest believes it is talking to a real device.

When the guest accesses device registers, the access traps to the VMM.

The VMM then emulates what the device would have done.

Device emulation is fully transparent.

The guest need not know it is virtualized.

It also decouples the emulated device from the physical hardware.

For example, the VMM may present an old or simple virtual device interface even if the physical device is different.

In some cases, the guest may have a driver for an emulated device even though the host does not have that exact physical device.

The VMM still needs enough understanding of the device interface to emulate register behavior and forward operations correctly.

Emulation is expensive and slow.

The VMM is in the critical path for device operations.

Every relevant register access may cause a trap.

Split device virtualization uses two drivers:

• a simple virtual driver inside the guest;
• the real device driver in the VMM or host.

The guest uses a special paravirtualized driver.

This driver talks to the VMM through a simpler virtual device interface.

The VMM translates those requests to operations on the real device.

Split/paravirtualized devices are:

• significantly faster than full emulation;
• easier to share among VMs;
• cleaner because the virtual device interface can be designed for virtualization.

They require special guest support.

The guest must have the paravirtualized driver installed.

This is similar in spirit to installing guest tools in a VM.

In pass-through, the VMM gives a VM direct access to a physical device.

The guest OS uses the real device driver for that specific device.

The VMM is mostly removed from the I/O path.

Pass-through can provide very high performance.

The VM can use the device almost as if it owned the hardware.

This is useful for performance-sensitive devices such as GPUs or high-speed network cards.

Pass-through is less flexible.

Problems include:

• the device is usually pinned to one VM;
• sharing is difficult or impossible;
• the host should not also use the device if correctness is required;
• the guest needs a driver for the specific device;
• VM migration becomes difficult because the VM is tied to physical hardware.

Pass-through requires an IOMMU for safety.

Devices often perform DMA, meaning they directly access memory.

Without protection, a passed-through device could read or write memory belonging to the host or other VMs.

The IOMMU translates and checks DMA addresses.

It is essentially memory protection for devices.

SR-IOV stands for Single Root I/O Virtualization.

It allows a PCIe device to expose multiple virtual copies of itself.

The hardware then helps multiplex the device among VMs.

For networking, this can include:

• per-VM transmit queues;
• per-VM receive queues;
• filtering;
• demultiplexing.

Modern NICs may implement functionality such as:

• switching;
• firewalling;
• queue management.

This pushes more complexity into the device hardware.

The lecture mentions Amazon Nitro as an example of modern cloud virtualization.

The idea is:

• VMs need direct device access for performance;
• pure direct pass-through is inflexible;
• therefore custom programmable hardware is used.

Nitro strips down the software hypervisor and offloads much functionality to dedicated hardware, such as ASICs or SoCs.

The hardware directly presents virtualized devices to VMs.

This improves:

• performance;
• isolation;
• synchronization;
• scalability.

The broader trend is that when software virtualization becomes too complex or slow, systems often push functionality into hardware.

A container should see only its own resources.

It should not see the whole host system.

For example:

• it should see its own process IDs;
• it should see its own network interfaces;
• it should see its own file-system view.

Resources should be allocated among containers.

A container should not be able to consume all CPU, memory, disk bandwidth, or kernel resources and thereby starve other containers.

chroot() restricts a process to a subtree of the file system.

This is a basic form of file-system isolation.

However, it is not flexible enough for full container environments.

Containers may need to share some files while keeping others private.

For example:

• shared Python interpreter or shared libraries;
• shared configuration templates;
• private application data;
• private writable directories.

File-system namespaces give each container an isolated view of the VFS.

They allow containers to mount and unmount file systems independently.

They also support overlay-style file systems, where common read-only layers can be shared while private writable layers remain isolated.

This is important for efficient container images.

Docker builds and manages file-system images.

A container image contains a file-system image for the container.

Images can be:

• transferred between machines;
• released online;
• shared between containers.

Docker uses copy-on-write techniques so containers can share common layers efficiently.

Docker also sets up and manages the container runtime environment.

It handles:

• mounting file systems;
• configuring networking;
• setting up the execution environment;
• logging;
• monitoring;
• exec into running containers;
• clean termination of containers.

Containers have much lower overhead than VMs.

Reasons:

• only one kernel instance;
• no separate guest kernel per container;
• only one level of address translation;
• direct use of host drivers;
• faster startup and shutdown;
• less memory and disk duplication.

Containers are more tightly coupled to the host.

Limitations include:

• same kernel version as the host;
• harder to encapsulate complete guest state;
• harder to migrate than a full VM in some respects;
• less strong abstraction boundary than a VM.

The security question is subtle.

Containers share the host kernel.

Therefore the trusted computing base includes:

• the full kernel;
• all relevant kernel subsystems;
• device drivers;
• namespace implementation;
• cgroup implementation;
• the huge system-call interface.

This is a much larger and more complicated interface than a minimal hypervisor interface.

Thus, containers are lightweight, but the isolation boundary may be weaker than a VM boundary depending on the threat model.

This is a traditional enterprise setting.

Employees run code of their choosing on the organization’s systems.

The tenants are not necessarily malicious outsiders.

Containers are often acceptable and useful here.

This is much more hostile.

Anonymous users may run arbitrary code on the same physical machine as valuable customers.

An attacker may try to co-locate with a target and break isolation.

In this setting, we must be more paranoid.

The key question is:

Which isolation mechanisms do we trust?

Possibilities include:

• OS isolation;
• hypervisor isolation;
• network isolation;
• hardware isolation.

There is no single universal answer.

In practice:

• enterprises often use containers for convenience and efficiency;
• cloud systems often use VMs as a stronger isolation boundary;
• containers are frequently run inside VMs.