Mar 182013

Sorry guys for another hiatus, my job at Calxeda keeps me busy. I was recently discussing ARM’s virtualization support with my friend Ali Hussain (yup, that’s our idea of a fun dinner conversation) and found some very interesting facts. I requested Ali to share his knowledge in a blog post series on this topic, so here you go. Ali is in ARM’s performance modeling team and has been working on ARM cores since 2008.

The idea for this blog post stemmed from talking to people that had the impression that ARM’s virtualization support, even with the virtualization extensions in Cortex-A15, is limited. I plan to write a few posts exploring virtualization, and the support for it in the ARM and x86 ISAs. This post will draw heavily on my understanding of the ARM architecture and operating systems.

What is Virtualization?
Before we can look at virtualization, we need to define a few key things. The first is virtualization itself. Virtualization in general is creating an environment in software to emulate something physical. More specifically, when we talk about hardware virtualization, it is running an operating system on a sandboxed virtual machine (VM) as opposed to having access to the physical hardware. This is done through the hypervisor which manages guest operating systems the same way an operating system manages applications.

Privilege levels

Virtualization does not require hardware support. It can be performed completely in software. An existing operating system can be patched to work at a lower privilege level and trap to the hypervisor. This is called paravirtualization. The advantage of hardware support is in simplifying the software work for the hypervisor and providing performance improvements. For this post, I want to discuss how hardware can help the hypervisor in performing its functions. Those interested in paravirtualization will find this talk about problems faced while implementing paravirtualization on ARM interesting by VMWare:

Let’s explore some of the duties of the hypervisor and how the hardware accomplishes them to better understand hardware-assisted virtualization.

Management Has Its Privileges
To sandbox an OS, the hypervisor has to be at a higher privilege level than the guest OS. ARM, for example, creates a higher privilege level called hypervisor mode. The hypervisor mode has access to its own set of system registers that are analogous to the registers present in the system mode. For example, just like the OS tracks process IDs using an Address Space IDentifier (ASID which is a part of the TTBR), the hypervisor tracks current VM using a VMID (which is a part of the VTTBR).

64 bit LPAE version of TTBR

ARM TTBR bit fields


ARM VTTBR bit fields

The hypervisor also has similar access controls as the supervisor mode. In addition,  the hypervisor can read and write the system control registers for the OS.

Having two layers of access control does create a lot of interesting scenarios where the hypervisor and the guest OS compete for a trap. ARM has gone with the philosophy that a good manager is one that intrudes into your work as little as possible. So, if an exception occurs, the guest OS is typically given the opportunity to service it before the hypervisor because the OS is better equipped to handle the applications’ requirements. Let me explain this with an example. Both the OS and the hypervisor provide a similar feature for disabling access to the floating point and SIMD units, using the CPACR (Coprocessor Access Control Register) and HCPTR (Hypervisor Coprocessor Trap Register) respectively. When an application running in the guest OS is disallowed access by both the CPACR and HCPTR, the CPACR has priority.

This is an interesting design choice. It has two advantages. First, it improves the response time of the exception. Second, it allows the OS to behave as normal. However, it also makes it extremely difficult to provide certain functionality, e.g., making the floating point unit invisible to the guest.

Check out the next post in this series, talking about memory management and interrupt handling in a hypervisor here.


1. All opinions expressed here are my own and not of ARM or any other entity.

2. I haven’t implemented a hypervisor or operating system myself so I would love to hear comments from experts in the field.


  6 Responses to “ARM Virtualization Extensions — Introduction (Part 1)”

  1. [...] ARM Virtualization Extensions [...]

  2. It’s really a nice and useful piece of information. I
    am glad that you just shared this helpful information with us.
    Please stay us informed like this. Thanks for sharing.

  3. I don’t even know how I ended up here, but I thought this
    post was great. I don’t know who you are but definitely you’re going to
    a famous blogger if you aren’t already ;) Cheers!

    • Thank you. Although both Aater and I are presently directing out energy towards establishing our company Flux7 Labs moreso than futurechips. You can read our posts at the Flux7 blog: There is a lot more focus there on the cloud but many posts there are related to chip design.

  4. There’s certainly a great deal to learn about this subject.
    I really like all of the points you made.

  5. First time swimming pool owners anxiousl anticipate having their
    owwn personal backyard oasis where they can cool off and
    relax on a hot summer day. Using an air purifier whose only by-product is fresh, clean air takes a lot
    of the worry out of getting rid of hair. Horses need
    to answer too an alpha, otherwise, they will feel the need to assume tthe role of the alpha.

 Leave a Reply



You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>