Tag Archives: virtualization

Using VCSA as remote syslog – Don’t forget the log rotation!

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Important note: It seems that vCenter Server Appliance updates revert the changes. Please check the settings after each update!

The VMware vCenter Server Appliance (VCSA) can act as a remote syslog destition for ESXi hosts. This is very handy for troubleshooting and I really recommend to use this feature.  But VMware ESXi hosts can be really chatty and therefore it’s a good idea to keep an eye on the free disk space of the VCSA.

Yesterday, a colleague had an interesting support case. A customer reported that his Veeam Backup & Replication jobs failed and that he was unable to login to the vCenter with the vSphere Client and vSphere Web Client. My colleague checked the VCSA VM and noticed that the VPXD failed to start (“Waiting for vpxd to initialize: ….failed”). Together we checked the appliance and the log files. The vpxd.log (/var/log/vmware/vpx) was updated weeks ago, but the last entry was interesting: No space left on device. But there was free disk space on /storage/log. I immediately checked the inode count with df -i and there it was: No free inodes. Why is this a problem? Each name entry in the file system consumes an inode. If there are no free inodes, no new directories and files can be created. The error message is the same as for missing disk space. Something had to have created a lot of files on /storage/log. Because /var/log/vmware is a symbolic linkt to /storage/log/vmware, it had to be something on the /storage/log partition. We checked the remote syslog location under /storage/log/remote and found gigabytes and an incredible number of logs. After removing the logs, the VPXD was able to start and the inode count was on a normal level.

But why were there so many logs? We checked the logrotate config and found a faulty config for the remote syslog files. Instead of rotating logs and remove old ones, this config rotated all logs every day and potentiated the number of logs. Please note that there is no logrotate config to rotate remote syslog files by default! This one was added manually.

This is the default config for the remote syslog-collector of the VCSA:

As you can see, with these settings a folder for each host and each month is created. According to this VMTN posting, we changed the syslog-collector config a bit:

With this settings, only a single file per host is created. We made also a change to /etc/logrotate.d/syslog and added this at the end:

With this configuration 30 log files will be preserved. The number of log files or how often log rotation should happen (weekly or daily) can easily be adjusted. But these settings should be sufficient for small environments.

It’s important to understand that the VCSA has different disks and that the disks are mountend to different mount points within the root filesystem. This is from a vSphere 5.5 VCSA:

/var/log/vmware and /var/log/remote are links to /storage/log/vmware and /storage/log/remote. Make sure that there is always enough free diskspace on ALL disks! I also want to highlight VMware KB2092127 (After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress). This error hit me a couple of times…

HP offers 1TB StoreOnce VSA for free

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A free StoreOnce VSA, like the well known 1 TB StoreVirtual VSA? That would be too cool to be real. But it is real! Since February, HP offers a free 1 TB version of their StoreOnce VSA. I totally missed this announcement, but thanks to Calvin Zito I noticed it today:

The link leads to another blog post from Ashwin Shetty (Can you protect your data for free? Introducing the new free 1TB StoreOnce VSA), in which he provides more information about the free 1 TB StoreOnce VSA.

HP StoreOnce VSA

HP StoreOnce VSA runs with the same software as the hardware-based StoreOnce appliances, but it’s delivered as a VM. You can run the VM on top of VMware ESXi, Microsoft Hyper-V or KVM. Beside the free 1 TB license, the StoreOnce VSA can purchased with 4 TB, 10 TB or 50 TB capacity (usable, non-deduplicated). In contrast to the hardware-based appliances, the StoreOnce VSA comes with licenses for replication and StoreOnce Catalyst. This makes the StoreOnce VSA a perfect fit for remote and branch offices. You can quickly deploy the StoreOnce VSA and replicate the backuped data to the central datacenter. But you can also deploy the VSA with the 4 TB, 10 TB or 50 TB license in your central datacenter and use it as a replication target for StoreOnce VSAs in the remote and branch offices (the replication target needs the replication license). A single VSA can act as replication target for up to 8 StoreOnce VSA and/ or StoreOnce appliances. You can scale the free 1 TB license with license upgrades to 4 TB, 10 TB and 50 TB. The StoreOnce VSA supports Catalyst, VTL (iSCSI) and as NAS (CIFS or NFS) backup targets. Take a look into the QuickSpecs for more information. I also recommend to read the two blog posts from Ashwin Shetty on Around the Storage Block:

Last year I’ve published several posts about the StoreOnce VSA. I recommend to download the free 1 TB StoreOnce VSA and to play with it. Some of my blog posts should help you get started.

Top vBlog 2015: vcloudnine.de placed on #133

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

What a great show by Eric Siebert, David Davis, Simon Seagrave and their special guests Scott Davis from Infinio and John Troyer from TechReckoning! If you missed it, watch the recording!

First, I want to thank Eric for his work. If you read tweets like these, you will get a bad conscience.

This is the seventh year that Eric has organized and conducted the annual Top vBlog contest. He put so much work into this contest and this should be be recognized. I also like to thank the sponsor Infinio for supporting this contest.

2015 was the second year in which I partaken the Top vBlog contest, but vcloudnine.de was on the voting list for the fist time. I started this blog in 2014 so I was on the “Newcomer” list of the contest. I’m always trying to create valuable content. This isn’t easy and often a draft is thrown to trash. I hope vcloudnine.de was chosen because of valuable content and not because voters like me. ;) This year’s Top vBlog poll brought us a lot changes. Eric has leaked some details in a blog post short before the announcement:

  • 60% more votes than 2014
  • 30% more blogs on the voting list
  • 7 changes in the top 10
  • 4 blogs in the top 25 that were not in there last year
  • 2 blogs in the top 25 that were newcomers this year
  • 1 blog new to the top 10

Congratulations to…

“Out of competition”: Duncan Epping (VCDX #007) and yellow-bricks.com for “defending” 1st place. Does anyone doubt it? Not really, right? ;) Congrats Duncan!

I am particularly happy for Derek Seaman (VCDX #125). His blog is a gold mine of content and he’s generating more and more (read his vSphere 6.0 series). Congrats Derek, #7 is totally deserved!

Congrats to Melissa Palmer for winning the “Best new blog” category. Keep on blogging, Melissa!

Congrats to Chris Wahl (VCDX #104) for winning the “Best indipendant blogger” category. Reading his blog is always a pleasure!

Also well deserved: Brian Madden has won the “Best VDI blog” category. His blog is an awesome resource if you deal with VDI!<in/p>

Honestly: That William Lam has won in the category “Best scriping blog” and Cormac Hogan in the “Best storage blog” category was no suprise for me. Totally deserved, guys!

I am very happy to see that some bloggers that I have on my reading list, ranked up in the list. You can find the results of the Top vBlog 2015 contest here. Congrats to all participant and thanks again to Eric Siebert!

To make the long story short…

I’m happy and disappointed at the same time. vcloudnine.de landed on place 133. Not the worst placement for a new blog. But I have missed my personal goal to be placed under the top 100. I’d like to thank all, that have voted for vcloudnine.de. This is a great motivation to work harder and to create more valuable content. Thank you all!

Tiering? Caching? Why it’s important to differ between them.

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago I talked to a colleague from our sales team and we discussed different solutions for a customer. I will spare you the details, but we discussed different solutions and we came across PernixData FVP, HP 3PAR Adaptive OptimizationHP 3PAR Adaptive Flash Cache and DataCore SANsymphony-V. And then the question of all questions came up: “What is the difference?”.

Simplify, then add Lightness

Lets talk about tiering. To make it simple: Tiering moves a block from one tier to another, depending on how often a block is accessed in a specific time. A tier is a class of storage with specific characteristics, for example ultra-fast flash, enterprise-grade SAS drives or even nearline drives. Characteristics can be the drive type, the used RAID level or a combination of characteristics. A 3-tier storage design can consist of only one drive type, but they can be organized in different RAID levels. Tier 1 can be RAID 1 and tier 3 can be RAID 6, but all tiers use enterprise-grade 15k SAS drives. But you can also mix drive types and RAID levels, for example tier 1 with flash, tier 2 with 15k SAS in a RAID 5 and tier 3 with SAS-NL and RAID 6. Each time a block is accessed, the block “heats up”. If it’s hot enough, it is moved one tier up. If it’s less often accessed, the block “cools down” and at a specific point, the block is moved a tier down. If a tier is full, colder blocks will to be moved down and hotter block have to be moved up. It’s a bit simplified, but products like DataCore SANsymphony-V with Auto-Tiering or HP 3PAR Adaptive Optimization are working this way.

Lets talk about caching. With caching a block is only copied to a faster region, which can be flash or even DRAM. The original block isn’t moved, only a copy of the accessed block is copied to a faster medium. If this block is accessed, the data is served from the faster medium. This also works for write I/O. If a block is written, the data is written to the faster medium and will be moved later to the underlying, slower medium. You can’t store block copies until infinity, so less accessed blocks have to be removed from cache if they are not accessed, or if the cache fills up. Examples for caching solutions are PernixData FVP, HP 3PAR Adaptive Flash Cache or NetApp Flash Pool (and also Flash Cache). I lead storage controller cache explicitly not appear in this list. All of the listed caching technologies (except NetApp Flash Cache) can do write-back caching. I wouldn’t recommend read-cache only solutions like VMware vSphere Flash Read Cache, except two situations: Your workload is focused on read I/O and/ or you already own a vSphere Enterprise Plus license, and you do not want to spend extra money.

Tiering or caching? What to choose?

Well… it depends. What is the main goal when using these techniques? Accelerate workloads and making best use of scarce and expensive storage (commonly flash storage).

Regardless of the workload, tiering will need some time to let the often accessed blocks heat up. Some vendors may anticipate this partially by writing data always to the fastest tier. But I don’t think that this is what I would call efficient. One benefit of tiering is, that you can have more then two tiers. You can have a small flash tier, a bigger SAS tier and a really big SAS-NL tier. Usually you will see a 10% flash / 40% SAS / 50% SAS-NL distribution. But as I also mentioned: You don’t have to use flash in a tiered storage design. That’s a plus. On the downside tiering can make mirrored storage designs complex. Heat maps aren’t mirrored between storage systems. If you failover your primary storage, all blocks need to be heaten up again. I know that vendors are working on that. HP 3PAR and DataCore SANsymphony-V currently have a “performance problem” after a failover. It’s only fair to mention it. Here are two examples of products I know well and both offer tiering: In a HP 3PAR Adaptive Optimization configuration, data is always written to the tier, from which the virtual volume was provisioned. This explains the best practice to provision new virtual volumes from the middle tier (Tier 1 CPG). DataCore SANsymphony-V uses the performance class in the storage profile of a virtual disk to determine where data should be written. Depending on the performance class, data is written to the highest available tier (tier affinity is taken into account). Don’t get confused with the tier numbering: Some vendors use tier 0 as the highest tier, others may start counting at tier 1.

Caching is more “spontaneous”. New blocks are written to the cache (usually flash storage, but it can also be DRAM). If a block is read from disks, it’s placed in the cache. Depending on the cache size, you can hold up a lot data. You can lose the cache, but you can’t lose the data ins this case. The cache only holds block copies (okay, okay, written blocks shouldn’t be acknowledged until they are in a second cache/ hose/ $WHATEVER). If the cache is gone, it’s relatively quickly filled up again. You usually can’t have more then two “tiers”. You can have flash and you can have rotating rust. Exception: PernixData FVP can also use host memory. I would call this as an additional half tier. ;) Nutanix uses a tiered storage desing in ther hyper-converged platform: Flash storage is used as read/ write cache, cost effective SATA drives are used to store the data. Caching is great if you have unpredictable workloads. Another interesting point: You can cache at different places in the stack. Take a look at PernixData FVP and HP 3PAR Adaptive Flash Cache. PernixData FVP is sitting next to the hypervisor kernel. HP 3PAR AFC is working at the storage controller level. FVP is awesome to accelerate VM workloads, but what if I have physical database servers? At this point, HP 3PAR AFC can play to its advantages. Because you usually have only two “tiers”, you will need more flash storage as compared to a tiered storage design. Especially then, if you mix flash and SAS-NL/ SATA.

Final words

Is there a rule when to use caching and when to use tiering? I don’t think so. You may use the workload as an indicator. If it’s more predictable you should take a closer look at a tiered storage design. In particular, if the customer wants to separate data from different classes. If you have more to do with unpredictable workloads, take a closer look at caching. There is no law that prevents combining caching and tiering. At the end, the customer requirements are the key. Do the math. Sometimes caching can outperform tiering from the cost perspective, especially if you mix flash and SAS-NL/ SATA in the right proportion.

My first impressions about PernixData FVP 2.5

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

On February 25, 2015 PernixData released the latest version of PernixData FVP. Even if it’s only a .5 release, FVP 2.5 adds some really cool features and improvements. New features are:

  • Distributed Fault Tolerant Memory-Z (DFTM-Z)
  • Intelligent I/O profiling
  • Role-based access control (RBAC), and
  • Network acceleration for NFS datastores

Distributed Fault Tolerant Memory-Z (DFTM-Z)

FVP 2.0 introduced support for server side memory as an acceleration resources. With this it was possible to use server side memroy to accelerate VM I/O operations. Server side memory is faster then flash, but also more expensive. With FVP 2.5, the support for adaptive memory compression. was added. DFTM-Z provides a more efficient use of the expensive resource “server side memory”.  Some of you may think “Oh no, compression! This will only cost performance!”. I don’t think that this is fair. ;) The PernixData engineers are focused on performance and I think that they haven’t during the development of DFTM-Z. DFTM-Z is enabled on hosts that use at least 20 GB memory for FVP. With increasing memory used for FVP, the area used for compression in the memory is also increased. So not the whole memory area used for acceleration is compressed, it’s only a part of it. With 20 GB contributing the FVP cluster, the compressed memory region is 4 GB. With more than 160 GB, the region is increased to 32 GB.

Intelligent I/O profiling

A VM usually has a specific I/O profile. Sometimes this I/O profile changes quickly, e. g. when doing backups (large sequential I/Os). With intelligent I/O profiling, such workloads can now be bypassed. This doesn’t disable acceleration! The active FVP footprint of the VM remains active and is used to accelerate I/O. The intelligent I/O profiling can be enabled on a per-VM basis using PowerShell.

Role-based access control (RBAC)

The access to FVP can now be controlled with a role-based model. For this, three different roles can be used.

  • Read and Write – View and change configuration, view performance charts
  • Read-Only – View configuration and performance charts only
  • No Access – no access

vCenter users with administrator permission have read/ write access to FVP. Users without administrator permission have only read-only access. All other users have no access to FVP.

Network acceleration for NFS datastores

In the past it was not possible to use the VM footprint, the “hot data”, after a vMotion, if the VM was stored in a NFS datastore. Now this VM footprint can used for read I/O over the network.

The update process

The update from FVP 2.0 to 2.5 is really easy:

  1. Transition the VMs to write through mode
  2. Update the FVP Management server
  3. Remove host extension on the hosts
  4. Install the new host extension on the hosts
  5. Enable vSphere Plugin (C# or Web Client)
  6. Transition the VMs to write back mode

I have performed this update in my lab, and the process went smooth. Be sure to take a look into the upgrade guide. Sometimes there are interesting things in it. ;)

Overall, I’m still totally convinced of PernixData and I hope to place it in a customer project soon.

vCenter Server Appliance: Troubleshooting full database partition

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer of mine had within 6 months twice a full database partition on a VMware vCenter Server Appliance. After the first outage, the customer increased the size of the partition which is mounted to /storage/db. Some months later, some days ago, the vCSA became unresponsive again. Again because of a filled up database partition. The customer increased the size of the database partition again  (~ 200 GB!!) and today I had time to take a look at this nasty vCSA.

The situation

vcsa_overview

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Within 2 days, the storage usage of the databse increased from 75% to 77%. First, I checked the size of the database:

 As you can see, the database had only 2 GB. The pg_log directory was more interesting:

 The directory was full with log files. The log files containted only one message:

The solution

This led me to VMware KB2092127 (After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress). And yes, this appliance was upgraded to U2 with high probability. The solution is described in KB2092127, and is really easy to implement. Please note that this is only a workaround. There’s currently no solution, as mentioned in the article.

Top vBlog 2015 Contest has started

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

If you are a frequent reader of virtualization blogs, then you may have heard about the vLaunchPad. It lists hundreds of VMware & virtualization blogs, as well as links to resources and other material. The vLaunchPad is managed by Eric Siebert (@ericsiebertvsphere-land.com) and he organizes year for year the annual Top vBlog voting contest. This year the Top vBlog contest is sponsored by Infinio.

In the 2014 voting my “old” blog was voted on place 292 of 320. I should mention that blazilla.de had only german-language content. In a community, where english is the predominating content language, this result may not surprise. If you are interested in last year’s results, you can find them here. In 2014 I have started vcloudnine.de, but I didn’t nominated it for the 2014 voting. Instead, I nominated blazilla.de for the Top vBlog 2014 contest. This year the tables turned and I have nominated vcloudnine.de for the categories:

  • Best new blog (Blog started in 2014), and
  • Best independent blogger (Can’t work for VMware or a hardware/software vendor)

As always all blogs that are listed on the vLaunchPad are included in the general voting. I don’t have a goal for the voting, but a place between #49 and #100 would be nice. ;)

Some short sentences about vcloudnine.de:

vcloudnine.de is the personal blog of Patrick Terlisten. The site has a strong focus on virtualization, storage, networking and IT infrastructure in general. The main driver of this blog is to share knowledge and write about topics, that I think is worth mentioning. The views expressed anywhere on this site are mine and not the opinions and views of my employer or a vendor.

The predominating topics on vcloudnine.de are VMware, HP Storage, HP Data Protector, networking in general and Microsoft Exchange.

Andreas Lesslhumer (@lessi001running-system.com) has created a nice statistic for 2014: Virtualization blogs 2014 by numbers. The statistic is based on the blogs, that are listed on the vLaunchPad. vcloudnine.de was one of the 28 blogs, that published more than 100 blog posts in 2014. In 2015 I published 13 blog posts so far. But to be honest: It’s not about the number of posts you publish – the content matters! So if you vote for a blog, vote for the content, not the number of published posts or the author.

Check out the Top vBlog 2015 landing page and don’t forget to vote for your favorite blogs! The voting will start soon!

The beginning of a deep friendship: Me & PernixData FVP 2.0

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

I’m a bit late, but better late than never. Some days ago I installed PernixData FVP 2.0 in my lab and I’m impressed! Until this installation, solutions such as PernixData FVP or VMware vSphere Flash Read Cache (vFRC) weren’t interesting for me or most of my customers. Some of my customers played around with vFRC, but most of them decieded to add flash devices to their primary storage system and use techniques like tiering or flash cache. Especially SMB customers had no chance to use flash or RAM to accelerate their workloads because of tight budgets. With decreasing costs for flash storage, solutions like PernixData FVP and VMware vSphere Flash Read Cache (vFRC) getting more interesting for my customers. Another reason was my lab. I simply hadn’t the equipment to play around with that fancy stuff. But things have changed and now I’m ready to give it a try.

The environment

For the moment I don’t have any SSDs in my lab servers, so I have to use RAM for acceleration. I will add some small SSDs later. Fortunately PernixData FVP 2.0 supports NFS and I can use host memory to accelerate my lab workloads.

The installation

I have installed PerniXata FVP 2.0 in my lab and deployed the host extension with the vSphere Update Manager to three of my lab hosts.

PernixData FVP consists of three components:

  • Host Extension
  • Management Server running on a Windows Server
  • UI Plugin for the vSphere C# and vSphere Web Client

The management server needs a MS SQL database and it installs the 64 bit version of Oracle Java SE 7. For a PoC or a small deployment, you can use the Express version of Microsoft SQL Server 2012. I installed the management server onto one of my Windows 2008 R2 servers. This server hosts also my vSphere Update Manager, so I had already a MS SQL database in place. I had some trouble right after the installation, because I missed to enable the SQL Browser service. This is clearly stated in the installation guide. So RTFM. ;)

NOTE: The Microsoft® SQL Server® instance requires an enabled TCP/IP protocol even if the database is installed locally. Additional details on enabling TCP/IP using the SQL Server Configuration Manager can be found here. If using a SQL Named Instance, as in the example above, ensure that the SQL Browser Service is enabled and running. Additional details on enabling the SQL Browser Service can be found here.

After I had fixed this, the management server service started without problems and I was able to install the vSphere C# client plugin. You need the plugin to manage FVP, but the plugin installation is only necessary, if you want to use the vSphere C# client. You don’t have to install a dedicated plugin for the vSphere Web Client.

To install the host extension, you can simply import the host extension into the vSphere Update Manager, build a host extension baseline, attach it to the hosts (or the cluster, datacenter object etc.) and remediate them. The hosts will go into the maintenance mode, install the host extension and then exit maintenance mode. A reboot of the hosts is not necessary!

Right after the installation, I created my first FVP cluster. The trial period starts with the installation of the management server. There is no special trial license to install. Simply install the management server and deploy the host extension. Then you have 30 days to evaluate PernixData FVP 2.0.

Both steps, the installation of the host extension using the vSphere Update Manager, as well as the installation of the Management server, are really easy. You can’t configure much, and you don’t need to configure much. You can customize the network configuration (what vMotion network or which ports should be used), you can blacklist VMs and select VADP VMs. Oh, and you can re-enable the “Getting started” started screen. Good for the customer, bad for the guy who’s payed to install FVP. ;) Nothing much to do. But I like it. It’s simple and you can quickly get started.

First impressions

My FVP cluster consists of three hosts. Because I don’t have any SSDs for the moment, I uses host memory to accelerate the workload. During my tests, 15 VMs were covered by FVP and they ran workloads like Microsoft SQL Server, Microsoft Exchange, some Linux VMs, Windows 7 Clients, Fileservices, Microsoft SCOM. I also played with Microsoft Exchange Jetstress 2013 in my lab. A mixed bag of different applications and workloads. A picture says more than a 1000 words. This is a screenshot of the usage tab after about one week. Quite impressive and I can confirm, that FVP accelerates my lab in a noticeable way.

pernixdata_results

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I’ve enabled FVP on Monday evening. Check the latency diagram, that I’ve taken from vCenter. See the latencies dropping on Monday evening? The peaks during the week were caused by my tests.

pernixdata_results_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Final words

Now it’s time to convince my sales colleagues to sell PernixData FVP. Or some customers read this blog post and ask my sales colleagues for PernixData. ;) I am totally convinced of this solution. You can buy PernixData FVP in different editions:

  • FVP Enterprise: No limit on the number of hosts or VMs
  • FVP Subscription: FVP Enterprise purchased using a subscription model
  • FVP Standard: No limit on the number of hosts or VMs. Perpetual license only. No support for Fault Domains, Adaptive Resource Management and Disaster Recovery integration (only in FVP Enterprise).
  • FVP VDI: Exclusively for VDI (priced on a per VM basis)
  • FVP Essentials Plus: FVP Standard that supports 3 hosts and accelerates up to 100 VMs. This product can only be used with VMware vSphere Essentials (Plus).

If you’re interested in a PoC or demo, don’t hesitate to contact me.

I’d like to thank Patrick Schulz, Systems Engineer DACH at PernixData, for his support! I recommend to follow him on Twitter and don’t foget to take a look at his blog.

Juniper publishes vMX

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This tweet from @JuniperNetworks has really inspired me yesterday. I liked Junipers Firefly Perimeter (vSRX) from the first day. I like the idea behind this product (yes, I like everything that can be run as a VM…). But yesterday Juniper has go one better.

Juniper Networks announced yesterday a virtualized and carrier-grade version of their MX Series 3D router. The Juniper Networks vMX is a virtual MX Series 3D Universal Edge Router and it’s optimized to run on x86 hardware. Juniper vMX can run on all major Hypervisors, including VMware ESXi and KVM. It was also mentioned, that vMX can be run in Docker containers or on bare-metal.

The development of vMX was relieved by Junipers acquisition of Contrail. Junipers physical MX series router is powered by Junipers Trio chipset and Juniper has virtualized their Trio chipset for vMX (now called vTrio). It was also optimized for x86 hardware. Depending on the number of physical resources, a vMX can achieve a throughput of 160 Gbps. vMX uses vTrio, Junos OS and supports the same feature set, so it feels and behaves like a physical MX series router. This ensures that customers can leverage their Juniper MX knowhow to run vMX in their environment. If a customer uses physical or virtual MX router is only a question of performance. Multiple vMX can be managed with Junos Space, Contrail SDN controller and OpenStack Cloud Manager. Customers will be able to buy vMX with beginning of Q1/2015 in a flexible license model (Pay-as-you-grow, perpetual or subscription license). Details about the pricing weren’t revealed by Juniper.

This short video was published by Juniper Networks and it’s available on YouTube.

VMware disables inter VM Transparent Page Sharing (TPS) for security reasons

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This morning I discovered a tweet from Derek Seaman in my timeline, that caught my attention.

TPS stands for Transparent Page Sharing and it’s one of VMware memory management technologies. VMware ESX(i) uses four different technologies to manage host and guest memory resources (check VMware KB2017642 for more information). The preference increases from TPS to swapping.

  • Transparent page sharing (TPS)
  • Ballooning
  • Memory Compression
  • Swapping

TPS is a technology by which redundant copies of memory pages are eliminated. You can understand TPS like some kind of memory deduplication. The hypervisor scans the memory periodically for memory pages that could be possibly  shared. For each candidate memory page a hash is calculated and it’s saved in a hash table. If a second candidate page has the same hash, a full bit-by-bit comparison for both pages is triggered. If both memory pages are identical, only one page is saved and the other memory page is reclaimed. TPS is enabled by default and shows good results, especially if you were running a lot VMs with the same OS, like in VDI or terminal server environments.

With the advent of hardware-assisted memory virtualization systems, like Intel EPT or AMD RVI, VMware changed the behaviour of TPS and how guest memory is backed to physical memory. Guest memory was now backed with larger memory pages (2MB instead of 4KB ) for better performance. But 4 KB pages were still used if there were no 2 MB continuous memory, e.g. in case of memory overcommitment or memory fragmentation. Using 2 MB memory pages has advantages, for sure, but in perspective of TPS it has two disadvantage:

  • small chance to find two identical memory pages
  • the expense of a bit-by-bit comparison is at 2 MB pages incredibly much higher than with 4 KB pages

The punchline is, that with hardware-assisted memory virtualization systems, TPS is only actively used if the host is under memory pressure. But it is still there and working.

Safety over performance

Yesterday VMware published KB2080735 (Security considerations and disallowing inter-Virtual Machine Transparent Page Sharing). The purpose of this KB:

This article acknowledges the recent academic research that leverages Transparent Page Sharing (TPS) to gain unauthorized access to data under certain highly controlled conditions and documents VMware’s precautionary measure of no longer enabling TPS in upcoming ESXi releases. At this time, VMware believes that the published information disclosure due to TPS between virtual machines is impractical in a real world deployment.

Because of this, TPS will be disabled by default with the release of:

  • ESXi 5.5 Update release (Q1/ 2015)
  • ESXi 5.1 Update release (Q4/ 2014)
  • ESXi 5.0 Update release (Q1/ 2015)
  • The next major version of ESXi (ESXi 6.0)

Prior these updates VMware will release patches that introduce additional TPS management capabilities and that WILL NOT change the existing settings for inter VM TPS (check KB2091682). As stated in KB2080735, the planned ESXi patch releases are:

  • ESXi 5.5 Patch 3
  • ESXi 5.1
  • ESXi 5.0

The patches for ESXi 5.0 and 5.1 are planned for Q4/ 2014. For ESXi 5.5 a patch the patch is already available (ESXi550-201410401-BG).

My 2 cents

Several years ago, the deactivation of TPS would have been fatal. Today, and in consideration of “safety over performance”, I think it was the right decision. If your design heavily relies on TPS, then you maybe have a bad design. ;)

Also a good read:

Frank DennemanFuture direction of disabling TPS by default and its impact on capacity planning
Magnus AnderssonChanges in ESXi Transparent Page Sharing (TPS) behaviour
Kenneth van SurksumVMware decides to disable TPS in future ESXi releases by default
Marcel van den BergVMware wil disable Transparant Page Sharing by default in future ESXi releases
Andrea MauroBye bye Transparent Page Sharing
Chris WahlTransparent Page Sharing Vulnerable, Yet Largely Irrelevant

More will follow, ping me on Twitter if you found a good one!