A few years ago when EVERYONE was going to move to the cloud, I remember all the people arguing over Opex vs CapEx. I pointed then that the discussion was mostly irrelevant, it came down to business requirements and what the business wanted. It’s why we, as technologist exist - to do what is right for our organization, not to pick tools because they are the new hotness (anyone burned by HashiCorp dropping Otto already?).
John Hildebrand just wrote a great post on whether DevOps is a load of BS. I largely agree with the post, that DevOps has been driven by developers find their own way to handle operational tasks associate with maintaining an application. As Jon pointed out, in greenfield deployment this is much easier than established corporations with existing applications that haven’t been written to be cloud native from the start.
So, if you’re still along for the ride so far, what is an established organization with developers and operations teams to do? I think we all agree at this point that DevOps is a culture, not a set of tools. Communication and empathy are key to DevOps succeeding an any organization.
The very first step should be to get developers and operations teams in the same room together every day. I am not saying to merge the teams (yet), but there needs to be open communication each and every day. Many developer teams have what are called standups, even if they are formed around Agile principles (or maybe they think they are). Give operations teams equal time each day in the standup to talk about what they are working on, what their challenges are. This opens up communications between the teams as now each understand what the other is up against on a day to day basis. Developers may not have insight into the struggles of an SSAE 16 or HIPAA audit, just as operations teams may not know about the struggles developers are going through to enable encryption end to end in their application.
This, again is a first step, something you can do today (or tomorrow) with minimal impact to existing organizational structures and SOPs. Would love to hear your thoughts! Ping me on Twitter @jfrappier.
At VMworld this year there was quite a bit of talk of multi-cloud or hybrid-cloud support from VMware. NSX, for example, being used to extend networks between on premises and public cloud providers such as AWS and Azure. Now don’t get me wrong, this is great but in terms of building a true hybrid cloud I am forced to build all of my applications around vRealize Automation. Forcing organizations to customize their pipelines and workflows around vRA, and not a native cloud provider API forces organizations to make a choice - lock into VMware or walk away (or even worse, maintain two sets of workflows).
Last year I had the opportunity to work on VxRack with Neutrino which is a appliance based OpenStack solution. While it was great to have hardware to play on, I wasn’t able to reproduce this setup at home.
I have long maintained that Ansible’s documentation is some of the best, if not best out there. However it is impossible to cover every single corner case in documentation which brings me to setting up Ansible to manage Windows, and authenticate via WinRM using Kerberos.
Photon is a minimal operating system designed to run containers (for example using Docker). You can learn more about Photon from the offical GitHub project page
Last week I had the privilige to keynote the annual Virtualization Technology User Group (VTUG) Spring Forward meetup. My topic was Ansible, with a goal of helping people be comfortable enough to write and run their first playbook.
Last week the team I work on ran into a problem with an installation of vRealize Automation 6.2, this installation has 2 appliances behind an NSX load balancer. After an ESXi host failure, one of the appliances did not come back online – when we tried to login we received the generic “Login failed. Please contact your System Administrator and report error code” completely-bs-string-that-means-nothing. Generally when we have seen this it was related to NTP, with the appliances and domain controllers being out of sync, however that wasn’t the case here.
The ability to transform your career are critical in the technology industry, even more so of late as roles transform from a limited scope such as a VMware engineer to the DevOps or Full Stack engineer roles. Making this transformation is not new, many readers of my blog are likely involved with some form of virtualization. This technology was just taking off between 2005-2010. If you were in a technology role before this time, virtualization was a skill you needed to add to your repertoire. For example, I started out as a desktop administrator tasked with creating images for systems and supporting users. From there, I moved into roles more focused on network and server technology, followed by brief a stint in management positions. I finally settled in to virtualization and cloud related work over the last several years. However times are a-changing once again and, except for a select few companies, having knowledge or specialty in just one area won’t be enough.
Disclaimer: My VMUG Advantage subscription was provided courtesy of the Boston VMUG. This post was not paid for, nor reviewed prior to publishing. This is simply my opinion after using the service. Additionally, I am an EMC employee
Disclaimer: My VMUG Advantage subscription was provided courtesy of the Boston VMUG. This post was not paid for, nor reviewed prior to publishing. This is simply my opinion after using the service. Additionally, I am an EMC employee
Disclaimer: My VMUG Advantage subscription was provided courtesy of the Boston VMUG. This post was not paid for, nor reviewed prior to publishing. This is simply my opinion after using the service. Additionally, I am an EMC employee
There were some new features recently added to the development branch on Ansible I wanted to use for testing, from what I could gather reading the source for one particular feature, adding validate_certs to the VMware modules, required Python 2.7.9 (I’m no Python programmer so don’t take my word for it). So I had two options, install Python and dependencies from source on Ubuntu 14.04 since it only has Python 2.7.6 available via the apt repos, or hop on up to 15.10 which has Python 2.7.10.
With the release on Ansible 2.0 came several new VMware modules that leverage pyVmomi, one of which was vmware_dns_config. This modules runs directly against ESXi hosts, so if you have small lab with no vCenter (because reasons?) you can still use this module to ensure that the host(s) have the desired DNS servers.
I always enjoy sharing my “ah-ha” moments, that is when you start to do something - or in this case are already doing something, that helps with other projects. This is a brief story about how we used Ansible to enable disaster recovery without needing to replicate virtual machines, or buy additional software.
If you need to install Varnish, here is a quick little Ansible playbook to get it installed. Updated your host file and user as needed. I am still reading up on Varnish, so haven’t got much past the install yet.
At the very last VMware User Conference of 2015, Sarah Zelechoski and I presented our session - DevOps IRL (In Real Life). In our talk we discussed experiences we have had at small to medium sized organizations where DevOps methodologies were embraced.
During the #vDM30in30 i’ve not been bashful with my blog posts that might upset some folks, VMware is just as complex as OpenStack… I’d never run KVM for an SMB and now this. The premise, just as “IT” people have been relegated to being classified as a basic provider of utility, so to will the software developer.
Stacki is a project I came across on Twitter that allows you to perform bare metal installations of Linux operating systems, there is a great introduction post on their project page so I am not going to rehash all of that but at a high level:
- Install linux for bare metal or virtual machines
- Configure RAID and network controllers
The main component of Stacki is the front end which you will install to manage your deployments that deploys “backends” which is simply the term used for the servers that Stacki builds.
Since I spend my days working with vRealize Automation, I like to understand what other platforms are out there such as CloudBolt C2 and RedHat CloudForms. While CloudForms does not have a trial or free(mium) download, there is an OpenSource version from the ManageIQ acquisition. There are several ways which you can get started with ManageIQ, I have opted for the vSphere deployment which comes as an OVA you can deploy - I did so in VMware Workstation.
One of my primary complaints with something like Nagios is adding monitors in, which is typically done via config files that need to be reloaded - if its not in the correct format sometimes it doesn’t work, sometimes Nagios doesn’t restart. Now that I have a host added to DataDog, I am going to take a look at how to create a monitor. Since the host I added is my Ansible master, I will look at adding a monitor to ensure that Ansible is running.
After a sad first experience with another “Software as a Service” monitoring vendor which I can’t sign up and get access to without first talking to a sales person, and because they don’t publish their pricing, I decided to give DataDog a try. DataDog is free for up to 5 hosts, sufficient maybe for very small SMBs or for a specific, small application instance or $15 per host billed annually (or $18 a month billed monthly). You can sign up for DataDog free or pro on their website, but for enterprsie support you’ll have to talk to the dreaded sales person.
In my last post, I wanted to point for people who think of OpenStack as “too complex” stacks up component for component to a VMware stack; now that is not to say they are equal in every way, far from it. They can, however, coexist as I still believe they have different purposes in the data center. This post lead to a conversation with Trevor pot which basically boiled down to KVM v ESXi - a very interesting topic, and not something we are likely to solve on Twitter or a blog post. But, as Trevor usually does, he made me think (shaking fist in air at Trevor - how dare you make me think!).
I am looking into various monitoring products, and since I may need to install them again, that means automation. With some help from Sarah Zelechoski and Larry Smith, I have the first pass done on an Ansible role for Sensu. There may be better ones out there, or you might just want to follow the directions manually but so far this role gets the install working up through the base install with examples. This is just as much about me getting better with Ansible, don’t like how I did something? PRs welcome as that is how I will learn from those with more experience.
Over the last 6 months or so I’ve had to spend more time with OpenStack that I had in my entire career combined. One thing I kept hearing was how complex OpenStack was/is, that there were to many components to keep track of. As I sat down to really think about that, especially as it relates to VMware I came up with this. Now not everything is a perfect 1-to-1 match, so please don’t tear me up on Twitter/comments, but I think you’ll get the point:
Update: If, like me, you only have the 6.0.0 version of the VCSA available to download, check out this post for how to install Update 1 to get the VAMI back. Thank you Christian for the reminder on the VAMI being brought back
Chalk this up to I should have paid more attention when RTFD (reading the documentation), but since I missed it tucked away in there, I thought others might have as well. vRealize Automation ships with several custom properties that you can use to delete Active Directory computer objects when destroying a virtual machine deployed through vRA. This is excellent for Windows shops who might otherwise have to build some other means of cleanup. One item to point out, however, is that this will delete the computer object immediately. If you have any type of retention period where you might have to restore these VMs, you would then also need to restore the AD object.
Disclaimer: I am an EMC employee. This post was not requested or required by my employer, it is simply my experience getting to know the product
I had been looking for a cloud hosted method of replacing my lab domain controller and DNS. Since Amazon offers IAM for free, I had been hoping Directory Services would also be free but it wasn’t. As I was turning back to my lab to refresh my Windows domain controller and update my Synology NAS I noticed something interesting; Synology offers both an LDAP and DNS server which can be run directly on the Synology NAS.
As part of my lab rebuild, and efforts to understand Amazon Web Services better, I am looking to setup Directory Services to use with my home lab, and potentially eliminate a virtual machine in my lab. It was pointed out on Google+, however, the dangers in doing this if I have no internet connection, or AWS is unavailable. In this post I am going to take a look at options from AWS and take into account potential pitfalls, like the one pointed out to me. Please note that I am still learning about AWS, so if I misinterprested something from the features available please let me know. Always learning and happy to have my mistakes corrected.
Ghost Inspector is a tool for generating and continuously monitoring web sites and web applications, since it is free for up to 100 tests, I thought I would give it a try to ensure certain elements of my blog are working as expected. You can sign up for your free account at ghostinspector.com
My poor lab has had enough, time to rebuild. I’ve always used a Windows Domain for authentication, and you can make that reasonably small but during this rebuild I am going to try and leverage Amazon Directory Services…because cloud? Before creating your directory service, however, you should setup accounts in Identity and Access Management, or IAM. IAM is a free service, so you can use this without incurring any additional cost.
In this post, I want to review something I have been using for a few years to stay productive at work. Before we get to that, however, there is something I need you to accept - you cannot multitask. Okay, maybe you can breath and chew bubblegum while working, but when it gets past some of the bodies basic functions, you simply can’t do more than one thing at a time. If you have manager who expects you to multitask, what they really mean is for you to be able to have multiple different projects or technologies in your work queue at the same time, but you can only ever do one of those things at a time.
As a Windows user in a DevOpsifed-Mac-Centric world, finding a good editor is a challenge - the best two native Windows text editors I have found are the good ‘ol Notepad++ and more recently Microsoft Visual Studio Code. Recently I was introduced to a web based IDE called CodeAnywhere. There are a ton of great features packed in, but maybe the most impressive to me is the free container they provide you to actually test the code you are writing.
I am starting to look at Amazon AWS and the various solutions available with it. For starters, I am looking at the various Virtual Private Cloud, or VPC, configuration options.
While Upguard (formerly known as ScriptRock) is great at monitoring and reporting on configuration state, it is not a configuration management tool in the way that Ansible or CFEngine is. They do, however give you the capability of generating what they called “automation snippets” for many popular configuration management tools to either correct an item that has failed the policy check, or even create entire “automation snippets” to full configure a node.
Upguard (formerly ScriptRock) recently announced a new, free, vulnerability scanner available within their solution. Years ago, Nessus was the defacto tool here but more recently has been commercialized. OpenVAS seems to be the new hotness for Open Source vulnerability scanning but is yet another product to introduce into the environment. In my previous two posts (compare / policies) I added two nodes to ScriptRock, I’ll use these again for testing the vulnerability scanner. If you have not already add a couple of hosts to ScriptRock before you proceed.
In my last post, I took a look at using ScriptRock to compare two nodes to see if their configuration state was identical. While this might be useful for troubleshooting scenarios, in my last example ensuring a Domain Controller was configured properly, you wouldn’t want to assume that a particular node is configured as desired at any given point in time - even if it is working as expected.
Recently ScriptRock announced their service is completely free for up to 10 nodes, when I say completely there are no features you have to pay for unlike other freemium offerings. ScriptRock allows you to monitor the configuration of your infrastructure including Windows, and Linux servers, AWS instances, and other cloud services. A really interesting new feature, if you have been interested but not sure how to get started with something like Ansible, is the ability to create an Ansible role/playbook from a monitored host. There is quite a bit you can do with ScriptRock, so let’s start by taking a look at how to compare two nodes.
**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**
Been thinking about Sysdig, and how it can be used for troubleshooting. One thought I had was to capture events during an Ansible playbook run in the event there were any problems. Now I’m not sure how practical that is just yet, but the first task was getting Sysdig installed. Of course, that meant writing an Ansible playbook to do so (really should have been a role probably but baby steps).
Ansible held a free online 2 hour introduction session, and while I’m not an expert I do feel I have a good handle on some of the items such as inventory files, and playbook formats. However there is always something to learn! One thing I took away from todays training was an ansible-galaxy command.
As I did in the previous post with vRealize Automation, it is now time to upgrade vRealize Application services, again based on KB2109760 this would be the second item to upgrade before upgrading vCenter with embedded SSO. Not that it is horribly difficult, but there is no management interface as we had with the vRealize Automation appliance so we will have to download the files, copy them to the appliance and start the upgrade.
With new versions of vRealize Automation and vSphere dropping, and seemingly being stable it is time to upgrade the home lab. Since this is a home lab, and somewhat basic there are just a few steps from KB2109760 that needs to be followed:
- Upgrade vRA (Appliance >> IaaS)
- Upgrade Application Services
- Upgrade vCenter
- Upgrade ESXi
In this post, I will cover the first step in the process, upgrade vRealize Automation to 6.2.latest. First, I have shut down services on my IaaS server. Now log into the VMware vCAC Appliance management interface on port 5480 - in my case https://vxprt-vcac01.vxprt.local:5480 for example and click on the update tab. Now, click on Check Updates. As you can see here, I have an available updated from 184.108.40.206 to 220.127.116.11
It was a bit over a year ago that I wrote about my 8-core home lab. I was asked if there were any updates to the build and I was curious to see how it stood up a year later. Happily for me, and anyone who has invested in this build, the same basic platform is still a solid option for your home lab. I have made a few tweaks below based on some new hardware being available. As I did last year, there was a focus on keeping cost down but having enough power to run a fully nested home lab.
Bob Plankers has a great post over at lonelysysadmin.net for preparing CentOS based virtual machines for being a template. As I’ve started working with Ubuntu more I decided to take that list and Ubuntu-ize it (mostly from proding by Sarah Zelechoski - one of the smartest people I’ve ever had the privilege to work with…so many thank you’s). Anyways here is that guide… Ubuntu-ized.
**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**
I’ve spent a fair amount of time over the last two years preparing for my VCDX, while the VCAP-DCA/VCIX-DCA is the last step before I fully dive in, I’ve been preparing myself not just for an exam/test but to be the best possible architect I can be. To that end, preparation is an on-going, ever evolving process.
**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**
*Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**
Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.
There are many tools out there that can help you manage your social media presence, unfortunately some of these tools take to the equivalent of spamming your followers with auto posted tweets, probably the most egregious of them is Sumall. You know the “How I did on twitter this week” tweets.
Boy, social media is proving tough to have some discussions for me lately - here is the first of
two three blog posts to set my position straight. Yesterday a conversation got started on Twitter based on a tweet shared by John Troyer “Devs Rool, IT Droolz” - in fact here is another point of view on the conversation from Rynardt Spies. Now as an “infrastructure” person you may think my take here is about saving my job, or staying relevant or some such thing but it is nothing at all about that - its about working together. In fact, those who know me well know that I am trying to push the “infrastructure people” into a more application focus, not necessarily development, but stop being infrastructure focused and work on being able to deliver applications and value to the business. I’ve never had a CFO walk into my office and say a virtual machine was down or a VLAN was misconfigured, but they know when their application is down.
Some time around the release of vSphere 5.5 (Update 2 maybe?) VMware officially(?) didn’t not support vCenter on a Windows Failover Cluster. I say didn’t not support because there still seems to be very limited documentation and KB’s on how to do this. The VMware vCenter Server Availability Guide documents available options such as using HA for vCenter availability, but also how to install vCenter on a Windows Failover Cluster, and configure the services appropriately since the application itself is not other cluster aware, for example like installing SQL on a failover cluster.
Working on building out a lab that is going to be used to demonstrate setting up a vCenter environment. We were fortuneate enough to be given some time to set it up “right” - meaning setup a SQL cluster for vCenter, SSO in HA behind a load balancer with valid certificates. I drew the SQL straw, and it s the first time I have setup SQL clustering. I had to pull from a few different resources, none were completely what I was trying to do but thank you to Derek Seaman’s blog and the MSDN blogs for being able to answer questions when they came up. You can find more information on Windows Failover Clustering on vSphere 5.5 here (nope not on 6 yet). An over view of our setup:
- Two Windows 2012 R2 virtual machines on separate hosts; SQL1 and SQL2
- Each virtual machines with two NICs; one for production/client access the other for cluster communication.
- Each virtual machine has 2 drives; 60GB "C" for OS and 20GB "D" for SQL installation
- 3 XtremIO drives presented via VPLEX
- AD accounts for SQL and SQL Agent were created in AD
- IP addresses for each of the SQL virtual machines, the Windows cluster, and the SQL cluster; for this setup that is 4 total.
Windows was installed, patched and joined to the domain. On each virtual machine I ensured that Windows Ethernet0 was first in the biding order and used for “production.” NIC1 would be used for cluster communication. Ensure RSS is not enabled on the NICs.
Disclaimer: After being a customer in my last role, I now work for EMC. This post is purely my opinion and was not requested, read, or approved by my employer
Duncan Epping recently posted an article called “How do I get to the next level” which was an interesting read, that I almost didn’t do. See in the beginning of the article he stated
If you can’t be bothered freeing up time, or have a too busy family schedule don’t even bother reading past this point.
Typically I’d stop there, because for me nothing is more important than family, and dedicating time to be with my daughter, wife, extended family, and friends. I’m not interested in reading an opinion if family schedule is a consideration, because for me it is. While I am no where near Duncan’s status or skill, I think there things everyone can do, regardless of family schedule to help you grow personally and professionally.
I wanted to share some of the example Ansible playbooks used during last Wednesday’s US #vBrownBag. During the show I went over examples of how you can use Ansible to create, clone, and update virtual machines in vCenter without the need for other provisioning tools. Based on my testing (and I’m still learning as well), the items noted in the comments are the bare minimum needed to run the playbook, even though the official documentation may currently state otherwise. If you are already using Ansible for configuration management, this is a handy option to have as you can perform the provisioning tasks without leaving Ansible.
During the #vBrownBag DevOps series after-show from my Using Ansible to provision VM’s in vCenter, Mike Marseglia asked about options for linting Ansible playbooks. Since I didn’t know, I thought it would be worthwhile to look into it. There is an Ansible-Lint repo on GitHub, reading through the information, it seemed straight forward. Here I am going to have a look at installing and using it against some example playbooks.
Scenario: You try to install the VMware vCenter Server Appliance (VCSA) or Platform Services Controller but receive an error during the installation. After correcting the problem during installation you attempt to re-install the appliance but receive the following error message:
Generally, installing virtual appliances has been pretty straight forward – import an OVA and enter the necessary details in the deployment wizard, or access the virtual appliances management interface (such as those typically on port 5480 from VMware). However, as of the Release Candidate for VMware vSphere 6.0, the vCenter Server Appliance (VCSA) installation takes a much different approach than what you’ve been used to.
During the installation of of the VMware vCenter Server Appliance (VCSA) 6.0 or the Platform Services Controller (PSC) Appliance 6.0, you receive the following message:
During the #vDM30in30 challenge I started playing with Ansible to get to know it a bit better. One of the things I was curious about is the ability for Ansible to provision virtual machines directly to vCenter. After all if I am using Ansible to manage the configuration of my servers, it would certainly be nice to have a playbook that also deploys my virtual machine, rather than another provisioning tool.
[display-posts category=”back to basics” display-posts posts_per_page=”15”]
Please note that the installation steps here and requirements are based on beta and release versions of ESXi 6.
Before you get started with virtualization in your environment there are a few things you will need to have in place.
Now that Nagios Log Server is installed, it’s time to get some log files in there. I got myself all fired up ready to comb through page after page of documentation to figure out how to set it up… then those nice folks over at Nagios did this…
When I think of syslog servers, I tend to think of VMware Log Insight and Splunk on the commercial side, and SyslogNG or an ELK solution like the one Larry Smith has blogged about in the past. I’ve never thought of Nagios; turns out they have a logging solution of their own, and it leverages the ELK stack. For many deployments, Nagios Log Server will fall into the commercial category, there is a free version which supports a single instance of Log Server running and a maximum of 500MB logged in a day (according to http://logfilemonitoring.com/ which appears to be affiliated with Nagios.com). However, for SMBs who may only have a few servers, or to support a specific application, Nagios Log Server may do the trick. Only one way to find out right? Let’s get it installed!
Chalk this up in the “useful error messages” column. When you attempt to enter a license key in the vRealize Automation appliance you receive “Error code: 500.”
Yesterday Tim Jabut forked my Ansible test repo on GitHub to help me get markdown working and I merged that back into my repository. Today it is time to learn how to fork a repository on GitHub myself. If you take a look at a repository on GitHub, you’ll see a Fork button on the upper right corner of the page:
Today I was chatting with Tim Jabaut in a Slack room Matthew Brender created for Commitmas (ping him or Josh Cohen to get in) and he shared a nice markdown cheat sheet (I seem to be all about cheat sheets during Commitmas). If you have looked at the Commitmas GitHub page you see that Matt and others have made his page pretty; it has been done using markdown. I tried adding some simple markdown to my README file on my Ansible Test Playbooks page but they were just coming over as ##, not as headings.
Day two of commitmas, I have my Windows computer setup and SSH keys added to my GitHub account. Time to clone my existing repository to make a few edits. First, I created a directory on my computer called ‘git’ where I’ll save all my work; cd to that directory and run git init
In part 4 we published an application blueprint through Application Serivces, that is pretty awesome but we still really haven’t done anything just yet. I mean its all just about working but the real hard part is creating the application blueprints. Just for fun, lets create a generic blueprint and run a deployment. While logged into Application Services go to Applications and click on the green + (plus) button to create a new application.
- Name the application and select a business group, if you've followed along my various home lab series you would select StarWars here since it is the only business group we gave permission to in vRealize Automation.
- Click save, click Create Application Version then click Save
- Now you are able to create a blueprint; click Create Blueprint
- Drag the logical template to the design pane, again if you're following along with me this would be the CentOS 64 logical template
- Now all this would do is create a virtual machine like you could do through vRealize Automation or vSphere; here however we also have several preconfigured services we can drag into our logical template to install applications.
- Let's do a typical single node web and database server
- Drag Apache, vFabric RabbitMQ and and vFabric Postgres into the logical template, it should look something like this:
Now one of the hardest parts about automating something is now all the dependencies. In this scenario I happen to know a few things are missing, not because I am a genius but because I went through several iterations of this blueprint before getting it to work. This, however also allows me to demo some other features of Application Services. In my CentOS template, SELinux is enabled - now I could convert my template to a virtual machine, disable it, clean up the virtual machine machine again and convert it back to a template. It’s what I would have done not 6-8 months ago. Now, however, I’ll simply use the tools available to me, tools like Application Services or Ansible to put the virtual machine into the state I want it:
- From the Application Components page, drag two "script" items into the logical template
- Edit the first script by clicking on it; name it (no spaces), click on Actions, click "Click here to Edit," copy the following into the window and click the reboot checkbox
#!/bin/bash # set SELinux disabled cp /etc/selinux/config /etc/selinux/config.bak sed -i s/SELINUX=permissive/SELINUX=disabled/g /etc/selinux/config
- SELinux will now be disabled upon reboot.
- We also have to tweak the EPEL install to allow it to pull data properly (seems to be a known issues right now). Rather than letting the EPEL package install as part of the services we used earlier, we can also do that in a script and configure the options we need for it to work.
- Edit the 2nd script as you did before but copy the following into the window
#!/bin/bash # install EPEL yum -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm sed -i “s/mirrorlist=https/mirrorlist=http/” /etc/yum.repos.d/epel.repo
- Click the OK button, you should now see something like this:
- Now click the deploy button, name the deployment, and select the business group
- Click Map Details, ensure all details match what you have setup, and click Next
- Provide a name to your virtual machine and edit CPU and memory as needed (and to match your vRA blueprint limits) - click Next
- Review the deployment blueprint and click Next
- Click the deploy button (you could also publish to vRA here as we did in part 4, but I'm just demonstrating the deployment)
- The deployment will start
Now at one point I wasn’t sure it was working, I could see Application Services say it was working (system was under 80-90% load consistently) however I wanted to see what vSphere was doing. As you an see in the two screenshots below, the virtual machines are being deployed as you might expect (they are from two different deployments so yes the dates are different)
When deploying an Application Services blueprint, you notice that the workflow does not move past the 2nd step in the provisioning process - agent_bootstrap node setup, however the previous step which renames the virtual machines appears to work fine. In this scenario you have also successfully installed the AppD agent in the vSphere template.
Now that CentOS7 is out, time to make sure I can setup my virtual machines with the VMXNET3 vmnic. As I documented in my previous post, CentOS 6.x using the VMXNET3 driver requires VMware Tools, VMware Tools needs Perl, Perl is not included in the minimal ISO so I need network access to get Perl to install VMware Tools to get network access. That order of operations doesn’t work very well.
In my last post, I showed you a simple example of an Ansible playbook using yum to update a package. Still really awesome, especially when you consider how often you might need to do that and how simple it is to handle that type of otherwise manual task. In this post, I am going to try and put together a slightly more complicated playbook to look at some of the other options available.
With Ansible installed, and a basic inventory file created we can now move beyond ad-hoc tasks (which by the way is still a great use case for Ansible) and take advantage of Playbooks. Playbooks are a set of commands organized as required to perform complicated tasks. Maybe you have provisioned dozens or hundreds of new virtual machines and you now need to make sure they are in the desired state - standardized versions of OpenSSL or MySQL for example, then deploy your custom software packages to those servers; that (simplistically) is where playbooks come in.
A question came up on Twitter recently about how to add a stand-alone Hyper-V server as an endpoint. What I can gather from the documentation is that you need to have an agent deployed for Hyper-V but the directions were otherwise unclear so this is an attempt to document the required steps. First, the assumption is you have a Windows Server with the Hyper-V role at a minimum available (if you are running as a virtual machine; make sure the OS type is set to Hyper-V).
Please note these directions were based on Ansible 1.8. Please check Ansible documentation for updated information
In my last post on Ansible, the installation documentation walked us through a simple example of how to issue a command on a host by putting 127.0.0.1 in the inventory file. Now as you know 127.0.0.1 is that server itself; the real power of an automation tool is working on multiple systems. You can manage which systems Ansible runs commands or playbooks on (more on playbooks in a future posts) by putting them in an inventory file - and what’s really cool; Ansible does this all agentless!
[display-posts category=”Application Services Lab” display-posts posts_per_page=”15”]
Application Services is configured, now its time to create and publish an Application Blueprint. During the installation I chose to install the sample content so I would have some an existing application blueprint available; I am going to take advantage of that sample content for my lab and edit one of the existing applications. If you are not already, log into Application Services as luke and use the pull down menu in the upper right to change to the Application view.
- Click on jPetStore
- In the Application Versions pane, click on the 1.0.0 version
- Click on the blueprint
- Click on CentOS32 v6.3
- Drag the CentOS64 v6.4 logical template into the application builder
- Drag the components from jPetStore to the CentOS64 operating sytem
- Delete the CentOS32 item and click the Save button
- Click the Deploy button in the upper right corner
- Name the new deployment profile and select the business group; Click the Deploy button
- Select the Deployment Environment, click the map details button then click Next
- Click into the hostname field and enter a name then click Next
- Review the execution plan and click next
- Click the publish button, name the item and click OK
Now we need to provide entitlements in vRealize Automation; log into vRA as tenantadmin:
- Click on the Administration tab >> Catalog Management >> Catalog Items
- Click on jPetStore (or whatever you named it), add it to the Clone Linux Template service and click Update
- Log out and log back in as luke
- Click on the Catalog tab; you should now have your basic VM template catalog item and the jPetStore catalog item.
In my last post we created the Cloud Provider, now we need to setup a Deployment Environment. We are really setting up logical constructs here to map to to services and resources we already have. So far we mapped a Cloud Provider to vRealize Automation Center and to our business group. Now we are going to create a deployment environment for Application Services that maps to the cloud provider we created, that maps to vRealize resources… If you are not already, log in as Luke and perform the following.
- Click on the Cloud Provider pull down menu in the top right corner (short aside, that menu name will change to show the context you are currently working in so if you logged out this may be different) and select Deployment Environments
- Click the Create A Deployment Environment button
- Provide a name
- Select a Cloud Provider from the pull down menu
- Click the Select button to select a reservation policy; click OK then Save
You screen should look similar to what is pictured below. Since Application Services is now part of vRealize Automation, most of the work we are doing here will map to what has already been configured in vRealize Automation. Next we need to create a logical template.
VMware Application Services (formerly Application Director) is now deployed, but we need to do a bit more integration with vRealize Automation / vCloud Automation Center so we can publish Application Services blueprints to the vRealize Automation catalog. First we need to define a cloud provider;
- While logged in as Luke, the user we gave all of the Application Services roles to, click on the Applications pull down menu and select Cloud Providers
- Click the Create Cloud Provider button/box
- Click the Cloud Provider Type pull down and notice what options are available - vCloud 5.x, vCAC and EC2. What about vCloud Air - can we use that? If you said yes you are correct because vCloud Air is based on vCloud Director.
- Enter the information like so (note some of the boxes appear "greyed out" - they are not, just a poor choice for background colors) and click the Validate Connection button
- Notice that you have to use an upper case domain, I'm curious as to why but in any case its the only way that worked for me
- Next in the lower half of the screen (not pictured above) in the templates section, click the green plus icon
- You should see the CentOS-Template catalog item we previously published in the vRealize Automation catalog; click the check mark next to the desired template and click OK
- Click the Save button in the upper right hand corner
We now have the first step in setting up Application Services complete, up next we will create a Deployment Enviornment
Home stretch, 15 posts and we are about to see our first catalog item published! Lets get going and create the entitlement which is how we define what can be done in vRealize Automation / vCloud Automation Center
- Log in as tenantadmin
- Click Administration >> Catalog Management >> Entitlements
- Click the Add button and fill in the information as follows
- Click the Next button
- Click the plus sign next to Entitled Services, select Clone Linux Template and click OK
- Click the plus sign next to Entitled Catalog items, select CentOS template and click OK
- Click the plus sign next to Entitled Actions, Select Machine from the pull down and chose all of the items, Select Virtual Machine from the pull down and select Destroy; click OK
- Click the Add button
Log out as tenantadmin and log back in as luke, you should now see your vSphere template, which is now a vRealize Automation / vCloud Automation Center blueprint published!
We’ve got our reservations done, but so far we haven’t created any catalog items for our Georgia and Alderaan employees to actually request. One of the simplest things to publish in the vRealize Automation / vCloud Automation Center catalog are virtual machine blueprints; which are created from vSphere templates. With our virtual machine converted to a template in vCenter, we should be ready to go.
- Log into vRealize Automation as iaasadmin
- Click on Infrastructure >> Blueprints >> Machine Prefixes
- Create a Prefix like we did for our business groups, call this one nix
- Click on Infrastructure >> Compute Resources >> Compute Resources
- Hover over cl01 >> Data Collection
- Wait a few moments and click the Refresh button at the bottom of the screen; status should be Succeeded
- Under Inventory, click Request Now
- Log out and log back in as tenantadmin
- Click on Infrastructure >> Blueprints >> Blueprints
- Hover over New Blueprint >> Virtual >> vSphere (vCenter)
- Fill in the build information similar to below
- Click the Build Information Tab
- Change Action to Clone
- Click the ellipse next to Clone From and select your linux template and click the OK button
- For testing in the lab, leave everything else as is and click the OK button at the bottom of the page
- Hover over the new Blueprint, click on Publish then click OK
- Navigate to Administration >> Catalog Management >>Services
- Click the Add button, name it Clone Linux Template, set it to Active, and click the Add button
- Highlight the new service and click the Manage Catalog Items button
- Click the green + icon, select CentOS-Template and click Add
- Click Close
Almost there I promise, now that the blueprint, service and catalog item is created, we just need to provide entitlements so our users can see it!
In order to use vSphere templates in vRealize Automation / vCloud Automation Center and Application Services / Application Director there is a bit of preparation you need to do, especially if you want to use Application Services. There are guest agents for both vRealize Automation and Application Services so lets get started. A quick assumption here, you already have a linux virtual machine installed with VMware Tools. I am going to cheat a bit here and use the e1000 NIC, if you want to use the VMXNET3 adapter see my post on how to install VMware Tools…which needs Perl…which needs network access…which needs Perl! Let’s get started with the specifics on configuring your Linux VM; I have a CentOS virtual machine called vxprt-centos-tmp that is powered on and ready to configure. Log in via the VMRC or SSH to get started:
One of my goals in the #vDM30in30 challenge was to expand my comfort level with vRealize Automation / vCloud Automation Center and Application Services / Application Director. To that end, it's time to deploy the AppS appliance. In vRealize Automation 6.1, Application Services (formerly Application Director) became a "component" of vRa. I guess what the marketing department meant by "component" was that it is a completely separate virtual appliance with its own management UI and integration with vRealize Automation :)
In 6.0, like the vCloud Automation Center appliance, the initial OVF import was important. While there are documented ways to manage the network settings that "should" allow you to make changes, I found that the changes were not always persistent so again I am careful here and probably carrying with me some bad memories. Log into the vSphere Web Client, if you are running your VMs on the VMware Workstation NAT'd network, do so from your DC.
- Click on vCenter >> Hosts and Clusters
- Right click on your cluster and select Deploy OVF Template, if prompted click allow
- Browse for the location to your Application Services OVF and click Next
- Click Next, Accept, and Next
- Name your appliance, I'll be keeping with my convention and use vxprt-apps01, and select the datacenter or folder you want to deploy to
- Select the datastore, then ensure you have selected Thin Provision
- Connect to the appropriate port, change the IP allocation pull down to Static and fill in the DNS, Gateway and Netmask fields; click Next
- Enter the IP address for the appliance and click Finish
- While the appliance is being deploy, open DNS manager and create an A record in both the forward and reverse lookup zones.
- If, like me you are limited to lab resources, change the amount of memory for the virtual machine to 3 or 4GB. I changed mine to 3GB and it seems to be working fine.
- Once the deployment finishes, power on the virtual machine and open the VMRC; you will see a prompt to enter the serial number for Application Services:
- Enter your serial number and press the enter key on your keyboard
- Enter the new OS root password when prompted (you can ignore the errors about weak passwords...not that mine is weak ... :)
- Enter the OS darwin_user account password
- The appliance will configure its initial configuration process - now would be a good time to also update your host file if necessary on your workstation. Services can take awhile to start since I dropped the memory of the system
- You will be asked if you want to use this instance for a migration from 6.0.1, in our case the answer is N
- Next, provide your vRealize Automation / vCloud Automation Center Server URL; in my case https://vxprt-vcac01.vxprt.local
- Enter email@example.com when prompted for the administrator username
- After a few moments you should get a prompt saying Registration is successful
- You will now be asked if you want to setup Out-Of-Box sample content; I am selecting Yes
- Next, provide the Tenant Name - in my case vsphere.local as I am using the default tenant for my lab
- Here, we need to switch back to the vRealize Automation web console for a moment as we need to give a user the appropriate roles to import content.
- Log in as tenantadmin, click on Administration >> Users; search for one of your Business Group admins, in my case either Rick or Luke, type in their name in the search box and click on the user when it appears
- Give the user the top 4 roles Application Architect, Catalog Administrator, Cloud Administrator, and Publisher/Deployer (probably don't need all, but can't decipher what specifically it needs from the documentation)
- Click Update; the user luke now has all Application Services related roles
- Switch back to the VMRC; enter luke as the username and then the password password
- Enter the business group that should have access to the sample content, in my case StarWars
- Once complete you should see that you can also import out of box content for other tenants again by running /home/darwin/tools/import_oob_content.sh
- Press any key to continue
- Enter the new password for the Application Services admin account (I know a lot of accounts huh)
- Setup will finish boot until you see the typical VMware appliance console
- Open a web browser and navigate to https://vxprt-apps01.vxprt.local:8443/darwin/org/vsphere.local
- You can log in as any tenant user that you configured; for example try luke
That is the basic deployment of VMware vRealize Automation / vCloud Automation Center Application Services - up next we have a bit more configuration to do to make the Application Services Available as a catalog item in the vRealize Automation / vCloud Automaton Center catalog.
So far we are doing pretty well, but we aren’t quite ready to turn vRealize Automation / vCloud Automation Center lose yet, next we will create reservations so users can’t consume all of our resources. Wait wait wait….why would we create a reservation to do that you fool - reservations “reserve” something for us - haven’t you ever read Jonathan Frappier’s book VMware vSphere Resource Management Essentials? Why yes, I have its a lovely book but a reservation in vSphere is not the same as a reservation in vRealize Automation / vCloud Automation Center - in fact they are generally used for opposite reasons (can you tell I’m working overnights in some of these posts :) ? )
We are cruising right along here in our vRealize Automation / vCloud Automation Center setup. So far we have everything installed, permissions assigned, a vCenter endpoint added and fabric group created with the cluster from our vCenter server. Now its time to setup business groups. Business groups are just a logical group of users, this may be done per department, per project or per external customer. We can publish catalog items to business groups, so when planning your business groups think of the things certain groups may or may not need. For example you may want a business group for your QA department that only has access to builds that are currently being tested so they do not chose the wrong version to deploy, or not want finance see HRs catalog items. Consider helpdesk users, you may want to publish certain catalog items for them to do certain tasks like create AD users and groups through vCenter Orchestrator workflows or PowerShell scripts - the possibilities are seemingly endless.
Time for Fabric Groups, and no a fabric group is not what Grandma does on Saturday afternoons at the senior center. Fabric groups in the vRealize Automation / vCloud Automation Center world is a collection of resources, this tends to send folks who have been storage focused for a long time down a different path as they start thinking about zoning and switches.
With administrative users setup so we can actually configure various options in vRealize Automation / vCloud Automation Center - its now time to add some compute resource, so we can actually deploy things! Endpoints in vRealize Automation / vCloud Automation Center can be several things:
- Hypervisor management platforms such as vCenter, vCenter Orchestrator, and SCVMM
- Cloud providers such as vCloud Air (formerly vCloud Hybrid Service), vCloud Director providers including vCloud Air, OpenStack using RHEV 3.1
- Physical hardware from Cisco, HP, and Dell
If you recall from the IaaS installation post, one of the options asked us to name the vCenter endpoint, now we are going to log in and configure our vCenter server as an endpoint so we can use it to deploy virtual machines through the catalog.
- Log into your vRealize Automation / vCloud Automation Center appliance (in my case https://vxprt-vcac01.vxprt.local/vcac) as.... do you recall from the last post who to log in as? That's right iaasadmin as that user was assigned the infrastructure administrator role which can manage endpoints
- Once logged in, click on the infrastructure tab >> Monitoring >> Log
- Notice here the errors related to the VRM agent occurring every minute, that is because we have not added our vCenter server yet
- Click Back to infrastructure, then click on Endpoints >> Endpoints
- Hover over New Endpoint >> Virtual and click on vSphere (vCenter)
- We will need to name the endpoint, the URL to the vCenter SDK and credentials. Since we have not already, I would create a user account in AD called svc_vra_vcbind and add it to the vcAdmins group (assuming you followed along on the home lab build, or give it admin permission directly to your vCenter)
- Fill in the following information:
- Name: vCenter (Do you recall why we are naming it vCenter?)
- Address: vcenterurl/sdk (in my case https://vxprt-vc01.vxprt.local/sdk)
- Credentials: Click the ellipse, then New Credentials and add a user account with permission to vCenter and click the green circle with the checkmark
- With all the information filled in, click the OK button
- Once vCenter has been added, return to the log view, you should notice that the VRM Agent errors are no longer occurring
Now you’ve added resources vRealize Automation / vCloud Automation Center can use to fulfill requests, however we still need to assign those compute resources to a fabric group, which will be next in my vRealize series!
With the installation out of the way, we can now start to configure permissions beyond the firstname.lastname@example.org account so we can log in and actually do cloudy things. First though, lets create a couple of accounts in active directory so we can drive home the different roles. Log into your DC and create two user accounts - tenantadmin and iaasadmin.
Almost time to install the IaaS components, but we still need a SQL server so lets get this party started, if you are installing SQL2012 you can check out my older post, in this post I will be installing 2014 with a bit less detail on selection choices.
- Navigate to your download location of SQL and launch the installer
- Click OK to allow the installer to extract the files to the requested location, the SQL Server Installation Center will launch
- Click on New SQL Server stand-alone installation
- Accept the license terms and click Next
- I am going to skip the updates here for the sake of time, so leave the update box unchecked and click Next
- On the Feature Selection screen I am only keeping Database Engine and sub components and Management Tools - Basic and Complete in the default directory
- On the Instance configuration select Default and click next
- On the Server Configuration page change SQL Server Browser to Automatic and click Next
- On the Database Engine Configuration page change to Mixed Mode and set the SA password
- On the Specify SQL Server administrators section add svc_vra_iaas (we can remove this user as a SQL admin after the installation of the IaaS components), click Next to proceed with the installation
- Once finished, click the Close button and close the Installation Center screen
Before we move on to installing the IaaS components, lets log into SQL and make sure everything is as we want
- Open the Start menu and type Management Studio
- Log in with with Windows Authentication
- Expand Security >> Logins
- Confirm VXPRT\svc_vra_iaas is listed, right click on that user and select Properties
- Click on Server Roles and ensure sysadmin is selected
With SQL now installed, it’s time to download and install the IaaS or Infrastructure-as-a-Service components. Log into your IaaS server as the service account you created and provided to the PreReq script - in my case svc_vra_iaas. Once you are logged in, close Server Manager, open a web browser and navigate to the URL of your vCloud Automation Center / vRealize Automation appliance, for example in my case https://vxprt-vcac01.vxprt.local. You should now be at the VMware vCloud Automation Center Appliance page where we can get started
Now that we have the vCloud Automation Center / vRealize Automation appliance deployed and working, it is time to get the Infrastructure-as-a-Service components installed. The IaaS components run on Windows and need to connect a Microsoft SQL server instance. In my case I am going to use SQL express installed on the same VM I will install the IaaS components on, again we are in a lab here - typically you would have a separate database server to host the DB. As always, and a point I’ve not referenced really at all this month, always check the documentation to ensure you are installing on a supported platform. The vCloud Automation Center / vRealize Automation documentation is excellent, as is the support matrix PDF.
In the last post we deployed the vCloud Automation Center / vRealize Automation appliance, while its mostly a straight forward installation I will likely hold on to the horrors (…okay horror is a storng word) of my first deployment where I could not resolve typos from the OVF wizard - take your time deploying the OVF for a smooth experience. Now its time to configure the appliance before we move on the IaaS installation. If you have not already done so, make sure you can resolve the FQDN of your appliance, if like me you are running the appliance behind the VMware Workstation NAT network you will need to likely add a host file entry or use your lab domain controller as your DNS server - I went with the former.
- Navigate to https://vxprt-vcac01.vxprt.local:5480 (replace with your URL)
- Log in as root and the password you set during the OVF deployment
- You will now be in the appliance VAMI with no services showing, don't work that is expected
- Click on the System tab >> Time Zone; set your time zone appropriately and click Save Settings
- Click on the Admin tab >> Time Settings; change the Time Sync Mode pull down menu to use time server then set the time server to the IP address of your domain controller (or dedicated NTP server) and click Save Settings
- Verify that NTP Status is Enabled: Yes, NTP Started: Yes,
- Click on the vCAC Settings tab; click Resolve Host Name - it should fill in the FQDN of your host, click Save Settings
- Click the SSL tab, from the Chose Action pull down menu select Generate Self Signed Certificate. Fill in the Common Name with the FQDN and other fields as appropriate, here is what mine looks like:
- Click the Replace Certificate button; after a few moments you will see a message that the certificate was successfully replaced
- Click the SSO tab - you will see the SSO status is Not Connected, because we haven't connected it yet :) Enter your vCenter server FQDN with port 7444 appended to the end, for example I am using vxprt-vc01.vxprt.local:7444. Now enter the SSO administrator username which if you've not created any other users in vsphere.local would be email@example.com and the password. Of course for production deployments I'd make an additional user account specific for this; sso_bind_vcac or similar. Once everything is entered, click Save Settings. When prompted click OK to accept the certificate.
- Don't panic here, this part takes a few minutes to complete...tick tock tick tock...
And here we are - SSO configuration updated successfully
I was going to do a post on NFS versus iSCSI, to be honest that is such old hat in my opinion it doesn’t really matter. Whether you use iSCSI or NFS is up to you, your application and business requirements along with any constraints in your infrastructure that may force you to lean one way or another. Since I am an NFS networking ninja, clearly I am going to go the NFS route. Let’s get started on setting up NFS, if you are not already log into your Synology DSM.
- Click on the main menu button on the upper left and open Control Panel
- Click on the File Services icon
- I have no need for CIFS or AFP at this time so I am going to disable those; expand the Windows File Service section and uncheck Enable Windows File Service; repeat for Mac File Service
- Expand NFS service and check enable NFS
- Click the Apply button
- In the left navigation window click on Shared Folder
- Click the create button
- Provide the necessary details for your folder I am naming my folder vxprt-silver01-ds01 which will be on the SATA drives; click OK
- Click on the NFS permissions tab and click the Create button
- In the hostname/IP field enter the range for your ESXi hosts, in my case its all the same network so 192.168.0.0/16
- Click OK twice
- Make note of the mount path value, we'll need that later
- Repeat for the folder on the SSD volume, I am naming htis folder vxprt-gold01-ds01
- You should now have two folders created
With the decision made to use vSphere SSO for vCloud Automation Center / vRealize Automation, it’s time to deploy the vRealize Appliance. The appliance comes as, well an appliance as the name suggests and is a straight forward OVF deployment. One thing to keep in mind, at least as of 6.0 it was very difficult to change the network settings that were supplied through the OVF properties during the deployment. If you fat finger something here just punt and redeploy - something I’m likely to hold on to for a while.
I just got a Synology DS1513+ and wanted to try out the SSD cache. Having never powered it on I pulled two of the 2TB Seagate drives and installed 2x Corsair SSDs. Once I powered on the device, it started beeping and wouldn’t stop. Turns out that when shipped with drives there is an existing volume already created. The beeping was an error because I basically broke the volume removing the two 2TB drives. To turn off the beeping, do the following:
- Log into DSM, since I am assuming this is a new deployment you can find the IP at https://find.synology.com
- Log in as admin with no password
- The control panel window will open
- Click on Beep off, take aspirin to fix the headache
- Close the control panel window
- In storage manager you will see Volume 1 in a crashed state, highlight it and click remove
- Click OK then yes to confirm deleting the volume
- You should now see no volumes in storage manager and the disk station health change to good
- You can now go about creating volumes as you see fit
Having purchased other Synology’s with no drives in them I didn’t expect the volume to already exist. If your Synology is beeping, log in and check it out!
In order to provide shared storage to my home lab, I am going to use a Synology DS1513+. In my lab I have my DS1513+ connected to a switch, which is connected to my home router, this allows me to use http://find.synology.com to start configuring my DS1513+.
One of the first steps generally in setting up vCloud Automation Center or vRealize Automation would be to deploy the identity appliance, however you can also use an existing vSphere SSO implementation. In fact, VMware has gone so far as to publish a technical white paper on how to configure vSphere SSO for high availability for use with vCloud Automation, now I won’t be doing that just yet but know its possible.
So you’ve got vCenter up and running and hosts added, it’s time to enable the cool things vCenter can do - namely vMotion, HA and DRS. I’ve gone back and forth on how I wanted to present vMotion and networking in the home lab. On one hand many existing deployments are likely running 1Gbps, though newer ones are likely to start with 10Gbps as prices have dropped. After a quick Twitter chat I decided to move forward as I would if I had 10Gbps networking and not have separate physical interfaces in my host for different traffic types.
vCenter is built, now we can start doing some of the cooler things VMware vSphere has to offer; up first - Dynamic Resource Scheduler. DRS can be run in either manual, partially automated or fully automated mode. Partially automated will make initial placements of new virtual machines and virtual machines during power on operations and suggest how to rebalance the cluster. Fully automated, well its fully automated. It will balance cluster resources based on how aggressive you want it to be. For a deeper dive into DRS, check out the Clustering Deep Dive book, basically the bible for all things HA and DRS.
The home lab is getting close! With the vCenter Server Appliance deployed and basic configuration done, its time to get vCenter setup - AD permissions, Data Center, Cluster and adding hosts to the cluster. While there are only 2 hosts so far in the home lab, its still good to get an idea of all of the functions / features so here we go.
All right - ESXi hosts built, datastores created (on at least 1 ESXi hosts) so lets import the vCenter Virtual Appliance. The VCSA should be a bit lighter weight for our home lab that vCenter on Windows + SQL. Before getting started, make sure you have download the VCSA from VMware and placed it in a location accessible to the vSphere Client.
- Launch the vSphere Client and connect to one of the ESXi hosts you added the local datastores to, in my case vxprt-esxi01
- Click on File >> Deploy OVF Template
- Browse to the location of the VCSA you downloaded from VMware and click Next, then Next again
- Name the VCSA, I'll keep to my naming conventions so vxprt-vc01 and click Next
- Select the storage you wish to place the VM on and click Next
- Select Thin Provision and click Next
- Here you could click finish, I am not as I also want to demonstrate importing the OVF using PowerCLI so I have clicked Cancel
In the last post we looked quickly at importing the vCenter Server Appliance through the vSphere Client, however its high time we introduce PowerCLI. PowerCLI is a set of PowerShell cmdlets to manage your VMware environments (vSphere, vCloud Director and View) and is quite powerful. So powerful in fact that this is going to be a pretty short post, the 7 bullet points needed just to import the OVF through the vSphere Client is now a single command!
Now that we know some of the tools that are available to manage ESXi hosts, lets use one to create a virtual machine, in this case importing a virtual appliance available as an OVF. Before we do that, one tiny piece of business to take care of; creating a datastore for our virtual ESXi hosts. In our home lab environment, adding a local datastore to an ESXi host is as easy as editing the virtual machine properties in VMware Workstation. Right click on the ESXi virtual machine and select Settings; click the Add.. button at the bottom of the Settings window and select hard disk. Follow the wizard to place the file in the desired location.
So had a need to clone a vApp several times, I finally got around to automating that task thanks again to PowerCLI. A few things I had to consider; with the New-VApp cmdlet you cannot select portgroups so I had to do that after the vApp was cloned and also needed to put the vApp into a specific folder after it was cloned. Other than those two considerations, it was actually kind of easy to figure out (at least based on what I needed to accomplish). Two thing I could not do in this script - place the cloned vApp into a datastore cluster and allow storage DRS to make the initial placement. Instead, I am relying to SDRS to balance the datastores after the power on operation. Also I could not force the virtual disk format size, I want them all thick, eager zeroed so instead I ensured the source vApp was set properly.
So now we’ve got two ESXi hosts and our domain controller running in the home lab, it’s almost time to setup vCenter however, in a real world scenario you would need a way to get vCenter onto the ESXi hosts (because of course you are virtualizing vCenter). Up until now what we have done through the DCUI would have been at a keyboard and mouse or virtual KVM (such as Cisco UCS or HP iLO) and we cannot create virtual machines via the DCUI. So, what tools are available to manage our ESXi hosts to start creating virtual machines?
So day to day I help maintain a bunch of vApps that run, but are also disposable. When we are finished with them I just want them gone. I am working on a PowerCLI script to help automate this process and saw the Stop-VApp cmdlet. Comparing that to the Stop-VM cmdlet there was no -kill switch in Stop-VApp so I wasn’t sure how to just “pull the plug” - after all I don’t care about the vApp at this point.
Now that we have finished building the template for our ESXi hosts used in our home lab setup, it’s time to start cloning. The process is not all that different from cloning the Windows virtual machine we did earlier, so a quick overview:
- In VMware Workstation, right click on your ESXi template virtual machine (if you've been following along it should be vxprt-esxi-tmp) >> Manage >> Clone
- Follow through the wizard, selecting Clone from "An existing snapshot..." and "Create a linked clone"
- Name your VM accordingly and place it in the desired folder/drive. I will name my ESXi virtual machines vxprt-esxi## so my first will be vxprt-esxi01
- Once the clone finishes, close the wizard, move the VM into your desired folder (if you are using folders) and power it on
After a few moments, our ESXi virtual machine will be powered on. Once it is clone, my preference is to give my ESXi hosts static IP addresses. Before doing that, log into your Domain Controller, open DNS manager and create A records for each of the ESXi virtual machines you plan to create by right clicking on the forward lookup zone for your domain and selecting New Host (A or AAA)… My IP scheme will be:
With the Windows template set and our first VM working, its time to make an ESXi template we will use in the home lab. I mean it is small - I typically install with only a 1GB OS drive but why not use the features in VMware Workstation, so I am going to setup a clone. Setting up the ESXi VM is pretty much the same as what we did in part 1, so I am not going to rehash that (hopefully you learned something in part 1) but I’ll note some tips here none the less.
Alight, so far we have built our Windows template in VMware Workstation that we will use for various home lab purposes, cloned it and got the first clone ready to be a domain controller. Given the limited resources in the lab, I’m not sure I want to tackle PKI at this time, though maybe I’ll try a lightweight opensource project at some point. Anyways back to why you are here, configuring Active Directory;
- The last thing to do before promoting the server to a DC is to give it a static IP address, after all we don't want that changing (even if we are using DNS for everything). Bring up the Start menu
- Click on Control Panel >> Network and Sharing Center >> Change Adapter settings
- Right click on Ethernet0 and select properties
- Double click on Internet Protocol Version 4 (TCP /IPv4)
- Change Obtain an IP address automatically to use the following and enter the IP information for your network. In my case I will set it to 192.168.6.5 with a subnet mask of 255.255.255.0 and a default gateway of 192.168.6.2 for my NAT'd VMware Workstation network. If it is not already, set the Preferred DNS server to 127.0.0.1
- Server Manager should still be open from the previous post - if not open it.
- Click on AD DS in the left navigation menu. You should have a yellow bar that says Configuration required... click the Yellow Triangle in the upper 1/3 of the window
- Click on Promote this server to a domain controller
- Select the Add a new forest radio button
- Specify your root domain name. If you are included to pay for SSL certificates use a valid TLD that you own as there are very few providers offering certificates for private domains such as .local. I am going with all self signed certificates in my lab (for now) so I've chosen vxprt.local (.lan has troubles with OSX...at least it used to) and click Next
- On the Domain Controller Options page, you can change the functional levels if you think you'd ever need to introduce and older domain controller, its unlikely so you should only need to add the DSRM password, then click Next
- On the remaining steps, just click next (or review information provided if you like)
- On the Prerequisites Check page, click Install. The VM will reboot.
- Log in with the domain administrator password you set
- Open the Start menu and click on Administrative Tools >> DNS
- Expand your DC >> Forward Lookup Zones and click on the zone for your domain (e.g. vxprt.local)
- Verify that your server appears with an A record for the IP previously set.
- Right click on Reverse Lookup Zone and click on New Zone
- Click Next, accepting defaults until you get to the Reverse Lookup Zone Name page
- Type in the first 3 octets of the IP subnet you are using, so for example I would type in 192.168.6, this will help generate the appropriate zone name, click Next two more times and click Finish. You now have a reverse lookup zone so hosts can be resolved by name and IP address.
- Go back to your forward lookup zone for your domain and double click on the A record for your DC
- Check the Update associate pointer PTR record and click ok; this will create a record in the reverse zone you just created
- The last step is to set a DNS forwarder since this server will server as the primary DNS server for all other servers.
- Right click on your server, just under DNS and select properties
- Click on the Forwarders tab and click the Edit... button
- Remove any local addresses from the list by highlighting it and selecting delete
- I will use the public Google DNS servers, but you could also use something like OpenDNS.
- Click where it says "Click here to add and IP address..." and enter 18.104.22.168 and 22.214.171.124 - those should resolve to google-public-dns-a and b; click OK and OK again, then close DNS Manager
- Open IE and verify you can get to the intenret, you should be all set!
So far we have setup our Windows template VM, created a Linked Clone and made it into a Domain Controller and NTP server, next we can get into setting up our virtual ESXi hosts.
With our Windows virtual machine built, patched, and cloned, its time to setup the Domain Controller for the home lab. We will use the Domain Controller for authentication throughout the home lab setup including the necessary service accounts for VMware vSphere, SSO and vCloud Automation Center/vRealize Automation. If you are here from part 2 you should be looking at a booted virtual machine clone here are the steps to finish off the Windows system wizard - if you already blew through this no worries you can skip the next section.
- Tick the I accept box and click the Accept button.
- Set your region, language and keyboard layout and click Next
- Set your administrator password and click Finish
- You should now be at the login screen.
Now its time to setup this Windows VM as our Domain Controller - it used to be quite easy - type
and follow the wizard, unfortunately Microsoft in all their wisdom decided to change the process after 12 years of it working flawlessly.
- Press CTRL-ALT-INS on your keyboard or click on the VM menu and select Send CTRL-ALT-DEL (CTRL-ALT-INS seems much easier to me)
- Log in with the password you just set
- First update the date/time in Windows so it is in the correct time zone. You can click on the clock in the lower right corner or bring up the Date and Time control panel item
- Once the date is set, click on the Internet Time tab, ensure it is set to automatically synchronize with time.windows.com and click OK
- In this setup, I will use the domain controller as an NTP server so I can point my ESXi virtual machines and other appliances here so time is synchronized properly (NTP is critical in any environment, even the lab). In order to use Windows as an NTP server there is a registry change we need to validate. Bring up the start menu and type regedit
- Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\Ntp - ensure that Enabled is set to 1 (this was already set to 1 for me)
- If you are not already there, open the Start menu, right click on Computer and select properties
- Under Computer name, domain and work group settings click the Change settings button the click the Change button
- Name your computer, I prefer short and simple so dc01 is pretty common for me but I'm going with vxprt-dc01 here. Leave the Workgroup selected and click the OK button
- When the Computer Name/Domain change popup opens click OK, click Close on the System Properties window and then click the Restart Now button
- Once the virtual machine restarts, log back in as administrator
- Now, open Server Manager (it may already be open - I didn't say simon says close the Server Manager window :)
- Click on Add roles and features
Now that you have your first Windows VM built and patched, you’re probably itching to get things built like turning the virtual machine it into your domain controller for the home lab which will be used for authentication throughout this setup. However, we want to be efficient with our time so we are going to take our Windows VM and use it to clone new VMs since at the very least I will need 3 Windows virtual machines for this lab; a Domain Controller, SQL server and web server for the vCloud Automation Center/vRealize Automation Infrastructure-as-a-Service server.
In what should be a multi-part series (unless work gets insane) I will be setting up the supporting infrastructure for my home lab. For this lab I will be using the 8-core home lab build I wrote about in the past. I am currently running Windows 8.1 with VMware Workstation 10. I have two volumes setup in Windows that will be dedicated for VMs - 1 is a single 120GB Neutron SSD that I will use for some of the “heavier” VMs such as SQL server and the vRealize IaaS server. The other is a ~1.3TB RAID0 dynamic volume built in Windows on 3x 500GB Seagate hybrid drives which will be used for common VMs such as the domain controller I am setting up here.
A great feature of the vSphere Web Client comes when you have two separate vCenter servers that share a common Single Sign-on (SSO) server. In this scenario you can see all vCenter servers connected to the shared SSO server - logging into the vSphere Web Client for either vCenter (or having a single web server running the Web Client bits) you can see any/all vCenter servers using the shared SSO server.
If you are running EMC ViPR SRM, and your license key expires you will no longer be able to log into the UI where you could have installed a new license key. Instead you will need to update the license(s) via the command line. The directions I had found had
a mistake were unclear, so thought I’d publish the steps that worked for me here.
In the Windows vSphere Client, when a wizard comes up, you need to finish or cancel that current task - not so in the Web Client! When start a wizard in the vSphere Web Client you can click outside of the wizard window back into the vSphere Web Client and your wizard window will disappear. Let’s take a look at an example, below is a screenshot of my lab, highlighted in red is the Work in Progress sidebar:
One option not available in the vCloud Automation Center (vCAC) appliance VAMI is the ability to send logs to a syslog server, such as Splunk or LogInsight (does anyone know if this has been exposed in vRA 6.1?), thankfully since the appliance is built on linux, its just a matter of configuring rsyslog. If you are using LogInsight you can use the LogInsight Content Pack. While I have LogInsight, I want to do this manually to send logs as if I were using a generic syslog server. As you can see here, none of my vCAC logs are here (my appliance is named vcacapp)
I am in a course this week and a question came up about how to configure the Group and User search base DN and its effect on access within vCAC / vRA. Ultimately permission will be granted as a combination of both fields. First and foremost when configuring vCloud Automation Center for vRealize Automation tenants this will control which users or groups you can assign tenant administrator or infrastructure administrator roles. Let’s look at some examples; if my domain is test.lab and I set my User and Group search base DN to dc=test,dc=lab I will be able to assign either of those roles to any user or group in the entire Active Directory, regardless of what organization unit or container they may be in. Easy enough, but that starts to open things up pretty wide.
Updated 4/20/2016 The under $800 home lab is now under $700! Because I am a geek, and my needs change, and this is what I like to do I often check out new hardware, cost features etc. One of the things I wish I did on my 8-core lab was go for a smaller form factor case. Pricing, as always is subject to change. I actually prefer to buy most of my hardware on Amazon because I seem to have an easier time returning items that are defective but I’ll link to NewEgg. You may be able to find a part or two for a bit less somewhere else which is always good.
Once the problem was corrected, and host once again had access to the storage and datastores VM’s were brought back online, however one VM would not start giving the error:
Failed to start the virtual machine (error-18)
The following errors also appeared in the logs
Vpxa: [3E879B70 info 'DiskLib' opID=29f52016-bc] DISKLIB-LINK : "/vmfs/volumes/531a13ef-0196038d-bcb7-44d3cabcb764/ehc-pod2-sql01/ehc-pod2-sql01-flat.vmdk" : failed to open (Failed to lock the file).
There are a couple KB’s that describe a potential problem, such as a missing descriptor file (http://kb.vmware.com/kb/1002511) which lead me to browse the datastore via SSH and see there was a .lck file. KB 1008728 got me a bit closer in helping me find which server thought it had a lock on the files by running, although it didn’t provide a direct solution:
vmkfstools -D pathtofile
Just a heads up to anyone looking to deploy vCAC using vSphere SSO instead of the vCAC SSO appliance, in 5.5.0 U1, U1a, and U1b you will have problems adding tenants or new users. The fix is to replace a JAR file which can be found at the link below along with a more detailed description of the error and solution. I ran up against this recently, and if it weren’t for another error I’m troubleshooting to bring my vCAC SQL server online I’d verify if it works.
Containers have been a hot topic of late, many are suggesting it is the next step in the virtualization evolution and spells doom for vSphere, Hyper-V or other “traditional” virtualization platforms. For those who are not aware of containers, its is a packaged application that can run isolated from other applications. It is not too dissimilar from ThinApps in a virtual desktop environment but focused on server applications such as Apache, Tomcat or other custom applications on top of a single Operating System.
Why, well because you don’t always have VMware Workstation or Fusion available, and VMware Player only let’s you do so much. Sometimes you just need to make due with VirtualBox. Be warned, however, ESXi has been quite unstable for me in VirtualBox, especially if trying to add new hardware. The build below was fairly stable until I tried to add a new virtual hard drive, now it PSODs on boot. Plan ahead and add all necessary hardware prior to ESXi installation.
I’ve seen this come up in more and more situations lately, especially as people move towards SD/USB media for ESXi installation so I thought I’d write up quick how-to for changing it. The error you receive is System logs on host are stored on non-persistent storage. This happens when there is no VMFS partition available during installation which causes log files to be written to RAM disk. When you log into vCenter, you would see a warning on your host, and the above message on the summary tab.
During a recent vCenter deployment using the VCSA I ran into an error I hadn’t run into before with the VCSA (or vCenter/SSO on Windows for that matter). After an error free install and setup wizard, I logged in to vCenter as firstname.lastname@example.org to set my roles and assign my AD groups permission. However I noticed that there was no identity source for my Active Directory domain, no problem add it in, boom now hop on over to my vCenter permissions tab and get people vCentering. This is where I ran into errors. When trying to search for a user I received a pop up that said
Cannot load users for the selected domain
Before I ran the setup wizard, I had SSH’d to the VCSA did some pings and digs to make sure the network bits were flowing properly and everything seemed fine. I could ping and dig both local and remote AD resources so I was confident that was all working fine. Easy fix I assumed, so I headed over to the global KB search tool known as Google and was lead to this KB, http://kb.vmware.com/kb/2033742 which suggested checking DNS, time synchronization and joining and re-joining to the domain. I manually re-checked DNS records were present, that the AD join process had worked correctly and the account was still enabled.
This post will walk through mounting an ESXi ISO via the UCS KVM and installing ESXi. It assumes that UCS Manager is configured properly with the necessary profiles, policies and pools, that all VLANs and uplinks have been properly configured and that storage switches and arrays have been properly zoned. You should also verify the hardware and firmware version to download the correct drivers and build a custom ESXi ISO using PowerCLI if one does not already exist.
- Launch the Cisco UCS Manager; if it was not already installed/launched navigate to the IP/URL of your UCS manager (ex https://10.10.10.10) and click the Launch UCS Manager button, a jnlp file will be downloaded, launch that file which will load the UCS Manager and login window.
- Once logged in, you will see your equipment tab. Expand Chassis, Chassis 1 and then Servers.
- Click on the server you want to install ESXi on and click on the KVM Console action item. A separate java application will launch. If prompted about an unencryted KVM session, select Accept for this session, and remember, then Apply.
- Click on the Virtual Media tab (accept again if prompted), click on Add image... and browse to your ISO directory, select the ISO and click the Mapped checkbox
- Power on the server, or if already booted click Reset (Click OK, Power Cycle, OK, OK)
- The UCS blades take several minutes to do checks, do not worry at this point.
- When the Cisco splash screen appears, press F6
- When the boot menu appears, select EFI: Cisco vKVM-Mapped vDVD and hit enter. The KVM needs to read the ISO, depending on the connection between your ISO directory and KVM, this will take a few moments.
- You will now see the ESXi installer start to load. From here, proceed with a normal ESXi installation.
I’ve been using the vCenter Server Appliance, AKA VCSA, in testing for several months now and ready to start using it for every day use. These are the installation steps to get a fresh vCenter Server Appliance installed using the embedded PostgreSQL database and Active Directory for SSO.
These examples are not meant to reflect best practices, or even efficiency, rather they are barebones PowerCLI examples of how multiple commands can be used within a script to perform common tasks. Only where necessary where items such as variables or inputs used.
The title is actually a bit misleading, its not actually the linux OS registering with DNS, but rather DHCP registering on behalf of the OS. In order to make this work, there just a couple config files on the linux side you need to edit and a DHCP setting on the Windows server. One common misconception I ran into often in trying to figure this out were people suggesting to set your DNS server to allow Secure and Non-Secure updates which I had done. It wasn’t until I realized that DHCP was actually handling the registration on behalf of the OS through the use of an authenticated domain account that I realized change the DNS setting wasn’t necessary, after all its a secure user account.
On most firewalls I have ever inherited, access control lists (ACL) are typically applied to inbound traffic on an interface, you can however have an ACL applied to outbound traffic. Some firewalls obscure this a bit and have you select the interface name where traffic is coming from and where its going to, on a Cisco ASA however you would apply the ACL either “in” or “out” of an interface. This can be a bit confusing, especially if you aren’t working with firewalls every day. Essentially you just need to visualize where the traffic is coming FROM and where it is going TO in relationship to the interface you are applying the ACL to. Lets take a look at an example; this is a fairly generic example you might see on most any ASA, we are applying an ACL to the “outside” interface for all “in” bound traffic to the interface, that is it is limiting traffic generated coming from one direction to another which will later be defined in the ACL itself:
access-group acl_outside_in in interface outside
Now the “outside” interface here is nothing more than the name given to a physical port on the ASA so it really could be anything. In your ASA config it would look something like this:
interface GigabitEthernet0/0 speed 1000 duplex full nameif outside security-level 0 ip address x.x.x.x y.y.y.y
If we changed nameif to homersimpson then the ACL would very simply be:
access-group acl_outside_in in interface homersimpson
The ACL name, “acl_outside_in” is also just a name that you give to the ACL, so again we could just as easily called the ACL “familyguy” which would turn the ACL into:
access-group familyguy in interface homersimpson
Typically I find that your ACLs and Interface names are descriptive of their purposes, unless you subscribe to the “security though obscurity” practice. Other than ensuring you are using the correct ACL and Interface name (because then it would not work), the only item left in the ACL is really the “in” or “out.”
I have seen a lot of back and forth via social media of late between people working for different vendors, each upset with one another over some apparent FUD smearing of each other. Let me just say if you are a vendor, and you are speaking negatively about your competitors - you are doing it wrong and you will quickly lose credibility. So if, for example, you offer a product that uses, oh I don’t know a VSAN like storage architecture should you really be saying anything negative about VSAN? I think VMware coming out with VSAN shows the mindset shift towards less complex infrastructures. If you as a vendor provide this VSAN like feature, this will ultimately only help your product if you market it correctly and don’t start in on bad mouthing competition.
Update: There is a newer version of this post at http://www.virtxpert.com/installing-vcenter-server-appliance-5-5-0b/ - the steps have changed a bit
I ran into a strange problem recently, a Cisco WLAN controller 5508 with 1142N APs (not sure the model and controller matter entirely as I found the fix on a support forum thread for a 4000 series) would allow clients to connect, get an IP address but NOT pass any traffic other than ICMP. I thought maybe the problem was Windows firewall related but disabled it still appeared. I thought maybe a driver problem but tried several revs of the driver, and it also happened with different model cards. A temporary work around was to disable, then re-enable the wireless card.
The opinions expressed here are my personal opinions and experiences. Content published here is not read or approved in advance by my current or any former employer and does not necessarily reflect the views and opinions of my current or any former employer.
I ran into this problem while setting up my lab. VMware has KB (link below) on this article but wasn’t much help for me, though I always start with the KB. The fix for me (I had two servers doing this) was to either restart the management agent via the DCUI and then add, or add the host via IP address then remove it from vCenter and re-add. These may not work for you, but since the KB article didn’t work for me, maybe this will help.
This is probably the best advice I have ever received, and unfortunately to many people don’t live by these rules. People worry about advancing their careers, but at what cost? I for one refuse to put anything in front of my family. Family, Friends, Health, Education, Work - in that order, every single time and if you ask me to change that order you will be sorely disappointed in the answer.
Guest Post by Kanji B.
Recently, I implemented a Security Information and Event Management (SIEM) tool called Alienvault / OSSIM to monitor servers/event logs to ensure compliance with several customer security agreements. I chose Alienvault because it combined several open source tools, providing a single pane of glass view into what would otherwise be several different tools (oh and I had no budget to do this with). During the implementation I hit a snag when configuring Alienvalt to monitor Windows Server event logs. After combing through the forums I found a combination of problems that needed to be fixed – hopefully this will help out others (while giving credit to all the posts we used to find the solution to our problem).
So you’re a small to medium size business, and after carefully consulting with your IT team (or me!) its been decided that you need a server – maybe its your first, maybe a replacement but in any case you only have this 1 single server. Before you start installing or preparing Windows on this server here are a few reasons why you should consider installing VMware ESXi first.