My New Journey – Into the Cloud

Happy New Year everyone! Over the last few years my work concentration has been, and still will be to a point, working to engineer solutions for my companies virtual infrastructure, and over the last couple years specializing in the vROPs suite of applications and now with the new year here, I have been assigned to work on what my companies direction and strategy will be for our cloud initiative. This doesn’t mean I am moving away from VMware, quite the opposite, I will be digging deeper into what VMware offers as a cloud platform and provider. I will also to some extent still have involvement in ESXi, vSphere, NSX and vROPs. My larger team still covers all of those products; however my focus will shift to private/public cloud. How cool is that right!? This is the industry trend and there is so much opportunity to be involved in engineering some awesome solutions and further grow the cloud footprint (cloudprint??) at work. Its not all going to be me, ill be part of a team, all of us working to provide real value. I’m very excited and feel privileged to be part of this movement.

So what does this all mean for this blog. Not much, except more cool and informative info as I work my way through this. Ill still focus on providing articles and write ups of solutions and information on how VMware deeply integrates with our cloud solutions, and the integration of VMware products. I betting you will still see some vROPs and capacity management stuff too as I transition. Over the last year the articles I have written here have been based around actual challenges or issue or decisions I have had to work through. Expect the same theme, just a focus change to cloud. I love sharing info with my VMUG/VMware community and I hope you will find the info I post will continue to interest you and maybe even help. ūüôā

See ya in the clouds!
Dan @anothergeek


Considerations for Capacity Management with vROps

Navigating your way around capacity management is not and easy task, especially at a large company where it seems almost impossible to get your arms wrapped around it. HA – I picture a large tree and trying to hug it, not quite able to lock your fingers on the other side! It’s really kind of like that. You got most of it, but you are always reaching. At times you need to step back and re-evaluate your angle or approach. Over the last year or so I’ve been working with the capacity management team to choose exactly the right metrics to determine the best way to evaluate capacity. Last week one cluster, according to vROPs, was in desperate need of capacity, we were running into our buffers; however when we looked closely in our review meeting we noticed that the reason we were out of capacity was due to CPU Demand. This spun off a number of weekly meetings to re consider our approach or angle to see if we can get our fingers locked. In all honesty, this wasn’t an oversight, we have a pretty smart group of people and we meet regularly to review. Everyone on our team has the same goal and these types of discussions make sure we are staying on target; however we did realize that we needed a deeper understanding of the different types of capacity models and how to apply them as policies across the virtual infrastructure. So let’s start with a quick level set and go from there. All right, here we go!

Allocation Model
This model is capacity based on the configured amount of resources assigned to a VM or VMs in a cluster. The consensus is that this model should be used for production environments where you have important workloads, and you want to be able to keep resources for fail-over, and you want to make sure you don’t over commit by too much. You decide your over commitment ratio and set that in the policy. This is the most conservative capacity model.

Demand Model
The Demand model is often used in Test/Development environments where you don’t necessarily care about over allocation, and you really want to get as many guests as possible in the environment. If you are using this model you probably don’t care if the hosts are running hot. You will likely be way over allocated but again you don’t care because you want to run this for highest possible VM density.

Memory Consumed model
This model allows you to see the memory resources used just like you would in the vSphere client. It shows the active memory, plus shared memory pages, plus recently touched memory. All the memory overhead.

So which one do we choose? That’s an excellent question. In all likely hood, we are going to look at all these models and how they affect capacity. We have, and I’m guessing you do too, clusters with mixed workloads or due to licensing considerations clusters where you have to mix test/dev hosts with production hosts. So its not so easy to just pick one or the other and go with it, especially when you have to scale up the environment to meet the needs of the company. Our team decided to start to implement different policies specific to the cluster and workloads in those clusters. The polices will include different allocation over-commit ratios for CPU/Memory and Disk. Some policies will account for all three models others will just be one or a combination. What’s really great is vRealize Operations is so flexible its really easy to dial in capacity just the way you want it. One other decision we made that you might want to consider is that we will only rely on the data in vROPs for capacity management. We wont look at what vCenter is showing for cluster resources used to determine if we can “fit” more VMs in. Capacity management is not easy, it takes time to collect metric data, analyze it and then tweak it so you are sure you can make the best decisions. Sometimes those decisions can save (or cost) your company a significant amount of money. The good news is there is no magic going on there. If you put in the work and use a great tool like vRealize Operation Manager you will get to a point where real value will be realized with vROPs. Now that our team has determined to use a combination of models, we can then begin to adjust policies and review data that’s already been collected to make sure we are using metrics that meet our needs. I’d love to hear how others are using vROPs to determine capacity and some of the challenges and success you have encountered. If you read this and want to share, add a comment.

I’d like to thank Hicham Mourad for his help with some questions and his guidance along the way. He is a really smart guy, and Im thankful I can reach out to him when I need to. ūüôā

VMware Announces General Availability – vSphere 6.5

Today VMware announced general availability for vSphere 6.5. Im really excited about this release, not only for vSphere 6.5, but I am also really looking forward to test out VSAN 6.5 and VROPs 6.4 along with Log Insight 4. All these products will go into my home lab first for me to play around with and try out new features, then Ill start to upgrade our Engineering Test lab at work, putting everthing through its paces, then onto test/dev/cert and into Prod. ¬†I’m really interested in the new fully supported HTML-5 Web Client, and the predictive DRS features. I also want to check out the new appliance managemnt and update manager.

Happy testing! ūüôā

Here is the link to the Official GA Announcement

Manually Increasing vSphere Web Client Heap Size

The other day when I was building a vSphere 6.0 environment up in my lab for testing I ran into an issue where performance was extremely slow in the web client and I was continually receiving an error that the VMware-dataservice-sca and vsphere-client status would change from green to yellow.¬† When I deployed the VCSA/PSC appliance I choose “Tiny” as the size option.¬† Even though my implementation is going to be under the 10 hosts and 100 VMs, I think this build¬†was just not enough, and performance in the web client was just really lacking.¬† Searching the VMware KB I came across 2144950.¬† I found out this is a known issue affecting¬†vCenter Server 6.0.¬†¬†Here are the steps that I used to work around the error and gain performance back in the web client.

First I added additional RAM to the appliance.  Pretty straight forward, no magic there.  Then I used SSH to connect to the appliance and ran the follow command:

cloudvm-ram-size -C XXX vsphere-client

Replace the XXX with the size in MB that you want to increase the heap size.

If you are running a Windows  vCenter Server, find C:\ProgramFiles\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat and run this command:

cloudvm-ram-size.bat -C XXX vspherewebclientsvc

Again swap out the XXX with the size in MB that you want to increase the heap size. Don’t forget to restart the vSphere client service.

Removing a PSC or vCenter Server in vSphere 6.x

The other day I’m bring up another vSphere 6.0 environment for our VDI team in our engineering test lab and for some reason im having all sorts of issues. ¬†I’m installing a VCSA with embedded PSC and connecting it to and existing SSO domain. ¬†I have no idea whats going on, its going horrible. One time the install will fail, then the next it will complete, but enhanced linked mode is just acting weird….Well unbeknownst to me the QIP team decided to cut over DNS to new appliances and that was reeking havoc across the environment. ¬†So now that I’ve killed (I kid) the guy who was doing this I’m left with a mess to clean up. ¬†Finally DNS is working properly so I’m going to re-deploy the PSC/VCSA again but before I do that, I have to clean up the one that I don’t want anymore. Lucky for us its a pretty easy job.

The first step I had to do was make sure that my appliance was powered down.  I knew that there was no other VCSA pointing to this PSC.  If you are unsure if any other vCenter is connected to the PSC you are removing, you can check by logging into the vSphere web client and go to the advanced vCenter server settings and look for a property called config.vpxd.sso.admin.url and the value of this setting is the PSC the vCenter server is using.  If you find any other vCenters VMware has KB 2113917 to help you re point your vCenter to a different PSC.

Once that is all sorted out, next we need to connect to another PSC is the same SSO domain via SSH and run the following command:

cmsso-util unregister --node-pnid Platform_Services_Controller_FQDN 
--username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

After that completes, delete the appliance from your inventory and check in Administration -> System Configration -> Nodes to make sure that its not listed there.

Removing a VCSA is just about the same as above just have to make one change in the command:

cmsso-util unregister --node-pnid vCenterServer_System_Name --username 
administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

If you need some additional info on these steps, check out KB 2106736

Project Home Lab – Part 1 – Hardware

A home lab can be a great resource for any App Dev, ¬†Sys Admin or Engineer. ¬†Its a great tool to learn about the products you are responsible for. I believe the value it will return to the company that you work for is ten fold. Think about it, you are learning at home on your own time, then bringing that knowledge back to your job to apply it towards development projects or support. Its really a win-win. My engineering team and I proposed to our department management a project to provide home labs to our engineering and app development teams. We thought it would be a great way to bridge the communication gap between the two teams and help reduce or eliminate shadow IT. One of the challenges we have come across working in a large company is knowing exactly what our development teams need to perform their job. ¬†Is it containers, OpenStack, or just some other product that allows them to move their projects and initiatives forward? ¬†The answer is probably yes to all or any of those questions and its more than likely already running under their desk. ¬†Our thought was to give the various teams a supportable (internal support) platform to work creatively and learn. Also a direct line of communication from App Dev to Engineering without going through the traditional channels. ¬†We are hoping this will provide a quicker turn around time to engineer the infrastructure to meet the needs of the developers and give them the tools they need. ¬†At least that’s the theory behind this pilot project. ¬†Over the next few posts ill share some of the cool things Im doing with my home lab and ill also let you know any feedback I receive from the teams using it and management. ¬†So lets get to it!

Part 1 – Hardware

When my team first set out to select the right hardware we looked around the internet. ¬†There are many choices and flavors of a home lab to choose from. ¬†Gone are the days where you need some big honking old decommissioned servers that suck power and cause your wife to complain about the sound and cost of electricity. ¬†Today’s home lab is small, quiet, powerful and efficient, and can provide a number of configuration choices for testing all sorts of builds and designs. ¬†Our requirements were pretty simple and standard. ¬†We wanted vSphere (no duh!) and some VSAN (yeah baby!) and a whole bunch of extra storage. ¬†Here is the run down of what we decided to get.

  • 3 Intel NUC kits (NUC6i5SYH) – that’s a Core i5 6260U 1.8 Ghz processor
  • 3 Crucial DDR4 32GB (2×16) DIMM kits – Each NUC will get 32GB of Memory
  • 3 Samsung 850 EVO (MZ-75E2T0B) 2TB¬†2.5″¬†SSD SATA 6Gb/s – one for each NUC
  • 3 Samsung 850 EVO M.2 (MZ-N5E120BW) SSD SATA 6Gb/s – one for each NUC (VSAN Cache)
  • 3 StarTech USB 3.0 to Gigabit Ethernet NIC adapter – VSAN traffic will go over this NIC
  • 3 Kingston Data Traveler G4 – USB Flash Drive 8GB – We’ll install ESXi and Boot from them
  • 1 Synology DiskStation 5 Bay DS1515+ NAS Server
  • 5 WD Red Pro NAS Hard Drives (WD8001FFWX) 8TB SATA 6Gb/s
  • 1 Linksys (SE3016) 16 port unmanaged Switch

Full disclaimer here…. I did not purchase the hardware with my own money, my company purchased the hardware for a pilot home lab project I previously mentioned. So yeah I know what you are thinking, what a deal. ¬†I agree, but I really believe my company will get value back for the purchase and with some conditions, they seem to believe that also.¬†You should expect to also purchase licensing. The licenses I’m using are my own. ¬†I have a VMUG Advantage, MSDN and I also get some free VMware licenses for being a vExpert. ¬†Really look at VMUG Advantage, its the best option and its very affordable. ¬†It goes without saying don’t use your production licenses. Info on VMUG Advantage can be found HERE. If your budget is tight, no worries, you can easily scale down (or up) to meet your needs and with all the options out there you should be able to build a really decent home lab.

All the hardware went together really nicely. ¬†I have to be honest, putting together all that stuff really gets my geek flag flying. Its almost a religious experience. ¬†Takes me back to when I was a kid home building PCs; but I digress. :). ¬†You shouldn’t really have any issues connecting all the pieces. ¬†One of the guys on my team did have one NUC only see 16GB of memory, he just needed to reset one DIMM and that was fixed.

In Part 2, Ill go over some design considerations and build out.

My VMworld 2016 Recap

One week a year geek enthusiasts get together to celebrate a passion that we all share. Administrators, Engineers, Bloggers, Executives and Employees converge on one city ready to learn about the latest in virtualization technology from VMware and its partners. All 23,000 of us. This year VMworld 2016 was held at Mandalay Bay in Las Vegas, NV.  Instead of writing about the news of the conference, Im going to recap my VMworld 2016. Hopefully you will get a different perspective than the normal Day 1, Day 2 summary.

This was my second VMworld.  The first one I attended was way back in 2007, and it was held in San Francisco, CA.  What a difference I experienced from almost ten years ago.  In 2007 I attended the conference by myself and I was with a smaller company.  This time up, the company I work for is a TAM (Technical Account Manager) customer, and is also a much larger company and that meant more vendor parties!  Yeah!  There was six of us going this year, two VMware Engineers, one VDI Engineer, one Network Engineer and two Ops folks, one of which is a VP. Most of us flew out on Saturday so we could check in at registration a day early and be able to attend the TAM Customer Day on Sunday.

TAM Customer Day

I had never been to a TAM Customer Day before, and I was really impressed. The day was filled with a morning Key note and a great discussion with Pat Gelsinger (CEO VMware) and Michael Dell (CEO of Dell Technologies).  The afternoon session was a really great CTO panel discussion with Paul Strong, Chris Wolf, Kit Colbert, Shawn Bass and Christos Karamanolis. We had a great lunch and then were able to meet and talk to a number of VMware experts on every technical topic you can imagine.

We also ate good!

YUM!! :)

YUM!! ūüôā

Solutions Exchange

After TAM Customer Day we were able to check out the opening of the Solutions Exchange and the reception party. ¬†The solutions exchange didn’t disappoint. ¬†It was packed to the brim with many vendors and they were all looking to exchange your badge scan for some swag! ¬†I scored a number of t-shirts and all sorts of items, like pens, headphones, battery chargers and a couple Google Cardboard VR. ¬†There were a few vendors giving out some nice stuff for being a VMware vExpert. ¬†Cohesity gave us a great Timbuk2 backpack filled with a Contigo water bottle, a battery charger, a USB drive and some connectors. ¬†Datrium totally kicked it up a notch with a Raspberry Pi3 and awesome case. I also received a geek version of Cards Against Humanity from Solidfire¬†and a medieval looking chalice from Catalogic and a nice baseball hat from Docker.


Sweet! Thanks Datrium!

We hit the Solutions Exchange a few times during the week to grab some extra stuff to bring back to our team.  Hey we like to share!

VMUG Member Party

Thee words. ¬†OFF THE HOOK! ¬†This was the fist year VMUG hosted a party for it’s members and it was awesome. ¬†It was held at House Of Blues at Mandalay Bay. ¬†The theme was an 80’s party. There was a live band that was totally kick’in. ¬†It was great to catch up with some fellow VMUG members and enjoy some drinks (a lot of drinks) and food. It was an excellent¬†start to the week. I had a really nice conversation with Paul Strong CTO for VMware, he even agreed to do a key note for a future UPState NY VMUG UserCon!¬†ūüôā

Session INF8097

On Tuesday I had my first presentation of the week with Tom Cronin (@Virtual_Tom) and our TAM Joe DePasquale (@DePasqualeJoe).  We spoke about meeting infrastructure high availability by deploying an out of band management cluster.  The session went very well.  I was a little nervous at the start, but once I got going, felt good.   I was happy I went through my slides with no issues.  Tom and Joe did a great job, we all did.

L to R - Joe, Me, Tom

L to R – Joe, Me, Tom

Tuesday night we hit up another vendor party at the House of Blues, which was not nearly as good as the VMUG party and then headed to Ri Ra Irish Pub for vBacon Vegas Style.  Hey it was beer and bacon, hell yeah it was fun!

Beer and Bacon Baby!!

Beer and Bacon Baby!!

Session MGT7924

Wednesday was a pretty big day.¬† My second presentation was with Hicham Mourad from VMware.¬† We were presenting on vRealize Operations Capacity Management.¬† The session had over 400 pre-registered attendees and ended up with over 300 showing up.¬† This time I wasn’t nervous at all.¬† I felt so comfortable on stage even though it was a much bigger room, stage and more people.¬† I did the welcome and opening introductions then handed it off to Hicham to go into a technical review of capacity terms, badges and a demo, then back to me to share some specific slides on how we do capacity management at the bank.¬† All during Hicham’s sections, I would interject with customer insight, and it worked really well.¬† I had about 15 minutes of material to present at the end, and Hicham gave me about¬†8 minutes! lol. So I had to condense my info.¬† Hicham and I killed it. We really did. We had a Top 10 presentation for Wednesday. Woo-Hoo!¬† After the presentation because my time was short, I had a number of people come up to me with questions, and handed out my card for follow up.¬† They were very interested in my experience with vROps¬†at work. ¬†It was awesome, what a rush. It was like something clicked, I really enjoyed speaking in front of so many people and¬†presenting to them my slides and information.¬† It was¬†such a thrill, I would do that full time!¬† You can check out my session here. Top 10 Sessions at VMworld 2016 for Wednesday¬† Look for vRealize Operations Capacity Explained.

That's me, killing it on stage with Hicham.

That’s me, killing it on stage with Hicham.

Wednesday was also the VMUG Leader Lunch.¬† They served us a nice meal, unfortunately I had to leave early for my presentation and didn’t get to stay for the speakers. Hopefully next year I wont have a conflict.¬† One highlight is that I had a chance to meet VIRT. (@vmug_virt).

Me and @vmug_virt

Me and @vmug_virt

VMworld 2016 Party

Wednesday night was the VMworld party at the Las Vegas Motor Speedway.¬† It was a super hot night, like 103 at the time of the party.¬† That was the first time I experienced elbow sweat!¬† The party was a lot of fun, food was good, lots of entertainment and I had a ride around the track in a car.¬† Capital Cities opened for Fall out Boy.¬† Both bands were great, but Capital Cities really was awesome.¬† I always feel like I don’t have enough time to experience everything the party has to offer, but its always fun and its great that VMworld provides rides to and from the event.

Last day of VMworld 2016

The conference wrapped up on Thursday September 1st.¬† I had a couple of sessions scheduled in the morning and MGT7770 was my favorite session of the week. ¬†It was VSAN Management using vROps plus the Management Pack for Storage Devices (MPSD) with 100% live demos on a multi-cloud private\public VMware cloud foundation infrastructure.¬† The speakers were Jeff Godfrey and Rawlinson Rivera.¬† I really enjoyed this session.¬† After that session I¬†went back to the VMvillage one last time before I headed out.¬† The end of VMworld is bitter/sweet.¬† The environment¬†and experiences are so much fun.¬† Seeing friends and meeting new people and¬†attending great sessions¬†are¬†some of the¬†best parts¬†of the week and this year speaking in two presentations was by far the highlight for me; however I’m really ready to get back home to my family and return to normal life.¬† Vegas was great, and a huge shout out to the blue shirts and event staff for really making the week great.¬† They were so¬†helpful and welcoming, always smiling; who dosent miss the box lunches, am I right?!¬† All in all it was a really educational and fun week.¬† I’m definitely looking forward to next year back in Las Vegas! Who knows you might even see me back up on stage! ūüėČ

Hear are some additional photos from my trip.