VMware Announces General Availability – vSphere 6.5

Today VMware announced general availability for vSphere 6.5. Im really excited about this release, not only for vSphere 6.5, but I am also really looking forward to test out VSAN 6.5 and VROPs 6.4 along with Log Insight 4. All these products will go into my home lab first for me to play around with and try out new features, then Ill start to upgrade our Engineering Test lab at work, putting everthing through its paces, then onto test/dev/cert and into Prod. ¬†I’m really interested in the new fully supported HTML-5 Web Client, and the predictive DRS features. I also want to check out the new appliance managemnt and update manager.

Happy testing! ūüôā

Here is the link to the Official GA Announcement


Manually Increasing vSphere Web Client Heap Size

The other day when I was building a vSphere 6.0 environment up in my lab for testing I ran into an issue where performance was extremely slow in the web client and I was continually receiving an error that the VMware-dataservice-sca and vsphere-client status would change from green to yellow.¬† When I deployed the VCSA/PSC appliance I choose “Tiny” as the size option.¬† Even though my implementation is going to be under the 10 hosts and 100 VMs, I think this build¬†was just not enough, and performance in the web client was just really lacking.¬† Searching the VMware KB I came across 2144950.¬† I found out this is a known issue affecting¬†vCenter Server 6.0.¬†¬†Here are the steps that I used to work around the error and gain performance back in the web client.

First I added additional RAM to the appliance.  Pretty straight forward, no magic there.  Then I used SSH to connect to the appliance and ran the follow command:

cloudvm-ram-size -C XXX vsphere-client

Replace the XXX with the size in MB that you want to increase the heap size.

If you are running a Windows  vCenter Server, find C:\ProgramFiles\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat and run this command:

cloudvm-ram-size.bat -C XXX vspherewebclientsvc

Again swap out the XXX with the size in MB that you want to increase the heap size. Don’t forget to restart the vSphere client service.

Removing a PSC or vCenter Server in vSphere 6.x

The other day I’m bring up another vSphere 6.0 environment for our VDI team in our engineering test lab and for some reason im having all sorts of issues. ¬†I’m installing a VCSA with embedded PSC and connecting it to and existing SSO domain. ¬†I have no idea whats going on, its going horrible. One time the install will fail, then the next it will complete, but enhanced linked mode is just acting weird….Well unbeknownst to me the QIP team decided to cut over DNS to new appliances and that was reeking havoc across the environment. ¬†So now that I’ve killed (I kid) the guy who was doing this I’m left with a mess to clean up. ¬†Finally DNS is working properly so I’m going to re-deploy the PSC/VCSA again but before I do that, I have to clean up the one that I don’t want anymore. Lucky for us its a pretty easy job.

The first step I had to do was make sure that my appliance was powered down.  I knew that there was no other VCSA pointing to this PSC.  If you are unsure if any other vCenter is connected to the PSC you are removing, you can check by logging into the vSphere web client and go to the advanced vCenter server settings and look for a property called config.vpxd.sso.admin.url and the value of this setting is the PSC the vCenter server is using.  If you find any other vCenters VMware has KB 2113917 to help you re point your vCenter to a different PSC.

Once that is all sorted out, next we need to connect to another PSC is the same SSO domain via SSH and run the following command:

cmsso-util unregister --node-pnid Platform_Services_Controller_FQDN 
--username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

After that completes, delete the appliance from your inventory and check in Administration -> System Configration -> Nodes to make sure that its not listed there.

Removing a VCSA is just about the same as above just have to make one change in the command:

cmsso-util unregister --node-pnid vCenterServer_System_Name --username 
administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

If you need some additional info on these steps, check out KB 2106736

Project Home Lab – Part 1 – Hardware

A home lab can be a great resource for any App Dev, ¬†Sys Admin or Engineer. ¬†Its a great tool to learn about the products you are responsible for. I believe the value it will return to the company that you work for is ten fold. Think about it, you are learning at home on your own time, then bringing that knowledge back to your job to apply it towards development projects or support. Its really a win-win. My engineering team and I proposed to our department management a project to provide home labs to our engineering and app development teams. We thought it would be a great way to bridge the communication gap between the two teams and help reduce or eliminate shadow IT. One of the challenges we have come across working in a large company is knowing exactly what our development teams need to perform their job. ¬†Is it containers, OpenStack, or just some other product that allows them to move their projects and initiatives forward? ¬†The answer is probably yes to all or any of those questions and its more than likely already running under their desk. ¬†Our thought was to give the various teams a supportable (internal support) platform to work creatively and learn. Also a direct line of communication from App Dev to Engineering without going through the traditional channels. ¬†We are hoping this will provide a quicker turn around time to engineer the infrastructure to meet the needs of the developers and give them the tools they need. ¬†At least that’s the theory behind this pilot project. ¬†Over the next few posts ill share some of the cool things Im doing with my home lab and ill also let you know any feedback I receive from the teams using it and management. ¬†So lets get to it!

Part 1 – Hardware

When my team first set out to select the right hardware we looked around the internet. ¬†There are many choices and flavors of a home lab to choose from. ¬†Gone are the days where you need some big honking old decommissioned servers that suck power and cause your wife to complain about the sound and cost of electricity. ¬†Today’s home lab is small, quiet, powerful and efficient, and can provide a number of configuration choices for testing all sorts of builds and designs. ¬†Our requirements were pretty simple and standard. ¬†We wanted vSphere (no duh!) and some VSAN (yeah baby!) and a whole bunch of extra storage. ¬†Here is the run down of what we decided to get.

  • 3 Intel NUC kits (NUC6i5SYH) – that’s a Core i5 6260U 1.8 Ghz processor
  • 3 Crucial DDR4 32GB (2×16) DIMM kits – Each NUC will get 32GB of Memory
  • 3 Samsung 850 EVO (MZ-75E2T0B) 2TB¬†2.5″¬†SSD SATA 6Gb/s – one for each NUC
  • 3 Samsung 850 EVO M.2 (MZ-N5E120BW) SSD SATA 6Gb/s – one for each NUC (VSAN Cache)
  • 3 StarTech USB 3.0 to Gigabit Ethernet NIC adapter – VSAN traffic will go over this NIC
  • 3 Kingston Data Traveler G4 – USB Flash Drive 8GB – We’ll install ESXi and Boot from them
  • 1 Synology DiskStation 5 Bay DS1515+ NAS Server
  • 5 WD Red Pro NAS Hard Drives (WD8001FFWX) 8TB SATA 6Gb/s
  • 1 Linksys (SE3016) 16 port unmanaged Switch

Full disclaimer here…. I did not purchase the hardware with my own money, my company purchased the hardware for a pilot home lab project I previously mentioned. So yeah I know what you are thinking, what a deal. ¬†I agree, but I really believe my company will get value back for the purchase and with some conditions, they seem to believe that also.¬†You should expect to also purchase licensing. The licenses I’m using are my own. ¬†I have a VMUG Advantage, MSDN and I also get some free VMware licenses for being a vExpert. ¬†Really look at VMUG Advantage, its the best option and its very affordable. ¬†It goes without saying don’t use your production licenses. Info on VMUG Advantage can be found HERE. If your budget is tight, no worries, you can easily scale down (or up) to meet your needs and with all the options out there you should be able to build a really decent home lab.

All the hardware went together really nicely. ¬†I have to be honest, putting together all that stuff really gets my geek flag flying. Its almost a religious experience. ¬†Takes me back to when I was a kid home building PCs; but I digress. :). ¬†You shouldn’t really have any issues connecting all the pieces. ¬†One of the guys on my team did have one NUC only see 16GB of memory, he just needed to reset one DIMM and that was fixed.

In Part 2, Ill go over some design considerations and build out.

My VMworld 2016 Recap

One week a year geek enthusiasts get together to celebrate a passion that we all share. Administrators, Engineers, Bloggers, Executives and Employees converge on one city ready to learn about the latest in virtualization technology from VMware and its partners. All 23,000 of us. This year VMworld 2016 was held at Mandalay Bay in Las Vegas, NV.  Instead of writing about the news of the conference, Im going to recap my VMworld 2016. Hopefully you will get a different perspective than the normal Day 1, Day 2 summary.

This was my second VMworld.  The first one I attended was way back in 2007, and it was held in San Francisco, CA.  What a difference I experienced from almost ten years ago.  In 2007 I attended the conference by myself and I was with a smaller company.  This time up, the company I work for is a TAM (Technical Account Manager) customer, and is also a much larger company and that meant more vendor parties!  Yeah!  There was six of us going this year, two VMware Engineers, one VDI Engineer, one Network Engineer and two Ops folks, one of which is a VP. Most of us flew out on Saturday so we could check in at registration a day early and be able to attend the TAM Customer Day on Sunday.

TAM Customer Day

I had never been to a TAM Customer Day before, and I was really impressed. The day was filled with a morning Key note and a great discussion with Pat Gelsinger (CEO VMware) and Michael Dell (CEO of Dell Technologies).  The afternoon session was a really great CTO panel discussion with Paul Strong, Chris Wolf, Kit Colbert, Shawn Bass and Christos Karamanolis. We had a great lunch and then were able to meet and talk to a number of VMware experts on every technical topic you can imagine.

We also ate good!

YUM!! :)

YUM!! ūüôā

Solutions Exchange

After TAM Customer Day we were able to check out the opening of the Solutions Exchange and the reception party. ¬†The solutions exchange didn’t disappoint. ¬†It was packed to the brim with many vendors and they were all looking to exchange your badge scan for some swag! ¬†I scored a number of t-shirts and all sorts of items, like pens, headphones, battery chargers and a couple Google Cardboard VR. ¬†There were a few vendors giving out some nice stuff for being a VMware vExpert. ¬†Cohesity gave us a great Timbuk2 backpack filled with a Contigo water bottle, a battery charger, a USB drive and some connectors. ¬†Datrium totally kicked it up a notch with a Raspberry Pi3 and awesome case. I also received a geek version of Cards Against Humanity from Solidfire¬†and a medieval looking chalice from Catalogic and a nice baseball hat from Docker.


Sweet! Thanks Datrium!

We hit the Solutions Exchange a few times during the week to grab some extra stuff to bring back to our team.  Hey we like to share!

VMUG Member Party

Thee words. ¬†OFF THE HOOK! ¬†This was the fist year VMUG hosted a party for it’s members and it was awesome. ¬†It was held at House Of Blues at Mandalay Bay. ¬†The theme was an 80’s party. There was a live band that was totally kick’in. ¬†It was great to catch up with some fellow VMUG members and enjoy some drinks (a lot of drinks) and food. It was an excellent¬†start to the week. I had a really nice conversation with Paul Strong CTO for VMware, he even agreed to do a key note for a future UPState NY VMUG UserCon!¬†ūüôā

Session INF8097

On Tuesday I had my first presentation of the week with Tom Cronin (@Virtual_Tom) and our TAM Joe DePasquale (@DePasqualeJoe).  We spoke about meeting infrastructure high availability by deploying an out of band management cluster.  The session went very well.  I was a little nervous at the start, but once I got going, felt good.   I was happy I went through my slides with no issues.  Tom and Joe did a great job, we all did.

L to R - Joe, Me, Tom

L to R – Joe, Me, Tom

Tuesday night we hit up another vendor party at the House of Blues, which was not nearly as good as the VMUG party and then headed to Ri Ra Irish Pub for vBacon Vegas Style.  Hey it was beer and bacon, hell yeah it was fun!

Beer and Bacon Baby!!

Beer and Bacon Baby!!

Session MGT7924

Wednesday was a pretty big day.¬† My second presentation was with Hicham Mourad from VMware.¬† We were presenting on vRealize Operations Capacity Management.¬† The session had over 400 pre-registered attendees and ended up with over 300 showing up.¬† This time I wasn’t nervous at all.¬† I felt so comfortable on stage even though it was a much bigger room, stage and more people.¬† I did the welcome and opening introductions then handed it off to Hicham to go into a technical review of capacity terms, badges and a demo, then back to me to share some specific slides on how we do capacity management at the bank.¬† All during Hicham’s sections, I would interject with customer insight, and it worked really well.¬† I had about 15 minutes of material to present at the end, and Hicham gave me about¬†8 minutes! lol. So I had to condense my info.¬† Hicham and I killed it. We really did. We had a Top 10 presentation for Wednesday. Woo-Hoo!¬† After the presentation because my time was short, I had a number of people come up to me with questions, and handed out my card for follow up.¬† They were very interested in my experience with vROps¬†at work. ¬†It was awesome, what a rush. It was like something clicked, I really enjoyed speaking in front of so many people and¬†presenting to them my slides and information.¬† It was¬†such a thrill, I would do that full time!¬† You can check out my session here. Top 10 Sessions at VMworld 2016 for Wednesday¬† Look for vRealize Operations Capacity Explained.

That's me, killing it on stage with Hicham.

That’s me, killing it on stage with Hicham.

Wednesday was also the VMUG Leader Lunch.¬† They served us a nice meal, unfortunately I had to leave early for my presentation and didn’t get to stay for the speakers. Hopefully next year I wont have a conflict.¬† One highlight is that I had a chance to meet VIRT. (@vmug_virt).

Me and @vmug_virt

Me and @vmug_virt

VMworld 2016 Party

Wednesday night was the VMworld party at the Las Vegas Motor Speedway.¬† It was a super hot night, like 103 at the time of the party.¬† That was the first time I experienced elbow sweat!¬† The party was a lot of fun, food was good, lots of entertainment and I had a ride around the track in a car.¬† Capital Cities opened for Fall out Boy.¬† Both bands were great, but Capital Cities really was awesome.¬† I always feel like I don’t have enough time to experience everything the party has to offer, but its always fun and its great that VMworld provides rides to and from the event.

Last day of VMworld 2016

The conference wrapped up on Thursday September 1st.¬† I had a couple of sessions scheduled in the morning and MGT7770 was my favorite session of the week. ¬†It was VSAN Management using vROps plus the Management Pack for Storage Devices (MPSD) with 100% live demos on a multi-cloud private\public VMware cloud foundation infrastructure.¬† The speakers were Jeff Godfrey and Rawlinson Rivera.¬† I really enjoyed this session.¬† After that session I¬†went back to the VMvillage one last time before I headed out.¬† The end of VMworld is bitter/sweet.¬† The environment¬†and experiences are so much fun.¬† Seeing friends and meeting new people and¬†attending great sessions¬†are¬†some of the¬†best parts¬†of the week and this year speaking in two presentations was by far the highlight for me; however I’m really ready to get back home to my family and return to normal life.¬† Vegas was great, and a huge shout out to the blue shirts and event staff for really making the week great.¬† They were so¬†helpful and welcoming, always smiling; who dosent miss the box lunches, am I right?!¬† All in all it was a really educational and fun week.¬† I’m definitely looking forward to next year back in Las Vegas! Who knows you might even see me back up on stage! ūüėČ

Hear are some additional photos from my trip.




vRealize Operations Manager 6.3 – What’s New!

Today VMware has released vRealize Operations Manager 6.3. vROps will maximize capacity utilization and enable optimum performance and availability of applications and infrastructures across vSphere, Hyper-V, Amazon, and physical hardware.  Streamline key IT processes with out-of-the-box and customizable policies, guided remediation and automated standards enforcement. Optimize performance, capacity and compliance while retaining full control of IT operations.

After I get back from VMworld I will be testing all the new features.¬† I’m looking forward to the DRS Dashbords!¬† Until then check out below all the new features.

What’s New in vROps 6.3

vRealize Operations Manager 6.3 is a release update that enhances product stability, performance, and usability.

Enhanced Workload Placement and vSphere Distributed Resource Scheduler (DRS) Integration:

  • Recommended Actions on landing page
  • Configure DRS from vRealize Operations Manager with Cluster/DRS Dashboard
  • Data Collection Notification toolbar
  • Perform Actions from Workload Utilization Dashboard

Improved Log Insight Integration:

  • Management Pack for Log Insight included in product
  • Improved Log Insight and vRealize Operations Manager alerting

Enhanced vSphere Monitoring:

  • Support vSphere 6.0 Hardening Guide
  • vRealize Operations Manager Compliance for vSphere 6.0 Objects
  • vRealize Operations Manager and vCenter Software-Defined Data Center (SDDC) Health Dashboards

General improvements:

  • New API Programming Guide
  • New licensing plan for vRealize Operations Manager Standard and Advanced Editions
  • Filtered SNMP Trap Alert Notifications
  • Enhanced SuperMetric capabilities
  • Reduction in default metrics collected

Source – vROps 6.3 release notes

Removing a Solution from vRealize Operations Manager 6.x

The great thing about having a lab environment is I get to test out a number of solutions for vROps. ¬†One that I¬†have been evaluating is the Cisco UCS Management¬†Pack for vROps. ¬† We started with a beta version for vROps 5.x then updated the pack for vROps 6.1 and now I wanted to install the newest version of the pack for 6.2. ¬†One problem, the old solutions are just kinda stuck in there. ¬†They don’t update and when you think you are removing the solution, you are really just deleting the adapter settings. ¬†In this article¬†ill go through the steps on how to remove the old management packs and get everything clean and ready for the new version.

***Caution!*** – we are going to be editing some sensitive files so you really should open up a service request with VMware support if you are doing this in production. ¬†I’m working in my lab environment ¬†so if things went FUBAR, its not a big deal. ¬† I will have to eventually do this in production, (well not me, the operations team) and so far I haven’t had any issues going through this, but ya never know. ¬†Open up an SR before touching your production vROps. ¬†This way in an event of an issue or a mistake VMware support can help guide you through fixing it. Okay, enough of that.

The fist step is to log into the vROps node that has the incompatible solution. In my lab I only have one node, so that’s pretty straight forward.

navigate to /storage/db/pakRepoLocal/  to determine which solution you want to uninstall.  I have a couple different UCS solutions installed.

Run this command to determine the actual adapter name and take note of the “Name” field.
cat /storage/db/pakRepoLocal/Adapter_Folder/manifest.txt

The next step to uninstall the solution pack is to change to /usr/lib/vmwarevcops/tools/opscli/   and run the opscli.sh with the uninstall option
./opscli.sh solution uninstall “Name_of_the_pak”

Once the process has completed you will see a return that states the uninstall has been successful. Like in the example below.

After the above step is complete, run this for some additional clean up.
$VMWARE_PYTHON_BIN $ALIVE_BASE/../vmwarevcopssuite/utilities/pakManager/bin/vcopsPakManager.py –action
cleanup –remove_pak –pak Name_of_Pak ¬† ¬† (replace Name_of_Pak with the name from above)

Next you will have to remove the solutions .pak file from the .pak files directory.
Go to $STORAGE/db/casa/pak/dist_pak_files/VA_LINUX  and rm the pack file name.

Now open¬†/storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json in a text editor and¬†delete all entries related to the removed solution from { to } ¬† Don’t forget to save it!

Lastly go back to the /storage/db/pakRepoLocal/  directory and remove the sub directories, files and parent directory for the solution you removed.  Use the rm and rmdir commands.   You may also have to delete any dashboards that were installed with the solutions pack from dashboards in the vRealize Operations Manager UI.  Also note that in order for the changes to take effect, you will need to log out and back into the UI.

Take your time running through the steps and you will see its not all that difficult. ¬†I’ve also used this process when a solution doesn’t install successfully before I try and reinstall it and remember to take caution when doing this in a production environment.