Diving into Powershell Azure Functions

Azure Functions are Microsoft’s “serverless” attempt to answer Amazon Lamda Function. I’ll avoid the fluff and simply say it’s a quick-and-dirty way to get some code on a publicly facing website. An ideal use case, for example, would be to process webhooks (i.e. consuming callback events).

Azure Functions support an array of languages, but we’ll focus primarily on their new support for Powershell (and some comparison to their C# .csx support).

At the time of this writing, powershell support was still labeled experimental and documentation was incomplete. This post attempts to serve as the hidden manual in the interim.

Read more Diving into Powershell Azure Functions

Removing Stubborn Orphaned Active Directory (Child) Domain

Say you’re using ntdsutil to remove an offline, orphaned domain as per KB230306 and you get an error like

DsRemoveDsDomainW error 0x2162(The requested domain could not be deleted because there exist domain controllers that still host this domain.)

There a few things you can still try. Again, we’re assuming the last DC in the domain is dead/offline and we’re in a completely orphaned state.

NOTE: These steps were performed on a child domain so millage may vary

Read more Removing Stubborn Orphaned Active Directory (Child) Domain

Dell PowerEdge Server Drivers Extraction for MDT (WinPE and OS)

Dell offers nicely bundled Client CAB Drivers for their line of workstations. They also offer a fairly large line of driver update and installation tools including the sometimes-hard-to-find Dell Server Update Utility (SUU), Dell Repository Manager (and it’s Client variant) which can be really handy lightweight, in-OS driver bundles, and of course Dell Support website.

What about MDT?

Although these are all great tools, we just want the raw INF files right? While Dell Repository Manager does offer an “export to RAW” option, I found it was really just an executable with some monolithic INF files that did not play well when importing MDT’s OOB store. I’m sure one could have chosen to package the whole thing up in application, but why do things differently and have potentially strange error codes?

Read more Dell PowerEdge Server Drivers Extraction for MDT (WinPE and OS)

PowerCli Helper Functions for your Profile

I’ve found the more I use PowerCLI the more I find the need to write some quick and easy helper functions for my $profile.

Using PsReadline to Show Which VMWare Servers You Are Connected To

 

Sometimes I might forget which vCenter I’m connected to via PowerCLI, especially if I have multiple tabs open. This is juast a quick modification of the prompt function using PsReadline.

See Also: PsReadline and ConEmu

viservers01

Script

Create A Quick Temp VM for Testing PXE Images (WDS/MDT)

Sometimes I need a quick VM to test out changes I’ve made to the WDS/MDT images. These functions can make this process a lot quicker.

Script

Other Useful PowerCLI Helper Functions

Some other functions that make using PowerCLI a bit easier.

Script

Powershell Script to Help Find Windows Uninstall Strings

Uninstallstring, uninstallstring ? { $_ } thou uninstallstring

Sometimes it can be a pain to find uninstall strings, and they can often be in a lot of places. I haven’t found the WMI calls to be terribly accurate in all cases.

This script simply searches the following registry subkeys for a partial string.

HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstal

HKLM:\SOFTWARE\WOW6432NODE\Microsoft\Windows\CurrentVersion\Uninstall

HKCU:\SOFTWARE\WOW6432NODE\Microsoft\Windows\CurrentVersion\Uninstall

HKCU:\SOFTWARE\WOW6432NODE\Microsoft\Windows\CurrentVersion\Uninstall

You will still need search through the keys to find plausible uninstall strings.

uninstallstringhelper01 Read more Powershell Script to Help Find Windows Uninstall Strings

VMWare Lost path redundancy to storage with EqualLogic Arrays / Initiator disconnected from target during login.

We were getting some transient alarms in VMWare similar to:

Lost path redundancy to storage device naa.64ed2ad569ca2e943e0a550600002009. Path vmhba37:C2:T0:L0 is down. Affected datastores: DATASTORENAME.

But all SAN HQ, VMWare host logs, and Dell LASO information didn’t shed much light on the situation. The host logs basically told us the same thing the alarms did; a path was dropping and coming back up.

tl;dr Working As Expected – ESX/ESXi hosts randomly drop and reconnect iSCSI connections to an EqualLogic array (2004432)

We had recently switched one of our vSphere installations from NAS to an EQL P6100 and also done some host upgrades to 5.5 within the week.  But, again everything was clean and there weren’t actually any noticeable degradation of performance.

Dell Multipath Extension Module

Now, I had used the Dell Image for ESXI 5.5 (VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A00.iso) to perform the upgrade. What I didn’t expect, however, is that the included version of Dell MEM would not be up to date. Granted, the version we were running prior was 1.1.2 where as the current is 1.2.0.

So, I checked the MEM version on the hosts themselves (via SSH):

So you can see the VIB dell-eql-routed-psp is on 1.1.1

After some reading, people had claimed this issue was alleviated by updating the MEM VIB on the hosts as well as updating the firmware on the SAN arrays. Pretty standard, low-hanging fruit advice but worth a shot on our non-production vSphere cluster. (Our new EQL PS6100 was still on firmware v6.x so that was upgraded to v7.x)

Results

So far, our vSphere cluster that has both the VIB and the SAN arrays upgraded no longer see this path degradation issue. It’s a pretty good comparison to our other vSphere cluster which is still seeing these alarms.

Remember to reboot the ESXi hosts after installing the new VIB and verify the VIB version using:

Don’t Forget LoginTimeout and DelayedAck

Unless you want your degraded paths to turn into lost paths, you should also make sure you follow the advice from Dell / EqualLogic and VMWae

Symptoms:

You will see generic lost connection errors in vSphere as well as generic errors in hostd.log

2014-09-24T08:03:22.931Z [29140B70 info ‘Hostsvc.VmkVprobSource’] VmkVprobSource::Post event: (vim.event.EventEx) {
–>    dynamicType = <unset>,
–>    key = 707785232,
–>    chainId = 707785232,
–>    createdTime = “1970-01-01T00:00:00Z”,
–>    userName = “”,
–>    datacenter = (vim.event.DatacenterEventArgument) null,
–>    computeResource = (vim.event.ComputeResourceEventArgument) null,
–>    host = (vim.event.HostEventArgument) {
–>       dynamicType = <unset>,
–>       name = “HOSTNAME”,
–>       host = ‘vim.HostSystem:ha-host’,
–>    },
–>    vm = (vim.event.VmEventArgument) null,
–>    ds = (vim.event.DatastoreEventArgument) null,
–>    net = (vim.event.NetworkEventArgument) null,
–>    dvs = (vim.event.DvsEventArgument) null,
–>    fullFormattedMessage = <unset>,
–>    changeTag = <unset>,
–>    eventTypeId = “esx.problem.scsi.device.state.permanentloss”,
–>    severity = <unset>,
–>    message = <unset>,
–>    arguments = (vmodl.KeyAnyValue) [
–>       (vmodl.KeyAnyValue) {
–>          dynamicType = <unset>,
–>          key = “1”,
–>          value = “naa.xxxxxxxxxxxxxxxxxxx”,
–>       },
–>       (vmodl.KeyAnyValue) {
–>          dynamicType = <unset>,
–>          key = “2”,
–>          value = “”VMFS_NAME””,
–>       }
–>    ],
–>    objectId = “ha-eventmgr”,
–>    objectType = “vim.HostSystem”,
–>    objectName = <unset>,
–>    fault = (vmodl.MethodFault) null,
–> }

and in EQL Group manager you’ll see something like this around the same time:

 Error    9/24/2014 2:38:22 AM  PS6100  7.4.3 | 7.4.23  iSCSI login to target ‘x.x.x.x:3260, iqn.2001-05.com.equallogic:x-xxxxx-HOSTNAME’ from initiator ‘X.X.X.X:30972, iqn.1998-01.com.vmware:HOSTNAME-XXXXX’ failed for the following reason: | Initiator disconnected from target during login.

Resolution:

Make sure to read the Deployment Considerations in the EqualLogic PDF. Aside from the ISCSI connection count, consider the sections as follows:

Deployment Considerations: iSCSI Login Timeout on vSphere 5.x
The default value of 5 seconds for iSCSI logins on vSphere 5.x is too short in some circumstances. For example: In a large configuration where the number of iSCSI sessions to the array is close to the limit of 1024 per pool. If a severe network disruption were to occur, such as the loss of a network switch, a large number of iSCSI sessions will need to be reestablished. With such a large number of logins occurring, some logins will not be completely processed within the 5 second default timeout period.
Dell therefore recommends applying patch ESXi500-201112001 and increasing the ESXi 5.0 iSCSI Login Timeout to 60 seconds to provide the maximum amount of time for such large numbers of logins to occur.

If the patch is installed prior to installing the EqualLogic MEM, the MEM installer will automatically set the iSCSI Login Timeout to the Dell recommended value of 60 seconds.
The iSCSI Login Timeout value can also be set using esxcli with the following syntax:
esxcli iscsi adapter param set –adapter=vmhba –key=LoginTimeout –value=60
See VMware KB 2009330 for additional information.

[…]

Disabling Delayed ACK: Delayed ACK is a TCP/IP method of allowing segment acknowledgements to piggyback on each other or other data passed over a connection with the goal of reducing IO overhead. One side effect of delayed ACK is that if the pipeline isn’t filled, acknowledgement of data will be delayed. In SANHQ this can be seen as higher latency during lower I/O periods. Latency is measured from the time the data is sent to when the acknowledgement is received. Since we are talking about disk I/O any increase in latency can result in poorer performance. Additional information can be found in VMware KB 1002598.
Note: While iSCSI Login Timeout is considered a best practice, it is also consider a requirement, and therefore will always be set to 60 seconds during installation.

You’ll need to reboot the host for the changes to take affect.

VMWare Tools Event Log Spam (Error 1000)

VMWare Tools (in 5.1 and 5.5 at least) can be a bit verbose in the Windows Event Logs. This is covered in KB2036350 Upgrading to VMware Tools 5.x causes log spew with the message: Error in the RPC receive loop: RpcIn: Unable to send and other such KBs.

Errors such as:

Error in the RPC receive loop: RpcIn: Unable to send.

-or-

vmusr[2116]: [warning] [vmusr] Error in the RPC receive loop: RpcIn: Unable to send.

-or-

[warning] [vmsvc:guestinfo] Conversion to UTF-8 failed
[warning] [vmsvc:guestinfo] Failed to get vmstats

vmwareevent1000 Read more VMWare Tools Event Log Spam (Error 1000)

A Better Powershell Console with Cmder (ConEmu) and PsReadLine

cmder_powershell
We’re making a better console. All of them, better consoles.

I don’t think I have to tell any Windows administrator that Powershell, while not perfect, is a God send compared to the old cmd days. But, my biggest gripe with Powershell has always been that its console (frankly) sucks compared to modern Linux terminals. And, the PS console has a very limited range of customization in terms of colors, fonts, tab completion, etc.

Trying to “natively” hack away at the console can be less than ideal from my experience. This is why ConEmu (or the prefab ConEmu setup Cmder) is the best solution I have seen thus far. It even allows you to attach easily to putty via something like putty -new_console

But, ConEmu and Cmder are both designed to improve the traditional CMD experience. Although a powershell session within ConEmu/Cmder are inherently better, it still lacks some things like Linux-style tab completion, coloring, location concatenation, etc.

This is where PsReadLine comes in. Assuming you’re running Powershell v3+, will allow us to manipulate the prompt, rebind hot keys, and customize coloring.

Read more A Better Powershell Console with Cmder (ConEmu) and PsReadLine

Powering off All Guests and Entering Maintenance Mode with VMWare PowerCli

We have a smaller, non-production cluster in our office that is, sadly attached to a Windows Storage server. And, given certain security requirements, we have to keep up to date (within 30 days) of Microsoft patches.

Yep, reboots on the entire VMFS.

There’s no -Evacuate switch on Set-VMHost to save us here.

This script will turn off all the VM guests in the set of VMHosts by trying the following in order:

  • Gracefully trying to shut down the guests – Shutdown-VMGuest
  • Forcefully shutting them down Stop-VM -Kill
  • Really forcefully trying to shut down via killing the PID on the ESXi host using esxcli.

It will then enter the VM Hosts in maintenance mode.

After you prompt it, it will take the hosts out of maintenance mode then try the following:

  • Bring the guets back online
  • Try to bring them up again, excerpt this time answering questions regarding “Did you copy this VM?” (I found sometimes this was an issue when completely shutting down the storage)

Read more Powering off All Guests and Entering Maintenance Mode with VMWare PowerCli

Powershell 32-bit and PowerCLI – Web Proxy Error

While trying to launch VMWare PowerCLI in 32-bit mode (to run certain commands like New-OSCustomizationSpec) I was getting an error

error creating the web proxy specified in the ‘system.net/defaultproxy’ configuration section

even though I had been using PowerCLI 64-bit for months. How odd.

Using the 32-bit shortcut in cmd.exe

and trying to manually load the snappin in Powerhsell x86 via Add-PsSnapin Vmware*Core resulted in the same error.

Turns out if you make this modification tot he x86 powershell.exe.config file it will resolve the issue.

e.g. C:\windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe.config

It also seems you can use the Set-PowerCliConfiguration as well – https://www.vmware.com/support/developer/PowerCLI/PowerCLI501/html/Set-PowerCLIConfiguration.html