Tuesday, November 28, 2017

Active Directory Replication: The active Directory property cannot be found in the cache

Last night I came across a problem with AD that I have never seen before. Normally, Active Directory Domain Services is an extremely robust application that will continue to work, regardless of not implementing best practices, or misconfiguration. However, I promoted a new domain controller and was surprised to see that it was not able to replicate with any other DCs.

Everything appeared to be configured properly. This seemed like the millionth domain controller I’ve created throughout my career, so I was accustomed to the process.

When clicking “Check Replication Topology” within Sites and Services, I received this error:

The active Directory property cannot be found in the cache

 

The problem was caused by a missing cname record in the _msdcs.<domain name>.<tld> forward lookup zone:



 

When a domain controller registers its name with another DNS server, it needs to create a CNAME record in this zone, which is then used by other domain controllers to locate replication partners. This is what the zone looks like normally:



 

There should be a CNAME record for each domain controller, mapping the DSA GUID to the FQDN of the domain controller. The newly promoted domain controller did not automatically create this CNAME record. After manually creating it, and waiting a little while, I was able to replicate all partitions throughout the domain.

 

To create this record, first obtain the DSA Guid. Open a cmd prompt on the problematic domain controller and type in:

Repadmin /showrepl



 

 

Then open DNS on another (working) domain controller and create a CNAME record in the _msdcs.<domain name>.<tld> forward lookup zone.



 

Also verify that the new domain controller is listed as a name server for the zone.



 

Just wait a little while, and then try to replicate all partitions.

 

Monday, November 13, 2017

Azure AD Connect: Health service data is not up to date

Noticed this error earlier today in the Azure Portal. This happened after renaming the server over the weekend. It looks like the Agent on the server uses a certificate to authenticate with Azure. Since the server name changed, the certificate is no longer valid.



I ran the "test-azureAdConnectHealthConnectivity" cmdlet as suggested, and got these results:



It was pretty obvious what the problem/solution was after seeing the "Invalid Credentials" response from the server. I wasn't sure how to update the certificates without completely re-installing Azure AD Connect, and couldn't find anything on Google that would help. But after poking through the Azure AD Connect Powershell module, I found this cmdlet: Register-AzureAdConnectHealthSyncAgent.



After running this cmdlet, and restarting the Azure AD Connect services, the error message was resolved in the Portal. Hope this helps anyone experiencing this issue.

Wednesday, November 8, 2017

Remove Stubborn PSC or vCenter Appliance from an SSO Domain

While attempting to decommission one of our vCenter sites, I ran into an issue removing one of the PSCs. This site consisted of two PSCs and one vCenter appliance. I removed the first PSC from the SSO domain successfully, and then removed the vCenter appliance. Things became a little tricky during the removal of the final PSC. This PSC did not get removed even after running the cmsso-util command. This article will detail the steps I took in decommissioning the site, as well as removing the stubborn PSC.

First, we need to check if vCenter is currently using the PSC we plan on decommissioning. If it is, we need to use the cmsso-util command to redirect vCenter to a different PSC. Instructions for redirecting vCenter can be found here: https://kb.vmware.com/s/article/2113917?language=en_US

To check what PSC your vCSA is currently pointing to, browse to the Advanced Settings for the vCSA in the vSphere Web Client. Filter by this key: config.vpxd.sso.admin.uri

To remove a PSC or vCSA from an SSO domain, connect to a PSC via SSH and run these commands:

To remove a PSC from the vSphere SSO domain:

cmsso-util unregister --node-pnid psc01.ad.vcplab.local --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

To remove a vCSA from the vSphere SSO domain:

cmsso-util unregister --node-pnid vcsa.ad.vcplab.local --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

After running these commands, delete the virtual appliances. You can also verify the appliances have been removed by browsing to Administration > System Configuration > Nodes in the vSphere Web Client.

If cmsso-util fails to remove any of the nodes, you can use this command to force the removal:

vdcleavefed -h vcsa.ad.vcplab.local -u Administrator -w Passw0rd!

Upon successful completion, you should see something like this:

/usr/lib/vmware-vmdir/bin/vdcleavefed -h vcsa.ad.vcplab.local -u administrator
password:
vdcleavefd offline for server vcsa.ad.vcplab.local
Leave federation cleanup done

When specifying the username, use the SSO admin (administrator). However, do not use the full UPN (administrator@vsphere.local). Doing so will cause the command to fail.

Tuesday, November 7, 2017

Exchange 2016 Hybrid Deploy Check - Username or Password Invalid

These days, it seems every Microsoft product comes with its own unique set of head scratchers. Microsoft Exchange Server is no exception to this. I was installing Exchange 2016 earlier today, to be used as a hybrid configuration server for Office 365 (no local mailboxes). I downloaded the self depackaging executable from Microsoft, and attempted to install it. If you currently have a hybrid configuration (which we did, with Exchange 2010), the Exchange 2016 installer will detect this and run some tests to verify that the Office 365 tenant is ready for Exchange 2016 (more info here: https://technet.microsoft.com/en-us/library/ms.exch.setup.hybridconfigurationstatuspage(v=exchg.160).aspx). You’ll be prompted for Office 365 credentials (the user must have the Organization Management role). Seems simple enough, right? Wrong.

After typing in the username and pasting the password into the password field, setup came back with an error message stating the username or password was wrong. I then clicked the back button, and it crashed. I ran through this process a few more times, all with the same outcome. I even rebooted the server, which (in my opinion) should never be the resolution to a software problem. I looked through setup logs and found no indication of what the problem could be…



It was on the fourth try that I typed in the password, and this seemed to work. I didn’t receive any error messages about the credentials being wrong. The Exchange setup seemed to continue on successfully. However, it then failed with a different error:



I again looked through the setup logs and found that this error happened anytime setup tried to run the “Get-OrganizationConfig” cmdlet. After troubleshooting for a little while, and no resolution in sight, I turned to Google. One of the posts I came across said that this is a bug in the Exchange installer, and to try and use the Cumulative Update installer instead. Apparently, with Exchange 2016, the Cumulative Update installer’s include all of the Exchange binaries, not just the updated binaries. I downloaded the installer for CU7 (all 6 gigabytes of it…) and successfully installed Exchange 2016. Hope this helps anyone out there struggling with this.

Friday, October 20, 2017

Enable SNMP in ESXi 6.5 via SSH

I was trying to enable SNMP on a few stand-alone ESXi 6.5 hosts the other day. Unfortunately, when attempting to enable this service using the vSphere Host Client, I received a very unhelpful error:

 



I wasn't able to find any helpful information in the logs that could help identify the cause. Interestingly enough, enabling the service through SSH worked. Here's how I did it. First, you'll need to connect to your host via SSH (you will need to enable the SSH daemon first).

Once connected, type in "esxcli system snmp get" to see the current snmp configuration.



To enable SNMP, we first need to configure the community strings. To do that, type in "esxcli system snmp get --communities=<community strings>". Then, enable the service by typing in "esxcli snmp set --enable=true".

You can then verify the new configure by typing in "esxcli system snmp get" again.



Now refresh the vSphere Host Client, and the service should be running.

Tuesday, September 26, 2017

Auditing Exchange Online Transport Rule Use

I recently came across a transport rule being unnecessarily used in Exchange Online. The transport rule in question was used for DLP, and encrypted messages based on the content. It searched message bodies for strings of characters matching credit card numbers, SSN's, etc.  I was surprised to see that there was no way to easily audit transport rule usage with Powershell, so I checked the Exchange Control Panel. There is an option for auditing in each of the transport rules:



So, what does this checkbox actually do?

Enabling this checkbox will cause this rule to appear in Message Trace logs when it is applied to a message. 

Let's look at an example:

Here we have a transport rule in Exchange Online that appends "Outbound" to the subject line of all messages sent to external recipients:



 

After sending a message to an external recipient, we can see the rule working:



Let's take a look at the message trace:



We can see two entries in the message trace log. The first is for applying the transport rule, and the second entry is for setting the audit severity level. I haven't found a good explanation of the audit severity levels, other than you can filter by them when doing a message trace.

If you know of any other use for the audit severity levels, leave a comment below!

Wednesday, July 12, 2017

vSphere - Unable to Apply DRS Resource Setting on Host

I noticed this error on one of my hosts today. After reading the full error message on the Events tab, and paying particular attention to the "Error Stack" section, it was obvious what the problem was.



"The configuration file of the virtual machine is corrupted". Well, that helps narrow things down, but what virtual machine is it?

After clicking on the host and going to the Related Objects -> Virtual Machines tab, the problem became even more clear.



This was a replica that we deleted last weekend. For whatever reason, vCenter didn't de-register it from the database when it was deleted. The VM no longer existed on the host. To fix the issue, just right click on the VM and click "Remove from Inventory".

After doing this DRS became healthy and all alarms were silenced. I hope this helps anyone experiencing this or a similar problem.

-Ryan

Tuesday, June 27, 2017

Migrate Windows Deployment Services to New Server

We have been making a great effort to move all of our internal services to Windows Server 2016. This past week, it was WDS' turn to get migrated. Migrating this role is extremely simple. Here are the steps that I took:

  1. Create new server and install WDS role.

  2. Stop WDS Service on old server

  3. Stop WDS Service on new server

  4. Use my "Copy-Files" PowerShell script (Available Here: Copy-Files.ps1) to copy RemoteInstall Share to new server

  5. Start WDS Service on new Server

  6. Shutdown old WDS Server completely

  7. Update option 66/67 in DHCP scopes to reflect new WDS Server

  8. Update any appropriate DNS records


Note:

If you are unable to start the WDS service, delete the WDS database and logs from the old server located at <drive letter>:\RemoteInstall\Stores\Metadata\*.*. You should be able to start the service after deleting these files.

Simple enough! :)

Monday, June 26, 2017

New Script: BulkAdd-SpamFilterWhitelist.ps1

This script is capable of adding a list of domains to an Exchange Online Spam Filter policy. It can be downloaded from TechNet or Github.

Github

TechNet

Unable to Power on Virtual Machine - A General System Error Occurred: Connection Refused

Hi there,

Earlier today I cloned a VM in one of our vCenter appliances. The cloning process completed without a hitch (as it usually does). However, I was not able to power on the VM. I looked at the "Tasks" view in vCenter and found this:



This is a fairly generic error message. However, the "Connection Refused" part prompted me to take a look at the vCenter services. You can view these services by browsing to Home > Administration > System Configuration > Services in the vSphere Web Client for vCenter. On the Summary tab, there is a handy "Services Health" section that will give you a high level overview of the overall health of the services. You can see here that three of the services are in a critical state on my vCenter appliance.



You can hover your mouse pointer over the critical services link and view the critical services. You can also click on each service to get a better view of the status. The Postgres database service on my appliance was one of the services in critical state. I clicked on it and viewed the screen below. You can see here that the problem is obvious, the filesystem holding the Postgres database service is completely out of space. I was able to identify that the log directory was the culprit.



Its simple enough to SSH into the appliance and delete the logs. However, this will not prevent the problem from happening again. After doing some Googling (I love that the Merriam-Webster dictionary identifies this as a real word in the English language!), I found these KB articles:

Follow this one to decrease the maximum backup size and maximum backup index size of the SSO logs in the log4j.properties configuration file:

/storage/log directory is full in vCenter Server Appliance 6.0 (2143565)


To offload your logs to a syslog server, follow this article:

Redirect vCenter Server Appliance Log Files to Another Machine

This will prevent the log directory from consuming the entire partition again. After following these steps I rebooted the vCSA for good measure and was able to power on the virtual machine after doing so.

Hopefully this will help someone who is having the same (or similar) problem!

Thursday, June 22, 2017

Windows 8 File History

File History


File history in Windows 8 allows users to automatically “backup” files that are in their libraries, contacts, favorites, Skydrive, and on the desktop. If the files are lost, they can be quickly restored. You can also restore a file from a specific point in time, being that File History creates a complete history of your files over time. You will need a separate drive other than the one you have Windows 8 installed on to use File History. To begin using it, connect an external drive and search for File History on the Start Screen:



After opening File History, you will see this screen:



To enable File History, click the ‘Turn On’ button. You can select the drive you want to use by clicking ‘Select Drive’ on the left hand side from this same screen. The first time you enable File History, it will create a full backup of all files on your computer, except for files that you do not access (system files), and files you have chosen to exclude. From then on, it will create a versioned copy of every file that has changed since the last backup.

You can use a locally attached drive or a network share for File History. To choose how often File History backs up files; choose ‘Advanced Settings’. From here you can also choose how much space on the drive is used, and how long saved versions of files should be kept.

When the total space allocated to File History has been used, the feature will delete the oldest versions of files to make room for newer versions.

User Account Control

User Account Control


Many people rarely pay close attention to those pesky User Account Control prompts that pop up when attempting to run a program as an administrator. When a user logs into her account, she is assigned a security token based on the level of access that she has (basic user or admin). This token is what defines what programs are allowed to do. Using this concept, the users session is incapable of making changes that would affect the entire system.

For a standard user, a token with the most basic privileges is assigned. Administrators have two tokens assigned, the first token contains all privileges usually awarded to an administrator (unrestricted), and the second is similar to that awarded to a basic user. This way, all programs that the administrator runs, including the Windows Shell, are run in basic user mode (this is why administrators still receive UAC prompts, but do not get asked for credentials). When an application requests higher privileges, the higher integrity token is used.

User Account Control prompts each have their own meaning based on the color (blue, grey, yellow, or red. The blue prompt (Figure 1) means that a built in Windows component that is signed by Microsoft is requesting administrative privileges to continue. This prompt has a multicolored shield in the upper left corner. The grey prompt (Figure 2) means a software application that is signed by a third party developer is requesting admin privileges. This prompt has a yellow shield with a black exclamation mark in the upper left corner. The yellow prompt (Figure 3) signifies that a unsigned executable is requesting administrator privileges.  This prompt also had a yellow shield with black exclamation mark in the upper left corner, and looks somewhat generic. Finally, the red prompt (Figure 4) means a program that was specifically blocked by an administrator has attempted to run.


Figure 1



Figure 2



Figure 3



Figure 4

Tuesday, June 20, 2017

How to Permanently Remove Office 365 Users

After deleting a user in Office 365, their account is moved to a 'recycle bin' for 30 days. This allows the user account to be easily recovered. This can often cause issues when attempting to recreate a mailbox while a hybrid configuration is in place.

To permanently delete the user within Office 365, first delete the user in the Office 365 Admin Portal or using Powershell. Then, connect to your Azure Active Directory environment with Powershell using the "Connect-MsolService" cmdlet.


To see a list of user accounts currently in the recycle bin, run this cmdlet:


Then, to permanently delete all accounts in the recycle bin, run this cmdlet:


To remove a specific user, run this cmdlet:

How to Permanently Remove Office 365 Users

After deleting a user in Office 365, their account is moved to a 'recycle bin' for 30 days. This allows the user account to be easily recovered. This can often cause issues when attempting to recreate a mailbox while a hybrid configuration is in place.

To permanently delete the user within Office 365, first delete the user in the Office 365 Admin Portal or using Powershell. Then, connect to your Azure Active Directory environment with Powershell using the "Connect-MsolService" cmdlet.

To see a list of user accounts currently in the recycle bin, run this cmdlet:

Then, to permanently delete all accounts in the recycle bin, run this cmdlet:

To remove a specific user, run this cmdlet:

Monday, June 12, 2017

"Access is denied" When Attempting to Delete a Dynamic Distribution Group

You may receive the error below when attempting to delete a dynamic distribution group.


To resolve this, open ADUC and show advanced features (Click View > Advanced Features). Then find the object for the dynamic distribution group and open the properties window. Browse to the "Object" tab and uncheck the "Protect object from accidental deletion" box. Wait for ADDS to replicate or force replication yourself. 


Go back to the ECP and you should be able to delete the group.

"Access is denied" When Attempting to Delete a Dynamic Distribution Group


You may receive the error below when attempting to delete a dynamic distribution group.





To resolve this, open ADUC and show advanced features (Click View > Advanced Features). Then find the object for the dynamic distribution group and open the properties window. Browse to the "Object" tab and uncheck the "Protect object from accidental deletion" box. Wait for ADDS to replicate or force replication yourself.



Go back to the ECP and you should be able to delete the group.

Sunday, May 28, 2017

Running vSphere in VMware Workstation 12

In this post I'll be walking through how to run a vSphere lab in VMware Workstation. I recently decided to obtain VCP6-DCV. Rather than driving up my electric bill like I've done in the past using physical servers, I'm attempting to run the entire lab on my workstation and a Synology NAS. 

If you've ever installed ESXi, installing it in Workstation will be a familiar process for you. VMware tools is included in the installation disc, which makes installing ESXi in Workstation dramatically easier than it used to be. . The process is very simple, so I won't be going through those steps here unless someone asks me to in the comments. I also will not be going through the process of installing Windows Server or configuring a domain controller/DNS/DHCP, as I am sure you have done so in the past if you are reading this. 

So that really only leaves us with installing vCenter. Most of the blogs I found for installing vCenter in VMware Workstation 12 were not accurate, and often left me with a broken installation. The process is somewhat straight-forward when deploying from the OVA. Let's get started.

First, download the OVA for vCenter here: Download vCenter

Once the download has completed, click File > Open in Workstation. Browse to the OVA, then give your new VM a name and location if necessary. Accept the EULA when prompted. 
Be sure to read it! 😎

Once the OVA finishes importing, do not power on the VM! There is some customization we need to do first. Close Workstation if it is open. Browse to the location on your PC that you imported the VM to. I'm using a Windows OS, so I will use File Explorer. Open the .VMX file (use Notepad or another text editor):


This is the configuration file for your virtual machine. We can use it to customize the name, IPv4/6 details, DNS domain, etc. Scroll down to the last line of text, and paste this in:

guestinfo.cis.vmdir.password = "vmware!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "10.0.0.15"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "10.0.0.10"
guestinfo.cis.appliance.net.gateway = "10.0.0.1"
guestinfo.cis.appliance.root.passwd = "vmware!"

Customize the above code to your needs. You will likely need to change the IPv4 details. Save the .VMX file and close your text editor. Now you can power on the virtual machine, and vCenter will run through the installation process. The installation can take around 10-15 minute in my experience. You may see generic login screens during the installation of Photon, do not login or interrupt the installation. Once it is complete, you should see the DCUI below:


You should now be able to browse to the IP address or DNS name of your vCenter server. Once you complete the configuration, you can login and see the page below:



In my lab I am running 3 ESXi hosts, 1 Windows Server, and one vCenter server. Plenty to study for the VCP lab. 


Good luck and be sure to leave a comment if you have any questions!

Running vSphere in VMware Workstation 12

In this post I'll be walking through how to run a vSphere lab in VMware Workstation. I recently decided to obtain VCP6-DCV. Rather than driving up my electric bill like I've done in the past using physical servers, I'm attempting to run the entire lab on my workstation and a Synology NAS.


If you've ever installed ESXi, installing it in Workstation will be a familiar process for you. VMware tools is included in the installation disc, which makes installing ESXi in Workstation dramatically easier than it used to be. . The process is very simple, so I won't be going through those steps here unless someone asks me to in the comments. I also will not be going through the process of installing Windows Server or configuring a domain controller/DNS/DHCP, as I am sure you have done so in the past if you are reading this.


So that really only leaves us with installing vCenter. Most of the blogs I found for installing vCenter in VMware Workstation 12 were not accurate, and often left me with a broken installation. The process is somewhat straight-forward when deploying from the OVA. Let's get started.


First, download the OVA for vCenter here: Download vCenter


Once the download has completed, click File > Open in Workstation. Browse to the OVA, then give your new VM a name and location if necessary. Accept the EULA when prompted.

Be sure to read it!


Once the OVA finishes importing, do not power on the VM! There is some customization we need to do first. Close Workstation if it is open. Browse to the location on your PC that you imported the VM to. I'm using a Windows OS, so I will use File Explorer. Open the .VMX file (use Notepad or another text editor):





This is the configuration file for your virtual machine. We can use it to customize the name, IPv4/6 details, DNS domain, etc. Scroll down to the last line of text, and paste this in:


guestinfo.cis.vmdir.password = "vmware!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "10.0.0.15"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "10.0.0.10"
guestinfo.cis.appliance.net.gateway = "10.0.0.1"
guestinfo.cis.appliance.root.passwd = "vmware!"


Customize the above code to your needs. You will likely need to change the IPv4 details. Save the .VMX file and close your text editor. Now you can power on the virtual machine, and vCenter will run through the installation process. The installation can take around 10-15 minute in my experience. You may see generic login screens during the installation of Photon, do not login or interrupt the installation. Once it is complete, you should see the DCUI below:




You should now be able to browse to the IP address or DNS name of your vCenter server. Once you complete the configuration, you can login and see the page below:


In my lab I am running 3 ESXi hosts, 1 Windows Server, and one vCenter server. Plenty to study for the VCP lab.



Good luck and be sure to leave a comment if you have any questions!

Wednesday, February 15, 2017

Azure AD Sync - Unable to Update this Object - Values Associated with Another Object

ERROR MESSAGE:

User@contoso.com<mailto:user@contoso.com>

Unable to update this object because the following attributes associated with this object have values that may already be associated with another object in your local directory services: [UserPrincipalName user@contoso.com<user@contoso.com>;]. Correct or remove the duplicate values in your local directory. Please refer to http://support.microsoft.com/kb/2647098 for more information on identifying objects with duplicate attribute values


When implementing SMTP soft-matching after a cutover migration, you may see this error if objects in the on-premises AD were not properly prepared to sync.

To fix this, you need access the object in ADUC and set the “E-mail:” attribute to the primary SMTP email address of the user in Office 365. This is how SMTP soft-matching works.


 


Then run another sync and the on-premises AD account should sync up with the Office 365 account. For help with forcing a sync, please see this article:


 

Follow by Email

Search This Blog