VMware NSX: Promoting a Secondary NSX manager with the REST API in cross-vCenter setup

Many, if not all and more, actions you can do via the VMware vCenter web client are also possible with the REST API. This design is perfect for people like me who like to automate everything. During this blog post I would like to explain the steps that are required in a cross-vCenter NSX failover scenario to promote a Secondary NSX manager to the Primary role.

The problem

In a cross-vCenter VMware NSX environment, multiple VMware NSX managers can be linked together. In such an environment there is always one Primary VMware NSX manager and one or more Secondary VMware NSX managers.
The cross-vCenter VMware NSX feature allows the use of universal objects (universal security groups, universal logical switches, …) which can be created via the Primary VMware NSX Manager and are synced to all connected Secondary VMware NSX managers.
When the Primary VMware NSX manager fails (think of a datacenter outage), it is necessary to promote one of the Secondary VMware NSX managers to the Primary role. This is fairly trivial via the VMware vCenter web client but of course it would be great to automate this process via the REST API.

In the solution section I will show you how to do it with the best REST API invocation tool ever created CURL (yes, I know Postman exists…)!

The solution

First I will explain the setup that I currently have running. I have two nested vCenters each linked to its own VMware NSX manager instance.
Following table will summarize the setup:

VMware vCenter
FQDN
VMware vCenter
IP
VMware NSX
FQDN
VMware NSX
IP
VMware NSX
role
T05VCA01 10.10.5.60 T05NSX01 10.10.5.64 Primary
T05VCA02 10.10.5.62 T05NSX02 10.10.5.65 Secondary

Assign the Primary role to the Secondary VMware NSX manager

We will pretend like the Primary NSX manager goes down and the Secondary NSX manager needs to be promoted to primary. Before we can do that, we should mark the Secondary VMware NSX manager as standalone.
This can be done with following command:

curl -X POST https://<<SECONDARY VMWARE NSX MANAGER>>/api/2.0/universalsync/configuration/role?action=set-as-standalone -k -u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml"

Filled in command:

curl -X POST https://t05nsx02/api/2.0/universalsync/configuration/role?action=set-as-standalone -k -u admin:SuperSecurePassword -H "Content-Type: application/xml"

Output:

<?xml version="1.0" encoding="UTF-8"?>
curl: (56) SSLRead() return error -9806
<error>
   <details>Unable to assign STANDALONE role. Universal objects of following types are present: LogicalSwitch,TransportZone,LogicalRouter,VNI Pool. Please delete these objects and try the operation again. NSX manager will remain in TRANSIT state until these objects are deleted.</details>
   <errorCode>125023</errorCode>
   <moduleName>core-services</moduleName>
</error>

The error returned is normal because in my current environment I have universal objects present which I do not want to delete in case of disaster recover. When we check the VMware vCenter web client, we see that the role is listed as Transit.
The transit state will allow us to give the former Secondary NSX manager the Primary role.

curl -X POST https://<<SECONDARY NSX MANAGER>>/api/2.0/universalsync/configuration/role?action=set-as-primary -k -u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml"
curl -X POST https://t05nsx02/api/2.0/universalsync/configuration/role?action=set-as-primary -k -u admin:SuperSecurePassword -H "Content-Type: application/xml"

Output:

<?xml version="1.0" encoding="UTF-8"?>
<universalSyncRole>
   <role>PRIMARY</role>
</universalSyncRole>

We now have a new Primary VMware NSX manager.

So we have our basic case ready! The secondary datacenter could now be used as primary site.

Reverting to the original setup

Of course image our original Primary VMware NSX manager comes back online, it is possible that we want to go back to the original condition.

First we need to remove the Primary role from the original Secondary VMware NSX manager. This can be done with the same REST call we used to remove the Secondary role in the first place.

curl -X POST https://<<SECONDARY NSX MANAGER>>/api/2.0/universalsync/configuration/role?action=set-as-standalone -k -u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml"
curl -X POST https://t05nsx02/api/2.0/universalsync/configuration/role?action=set-as-standalone -k -u admin:SuperSecurePassword -H "Content-Type: application/xml"

output:

<?xml version="1.0" encoding="UTF-8"?>
curl: (56) SSLRead() return error -9806 
<error>
   <details>Unable to assign STANDALONE role. Universal objects of following types are present: LogicalSwitch,TransportZone,LogicalRouter,VNI Pool. Please delete these objects and try the operation again. NSX manager will remain in TRANSIT state until these objects are deleted.</details>
   <errorCode>125023</errorCode>
   <moduleName>core-services</moduleName>
</error>

This command will bring the Secondary VMware NSX manager to the Transit role once again.

Before we turn our attention to the Primary VMware NSX manager, we execute following commands on the Secondary VMware NSX manager (now in Transit role).

curl -X GET https://<<SECONDARY VMWARE NSX MANAGER>>/api/1.0/appliance-management/certificatemanager/certificates/nsx  -k –u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml"

After we fill in the parameters with our correct values:

 curl -X GET https://t05nsx02/api/1.0/appliance-management/certificatemanager/certificates/nsx -k -u admin:SuperSecurePassword -H "Content-Type: application/xml" 

We retrieve the following return message:

<?xml version="1.0" encoding="UTF-8"?>
<x509Certificates>
   <x509certificate>
      <subjectCn>t05nsx02</subjectCn>
      <issuerCn>t05nsx02</issuerCn>
      <version>3</version>
      <serialNumber>65c2c4ea</serialNumber>
      <signatureAlgo>SHA256WITHRSA</signatureAlgo>
      <signature>8b 19 fa 52 89 da 10 ff 31 c2 90 ce 90 0d fa 42 f7 88 a2 dc 3b 45 11 c5 9d 5f 02 3f a1 7d 63 d3 58 e3 32 3b ff 82 55 d0 3e 88 9c a9 da 07 82 c0 fe df 6c ea c3 5f 5a 7c 79 97 78 f4 22 21 04 cc d0 72 a1 89 3f 31 c1 b8 03 46 e4 43 a4 4b 41 d0 d4 51 45 1c 97 6a 31 85 da 92 69 70 9f df 4a 26 d7 67 4d 1c 0c 16 38 70 20 c4 09 83 cc 17 4d 76 79 63 da 7f fe 82 c2 3c bf 1d 5c ee aa 2e c5 0d 6b f9 bf 23 b7 2b c2 bc 18 3b 38 56 f3 69 a3 72 a8 1c a4 bf 02 3d ec d0 95 1c 56 be fd a4 30 9e 1d ba 25 b1 1f 77 c6 11 db ab 39 24 b5 4c 21 9c 55 c6 e0 ea cb 93 16 7f b3 41 6c 6d 3c 82 b5 ff c9 40 3f f2 3e 08 d7 a6 a6 cb df bc 19 04 0f 70 9a cf 46 0a 33 a3 af e1 7c f9 76 91 a2 de b9 b8 fc 88 61 f6 80 ce 0e 5f 3f 00 e1 ee 86 95 9e 18 e0 16 ea 13 c4 21 8b 79 23 68 99 2d 16 93 9c b2</signature>
      <notBefore>1470217814000</notBefore>
      <notAfter>1785577814000</notAfter>
      <issuer>CN=t05nsx02,OU=NSX,O=VMware Inc.,L=Palo Alto,ST=CA,C=US</issuer>
      <subject>CN=t05nsx02,OU=NSX,O=VMware Inc.,L=Palo Alto,ST=CA,C=US</subject>
      <publicKeyAlgo>RSA</publicKeyAlgo>
      <publicKeyLength>2048</publicKeyLength>
      <rsaPublicKeyModulus>00 b0 97 0a c3 be b9 57 0c ee 48 b4 8b e1 eb 13 81 12 18 e8 e4 41 1a df 13 2c dc a1 24 24 9d f4 ad af 43 62 86 99 91 a5 c1 43 86 32 3f f1 20 e2 6e ff 56 38 30 90 b4 b7 9c 81 c9 5c 33 75 32 b2 fc ee 2c de 0f 4a 42 ca 88 be 18 da dd 04 1b 53 e7 b1 bf b7 6b fc 01 4b b8 23 18 09 e6 33 db 94 e9 d2 4e fb 27 8c c3 3b a4 10 47 cf 0b 31 2c 5e 2c 08 aa d5 0a 75 35 14 b6 d7 0a 6b 20 5d 40 88 6f 14 55 d4 c7 7c 1b 15 af 5e ec 8b b8 f6 ac 49 d7 2b 19 b8 a5 ec fd 13 23 cf c5 8a 1a 93 f6 45 47 56 b1 94 4a 6e c0 82 d1 9d 5c d3 32 fc 23 a8 c7 8c c0 83 83 62 a6 4f 32 40 a4 72 fb 00 a3 7b d2 05 21 74 64 12 7d 86 bd 2a f9 17 9f cf 19 24 ff d5 4c b7 5a a1 9e 77 c4 96 d5 f9 3b 31 8f 1b 8a fe 6b 24 ca 9f be d6 37 b7 32 a3 af 21 4c 9b 76 93 e2 56 f8 d2 d8 6e 7a 88 76 a4 55 c6 9f bf 6b</rsaPublicKeyModulus>
      <rsaPublicKeyExponent>10001</rsaPublicKeyExponent>
      <sha1Hash>ab:45:b0:8f:ef:32:a9:a7:54:14:85:31:a4:18:b4:01:fa:3e:5b:ae</sha1Hash>
      <md5Hash>9f:d2:84:1e:09:17:89:ba:f8:d2:00:0d:4d:a6:87:4c</md5Hash>
      <isCa>false</isCa>
      <isValid>true</isValid>
   </x509certificate>
</x509Certificates>

Note down the sha1Hash field (in this case ab:45:b0:8f:ef:32:a9:a7:54:14:85:31:a4:18:b4:01:fa:3e:5b:ae) because we will need it for our next step in the workflow.

Our Primary VMware NSX manager is not aware of the different steps we performed on the Secondary VMware NSX manager. For this reason, we will first need to remove Secondary VMware NSX manager before we can add it again.

In order to do this, we will need to invoke a DELETE REST call on the Primary VMware NSX manager.

First we need to get the uuid of the Secondary VMware NSX manager and use it in our next DELETE curl command:

curl -X GET https://<<PRIMARY VMWARE NSX MANAGER>>/api/2.0/universalsync/configuration/nsxmanagers -k -u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml

output:

<?xml version="1.0" encoding="UTF-8"?&gt
<nsxManagerInfos>
   <nsxManagerInfo>
      <uuid>a6fd5275-738e-4d86-9c0b-d463157bcd9f</uuid>
      <nsxManagerIp>10.10.5.65</nsxManagerIp>
      <nsxManagerUsername>replicator-a6fd5275-738e-4d86-9c0b-d463157bcd9f</nsxManagerUsername>
      &ltcertificateThumbprint>AB:45:B0:8F:EF:32:A9:A7:54:14:85:31:A4:18:B4:01:FA:3E:5B:AE</certificateThumbprint>
      <isPrimary>false</isPrimary>
   </nsxManagerInfo>
</nsxManagerInfos>

Extract the UUID value from the ouput and use is to construct the DELETE curl request:

curl -X DELETE https://<<PRIMARY VMWARE NSX MANAGER>>/api/2.0/universalsync/configuration/nsxmanagers/a6fd5275-738e-4d86-9c0b-d463157bcd9f?force=true -k –u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml"
curl -X DELETE https://t05nsx01/api/2.0/universalsync/configuration/nsxmanagers/a6fd5275-738e-4d86-9c0b-d463157bcd9f?force=true -k -u admin:SuperSecurePassword -H "Content-Type: application/xml"

Now we can add the Secondary VMware NSX manager again. Create following XML data with the correct values for your environment and invoke a POST request:

<?xml version="1.0" encoding="UTF-8"?>
<nsxManagerInfo>
   <nsxManagerIp><<SECONDARY VMWARE NSX MANAGER>></nsxManagerIp>
   <nsxManagerUsername><<USERNAME>></nsxManagerUsername>
   <nsxManagerPassword><<PASSWORD>></nsxManagerPassword>
   <certificateThumbprint><<SHA1HASH>></certificateThumbprint>
   <isPrimary>false</isPrimary>
</nsxManagerInfo>
<?xml version="1.0" encoding="UTF-8"?>
<nsxManagerInfo>
   <nsxManagerIp>10.10.5.65</nsxManagerIp>
   <nsxManagerUsername>admin</nsxManagerUsername>
   <nsxManagerPassword>SuperSecurePassword</nsxManagerPassword>
   <certificateThumbprint>ab:45:b0:8f:ef:32:a9:a7:54:14:85:31:a4:18:b4:01:fa:3e:5b:ae</certificateThumbprint>
   <isPrimary>false</isPrimary>
</nsxManagerInfo>
curl -X POST https://<<PRIMARY VMWARE NSX MANAGER>>/api/2.0/universalsync/configuration/nsxmanagers?force=true -k -u <<USERNAME>>:<<PASSWORD>> -H "Content-Type: application/xml -d '<<DATA>>'
curl -X POST https://t05nsx01/api/2.0/universalsync/configuration/nsxmanagers?force=true -k -u admin:SuperSecurePassword -H "Content-Type: application/xml" -d '<?xml version="1.0" encoding="UTF-8"?><nsxManagerInfo><nsxManagerIp>10.10.5.65</nsxManagerIp><nsxManagerUsername>admin</nsxManagerUsername><nsxManagerPassword> SuperSecurePassword</nsxManagerPassword><certificateThumbprint>ab:45:b0:8f:ef:32:a9:a7:54:14:85:31:a4:18:b4:01:fa:3e:5b:ae</certificateThumbprint><isPrimary>false</isPrimary></nsxManagerInfo>'

This was the last step in our procedure. We now again have a Primary VMware NSX manager linked to a Secondary VMware NSX manager.

Conclusion

Via the VMware NSX REST API it is possible to automatically change the roles of the manager instances. This can be convenient when automating disaster recovery scenarios by using VMware vRealize Orchestrator or other automation tools.

I hope this was helpful and feel free to give feedback if you have remarks concerning the procedure

yannickstruyf