Introduction
Over time, vSEC:CMS database will grow and depending on the size of your deployment, the data can grow to a level where it impacts the overall performance of the system. This article will provide some tips on how you can determine where the bottlenecks are and how you can make adjustments to improve overall system performance.
Before going deeper into what tools are available, we should define what we mean by poor performance. Poor performance is observable long timings when performing common use cases such as credential issuance, credential PIN resets or credential certificate reissuances. It can also be around extremely long times for an operator to log onto the Admin or Agent applications.
System Health Checks
There are several checks that can be performed to determine possible reasons why your system is performing poorly. This section will describe what checks can be performed and what can be done to remedy them.
It is recommended that for any of the task described below, where an operator needs to log onto the Admin console, that they do so with an operator that has System Administrator role.
System Status Check
From version 6.10 you can use a REST API function to determine the current status of the system to ensure that the system is available and operational. You will need to have the REST API configured in order to use this (see the article vSEC:CMS REST API for details on this).
From the Swagger UI look in the Management API section for examples of how you can use this function.
Client Side Polling
It may be configured in your environment that vSEC:CMS User applications are polling on the server for updates. This is not recommended as this will put excessive traffic and load on the server which leads to poor overall performance.
To check this RDP to the server where vSEC:CMS is installed and log onto the Admin console. Go to Help - Diagnostic and check what is set for Wait4Events. It should be set to off (1) like below. If this is not set then follow the instructions in the Resolve section.
Resolve
To resolve the above issue, you need to RDP to the server where vSEC:CMS is installed and perform the following steps:
- Close vSEC:CMS Admin console if it is open
- Open Regedit and add registry key DWORD named startup.tasks here [HKEY_CURRENT_USER\Software\Versatile Security\vSEC_CMS_T] with a HEX value of 10
- Open vSEC:CMS Admin console and go to Help - Diagnostic and verify that you see Wait4Events is set to off
- Lastly, if clients are configured to be running in system tray mode then they need to be restarted.
Check Database Size
Over time the database size will grow and sometimes this can lead to performance issues and some tables in the database may need to be resized to improve overall performance.
To check this RDP to the server where vSEC:CMS is installed and log onto the Admin console. Go to Help - Diagnostic and check what is set in the Database section.
Additionally, you can check the number of records and size of these tables which can be useful in determining what tables may be oversized and need reducing.
To check this RDP to the server where vSEC:CMS is installed and log onto the Admin console. Hold down the Ctrl key and go to Help - Diagnostic. Select the Copy button and select the Including Minidriver(s) option. Save the contents to a file (should be html format).
Resolve
In order to try to resolve issues related to large database sizes we recommend that you open a Support ticket and provide the information above as part of the ticket. Then the Versasec support engineers will guide on next steps depending on the information provided.
Performance Monitoring Checks
It maybe useful to look at the performance data captured on the server-side in order to determine where potential bottlenecks exist. On the server where vSEC:CMS is installed you can navigate to the folder where these files are written to. The files will be stored in the AppData folder for the service account: C:\Users\<cms_service_account>\AppData\Local\Versasec\vSEC_CMS
If vSEC:CMS service is running under Windows local SYSTEM account then the file location will be C:\Windows\System32\config\systemprofile\AppData\Local\Versasec\vSEC_CMS
where <cms_service_account> is the dedicated Windows account that vSEC:CMS service runs under.
In there you will see several files starting with name perfmon_. These files can be sent to Versasec for analysis. In such cases we recommend that you open a Support ticket and provide the information above as part of the ticket. Then the Versasec support engineers will guide on next steps depending on the information provided.
perfmon_ capturing is enabled by default. If log analysis shows that the performance overhead is too high, you can disable the feature via the server’s registry settings. On the server where vSEC:CMS is installed the following values can be applied:
DWORD: perfmon.vs.booking.pmmode where values can be: 0 (zero) no perfmon_ data captured; 1 basic information is capture; 2 this is the default setting if no registry key is set; 3 extensive performance related data is captured.
Location: [HKEY_LOCAL_MACHINE\SOFTWARE\Versatile Security\vSEC_CMS_T\Service]
You will need to restart the client in system tray mode for this change to become effective.
Additionally, clients that are running vSEC:CMS User application will by default send performance related data to the server. This can bring considerable overhead in terms of traffic and load if you have many clients running in system tray mode. It is recommended to switch this off if you plan to have your clients running in system tray mode. You can disable this by setting a registry key on the clients:
DWORD: perfmon.vs.enabled value 0 (zero)
Location: [HKEY_LOCAL_MACHINE\SOFTWARE\Versatile Security\vSEC_CMS_T]
You will need to restart the client in system tray mode for this change to become effective.
Timing Checks
If you are experiencing long delays in opening the Admin, Agent or User client consoles it is possible to record where the time is being spent when opening them.
On a client where this is encountered open a command prompt as administrator. Navigate to the location where the client component is installed.
For Admin or Agent, if the application was installed into the default location, change directory to:
C:\Program Files\Versasec\vSEC_CMS S-Series
For User, if the application was installed into the default location, change directory to:
C:\Program Files\Versasec\vSEC_CMS Self-service
Then for Admin application run:
vSEC_CMS_T.exe -apptiming:15
Log onto the console as normal. You should see a dialog similar to below.
Select the Duration link which should open a dialog like below.
Copy this information and we recommend that you open a Support ticket and provide the information above as part of the ticket. Then the Versasec support engineers will guide on next steps depending on the information provided.
For Agent application run:
vSEC_CMS_T_LITE.exe -apptiming:15
Log onto the console as normal. You should see a dialog similar to below.
Select the Duration link which should open a dialog like below.
Copy this information and we recommend that you open a Support ticket and provide the information above as part of the ticket. Then the Versasec support engineers will guide on next steps depending on the information provided.
And for User application run:
vSEC_CMS_T_USS.exe -apptiming:15
When the console finishes opening up you will see dialog like below.
Select the Duration link which should open a dialog similar to below.
Copy this information and we recommend that you open a Support ticket and provide the information above as part of the ticket. Then the Versasec support engineers will guide on next steps depending on the information provided.
Flush Cache
Over time it maybe required to flush the cache in vSEC:CMS to clear up inconsistent behaviour or corrupted caches. This section will describe how you can do this.
Server-Side Cache
On the server where vSEC:CMS is installed open Windows Services console and make a note of the account that vSEC:CMS Service is running under.
Stop ALL the vSEC:CMS services.
Open File Explorer and:
- If vSEC:CMS service is running under a dedicated Windows account navigate to 'C:\Users\<cms-service-account-name>\AppData\Local\Versasec\vSEC_CMS' and delete ALL files from this folder.
- If vSEC:CMS service is running under Windows local SYSTEM account navigate to 'C:\Windows\System32\config\systemprofile\AppData\Local\Versasec\vSEC_CMS' and delete ALL files from this folder.
Restart the vSEC:CMS services and see if that resolves any issues you may have encountered.
Client-Side Cache
On clients where you run vSEC:CMS applications (Agent or Admin console, or User application) close any open application.
Open File Explorer and navigate to 'C:\Users\<logged-on-user-account>\AppData\Local\Versasec\vSEC_CMS' and delete ALL files from this folder.
Open application and see if that resolves any issues you may have encountered.