VQ Conference Manager commissioning checklist

Introduction

This chapter provides a check list of things to check when deploying (or commissioning) a new VQ Conference Manager (VQCM) instance.

As a general rule, once the system has been configured and certain things have been checked to be working correctly, the system will work as expected and consistently.

The document does not provide an extensive list of complicated tests; we do those here at VQ and only release a new version once the software has passed an extensive set of automated and manual tests that reflect the workload of our largest customers for extended levels of operation. Approximately 500 automated tests validate the user interface; another 500 test the core system functionality and two load tests check that the package as a whole will run production workloads for approximately 1 year. One test repeatedly adds 2000 participants into 50 Spaces and does this for 5000 cycles (giving a total of 10 million call participants). The test runs for nearly 5 days (24/7). The second test runs with 6000 participants over 5000 cycles (30 million call participants) and runs for 15 days (24/7).

Operational Basics

Check that the customers operational team know the importance of backing up both VQ Conference Manager and the CMS nodes on a regular basis and they also know how to do it (and practice a restore or both VQ Conference Manager and the CMS nodes).

CMS database backup and restore
VQCM/AM database back-up and restore

When performing a backup, please ensure that both VQCM and CMS are backed up at the same time. A considerable amount of state information is shared between the two systems; backing up at different times can result in equally substantial differences between the two backup data sets rendering them, essentially, useless.

VQ Customer Support

Support Contact Details:

Email: support@vqcomms.com

To raise a support ticket email support@vqcomms.com. This will generate a support ticket with a reference number that you can track and respond to.

Cluster Connect Script

If you are provisioning a CMS cluster against VQ, please contact support@vqcomms.com and ask for the “CMS Cluster Connect script”. This script connects to CMS, gets all the CMS nodes and then provisions each CMS node within VQ. It helps avoid problems with incorrect CMS node names that can affect how VQCM processes state change events from CMS leading to incomplete data being presented in the VQCM Activity Call Monitoring page.

Cluster Test Script

We’ve produced a bash-script called “clustertest” that allows commissioning engineers to check that each CMS node is correctly configured and place calls on a node-by-node basis to check that VQCM correctly sees state changes (confirms the firewall is OK) and that mute/disconnect commands can correctly be performed.

Please run the tool (contact support@vqcomms.com), follow its instructions and confirm that Activity sees state changes from each node and that mutes/disconnects work correctly. Longer term, the tool will become integrated into the Conference Manager Administration interface.

This is the only test you need to make to ensure VQCM is correctly functioning with CMS. Once this test has been performed, your testing should be to confirm that the functionality you require is configured on your Space Templates and Participant Roles.

Multi node operation

Multi-node operation at VQCM 3.x is not supported.

Multi-node operation and High Availability (“HA”) will be introduced after the initial release of VQCM 4.x.

Adding a new Call Bridge to VQCM

When adding a new Call Bridge to VQCM, take note of the “Cluster Priority” setting. This tells VQCM which is the “best” CMS node to send commands to. “Better” nodes have lower numbers; the best has the lowest number.

Cluster Priority is used to ensure that VQCM sends API commands to the nearest, or best, available, online, CMS.

As an example, if your VQCM is hosted in New York and there are CMS nodes in New York and Tokyo, the round-trip delay sending an API command to Tokyo is substantially longer than send it to New York.

The throughput of API commands to the CMS node[s] is directly proportional to how quickly things happen on the user interface (for example, muting a participant).

Make sure, therefore, that your MCUs are configured within VQCM to have the “nearest” (or lowest network latency) MCUs set to the lowest Cluster Priority.

CMS things to check

There are several key things that need to be working correctly.

VQCM needs to receive the state change events from each CMS. These are events like “Call Started” or “Participant joined call”. If these events are not received by VQCM, no calls will appear in the Activity page and no historical data will be produced.

It is therefore critical that VQCM and CMS are talking to one another correctly; VQCM sending commands to CMS and CMS sends state changes back to VQCM via the CDR POST mechanism.

VQCM 3.x configures the CMS CDR settings with the objective of reducing the likelihood of an error being introduced (incorrect VQCM address or Port or forgetting it completely). It still leaves the possibility, however, that a firewall is blocking the HTTPS POST messages from CMS to VQCM.

Please check that no firewalls are blocking traffic between your CMS node[s] and CMS on port 443 (HTTPS).

When VQCM configures the CMS CDR settings, it writes a FQDN value. If there is no DNS server configured on the CMS node, the CMS server will not be able to resolve the FQDN address.

Please check your CMS has a DNS configured.

In a clustered CMS deployment, the node names in VQCM need to match exactly (case sensitive) the node names as configured in CMS [Note: we plan to auto-detect the cluster node names in a future release and remove this potential for a mis-configuration].

Symptoms of an incorrect node name include mutes or disconnects not working (for example, a user attempts to disconnect a participant via VQCM, the participant goes to a disconnecting state, nothing happens at the device end and seconds later, VQCM shows the participant as incall again).

When VQCM connects to a CMS node, it performs a long list of actions to discover the state of the node; one of the actions is to check that the CMS node has the same time as VQCM. If the clocks are out by more than 60 seconds, VQCM will reject the connect attempt and display the MCU in an offline state with an error state of “CallBridgeClockOutofSyncWithAM”. This may feel painful but it highlights any clock synchronization differences and prevents subtle and more difficult to track down issues caused by state change events from CMS being discarded because of NTP time differences between VQCM and CMS.

If you attempt to connect to a CMS node and see the “CallBridgeClockOutofSyncWithAM”, please check that both VQCM and the ESXi VM Host that hosts CMS are configured to use the same NTP server.

Please also note that we have experienced cases where CMS showed the correct time on its MMP interface but the time stamps included in state change events (CDR events) was different; VQCM checks against the time stamp in CDR events. Rebooting CMS normally fixes the problem.

The cdrTime value used by CMS for CDRS can be seen from the following URI on your CMS cluster:

http://<Your CMS URI>/api/v1/system/status

Result:

<clusterEnabled>false</clusterEnabled>

</status>

Time Zones and locales

Make sure the Time Zones are configured to your time zones. All times in VQCM are displayed in ‘User’ time (the exception being Call Data Records using Analytics 1 in AM 2.x and VQCM 3.x. CDR times can be displayed in local time using Analytics 2 in VQCM 3.x).

Default System Time Zone (System->Settings)
Tenant Time Zones(Tenant->Settings)
LDAP config time zones (LDAP Config->Settings)

Locales – en-US and en-GB

Locales control how dates and times are displayed. Currently, we support 2 options:

En-US: Dates MM/DD and times AM/PM
En-GB: Dates DD/MM and times 24 hour

Which can be specified:

System->Settings
Tenant->Settings
User->Settings
LDAP Config->Settings

License

Check it expires when you think it will (System->License).

As license expiry approaches, warnings and a license count down will display in the user interface. Once the license has expired, key functionality will no longer be available although the software will continue to run.

User Experience Templates

Does each user type enable the level of functionality required?
Does each user type see the Space types expected?
Are Super-User users required (Admin/Operators) and have they been configured?

Space Templates

Have the default Space Templates been copied/recreated so that if changes are required to Roles/Space Templates in the future, this can be done (Space Templates shipped with the product are read-only) quickly and at low risk without requiring a copy/recreate update on a production service?

Have deployment preferences for Call level settings such as Call Size (number of participants, Recording, Streaming) been set? If the deployment is using the Cisco Meeting App (CMA), has the Invitee Role been set so that the CMA client displays the Participant Invite details (including Passcode/Pins) rather than the Chairpersons?

Have deployment preferences for Role settings such as single, dual channel presentation mode, video call quality, amount of time Participants will remain in call after the Chair-Person leaves been set?

Check your Role properties for TIP. TIP, TelePresence Interoperability Protocol, is the method used by immersive, three-screen TelePresence systems such as the Cisco IX5000 to encode three separate video streams (one from each camera) into a single H.264 video channel. Cisco invented this protocol and made it available royalty-free in order to promote immersive telepresence interoperability between different vendors. Support for this protocol is built in to CMS so that it can host multi-party conferences where IX5000s might be participants. But it is optional. We have observed issues with content share and with participant join when this setting is disabled therefore we recommend enabling it unless there is a good reason not to.

Understand the customer’s requirement for compliance; do they have a compliance team and will compliance sign-off be required before the service can be deployed. The big question is passcode and Pins protection for call calls. Assume you will need them.

Check that the CMS Global (Top-level) and Tenant Call and Call Leg Profiles are consistent with the Role definitions in the VQCM Space Templates being used. We have seen instances where customers have been using a CMS, upgrade to use VQCM and find that some functionality behaves differently – specific examples being TIP, BFCP and SIP presentation mode.

Please contact support@vqcomms.com for guidance on how to do this. We also now have a tool “CMS Migration tool” that can be used to migrate settings from CMS into VQ Space Templates.

An outline of the process steps is:

API Get /api/v1/tenants
API Get /api/v1/tenant/<tenant guid>
API Get /api/v1/callProfiles/<call profile guid>
API Get /api/vq/calllegProfile/<call leg profile guiod>

System:

API GET /api/vq/system/profiles
As 3-4 above

Check the results of the Call and Call Profile Gets with the definitions in the Roles for each SpaceTemplate being used.

Please contact support@vqcomms.com for guidance on accessing and using this tool.

Dial Plan

By default, VQCM will create Call Identifiers prefixed by “700”. If a different prefix is required, this can be changed (see “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com); each new Call Id is the previous value +1 (so, default values would be 7001, 7002, 7003 etc.).

In many situations, more control over URIs and Call Ids is required; for that reason, generators have been added to the LDAP Configuration and Space Template definition pages.

The LDAP Configuration page supports the Auto-Increment (“AI”) keyword and the Space Template page supports both Auto-Increment and Random (“RND”) keywords.

When Call Id’s and URIs need to be generated (e.g., they are not driven by AD/LDAP Attributes), the “Auto-Increment (‘%AI(xx)%’) ” and “Random” (‘%RND(xx)%’) keywords can be used along with prefixes/postfixes) to create Call Ids and URIs (of given length; xx) with values that conform to the dial-plan requirements of the deployment.

When defining the length component for generators (AI and RND), please ensure you specify a length value (the generated values are leading zero filled) that will give you sufficient values – for example, length 4 would give a number range of 0…999 (one thousand values). This might sound a lot but isn’t; you should be considering 6 or 7 digits. Please ensure the number ranges generated work with your dial-plan.

Please also note that the generator functions do not currently ‘wrap-around’ and re-use unused numbers. Prefix chains can be defined so the values generated can go thru a list of defined ranges. For example, %66,67AI(6)% would generate values 66000000 thru 66999999 and 67000000 thru 67999999.

Once the generator value has wrapped around over the available set of values, it’ll continue incrementing and therefore (in this example) start generating 7-digit values which will be appended to the last prefix (in this case 67) giving a 9-digit result rather than a 8-digit result.

For more details on Generators, please see document “Generator keywords for URI and URI values”; available for download from our Customer Portal - vqcomms.com/login.

International Gateways

If your deployment is on a larger scale, you might have participants dialing into calls using different SIP trunks.

Please make sure you make tests calls through each SIP trunk connected to your CMS. We had one instance, where, for example, inbound calls (calls to CMS) in most cases worked but calls from a particular region connected as far as the CMS IVR but weren’t able to join a call. The root cause of the problem turned out to be that all the failing calls were routed through the same SIP trunk and the SIP server on the failing SIP trunk was an older version of Asterisk. The Asterisk SIP server didn’t support Video or BFCP in the SIP SDP headers and ignored the RE-INVITES containing them issued by CMS as part of the IVR exchange. In this example, there were two solution options:

Update the Asterisk SIP server to a later version that supported Audio, Video and Content in the RE-INVITE SDP headers
Update the CMS version from 2.2.28 to CMS 2.3; CMS 2.2.28 always issued a RE-INVITE with all media types; CMS 2.3 only issued the RE-INVITE with the media headers already selected (in this case, Audio only)

LDAP/AD Configuration

Check users are being imported and have the right UX Profile, Space Template type, Time Zone and Locale.

The LDAP Configuration page contains a couple of easy to miss settings. Check you have the one you need:

Sync with Call Bridge

Normally, this option is checked and instructs VQCM to share LDAP/AD configuration settings with CMS, allows the VQCM LDAP Importer to run and, as part of that, instruct CMS to Synchronize with LDAP/AD.

Note that if the “Use PMP Licenses” is set, VQCM will set the “PMP license” on each provisioned user by the LDAP Import (the request will be rejected by CMS if insufficient PMP licenses have been purchased from Cisco).

When unchecked, VQCM shares no LDAP/AD configuration settings with CMS and, importantly, never issues a LDAP Synchronize command to CMS.

Uncheck this option if you are not using the CMA client from Cisco, do not require PMP licensing and want to use your CMS cluster as “a big MCU” with none of the valued added features offered by CMA.

Create Endpoint for User

VQCM’s default is to create a CMA compatible Endpoint (device) URI for each user imported from LDAP/AD. This populates CM’s address books and makes it easy to add members to Spaces or scheduled calls. In some cases, you might want to associate endpoints (devices) with users that don’t use the same address convention as used by CMA (the device URI is the same as the user login name): $sAMAccountName$@yourdomain.com

Create Spaces on LDAP import

Normally this option is checked; VQCM will create the Spaces using VQCM’s Space Templates mechanisms and gives complete control of how calls perform without getting anywhere near the CMS APIs.

If this option is unchecked and Space mapping options have been defined (an attribute mapping has been defined for Space name and URI), CMS will create a Space for each user using default settings configured on CMS.

Do you need the LDAP Importer to run on a regular basis?

Please read the Automated LDAP Import document available from the VQ Customer Portal. You can also contact support@vqcomms.com and we’ll guide you thru the process.

Check the Operations team know how to:

Run the Importer
Run a destructive Import (to mass delete users)
Look at the Importer logs to check that imports have run as expected

Updating Space Templates after Users have been imported and Spaces created

The API object model of each Space on CMS is complex; the following additional objects are associated: CallProfile, CallLegProfile[s] and AccessMethods.

When VQCM creates a new Space, all the complexity of these objects is taken care of and if the Space is updated, any changes are also made. If the Space is deleted, we delete all the associated objects.

The thing to be aware of is that the values used to create things like the CallProfile and CallLegProfile are obtained from the Space Template and Roles. VQCM uses the values as defined at the point in time when the Space is created.

So, if you want to then change a Space Template value or Role definition, the new values must be applied to all Spaces that exist on CMS that use the updated Space Template.

A utility called “TemplateManager” is available for this task. TemplateManager and documentation can be downloaded from our Customer Portal - vqcomms.com/login.

Provisioning Mails to new users

One of the key enablers for users using a service is to know how to. What are the details for service access and what, for example, are the details (address and passcodes) for their Space?

Two tools are provided to facilitate this; ImportEmailer and BulkEmailer. ImportEmailer works with the LDAP Importer and mails each new user created with the details for their account. BulkEmailer can be used to send emails to, for example, all the users on a Tenant.

Also note that for customers using the iOS phone application, the email provisioning tools also include options on how to automatically configure each users iOS application via their provisioning email.

Please see the BulkEmailer and ImporterEmailer documentation for details.

The Message Templates sent can be customized to meet your needs. The tools and documentation can be downloaded from our Customer Portal - vqcomms.com/login.

Importing Endpoint (Room system) details

The EndpointImporter utility is now available for VQCM 3; it adds, updates and deletes devices based on data in a comma separated file.

The tool and documentation can be downloaded from our Customer Portal - vqcomms.com/login.

Message Templates

Manually edit templates to change “myDomain.com” to your domain.

Replace hard coded example telephone numbers with those for your network. For example, the “Space Details” family of example templates include the telephone dial-in number 1-805-419-2627. Please replace this number with yours.

In most deployments, the Templates that will need updating are the “URIReservationDetailsIcons.html” and “URIReservationDetailsWithoutIcons.html” templates. Two variants exist, one with icons and one without. Chose the one most appropriate in your environment.

If your system is using Scheduled calls, the CallBooked.html and CallUpdated.html examples might also be of interest.

The Templates can be updated (for example, adding telephone dial in numbers). We caution against changing the style too radically; Outlook’s support of HTML is limited and the styles and examples provided have been tested against all Outlook variants and work (as in, render cleanly).

Please also note that if you add Icons (graphics) to the Templates, they must be accessible from a publicly accessible web site to avoid problems with subsets of users not being able to see them.

From VQCM 3.5, the scripting language models were simplified for Spaces based on single Roles. From 3.5 Call Id, Pin, Uri and Secret are available at the root level of the McuUriReservation object and there is no need to loop thru the Roles.

For example, {{#is UriReservation.Pin '' }}{{else}}and then PIN:{{UriReservation.Pin}}#{{/is}}.

This example checks whether the Pin is set to blank (‘’) and if it is not, displays the text ‘and then PIN:6666#’.

From VQCM 3.6, we have started making example Message Templates available on the vqcomms.com portal. From the Downloads tab, expand Message Templates.

The first example is SpaceDetailsWithIconsAndSecretSingleRole.

This example shows how to add a secret for a Space and therefore allow easy access from WebRTC for a single Role Space.

It also shows how to add an iPhone ‘click to call’ link and how to specificy a ‘click to call’ link for WebEx Teams clients.

Please replace mydomain.com with the FQDN of your CMS sysrem.

Please replace the telephone number 1-805-419-2627 with that appropriate one for your system (or delete)

System Configuration Considerations

Self-Service mode

This mode describes the usage model where users dial into their Spaces and VQCM is working in a reactive mode; as the Participants dial into the Space, a call on the Space is dynamically created by CMS and VQCM detects the new Call coming into existence via events posted by CMS to VQCM.

This is the mode that allows the greatest call volumes; customers have systems deployed hosting over a million call participants per month (1.1 and 1.3 million; 50k/70k call participants per day respectively).

On systems hosting large numbers of calls per month in Self-Service mode, we advise changing the ‘CallAgeing’ setting in the System Configuration database (see “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login) to a value of 1 (one) from its default of 40.

The CallAgeing setting controls when Calls are deleted once the call has completed; setting the value to one results in completed calls being deleted 24 hours after completion rather than 40 days later. On large systems, substantial numbers of Calls are generated which results in two things:

Over 40 days, the system becomes slower and less responsive because it can have tens of thousands of Calls that it is repeatedly needing to scan and loop through
If the system restarts, it can take the system minutes to initialize on account of the large number of calls. This can result in 504 Errors being displayed on the User Interface because the Core VQ service is unresponsive (it’s still initializing)

Schedule Call mode

This mode describes the usage model where Meetings are scheduled via the “Meetings List” or the “Schedule New Meeting” coApp.

Systems in this usage mode will typically be limited to less than 250 scheduled calls per day or 2500 per month.

In Scheduled Call mode, you might want to consider enabling “Call Ending“ messaging where VQCM will display "The call is about to end” messages into the video stream of each video participant when the call reaches 10 minutes before its scheduled completion time. The Message repeats at 5 minutes and then every minute to call end time. The message content and intervals can be configured. Please see the “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login.

Default Call Times for non-scheduled calls starting on Spaces

Changing the maximum call length of calls initiated on Spaces (inbound first participant)

When the first Participant joins a call (dials into a Space) on an idle Space, a new call is instantiated on CMS and within VQ. The default length of the new call created within VQCM is 12 hours. For most customers, this maximum limit works but for some it does not. When the call reaches its duration limit, VQCM shuts the call down.

The maximum call length can be increased via the InboundAdHocCallLength setting. The setting defines the maximum call length (in hours) that will be used.

NOTE: the InboundAdHocCallLength settng is in hours as opposed to the DefaultDurationMinutes setting described below (Section 2.9) which is in minutes.

Please see the “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login.

Changing the maximum call length of calls initiated on Spaces (outbound first participant)

If the Call coApp is used to make an outbound call to a Participant from an Idle Space, a new call is instantiated on CMS and within VQ. The default length of the new call created within VQCM is 120 minutes (two hours). For most customers, this maximum limit works but for some it does not. When the call reaches its duration limit, VQCM shuts the call down.

NOTE: the DefaultDurationMinutes setting is in minutes as opposed to the InboundAdHocCallLength setting described above (Section 2.8) which is in hours.

Please see the “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login.

Adding audio prompt for Reactive Calls/Blast Dial

As a default, Reactive Calls/Blast Dial places an outbound call to the Members and Endpoints associated with the Space.

You can configure and audio confirmation prompt. The dialled participant will initially hear a ‘welcome’ message and then be asked to press 1 to join the call.

Please see the “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login.

System Wide Video Messaging

On some systems, it is a requirement that all Participants know the confidentiality level of the service. For example, Secret or Restricted.

Please see the “VQ Conference Manager Configuration Options” document available from our Customer Portal - vqcomms.com/login.

Your Configuration Settings

Please save the password settings for your system in a secure location. We get a steady string of support tickets from customers asking us to recover lost passwords. VQCM stores passwords securely and there is no way for us to recover forgotten passwords.

Please also remember to back up your VQCM system. The easiest and quickest way is by performing a VM Snapshot.

Always take a backup of your VQCM and CMS systems before performing upgrades.