EzCloudInfo

This article outlines the unplanned failover and fail back for Skype for business. However the DR setup must be provisioned in order for the DR activation to happen.

The deployment must have the below setup :

1)Skype for Business HQ Front end, HQ SQL ,HQ OOS will be part of the HQ active directory site.

2) HQ site will have its dedicated Edge server.

3) Skype for Business DR Front end, DR SQL and DR OOS will be part of the DR active directory site.

4) DR site will have its dedicated Edge server.

5) DR front end, edge servers will be in the same Skype for Business Site since the site is a standby site.

6) Synchronous data Replication will be enabled between the HQ FE pool and the DR FE pool .

7) DR sql store information must be published in the topology builder.

8) Associated backup pool must be specified as DR Skype for business FE pool in the topology builder.DR file stores must be published in the topology builder.

9) HQ and DR site edge servers DNS name spaces can be load balanced. DR site must be made unavailable during normal scenario and connections  to DR edge must be allowed only during DR scenario.

10) Required communication from HQ to DR FE,SQL should be present for the Pool replication to happen.

Example of DR setup with main site:

Procedure to activate unplanned DR failover:

In case of unplanned failover its a total disaster where the main site will be completely unavailable.

So the  CMS (Central Management Store) ,HQ fe pool and HQ edge services will not be accessible during this scenario.

Below steps can be used:

1) Configure in the DNS load balancer and make sure the edge server DNS name spaces are ready to accept connections in the DR site edge server. There are multiple ways to achieve this based on the network setup. As a last resort also we can add simply 2 entries (hq & dr) on the DNS name spaces and stop the DR edge services. We can activate the DR edge services only during the DR scenarios.

2) Activate the CMS

We can try to run the below command to see the CMS status

Invoke-CsManagementServerFailover -Whatif

This command will throw an error because this CMS is not available since it was present in the main site and main site is totally in accessible.

In a normal state when the main site is available in a planned failover the result of the command will be the below

It will let us know the current state of the CMS and the proposed state of the CMS after the failover.

3) In this scenario the CMS needs to be activated forcefully by the below command

Invoke-CsManagementServerFailover -BackupSqlServerFqdn “DRSQLFQDN” –BackupSqlInstaceName “BACKUPDRSQLINSTANCE” –Force:$true

4) Wait for the replication status to be completed:

We can check the replication status by below command

Get-CsManagementStoreReplicationStatus | ft

5) Reconfigure Edge Federation Route  via DR edge and publish topology and run the setup on all edge servers.

Enable the federation on DR edge and modify the federation route via DR edge.

6) Failover the Pool using disaster mode switch.

Invoke-CsPoolFailOver -PoolFqdn “poolfqdn” -Force -DisasterMode

Failback to HQ site:

Once after the main site is back  make sure the  DNS name spaces are available in the main site

1)  Failover the CMS

Invoke-CsManagementServerFailover

Wait for the CMS replication to complete in the main site.

2) Failback the FE pool to the main site.

Invoke-CsPoolFailBack -PoolFqdn “poolfqdn”

3) Reconfigure Edge Federation Route and publish topology and run the setup on all edge servers.

Note: The DNS routing and the VOIP component SIP/PSTN integration will vary in each and every deployment .The DR setup and failover needs to be taken into consideration  according to these configuration.

Thanks & Regards
Sathish Veerapandian

This might occur when multiple meetings with high number of participants is initiated the Skype for Business pool is unable to process this request.

We can also see the following logs recorded in the FE servers also:

Log Name: SFB Server
Source: LS ApplicationSharing Conferencing Server
Event ID: 32032
Task Category: (1304)
Level: Warning
Description:
Internal Application Sharing Server health monitoring has detected that Application Sharing Server is running at a moderate capacity.

Application Sharing Server Health State: Loaded
Cause: Application Sharing Server has reach moderate capacity.

This application sharing parameter is controlled by MaxBandwidthPerAppSharingServiceMb

What is MaxBandwidthPerAppSharingServiceMb parameter used for ?

This value Indicates the maximum amount of bandwidth (in megabytes) set aside(allocated) for the Application Sharing Conferencing service.
MaxBandwidthPerAppSharingServiceMb can be set to any integer value between 50 and 100000, inclusive. The default value is 375 megabytes.

This setting is applied and restricted to share the bandwidth within the pool at the org level and not in the server level.
Changing this value cannot be scoped only on specific servers.

Solution for this problem:

Run Get-CsConferencingConfiguration
Look for the value MaxBandwidthPerAppSharingServiceMb.

The default value is 375 MB . Increase them to some extra value and see the results.

The value can be increased by running the below command:

Set-CsConferencingConfiguration -Identity conferencingconfigurationname -MaxBandwidthPerAppSharingServiceMb “MBSizeValue”

But the strange thing is really hard to identify this actual consumed value from the default 375MB which should be sufficient for a conferencing type for the Application sharing like a powerpoint in the presentation using Remote Desktop Protocol (RDP) protocol.

Whenever a meeting is schedule in Skype for Business server will create an entry in the Conferencing database but it will not reserve any memory in prior for the meeting which is going to occur in future. It will use the built in load balancing logic which will dynamically/instantly allocate conferencing resources on the FE servers based on the load on all FE servers and number of meeting that is occurring at the moment.

So here the value is exceeding because the pool is already running at its peak threshold level of its allocated capacity.May be if more meeting is initiated with more than 300 participants the Pool might run out of resource and this issue might occur.

The Conferencing CDR tables has the below records:

Conferences table in Skype for Business Server 2015:
Stores information about all conferences that were archived or whose details were recorded, including ConferenceURI, and start and end time.
ConferenceSessionDetails table in Skype for Business Server 2015:
Stores information about every SIP-based conference session, including start and end time, user ID, response code, and diagnostic ID for each session.
FocusJoinsAndLeaves table in Skype for Business Server 2015:
Stores information about conference joins and leaves, including users’ role and client version.
McuJoinsAndLeaves table in Skype for Business Server 2015:
Stores information about the A/V Conferencing Servers that are involved in a conference and the user join and leave times.

If the  organization requires meetings with greater than 250 participants, we need to plan for a dedicated pool to support the load

We can use the capacity planning conferencing for meeting – https://technet.microsoft.com/EN-US/library/mt297718.aspx

Thanks & Regards
Sathish Veerapandian

All Skype for Business Clients from remote locations were unable to present the screen sharing through meet now ,peer to peer and conference.
This a new deployment and users were unable to present desktop.

Below were the test scenarios:

1st test – from remote users n/w to my home n/w – received error (we couldn’t connect to the presentation because of n/w issues. Please try again later)
2nd test – from remote users n/w to my office n/w – received error (we couldn’t connect to the presentation because of n/w issues. Please try again later)

Below troubleshooting were done :

1)Did a telnet to lyncdiscover.domain.com on port 80 and 443 – ( This was done just to make sure the clients when logging in gets all the updated info of the pool,SFB config etc..,)
2)Did a telnet to meet.domain.com on port  443 – successful
3)Did a telnet to join.domain.com on port  443 – successful
4)Did a telnet to av.domain.com on 443 successful

Assume the below scenario deployment:
1)The edges were in DNSLB and were in scaled consolidated topology using NAT.
2)UDP 3478 for AV service external IP.
3)TCP 443 for external IP’s.
4)Port 50k was blocked in my case since no legacy OCS clients.
5)No edge hair pin traffic is allowed for Audio and Video Public Ips.

Did a Snooper trace from the affected remote client and got the following info on the snooper logs

Getting  error as call failed due to media connectivity failure when both the end points are remote.

Now this is the time for me to dig into the analysis of in which protocol fashion the SFB clients establishes the connection.So started to explore on STUN,TURN & ICE since ever i was having a glossy look on these topics.

So what kind of protocols they use:

SFB/Lync uses all these 3 protocols to establish a media connectivity:

ICE:
The stands for Interactive Connectivity Establishment protocol for communications. All Lync/SFB clients are ICE clients and use ICE to try and establish connectivity between itself and another ICE client.Remember this is the main protocol which functions as the core and wraps the other 2 to establish a path.

STUN:
The new name for this acronym is Session Traversal Utilities for NAT.
This will allow the SFB client to discover the available public IP for the SFB media path inorder to establish the connectivity.

TURN:
Traversal Using Relay around NAT.
This will establish a chain of connection between the external client and the client inside the network.By using this edge servers will create a chain and will offer ports on UDP and TCP for the media path. Once this chain is established it promises the remote client to send its media connection to the internal network client.

So now we can understand clearly that the External Corporate firewall requires a Hairpin traffic to be allowed for the A/V edge Public Ips for the STUN and TURN to work in the required  UDP  TCP path.

Since these are the most commonly used RFC standard protocols SFB clients also uses them. These all are IETF standards protocols and hence Microsoft also uses them.
Now the SFB clients will use the below process to establish a media connectivity with the remote client:

Candidate Discovery:
Where the clients discover their available public IP addresses for media connectivity. These include both STUN and TURN addresses of the Edge server.

Candidate Exchange:
This is the place where both the SFB clients sends each other list of addresses on which they can be communicated for this media path.
Remember this will happen bidirectional.

Connectivity Checks:
This is where both the candidates(clients) try to establish a connection on all these addresses simultaneously (not one by one).
Finally the result would be the SFB client will pick any one of the available route and establish a connection with the client whoever is responding first.

Candidate Promotion:
This is the Final stage of the SFB client and happens after the call is established and its running.
Here the clients if identify any path which is more optimum and quick they decide to change that route which gives the better experience to the user.

These candidate information can be seen in snooper logs

We can see 3 types of candidate information

The first one below is for port 50k and can be ignored if you are not having this option

The second one is for audio and last one will be for video. We will have the same like one for audio with label main mentioned as audio.

Lets say if we have only port 50k opened and not 443 for UDP then we can see only those  50k candidate lists.

TCP-ACT indicates that with this candidate pair the client is able to send RTP and RTCP traffic

By having a look at it we can confirm that the candidate is a STUN pair. TCP-ACT and typ srfx raddr is the thing that indicates they are STUN pair.

In this case if the candidate discovery fails in all the cases we can find  BYE sip in the snooper logs and which mentions opaque=epid followed by guid

There are 2 solutions for this problem to work:

Allow Port 50k inbound:

We can  allow the media communications on this edge Audio/Video Ip only on port 50 K. But at real times when users connecting from different network for the media path they need to cross firewalls where they might have only the standard 80 & 443 allowed and these ports might be blocked.

Allow the hair pin edge traffic:

Allow the traffic on the edge server external firewall  to traverse the traffic between the two AV Edge servers public IP addresses. This will give the appropriate candidate lists for the clients connecting via different edge servers on UDP port 3478 through this hair pin traffic.

Note:

1)If we have only one edge server installed we do not need to follow this steps since all the clients will connect only to one edge server node and no issues will be identified. Just make sure the UDP 3478 is opened for this communication.

2)SFB  clients will always try to establish media path  via UDP as preffered if its available. If UDP isn’t available it tries to switch to TCP and establishes the connectivity.

Thanks & Regards
Sathish Veerapandian
MVP- Office Servers & Services.

In-order for the users to connect externally from the organization’s network we need to publish the Skype for business services.In this article we will have a look at best ways to publish the Skype for Business Edge servers over the internet.
By doing this the users can participate from external N\W in IM,AV ,web conferencing sessions.

There is lot of confusion in the architectural part of load balancing the Skype for Business Edge servers and cannot be taken as easy deployment. If the SFB deployment is extended to communicate with federated partners, remote connected users and Public Instant Messaging users then a real proper planning of the edge servers deployment needs to be carried over.

If we have 2 or more edge servers deployed in the DMZ they need to be load balanced to equally distribute the load in all the edge interfaces.
In general Microsoft recommends to use a DNS Load Balancer for Edge High Availability.

Load balancing distributes the traffic among the servers in a pool so that the services are provided without any delay.

Below are 3 types of load balancing solution that we can use based on our requirement:

DNS Load Balancer Using NAT :

This is the best recommended approach.
We are actually load balancing each edge services namespace over the internet with multiple A records NATTING them via firewall and then to Edge servers.
These Ip addresses are bound to each services seperately routed to internal individual Ip’s assigned to the external NIC.
Three private IP addresses are assigned to this network adapter, for example 131.107.155.10 for Access Edge service, 131.107.155.20 for Web Conferencing Edge service, 131.107.155.30 for A/V Edge service. These private Ip’s listen individual public IPs Natted from the f/w.
These Ips are not participated in the load balancer and used only for NATing.
They are basically behind a port forwarding firewall which is good.

Advantages of doing this:

1) We are assigning a separate public IP’s for each service and using standard ports. So the remote users will not have any issues on connecting behind their firewall since all are standard ports.
2) Its very good to troubleshoot in analyzing a particular service traffic statistics, Logging and easy to identify the issues with the logs packet capture etc..,

Disadvantages of doing this:

1) The edge services rely on multiple A records with the same name but different IP addresses. So its not service aware configuration and failure detection rate and routing to the available server is not possible.

But still i would go with this option considering the failure detection rate is very minimal in a well planned deployment and strong n/w considering very helpful and easy during any troubleshooting scenarios.

Below is the example of DNS load balancing using NAT

Lets assume i need to load balance 2 edge servers using DNS Load-balancing NAT as per below environment.

Below is the DNS configuration

DNS Load balancer using Public Ip Addresses:

By doing this we are using one public IP for all 3 services on each server and differentiate them by TCP/UDP port value.
We are directly assigning the public IP’s on the edge servers one of the 2 NIC’s which should be external NIC.
Three private IP addresses are assigned to this network adapter, for example 131.107.155.10 for Access Edge service, 131.107.155.20 for Web Conferencing Edge service, 131.107.155.30 for A/V Edge service.
The Access Edge service public IP address is primary in the NIC with default gateway set to the external Firewall.
Web Conferencing Edge service and A/V Edge service private IP addresses are additional IP addresses in the Advanced section of the properties of Internet Protocol Version 4 (TCP/IPv4)

Disadvantages of doing this:
It is not recommended, to use a single public IP address for all three Edge service interfaces.
Though this does save IP addresses, it requires different port numbers for each service.

Access Edge – 5061/TCP
Web Conferencing – 444/TCP
A/V Edge – 443/TCP

These might cause issues for remote users connecting externally from a n/w where their firewall doesn’t allow the traffic over TCP 5061 port.
Having three unique IP addresses will help us in easily doing a packet filtering to identify and resolve the issues.

Hardware load balancing using public Ip Address:

Load balancing is only need for old OCS clients and xmpp, but works fine if both edge server are up. From Lync 2010 Microsoft does not recommends to load balance the Edge services from internet.

We are creating a virtual Ip address for each services that edge serves (Access, WebConferencing, A/V) on the load balancer like F5, KEMP etc..,
Behind this Virtual Ip’s we need to add the edge servers associated for the services.
The main benefit of this is failure detection rate is very quicker since it detects the failure from the server side.

Disadvantages:

1) The A/V services will not see the client’s true IP ( for example in a peer to peer audio call for a user connected from external to internal)
2)Few challenges in configuring the outbound client connections going from the edge to internet (Routing & SNAT)

Thanks & Regards
Sathish Veerapandian
MVP – Office Servers & Services

From build 16.0.3331.1000 Skype for Business client 2016 there is an option to send IM to people who are offline. When the users sign in to the Desktop  client all the missed IM conversations will be notified.

We need to follow the below steps to enable this feature for all users.

Basically we require 2 parameters that needs to be enabled on the client policy in order for this feature to work.

EnableIMAutoArchiving

DisableSavingIM

By default this value will be set to null with no values as below.

The default option is set to null which means it will save the conversation history locally on the PC and mobile devices and not on the server side unless the option EnableServerConversationHistory value is set to True.

We have 3 options to set:

1) DisableSavingIM value to Null 

When set end users will have the option either to select or uncheck the option save IM conversations in my email conversation history folder.

2) DisableSavingIM  value to True 

When set end users will not have the option either to select or uncheck the option save IM conversations in my email conversation history folder. The option will be greyed out.

3) DisableSavingIM value to False 

Setting this value will not

In-order for this feature to work we need to set this value to True  since with the null option and false  it will not work.

post enabling this end users will get this notifications icon on the Skype for business 2016 client.

If the exchange server integration is enabled for archiving then all these archiving data will be stored on the associated user Exchange Mailbox.

The versions of exchange it supports to integrate the oauth setup is from Exchange 2013 and Exchange 2016 or Exchange Online.
If the version of exchange is 2010 then we do not have this option to store this archiving data on the Exchange.
In this scenario the data will be stored in the Archiving SQL server database.

Below sample Dashboard report shows about IM information contained in the archiving database for IM stored on Archiving SQL database.

If we have the server side enabled on the Archiving SQL DB its very important we need to look for two values

CachePurging Interval

The system will look for the participants who doesn’t have archive enabled and for them the transcript will be deleted from the database.

Keep Archiving data

By setting this value the system will have only the logs of specified value and purge other records which are older than the specified value.

If in case the data is stored on the Exchange mailbox then we need to make sure a retention policy with the retention tag corresponding to this folder is created which will not increase the Mailbox Quota for end users.

Below are the limitations of this sending Offline IM  feature at this moment:

1. This feature available only for peer to peer Instant messages at this moment.
2. This feature is not available for users sending IM to offline persons through mobile devices.
3. The IM (senders) should be sent from a desktop/laptop thick client. Microsoft might extend this feature to all the clients in future.

Thanks & Regards

Sathish Veerapandian

MVP – Office Servers & Services