This has resulted in many Citrix administrators having to deploy Teams into their Virtual Apps and Desktops environments – often quite rapidly. Teams is Microsoft’s primary Office365 collaboration tool and is written in Electron, which makes it a bit of a pain to manage, like other Electron apps. Teams HDX optimization will not be backported to the LTSR 7.15. Step-by-step: Microsoft Teams optimization mechanisms can be divided into 5 main areas. Media Engine instantiation (HdxTeams.exe or HdxRtcEngine.exe, depending on CWA version). The Citrix® HDX™ Optimization for Microsoft® Teams offers clear, crisp 720p high-definition video calls @30 fps, in an optimized architecture. Users can seamlessly participate in audio-video or audio-only calls to and from other Teams users, Optimized Teams’ users and other standards-based video desktop and conference room systems.

• Disable Citrix Hdx Optimization For Microsoft Teams
• Citrix Optimization For Microsoft Teams Login

With many customers on Office 365 or migrating to Office 365, one of the benefits of this feature release is the ability to take advantage of an intelligent communication solution, as Microsoft Teams is bundled into Office 365.

With built-in tools within Microsoft Teams, people can work more efficiently and stay connected wherever they are, on any device, through centralized delivery optimized by Citrix.

With the upcoming release of HDX optimization for Ms Teams, customers will get a fully native, fully featured Ms Teams experience within Citrix Virtual Apps and Desktops. With a single point of authentication, this essentially also improves reliability and ease of use. The Citrix Workspace app has a built-in multi-platform HDX Media Engine that ensures optimized device and media handling, with audio, video, and screen sharing offloaded to the users device.

• This Preview may be used to install, configure, and explore the Citrix Virtual Apps and Desktops solutions for optimizing Microsoft Teams with HDX. It is available to Citrix Partners and Citrix Virtual Apps and Desktops (CVAD) customers with current Customer Success Services (CSS) as of Feb 13, 2019.
• You would also need access to a Preview build of the Teams application that Microsoft is making available through their TAP Program.
• The HDX optimization for Teams allows you to run multimedia workloads in virtual desktops without sacrificing user experience and server scalability, which is critical for Microsoft Teams adoption in the Enterprise.
• With this preview of Workspace app version 19.5.0.19, you can offload the processing of audio, video and screen sharing to the user’s device, which results in lower CPU, RAM, I/O and Bandwidth consumption in the virtual machine hosting the application.
• Citrix Workspace app will fetch, decode and render the audio and video streams directly, leveraging the processing power of the endpoint and Internet break-out at the branch office (connectivity back to Azure services required).

Download the Citrix Workspace App tech preview file and follow the installation instructions in the Microsoft TAP Guide.

A 1906 VDA is required.

Please note that, generally, fully quitting Teams (Systray icon / right click / quit) and restarting it clears all the components involved and can fix whatever issue you were experiencing.

[Warning: If your VDA is 7.15 (any CU) and you still want to use Teams in unoptimized mode (the only mode supported in these VDA versions), scroll here. Teams HDX optimization will not be backported to the LTSR 7.15].

Step-by-step:

Microsoft Teams optimization mechanisms can be divided into 5 main areas

1. Media Engine instantiation (HdxTeams.exe or HdxRtcEngine.exe, depending on CWA version)

2.Media Engine peripheral acquisition

3.Media Engine transport layer establishment

4. Media Engine multimedia stream processing

5. Monitoring
More information on how to leverage HDX Monitor to monitor Teams optimization can be found here.

Proper troubleshooting requires the Admin to verify each area progressively, moving down the list.

Please note that the presence of a Citrix Gateway as an HDX proxy should have no impact, so you can discard that from your troubleshooting. If you are also using the Gateway for VPN, make sure you are allowing the client machine to reach the O365 Teams servers.

#1: Media Engine instantiation

The quickest way to determine if the user is running in optimized mode is to look for HdxTeams.exe (or HdxRtcEngine.exe) running on the user’s client machine:

If the process is not found, then point #1.1 should be investigated first.

Click on the avatar / About / Version. “Citrix HDX Optimized” should be displayed. This means Teams is ready for an optimized call.

If the legend is not there, and instead you see “Citrix HDX Not Connected” then Teams did not load in VDI mode correctly.

Citrix HDX Not Connected means Teams loaded the Citrix API correctly, but it could not complete an end-to-end check because there was an error in subsequent parts of the stack. Most likely, VDA Services (Citrix HDX HTML5 Video Redirection, Citrix HDX Teams Redirection) or in Workspace app.

If there is no legend with the word Citrix, then Teams did not load in VDI mode at all:

The VDA regkey MsTeamsRedir is set by a Citrix service (CtxSvcHost.exe) when the user logs on or reconnects to the virtual desktop. Teams will read this key when starting, and if the value is 1, it will load in VDI mode.

The value is set to 1 automatically if and only if:

1. Microsoft Teams Redirection policy is enabled in Studio (and by default is Allowed)
2. CWA has loaded the Teams Virtual Channel (VDTeamsn.dll) and the CWA version is 1907 or higher

If = 0 or missing – trouble!

Action: Quit Teams from the systray itself (right click on the icon / Quit).
Sometimes, if Teams auto-updated while the session was in disconnected state, it can fail to read the key properly upon reconnections.

Lastly, if the regkey value is 1 but Teams still fails to display “Citrix HDX Optimized” or “Citrix HDX Not Connected”, there is likely a problem with Teams itself. Common issues could be erroneous cached items (settings.json and storage.json) in AppDataRoamingMicrosoftTeams , or even dual Teams installations (under %appdata% and Program Files). Customers are advised to contact Microsoft support in this case.

#1.2: WebSocket phase

The VDA Service “Citrix HDX HTML5 Video Redirection” [WebSocketService.exe] is the point of entry for Teams into the VDA. In other words, Teams must open a secure websocket connection to this service, which listens at 127.0.0.1:9002 in the VDA.
If WebSocketService.exe is not actively listening, optimization will fail. In some cases, the Service will show as “Running” in the services.msc console, but when you run netstat you will not see the socket.
Therefore, you must make sure it is running and LISTENING :

`

After a successful connection, you will see the state change to “ESTABLISHED”:

[Note: WebSocketService.exe listens also at 9001 – this socket is used for other HDX features (Browser Content Redirection and HTML5 Video Redirection, and they are independent of Teams)]

–Proxies: If an explicit proxy server is configured in the VDA, make sure connections to localhost are not routed through it. Otherwise, redirection will fail. You must “bypass proxy servers for local address” 127.0.0.1:9002

–PAC files: make sure your VDA pac file script is returning DIRECT for wss://127.0.0.1:9002. If not, optimization will fail. One way to do this:

shExpMatch(url, “wss://127.0.0.1:9002/*”)

–Anti-Virus: Some security software products interfere with the proper functioning of WebSocketService.exe and its certificates. While the actual service (Citrix HDX HTML5 Video Redirection) might be running in the services.msc console, the localhost 127.0.0.1:9002 TCP socket is never in listening mode as seen in netstat. Trying to restart the service causes it to hang (“Stopping…”). Ensure you apply the proper exclusions for the WebScoketService.exe process.

Once there is a successful secure WebSocket (wss) connection, WebSocketService.exe will create a new process in the user’s session called WebSocketAgent.exe:

This process can also be spotted in Director:

[Note: WebSocketAgent.exe is also used for BCR and HTML5 Video Redirection]

Action: if WebSocketAgent.exe is not being created while 127.0.0.1:9002 is ESTABLISHED, restart the WebSocketService.exe service. Relaunching Teams might be required. Also, in some rare cases, you might see WebSocketAgent.exe coming up but closing within a second, and in this case this could be related to a crash in HdxTeams.exe in the client machine (hence make sure to collect EventViewer entries on the client).

The last part of this second phase is the establishment of the Teams virtual channel (CTXMTOP), and this process is ultimately handled by the “Citrix HDX Teams Redirection” Service (CtxSvcHost.exe in the VDA).
CTXMTOP is a Command & Control virtual channel only – no media is exchanged between CWA and VDA. All the media is handled by CWA in a Client-Fetch-Client-Render manner.

Action: If WebSocketAgent.exe is running on the VDA but HdxTeams.exe is not running on the client, restart “Citrix HDX Teams Redirection”.

Once the virtual channel is established, the Citrix HDX Engine in CWA (wfica32.exe) will spawn the media engine on the client:

At this point, Teams is ready to handle calls.

If HdxTeams.exe / HdxRtcEngine.exe launches, then the respective webrpc log will be created.

Windows Logs:
They can be found on the user’s machine in %userprofile%AppDataLocalTempHDXTeams folder.
(Newer versions of CWA, e.g. 2009.6 or higher, store the logs in AppDataLocalTempHdxRtcEngine)
Each session will create its own separate folder.

Mac Logs:
1. vdwebrtc log — records the execution of Virtual Channel vdwebrtc
Location： /Users/<User Name>/Library/Logs/Citrix Workspace/CitrixViewer_<Y_M_D_H_M_S>.txt

2. HdxRtcEngine log — records the execution of process HdxRtcEngine
Location: $TMPDIR/hdxrtcengine/<W_M_D_H_M_S_Y>/hdxrtcengine.log
Note that Hdxrtcengine log is enabled by default.

Linux Logs:
The /tmp/webrpc/<current date>/ and /tmp/hdxrtcengine/<current date>/ directory will have log files.

#2:Media Engine peripheral acquisition

In Teams, go to Settings à Devices. All the peripherals should now be mapped, and the video self-preview visible. You should be able to toggle between the cameras too.
The “Make a test call” button is a very handy tool to assess peripheral acquisition and performance. Go to Settings/Devices/Audio devices.

Note: when Teams enumerates Playback/Recording peripherals, it will select the ones that are defaults at the endpoint. Open mmsys.cpl on the endpoint, and you can see what the default devices are in the user’s client machine. These are the ones that will be auto-selected by Teams.
You can toggle these devices in the middle of a call – in that case, a 1 sec audio glitch might be experienced while the new devices are remapped.

The peripheral acquisition process is captured in the webrpc.txt logs.
All device enumeration entries will have “webrtcapi.RPCStubs” in the respective lines.

#2.1 Errors
Any error in the acquisition process will be displayed in the webrpc logs.
E.g.:

Important: The error in the table above is fixed in Citrix Workspace app 2009.6 for Windows - so any peripheral (even Studio-Quality soundcards) is mapped.

If Teams loaded in unoptimized mode, it will try to acquire the endpoint’s peripherals leveraging the traditional HDX technologies for Audio, Microphone and Webcam redirection (see section on 7.15 for a description of unoptimized mode).
In this case, the camera name will show the “Citrix HDX Web Camera” legend – this is an unequivocal sign that Teams is unoptimized.

In this phase, errors are generally equivalent to call failures, or one-way audio/video only where one peer cannot hear/see the other.

The call setup process is handled primarily by Teams, not HDX WebRTC engine In other words, the hosted Teams app will utilize the signaling channel back to O365 Teams servers who will in turn contact the other peer.
Teams does rely on HDX WebRTC engine for the call negotiation by requesting an SDP file, which is a compilation of all the media and transport address characteristics of the endpoint.
HDX WebRTC engine relies on hole punching techniques (ICE/STUN/TURN) to eventually establish the connection to the other peer or conference server.

Enterprise network environments can be a challenge in this phase.
Proxies, Firewalls, symmetric NATs, VPNs, Deep packet inspectors, Session Border Controllers, etc etc can all affect the call establishment process; hence it is important you understand your network topology and consult with your networking team if necessary.

Teams conference servers are all hosted in O365 with the IP Blocks 52.114.0.0/12 52.120.0.0/12 and UDP ports 3478-3481. TCP 443 can be used as a fallback.
Peer to peer calls will be attempted via direct connections, but if that fails a TURN server (also hosted in O365, same IP:PORTS as above) might be used.

Important note for DNS : make sure the endpoint can do DNS resolutions. The Teams TURN Servers FQDN must be resolved by the endpoint (e.g. worldaz.turn.teams.microsoft.com and euaz.turn.teams.microsoft.com)

#3.1 When the user initiates or receives a call, the signaling channel between the Teams hosted app and the Teams servers in O365 will negotiate an SDP file, containing all the required information for HdxTeams.exe or HdxRtcEngine.exe to execute the call.
Please note that the SDP file can be updated multiple times after the initial single pass negotiation,
containing new info about ICE candidates or new call parameters (resolution, fps, etc).

Each peer maintains two simultaneous descriptions:

• a local one describing itself ('webrtcapi.RTCPeerConnection Info: setLocalDescription”)

• a remote one describing the other peer (setRemoteDescription)

The SDP files are logged in webrpc.txt. Look for the “>>> begin:sdp” entries.
The caller will ‘set’ his SDP offer first (without any ICE candidates), which can be seen in the webrpc.txt logs as setLocalDescription.

#3.2 Interactive Connectivity Establishment
The caller now will ask STUN servers to generate ICE candidates. It will then send -over the signaling channel- individual ICE candidates as soon as they are discovered (a.k.a “trickle ICE”) by the local ICE agent .

Example #1 : VDI caller (behind Firewall blocking UDP) joining a conference

The VDI user caller will register remote ICE candidate and begin connectivity checks (STUN Bind request/response). Sometimes, the calling peer can be behind a Firewall that blocks STUN/UDP packets, and in such a case there will no answer to the STUN Bind requests:

[Warning: STUN Bind requests are essential for a healthy call setup, as this allows candidate discovery with optimal call routing (especially in p2p calls). If your branch office Firewalls are blocking STUN and UDP destination ports 3478-81, then a transport relay in O365 must be contacted using TCP which adds additional latency and degrades call quality experience].

Since the Firewall is blocking UDP, the calling peer must attempt a TCP connection to the other candidate:2
If that also fails, it will have to use his transport relay (“relay raddr”) seen in the Trickle ice snippet above (52.114.188.57).

Note: HdxTeams.exe cannot read explicit proxy configurations on the endpoint.

#3.3 RTP media flows
If the connection was established using TCP through a TURN server, you will not be able to decode the traffic in Wireshark as RTP since it is encrypted using TLS.

In case of a peer-to-peer connection or conference via UDP, then Wireshark will show more detailed info that can be used to understand the traffic flow between the participants.

Wireshark recommendations:

RTP cannot be automatically analyzed by Wireshark. Applying the Display filter ‘rtp’ won’t result in any RTP packets displayed.
Instead, first set the Display Filter to ‘udp’ to spot UDP traffic. After the connection is established, there will be UDP traffic sent regularly between the peers. RTP over UDP traffic will begin with 90 hexa (or 80 hexa if no RTP header extensions are present) in Data, the first octet of the RTP header.

By selecting one of these UDP packets and selecting 'Analyze/Decode As' … then selecting RTP, Wireshark will process the packets as RTP.

The WebRTC media engine in Workspace app (HdxTeams.exe) uses the Secure RTP protocol for multimedia streams that are offloaded to the client. SRTP provides confidentiality and authentication to RTP, using symmetric keys to encrypt media and control messages, and uses the AES encryption cipher in Counter Mode.

4. Media Engine multimedia stream processing

Once the call set up is finalized and the ICE connectivity checks are completed, the media will be exchanged between the parties.
Currently we support video resolutions up to 720p@30fps, with 360p being common [H264 and VP9].

-The video streams are captured in the webrp.txt logs as 'DxVideoRenderer ':
webrtcapi.DxVideoRenderer Info: configure. window = 657170 frameWidth = 640 frameHeight = 360
-The webcam captures entries in webrp.txt are'CameraCapture':
webrtcapi.CameraCapture Info: start. capabilities: width = 640 height = 360 fps = 30 fmt = 9

Lip sync issues might be caused by high CPU usage on the endpoint. The encoding process (camera capture and subsequent encoding) is more expensive in CPU terms than decoding. Some endpoints might not have enough processing power (see here for system requirements), resulting in 100% CPU utilization and causing audio/video to be out of sync.

The RTP protocol can be decoded in Wireshark to understand what type of codecs are being used and how much bandwidth is consumed.

#4.1 As an example we will investigate a p2p audio call between two VDI users. HdxTeams will select the Opus codec for this scenario, which can be seen clearly in the SDP Offer/Answer negotiation.
The calling party will state all his supported codecs, and the callee will then reply with his preferred choice in the SDP answer.

[Note: The notification audio (ringtone) is played by Teams out of the VDA’s default playout device.]

The call bandwidth usage can then be plotted in Wireshark using the Statisticsà IO Graph option.
Apply the proper filters to display only the traffic associated with the call.
We can see that the typical bandwidth for a voice call is ~40-70 kbps (each way).

Green: outgoing | Blue: incoming | Red: outgoing + incoming

Screensharing

Disable Citrix Hdx Optimization For Microsoft Teams

In the Skype for Business optimization pack (RTOP), screensharing was not optimized. In other words, there was no offloading of such streams and therefore they were rendered server-side natively by Skype for Business.

But in terms of HDX optimization for Microsoft Teams, incoming screensharing is no different than incoming webcam video streams, since Teams relies on video-based-screen-sharing. Hence it is optimized by default.

However, a VDI user sharing his monitor (i.e. outgoing screensharing) will result in HdxTeams.exe / HdxRtcEngine.exe performing display captures at the endpoint (with H.264). Currently it is only possible to share the primary (default) monitor.
HdxTeams.exe / HdxRtcEngine.exe crops the sharing to the Desktop Viewer window only (if CWA's Desktop Viewer is in Windowed mode).

If Desktop Viewer (cdviewer.exe) is in full screen mode, then of course the entire monitor is shared.
In multi-monitor setups, only the primary monitor is shared, and the screen picker button in Teams will label the preview as 'Screen #1'.

Citrix Optimization For Microsoft Teams Login

Common issues with outgoing screensharing, as perceived by the other peer, are:
- low resolution
- black window instead of the actual screen
- screen freeze on the sender's side

The webrpc.txt logs are the best source for troubleshooting such issues.

HdxTeams.exe / HdxRtcEngine.exe contains a bandwidth estimator that will allow Teams to pick the best resolution that can be sustained by the network, and sometimes it can take a few seconds for it to converge to the right value. So screensharing might actually start with a low resolution before it jumps to a higher one.
In the logs, the parameter 'webrtcapi.DesktopCapture' can be used for quick finds, and if there are errors they would be logged as 'webrtcapi.DesktopCapture Error:'.
Black windows can be caused by issues on the sender's side (user capturing) or the receiver's side (user rendering the stream).
For example, if the error is on the sender's side the following entry could be seen:
webrtcapi.DesktopCapture Error: OnCaptureResult. invalid cropping rectangle (x = 0 y = 0 width = 0 height = 0)

Monitoring Teams with HDX Monitor

On 1912 VDAs (or higher), you can monitor an optimized active call using HDX Monitor (3.5.2 or higher).
Important : make sure you are matching the HDX monitor version with the right VDA version (HDX Monitor is packaged with the CVAD .iso)
To turn this on, create the following WebrtcDirectorIntegration regkey on the VDA:

HKLMSoftwareCitrixHDXMediaStream
Or
HKLMSOFTWAREWOW6432NodeCitrixHdxMediaStream
- reg key value:
name: WebrtcDirectorIntegration
type: DWORD
value: enable(1), disable(0)

The information is displayed under the WebRTC icon :

The following data content will be populated when an active Teams redirection is going on.
Please see below for the detailed field description:

• Call Device Used Audio: audio device used for the last active call
• Call Device Used Video: video device used for the last active call
• Call Direction: incoming or outgoing
• Call Duration Audio: last active audio call duration in seconds
• Call Duration Video: last active video call duration in seconds
• Call Duration Conference: last active conference call duration in seconds
• Call Duration Screenshare: last active desktop sharing call duration in seconds
• Call Establishment Duration Incoming: last active incoming call establishment in sec
• Call Establishment Duration Outgoing: last active outgoing call establishment in sec
• Call State: Active or Idle
• Call Type: Audio, Video, Conference, Screenshare
• Version - Endpoint OS: not filled currently
• Version - Receiver: CWA version
• Version - Typescript: API version
• Version - Vda: not filled currently
• Version - Webrpc: webrpc version
• Version - WebrtcCodec: webrtc codec version

Guidelines for 7.15 LTSR

If you are using 7.15 LTSR (with any CU), please note that the new HDX Optimization for Microsoft Teams is not (and will not be) available in this version and the only way to use Teams is through the legacy HDX technologies:

• Client Audio Redirection (we recommend using the RTP/UDP policy, which requires the Audio quality to be set to Medium)
• Multimedia Conferencing (for Webcam redirection, a.k.a HDX RealTime Webcam Video Compression)

While these technologies do offload some of the CPU/RAM/BW from the VDA, they do not fully optimize the experience for Real Time traffic and will create additional loads on the VDA resources.
The best example is the rendering of a video feed from another peer / meeting. This is done using Thinwire graphics technologies and it renders on the VDA, which is expensive in CPU terms.
Admins concerned about Single Server Scalability might want to disable video conferencing policies on the Microsoft Teams admin center. See here for more info.

A secondary method for redirecting peripherals is available, and some customers might prefer to use Generic USB Redirection.
With HDX Generic USB Redirection technology, the peripherals (e.g webcam or headset) is virtually detached from the client device and attached to the XenApp/XenDesktop session. This provides all the native functionalities of the webcam/peripheral in the virtual session.
HDX Generic USB Redirection requires the device drivers for the webcam/peripheral to be available on both the client device as well as on the VDA.
Bandwidth usage for webcams using HDX Generic USB Redirection technology can vary based on the vendor and model of the device, but it is significantly higher compared to use it over HDX RealTime Webcam Video Compression. HDX Generic USB for webcams/peripherals is recommended to be used only under LAN conditions where bandwidth and specially latency are not constraints.
Refer the following link regarding more information on HDX Generic USB configuration: Configure USB Support.

Known Issues with Microsoft Teams and 7.15 VDA:

• Webcam redirection will not work with 64-bit Teams, only with 32-bit. This is known limitation in 7.15 (any CU) - see here for more details.
• In Windows Server, Teams.exe might crash, or the Teams logon screen might become blank and no input fields are presented to the user. Please add Teams.exe to the VDA registry under SfrHook, and make sure the Flag value REG_DWORD is set to 204 Hexadecimal
• After you add the SfrHook key, you must reboot your VDA.
• As a last resort, you can disable ALL hooking for Teams by following CTX107825

Related:

• No Related Posts