Live Video Streaming over Solace, Part 2

In part 1 of this series of blog posts, I introduded the concepts of using Solace for video streaming .In this second installment of the live video streaming over Solace series, let’s take a look at a simple streaming demo application called BroadcastMe.  Its designed to show how live video streaming can be implemented over Solace.  The usage scenario is to multicast video from a single broadcaster to multiple stream viewers. The source code can be downloaded from Github.

Live video streaming is a little different from conventional video file playback. Stream video is usually transported using MPEG transport stream (MPEG-TS, MTS or TS).  Very briefly, a transport stream specifies a container format which describes how to encapsulate packetized elementary streams with error correction and stream synchronization features. Within the transport stream, there are a number of sub-streams. For instance, a transport stream can contain a main data stream of MPEG codec (such as mpeg2, mp4, h264), as well as any number of non-MPEG codecs (such as ACS, DTS audio) or even text information like electronic program guide. The transport stream is then wrapped at the IP layer using UDP or RTSP (Real Time Streaming Protocol) for IP network delivery.

As the method of encoding and decoding transport streams for video/live stream playback has matured over the past decade, it does not make sense to break what already works by modifying codec to fit into the Solace messaging ecosystem. A more non-intrusive way to apply Solace’s data movement technology to live video streaming is to focus at the network IP layer.  Specifically, I am referring to a streaming codec agnostic way of wrapping the payload of the transport layer protocol (such as UDP, RTSP) with Solace’s Solace Message Format (SMF). Here, the UDP payloads are the transport stream containers.

So, instead of delivering the transport stream container via the UDP protocol between a broadcaster and stream viewers, let’s intercept the UDP packet streams and re-inject its payload, encapsulated in SMF, into Solace for delivery.  This way, Solace’s data movement features can be exploited to address various data/stream delivery problems like fan-out efficiency, geographical content routing and caching or even WAN optimization.

How it Works

BroadcastMe in its basic form is a proxy.  It consists of two parts, an InputProxy and an OutputProxy. Typically, a live video stream will enter into Solace via InputProxy where it bridges the live stream source and Solace, and will exit Solace via OutputProxy where it bridges the stream viewer and Solace.

At the InputProxy, UDP data is read off from the socket and its content, i.e. the transport stream container, is then added as an attachment of an outgoing BytesXMLMessage (see snippets below).  This is the encapsulation process.

// Read data off socket
dsocket.receive(packet);
data = new byte[packet.getLength()];
System.arraycopy(packet.getData(), packet.getOffset(), data, 0,
packet.getLength());
// Create SMF message and encapsulate
BytesXMLMessage msg = JCSMPFactory.onlyInstance().createMessage(BytesXMLMessage.class);
msg.writeAttachment(data);

if (mode.equals("persistent")) {
msg.setDeliveryMode(DeliveryMode.PERSISTENT);
} else {
// Everything else, even if wrong spelling, just default to direct.
msg.setDeliveryMode(DeliveryMode.DIRECT);
}
// Send message
prod.send(msg, topic);

At the OutputProxy, the attachment part of the received BytesXMLMessage is read out and used as the message body of outgoing UDP packets (see snippets below). This is the decapsulation process.

// Pass msg content to UDP stream
DatagramPacket packet = new DatagramPacket(message.getAttachmentByteBuffer().array(),
message.getAttachmentContentLength(), address, port);
// Decapsulated and send to UDP
try {
dsocket.send(packet);
} catch (Exception ex) {
System.err.println("Encountered an Exception... " + ex.getMessage());
}

So, what about the source of the live video stream? Are there any changes required?  No, there are no modification required at the source side.  The InputProxy picks up the video feed just like a normal video stream viewer device would, though transport protocols like UDP or RTSP.  This design is intentional.

Likewise, what about at the destination, i.e. at the viewing device?  You don’t need to change anything at the destination either. The OutputProxy delivers the video feed just like a normal video source device would through transport protocol like UDP or RTSP.

Let’s Get the Demo Running

To get BroadcastMe to showcase live video stream over Solace you will need:

  • laptop with a video camera (for capturing live feeds)
  • FFMpeg program (for encoding feeds)
  • VLC program (for decoding feeds and viewing)
  • BroadcastMe binary (get from Github and follow the included build instructions)
  • JDK Environment 1.6 and above
  • Solace Virtual Message Router (VMR) (download the VMR community edition from dev.solace.com)

Overview of Steps

  1. Prepare the Solace Virtual Message Router
  2. Link up the VMRs using Solace’s Multi-Node Routing (MNR) protocol
  3. Start the InputProxy
  4. Start live video stream capturing using FFMpeg
  5. Start the OutputProxy
  6. Start the VLC viewer

Detailed Procedure

  1. Prepare the Solace Message Router
    1. Install 2 VMRs (e.g. run VMR on Amazon AWS in 2 different availability zones). Refer to this link for installation procedures.
    2. Ensure the firewall is not blocking on port 55555, 55556 (default SMF ports)
    3. Ensure that message-vpn’s “Maximum Spool Usage” is configured (the OutputProxy uses temporary queue with topic subscription)
  2. Link up the VMRs using Solace’s Multi-Node Routing (MNR) protocol
    The purpose is to show that the live video stream can, in fact, traverse through two different VMRs which can be geographically far apart.

    1. Configure MNR between 2 VMRs. One with the host name ip-172-31-19-216 and the other ip-172-31-13-92.
      ip-172-31-19-216# configure
      ip-172-31-19-216(configure)# routing
      ip-172-31-19-216(configure/routing)# create cspf neighbor ip-172-31-13-92
      connect-via 52.56.121.135
      
      ip-172-31-13-92# configure
      ip-172-31-13-92(configure)# routing
      ip-172-31-13-92(configure/routing)# create cspf neighbor ip-172-31-19-216
      connect-via 52.221.59.15
      
    2. Confirm MNR is established. Run the ‘show cspf neighbor’ command and make sure that the state is showing ‘OK’.
      ip-172-31-19-216# show cspf neighbor *
      Neighbor                        Ports    State    Cost Cost     Uptime
      Data/Ctrl            cfg  act
      ---------------------------- ----------- -------- ---- ---- ----------------
      ip-172-31-13-92              55555/—— Ok        100  100   0d 04h 41m 18s
      ip-172-31-13-92# show cspf neighbor *
      Neighbor                        Ports    State    Cost Cost     Uptime
      Data/Ctrl            cfg  act
      ---------------------------- ----------- -------- ---- ---- ----------------
      ip-172-31-19-216             55555/—— Ok        100  100   0d 04h 41m 26s
      
  1. Start the InputProxy
    1. Download and compile BroadcastMe (see README.md for detail)
    2. Start the InputProxy from a shell environment of your choice. The command below instructs the InputProxy to connect to VMR with IP of 52.221.59.15 (i.e. the public IP address of host ip-172-31-19-216) using username default to message-vpn default, and to publish received video, from localhost port 1235, using direct messaging to topic channel/1.
      $ ./inputProxy 52.221.59.15:55555 default default channel/1 direct 1235 verbose
      Input initializing...
      Jan 25, 2017 10:49:21 PM com.solacesystems.jcsmp.protocol.impl.TcpClientChannel call
      INFO: Connecting to host 'orig=52.221.59.15:55555, host=52.221.59.15, port=55555' (host 1 of 1, smfclient 2, attempt 1 of 1, this_host_attempt: 1 of 1)
      Jan 25, 2017 10:49:21 PM com.solacesystems.jcsmp.protocol.impl.TcpClientChannel call
      INFO: Connected to host 'orig=52.221.59.15:55555, host=52.221.59.15, port=55555' (smfclient 2)
      Connected.
      Control-C to exit
      
  1. Start live video stream capturing using FFMpeg
      1. Determine the laptop’s video capturing input device name with –list_devices option. For example, on a Macbook, the following indicates that “FaceTime HD Camera” is on “0”.
    $ ./ffmpeg -f avfoundation -list_devices true -i ""
    ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
    built with Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
    configuration: --prefix=/Volumes/tempdisk/sw --as=yasm --enable-gpl --enable-version3 --enable-pthreads --disable-ffplay --disable-ffserver --disable-shared --enable-static --enable-libvpx --disable-decoder=libvpx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libx265 --enable-libxvid --enable-zlib --enable-avfilter --enable-fontconfig --enable-libfreetype --enable-libass --enable-libvidstab --enable-libsnappy --enable-filters --enable-postproc --enable-runtime-cpudetect --disable-indev=qtkit --disable-indev=x11grab_xcb
    libavutil      55. 34.100 / 55. 34.100
    libavcodec     57. 64.101 / 57. 64.101
    libavformat    57. 56.100 / 57. 56.100
    libavdevice    57.  1.100 / 57.  1.100
    libavfilter     6. 65.100 /  6. 65.100
    libswscale      4.  2.100 /  4.  2.100
    libswresample   2.  3.100 /  2.  3.100
    libpostproc    54.  1.100 / 54.  1.100
    [AVFoundation input device @ 0x7fae53c0e940] AVFoundation video devices:
    [AVFoundation input device @ 0x7fae53c0e940] [0] FaceTime HD Camera
    [AVFoundation input device @ 0x7fae53c0e940] [1] Capture screen 0
    [AVFoundation input device @ 0x7fae53c0e940] AVFoundation audio devices:
    :
    :
    
    1. Use ffmpeg to capture the video feed from a laptop camera and push the encoded output to an UDP port and address that matches the setting on InputProxy. In this demo, both processes are running on the same physical host.
      ./ffmpeg -f avfoundation -framerate 30 -video_size 640x480 -i "0" -f mpegts udp://localhost:1235
      

      Under Linux environment, “video4linux2” should be passed in as the argument for -f and the input device (-i) normally defaults to /dev/video0.

  1. Start the OutputProxy
    1. Start the OutputProxy from a shell environment of your choice. The command below instructs the OutputProxy to connect to VMR with IP of 52.56.121.135 (i.e. the public IP address of host ip-172-31-13-92) using username default to message-vpn default, and to subscribe to topic channel/1 to receive messages and redirect them to IP address 127.0.0.1 at port 1239.
      $ ./outputProxy 52.56.121.135:55555 default default channel/1 127.0.0.1 1239 verbose
      OutputProxy initializing...
      Jan 25, 2017 10:49:23 PM com.solacesystems.jcsmp.protocol.impl.TcpClientChannel call
      INFO: Connecting to host 'orig=52.56.121.135:55555, host=52.56.121.135, port=55555' (host 1 of 1, smfclient 2, attempt 1 of 1, this_host_attempt: 1 of 1)
      Jan 25, 2017 10:49:24 PM com.solacesystems.jcsmp.protocol.impl.TcpClientChannel call
      INFO: Connected to host 'orig=52.56.121.135:55555, host=52.56.121.135, port=55555' (smfclient 2)
      Connected.
      Control-C to exit
      
    2. Confirm that topic subscriptions (e.g. channel/1 etc.) are propagated in the MNR network with the ‘show smrp subscriptions’ CLI command.
      ip-172-31-19-216# show smrp subscriptions
      
      Flags Legend:
      T - Destination Type (C=local-client, Q=local-queue
      R=remote-router)
      P - Subscription Persistence (P=persistent, N=non-persistent)
      R - Redundancy Type for Local Destinations (P=primary, B=backup
      S=static -=not-applicable)
      
      Message VPN : default (exported: Yes; 100% complete)
      Destination Name         Flags BlkID DTO  Subscription
      T P R       Prio
      ------------------------ - - - ----- ---- ------------------------------------
      #client                  C N S     0   P1 #MCAST/>
      #client                  C N S     0   P1 #SEMP/ip-172-31-19-216/>
      #client                  C N S     0   P1 #P2P/ip-172-31-19-216/#client/>
      #client                  C N S     0   P1 #P2P/v:ip-172-31-19-216/#client/>
      #client                  C N S     0   P1 #SEMP/v:ip-172-31-19-216/>
      ip-172-31-13-92          R N -     0   DA #MCAST/>
      ip-172-31-13-92          R N -     0   DA #SEMP/ip-172-31-13-92/>
      ip-172-31-13-92          R N -     0   DA #P2P/ip-172-31-13-92/#client/>
      ip-172-31-13-92          R N -     0   DA #P2P/v:ip-172-31-13-92/#client/>
      ip-172-31-13-92          R N -     0   DA #SEMP/v:ip-172-31-13-92/>
      v:ip-172-31-13-92        R N -    21   DA channel/1
      
      
      ip-172-31-13-92# show smrp subscriptions
      
      
      Flags Legend:
      T - Destination Type (C=local-client, Q=local-queue
      R=remote-router)
      P - Subscription Persistence (P=persistent, N=non-persistent)
      R - Redundancy Type for Local Destinations (P=primary, B=backup
      S=static -=not-applicable)
      
      Message VPN : default (exported: Yes; 100% complete)
      
      Destination Name         Flags BlkID DTO  Subscription
      T P R       Prio
      ------------------------ - - - ----- ---- ------------------------------------
      #client                  C N S     0   P1 #MCAST/>
      #client                  C N S     0   P1 #SEMP/ip-172-31-13-92/>
      #client                  C N S     0   P1 #P2P/ip-172-31-13-92/#client/>
      #client                  C N S     0   P1 #P2P/v:ip-172-31-13-92/#client/>
      #client                  C N S     0   P1 #SEMP/v:ip-172-31-13-92/>
      #P2P/QTMP/v:ip-172-31-13 Q P P    21   DA channel/1
      -92/29f6f3a8-08be-4014
      -8e72-e205af594849
      ip-172-31-19-216         R N -     0   DA #MCAST/>
      ip-172-31-19-216         R N -     0   DA #SEMP/ip-172-31-19-216/>
      ip-172-31-19-216         R N -     0   DA #P2P/ip-172-31-19-216/#client/>
      ip-172-31-19-216         R N -     0   DA #P2P/v:ip-172-31-19-216/#client/>
      ip-172-31-19-216         R N -     0   DA #SEMP/v:ip-172-31-19-216/>
      

      The OutputProxy create a temporary queue (#P2P/QTMP/v:ip-172-31-13-92/29f6f3a8-08be-4014-8e72-e205af594849) and maps topic subscription “channel/1” to attract the messages.

    3. Confirm that video streams are flowing between the VMRs by checking the Total Message/Bytes Sent and Received counter in neighbor statics.
      ip-172-31-19-216# show cspf neighbor ip-172-31-13-92 stats
      Neighbor     : ip-172-31-13-92
      Connect Via  : 52.56.121.135:55003
      Control Port : unspecified
      Received                 Sent
      -------------------- --------------------
      Total Messages                                 3399                 6118
      Control Messages                             2486                 2482
      Data Messages                                 913                 3636
      Total Bytes                                 1380320              4010413
      Control Bytes                              461506               462290
      Data Bytes                                 918814              3548123
      
      Ingress (msg/sec)     Egress (msg/sec)
      -------------------- --------------------
      Current Rate (1 sec sample)                       1                   48
      Average Rate (60 sec interval)                    0                    9
      Ingress (byte/sec)    Egress (byte/sec)
      -------------------- --------------------
      Current Rate (1 sec sample)                     181                44353
      Average Rate (60 sec interval)                   28                 9824
      Total Ingress Discards                                                 0
      Total Egress Discards                                                  0
      
      
      ip-172-31-13-92(admin)# show cspf neighbor ip-172-31-19-216 stats
      
      Neighbor     : ip-172-31-19-216
      Connect Via  : 52.221.59.15:55003
      Control Port : unspecified
      Received                 Sent
      -------------------- --------------------
      Total Messages                                 7808                 3396
      Control Messages                             2478                 2483
      Data Messages                                5330                  913
      Total Bytes                                 5669019              1379594
      Control Bytes                              461491               460780
      Data Bytes                                5207528               918814
      Ingress (msg/sec)     Egress (msg/sec)
      -------------------- --------------------
      
      Current Rate (1 sec sample)                      43                    1
      Average Rate (60 sec interval)                   26                    0
      Ingress (byte/sec)    Egress (byte/sec)
      -------------------- --------------------
      Current Rate (1 sec sample)                   35315                  181
      Average Rate (60 sec interval)                25662                   42
      Total Ingress Discards                                                 0
      Total Egress Discards                                                  0
      
  1. Start the VLC viewer
    1. Start VLC and choose Media/Open Network Stream (usually under File -> Open Network)
    2. Enter, in the address field, the URL matching the redirected address and port specified by the OutputProxy, e.g. udp://@127.0.0.1:1239. Don’t forget the “@” symbol after the “udp://” and before the address.