Sending Video from a Public Stream to an Amazon Kinesis Video Stream Using GStreamer

August 3, 2019

Figure showing the Kinesis Video Streams service that receives video from a public stream.

Last updated: September 12, 2019

The Amazon Kinesis Video Streams service makes it easy to securely stream video from different devices to the AWS Cloud for real-time video processing, machine learning (ML), analytics, playback, and storage.

This blog post describes how to build a Docker image containing the GStreamer multimedia framework and the Amazon Kinesis Video GStreamer plugin (kvssink). You launch a GStreamer pipeline in a running Docker container to send video from a public stream, for example, from a Periscope live stream to the Amazon Kinesis Video Streams service.

To get further information about the Amazon Kinesis Video Streams service and learn how to stream video from a PC webcam or a media file to an Amazon Kinesis Video stream using the GStreamer multimedia framework, see Streaming Video from a PC Webcam to an Amazon Kinesis Video Stream Using GStreamer.

How to Send Video from a Public Stream to a Kinesis Video Stream Using GStreamer

The following diagram provides a general architectural overview of the solution that you implement using this procedure.

Flow diagram showing a Docker container running on an EC2 instance or PC. GStreamer running in the Docker container gets media data from a public HLS/RTSP stream an puts it to Kinesis Video Streams.

To send video from a public stream to an Amazon Kinesis Video stream using the GStreamer multimedia framework, perform the following steps:

Build a Docker image containing GStreamer and the Amazon Kinesis Video GStreamer plugin. You can build and use this image on your PC or in the cloud, on an EC2 instance.
See Appendix A, How to Build a Docker Image Containing GStreamer and the Amazon Kinesis Video GStreamer Plugin.
Open the Kinesis Video Streams Console at https://console.aws.amazon.com/kinesisvideo and switch to the AWS region where you want to create a Kinesis Video stream. Then, create a new stream with the name example-video-stream and default settings.

If you want to run the Docker image on your PC, create a new IAM user.
If you want to run the Docker image on an EC2 instance, create a new IAM role and attach this role to your EC2 instance.
In both cases, use the following policy:

{ 
  "Version": "2012-10-17",
  "Statement": [
    { 
      "Action": [
        "kinesisvideo:DescribeStream", "kinesisvideo:GetDataEndpoint",
        "kinesisvideo:PutMedia"
      ],
      "Resource": "arn:aws:kinesisvideo:*:*:stream/example-video-stream*",
      "Effect": "Allow"
    }
  ]
}

Run a new Docker container and get shell access inside this container.
```
docker run -it --rm --network='host' \
kinesis-video-producer-sdk-cpp-amazon-linux /bin/bash
```
- Do not close this shell prompt. In the following steps you use it for executing commands within the running container.
Set up the credentials file that is used by the Kinesis Video Streams GStreamer plugin (kvssink).
- If you run the Docker image on your PC, in the running Docker container create a new file /root/credentials.txt and put in this file the <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> of the IAM user that you created in step 3.
  The file must contain only one following line:
```
CREDENTIALS	<AWS_ACCESS_KEY_ID>	<AWS_SECRET_ACCESS_KEY>
```
- If you run the Docker image on an EC2 instance, in the running Docker container execute the following command to create the credentials.txt file containing the credentials of the IAM role of your EC2 instance.
```
python << EOF > /root/credentials.txt
from __future__ import print_function
import urllib2
import json
metadata_url = 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
iam_role = urllib2.urlopen(metadata_url).read()
credentials = json.load(urllib2.urlopen(metadata_url + iam_role))
print('CREDENTIALS', credentials['AccessKeyId'], credentials['Expiration'],
      credentials['SecretAccessKey'], credentials['Token'])
EOF
```

Send video from a public stream to your Amazon Kinesis Video stream.
In the following commands replace <AWS Region> with the region name where you created your Amazon Kinesis Video stream (for example, us-east-1).

If you want to send video from a HTTP (HLS) live stream, launch the following GStreamer pipeline in the running Docker container:

gst-launch-1.0 souphttpsrc location='<https://Stream URL>' ! \
hlsdemux ! decodebin ! \
x264enc bframes=0 key-int-max=15 bitrate=300 ! \
capsfilter caps='video/x-h264,stream-format=avc,alignment=au' ! \
kvssink stream-name='example-video-stream' \
storage-size=32 credential-path=/root/credentials.txt \
aws-region='<AWS Region>'

Refer to Appendix B, How to Retrieve the URL of a Periscope Live Stream if you want to send video from a Periscope live stream.

If you want to send video from a RTSP live stream which is already H.264 compressed, launch the following GStreamer pipeline:

gst-launch-1.0 rtspsrc location='<rtsp://Stream URL>' ! \
rtph264depay ! \
capsfilter caps='video/x-h264,stream-format=avc,alignment=au' ! \
kvssink stream-name='example-video-stream' \
storage-size=32 credential-path=/root/credentials.txt \
aws-region='<AWS Region>'

Appendix A, How to Build a Docker Image Containing GStreamer and the Amazon Kinesis Video GStreamer Plugin

To build a Docker image containing the GStreamer multimedia framework, the Amazon Kinesis Video GStreamer plugin (kvssink), and the souphttpsrc GStreamer element that is required to read media data from HTTP (HLS) video streams, perform the following steps:

Clone the repository of the Kinesis Video Streams Producer Library for C++. Then, switch to the latest commit that existed at the time of the last update of this procedure.

git clone https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp;
cd amazon-kinesis-video-streams-producer-sdk-cpp;
git checkout $(git rev-list master -n 1 --first-parent --before='2019-09-12');

Go to the directory that contains the Dockerfile with build instructions. Then, patch the original Dockerfile to build additionally the souphttpsrc GStreamer element.

cd docker_native_scripts/amazonlinux-docker

patch -p0 Dockerfile << EOF
@@ -21,0 +22,2 @@
+        yum install -y libsoup-devel && \\
+        yum install -y gnutls-devel && \\
@@ -42,0 +45,2 @@
+RUN sed -i '/^export PKG_CONFIG_LIBDIR=/a export PKG_CONFIG_LIBDIR="\\\$PKG_CONFIG_LIBDIR:\/usr\/lib64\/pkgconfig"' ./install-script
+RUN sed -i 's/^max_parallel=2/max_parallel=\\\$(grep -c "^processor" \/proc\/cpuinfo)/' ./install-script
@@ -48,0 +53 @@
+ENV GIO_MODULE_DIR=/usr/lib64/gio/modules
EOF

Build a new Docker image.
```
docker build -t kinesis-video-producer-sdk-cpp-amazon-linux .
```
- Build time on a c5.xlarge EC2 instance that has 4 vCPUs is approximately 15 minutes.

Appendix B, How to Retrieve the URL of a Periscope Live Stream

Each Periscope.tv stream has a unique Stream ID. You can find it in the address bar of your browser when you open a stream. For example, in the following URL, the Stream ID is 1P3KQLuedBixE.

https://www.periscope.tv/w/1P3KQLuedBixE?channel=...

To retrieve the URL of a Periscope stream, you need to send a request containing the Stream ID to the Periscope API endpoint and extract the value of hls_url. For example, for the stream 1P3KQLuedBixE, execute the following command:

curl -s 'https://api.periscope.tv/api/v2/accessVideoPublic?broadcast_id=1P3KQLuedBixE' | \
python -c 'import sys, json; print json.load(sys.stdin)["hls_url"]'

Kinesis Video Streams

Sending Video from a Public Stream to an Amazon Kinesis Video Stream Using GStreamer

How to Send Video from a Public Stream to a Kinesis Video Stream Using GStreamer

Appendix A, How to Build a Docker Image Containing GStreamer and the Amazon Kinesis Video GStreamer Plugin

Appendix B, How to Retrieve the URL of a Periscope Live Stream

Table of Contents