Working With Media¶

You can use a variety of different media inputs and outputs with alwaysAI, as well as perform different media manipulations. This guide will outline using the different media options in your computer vision applications.

Capture Video From a Camera
- General Usage
Play Video From a File
- Simulate a Real-Time Video Stream
Store Video Clips
- Capture Video on an Event Trigger
Publish a Video Stream
Manipulate Images and Video Frames

Capture Video From a Camera ¶

The video stream class to use is based on the target you’re running on as well as the type of camera. Use this table to determine which class to use.

Target	USB Webcam	CSI (Ribbon) or Built-in Camera
Mac	`WebcamVideoStream`	`WebcamVideoStream`
Windows	`WebcamVideoStream`	`WebcamVideoStream`
Raspberry Pi	`WebcamVideoStream`	`WebcamVideoStream`
NVIDIA Jetson	`WebcamVideoStream`	`JetsonVideoStream`
Other devices	`WebcamVideoStream`	`GStreamerVideoStream`

Additionally, the following are supported across all targets:

Intel RealSense camera: RealSense
IP video streams: IPVideoStream

Head to the Supported Devices page to see the full list of supported boards and cameras.

General Usage ¶

All the video stream classes follow the same operation. Using WebcamVideoStream, the video stream can be started in the following way:

with edgeiq.WebcamVideoStream(cam=0) as video_stream:

The cam parameter is the camera index of the webcam to use, where 0 is often the default camera. The most recent frame can be read by using the read() function in the code block:

frame = video_stream.read()

Note that frames that become outdated are dropped to enable real-time analysis. The camera connection is automatically cleaned up upon exiting the code block.

Play Video From a File ¶

The FileVideoStream class enables loading video from a local file. To generate a list of all video files in a directory, the list_files() function can be used.

video_paths = edgeiq.list_files(base_path="some/video/dir", valid_exts=(".avi", ".mpeg"))

The video stream can be started in the following way, using either a known path to a video file, or a path returned by list_files():

with edgeiq.FileVideoStream("video.mpeg") as video_stream:

Within the following code block, the next frame can be read out using the read() function:

frame = video_stream.read()

Unlike WebcamVideoStream, the frames are queued up when the app read gets behind the video loading. This means there are no dropped frames when using FileVideoStream. The file connection is automatically cleaned up upon exiting the code block.

To check if there are more frames left in the video, use the more() function. Alternatively, you can catch the NoMoreFrames exception when reading the next frame:

try:
  frame = video_stream.read()
except edgeiq.NoMoreFrames:
  # Do something

If you’d like the video to loop endlessly, this is a good spot to restart the video by calling the start() function again:

while True:
  try:
    frame = video_stream.read()
  except edgeiq.NoMoreFrames:
    video_stream.start()
    continue

Simulate a Real-Time Video Stream ¶

In the process of building your app, it can be useful to test against a video but simulate a real-time video stream like you would get with a webcam. This can be done by setting the play_realtime argument to True when instantiating the FileVideoStream object:

with edgeiq.FileVideoStream("video.mpeg", play_realtime=True) as video_stream:

This works by playing the video file back at the encoded FPS and dropping frames that aren’t read out in time.

Store Video Clips ¶

The VideoWriter class will save a video clip to a file. The input parameters are the output path at which to save the video file, the frame rate to save, whether to keep colors, and the codec to use. Start by instantiating the VideoWriter object, either by using a with statement, or by directly instantiating the object.

with edgeiq.VideoWriter(output_path="video_clip.avi") as video_writer:

or:

video_writer = edgeiq.VideoWriter(output_path="video_clip.avi")

For each frame that is ready to be added to the clip, use the write_frame() function.

video_writer.write_frame(frame)

For the context manager, the file will be written and closed when the code block exits. Otherwise, the close() function must be called.

Capture Video on an Event Trigger ¶

The EventVideoWriter makes it simple to capture specific events in video clips. When instantiating the object, choose the frames-per-second you’d like to encode the video at, and the amount of pre-roll and post-roll to record for each event.

# Capture 5 seconds of pre-roll and post-roll at 30 FPS
event_video_writer = edgeiq.EventVideoWriter(pre_roll=150, post_roll=150, fps=30)

Then for each frame call the update() function.

event_video_writer.update(frame)

When an event occurs, start recording the incoming frames, along with any pre-roll frames with the start_event() function.

if event_occurred:
  output_path="video-event{}.avi".format(event_index)
  event_video_writer.start_event(output_path=output_path, callback_function=cb, callback_args=(arg1, arg2))

The callback function is called with the provided arguments when the video capture is complete. Finally, when the event completes, finish saving the video file.

if event_complete:
  event_writer.finish_event()

Due to recording the post-roll, the video will not be saved immediately after finish_event() is called. Use the callback function to perform actions once the video capture completes. For example, to print out the path of the video that completed use the following callback function and arguments.

def cb(output_path):
  print("Video saved: {}".format(output_path))

...

if event_occurred:
  output_path="video-event{}.avi".format(event_index)
  event_video_writer.start_event(output_path=output_path, callback_function=cb, callback_args=(output_path))

The EventVideoWriter class intelligently handles calls to start_event() while it is already recording (during an event or post-roll). In those cases, the current video clip will be extended to capture the new event as well.

Publish a Video Stream ¶

The MjpgVideoWriter class can be used to publish an MJPG video stream that can be consumed by third-party apps such as VLC. The parameters enable you to control the latency and likelihood of dropped frames, as well as the output port and the image size and quality. Instantiating this class starts an additional process that runs the server to host the stream.

Manipulate Images and Video Frames ¶

Translate an Image ¶

To translate an image by shifting it on the x and y axes, use the translate() function.

translated_image = edgeiq.translate(image, x=x_shift, y=y_shift)

Rotate an Image ¶

To rotate an image about a specific point use the rotate() function. The center defaults to the actual center of the image, and the scale factor defaults to 1.0 resulting in no scaling.

rotated_image = edgeiq.rotate(image, angle=90)

Resize an Image ¶

To resize an image to be a specific width or height, use the resize() function; provide only the desired dimension and leave keep_scale set to True. For example, to resize an image so that its width is now new_width:

resized_image = edgeiq.resize(image, width=new_width)

To resize an image so that it fits in a space no larger than max_width and max_height, do the following:

resized_image = edgeiq.resize(image, width=max_width, height=max_height)

Display Bounding Boxes ¶

To display bounding boxes on an image, use the markup_image() function. The function will draw the bounding boxes on the image, optionally along with labels and confidences. Custom colors can be provided, and will be indexed according to the master label list of the model. One method to define custom colors would be to first create a dictionary of colors you’d like to set for specific labels. We’ll set “person” to blue, “pottedplant” to green, and all other detections to red. Note that the color values are in (B, G, R) format.

default_color = (0, 0, 255)
color_map = {
    "person": (255, 0, 0),
    "pottedplant": (0, 255, 0)
  }

Now make the full colors list to pass into markup_image().

colors = [color_map.get(label, default_color) for label in obj_detect.labels]

Finally, pass the color list into markup_image() and your detections will be displayed with your custom colors.

new_image = edgeiq.markup_image(image, predictions, colors=colors)

Cut Out a Segment of an Image ¶

A bounding box can also be used to cut out a segment of an image for further processing. The cutout_image() function takes a bounding box and returns the segment of the image outlined by the box.

image_segment = edgeiq.cutout_image(image, prediction.box)

Related Tutorials

How to Build a Simple Computer Vision Texting App Using Twilio