Skip to content

asmirnou/watsor

BranchesTags

Repository files navigation

logo

Watsor

Watsor detects objects in video stream using deep learning-based approach. Intended primarily for surveillance it works in sheer real-time analysing the most recent frame to deliver the fastest reaction against a detected threat.

Table of contents

What it does

  • Performs smart detection based on artificial neural networks significantly reducing false positives in video surveillance.
  • Capable to limit detectionzonesusingmask imagewith alpha channel.
  • Supports multiple hardware accelerators such asThe Coral USB AcceleratorandNvidia CUDA GPUsto speed up detection algorithms.
  • Reports the detected objects viaMQTTprotocol primarily for integration withHomeAssistant.
  • Allows controlling video decoder using the commands published over MQTT.
  • Broadcasts video stream with rendered object detections inMPEG-TSandMotion JPEGformats over HTTP.
  • Captures video from any source and encodes video with rendered object detections in any format supported byFFmpeg.

Being applicable in CCTV, Watsor also suits other areas, where object detection in video stream is required.

Getting started

Watsor is written in Python 3 and ships mainly asPython module.The easiest way, however, is trying it inDockeras far as hardware drivers and models are bundled in the images.

Regardless of how you run Watsor, you need to prepare configuration file, describing the cameras you've got and few other options such as types of detections and zones. Refer to the guide below.

Configuration

Watsor uses theYAMLsyntax for configuration. The basics of YAML syntax are block collections and mappings containing key-value pairs. Each item in a collection starts with a-while mappings have the formatkey: value.Watsor performs config validation and, if you specified duplicate keys, missed some values or have wrong YAML syntax, prints the error message at the start.

Please find an exampleconfigurationfile with explanatory comments that you can use as a template.

Cameras

camerasblock in document root is a list of cameras you gonna use for surveillance. Each camera must have unique name distinguishing it from others. The name of the camera appears in logs and also is included in path to a video stream over HTTP.

Expand code snippet 
cameras:
-camera1:
width:640
height:480
input:rtsp://192.168.0.10:554
#output:!ENV "${HOME}/Videos/camera1.m3u8"
#mask: camera1.png
#detect: []
#ffmpeg: []
-...

A camera must specifywidth,heightandinputsource of a video feed. Refer to the camera setting to figure out the proper values.widthandheightare supposed to reflect the resolution of the actual video feed. They don’t change the size of the video, but are used to configure the frame buffers and other things needed for Watsor to work. Ifwidthorheightvalues do not match video feed, the image can’t be displayed correctly. The input source URL usually starts withrtsp://orhttp://,but can be anything FFmpeg can consume, a.mp4file for instance.

Optionalmaskis a filename of a mask image used to limit detection zone. Refer to thefollowingsection.

Optionaldetectblock overridesthe defaultsto specify the types of objects of interest and set of filters to sift less significant detections.

Optionalffmpegblock overridesthe defaultsto specify the arguments for FFmpeg application used to decode input video stream and also to encode the output when necessary.ffmpeg.encoderis absolutely optional, define it only if you'd like to record or broadcast video stream with rendered object detections.

Whenffmpeg.encoderis used to record video file, useoutputkey to specify the location. Refer toFFmpeg formatsfor all available options. Whenoutputkey is absent, Watsor runs lightweight HTTP server for streaming the video over the network. The streaming is performed inMPEG-TSformat, so make surempegts formatis set inffmpeg.encoderoutput arguments. Whenoutputis set the encoded stream is being written to file and the broadcast in MPEG-TS over HTTP is not possible.

Regardless of whether or notffmpeg.encoderis enabled andoutputis defined, the broadcast of video stream inMotion JPEGformat over HTTP is available all the time.

FFmpeg decoder and encoder

ffmpegblock present in document root is used as defaults for decoding and encoding a video stream if a camera does not set its own.

FFmpegis a very fast video converter that grabs a live video from source. Watsor runs one or two FFmpeg subprocesses for each camera: mandatory decoder and optional encoder. The decoder reads from the source specified by the camera'sinputkey (which can be an URL of camera feed, a regular file or a grabbing device), following the-ioption. The encoder writes to the output file, if camera'soutputis set, orstdout,if not. The options in the command line before and after-idetermine the parameters for input and output of each FFmpeg subprocess. The input parameters, for example, can enable hardware accelerated decoding. The output of decoder and the input of encoder must include-f rawvideo -pix_fmt rgb24as Watsor reads raw video frames (24-bit) from first FFmpeg'sstdoutand after detection complete can write raw video frames instdinof another FFmpeg subprocess.

You must include-ioption in the command line listnot followedby the input as Watsor includes camera'sinputkey after-iautomatically. For encoder it also includes other required options such as-sto define explicitly the size of raw video frames.

Expand code snippet 
ffmpeg:
decoder:
-...
--i
--f
-rawvideo
--pix_fmt
-rgb24
encoder:
--f
-rawvideo
--pix_fmt
-rgb24
--i
--f
-mpegts
-...

ffmpeg.encoderis optional and intended for the recording or broadcasting of video stream with rendered object detections. The recording is suitable for on demand streaming as the video is stored in a file such as.m3u8(HLS) or.mp4and can be re-watched. The broadcasting means the video is being recorded in real time and the viewer can only watch it as it is streaming. To broadcast the video with rendered object detections over HTTP set-f mpegtsoption in the encoder and remove the camera'soutputkey. The link to the video stream can be grabbed from Watsor's home page and opened in media player such asVLC.

When broadcasting live video stream in MPEG-TS be aware of noticeablelatency,which is unavoidable due to the nature of video encoding algorithm. Bear in mind the media player introduces a latency of its own as it buffers the feed. To watch the video with rendered object detections in sheer real-time with zero latency open Motion JPEG URL of the camera feed.

Broadcasting in MPEG-TS format is perfect for streaming over the network as the video sequencing compression transports only the changes in the sequence thus uses less network bandwidth (and storage). Motion JPEG compresses each video frame into a JPEG image making them available as a continuous flow of images over a network. As all frames are compressed independently and entirely, the amount of data transmitted across the network is far more than that of MPEG-TS.

Detection classes and filters

detectblock present in document root is used as defaults if a camera does not set its own. The block is a list ofCOCO classes(90 at max) used for detection each specifying three filters:area,confidenceandzones.

Expand code snippet 
detect:
-person:
area:20
confidence:60
zones:[1, 3, 5]
-car:
-...
  • Theareais the minimum area of the bounding box an object should have in order to be detected. Defaults to 10% of entire video resolution if not set.
  • confidenceis a minimum threshold that a detection is what it's guessed to be, otherwise it's ruled out. 50% if not set.
  • zonesis a list of indexes designating the zones on mask image, where detection is allowed. Not set by default meaning all zones are allowed.

Zones and masks

To limit detection zonestake a snapshotfrom your camera. The width and height of the image must be of the same size as your video feed.

Open the image in graphics editor such as GNU Image Manipulation Program (GIMP) and select the desired zone. Make the selection floating, then create a new layer from the given floating selection. That layer represents a zone, outside of which the objects are not being detected.

Select main layer with the rest of the image and change theopacityof the layer to~85%.The opacity of the layers from floating selection must remain100%to let Watsor distinct them from the background. Save image in PNG 32-bit format with alpha channel. Specify the file name and path (can be relative) in camera configuration.

The following depicts said above:

You can select many zones and of any shape, but make sure they don't overlap, otherwise several zones merge in one.

When thebounding boxof an object intersects with the shape of a zone, detection occurs and the zone highlights in yellow. The index of a zone is then transmitted over MQTT. The zones are indexed in mask image depending on the proximity of their center to the origin. On the sample above there are two zones and a car was detected in2(shown in red). If not very clear of what zone is the index, the following tool will display it for you:

python3 -m watsor.zones -m config/porch.png

Tips

  • Not usinghardware acceleratorresults in high CPU load. In order to reduce CPU load change theframe rateof an input video stream either in the setings of your camera or using FFmpegfilteras follows:

    FPS filter 
    --filter:v
-fps=fps=10

    Ideally, the input frame rate of the cameras should not exceed the capabilities of the detectors, otherwise all available CPU units will be used for detection.

  • Unless you record video with rendered object detections choosesmaller resolutionin camera settings. 300x300 is the size of the images used to train the object detection model. During detection the framework automatically converts an input image to the size of the model to match the size of its tensors. As the resize happens anyway during the detection, feeding the stream of high resolution doesn’t make much sense unless the output stream the Watsor produces (the size of which matches the input resolution) is being recorded for later review (where the high resolution video is obviously desired).

    Sometimes camera doesn't provide much options to change resolution. Use FFmpegscale filterthen:

    scale filter 
    --filter:v
-scale=640:480

    If you need to combine scale filter with FPS filter, separate them withcommas:

    two filters together 
    --filter:v
-fps=fps=10,scale=640:480

  • Consider configuringhardware accelerationfor decoding H.264 video (or whatever your camera produces) in FFmpeg. The command line options withhwaccelprefix are right for that. Refer to the followingwikito learn what methods are available on your device.

  • Playing the video in VLC (or other player) require constant rate of 25 frames / sec. Having many cameras or very few detectors processing all video sources sometimes can not provide such an output speed. Let's say you've got 2 cameras each producing 30 FPS and only one CPU-based detector capable to cope only with 25 FPS. The output speed of MPEG-TS video stream then will be only 12.5 FPS for each camera, resulting in jerks and pauses while viewing a video. To make a video fluent, the frame rate has to be changed, such that the output frame rate is higher than the input frame rate.

    The following trick may be used (expand code snippet) 
    encoder:
--hide_banner
--f
-rawvideo
--pix_fmt
-rgb24
--r
-10
--vsync
-drop
--i
--an
--f
-mpegts
--r
-30000/1001
--vsync
-cfr
--vcodec
-libx264
--pix_fmt
-yuv420p

    First the rate is lowered even more down to 10 FPS in order to guarantee constant feed for FFmpeg encoder (-r 10). The frames exceeding 10 FPS are dropped (-vsync drop) in order to match the target rate. Then output speed is set to be standard30000/1001(~30 FPS) and constant (-vsync cfr) to produce fluent video stream, duplicating frames as necessary to achieve the targeted output frame rate.

Environment variables

You can include values from your system’s environment variables either likeHome Assistant Core does:

password:!env_var PASSWORD default_passwordor

input:!ENV "rtsp://${RTSP_USERNAME}:${RTSP_PASSWORD}@192.168.0.10:554"

Secrets

You can remove any private information from your configuration file. The entries for password can be replaced with!secretkeyword followed by an identifier. The password will be then looked atsecrets.yamlfile containing the corresponding password assigned to the identifier:

config.yaml 
mqtt:
username:!secret mqtt_username
password:!secret mqtt_password
secrets.yaml 
mqtt_username:"john"
mqtt_password:"qwerty"

Asecrets.yamlwill be resolved first by looking in the same folder as the YAML file referencing the secret, next, parent folders will be searched for asecrets.yamlfile. The logic inherits from Home Assistant and you can place your config along with HA files reusing singlesecrets.yamlfile for both apps.

HomeAssistant integration

The easiest way to get started is using Watsoradd-onsfor Home Assistant.

Watsor can communicate withHomeAssistantviaMQTT- "Internet of Things" connectivity protocol. Please refer todemo projectto examine the configuration files.

To configure optional MQTT client add the following lines (expand the snippet): 
mqtt:
host:localhost
#port: 1883
#username:!secret mqtt_username
#password:!secret mqtt_password

List of MQTT topics:

  • watsor/cameras/{camera}/available
  • watsor/cameras/{camera}/command
  • watsor/cameras/{camera}/sensor
  • watsor/cameras/{camera}/state
  • watsor/cameras/{camera}/detection/{class}/state
  • watsor/cameras/{camera}/detection/{class}/details

The binary state (ON/OFF) of a detected object class is published at/detection/{class}/statetopic, confirming the state every 10 seconds. This signals about a detected threat and is supposed to trigger an alarm.

Subscribe to the topicavailableto receive availability (online/offline) updates about specific camera. The camera isonlinewhen Watsor starts and goesofflinewhen the application stops.

The camera can be controlled via topiccommandallowing to start/stop the decoder, limit frame rate and enable/disable the reporting of detection details.

  • When alarm control panel is armed, sendONto start the decoder and detection processes. When disarmed - sendOFFto stop analysing the camera feed. The camera notifies about its running state through thestatetopic.
  • If nothing is detected for more than 30 seconds, you can slow down the camera by sendingFPS = 5(or whatever rate you prefer) to limit frame rate in order to reduce the load on CPU or hardware accelerated detector. The camera will reset FPS limit itself and will reach full speed as soon as something's detected.
  • The reporting of detection details including confidence, coordinates of the bounding box and zone index along with the timestamp of the frame can be turned on / off by sendingdetails = on/details = offaccordingly. The details are published at another topic in JSON format as follows: 
    {
"t":"2020-06-27T17:41:21.681899",
"d":[{
"c":73.7,
"b":[54,244,617,479]
}]
}
    wheretis timestamp of a frame,d- array of all detections of the given class of an object designated in topic path. Each detection has confidencecand bounding boxbconsisting ofx_min,y_min,x_max,y_max.

The topicsensoris used to send the updates about camera current input and output speeds. Monitoring the speed is useful to trigger the alert, if camera is broken or disconnected suddenly.

The sensor values and detection details can be transmitted over MQTT very often up to tens times per second. Therecorderintegration in HomeAssistant constantly saves data, storing everything by default from sensors to state changes. Fortunately, HomeAssistant allows to customize what needs to be written and not. A good idea is to include in recoder only those measurements that are really needed to avoid degradation of HomeAssistant's performance.

Running Watsor

Docker

To run Watsor in Docker mountconfigurationfiles at/etc/watsorfolder of a container. Add host devices for video and detection hardware acceleration (if available). As far as Watsor runs several processes, which share the memory, increase the default shared memory size of 64m to a bit more depending on the number of cameras. Here is example ofdocker-compose.yaml:

Expand code snippet 
version:'3'

services:
watsor:
container_name:watsor
image:smirnou/watsor:latest
environment:
-LOG_LEVEL=info
volumes:
-/etc/localtime:/etc/localtime:ro
-../config:/etc/watsor:ro
devices:
-/dev/bus/usb:/dev/bus/usb
-/dev/dri:/dev/dri
ports:
-8080:8080
shm_size:512m

To run GPU acceleratedwatsor.gpuDocker image use theNVIDIA Container Toolkit. Pass--gpus allflag to add GPU devices to the container:

Expand code snippet 
docker run -t -i \
--rm \
--env LOG_LEVEL=info \
--volume /etc/localtime:/etc/localtime:ro \
--volume$PWD/config:/etc/watsor:ro \
--device /dev/bus/usb:/dev/bus/usb \
--device /dev/dri:/dev/dri \
--publish 8080:8080 \
--shm-size=512m \
--gpus all \
smirnou/watsor.gpu:latest

If your GPU supports Half precision (also known as FP16), you can boost performance by enabling this mode as follows:

docker run --gpus all --env TRT_FLOAT_PRECISION=16...

Models for CPU/GPU and EdgeTPU (Coral) are bundled in Docker images. You can use your own models, trained based on those listed inobject detection modelssection, by mounting the volume at/usr/share/watsor/model.

The following table lists the available docker images:

Image Suitable for
watsor x86-64
watsor.gpu x86-64 with Nvidia CUDA GPU
watsor.jetson Jetson devices (Xavier, TX2, and Nano)
watsor.pi3 Raspberry PI 3 or 4 with 32-bit OS
watsor.pi4 Raspberry PI 4 with 64-bit OS

Kubernetes

To deploy Watsor on Kubernetes cluster useHelm chart:

helm repo add asmirnou https://asmirnou.github.io/watsor-helm-chart
helm repo update
helm install watsor asmirnou/watsor

Python module

Watsor works well on Pyhton 3.6, 3.7, 3.8. Use avirtual environmentwhen installing Python packages.

  1. Install module:

    python3 -m pip install watsor

    If you've got ahardware acceleratoror are installing the application on a tiny board like Raspberry Pi, take a look atextraprofilescoral,cudaorliteinsetup.py.The dependencies listed in those profiles need to be installed in advance. Refer to the documentation of the device or take a look at one of theDocker filesto understand how the dependencies are installed.

  2. Createmodel/folder, download, unpack and prepare theobject detection models(see the section below).

  3. Write the config file by following theguideabove, takethis configas an example.

  4. Make sureffmpegis installed on your operating system.

  5. Run watsor as follows:

    python3 -m watsor.main --config config/config.yaml --model-path model/

    or if NVIDIA CUDA GPU is present andproperly set up:

    python3 -m watsor.main_for_gpu --config config/config.yaml --model-path model/

Openhttp://localhost:8080to navigate to a simple home page, where you'll find the links to the cameras video streams, snapshots of object detected classes and metrics.

Object detection models

Watsor uses convolutional neural network trained to recognize 90 classes of object. The detection model has several types providing the trade-off between accuracy and speed. For example,MobileNet v1is the fastest, but less accurate thanInception v2.The models recommended below are reasonable for video surveillance in real-time, however you are not limited to use only these models as Watsor supports any one fromTensorFlow Model Garden.It's worth trying their newest models with higher average precision.

The models are available in several formats depending on the device the inference is being performed on.

  • If you've gotThe Coral USB Acceleratordownload one of the models built for Edge TPU (MobileNet v1/v2), rename the file and put inmodel/folder asedgetpu.tflite.

  • HaveNvidia CUDA GPUon board - download one of the models, rename the file and put inmodel/folder asgpu.uff.Watsor also works with TF2 models in ONNX format, these you need to convert yourself usingthe following guide,then save asmodel/gpu.onnx.

  • CPU is used only when there are no accelerators or their models provided. Inside of TensorFlow archive you will find several files, you only needfrozen_inference_graph.pbrenamed ascpu.pband placed in themodel/folder. TF2 models do not have frozen graph, but there is asaved_modelfolder that needs to be copied in full to themodel/folder of Watsor.

    Please note thatsaved modeltakes longer to start. If TensorFlow is configured properly, GPU will be involved.

  • For single board computer such as Raspberry Pi orJetson Nanolightweight model is more efficient. Download and unpack if needed, rename the file and put inmodel/folder ascpu.tflite.

Device Filename MobileNet v1 MobileNet v2 Inception v2
Coral model/edgetpu.tflite MobileNet v1 MobileNet v2 N/A
Nvidia CUDA GPU model/gpu.uff MobileNet v1 MobileNet v2 Inception v2
CPU model/cpu.pb MobileNet v1 MobileNet v2 Inception v2
Raspberry Pi & others model/cpu.tflite MobileNet v1 MobileNet v2 N/A

Hardware acceleration drivers

Use of hardware accelerator is optional, but highly recommended as object detection as such requires much computation. Speaking of number of frames per second the CPU of a desktop computer can process just 24 FPS of the lightest model, but an accelerator can boost the performance up to 5 times. Two accelerators connected or installed can handle 200 FPS, which is more than enough for handling several video cameras.

Watsor supportsmultipleaccelerators, equally balancing the load among them. Having at least one accelerator connected Watsor uses the hardware for computation, less loading the CPU. It falls over to CPU running Tensorflow if no accelerator is available.

To restrict the devices that Watsor sees setCORAL_VISIBLE_DEVICESorCUDA_VISIBLE_DEVICESenvironmental variables to a comma-separated list of device IDs to make only those devices visible to the application. The device ID can be known from application logs after enabling debug mode with the following command-line option:--log-level debug

If you've got the Coral install theEdge TPU runtimeandPyCoral API.

HaveNvidia CUDA GPUon board - install the following components (now the hard part, you'd better considerDocker:

Building from source

Clone Git repository:

git clone https://github.com/asmirnou/watsor.git

Create virtual environment and install Watsor dependencies:

make venv

sourcevenv/bin/activate

make install

Troubleshooting

If you encounter issues installing Watsor prerequisites, try to seek for solution over the web first. I'm glad to help, but most of the times the answer is being composed from the findings onhttps://stackoverflow.com/and etc. Be bold andfile an issuehaving not found a solution.

Credits

License

MIT License

About

Object detection for video surveillance

Topics

python mqtt video stream camera ffmpeg surveillance gpu detection realtime cuda ip mpegts hardware-acceleration zones homeassistant coral tensorrt tensrflow person-detector

Resources

Readme

License

MIT license
Activity

Stars

254 stars

Watchers

15 watching

Forks

32 forks
Report repository

Releases 7

Tensorrt 8 support Latest
Nov 14, 2023
+ 6 releases

Packages

No packages published

Languages