[MOT] Add BoT-SORT and MOT speed optimization (#7591)

* Add BOTSORTTracker
* pptracking/python/mot/tracker/botsort_tracker.py
* ppdet/modeling/mot/tracker/botsort_tracker.py

fix OCSORTTracker

* * botsort kalman_filter.py float64->float32
* requirements.txt add numba

* [Mot] kalman_filter.py lazy import numba
* add code source and modify comments

* [MOT] fix numba lazy import bug

* [MOT] botsort add README.md

* delete comment code

Author: LokeZhou

configs/mot/botsort/README.md

@@ -0,0 +1,89 @@
+English | [简体中文](README_cn.md)
+
+# BOT_SORT (BoT-SORT: Robust Associations Multi-Pedestrian Tracking)
+
+## content
+- [introduction](#introduction)
+- [model zoo](#modelzoo)
+- [Quick Start](#QuickStart)
+- [Citation](Citation)
+
+## introduction
+[BOT_SORT](https://arxiv.org/pdf/2206.14651v2.pdf)(BoT-SORT: Robust Associations Multi-Pedestrian Tracking). The configuration of common detectors is provided here for reference. Because different training data sets, input scales, number of training epochs, NMS threshold settings, etc. will lead to differences in model accuracy and performance, please adapt according to your needs
+
+## modelzoo
+
+### BOT_SORT在MOT-17 half Val Set
+
+|  Dataset      |  detector     | input size  | detector mAP  |  MOTA  |  IDF1  |  config |
+| :--------         | :-----      | :----:  | :------:  | :----: |:-----: |:----:   |
+| MOT-17 half train | PP-YOLOE-l  | 640x640 |  52.7    |  55.5  |  64.2 |[config](./botsort_ppyoloe.yml) |
+
+
+**Attention:**
+  - Model weight download link in the configuration file ` ` ` det_ Weights ` ` `, run the verification command to automatically download.
+  - **MOT17-half train** is a data set composed of pictures and labels of the first half frames of each video in the MOT17 train sequence (7 in total). To verify the accuracy, we can use the **MOT17-half val** to eval，It is composed of the second half frame of each video，download [link](https://bj.bcebos.com/v1/paddledet/data/mot/MOT17.zip)，decompression `dataset/mot/`
+
+  - BOT_ SORT training is a separate detector training MOT dataset, reasoning is to assemble a tracker to evaluate MOT indicators, and a separate detection model can also evaluate detection indicators.
+  - BOT_SORT export deployment is to export the detection model separately and then assemble the tracker for operation. Refer to [PP-Tracking](../../../deploy/pptracking/python)。
+  - BOT_SORT is the main scheme for PP Human, PP Vehicle and other pipelines to analyze the project tracking direction. For specific use, please refer to [Pipeline](../../../deploy/pipeline) and [MOT](../../../deploy/pipeline/docs/tutorials/pphuman_mot.md).
+
+
+## QuickStart
+
+### 1. train
+Start training and evaluation with the following command
+```bash
+#Single gpu
+CUDA_VISIBLE_DEVICES=0 python tools/train.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --eval --amp
+
+#Multi gpu
+python -m paddle.distributed.launch --log_dir=ppyoloe --gpus 0,1,2,3,4,5,6,7 tools/train.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --eval --amp
+```
+
+### 2. evaluate
+#### 2.1 detection
+```bash
+CUDA_VISIBLE_DEVICES=0 python tools/eval.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml
+```
+
+**Attention:**
+ - eval detection use ```tools/eval.py```，eval mot use ```tools/eval_mot.py```.
+
+#### 2.2 mot
+```bash
+CUDA_VISIBLE_DEVICES=0 python tools/eval_mot.py -c configs/mot/botsort/botsort_ppyoloe.yml --scaled=True
+```
+**Attention:**
+ - `--scaled` indicates whether the coordinates of the output results of the model have been scaled back to the original drawing. If the detection model used is JDE YOLOv3, it is false. If the universal detection model is used, it is true. The default value is false.
+ - mot result save `{output_dir}/mot_results/`,each video sequence in it corresponds to a txt, and each line of information in each txt file is `frame,id,x1,y1,w,h,score,-1,-1,-1`, and `{output_dir}` could  use `--output_dir` to set.
+
+### 3. export detection model
+
+```bash
+python tools/export_model.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --output_dir=output_inference -o weights=https://bj.bcebos.com/v1/paddledet/models/mot/ppyoloe_crn_l_36e_640x640_mot17half.pdparams
+```
+
+### 4. Use the export model to predict
+
+```bash
+# download demo video
+wget https://bj.bcebos.com/v1/paddledet/data/mot/demo/mot17_demo.mp4
+
+CUDA_VISIBLE_DEVICES=0 python deploy/pptracking/python/mot_sde_infer.py --model_dir=output_inference/ppyoloe_crn_l_36e_640x640_mot17half --tracker_config=deploy/pptracking/python/tracker_config.yml --video_file=mot17_demo.mp4 --device=GPU --threshold=0.5
+```
+**Attention:**
+ - You must fix `tracker_config.yml` tracker `type: BOTSORTTracker`，if you want to use BOT_SORT.
+ - The tracking model is used to predict videos. It does not support prediction of a single image. By default, the videos with visualized tracking results are saved. You can add `--save_mot_txts` (save a txt for each video) or `--save_mot_txt_per_img`(Save a txt for each image)  or `--save_images` save the visualization picture of tracking results.
+ - Each line of the trace result txt file format `frame,id,x1,y1,w,h,score,-1,-1,-1`。
+
+
+## Citation
+```
+@article{aharon2022bot,
+  title={BoT-SORT: Robust Associations Multi-Pedestrian Tracking},
+  author={Aharon, Nir and Orfaig, Roy and Bobrovsky, Ben-Zion},
+  journal={arXiv preprint arXiv:2206.14651},
+  year={2022}
+}
+```

configs/mot/botsort/README_cn.md

@@ -0,0 +1,89 @@
+简体中文 | [English](README.md)
+
+# BOT_SORT (BoT-SORT: Robust Associations Multi-Pedestrian Tracking)
+
+## 内容
+- [简介](#简介)
+- [模型库](#模型库)
+- [快速开始](#快速开始)
+- [引用](#引用)
+
+## 简介
+[BOT_SORT](https://arxiv.org/pdf/2206.14651v2.pdf)(BoT-SORT: Robust Associations Multi-Pedestrian Tracking)。此处提供了常用检测器的配置作为参考。由于训练数据集、输入尺度、训练epoch数、NMS阈值设置等的不同均会导致模型精度和性能的差异，请自行根据需求进行适配。
+
+## 模型库
+
+### BOT_SORT在MOT-17 half Val Set上结果
+
+|  检测训练数据集      |  检测器     | 输入尺度  | 检测mAP  |  MOTA  |  IDF1  |  配置文件 |
+| :--------         | :-----      | :----:  | :------:  | :----: |:-----: |:----:   |
+| MOT-17 half train | PP-YOLOE-l  | 640x640 |  52.7    |  55.5  |  64.2 |[配置文件](./botsort_ppyoloe.yml) |
+
+
+**注意:**
+  - 模型权重下载链接在配置文件中的```det_weights```，运行验证的命令即可自动下载。
+  - **MOT17-half train**是MOT17的train序列(共7个)每个视频的前一半帧的图片和标注组成的数据集，而为了验证精度可以都用**MOT17-half val**数据集去评估，它是每个视频的后一半帧组成的，数据集可以从[此链接](https://bj.bcebos.com/v1/paddledet/data/mot/MOT17.zip)下载，并解压放在`dataset/mot/`文件夹下。
+
+  - BOT_SORT的训练是单独的检测器训练MOT数据集，推理是组装跟踪器去评估MOT指标，单独的检测模型也可以评估检测指标。
+  - BOT_SORT的导出部署，是单独导出检测模型，再组装跟踪器运行的，参照[PP-Tracking](../../../deploy/pptracking/python)。
+  - BOT_SORT是PP-Human和PP-Vehicle等Pipeline分析项目跟踪方向的主要方案，具体使用参照[Pipeline](../../../deploy/pipeline)和[MOT](../../../deploy/pipeline/docs/tutorials/pphuman_mot.md)。
+
+
+## 快速开始
+
+### 1. 训练
+通过如下命令一键式启动训练和评估
+```bash
+#单卡训练
+CUDA_VISIBLE_DEVICES=0 python tools/train.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --eval --amp
+
+#多卡训练
+python -m paddle.distributed.launch --log_dir=ppyoloe --gpus 0,1,2,3,4,5,6,7 tools/train.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --eval --amp
+```
+
+### 2. 评估
+#### 2.1 评估检测效果
+```bash
+CUDA_VISIBLE_DEVICES=0 python tools/eval.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml
+```
+
+**注意:**
+ - 评估检测使用的是```tools/eval.py```, 评估跟踪使用的是```tools/eval_mot.py```。
+
+#### 2.2 评估跟踪效果
+```bash
+CUDA_VISIBLE_DEVICES=0 python tools/eval_mot.py -c configs/mot/botsort/botsort_ppyoloe.yml --scaled=True
+```
+**注意:**
+ - `--scaled`表示在模型输出结果的坐标是否已经是缩放回原图的，如果使用的检测模型是JDE YOLOv3则为False，如果使用通用检测模型则为True, 默认值是False。
+ - 跟踪结果会存于`{output_dir}/mot_results/`中，里面每个视频序列对应一个txt，每个txt文件每行信息是`frame,id,x1,y1,w,h,score,-1,-1,-1`, 此外`{output_dir}`可通过`--output_dir`设置。
+
+### 3. 导出预测模型
+
+```bash
+python tools/export_model.py -c configs/mot/bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml --output_dir=output_inference -o weights=https://bj.bcebos.com/v1/paddledet/models/mot/ppyoloe_crn_l_36e_640x640_mot17half.pdparams
+```
+
+### 4. 用导出的模型基于Python去预测
+
+```bash
+# 下载demo视频
+wget https://bj.bcebos.com/v1/paddledet/data/mot/demo/mot17_demo.mp4
+
+CUDA_VISIBLE_DEVICES=0 python deploy/pptracking/python/mot_sde_infer.py --model_dir=output_inference/ppyoloe_crn_l_36e_640x640_mot17half --tracker_config=deploy/pptracking/python/tracker_config.yml --video_file=mot17_demo.mp4 --device=GPU --threshold=0.5
+```
+**注意:**
+ - 运行前需要手动修改`tracker_config.yml`的跟踪器类型为`type: BOTSORTTracker`。
+ - 跟踪模型是对视频进行预测，不支持单张图的预测，默认保存跟踪结果可视化后的视频，可添加`--save_mot_txts`(对每个视频保存一个txt)或`--save_mot_txt_per_img`(对每张图片保存一个txt)表示保存跟踪结果的txt文件，或`--save_images`表示保存跟踪结果可视化图片。
+ - 跟踪结果txt文件每行信息是`frame,id,x1,y1,w,h,score,-1,-1,-1`。
+
+
+## 引用
+```
+@article{aharon2022bot,
+  title={BoT-SORT: Robust Associations Multi-Pedestrian Tracking},
+  author={Aharon, Nir and Orfaig, Roy and Bobrovsky, Ben-Zion},
+  journal={arXiv preprint arXiv:2206.14651},
+  year={2022}
+}
+```

configs/mot/botsort/botsort_ppyoloe.yml

@@ -0,0 +1,75 @@
+# This config is an assembled config for ByteTrack MOT, used as eval/infer mode for MOT.
+_BASE_: [
+  '../bytetrack/detector/ppyoloe_crn_l_36e_640x640_mot17half.yml',
+  '../bytetrack/_base_/mot17.yml',
+  '../bytetrack/_base_/ppyoloe_mot_reader_640x640.yml'
+]
+weights: output/botsort_ppyoloe/model_final
+log_iter: 20
+snapshot_epoch: 2
+
+metric: MOT # eval/infer mode, set 'COCO' can be training mode
+num_classes: 1
+
+architecture: ByteTrack
+pretrain_weights: https://bj.bcebos.com/v1/paddledet/models/ppyoloe_crn_l_300e_coco.pdparams
+ByteTrack:
+  detector: YOLOv3 # PPYOLOe version
+  reid: None
+  tracker: BOTSORTTracker
+det_weights: https://bj.bcebos.com/v1/paddledet/models/mot/ppyoloe_crn_l_36e_640x640_mot17half.pdparams
+reid_weights: None
+
+YOLOv3:
+  backbone: CSPResNet
+  neck: CustomCSPPAN
+  yolo_head: PPYOLOEHead
+  post_process: ~
+
+# Tracking requires higher quality boxes, so NMS score_threshold will be higher
+PPYOLOEHead:
+  fpn_strides: [32, 16, 8]
+  grid_cell_scale: 5.0
+  grid_cell_offset: 0.5
+  static_assigner_epoch: -1 # 100
+  use_varifocal_loss: True
+  loss_weight: {class: 1.0, iou: 2.5, dfl: 0.5}
+  static_assigner:
+    name: ATSSAssigner
+    topk: 9
+  assigner:
+    name: TaskAlignedAssigner
+    topk: 13
+    alpha: 1.0
+    beta: 6.0
+  nms:
+    name: MultiClassNMS
+    nms_top_k: 1000
+    keep_top_k: 100
+    score_threshold: 0.1 # 0.01 in original detector
+    nms_threshold: 0.4 # 0.6 in original detector
+
+
+BOTSORTTracker:
+  track_high_thresh: 0.3
+  track_low_thresh: 0.2
+  new_track_thresh: 0.4
+  match_thresh: 0.7
+  track_buffer: 30
+  min_box_area: 0
+  camera_motion: False
+  cmc_method: 'sparseOptFlow' # only camera_motion is True,
+                              # sparseOptFlow | files (Vidstab GMC) | orb | ecc
+
+
+# MOTDataset for MOT evaluation and inference
+EvalMOTDataset:
+  !MOTImageFolder
+    dataset_dir: dataset/mot
+    data_root: MOT17/images/half
+    keep_ori_im: True # set as True in DeepSORT and ByteTrack
+
+TestMOTDataset:
+  !MOTImageFolder
+    dataset_dir: dataset/mot
+    keep_ori_im: True # set True if save visualization images or video

configs/mot/ocsort/ocsort_ppyoloe.yml

@@ -60,13 +60,14 @@ OCSORTTracker:
   vertical_ratio: 0
   min_box_area: 0
   use_byte: False
+  use_angle_cost: False
 
 
 # MOTDataset for MOT evaluation and inference
 EvalMOTDataset:
   !MOTImageFolder
     dataset_dir: dataset/mot
-    data_root: MOT17/images/half
+    data_root: MOT17_test/images/half
     keep_ori_im: True # set as True in DeepSORT and ByteTrack
 
 TestMOTDataset:

deploy/pptracking/python/mot/matching/ocsort_matching.py

@@ -57,7 +57,8 @@ def linear_assignment(cost_matrix):
     try:
         import lap
         _, x, y = lap.lapjv(cost_matrix, extend_cost=True)
-        return np.array([[y[i], i] for i in x if i >= 0])  #
+        match = np.array([[y[i], i] for i in x if i >= 0])
+        return match
     except ImportError:
         from scipy.optimize import linear_sum_assignment
         x, y = linear_sum_assignment(cost_matrix)
@@ -125,3 +126,44 @@ def associate(detections, trackers, iou_threshold, velocities, previous_obs,
         matches = np.concatenate(matches, axis=0)
 
     return matches, np.array(unmatched_detections), np.array(unmatched_trackers)
+
+
+def associate_only_iou(detections, trackers, iou_threshold):
+    if (len(trackers) == 0):
+        return np.empty(
+            (0, 2), dtype=int), np.arange(len(detections)), np.empty(
+                (0, 5), dtype=int)
+
+    iou_matrix = iou_batch(detections, trackers)
+
+    if min(iou_matrix.shape) > 0:
+        a = (iou_matrix > iou_threshold).astype(np.int32)
+        if a.sum(1).max() == 1 and a.sum(0).max() == 1:
+            matched_indices = np.stack(np.where(a), axis=1)
+        else:
+            matched_indices = linear_assignment(-iou_matrix)
+    else:
+        matched_indices = np.empty(shape=(0, 2))
+
+    unmatched_detections = []
+    for d, det in enumerate(detections):
+        if (d not in matched_indices[:, 0]):
+            unmatched_detections.append(d)
+    unmatched_trackers = []
+    for t, trk in enumerate(trackers):
+        if (t not in matched_indices[:, 1]):
+            unmatched_trackers.append(t)
+
+    # filter out matched with low IOU
+    matches = []
+    for m in matched_indices:
+        if (iou_matrix[m[0], m[1]] < iou_threshold):
+            unmatched_detections.append(m[0])
+            unmatched_trackers.append(m[1])
+        else:
+            matches.append(m.reshape(1, 2))
+    if (len(matches) == 0):
+        matches = np.empty((0, 2), dtype=int)
+    else:
+        matches = np.concatenate(matches, axis=0)
+    return matches, np.array(unmatched_detections), np.array(unmatched_trackers)

deploy/pptracking/python/mot/motion/__init__.py

@@ -15,3 +15,5 @@
 from . import kalman_filter
 
 from .kalman_filter import *
+from .gmc import *
+from .ocsort_kalman_filter import *