案例｜边缘智能：EdgeX 集成 OpenVINO™ Model Server 之目标检测

边缘智能：EdgeX 集成 OpenVINO™ Model Server 之目标检测，可以用在很多场景中，如：入侵检测，安全合规，杂物堆放，物品挪动等。

本文使用到的一些软硬件信息：

EdgeXFoundry（V3.1.0）
OpenVINO™ 和 OVMS（OpenVINO™ Model Server）v2024.0.0
SSDLITE MobileNet V2 模型（来自 Intel Model Zoo）
Device Services for OVMS
- device-openvino-ssdlite-mobilenet-v2（亿琪软件版权所有）
- 其他模型微服务
Intel® CPU（内置 iGPU）
- Intel® Celeron 赛扬
  - CPU: Intel® Celeron 6305E, 2 Cores @ 1.80 GHz
  - iGPU: Intel® UHD Graphics 1.25 GHz, 48 EU(Execution Units)

以上信息除了 device-openvino-ssdlite-mobilenet-v2 以外，都可以在网络上查询到，请自行查阅相关详细内容。

设计架构图

系统架构

本系统设计硬件设计采用 Intel® CPU(iGPU) + GPU[可选] ，CPU 必须为 Intel® 架构，独立 GPU 可根据实际需要灵活扩展。

本系统全部采用 Docker 微服务运行，描述如下：

图例中，① 作为 AI 推理服务器，运行 OpenVINO™ Model Server 容器；
图例中，② 作为流媒体服务器，负责流媒体的编解码，实时查看视频流，运行亿琪软件产品 YiMEDIA，同样也采用容器运行；
图例中，③ 作为边缘计算，运行 EdgeXFoundry 容器，负责整个系统的协调和业务驱动；

软硬件使用情况（有图有真相）

Intel CPU Spec

推理结果演示

通过 Web MJPEG 方式实时查看推理结果，例如： http://localhost:18080/OpenVINO-HK2

Intel CPU 型号和 iGPU 使用情况

推理进行时，查看 iGPU 使用情况，通过运行命令 intel_gpu_top 即可查看到 iGPU 使用率。

准备工作

硬件环境

准备一台可以作为整个系统运行的硬件设备，笔者使用了开头提到的两种类型的硬件设备分别进行了测试。

Intel 赛扬处理器

Intel Celeron 6305E 主机一台。下图中，右边长方形的芯片就是 iGPU，

工业整机

选用深圳铂盛公司产的 AIPC 来完成这个任务，非常合适。可以根据业务需要，更换 CPU 和加装其他辅助配件。

软件环境

操作系统和硬件配置

采用 Ubuntu 22.04 作为 Docker 宿主主机，并且已经成功完成 Docker 运行环境的安装和测试。

# uname -a
Linux YiFUSION-N100 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb  9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy

采用 CPU 内置 iGPU 作为推理

# ll /dev/dri/
total 0
drwxr-xr-x   3 root root        100  2月 23 17:34 ./
drwxr-xr-x  20 root root       4620  2月 23 20:19 ../
crw-rw----+  1 root video  226,   0  2月 23 20:58 card0
crw-rw----+  1 root render 226, 128  2月 23 17:39 renderD128

① OpenVINO™ 服务

OpenVINO™ Model Server 运行环境

OpenVINO™ 相关配置 Configuration JSON

模型库配置：

name: 模型名称
base_path: 模型文件路径
target_device: 目标推理设备
layout: 需要的图片格式(可选)

# more models/config.json
{
  "model_config_list":[
      {
        "config":{
            "name":"ssdlite_mobilenet_v2",
            "base_path":"/models/ssdlite_mobilenet_v2",
            "target_device": "GPU"
        }
      },
  ]
}

OpenVINO™ 相关配置 Local Model Repository

既可以用云存储作为模型库存放位置，也可以使用本地磁盘作为模型库存放位置（本例子使用）。

一级目录是对应的模型名称
二级目录是模型的版本，比如：1，2...
xml 是 OpenVINO™ 模型库相关配置，bin 是 OpenVINO™ 模型 IR 文件

models 目录结构如下：

models/
├── ssdlite_mobilenet_v2
│   └── 1
│       ├── coco_91cl_bkgr.txt
│       ├── ssdlite_mobilenet_v2.bin
│       ├── ssdlite_mobilenet_v2.mapping
│       └── ssdlite_mobilenet_v2.xml

启动 Model Server 容器

Model Server 容器

docker run -d \
--name model_server \
--network edgex_edgex-network \
--device /dev/dri \
-v $(pwd)/models:/models \
-p 9000:9000 \
-p 8000:8000 \
openvino/model_server:latest-gpu \
--config_path /models/config.json \
--port 9000 \
--rest_port 8000 \
--log_level DEBUG

Model Server 服务测试

验证 Model Server 运行状态

通过 Kserve RESTful API 访问 OVMS 工作状态，这里访问已经启动的模型：ssdlite_mobilenet_v2。

curl http://192.168.198.8:8000/v1/config
{
  "ssdlite_mobilenet_v2": {
    "model_version_status": [
      {
        "version": "1",
        "state": "AVAILABLE",
        "status": {
          "error_code": "OK",
          "error_message": "OK"
        }
      }
    ]
  }
}

模型参数可视化

同样通过 Kserve RESTful API 可以访问模型的详细输入和输出参数，主要是 inputs 和 outputs 里面的 dim。

inputs: 需要输入图片名称是 image_tensor，格式是 DT_INT8，可以用 byte。图片纬度是 [1,300,300,3]， width:300, height:300, color:3；
outputs: 输出格式是一个四维矩阵 [1,1,100,7]，只需要后面两个维度：rows:100, cols:7；

{
  "modelSpec": {
    "name": "ssdlite_mobilenet_v2",
    "signatureName": "",
    "version": "1"
  },
  "metadata": {
    "signature_def": {
      "@type": "type.googleapis.com/tensorflow.serving.SignatureDefMap",
      "signatureDef": {
        "serving_default": {
          "inputs": {
            "image_tensor": {
              "dtype": "DT_UINT8",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "3",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "image_tensor"
            }
          },
          "outputs": {
            "detection_boxes": {
              "dtype": "DT_FLOAT",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "100",
                    "name": ""
                  },
                  {
                    "size": "7",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "detection_boxes"
            }
          },
          "methodName": "",
          "defaults": {

          }
        }
      }
    }
  }
}

② YiMEDIA 流媒体服务

笔者采用亿琪软件公司自己的流媒体服务器软件：YiMEDIA，来实现视频流的编解码。

当然，用户也可以使用一些其他框架来支撑，比如：

③ EdgeXFoundry 服务

部署 EdgeXFoundry Docker 容器

建议

根据 EdgeXFoundry edgex-compose 仓库手册，创建自己的 EdgeX 运行环境。本测试案例，只需要核心服务和 mqtt-broker 服务即可。

Device Service for ONVIF Camera

注意

Device Service for ONVIF Camera 是否需要使用，取决于是否由 EdgeX 来管理网络摄像头(IPC)，本例中暂时未启用此服务，采用手动设置。

Device Services for OVMS

测试

这是一个系列微服务，包含了各种算法库，支持算法调用和推理。

SSDLITE_MOBILENET_V2：提交一张图片，返回目标识别结果；

EdgeX Device Service: device-openvino-ssdlite-mobilenet-v2

此服务代码未开源，仅供商业服务。

设备服务配置参考

deviceList:

- name: OpenVINO-Object-Detection
  profileName: OpenVINO-Device
  description: Example of OpenVINO Device
  labels: [AI]
  protocols:
    ovms:
      Host: 192.168.198.8
      Port: 9000
      Model: ssdlite_mobilenet_v2
      Version: 1
      Uri: rtsp://192.168.123.12:18554/test
      Snapshot: false
      Record: false
      Score: 0.3

这是一个设备服务基础配置例子，用户可根据实际业务需求配置：

Host: OVMS 服务地址
Port: OVMS 服务端口
Model: 模型名称
Version: 模型版本
Uri: IPC 摄像头流地址
Snapshot: 是否建立快照
Record: 是否记录推理视频
Score: 推理结果最低信任度

以下是完整的 EdgeX metadata：

curl http://192.168.123.15:59881/api/v3/device/all

{
  "apiVersion": "v3",
  "statusCode": 200,
  "totalCount": 1,
  "devices": [
    {
      "created": 1708691437289,
      "modified": 1708693207326,
      "id": "6b02c540-eeae-4b61-b741-cdefcf93fd09",
      "name": "OpenVINO-Object-Detection",
      "description": "Example of OpenVINO Device",
      "adminState": "UNLOCKED",
      "operatingState": "UP",
      "labels": ["AI"],
      "serviceName": "device-openvino-ssdlite-mobilenet-v2",
      "profileName": "OpenVINO-Device",
      "autoEvents": [{}],
      "protocols": {
        "ovms": {
          "Host": "192.168.198.8",
          "Model": "ssdlite_mobilenet_v2",
          "Port": 9000,
          "Record": false,
          "Snapshot": false,
          "Uri": "rtsp://192.168.123.12:18554/test",
          "Version": 1,
          "Score": 0.3
        }
      }
    }
  ]
}

YiCONNECT 云平台对接

这里，使用 app-service-gateway 应用对接亿琪软件公司的云端服务平台：YiCONNECT。

验证工作

容器微服务状况

容器

所有的容器都运行成功后，可看到类似以下的结果：

# docker ps --format 'table {{.Image}}\t{{.Names}}'

IMAGE                                                NAMES
①
openvino/model_server:latest-gpu                     model_server

②
yiqisoft/YiMEDIA                                     yimedia

③
edgexfoundry/device-openvino-ssdlite-mobilenet-v2:0.0.0-dev       edgex-device-openvino-resnet
edgexfoundry/app-service-configurable:3.1.0                       edgex-app-rules-engine
edgexfoundry/core-data:3.1.0                                      edgex-core-data
edgexfoundry/core-command:3.1.0                                   edgex-core-command
edgexfoundry/support-scheduler:3.1.0                              edgex-support-scheduler
edgexfoundry/core-common-config-bootstrapper:3.1.0                edgex-core-common-config-bootstrapper
edgexfoundry/support-notifications:3.1.0                          edgex-support-notifications
edgexfoundry/core-metadata:3.1.0                                  edgex-core-metadata
eclipse-mosquitto:2.0.18                                          edgex-mqtt-broker
edgexfoundry/edgex-ui:3.1.0                                       edgex-ui-go
redis:7.0.14-alpine                                               edgex-redis
hashicorp/consul:1.16.2                                           edgex-core-consul

容器启动后，自动实现 1秒1次的推理请求。

device-openvino-ssdlite-mobilenet-v2 容器微服务日志

日志

edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.873363754Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/all-services"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.885937004Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:442 msg="loading the common configuration for service type device-service"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.886617712Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/device-services"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.893988962Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:173 msg="Common configuration loaded from the Configuration Provider. No overrides applied"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.894078212Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/device-openvino-ssdlite-mobilenet-v2"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.898733712Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:218 msg="Private configuration loaded from the Configuration Provider. No overrides applied"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.89878342Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:269 msg="listening for private config changes"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.898789837Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:271 msg="listening for all services common config changes"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.898793879Z app=device-openvino-ssdlite-mobilenet-v2 source=config.go:278 msg="listening for device service common config changes"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.898803254Z app=device-openvino-ssdlite-mobilenet-v2 source=registry.go:61 msg="Using Registry access token of length 0"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.89886942Z app=device-openvino-ssdlite-mobilenet-v2 source=registry.go:89 msg="Using Registry (consul) from http://edgex-core-consul:8500"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.920700504Z app=device-openvino-ssdlite-mobilenet-v2 source=httpserver.go:149 msg="Web server starting (localhost:61803)"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.92074342Z app=device-openvino-ssdlite-mobilenet-v2 source=messaging.go:66 msg="Setting options for secure MessageBus with AuthMode='usernamepassword' and SecretName='redisdb"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.920965504Z app=device-openvino-ssdlite-mobilenet-v2 source=messaging.go:104 msg="Connected to redis Message Bus @ redis://edgex-redis:6379 with AuthMode='usernamepassword'"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.921013879Z app=device-openvino-ssdlite-mobilenet-v2 source=command.go:36 msg="Subscribing to command requests on topic: edgex/device/command/request/device-openvino-ssdlite-mobilenet-v2/#"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.92102142Z app=device-openvino-ssdlite-mobilenet-v2 source=command.go:40 msg="Responses to command requests will be published on topic: edgex/response/device-openvino-ssdlite-mobilenet-v2/<requestId>"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.924235045Z app=device-openvino-ssdlite-mobilenet-v2 source=callback.go:34 msg="Subscribing to System Events on topic: edgex/system-events/core-metadata/+/+/device-openvino-ssdlite-mobilenet-v2/#"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.925392587Z app=device-openvino-ssdlite-mobilenet-v2 source=validation.go:31 msg="Subscribing to device validation requests on topic: edgex/device-openvino-ssdlite-mobilenet-v2/validate/device"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.925427754Z app=device-openvino-ssdlite-mobilenet-v2 source=validation.go:35 msg="Responses to device validation requests will be published on topic: edgex/response/device-openvino-ssdlite-mobilenet-v2/<requestId>"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.92751767Z app=device-openvino-ssdlite-mobilenet-v2 source=manager.go:128 msg="Metrics Manager started with a report interval of 30s"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.928356087Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:190 msg="Using registry for URL for 'core-metadata': http://edgex-core-metadata:59881"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.92839667Z app=device-openvino-ssdlite-mobilenet-v2 source=restrouter.go:56 msg="Registering v2 routes..."
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.931419462Z app=device-openvino-ssdlite-mobilenet-v2 source=devices.go:78 msg="LastConnected-OpenVINO-TP2 metric has been registered and will be reported (if enabled)"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:10.931474712Z app=device-openvino-ssdlite-mobilenet-v2 source=devices.go:78 msg="LastConnected-OpenVINO-HK2 metric has been registered and will be reported (if enabled)"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.302464173Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 37.834208ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.339857923Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 20.645875ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.368987339Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 20.203584ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.401820923Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 23.99275ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.434832881Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 21.039792ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.464537798Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 20.51225ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.493459548Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 19.709ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.523483173Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 20.954292ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.641823423Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 108.266916ms"
edgex-device-openvino-ssdlite-mobilenet-v2  | level=INFO ts=2024-04-11T04:25:15.681794381Z app=device-openvino-ssdlite-mobilenet-v2 source=clients.go:162 msg="Inference time: 24.686125ms"

可以看到推理结果：msg="Inference time: 24.686125ms" 代表了每一帧的推理速度，综合平均值，在 20ms 左右，可以达到 50fps。

OpenVINO™ Model Server 日志

日志

# docker logs -f model_server

model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.022 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.943][131][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.492 ms
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.547 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.226 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4070 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.850 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.953][128][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.192 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.519 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.052 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 12.787 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.004 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.976][132][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.705 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.461 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.049 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4050 ms

可以看到推理使用设备：Used device: CPU，也可以在 config.json 配置文件中修改为： AUTO 或 GPU
整个请求所花的时间：Total gRPC request processing time: 10.4050 ms，综合平均值，也就是10-15ms 之间，性能非常好。

亿琪云接收推理结果

云端效果请参考以往文章：案例｜边缘智能：YiFUSION AI 推理应用之 EdgeX 集成 OpenVINO™

产品应用

YiFUSION--工业边缘智能融合网关

EdgeX 框架应用于我司的边缘计算产品 YiEDGE（边缘计算软件）和 YiFUSION（工业边缘智能融合网关）。

YiFUSION: 边缘融合智能网关是指具备边缘计算、智能分析、数据融合等功能的智能网关，能够在边缘侧实现数据采集、处理、分析、融合和应用，为用户提供更加智能、高效和低成本的服务。

简洁定义

英文名称：YiFUSION
中文名称：工业边缘智能融合网关
产品形态：软硬件一体机
硬件架构：X86/ARM/超融合/国产化
软件架构：全微服务化

关于我们

亿琪软件

上海亿琪软件有限公司，国家级高新技术企业，全球开放边缘计算和物联网领域的领导者，全球领先的工业物联网软件开发商和解决方案提供商，助力企业和组织实现数字化转型。公司专注于 5G 通信、AI 人工智能、边缘计算和大数据网络安全多项技术领域，致力于物联网领域前沿技术的创新，为用户提供全方位、智能化和安全的物联网解决方案。

2023 年，公司发布“ YiFUSION ｜工业边缘智能融合网关 ”产品，为工业客户提供一整套的边缘计算+AI 能力：高性能数据采集、多类型数据融合、AI 算法集成、云端业务对接。在边缘网关的基础上，集成了 IoT 平台的边缘协同能力、本地 Web SCADA 和 HMI 功能、本地数据存储、边缘 AI 视频分析、行业应用集成等。
2022 年，公司推出 “ YiCLOUD ｜亿琪云 ”一站式物联网应用解决方案。公司的业务涵盖了智慧城市、智慧农业、智能工厂和智慧园区等多个领域，公司软硬件产品和解决方案获得华为技术认证，得到中国移动 OCP 认证，公司还是边缘计算产业联盟 ECC 成员。

关注我们

联系我们--商业服务

网站：http://yiqisoft.cn
邮件：support@yiqisoft.cn
电话：021-68863086
手机：186-1666-9123