Rapiro Robot

This chapter introduced how to build a robot that controlled by HS face detector.

Hardware requirement:

  • Horned Sungem Device
  • Raspberry Pi 3b
  • Rapiro Robot
  • 12V to 5V DC-DC module (Optional)
  • Cooling fan (Optional)

Workflow

2-1

The built-in model Mobilenet-SSD face detector is used in this DIY demo. The logic is illustrated as below:

2-2

Implementation

Firstly assemble and power the Rapiro robot, then use robot to supply the power to a Raspeberry Pi. The signal is sent through USB port.

Please note that due to the low stability of Raspberry Pi USB power transfer, it might be sometimes insufficient to power the HS device. However, it’s possible to use an extra cable to power the HS.

To make to robot looks good, we put all things into his head:

3-1

The controlling code for rapiro robot:

class RapiroProcessor(Thread):

    class Flags:
        face = None
        face_frame_cnt = 0
        head_angle = 90
        arm_raised = False
        light_on = False
        light_need_change = False

    def __init__(self, connection):
        Thread.__init__(self)
        self.daemon = True
        self.rapiro = Rapiro(connection)
        self.flags = RapiroProcessor.Flags()

    def reset(self):
        self.rapiro.execute(GO_TO_INITIAL_POSITION)
        time.sleep(3)

    def run(self):
        self.reset()
        try:
            while True:
                time.sleep(0.1)
                has_face = self.flags.face is not None
                frame_cnt = self.flags.face_frame_cnt
                comm = ""
                if has_face:  # Head pose
                    x1 = self.flags.face[2]
                    x2 = self.flags.face[4]
                    x_mid = (x1 + x2) / 2
                    if x_mid > 0.6 or x_mid / 2 < 0.4:
                        angle = self.flags.head_angle - int((x_mid - 0.5) / 0.1)
                    else:
                        angle = self.flags.head_angle
                    angle = 180 if angle > 180 else angle
                    angle = 0 if angle < 0 else angle
                    self.flags.head_angle = angle
                    eye = (246, 124, 1)
                    frame_cnt += 1
                else:
                    eye = (2, 220, 255)
                    frame_cnt -= 1

                self.flags.light_need_change = (not has_face if self.flags.light_on else has_face)
                self.flags.light_on = has_face

                if frame_cnt > 5:  # Raise hand
                    self.flags.face_frame_cnt = 5
                    if not self.flags.arm_raised:
                        comm += r_shoulder_yz(90, 1)
                        self.flags.arm_raised = True
                elif frame_cnt < -5:  # Put hand down
                    self.flags.face_frame_cnt = -5
                    if self.flags.arm_raised:
                        comm += r_shoulder_yz(0, 1)
                        self.flags.arm_raised = False
                else:
                    self.flags.face_frame_cnt = frame_cnt

                if self.flags.light_need_change:  # Change eyes colour
                    comm += eye_color(eye[0], eye[1], eye[2], 1)
                comm += head_yaw(self.flags.head_angle, 1)

                self.rapiro.execute(comm)
        finally:
            self.reset()

the python code for HS:

#!/usr/bin/env python3
# coding=utf-8

import sys
sys.path.append("../../SungemSDK-Python")
import hsapi as hs
import rapiro
import video


def process(ret):
    img = ret[0]
    face = []
    img_w = img.shape[1]
    img_h = img.shape[0]
    for box in ret[1]:
        if (box[4] - box[2] > img_w * 0.8) \
                and (box[5] - box[3] > img_h * 0.8):
            continue
        box[2] /= img_w
        box[4] /= img_w
        box[3] /= img_h
        box[5] /= img_h
        face.append(box)
    return face


if __name__ == '__main__':
    t_rapiro = rapiro.RapiroProcessor("/dev/ttyUSB0")  # 控制线程
    t_video = video.VideoProcessor(("192.168.50.202", 10600))  # 视频线程

    try:
        net = hs.FaceDetector(zoom=True, verbose=0, threshSSD=0.55)
        t_rapiro.start()
        t_video.start()
        while True:
            result = net.run()
            image = net.plot(result)
            
            faces = process(result)
            t_rapiro.flags.face = faces[0] if len(faces) > 0 else None

            if t_video.has_client():
                t_video.input_queue.put(image)
    finally:
        t_rapiro.reset()

Outcome

When face is detected, the robot will turn his head to the face, raises right hand and his eyes will turn yellow. If no face detected, he will return to the default status.

4-1 4-2