手绘识别

本文主要给大家介绍下在Android平台下使用角蜂鸟实现手绘识别功能

准备工作

  1. 配置环境等详情请参照Hello 2018里的快速开始,此处不具体阐述
  2. 下载物体检测所需模型graph_sg文件,在您Android Studio中,当前module下新建assets包,将下载的模型文件复制到该目录下
  3. 因工程需要处理图像,所以使用了javacv库,可从GitHub自行下载或点击链接从示例工程中libs下拷贝到自己工程
  4. class_list_chn.txt文件拷贝到Android设备的存储空间下,用于345种物体的比对

具体实现

  • 打开设备通信

    int status = openDevice();
    if (status != ConnectStatus.HS_OK) {
      return;
    }
    
  • 将graph文件传输到角蜂鸟里

    int id = allocateGraphByAssets("graph_sg");
    
  • 读取物体分类文件

    try {
    BufferedReader bufferedReader = new BufferedReader(new FileReader(Environment.getExternalStorageDirectory().getAbsolutePath() + "/hs/class_list_chn.txt"));
    for (int i = 0; i < 345; i++) {
        String line = bufferedReader.readLine();
        if (line != null) {
            String[] strings = line.split(" ");
            mObjectNames[i] = strings[0];
        }
    }
    bufferedReader.close();
    } catch (FileNotFoundException e) {
    e.printStackTrace();
    Log.e("SketchGuessThread", "FileNotFoundException");
    } catch (IOException e) {
    e.printStackTrace();
    Log.e("SketchGuessThread", "IOException");
    }
    
  • 获取角蜂鸟内置的摄像头图像

    byte[] bytes = deviceGetImage();
    

注意:开发者也可以根据自己需求选择自己外部摄像头数据。

  • 处理图像,并将处理好的tensor传给角蜂鸟

    opencv_core.IplImage bgrImage = opencv_core.IplImage.create(FRAME_W, FRAME_H, opencv_core.IPL_DEPTH_8U, 3);
    bgrImage.getByteBuffer().put(bytes_frame);
    int sg_weight = (int) (FRAME_W * roi_ratio);
    //crop
    opencv_core.CvRect cvRect = opencv_core.cvRect((int) (FRAME_W * (0.5 - roi_ratio / 2)), (int) (FRAME_H * 0.5 - sg_weight / 2), sg_weight, sg_weight);
    cvSetImageROI(bgrImage, cvRect);
    opencv_core.IplImage cropped = cvCreateImage(cvGetSize(bgrImage), bgrImage.depth(), bgrImage.nChannels());
    cvCopy(bgrImage, cropped);
    //canny
    opencv_core.IplImage image_canny = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 1);
    cvCanny(cropped, image_canny, 120, 45);
    //dilate
    opencv_core.IplImage image_dilate = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 1);
    //kernel = np.ones((4,4),np.uint8)
    opencv_core.IplConvKernel iplConvKernel = cvCreateStructuringElementEx(4, 4, 0, 0, CV_SHAPE_RECT);
    cvDilate(image_canny, image_dilate, iplConvKernel, 1);
    opencv_core.IplImage image_dilate_rgba = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 4);
    cvCvtColor(image_dilate, image_dilate_rgba, CV_GRAY2RGBA);
    //resize
    opencv_core.IplImage image_load = opencv_core.IplImage.create(28, 28, opencv_core.IPL_DEPTH_8U, 4);
    cvResize(image_dilate_rgba, image_load);
    Bitmap bitmap_tensor = IplImageToBitmap(image_load);
    Message message1 = new Message();
    message1.arg1 = 2;
    message1.obj = bitmap_tensor;
    mHandler.sendMessage(message1);
    int[] pixels = new int[28 * 28];
    bitmap_tensor.getPixels(pixels, 0, 28, 0, 0, 28, 28);
    float[] floats = new float[28 * 28 * 3];
    for (int i = 0; i < 28 * 28; i++) {
    floats[i] = Color.red(pixels[i]) * 0.007843f - 1;
    floats[3 * i + 1] = Color.green(pixels[i]) * 0.007843f - 1;
    floats[3 * i + 2] = Color.blue(pixels[i]) * 0.007843f - 1;
    }
    int status_tensor = loadTensor(floats, floats.length, id);
    
  • 获取返回的处理结果

    float[] result = mHsApi.getResult(id);
    
  • 处理result返回值,排列取出最大的5个数对应的idex.

    public String[] sortMax5(float[] result) {
    
       HashMap<Integer, Float> integerFloatHashMap = new HashMap<>();
       String[] object_names = new String[5];
       for (int i = 0; i < result.length; i++) {
           integerFloatHashMap.put(i, result[i]);
       }
       Arrays.sort(result);
       for (int i = 0; i < result.length; i++) {
           if (integerFloatHashMap.get(i) == result[result.length - 1]) {
               object_names[0] = mObjectNames[i] + " " + result[result.length - 1];
           }
           if (integerFloatHashMap.get(i) == result[result.length - 2]) {
               object_names[1] = mObjectNames[i] + " " + result[result.length - 2];
           }
           if (integerFloatHashMap.get(i) == result[result.length - 3]) {
               object_names[2] = mObjectNames[i] + " " + result[result.length - 3];
           }
           if (integerFloatHashMap.get(i) == result[result.length - 4]) {
               object_names[3] = mObjectNames[i] + " " + result[result.length - 4];
           }
           if (integerFloatHashMap.get(i) == result[result.length - 5]) {
               object_names[4] = mObjectNames[i] + " " + result[result.length - 5];
           }
       }
       return object_names;
    }
    
  • 结果显示在view上,效果如图:

sketchGuess

友好提醒:因Android设备基本都是USB2.0,所以不建议使用1080P的图像,传输比较耗时,会有卡顿感,可以使用360P的图像,铺满屏幕即可

具体代码可去GitHub下载,地址如下 SungemSDK-AndroidExamples