手绘识别

本文主要给大家介绍下在Android平台下使用角蜂鸟实现手绘识别功能

准备工作

  1. 配置环境等详情请参照Hello 2018里的快速开始，此处不具体阐述
  2. 下载物体检测所需模型graph_sg文件，在您Android Studio中，当前module下新建assets包，将下载的模型文件复制到该目录下
  3. 因工程需要处理图像，所以使用了javacv库，可从GitHub自行下载或点击链接从示例工程中libs下拷贝到自己工程
  4. 将class_list_chn.txt文件拷贝到Android设备的存储空间下，用于345种物体的比对

具体实现

  • 打开设备通信

    int status = openDevice();
if (status != ConnectStatus.HS_OK) {
  return;
}

  • 将graph文件传输到角蜂鸟里

    int id = allocateGraphByAssets("graph_sg");

  • 读取物体分类文件

    try {
BufferedReader bufferedReader = new BufferedReader(new FileReader(Environment.getExternalStorageDirectory().getAbsolutePath() + "/hs/class_list_chn.txt"));
for (int i = 0; i < 345; i++) {
    String line = bufferedReader.readLine();
    if (line != null) {
        String[] strings = line.split(" ");
        mObjectNames[i] = strings[0];
    }
}
bufferedReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
Log.e("SketchGuessThread", "FileNotFoundException");
} catch (IOException e) {
e.printStackTrace();
Log.e("SketchGuessThread", "IOException");
}

  • 获取角蜂鸟内置的摄像头图像

    byte[] bytes = deviceGetImage();

注意：开发者也可以根据自己需求选择自己外部摄像头数据。

  • 处理图像，并将处理好的tensor传给角蜂鸟

    opencv_core.IplImage bgrImage = opencv_core.IplImage.create(FRAME_W, FRAME_H, opencv_core.IPL_DEPTH_8U, 3);
bgrImage.getByteBuffer().put(bytes_frame);
int sg_weight = (int) (FRAME_W * roi_ratio);
//crop
opencv_core.CvRect cvRect = opencv_core.cvRect((int) (FRAME_W * (0.5 - roi_ratio / 2)), (int) (FRAME_H * 0.5 - sg_weight / 2), sg_weight, sg_weight);
cvSetImageROI(bgrImage, cvRect);
opencv_core.IplImage cropped = cvCreateImage(cvGetSize(bgrImage), bgrImage.depth(), bgrImage.nChannels());
cvCopy(bgrImage, cropped);
//canny
opencv_core.IplImage image_canny = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 1);
cvCanny(cropped, image_canny, 120, 45);
//dilate
opencv_core.IplImage image_dilate = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 1);
//kernel = np.ones((4,4),np.uint8)
opencv_core.IplConvKernel iplConvKernel = cvCreateStructuringElementEx(4, 4, 0, 0, CV_SHAPE_RECT);
cvDilate(image_canny, image_dilate, iplConvKernel, 1);
opencv_core.IplImage image_dilate_rgba = opencv_core.IplImage.create(sg_weight, sg_weight, opencv_core.IPL_DEPTH_8U, 4);
cvCvtColor(image_dilate, image_dilate_rgba, CV_GRAY2RGBA);
//resize
opencv_core.IplImage image_load = opencv_core.IplImage.create(28, 28, opencv_core.IPL_DEPTH_8U, 4);
cvResize(image_dilate_rgba, image_load);
Bitmap bitmap_tensor = IplImageToBitmap(image_load);
Message message1 = new Message();
message1.arg1 = 2;
message1.obj = bitmap_tensor;
mHandler.sendMessage(message1);
int[] pixels = new int[28 * 28];
bitmap_tensor.getPixels(pixels, 0, 28, 0, 0, 28, 28);
float[] floats = new float[28 * 28 * 3];
for (int i = 0; i < 28 * 28; i++) {
floats[i] = Color.red(pixels[i]) * 0.007843f - 1;
floats[3 * i + 1] = Color.green(pixels[i]) * 0.007843f - 1;
floats[3 * i + 2] = Color.blue(pixels[i]) * 0.007843f - 1;
}
int status_tensor = loadTensor(floats, floats.length, id);

  • 获取返回的处理结果

    float[] result = mHsApi.getResult(id);

  • 处理result返回值,排列取出最大的5个数对应的idex.

    public String[] sortMax5(float[] result) {

   HashMap<Integer, Float> integerFloatHashMap = new HashMap<>();
   String[] object_names = new String[5];
   for (int i = 0; i < result.length; i++) {
       integerFloatHashMap.put(i, result[i]);
   }
   Arrays.sort(result);
   for (int i = 0; i < result.length; i++) {
       if (integerFloatHashMap.get(i) == result[result.length - 1]) {
           object_names[0] = mObjectNames[i] + " " + result[result.length - 1];
       }
       if (integerFloatHashMap.get(i) == result[result.length - 2]) {
           object_names[1] = mObjectNames[i] + " " + result[result.length - 2];
       }
       if (integerFloatHashMap.get(i) == result[result.length - 3]) {
           object_names[2] = mObjectNames[i] + " " + result[result.length - 3];
       }
       if (integerFloatHashMap.get(i) == result[result.length - 4]) {
           object_names[3] = mObjectNames[i] + " " + result[result.length - 4];
       }
       if (integerFloatHashMap.get(i) == result[result.length - 5]) {
           object_names[4] = mObjectNames[i] + " " + result[result.length - 5];
       }
   }
   return object_names;
}

  • 结果显示在view上,效果如图：

sketchGuess

友好提醒：因Android设备基本都是USB2.0,所以不建议使用1080P的图像，传输比较耗时，会有卡顿感，可以使用360P的图像，铺满屏幕即可

具体代码可去GitHub下载，地址如下 SungemSDK-AndroidExamples