> For the complete documentation index, see [llms.txt](https://doc.some3c.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.some3c.com/iphone-farm-setup/xp-api-documentation/picture-text-recognition.md). # Picture Text Recognition ## 1.Take a screenshot In special cases, you can use the shortcut (shortcut version 1.3 or higher) /shortcut/other/screenshot ### Calling * **Request Parameters** | parameter | type | illustrate | Remark | | ---------- | ------------- | ------------------------------- | ------------------------------------------------- | | fun | String | /pic/screenshot | Request function name | | id | String | Device ID | Multiple is not allowed | | binary | Boolean | If true, returns binary data. | By default, json is used for return | | jpg | Boolean | If true, returns the jpg format | Default bmp format | | rect | Integer Array | Leave full screen blank | Left x, Up y, Right x, Down y | | save\_path | String | Save to local path | If left blank, return using memory stream or json | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/screenshot?id=5C:F7:E6:CE:65:22&save_path=D%3A%5CiMouseXP%5CScreenshot%5C' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'content-type: multipart/form-data' \ --form fun=/pic/screenshot \ --form id=5C:F7:E6:CE:65:22 ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/screenshot", "data": { "id":"5C:F7:E6:CE:65:22", "binary":true } }' ``` ### Response When the binary parameter is true, this json will not be returned, and the binary image data will be returned directly ``` { "data": { "code": 0, "id": "6C:72:E7:EA:CD:95", "jpg": false, "binary": false, "rect": "", "image": "D:\\iMouseXP\\Screenshot\\1128935437.bmp", "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/screenshot" } ``` * **Response data field description** | Fields | illustrate | Remark | | ------ | ---------- | ----------------------------------------------------------------------------------------- | | jpg | - | - | | binary | - | - | | rect | - | - | | image | Image data | If save\_path is used, the image path is returned, otherwise the base64 string image data | ## 2.General image search In special cases, you can use the shortcut (shortcut version 1.3 or higher) /shortcut/other/find-image ### Calling This image search mode is exactly the same as DM * **Request Parameters** | parameter | type | illustrate | Remark | | ------------ | ---------------------- | ------------------- | ----------------------------------------------------------------------------------------------- | | fun | String | /pic/find-image | Request function name | | id | String | Device ID | Multiple is not allowed | | target\_img | String | Big picture of goal | Generally, you don't need to fill it in. Our internal screen image will be used by default. | | delta\_color | String | Color Cast | Reference DM | | all | Boolean | Have you found all? | false will only return one of the images in the list, true will find all the images in the list | | direction | String | Find directions | Reference DM | | img\_list | String Array | Thumbnail | Base64 string image list or local path list | | rect | Integer Array | Find Region | Left x, Up y, Right x, Down y | | similarity | Floating point numbers | Similarity | Greater than 0 and less than 1 | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/find-image?id=6C:8D:C1:67:04:29&similarity=0.8&img_list=%5B%22D%3A%5C%5CiMouseXP%5C%5CScreenshot%5C%5C1.bmp%22%5D' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header "content-type: multipart/form-data" \ --form "fun=/pic/find-image" \ --form "id=6C:8D:C1:67:04:29" \ --form "similarity=0.8" \ --form "img_list=[\"D:\\\iMouseXP\\\Screenshot\\\1.bmp\"]" ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/find-image", "data": { "id": "6C:8D:C1:67:04:29", "img_list": ["D:\\iMouseXP\\Screenshot\\1.bmp"], "similarity": 0.85 } }' ``` ### Response ``` { "data": { "code": 0, "id": "6C:8D:C1:67:04:29", "list": [ { "index": 0, "centre": [ 457, 1231 ], "rect": [ 425, 1206, 489, 1257 ], "code": 0 } ], "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/find-image" } ``` * **Response data field description** | Fields | illustrate | Remark | | ------ | --------------------------------------- | ------------------------------------------- | | index | index | The first picture is found, starting from 0 | | centre | Find the center point of the image | x and y | | rect | Find the rectangular region of an image | Left x, Up y, Right x, Down y | ## 3.OpenCV Image Search In special cases, you can use the shortcut (shortcut version 1.3 or higher): /shortcut/other/find-image-cv ### Calling * **Request Parameters** | parameter | type | illustrate | Remark | | ----------- | ---------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------- | | fun | String | /pic/find-image-cv | Request function name | | id | String | Device ID | Multiple is not allowed | | target\_img | String | Big picture of goal | Generally, you don't need to fill it in. Our internal screen image will be used by default. | | same | Boolean | Do you want to find the same image | If true, all the same images in the image will be found | | all | Boolean | Have you found all | false will only return one of the images in the list, true will find all the images in the list | | img\_list | String Array | Thumbnail | Base64 string image list or local path list | | rect | Integer Array | Find Region | Left x, Up y, Right x, Down y | | similarity | Floating point numbers | Similarity | Greater than 0 and less than 1 | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/find-image-cv?id=6C:8D:C1:67:04:29&similarity=0.8&img_list=%5B%22D%3A%5C%5CiMouseXP%5C%5CScreenshot%5C%5C1.bmp%22%5D' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header "content-type: multipart/form-data" \ --form "fun=/pic/find-image-cv" \ --form "id=6C:8D:C1:67:04:29" \ --form "similarity=0.8" \ --form "img_list=[\"D:\\\iMouseXP\\\Screenshot\\\1.bmp\"]" ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/find-image-cv", "data": { "id": "6C:8D:C1:67:04:29", "img_list": ["D:\\iMouseXP\\Screenshot\\1.bmp"], "similarity": 0.85 } }' ``` ### Response ``` { "data": { "code": 0, "id": "6C:8D:C1:67:04:29", "list": [ { "index": 0, "centre": [ 457, 1231 ], "rect": [ 425, 1206, 489, 1257 ], "code": 0 } ], "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/find-image-cv" } ``` * **Response data Data field description**\ list returns a list of all found text | Fields | illustrate | Remark | | ---------- | --------------------------------------- | ------------------------------------------- | | index | index | The first picture is found, starting from 0 | | centre | Find the center point of the image | x and y | | rect | Find the rectangular region of an image | Left x, Up y, Right x, Down y | | similarity | Similarity | | ## 4.OCR text recognition ### Calling In special cases, shortcuts can be used (shortcut version 1.3 or higher required): /shortcut/other/ocr /shortcut/other/ocr-ex\ There's another feature called /pic/ocr-ex, an enhanced OCR version with more accurate recognition, but it uses more resources. * **Request Parameters** | parameter | type | illustrate | Remark | | ----------- | ------------- | ------------------- | ------------------------------------------------------------------------------------------- | | fun | String | /pic/ocr | Request function name | | id | String | Device ID | Multiple is not allowed | | target\_img | String | Big picture of goal | Generally, you don't need to fill it in. Our internal screen image will be used by default. | | rect | Integer Array | Find Region | Left x, Up y, Right x, Down y | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/ocr?id=6C:8D:C1:67:04:29' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header "content-type: multipart/form-data" \ --form "fun=/pic/ocr" \ --form "id=6C:8D:C1:67:04:29" ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/ocr", "data": { "id": "6C:8D:C1:67:04:29" } }' ``` ### Response ``` { "data": { "code": 0, "id": "6C:8D:C1:67:04:29", "list": [ { "text": "Screen mirror output", "centre": [ 374, 475 ], "rect": [ 270, 457, 478, 494 ], "similarity": 0.82 } ], "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/ocr" } ``` * **Response data Data field description**\ list returns a list of all found text | Fields | illustrate | Remark | | ---------- | ---------------------------- | ----------------------------- | | text | Recognized text | | | centre | The center point of the text | x and y | | rect | Rectangular area of ​​text | Left x, Up y, Right x, Down y | | similarity | Similarity | | ## 5.Find text ### Calling In special cases, shortcuts can be used (shortcut version 1.3 or higher required): /shortcut/other/find-text /shortcut/other/find-text-ex There is also a function called /pic/find-text-ex that uses an enhanced version of OCR that is more accurate but takes up more resources. * **Request Parameters** | parameter | type | illustrate | Remark | | ----------- | ---------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- | | fun | String | /pic/find-text | Request function name | | id | String | Device ID | Multiple is not allowed | | target\_img | String | Big picture of goal | Generally, you don't need to fill it in. Our internal screen image will be used by default. | | rect | Integer Array | Find Region | Left x, Up y, Right x, Down y | | similarity | Floating point numbers | Similarity | It is best to use OCR to recognize it once to see how similar it is. | | contain | Boolean | Contains | | | text | String Array | The text array to be searched | | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/find-text?id=6C:8D:C1:67:04:29&contain=true&similarity=0.8&text=%5B%22D2B6%22%2C%22DCA1%22%5D' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header "content-type: multipart/form-data" \ --form "fun=/pic/find-text" \ --form 'text=["aaa","bbb"]' \ --form "similarity=0.8" \ --form "id=FA:9E:10:3A:FE:E8" ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/find-text", "data": { "id": "FA:9E:10:3A:FE:E8", "text":["时钟","邮件"], "similarity":0.8 } }' ``` ### Response ``` { "data": { "code": 0, "id": "6C:8D:C1:67:04:29", "list": [ { "text": "iMouseXP-D2B6", "centre": [ 283, 581 ], "rect": [ 153, 565, 413, 598 ], "similarity": 0.98 }, { "text": "iMouseXP-DCA1", "centre": [ 281, 685 ], "rect": [ 153, 671, 409, 700 ], "similarity": 0.91 } ], "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/find-text" } ``` * **Response data Data field description**\ list returns a list of all found text | Fields | illustrate | Remark | | ---------- | ---------------------------- | ----------------------------- | | text | Recognized text | | | centre | The center point of the text | x and y | | rect | Rectangular area of ​​text | Left x, Up y, Right x, Down y | | similarity | Similarity | | ## 6.Find Multi Color In special cases, you can use the shortcut (shortcut version 1.3 or higher) /shortcut/other/find-multi-color ### Calling * **Request Parameters** | parameter | type | illustrate | Remark | | ----------- | ----------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------- | | fun | String | /pic/find-multi-color | Request function name | | id | String | Device ID | Multiple is not allowed | | target\_img | String | Big picture of goal | Generally, you don't need to fill it in. Our internal screen image will be used by default. | | same | Boolean | Do you want to find the same image? | If true, all the same color tables in the image will be found | | all | Boolean | Have you found all? | false will return only one of the colors in the list, true will find all the colors in the list | | list | json object array | - | - | Parameter description of json object array in list: | parameter | type | illustrate | Remark | | ------------ | ---------------------- | ------------------------------ | -------------------------------------------------------------------- | | first\_color | String | The color table being searched | - | | rect | Integer Array | Find Region | Left x, top y, right x, bottom y are empty and search in full screen | | similarity | Floating point numbers | Similarity | - | * **httpGet request example** ``` curl --request GET \ --url 'http://192.168.9.9:9911/api/pic/find-multi-color?id=6C:8D:C1:67:04:29&list=%5B%7B%22first_color%22%3A%20%22284%7C1224%7C2AB9F3-000000%2C310%7C1235%7C29ADF1-000000%2C281%7C1225%7C29B8F2-000000%2C288%7C1236%7CEB2A31-000000%2C286%7C1258%7C2693EC-000000%2C262%7C1253%7C2699EE-000000%2C265%7C1228%7C2AB5F2-000000%2C305%7C1232%7CC25B78-000000%2C336%7C1253%7C2997EE-000000%2C301%7C1258%7C2693EC-000000%2C281%7C1245%7CFFFFFC-000000%22%2C%22similarity%22%3A%200.85%7D%5D' ``` * **httpPost request example** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header "Content-Type: multipart/form-data" \ --form "fun=/pic/find-multi-color" \ --form "id=6C:8D:C1:67:04:29" \ --form "list=[{\"first_color\":\"284|1224|2AB9F3-000000,310|1235|29ADF1-000000,281|1225|29B8F2-000000,288|1236|EB2A31-000000,286|1258|2693EC-000000,262|1253|2699EE-000000,265|1228|2AB5F2-000000,305|1232|C25B78-000000,336|1253|2997EE-000000,301|1258|2693EC-000000,281|1245|FFFFFC-000000\",\"similarity\":0.85}]" ``` * **httpPost json request example \[websocket can also send the same json request]** ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/find-multi-color", "data": { "id": "60:9A:C1:9A:22:A1", "list": [ { "first_color": "284|1224|2AB9F3-000000,310|1235|29ADF1-000000,281|1225|29B8F2-000000,288|1236|EB2A31-000000,286|1258|2693EC-000000,262|1253|2699EE-000000,265|1228|2AB5F2-000000,305|1232|C25B78-000000,336|1253|2997EE-000000,301|1258|2693EC-000000,281|1245|FFFFFC-000000", "similarity": 0.85 } ], "same": false, "all": false } }' ``` ``` curl --request POST \ --url http://192.168.9.9:9911/api \ --header 'Content-Type: application/json' \ --data '{ "fun": "/pic/find-multi-color", "data": { "id": "6C:8D:C1:67:04:29", "list": [ { "rect": [ 0, 0, 0, 0 ], "first_color": "284|1224|2AB9F3-000000,310|1235|29ADF1-000000,281|1225|29B8F2-000000,288|1236|EB2A31-000000,286|1258|2693EC-000000,262|1253|2699EE-000000,265|1228|2AB5F2-000000,305|1232|C25B78-000000,336|1253|2997EE-000000,301|1258|2693EC-000000,281|1245|FFFFFC-000000", "similarity": 0.85 } ], "same": false, "all": false } }' ``` ### Response ``` { "data": { "code": 0, "id": "60:9A:C1:9A:22:A1", "list": [ { "index": 0, "centre": [ 286, 1222 ] } ], "message": "成功" }, "status": 200, "message": "成功", "msgid": 0, "fun": "/pic/find-multi-color" } ``` * **Response data Data field description**\ list returns a list of all found text | Fields | illustrate | Remark | | ------ | ---------------------------------- | -------------------------------------------------------------- | | index | index | The first color search configuration is found, starting from 0 | | centre | Find the center point of the image | x and y | ##