Trex - Interactive Inference

The TRex Interactive Inference algorithm enables user prompting on image and get the boxes and scores on the same image.

This algorithm supports batch inference for multiple images, and for every image, multiple prompts are supported.

However, the prompt type for every image is limited to either point or rect.

Usage Pattern

First of all, make sure you have installed this SDK by pip:

pip install dds-cloudapi-sdk

Then trigger the algorithm through TRexInteractiveInfer class:

from dds_cloudapi_sdk import Client
from dds_cloudapi_sdk import Config
from dds_cloudapi_sdk import BatchRectInfer
from dds_cloudapi_sdk import BatchRectPrompt
from dds_cloudapi_sdk import TRexInteractiveInfer

# Step 1: initialize the config
token = "Your API token here"
config = Config(token)

# Step 2: initialize the client
client = Client(config)

# Step 3: run the task by TRexInteractiveInfer class
infer_image = "https://dev.deepdataspace.com/static/04_a.ae28c1d6.jpg"
# if you are processing local image file, upload them to DDS server to get the image url
# infer_image = client.upload_file("/path/to/your/infer/image.png")

infer_1 = BatchRectInfer(
    image=infer_image,
    prompts=[
        BatchRectPrompt(category_id=1, rects=[[475.18413597733706, 550.1983002832861, 548.1019830028329, 599.915014164306]])
    ]
)

task = TRexInteractiveInfer([infer_1])  # the interactive infer task supports batch inference

client.run_task(task)
for image_objects in task.result.object_batches:
    for obj in image_objects:
        print(obj.score)
        print(obj.bbox)
        print(obj.category_id)
        break
    break

API Reference

class TRexInteractiveInfer(batch_infers)[source]

Trigger the Trex Interactive Inference algorithm.

This task can process prompts from multiple images, and each image can have several prompts. However, each task is limited to one type of prompt, either point or rect.

Parameters:

batch_infers (List[BatchPointInfer] | List[BatchRectInfer]) – list of BatchPointInfer objects or BatchRectInfer.

property result: TaskResult

Get the formatted TaskResult object.

class BatchPointInfer(*, image, prompts)[source]

An infer image with batch point prompts.

Parameters:
image: str

the image url to be inferred on

prompts: List[BatchPointPrompt]

a list of BatchPointPrompt

class BatchRectInfer(*, image, prompts)[source]

An infer image with batch rect prompts.

Parameters:
image: str

the image url to be inferred on

prompts: List[BatchRectPrompt]

a list of BatchRectPrompt

class BatchPointPrompt(*, points, image=None, category_id=None)[source]

A batch of point prompts.

Parameters:

  • points (List[List[float]]) – a list of point locations in [[x1, y1], [x2, y2]]

  • image (str) – the image url the point prompts are acting on, if not provided, the infer image url in context will be used

  • category_id (int) – the category id of the points

points: List[List[float]]

a list of point locations in [[x1, y1], [x2, y2]]

image: str

the image url the point prompts are acting on

category_id: int

the category id of the points, only required for TRexInteractiveInfer task.

class BatchRectPrompt(*, rects, image=None, category_id=None)[source]

A batch of rectangle prompts.

Parameters:

  • rects (List[List[float]]) – a list of rect locations in [[[upper_left_x, upper_left_y, lower_right_x, lower_right_y], …]

  • image (str) – the image url the rectangle prompts are acting on

  • category_id (int) – the category id of the rects

rects: List[List[float]]

a list of rect locations in [[[upper_left_x, upper_left_y, lower_right_x, lower_right_y], …]

image: str

the image url the rectangle prompts are acting on, if not provided, the infer image url in context will be used

category_id: int

the category id of the rects, only required for TRexInteractiveInfer task.

class TaskResult(*, object_batches)[source]

The task result of TRexInteractiveInfer task.

Parameters:

object_batches (List[List[TRexObject]]) – a 2D list of detected objects of TRexObject, each inner list is the detected objects of one image

object_batches: List[List[TRexObject]]

a 2D list of detected objects of TRexObject, each inner list is the detected objects of one image

class TRexObject(*, score, bbox, category_id)[source]

The object detected by TRexInteractiveInfer task.

Parameters:

  • score (float) – the prediction score

  • bbox (List[float]) – the bounding box, [upper_left_x, upper_left_y, lower_right_x, lower_right_y]

  • category_id (int) – the category id of the object

score: float

the prediction score

bbox: List[float]

the bounding box, [upper_left_x, upper_left_y, lower_right_x, lower_right_y]

category_id: int

the category id of the object