Evaluations: create a human feedback loop

Deeploy is build around the believe that it is important to explain what deployed models do when models make predictions and for for future reference. Therefore every deployment in Deeploy has an endpoint to collect feedback from human experts. Feedback is provided on the level of a single prediction. The human evaluation is saved as metadata to the prediction. Below we will elaborate on human evaluation in a step-by-step example.

Step 1: Get the deployment base API

In this article we assume you already created a deployment in Deeploy. If this is not the case we advise to follow the walkthrough first. Navigate to the deployment details and copy the deployment API url by expanding the API box and clicking the url (see screenshot below).

Step 2: Inference the deployment

You can use the Deeploy app to create a sample inference call, see this article for more information. More more common scenario is that you will use the Deeploy Python client or API to inference the deployment. Code snippets are available in the Deeploy app, see this article for more information. You can use these code snippets from you local environment. A complete overview of the deployment API documentation is available here.

Example inference call for explanation on deployment URL in Python. 1st code snippet uses the Python client (see here for more information about the Python client), the second example uses the API from Python:

in this example we use a deployment in workspace with id: e7942eeb-3e7e-4d27-a413-23f49a0f24f3 and deployment with id: c2bd2a1b-6909-4c42-b3ce-d1d86631e0bf

Example 1:

import deeploy

workspace_id = 'e7942eeb-3e7e-4d27-a413-23f49a0f24f3'
deployment_id = 'c2bd2a1b-6909-4c42-b3ce-d1d86631e0bf'

client_options = { 
    'access_key': 'AKIAEXAMPLEKEY', # see https://deeploy-ml.zendesk.com/hc/en-150/articles/6505862346386-Personal-key-pairs
    'secret_key': 'examplesecretkey', 
    'host': 'example.deeploy.ml', 
    'workspace_id': workspace_id
}

request_body = { 
    "instances": [[39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]] 
}

client = Client(**client_options)
explanation = client.explain(deployment_id, request_body)

Example 2:

import requests

workspace_id = 'e7942eeb-3e7e-4d27-a413-23f49a0f24f3'
deployment_id = 'c2bd2a1b-6909-4c42-b3ce-d1d86631e0bf'
token = '' # see https://deeploy-ml.zendesk.com/hc/en-150/articles/360021070580-Integrating-a-Deployment

deployment_url = f"https://api.demo.deeploy.ml/workspaces/{workspace_id}/deployments/{deployment_id}/explain" 

request_body = { 
    "instances": [[39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]]  
} 

headers = { 'Authorization': 'Bearer %s' % token, } 
response = requests.post(deployment_url, headers=headers, json=request_body) 
explanation = response.json() 

Step 3: Get the request log ID and prediction log ID

The request log ID is needed to connect the evaluation to the right prediction log. In the example below we use the output from step 2.

request_log_id = explanation['requestLogId']
prediction_log_id = explanation['predictionLogIds'][0]

Step 4: Evaluate a prediction

In order to evaluate a prediction you will use your deployment API or Python client. It is common that these act as a machine learning backend to you (web) application to your business application.

Example evaluation for explanation on deployment URL in Python. 1st code snippet uses the Python client, the second option uses the API from Python:

Example 1:

evaluation_input = {
 "result": 1, 
 "value": {"predictions": [true]}, 
} 

explanation = "<YOUR_EVALUATION>" 

client.evaluate(deployment_id, request_log_id, prediction_log_id, evaluation_input, explanation)

Example 2:

deployment_url = f"https://api.demo.deeploy.ml/workspaces/{workspace_id}/deployments/{deployment_id}/requestLogs/{request_log_id}/predictionLogs/{prediction_log_id}/validations" 
 
evaluation_input = {
    "result": 1, 
    "value": {"predictions": [true]}, 
    "explanation": "<YOUR_EVALUATION>" 
} 

headers = { 'Authorization': 'Bearer %s' % token, } 
response = requests.post(deployment_url, headers=headers, json=evaluation_input) 
evaluation_output = response.json() 

Step 5: Monitor expert evaluations

Navigate to the deployment details and monitoring deployment screens to monitor the feedback loop (see screenshots).

Evaluation in prediction log details:

Expert evaluation monitoring: