Monitor and Analyze - TBD

Этот функционала пока недоступен для Base Edition. Очень надеемся, что он появиться в ближайшее время.

Data Validation 🔬

Data schemas

MLPanel supports optional data schemas associated with models. Data schema for model is training data schema (without target column(s)).

What is schema?

Now schema is TensorFlow Data Validation (TFDV) statistics generated from pandas.DataFrame/csv/tfrecords.

So, you data must be in one of these formats or converted to it.

Motivation

Data schema associated with model allows:

• get schema to know columns names and types;

• validate data sent for prediction.

How to create and save data schema

You must generate TFDV statistics, save it into file stats.tfdv and log this file to MLflow tracking server as artifact. Example:

import  pandas as pd 
import tensorflow_data_validation as tfdv

...

# Read dataset
dataset = pd.read_csv(DATASET)

# Generate statistics from dataset WITHOUT target column
statistics = tfdv.generate_statistics_from_dataframe(dataset.drop(TARGET_COLUMN, axis=1))

# Fixed schema filename
STATISTICS_FILE = 'stats.tfdv'

# Serialize and save statistics to file
with open(STATISTICS_FILE, 'wb') as out_stats:
    out_stats.write(statistics.SerializeToString())

mlflow.log_artifact(DATASET)
mlflow.log_artifact(STATISTICS_FILE)

...

Use schema on inference

Schema is optional. What does it mean?

• if schema exists then:

  • you can get calling endpoint GET /deployments/{deployment_id}/schema;

  • data for prediction (endpoint POST /deployments/{deployment_id}/predict) will be validated (column names and types will be checked);

• if schema does not exists then:

  • you'll get empty schema calling endpoint GET /deployments/{deployment_id}/schema;

  • data for prediction will not be validated - data will be sent to MLflow deploy server without validation.

Required data format sending for prediction

To predict data calling endpoint POST /deployments/{deployment_id}/predict the data must be json string with following structure:

{
  "schema": {
    "fields":[
          {"name":"<col1>","type":"<type1>"},
          {"name":"<col2>","type":"<type2>"},
          ..................................,
          {"name":"<colN>","type":"<typeN>"},
    ]
  },
   "data": [
      {"index":0,"col1": v11,"col2": v21, ..., "colN": vN1},
      {"index":1,"col1": v12,"col2": v22, ..., "colN": vN2},
      .....................................................,
      {"index":P,"col1": v1P,"col2": v2P, ..., "colN": vNP},
   ]
}

To get json with such structure just convert pandas DataFrame to json, orient=table:

import pandas as pd

df = pd.read_csv(...)
json_dataset = df.to_json(orient='table')

About reading json by pandas: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html