AI - Monitoring

Data drift

• What
• Feature
• Population
• Covariate shift
• How
• Enough of the data needs to be labeled to introduce new classes
• Retrain the model

Concept drift

• What
• Pt(X, y) != Pt+1(X, y)
• Sudden
• Gradual
• Incremental
• Recurring
• How
• The old data needs to be relabeled. Retrain the model
• Use an ensemble approach to train your new model

Prediction drift

Label drift

Training, serving skew

• Tensorflow data validation