Model bana liya — par wo kitna accha hai? Sirf accuracy dekhna kaafi nahi, aur kabhi-kabhi to bilkul dhokha hai.
Accuracy ka dhokha
Maan lijiye 100 transactions me se sirf 5 fraud hain. Ek 'model' jo hamesha bolta hai fraud nahi hai:
Example
import numpy as np
from sklearn.metrics import accuracy_score, recall_score
y_true = np.array([1] * 5 + [0] * 95) # 1 = fraud
y_lazy = np.zeros(100, dtype=int) # hamesha "fraud nahi"
print("Accuracy:", accuracy_score(y_true, y_lazy))
print("Kitne fraud pakde (recall):", recall_score(y_true, y_lazy))
Output
Accuracy: 0.95
Kitne fraud pakde (recall): 0.0
95% accuracy, par ek bhi fraud nahi pakda! Jab classes imbalanced hon, accuracy se aage dekhna padta hai.
Confusion matrix
Chaliye pichhle lesson ka breast cancer model lete hain. Is baar hum malignant (cancer) ko positive class maanenge, isliye target ulta kar rahe hain:
Example
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
cancer = load_breast_cancer(as_frame=True)
X, y = cancer.data, cancer.target # 0 = malignant, 1 = benign
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix
y_train_m = 1 - y_train # ab 1 = malignant
y_test_m = 1 - y_test
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
clf.fit(X_train, y_train_m)
y_pred = clf.predict(X_test)
cm = confusion_matrix(y_test_m, y_pred)
print(pd.DataFrame(cm, index=["Asli: benign", "Asli: malignant"],
columns=["Predicted: benign", "Predicted: malignant"]))
Output
Predicted: benign Predicted: malignant
Asli: benign 71 1
Asli: malignant 1 41
Predicted: Negative | Predicted: Positive | |
|---|---|---|
Asli: Negative | TN (True Negative) | FP (False Positive) — jhootha alarm |
Asli: Positive | FN (False Negative) — miss ho gaya | TP (True Positive) |
Precision, Recall aur F1
Metric | Formula | Sawaal |
|---|---|---|
Precision | TP / (TP + FP) | Jinhe positive bola, unme se kitne sach me positive the? |
Recall | TP / (TP + FN) | Saare asli positives me se kitne pakde? |
F1-score | 2 × P × R / (P + R) | Precision aur recall ka balance |
Example
from sklearn.metrics import classification_report
print(classification_report(y_test_m, y_pred, target_names=["benign", "malignant"]))
Output
precision recall f1-score support
benign 0.99 0.99 0.99 72
malignant 0.98 0.98 0.98 42
accuracy 0.98 114
macro avg 0.98 0.98 0.98 114
weighted avg 0.98 0.98 0.98 114
- Recall zyada important jab miss karna mehenga ho — cancer detection, fraud detection.
- Precision zyada important jab jhootha alarm mehenga ho — spam filter (zaroori email spam me na chala jaaye).
ROC-AUC
AUC batata hai ki model positive aur negative ko kitne achhe se alag kar paata hai, har threshold pe. 0.5 = sikka uchhaalne jaisa, 1.0 = perfect:
Example
from sklearn.metrics import roc_auc_score
proba = clf.predict_proba(X_test)[:, 1]
print("ROC-AUC:", round(roc_auc_score(y_test_m, proba), 4))
Output
ROC-AUC: 0.9954
Cross-validation
Ek hi train-test split kismat pe depend karta hai. K-fold cross-validation data ko K hisson me baant ke K baar train-test karta hai — har hissa ek baar test banta hai. Result zyada bharosemand hota hai:
Example
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, 1 - y, cv=5, scoring="recall")
print("Har fold ka recall:", scores.round(3))
print(f"Average: {scores.mean():.3f} ± {scores.std():.3f}")
Output
Har fold ka recall: [0.977 0.953 0.929 0.952 1. ]
Average: 0.962 ± 0.024
Hyperparameter tuning: GridSearchCV
Model ki settings (jaise Logistic Regression ka C) ko hyperparameters kehte hain. GridSearchCV har combination ko cross-validation se try karke best chunta hai:
Example
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(
clf,
param_grid={"logisticregression__C": [0.01, 0.1, 1, 10]},
cv=5,
scoring="f1",
)
grid.fit(X_train, y_train_m)
print("Best params:", grid.best_params_)
print("Best CV F1:", round(grid.best_score_, 4))
print("Test F1:", round(grid.score(X_test, y_test_m), 4))
Output
Best params: {'logisticregression__C': 1}
Best CV F1: 0.9731
Test F1: 0.9762
Regression ke metrics yaad hain?
Regression models ke liye MAE, RMSE aur R² use hote hain — ye humne Linear Regression wale lesson me dekhe the.
Aur detail me: Understanding Accuracy and Evaluation Metrics in Machine Learning.