¶ (c) Technocore - All Rights Reserved.
NOTICE: All information contained herein is, and remains
the property of TechnoCore.
The intellectual and technical concepts contained
herein are proprietary to TechnoCore and dissemination of this information or reproduction of this material
is strictly forbidden unless prior written permission is obtained
from TechnoCore.
¶ ObjML
The ObjML class provides a foundational framework for machine learning operations within the system. It encapsulates functionalities for data preprocessing, model training, prediction, and evaluation, with a focus on integrating seamlessly with various data sources and machine learning algorithms.
¶ Key Features
- Data Preprocessing: Handles tasks such as one-hot encoding for categorical features and splitting data into features and target variables.
- Model Training & Evaluation: Provides generic methods for training and evaluating machine learning models.
- Cost-Sensitive Classification: Includes specialized methods for training models where misclassification costs vary between different classes.
- Data Integration: Supports reading data from various sources, including MySQL databases.
¶ Cost-Sensitive Classification with Logistic Regression
The ObjML class supports cost-sensitive classification, particularly through its train_cost_sensitive_classifier method. This is crucial in scenarios where the cost of misclassifying one class is significantly higher than misclassifying another (e.g., in fraud detection, medical diagnosis).
¶ Logistic Regression (sklearn.linear_model.LogisticRegression)
Logistic Regression is a linear model used for binary classification. Despite its name, it is a classification algorithm rather than a regression algorithm. It models the probability that a given input belongs to a particular class.
Role in Cost-Sensitive Classification:
In cost-sensitive scenarios, LogisticRegression can be configured using the class_weight parameter. This parameter assigns different weights to classes, effectively penalizing misclassifications of certain classes more heavily during the model training process. This helps the model to prioritize correctly classifying the more "costly" class.
Key Parameters for Cost-Sensitive Use:
class_weight:- Purpose: Dictionary or 'balanced'. If a dictionary, it specifies the weights associated with classes. For example,
{0: 1, 1: 5}means that misclassifying class1is 5 times more costly than misclassifying class0. - Impact: During training, the algorithm will give more importance to samples from classes with higher weights, leading to a model that is more cautious about misclassifying those classes.
- Purpose: Dictionary or 'balanced'. If a dictionary, it specifies the weights associated with classes. For example,
solver:- Purpose: Algorithm to use in the optimization problem. Common choices include 'liblinear', 'lbfgs', 'saga'.
- Recommendation: 'liblinear' is often a good choice for smaller datasets and when
class_weightis used, as it supports L1 and L2 regularization.
max_iter:- Purpose: Maximum number of iterations taken for the solvers to converge.
- Impact: Increasing this value can help the model converge, especially with complex datasets or when class weights are applied, but it also increases training time.
random_state:- Purpose: Controls the randomness of the solver.
- Impact: Ensures reproducibility of results.
Example Usage (within ObjML.train_cost_sensitive_classifier):
# Example of how LogisticRegression is instantiated within ObjML
model = LogisticRegression(
class_weight={1: 1, 0: 5}, # Example: class 0 is 5 times more costly to misclassify
max_iter=1000,
solver='liblinear',
random_state=42
)
model.fit(X_train, y_train)
This configuration guides the LogisticRegression model to be more sensitive to errors on class 0 (e.g., "bad credit" in the German credit dataset, which was mapped from original class 2), thereby aligning the model's predictions with the business costs of misclassification.
¶ Random Forest Classifier (sklearn.ensemble.RandomForestClassifier)
Random Forest is an ensemble learning method for classification, regression, and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output is the class selected by most trees.
Role in Cost-Sensitive Classification:
Similar to Logistic Regression, RandomForestClassifier can be made cost-sensitive through its class_weight parameter. This allows the model to give more importance to samples from certain classes, which is particularly useful for imbalanced datasets or when misclassification costs differ.
Key Parameters for Cost-Sensitive Use:
class_weight:- Purpose: Dictionary or 'balanced'. If a dictionary, it specifies the weights associated with classes. For example,
{0: 1, 1: 5}means that misclassifying class1is 5 times more costly than misclassifying class0. - Impact: Samples from classes with higher weights will have a greater influence on the training of individual decision trees, leading to a forest that is more biased towards correctly classifying those costly classes.
- Purpose: Dictionary or 'balanced'. If a dictionary, it specifies the weights associated with classes. For example,
n_estimators:- Purpose: The number of trees in the forest.
- Impact: A higher number of trees generally improves performance but also increases computation time.
random_state:- Purpose: Controls the randomness of the bootstrapping of the samples and the features considered when looking for the best split.
- Impact: Ensures reproducibility of results.
Example Usage (within ObjML.train_cost_sensitive_classifier):
# Example of how RandomForestClassifier is instantiated within ObjML
model = RandomForestClassifier(
class_weight={1: 1, 0: 5}, # Example: class 0 is 5 times more costly
random_state=42
)
model.fit(X_train, y_train)
¶ Gradient Boosting Classifier (sklearn.ensemble.GradientBoostingClassifier)
Gradient Boosting is a powerful ensemble technique that builds a model in a stage-wise fashion, typically using decision trees as weak learners. It iteratively improves by fitting new models to the residuals of previous models.
Role in Cost-Sensitive Classification:
GradientBoostingClassifier does not have a direct class_weight parameter in its constructor. Instead, cost-sensitivity is typically introduced by providing sample_weight to the fit method. By assigning higher weights to samples from more costly classes, the algorithm focuses more on correctly predicting those samples.
Key Parameters for Cost-Sensitive Use:
sample_weight(passed tofitmethod):- Purpose: Individual weights for each sample. Samples with higher weights will have a greater impact on the model's training.
- Impact: When
sample_weightis derived fromclass_weight(e.g., samples from class0get a weight of 5, while samples from class1get a weight of 1), the boosting algorithm will prioritize reducing errors on the more costly samples.
n_estimators:- Purpose: The number of boosting stages to perform.
- Impact: A higher number generally leads to better performance but can also lead to overfitting.
learning_rate:- Purpose: Shrinks the contribution of each tree.
- Impact: A lower learning rate requires more
n_estimatorsbut can lead to more robust models.
random_state:- Purpose: Controls the random seed given to each Tree estimator at each boosting iteration.
- Impact: Ensures reproducibility of results.
Example Usage (within ObjML.train_cost_sensitive_classifier):
# Example of how GradientBoostingClassifier is instantiated and fitted within ObjML
model = GradientBoostingClassifier(random_state=42)
# sample_weights would be computed based on class_weights and y_train
model.fit(X_train, y_train, sample_weight=sample_weights)
¶ Multi-layer Perceptron Classifier (sklearn.neural_network.MLPClassifier)
A Multi-layer Perceptron (MLP) is a type of artificial neural network. It consists of at least three layers of nodes: an input layer, a hidden layer, and an output layer. MLPs are capable of learning non-linear functions and are widely used for classification tasks.
Role in Cost-Sensitive Classification:
Similar to Gradient Boosting, MLPClassifier does not directly accept a class_weight parameter in its constructor. Cost-sensitivity is achieved by passing sample_weight to the fit method. By weighting samples from more costly classes more heavily, the neural network's optimization process (e.g., backpropagation) will focus more on minimizing errors for those samples.
Key Parameters for Cost-Sensitive Use:
sample_weight(passed tofitmethod):- Purpose: Individual weights for each sample. Samples with higher weights will have a greater impact on the model's training.
- Impact: When
sample_weightis derived fromclass_weight, the network's weights and biases will be adjusted more significantly based on the errors made on the more costly samples.
hidden_layer_sizes:- Purpose: The ith element represents the number of neurons in the ith hidden layer.
- Impact: Defines the complexity of the model.
max_iter:- Purpose: Maximum number of iterations (epochs) for the solver to converge.
- Impact: Increasing this can help the model learn more complex patterns but also increases training time and risk of overfitting.
early_stopping:- Purpose: Whether to use early stopping to terminate training when validation score is not improving.
- Impact: Helps prevent overfitting and reduces training time.
random_state:- Purpose: Determines random number generation for weights and bias initialization, and for data splitting if
early_stoppingis true. - Impact: Ensures reproducibility of results.
- Purpose: Determines random number generation for weights and bias initialization, and for data splitting if
Example Usage (within ObjML.train_cost_sensitive_classifier):
# Example of how MLPClassifier is instantiated and fitted within ObjML
model = MLPClassifier(
hidden_layer_sizes=(100,),
max_iter=500,
random_state=42,
early_stopping=True
)
# sample_weights would be computed based on class_weights and y_train
model.fit(X_train, y_train, sample_weight=sample_weights)
¶ Weight of Evidence (WOE) and Information Value (IV)
Weight of Evidence (WOE) and Information Value (IV) are statistical techniques used in credit scoring and risk modeling to:
- Assess the predictive power of individual features
- Transform categorical variables into continuous scores
- Establish monotonic relationships between features and target variables
¶ Understanding Weight of Evidence (WOE)
WOE measures the strength of a categorical variable in separating "good" outcomes from "bad" outcomes.
Formula:
WOE = ln(% of Goods / % of Bads)
Where:
- % of Goods = (Number of Goods in category / Total Goods)
- % of Bads = (Number of Bads in category / Total Bads)
Interpretation:
- Positive WOE: Category has higher proportion of goods than bads (favorable)
- Negative WOE: Category has higher proportion of bads than goods (unfavorable)
- WOE ≈ 0: Category has similar proportions of goods and bads (neutral)
Example:
Suppose we're analyzing "Employment Status" for credit default prediction:
| Employment Status | Total | Bad (Default) | Good (No Default) | % Bad | % Good | WOE |
|---|---|---|---|---|---|---|
| Full-time | 500 | 50 | 450 | 25% | 56.25% | ln(56.25/25) = 0.81 |
| Part-time | 200 | 80 | 120 | 40% | 15% | ln(15/40) = -0.98 |
| Unemployed | 300 | 70 | 230 | 35% | 28.75% | ln(28.75/35) = -0.20 |
In this example:
- Full-time workers have positive WOE (0.81) → lower default risk
- Part-time workers have negative WOE (-0.98) → higher default risk
- Unemployed have slightly negative WOE (-0.20) → slightly higher default risk
¶ Understanding Information Value (IV)
IV measures the overall predictive power of a feature. It aggregates WOE across all categories.
Formula:
IV = Σ [(% of Goods - % of Bads) × WOE]
Interpretation Rules:
| IV Range | Predictive Power |
|---|---|
| < 0.02 | Useless |
| 0.02 - 0.1 | Weak |
| 0.1 - 0.3 | Medium |
| 0.3 - 0.5 | Strong |
| > 0.5 | Suspicious (may be overfitting) |
Example Calculation:
Using the employment status example above:
| Category | % Good - % Bad | WOE | IV Contribution |
|---|---|---|---|
| Full-time | 56.25% - 25% = 0.3125 | 0.81 | 0.3125 × 0.81 = 0.253 |
| Part-time | 15% - 40% = -0.25 | -0.98 | -0.25 × (-0.98) = 0.245 |
| Unemployed | 28.75% - 35% = -0.0625 | -0.20 | -0.0625 × (-0.20) = 0.0125 |
Total IV = 0.253 + 0.245 + 0.0125 = 0.51 → Strong predictor (but borderline suspicious)
¶ WOE/IV for Multi-Class Classification
For multi-class targets (e.g., Credit Score: Good, Standard, Poor), we use a one-vs-rest approach:
# Example: Calculate WOE/IV for "Good" vs. "Standard + Poor"
obj_ml = ObjML()
woe_map_good, iv_good = obj_ml.calculate_woe_iv_multiclass(
df=df,
feature='employment_status',
target='credit_score',
target_class='Good'
)
# Repeat for each class
woe_map_standard, iv_standard = obj_ml.calculate_woe_iv_multiclass(
df=df,
feature='employment_status',
target='credit_score',
target_class='Standard'
)
¶ Credit Scorecards
A credit scorecard translates model coefficients into easy-to-interpret point values. Each feature contributes a certain number of points, and the total score determines the credit decision.
¶ Scorecard Scaling Parameters
-
PDO (Points to Double the Odds): How many points it takes to double the odds of being "good"
- Common value: 20 points
- Example: If score 600 = 50:1 odds, then score 620 = 100:1 odds
-
Base Score: The score at a specific base odds
- Common value: 500-600 points
- Example: 500 points at 50:1 odds
-
Base Odds: The odds of being "good" at the base score
- Common value: 50:1 (meaning 50 goods for every 1 bad)
¶ Binary Scorecard Example
from sklearn.linear_model import LogisticRegression
# Train model
model = LogisticRegression()
model.fit(X_train, y_train)
# Create scorecard
obj_ml = ObjML()
scorecard = obj_ml.scorecard_scaling(
model=model,
pdo=20,
base_odds=50.0,
base_score=500
)
# Result: {'Intercept': 485.2, 'age': 12.5, 'income': 8.3, ...}
¶ Multi-Class Scorecards
Multi-class scorecards output separate scores for each class, allowing ranking (e.g., "Good", "Standard", "Poor").
¶ Complete Example: Credit Score Classification
This example demonstrates building a 3-class scorecard using the Credit Score Classification dataset (100,000 samples with classes: Good, Standard, Poor).
¶ Step 1: Load and Prepare Data
import sys
import os
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
# Add paths
base_path = os.getcwd()
paths = ["", "/factory.core", "/factory.service", "/factory.learn"]
for relative_path in paths:
if (base_path + relative_path) not in sys.path:
sys.path.append(base_path + relative_path)
from ObjMLDatasets import ObjMLDatasets
# Initialize and download dataset
obj_datasets = ObjMLDatasets(0)
obj_datasets.download_data(dataset_name="credit_score_classification")
# Load dataset using Kaggle API
df = obj_datasets.load_data_from_kaggle(
dataset_name="credit_score_classification",
kaggle_dataset="parisrohan/credit-score-classification",
kaggle_file="train.csv"
)
print(f"Dataset shape: {df.shape}")
print(f"Target distribution:\n{df['Credit_Score'].value_counts()}")
¶ Step 2: Feature Engineering with WOE
from ObjML import ObjML
obj_ml = ObjML()
# Select categorical features for WOE transformation
categorical_features = ['Occupation', 'Credit_Mix', 'Payment_of_Min_Amount',
'Payment_Behaviour']
target_column = 'Credit_Score'
# Calculate WOE for each class using one-vs-rest
classes = ['Good', 'Standard', 'Poor']
woe_maps = {}
for class_label in classes:
woe_maps[class_label] = {}
print(f"\n=== Calculating WOE/IV for class: {class_label} ===")
for feature in categorical_features:
woe_map, iv = obj_ml.calculate_woe_iv_multiclass(
df=df,
feature=feature,
target=target_column,
target_class=class_label
)
woe_maps[class_label][feature] = woe_map
print(f"{feature}: IV = {iv:.4f} ", end="")
if iv < 0.02:
print("(Useless)")
elif iv < 0.1:
print("(Weak)")
elif iv < 0.3:
print("(Medium)")
elif iv < 0.5:
print("(Strong)")
else:
print("(Very Strong/Suspicious)")
# Apply WOE transformation for one class (e.g., 'Good')
df_woe = df.copy()
for feature in categorical_features:
woe_col_name = f"{feature}_WOE_Good"
df_woe[woe_col_name] = df_woe[feature].map(woe_maps['Good'][feature])
df_woe[woe_col_name].fillna(0, inplace=True) # Handle unseen categories
¶ Step 3: Train One-vs-Rest Models
# Prepare features
numeric_features = ['Age', 'Annual_Income', 'Monthly_Inhand_Salary',
'Num_Bank_Accounts', 'Num_Credit_Card', 'Interest_Rate',
'Num_of_Loan', 'Delay_from_due_date', 'Num_of_Delayed_Payment',
'Num_Credit_Inquiries', 'Outstanding_Debt',
'Credit_Utilization_Ratio', 'Total_EMI_per_month',
'Amount_invested_monthly', 'Monthly_Balance']
woe_features = [f"{f}_WOE_Good" for f in categorical_features]
all_features = numeric_features + woe_features
X = df_woe[all_features]
y = df_woe[target_column]
# Handle missing values
X = X.fillna(X.median())
# Split data
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
# Train one-vs-rest models
models = {}
for class_label in classes:
print(f"\nTraining model for class: {class_label}")
# Create binary target (1 if class_label, 0 otherwise)
y_binary = (y_train == class_label).astype(int)
# Train logistic regression
model = LogisticRegression(
max_iter=1000,
solver='liblinear',
random_state=42
)
model.fit(X_train, y_binary)
models[class_label] = model
# Evaluate on training set
train_accuracy = model.score(X_train, y_binary)
print(f"Training accuracy for {class_label}: {train_accuracy:.4f}")
¶ Step 4: Generate Multi-Class Scorecard
# Generate scorecard for all classes
scorecard_multiclass = obj_ml.scorecard_scaling_multiclass(
models=models,
pdo=20,
base_odds=50.0,
base_score=500
)
# Display scorecard summary
for class_label, scorecard in scorecard_multiclass.items():
print(f"\n=== Scorecard for {class_label} ===")
print(f"Intercept: {scorecard['Intercept']:.2f}")
print("\nTop 10 influential features:")
# Sort features by absolute point value
feature_points = {k: v for k, v in scorecard.items() if k != 'Intercept'}
sorted_features = sorted(feature_points.items(),
key=lambda x: abs(x[1]),
reverse=True)[:10]
for feature, points in sorted_features:
print(f" {feature}: {points:+.2f} points")
¶ Step 5: Apply Scorecard and Predict
# Apply scorecard to test set
scores = {}
for class_label, scorecard in scorecard_multiclass.items():
# Start with intercept
score = np.full(len(X_test), scorecard['Intercept'])
# Add feature contributions
for feature, points in scorecard.items():
if feature != 'Intercept' and feature in X_test.columns:
score += X_test[feature].values * points
scores[class_label] = score
# Create DataFrame with scores
scores_df = pd.DataFrame(scores, index=X_test.index)
scores_df['Predicted_Class'] = scores_df.idxmax(axis=1) # Highest score wins
scores_df['Actual_Class'] = y_test
print("\n=== Sample Predictions ===")
print(scores_df[['Good', 'Standard', 'Poor', 'Predicted_Class', 'Actual_Class']].head(10))
# Calculate accuracy
accuracy = (scores_df['Predicted_Class'] == scores_df['Actual_Class']).mean()
print(f"\nMulti-class scorecard accuracy: {accuracy:.4f}")
# Confusion matrix
from sklearn.metrics import confusion_matrix, classification_report
cm = confusion_matrix(y_test, scores_df['Predicted_Class'], labels=classes)
print("\n=== Confusion Matrix ===")
print(pd.DataFrame(cm, index=classes, columns=classes))
print("\n=== Classification Report ===")
print(classification_report(y_test, scores_df['Predicted_Class']))
¶ Step 6: Interpret Results
# Analyze score distributions by class
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
for idx, class_label in enumerate(classes):
scores_df_class = scores_df[scores_df['Actual_Class'] == class_label]
axes[idx].hist(scores_df_class['Good'], alpha=0.5, bins=30, label='Good Score')
axes[idx].hist(scores_df_class['Standard'], alpha=0.5, bins=30, label='Standard Score')
axes[idx].hist(scores_df_class['Poor'], alpha=0.5, bins=30, label='Poor Score')
axes[idx].set_title(f'Score Distribution for Actual {class_label}')
axes[idx].set_xlabel('Score')
axes[idx].set_ylabel('Frequency')
axes[idx].legend()
plt.tight_layout()
plt.savefig('multiclass_scorecard_distributions.png')
print("\nScore distribution plot saved as 'multiclass_scorecard_distributions.png'")
¶ Key Insights from Multi-Class Scorecards
-
Class Separation: Good scorecards show clear separation between score distributions for different classes
-
Feature Importance: Features with larger absolute point values have more influence on the final score
-
Decision Rules: Instead of binary accept/reject, multi-class scorecards enable:
- Tiered pricing (different interest rates for Good/Standard/Poor)
- Approval thresholds (auto-approve Good, manual review Standard, auto-reject Poor)
- Risk-based limits (higher credit limits for Good scores)
-
Interpretability: Stakeholders can understand:
- "This applicant scored 520 for Good, 480 for Standard, 430 for Poor → Classified as Good"
- "Adding 5 years of credit history would increase the Good score by 45 points"
¶ Best Practices
-
Feature Selection: Only include features with IV > 0.02 (avoid useless predictors)
-
WOE Stability: Ensure WOE values are monotonic (consistently increasing or decreasing)
-
Sample Size: Each category should have sufficient samples (at least 5% of total)
-
Missing Values: Handle with separate category or domain-specific imputation
-
Validation: Always validate on out-of-time samples to ensure scorecard stability
-
Documentation: Document:
- WOE/IV for each feature
- Point allocation logic
- Score cutoffs and decision rules
- Model performance metrics
¶ Model Persistence
The ObjML class provides robust model persistence functionality, allowing you to save trained models to the database with full metadata tracking, version control, and easy retrieval. This is essential for production deployments, model monitoring, and regulatory compliance.
¶ Key Features
- Database Storage: Models are serialized and stored in MySQL with LONGBLOB support
- Version Control: Track multiple versions of the same model
- Metadata Tracking: Automatically captures training date, Python version, scikit-learn version
- Preprocessor Support: Save and load encoders (OneHotEncoder, etc.) alongside models
- Training Metrics: Store evaluation metrics with each model version
- CLI Management: Command-line tools for listing and inspecting models
- Active Model Flagging: Mark models as active for production use
¶ Database Schema
Models are stored in the def_ml_models table:
CREATE TABLE def_ml_models (
model_id VARCHAR(36) NOT NULL,
model_name VARCHAR(255) NOT NULL,
version VARCHAR(50) NOT NULL,
package VARCHAR(255) NOT NULL,
model_type VARCHAR(100),
model_binary LONGBLOB NOT NULL,
encoder_binary LONGBLOB,
metadata_json JSON,
feature_names JSON,
training_metrics JSON,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
created_by VARCHAR(255),
is_active BOOLEAN DEFAULT FALSE,
PRIMARY KEY (model_id, package),
UNIQUE KEY unique_name_version (model_name, version, package)
);
Note: The package field is automatically populated using self.get_package() and is part of the primary key to support multi-package environments.
¶ Saving Models
¶ Basic Model Saving
from ObjML import ObjML
from sklearn.linear_model import LogisticRegression
import numpy as np
# Train a model
X_train = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y_train = np.array([0, 1, 0, 1])
model = LogisticRegression()
model.fit(X_train, y_train)
# Save to database
obj_ml = ObjML(0)
model_id = obj_ml.save_model_to_db(
model=model,
model_name="credit_risk_model",
version="v1.0",
feature_names=["age", "income"],
training_metrics={
"accuracy": 0.95,
"precision": 0.93,
"recall": 0.92,
"f1_score": 0.93
}
)
print(f"Model saved with ID: {model_id}")
¶ Saving with Encoder
from sklearn.preprocessing import OneHotEncoder
# Create and fit encoder
encoder = OneHotEncoder()
encoder.fit([['A'], ['B'], ['C']])
# Train model
model = LogisticRegression()
model.fit(X_train, y_train)
# Save model with encoder
model_id = obj_ml.save_model_to_db(
model=model,
model_name="credit_risk_model_with_encoder",
version="v1.0",
feature_names=["category", "amount"],
training_metrics={"accuracy": 0.90},
encoder=encoder,
created_by="data_scientist_1",
is_active=True # Mark as active version
)
¶ Saving Multiple Versions
# Save v1.0
model_id_v1 = obj_ml.save_model_to_db(
model=model_v1,
model_name="fraud_detection",
version="v1.0",
feature_names=feature_list,
training_metrics={"auc": 0.85}
)
# Later, save v2.0 with improvements
model_id_v2 = obj_ml.save_model_to_db(
model=model_v2,
model_name="fraud_detection",
version="v2.0",
feature_names=feature_list,
training_metrics={"auc": 0.92},
is_active=True # Make v2.0 the active version
)
¶ Loading Models
¶ Load Latest Version
# Load the most recent version
model, metadata = obj_ml.load_model_from_db(
model_name="credit_risk_model",
version="latest"
)
# Make predictions
X_test = np.array([[2, 3], [4, 5]])
predictions = model.predict(X_test)
# Access metadata
print(f"Model ID: {metadata['model_id']}")
print(f"Model Type: {metadata['model_type']}")
print(f"Features: {metadata['feature_names']}")
print(f"Training Metrics: {metadata['training_metrics']}")
print(f"Created: {metadata['created_at']}")
¶ Load Specific Version
# Load a specific version
model, metadata = obj_ml.load_model_from_db(
model_name="fraud_detection",
version="v1.0"
)
# Use loaded encoder if available
if metadata['encoder']:
encoder = metadata['encoder']
transformed = encoder.transform(categorical_data)
¶ Production Deployment Example
def predict_credit_risk(customer_data):
"""Production prediction function."""
obj_ml = ObjML(0)
# Load active model
model, metadata = obj_ml.load_model_from_db(
model_name="credit_risk_model",
version="latest"
)
# Prepare features in correct order
features = customer_data[metadata['feature_names']]
# Apply encoder if present
if metadata['encoder']:
features = metadata['encoder'].transform(features)
# Make prediction
prediction = model.predict(features)
probability = model.predict_proba(features)
return {
"prediction": prediction[0],
"probability": probability[0][1],
"model_version": metadata['model_id'],
"model_type": metadata['model_type']
}
¶ Listing Models
¶ List All Models
# Get all models
models = obj_ml.list_models()
for m in models:
print(f"{m['model_name']} v{m['version']} - {m['model_type']}")
print(f" Created: {m['created_at']}")
print(f" Features: {m['feature_count']}")
print(f" Active: {m['is_active']}")
¶ Filter by Model Name
# Get all versions of a specific model
models = obj_ml.list_models(model_name="fraud_detection")
# Display versions
for m in models:
status = "ACTIVE" if m['is_active'] else ""
print(f" {m['version']} - {m['created_at']} {status}")
¶ Limit Results
# Get 10 most recent models
recent_models = obj_ml.list_models(limit=10)
¶ CLI Commands
¶ Create Table
python factory.core/ObjML.py create-table
¶ List Saved Models
# List all models
python factory.core/ObjML.py list-saved-models
# Filter by name
python factory.core/ObjML.py list-saved-models --model credit_risk_model
# Limit results
python factory.core/ObjML.py list-saved-models --limit 5
Note: Package is automatically determined from self.get_package() - no need to specify it.
Output example:
Saved ML Models
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Model Name ┃ Version ┃ Type ┃ Features ┃ Created ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ credit_risk_model │ v2.0 │ LogisticRegression │ 15 │ 2025-12-26 10:30:00 │ ✓ │
│ credit_risk_model │ v1.0 │ LogisticRegression │ 12 │ 2025-12-25 09:15:00 │ │
│ fraud_detection │ v1.5 │ RandomForest... │ 20 │ 2025-12-24 14:20:00 │ ✓ │
└─────────────────────┴─────────┴────────────────────┴──────────┴─────────────────────┴────────┘
Total: 3 model(s)
¶ Show Model Details
# Show latest version
python factory.core/ObjML.py show-model credit_risk_model
# Show specific version
python factory.core/ObjML.py show-model fraud_detection --version v1.0
Output example:
Model: credit_risk_model
Model ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Type: LogisticRegression
Created: 2025-12-26 10:30:00
Metadata:
training_date: 2025-12-26T10:30:00.123456
python_version: 3.12.3
sklearn_version: 1.3.0
model_class: LogisticRegression
Features (15):
1. age
2. income
3. credit_score
4. employment_length
5. debt_to_income
... and 10 more
Training Metrics:
accuracy: 0.95
precision: 0.93
recall: 0.92
f1_score: 0.93
¶ Complete Workflow Example
from ObjML import ObjML
from ObjMLDatasets import ObjMLDatasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score
# 1. Load dataset
datasets = ObjMLDatasets()
df = datasets.load_dataset("german_credit")
# 2. Prepare data
X = df.drop("class", axis=1)
y = df["class"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
# 3. Train model
obj_ml = ObjML(0)
model, metrics, y_pred, X_test, y_test = obj_ml.train_cost_sensitive_classifier(
X=X_train,
y=y_train,
class_weights={0: 1, 1: 5},
model_type="LogisticRegression"
)
# 4. Calculate evaluation metrics
y_pred_test = model.predict(X_test)
training_metrics = {
"accuracy": accuracy_score(y_test, y_pred_test),
"precision": precision_score(y_test, y_pred_test),
"recall": recall_score(y_test, y_pred_test),
"confusion_matrix": metrics["confusion_matrix"]
}
# 5. Save model
model_id = obj_ml.save_model_to_db(
model=model,
model_name="german_credit_model",
version="v1.0",
feature_names=list(X_train.columns),
training_metrics=training_metrics,
created_by="ml_engineer",
is_active=True
)
print(f"Model saved: {model_id}")
# 6. Later: Load and use model
loaded_model, metadata = obj_ml.load_model_from_db(
model_name="german_credit_model",
version="latest"
)
# Make predictions with loaded model
new_predictions = loaded_model.predict(X_test)
print(f"Predictions made using model {metadata['model_id']}")
¶ Best Practices
-
Version Naming: Use semantic versioning (v1.0, v1.1, v2.0) or date-based versioning (v2025-12-26)
-
Metadata Tracking: Always include comprehensive training metrics for model comparison
-
Feature Consistency: Store feature names to ensure correct order during prediction
-
Encoder Management: Save encoders with models to maintain preprocessing consistency
-
Active Flag: Use
is_active=Truefor production models -
Model Documentation: Document:
- Training dataset details
- Feature engineering steps
- Class weights or sample weights used
- Performance on test/validation sets
- Business rules and thresholds
-
Model Monitoring: Regularly check:
- Prediction performance drift
- Feature distribution changes (PSI)
- Model accuracy over time
-
Rollback Strategy: Keep previous model versions for quick rollback if needed
-
Testing: Always test loaded models on validation data before production deployment
-
Security: Limit database write access to authorized users only
¶ Troubleshooting
¶ Model Load Fails
try:
model, metadata = obj_ml.load_model_from_db(
model_name="my_model",
version="v1.0"
)
except ValueError as e:
print(f"Model not found: {e}")
# List available models
models = obj_ml.list_models(model_name="my_model")
print(f"Available versions: {[m['version'] for m in models]}")
¶ Version Mismatch
If you get warnings about sklearn version mismatch:
model, metadata = obj_ml.load_model_from_db(model_name="my_model")
print(f"Model trained with sklearn {metadata['metadata']['sklearn_version']}")
print(f"Current sklearn version: {sklearn.__version__}")
# Models are generally compatible across minor versions
# For major version differences, consider retraining
¶ Large Model Storage
For very large models (>100MB):
# Consider:
# 1. Model compression techniques
# 2. Feature selection to reduce model complexity
# 3. Increasing MySQL max_allowed_packet setting
# 4. Using separate file storage with database references
¶ Further Reading and Resources
¶ Weight of Evidence (WOE) and Information Value (IV)
-
Listendata - Weight of Evidence (WOE) and Information Value (IV) Explained
- https://www.listendata.com/2015/03/weight-of-evidence-woe-and-information.html
- Comprehensive tutorial with examples and Python code
- Covers WOE calculation, IV interpretation, and binning strategies
-
Towards Data Science - Building a Scorecard using WOE and IV
- https://towardsdatascience.com/building-a-scorecard-with-woe-and-information-value-iv-6b9e5e8c7f15
- Step-by-step guide to building credit scorecards
- Includes practical Python implementations
-
Analytics Vidhya - WOE and IV in Credit Risk Modeling
- https://www.analyticsvidhya.com/blog/2021/06/understand-weight-of-evidence-and-information-value/
- Detailed explanation with credit risk examples
- Discusses binning techniques for continuous variables
-
Sunscrapers - WOE and IV for Feature Selection
- https://sunscrapers.com/blog/weight-of-evidence-and-information-value-explained/
- Focuses on using WOE/IV for feature engineering
- Includes performance comparisons
¶ Credit Scorecards
-
Credit Scoring Toolkit (Naeem Siddiqi)
- Book: "Credit Risk Scorecards: Development and Implementation Using SAS"
- Industry standard reference for scorecard development
- Covers business requirements, model validation, and implementation
-
Scorecard Development Best Practices
- https://www.federalreserve.gov/supervisionreg/srletters/sr0319a1.pdf
- Federal Reserve guidance on model risk management
- Best practices for scorecard validation and monitoring
-
FICO Scorecard Development
- https://www.fico.com/blogs/credit-scoring-101-what-are-scorecards
- Overview of FICO scoring methodology
- Industry perspective on scorecard design
-
Scorecardpy - Python Package Documentation
- https://github.com/ShichenXie/scorecardpy
- Open-source Python package for scorecard development
- Includes WOE binning, IV calculation, and scorecard scaling
¶ Multi-Class Classification and One-vs-Rest
-
Scikit-learn - Multi-class Classification Strategies
- https://scikit-learn.org/stable/modules/multiclass.html
- Official documentation on one-vs-rest and one-vs-one approaches
- Implementation details for multi-class models
-
Machine Learning Mastery - One-vs-Rest and One-vs-One
- https://machinelearningmastery.com/one-vs-rest-and-one-vs-one-for-multi-class-classification/
- Clear explanation of multi-class strategies
- Practical examples with scikit-learn
-
Towards Data Science - Multi-class Logistic Regression
- https://towardsdatascience.com/multi-class-classification-one-vs-all-one-vs-one-94daed32a87b
- Comparison of multi-class approaches
- Performance considerations
¶ Logistic Regression for Credit Scoring
-
Penn State - Logistic Regression (STAT 504)
- https://online.stat.psu.edu/stat504/lesson/6
- Academic introduction to logistic regression
- Covers odds ratios and interpretation
-
Logistic Regression in Credit Risk
- https://www.mdpi.com/2227-7072/9/4/53
- Academic paper on logistic regression for credit scoring
- Discusses model performance and interpretation
-
Interpreting Logistic Regression Coefficients
- https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faq-how-do-i-interpret-odds-ratios-in-logistic-regression/
- UCLA Statistical Consulting guide
- Explains odds ratios and coefficient interpretation
¶ Model Validation and Monitoring
-
Model Risk Management (SR 11-7)
- https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
- Federal Reserve guidance on model risk management
- Covers validation, testing, and ongoing monitoring
-
Gini Coefficient and Model Performance
- https://www.listendata.com/2019/08/gini-coefficient.html
- Explains Gini coefficient for credit scoring
- ROC curves and performance metrics
-
Population Stability Index (PSI)
- https://www.listendata.com/2015/05/population-stability-index.html
- Guide to monitoring scorecard stability over time
- Detecting population drift
¶ Practical Tutorials and Code Examples
-
Kaggle - Credit Scoring Notebooks
- https://www.kaggle.com/code/willcanniford/predict-loan-defaults-using-scorecards
- Real-world credit scoring examples
- Community-contributed implementations
-
GitHub - Credit Risk Modeling
- https://github.com/topics/credit-scoring
- Open-source credit scoring projects
- Various implementations and techniques
-
DataCamp - Building Credit Scorecards in Python
- https://www.datacamp.com/courses/credit-risk-modeling-in-python
- Interactive tutorial (requires subscription)
- Hands-on exercises
¶ Academic Papers
-
Thomas, L.C. - "A survey of credit and behavioural scoring"
- International Journal of Forecasting (2000)
- Foundational paper on credit scoring techniques
- Historical perspective and evolution
-
Baesens, B. et al. - "Benchmarking state-of-the-art classification algorithms"
- Expert Systems with Applications (2003)
- Comparison of classification methods for credit scoring
- Performance analysis
¶ Industry Standards and Regulations
-
Basel II Credit Risk Management
- https://www.bis.org/publ/bcbs107.htm
- International regulatory framework
- Internal ratings-based approach
-
Fair Lending and Model Fairness
- https://www.consumerfinance.gov/fair-lending/
- CFPB guidance on fair lending
- Considerations for model bias and discrimination
-
FICO Model Documentation Standards
- https://www.fico.com/en/latest-thinking/white-paper/model-documentation-best-practices
- Industry best practices for model documentation
- Regulatory compliance considerations
¶ YouTube Channels and Video Tutorials
-
StatQuest with Josh Starmer - Logistic Regression
- https://www.youtube.com/c/joshstarmer
- Clear visual explanations of statistical concepts
- Logistic regression, odds ratios, and classification
-
Krish Naik - Credit Score Classification
- https://www.youtube.com/c/Krishnaik06
- Practical ML tutorials with credit scoring examples
- End-to-end project walkthroughs
¶ Community Forums and Discussion
-
Cross Validated (Stack Exchange)
- https://stats.stackexchange.com/questions/tagged/credit-scoring
- Q&A on credit scoring and WOE/IV
- Expert answers to technical questions
-
r/MachineLearning - Credit Scoring Discussions
- https://www.reddit.com/r/MachineLearning/
- Community discussions on ML for credit risk
- Latest research and techniques
-
LinkedIn Learning - Credit Risk Analytics
- https://www.linkedin.com/learning/topics/credit-risk-analysis
- Professional courses on credit risk modeling
- Industry perspectives and case studies
Note: When implementing credit scorecards in production, always consult with legal and compliance teams to ensure adherence to:
- Fair Credit Reporting Act (FCRA)
- Equal Credit Opportunity Act (ECOA)
- General Data Protection Regulation (GDPR) if applicable
- Local consumer protection laws
Model documentation, validation, and ongoing monitoring are critical for regulatory compliance and ethical AI practices.