Logistic Regression

Logistic Regression solves binary classification problem. It uses Sigmoid function as activation function on top of Linear Regression(can be others) to achieve binary classification. And Binary Cross-Entropy Loss is the corresponding loss function.

So the logit will be:

And putting Sigmoid as activation function on the top:

Binary Cross-Entropy Loss:

Confusion Matrix

Due to the native of binary classification, a model always return 1 can get 50% accuracy rate, which does not make much sense. Thus, in many cases, simply looking at error rate won’t be enough when diagnosing binary classification model, since prediction errors with different cause will be treated differently.

The prediction output and actual output have four type of combination:

Several metrics can be calculated out of the matrix:

• Accuracy:
• Recall:
• Specificity:
• Fall-out:
• Precision:
• Miss rate:
• F-1 Score:
  • Indicating the overall performance of the binary classification.
• Area Under Curve (AUC): Plot FPR and TPR as a curve (called Receiver-Operating Characteristics Curve (ROC)) and calculate the area under the curve.

Multi-Class Classification

Instead of output one value, the model output n(number of classes) value. And by applying a Softmax as activation function on top to get the class and score.

Similar to Logistic Regression, the logit looks like, where k is the kth category / output logit:

Multi-class Cross-Entropy Loss:

Multi-Label Classification

Different from Multi-Class Classification, multi-label classification can output multiple labels as 1.