In machine learning, local case-control sampling is an algorithm used to reduce the complexity of training a logistic regression classifier. The algorithm reduces the training complexity by selecting a small subsample of the original dataset for training. It assumes the availability of an (unreliable) pilot estimation of the parameters. It then performs a single pass over the entire dataset using the pilot estimation to identify the most "surprising" samples. In practice, the pilot may come from prior knowledge or training using a subsample of the dataset. The algorithm is most effective when the underlying dataset is imbalanced. It exploits the structures of conditional imbalanced datasets more efficiently than alternative methods, such as case control sampling and weighted case control sampling.
Imbalanced datasets In classification, a dataset is a set of N data points ( x i , y i ) i = 1 N {\displaystyle (x_{i},y_{i})_{i=1}^{N}} , where x i ∈ R d {\displaystyle x_{i}\in \mathbb {R} ^{d}} is a feature vector, y i ∈ { 0 , 1 } {\displaystyle y_{i}\in \{0,1\}} is a label. Intuitively, a dataset is imbalanced when certain important statistical patterns are rare. The lack of observations of certain patterns does not always imply their irrelevance. For example, in medical studies of rare diseases, the small number of infected patients (cases) conveys the most valuable information for diagnosis and treatments. Formally, an imbalanced dataset exhibits one or more of the following properties:
Marginal Imbalance. A dataset is marginally imbalanced if one class is rare compared to the other class. In other words, P ( Y = 1 ) ≈ 0 {\displaystyle \mathbb {P} (Y=1)\approx 0} . Conditional Imbalance. A dataset is conditionally imbalanced when it is easy to predict the correct labels in most cases. For example, if X ∈ { 0 , 1 } {\displaystyle X\in \{0,1\}} , the dataset is conditionally imbalanced if P ( Y = 1 ∣ X = 0 ) ≈ 0 {\displaystyle \mathbb {P} (Y=1\mid X=0)\approx 0} and P ( Y = 1 ∣ X = 1 ) ≈ 1 {\displaystyle \mathbb {P} (Y=1\mid X=1)\approx 1} .
Algorithm outline In logistic regression, given the model θ = ( α , β ) {\displaystyle \theta =(\alpha ,\beta )} , the prediction is made according to P ( Y = 1 ∣ X ; θ ) = p ~ θ ( x ) = exp ( α + β T x ) 1 + exp ( α + β T x ) {\displaystyle \mathbb {P} (Y=1\mid X;\theta )={\tilde {p}}_{\theta }(x)={\frac {\exp(\alpha +\beta ^{T}x)}{1+\exp(\alpha +\beta ^{T}x)}}} . The local-case control sampling algorithm assumes the availability of a pilot model θ ~ = ( α ~ , β ~ ) {\displaystyle {\tilde {\theta }}=({\tilde {\alpha }},{\tilde {\beta }})} . Given the pilot model, the algorithm performs a single pass over the entire dataset to select the subset of samples to include in training the logistic regression model. For a sample ( x , y ) {\displaystyle (x,y)} , define the acceptance probability as a ( x , y ) = | y − p ~ θ ~ ( x ) | {\displaystyle a(x,y)=|y-{\tilde {p}}_{\tilde {\theta }}(x)|} . The algorithm proceeds as follows:
… excerpt ends here. Continue reading the full article.
