Stochastic Gradient Descent (SGD) is an optimization algorithm widely used in machine learning and deep learning. Unlike traditional gradient descent, which computes the gradient of the loss function using the entire dataset, SGD updates the model parameters using only a single training example at each iteration. This approach significantly reduces the computational cost, especially for large datasets, as the cost per iteration drops from (\mathcal{O}(n)) to (\mathcal{O}(1)) ².

In SGD, at each iteration, a random index is selected from the training dataset, and the gradient is computed based on that single example. This randomness introduces noise into the optimization process, which can help escape local minima and potentially lead to better convergence in some cases. However, it may also result in a more erratic convergence path compared to traditional gradient descent ³.

SGD is particularly effective for fitting linear classifiers and regressors under convex loss functions, making it a popular choice in various machine learning frameworks, including Scikit-learn ⁴.