Machine Learning from Scratch: A PhD Researcher's Beginner Guide

I work as an AI engineer at the Defense Threat Reduction Agency. Before that I spent years doing physics simulations and data analysis in nuclear engineering. When people ask me how to get started with machine learning, they usually expect me to recommend a massive online course or point them to a textbook. My actual advice is different: start by understanding what ML is doing mathematically, build intuition with small examples, and only then reach for the big frameworks.

This guide is for people who have zero or near-zero ML background. We will cover the conceptual landscape, then go deep on the two most important ideas — linear regression and neural networks — with enough mathematical detail to understand what is actually happening. Python pseudocode is included throughout so you can see how concepts translate to code.

What Machine Learning Actually Is

Here is the honest definition: machine learning is the process of fitting a mathematical function to data. That function can be simple (a line) or extraordinarily complex (a deep neural network with billions of parameters), but the core idea is the same. You have inputs, you have desired outputs, and you want a function that maps inputs to outputs in a way that generalizes to data you have not seen yet.

Traditional programming is explicit: you write rules. If the input is X, do Y. Machine learning is implicit: you show the algorithm examples and let it infer the rules. This works extremely well for problems where the rules are too complex to write explicitly — recognizing faces, translating language, predicting protein structures.

The Three Questions Every ML Project Must Answer

1. What is the model? What mathematical structure will learn from the data? (Linear model, decision tree, neural network, etc.)
2. What is the loss function? How do we measure how wrong the model's predictions are?
3. What is the optimizer? How do we adjust the model's parameters to reduce the loss?

Almost every ML paper you will ever read answers these three questions. Once you recognize the pattern, reading the literature becomes much easier.

Supervised vs. Unsupervised Learning

The most important distinction in ML is between supervised and unsupervised learning. The difference comes down to whether your training data includes labels.

For beginners, supervised learning is the right place to start because the feedback signal (the label) makes training concrete and measurable. Most practical ML applications are supervised.

Linear Regression: The Foundation

Linear regression is the simplest ML model. It assumes the relationship between input x and output y is linear: y = w · x + b, where w is the weight (slope) and b is the bias (intercept). These two parameters are what the model learns from data.

The Loss Function

For regression, the standard loss is mean squared error (MSE). Given n training examples with true outputs y_i and predicted outputs ŷ_i:

MSE penalizes large errors heavily (because of the square) and is zero only when every prediction is perfect. The goal of training is to find w and b that minimize MSE over the training data.

Gradient Descent

Gradient descent is the optimizer. The intuition: imagine you are standing on a hilly landscape and you want to walk to the lowest point. At each step, look around, find the direction of steepest downhill, and take a small step that way. In ML, the "landscape" is the loss function over all possible parameter values, and the gradient (the multivariable derivative) tells us which direction is "uphill." We move in the opposite direction.

We repeat this update many times (many epochs) until the loss stops decreasing meaningfully. The learning rate is a hyperparameter you choose — too large and the optimizer overshoots and diverges; too small and training takes forever. Choosing a good learning rate is one of the most important practical skills in ML.

Classification and Logistic Regression

When the output is a category rather than a number (spam vs. not spam, cat vs. dog), we use classification. Logistic regression is the simplest classifier. It takes the linear model output and squashes it through the sigmoid function to produce a probability between 0 and 1.

The binary cross-entropy loss penalizes confident wrong answers very heavily, which is the right behavior for classification. If the model says 99% probability and it is wrong, that is much worse than saying 60% probability and being wrong.

Neural Networks: Stacking Linear Models

A neural network is a sequence of linear transformations with nonlinear activation functions applied between them. That is it. The power comes from depth — stacking many layers allows the network to represent extraordinarily complex functions.

A Single Neuron

One neuron takes a vector of inputs x, computes a weighted sum plus bias, then applies an activation function f:

Without nonlinear activation functions, stacking layers would be mathematically equivalent to a single linear model. The activation functions are what give neural networks their ability to approximate any continuous function (the Universal Approximation Theorem).

Feedforward Network Architecture

A standard feedforward network has three types of layers. The input layer receives raw data. Hidden layers learn intermediate representations. The output layer produces the final prediction. Each layer is a collection of neurons, all receiving outputs from the previous layer.

Backpropagation

Training a neural network means adjusting all weights and biases to minimize the loss. We still use gradient descent, but computing the gradients for each parameter requires the chain rule from calculus, applied layer by layer from the output backward through the network. This process is called backpropagation. You almost never implement it by hand — frameworks like PyTorch and TensorFlow handle it automatically through automatic differentiation — but understanding what it is doing conceptually is essential for debugging and designing networks.

Overfitting and the Bias-Variance Tradeoff

The most important concept in practical ML is the distinction between training performance and generalization performance. A model that memorizes training examples but fails on new data is useless. This is called overfitting.

• Underfitting (high bias): The model is too simple to capture the true pattern. A linear model on data with a quadratic relationship will underfit.
• Overfitting (high variance): The model is too complex and memorizes noise in the training data rather than learning the underlying pattern.
• The goal: Find the sweet spot where the model is complex enough to capture the pattern but not so complex that it memorizes noise.

The primary tools for combating overfitting are regularization (adding a penalty term to the loss that discourages large weights), dropout (randomly disabling neurons during training), early stopping (stopping training before the validation loss starts increasing), and more data (always the best solution when available).

Recommended Books