From Machine Learning to YOLO: How Robots See

Traditional code spells out rules. For example: “If the temperature is above 30°C, turn on the fan.” Machine learning instead uses examples to fit a model: a mathematical function with adjustable parameters.

prediction = f(input; parameters)
ŷ = f(x; θ)

x = input features
ŷ = predicted output
θ = parameters learned from data

Learning means adjusting θ so the model performs well on its task. The model does not memorize a universal truth; it learns patterns from a particular dataset. Whether those patterns hold up in the real world is an engineering question.

Supervised learning learns from examples with target answers. Unsupervised learning discovers patterns without those target answers. These are two major categories—not all of machine learning. Reinforcement learning, for instance, learns from rewards associated with actions.

Suppose we want to predict a delivery robot’s trip duration from distance. Our training data includes pairs of inputs and measured outcomes: distance x and duration y.

distance (meters)    duration (seconds)
10                   9
20                   14
30                   19

Simple model: ŷ = wx + b
w = 0.5 seconds/meter, b = 4 seconds
At x = 40 meters: ŷ = 0.5 × 40 + 4 = 24 seconds

This is linear regression: predict a number with a line. The slope w and intercept b are learned parameters. Real journeys also depend on turns, obstacles, speed limits, and terrain; this deliberately simple model ignores those factors.

How do we judge a proposed line? Use a loss function that measures prediction error. Mean squared error averages the squared difference between predicted and observed values:

MSE = (1/n) Σᵢ (ŷᵢ − yᵢ)²

Observed:  [9, 14, 19]
Predicted: [8, 13, 18]
MSE = (1² + 1² + 1²) / 3 = 1

Gradient descent nudges parameters in the direction that reduces loss: θ ← θ − η∇L(θ). The gradient ∇L tells us which direction increases the loss; η is the learning rate. Too large a step may overshoot. Too small a step may learn slowly.

Not every supervised task predicts a number. Classification predicts a category: spam or not spam, a faulty component or a good one. Object detection adds another target: both the category and the object’s location. A supervised detector learns from images annotated with class labels and bounding boxes.

A model that excels on training images but fails on new images is overfitting. Split datasets carefully: near-identical frames from the same video in both training and test sets can make results look better than they are.

Imagine collecting thousands of robot trips, each described by distance, average speed, and number of stops—but without labels such as “easy” or “congested.” Clustering can group similar trips and help us discover recurring operating conditions.

K-means is a simple example. Choose K cluster centers, assign each point to its nearest center, then replace each center with the average of its assigned points. Repeat until the assignments settle.

Squared distance: d²(x, μ) = Σⱼ (xⱼ − μⱼ)²
Choose centers to minimize: Σᵢ ||xᵢ − μassigned(i)||²

One-dimensional example, K = 2:
Trip distances: [1, 2, 8, 9]
Groups: [1, 2] and [8, 9]
Centers: 1.5 and 8.5

The algorithm finds groups; it does not know their meaning. Calling a group “warehouse traffic” requires human interpretation. Scale features before comparing them: otherwise a large-valued feature, such as distance in meters, may dominate a smaller-valued one.

Applications include customer segmentation, grouping sensor patterns, and exploratory analysis. Anomaly detection can flag observations that differ from normal behavior, but an unusual observation is not automatically a fault.

A color image is an array of red, green, and blue measurements. A 640 × 480 RGB image contains 921,600 channel values. A single line is not enough to model the relationships between all those values.

A neural network stacks learned transformations. A simplified layer computes h = σ(Wx + b): multiply inputs by weights, add a bias, then apply a nonlinear activation σ. Without nonlinearity, stacking linear layers would still be just a linear transformation.

Convolutional layers reuse small learned filters across an image. Early features can respond to local edges and textures; deeper features combine information over larger regions. Many YOLO architectures use these ideas to predict objects at multiple scales.

Training uses labeled examples, a loss, and backpropagation to change weights. Inference applies already-learned weights to a new image. Our webcam experiment runs inference—it does not train a new model or learn your identity.

YOLO stands for You Only Look Once. The original 2016 paper treated detection as a single network prediction over the full image, rather than separately running a classifier on many proposed regions. This single-stage approach made real-time detection a practical focus.

“Once” does not mean one operation or perfect instantaneous vision. Image preparation, many neural-network layers, and output processing still take time. YOLO is now a family of architectures; details such as anchors, heads, losses, and postprocessing differ across versions.

A typical detection includes a class, a bounding box, and a score. A box can be represented by its center and size, (cx, cy, w, h), or by corners, (x₁, y₁, x₂, y₂). A score is not a guarantee or necessarily a calibrated probability.

class: person
score: 0.87
box: [120, 40, 330, 450]  # left, top, right, bottom

For the YOLOX model in this experiment:
person score = objectness × person-class score

Many candidate boxes may describe the same object. Non-maximum suppression (NMS) keeps a strong candidate and removes overlapping, weaker candidates. Intersection over union measures how much two boxes overlap:

IoU(A, B) = area(A ∩ B) / area(A ∪ B)

Two boxes each have area 100; their intersection is 60.
Union = 100 + 100 − 60 = 140
IoU = 60 / 140 ≈ 0.43

A higher confidence threshold removes more weak detections but may miss people. A lower one may detect more people but add false positives. An NMS threshold controls overlap suppression, not confidence. Overlapping people can be challenging because suppression may remove a valid neighboring box.

Evaluate more than speed: precision asks how many detections were correct; recall asks how many real objects were found. Mean average precision (mAP) summarizes detection performance across classes and matching criteria. FPS says nothing by itself about missed people.

YOLO is a perception tool, not a complete autonomous robot. It provides observations that other parts of the robot interpret and act upon.

To estimate 3D position with depth Z, a calibrated pinhole-camera approximation uses X = (u − cₓ)Z/fₓ and Y = (v − cᵧ)Z/fᵧ. Here (u, v) is a pixel, (cₓ, cᵧ) is the camera’s principal point, and (fₓ, fᵧ) are focal lengths in pixels. Without depth or additional assumptions, a 2D box cannot tell us how many meters away an object is.

Occlusion, poor lighting, motion blur, unusual clothing, and changes in environment can all degrade detection. Robots need timestamps, stale-data checks, conservative fallback behavior, and appropriate independent sensing or safety-rated controls—not just a bounding box.

This experiment uses YOLOX-Nano, a lightweight, Apache-2.0-licensed member of the YOLO family. Its pretrained COCO labels include person, not a separate face class. Finding a person is not identifying who they are; face detection and face recognition are different tasks.

Python runs in your browser through Pyodide. ONNX Runtime executes the neural network, while the editable Python code decodes predictions, applies the person-confidence threshold, and removes duplicate boxes. Browser camera APIs supply frames. This is real Python execution, not a server accessing your webcam.

Python webcam experiment

Three experiments worth trying

1. Change CONFIDENCE from 0.45 to 0.75. Compare how many detections remain.
2. Step farther from the camera or partially occlude yourself. Watch for missed detections instead of assuming the model always sees you.
3. Change the lighting and compare the results and time per frame. The displayed latency includes inference and Python processing, not a guaranteed robot response time.

The editor is a focused browser experiment, not a full desktop Python environment. The example uses Python’s standard library; desktop OpenCV camera capture and PyTorch training do not run here. Stop the camera before editing, then run the revised code. Loading time and speed depend on your device and connection.

For faces, choose and validate a face-specific model rather than cropping the top of every person box and calling it face detection. That requires a separate model, preprocessing pipeline, and evaluation.

A demo answers “Can the model detect me in this room?” Deployment asks harder questions: which people and objects must it detect, in which environments, at what latency, and what happens when it fails?

• Data: collect representative examples and labels, respecting consent and privacy.
• Model: choose size, training data, accuracy, export format, and a license appropriate for your use.
• Hardware: test on the actual robot. CPU browser speed does not predict GPU, edge accelerator, or battery behavior.
• Integration: connect time-stamped detections to tracking, calibrated coordinates, planning, and control.
• Failure: measure missed detections and false alarms, then define a safe fallback.

A box on the screen is the beginning of understanding, not the end. Learn where the data came from, what the model predicts, and what it cannot promise. Then build a system that treats those limitations honestly.

Don’t just learn. DO.

• The original YOLO paper — You Only Look Once (2016)
• YOLOX paper — Exceeding YOLO Series in 2021
• Official YOLOX project, model exports, and Apache-2.0 license
• YOLOX ONNX preprocessing and inference
• Scikit-learn: supervised learning and clustering
• Pyodide: Python in the browser
• ONNX Runtime Web