The perception stack, and exactly how to build it.
This is the deepest technical page on the site, on purpose. If you're evaluating whether this is real engineering or a wrapper around a generic API, everything you need to judge that is below — including the parts that are hard.
General media-AI services vs. the NVIDIA vision stack
| Capability | General media-AI service | NVIDIA vision stack |
|---|---|---|
| Player detection | Generic person detection tuned for consumer video | PeopleNet / PeopleNet-Transformer, pretrained on dense crowd and field scenes |
| Identity across occlusion | Frame-by-frame detection; identity is not maintained | ReID feature vectors + state estimation in gst-nvtracker |
| Re-entry after leaving frame | New object, new ID, history lost | Target re-association matches terminated tracks on motion + appearance |
| Biomechanics | Not available — no 3D skeletal output | BodyPose3DNet, 34 joints in 3D per athlete per frame |
| Action recognition | Generic label taxonomy, not sport-specific | PoseClassificationNet fine-tuned on your sport with TAO |
| Cost model | Per-minute API pricing that scales linearly with footage | Fixed GPU-hour cost, or near-zero marginal cost on an edge appliance |
Eleven components, one pipeline
NVIDIA DeepStream SDK
The GStreamer-based streaming video analytics pipeline. Open source, hardware-accelerated decode through inference through post-processing. Sports analytics is a named use case. This is the spine everything else plugs into.
PeopleNet / PeopleNet-Transformer
TAO pretrained detection models. They find every body on the field — athletes, officials, and technical staff — at full match resolution without us collecting a detection dataset first.
Gst-nvtracker (NvDCF + NvDeepSORT)
Applies a re-identification network to extract a feature vector per athlete, compares candidates by cosine distance, and combines that with a state estimator for frame-to-frame association. This is what survives occlusion in a crowded penalty box.
Target re-association
When an athlete leaves the frame and comes back, terminated tracks are matched to new detections using both motion history and visual features — so their minutes and their metrics stay on one identity.
BodyPose3DNet
3D pose network predicting 34 body joints — pelvis, hips, knees, ankles, heels, toes, shoulders, elbows, wrists and more. This is the biomechanics engine: symmetry, valgus, landing, and deceleration all derive from it.
PoseClassificationNet
ST-GCN over skeleton sequences, fine-tuned on sport-specific actions. Classifies what the movement was, not just that movement occurred.
NVIDIA TAO Toolkit
Transfer learning that fine-tunes the pretrained models on your own footage — without a large training dataset or a data science team on staff.
Triton Inference Server
Model serving. Multiple models, versioning, dynamic batching, and consistent throughput across cloud GPU and Jetson deployments.
RAPIDS / cuML
GPU-accelerated scoring and the similarity index. Nearest-neighbour search over millions of movement embeddings stays interactive instead of becoming a batch job.
NIM microservices
The LLM that writes the plain-English scout narrative attached to every athlete report, deployed as a containerized inference microservice.
Jetson Orin
Optional pitchside edge appliance so footage never leaves the building. The same containers, deployed on-site, for programs with strict youth-data constraints.
Nine steps, in order
This is the build order we'd follow ourselves. Steps 3 and 5 are where projects die; budget accordingly.
- STEP 01
Apply to NVIDIA Inception
Free — no equity, no fees. Requires an incorporated company under 10 years old, at least one active developer, and a working website. Unlocks preferred hardware and software pricing, partner cloud credits, DLI training credits, and access to the VC network.
VerifyProgram terms and partner offers change. Confirm current requirements and benefits on NVIDIA's official startup program page before you plan around them. - STEP 02
Provision GPU compute
Start on a cloud GPU VM — it's the fastest path to a working pipeline and you can throw it away. If the target deployment is on-premise, order a Jetson AGX Orin dev kit in parallel; edge constraints are easier to design for than to retrofit.
- STEP 03
Install DeepStream and run the stock reference pipeline
End to end, on one real sample game, before writing any custom code. If the reference app can't decode and process your footage, no amount of custom code will fix that — and you want to find out on day one.
# verify the environment and run the reference app deepstream-app --version deepstream-app -c configs/source1_game_sample.txt # confirm decode + inference are actually on the GPU nvidia-smi dmon -s um - STEP 04
Pull the models and build TensorRT engines
Pull PeopleNet and BodyPose3DNet from the NGC catalog, build FP16 TensorRT engines for your exact GPU, and verify throughput before you build anything on top. Engines are hardware-specific — rebuild them when you change GPU class.
ngc registry model download-version \ "nvidia/tao/peoplenet:deployable_quantized_v2.6.1" ngc registry model download-version \ "nvidia/tao/bodypose3dnet:deployable_accuracy_v1.0" trtexec --onnx=bodypose3dnet.onnx \ --saveEngine=bodypose3dnet_fp16.engine \ --fp16 --workspace=4096 # target: sustained real-time or better on a full-resolution match feed - STEP 05
Configure nvtracker with NvDCF + ReID
Enable the re-identification network, tune the cosine-distance threshold, and set the re-association window for how long an athlete can be gone before their track is retired.
[tracker] tracker-width=960 tracker-height=544 ll-lib-file=libnvds_nvmultiobjecttracker.so ll-config-file=config_tracker_NvDCF_accuracy_ReID.yml enable-past-frame=1 enable-batch-process=1WarningTune re-association parameters against your specific sport. Same-uniform athletes in dense occlusion is the hard case, and it is where most projects fail. Budget more time here than feels reasonable — then add more. - STEP 06
Build homography calibration
Pixel coordinates are meaningless; field coordinates are the whole point. Use a semi-automatic approach — an operator clicks four known field corners once per camera position — which is far more robust than auto-detecting field lines on worn grass, in rain, or under floodlights.
import cv2, numpy as np # operator clicks 4 field corners once per camera position src = np.float32(clicked_pixel_points) # image space dst = np.float32([[0,0],[105,0],[105,68],[0,68]]) # metres H, _ = cv2.findHomography(src, dst) field_xy = cv2.perspectiveTransform(track_points, H) - STEP 07
Fine-tune PoseClassificationNet with TAO
Roughly 200 labeled clips per action class is enough to get useful sport-specific classification out of transfer learning. Label from your own footage — clips from a different competition level generalize worse than people expect.
tao model pose_classification train \ -e specs/train_sport_actions.yaml \ -r results/pose_cls_v1 \ -k $NGC_KEY # ~200 labeled clips per class; hold out one full match for evaluation - STEP 08
Fuse optional wearable data by timestamp
If the program runs GPS or heart-rate wearables, pull them via API and align to match clock. Physiological Response uses this pillar's richest inputs when it's present and degrades gracefully to video-derived intensity when it isn't.
- STEP 09
Compute the score, persist, publish
Run the five pillars on RAPIDS, apply maturation normalization, write to Azure SQL against the unified athlete ID, and trigger the Power BI refresh. From here it's a data product, not a vision problem.
One match, end to end
A single 90-minute game file, from the moment an assistant coach drags it into a folder on Sunday evening to the moment the staff's tablets refresh 48 minutes later. No human touches it in between.
Athlete report card
Braking load consistently higher through the right limb across 34 decel events above 3 m/s².
Knee valgus on landing increases after minute 62 relative to this athlete's own first-half baseline.
"#14 controls tempo from deep and scans more frequently than any midfielder in this age group's dataset, which is why his first touch survives pressure that turns his teammates over. His deceleration asymmetry and late-match landing mechanics are worth a look from performance staff before his minutes increase."
Generated by an NIM-hosted LLM from the computed pillar data. Decision support only — not a medical assessment.
Bring us one game and we'll run this exact pipeline on it.
Same nine steps, your footage, your athletes, your report card. That's the pilot.