Running reasoning and agentic AI at the edge has been harder than it needs to be. Until recently, models capable of multi-step reasoning were too large to run locally on edge hardware. Developers building agents have had to route inference through a data center, adding network dependency, increasing costs, and exposing data that may need to stay on device.
That constraint is lifting. Several model families released throughout the summer have collectively marked a turning point for edge AI. This new generation of compact open models now delivers reasoning and agentic capabilities that required large data center systems only a few months ago, and NVIDIA Jetson can run them today.
This could power in-cab assistants, real-time anomaly detection, and robots working in harsh or remote environments. On-site experts could spend less time troubleshooting, and critical systems could keep running when connectivity is limited or unavailable.
This post walks through what you need to know to deploy this new generation of open models on Jetson, using Nemotron 3.5 Lightning and Qwen3.8-27B as examples. You’ll learn what to look for when comparing model architectures, how to apply inference optimization techniques to get the most out of your hardware, and how to validate the configuration for your workload.
Specifically, this post answers the following developer questions:
- How do you choose a reasoning model for Jetson?
- How do NVFP4 quantization and speculative decoding improve inference performance?
- How do you serve Nemotron 3.5 Lightning and Qwen3.8-27B with vLLM?
- How do you validate the configuration for your application?
Figure 1, below, shows this shift. It plots the Artificial Analysis Intelligence Index by model size and release date. Open models released in 2026 now reach scores similar to leading models from 2025, while using far fewer parameters.
How do you choose a reasoning model for Jetson?
Better training methods and more efficient architectures are driving this change. For example, distillation transfers some of Nemotron 3 Ultra’s capabilities into the smaller Nemotron 3.5 Lightning model. Different architectures also create different tradeoffs in capability, memory use, and generation speed.
Qwen3.8-27B is a dense model, so it activates all 27 billion parameters for each token. Nemotron 3.5 Lightning uses a mixture-of-experts (MoE) architecture. It has 30 billion total parameters but activates only 3 billion for each token. This makes the two models suitable for different types of workloads.
These differences become especially important for long-running agents. For example, an agent can monitor a system using live sensor data and device logs, take approved corrective actions, verify the results against predefined tests, and escalate to an expert only when needed. All of this runs locally at the edge without an internet connection, keeping latency low.
Nemotron 3.5 Lightning is a good fit for these response-heavy workflows, where faster token generation can shorten the overall process. Qwen3.8-27B is a better fit for tasks that require fewer, harder decisions and allow the agent to spend more time generating each response.
Benchmark both models on the decisions, tools, and response patterns required by your application before selecting one.
On Jetson, these agent loops can run next to the sensors and systems they work with. You can deploy models locally through popular frameworks, including vLLM and llama.cpp, so the reasoning loop does not depend entirely on the data center.
Gemma 4 E4B is a strong starting point for Jetson Orin Nano. For Jetson AGX Orin and Jetson AGX Thor, Nemotron 3.5 Lightning and Qwen3.8-27B are strong options. These model families have high-quality quantized checkpoints and optimized deployment options across popular inference engines.
How do you optimize reasoning-model inference on Jetson?
Two complementary techniques can improve inference performance: NVFP4 quantization reduces the work and memory required for model operations, while speculative decoding can generate multiple accepted tokens per verification step.
Model architecture sets the starting point, but serving choices also affect performance. Figure 2, below, compares both models in BF16, NVFP4, and NVFP4 with the fastest speculative decoding configuration we tested for each model.
As shown in Figure 2, above, we add two optimizations, one at a time. BF16 provides the baseline. NVFP4 adds quantization. The final configuration combines NVFP4 with the fastest speculative decoding configuration we tested for each model.
During decoding, a model generates its response one token at a time. Each token usually requires another pass through the model. This gives you two ways to improve performance: reduce the work in each pass or produce more tokens from each pass.
Quantization takes the first approach. Lower-precision values reduce the amount of data the GPU must move and process during each pass. With a format such as NVFP4, you can improve generation speed and reduce memory use while keeping quality close to BF16.
Speculative decoding takes the second approach. A smaller draft model proposes several tokens, and the main model verifies them together. The main model still makes the final decision. If it accepts several proposed tokens, generation advances by several tokens in one verification step.
Video 1, below, shows a response generation with and without speculative decoding, illustrating the resulting speedup.
Several methods can produce these drafts, including MTP, DFlash, and DSpark. All three run on Jetson, but they generate and evaluate proposals differently. We tested the available methods and draft checkpoints instead of assuming that one configuration would perform best for every model.
The two optimizations complement each other. NVFP4 reduces the cost of each pass, while speculative decoding increases the number of accepted tokens produced from each pass. Together, they improve performance more than either optimization alone.
The fastest speculative decoding configuration differed between the two models. Nemotron 3.5 Lightning performed best with DSpark, while Qwen3.8-27B performed best with DFlash2. Test the method and draft checkpoint with the model you plan to deploy instead of assuming that one configuration will work best for every model.
Prerequisites
Before running the following commands, confirm that you have:
- A Jetson AGX Thor or Jetson AGX Orin
- JetPack 7.2 with NVIDIA Container Runtime and Docker configured.
- Enough storage for the model and draft checkpoints
- Accepted the license terms for NVIDIA Nemotron and Qwen3.8 checkpoints
For Nemotron 3.5 Lightning, you can run the fastest configuration we tested, which combines NVFP4 with DSpark, on Jetson AGX Thor or Jetson AGX Orin.
First, start the vllm/vllm-openai:v0.28.0 container:
docker run --pull=always --runtime nvidia --rm -it \ --network host \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --entrypoint bash \ vllm/vllm-openai:v0.28.0
Then, run the following command inside the container:
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
--reasoning-parser nemotron_v3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-model-len 128000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.7 \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--enable-prefix-caching \
--speculative-config '{"method":"dspark","model":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark","num_speculative_tokens":5}' \
--mamba-backend flashinfer \
--mamba-ssm-cache-dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--mamba-cache-mode align
For Qwen3.8-27B, you can use the same container launched with the command above and run the fastest configuration we tested, which combines NVFP4 with DFlash2, on Jetson AGX Thor or Jetson AGX Orin using the following command:
VLLM_GDN_DECODE_KERNEL=triton vllm serve Inferact/Qwen3.8-27B-NVFP4 \
--served-model-name qwen38 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-model-len 50000 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.85 \
--trust-remote-code \
--speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'
Each method creates and evaluates draft tokens differently, which affects the cost and accuracy of its proposals.
MTP uses prediction heads trained with the main model to propose several future tokens. You can now use MTP with many mainstream model families, including Qwen, Gemma, and Nemotron.
DFlash uses a separate diffusion-based draft model to propose a block of tokens in parallel. It currently supports the widest selection of compatible draft checkpoints. DSpark builds on DFlash by correcting drafts and stopping weak proposals early. DSpark can be faster when a matching checkpoint is available, but it supports fewer checkpoints.
Validate performance with representative workloads
A model-level benchmark can help you identify a strong speculative decoding configuration. However, applications generate different kinds of text, and performance can change with the workload. To measure this effect, we kept the fastest configuration for each model fixed and tested four SpeedBench categories: writing, reasoning, summarization, and retrieval-augmented generation.
The same method remained the fastest for each model across the categories we tested, but throughput still varied by workload. Nemotron 3.5 Lightning with DSpark ranged from 123.01 to 138.02 output tokens/s, while Qwen3.8-27B with DFlash2 ranged from 27.69 to 34.44 output tokens/s.
Validate your speculative decoding configuration with prompts that represent the intended application. A representative dataset will help you choose the method and draft checkpoint that work best for your model and use case.
When should you train a custom checkpoint?
For most applications, start with an existing quantized checkpoint and draft model. This is usually enough to get strong accuracy and a useful speedup without training anything yourself. Before deploying the configuration, test it with prompts from your application. General benchmarks cannot tell you whether a checkpoint preserves the behavior that matters for your data.
If quantization reduces accuracy, use NVIDIA Model Optimizer to fine-tune the quantized model with quantization-aware training or distillation. Quantization-aware training simulates lower precision during training. Quantization-aware distillation also uses a higher-precision teacher model to help the quantized model retain the original model’s behavior.
This additional tuning is most useful for specialized workloads where small accuracy changes matter. To learn how to train a quantized model with either approach, follow the NVIDIA Model Optimizer QAT and QAD tutorial.
Apply the same approach to speculative decoding. A public draft checkpoint can be compatible with your model but still provide less acceleration than expected. The speedup depends on how often the main model accepts the proposed tokens. When the acceptance rate is low, the cost of producing and verifying drafts can reduce the benefit. If this happens, follow the vLLM Speculators training guide to train a compatible speculator with representative application data. Speculators supports methods including MTP, EAGLE-3, DFlash, and DSpark. After training, measure both the draft-token acceptance rate and decode throughput on your own prompts.
Most applications will not need custom training. Start with the available checkpoints, measure their accuracy and performance, and train your own only when those results show a clear gap.
Get started
Jetson supports the latest open models with optimized runtimes, quantized checkpoints, and speculative decoding. With this foundation in place, you can move from testing models to building and shipping edge applications.
To learn more about these models, see how to benchmark them, find recommended recipes, and compare performance across Jetson platforms, visit the Jetson AI Lab Models page.
For more hands-on guidance, see our tutorials on running LLMs and VLMs on Jetson, benchmarking generative AI models, and getting started with speculative decoding on popular frameworks.