Building
On-Device AI Apps
with Flutter & Gemma
How most AI features work today
Every AI feature you've shipped
works the same way.
The round trip
That round trip isn't free.
The user waits for the network before the first word appears, and longer on a weak connection.
You pay per token, so the bill grows as more people use the feature.
Whatever the user typed, photographed or recorded leaves their phone.
No signal means no feature, whether that's on a flight or on patchy mobile data.
So here's the question
What if the model
lived
inside
your app?
Meet
Gemma
Google's family of open models, including the one we'll run on the device today.
The Gemma 4 lineup
Five sizes, and two are made for phones.
128K-token context. Takes text, images and audio.
Google says the “E” stands for “effective” parameters. Per-Layer Embeddings keep the working memory close to that of a much smaller model.
256K-token context.
In 2024, the smallest Gemma could only read text
running on a phone.
It understands text, images and audio, and was pre-trained on 140+ languages.
On an iPhone 17 Pro it starts replying in 0.3 s and writes 56.5 tokens a second, faster than anyone can read.
Speeds from the litert-community/gemma-4-E2B-it-litert-lm model card: 1,024-token prompt, 256 generated tokens, 2,048-token context. Parameters and languages from Google's Gemma 4 E2B model card.
Who has no other option
Some apps can't use the cloud.
Summarise a consultation while it happens.
Patient notes are not allowed to reach someone else's server.
An assistant for the technician on site.
Basements, rigs, aircraft, rural roads — there is no signal where the work is.
Read an ID or a claim form from a photo.
The document never has to leave the customer's phone.
Describe what the camera is pointed at.
It has to answer in the moment, even with no connection at all.
How it
works
From your Dart code down to the chip.
What's inside the device
You only write the top layer.
Google's runtime for running LLMs on devices. It also powers on-device AI in Chrome, Chromebook Plus and Pixel Watch.
flutter_edge_ai talks to the runtime straight from Dart through dart:ffi, on Android, iOS and desktop.
How the package is put together
A small core, plus what you actually use.
flutter_edge_aithe core, always neededflutter_edge_ai_litertlm.litertlm engine: mobile, desktop, webflutter_edge_ai_speechspeech-to-text and text-to-speechflutter_edge_ai_mediapipe.task engine: mobile, webflutter_edge_ai_ragon-device RAGflutter_edge_ai_agentagent skillsflutter_edge_ai_onnxONNX RuntimeEvery engine comes with its own native libraries. Splitting them up means your app only ships the ones it uses.
The core doesn't register an engine for you. Forget to add one and the app still compiles, then fails at runtime.
Picking a model format
One model file for every platform.
.litertlm
The LiteRT-LM format, and what this demo uses.
.task
The MediaPipe format, used by older models.
Same code, different weights
Gemma is one choice, not the only one.
SmolLM 135M135 MBLFM2.5 230M168 MBFunctionGemma 270M284 MBFunctionGemma does one job: it calls your functions.
Qwen3 0.6B586 MBQwen 2.5 1.5B1.6 GBDeepSeek R1 1.5B1.7 GBPhi-4 Mini3.9 GBQwen3 and DeepSeek R1 also stream their reasoning.
SmolVLM2 500M360 MBFastVLM 0.5B0.5 GBQwen2-VL 2B1.8 GBPick the smallest one that can do the job.
Gemini Nano on Android, Apple Foundation Models on iOS and macOS, Phi Silica on Windows. Nothing to download — but it is a different model on every platform, and it may be switched off.
The file you install, and the ModelType you pass. Everything after that is the same code.
Let me
show you
One Flutter app running Gemma 4 E2B, entirely on the device.
Show it a photo.
assets/demo/vision.mp4Vision chat: attach a photo, ask “What's in this image?”, record until the answer finishes. 20–40 s.
Watch it think.
assets/demo/thinking.mp4Thinking mode: tap the bat-and-ball suggestion, expand “Reasoning” while it streams. 30–60 s.
It calls your Dart.
assets/demo/function-calling.mp4Function calling: send “Add buy milk, then tell me what's on my list”. Show both tool bubbles and the task strip update. 20–40 s.
Talk to it.
assets/demo/voice-loop.mp4Voice loop, push-to-talk, recorded WITH system audio: ask a short question and let it speak the reply. 20–40 s.
Now turn off the Wi‑Fi.
assets/demo/airplane-mode.mp4Turn Wi-Fi off in Control Center on camera, then send a new prompt in Vision or Thinking. 20–40 s.
What you just watched
Three models, one download each.
Whisper Tiny151 MB
Your voice becomes text. It listens in 30-second windows.
Gemma 4 E2B2.6 GB
Text and images in, the answer out. Every demo you just saw used this one.
Inflect-Nano36 MB
The answer becomes speech. English only, and fast enough to keep a conversation going.
About 2.8 GB in total. Downloaded once, then nothing leaves the device.
Build it
The code behind the demo, exactly as it runs.
Step 1 · Packages and engine
Add the packages, then register the engine.
pubspec.yamldependencies: flutter_edge_ai: ^2.1.1 flutter_edge_ai_litertlm: ^1.10.1
flutter_edge_ai needs Flutter 3.44+ and Dart 3.12+, because its native libraries come through Native Assets build hooks.
main.dartFuture<void> main() async { WidgetsFlutterBinding.ensureInitialized(); await FlutterEdgeAi.initialize( inferenceEngines: const [LiteRtLmEngine()], ); runApp(const App()); }
No Hugging Face token needed. The litert-community build of Gemma 4 E2B is public.
Step 2 · Download, load, open a chat
Download it once, then load it on the GPU.
gemma_service.dartawait FlutterEdgeAi.installModel( modelType: ModelType.gemma4, fileType: ModelFileType.litertlm, ).fromNetwork(gemma4E2bUrl).withProgress(onProgress) .install(); final model = await FlutterEdgeAi.getActiveModel( maxTokens: 4096, preferredBackend: PreferredBackend.gpu, supportImage: true, ); final chat = await model.createChat( modelType: ModelType.gemma4, supportImage: true, enableThinking: true, );
Downloads about 2.6 GB the first time. If the file is already there, it just marks the model as active.
This is the context window: prompt, history and reply combined. To limit the reply, use maxOutputTokens.
Get either one wrong and it still compiles — the app fails later, at runtime.
Step 3 · Send a message, stream the reply
Stream the reply as it's written.
chat_screen.dartawait chat.addQueryChunk( Message.text(text: prompt, isUser: true), ); await for (final r in chat.generateChatResponseAsync()) { switch (r) { case TextResponse(:final token): answer += token; case ThinkingResponse(:final content): reasoning += content; case FunctionCallResponse(): case ParallelFunctionCallResponse(): break; } }
ModelResponse is sealed, so the switch has to handle every case. A new response type gives you a compile error instead of silently lost tokens.
Message.withImage( text: 'What do you see?', imageBytes: photo, isUser: true, )
What broke
The parts most talks skip. All of these happened while I was building this app.
Lesson 1
Speech models listen for a fixed length.
My first voice turns were cut off mid-sentence, and nothing in the logs said why. A speech-to-text model is exported with one fixed window. Anything longer than that is dropped, quietly.
The filename says it: moonshine_tiny_5s_f32.tflite. Say a sentence and a half, and only the first five seconds reach the model.
A 30-second window, and multilingual. That is what makes a real conversational turn possible, so this is what the demo ships.
Cap the recorder below the window — mine stops at 25 seconds — and show the countdown while it listens. Short audio is padded, so there is no cost to stopping early.
The same holds for whichever model you pick: read the window it was exported with.
Lesson 2
Small models slip into other languages.
Yes, I know about Flutter. It is an open-ソース software development kit created by Google…
The model knows the same word in many languages. ソース is Japanese for “source”, and with loose sampling it can beat the English word.
- Tighter sampling, such as temperature 0.7 for chat and 0.3 for voice
- Say which language to use in the system instruction
- For speech, remove characters the voice can't pronounce
That made it rare, but it still happens now and then.
Not a prototype
Shipping today, on-device.
Summarises your recording with Gemini Nano, with no internet connection.
Rewrites and summarises your text on a 3-billion-parameter model, on the phone.
Gemini Nano built into the browser, reachable from JavaScript.
Google’s own app. Download a model, turn off your Wi-Fi, and keep chatting. The same Gemma 4 E2B you just watched is in there.
Your turn
Get it running tonight.
flutter_edge_ai needs Flutter 3.44+ and Dart 3.12+.
flutter pub add flutter_edge_ai flutter_edge_ai_litertlm
Call FlutterEdgeAi.initialize(inferenceEngines: …) in main.
iOS 15+. Android: minSdk 30 and arm64. macOS: Apple Silicon, with Swift Package Manager off and the Podfile step from the docs.
It's public, so no token. About 2.6 GB, downloaded once.
It keeps working.
The codelab on the next slide walks through every one of these steps, including the platform setup.
Build it yourself
Build this app, step by step.
Scan to open
Everything from this talk.
On-device AI isn't replacing cloud AI.
It's unlocking a different class of app — one that's private, offline-capable, and free to scale.
Thank you
