AI integration

Adding generative model features to a Flutter app: where the call should originate, how to stream a response into the interface, how to get structured data back, and what to do when it fails or costs too much.

Contents

• Architecture first
• Never ship the key
• Provider options
• Calling through your backend
• Streaming into the UI
• Structured output
• Multimodal input
• On-device models
• Designing for failure
• Cost and latency
• Evaluating quality
• AI-assisted development

Architecture first

There are two shapes, and choosing the wrong one is expensive to undo:

Shape	The app calls	Use when
App to backend to model	Your own API	Almost always: you control keys, prompts, rate limits, logging, and cost
App to model directly	The provider SDK	Prototypes, internal tools, or a managed SDK designed to hold no secret in the client

The second is tempting because it is fewer moving parts. It also means your prompt, your model choice, and your quota live in an artefact that anyone can decompile, and changing a prompt requires shipping an app update.

Never ship the key

An API key compiled into a Flutter app is extractable — from the binary, from --dart-define values, from network inspection on a rooted device. Treat any key that reaches the client as public.

Consequences, in practice:

• Put the provider call behind your own endpoint, and authenticate the user to your endpoint.
• Enforce rate limits and spending caps server-side. A leaked key with no cap is a bill, not an inconvenience.
• If you use a managed client SDK, use the mode designed for untrusted clients — one that mints short-lived, scoped tokens or proxies through the vendor's own backend — and enable its abuse protection.

See Security and Local storage for token handling on the device.

Provider options

Option	Notes
Provider HTTP API through your backend	Portable; you are not tied to a Dart SDK's release cadence
Vendor Dart SDK in the app	Fastest prototype; check whether it is intended for client use
Firebase-hosted model access	Handles client attestation and keeps the key server-side; ties you to Firebase
On-device model	No network, no per-call cost, strong privacy; limited capability

Pick the provider on capability, latency, price, and data-handling terms — then isolate it behind an interface in your own code so swapping is a repository change, not a rewrite. Model names, context limits, and prices change frequently; read the provider's current documentation rather than any figure written in a tutorial, including this one.

Calling through your backend

Keep the model behind a repository, exactly like any other remote data source — see Networking and serialization.

abstract interface class AssistantRepository {
  Future<Result<String>> complete(String prompt);
  Stream<String> stream(String prompt, {CancelToken? cancelToken});
}

class HttpAssistantRepository implements AssistantRepository {
  HttpAssistantRepository(this._dio);
  final Dio _dio;

  @override
  Future<Result<String>> complete(String prompt) async {
    try {
      final response = await _dio.post<Map<String, Object?>>(
        '/assistant/complete',
        data: {'prompt': prompt},
        options: Options(receiveTimeout: const Duration(seconds: 60)),
      );
      return Ok(response.data!['text']! as String);
    } on DioException catch (e) {
      return Err(Failure.fromDio(e));
    }
  }
}

Note the timeout: model calls are slow by the standards of a REST API, and the default timeouts you use elsewhere will cut off a legitimate response.

Streaming into the UI

Streaming is the single biggest perceived-latency improvement available. Have the backend forward server-sent events and expose them as a Dart Stream:

@override
Stream<String> stream(String prompt, {CancelToken? cancelToken}) async* {
  final response = await _dio.post<ResponseBody>(
    '/assistant/stream',
    data: {'prompt': prompt},
    cancelToken: cancelToken,
    options: Options(responseType: ResponseType.stream),
  );

  final lines = utf8.decoder
      .bind(response.data!.stream.map((chunk) => chunk.toList()))
      .transform(const LineSplitter());

  await for (final line in lines) {
    if (!line.startsWith('data:')) continue;
    final payload = line.substring(5).trim();
    if (payload == '[DONE]') return;
    yield jsonDecode(payload)['delta'] as String;
  }
}

In the widget, accumulate rather than rebuilding the whole conversation per token:

StreamBuilder<String>(
  stream: _tokens,
  builder: (context, snapshot) => Text(_buffer.write(snapshot.data ?? '')),
)

Practical details that make streaming feel right: render partial text immediately, keep the scroll pinned to the bottom only while the user has not scrolled away, and give the user a stop button wired to a CancelToken — see Async and isolates.

Structured output

When you want data rather than prose, ask the provider for JSON (most support a JSON or schema-constrained mode), then parse it with the same machinery as any other response:

@freezed
class ExtractedReceipt with _$ExtractedReceipt {
  const factory ExtractedReceipt({
    required String merchant,
    required double total,
    required DateTime date,
    @Default(<String>[]) List<String> items,
  }) = _ExtractedReceipt;

  factory ExtractedReceipt.fromJson(Map<String, Object?> json) =>
      _$ExtractedReceiptFromJson(json);
}

Always validate. A model can return well-formed JSON with a nonsensical value, so range-check numbers, verify enums, and treat a parse failure as a normal error path with a retry — not as an exception that crashes the screen.

Multimodal input

For image input, keep the upload off the UI thread and downscale before sending: a 12-megapixel photo costs latency and tokens for no accuracy gain.

final bytes = await Isolate.run(() => _downscaleJpeg(originalBytes, 1024));

Ask for camera and photo permissions with a clear rationale, and tell the user what leaves the device. If the image contains personal data, that is a privacy disclosure, not an implementation detail.

On-device models

Small models can run locally through platform ML runtimes or dedicated Flutter packages. The trade is capability for privacy, offline support, and zero marginal cost.

Worth considering when: the task is narrow (classification, extraction, summarising a short text), the data is sensitive, or the feature must work offline. Not worth it when you need broad reasoning or long context.

Costs: a large model file to download or bundle, meaningful memory use, slower first run, and per-device performance variance. Run inference off the UI thread and measure on a low-end device.

Designing for failure

Model calls fail more often and in more ways than ordinary API calls. Handle all of these explicitly:

Failure	Response
Timeout	Cancel and offer retry; never leave a spinner forever
Rate limited	Back off with jitter; tell the user it is temporary
Content filtered	Show a specific message; do not present it as a crash
Malformed structured output	Retry once, then fall back to a manual path
Offline	Detect early and skip the call
Slow first token	Show a token-level progress indicator, not an indeterminate spinner

Give the user a way out at every step: cancel, retry, edit the input, or do the task manually. A feature that can only succeed is a feature that hangs.

Cost and latency

• Every call costs money proportional to input plus output length. Trim what you send: summarise history rather than resending it, and cap output length.
• Cache aggressively. Identical inputs should not be billed twice — cache on the server, keyed by a hash of the normalised request.
• Debounce input-driven calls so typing does not fire a request per keystroke.
• Enforce per-user quotas server-side, and monitor spend from day one.
• Measure end-to-end latency and time-to-first-token separately; they need different fixes.

Evaluating quality

Treat prompts as code:

• Version them, keep them server-side, and change them without an app release.
• Keep a fixed set of representative inputs with expected properties, and re-run it whenever the prompt or model changes.
• Assert on properties — "returns valid JSON", "total matches the sum of items", "does not contain the customer's phone number" — rather than exact strings, since output varies between runs.
• Log inputs and outputs (with consent and appropriate redaction) so you can diagnose complaints.

AI-assisted development

Coding assistants are useful for boilerplate, tests, and unfamiliar APIs. Two cautions specific to Flutter: generated code frequently targets an older API surface (deprecated widget parameters, superseded state management idioms), and package versions in generated snippets are often invented. Compile it, run the analyser, and check the package's own documentation before trusting it.

General background on models and providers is in AI.

Next: Resources.