Should the AI model run on the device or on a server?

For most features, a hosted model called from your own server is the practical starting point: it gives the best output quality and you can change it without an app release. On-device models suit features that must work offline, respond instantly, or keep sensitive data on the phone.

Will adding AI make my app slower?

Not if it is isolated. The AI feature should load only when used, do its work off the main thread, and never block launch or core screens. The slow part is the model's response time, which you manage with streaming and honest progress states rather than by making the rest of the app wait.

Can I call an AI provider directly from the app to save building a backend?

Not safely. Any key shipped inside an app can be extracted and used at your expense. Even a minimal server endpoint that holds the key, checks who is calling and applies limits is enough, and it also gives you one place to change prompts and models.

Adding AI to an existing mobile app without wrecking performance

Start from what the app already does well

An existing app has users who rely on it being quick and predictable. An AI feature is the opposite by nature: slower than a normal request, variable in its output, dependent on a service you don't run, and charged by usage.

So the goal is containment. The feature should add something useful without touching launch time, core screens or reliability. Everything below follows from that.

Decide where the model runs

There are two real options, and they suit different features.

	Hosted model via your server	On-device model
Output quality	Highest; access to the most capable models	Limited by what the phone can run
Works offline	No	Yes
Response time	Network plus generation time	No network delay; depends on the device
Cost	Charged per use	No usage charge; costs battery, memory and app size
Data	Leaves the device	Stays on the device
Changing the model	Server change, no app release	Usually an app update or a model download
Device coverage	Every device	Newer, more capable devices only

Hosted, through your own server, is where most features should begin. The app talks to your backend; your backend talks to the model provider. You get the best available models and can change prompts, models and providers without anyone updating the app.

On-device makes sense for features that must work without a connection, need to respond instantly, or handle data that should not leave the phone. Both Apple and Google provide on-device machine learning frameworks, and classic tasks such as text recognition, image classification and speech are well served. The costs are different: models take up storage and memory, heavy inference drains the battery and warms the device, and older phones may not cope at all. You need a plan for the devices that can't run it.

A mix is common: a small on-device model for quick, private tasks, and a hosted model for anything that needs real reasoning.

Never ship API keys in the app

This is the one rule with no exceptions. A key embedded in a mobile app is not secret. App packages can be unpacked and network traffic can be inspected, and obscuring the key only slows a determined person down. Once extracted, the key can be used by anyone, and the bill is yours.

The fix is the server in the middle. It holds the provider key, and the app authenticates to it as a signed-in user. That endpoint is also where you:

check who is calling and whether they are allowed to use the feature;
apply rate limits and usage caps;
build the prompt, so the instructions cannot be altered from the client;
log usage and errors;
swap the model or provider when you need to.

Both platforms also offer attestation services that help your server confirm a request came from a genuine copy of your app. They are worth adding for a feature that costs money per call, as an extra layer rather than a replacement for authentication.

Design for the wait

A language model can take several seconds to produce a full answer. How that wait feels is a design problem as much as an engineering one.

Stream the response. Showing text as it is generated makes the feature feel responsive, even though the total time is the same. Your server needs to pass the stream through, and the app needs to render partial content without the layout jumping about.

Show honest progress. A plain spinner with no context feels broken after a few seconds. Say what is happening, and for multi-step work, show the steps.

Let the user leave. They should be able to cancel, navigate away and come back. Cancelling in the app should also cancel the work on the server, so you don't pay for an answer nobody reads.

Keep the rest of the app out of it. Load the feature's code and assets only when it is opened. Parse and render off the main thread. Never make launch or a core screen wait on an AI call.

Plan for failure and for no connection

Mobile networks drop. Providers have outages and rate limits. Models occasionally return something unusable. Treat all three as normal rather than exceptional.

Set a timeout and show a clear message with a retry, not an endless spinner.
Keep what the user typed. Losing a carefully written prompt to a network error is the quickest way to lose trust.
When offline, say so up front and disable the feature, rather than letting the user try and fail.
Where there was a non-AI way to do the task, keep it. The AI feature should be an addition, not the only route.
Validate the model's output on your server before the app relies on it, particularly if you expect structured data.

Control the cost

Hosted models are charged by the amount of text processed, so cost grows with usage and with how much context you send. A few habits keep it predictable:

Limit use per user. A daily or monthly cap, enforced on the server, protects you from both abuse and enthusiasm.
Cache what repeats. If many users ask for the same thing, or one user asks twice, serve the stored result.
Send less. Trim the context to what the task needs. Long histories and whole documents are the usual cause of surprising bills.
Use the smallest model that does the job. Reserve the most capable model for the requests that need it.
Watch it. Track cost per user and per feature from the first day, and set an alert on spend.

Be clear about what leaves the device

Adding a hosted model usually means user content now goes to a third party. That changes your privacy position, and you should handle it deliberately:

Send only what the feature needs. Strip identifiers and unrelated fields on the server.
Tell users plainly what is sent and to whom, at the point they use the feature.
Update your privacy policy and the privacy disclosures in both app stores to match.
Read your provider's terms on data retention and on whether content is used for training, and choose settings that fit your users.
For sensitive categories such as health, finance or children's data, take advice before launch rather than after.

Check the store rules before you submit

Both stores have expectations for apps that generate content, and they have been changing as AI features spread. In general, expect to need a way to prevent clearly harmful output, a way for users to report or flag what the feature produces, and accurate disclosure of how personal data is shared with third-party AI services. If generated content can be seen by other users, the usual rules for user-generated content apply too.

I won't quote specific clauses, because they change. Read the current App Store Review Guidelines and Google Play's developer policies before each submission that adds or changes an AI feature.

Ship it behind a feature flag

A feature flag is a switch on your server that the app checks before showing the feature. For AI it is close to essential, because the things that can go wrong, such as a provider outage, a cost spike or a bad output pattern, cannot wait for an app release and store review.

With a flag you can release to a small group first, compare behaviour, widen gradually, and turn the feature off in minutes if you need to. It belongs alongside a wider plan for how you deploy and roll back, which is one of the product decisions that decide whether you scale cleanly.

What to read next

For wider product decisions, see the mobile architecture series.