BuildingResona
A full-stack AI voice product built to study the hard parts of shipping software: orchestration, tenancy, media delivery, billing, and the operational edges that prototypes usually skip.
Inference
Self-hosted
Tenancy
Org-scoped
Delivery
Signed URLs
Why I Built This
Most AI side projects stop at a model call wrapped in a form. I wanted to build something that forced me to understand the pieces that make a product durable: identity, org boundaries, data flow, storage, billing, and deployment. The interesting work was never the prompt-to-audio path by itself, but everything around it.
Resona became a vehicle for that investigation. Voice generation was the domain; production engineering was the subject.
System Architecture
The application is easiest to understand if you split it into two flows: the interactive dashboard and the generation pipeline. They share the same product boundary, but they fail and scale for very different reasons.
Web Application Flow
Voice Generation Pipeline
Deployment Topology
Frontend / API
Vercel
Inference
Modal (FastAPI)
Database
PostgreSQL
Object Storage
AWS S3
Auth
Clerk
Billing
Polar SDK
Error Monitoring
Sentry
Database Schema
The schema is intentionally small and opinionated. Each entity owns a single concern, and organization scope is the default context rather than an afterthought.
Clerk User + Organization (auth only — not Prisma tables) Voice (SYSTEM | CUSTOM) ├── orgId? // null for SYSTEM, set for CUSTOM ├── objectKey? // reference clip in S3 └── generations[] Generation ├── orgId // tenant scope on every row ├── voiceId? // SetNull if voice deleted ├── voiceName // snapshot at generation time ├── text, objectKey? // WAV in S3 when upload completes └── inference params (temperature, topP, topK, …) Billing: Polar (subscriptions + usage events at request time)
The important part is the shape of the relationships: generations hang off voices, Clerk carries org identity, and Polar handles subscription state outside Postgres.
Technical Decisions & Tradeoffs
Why self-host inference instead of an API?
Self-hosting Chatterbox TTS shifts generation into owned infrastructure instead of a voice API contract. The product gains control over model behavior, storage access, and deployment boundaries. The tradeoff is operational work: Modal, bucket mounts, API keys, and failure handling all become part of the application.
Why Clerk instead of rolling auth?
Rolling auth from scratch would mean building and maintaining the whole security surface: sessions, password flows, OAuth, org switching, and edge cases around route protection. Clerk removes that maintenance burden and gives a mature organization model that fits the product's tenancy requirements without custom security plumbing.
Why tRPC instead of REST?
tRPC keeps the request and response contract in one place, so refactors surface as compiler errors instead of runtime surprises. That matters in a codebase where the frontend, API layer, and database schema evolve together. Prisma reinforces the same pattern by making the data model itself type-aware.
Why AWS S3 for storage?
S3 is the standard object store for voice samples and generated WAVs in this stack. The AWS SDK handles uploads and short-lived signed URLs, so buckets stay private while the app serves audio through authenticated routes.
Why Polar for billing?
Polar gives the project metered billing without rebuilding a full payments and webhook stack from scratch. The useful part is not only checkout, but the ability to emit usage from business logic and keep subscription state in the server layer where enforcement actually matters.
Billing Architecture
Billing is a state machine, not a button. The important part is the order of events: verify access, attempt generation, record usage only after success, and expose the portal as a management surface rather than the source of truth.
Protected premium actions
Security Considerations
The security model is layered, with controls split between routing, storage, validation, and billing enforcement. That keeps the product safe even when a user bypasses the normal UI flow.
Tenant Isolation
All tenant queries filter by ctx.orgId from Clerk. Custom voices and generations never leak across organizations.
Signed URL Delivery
Audio assets in S3 are never publicly accessible. Signed URLs expire after a short window, preventing unauthorized hotlinking or enumeration.
Proxy Route Protection
Clerk-backed Proxy enforces authentication and organization selection on protected page routes; API and tRPC handlers keep their own server-side guards.
Upload Validation
MIME type, file size (~20MB cap), and audio duration are validated both client-side and server-side. Server validation is the authoritative check.
Premium Enforcement
Feature gates run server-side in tRPC procedures — not just in the UI. Bypassing the frontend doesn't bypass billing enforcement.
Error Monitoring
Sentry captures failures, while critical paths log opaque org and voice identifiers. Sensitive values are never logged.
Environment Isolation
All required secrets are environment-scoped and validated at startup. The code expects credentials from deployment environments, not hardcoded values.
Polar at request time
Subscription checks call Polar during mutations (generate, create voice). Usage events are emitted after successful work, not before.
Multi-Tenancy in Practice
Multi-tenancy is mostly about discipline. The code has to carry the organization boundary everywhere, and the easiest place to lose that boundary is in the edge cases.
// orgProcedure attaches ctx.orgId from Clerk on every tenant mutation.
const generation = await prisma.generation.findUnique({
where: { id: input.id, orgId: ctx.orgId },
});
const voice = await prisma.voice.findUnique({
where: {
id: input.voiceId,
OR: [
{ variant: "SYSTEM" },
{ variant: "CUSTOM", orgId: ctx.orgId },
],
},
});Challenges I Solved
Prisma connection exhaustion in development
Next.js hot reload recreates module instances on every save. Without a singleton pattern, each reload spawns a new Prisma client and a new database connection pool, which quietly exhausts local resources. The fix is to reuse the existing client from a process-level singleton whenever it already exists.
// src/lib/db.ts
const globalForPrisma = global as unknown as { prisma: PrismaClient };
export const prisma =
globalForPrisma.prisma ?? new PrismaClient({ adapter });
if (process.env.NODE_ENV !== "production") {
globalForPrisma.prisma = prisma;
}Next.js route groups and layout confusion
Route groups affect layout organization, not URL structure. Misunderstanding that distinction can accidentally attach an authenticated layout to a public route or expose a protected route in the wrong shell. The fix was to treat route groups as structure only and keep access control in proxy.ts and server handlers.
Signed URL expiry during long playback sessions
Audio signed URLs have a limited TTL for security. If a user starts playback near expiry, the URL can become invalid mid-session, which feels like a broken player. Generating fresh URLs at playback time keeps security intact without degrading the listening experience.
Browser recording compatibility
MediaRecorder behavior differs across browsers, especially around supported MIME types and blob handling. RecordRTC hides a lot of that variance, but upload handling still needs MIME normalization so the server can treat every recording consistently before it reaches storage.
Metered billing accuracy
Usage events must be emitted after successful generation, not before. If the generation fails but the meter fires, the user gets charged for nothing. Emitting the event only after the S3 upload confirmation keeps billing aligned with actual output.
Key Engineering Learnings
Owning inference changes the economics
Self-hosting TTS moves generation cost, reliability, and model behavior into infrastructure the product controls. The operational complexity is real, but the system can evolve without being constrained by a third-party voice API surface.
Multi-tenancy has to be foundational
Retrofitting tenant isolation into an existing codebase is painful. If organizations are part of the product, they need to be treated as a first-class database, routing, and server-guard concern from day one.
Type safety is a productivity multiplier
tRPC + Prisma + TypeScript means a schema change propagates visibly through the entire stack at compile time. Refactors that would take days of cross-referencing become manageable because the compiler points to the breakage immediately.
Billing is architecture, not UI
Feature gates, usage metering, and subscription enforcement belong in the application layer. If they exist only in the frontend, they are not policy — they are decoration.
Observability before production, not after
Sentry was set up before the first deployment. When things broke, stack traces and structured logs around generation and billing paths made debugging a fast exercise instead of a forensic one.
Signed URLs are the correct default for private media
Public buckets are the wrong default for any user-generated content. Time-limited signed URLs give controlled access without the overhead of a full proxy layer or the risk of permanent public access.
Performance Considerations
Signed URL delivery
S3 holds private blobs; signed URLs and app proxies gate playback so the bucket never needs to be public.
Loading skeletons
Skeleton states prevent layout shift and communicate progress during async operations instead of making the page feel frozen.
React Server Components
Data-fetching components run server-side where possible, reducing client-side JS and avoiding extra waterfall requests.
Debounced search
Voice search is debounced so typing does not trigger a tRPC query on every keystroke.
Nuqs for URL state
Filter and search state lives in the URL, which keeps it bookmarkable, shareable, and cheaper than mirroring everything in component state.
Streamed audio previews
WaveSurfer.js streams audio progressively rather than waiting for the full file to download before playback starts.
Future Improvements
These are the next real gaps to close if the product grows, not a generic backlog of nice-to-haves.
Background job queue for voice generation
Moves inference out of the HTTP request lifecycle and removes timeout pressure.
Redis caching for org/subscription state
Cuts repeated lookups on authenticated requests that do the same checks over and over.
Webhook retry strategy
Prevents billing state drift when provider events arrive late or fail once.
Granular RBAC within orgs
Separates member and admin powers without widening the auth model everywhere.
Usage analytics dashboards
Gives organizations visibility into what they are consuming and when.
Rate limiting per org
Adds a safety valve against runaway usage and accidental abuse.
API key system
Would let external developers access the inference pipeline directly.
Distributed inference workers
Scale Modal/Chatterbox workers horizontally beyond a single inference container.
Error Handling Patterns
A production product needs a coherent error strategy, not ad-hoc try/catch blocks scattered across the codebase. The pattern here is layered: tRPC procedures throw typed errors, the frontend catches them with consistent toast feedback, and Sentry captures everything with enough context to debug without reproducing.
Why typed tRPC errors instead of generic status codes?
tRPC's TRPCError carries a code (UNAUTHORIZED, FORBIDDEN, NOT_FOUND, etc.) that maps directly to the business domain. The frontend can pattern-match on error codes to show contextual messages — 'upgrade your plan' vs 'voice not found' — instead of a generic 'something went wrong'. This makes error handling a product feature rather than a debug afterthought.
How does Sentry context get enriched?
tRPC runs through Sentry middleware with RPC input capture, and critical generation paths log opaque org and voice identifiers. When an error bubbles up, the stack trace and nearby structured logs give enough context to identify the failing path without logging sensitive content like voice audio or billing tokens.
What about client-side error boundaries?
React error boundaries catch rendering crashes and show a recovery UI instead of a blank screen. Combined with tRPC's onError callback and toast notifications, the user always sees feedback — whether the failure is a network timeout, a validation rejection, or an unexpected server error.
State Management Strategy
State management in Resona is deliberately split across three boundaries. Each boundary owns a specific responsibility, and mixing them is where most React apps accumulate unnecessary complexity.
Server State
Owned by tRPC + React Query. Voices, generations, and subscription status are fetched, cached, and invalidated through query keys. No manual Redux-style stores.
URL State
Owned by Nuqs. Search filters, active voice selection, and pagination live in the URL. This makes the UI bookmarkable, shareable, and free from component-level sync issues.
Ephemeral UI State
Owned by React useState/useRef. Modal open/close, recording status, playback position — state that has no meaning outside the current session and should never be persisted.
// Example: URL state with Nuqs keeps filters shareable and persistent.
// No useEffect sync — the URL is the single source of truth.
const [search, setSearch] = useQueryState("q", parseAsString.withDefault(""));
const [voiceType, setVoiceType] = useQueryState("type", parseAsString);
// tRPC query reads directly from URL state:
const { data } = trpc.voice.list.useQuery({
search,
variant: voiceType ?? undefined,
orgId: ctx.orgId,
});The key discipline is never duplicating server state in local component state. If it came from a query, it stays in the query cache. If it belongs in the URL, Nuqs owns it.
Inference Orchestration & Failure Recovery
The generation pipeline crosses three systems (tRPC → Modal/FastAPI → S3), so the orchestration has to handle partial failures at every boundary. The pattern is a two-phase write with rollback, plus fire-and-forget metering so billing telemetry never blocks the user response.
// Simplified generation orchestration from generations.ts
// Phase 1: Create DB row (no objectKey yet)
const generation = await prisma.generation.create({
data: { orgId, text, voiceName, voiceId, ...params },
});
try {
// Phase 2: Upload WAV to S3, then link to DB row
await uploadAudio({ buffer, key: objectKey });
await prisma.generation.update({
where: { id: generation.id },
data: { objectKey },
});
} catch {
// Rollback: delete orphaned DB row so clients never see
// a generation without audio
await prisma.generation.delete({ where: { id: generation.id } });
throw new TRPCError({ code: "INTERNAL_SERVER_ERROR" });
}
// Fire-and-forget: metering cannot slow or fail the response
polar.events.ingest({ ... }).catch(() => {});Two-Phase Write with Rollback
The DB row is created before the S3 upload. If the upload fails, the row is deleted so clients never see a generation without audio. This avoids orphaned records without requiring distributed transactions.
Fire-and-Forget Metering
Polar usage events emit after success but are non-blocking. The event names are part of the billing contract: tts_generation sums the characters metadata property, while voice_creation counts successful clone creations.
Cross-Language Type Safety
The FastAPI inference server exposes an OpenAPI spec. scripts/sync-api.ts fetches it and regenerates TypeScript types, so changing chatterbox_tts.py requires syncing the client contract.
Modal CloudBucketMount
The GPU container mounts the S3 bucket read-only at /storage. Voice reference audio is accessed as local files inside the container — no per-request S3 fetches during inference.
The important constraint is ordering: verify subscription → generate audio → store in S3 → write objectKey → emit usage event. Each step depends on the previous one succeeding, and failure at any point rolls back cleanly without leaving inconsistent state.
Closing Thoughts
The hardest parts of Resona had little to do with AI itself. They were the same hard parts any production product faces: clear boundaries, reliable billing, private media handling, and making failures visible early enough to act on them.
Inference ownership is a meaningful architectural choice, but it is only one part of the system. The product becomes real when tenancy, billing, and observability work together without leaking complexity back into the user experience.