Generative AI for 3D Digital Assistants: An End-to-End Methodology from Scalable Creation to Safe, Dynamic Orchestration in Production

Abstract

As digital assistants become a growing part of people's everyday digital lives, the limitations of traditional chatbots become apparent, falling short of the empathy and expressiveness users look for. Within BrightFactory, this expectation is embodied in Buddy, a multimodal digital assistant that combines text-based communication with an animated visual presence to foster genuine human connection. Scaling this kind of system, however, demands solutions to challenges that manual workflows struggle to meet, including 3D asset creation, limited animation adaptability, and real-time behavioral coordination. This dissertation investigates how Generative AI can be integrated into Buddy's production pipeline to address these challenges. To this end, the work is structured around three research phases, each validated through the Buddy Forum, a live internal team chat built to serve as the primary case study. The first phase employs text-to-3D models to automatically produce textured meshes from natural language prompts. The second extends the pipeline with an AI-assisted animation system that selects and applies the most appropriate motion strategy for each asset. The third phase integrates these outputs into Buddy's live behavior, using a large language model to select the most contextually appropriate assets and a coordination layer for character animation. Across all phases, structural guardrails are enforced to prevent invalid states and ensure the pipeline produces physically plausible outputs. The results show that the pipeline is capable of generating, rigging, and animating 3D assets with reasonable semantic alignment and temporal coherence, while the orchestration layer demonstrates promising behavioral-level asset selection. The evaluation further identifies the practical boundaries of current automation, pointing to concrete directions for future refinement. Overall, the findings suggest that integrating generative AI into a production pipeline for digital assistants is a viable path, with the proposed architecture offering a meaningful step toward more adaptive and expressive behavior in Buddy.