Skip to main content

3d-ai-tv 2024-12-04

Summary​

The chat segment revolves around the community members discussing their creative projects, specifically focusing on making scenes inspired by various artists. Boom shared a scene they created and suggested creating another one based on Nam June Paik's TV arrangements.

FAQ​

  • Could you make an endpoint that returns the trailer review in 1 JSON object? (Then I can convert it to one of these show scripts on the fly with another AI call.) (asked by SM Sith Lord)
  • Can we pull JSON from a URL? Can Firebase real-time database be used instead of something more local? (asked by [SM Sith Lord, 08:42])
  • Do I need markers for phenoms? Do you just need the audio & length to do that? (asked by boom)
  • Would delivering phenoms work best if 3D stage handles TTS, and can it handle real-time processing of an interrupted scream from a window? (asked by sm sith lord)
  • Can we play the show directly in a web page? Why would it be slow comedy if there's no multi-thread voice support? What are visemes and how can they replace audio eventually? (asked by [SM Sith Lord])
  • How does each scene maintain its unique vibe with only four characters, considering the limitations of Unity integration for now? (asked by boom)
  • How does the web page app running on your PC talk to the Unity app running on your PC? How can I implement this communication efficiently, considering my limited development skills and time constraints? (asked by whobody)
  • How do we handle updates in our JSON file within the Unity script to ensure only new events are processed? What's an efficient way of doing this without overloading resources or causing delays? (asked by boom)
  • 'Unity can support it?' referring to the Unity showrunner web page app with AI calls via node.js server and Claude & OpenAI APIs (asked by [boom (08:54)])
  • Will SM use Firebase or host on a website for their project? (asked by [SM Sith Lord (08:54)])

Who Helped Who​

  • @boom helped with Scene creation by providing Boom provided guidance on creating a scene and shared inspiration from Nam June Paik's work
  • @whobody helped @boom with Improving application functionality by providing Discussing script generator and JSON implementation
  • @whobody helped @boom with Enhancing user experience by providing Sharing thoughts on screen grabs, using Eliza plugins for real-time interaction.
  • [boom] helped Creating an endpoint for trailer review in JSON format. with ] by providing SM Sith Lord
  • [SM Sith Lord (08:43)] helped [boom(08:42)] with Technical discussion on handling text-to-speech within the game engine for better synchronization with visuals. by providing Discussing TTS and blendshapes for Unity project to improve audio-visual sync.
  • [boom] helped [sm sith lord] with Understanding phenoms processing requirements by providing SM Sith Lord explained the need for audio markers in handling 'phenomes'.
  • SM Sith Lord helped [boom] with Integration of audio API and scene management in Unity. by providing SM Sith Lord provided technical guidance on integrating web page app events into Unity.
  • boom helped whobody with Implementing efficient communication between web page and Unity apps by providing SM Sith Lord suggests a method for polling JSON files from URLs within Unity scripts to efficiently process new events.
  • [boom (08:54)] helped [SM Sith Lord (08:56)] with Implementing fetch JSON from URL in Unity showrunner web page app, considering hosting options and potential elimination of bridge by providing Boom provided technical advice on using Unity, node.js server for AI calls via Claude & OpenAI APIs
  • [SM Sith Lord (08:58)] helped [boom(08:59)], [memealot] with Exploring alternatives to Firebase for data management by providing Discussing the possibility of running a node.js server on own PC

Action Items​

Technical Tasks​

  • Implement Avatars as feeds onto TV with a main content center. (mentioned by @boom)
  • Create an endpoint that returns trailer review as a single JSON object. (mentioned by SM Sith Lord)
  • Assign cameras based on actor names (mentioned by [boom, 08:40])
  • Handle TTS within Unity for better audio-visual sync (mentioned by [boom (08:43)])
  • Implement audio markers for phenoms (mentioned by [SM Sith Lord])
  • Processing the audio stream in real-time. (mentioned by [SM Sith Lord])
  • Integrate loadScene & speak events from web page app into Unity (mentioned by SM Sith Lord)
  • Poll a JSON file for new event updates within Unity script (mentioned by SM Sith Lord)
  • Evaluate hosting options for the Unity showrunner and AI call server, considering potential elimination of bridge (mentioned by [SM Sith Lord (08:54)])
  • Implement fetching JSON from a URL for Unity showrunner web page app, using node.js server (mentioned by [SM Sith Lord (08:56)])
  • Develop a Unity app that pulls data from Firebase object (mentioned by [SM Sith Lord (08:57)])
  • Create an AMode for loading files vs API and override functionality (mentioned by [boom(08:57,09:13)], [SM Sith Lord (08:58)])
  • Implement a flattened structure for event stream data to be used with Unity (mentioned by @SM Sith Lord)

Documentation Needs​

  • Design the scenemanager to handle payloads one after another in real-time. (mentioned by boom)
  • Parse entire scene or show for new parser implementation (mentioned by [boom, 08:41])
  • API audio length and drive related actions in Unity. (mentioned by boom)
  • Create an efficient data structure to search and process newer events in the JSON file. (mentioned by boom)

Feature Requests​

  • Create a scene inspired by Nam June Paik's TV arrangements with multiple televisions (mentioned by @boom)
  • Consider using external plugins like Meta's deprecated one or Eleven Labs API (mentioned by [SM Sith Lord (08:45)], [boom(08:43)])
  • Consider enum for configs to differentiate between local prewritten JSON mode vs web API calls (mentioned by [boom (08:56)])