ABOUT

ABSTRACTThis project investigates how Artificial Intelligence (AI) is used in 3D animation and how it changes the creative process and the way things are created. AI makes animation more efficient by automating jobs that are done. It also allows new ways to be creative. The study's goal is to find out how AI tools such as Stable Diffusion, Meshy AI, and ComfyUI can make the 3D modelling process more efficient.The purpose of this study is to discover how AI is used in the animation industry and explore the issues that prevent individuals from using it. Through this, AI's potential can be seen through different animation projects. As part of the study, AI-powered videos synchronised with music were created alongside a survey, which was presented to 29 people to discover their opinion on AI creating content.The results show that even though AI speeds up results, some people are still concerned about the ethics and how it might impact creativity. People are also concerned that AI could replace artists who collaborate with humans. The study shows that AI tools should only be used to help people be unique rather than to replace them completely. This way, both creativity and artistic integrity can be kept.It concludes that AI will likely make working together on creative projects easier in the future, but people will still have to manage the process so that AI will not take over and prevent them from being creative.
FINAL VIDEO
Here is the production process!
PRE PRODUCTIONImplementing The Background Music and Creating The Shot List in Adobe Premiere ProThe music was selected from "Epidemicsounds," which worked perfectly with this kind of video. While editing, it's easier to have all the stems together because of "Epidemicsounds." During the editing process, shot lists were marked up in Adobe Premiere, and the duration of each shot was determined for the final video. Everything had been done to design the research component of the videos when all of these processes finished. The latest idea for final 3D video was to use these created videos as animated textures.
With Adobe After Effects, some sound-based animated videos were created. The videos were made by using universal plugins. The video and music were synced in After Effects using a particular plugin, as seen in below. Following, a variety of styles were applied to the completed video to provide a different appearance, as seen in below. Four distinct movies were initially planned for the research; some of these were created with animation tools, and the other videos have already been sent to the Stable Diffusion's deforum extension.​​​​​​​
The first videos have been used as input (init video) on the Deforum platform in order to create new videos. After that, prompts were given, as seen in below. With the use of recommendations, Deforum's ControlNet feature allows you to apply a different style but still preserve the original movements from the input video. Below image displays the parameters for this process.
Plenty of experimentation and error required throughout the way, which unfortunately demonstrates to be very time-consuming. Stable Diffusion optimizes production by using the CPU and GPU at the same time; if the machine cannot satisfy the system requirements, tasks may take longer. The videos were first produced at a lower resolution to save time. They were then upscaled with Topaz Gigapixel AI. The videos were ready and everything was in place for the modeling process to start at this point.
Modelling with AI ToolsThe modelling process began with creating shoe designs in MidJourney, involving multiple iterations to select the best options. After this phase, the focus shifted to 3D modelling as it can be seen in below.
In the modeling phase, the chosen shoe images were initially modeled using the ComfyUI plugin and Stable Diffusion. It was shown that ComfyUI also features a node-based design set up, with multiple extensions available. Unfortunately, the initial attempts were unsatisfactory. The results became better once understanding the functionality of each tool in ComfyUI and after finding different Stable Diffusion methods online. The results were poor quality and the AI had errors, even after agreeing on what to do and trying different approaches. ​​​​​​​
Moreover, alternative methods were examined into, for instance the "meshy.ai" website, which provides “Textto3D” and “Imgto3D” conversion tools. Meshy AI was used to make models using the photos generated in MidJourney. As seen in below, the early results from the AI tool were not flawless, but they were sufficient to make improvements.
Every model was then imported into Cinema4D in order to improve the quality. The topology was cleaned by exporting the 3D geometry of the model and utilizing the “remesh” tool in Cinema4D. Nevertheless, a problem evolved when textures were distorted while remeshing. Applying UV texturing over the 3D model was challenging because the models had just front-facing image references available. The initial approach for doing this required unpacking the UVs and painting them in Substance Painter. However, this took a lot of work as well as results that were average. An additional method for projecting the texture on the front face onto the model was found through a search on the internet. The shoes' front and back sides have the same appearance, which made the edges perfect to use with a texture projector. Although this approach was effective for the shoe design, it might not be applicable to other model types.Below image shows three different versions of the same mesh: one as extracted with meshy.ai, the second after remeshing and the last after texture projection with considerable enhancement of model quality​​​​​​​
Following, a full texture setup was made, and Redshift was selected as the render engine to make the model more realistic. It helped to examine what was written on Maxon's website to learn more about Redshift's node system and how it could make the visuals improved. For this, PBR (Physically Based Rendering) materials were needed. These included a normal diffuse map, a metallic map, a glossy map, and a roughness map. MidJourney only provided the diffuse map, so a tool called Laigter was used to make the other maps, which  can be seen in the figure below.
Since the AI couldn't accurately replicate features like heel loops and air bubbles, an opacity map had to be created by hand. This opacity map, which was made in Photoshop, was then loaded into Redshift to generate transparent textures that emphasized the model's most important regions. The figure below provides an illustration of this process.​​​​​​​
As seen in below, this arrangement was used for all other shoe designs to ensure that each model was prepared before going on to the garment design stage.
Designing The Clothes in Marvelous Designer Users may create 2D sewing patterns with Marvelous Designer, stitch them together exactly like they would with real fabric, and then simulate sewing that outfit onto a character. The application's physics engine enables the fabric to move and respond as if it were being affected by gravity, wind, etc. Because of the character's anatomy, it can also be slightly difficult to get the garments to fit the character properly. Selecting the right character for the task at hand was therefore crucial.
Character Creator was used to create the character model. The “character mesh form customizer” is an amazing function that allows you to alter a human body type with a few well-designed sliders. This makes the process of coming up with an original character quick and easy.
Character Creator also offers a method for adjusting body proportions, allowing users recreate clothing on the 3D model in an even accurate way. Since clothing design and simulation believe that garments are designed off these standard poses, it was necessary to import the character into Marvelous Designer in either an A-pose or T-pose once it was exported. The clothing may not fit correctly if the character is imported at another pose, which could lead to issues later on in the simulation.The character needs to be animated in such a way that it begins standing in a position prepared for thirty to forty frames when it is imported into Marvelous Designer. This offers a solid foundation on which to form the garment correctly. To provide a seamless transition between the T-pose and the actual animation start, a buffer of roughly 10–20 frames is required. This can sometimes cause the simulation "to break," which results in mistakes in how the clothes interact with motion content.There were a few manual character posture fixes where having smooth animation and simulations was critical. Furthermore, the character's motion capture data was handled using Cinema4D's motion tools, which assisted in merging and baking the motion data into the timeline for the character's animation. Cinema4D makes it quick and easy to adjust and improve the motion data before the simulation. As it can be seen below.
Clothes design started as soon as the character and animation were properly setup. Creating clothes in Marvelous Designer requires a knowledge of stitching points and sewing patterns. Designers can customize how the clothes fits and moves on the character by accurately placing. While some of the clothes for this project were altered from Marvelous Designer's pre-existing clothing library, others were made entirely from scratch. Faster testing and iteration were made possible with this method. The final dress design used in the final video is shown in below.
To make sure the clothes reacted to the character's movements accurately, they were simulated several times. Different designs were also evaluated to determine which produced the greatest outcomes. Marvelous Designer has tools to configure UV maps in addition to support exporting the clothing in a variety of formats, including .fbx and .abc (Alembic).  Converting the garment's topology from triangulated to quad based is a crucial stage in this procedure is shown in below.
By default, cloth exported from Marvelous Designer consists of a triangulated topology. Triangle-shaped surfaces are not a good fit for displacement and bump maps in rendering. Upon transforming the topology into quads, the simulation displays more natural behaviour and can easily apply maps. It is practical to achieve results that are realistic in this approach. Texturing came next after the model was dressed. Marvelous Designer's AI Texture Generator is one of its best features; as below image illustrates, it can generate textures by giving prompts. Customizing textures has made it possible to quickly produce high-quality apparel and enhanced the final designs' rendered appearance.
In the figure below, additional AI tools such as ChatGPT and Midjourney have been used for texture development. The design process moved faster with the addition of these AI technologies.
There were specific challenges observed while executing the simulations. For example, clothing would sometimes tear itself if the character's hand moved too close to the body, regardless of if it was touching it or not. In both situations, the solution was going back to C4D and changing the animation data by hand in order to fix the issues. This required a lot of back-and-forth between Cinema4D's animation editing and Marvelous Designer for simulation. Although it took a while, this procedure was required to get the clothes to simulate correctly without collapsing.
The configuration of the simulations is simply illustrated in below. After they were done, modifications had to be done to the UV maps for the clothing to avoid texture overlap. This was necessary since there would have been rendering issues and a stretched or warped aspect for the textures if there had been overlap in the UV layout. In order to verify that there would be no rendering issues during the final export, the Marvellous Designer UV editor was also used to change the UV maps, with all of its pieces put inside a 2D space, as illustrated in below.
There are certain limitations on how the animation can be exported from Marvelous Designer. For instance, the clothing's animation data is lost while exporting in the .fbx format; therefore, the animation had to be included using the .abc (Alembic) format. The clothes have to be exported twice, once for the animation in Alembic format and another for the textures in .fbx format because Alembic does not include the textures. The imported FBX file in Cinema4D was then given the same animation layers from the Alembic, Ensuring that the final garment models would accurately display the textures and animation required this technique. When those clothes initially entered Cinema4D, they appeared poor and inefficient. The materials were optimized using a Redshift node-based approach to give them a more attractive look, as seen in below. To give the textures more realism, some fabric elements from the Grayscalegorilla texture library were mixed with bump, roughness, and height maps. This is how Redshift is typically used in Cinema4D workflows.
A simple daylight system-based lighting setup with a few spot lights to highlight the clothing was used for the last scene. Final renders were created after the lighting had been set up. The integration of Marvelous Designer, C4D, and Redshift for the process played an important role in improving the visuals in the current production. In summary, producing outstanding, physically realistic simulated clothes for this project required more than only creating outfits in Marvelous Designer. To produce the finest result, it required careful balancing between rendering in Redshift and animating in Cinema4D. By integrating simulation methods and AI tools with human adjustments, the clothes designs were able to work correctly in the animation and have a suitable visual impact.
Animation PhaseThe character from Character Creator was already pre-rigged, so there was no need to rig it alone. The animation was finished in Cinema 4D. However, motion capture was necessary for the walk cycle animation in the final part of the video in order to enhance the look and movement by using the AI-based program DeepMotion, which can analyze videos and animate the character.A tripod was used to record a video of a person walking in front of the camera in order to catch the walk cycle. After the video was recorded, it was uploaded to the DeepMotion platform, where artificial intelligence (AI) analyzed and generated animation data. This data was saved as a .fbx file for export. After being exported as a .fbx file, this data was imported into the Cinema 4D rigged character, which was originally in a T-pose.While integrating, some problems were found because of potentially inaccurate motion capture data caused by less than perfect lighting conditions in the video-based recordings. These parts had to be manually fixed in C4D itself before it could work on the cloth simulation directly. This animation was prepared for further processing after the necessary adjustments.
Manual animation techniques were also used in other parts of the video. Cinema 4D was used to produce animations by keyframing various movements directly into the program. One useful feature was the ability to apply the vibrating tool to all of the static character's bones. The tool was used to perform an automatic movement function on the bones, adding a small portion of randomization, as many of the characters in these scenarios are supposed to be kept static. That gave the otherwise static visuals an additional amount of motion, which significantly enhanced the way it looked. Using the bend deformer together with a shoe model and adjusting the degree of bends throughout the scene, moving animations were produced for the shoe animation scenes.Cloth simulation tags were applied to a plane object that had been cloned using the cloner tool in order to create dynamic simulations in the clothing scenes. For certain abstract visual effects, one may also curl and twist the splines using deformers like turbulence and attractor.
The scenes' background movement was created using an After Effects video that was output as a .png sequence. The walk cycle situation went through the same process, with more video sequences placed on displays behind the actor. In this way, it improved the visual diversity in and around that area.
Once every scene has been animated, Redshift is used for the final texturing and lighting. In order to achieve detailed texturing and rendering with realistic lighting effects, particular focus was given to the lighting. Subsequently, each scene was individually rendered to produce a professional and dramatic end result.To sum up, the animation method was a combination of dynamic simulations, manual keyframing, and motion capture, all of which were done in Cinema 4D. Many visually appealing sequences were produced by combining AI motion capture tools like DeepMotion with the variety of deformers and animation tools found in Cinema 4D. Redshift was used for rendering, providing a technically sound and visually vibrant end result.
Approaching the Final VideoIn order to create the final video, each scene was finally saved as an .exr file. Subsequently, Premiere Pro was used to import the image files and arrange them according to the shot list. Following the initial Premiere edits, the scenes were forwarded to Adobe After Effects for additional corrections.
After the video was produced, effects were added to improve the final product's appearance and color corrections were made in After Effects to fix the grading. At this point, it had smooth control over the look of the whole project from beginning to end.The video production process is eventually completed after every scene has been edited, every VFX shot has been post-produced, and every file has been exported as an .mp4 using Adobe Media Encoder.