Virtual production (VP) blends traditional filmmaking with cutting-edge technology such as LED walls, motion capture, augmented reality (AR), and advanced computer graphics. Even perceived ‘old’ tech such as green screens have developed, allowing for stunning live-to-tape filmmaking. Whether filming a movie, music video, advert, corporate explainer, or social media reel, VP allows directors to push creative boundaries beyond budget when combined with locations and sets, or as an entire virtual production.
However, this is a double-edged sword. The greater creative freedom given by VP also carries greater technical knowledge and risks. This can be a challenge for directors, who are responsible for the overall success of any type of production.
Although, this shouldn’t deter directors from dipping their toes into VP for all or part of a production (or taking the whole plunge). With careful preparation and proper management, directors can harness the full potential of virtual production while mitigating the risks associated with unfamiliar technology.
The Director’s Role in Virtual Production
In any type of traditional production (movie, music video, advert, corporate explainer, or social media reel etc), the director is the creative leader who shapes the narrative, guides talent, makes artistic decisions, and oversees the film's overall visual style to bring the script to life.
In virtual production, the director must do all of that – and more. They must not only understand the narrative and performance aspects but also have a solid grasp of ‘the art of the possible’ when it comes to VP.
This new addition to their craft includes but is not limited to understanding how camera movements interact with CGI elements in real-time, impact of physical and virtual lighting, when and how to use 2D vs 2.5D vs 3D backplates, the numerous ways to capture ‘in camera’ or take it to post, the effective communicating with the technical team to deliver the vision and being familiar with the capabilities, limitations and when to use the VP tools and workflows to hand. (The list goes on, and there’s far too much to put here).
Once a director has a solid grasp of the tech possibilities (and challenges), their ability to fulfil their traditional role is greatly enhanced. For example, through the use of Unreal Engine and VR headsets, directors can explore the virtual environment development as they are built, long before the shoot. Also, if so inclined, armed with some basic know-how, a director can contribute to or own the physicality of a virtual environment themselves. Specifically for indie filmmakers, this is an incredibly democratising use of the technology.
Furthermore, unlike a traditional film set, virtual environments can be adjusted on set, lighting, movement of windows, deleting a tree etc, allowing space for instant creative decisions. However, this approach has to be taken with great caution, and requires experience and clear communications with the on-set VP tech team. Moving a virtual prop a just a few metres seems only to take seconds, but the untested knock-on effect may cause lighting/ colour abortions or hardware performance issues that will inevitably cause delays on set.
This is why collaboration is so much more important in a VP shoot. Directors must work more closely with visual effects teams, virtual art departments (VAD), technical directors and virtual production supervisors throughout the production, especially in the pre-production phase. This means balancing their needs, tapping into expertise and communicating clearly and effectively in a language that can be understood on both sides. For example, actors and expense have a very different meaning in the world of game development.
Just like traditional filming, the type, complexity and style of a production dictates how hands-on a director can or should be. Larger productions have more roles and teams. The same goes for VP, allowing for roles and departments such as VAD (virtual art department), VP supervisors, Head of Virtual Production, VP Producer, Virtual Cinematographer and even niche roles such as Virtual Lighting Gaffer. Small productions will inevitably combine roles, the VP supervisor may also be the UE tech artist, virtual gaffer and key-puller. Just like a DP will cam op and gaffer. Likewise the director maybe the scriptwriter, UE VFX artist and colourist/editor.
One key stumbling block to consider is who is responsible for what and when. Currently, even within the key VFX companies, there is an inconsistency to VP related job titles,roles and responsibilities. The good news is that ‘Virtual production supervisor’ is seen as the key title on a production, however, some see this as an on-set hands-on role, others (and especially those in the VP world) see the role as being integral part of a production from the very start and wholly responsible for the virtual production aspect. Quite often, they straddle the two, especially with smaller productions.
As a result, directors can become paralysed by analysis, and rightly so, unclear who will be doing what and when, with a huge reliance on trust.
One key role the director needs to understand the role of VP is the producer. A simple misunderstanding can lead to the VP supervisor being engaged too late to advise on technology choices, creative options, and methodologies which may have saved vast amounts of time and money to achieve the director's vision. This is typically seen in the commercial productions, as producers are typically engaged after strategy and creative decisions have been made, when they are tasked with the delivery. To get the most out of the magic of virtual production, briefing media strategists and creative teams on ‘the art of the possible’ will not only save time and money, but broaden the creative sandpit, increasing variations and allowing for innovation and exceeding client expectations.
Pre-Production Challenges and Solutions
In traditional pre-production planning, directors focus on script breakdown, storyboarding, scheduling, casting, location scouting, and collaborating with the heads of departments to develop the visual style and feel of the film.
Much of this is the same for VP. For any VP parts of the shoot, there will be less physical location scouting due to the ability to use virtual environments. These environments can be 3D scans (the use of gaussian splatting has greatly reduced the entry point cost), early drafts of fictional realistic locations, abstract sets, digital twins of a real location )for call-backs or unrealistic to film in (eg Time Sq at any time). So phases like storyboarding, the use of virtual location scouting (via a VR headset) and pre-viz within Unreal Engine will positively impact time, budget and CO2 impact.
The biggest difference, however, is that in VP, pre-production planning takes on even greater importance due to the VP workflow which moves from a traditional “fix in post” methodology to a “fix in pre” one. Taking this approach dramatically increases productivity, saving budget and time.
However, this means directors must have a clear vision much earlier in the process. They must work closely with VP Supervisor or VP HOD from the start to understand the capabilities and limitations of the VP tools and shots on a VP stage at their disposal.
This shift brings about new challenges, too. Directors must ensure that the technology serves the story rather than dictates it, manage the often higher costs associated with tech, and plan with detail to avoid costly changes later.
Directors must be involved in decisions about the scale, limitations and technical specifications of LED volumes, why and when green screen virtual production is the right choice (no matter how many groans there are), the camera setup, and the overall visual aesthetic of the virtual environment. This process also includes a highly valuable pre-visualization (previs) phase to shoot scenes, shot block and test camera movements, resulting in a pre-viz roughcut that significantly reduces the need for on-set adjustments.
Pre-light, tech, shot and virtual scout days are equally, if not more, important, as they provide an opportunity to test the combination of physical and virtual lighting, camera movements, camera settings, set layout, and shot angles, ensuring that the filming days aren’t delayed by foreseeable and assumptive challenges.
Likewise, understanding colour management in virtual production is crucial. The approach needs to be fully considered before the technology and type of virtual production are chosen. Another important reason to get a VP supervisor involved at an early stage.
Size, set-up, and format of an LED volume will impact the type of in-camera photography, while green screen VP is more flexible in post while affording on-set PreVis and instantly available slap comps for rough cuts. Awareness of these options is key. While making decisions too late will inevitably impact time and budget resources, but also a missed opportunity to really lean in to the possibilities that combining live action with immersive CGI can offer.
Remember that these decisions can be unique to the scene or even shot. Like a decision on location vs set build, virtual production gives a director options, and it doesn’t need to be one choice for the entire production. This is being helped by the growth of ‘smart stages’ where manageable LED volumes are located in the same space as a green or blue screen, allowing for incredible flexibility and agility.
Innovative Directing Techniques
Virtual production adds innovative directing techniques that directors can employ on shoot days. For example, they can use real-time rendered environments to experiment with dynamic scenes, allowing immediate visual feedback and adjustments. This is limited due to colour and lighting considerations, but it still gives some flexibility that a traditional shoot might not have.
Likewise, performance capture adds a new dimension to VP shoots. Directors can work with actors wearing mocap suits to create complex, digitally rendered characters or scenes in real time. This approach means directors can better visualise the final outputs, enhancing their ability to bring their creative vision to life.
Directors can also use augmented reality (AR) on set. AR can overlay digital information or digital props onto the physical set, providing directors and actors with a more immersive and interactive environment. This can be particularly useful for scenes where actors interact with elements that will be added in post-production, giving them a more tangible reference during the shoot.
These are just a few examples of what directors can do on the shoot days, but not all of them.
Post-Production Considerations
While virtual production can significantly streamline the post-production process (as long as the pre-production phase is done right), directors still play a crucial role in this phase.
As always, Directors will work closely with editors and VFX artists in the post-production phase. In VP, the focus will be to integrate live-action footage with the virtual elements. They must keep a keen eye on how these elements interact, ensuring the blend is technically sound and artistically cohesive with their vision.
A critical aspect of this phase is the colour grading process, especially in virtual environments where ensuring consistency across scenes can be challenging. The director must ensure that the colour grading aligns with the overall tone and mood of the film. This can be the trickiest part of the process. It not only requires a delicate balance between the artistic vision and the technical limitations of blending real and virtual footage. But an understanding of what approach is needed in order to achieve the required results.
Directors are generally aware of the potential for re-shoots or additional virtual effects that may be required after the initial filming. VP can come to save the day as back plates can be rendered in post, with updated CGI or additional VFX which also uses camera tracking to mirror the physical movements of the camera at time of shooting. In addition, for call-backs, simple CUs and ECUs can be easily achieved on a VP stage where a digital twin has been created of the location or set.
As always, VP adds a new dimension to the director’s responsibilities. However, with these added complexities comes an expanded scope for creativity and innovation.
Future of Directing in Virtual Productions
As virtual production technologies evolve, directors who are committed to adopting technology to support their storytelling vision must stay ahead of the latest developments in real-time rendering, augmented reality (AR), and extended reality (XR).
These technologies blur the lines between physical and digital filmmaking, offering even more creative possibilities. Directors must be prepared to adapt their skills and knowledge continually, embracing these new tools to push the boundaries of storytelling.
The future of virtual production promises more immersive and visually spectacular cinematic experiences, requiring directors to be innovative, flexible, and technically adept.
Directing is hard at the best of times, but the future of virtual production presents an exciting challenge for directors. The future of directing in virtual productions isn't just about managing a set or guiding a narrative; it's about leading the charge in a rapidly changing digital landscape where imagination meets reality in unprecedented ways.