INTRO

Throughout the history of art, the tension between control and randomness has been integral to the creative dynamic. Classical art emphasizes meticulous control, striving for idealized beauty and perfect representation, whereas movements like Dadaism and Surrealism embraced chance and the subconscious. Techniques like automatism, uncontrolled materials and glitch art challenge traditional notions of artistic skill and intent. We see this across multiple mediums, from the carefully planned brushstrokes of the Dutch masters to the unpredictable splatters of abstract expressionism. Even a cursory review of humanity's collective output shows both control and randomness can serve as powerful tools for expression, each offering unique avenues for conveying meaning, intent and emotion.

Every act of creation is a testament to human will, yet its final form is always deformed by the contours of the world it seeks to capture. These constraints define a spectrum of artistic capacity, from those who strive to eliminate any deviation from their preconceived visions to those who eagerly embrace the stubborn and chaotic intrusion of reality’s imperfections.

NOTE: Throughout this document, "artist," "creative," and "user" are used interchangeably to refer to anyone contributing creative output to the games domain, including engineers, artists, musicians, and advertising professionals. Engineers are ‘artists’!

MAIN

In some ways AI is no different. Creatives have identified the same core tension at the heart of the AI-art revolution: the push-and-pull between artistic control and serendipitous collaboration. Right now (Q1, 2025), I think it’s fair to say most traditional artists would say “not enough control”. Some writers, artists, musicians and engineers are learning to grapple with how much they want to steer the AI. Do you meticulously guide every parameter to achieve a preconceived output (often finding frustration when the ‘black box’ doesn’t obey) OR is it better to embrace the unexpected and allow the AI to contribute its own "ideas" (for lack of a better term) to the creative process?

In my own visual work (see HERE), I have been exploring this dynamic extensively for a few years.

Let's break down these two sides of the coin:

IntentION: The Need for Control

Lets consider why Control is so important. Using AI with this approach (mindset) has some advantages:

Realizing a Vision: Many creative people are driven by a specific vision in their head they want to bring to life. They see AI as a tool to execute that vision, a sophisticated brush or sculpting program. They want/need to control variables like style, composition, subject matter, color palettes, and even emotional tone to stay true to their goal. The brief history of these tools shows an ever-improving set of controls. In a professional context these tools must be in-service of a work product that focuses their purpose and predicates a fixed intention.
Authorship and Authenticity: The question of authorship is crucial. If an AI generates an image based on a vague prompt, is it truly the artist's work? Artists may feel the need to maintain control to assert their authorship and ensure the piece reflects their unique perspective and skill. In a world where anyone can type "dragon" into Midjourney, how do you imprint originality and stand out? This technology will soon be completely ubiquitous (in some domains like still images it already is); how do we find our own authentic voices in a sea of AI mediocrity? One way is to stake-out our own individually unique position on this spectrum. Consider the millions of blogs out there; how many are just Ctrl-C/Ctrl-V from ChatGPT? What's the point when people can just ask the AI themselves?
Avoiding Clichés and Generic Outputs: Generative AI, by its nature, is trained on vast datasets. Without careful guidance, it can easily produce bland, generic, predictable outputs based on prevalent trends. This is ‘AI Slop’ and it needs to stop. Artists who value originality need to steer the AI away from these clichés. I submit to you that humanity does NOT need another .jpg of a hot anime girl staring at the camera with zero emotion….GTFO.

Develop a Signature Style - Find your Authentic Voice: We are seeing more artists use AI to enhance their existing style or explore new avenues within their established aesthetic. One of the most interesting techniques is to 'train' a LoRA (Low-Rank Adaptation) on their own traditional work. This method has the added benefit of side-stepping legal and ethical issues associated with large commercial (and most open-source) models. We'll be talking much more about proprietary data-sets (including for language, 2D & 3D asset creation) in future blogs.

Collaboration and Serendipity

Let’s call this ‘Embracing the AI's "Voice"‘. Working with AI in this way has some advantages:

Breaking Creative Blocks: When facing creative block, artists might use AI to generate unexpected imagery, acting as a source of inspiration and prompting new ideas. The AI can provide a fresh perspective and break them out of familiar patterns. Every creative has a 'bag of tricks' that have been successful in the past they will reach for instinctually. While useful, these well-worn-paths can lead to creative stagnation; AI can help re-forge new paths and break out of habitual thinking. In my own work, I try to push image-generation AI as far as possible in this regard; I'm particularly interested in exploring the "subconscious" of the AI, trying to get at what it might “see” if it were to dream. I’m making an active effort to subvert expectations with these tools:

Happy Accidents: The AI can generate results that the artist would never have conceived of on their own. This can lead to entirely new creative directions and the exploration of uncharted artistic territory. Serendipity refers to the fortunate and unexpected discovery of valuable design elements, mechanics, or narrative ideas during the game development process.

Remember the great Bob Ross' 'happy accidents'? Not just a technique for painting trees, but an actual life philosophy!

AI can be a great engine for these kinds of discoveries, and if you embrace this approach it can be extremely benficial to your mental health when using these (often exasperating) tools. It can be very zen to stop fighting with the AI and let it take the wheel (a bit).

Remove Tedium: AI can handle complex tasks like generating intricate patterns, simulating physics, or manipulating textures in ways that would be incredibly difficult or time-consuming to do manually. This opens up a whole new range of creative possibilities.

Experimentation and Playfulness: Embracing the unpredictable nature of AI can (in theory) lead to a more playful and experimental creative process. Artists can treat the AI as a collaborator, engaging in a dialogue and responding to its outputs in an iterative fashion. Have you ever partnered with someone who was enthusiastic, creative, intelligent and just....inspiring? Ideally these AI tools will begin to function like truly inspiring collaborators encouraging us to do our very best work.

The Spectrum of Control:

It's important to note that this isn't an either/or situation. Artists & engineers working with these tools each operate somewhere on a spectrum between complete control and total surrender to the AI's generative abilities. Let’s consider a few examples:

EXAMPLE #1 - Writing Fiction

Let's consider the task of writing a story scene for your game using something like NovelAI. NovelAI is designed to be more than just a text generator; it aims to be a collaborative storytelling partner, offering more control over style, tone, and narrative structure than a general-purpose LLM (like ChatGPT native).

Scenario: Your lead writer needs to construct a short story within the game about a mysterious sentient forest.

OPTION 1: Strong Creative Intent (Control-Focused Approach):

The Writer's Vision: A dark, ancient forest where the trees communicate telepathically and act as guardians against an encroaching industrial power. The forest has a melancholic, almost mournful personality, aware of its slow decline.

NovelAI features used:

Lorebook: Creates a detailed lorebook entry describing Elderwood, including its history, the personalities of key trees (like the Grand Speaker), the nature of the telepathic communication, and the specific details of the encroaching industry. This acts as a structured reference for NovelAI.
Story Cards/Memory: Uses "story cards" or memory to explicitly instruct the AI about the current scene's context, character motivations, and desired tone. For example, "Scene: Grand Speaker confronts the young tree about their recklessness. Tone: Somber, tinged with regret."
Fine-Tuning with Prompts: While still using prompts, the author is extremely precise: "Describe the Grand Speaker's voice. Use metaphors related to the rustling of ancient leaves and the creaking of old wood. The voice should convey wisdom but also weariness…include a slight sense of sadness."
Editing and Rerolls: Relentlessly edits the generated text and uses NovelAI's reroll/regenerate function to fine-tune specific sentences or paragraphs until they perfectly match the intended vision. They might use the "instruct" feature to nudge the AI toward a more specific style after the first draft.

Results Option #1:

The author is using NovelAI as a powerful, customizable writing engine to help only in certain areas. The lorebook and story cards guide the AI to generate text that consistently aligns with the established world and character motivations. The extensive editing and rerolling ensure that the final product feels authentically "authored" by the human, with NovelAI primarily serving as a tool to bring their vision to life. The AI is kept on a tight leash, preventing it from straying too far from the established guidelines.

OPTION 2: Collaboration and Serendipity (AI as a Creative Partner):

The Writer's Vision: The writer knows they wants a sentient forest, but they want to be surprised. They're willing to let NovelAI take the story in unexpected directions, adding its own "flavor" to the narrative. In this approach, curation becomes a key function for the writer; choosing larger ideas/themes than re-rolling minute details like the phrasing of a particular sentence (that likely will come later here too).

NovelAI features used:

Minimal Lorebook/Memory: The novelist creates a basic lorebook with only essential information, leaving room for NovelAI to fill in the gaps. Likely the best ideas are rolled back into the Lorebook for future reference.
Open-Ended Prompts: Uses broader prompts like: "What secrets does Elderwood hold?" or "What are the trees' dreams?" or "Write a scene where the forest reacts to the arrival of the iron machines in an unexpected way." The writer can even be explicit about the interest in novelty "write a surprise ending for this scene" or "be extremely creative with this line of dialogue" which (in some LLMs) can work surprisingly well.
Embracing the Unexpected: Instead of immediately editing out elements that don't fit their initial vision, the novelist explores those elements further. If NovelAI introduces a strange new character or a bizarre plot twist, the novelist considers how it could enrich the story. One extreme approach is to grow the story branches organically in all sorts of wild direction, then prune back to only the best.
Exploration of Different Styles: Experiments with different AI models and stylistic settings within NovelAI to see what unique voices the AI can bring to the story. Some models are better for dialogue, some for story beats, some fore story/background lore. Using multiple models can really open up the creative possibility space.

Results Option #2:

NovelAI might introduce elements that the writer never considered: perhaps the forest is connected to a parallel dimension, or maybe the trees have a complex social hierarchy based on the types of birds that nest in their branches. Maybe the encroaching industry is not simply destructive, but offers a twisted form of symbiosis with the forest. The writer is then free to take whichever surprising elements to weave into the narrative. They might have to adjust their original plans, but the resulting story becomes richer and more unpredictable, a true collaboration between human and AI.

BONUS: the writer could use NovelAI's image generation feature (or another tool like Midjourney or Grok) to create visuals of the forest and use those to inspire new narrative directions and organize in a free PKM like Obsidian.

Example of a specific collaborative moment:

Writer: "Write about the trees' perception of time."

NovelAI: "Time in Elderwood was not a linear progression, but a vast, echoing chorus of seasons. The trees remembered not just their own lives, but the lives of their ancestors, their sap mingling in a shared stream of consciousness that stretched back centuries. They experienced time as a series of concentric rings, each ring representing a different era in the forest's history."

Writer's Reaction: "The 'concentric rings' metaphor is brilliant! I'm going to use that as the basis for a unique system of forest magic, where the trees can manipulate the past by accessing different rings of time."

In this case, NovelAI's description of time inspires a completely new element in the story – a system of time-based magic – which the novelist then develops and integrates into the narrative. This showcases how a purpose-built AI tool can provide truly innovative ideas that go beyond simply executing pre-determined commands.

Key Differences using a tool like NovelAI compared to ChatGPT native:

Granular Control: NovelAI provides much more granular control over the AI's behavior through features like lorebooks, story cards, and stylistic settings.

Narrative Focus: It's specifically designed for storytelling, so it understands narrative structures and character development in a way that ChatGPT doesn't.

Iterative Process: NovelAI encourages a more iterative and experimental creative process, with features that make it easy to regenerate and refine text.

Multiple LLMs: As mentioned, this is incredibly important for creatives who really need maximum quality.

In the above example, the artist can fine-tune the AI's "voice" and guide it to create a story that aligns with their vision, or they can embrace the AI's unexpected suggestions and allow it to shape the narrative in surprising ways. The degree of control and collaboration is ultimately up to each individual and their creative goals.

EXAMPLE #2 - 3D Level Design

Consider the popular Unreal/Unity plugin Dungeon Architect . This plugin lets Level Designers (LDs) define the layout of a procedural "dungeon" (can be any kind of 3D world) using their flow editor. Once defined, the AI will create an infinite number of procedural dungeons that follow the layout rules. Dungeon Architect was designed with a powerful understanding that game designers need the flexibility to automate some things and lovingly hand-craft others.

Node graphs are generated using Graph Grammar rules, which is a both intuitive and powerful way of designing layouts. It gives LDs complete control on the layout and opportunities to rewrite and grow the graph as needed. In this tool we see a perfect example of control (LDs define what they care about) vs. AI randomness. Humans can define what is important and the computer can handle the tedium. A heavy handed approach might define large portions of the level traditionally, and choose to use the tool only to 'add color around the edges'. A more hands-off approach will define only critical components (maybe spawners, exit etc.) and give the AI free reign to build (including a likely subsequent curation/culling function by LDs).

We are going to see a LOT of innovation in the level layout & design domain, including some native tools inside Unreal/Unity soon. The ability to rapidly build fun environments that satisfy the complex constraints associated with the very best 3D games will address a massive pain-point (cost) for our industry. I sincerely hope this leads to fascinating and innovative level-designs and not job loss for the talent that has traditionally built these worlds.

Conclusion

The relationship between artists and AI is a dynamic and evolving one. I submit these tools are at their best when they are ‘inspiration engines’ not just task-execution machines. There's no right or wrong way to approach it the control<->serendipity spectrum; both ‘methods’ are extremely valid and each have their place. The key for creatives of all kinds is to be mindful of our intentions, experiment with different approaches, and ultimately find a way to use AI that allows us to develop our unique voice. I believe this ongoing discussion about control-versus-collaboration is crucial for shaping the future of AI.

#004 CONTROL

INTRO