generative artificial intelligence for early stage architectural prototyping

for early stage architectural architectural prototyping prototyping

tomáš bajer

VIA.DK · AT&CM AH41-23S · LPE 26 · 11 · 2023

generative artificial artificial intelligence intelligence

different

Like CAD software revolutionising the project documentation phase, the quickly developing generative artificial intelligence has the potential to shape our future outline phase, including our very own creative process.

HOW IS IT

motivation

TECHNOLOGICAL

introduction introduction

legend legend

Neural networks allowing this kind of abstraction and chance for subjective perception is something new, in my opinion, a revolutionising approach, which earlier tools just didn’t allow us to use. Even though genAI still works on predefined mathematical operations, the complexity is a bit overwhelming... For us, its deep processes might appear unclear - similar to human brains.

OBJECTIVE Sourced summary

Research environment, variables

Community based opinion

SUBJECTIVE Practices ··· tips and tricks All images used in this research are available to download on address tobacreations.cz/link/genAI Generation metadata are embedded in the image, so it is possible to extract the original prompt and settings using any PNG chunk inspector.

Personal opinion and evaluation

Evaluation ··· cons

Evaluation ··· pros

change us?

abstraction

I consider it a major technological shift thanks to two aspects… · The unprecedently short time in which researchers enhanced its capabilities and allowed public access. · It is the first of the tools which allows us to work with a lot of abstraction

Generative AI introduces a change to the psychological and workflow approach: We can provide very generic input with more freedom - with space for creativity, imprecision and even mistakes. In return, we are not expecting precise results - we often even prefer, to some degree, non-precise, unexpected and creative results.

This workflow allows for graphical and contextual abstraction from the beginning to the end.

WILL GENERATIVE AI

shift

TECHNOLOGICAL

So far, most software tools we have used aim for more precise, effective or complex workflow by automating subtasks. Moreover, genAI is no different to them it still falls within these categories. However, unarguably, there is something that makes us feel like it is different this time.

DETERMINISM VS.

Regular tools work deterministically - by preprogrammed operations on specific data formats as input and output. For the process to produce precise results, we need to enter precise data.

FEATURES

How can we incorporate this tool into our workflow. Not to replace it but rather enhance it. Not to give up our creative thinking, but rather utilise it in a more effective way.

Question

···

Objective

Limitation ··· Important feature The tools are shifting, and we need to advance with them. The generative AI will not replace us, but people who will know how to work with it might.

ABBREVIATIONS genAI = generative aritfical inteligence GAI = general artificial inteligence SD = Stable Diffusion

PROMPT SETTINGS

Image genAI at this very moment, works as a noise predictor. It does not understand what it generates, so any viable statical structure or typological relations can not connected with understanding or awareness. So, it is suitable for early-stage concepts where these features still remain in question.

But not so much for creating working floor plans, structural solutions or even full building forms... That is a challenge that will take a few years to overcome

SEAMLESS GRIDS

Controlled iteration for prototyping, morphing, alternating or developing concept features. Methods of guiding the image generation for purposes like form-finding, structure development or material iteration whilst keeping consistent results

ARE THERE ANY

feature

iteration

CONTROLLED

integration

Seamless integration into existing architectural process. The workflow has to be flexible to accommodate one’s individual needs in different stages with varying levels of detail. Workflow should be applicable to different input styles.

Recognize and maintain often distinct features of the concept. Either utilizing an automatic feature extraction or finding workflows to communicate objective geometry. Adjusting generation guidance for finding a preferred balance between creativity and fidelity to the original concept.

limitations?

The focus is on the full process, from source draft, sketch or model across processing the input, working with features, intents and expectations.

Iterating over results with different methods and generation settings.

SEAMLESS

Generative AI provides us with nearly infinite possibilities in the context of adjusting inputs and settings. For this very reason, it is not reasonable to try to explore every aspect. Setting up the generation often requires trial and error, even for skilled professionals.

This research aims not to become a guide for every possible situation given the amount of independent variables.

PRESERVATION

usage

PRACTICAL

The main focus of this thesis is exploring different possibilities and workflows for transforming architectural concepts into early-stage renderings.

Providing basic overview and possibly guidance for selected workflows in the field of AI-driven generative archviz.

SEAMLESS

working now

HOW ARE WE

Architectural workflow is an iterative process of developing an idea. Many possible approaches range from using a drawing medium to fully prototyping the volumes in virtual space environments. These processes are often combined and switched in different phases of the development. [empiri]

WHAT‘D BE NEXT?

delimitation delimitation

problem formulation problem formulation

In my opinion, successful workflow is based on utilizing effective communication channels with the genAI - meaning using appropriate tools and settings for the intended objective.

I have selected a few of them in the thesis. The following concepts were out of scope but would be definitely relevant to look into next.

Cascading iterative prompt engineering and finetuning, using bracket weight expressions, keyword blending Exploring different upscalers, fine tuners and samplers and settings such as CFG scale, and number of steps. Evaluating more extensions, resources, models and practices Programmatically generating more extensive sample grids and using this to select viable methods for future uses Utilizing more features of ControlNet - testing preprocessors and corresponding models (softedge, MLSD, p2p, shuffle) and its settings combination Generating user reference image embeddings - in the form of reference through ControlNet, inpaint reference or training custom LORAs. Training own model using Dreambooth, e.g.

copyright

All used input graphical materials are the author’s work. Copyright debate around AI-generated images is not entirely settled; for now the StableDiffusion model itself is an opensource under Open-RAIL license. Generated images are currently comparable to authors work licensed under CC BY 4.0 license.

The most well-known models of genAI at the moment are Midjourney, DAL·E 3 and StabIeDiffusion. All of them are powerful tools, but only StableDiffusion (SD) has tools and parameter settings to allow experimentation to be this robust.

controlNet

My PC weak so I used GoogleColab Pro to run Stable Diffusion in the cloud.

Controlled by web graphical interface Stable Diffusion web UI (A1111). Base model is consistently SD 1.5 because of the widest variety of checkpoint models and LORAs. All images resolution 512x512. Utilizing ControlNet and RegionalPrompter extensions.

GEOMETRY GUIDANCE WITH

to use

WHICH TOOL

environment

It’s impossible to judge the quality of images objectively as they consist of so many aspects ranging from texture details to overall lighting of the scene.

Subjective preferences have a significant influence on the perceived quality of the image. For these very reasons, it is not the quality being evaluated but rather the features, practices and consistency of outputs for the purpose of creating workflows for the early architectural process.

to consider

It is problematic to achieve objective representation and even harder ratings of showcased images because of the combined factors of the subjective preferences, network complexity, and partial randomness introduced to the input. Even inexact input might, by random chance, produce excellent output and the other way around.

Judging every quality aspect of each photo is not very practical because of the mentioned subjectivity and the simple reason that the photo can often speak for itself better than words. So I invite you as a reader to pay attention not just to the general mood and realism of generated images but also noticing: · Precision and creative development of defining features from input · Blending between building segments, consistency across the image · Application of material – suitability, texture scale, orientation, structure · Context of surroundings, scale, composition and scenery aspects, lighting

RESEARCH

The following measures were undertaken: · 1:1 image ratio and aligned position of used inputs, meaning the position and scale of objects is consistent · using identical prompt, model, resolution/ratio, CFG scale, iteration steps and other non-variable settings for each evaluated series · generating two variants for each case and selecting the preferred one by means of subjective quality.

OTHER IMPORTANT FEATURES

OF OUTPUTS

representation

I have selected three of my earlier projects with distinct features to demonstrate possible developments of different architectural concepts in different contexts. This way, I have the possibility to evaluate showcased methods on different objectives and assess their benefits and shortcomings.

evaluated

For purposes of my research, I have selected three differently developed ideas, meaning used various mediums and stages. This is useful for comparing different feature extraction procedures.

WHAT IS BEING

STAYING RELEVANT FOR

diverse workflows

How to examine various techniques to explore a wide range of usages?

AND IMAGE SOURCES

methodology methodology

Guides diffusion towards an image that aligns with the graphical input.

Functions on a principle of hyper neural network that enables conditional control signal in the diffusion process.

[6][7] Helps to guide the diffusion on a graphical level by providing additional graphical information, such as: · edge maps that outline the shapes · segmentation maps that identify different regions in the image · key points that mark object locations

inputs inputs HYBRID DRAFTING, COLLAGE

a

USED SD 1.5 MODEL BUILDING CONCEPT CONTEXT Contemporary art museum with public space in parterre. Organic architecture with sharp flowy edges Embankment of developing city area, possibly on a river side or an man-made island

INPUT DETAILING More advanced conceptual phase incorporating detailed aspects like glazed areas, material texture drafts, or e.g., furniture.

Objects are smoother but still carry a high degree of abstraction in relations between them. Possibly sketch underlayed with photo, 3D model of environment, or pure collage

OBJECTIVE OF PROTOTYPING FEATURES TO PRESERVE What possible facade types and shapes could work on this concept. Developing design and prototyping outer shell while keeping outline curves

Circular water area in front with access bridge, big glazed curtain walls filling segments of the façade. Supporting outer shell appearing thin but still statically viable

dreamshaper_8

3D VOLUME PROTOTYPING

Recreational centre with circular floor plan opening towards the water area and allowing access from a circular pier. Unconventional glamping shed with wooden shingle facade

Conceptual depiction of building volume from 3D model environment. Precise and specific building outlines concerning shape form, varying degree of façade internal detail.

Nearly flat terrain, empty area enclosed by open field areas and coniferous forests and nearby village.

Exploring different ideas and alternative solutions to design and connection of buildings while keeping overall shape and composition

external supporting system of columns, glamping shed in foreground island, circular pier with shading

DRAFT SKETCHING

Modern residential district, enclosed by similarly tall building, sloped terrain with retaining wall and roll on top, behind the building Residential building based on ortholinear box grid, matching horizontal planes, offseted vertical volumes.

Basic handdrawn conceptual outline, possibly original on paper medium, high degree of abstraction, nonprecise, multiple lines overlaying each other, little to none shading, only few detailed segments

Experimenting with facade structure, segments and composition while keeping somewhat true to the original proportions

Distinct groundfloor with lot of glazed areas and lamelle elements. Cantilevered balcony boxes are regularly placed on the façade

of the rest of the façade are coated with reflective(white metal panels:1.5), highlighting the organic shapes BREAK Located on an open artificial island on a river inside a big city, connecting and blending with the water creates stunning reflections.

Building is surrounded by greenery and cityscape in the background. BREAK Exterior professional photo of the building shows futuristic vision of the structure in soft diffused hyper realistic light and vibrant colours. Cold tones emphasize sophistication and complex details.

‘HDArchiMaster- v1.0 | Stable Diffusion Checkpoint | Civitai’ Accessed: Nov. 08, 2023. [Online]. https://civitai.com/models/130811?modelVersionId=143619

Private circular residence on the lakeside in close connection with surrounding nature. Fluid shape inspirated by elegance of Santiago Calatrava. The building has many bent glazed areas and visually captivating external support structure consisting of subtle wooden columns intersected by massive wooden beams. Hosting restaurant, club area and spacious apartments overlooking the reflective surface of the lake. BREAK Wedged into the terrain, allowing access to the circular pier elevated on complex concrete and wooden construction. This

architectureExterior_v60

c

‘DreamShaper- v8.0 | Stable Diffusion Checkpoint | Civitai’ Accessed: Nov. 08, 2023. [Online]. https://civitai.com/models/4384?modelVersionId=128713

Isolated tall building in organic contemporary architecture style, showcasing fluid shapes while maintaining sharp edges. Influenced by the harmonic and dynamic shapes of Zaha Hadid and usage of white metal facades by Frank Gehry Following trends and architectural quality of modern concert halls, art museums, and museums. Its shape resembles waves with futuristic and extravagant expression. BREAK The building is captivated by its extensive fluid glass façade, consisting of big glass panes mounted on steel structure. Segments

hdarchimaster_v10

b

USED PROMPT

dynamic wooden platform is shaded by futuristic looking textile curtain hanged on curved columns.

The site is located in the middle of meadows enclosed by forests BREAK Untypical wooden shingle covered glamping shed in front on a small island accessible by pier. It is elegant, curved shape is resembling atmosphere of a yurt or teepee. BREAK The architecture expresses connection with the raw power of nature, the shape follows circular arcs and golden ratio while showcasing transparency, luxury, and openness.

‘architecture_Exterior_SDlife_Chiasedamme_V6.0 - v6.0 | Stable Diffusion Checkpoint | Civitai’. Accessed: Nov. 08, 2023. [Online] https://civitai.com/models/126660?modelVersionId=143619

Modern residential building with multiple terraced stories full of extensive greenery inspired by the works of studio BIG - Bjarke Ingels architects.

Significant horizontal lines, sharp edges, multiple levels, fragmented façade combining metal, big glazed areas and white brick covering the facade.

BREAK The concept aims to provide an ideal environment both inside and outside, creating form of retreat in the centre of pulsating modern city. Well maintained public spaces with trees creating blend of modern architecture with natural environment

BREAK Two story balconies in form of blocks shielded by dark matte metal sheets extending out of the segmented façade. Ground floor has wide glass façade with tall metal lamellas as shading elements. BREAK Highly detailed exterior scene of the building with cityscape in background. Soft diffused light coming through the clouds, contrast shadows helping to the depth perception, warm colour temperature adding vibrancy, award winning architectural photo, architecture embodies an urban oasis concept

theory theory

How does it pick which features of original images to generate?

resources resources

academic papers [1]

How does it generate images? Are they original or just segments of existing?

A detailed description of the inner workings of these networks is out of the scope of this thesis. But a general understanding of the concept is crucial for structured and documented research and successful application of the methods.

By manipulating the latent space, the model can explore and generate variations of the desired image, ensuring that the generated image closely aligns with the textual prompt. [3]

EMBEDINGS

Stable Diffusion operates within the latent space of images, a high-dimensional space where each point represents a unique image, and point coordinates express an aspect of the image.[3] Competing models like Midjourney and DALL-E, in contrast, work on pixel space of images.

navigation

StableDiffusion utilizes a specialized neural network called U-Net to perform a diffusion process. The model starts with an image of pure noise with normal distribution. Then, it iteratively reduces the noise by predicting the initial noise and subtracting a specified amount of it from the image.

This works thanks to the GAN model structure – generative adversarial network, which is trained on denoising real images whilst receiving feedback on its success. StableDiffusion was then trained on a dataset of captioned images (LAION-5B =5.9 billion) [3][6][7]

LATENT SPACE

diffusion

BASED IMAGE GENERATION

Stable Diffusion is a generative model, meaning it can create new data, in this case, images. Simply explanied, it works by denoising an image of random noise whilst being guided by the text prompt. [3]

role of text

How does it pick which features of original images to generate?

Words and relations between them are encoded into text token embeddings using a transformer language neural network. Text embeddings are also based on latent high-dimensional space, which allows the expression of meaning by a mathematical vector, where each dimension corresponds to a different aspect of the text. Similar words produce similar vectors. These tokens then provide guidance in multiple layers of the U-net - the image generation process.[1] [3]

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, ‘Hierarchical Text-Conditional Image Generation with CLIP Latents’, Apr. 2022, [Online]. Available: https://arxiv.org/abs/2204.06125v1 [2] T. Salimans and J. Ho, ‘Progressive Distillation for Fast Sampling of Diffusion Models’, ICLR 2022 10th International Conference on Learning Representations, Feb. 2022 [Online]. Available: https://arxiv.org/abs/2202.00512v2 [3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, ‘High-Resolution Image Synthesis With Latent Diffusion Models’ [Online]. Available: https://arxiv.org/abs/2112.10752 [4] L. Huang, D. Chen, Y. Liu, Y. Shen, D. Zhao, and J. Zhou, ‘Composer: Creative and Controllable Image Synthesis with Composable Conditions’, Feb. 2023 [Online]. Available: https://arxiv.org/abs/2302.09778v2 [5] Y. Vinker, Y. Alaluf, D. Cohen-Or, and A. Shamir, ‘CLIPascene: Scene Sketching with Different Types and Levels of Abstraction’, Nov. 2022 [Online]. Available: https://arxiv.org/abs/2211.17256v2 [6] A. Voynov, K. Aberman, and D. Cohen-Or, ‘Sketch-Guided Text-to-Image Diffusion Models’, 2023.

[Online]. Available: https://arxiv.org/abs/2211.13752 [7] L. Zhang, A. Rao, and M. Agrawala, ‘Adding Conditional Control to Text-to-Image Diffusion Models’, Feb. 2023, Accessed: Oct. 21, 2023. [Online]. Available: https://arxiv.org/abs/2302.05543v2

environment & extensions [10] GitHub user lllyasviel, ‘lllyasviel/ControlNet: Let us control diffusion models!’

[Online]. Available: https://github.com/lllyasviel/ControlNet [11] GitHub user Mikubill, ‘Mikubill/sd-webui-controlnet: WebUI extension for ControlNet v 1.6.0.’

[Online]. Available: https://github.com/Mikubill/sd-webui-controlnet [12] GitHub user AUTOMATIC1111, ‘AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI’.

[Online]. Available: https://github.com/AUTOMATIC1111/stable-diffusion-webui [13] GitHub user hako-mikan, ‘hako-mikan/sd-webui-regional-prompter: set prompt to divided region’.

[Online]. Available: https://github.com/hako-mikan/sd-webui-regional-prompter

community sourced [20] ‘Stable Diffusion: Basic Intuition Behind Generative AI - KDnuggets’. Accessed: Nov. 08, 2023. [Online]. Available: https://www.kdnuggets.com/2023/06/stable-diffusion-basic-intuition-behind-generative-ai.html [21] ‘ControlNet v1.1: A complete guide - Stable Diffusion Art’. Accessed: Oct. 21, 2023. [Online]. Available: https://stable-diffusion-art.com/controlnet/ [22] ‘Regional Prompter: Control image composition in Stable Diffusion’. Accessed: Oct. 25, 2023. [Online]. https://stable-diffusion-art.com/regional-prompter/ [23] Stable Diffusion prompt: a definitive guide‘. Accessed: Oct. 24, 2023. [Online]. https://stable-diffusion-art.com/prompt-guide/ [...] Inspirational YouTube channels: Papers and news: ColdFusion, Two Minute Papers, Computerphile, Practical showcases: Nerdy Rodent, Futurepedia, altArch, DevCreates, Sebastian Kamph, koiboi

Chaining keywords into contextual sentences. Text transformer works by capturing relations between words, and they tend to produce more consistent results with the usage of sentences.

your prompt chunks

SAY WHAT YOU DON‘T WANT

Reaction to the prompt depends on how the model was trained. Basically on the text descriptions of images and structure of the training dataset.

Try to use common words or phrases which could have appeared as captions for training images.

rabbit hole

BUT FAMILIAR

Talking in beautiful, extensive descriptions can actually pay off. Using abstracts and emphasizing keywords in the prompt chain can effectively lead to better results. Even though the network does not understand these “abstracts”, it helps it to navigate to the area of image latent space, which more consistently produces better results.

STRUCTURE

Linking to a specific author or resource, for example, an artist, celebrity, or image source website, can serve as a good shortcut for achieving the desired style and better consistency.

But it also carries the risk of unwanted correlation and stereotypes.

DOWN THE

Use a multi-category detailed approach to the prompt. The StableDiffusionArt guide provides us with the following general categories which are worth considering: Subject, Medium, Style, Artist, Website, Resolution, Additional details, Colour, Lighting

The usage of negative prompt helps guide the generation, for SD 2.0+ generation models, it is almost imperative.

USE SENTENCES

BE DIVERSE USE EXAMPLES

nice phrases

experiment

GET INSPIRED &

There are no right or wrong how-to-write prompts; they all kind of work.

The only bad practice would be not experimenting with it and not getting experience. It is often practised to reuse the same segments (for example, scene settings and negative prompts), which helps with consistency. Get inspired on sites like CivitAI and experiment on your own!

Creating a stable prompt can become very unintuitive as only changing the order or rephrasing words can lead to considerable changes in results. As good practice appeared, engineering the prompt and then fine-tuning different chunks of it on their own or in full context. Generating a batch of results in lower quality to understand the probabilities.

EXAGGERATE A BIT WITH

what works

Text description which guides the image generation. It can range from a few words to a long-engineered keyword chain.

HOW TO TELL

summary

Which role does prompting play in the workflow with genAI

Prompt engineering aims to achieve a more controlled generation of specific features (such as object, style, or detail) and increase consistency. Prompting is critical to achieving overall consistency of the results. Summary of principles, which are generally accepted within the community around StableDiffusion.[23]

be creative

practices

Prompt means text input for the genAI. It allows us to communicate our intents through words in virtually any form - as isolated keywords, long sentences, in multiple languages... The model then uses a text transformer to understand relations between words and create tokens which use to navigate through its image latent space. [3][5]

GENERAL

BEHIND

prompt engineering

complexity

THEORY OF

The longer prompt is divided into chunks of 75-word tokens; web UI displays this number above the text field as tokens in the latest chunk/ total token space.

The beginning of each token can be described as a “spike of attention”. I consider it even more effective than using weights. Important keywords placed at the front will have a much higher chance of appearing in the image.

Prompting can utilize advanced techniques like keyword weights and keyword blending and is also influenced by the model structure and training itself, keyword correlation, stereotyping, and bias. [1][3]

PRACTICAL

prompt engineering

[extraction and analysis of selected images at https://civitai.com/search/models?query=architecture]

abstracts

NEGATIVE blend of innovation and aesthetic elegance, award-winning, professional, visionary architect, innovative design, architectural marvel, masterpiece, spacious public areas, optimistic atmosphere, sophistication, promise of tomorrow, harmonious blend of luxury, seclusion, connection with the raw power of nature, golden ratio, architecture embodies an urban oasis concept, visually captivating and emotionally resonant space, blend of innovation and aesthetic elegance, showcase transparency and openness

NEGATIVE

abstracts

highly detailed, soft diffused lighting, soft shadows, warm color temperature adding vibrancy, highlighting the architectural details, realistic, 8k, best quality, depth of field, complex details, natural light pouring in, vibrant cloudy sunset sky, day time, front view, aerial view, Soft ambient lighting casts a warm and welcoming glow. The medium for this artwork is an architectural blueprint rendered in high-definition 3D graphics, golden hour, hyperrealism, ultra wide shot

objects

street-facing, exterior view of building, no humans, cityscape in background, blurred simple scenery background, building, outdoors, road, sky, street, lamppost, tree, power lines, super high-rise buildings in the block, breathtaking location perched on the edge of a cliff, reflective paved area around, complemented by lush greenery and reflective water areas. visually captivating, emotionally resonant,

POSITIVE

& CONTEXT

settings

towering glass and steel skyscraper reaching for the sky, floor-to-ceiling windows offering unobstructed views, glass curtain walls with white metal frame, dynamic grey metal façade, walls made of pristine concrete, incorporating advanced energy-efficient technologies and dynamic facade elements, expansive terraces with greenery and reflective paving, open-space concept living areas, multiple storey spacious entry hall with reception, suspended walkways and floating gardens, seamlessly integrates natural elements into its design, rounded corners and angles, beveled edges

scenery

Reflecting elegant and sophisticated architecture of Frank Gehry, showing characteristics of Zaha Hadid, inspired by the works of studio BIG - Bjarke Ingels architects, envisioned by Zhang Lei, Inspired by the works of Syd Mead

SCENE / CAMERA

OR FEATURE

spatial aspects

modern organic architecture with assymetric shapes, futuristic design elements, modernist architecture, minimalist design with clean lines and geometric shapes, New York, Japan, sleek lines, sharp edges, European style, inspired in natural shapes.

architect

high-rise shopping centre, residential district, luxury villa, cliffside vacation retreat, contemporary art gallery, modern museum, office building Some keywords tend to carry an influential stereotypes, for example “office building” tends to produce very monotonous rectangular buildings

BUIDLING DETAILS &

style

ARCHITECTURAL

type

BUILDING

FOR EXTERIORS

interior, inside the building, glass wall, people, dense columns, levitating elements, roof construction above, huge canopy, cantilever, roofed plaza, part of a building

low quality, paintings, sketches, signature, soft, blurry, flat image, drawing, sketch, poor quality, text, word, logo, pixelated, low resolution, high contrast, oversharpened, dirt, monotonous foreground, cropped image, out of frame, amateur, cut-off, watermark, underexposed, overexposed, surrounded, out of scale

MORPHING

image to image img2img

a

0

b

c

original

1

denoising strength 0,2

2

denoising strength 0,4

3

denoising strength 0,6

4

denoising strength 0,8

METHOD The model does not start from noise, but partially noised input image. denoising strength specifies how much noise is added. [3]

EVALUATION Ensures overall similarity including colour palette and object shape. Very usefull for gentle iterative reshaping and stylizing of existing concept with low denoising amount

Fine detail and and curve shape is usually deformed or lost before achieving appropriate texturing or desired spatial or form changes.

Takes long time to developed correct textures on pure outline drafts with empty background...

...but even basic shading helps

BEHAVIOUR Noise tends to influence the most fine details on the edges and could result in morphing of corners or change of perspective

Overall shape is the last thing to disappear. By dimming the eyes, one can identify the most contrastful parts. This blurred outline is somewhat similar to what could the genAI work with.

GUIDING WITH

scribble

METHOD We tend to think like each curve has its meaning and feature such as bounding, enclosing or leading... But all the genAI sees is an edge where something should end - no matter if precise facade element, complex skyline silhouette, or unclear tree edge. Account for it.

controlnet control_v11p_sd15_seg 0

a

b

INPUT ··· scribble

1

weigh 0,2 ··· end step 0,2

2

weigh 0,5 ··· end step 0,5

3

weigh 1,0 ··· end step 0,8

ControlNet extension introduces geometrical guidance to the generation process. [7]

FEATURE EXTRACTION Guiding curves can be generated by preprocessor using algorithmic edge detection (great for photos) or directly providing image lines (ideal for sketches) [7]

The preprocessor fails to keep a balanced level of detail. It often includes too many or too few edges.

Direct sketch guidance input is created most easily from all the showcased ControlNet methods, but has to be kept clean and reasonably detailed. Impact of detailing was explored on page 12

BEHAVIOUR We can achieve very distinct and vibrant designs while keeping selected level of precision. But the outline on its own is never able to fully express its original meaning. Only where geometrical or segmentational edge should appear

CONTROL WEIGHTS To follow shapes and explore their possible interconnections iterate with end control step To generate object and morph them by following the shapes iterate with start control step

EVALUATION The looser the control weight, the more the curves got disjointed and incorporated into different segments

c INPUTS Inputs a,c from digital sketching in Concepts app (use of layers to speed up process). Input b using Photoshop to extract edges from Rhino screenshot

Great for guiding statical element orientation, textures, depth, overall geometry.

Does not carry information about source object type which can lead to unwanted associations (horizon line becomes roof of the building), extensions (the building is enclosed) or splitting of objects

DETAILED

scribble

Sometimes original geometry could be hardly visible, try overlaying with the input outline, if you can see its features in there.

This could help you explain the unexpected development.

controlnet control_v11p_sd15_seg

a

0

INPUT ··· scribbleAuto

1

weigh 1,0 ··· end step 0,8

2

INPUT ··· scribbleDetailed

Even small curve extension are often transferred to the outputs (2a 2a → 4a, 5a).

5a On the other hand, gaps, dashes and non touching lines do not pose a challenge for the generation and their interconnection is smooth and quite precise (2c 2c → 4c, 5c).

5c

3

weigh 0,2 ··· end step 0,2

4

weigh 0,5 ··· end step 0,5

5

weigh 1,0 ··· end step 0,8 ControlNet works by providing simply said anchoring geometry.

It does not specify explicitly boundary of objects, only where some kind of edge should occur either internal (crease) or naked edges (segment boundary) [7][21]

b

c

For this very reason, the buildings were often confused as a part of bigger structure, extending on top or bottom.

For smooth and clean corners in outputted geometry is generally better practice to remove the corners than to use input with them being over-extended or as dense nods from multiple lines.

scribble detailing HOW DETAILED IS TOO DETAILED

a

0

Unenclosed shapes without apparent border forces the genAI to find a way how to connect and cap the open curves. As their number decreases, cleanear connections appeared

RAW INPUT ··· original Both thin and bold curves in inputs 1, 2 were often ignored due to unclarity caused by shading, in example 3, 4 only few thin dense lines were not identifiable in generated image. In 5, 6 every input curve is clearly visible in the result.

Curve in segmentation only explicitly defines the edge itself, not any context or amount of nearby edges to generate. Therefore is often confused as part of different elements than intended [6][21]

controlnet control_v11p_sd15_seg control weight: 1.0 ending control step: 0.8 0

autogenerated, resolution 1600px

1

PROCESSED INPUT

PREPROCESSED INPUT

i

o

overdetailed, unenclosed, shaded

2

overdetailed, unenclosed, highlited

3

overdetailed, unenclosed

4

overdetailed, enclosed

5

detailed, enclosed

6

surface outlines, enclosed

WORKING WITH

segmentation AUTO SEGMENTATION

GENERALLY

a

INPUT ··· segmentsAuto

USER SEGMENTATION

Preprocessor uses „semantic segmentation“ which distinguishes different object types. [7]

Complex process with multiple possible levels, based on identifying objects in scene using computer vision. Objects are usually represented by colour filled areas. In SD is used for both generation of image and preprocessing graphical guidance. [3][7]

0

INPUTS

1

With direct input (without preprocessor) of segmentation maps, we can even utilize instance segmentation by further separating independent objects with slightly offsetted colors from base category. [7]

Combination with scribble is effective choice how to provide correct meaning to shapes whilst maintaining geometry of their edges.

autosegmentation output weight 1,0 ··· end step 0,8

control_v11p_sd15_seg control_v11p_sd15_scribble

b

INPUT ··· segmentsAuto

autosegmentation output weight 1,0 ··· end step 0,8

c

manual segmentation outputs 2 both weigh 1,0 ··· end step 1,0

Autogenerated segmentation maps were extremely irrelevant.

For this reason, I did not showcase them. Using 150 predefined categories with color code (table attached in Excel and PNG). Input a post-colored in Photoshop, b is original export from Rhino (useful script attached), c is original export from Concepts app.

EVALUATION Great for guiding statical element orientation, textures, depth, overall geometry.

Segmentation carries information only about border and object type, delicate details are lost, spatial orientation is guessed.

Segmentation at the moment is not very optimized for architectural purposes - only limited amount of predefined regions not allowing material or structural specification Only basic exterior element such as column and window are assigned, and general concepts such as house, building and skyscraper.

Despite this, it is still very much usable. With extended segmentation table would this method be extremely effective.

wall

cushion

bar

tent; collapsible; shelter

(120, 120, 120)

(255, 194, 7)

(0, 255, 153)

(112, 224, 255)

building; edifice

base; pedestal; stand

arcade; machine

bag

(180, 120, 120)

(255, 122, 8)

(255, 92, 0)

(70, 184, 160)

sky

box

hovel; hut; hutch; shack; shanty

minibike; motorbike

(6, 230, 230)

(0, 255, 20)

(255, 0, 255)

(163, 0, 255)

floor; flooring

column; pillar

bus; autobus; coach; charabanc;

cradle

(80, 50, 50)

(255, 8, 41)

double-decker; jitney; motorbus;

(153, 0, 255)

tree

signboard; sign

motorcoach; omnibus; passenger;

oven

(4, 200, 3)

(255, 5, 153)

(71, 255, 0)

ceiling

chest; of; drawers; chest; bureau;

vehicle (255, 0, 245) towel

(120, 120, 80)

dresser (6, 51, 255)

(255, 0, 102)

(255, 0, 163)

road; route

counter

light; light; source

food; solid; food

(140, 140, 140)

(235, 12, 255)

(255, 173, 0)

(255, 204, 0)

bed

sand

truck; motortruck

step; stair

(204, 5, 255)

(160, 150, 20)

(255, 0, 20)

(255, 0, 143)

windowpane; window

sink

tower

tank; storage; tank

(230, 230, 230)

(0, 163, 255)

(255, 184, 184)

(0, 255, 235)

grass

skyscraper

chandelier; pendant; pendent

trade; name; brand; name; brand;

(4, 250, 7)

(140, 140, 140)

(0, 31, 255)

marque (133, 255, 0)

cabinet

fireplace; hearth; open; fireplace

awning; sunshade; sunblind

microwave; microwave; oven

(224, 5, 255)

(0250, 10, 15)

(0, 255, 61)

(255, 0, 235)

sidewalk; pavement

refrigerator; icebox

streetlight; street; lamp

pot; flowerpot

(235, 255, 7)

(20, 255, 0)

(0, 71, 255)

(245, 0, 255)

person; individual; someone;

grandstand; covered; stand

booth; cubicle; stall; kiosk

animal; animate; being; beast;

somebody; mortal; soul

(31, 255, 0)

(255, 0, 204)

brute; creature; fauna (255, 0, 122)

(150, 5, 61)

path

television; television; receiver;

bicycle; bike; wheel; cycle

(255, 31, 0)

television; set; tv; tv; set; idiot; box;

(255, 245, 0)

earth; ground

stairs; steps

boob; tube; telly; goggle; box

lake

(120, 120, 70)

(255, 224, 0)

(10, 190, 212)

door; double; door

runway

(0, 255, 194) airplane; aeroplane; plane

(8, 255, 51)

(153, 255, 0)

(0, 255, 82)

dishwashing; machine (214, 255, 0)

table

case; display; case; showcase;

dirt; track

screen; silver; screen; projection;

(255, 6, 82)

vitrine (0, 0, 255)

(0, 10, 255)

screen (0, 204, 255)

mountain; mount

pool; table; billiard; table; snooker;

apparel; wearing; apparel; dress;

blanket; cover

(143, 255, 140)

table (255, 71, 0)

clothes (0, 112, 255)

(20, 0, 255)

plant; flora; plant; life

pillow

pole

sculpture

(204, 255, 4)

(0, 235, 255)

(51, 0, 255)

(255, 255, 0)

curtain; drape; drapery; mantle; pall

screen; door; screen

land; ground; soil

hood; exhaust; hood

(255, 51, 7)

(0, 173, 255)

(0, 194, 255)

(0, 153, 255)

chair

stairway; staircase

bannister; banister; balustrade;

sconce

(204, 70, 3)

(31, 0, 255)

balusters; handrail (0, 122, 255)

(0, 41, 255)

car; auto; automobile; machine;

river

escalator; moving; staircase;

vase

motorcar (0, 102, 200)

(11, 200, 200)

moving; stairway (0, 255, 163)

(0, 255, 204)

water

bridge; span

ottoman; pouf; pouffe; puff;

traffic; light; traffic; signal; stoplight

(61, 230, 250)

(255 ,82, 0)

hassock (255, 153, 0)

(41, 0, 255)

painting; picture

bookcase

bottle

tray

(255, 6, 51)

(0, 255, 245)

(0, 255, 10)

(41, 255, 0)

sofa; couch; lounge

blind; screen

buffet; counter; sideboard

ashcan; trash; can; garbage; can;

(11, 102, 255)

(0, 61, 255)

(255, 112, 0)

wastebin; ash; bin; ash-bin; ashbin;

shelf

coffee; table; cocktail; table

poster; posting; placard; notice; bill;

dustbin; trash; barrel; trash; bin

(255, 7, 71)

(0, 255, 112)

card (143, 255, 0)

house

toilet; can; commode; crapper; pot;

stage

(173, 0, 255) fan

(255, 9, 224)

potty; stool; throne (0, 255, 133)

(82, 0, 255)

(0, 245, 255)

sea

flower

van

pier; wharf; wharfage; dock

(9, 7, 230)

(255, 0, 0)

(163, 255, 0)

(71, 0, 255)

mirror

book

ship

crt; screen

(220, 220, 220)

(255, 163, 0)

(255, 235, 0)

(122, 0, 255)

rug; carpet; carpeting

hill

fountain

plate

(255, 9, 92)

(255, 102, 0)

(8, 184, 170)

(0, 255, 184)

field

bench

conveyer; belt; conveyor; belt;

monitor; monitoring; device

(112, 9, 255)

(194, 255, 0)

conveyer; conveyor; transporter

(0, 92, 255)

armchair

countertop

(133, 0, 255)

bulletin; board; notice; board

(8, 255, 214)

(0, 143, 255)

seat

stove; kitchen; stove; range; kitchen;

canopy

shower

(7, 255, 224)

range; cooking; stove

(0, 255, 92)

(0, 133, 255)

fence; fencing

(51, 255, 0)

washer; automatic; washer;

radiator

washing; machine (184, 0, 255)

(255, 214, 0)

ball

dishwasher; dish; washer;

(184, 255, 0)

(255, 184, 6) desk

palm; palm; tree

plaything; toy

glass; drinking; glass

(10, 255, 71)

(0, 82, 255)

(255, 0, 31)

(25, 194, 194)

rock; stone

kitchen; island

swimming; pool; swimming; bath;

clock

(255, 41, 10)

(0, 255, 41)

natatorium (0, 184, 255)

(102, 255, 0)

wardrobe; closet; press

computer; computing; machine;

stool

flag

(7, 255, 255)

computing; device; data; processor;

(0, 214, 255)

(92, 0, 255)

lamp

electronic; computer; information;

barrel; cask

(224, 255, 8)

processing; system (0, 255, 173) swivel; chair

(255, 0, 112)

bathtub; bathing; tub; bath; tub (102, 8, 255)

(10, 0, 255)

(92, 255, 0)

railing; rail

boat

waterfall; falls

(255, 61, 6)

(173, 255, 0)

(0, 224, 255)

basket; handbasket

formatted by Tomáš Bajer, TobaCreations © 2023 based on https://docs.google.com/spreadsheets/d/1se8YEtb2detS7OuPE86fXGyD269pMycAWe2mtKUj2W8

GUIDING USING

segmentation controlnet control_v11p_sd15_seg

a

0

b

c

Precesion difference between 5a and 5c 5c. Frequent object boundary expansion with c might have been caused by space above separated by easily followable lines. Influence by used model also often occurs.

More conceptual and unprecise inputs work better with looser guidance weights High values of control weight result in strong guidance of fine details.

Beneficial only for originally precise b2, b3. b3

1

weigh 0,5 ··· end step 0,5

2

weigh 1,0 ··· end step 1,0

3

weigh 2,0 ··· end step 1,0

Test how each model reacts to weights of ControlNet guidance

controlnet control_v11p_sd15_scribble 4

5

scribble ··· w 0,3 ··· end 0,3 segmentation ··· w 0,5 ··· end 0,5

6

weigh 0,5 ··· end step 0,5 weigh 0,8 ··· end step 0,8

7

weigh 0,5 ··· end step 0,5 weigh 0,8 ··· end step 0,8

DETAILED

segmentation

in all b outputs the segmentation failed to create any kind of textile membrane or cloth material. Strongest guidance achieved translucent material, but still no cloth. Notice, how SD was able to generate correctly curved suppport columns for the shed, eventhough they were not showed anywhere - just by using outer shape.

controlnet control_v11p_sd15_seg

a

0

b

c

1

weigh 0,5 ··· end step 0,5

2

weigh 1,0 ··· end step 1,0

3

weigh 2,0 ··· end step 1,0

Every model reacts differently to strenght of guidance. Stronger guidance would help row c , but would make row a a bit worse

controlnet control_v11p_sd15_scribble 4

5

scribble ··· w 0,3 ··· end 0,3 segmentation ··· w 0,5 ··· end 0,5

6

weigh 0,5 ··· end step 0,5 weigh 0,8 ··· end step 0,8

7

weigh 1,0 ··· end step 1,0 weigh 0,8 ··· end step 0,8

UTILIZING POWER OF

depth map controlnet control_v11p_sd15_depth

a

0

RAW INPUT ··· original

preprocessor depth_midas 1

INPUT ··· depthAuto

2

weigh 1,0 ··· end step 1,0

3

INPUT ··· depth

4

weigh 1,0 ··· end step 1,0

ESTIMATED DEPTH MAP Preprocessor is able to approximate depth map from existing 2D image [7] Limited depth resolution. Deeper sceneries (larger rooms for e.g.,) are analysed only in foreground area. Thin elements are often evaluated as part of background

TRUE DEPTH MAP Generated from 3D software modeler, precise and but requires actual model of concept. Depth resolution limited only by image channel depth (8 bit/channel = 256 layers) [wikiwand.com/en/en/Channel_(digital_image)]

b

EVALUATION Very effective for depicting correct spatial relations within scenery and defining larger smoothly curved areas, where previous methods fail to recognize these features

c INPUTS All inputs depthAuto are based on preprocessing RAW inputs. a3,c3 are manually cleaned a1, c1 in Photoshop, b3 is export from Rhino Z-buffer

Does not guide edges very well.

Challenging to use without 3D model - with sketches and drawings where processed image often requires manual clean up Combination with scribble could produce precise volume topology

camera elevation CONTEXT By the nature of model training datasets, the amount of architectural photos captured from human perspective is larger, than those captured from highly elevated platform or from drone shots. [3] [23]

S

HYPOTHESIS Therefore generated images from these aerial perspectives should be of worse quality. Either in terms of realistic depictions or would confuse model into generating different scenery and illogical connections

TESTING METHOD four different camera altitudes capturing 3D model, extracting edges and true depth map. controlnet „S“ ROW ··· control_v11p_sd15_seg „D“ ROW ··· control_v11p_sd15_depth control weight: 1.0 ending control step: 0.8

CONCLUSION The hypothesis is strongly supported by the scribble input. An addition to negative prompt had to occur from the second image as generated pictures were generally interiors. When provided a depth map - more spatially representing input, the results were more precise, but decline in quality is still observable

D

original angle

2a

with horizon line

2b

without horizon line

2b

negative prompt adjustment

HOW MATTERS

1

without horizon line + adjusted negative prompt

3

4

BASIC

METHOD

regional prompting controlnet control_v11p_sd15_seg

a

0

regionsDetailed

Due to compatibility issues, I only used Regional Prompter and its „mask“ option. Compared to similar extensions like „latent couple“ or „paint-with-words“ is less intuitive but more popular

controlnet control_v11p_sd15_scribble 1

base prompt weight 0.2

2

scribbleDetailed

3

scribble ··· w 0,5 ··· end 0,5 no base prompt

4

weigh 0,5 ··· end step 0,5 base prompt weight 0.2

5

A method of targeting different prompts to specified parts of image. Uses principle of segmentation with customized meanings. Grid or geometrical = shape mask can be used.[22]

weigh 0,8 ··· end step 0,8 base prompt weight 1.0

BEHAVIOUR Does not specify boundary of the segment as previous methods did. Only guides towards executing prompt chunk in the area.

Relevant results achieved with simultaneous use of ControlNet.

Regional prompter uses prompt chunks to target different regions in predefined order of colors.

Make sure each region’s prompt is only one chunk separated by BREAK statements. Negative prompt can be only single chunk.

Baseprompt applies to unassigned regions and general image.

Error „index out of range“ could mean you don‘t have enough chunks or multiple in negative prompt. Colors have to be precise, I have created a graphical guide and Python script for Rhino models that could help (attached)

b

REGION PROMPTER COLOR INDEXES

c

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

regional prompting regional prompter

a

0

regionsDetailed

Guided generation of smaller zones with atypical objects in scenery requires a lot of trial and error to be perfected (even with usage of further guidance)

controlnet control_v11p_sd15_scribble 1

base prompt weight 0.2

2

scribbleDetailed

3

scribble ··· w 0,5 ··· end 0,5 no base prompt

4

weigh 0,5 ··· end step 0,5 base prompt weight 0.2

in all b outputs regional prompting failed to create any kind of textile membrane or cloth. It often slipped to generating plank construction or tree row.

5

weigh 0,8 ··· end step 0,8 base prompt weight 1.0

EVALUATION

DETAILED

Great for bigger areas of atypical elements, textures and further guiding detailing on objects.

Increased probability of correctly generating in areas, that have enough space around and therefore don‘t get covered by neighbors

Did not prove as effective as segmentation for precise preservation of edges. The guidance is looser and does not guarantee the generation of all specified elements. Custom mask input has to accurately follow predefined colour (inlcuded script and guide that could simplify this)

SEGMENTATION COMPARISON

b

Other extensions such as „paint-with-words“ or „latent couple / two shot„ might be more successful with this task.

INPUTS

c

Using predefined color sequence (guide and script for Rhino attached). All inputs a, b, c post-colored in Photoshop

HOW TO

unstuck workflow

change model Models behave quite consistently under well engineered prompts. When relevant results are not returned, it is worth trying different model or LORA

Prompt text processing is very complex and carries a lot of keyword correlation, stereotyping, and bias.

The „butterfly effect“ in prompt is in my experience very common.

Following images have same baseprompt, only differ in ordering or parahrasing of keywords in prompt, as shown below. Yet completely different probabilities of results.

island

c

High probability of being generated isolated on an island

change approach to prompt

a

...which follows prompt If model is trained on images of parts of building without text tag specifying this context, negative prompt will be not very helpful. It is much easier to use guidance or simply model, that is trained on more suitable dataset

checkpoint model architectureExterior_v60

checkpoint model darchimaster_v10

The building is captivated by its extensive fluid glass façade, consisting of big glass panes mounted on steel structure. Segments of the rest of the façade are coated with reflective (white metal panels:1.5)

This uncertainty cannot eliminated, but can be worked with by not sticking to single baseprompt and finetuning it when it does not often produce expected results

change in prompt

a

wall

c

Higher probability of being generated in perceived inside, enclosed by walls or wedged into part of the building

Try different base models regularly and in different situations.

There‘s no need to use only one

...which goes against both positive and negative prompt

The building is captivated by its extravagant façade with surfaces of glass and (white matte metal panels:1.5). Glazed areas consist of big glass panes mounted on steel structure, change in prompt

STYLIZATION WITH

checkpoint models and LORAs 0

LORA ··· HouseBlurprint_v01

1

LORA ··· aki

2

LORA ··· archi_draw-v4

3

LORA ··· sandiese

4

LORA ··· xsarchitectural-9Science

5

LORA ··· XSArchi_127 This demonstration shows how huge impact can utilization of custom embeddings in form of LORAs have. Constant prompts with constant settings, only specific LORA is added to prompt. Switching of base models also provided high degree of freedom for experimentation.

a

6

LORA ··· 0513sketch_v2_01

7

LORA ··· organic-v2

8

LORA ··· MIR-v3

9

LORA ··· lwmirXL-V1.0fp16

10

LORA ··· Archiveria_yiu_v30

Don‘t be afraid to explore additional approaches for fine-tuning the image and for the generation itself. Some are listed in delimination

··· 11

MODEL··· anything-v3-fp16-pruned

12

MODEL ··· dreamshaper_8

13 MODEL··· hdarchimaster_v10

14 MODEL··· architectureExterior_v60 15

MODEL··· dvarch

16 MODEL··· SD v1-5-pruned-emaonly

17

MODEL··· Realistic_Vision_V5.1

···

revolution

JUST A BUBBLE OR A

c

The images were not inteded as graphical asset to present the final project. Yet, some of them achieved such unexpected qualities that could potentially be used in this way. You can compare how original concepts contrast with generated images and final renderds (with postproduction) The hypothesis of technological shift is further supported by the time demands between these results.

and beyond

1 minute vs. hours of potential concept modelling...

With SD over 800 unique prototype iterations during 20 hours including adjusting and experimenting.

of the future

fast?

HOW FAST IS

b

TO INFINITY

1

graphics for final project 2 30-60 hours for the final model, rendering and edits

use cases

0

original input 2-8 hours for potential model

POTENTIAL

a

selected generated image 1 minute

stay stay visionary visionary

ECHOES

conclusion conclusion

The artificial intelligence proved as an extremely powerful tool for quickly shaping and prototyping ideas. There is great space to utilizing these tools in professional world thanks to their flexibility and speed. It is very relevant experimenting with them and implementing them into existing creative prototyping process. I was able to achieve various levels of control over preserved and developed elements which were unfornutately sometimes correlated with decreased realism of outputted image. The concepts did share different developments, some stayed true to the last extent, others kept only significant aspects.

Generative design has in my opinion very little to no bounds. It is based on principle how our brains work - forming and morphing stream of intangible noise into basic shapes and further enhancing them based on conditions. It works by utilizing a generative process - which comes from empiri in terms of human brains, learnt connections from datasets for genAI or from algorithms and parameters for generative design. Amplification of nuance often finds its path to final design, at least in my experience.

And genAI is basically built on this principle.

When we develop generative network complex enough to perceive not just the image, but also it‘s context and conditions. It might be able become the design tool of future - a connection of parametric conditional (generative) design and genAI. It will not only look into a noise and try to see something in it. It will be able to also perceive relations, interlock segments and start generating around in them. Iteratively navigating through solutions, nuancing their aspects, evaluating their relevance and step by step guiding itself towards working solution... Similarly to what we do... Only in a fraction of time.