Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Material Suggestion & Optimization with Biocomputation A Project for Intelligent Material Selection in Architecture
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Introduction
This project aims to redefine the way architects select and optimize building materials by leveraging advanced biocomputational techniques. The goal is to merge sustainability, aesthetics, and high performance to offer a data-driven approach that not only enhances the design process but also respects environmental contexts. Throughout this presentation, we will explore the target users, the underlying workflow, and the tools we've used to achieve preliminary results. We will also discuss our progress, the advantages of our local approach, and the steps left to complete this journey.
Cornell University| Design Tech
2024
Project Overview The ArchitectureRealMix v1.1 model specializes in generating realistic architectural designs, encompassing various scenarios such as buildings, landscapes, and interior designs
Input Parameters • Architectural Images & Natural Patterns: Foundational dataset for material generation. • Design Intent: Keywords such as 'transparency,' 'natural,' or 'biomorphic aesthetic.' • Architectural Context: Building type, scale, and design phase guide selection.
Models • • • • • •
Architecturerealmix_V1repair.safetenso Dreamshaper_8.safensors Canny model_controlnet Depthanything_V2 Lineart Ultimate SD Upscale_Esrgan/4x-UltraScape
https://civitai.com/models/4384/dreamshaper
DreamShaper 8 offers versatility in producing both photorealistic and artistic images, making it suitable for diverse applications within architectural visualization. To guide the image generation process, the ControlNet model, particularly with the Canny edge detection preprocessor, is utilized. This integration allows for precise control over image structures by extracting edge information from input images The Depth Anything v2 model contributes depth information, enhancing the three-dimensional realism of the generated images. The Lineart model further assists in defining clear structural outlines, facilitating the creation of detailed and accurate architectural visualizations.
https://civitai.com/models/84958/architecturerealmix
https://github.com/Mikubill/sd-webui-controlnet/issues/43
Cornell University| Design Tech
2024
AI Models Architecturereal Canny model_controlnet mix_V1repair
Depthanything_V2
Ultimate SD
The project utilizes a variety of advanced AI models and tools designed to enhance the architectural visualization process. These models focus on generating high-quality imagery, controlling structure, adding depth, and refining image clarity to better communicate design intent. By integrating these AI capabilities, the workflow aims to produce adaptive, contextaware architectural outputs that blend artistic expression with technical precision.
Upscale_Esrgan/4x-UltraScape
Dreamshaper
Lineart
Target Users The tool aims to assist architects in optimizing material
selection by providing data-driven, adaptive material suggestions that improve efficiency and enhance the aesthetic quality of their projects. By leveraging advanced AI models, the tool integrates natureinspired designs with cutting-edge computational methods to better meet the needs of forward-thinking architectural practices.
Architects, particularly those focused on: • Sustainability • Responsive Design • Biophilic Design Data-driven material suggestions to improve efficiency, sustainability, and aesthetics.
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
1.Data Preprocessing: 1. The process begins with the AIO_Preprocessor to prepare images for further use. The input images consist of architectural textures and patterns found in nature. These images are essential as they provide the foundational data used to inspire and generate new material textures. 2. The Load Image Batch node is used to batch-load architectural and nature-based textures from a specified path. This step ensures a variety of images are available for further generation and analysis. 2.Model Loading: 1. Multiple models are loaded into the workflow to perform specific tasks. The Dreamshaper_8 model is utilized for generating material visuals, while the Architecturerealmix v1.1 model is used for producing stylistic variations of architectural elements. These models have been pre-trained and are loaded using the CheckpointLoaderSimple. 2. ControlNet models such as canny and lineart are employed to extract structural features from the input images, enabling more precise control during the image generation process. 3.Image Processing and Conditioning: 1. ControlNetLoader modules are loaded to control the generation process by providing specific image structures as guidance. The Canny Edge Detection is applied to extract outlines, while DepthAnythingV2Preprocessor captures depth information from the images, adding a layer of spatial realism to the visualizations. 2. These models help in effectively conditioning the generation process, ensuring that the output maintains a connection to both architectural geometry and natural textures.
Project Execution Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
1.Image Generation: 1. KSampler is used to sample generated content, while the VAEDecode node decodes the latent space into visually interpretable images. These nodes are responsible for converting the processed and conditioned data into architectural images that incorporate the input features. 2. The IPAdapterAdvanced node also facilitates integration and style adaptation to bring together architectural textures and natural forms cohesively. 2.Image Enhancement and Upscaling: 1. After generation, the images are further enhanced using the Ultimate SD Upscale model. This module upscales the generated images, improving their resolution while preserving important details like textures and outlines. 2. This step is critical for preparing images for 3D reconstruction, ensuring that the level of detail is sufficient for realistic modeling of building facades. 3.Output and UI Development: 1. The generated images are then used to create multiple façade representations for architectural analysis. The workflow also involves fine-tuning the ControlNet models to better outline the geometry, ensuring the images capture detailed features. 2. Finally, a user interface is being developed to facilitate interaction with these generative tools, allowing architects to intuitively generate and modify material suggestions based on their design needs.
Project Execution Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Key Components Stage
Description
Tools/Models Used
Data Preprocessing
Load and prepare architectural and natureAIO_Preprocessor, Load Image Batch based images Load pre-trained models for generation
Dreamshaper_8, Architecturerealmix, ControlNetLoader
Image Processing and Conditioning
Extract features for image guidance
ControlNetLoader, DepthAnythingV2Preprocessor
Image Generation
Generate new material visuals using pretrained models
KSampler, VAEDecode, IPAdapterAdvanced
Image Enhancement and Upscaling
Enhance and upscale generated images
Ultimate SD Upscale
Output and UI Development
Create façades and develop UI for tool usability
ControlNet fine-tuning, UI Development
Model Loading
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Prompt conditioning
Load image
Model Loading
Data Preprocessing
IP Adapter, Style transfer
Control Net
Image Processing and Conditioning
Cornell University| Design Tech
Image Enhancement and Upscaling
Appiah Acheampong Emmanuel
3d Reconstruction
2024
Input :Webcam pictures| Output: Elevation views of the object
Input :Loading interactive Video| Output: Elevation views of the object
Input : Elevation views of the object | Output: Rescale image for latent space
Input :Dataset from directory | Output: Loaded output images
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Input :Text Prompt| Output: Prompt from Chatgpt API
Workflow Cornell University| Design Tech
Input :Loading the checkpoint and IP adapter| Output: Style transfer from the Latent Image
Appiah Acheampong Emmanuel
2024
Input :Removed background image | process: 3d generation |Output : 3d form with textures
Input :Rescale image Process: Control Net Output: Geometry tracing and depth extraction
Workflow
Input :Processed image from K sampler | process: Upscaling |Output : upscaled image
Cornell University| Design Tech
Input :Processed image from K sampler | process: Background removal |Output : Image without background
Appiah Acheampong Emmanuel
2024
UI Interface
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Outputs
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Outputs
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Outputs
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Outputs
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Outputs
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Preliminary Results Material Suggestions: Generated a range of material options combining architectural & natural inspirations. Visualizations show dynamic, adaptive materials based on context. Interface Implementation: Initial interface developed using ComfyUI to present material options.
Dataset Input
interconnected architecture, showcasing how modules like preprocessors, generative models, and post-processing nodes interact to produce high-quality outputs. Key nodes include Canny edge detection for structural refinement,
Style Transfer
ComfyUI workflow illustrates the
Input Image
Final Results
Bio-Tile
Depth-AnythingV2 for spatial mapping, IPAdapter for bio-material style blending, enhancement. Each node's function is seamlessly linked to others, reflecting a well-structured and efficient pipeline.
RemGB
and Ultimate SD Upscale for resolution
3D Obj File
This tool aims to empower architects with adaptive, data-rich material options. Next Steps: - Focus on refining optimization and finalizing the user interface. The focus is on refining optimization processes and completing the interface.
Summary ComfyUI: comfyanonymous (2023) ComfyUI. Available at: https://github.com/comfyanonymous/ComfyUI (Accessed: 1 December 2024). ControlNet Models:
Mikubill (2023) sd-webui-controlnet. Available at: https://github.com/Mikubill/sd-webui-controlnet (Accessed: 1 December 2024). Depth-Anything-V2:
References
DepthAnything (2023) Depth-Anything-V2. Available at: https://github.com/DepthAnything/Depth-Anything-V2 (Accessed: 1 December 2024). Lineart Model: Civitai (2023) ControlNet Pre-Trained Models. Available at: https://civitai.com/models/9251/controlnet-pre-trained-models (Accessed: 1 December 2024). Ultimate SD Upscale Extension: atdigit (2023) ComfyUI_Ultimate_SD_Upscale. Available at: https://github.com/atdigit/ComfyUI_Ultimate_SD_Upscale (Accessed: 1 December 2024). AI-Driven Architectural Design: Parametric Architecture (2023) 'Taking Control 4.0: ControlNet x ComfyUI in Architecture', Parametric Architecture. Available at: https://parametric-architecture.com/taking-control-4-0-controlnet-x-comfyui-in-architecture-studio-carlos-banon/ (Accessed: 1 December 2024). 3D Reconstruction and Facade Generation: Ma, H. (2023) 'Text Semantics to Image Generation: A method of building facades design based on Stable Diffusion model', arXiv. Available at: https://arxiv.org/abs/2303.12755 (Accessed: 1 December 2024).
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024
Results :Video Video demonstrating the use of the Workflow
Results :Video Video demonstrating the use of the Workflow
Thank You
Cornell University| Design Tech
Appiah Acheampong Emmanuel
2024