A workflow for image-to-3D generation and multi-objective optimisation of generated 3D models

Journal of Computational Design & Engineering

accepted version

Title A multi-modal workflow for image-to-3D generation and multi-objective optimisation of generated 3D models Authors (1,2) Kapsalis, E.; (1) Jaeger, N. (1): University of Nottingham, UK (2): University of Derby, UK

0. Abstract Form-finding and optimisation tools enable architects and engineers to draw inspiration from precedent projects and generate a multitude of design alternatives, based on certain criteria, during the early stages of design. Despite the recent advancements in the fields of generative AI and multi-objective evolutionary computation, there remains significant potential for further development and integration of these techniques in the design exploration process. In this paper, we propose a new technique of generating 3D models from 2D images or technical drawings, and subsequently optimising their design based on sets of multiple objectives. We also present a case study to demonstrate the practical applicability and effectiveness of the proposed workflow. Results from the case study indicate the usability of our approach in generating a diverse set of design alternatives, optimising them based on multiple criteria, and ultimately selecting a design solution that is well-aligned with project goals and constraints. We conclude by discussing the practical, technical, and ethical implications of our approach.

Keywords: generative AI, image-to-3D, evolutionary algorithms, multi-objective optimisation, design space exploration, form-finding

Journal of Computational Design & Engineering

accepted version

1. Introduction Design exploration encompasses the intricate process of form-finding, wherein architects and engineers utilise computational methods to generate a plethora of 3D models, allowing for the discovery of novel and innovative solutions (Navinchandra & Riitahuhta, 2012). Often drawing inspiration from precedent projects, this approach facilitates the identification of design alternatives that satisfy functional, aesthetic, and performance criteria (ibid.). Design exploration also includes the optimisation of structural forms, fostering an iterative refinement process that ultimately yields more efficient, usable, and impactful designs (Martins & Ning, 2021).

Advanced computational tools and 3D modelling software (e.g., Building Information Modelling – BIM) enable architects and engineers to effectively navigate the design exploration process by generating, analysing, and evaluating numerous design alternatives in a streamlined and convincing manner (Navinchandra & Riitahuhta, 2012). These sophisticated technologies facilitate the integration of multi-disciplinary performance criteria, such as structural integrity and contextual requirements, within the form-finding process. As a result, architects and engineers can draw inspiration from precedent projects and subsequently optimise design solutions, drawing, while simultaneously addressing complex form-finding challenges in a responsive way.

1.1. Generative AI for 3D model creation Generative AI refers to a subset of artificial intelligence techniques that focus on creating new content, such as images, text, audio, or video, by learning patterns and structures present in existing data (Foster, 2019). The goal of generative AI is to generate novel outputs that are both realistic and highly similar to the training data, while not being exact replicas of any specific examples from the dataset (Sikka, 2021). Some of the most popular generative AI models include Generative Adversarial Networks (GANs) (Goodfellow et al., 2016), Variational Autoencoders (VAE) (Kingman & Welling, 2019), and diffusions models (Tomczak, 2022).

By leveraging similar models, generative techniques can effectively analyse and process visual data to create realistic and accurate 3D models for a wide range of applications, including architecture and structural engineering. Several prior works have explored 3D model generation over a number of different techniques.

Specifically, a growing body of work has aimed to train 3D-aware Generative

Journal of Computational Design & Engineering

accepted version

Adversarial Networks (GANs) from datasets of 2D images (Schwarz et al., 2020; Chan et al., 2021; Or-El et al., 2021; Gu et al., 2021). These GANs are typically applied to the problem of novel view synthesis in forward-facing scenes, and do not attempt to reconstruct full 360-degree views of objects. In a recent study, Gao et al.

(2022) have developed a GAN that is capable of generating complete 3D meshes.

The GAN is paired with a discriminator that utilises differentially-rendered views of the generated meshes, as proposed by Laine et al. (2020). Elsewhere, Bautista et al.

(2022) have produced comprehensive three-dimensional (3D) scenes through a twostep process. Firstly, they acquired a representation space that decodes into Neural Radiance Fields (NeRFs), as proposed by Mildenhall et al. (2020). Subsequently, they trained a diffusion prior on this representation space. Overall, generative techniques, including 3D-aware GANs and NeRFs, enable the generation of realistic and accurate 3D models for various applications, with recent advancements focusing on complete 360-degree views and comprehensive 3D scenes.

In the fields of architecture and structural engineering, generative AI has recently begun to gain traction with respect to reconstructing 3D models (Shrestha et al., 2018; Ji et al., 2019). Specifically, generative techniques have been used to create accurate 3D models of buildings and structures from 2D images, which can assist with building information modelling (BIM) tasks, virtual reality simulations, or retrofitting projects. In this context, the vast majority of research efforts conducted have focused on generating 3D building models from 2D floorplan drawings using various techniques, such as Convolutional Neural Networks (Jang et al., 2020), GANs (Dong et al., 2021), Graph Neural Networks (Vidanapathirana et al., 2021), or a combination of rule-based and data-based methods (Park & Kim, 2021). An interesting work by Justo et al. (2021) who extracted point cloud data from scanned images – i.e., discrete sets of data points in space that represent 3D models – and then used a supervised learning algorithm to visualise road infrastructure assets in BIM. In a similar fashion, Poku-Agyemang and Reiterer (2023) converted 2D technical plans of bridges into 3D objects through point cloud data extraction, employing a set of supervised learning algorithms to estimate discrete parts of input drawings, such as lines and corners. Despite these advancements, generation of 3D models from 2D images has not yet reached its full potential with respect to the fields of architecture and structural engineering, indicating ample room for further

Journal of Computational Design & Engineering

accepted version

innovation and improvement in utilising generative AI for architectural or engineering purposes.

1.2. Evolutionary computation for multi-objective design optimisation Evolutionary computation is a sub-field of AI and involves the use of algorithms that can evolve with the availability of more data and experience (Baeck, 1997; Fogel, 2000). Computational models using evolutionary algorithms can program themselves by taking inspiration from Darwinian evolution theory and simulating biological breeding (Slowik, 2020). Evolutionary computation has been used to solve problems that have too many variables for traditional algorithms, such as optimisation problems (Chanal, 2021). In an optimisation context, computation begins with a possible set of solutions to a problem, then evaluates and modifies the possible set to arrive at the most suitable output (Deb, 2001). Over recent years, several approaches have been tested and applied to evolutionary computation, such as the genetic algorithms, genetic programming, differential evolution, evolution strategy, and evolutionary programming (Sen, 2015; Slowik, 2020).

Genetic algorithms were firstly developed by Holland (1992) have since been tried on various optimization problems with a high degree of success (Norouzi, 2014; Yongbin, 2021; Sang, 2021). In genetic algorithms, the evolution usually starts from a population of randomly generated individuals, and is an iterative process, with the population in each iteration called a generation (McCall, 2015). In each generation, the fitness of every individual in the population is evaluated; the fitness is usually the value of the objective criterion in the optimisation problem being solved (ibid). The new generation of candidate solutions is then used in the next iteration of the algorithm (Mitchell, 1998). In this way, the process keeps “evolving” better solutions over generations, till it reaches a stopping condition (ibid.). Genetic algorithms often handle multi-objective optimisation problems, which involve finding solutions that satisfy multiple, often conflicting, objectives simultaneously. In such cases, the aim is to find a set of solutions that represent trade-offs between the conflicting objectives (Coello et al., 2007). These trade-off solutions are known as Pareto-optimal solutions, and the set of all such solutions is called the Pareto-optimal front (ibid.). A Pareto-optimal solution is one where it is not possible to improve one objective without deteriorating at least one other objective (Mirjalili & Dong, 2019). In other

Journal of Computational Design & Engineering

accepted version

words, no other solution in the search space can dominate a Pareto-optimal solution.

A solution dominates another if it is better or equal in all objectives and strictly better in at least one objective (ibid.). Therefore, multi-objective optimisation has widespread applications in various fields, including engineering design, finance, logistics, energy management, and environmental management, where decisionmakers need to balance multiple conflicting objectives.

In architectural and structural engineering, multi-objective optimisation plays a crucial role in finding optimal solutions for designing buildings and infrastructure that cater to multiple, often conflicting, criteria, such as cost minimisation, performance maximisation, and environmental impact reduction. A significant amount of research has centred around designing building structures that balance multiple objectives, such as energy efficiency, occupant comfort, aesthetics, and cost, through utilising genetic algorithms (Showkatbakhsh & Kaviani, 2020; Chi et al., 2021; van Ameijde et al., 2022). Another research strand has utilised genetic algorithms during early stages of design in order to consider multiple objectives with respect to form-finding for architectural or urban design tasks, such as maximizing usable space, minimizing circulation paths, and adhering to various regulations and guidelines (Cheung et al., 2022; Lima et al., 2022; Makki et al., 2022). Other researchers have focused on topology optimisation, where their goal has been to generate the most efficient layout of material in structures while considering multiple objectives, such as minimising weight and maximising structural performance, e.g., stiffness or strength (García Menéndez et al., 2021; Mostafavi et al., 2020). Overall, by using multi-objective optimisation techniques, architectural and structural engineers can effectively explore a wide range of design alternatives, identify trade-offs between conflicting objectives, and ultimately select the most suitable solutions for their specific projects.

1.3. Purpose The purpose of this paper is to propose a novel workflow for the early stages of architectural or structural engineering processes. The workflow presents a new technique of generating 3D models from 2D images or technical drawings, and subsequently optimising their design based on sets of multiple criteria. The paper also includes a case study to demonstrate the applicability of the proposed workflow.

Journal of Computational Design & Engineering

accepted version

The significance of this work pertains to two parameters of design space exploration, i.e., inspiration from precedents and multi-objective optimisation, which are crucial for the early stages of architectural or structural engineering processes. Specifically, generating 3D models from 2D technical drawings or images representing precedent projects is conducive for fostering design inspiration. By transforming 2D objects into 3D models, architects and engineers can better visualise and understand the spatial relationships, forms, and details of precedent projects. Vectorising raster graphics from precedents is likely to enable them to seamlessly integrate previous designs into their own BIM workflows, by converting pictorial information into manipulatable data, based on which numerous new design options can be developed. Drawing on this, multi-objective optimisation allows for more thorough evaluation of different design options by considering various aspects, such as structural performance, aesthetics, cost, and energy efficiency. This approach contributes a deeper understanding of the relationships between design criteria and helps in selecting a design solution that best aligns with project goals and constraints.

2. Methods used The proposed workflow consists of two parts – i.e., (a) generation of 3D models from 2D images, and (b) multi-objective optimisation – for which different architectures have been utilised. Specifically, we built on the pre-trained Point-E diffusion system for 3D model synthesis (Nichol et al., 2022) to implement the first part, and utilised the Wallacei multi-objective evolutionary optimisation engine (Makki et al., 2022) to develop the second part. The following lines describe the main architectures used.

2.1. Point-E for 3D model synthesis The Point-E system was recently developed as a 3D model generation method (Nichol et al., 2022). Point-E was chosen as it is capable of efficiently producing diverse and complex 3D shapes conditioned on 2D images. The system samples from some distribution q(x0) using a neural network approximation pθ(x0). This is generally based on the Gaussian diffusion setup of Ho et al. (2020), through which a noising process is defined as in Equation 1: (1) ,

Journal of Computational Design & Engineering

accepted version

for integer timesteps t ∈ [0,T]. The system is pre-trained on a dataset of several

million 3D models and associated metadata by approximating q(xt−1|xt) as a neural network pθ(xt−1|xt). In this study, we make use of three publicly available imageconditioned models, specifically the 40-million (40M), 300-million (300M), and 1billion (1B) parameter models.

Point-E first breeds a single synthetic view of an input image using a diffusion-based model. To generate text-conditional synthetic views, we use a 3billion parameter GLIDE model (Nichol et al., 2021) fine-tuned on rendered 3D models from the trained dataset. After that, the system produces a coarse point cloud (1,024 points) conditioned on the synthetic view. To generate the low-resolution point clouds, the authors used a conditional, permutation invariant diffusion model (Nichol et al., 2022). Finally, the system generates a fine point cloud (4,096 points) conditioned on the low-resolution point cloud and the synthetic view. To up-sample these lowresolution point clouds, the authors use a similar (but smaller) diffusion model which is additionally conditioned on the low-resolution point cloud (ibid.). The system is also capable of producing meshes from the generated point cloud using a regression-based approach (Lorensen & Cline, 1987). Overall, this method can generate 3D mesh models from 2D images or technical drawings efficiently and in a timely manner.

2.2. Wallacei optimisation engine Wallacei is an engine that allows users to run evolutionary simulations in Grasshopper (Makki et al., 2022). It was chosen because it offers various options while setting up the design problem, analysing the outputted results, and selecting solutions. Wallacei employs the NSGA-2 genetic algorithm (Deb et al. 2002), as its structural evolutionary component. The main advantage of the NSGA-2 algorithms lays on its ability to emphasise non-dominated solutions using an elitist principle while preserving diversity – i.e., the elites of a population are carried to the next generation (ibid.).

The Wallacei engine requires three sets of inputs: an initial 3D model or set of models (Phenotype), the design variables (Genes) and the fitness objectives (FO).

The phenotype serves as the starting point for the optimisation process. It defines the geometry, topology, and other physical properties of the design. They are the

Journal of Computational Design & Engineering

accepted version

inputs that determine the variations in the phenotype. The genes can be continuous or discrete values, and they can be either independent or dependent on other variables. Examples of genes could be the length of a beam, the height of a building, or the thickness of a shell. The fitness objectives are the performance metrics that are used to evaluate the quality of the design solutions generated by the optimisation process. They represent the goals and constraints that the designer wants to achieve or satisfy.

Once the user provides the required input into Wallacei, the engine initiates an evolutionary process to simulate optimal solutions based on the given design variables and fitness objectives. The engine breeds an initial population of design solutions, which are randomly generated based on the given design variables.

Wallacei analyses the results from the simulation in order to select the fittest solutions by evaluating each design solution against the given fitness objectives, and a fitness value is assigned to each solution based on how well it satisfies the objectives. The engine uses a mix of statistical analysis (i.e., mean and standard deviation of fitness values), Pareto analysis (i.e., this is to identify trade-offs between fitness objectives), clustering, and sensitivity analysis to evaluate the fitness values of a single solution or the whole population. Based on the analysis findings, the user selects the fittest solutions - i.e., those that correspond most to the fitness objectives.

The user repeats the simulation process for the selected solutions as many times as needed until a satisfactory solution is found, which would optimise the initial design according to the design intentions, requirements, or constraints.

2.3. Workflow design The proposed workflow draws on the main benefits of the architectures described in sections 2.1 and 2.2. Specifically, it harnesses the robustness of the Point-E system to allow for 3D model synthesis from pictorial 2D data. It also utilises the Wallacei engine to optimise the generated 3D model against multiple objectives in an evolutionary way.

The process begins by identifying the 2D object for reconstruction. The 2D object can come in any pictorial form (e.g., image or technical drawing), as a representation of architectural or structural engineering projects (e.g., buildings in section or plan views, bridges in axonometric views). Those are likely to originate from relevant

Journal of Computational Design & Engineering

accepted version

databases that include realised or conceptual projects, such as the RIBA pix collection 1. Once the user has identified the 2D object for reconstruction, they insert it into the Point-E system for 3D generation. Building on the Point-E pre-trained system and architecture, which includes the creation of a synthetic view, a coarse point-cloud model, a fine point-cloud model sequentially, the system finally returns two outputs: (a) a 3D point-cloud model and, (b) a 3D mesh model of the inserted 2D object. We have created a user-friendly interface 2 on the Hugging Face datascience platform, where the user only needs to insert the 2D object of interest in pictorial format (e.g., PNG, JPEG files), choose between different training basedmodels and degrees of guidance in the point-cloud synthesis process, and finally retrieve the generated outputs. Alternatively, the user can integrate this part into their own workflows by using the code we provide on GitHub 3.

Once this part is completed, the user should import the mesh model into a CAD/BIM environment for further refinement and optimisation purposes. For this step, we strongly recommend the Rhinoceros 3D modelling software, as this program includes the Grasshopper 3D visual-scripting tool, which is the hosting environment for the Wallacei engine. The user should be now able to parametrise the model in a Grasshopper environment. This step refers to translating the topological properties of the modelled object (e.g., dimensions, compactness, connectedness) into mathematical variables that can be adjusted – through numbers sliders or inputs and Boolean toggles – to control the behaviour or appearance of the object.

Parametrisation enables the user to manipulate the model so that it that can respond to changing requirements or constraints and allows for easy iteration and exploration of design options. This step is vital for integrating the 3D model into the user’s design space and relating it to their requirements or constraints, upon which the optimisation process will be implemented.

In most cases, the model parameters will define the genes required for the Wallacei simulations, though this is subject to the nature of design problem the user is looking to resolve. The user also needs to define the fitness objectives, which will drive the optimisation process. Once those two sets have been defined, the Wallacei engine The collection is available here: https://www.ribapix.com/ The interface is publicly accessible here: https://huggingface.co/spaces/timo-kap/point-e 3 The code is publicly accessible here: https://github.com/timo-kapa/GenAI-MOEO_DesignExploration 1 2

Journal of Computational Design & Engineering

accepted version

will run the evolutionary simulation process, as described in Section 2.2. After analysing the results of the initial simulation, the user can either feed the fittest solutions – according to predefined objectives – into another round of simulations or terminate the process by selecting the solution (or, solutions) that optimise the model against one or many objectives. Figure 1 provides an overview of the proposed workflow and main architectures used.

Figure 1: Proposed workflow and architectures included. Source: the authors.

3. Results To validate the workflow proposed in Section 2, we conducted a case study to apply our approach to a real-world design problem and evaluate its effectiveness. The case study addresses the design of a new access ramp, which is intended to facilitate access of mobility-disabled people to train carriages. The study has been developed in relation to the doctoral research of one of the co-authors, wherein it realises a significant part of the research objectives. The computational setup for the experiment included a high-performance desktop computer with the following specifications: o CPU: Intel Core i9-12900 3.8 GHz, 16-core processor

Journal of Computational Design & Engineering

accepted version

o GPU: NVIDIA GeForce RTX 3080 graphics card, with 12 GB VRAM o RAM: 64 GB DDR5 memory o Storage: 1 TB SSD storage o Operating System: Windows 11 3.1. Design problem In this study, the design problem revolves around the platform train interface of train stations. Specifically, previous work has indicated that the physical gaps between train carriages and platforms can have detrimental effects on quality-of-life aspects of mobility-disabled people, including life-threatening accidents (Kapsalis et al., 2022). Existing solutions, such portable ramps, might be ineffectual in certain cases (e.g., in unmanned stations), thus inhibiting mobility-disabled individuals from accessing trains. For this reason, we have been ideating a robotic ramp, which can structurally adapt to different types of gaps and move along the platform to cater for different access points autonomously. The following lines describe the main body of our design exploration, regarding the form-finding process of the robotic ramp, based on the workflow proposed in Section 2.

3.2. Data collection To draw inspiration for the design ideation stage, we sought for relevant projects in architectural databases and patent libraries. We trained a content-based image retrieval system based on deep neural networks and search over 100,000 technical drawings (Kapsalis et al., 2023). Eventually, 36 drawings were qualified for further investigation after the application of quantitative and qualitative filters. The drawings represented ramps or ramp-type structures in various 2D views (i.e., plan, section, axonometric); Figure 2 illustrates an example of these drawings. Of the 36 drawings, only four would satisfy user requirements or preferences identified in previous ethnographic studies (ibid.). We apply the proposed workflow to generate 3D models from all four drawings and subsequently optimise their design.

Figure 2: Two-dimensional drawing selected for further design investigation.

Journal of Computational Design & Engineering

accepted version

3.3. 3D model synthesis For generating 3D models, we inserted the four images containing 2D technical drawings into the Point-E system and ran 30 iterations apiece. Our intention was to experiment with three base-models, on which Point-E had been trained (see Section 2.1); therefore, we tasked each base-model with 10 iterations per image. For each iteration, Point-E models produced a pair of outputs consisting of a 3D-like pointcloud model and a 3D mesh model. In total, we produced 120 pairs. Figure 3 provides an example set of outputs from this process.

Figure 3: An example from the 3D model generation process - a 2D drawing (a) is fed into Point-E, which produces a point-cloud model (b) and a 3D mesh model (c).

Our findings indicated that the Point-E system frequently generates coherent and superior quality 3D mesh models in response to intricate images, such as technical drawings. However, the point cloud generation occasionally encountered difficulties in comprehending or extrapolating the conditional image, culminating in a shape that diverges from the initial drawing.

Based on the overall results of the 120 generations conducted, we compared PointE’s ability to generate accurate 3D content from conditioned images to other generative architectures, using the COCO evaluation framework. Evaluation results are presented in Table 1.

Method

ViT-B/32

ViT-L/14

Latency

DreamFields

78.6%

82.9%

CLIP-Mesh

67.8%

74.5%

∼200V100-hr

DreamFusion

75.1%

79.7%

∼17V100-min

Point-E (40M)

62.4%

68.2%

1.4 V100-min

∼12V100-hr

Journal of Computational Design & Engineering

accepted version

Point-E (300M)

65.1%

69.7%

1.7 V100-min

Point-E (1B)

70.2%

75.3%

2.1 V100-min

Table 1: Comparison of Point-E to other 3D generative techniques as measured by CLIP R-Precision (with two different CLIP base models) on COCO evaluation prompts.

Although Point-E yielded inferior results in comparison to state-of-the-art methodologies as shown in Table 1, it generated samples in a considerably reduced time span. This characteristic renders this method potentially more feasible for specific applications that require time-efficient workflows. It can also facilitate the identification of superior-quality 3D models by examining numerous objects and selecting the optimal one based on a predetermined heuristic.

Table 1 has also indicated that 3D generations derived from the one-billion (1B) parameter base-model were more accurate in representing the conditioned images.

As such, we omitted generations produced from the other two base-models (i.e., those built on 40M and 300M parameters, respectively) of Point-E for next stages of our study. After removing duplicates – i.e., two or more generated 3D obejcts that referred to the same 2D drawing – and eliminating erroneous interpretations of the input forms, we only kept the 3D objects that that most accurately reconstructed the input form according to our design perception. As such, the final output of this procedure encompassed four 3D mesh objects (Figure 4), each corresponding to the respective conditioned images.

Figure 4: 3D mesh objects selected for further evaluation.

3.4. Evolutionary optimisation We imported the mesh objects generated through Point-E into the Rhinoceros 3D environment – i.e., the CAD software where we conducted the optimisation part of our study. A common challenge for all four objects was that the constituent meshes were produced in a semi-refined way, and thereby they appear either slightly deformed or perforated. This is because Point-E cannot capture fine-grained details (e.g., complex shapes or textures), which are commonly found in engineering

Journal of Computational Design & Engineering

accepted version

imagery or drawings. Using Rhinoceros “Repair Mesh” command, we were able to refine the meshes and fill all holes, in such a way that the 3D objects almost precisely represented the initial 2D drawings.

Another issue concerned the contextualisation of the mesh objects into our design space, as meshes are innately difficult to parametrise, and subsequently link to local design factors (e.g., user needs and spatial constraints). For this reason, we wrote a small script to convert meshes into surface boundary representations (i.e., brep) in Grasshopper, as surfaces are easier to manipulate. Figure 5 visualises the script we wrote in a Grasshopper viewport.

Figure 5: Converting mesh objects to brep. Source: the authors

Once we completed the conversion from meshes to surfaces, we integrated the 3D objects into our design space. To achieve that, we have previously developed a script that returns optimal ramp width and length based on a number of design criteria that influence the design environment, namely platform characteristics (i.e., width, length, height, and distance from rail tracks), train step characteristics (i.e., height, distance from rail tracks), and user requirements (i.e., accessibility gradient).

The developed script, though, does not take into account some other crucial design criteria, such as handrail integration and positioning, ramp movement, and structural deployment of ramp. Since the proposed solution is intended to be structurally adaptable, movable, and responsive to user requirements, we utilised the Wallacei evolutionary engine to complement our script and incorporate the missing criteria into the form-finding process for the robotic ramp. Our intention was to optimise the design of a ramp to accommodate various gaps across the UK rail network, considering the diverse characteristics of trains and platforms.

We linked each of the four objects to Wallacei as distinct phenotypes and carried out separate simulations and analyses for each one. We inputted the same genes and fitness objectives for all simulations, as described in Table 2. The genes referred to design characteristics defining the ramp form as well as train-steps and platforms,

Journal of Computational Design & Engineering

accepted version

while the fitness objectives represented design criteria or user requirements (e.g., maximal access gradient) that we wanted to fulfil. In total, we fed 11 genes and 4 fitness objectives into the engine, resulting in a Z design space. Table 2 provides an outline of the genes and fitness objectives used. To illustrate the optimisation process, the following lines demonstrate the outcomes of the evolutionary simulation conducted for one of the ramp-type objects (Figure 6).

Genes

Fitness objectives (FO)



Platform width (PW)



Platform length (PL)



Platform offset from rail tracks (PO)



Platform height (PH)



Train-step offset from rail tracks (TO)



Train-step height (PH)



Ramp length (RL)



Ramp width (RW)



Handrail type (truss) (HT)



Handrail height (HH)



Deployment angle (DA)



FO1: minimise deployment angle (5o≤ angle (deg.) ≤10o)



FO2: maximise ramp width (length ≤ 1.0 m)



FO3: maximise handrail height (0.70m ≤ handrail height ≤ 0.90m)



FO4: minimise ramp length (length ≤ 2.5m)

Table 2: Genes and fitness objectives for optimisation through the Wallacei engine

Figure 6: Example object selected for evolutionary optimisation.

Journal of Computational Design & Engineering

accepted version

In this task, crossover probability was set at 0.9, mutation probability at 1/n - where n is the number of variables in the design problem (Deb et al. 2002), mutation distribution index at 10, and crossover distribution index at 20. Given the topological range of the evaluated geometries comprising the phenotype, the simulation time per iteration varied substantially, averaged at 32sec in our computing system. Three optimization runs in Wallacei assisted in optimising the process by looking at the following graphs (Makki et al., 2022): 

“Standard Deviation (SD) Graph: The SD curves are plotted for each generation separately from the first (red) to the last (blue). They represent the distribution of the values from the mean. A low SD factor indicates less variation within the population., while a high SD factor indicates more variation.



Mean Trendline Chart: The mean trendline chart presents the mean fitness value for each fitness objective independently for each generation across the entire simulation from start to finish.



Fitness Values (FV) Chart: The FV chart analyses the FVs for each FO independently across the entire population. The aim is to visualize how the solutions are performing in relation to one another.”

The population size was optimised in two separate runs, taking into account the duration of the simulation, the standard deviation trend, and the fitness values. In the first run, a population of 1000, 20 generations, and 50 generation counts resulted in a simulation time of 8 hours and 50 minutes. The second run featured a 200-member population, 10 generations, and 20 generation counts, which reduced the simulation time to 2 hours and 9 minutes. Figure 7 illustrates the Parallel Coordinate Plots for both runs, which analyse each solution in the population by comparing their fitness values across all fitness objectives (FOs), with red representing the first and blue the last.

A consistent pattern emerges for FO1, FO2, and FO3, where stability is achieved after the sixth generation. However, FO4 (ramp length) displays no discernible pattern and varies throughout the generations. The best fit values for FO1 (deployment angle), FO2 (ramp width), and FO3 (handrail height) exhibit an absolute difference of 0.6, 0.04, and 0.03, respectively.

Journal of Computational Design & Engineering

accepted version

Figure 7: Parallel Coordinate Plot for run 1 (R1) and run 2 (R2)

The second experiment focused on optimising fitness objectives (FOs) and determining their priority. A third run followed, which was identical to the second one but excluded FO4 to assess the impact of deployment angle. Fitness values were monitored, and their performance is documented in Figure 8. The average trendlines for FO1, FO2, and FO3 indicate enhanced stability after the second generation and improved fitness values (5.9, 0.92, and 0.85). The last 74 solutions feature geometries with the maximum lengths, implying that largest shapes do not negatively influence the results; however, this would not be preferred due to spatial restrictions in station arrangements.

Figure 8: Comparative graphs for fitness values in run 2 (R2) and run 3 (R3)

The second run was further analysed as it is time-efficient, accurate, and employs the deployment angle criterion to regulate excessive lengths. The objectives are prioritised in the following order: deployment angle (FO1) and ramp width (FO2)

Journal of Computational Design & Engineering

accepted version

receive the highest priority, handrail height (FO3) is secondary, and the ramp length (FO4) is the lowest indicator for selecting the solution to be validated.

Six design solutions are chosen for further examination using Wallacei selection analysis methods, including the best-ranked solutions for FO1, FO2, and FO3, the solution with the best relative difference between fitness ranks, the solution representing the average fitness ranks, and the best average solution from the last generation.

These chosen solutions are plotted on the Parallel Coordinate Plot (PCP) (Figure 9, left), showing a good fit in deployment angle (values under 5.9) and width (values above 0.92), a medium fit in handrail height (values between 0.80 and 0.85), and a medium to low fit in the ramp width criterion (values between 1.96 and 2.5). The entire population is displayed in the objective space (Figure 9, right), with FOs 1, 2, and 3 on the X, Y, and Z axes, respectively, and FO 4 represented by color (Green (fittest) to Red (least fit)). The chosen solutions are highlighted in blue.

Figure 9: Parallel Coordinate Plot with the selected solutions highlighted in black (left). On the right, the objective space of all populations and the selected solutions highlighted in blue.

Figure 10 presents the phenotypes and their diamond fitness charts for all selected solutions, detailing fitness values and rankings for each objective. Diamond fitness charts compare the performance of different fitness objectives for a specific solution; the closer an objective is to the chart's centre, the fitter it is. The evolutionary formfinding process demonstrated significant design variability among the solutions, occasionally with minimal differences in performance results. This multi-objective evaluation did not provide a single "ideal” solution; rather, it offered substantial directions in the early design stages of the robotic ramp.

Journal of Computational Design & Engineering

accepted version

Figure 10: Selected solutions, fitness values, and objective rankings

4. Discussion This paper presented a novel workflow for the early stages of architectural or structural engineering processes. The workflow described a new technique of generating 3D models from 2D images or technical drawings based on Point-E system, and subsequently optimising their design against multiple criteria through the Wallacei engine. The paper also included a case study to demonstrate the applicability of the proposed workflow.

To the best of our knowledge, this is the first effort to incorporate generative, imageto-3D techniques along with evolutionary algorithms into the process of design exploration in an architectural and structural engineering context. Building on this foundation, our work makes several important contributions to the field. Converting 2D raster images or scanned technical drawings into fully-realised, vectorised 3D models vector graphics can spur the design form-finding process. Specifically, it enables architects and engineers to readily explore and manipulate precedent designs in a time-efficient manner, enhancing their understanding of complex spatial relationships, forms, and details that might not be apparent in 2D representations.

Secondly, the employed set of evolutionary algorithms help professionals efficiently navigate the design space. This allows for the simultaneous consideration of multiple design criteria, such as spatial constraints, user needs, and structural parameters, in the early stages of the design process. Our approach not only streamlines decisionmaking but also uncovers valuable insights into the trade-offs and synergies

Journal of Computational Design & Engineering

accepted version

between different design objectives, promoting the development of more informed and balanced solutions. Finally, by combining these novel techniques into a cohesive workflow, we provide a powerful tool for architects and structural engineers, which can be easily incorporated into their own workflows. This integrated approach increases efficiency and fosters creativity by facilitating the seamless integration of knowledge from precedent projects, advanced 3D visualisation, and multi-objective optimisation.

In this paper, we also included a case study to demonstrate the practical applicability and effectiveness of the proposed workflow. This example showcased the usability of our approach in generating a diverse set of design alternatives, optimising them based on multiple criteria, and ultimately selecting a design solution that is wellaligned with project goals and constraints. In the case study presented, our approach demonstrated a strong commitment to inclusive and universal design paradigms by integrating human functioning parameters (e.g., accessible slope and reach range expressed though the handrail height criterion), specifically tailored to mobilitydisabled individuals, as fitness objectives within the evolutionary engine. By prioritising these human-centred criteria during the design optimisation process, we ensure that the resulting solutions are better suited to accommodate the needs of individuals with diverse abilities, thereby promoting accessibility and inclusivity within the built environment.

Our methodology not only advances a more empathetic design approach in the architectural and structural engineering domains but also exemplifies the potential of harnessing AI in an ethical way. By employing evolutionary algorithms in conjunction with human functioning parameters, we effectively align the goals of AI-driven optimisation with the broader social and ethical imperatives of universal design. Our approach highlights the potential for AI to be a powerful catalyst for positive change, fostering the creation of more inclusive spaces that cater to the diverse needs and experiences of all users. By leveraging the power of ethical AI by virtue of humancentred evolutionary algorithms, we set an example for how technology can be responsibly harnessed to create environments that are not only aesthetically pleasing and structurally-robust, but also fundamentally inclusive and accessible to all.

Journal of Computational Design & Engineering

accepted version

5. Limitations and future work Having run some pilot studies, we noticed that some users faced challenges in navigating the image-to-3D part of the workflow when accessing our space on the Hugging Face platform. For this reason, it is required that we refine the user interface, added clear instructions, and incorporated more intuitive controls to facilitate easier manipulation of the images and 3D models.

During the case study, we found that certain aspects of the workflow, such as the image-to-point cloud conversion and the multi-objective simulation processes, were computationally demanding. To address this issue, we plan to introduce parallel processing techniques and streamline the employed architecture, thus significantly reducing the computation time and enhancing the overall efficiency of the workflow.

The case study highlighted instances where the point-to-mesh conversion tasks encountered issues with output data quality or format, leading to errors or unexpected or severely deformed results. It is possible that such discrepancies stem from two underlying factors: (a) the system’s erroneous interpretation of the object forms as portrayed in the image, or (b) the system’s inaccurate inference of a concealed portion of the shape within the image. In any case, we plan to incorporate additional error-handling mechanisms and data validation checks, ensuring more reliable performance and accurate outcomes in the future.

Most importantly, the case study helped us detect a few flaws in the developed workflow and refine it accordingly, ultimately resulting in a more robust and effective solution. Some of the identified shortcomings include the retrieval of semi-refined meshes when converted from point-cloud formats and the challenging parametrisation of meshes. By using a common Rhinoceros 3D command and developing a small script, we managed to overcome the respective flaws and improve the workflow. As such, we provide an updated version of the proposed workflow in Figure 11.

Journal of Computational Design & Engineering

accepted version

Figure 11: Updated version of the proposed workflow after refinements.

References Bautista, M. A., Guo, P., Abnar, S., Talbott, W., Toshev, A., Chen, Z., Dinh, L., Zhai, S., Goh, H., Ulbricht, D., Dehghan, A., & Susskind, J. (2022). GAUDI: A Neural Architect for Immersive 3D Scene Generation. ArXiv. Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., Shalini De Mello, Gallo, O., Leonidas Guibas, Tremblay, J., Khamis, S., Karras, T., & Wetzstein, G. (2021). Efficient geometry-aware 3D generative adversarial networks.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June, 16102–16112. https://doi.org/10.1109/CVPR52688.2022.01565 Cheung, L., Xu, Z., Chen, P., & Makki, M. (2022, April). AN ALTERNATIVE MODEL FOR URBAN RENEWAL: A GENERATIVE APPROACH TO THE (RE)DEVELOPMENT OF XIAN VILLAGE. Proceedings of the 27th International Conference of the Association for Computer- Aided Architectural Design Research in Asia (CAADRIA) 2022. https://opus.lib.uts.edu.au/handle/10453/158256 Chi, D. A., M. Edwin González, Valdivia, R., & J. Eduardo Gutiérrez. (2021).

Parametric design and comfort optimization of dynamic shading structures.

Sustainability 2021, Vol. 13, Page 7670, 13(14), 7670. https://doi.org/10.3390/SU13147670

Journal of Computational Design & Engineering

accepted version

Coello, C. C., Lamont, G. B., & Veldhuizen, van. (2007). Evolutionary algorithms for solving multi-objective problems. Springer US. https://books.google.co.uk/books?id=2murCij_wHcC Dong, S., Wang, W., Li, W., & Zou, K. (2021). Vectorization of floor plans based on EdgeGAN. Information, 12(5), 206.

Foster, D. (2019). Generative deep learning: Teaching machines to paint, write, compose, and play. O’Reilly Media. https://books.google.co.uk/books?id=RqegDwAAQBAJ Gao, J., Shen, T., Wang, Z., Chen, W., Yin, K., Li, D., Or Litany, Zan Gojcic, & Fidler, S. (2022). GET3D: A generative model of high quality 3D textured shapes learned from images.

Gu, J., Liu, L., Wang, P., & Theobalt, C. (2021). StyleNeRF: A style-based 3D-Aware generator for high-resolution image synthesis.

Ho, J., Jain, A., & Pieter Abbeel. (2020). Denoising diffusion probabilistic models.

Advances in Neural Information Processing Systems, 2020-December. https://arxiv.org/abs/2006.11239v2 Jang, H., Yu, K., & Yang, J. (2020). Indoor reconstruction from floorplan images with a deep learning approach. ISPRS International Journal of Geo-Information, 9(2), 65.

Jeroen van Ameijde, Chun Yu Ma, Goepel, G., Kirsten, C., & Wong, J. (2022). Datadriven placemaking: Public space canopy design through multi-objective optimisation considering shading, structural and social performance. Frontiers of Architectural Research, 11(2), 308–323. https://doi.org/10.1016/J.FOAR.2021.10.007 Ji, S., Wei, S., & Lu, M. (2019). A scale robust convolutional neural network for automatic building extraction from aerial and satellite imagery. International Journal of Remote Sensing, 40(9), 3308–3322.

Justo, A., Soilán, M., Sánchez-Rodríguez, A., & Riveiro, B. (2021). Scan-to-BIM for the infrastructure domain: Generation of IFC-compliant models of road infrastructure assets and semantics using 3D point cloud data. Automation in Construction, 127, 103703. https://doi.org/10.1016/J.AUTCON.2021.103703 Kapsalis, E., Jaeger, N., & Hale, J. (2022). Disabled-by-design: effects of inaccessible urban public spaces on users of mobility assistive devices–a systematic review. Disability and Rehabilitation: Assistive Technology. https://doi.org/10.1080/17483107.2022.2111723

Journal of Computational Design & Engineering

accepted version

Kapsalis, E., & Jaeger, N., (2023). Disabled-by-design: effects of inaccessible urban public spaces on users of mobility assistive devices–a systematic review.

Disability and Rehabilitation: Assistive Technology.

Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders.

Now Publishers. https://books.google.co.uk/books?id=pLX0ywEACAAJ Laine, S., Janne Hellsten, Karras, T., Seol, Y., Lehtinen, J., & Timo Aila. (2020).

Modular primitives for high-performance differentiable rendering.

Lima, F., Brown, N., & Duarte, J. (2021, March). Urban design optimization: generative approaches towards urban fabrics with improved transit accessibility and walkability. Caadria. https://doi.org/10.52842/conf.caadria.2021.2.719 Lorensen, W. E., & Cline, H. E. (1987). Marching cubes: A high resolution 3D surface construction algorithm. ACM Siggraph Computer Graphics, 21(4), 163–169.

Lucas García Menéndez, Mar, del, & Manuel Domínguez Somonte. (2021). Design of a cylindrical container using the Wallacei evolutionary engine. Técnica Industrial, ISSN 0040-1838, No 328, 2021, Págs. 38-43, 328, 38–43. https://doi.org/10.23800/10502 Madhawa Vidanapathirana, Wu, Q., Furukawa, Y., Chang, A. X., & Savva, M. (2021). Plan2Scene: Converting floorplans to 3D scenes. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 10728–10737. https://doi.org/10.1109/CVPR46437.2021.01059 Makki, M., Navarro-Mateu, D., & Milad Showkatbakhsh. (2022a). Decoding the architectural genome: Multi-objective evolutionary algorithms in design.

Https://Doi.org/10.1080/24751448.2022.2040305, 6(1), 68–79. https://doi.org/10.1080/24751448.2022.2040305 Makki, M., Navarro-Mateu, D., & Milad Showkatbakhsh. (2022b). Decoding the architectural genome: Multi-objective evolutionary algorithms in design.

Https://Doi.org/10.1080/24751448.2022.2040305, 6(1), 68–79. https://doi.org/10.1080/24751448.2022.2040305 Miguel Angel Bautista, Guo, P., Abnar, S., Talbott, W., Toshev, A., Chen, Z., Dinh, L., Zhai, S., Goh, H., Ulbricht, D., Dehghan, A., & Susskind, J. (2022). GAUDI: A neural architect for immersive 3D scene generation. In ArXiv. Milad Showkatbakhsh, & Kaviani, S. (2020). Homeostatic generative design process: Emergence of the adaptive architectural form and skin to excessive solar radiation. Https://Doi.org/10.1177/1478077120951947, 19(3), 315–330. https://doi.org/10.1177/1478077120951947

Journal of Computational Design & Engineering

accepted version

Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ravi Ramamoorthi, & Ng, R. (2020). NeRF: Representing scenes as neural radiance fields for view synthesis.

Mostafavi, S., Kastrati, V., Badr, H., & Shazwan Mazlan. (2020). Design computation to robotic production methods for reciprocal tessellation of free-from timber structures - design, production, and assembly of 100 years bauhaus wood pavilion. Anthropologic: Architecture and Fabrication in the Cognitive Age Proceedings of the 38th ECAADe Conference - Volume 2, 2, 413–421. https://doi.org/10.52842/CONF.ECAADE.2020.2.413 Nichol, A., Jun, H., Prafulla Dhariwal, Mishkin, P., & Chen, M. (2022, December).

Point-e: A system for generating 3D point clouds from complex prompts. https://arxiv.org/abs/2212.08751v1 Nichol, A., Prafulla Dhariwal, Ramesh, A., Pranav Shyam, Mishkin, P., McGrew, B., Ilya Sutskever, & Chen, M. (2021, December). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. https://arxiv.org/abs/2112.10741v3 Or-El, R., Luo, X., Shan, M., Shechtman, E., Jeong Joon Park, & KemelmacherShlizerman, I. (2021). StyleSDF: High-resolution 3D-Consistent image and geometry generation.

Park, S., & Kim, H. (2021). 3DPlanNet: Generating 3D models from 2D floor plan images using ensemble methods. Electronics, 10(22), 2729. https://doi.org/10.3390/electronics10222729 Prince, S. J. D. (2012). Computer vision: Models, learning, and inference. Cambridge University Press. https://books.google.co.uk/books?id=PmrICLzHutgC S Mirjalili, & Dong, J. S. (2019). Multi-objective optimization using artificial intelligence techniques. Springer International Publishing. https://books.google.co.uk/books?id=wAilDwAAQBAJ Schwarz, K., Liao, Y., Niemeyer, M., & Geiger, A. (2020). GRAF: Generative radiance fields for 3D-Aware image synthesis. In H. Larochelle, M Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems (Vol. 33, pp. 20154–20166). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2020/file/e92e1b476bb5262d 793fd40931e0ed53-Paper.pdf Shrestha, S., & Vanneschi, L. (2018). Improved fully convolutional network with conditional random fields for building extraction. Remote Sensing, 10(7), 1135. https://doi.org/10.3390/rs10071135

Journal of Computational Design & Engineering

accepted version

Sikka, B. (2021). Elements of deep learning for computer vision: Explore deep neural network architectures, PyTorch, object detection algorithms, and computer vision applications for python coders (english edition). BPB Publications. https://books.google.co.uk/books?id=n8U0EAAAQBAJ Spencer, B. F., Vedhus Hoskere, & Yasutaka Narazaki. (2019). Advances in computer vision-based civil infrastructure inspection and monitoring.

Engineering, 5(2), 199–222. https://doi.org/10.1016/J.ENG.2018.11.030 Starzyńska-Grześ, M. B., Roussel, R., Jacoby, S., & Asadipour, A. (2022). Computer vision-based analysis of buildings and built environments: A systematic review of current approaches. ACM Computing Surveys. https://doi.org/10.1145/3578552