MINGYANG SUN Selected Works 2019-2023
CONTENTS
01
URBAN ECHOES AI-Guided Exploration of Istanbul’s Cityscape Similarities
02
COGNITIVE LANDSCAPES Exploring Emotional Responses in VR-EEG Driven Art Therapy
03
RECITYGEN Interactive and Generative Participatory Urban Design Tool
04
SECTIONAL SAMPLING Controlled Generation of High-Resolution Architecture Design
05
SPATIAL SYMMETRY Algorithmic Exploration of Calabi-Yau Forms
06
INCLUSIVE AND CONNECTED LANDSCAPES For Self-Constructed Settlements in Quito, Ecuador
07
OTHER WORKS 3D Scan, Speculative Design, GIS Analysis, Community Engagement
2
URBAN ECHOES AI-Guided Exploration of Istanbul’s Cityscape Similarities Independent Research | Suzhou, China | Summer, 2023 Tutor: Han Tu My Role: 80% Concept Design, 30% Machine Learning, 90% Drawing Team: Yaluo Wang, Lingkai Wang, Shiqi Dai Confronting the challenge of understanding urban complexity, the Urban Echos project in Istanbul set out to reveal the historical connections of the city with Athens, Rome, Paris, and Bursa. The goal is to discern similarities across these cities by examining their urban characteristics in depth. This comprehensive analysis involves three key techniques: satellite images are processed using U-Net for segmentation, revealing urban textures like buildings, streets, green spaces, and water bodies. Street views are dissected using DPT model, segmenting images into elements such as buildings, streets, sidewalks, sky, and greenery. Lastly, the ResNet model is used to classify architectural styles, identifying 52 styles. These components are compared with similar elements in the other cities, using standardized scores to measure similarities. This analysis shows that certain urban elements in Istanbul closely resemble those in the other cities. The Urban Echoes project provides insights into the complex structure of urban environments and enhances the understanding of residents and urban designers. By using machine learning and data analysis, the project sheds light on the detailed nature of urban features.
3
CONCEPT: HISTORY & SIMILARITY Conceptual Process
Urban Feature
Dataset
Model
Step 1
Step 2
Step 3
Satellite Urban Texture
Facade Architectural Style
Street View Street Features
Satellite images of 5 cities’ historical districts divided to 300m grids
1000 Google Street View images from 5 selected streets in each city
5000 Google Street View images of each city's historical districts
U-Net Image Classification
ResNet Architectural Style Classification
DPT Image Segmentation
Street
Categories
Green
Istanbul:
Istanbul:
Percentage of each satellite
Similarity Score
Sky
Road
Green
Others
Sidewalk
Building 18 Architectural Styles
Water
Building
Istanbul:
Percentage of each facade
Percentage of each street view
90%
13%
...
20%
80%
75%
...
10%
90%
62%
...
10%
90%
90%
70%
95%
95%
99%
60%
80%
80%
65%
50%
20%
Average percentage of each city
Paris
Bursa
Rome
Athens
Average percentage of each city
Paris
Bursa
Rome
Athens
Average percentage of each city
Paris
Bursa
Rome
Athens
100
69
Calcualte Standard Deviation of Each Category & Remapping Average Score 94
97
95
95
85
100
52
78
77
85
4
URBAN TEXTURE RESEARCH: SATELLITE Classification Method Satellite Classification Model: U-Net & Siamese Network
Data Mining: Scraping Satellite from Google Map
Istanbul
Athens - 3*3 km Sample Areas
3
32
Satellite Classification with U-Net
32
64
32
Average Similarity Results
Similarity learning with Siamese Network
32
Athens
Rome Crop into 10x10 grids as training dataset
64
Rome Istanbul every individual grid
64
8x8
512
16x16
32x32
256 256
16x16
Paris
64x64
128 128
32x32
64x64
128x128
128
128x128
64
Bursa
94.10
256x256
256x256
3 km
512
1024
Average
96.83 Similarity score for each grid
256 128
Bursa
Differencing layer
94.92
512 256
Conv 3x3 ReLu Copy MaxPool 2x2 Up-conv Pool 2x2 Conv 1x1
512
1024
Paris Each category of Athens, Rome, Bursa, Paris
Conv Network
Encoding
94.96
Classification Analysis
Athens
Istanbul
Building
Street
Green
Aerial Images of Other Cities Reflecting Back to Istanbul’s Map
Water
57 16
12
19
Rome
Bursa 0.9444
13 N/A
Bursa
0.9409
Rome 0.9508
60 31
Rome
Rome 0.9472
Bursa 0.9415
Rome 0.9488
53 27
8
12 Bursa
Rome
0.9411
0.9683
65 26
9 N/A Athens 0.9428
Paris
81
8
6
Paris 0.9215
5 Sun, Y., Bi, F., Gao, Y., Chen, L., & Feng, S. (2022). A multi-attention unet for semantic segmentation in remote sensing images. Symmetry, 14(5), 906. doi:10.3390/ Koch, G.R. (2015). Siamese Neural Networks for One-Shot Image Recognition. ICML deep learning workshop, vol. 2.
5
ARCHITECTURAL STYLE RESEARCH: FACADE Prediction Method ResNet Models Tryout (ResNet 34, 50, 101)
Data Mining: Scraping Facades Views from Google Map & Data Augmentation
CONV_1
Athens
CONV_3
ResNet 34
De-colorized
3*3, 64 *3 3*3, 64
Rome
ResNet 50
X 18 Styles
Bursa
CONV_3
1*1, 64 3*3, 64 *3 1*1, 256
Rotation Data Augmentation
CONV_4
CONV_1
1*1, 256 3*3, 256 *6 1*1, 1024
1*1, 128 3*3, 128 *4 1*1, 512
1*1, 64 3*3, 64 *3 1*1, 256
Crop
CONV_4
3*3, MaxPool, Stride 2
ResNet 101
7*7, 64 Stride 2
Line Romanesque Architecture ...
1*1, 512 3*3, 512 *3 1*1, 2048
FLOPS Average Poll, 1000-d fc, 3.8 x 10
83.43
9
CONV_2 CONV_3
Paris
CONV_5
Convolution +ReLU
... 1*1, 256 3*3, 256 *23 1*1, 1024
1*1, 128 3*3, 128 *4 1*1, 512
Max pooling
CONV_5
FLOPS
1*1, 512 3*3, 512 *3 1*1, 2048
Average Poll, 1000-d fc, 7.6 x 10
Fully connected+ReLU
72.46
9
Architecture
De-texturized
Georgian Architecture
Heading 2 = 90+a
9
CONV_2
7*7, 64 Stride 2
Heading 1 = 90-a
Average Poll, 1000-d fc, 3.6 x 10
70.81
Style of Image
Byzantine Architecture
Field of view = 90 m
3*3, 512 *3 3*3, 512
FLOPS
CONV_1
N
50
CONV_5
3*3, 256 *6 3*3, 256
3*3, 128 *4 3*3, 128
3*3, MaxPool, Stride 2
Scrape facades: Turgut Ozal Millet Cd Road
Art Deco Architecture
CONV_4
3*3, MaxPool, Stride 2
3 km
Confusion Matrix
Accuracy (%)
CONV_2
7*7, 64 Stride 2
Istanbul
Predication Accuracy
153
0
0
3
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
110
1
0
0
0
0
0
2
0
0
2
0
0
0
0
2
0
0
11
72
0
0
0
0
0
0
0
0
3
0
0
2
0
3
1
5
0
0
115
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
110
4
0
4
2
1
0
0
2
0
0
1
0
1
0
0
0
0
5
141
1
0
2
1
0
1
0
0
0
0
2
0
0
0
0
0
1
1
111
0
2
1
0
1
0
1
2
1
1
0
0
0
0
0
4
3
0
85
1
0
0
0
0
0
0
0
0
0
0
0
0
0
3
8
5
0
105
0
6
0
0
0
2
0
0
0
1
0
0
0
1
0
3
0
0
82
0
1
0
0
0
0
0
0
0
0
2
0
3
1
0
0
3
0
61
0
0
0
0
0
0
3
0
1
2
0
1
0
0
0
0
0
0
140
0
0
2
0
7
0
0
0
0
0
1
0
0
3
0
0
0
0
82
0
0
0
0
0
0
1
2
0
1
2
4
0
1
1
1
3
0
80
7
0
0
0
0
2
1
0
0
1
2
0
2
0
0
5
0
2
105
0
2
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
107
0
0
0
1
0
0
1
0
0
0
5
0
2
2
0
0
1
0
120
0
0
0
0
0
12
0
0
16
0
0
3
0
1
0
0
1
0
88
Predicted Style
Softmax
Similarity Analysis Palladian Architecture
Art Deco Architecture
Art Nouveau Architecture
American Craftsman
Romanesque Architecture
Chicago School Architecture
Athens
Byzantine Architecture
Rome Bursa Paris
Bursa
Paris
Paris
Athens
Bursa
Paris
Achaemenid architecture American craftsman style
Bursa
Art Deco architecture Art Nouveau architecture
Pa
ris Istanbul
Istanbul
Istanbul
Istanbul
Istanbul
Istanbul
Bauhaus architecture Beaux-Arts architecture
Istanbul
Byzantine architecture Chicago school architecture Colonial architecture Deconstructivism Edwardian architecture Georgian architecture International style Novelty architecture Palladian architecture Postmodern architecture
Bu
rsa
Ro
me
Romanesque architecture Tudor Revival architecture
Istanbul
Istanbul
Istanbul
Rome
Bursa
Athens Athens
Georgian Architecture
Achaemenid Architecture
Bursa
Paris
Rome
Paris
Tudor Revival Architecture Xu, Z., Tao, D., Zhang, Y., Wu, J., & Tsoi, A. C. (2014). Architectural Style Classification using Multinomial latent logistic regression. Computer Vision – ECCV 2014, 600–615. doi:10.1007/978-3-319-10590-1_39
6
STREET FEATURES RESEARCH: STREET VIEW Segmentation Method Data Mining: Scraping Google Street View Images
Istanbul
Image Segmentation Training Model: DPT Model
Athens: 3*3 km Sample Areas Downloaded 4860 Google street view images
3 km
Input
Fusion
Reassemble 32
Fusion
Transformer
Heading = 180-a
Resample
Residual Conv Unit
0.45
0.09 0.06 0.23 0.03 0.14
0.49
0.10 0.09 0.20 0.050.08
+
Residual Conv Unit
Paris
Project
Concatenate
Embed
0.10 0.12 0.20
Rome
Resample 0.5
Paris
0.12 0.07
0.39
Bursa
Project
2300 images
Field of view = 90
Green Road Sidewalk Others
Athens
Output Read
Bursa a
Reassemble 32
Building Sky
Transformer
2100 images
m
Fusion
Transformer
Rome
50
Reassemble 32 Transformer
1830 images
N
Average Segmentation Results
0.37
0.11 0.10 0.26
0.38
0.14
0.05 0.11
Head Istanbul
3300 images
Reassemble
Tokens
Fusion
0.08 0.08 0.20 0.02
Similarity Analysis 100
0.07 29 0.12 23 0.20 16
50
0.14 15 0.03 12 0.23 13
x 2100
0.06 93 0.09 41 0.45 30
34
Rome 0.08 28 0.05 13 0.20 16
x 2300 Average
0.09 100 0.10 45 0.49 18
64
Bursa 0.11 19
Building
Average
0.10 100 0.11 63 0.37 100
62
Sky
Green
Road 93
...
Building
85
41
30
13 Sky
Green
Road 100
80 ...
100
45 17 Building
Sky
Green
Road
100
0.05 13 0.26 10
x 3300
49
45
16
Athens
Average
...
Remapping
0.12 49 0.10 45 0.39 100
Calculate with each point of street view in Istanbul
x 1830 Average
77
69
63
...
50 11
Paris
Building
Sky
Green
Road
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021). Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1217912188).
7
SIMILARITY RESULT ANALYSIS
Athens - Street View Similarity
Rome - Street View Similarity
Bursa - Street View Similarity
Paris - Street View Similarity
Istanbul - Street View Similarity
Athens - Architectural Style Similarity
Rome - Architectural Style Similarity
Bursa - Architectural Style Similarity
Paris - Architectural Style Similarity
Istanbul - Architectural Style Similarity
Athens - Satellite Similarity
Rome - Satellite Similarity
Bursa - Satellite Similarity
Paris - Satellite Similarity
Istanbul - Satellite Similarity 8
TOP SIMILARITY SCORES MAPPING Bursa 94.1
Rome 94.6
35
Rome 94.5
Rome 94.4
Athens 93.0
Focus Area for Similarity Assessment Rome 94.5
Similarity Score Satellite
Architectural Style
Street View Athens
Rome Rome 94.3
Rome 94.2
Bursa
Paris Legend: Score Scale for Each Category
Bursa 94.1
Paris 93.6
Satellite: 0
100
0
100
Architectural Style:
Street View:
Highlighted Grid for Scores >93 9
COGNITIVE LANDSCAPES Exploring Emotional Responses in VR-EEG Driven Art therapy Independent Research | Suzhou, China | Summer, 2023 Tutor: Tu Han My Role: 100% VR Design, 40% User Test, Group Data Analysis Team: Hansen, Zihe Wang, Qingyun Liu This project addresses the escalating mental health crisis by integrating Electroencephalogram (EEG) technology with Virtual Reality (VR) in art therapy. Traditional art therapy faces challenges in medium selection and measuring progress, which are addressed by our VR-EEG platform. This immersive environment enhances therapeutic engagement and facilitates emotional expression. The VR-based art therapy procedure unfolds in three phases: planning, creation, and sharing, each crafted to elicit specific emotional responses. Various VR scenarios, encompassing diverse contexts, textures, and models, were developed to explore their impact on participants’ perceptions and emotions. In this project, interactions of 12 participants within VR settings were tracked using EEG technology. The focus was on examining shifts in alpha, beta, and theta brainwaves, which are critical indicators of emotion states. Integration of EEG allows for real-time brain activity monitoring, offering deep insights into the emotional effects of the therapy. The project also includes a sharing platform, enabling participants to share art creations, give feedback, and communicate feelings with others.
10
RESEARCH BACKGROUND Emotion Problems
Research Process
Global mental health needs are surging due to social disparities, pandemic impacts, and modern stressors, with 40% of American adults facing mental challenges and over 275 million people globally suffering from anxiety or depression.
Creating
Planning
Style
Context
Ancient Painting
Natural
Model
Move
Rotate
Sharing
Delete
Review
Share
Review and Browse shared share your own scenes published scene created by others Realistic
Anxiety
Distraction
Comment
Comment and communicate with friends
Urban
Loneliness
Art Therapy Challenges
Scanning & Photogrammetry
Art therapy, a form of psychotherapy that primarily employs various forms of art as a means of communication, is well-regarded for its efficacy in facilitating the articulation and therapeutic processing of multifaceted emotional states. Nonetheless, it encounters specific challenges:
VR
EEG Monitor
Review
3 Reference Sensors
2 Forehead Sensors Alpha: 8Hz-12Hz
Online Sharing Platform
2 SmartSense Conductive Rubber Ear Sensors
Type Here ...
Share
α Rising Trend Relaxed
Beta: 12Hz-40Hz
β
Steady Focused
Media Selection
Emotion Measurement
Emotion Expression
Hard to select mediums aligning with emotional needs
Hard to measure emotional changes
Need guidance to express emotions
VR-EEG Platform
Comment Comments
Theta: 4-8Hz
θ Rising Trend
Type Here ...
Communicative
Test Environment
This research integrates EEG with VR art therapy, aiming to fill existing gaps in the traditional art creation process. EEG's non-invasive brain activity monitoring offers real-time emotional insights, offering a novel dimension to art therapy.
EEG Headset
VR Headset VR View Recording
Testee 1 Planning
Creating
Sharing
Personalized choices in VR Based on Individual Preferences
EEG’s monitoring to understand emotional changes
VR social platform for sharing and emotion expressions
Testees exhibit higher negative affect than positive on Positive and Negative Affect Schedule (PANAS) [1] [1] Roemer, A., & Medvedev, O. N. (2023). Positive and Negative Affect Schedule (PANAS). In Handbook of Assessment in Mindfulness Research (pp. 1-11). Cham: Springer International Publishing.
11
CONTEXT AND MODEL COLLECTION Study Process
Nature + Urban
Context Categorization
Urban
Scanning and Photogrammetry
Nature
Scanned from Panmen Scenic Area
Aerial Photography DJI Mini 3 Pro
Context Collection
Metashape 24.86 Hectares a4: Urban
a3: Nature + Urban Typical Episode Extraction
8 Models Pavilion
Pan City Gate
a4: Urban
a3: Nature + Urban
Ruiguang Tower
Sirui Hall
Landscape
Sirui Hall
Bridge
Model Collection
5000 Photos
Ruiguang Tower
Furniture
Ancient Building Identification
a2: Nature
Pan City Gate
Pavilion
Bridge
a2: Nature
Furniture
Landscape
Sculpture Point Cloud
b1: Scanning and Model Generation Model Wireframe
b2: Model Stylization
Model Confidence
Combination
Realistic Combination
a1b1: Empty Context + Realistic Model
a2b1: Nature Context + Realistic Model
a3b1: Nature & Urban Context + Realistic Model
a4b2: Urban Context + Stylized Model
Model Texture Ancient Painting Combination
a1b2: Empty Context + Stylized Model
a2b2: Nature Context + Stylized Model
a3b2: Nature & Urban Context + Stylized Model
a4b1: Urban Context + Realistic Model
12
DESIGN STUDY Planning
Creating
Sharing
Research Question 1: What kind of brainwave dominate during the initial planning process?
Research Question 2: What style-context pairs best enhance relaxation, focus or communicative state[1]?
Research Question 3: How do the emotions change During different states of the creating process?
Research Question 4: Does the process of reviewing, selecting favorite scenes, and sharing contribute to therapeutic outcomes?
Collect EEG data on wandering and planning in blank scene.
Calculate the normalized values per scene, identify top 3 effective scenes, and select a3b2 as the sample.
Contrast brainwave data with screen recordings to assess brainwaves during creating operations.
Collect EEG data during reviewing and favorite scene-choosing process.
Summarize and categorize changes in brainwaves corresponding to different operations. Large objects manipulation
Compare the average value of α, β and θ of 30s resting state, integrating questionnaires. Emotional States Scale Relaxed
-3 to 3
Difference in mean scores preand post- test
Communicative -3 to 3
Objects selection Compare the average value of α, β and θ of resting state, and integrate questionnaire data.
Top 3 scenes: α wave
Top 3 scenes: β wave
Focused
-3 to 3
Top 3 scenes: θ wave
Figures and landscape elements placement
Research Question 5: Do testees experience positive emotion changes after the entire design process? Planning 1:
Planning 2:
Planning in Mind Planning for Style and Context Selection
Creating:
Sharing:
Wander, place, move, rotate, delete, select
Reviewing, favorite scene choosing, sharing
α β θ
Wandering and Planning in the blank scene for 1.5 min.
[1] Fischer, N.L., Peres, R., Fiorani, M., 2018. Frontal Alpha Asymmetry and Theta Oscillations Associated With Information Sharing Intention. Front. Behav. Neurosci. 12.
13
PLANNING Planning 1
Planning 2
Grow rates of average values :
Choose mood-enhancing combinations:
Judging from the results, most people will increase the average beta wave after the planning1 stage, and it can be seen that they will be more focused.
+106.43%
0.6 0.4 0.2
1.2
<35%
a1b0
47%
+219.99%
0.6 0.4 0.2
1.0
1.0 0.8
0.0 1.4 1.2
0.0 1.4 1.2 1.0
+223.75%
0.8 0.6 0.4
44%
0.0
Rest 30s
Planning 60s
Rest 30s
+118.66%
0.6 0.4
0.0 1.4
a1b0’
46%
1.2
0.8
0.8
43%
0.6 0.4
ment; a3b2 with 68% normalization is most effective. a2b1
60%
52% Rest 30s
Planning 60s
Rest 30s
a1b2
58%
a1b2
58%
57%
0.6 0.4
49% Rest 30s
Planning 60s
Rest 30s
1.2
50%
0.2 0.0 1.4
a2b0
42%
1.0
Rest 30s
Planning 60s
a3b2
Rest 30s
0.0
52%
0.6
1.4
0.4
1.2 1.0
+141.12%
0.8 0.6 0.4
1.4
0.2 0.0
1.2
1.4
1.0
1.2
Rest 30s
Planning 60s
Rest 30s
+6.21%
0.6
59%
59%
0.6
0.0
1.2
1.4
1.0
+8.19%
0.8 0.6
1.2
Rest 30s
Planning 60s
Rest 30s
0.2 0.0
0.8
1.4
0.6
1.2 1.0
-58.69%
0.6
a3b0
62%
0.4 0.2
1.4
0.0
1.2
1.4
38%
68%
43%
49%
a2b1 48%
0.2
1.2
Rest 30s
Planning 60s
Rest 30s
a3b1
65%
58%
0.8
37%
0.6
a1b1
0.4 51% Rest 30s
Planning 60s
Rest 30s
1.4
a3b0’
52%
51%
0.2 0.0
1.2
Rest 30s
Planning 60s
61%
Rest 30s
a3b2
52%
1.0
1.0
1.2
a2b2
1.0
0.2 0.0
a3b2
Rest 30s
0.6
0.0
0.4
0.8
Planning 60s
0.8
1.4
1.0
0.4
1.2
Rest 30s
0.4 56%
0.2
1.4
48%
0.2
1.0
0.2 0.0
65%
0.6
0.0
0.4
0.4
52%
60%
0.8
1.4
a2b0’
0.8
1.0 0.8
a2b1
0.4 41%
0.2 0.0
1.2 1.0
0.8
0.2
Testee 9
Communicative
61%
0.2
1.0
1.4
1.2 1.0
Testee 8
Compare and select top 3 scenes for mood enhance-
Focused
1.4
0.8
Testee 7
Relaxed
47%
0.6
1.0
0.0
a1b1
1.6 1.2 0.8 0.4 0.0
0.4
48%
0.2
0.2
1.2
0.8
0.2
0.8
Testee 6
35~40%
0.4
1.4
1.0
Testee 5
40~45%
0.6
0.0
Testee 4
and a3b1 less positive than blanks, others more effective.
45~50%
1.2
1.0
Testee 3
50~55%
Relaxed
1.2
0.8
Testee 2
55~60%
Focused
Testee 1
Analyze EEG and questionnaire data from 16 scenes; a2b2
60~65%
1.4
Selecting top 3 effective scenes of each emotion:
Questionnaires:
>65%
1.4
1.4
Normalized values’ extent:
1.0
0.6 0.4
48%
0.6 0.4
0.2 0.0
Testee 10
0.8
0.0
1.2 1.0
+87.26%
0.8 0.6
1.4 1.2
55% Rest 30s
Planning 60s
Rest 30s
0.0
Testee 11
a4b0
35%
1.0
-36.93%
0.4
0.0 1.4
1.2
1.2 1.0
-87.47%
0.8 0.6 0.4
Planning 60s
Rest 30s
Rest
55 s
2 min
50 s
45 s
40 s
35 s
30 s
25 s
20 s
15 s
5s
Planning
10 s
55 s
1 min
50 s
45 s
40 s
35 s
30 s
25 s
20 s
5s
10 s
15 s
Rest
a4b1
51%
59%
0.6
a4b0’
0.0
47%
1.2 1.0
0.8
0.8
61%
64%
0.2
1.0
Rest 30s
Planning 60s
Rest 30s
a4b2
51%
41%
0.6 0.4
52%
0.2 0.0
64%
Rest 30s
0.8
1.4
0.4
0.0
0 min
Rest 30s
0.6
0.2
Beta average data
51%
0.2
1.4
1.2
Planning 60s
0.4
0.4
0.0
0.2
Beta original trend
52%
0.6
0.6
56% Rest 30s
1.0
0.8
1.2
0.8
Testee 12
0.0
1.0
1.4
a4b1
0.2
1.4
0.4 0.2
68%
0.6 0.4
0.2
1.4
0.8
Rest 30s
Planning 60s
Rest 30s
a3b2
56%
a1b1
52%
48%
0.2 0.0
Communicative
+6.92%
0.8
Rest 30s
Planning 60s
Rest 30s
14
CREATING: SPACE CONSTRUCTION Chose a3b2 as the sample scene for creating: All α, β and θ waves show a gradual upward trend in fluctuations, with β waves increasing the most, meaning that the users become more and more focused.
15
g
VR-EEG ART THERAPY PLATFORM Select from Context Collection Planning Anxiety
Choose Creating Mode Creating
Distraction
Planning
Comment & Browse Others’ Posts
Sharing
Loneliness
Creating
Sharing
Planning
Select from Model Collection Planning
Creating
Sharing
Place, Edit, Delete Models Creating
Architecture
Planning
Tree
Creating
Bridge
Furniture
Building Part: Temple
Sharing
Planning
Plan in Blank Scene Planning
Creating
Sharing
Share Your Creation Creating
Cancel
Planning
Creating
Sharing
Planning
Creating
Share
Sharing
Planning
Creating
Sharing 16
RECITYGEN Interactive and Generative Participatory Urban Design Tool Independent Research | Beijing, China | Spring, 2023 Tutor: Runjia Tian My Role: 100% App Development, 80% Data Analysis, Group User Test Team: Di Mo, Chengxiu Yin RECITYGEN is a tool aiming at enhancing urban design through participatory methods. Its primary goal is to enable collaboration between residents and urban planners, ensuring that community needs and preferences are incorporated into urban development. The tool harnesses latent diffusion models and the Segment Anything Model (SAM) to create an accessible platform for public input in urban design. Users interact with RECITYGEN by uploading street view images to its online interface and using SAM to mark areas for improvement. A key feature is the tool’s AI-driven visualization: users provide text descriptions of desired changes, and RECITYGEN uses a latent diffusion model to quickly generate a visual preview of these changes. The tool, first tested in a Beijing urban renewal project, received positive feedback. This suggests it is effective in aligning urban design with the expectations of the public. Future plans include broadening its application, improving user interaction, and encouraging more community participation in the urban design process.
17
RESEARCH BACKGROUND Residents Facing Communication Barriers I've got tons of ideas to improve our neighborhood, but there's just no easy way to share what we're thinking.
Traditional Process Traditional Process
Proposed Process Proposed Process
Public
Public
Agree. I see so many areas needing improvement, like the old community center. But there's a huge gap between our wishes, and what the urban planners understand.
Initial Concerns
Conceptual Process
User Interface
Technical Process
Locate Sites
Pin Sites
Baidu mapping API
Select Sites
Take Site Photos
Upload Site Photos
Identify Concerns
Add Points to Set Mask
Segement Anything Model
Generate Images
Input Text Prompts
Latent Diffusion Model
Evaluate/ Regenerate
Evaluate/ Rate Results
Upload Data to Cloud
Interview with Designers
Comment and Take Survey
Data Analysis
Initial Concerns
Data Collection
Designers Struggling with Information Gathering
Concerns Indentification Quick, cost-effective input is hard with our methods. We're always constrained by resources and time.
Concerns Indentification Workshops Public Forums Stakeholder Engagement
RECITYGEN APP
Online Surveys Interviews This area has potential, but figuring out what the community really wants? It's like we're guessing half the time without their input.
Feedback and Refinement Image Generation
Collaborative Communication Gap This draft plan is not quite what we want. If only there was a way to show you my ideas directly, with images or something tangible, to make my points clearer.
Evaluation/ Regeneration
I understand the challenge. We need a more engaging way to collaborate and refine our approach with real-time feedback. Initial Design Concepts
Designer
Concerns Indentification
Designer 18
CONCEPTUAL PROCESS Step1: Locate Sites
Step2: Select Photos
Step3: Identify Concerns
Step4: Generate Images from Texts
Step5: Evaluate and Regenerate
Step6: Share with Designers
Accessible green space
+
Active edge with planters
...
TECHNICAL PROCESS Output Image
Prompt Encoder
Points
Image Encoder
...
Lightweight Mask Decoder
Image Embeddings
Input Image
Mask
512 X 512
1 - Mask
Valid Mask
Step3: Segment Anything Model
VAE Decoder Outside Mask
Inside Mask
Step4: Latent Space
VAE Encoder
... z0
z1
z2
zt Forward Diffusion
Lantent Vector
3X 512 X 512
zt-1 Denoising UNET
Noisy Lantent Vector
z0 Lantent Vector
...
Prompt Encoder
Prompt Segment Everything
t-1
Mask Catogaries Architecture
Structure
Results A towering, sleek skyscraper with a glass facade ...
8
A contemporary steel footbridge with ...
Pavement
A winding garden path made of patterned stone ...
Green
A vibrant community garden street pocket park ...
5 4 2
19
INFORMATION ARCHITECTURE
USER INTERFACE
20
RESULT ANALYSIS Pilot Test Mapping
Score
0
Mask
Prompt
Generated Results Summary
Top Rated Results
A residential complex with a glass facades ...
6
6
3
4
A sleek, straight concrete pavement , minimalist garden ...
4
4
6
8
A tranquil community park elements of a Japanese garden ...
7
2
5
8
A circular cobblestone path in a garden, smooth pebbles ...
6
8
6
7
A tall, artistic residential tower with a unique geometric facade ...
4
5
6
4
A sleek, futuristic pedestrian bridge with glass walkways ...
1
3
5
3
A circular cobblestone path in a serene Zen garden...
4
9
8
6
A street pocket park with colorful flowers and tall grasses ...
4
5
5
8
An office building with active store fronts, modern style buildings ...
4
6
4
2
A traditional Chinese style store front accessible from the path ...
8
6
6
7
A cozy urban oasis pocket park, peddle stone pavement ...
6
8
6
4
A modern park in an urban setting, minimalist landscape ...
2
4
5
4
5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
0 5 10
21
SECTIONAL SAMPLING Controlled Generation of High-Resolution Architecture Design Independent Research | Boston, MA | Summer, 2023 Tutor: Runjia Tian My Role: 30% Data Collection, 50% Model Training, 50% Writing Team: Zheng Fang, Xuechen Li This paper addresses the challenge of algorithmically generating 3D architectural designs. Conventional rule-based methods and GAN-based 3D generative models offer limited extrapolation and control and are limited to generating low-resolutional results. Our method utilizes fine-tuned latent diffusion models, specifically Stable Diffusion, applied to an architectural dataset, to produce detailed architectural sections. These are then synthesized into 3D models using latent interpolation. The pipeline starts with creating sectional poché datasets for various building typologies—houses, museums, and offices. The fine-tuned Stable Diffusion models, enhanced through LoRA, generate new sections, conditioned on section profiles via ControlNet. This ensures fluid transitions between sections. The generated sections are reconstructed into 3D models using the Marching Cubes algorithm within the Grasshopper. Our research contributes significantly to AI-generated architectural content, providing designers with advanced tools for creating high-quality, controlled models during the initial stages of a project.
22
CONCEPTUAL FRAMEWORK Data Collection
Step1: Precedent Dataset Rhino Model x 9
2D Sections Generation
Step2: LoRA Model Training Serial Sections x 60
Step3: ControlNet Conditioned Key Sections x 10
3D Construction
Step4: Diffusion 2D Inference Interpolation 30 x 10
Step5: Sampled Point Cloud Point Cloud x 3
Step6: Mesh Reconstruction Generated Model x 3
Data Collection House
Museum
Office
Motherhouse
Poli House
House In Leiria
Yale Center for British Art
Ningbo Museum
Museum of Image and Sound
UC Innovation Center
Netherlands Institute
Larkin Building
Independent Architecture
Pezo von Ellrichshausen
Aires Mateus
Louis I. Kahn
Wang Shu
Diller Scofidio + Renfro
Alejandro Aravena
Neutelings Riedijk
Frank Lloyd Wright
23
TECHNICAL PROCESS
24
RESEARCH BACKGROUND Profile: Mother House
Monochrome, Greyscale, Archihtectural Section, House in Leiria, Single Family Home No Humans
Prompt: Single Family
+
LoRA: House in Leiria
Profile: Yale Center for British Art
Monochrome, Greyscale, Archihtectural Section, Museum of Image and Sound, No Humans
Prompt: Museum
+
LoRA: Museum of Image and Sound
Profile: UC Innovation Center
Monochrome, Greyscale, Archihtectural Section, Larkin Building, No Humans, White Background
Prompt: Office
+
LoRA: Larkin Building
25
3D CONSTRUCTION RESULTS House
Museum
Office
The algorithm captures single family house design, reflecting wall thickness and slab artic-
For museums, the method mirrors unique spatial distribution, showcasing its ability to
In office design, the algorithm mirrors typical office building scale, proportion, and struc-
ulation, essential in residential architecture.
reproduce the complex composition of public cultural spaces.
tural integrity, highlighting its potential in designing functional office spaces.
X1
X2
X3
X1
X2
X3
X1
X2
X3
Y1
Y2
Y3
Y1
Y2
Y3
Y1
Y2
Y3
Y1
Z1
Z3
Z2
X2 X2
X1
Y1
Z1
Z3
Z2
X1
Z1
Y1
X1 Y2
Y2
X2
Y3
Z2
X3
Y3
X2
Z3 Y1 Y2 Y3
X3 Z3 Z3 Z2 Z1
Z2 Z3 Z2
Z1
Z1
26
SPATIAL SYMMETRY Algorithmic Exploration of Calabi-Yau Forms Arch 743 Seminar | University of Pennsylvania | Fall 2020 Tutor: Ezio Blasetti My Role: Group Research, 100% Algorithm, 100% Generative Design Team: Bohao Sun, Xianlong Deng This project merges the disciplines of architecture, geometry, and computational science to utilize generative systems in transforming Calabi-Yau manifolds, which are pivotal in algebraic geometry and superstring theory, into innovative three-dimensional architectural models. The process begins with conformal mapping to maintain the local angles of Calabi-Yau manifolds, thereby streamlining the complexity of their geometrical attributes. Subsequently, a 4D conformal Calabi-Yau manifold is mapped and subjected to stereographic projection to achieve a 3D visual representation. The final step involves the 3D reconstruction of point clouds into a mesh format. Emphasizing algorithmic design principles, the project adopts an iterative, experimental approach to leverage computational capabilities for creating diverse spatial structures. This method underscores the importance of spatial dimensions in architectural design and highlights the role of advanced technology in visualizing and interpreting intricate spatial forms. The project aims not just to explore theoretical concepts but also to offer practical applications in architectural design, pushing the boundaries of how we understand and utilize space in the built environment.
27
CONCEPTUAL FRAMEWORK
N=2
Conformal Mapping
Top
Right
Axon
4D Projection
3D Representation
f(z)=Z
f(z)=1/z N=3
f(z)=z^2/2
N=4
f(z)=√2z
f(z)=sin z
N=5 f(z)=cos z
f(z)=e^z
28
CONFORMAL MAPPING
HYPERDIMENSIONAL TRANSFORMATION
By changing the functions of x and y values of points, a catalog of possible transformed forms of 5-dimension calabi-yau is created.
5-Dimension Manifold
f(z)=1/z
Conformal Manifold
f(z)=z^2/2
f(z)=√2z
#Function to create a 3D parametric plot of a conformal shape def ParametricPlot3D(step=0.1): maxB = math.pi/2 + step for k1 in range(n): for k2 in range(n): for a in qrange(-1,1,step): for b in qrange(0,maxB,step): z1= Complex_z1(a,b,n,k1) z2= Complex_z2(a,b,n,k2) x_new = z1.real y_new = z2.real #Calculate the new z coordinate by combining the imaginary parts of z1 and z2 with a rotation angle Alpha. z_new = math.sin(Alpha)* z1.imag + math.sin(Alpha) * z2.imag points.append(rhp3(x_new,y_new,z_new)) return points
f(z)=sin z
f(z)=cos z
Map a 4D Calabi-Yau manifold, project into 3D, rotate across xy, xz, xw, yz, yw, zw planes, and catalog resulting geometries.
Conformal Manifold
0o
f(z)=e^z
f(z)=1/z
XY
f(z)=z^2/2
XZ
f(z)=√2z
XW
f(z)=sin z
YZ
f(z)=cos z
YW
f(z)=e^z
ZW
4D Manifold Projection
30o
60o
def rotate(self, XY, XZ, XW, YZ, YW, ZW): vec = [self.x,self.y,self.z,self.w] angles = [XY, XZ, XW, YZ, YW, ZW] for i in range( (angles)): angle = m.radians(angles[i]) c = m.cos(angle) s = m.sin(angle) rotationMatrix = [ [[c,s,0,0] ,[-s,c,0,0] ,[0,0,1,0] ,[0,0,0,1] ], [[c,0,-s,0] ,[0,1,0,0] ,[s,0,c,0] ,[0,0,0,1] ], [[c,0,0,s] ,[0,1,0,0] ,[0,0,1,0] ,[-s,0,0,c] ], [[1,0,0,0] ,[0,c,s,0] ,[0,-s,c,0],[0,0,0,1] ], [[1,0,0,0] ,[0,c,0,-s] ,[0,0,1,0] ,[0,s,0,c] ], [[1,0,0,0] ,[0,1,0,0] ,[0,0,c,-s],[0,0,s,c] ] ] matrix = rotationMatrix[i] vec = transformVec(vec, matrix) return HyperVec(vec[0],vec[1],vec[2],vec[3])
90o
120o
150o
29
3D REPRESENTATION Selected Geometries Matrix f(z)=sin z
f(z)=√2z
f(z)=z^2/2
8-dimension calabi-yau manifold by conformal transformation of sin(z) and e^z:
N=3 Top
Right
Axon
N=4 Top
Right
Axon
N=5 Top
Right
Axon
30
INCLUSIVE AND CONNECTED LANDSCAPES For Self-Constructed Settlements in Quito, Ecuador Larp 702 studio | University of Pennsylvania | Spring 2021 Tutor: David Gouverneur My Role: Group Research, 80% Urban Design, 100% Drawing Team: Shiqi Ming, Siying Xu Over a billion people around the globe live in unplanned of self-constructed settlements frequently at peri-urban locations, excluded from the benefits of city life, and occupying high-risk sites. Latin America is still considered the continent with the highest social inequalities, manifested in the disconnection between the formal and the informal areas. These areas are characterized by environmental/health problems, the lack of communal services, infrastructure and public spaces, poor accessibility, stigmatized by the formal city, and presenting low income and high levels of violence. This project posits that landscape-driven approaches are powerful tools to address the aforementioned concerns, fostering social inclusion, and breaking down physical and cultural barriers, connecting the settlements to their natural systems, linking formal areas to the informal ones, improving physical and performative relations among the informal settlements, and establishing productive local, urban, and metropolitan networks. Supported by Quito municipality, a composite of informal enclaves, including Comité del Pueblo, La Bota, and Puertas del Sol, located on rather flat land in the north-east of Quito, Ecuador was selected as the site.
31
BACKGROUND Environmental Challenges
Social Conditions
In Quito, Ecuador, neighborhoods face environmental degradation due to deep ravines and biodiversity loss. Ravines, rich in habitats, are
Highly consolidated informal settlements are isolated from the formal city and each other, lacking community facilities and public spaces.
neglected and inaccessible. Some housing is built on environmentally unstable, high-risk terrain.
Limited connectivity and congested commercial corridors exacerbate these challenges, contributing to a high poverty rates.
Steep Slope
Surface Runoff
Degraded Habitat
Unstable Structures
Site Landslide Risk Map
Isolation from the City
Disconnection
Lack of Amenities
Economic Challenges
Site Connection Map
PUERTAS DEL SOL
PUERTAS DEL SOL
Land Risk Proportion
1 in 24 person under poverty level
LA BO TA
LA BO TA
CO MI TE
CO MI TE
DE LP UE BL O
EL CARMEN
DE LP UE BL O
EL CARMEN
Site Boundary
Disconnection
Risky Structures
Road Connection
Risky Area
Educational
Landslide Events
Commercial
Ravines
Ravines
32
DESIGN STRATEGIES 1. Reduce Risks & Address Environmental Concerns
Site Plan
1. Tech School
Relocation: Densified Community
2. Agriculture School 3. Church 4. Farmer's Market
Natural Preservation
5. Park Library 6. Ecology Exhibition & Education
10
7. Recycling Center 8. School
Street Green
9. Water Management Facility
9
10. Community Center 11. New Campus Urban Agriculture
2. Enhance Accessibility & Mobility
Proposed Metro Bridge
Central Corridor
4 2
5
3
7 6
1 Gandola Station 8
3. Celebrate the Community
Community Open Space 11 District Park
Commercial Street Campus
0
75
150
300m
33
RISK MANAGEMENT STRATEGY A Green Armature: As a Tool to Reduce Risks and Address Environmental Concerns
Agriculture Runoff Habitat Restoration
ation
Reloc
re
icultu
n tio
n1
n Agr
Urba
Sec
3
tio
Natu
c Se
s
ral Re
Section 1
Trail
ion torat
Community Raingarden Agroforestry
on
i ect
S
Section 2
ral Natu
n
ratio
Resto
k
n Par
ventio
Inter
re
icultu
n Agr
Urba
2
Terrace Agriculture
Bio-retaining Wall
Trail
Section 3
ion
ificat
Dens
k
& Par
Road
34
DIGITAL PRESERVATION: 3D Scanning of Xi’an’s Dongyue Temple Arch Studio | Southeast University | Fall 2018 This project entailed a detailed 3D scan of Xi'an's Dongyue Temple, using Faro technology. Over three days, our ten-student team captured
The Dongyue Temple, located near Xi'an's Ming City Wall, was originally established in 1116 during the Song Dynasty and underwent several
the temple's architectural intricacies, aiming to digitally preserve and study this historic site for future conservation.
renovations and expansions during the Ming and Qing Dynasties.
100m
Elevation: Specifies the color of points based on heights. 50m
Dou gong
Cha shou
Liang jia
Colored murals
Middle Hall
0m
Rear Hall
100%
Intensity:
Main Hall
Specifies the color of points based on the laser return intensity. 50%
Shanmen (Gate)
0%
Elevation: Specifies the color of points based on surface orientation.
35 35
WATER MINER: Speculative Urban Design on Johannesburg 2050 LARP 701 studio | University of Pennsylvania | Fall 2020 This studio project employs multimedia to imagine Johannesburg 2050, where extreme weather drive residents to an underground city within former mines. This speculative design emphasizes ecological restoration for sustainable living in challenging environments.
Storyboard
Water level
Camera height
Water level
Camera height
Water level
Camera height
Water level
Camera height
Water level
Camera height
36 36
DYNAMIC EDGES: Conservation and Development of New York State LARP 601 studio | University of Pennsylvania | Fall 2020 This research project reenvisions the borders of New York State as dynamic, functional areas. Utilizing GIS data analysis, it proposes strategies
Key approaches include expanding conservation easements to enhance activities and broadening
that balance conservation efforts with development needs.
protected areas for ecological restoration, aiming to transform borders into sustainable spaces.
Conflicts
Existing Edges
Dynamic Edges
The boundaries of conservation zones
Identify approaches for managing
and areas of high population density
boundaries based on their ecological
suggest a possible conflict.
and economic significance.
Expand activities Expand protected area
Strategies
Albany Pine Bush Preserve
Greenport Conservation Area
Phoenicia Wild Forest
Highland Lakes State Park
Blue Mountain Reservation Park GIS Data Source: McGarigal K; Compton BW; Plunkett EB; DeLuca WV; Grand J; Ene E; Jackson SD. 2018. A landscape index of ecological integrity to inform landscape conservation. Landscape Ecology 33:1029-1048.
37
UNIVERSITY OF NEW MEXICO INTERGRATED CAMPUS PLAN Online Survey / On Site Open Houses for Design Reflection As a part of Sasaki Urban Design Team, I developed the UNM’s 2050 campus plan focusing on community engagement. Utilizing the Comap
This extensive engagement helped identify welcoming spaces, areas of safety concern, and accessibility issues on campus. The analysis of this
platform, we gathered 435 online responses and over 1,300 comments, plus feedback from 500+ participants in Albuquerque open houses.
rich community input was pivotal in shaping our design framework.
CoMap Platform Result Analysis for Online Survey
38
Perception - Design - Technology
39