Methodology and Framework

Data identification and generation

SVI is a technology that represents urban streetscape images from a human perspective. Through GPS positioning, the location of each photo is determined, and then the Geo-reference extracts the geographic information coordinates of each photo and the main database of Google Map to make a correlation, associating each point with these photos on the map.

Google Street View, a developer of technology, launched the Google Street View (GSV) feature in 2007. Currently, in addition to Google, Baidu and Tencent also provide related services. Due to the huge number of street view images and their uneven quality, some photos need to be adjusted by the recognition algorithm for data after taking. Moreover, Street View photos related to privacy also need to be excluded or post-processed. Street View Supply Channel website not only locates and labels the photos back for clean collection, but even scans the massive photo data and reads the related text information to fix the map information.

Image Extraction

The most open and easily accessible street view resources should be Google Street View. In the Google Street View interface people are able to change their position and rotate their visual field freely to simulate the real feeling of street view. However, Google Street View is web-based and unable to be downloaded and processed directly.

The group chooses to use Street View Download 360 Pro, an app for downloading 360° panoramas (example see Figure 4 2) from Google Street View, to extract the necessary images. With the area download tool it is easy to get all panoramas within an area during a certain period at one time. Moreover, the exif metadata are added to the image such as geolocation, elevation, date taken, camera rotation, etc. This may facilitate some future analysis.

As the purpose of street image view analysis is to simulate what people see and feeling, panoramas are actually not available to analyze. Things have to be done to crop the panoramas into single view images i.e. looking front, back, right or left. Fortunately, although large distortion occurs in the upper and lower parts of the panoramas, the middle parts looks normal like human vision. As a result, as long as you zoom in and just cut the panoramas, it will definitely work.

This is a simple, mechanical and repetitive job, and computers are best at it. This is not difficult if you have some programming experience. And as long as someone else completes the program, even people who know nothing about computers can easily complete the process.

2011 Images

2021 Images

Generally speaking, we now use the semantic segmentation method based on deep learning to analyse the street view. Image segmentation is an important subject in the field of image processing and computer vision. It is widely used in the fields of scene understanding, medical image analysis, robot perception, video surveillance, augmented reality, image compression and so on. In recent years, due to the success of deep learning model in visual application, a lot of work has been devoted to developing image segmentation methods using deep learning model. SVI benefit from the development of this area a lot (Xiaobaixueshijue, 2021).

The basic concept of semantic segmentation is to label each point of the target category on the image according to the "semantics", so that different kinds of things can be distinguished on the image, which can be understood as the classification task at the pixel level (Xiaobaixueshijue, 2021).

Due to the use of different deep learning model (algorithms) and databases, different semantic segmentation methods will have different accuracy, different semantic classification and of course different results. And in this study, we have tried two different method with different deep learning model and datasets to segment the images. The datasets are ADE_20K dataset and Cityscapes dataset.

ADE_20K has 25,574 images in the training set and 2,000 in the validation set (Zhou, et al., 2019). The annotated images cover the scene categories from the SUN and Places database. Below there are some examples showing the images, object segmentations, and parts segmentations (see Figure 4 4). This dataset includes 150 classes with 100 thing classes and 50 stuff classes. It’s very detailed and rich. However, this dataset includes not only “street” stuff but also some other indoor elements. To some extent, this complexity may reduce the accuracy of street view parts

Cityscapes has 5000 images of driving scenes in urban environment (2975train, 500 Val, 1525test). Cityscapes dataset, namely urban landscape dataset, is a new large-scale dataset, which contains a set of different stereo video sequences, recording street scenes in 50 different cities. It includes 30 classes. Though the amount of images are much smaller than ADE_20K, its advantage is that it focus on the “driving scenes”, on the other words streets, and as a result leading to more concise and accurate results for further analysis.

Anyway, whatever the dataset we use, we have to use a software or a program to complete the process of deep learning, AI training and segmentation. For ADE_20K dataset, we’ve got a software (Yao, et al., 2019) provided by UrbanComp from China University of Geosciences. And for Cityscapes dataset thanks to Ghoshan. He wrote the script for us.

This step is much easier to understand – with the segmetation results, the only thing we are going to do is to assign a color to each class the software identified. Then we can overlay the segmentation images with original images to see the results. Different softwares may have different formats of outputs but this is the basic concept of visualization. And usually the softwares will give a sheet with data telling the proportion of each classes in each picture for further statistics analysis.

This is a very convenient tool. With the free version, you can download single google street view panorama directly by coping the URL. With the paid pro version, multiple download are allowed, and the software provides the function to capture the URLs of the street view panoramas of the geographical area specified by the user at one time. It can even help users filter the street view panoramas taken in a specific period and add eExif metadata (geolocation, elevation, date taken, camera rotation, etc.) to the image.

(1) To Capture URLs

For single panorama, directly open the google map and copy the URL. For multiple panoramas, we must use the area download tool of SVD360 Pro. The interface provide a window of Google map and we can create our polygon to highlight the area we focus.

After the polygon is created, the software will show the diagonal distance and area of the polygon. The third parameter “step distance” is related to the right user setting parameter, which will be discussed in the next part. We made it 0.000375 then 40.5m. And users can choose whether to download outdoor street views only. Click “download” to download URLs.

After the download finishes, we need to export the URLs. Filter can help us to select a certain period of time and we can choose to copy to clipboard or save into a file.

Method 1: Python Script for image cropping, segmentation analysis, visualising, and data extracting

After all of us failed in the local script running, Ghoshan created a shared one via google colab. First it was just a cropping script and after we failed again with the ADE_20K software, he updated it and added segmentation function based on Cityscapes dataset. The following steps are for the latest version. And the website is: https://colab.research.google.com/drive/1qgaiVSnuK0nSoKf4s7HJpRqNqyZu97WG?usp=sharing#scrollTo=MPf1PKK55lcj

Operation Steps

We just need to run these cells one by one and input our panoramas properly. And for the input part first we active the upper cell to import our drive space to this script, and then we can open the side panel to see our files in drive. When the mouse is on a thing there will be a button seems like three dots. Click the button then we can copy the path of the folders.

Paste the path to replace the orange underlined text to make sure the images will be where we want them to be. Then we can change the “mode” to get a certain direction.

Moreover if we want we can choose the deep learning model we are going to apply between “mobilenetv2_coco_cityscapes_ trainfine” and “xception65_cityscapes_trainfine”. According to the reference web in this script, the former has an accuracy of 70.71% and 73.57% with 23MB file size, and the latter has an accuracy of 78.79% and 80.42% with 439MB file size. In our study we just keep it as it was. Finally we will get the cropped images and segmentation images in the set folders and we can download them.

Method 2: Visual Image Semantic Segmentation Software Based on Deep Learning Full Convolution Network (FCN)

The network is a deep learning full convolution network based on ADE_20K dataset training by Professor Guan Qingfeng of the school of information engineering of China University of Geosciences (Wuhan), and the software programming language is C++. The program is provided by the High Performance Computing Laboratory (CUG. HPSCIL).

The pixel contrast accuracy of the network in the training dataset is 0.814426, and that in the test dataset is 0.66839. (Yao, et al., 2019) And this software needs the running environment with Windows 8 / 10 (64 bit), Microsoft Visual C++ 2015/2017 Redistributable and NVDIA CUDA 9. X (recommended).

Operation Steps

The software is a console program opened with Windows notepad or notepad++. We must save our cropped images we get from the former step in “Data” folder – “Image” folder and then run “run.bat”. Then the segmentation images will occur in the “Data”“Image” – “seg_files” folder, and the sheet in “Data” folder.

Limitations

The accuracy of this software is about 66.84%, lower than each of the models the script is applying. And as has been discussed in Chapter 4.3, this software uses ADE_20K dataset to train, which means it doesn’t focus on outdoor street environment and this brings some further inaccuracy.

Even more unfortunately, this software doesn’t work in most of our students’ personal laptops with errors below in Figure 5 9. After reduce the size of images only one of us succeeded and then we turned to the script.

Total images downloaded for different streets through SVD260 Pro software.

(2) https://www.cityscapes-dataset.com

Image segmentation using the aforementioned 2 methods

2.34 Results, categories, meanings, and discussions

According to the report of Image Semantics Segmentation with Cityscapes Dataset, the following seven group of features can be studied individually or in combination with each other for a specific urban issue.

Flat (road, sidewalk, parking, rail track)

Urban Road System and pedestrian network: By applying the SVI segmentation, we obtained the proportions of road, sidewalks, parking and rail track, and by using these data, we can study the road composition and road hierarchy of the area. Amount of sidewalk can also offer the opportunity for us to do research on pedestrian walkway continuity.

Human (person, rider)

The location and number of humans on the image is not a flow statistic, but it can be used as a reference for other studies. The data suggests that the different road sections and compositions are attractive to the crowd and the preliminary information on crowd usage.

Vehicle (car, truck, bus, on rails, motorcycle, bicycle, caravan, trailer)

Traffic noise & traffic pollution: High traffic noise loading zones are affected by all road-related factors, i.e., total road traffic, above-ground metro, transit and rail traffic (Honold, et al., 2012). Urban pollution caused by excessive traffic flow has always been a hot issue in cities. Amount of vehicle can provide preliminary information on the traffic flow and the transport diversity on the roads in this area.

Construction (building, wall, fence, guard rail, bridge, tunnel)

Thermal comfort: Appropriate urban geometry has a positive impact on reducing heat island effects and improving pedestrian thermal comfort (Jamei, et al., 2016), of which urban canyon geometry plays a key role in saving up to 30% of energy used in commercial buildings and 19% in residential buildings (Strømann-Andersen & Sattrup, 2011). One of the key parameters in determining canyon geometry is aspect ratio, which is defined as the ratio between the average height (H) of the canyon walls and the canyon width (W) (Oke, 1988). By comparing the proportions of the buildings in the different images, we can get a first idea of the urban canyon geometry of the road.

Object (pole, pole group, traffic sign, traffic light)

Road objects, such as traffic sign, traffic light and pole group, can have an impact on both vehicular traffic and human movement. The configuration and amount of road objects is also used as auxiliary information for the study of traffic problems.

Nature (vegetation, terrain)

Thermal comfort: The difficult issue for urban designers and planners in accommodating a growing population is to reduce cities' vulnerability to future climate change, where the use of vegetation has shown to be effective in reducing heat island effects and improving pedestrian thermal comfort (Jamei, et al., 2016). The research can obtain the on-site data at specific locations by comparing amount and proportion of green space in the images with the Urban Thermal Data, and to examine how the use of vegetation reduce the heat island effects and improve pedestrian thermal comfort. Or based on the current research, we can identify and calculate the amount and proportion of green space in the images to verify that the urban planning and design is using the appropriate vegetation that has a positive effect on the urban thermal environment.

Pollution Vegetation and human health, along with air pollution and human health, are frequently investigated together. Green infrastructure has a further positive effect on human health by mitigating air pollution (Kumar, et al., 2019). We can identify and compute the amount and proportion of green space in the photos based on the existing study to verify that the urban planning and design is employing appropriate vegetation to mitigate urban pollution.

Sky

Thermal comfort: The level of visibility toward the sky is defined by the level of urban obstructions such as buildings and vegetation (Correa, et al., 2012). The SVF is a critical metric for determining the geometry, density, and thermal balance of metropolitan environments. SVF influences pedestrian-level thermal comfort and wind speed to influence pedestrian comfort. The proportion of sky reflects SVF to a certain extent.

2.35 Limitations of the Study

The highlight of this study is the use of big data and machine learning for quality evaluation and overlay analysis with accessibility to assess the quality and micro-renewal potential of streets. At the same time, however, there are limitations to this study due to the data sources and research techniques. On the one hand, the data source is limited by the fact that the streetscape data is captured on the driveway, while the public spaces that people use on a daily basis are mostly combined with pedestrian spaces, so there are deviations between the viewpoints and the actual viewpoints. On the other hand, due to the limitations of the machine learning algorithm and the resolution of the streetscape data, some key elements that also have an impact on spatial quality, such as "landmark skyline" and "cleanliness and maintenance", which have been mentioned in many papers, were not included in this round of research. This can lead to some errors and limitations in the evaluation of quality. In the future, it is planned to introduce new analysis algorithms to gradually address this shortcoming. Again, In addition, the images after segmentation are partially misidentified, resulting in less accurate statistics of the final image data.

List of References

• Barbierato, E., Bernetti, I., Capecchi, I., & Saragosa, C., 2020. Integrating remote sensing and street view images to quantify urban forest ecosystem services. Mdpi.Com.

• Biljecki, F. and Ito, K., 2021. Street view imagery in urban analytics and GIS: A review. Landscape and Urban Planning , p. 104217.

• Chacra, D. A., & Zelek, J. 2017. Fully automated road defect detection using street view images. Ieeexplore.Ieee.Org.

• Gage, R., Wilson, N., Signal, L., & Thomson, G. 2019. Shade in playgrounds: findings from a nationwide survey and implications for urban health policy [Springer Verlag]. In Journal of Public Health (Germany) 27(5).

• Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V. and Garcia-Rodriguez, J., 2017. A review on deep learning techniques applied to semantic segmentation. arXiv preprint arXiv:1704.06857.

• Gebru, T., Krause, J., Wang, Y., & Chen, D., 2017. Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. National Acad Sciences.

• Gong, Z., Ma, Q., Kan, C., & Qi, Q., 2019. Classifying street spaces with street view images for a spatial indicator of urban functions. Mdpi.Com..

• Gorgul, E., Chen, C., Wu, K. K., & Guo, Y. 2019. Poster: Measuring Street Enclosure and Its Influence to Human Physiology through Wearable Sensors. Dl.Acm.Org, pp. 65-68.

• Hu, C., Zhang, F., Gong, F., Ratti, C., & Li, X. 2020. Classification and mapping of urban canyon geometry using Google Street View images and deep multitask learning.. In Elsevier.

• J. Long, E. Shelhamer, and T. Darrell, 2015. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 3431–3440.

• Kato, Y., & Matsukawa, S., 2019. Development of generating system for architectural color icons using Google map platform and Tensorflow-segmentation..

• Law, S., Paige, B., & Russell, C. 2019. Take a look around: Using street view and satellite images to estimate house prices. ACM Transactions on Intelligent Systems and Technology, 10(5).

• Li, X., Zhang, C., Li, W., Ricard, R., & Meng, Q., 2015. Assessing street-level urban greenery using Google Street View and a modified green view index. Elsevier.

• Mahabir, R., Schuchard, R., Crooks, A., Croitoru, A. and Stefanidis, A., 2020. Crowdsourcing street view imagery: a comparison of mapillary and OpenStreetCam. ISPRS International Journal of Geo-Information, 9(6), p. 341.

• Min, W., Mei, S., Liu, L., & Wang, Y., 2019. Multi-task deep relative attribute learning for visual urban perception. Ieeexplore.Ieee. Org.

• Miranda, F., Hosseini, M., Lage, M., Doraiswamy, H., Dove, G., & Silva, C. T., 2020. Urban Mosaic: Visual Exploration of Streetscapes Using Large-Scale Image Data. Conference on Human Factors in Computing Systems - Proceedings.

• Steinmetz-Wood, M., El-Geneidy, A., & Ross, N, 2020. Moving to policy-amenable options for built environment research: the role of micro-scale neighborhood environment in promoting walking. Elsevier.

• Wang, R., Liu, Y., Lu, Y., Yuan, Y., Zhang, J., Liu, P., & Yao, Y., 2019. The linkage between the perception of neighbourhood and physical activity in Guangzhou, China: Using street view imagery with deep learning techniques. International Journal of Health Geographics, 18(1).

• Xiao, J. and Quan, L., 2009. Multiple view semantic segmentation for street view images.. In 2009 IEEE 12th international conference on computer vision 9, pp. 686-693.

• Yin, L., Cheng, Q., Wang, Z., & Shao, Z., 2015. ’Big data’for pedestrian volume: Exploring the use of Google Street View images for pedestrian counts. Elsevier.

• Zhou, H., He, S., Cai, Y., Wang, M., & Su, S. (n.d.). , n.d. Social inequalities in neighborhood visual walkability: Using street view imagery and deep learning technologies to facilitate healthy city planning. Elsevier.

• Correa, E., Ruiz, M., Canton, A. & Lesino, G., 2012. Thermal comfort in forested urban canyons of low building density. An assessment for the city of Mendoza, Argentina. Build Environ, 58, pp. pp. 219-230.

• Herzog, T. R., Black, A., Fountaine, K. & Kno, 1997. Reflection and attentional recovery as distinctive benefits of restorative environments. Journal of environmental psychology, 17(2), pp. pp.165-170..

• Honold, J., Beyer, R., Lakes, T. & van der Meer, 2012. Multiple environmental burdens and neighborhood-related health of city residents. Journal of environmental psychology, 32(4), pp. pp.305-317.

• Jamei, E., Rajagopalan, P., Seyedmahmoudian, M. & Jamei, Y., 2016. Review on the impact of urban geometry and pedestrian level greening on outdoor thermal comfort.. Renewable and Sustainable Energy Reviews, 54, pp. pp.1002-1017.

• Kumar, P. et al., 2019. The nexus between air pollution, green infrastructure and human health. Environment international, 133, pp. p.105-181.

• Oke, T., 1988. Street design and urban canopy layer climate. Energy Build, 11 pp. pp. 103-113.

• Strømann-Andersen, J. & Sattrup, P., 2011. The urban canyon and building energy use: Urban density versus daylight and passive solar gains. Energy and buildings, p. 43(8).

• Xiaobaixueshijue, 2021. A survey of semantic segmentation based on deep learning (in Chinese). [Online]

• Available at: https://blog.51cto.com/u_14411234/3110792?b=totalstatistic

• [Accessed 01 04 2022].

• Xiaobaixueshijue, 2021. Notes | deep understanding on deep learning semantic segmentation (in Chinese). [Online]

• Available at: https://blog.51cto.com/u_14411234/3112863?b=totalstatistic

• [Accessed 01 04 2022].

• Yao, Y. et al., 2019. A human-machine adversarial scoring framework for urban perception assessment using street-view images. International Journal of Geographical Information Science, 33(12.), pp. 2363-2384.

• Zhou, B. et al., 2019. Semantic Understanding of Scenes through ADE20K Dataset. International Journal of Computer Vision, 127(3), pp. 302-321.

Spatial analyses Part 3

For further details please get in touch with:

• Alain Chiaradia (alainjfc@hku.hk)

• Siddharth Khakhar (khakhar@hku.hk)

Spacial Analysis :

An analysis on the spatial distribution of activities on a selected piece of urban fabric. The analysis focuses on the exploration of landuse, territorial organization ,linkages and connectivity in pedestrial and transportational networks between them; with the knowledge of social , economical and eco factors. Spacial Analysis aims to co-ordinate and impact spacial plannings, by providing a possible soltion of sustainable development.

Network Distance :

Network distance is the span of the shortest path (measureed in m) , measured between any two locations on a spacial network. The shortest path is calculated based on the travel distance or travel time of the spacial network.

Buffer Distance :

Buffer distance is the measure of uniform distance (measured in m ) within a transitional area (the buffer) of land between two zones. This involves measuring distance outward in directions from an object. Buffering can be done on all three types of vector data: point, line, area.

Effective Density:

Effective Density is the concept of relationship between mass and mobility. In Urban studies, it is often used to describe the intensity of certain acivity on a network within a defined unit of area on the urban fabric.

Network Analysis Diagrams

Network Analysis of a building : Exploring from A polygon

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer.The distance is measured from a point of interest, in this case a selected school -Baptist Lui Ming Choi Secondary School. The school is located within an area of housing estate complexs, the diagram shows the school is also in close proximity of other schools and community infrastrctures in the area.

Network Analysis Diagrams

Network Analysis on subway exits: Exploring from points

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer from the Tai Wai MTR Station Exits , all exit points are taken into account in this analysis.

The distance is measured from the 7 exists of Tai Wai MtR Station.As shown in the diagram , the exits are all closely located to the main station, therefore they are centered vantage points that allow access to the Tai Wai community, that consists mainly of housing estates and industrial buildings.

Network Analysis Diagrams

Network Analysis on a housing complex : Exploring from an area

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer from the Wo Che Housing Estate.

The distance is measured from the Wo Che Estate Housing complex boundary, therefore this analysis is done based on a larger coverage of area . As shown in the diagram ,Wo Che Estate is in close proximity with other housing estates such as Lek Yuen Estate , and a number of schools.

Network Analysis Diagrams

Network Analysis on a housing complex : Exploring from an area

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer from the Wo Jat Ming Chuen.

The distance is measured from the Jat Ming Chuen Housing complex Boundary therefore this analysis is done based on a larger coverage of area . As shown in the diagram , Jat Ming Chuen is in close proximity with another public housing estate-Sha Kok Estate , a number of schools, markets and private housing estates.

Network Analysis Diagrams

Network Analysis on a a footbridge : Exploring from linear networks

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer from a foorbridge in Shatin. The distance is measured from the footbridge that connects from Shatin Market to Wo Che Estate through Lek Yuen and Fung Wo Estate . This analysis comparitively shows a much higher spacial coverage. As the footbridge itself provides a linear and horizontal link to spaces, the service area analysis shows the pedestrian service area to be developing in a vertical manner, spanning from the footbridge.

Network Analysis Diagrams

Network Analysis on a a footbridge : Exploring from linear networks

This network analysis explores the span in the pedestrian network within a service area of a 500m buffer from a foorbridge in Shatin.

The distance is measured from the footbridge that connects from Jat Ming Chuen to Sha Kok Estate . This footbridge provides linkage in a more ‘closed off’ and circular manner. As a result, the service area analysis shows the pedestrian service area to be developing in a ‘web-like’ manner, spanning from the footbridge as a center point, linking to areas outside of the footbridge perimeter.