Engineering Quarterly VOL.VI. . . No.2
H AROLD H AMM B OOK S IGNING
Harold Hamm visited campus on October 6, 2023 to meet the students and sign copies of his new book, Game Changer, an autobiographical account of the progress made in the oil & gas industry over the past three decades which was brought about by the discovery of horizontal drilling and hydraulic fracturing.
WINTER 2024
America, his thoughts on the value of education, and his advice for our engineering students as they embark on their careers. His advice for students was find your passion and pursue it wherever it leads and, with a lot of hard work and a bit of luck, you will find success.
N EW E NVIRONMENTAL E NGINEERING D EGREE By ASSAL HADDAD
During the book signing the students got a free copy of the book and had the opportunity to have their picture taken with Mr. Hamm. The faculty of the Hamm School of Engineering also took the opportunity to get a group photo with Mr. Hamm. After the book signing there was a fireside chat in Arno Gustin auditorium where Senator Kevin Cramer interviewed Mr. Hamm about book, his thoughts on the future of energy in
The Hamm School of Engineering is a rapidly growing academic body that is firmly resting on the shoulders of a number of professors of various engineering backgrounds. That is one of its appealing characteristics which not only provides a multi-disciplinary education and experience to students, but also allows us to easily grow in any direction that fills a void in the market and industry. The recent boom in the oil industry in North Dakota, coupled with an inevitable population growth and stress on local resources, demand conscious environmental decisions that allow this prosperous development to grow within the natural laws of sustainability. The Since its inception, the Hamm School of Engineering noticed the void in the market but intentionally waited to start an Environmental Engineering
University of Mary
FREE
program until the ABET accreditation of its civil, mechanical, and electrical engineering programs to establish a clear path and vision for the birth of this new program. The Environmental Engineering program at the Hamm School of Engineering was opened for student enrollment at the beginning of Fall semester in 2023. It is a natural extension of the Civil Engineering program where students are exposed to the basics of mechanics and hydrology and it is also linked to some of the science programs at University of Mary to allow students to learn about microbiology and organic chemistry from an abstract point of view before delving into the engineering use of these scientific equations and concepts. The Environmental Engineering program is designed to grow and breathe inside the multi-disciplinary atmosphere cherished at the Hamm School of Engineering, where design project courses deliberately group engineering students from various backgrounds together to work in teams, and where basic fluid mechanics and thermodynamics classes are taken by civil, mechanical, and Environmental Engineering students in the same classroom. This was only achievable due to the rich background of the school’s faculty who are able to address all these wide engineering and scientific subjects in different contexts. The Environmental Engineering program
Engineering Quarterly WINTER 2024
VOL.VI. . . No.2
offers courses that address design issues in air pollution, water quality, wastewater treatment, solid waste management, and landfill design. It also establishes a strong mathematical vision for students to model contamination spread, accumulation, and treatment. Upon finishing the course load of the new program students will be able to accurately and quantitatively assess the environmental impacts of any engineering project and will be able to pinpoint the most efficient and economic solution to marginalize those impacts. The University of Mary’s Catholic identity and Catholic teachings of environmental stewardship is evident in each and every one of those courses where it is believed that preserving the environment (locally, nationally and globally) is not merely a scientific endeavor but also a moral and ethical duty.
WAPA S UBSTATION T OUR By FR. ATHANASIUS OWEIS
2
dropower plants operated by the Bureau of B ISMARCK WATER T REATMENT P LANT Reclamation, U.S. Army Corps of Engineers, T OUR and the International Boundary and Water By ASSAL HADDAD Commission. Together, these plants have an installed capacity of 10,504 megawatts. They sell that hydropower to preferred customers On Thursday, October 26th, a number of our such as Federal and state agencies, cities and Engineering students and professors traveled towns, rural electric cooperatives, public util- to the Bismarck water treatment plant for a group tour. The ity districts, irrigation districts and Native tour was given by American tribes. Those wholesale customers, Jim Kershaw, the in turn, provide retail electric service to milWater Plant Superlions of consumers in the West. intendent for the City of Bismarck, and Jarrett Hillius, Water Engineer and Project Manager from HDR Engineering. The tour began with a presentation along with coffee and donuts followed by a walk around the facility. The City of Bismarck Water Treatment Plant has a staff of sixteen. It takes water from the Missouri River, filters it, purifies it, softens it, and The substation in Bismarck takes the power then distributes generated by the Garrison Dam in northern it to over 21,000 North Dakota and sells it to Capital City Elec- metered connectric and Montana-Dakota Utilities, each of tions across Biswhich have substations which are adjacent to marck, Lincoln, the WAPA substation. and South Central Regional Water District. The current treatment capacity of the water treatment plant is 28.6 million gallons per day (MGD) but the City has contracted HDR Engineering to design a new filtration facility which will increase the capacity to 40 MGD. The water distribution system is an elaborate network consisting of 360 miles of water pipe, 4400 Walking among the huge circuit elements in hydrants, 6200 the substation and hearing the crackle and valves, 4 pump buzz of the electricity flowing through it gave stations, and 4 both the students and the professors a feeling sites consisting of awe. It almost felt like we were ants crawlof 8 underground ing on a giant electric circuit board. reservoirs and 3 The Electrical Engineering department elevated tanks that at the Hamm School of Engineering is curprovide approxirently adding a multitude of power systems mately 22 million laboratory modules to its curriculum to better prepare our graduates for careers in the gallons of water storage. The water is taken power industry and we hope that some of the from the Missouri River and first pretreated students who were on the tour will choose to to clarify and remove suspended solids. They work for local and national power companies showed us the pretreatment facility which has such as the Western Area Power Administra- a capacity of 13 MGD.
On Friday, September 29th, a number of our Engineering students traveled to the Bismarck substation of the Western Area Power Administration (WAPA) for a group tour of their substation. The tour was given by Travis Keller who is the Supervisory Electrical Engineer at that substation. The tour was informative and interesting and was enjoyed by everyone. Travis told us about the nation’s transmission system, showed us the huge transformers, circuit breakers, switches, and the two 230 kV and the 115 kV transmission lines connected tion. to that substation. WAPA is one of four power marketing administrations within the U.S. Department of Energy. Their role is to market and transmit wholesale electricity from multiuse water projects across a 15-state region of the central and western U.S. WAPA has more than 17,000 miles of electrical transmission which carries electricity from 57 hy-
The pretreated water is sent to giant softening basins where it is mixed with lime to remove hardness (minerals). The capacity of the the softening basins is 30 MGD. They looked like giant round swimming pools and had an
VOL.VI. . . No.2
intricate system of motors to mix in the lime and to skim off the soft water from the top to return to the pipes. The water is then disinfected and stabilized, meaning chlorine and chloromines are added to remove any viruses and other biologicals. Then the CO2 levels and PH are adjusted (stabilization). Next it is sent through a filtration system consisting of a rapid sand filter and 14 single and dual bay filters to remove the very fine suspended solids that remain.
Engineering Quarterly WINTER 2024 There are 100 lockers that line the main hallway of St. Mary’s Central High School in Bismarck. Every night, the school principal makes sure that all of the lockers are closed so that there will be an orderly start the next morning. One day, 100 prankster students met up before school started in the morning and lined up. The first student then walked down the hallway and opened every locker. The next student follows and closes every second locker starting at the second locker. Then student 3 goes to every third locker (starting with locker number 3) and opens it if it’s closed and closes it if it is open. Student 4 follows by opening every fourth locker if it’s closed and closing it if it’s open. The continues until Student 100 finally goes to the hundredth locker. When the principle arrives later in the morning, which lockers does she find open? First write a C program to simulate the problem so that you know the correct answer. Then devise a mathematical proof of the answer.
After a final stage consisting of a gravity E LECTRONICS P ROBLEMS thickener the water is finally sent in to the By FR. ATHANASIUS OWEIS distribution system consisting of a series of pumps, reservoirs, and elevated storage to provide service to five pressure zones on the Determine the output voltage waveform in network. the figure:
3
At this year’s event we had a great turnout of alumni and seniors attending. Beer and nachos were enjoyed by all and everyone got the chance to catch up on old friends and make some new ones.
BNI C OAL AND M INNKOTA P OWER V ISIT By RODRIGO APARECIDO
BNI Coal and Minnkota Power visited our engineering school on Thursday, November 16th. They set up a table in our lobby and talked to students as they walked by to and from classes.
BNI and Minnkota are interested in hiring interns and graduates and find that this is a very effective way to get to meet them and interview them. The Hamm School of Engineering offers this method for student interaction to our corporate sponsors.
S ENIOR D ESIGN P ROJECTS The faculty and students thoroughly enjoyed the tour of the facility. It was a fascinating example of how engineering meets crucial heath and safety needs of every community across the country.
P ROBLEMS TO TRY Here are some problems from our courses this semester that you might like to try. The solutions can be found later on in this issue. O PEN L OCKERS By TERRY PILLING
Here is a problem written for Popular Mechanics by MIT professor Laura Feiveson.
By JAMES CARRICO AND FR. ATHANASIUS OWEIS
The seniors completed their semester one design projects and gave their group presentations on Friday, December 15, 2023. The mechanical engineering students and On Friday, November 17, 2023 we held electrical engineering students gave joint our annual Fall alumni event on campus. The presentations of their first semester design projects on Friday, December 15, 2023. The Fall alumni event first group consisted of Evan Caroll, Joseph is designed to be Cunningham, Thomas Freas, Andrew Merz, a meet-and-greet and Tylen Small. between our past graduates and our current graduating seniors. The hope is for our students to continue to develop contacts with each other across disciplines and across years and build a network of engineers that can be accessed throughout their future careers.
A LUMNI E VENT
VOL.VI. . . No.2
Their project was to design an assisted bowling device that would allow a wheelchair bound individual to bowl and control the spin on the ball and the direction in which the ball is aimed. The next group consisted of Josephine Bowden, Brenna Jones, Connor Murphy, and Jose Reyes. Their project was to design a vertical axis wind turbine or VAWT optimized to scavenge wind energy along North Dakota freeways from both passing cars and wind gusts.
Engineering Quarterly WINTER 2024 appropriately. This feature is crucial for integrating the server with various low compute thin client devices such as the speaker. Once a command is recognized, it is parsed and an MQTT message is sent to the respective device to execute the desired action. In situations where no specific command is given, the server’s LLM can generate helpful responses or engage in friendly conversation. The server software side contains the following elements: 1) LLM and Neural Network Engine. 2) API for Text-to-Speech (TTS). 3) Vosk STT API. 4) HTTP Request Handling. 5) Smart Device Control. and 6) Fallback for Non-Command Interactions. The server’s performance is supported by its powerful hardware, notably an Nvidia Tesla P40 GPU. The server is shown below:
4
integrates voice control into the core functionality of the device. The intelligence of the controller is facilitated by key components: 1) ESP32 (ESP-WROOM-32). 2) Relay Module (Tongling 5VDC Relay Shield). 3) Breadboard. 4) LEDs in addition to Resistor to limit the current to protect the LED. The controller prototype and enclosure is shown below.
E LECTRONICS I D ESIGN P ROJECTS By FR. ATHANASIUS OWEIS
The Electrical Engineering Students built an AI Home Assistant for their Electrical Senior Design. This was built by John Soupir, Stephanie Benavides and Brian Biver. The project was designed for smart home automation, and bridges an important gap in the existing solutions, as it challenges the conventional cloud-based AI systems and presents a research question of paramount importance: “Can a self-hosted AI home assistant offer The Speaker is the engagement hub of the greater privacy, security, and versatility while AI Home Assistant, allowing the user to conmaintaining the convenience and functionaltrol volume of the speaker as well as speak ity that users have come to expect from tradivoice commands to turn on different switches tional home automation systems?” throughout your house and ask the AI questions that you might have. The Speaker is able to send and receive audio from the AI Server that performs all the AI processing and switch control. The Speaker that the student built contains the following components: ESP32, Speaker, SiPeed Microphone Array, LED Light Ring and buttons . This speaker is shown below:
The students answered this question successfully and they built their own solution, which features a server, a speaker and a controller. Utilizing an ESP32 microcontroller, the controller employs AI voice commands for seamless device control, with an innovative integration of llama, an open and efficient foundation language model. Results demonstrate successful implementation, enabling users to effortlessly command devices through natural language, achieving a high success rate in controlling diverse appliances. This project establishes a foundation for privacy-centric, cost-effective, and locally-operated AI home automation systems, emphasizing the potential for future enhancements and widespread applicability. The server that the students built is equipped to handle complex AI tasks, primarily centered around a large language model (LLM). It is capable of understanding and generating human-like text, making it essential for natural language processing. The server integrates a text-to-speech (TTS) API and Speech-to-Text (STT) API. It is also capable of handling HTTP POST and GET requests, allowing it to receive prompts and instructions from external sources and respond
As for the controller, this smart home controller employs the ESP32’s Wi-Fi capabilities to establish seamless connections with relay-connected devices, such as lamps or fans, through a user-friendly app or web interface. The core of the design, the Arduino program, has undergone enhancements to include AI voice command recognition. The setup function initializes the GPIO pin for relay control, and the loop function now integrates a voice command module. This module interprets natural language commands, allowing users to interact with the controller through spoken instructions. The result is a sophisticated yet user-friendly system that seamlessly
In Electronics I with Lab (EEL321/L), the students are divided into groups where each group can build a certain project with electronic devices and components throughout the semester. The first project was an “LED Connect 4 Board” by students: Megan Achbach, Benedict Brophy, and Gannon Steffes. They chose to design, simulate, and construct a playable electronic Connect Four board. The board’s structure was 3D-printed, and an Arduino Nano was used for the controls.
The physical design of this project was designed and modeled from scratch in SolidWorks. After designing, each component of the model was then 3D printed. These components needed to be sized and implemented precisely to ensure that space was efficiently used. The major goal in designing the structure of the board was to mimic the traditional size and feel of a typical connect four board. There were a few connection methods considered when designing the enclosure of the board: The student researched several methods and finally opted to use the dovetail connection method. The dovetail method was used on each shelf, the face plate, and the front plates of the board. This method allowed movement on only one axis and acted as a puzzle piece connection for each piece and worked extremely well in conjunction with pressure fitting. An image of the dovetail connection method can be seen above. Once the structure was built, there were seven buttons installed, one per column, that operate as how the players “drop” a token. These buttons are set up as active high components such that when the button is pushed, it sends a +5V signal to a digital pin. These
VOL.VI. . . No.2
pins are also connected by a pull-down resistor to ground. To install these buttons, one +5V wire is routed to the buttons which are chained together, and then each button has a wire returning back to the Arduino. This return wire is a wire from a Cat6 Ethernet cable. This wire was selected due to its very thin insulation and the structure’s space limitations. These wires route down the side of the board in channels. The other player inputs come in the form of six potentiometers to control the red, green, and blue values of each player’s color choice. These potentiometers each have a +5V supply and ground connection. Each potentiometer has an analog signal wire returning to the Arduino as well.
Engineering Quarterly WINTER 2024 finally displays sections of this buffer frame, shifting it periodically. Both groups presented their projects to a number of HAMM School of Engineering Faculty and allowed them to test these projects and some faculty had fun playing Connect-4 with the project that the students built.
S OPHOMORE D ESIGN P ROJECTS By ANTHONY GARCIA
The sophomores completed ENR 280, the first of a two-class design series of ENR 280 (Fall) and ENR 281 (Spring). Students were tasked with building a racetrack and remotecontrol cars for a final race that will take place in late April. The racetrack consists of both dynamic and static structures that introduce various obstacles to the race cars.
5
number. I.e. all locker numbers are divisible by 1 and so the first student opens all of them. The even numbered lockers are divisible by 2 and so student number 2 closes all of those. The lockers which are divisible by three are then state-changed by student number 3, and so on. How many factors do the locker numbers have? Well, they each have a factor of themselves and 1. For example locker number 8 has 1 × 8 = 8. However, the 1 opens the locker and the 8 closes it again. In other words, pairs of factors which are different will not change the state of the locker. So the only lockers which will change their state – and therefore be open – are the ones that have an odd number of factors. This will only happen when the locker number is a square. For example, the factors of 16 are 16, 8, 4, 2, 1 and the reason there are 5 factors instead of 6 is because 16 is a square. The state of locker 16 will be changed by student number 1, 2, 4, 8, and finally 16. So it will end up open. On the other hand, locker 15 has factors 15, 5, 3, 1. Student 1 opens it, student 3 closes it, student 5 opens it, and finally student 15 closes it. Since it has an even number of factors it ends up closed. Factors come in pairs and so the only lockers with an odd number of factors will be the ones whose number is a perfect square. Here is a computer program in C which will do a simulation to figure out the correct answer: lockers.c #include <stdio.h> #include <stdlib.h>
int main (int argc, char **argv){ The other project was a Large LED screen if(argc != 2){ that was built by students: Isaac Trefz and fprintf(stderr, "Usage: %s <num lockers>\n", argv[0]); Cole Sebastian. The purpose of this project return 1; } was to create a modular LED screen that can int i,j,N = atoi(argv[1]); be stand-alone or used with other screens to int locker[N+1]; for (i=0; i<=N; i++) locker[i]=0; expand the size of the display. As well as for (i=1; i<=N; i++){ the projection of data such as time, temperfor (j=i; j<=N; j+=i) locker[j]++; } ature, brightness, and any other sensor that for (i=1; i<=N; i++){ works with an ESP32 microcontroller. The if(locker[i]%2) printf("locker[%d] = open\n",i); LED board can use display any sensor data. } Across the two sections, there were 11 teams The students chose a KY-015 temperature and return 0; assembled, each guided by a member of the } humidity sensor as a proof-of-concept. Engineering faculty. In ENR 280, the sophomore students started the design process with some background research and generating Voltage Waveform: some initial designs which they presented at the end of the semester. They will continue 10kΩ Vout = Vout with their respective teams into Spring 2024 10kΩ + 47kΩ where they will complete the designs and ± (4.7V + 0.7V ) manufacturing of their structures and remoteVout = 0.175Vout ± 5.4V controlled cars. ±5.4V Vout = = ±6.55V 1 − 0.175 VU T P = (0.175)(+6.55V ) = +1.15V VLT P = (0.175)(−6.55V ) = −1.15V To drive the individual pixels, the pixel needs to receive 24 bits to display a color at a certain brightness. The students built a new and library from scratch to present the data on this screen. This new library has a 96x84 array that contains the last 96 ASCII characters (the only characters that are possible to display) in order on one axis, and the 84-pixel states S OLUTIONS TO P ROBLEMS required for a 7x12 matrix on the other axis. The program iterates through each letter of the input function and converts the character to it’s ASCII value. The program then creates a Open Lockers: Mathematically, simply note buffer frame of the entire message, which may that the state of each locker changes whenever be larger than the LED board. The function the student number is a factor of the locker
VOL.VI. . . No.2
C REATING YOUR OWN AI By TERRY PILLING
Engineering Quarterly WINTER 2024
6
no consensus on what are the “right” answers minimum” since there may be several minima of a given function and this method will only here. find one of them. The one that it finds may Starting with a given input and given net- not be the global or absolute minimum of the work parameters and then computing what function but it will be a local minimum. comes out the other side is called forward We perform the adjustments to the parampropagation. The way we compute it is by first setting layer zero (t = 0) to the input values eters in the following way: and then we compute X µ (t) for each neuron and all layers in the network, sequentially, un∂C til we reach the output layer. The following is δB µ (t) = −ϵ ∂Bµ (t) the formula that is used for forward propagation ∂C δRνµ (t + 1, t) = −ϵ ν ∂Rµ (t + 1, t) N (t) X X µ (t+1) = σ
Rνµ (t + 1, t)X ν (t) + B µ (t + 1)
ν=0
In my ENR 338: Advanced Engineering Mathematics course this past Fall semester I taught the students how to write their own neural network in C and then train that network to solve given problems. Deep learning Neural networks play a crucial role in artificial intelligence and are a subset of machine learning algorithms. They are a computational model inspired by the structure and functioning of the human brain. In artificial intelligence, neural networks are used to process complex data and learn patterns from it, enabling the system to make decisions, recognize patterns, and perform tasks without being explicitly programmed for each step. A neural network consists of layers, neurons, transition functions, and bias functions and the entire network is a machine that takes inputs and produces outputs. We have defined the inputs as layer t = 0 and the outputs as layer t = T . When you are ‘training’ the neural network you have a desired output that you would like the network to produce. For example, you may have a collection of hand-written numbers, each labeled by the number they are supposed to show. You would put the pictures in as the input and the network would generate numbers as the output. You would then compare the numbers that it generated with the correct answers from the labels to see how well the network did. We will use the special label Y µ to denote the ‘correct answer’ when we are training the network. We will compare the output of the network: X µ (T ) with the correct answer Y µ for each value of µ to see how accurate our network is. The transition and bias functions Rνµ (t + 1, t) and B µ (t) are the adjustable parameters of the network. You adjust them with a set of training data until your network is getting the answers correct to the desired accuracy and then your network is ‘trained’. You can then feed it data for which the answer is unknown and have a certain degree of trust that the answers that come out are the correct ones even though you no longer have a known correct answer to check it with. Exercise 1: GPT-4 has 120 layers and 100 billion neurons. How many parameters does it have? Exercise 2: The human brain has about 86 billion neurons, which is less that that of GPT-4. One might assume that GPT-4 should be smarter than we are. However, in most ways it is not. What are some of the most significant differences between the human brain and GPT-4 that could account for the fact that we are conscious and GPT-4 isn’t? Defend your answers to be convincing since there is
After calculating these changes, we adjust where σ(z) is a particular non-linear function our parameters via of its argument. Now that we have seen how B µ (t) −→ B µ (t) + δB µ (t) forward propagation works we can already write a perfectly working neural network in Rνµ (t + 1, t) −→ Rνµ (t + 1, t) + δRνµ (t + 1, t) that it can take input data and produce output data. However, in order to find the correct where the change is in the direction of the negparameters (i.e. transition functions and bias ative gradient of the cost function with respect functions) which will allow us to solve spe- to the parameter and scaled by the value of the cific problems we need to train the network. learning rate ϵ. We perform this adjustment We use a set of known inputs for which the an- over and over until the cost function has fallen swers are already known. These could be im- below some predefined minimum threshold. ages along with the descriptions of what they Back propagation is the method that we are – such as handwritten numerals and their use to find the negative gradient of the cost digital values, or pictures which may or may function for each of the parameters in the netIt uses the fact the the output values of not contain fire hydrants, etc. We take this work. the neurons are actually functions of all of the data set of solved problems and we feed them parameters in the network and so one can use into our network. Then we compare the out- the chain rule to take the derivatives and the put with the desired answers and, if they are change to any of the parameters is given inductively. To find each layer we move back not the same to within a specified tolerance, through the network. Define A (T − k) as ak we adjust the parameters of the network until they are. ∂X a0 (T ) ∂X a1 (T − 1) ∂C Aak (T − k) ≡ −ϵ To accomplish this training we define a ∂X a0 (T ) ∂X a1 (T − 1) ∂X a2 (T − 2) ∂X ak−1 (T − [k − 1]) Cost Function to compare the output we get ··· ∂X ak (T − k) from the network with the desired output. We then consider the network to be trained when the value of the cost function is minimized then the general formula for layer k is across the training data. Here is an example δB µ (T − k) = Aµ (T − k)X µ (T − k) [1 − X µ (T − k)] of a least squares cost function: δRµ (T − k, T − k − 1) ν
N (T )
1X µ [X (T ) − Y µ ] [Xµ (T ) − Yµ ] C= 2 µ=0 This is a function of the last layer in the network and, for each neuron in the layer, it adds the squared difference with the corresponding value in the desired answer. Next we train the network in tiny increments to achieve smaller and smaller values of the cost function until we reach it’s minimum value. This is called the method of steepest descent, or gradient descent. This method adjusts each of the parameters across the network according to its effect on the cost function and the adjustment is made in the direction of the negative rate of change of the cost function with respect to that parameter. The parameter is adjusted by a tiny fixed amount that we define as ϵ. We call this the learning rate. Note that this is exactly the same thing that we do in normal calculus when we are finding the minimum of a function. We travel in the direction of the negative gradient of the function with respect to position. The gradient of a function points in the direction of fastest increase of the function and so the negative gradient will point in the direction of fastest decrease. If we continue to travel by small amounts in the direction of fastest decrease we will eventually arrive at a local minimum of the function. Note that we say “local
= Aµ (T − k)X µ (T − k) [1 − X µ (T − k)] Xν (T − k − 1)
and now we can start at k = 0 for the output layer, and increment to k = [T −1] always using the results from the previous computation to get the next until all of the parameters are updated. After we have updated all of the parameters, we then run the forward propagation again with the new parameters, compare the result with our training data, and if the result is still too far away from the training data we do another back propagation to update the parameters again. We continue in this fashion, over and over, until the output of the network matches the desired output to within our specified error. In order to train the network on a large collection of training data we feed questionanswer pairs in, one at a time, modifying the parameters of the network for each one with the hope that, if we have enough layers and enough neurons, the parameters will converge to a solution of the ‘meta problem’ that we are trying to solve. For example suppose we would like to design a neural network that acts like an XOR gate. This means that when we input a (1, 0) or a (0, 1) the output should be 1, but if we input a (0, 0) or a (1, 1) the output should be 0. Here is a C program that creates an XOR gate neural network and tests it.
VOL.VI. . . No.2 xor.c #include <stdio.h> #include <stdlib.h> #include <math.h> #include <time.h> #define EPS 0.2 #define ERROR 0.00001 double sigma(double x) return 1/(1+exp(-x)); int main (int argc, char **argv){ if (argc != 3){ fprintf(stderr,"Ex: %s 4 2\n", argv [0]); fprintf(stderr,"4 neurons/layer\n"); fprintf(stderr,"2 hidden layers\n"); return 1; } srand(time(NULL)); int N = atoi(argv[1]); int T = atoi(argv[2]); int mu, nu, t=0, cycles=0; double Y[N],cost=ERROR+1; double R[N][N][T+1],B[N][T+1]; double X[N][T+1],Z[N][T+1],dB[N][T+1]; double IN[4][N],OUT[4][N]; for(int i = 0; i < 4; i++){ for (mu = 0; mu < N; mu++){ IN[i][N] = 0.001; OUT[i][N] = 0.001; } } // inputs + desired outputs for XOR // Only care about first two neurons // Output: all 1’s is a 1 // all 0’s is a 0 IN[0][0]=1; IN[0][1]=1; OUT[0][0]=0; IN[1][0]=1; IN[1][1]=0; OUT[1][0]=1; IN[2][0]=0; IN[2][1]=1; OUT[2][0]=1; IN[3][0]=0; IN[3][1]=0; OUT[3][0]=0; // randomly initialize for (t = 1; t <= T; t++){ for (mu = 0; mu < N; mu++){ B[mu][t] = (double) (rand()%100) /100.0; for (nu = 0; nu < N; nu++){ R[mu][nu][t] = (double) (rand() %100)/100.0; } } } while( cost > ERROR ){ // loop over training data for(int i = 0; i < 4; i++){ for (mu = 0; mu < N; mu++){ X[mu][0] = IN[i][mu]; Y[mu] = OUT[i][0]; } // forward propagate for (t = 1; t <= T; t++){ for (mu = 0; mu < N; mu++){ Z[mu][t] = B[mu][t]; for (nu = 0; nu < N; nu++){ Z[mu][t] = Z[mu][t] + R[mu][nu][t ]*X[nu][t-1]; } X[mu][t] = sigma(Z[mu][t]); } } // back propagate // k=0 layer for (mu = 0; mu < N; mu++){ dB[mu][T] = -EPS*(X[mu][T]-Y[mu])*X [mu][T]*(1-X[mu][T]); B[mu][T] = B[mu][T] + dB[mu][T]; for (nu = 0; nu < N; nu++){ R[mu][nu][T] = R[mu][nu][T] + dB[ mu][T]*X[nu][T-1]; } } // k = 1...T-1 layers for (int k = 1; k < T; k++){ for (mu = 0; mu < N; mu++){ dB[mu][T-k] = 0; for (int a = 0; a < N; a++){ dB[mu][T-k] = dB[mu][T-k] + dB[a ][T-k+1]*R[a][mu][T-k+1]*X[mu][T-k ]*(1-X[mu][T-k]); } B[mu][T-k] = B[mu][T-k] + dB[mu][T -k]; for (nu = 0; nu < N; nu++){ R[mu][nu][T-k] = R[mu][nu][T-k] + dB[mu][T-k]*X[nu][T-k-1]; } } } // calculate cost function cost = 0; for (mu = 0; mu < N; mu++){ cost = cost + (X[mu][T]-Y[mu])*(X[ mu][T]-Y[mu]); } cost = 0.5*cost; // increment back propagations cycles++; } // end training data loop } // end cost while loop // The network should be trained. // print out values of R and B fprintf(stdout,"\nThe trained network parameters\n");
Engineering Quarterly WINTER 2024
7
for banks as well as a multitude of other applications. There is a free data set, called MNIST, that can be found online containing 50,000 images of handwritten numerical digits labeled by the number that the image is showing. Write a neural network that takes the sequence of greyscale pixels from the images – i.e. each pixel has a number assigned to it depending on how light or dark it is, these num// Test the trained network bers can be lined up into a long vector and fprintf(stdout,"\nTesting the trained then used as input to the neural network. The network...\n"); for(int i = 0; i < 4; i++){ output of the neural network is a vector Y µ for (mu = 0; mu < N; mu++){ with ten components. So, for example, if the X[mu][0] = IN[i][mu]; Y[mu] = OUT[i][0]; digit is a 0 then the output will have a 1 in the } zeroth component and zeros everywhere else, for (t = 1; t <= T; t++){ for (mu = 0; mu < N; mu++){ if the digit in the picture is a 1 then the output Z[mu][t] = B[mu][t]; should have a 0 in the zeroth component, a 1 for (nu = 0; nu < N; nu++){ Z[mu][t] = Z[mu][t] in the first component, and zeros everywhere + R[mu][nu][t]*X[nu][t-1]; else. } X[mu][t] = sigma(Z[mu][t]); Note that there are fewer output neurons } } // print out results than input neurons so you will have to modify fprintf(stdout,"\nINPUT\t\tOUTPUT\n"); your code to allow different number of neufprintf(stdout,"-----\t\t------\n"); for (mu = 0; mu < N; mu++){ rons in different layers (at least the last layer) fprintf(stdout,"X[%d][0] = %0.2lf\tX or else simply pad the output vector with zeros [%d][%d] = %0.2lf\n", mu, X[mu ][0], mu, T, X[mu][T]); after the first ten. } } You should be able to train your network return 0; } with the first 40,000 images in the data set and then test it with the remaining 10,000 to see if The result of running the code with 2 neu- it gives the correct answers – or at least derons per layer and 3 hidden layers is as fol- cide what percent accuracy it has. You should lows: then try different numbers of layers and neurons per layer to see if you can improve your terry@andromeda:Code$ gcc xor.c -lm networks accuracy. terry@andromeda:Code$ ./a.out 2 3 Electrical Engineering student, John The trained network parameters Soupir, solved the Project above, essenafter 848820 back propagations: tially creating a e-banking AI that can read the writing on a picture of a hand writB[mu](1) R[mu][nu](1,0) ten check. You can find his complete program on GitHub. To download John’s deep |-2.642 | | 6.273 6.358 | |-6.080 | | 3.967 3.981 | learning network from Github use: git clone https://github.com/johnsoupir/neural-reader fprintf(stdout,"after %d backpropagations:\n", cycles); for (t = 1; t <= T; t++){ fprintf(stdout,"\nB[mu](%d)\t R[mu][ nu](%d,%d)\n", t, t, t-1); for (mu = 0; mu < N; mu++){ fprintf(stdout,"\n|%6.3lf |\t|", B[ mu][t]); for (nu = 0; nu < N; nu++){ fprintf(stdout," %6.3lf", R[mu][nu ][t]); } fprintf(stdout," |"); } fprintf(stdout,"\n"); }
B[mu](2)
R[mu][nu](2,1)
|-3.814 | | 1.309 |
| 8.082 -8.717 | | -2.163 2.269 |
B[mu](3)
R[mu][nu](3,2)
|-3.507 | |-3.670 |
| 11.120 -3.470 | | 11.191 -3.247 |
Testing the trained network... INPUT ----X[0][0] = 1.00 X[1][0] = 1.00
OUTPUT -----X[0][3] = 0.00 X[1][3] = 0.00
INPUT ----X[0][0] = 1.00 X[1][0] = 0.00
OUTPUT -----X[0][3] = 1.00 X[1][3] = 1.00
INPUT ----X[0][0] = 0.00 X[1][0] = 1.00
OUTPUT -----X[0][3] = 1.00 X[1][3] = 1.00
INPUT ----X[0][0] = 0.00 X[1][0] = 0.00
OUTPUT -----X[0][3] = 0.00 X[1][3] = 0.00
which we can see acts exactly like a XOR gate for all of the possible inputs. Project: Classifying hand-written digits is often called the ‘hello world’ program for neural networks. It is used in mobile deposits
Contact Information For more information about our engineering program please feel free to email us at Engineering@umary.edu The digital version of this issue as well as all previous issues can be found at https://issuu.com/ umary-engineering Next issue: April 2024
VOL.VI. . . No.2
Engineering Quarterly WINTER 2024
8