Compare commits

...

3 Commits

Author SHA1 Message Date
cf203978c6 Waiter group
Report from my individual part
2020-05-04 06:34:26 +00:00
74e6baecfa Individual Project #1 implementation; s444523
A Convoultional Neural Network classyfing customers plates into three categories. Documentation inside s444523.rar
2020-04-30 07:42:18 +00:00
08def183f7 Individual Project #1 implementation
CNN classifying images of plates
2020-04-29 19:47:30 +00:00
5 changed files with 529 additions and 0 deletions

View File

@ -0,0 +1,91 @@
# CNN Plates Classification
Author: Weronika Skowrońska, s444523
As my individual project, I decided to perform a classification of plates images using a Convolutional Neural Network. The goal of the project is to classify a photo of the client's plate as empty(0), dirty(1) or full(2), and assign an appropriate value to the given instance of the "Table" class.
# Architecture
Architecture of my CNN is very simple. I decided to use two convolutions, each using 32 feature detectors of size 3 by 3, followed by the ReLU activation function and MaxPooling of size 2 by 2.
```sh
classifier = Sequential()
classifier.add(Convolution2D(32, (3, 3), input_shape =(256, 256, 3), activation = "relu"))
classifier.add(MaxPooling2D(pool_size = (2,2)))
classifier.add(Convolution2D(32, 3, 3, activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
classifier.add(Flatten())
```
After flattening, I added a fully connected layer of size 128 (again with ReLU activation function). The output layer consists of 3 neurons with softmax activation function, as I am using the Network for multiclass classification (3 possible outcomes).
```sh
classifier.add(Dense(units = 128, activation = "relu"))
classifier.add(Dense(units = 3, activation = "softmax"))
```
The optimizer of my network is adam, and categorical cross entropy was my choice for a loss function.
```sh
classifier.compile(optimizer = "adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
```
# Library
I used keras to implement the network. It let me add some specific features to my network, such as early stopping and a few methods of data augmentation.
```sh
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
width_shift_range=0.2,
height_shift_range=0.1,
fill_mode='nearest')
```
This last issue was very important to me, as I did not have many photos to train the network with (altogether there were approximately 1200 of them).
# Project implementation
After training the Network, I firstly saved the model which gave me the best results (two keras callbacks, EarlyStopping and ModelCheckpoint were very useful) to a file named "best_model.h5".
```sh
# callbacks:
es = EarlyStopping(monitor='val_loss', mode='min', baseline=1, patience = 10)
mc = ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True, verbose = 1, period = 10)
```
It occured though, that the file is to big to upload it to git, so I modified the code a little bit, and instead of saving the model, I saved the weights:
```sh
mc = ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True, verbose = 1, period = 10, save_weights_only = True)
```
To be honest, it was not a very good idea either, as the new file is also to big to upload it. I managed to solve the probem in another way: I added the h5 file to my google drive, and added a link to download it to the project files.
To use the saved weights, I created the CNN model inside our project:
```sh
#initializing:
classifier = Sequential()
#Convolution:
classifier.add(Convolution2D(32, (3, 3), input_shape =(256, 256, 3), activation = "relu"))
#Pooling:
classifier.add(MaxPooling2D(pool_size = (2,2)))
# Adding a second convolutional layer
classifier.add(Convolution2D(32, 3, 3, activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
#Flattening:
classifier.add(Flatten())
#Fully connected layers::
classifier.add(Dense(units = 128, activation = "relu"))
classifier.add(Dense(units = 3, activation = "softmax"))
# loading weigjts:
classifier.load_weights('s444523/best_model_weights2.h5')
#Making CNN:
classifier.compile(optimizer = "adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
```
After coming to each table, the Agent (the waiter) evaluates a randomly selected photo of a plate using the provided model, and assigns the number of predicted class into the "state" attribute of a given table. This information will let perform further actions, based on the predicted outcome.
I noticed that my program has difficulties in distinguishing a full plate from a dirty one - interestingly, this was also a problem for me and my friends when we worked as real waiters in the restaurant. Therefore, if the plate is classified by the waiter as dirty, he asks politely if the client already has done eating, and acts accordingly to his answer:
```sh
if result[1] == 1:
result[1] = 0
x = int(input("Excuse me, have You done eating? 1=Yes, 2 = No \n"))
result[x] = 1
```

BIN
plates.rar Normal file

Binary file not shown.

BIN
s444523.rar Normal file

Binary file not shown.

369
waiter_v3.py Normal file
View File

@ -0,0 +1,369 @@
import pygame
import numpy as np
import math
########################
### WS ###
########################
# For CNN:
import keras
from keras.preprocessing import image
from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
#initializing:
classifier = Sequential()
#Convolution:
classifier.add(Convolution2D(32, (3, 3), input_shape =(256, 256, 3), activation = "relu"))
#Pooling:
classifier.add(MaxPooling2D(pool_size = (2,2)))
# Adding a second convolutional layer
classifier.add(Convolution2D(32, 3, 3, activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
#Flattening:
classifier.add(Flatten())
#Fully connected layers::
classifier.add(Dense(units = 128, activation = "relu"))
classifier.add(Dense(units = 3, activation = "softmax"))
# loading weigjts:
classifier.load_weights('s444523/best_model_weights2.h5')
#Making CNN:
classifier.compile(optimizer = "adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
########################
### WS ###
########################
# Colors:
# Define some colors
BLACK = (0, 0, 0)
WHITE = (255, 255, 255)
GREEN = (0, 255, 0)
RED = (255, 0, 0)
BLUE = (0, 0, 240)
#Width and Height of each square:
WIDTH = 20
HEIGHT = 20
#Margin:
MARGIN = 5
grid = [[0 for x in range(16)] for y in range(16)]
def change_value(i, j, width, n):
for r in range (i, i+width):
for c in range (j, j+width):
grid[r][c] = n
# the class "Table"
class Table:
def __init__(self, coordinate_i, coordinate_j, state = 0):
self.coordinate_i = coordinate_i
self.coordinate_j = coordinate_j
self.state = state
change_value(coordinate_i, coordinate_j, 2, 1)
def get_destination_coor(self):
return [self.coordinate_i, self.coordinate_j-1]
########################
### WS ###
########################
# The finction "state of meal" chooses a photo of a plate at the given table.
def state_of_meal(self):
## !!!!!!###
num = np.random.randint(67, 100)
## !!!!!!###
if num<=67:
img_name = 'plates/{}.png'.format(num)
else:
img_name = 'plates/{}.jpg'.format(num)
return img_name
# The function "change state" changes the value of the state variable.
# It informs, whether the client are waiting for the waiter to make an order
# (0 - empty plates) are eating (2 - full plates) or are waiting for the
# waiter to get a recipt (1 - dirty plates)
def change_state(self, st):
self.state = st
########################
### /WS ###
########################
class Kitchen:
def __init__(self, coordinate_i, coordinate_j):
self.coordinate_i = coordinate_i
self.coordinate_j = coordinate_j
change_value(coordinate_i, coordinate_j, 3, 2)
class Agent:
def __init__(self,orig_coordinate_i, orig_coordinate_j):
self.orig_coordinate_i = orig_coordinate_i
self.orig_coordinate_j = orig_coordinate_j
self.state = np.array([1,2])
change_value(orig_coordinate_j, orig_coordinate_j, 1, 3)
self.state_update(orig_coordinate_i, orig_coordinate_j)
self.previous_grid = np.array([-1, -1])
def state_update(self, c1, c2):
self.state[0] = c1
self.state[1] = c2
def leave(self):
change_value(self.state[0], self.state[1], 1, 0)
def previous_grid_update(self):
self.previous_grid[0] = self.state[0]
self.previous_grid[1] = self.state[1]
def move_to(self, nextPos):
self.previous_grid_update()
self.leave()
self.state_update(x + nextPos[0], y + nextPos[1])
change_value(self.state[0], self.state[1], 1, 3)
########################
### WS ###
########################
#The function "define_table" serches for the apropriate table in the
# table_dict (to enable the usage of class attributes and methods)
def define_table(self, t_num):
t_num = 'table{}'.format(t_num)
t_num = table_dict[t_num]
return t_num
# The function "check_plates" uses the pretrained CNN model to classify
# the plate on the table as empty(0), full(2) or dirty(1)
def check_plates(self, table_number):
table = self.define_table(table_number)
plate = table.state_of_meal()
plate= image.load_img(plate, target_size = (256, 256))
plate = image.img_to_array(plate)
plate = np.expand_dims(plate, axis = 0)
result = classifier.predict(plate)[0]
print (result)
if result[1] == 1:
result[1] = 0
x = int(input("Excuse me, have You done eating? 1=Yes, 2 = No \n"))
result[x] = 1
for i, x in enumerate(result):
if result[i] == 1:
table.change_state(i)
########################
### /WS ###
########################
# check the next grid is not the previous grid to prevent the loop
def next_is_previous(self, x, y):
return np.array_equal(self.previous_grid, np.array([x, y]))
def check_done():
for event in pygame.event.get(): # Checking for the event
if event.type == pygame.QUIT: # If the program is closed:
return True # To exit the loop
def draw_grid():
for row in range(16): # Drawing the grid
for column in range(16):
color = WHITE
if grid[row][column] == 1:
color = GREEN
if grid[row][column] == 2:
color = RED
if grid[row][column] == 3:
color = BLUE
pygame.draw.rect(screen,
color,
[(MARGIN + WIDTH) * column + MARGIN,
(MARGIN + HEIGHT) * row + MARGIN,
WIDTH,
HEIGHT])
# calculate the distance between two points
def distance(point1, point2):
return math.sqrt((point2[0] - point1[0])**2 + (point2[1] - point1[1])**2)
## default positions of the agent:
x = 12
y = 12
agent = Agent(x, y)
table1 = Table(2, 2)
table2 = Table (2,7)
table3 = Table(2, 12)
table4 = Table(7, 2)
table5 = Table(7, 7)
table6 = Table(7, 12)
table7 = Table(12, 2)
table8 = Table(12, 7)
################### WS #####################
# I added the dict to loop through tables.
table_dict = {"table1":table1, "table2":table2, "table3":table3,"table4":table4,
"table5":table5,"table6":table6,"table7":table7,"table8":table8
}
################### WS #####################
#class Kitchen:
kitchen = Kitchen(13, 13)
# destination array
destination_tables = []
pygame.init()
WINDOW_SIZE = [405, 405]
screen = pygame.display.set_mode(WINDOW_SIZE)
pygame.display.set_caption("Waiter_Grid3")
done = False
clock = pygame.time.Clock()
# -------- Main Program Loop -----------
while not done:
screen.fill(BLACK) # Background color
draw_grid()
done = check_done()
for value in table_dict.values(): destination_tables.append(value.get_destination_coor())
# We need to define the number of the table by which we are in:
num_of_table = 1
while len(destination_tables) != 0:
# set the first element(table) in array as currDestination
currDestination = destination_tables[0]
# from kitchen to table
while agent.state[0] != currDestination[0] or agent.state[1] != currDestination[1]:
#///////////////////////////////////////
x = agent.state[0]
y = agent.state[1]
# set a huge default number
minDis = 9999
nextPos = []
# check whether the agent goes left
if y-1 >= 0 and grid[x][y-1] != 1 and not agent.next_is_previous(x, y-1):
minDis = distance([x, y-1], currDestination)
nextPos = [0, -1] # means go left
# check whether the agent goes right
if y+1 <= 15 and grid[x][y+1] != 1 and grid[x][y+1] != 2 and not agent.next_is_previous(x, y+1):
d = distance([x, y+1], currDestination)
if d < minDis:
minDis = d
nextPos = [0, 1] # means go right
# check whether the agent goes up
if x-1 >= 0 and grid[x-1][y] != 1 and not agent.next_is_previous(x-1, y):
d = distance([x-1, y], currDestination)
if d < minDis:
minDis = d
nextPos = [-1, 0] # means go up
# check whether the agent goes down
if x+1 <= 15 and grid[x+1][y] != 1 and grid[x+1][y] != 2 and not agent.next_is_previous(x+1, y):
d = distance([x+1, y], currDestination)
if d < minDis:
minDis = d
nextPos = [1, 0] # means go down
# print(agent.previous_grid)
agent.move_to(nextPos)
#////////////////////////////////////////////////
pygame.time.delay(100)
screen.fill(BLACK) # Background color
draw_grid() # Drawing the grid
clock.tick(60) # Limit to 60 frames per second
pygame.display.flip() # Updating the screen
########################
### WS ###
########################
#pygame.time.delay(100)
print("I'm at a table no. {}".format(num_of_table))
## Checking at what state are the plates:
agent.check_plates(num_of_table)
num_of_table +=1
########################
### /WS ###
########################
# set the kitchen as currDestination
currDestination = [13, 12]
# from table to kitchen
while agent.state[0] != currDestination[0] or agent.state[1] != currDestination[1]:
#///////////////////////////////////////
x = agent.state[0]
y = agent.state[1]
# set a huge default number
minDis = 9999
nextPos = []
# check whether the agent goes left
if y-1 >= 0 and grid[x][y-1] != 1 and not agent.next_is_previous(x, y-1):
minDis = distance([x, y-1], currDestination)
nextPos = [0, -1] # means go left
# check whether the agent goes right
if y+1 <= 15 and grid[x][y+1] != 1 and grid[x][y+1] != 2 and not agent.next_is_previous(x, y+1):
d = distance([x, y+1], currDestination)
if d < minDis:
minDis = d
nextPos = [0, 1] # means go right
# check whether the agent goes up
if x-1 >= 0 and grid[x-1][y] != 1 and grid[x-1][y] != 2 and not agent.next_is_previous(x-1, y):
d = distance([x-1, y], currDestination)
if d < minDis:
minDis = d
nextPos = [-1, 0] # means go up
# check whether the agent goes down
if x+1 <= 15 and grid[x+1][y] != 1 and grid[x+1][y] != 2 and not agent.next_is_previous(x+1, y):
d = distance([x+1, y], currDestination)
if d < minDis:
minDis = d
nextPos = [1, 0] # means go down
agent.move_to(nextPos)
#////////////////////////////////////////////////
pygame.time.delay(100)
screen.fill(BLACK) # Background color
draw_grid() # Drawing the grid
clock.tick(60) # Limit to 60 frames per second
pygame.display.flip() # Updating the screen
destination_tables = destination_tables[1:] # remove the first element in the list
# After each fool loop, we can quit the program:.
if len(destination_tables) == 0:
play_again = 1
play_again = int(input("Exit? 0=No, 1=Yes \n"))
if play_again:
pygame.quit()
pygame.quit()

69
which_plate_CNN.py Normal file
View File

@ -0,0 +1,69 @@
##My cnn, classyfing the plates as dirty, clean or full.
#imports
from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
#initializing:
classifier = Sequential()
#Convolution:
classifier.add(Convolution2D(32, (3, 3), input_shape =(256, 256, 3), activation = "relu"))
#Pooling:
classifier.add(MaxPooling2D(pool_size = (2,2)))
# Adding a second convolutional layer
classifier.add(Convolution2D(32, 3, 3, activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
#Flattening:
classifier.add(Flatten())
#Fully connected layers::
classifier.add(Dense(units = 128, activation = "relu"))
classifier.add(Dense(units = 3, activation = "softmax"))
#Making CNN:
classifier.compile(optimizer = "adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
#From KERAS:
from keras.preprocessing.image import ImageDataGenerator
#Data augmentation:
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
width_shift_range=0.2,
height_shift_range=0.1,
fill_mode='nearest')
test_datagen = ImageDataGenerator(rescale=1./255)
training_set = train_datagen.flow_from_directory('plates/training_set',
target_size=(256, 256),
batch_size=16,
class_mode='categorical')
test_set = test_datagen.flow_from_directory('plates/test_set',
target_size=(256, 256),
batch_size=16,
class_mode='categorical')
# callbacks:
es = EarlyStopping(monitor='val_loss', mode='min', baseline=1, patience = 10)
mc = ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True, verbose = 1, period = 10)
classifier.fit_generator(
training_set,
steps_per_epoch = 88,
epochs=200,
callbacks=[es, mc],
validation_data=test_set,
validation_steps=10)