Overall Equipment Utilisation (OEU) Monitoring and Remote Quality Check in Legacy Machine with Raspberry Pi

Overall Equipment Utilisation (OEU) plays an important role as a benchmark for manufacturing companies to determine each machine's efficiency. Currently, there is no proper OEU measurement system in legacy machines and only relies on human observation. This project aims to develop a measurement of OEU system by using Optical Character Recognition (OCR). An efficient Optical Character Recognition (OCR) algorithm is needed to have a high percentage of recognition rate. The outcome of this project will be a Graphical User Interface (GUI) that display realtime OEU monitoring and remote quality check for legacy machines. PytesseractOCR Version 4 classifier using the Recurrent Neural Network (RNN) method has been proposed in this paper. Furthermore, an error detection feature is designed from OCR output. This is an open access article under the CC BY-SA license


INTRODUCTION
In most manufacturing companies, the measurement of Overall Equipment Efficiency (OEE) is widely used to analyse the percentage of machine productivity. OEE has been a crucial key strategy to maximise production time and minimise machine downtime [1] [2]. This evaluation becomes a benchmarking performance for production planning to reduce their losses time. Losses of time might be caused by machine error, repair delay, production test, preventive maintenance and many more [3]. The critical factor of OEE is Availability, Performance and Quality. Figure 1 shows how OEE can be measured [4] [5].
In Texas Instrument, an overall Overall Equipment Utilization (OEU) is used to get equipment productivity. There are some differences between OEE and OEU in terms of formula. OEU is equal to Availability which can be calculated as (1).
In order to determine OEU for tools, a communication protocol called Semiconductor Equipment Communication Standard and Generic Equipment Model (SECS/GEM) has been implemented worldwide as it allows manufacturers to monitor machine state remotely. However, this will be challenging for the company's own legacy machine that does not have SECS/GEM protocol.

MATERIAL AND METHOD
Many text detection methods have been proposed and can be categorised based on several machine learning methods. These methods include Artificial Neural Network (ANN) [6], Support Vector Machine (SVM) [7], K Nearest Neighbor [8], and. Template Matching [9].

Artificial Neural Network (ANN)
ANN is one of the OCR algorithms used to classify the character in images. ANN is used for printed and typewritten character recognition [10]. The suitability of ANN as intelligent learning lies in its ability to demonstrate the human brain that contains millions of complex neurons [11]. The neural network model is called feed-forward as the processing is in the forwarding path only. Figure 2 shows the neural network model consisting of 3 layers: input images, hidden layer, and output text. The hidden layer is the main data processing connecting between input and output. The neurons in the hidden layer trained by the training network represent OCR.

Support Vector Machine (SVM)
SVMs principle is to separate linear or non-linear hyperplane with a maximum margin between two classes in feature space by projecting the input data into higher dimensional [12]. There are two types of mapping in feature space: linear and non-linear, as shown in Figure 3.

K Nearest Neighbor
K-Nearest Neighbor Classifier (k-NN) is a distance-based classification method that computes a Euclidean distance between testing data with all data samples of a training set [15]. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbor. The Euclidean distance D between two feature vectors X and Y is: where and are elements of X and Y, respectively.

Template Matching
Template matching works by comparing two images where the sub-image called templated is inside an image [16]. The centre of the template is used as a reference point. The template will be a data set of characters that will be saved and compared to input characters. It will return the text output if there is a similarity between the input character and template. However, template matching only work with typewritten character recognition as the font already have templates.

System Design Analysis Experimental Setup
The pilot machine involved in this project is the die bond legacy machine model ESEC 2008, as shown in Figure 4. With the help of technicians in Texas Instruments Sdn Bhd, the hardware component is set up behind the machine. This board installation was done during machine setup time and did not interfere machine's operation on a wafer. Figure 5 shows how this system architecture is designed and operated. Starting from Raspberry Pi, the frames from the machine will be grabbed and captured using a USB capture card before sending for image processing on OCR. OCR output log file will be sent to central PC through private WIFI modem. After processing the data, the operator can only monitor multiple machine frames and OEU summaries through central PC. Hence, it can reduce human resources since only one operator is needed to monitor multiple machines. Figure 6 shows the circuit design of the hardware component. The power supply for raspberry pi is taken from a 24V DC machine. In order to power up the video capture card, 5V DC is consumed from the nearest plug. HDMI video capture card will grab the images from machine's screens 1 and 2 taking place from some adapters available in the machine. The image then will be stored and processed inside Raspberry Pie before the OCR result is sent to the host PC.

Software and Library
The compiler system used in this project is Python 3.8. It has been used for image processing in Raspberry Pie central PC data processing. The Thonny tool and Spyder (Anaconda) tool were used in Raspberry Pie and Central PC.
One of the OCR engines is Tesseract. In Python, the wrapper for OCR-Tesseract is the Pytesseract library which is used in this project. Pytesseract is used to read text from images extracted from the machine monitor. This processing is done in raspberry pi for every second. For image processing starting from capturing the image using a video capture card and cropping is using OpenCV (Open-Source Computer Vision Library). For text processing, Pandas and Regex (Regular Expression) libraries are used to sort the extracted data and calculate OEU. This process is done on a central PC. Lastly, to design Graphical User Interface (GUI) framework, Tkinter (Tk interface) library is used while the Matplotplib function is used to create a chart to display OEU in summary.

Data Training
The software used for OCR training is JtessBoxEditor, requiring a Java Runtime environment. JtessBoxEditor is an open-source trainer for Tesseract OCR. Tesseract uses template matching in pixels, consisting of 2 step classification [17]. The first is the adaptive recognition step, whereas the character pixel matches completely with the data set. The second step is when the character does not match with any data set, it will recognise again and match with the character with the highest percentage of similarity pixel.
The editor in this software will be in a box file where each alphabet or number must be checked whether is it correct or not. Each box will save the x and y coordinate information at the top left edge, width and height. There are some criteria needed to generate the box. Firstly, the source image must be excellent and high resolution with at least 300 DPI (dots per inch). Next, the character must be fit entirely inside the box. Lastly, the spacing between characters must be adjusted to avoid box overlapping issues. These criteria must be followed to ensure OCR can accurately recognise the character. Figure 7 shows the drawing box process.  Figure 8 shows the circuit design currently applied on the site to collect images data through the USB camera. Figure 9 shows the data consisting of OCR output (machine status) with a timestamp in the text file. The data was then analysed in an Excel file to determine OEU. The script was designed to extract the count bin from OCR output. Usually, if the machine operates, the count bin will be decreasing. The condition is when the current count bin is less than the previous count bin, then the current time minus the previous time. Hence, the machine run time can be determined. If the count bin is not detected (downtime), it will show the machine status error. Further analysis is being done using keyword matching to detect the causes of downtime. From the data, a pie chart can be created to show OEU results.

OCR Accuracy
The equation for OCR accuracy is: Table 1, Table 2, Table 3 and Table 4 show recognition rates for different machine statuses. Each machine status is divided into three datasets, whereby each dataset has ten samples.

Discussion
Based on Table 1, "The N2H2 pressure is low" has an accuracy of 100% for all data set. This can be analysed that OCR is able to read this machine status accurately from an image. Based on Table 2, the accuracy for "Map Display Reference" is less than 80% for all datasets. The noise on the image might be one of the causes of a lower accuracy rate. Hence, more image processing steps need to be done to filter the noise. Moreover, more training data need to be done on images that contain this machine status. Based on Table 3, "Upload map successful" has more than 80% accuracy for datasets 2 and 3. The classifier is quite unstable as the first dataset has poor performance (<80%), but for the second and the third dataset, it has good performance (>80%). For the first dataset, the OCR cannot read character 'U' correctly. This might be confusing for the classifier to determine whether it is "U "or "J ". Therefore, more training font needs to be done on images containing this machine status. Table 4 show the error percentage for the machine status and its sample errors. The mismatching word can be seen when some alphabet is wrongly classified. For "References", sometimes it classifies the small letter as a big letter. However, the classifier can read "References" correctly for some data sets. For "Cassette" and "Wafer", only one letter was mismatched, but the error percentage was 100% as it read wrongly in all data set. This error might be because the classifier is unstable and needs to train more images containing these machines' status.

CONCLUSION
These text recognition objectives with OCR for semiconductor manufacturing applications have been successfully fulfilled and delivered. A comprehensive literature review on the application and method of OCR has been discussed. The literature reviews it, including the importance of the manning ratio in the ergonomic aspect. For this project, Tesseract 4 classifier p-ISSN: 2774-602X e-ISSN: 2774-6038 was successfully applied in Raspberry Pi for OCR processing. Data were collected from the machine's screen, and OCR is being done in Raspberry Pi 4. The collected data undergo some image processing steps such as converting to grayscale and cropping before OCR. The OCR output is then sent to the host PC for data analysis, OEU calculation, and GUI development. GUI has been successfully developed in executable file format The limitation of this project is the accessibility of raw data collection. For example, having a text recognition project requires a lot of data from a machine for classifier training and testing. For this type of machine (ESEC 2008), there are 1475 machine status or errors that need to be classified by OCR. Hence, for the OCR accuracy to be above 80% for every machine status is quite difficult. Furthermore, because of the pandemic Covid-19, entering Texas Instruments Sdn. Bhd. is not allowed. Hence, limited data that can be accessed is used for this project.