Quality Prediction and Classification of Process Parameterization for Multi-Material Jetting by Means of Computer Vision and Machine Learning

Abstract

Multi-Material Jetting (MMJ) is an additive manufacturing process empowering the printing of ceramics and hard metals with the highest precision. Given great advantages, it also poses challenges in ensuring the repeatability of part quality due to an inherent broader choice of built strategies. The addition of advanced quality assurance methods can therefore benefit the repeatability of part quality for widespread adoption. In particular, quality defects caused by improperly configured droplet overlap parameterizations, despite droplets themselves being well parameterized, constitute a major challenge for stable process control. This publication deals with the automated classification of the adequacy of process parameterization on green parts based on in-line surface measurements and their processing with machine learning methods, in particular the training of convolutional neural networks. To generate the training data, a demo part structure with eight layers was printed with different overlap settings, scanned, and labeled by process engineers. In particular, models with two convolutional layers and a pooling size of (6, 6) appeared to yield the best accuracies. Models trained only with images of the first layer and without the infill edge obtained validation accuracies of 90%. Consequently, an arbitrary section of the first layer is sufficient to deliver a prediction about the quality of the subsequently printed layers.

1. Introduction

Following the initial use of additive manufacturing (AM) technology for prototype production, AM technologies have evolved to become suitable for the production of small batches and special parts, thus specifically addressing the needs of niche products [1]. Three-dimensional components are created in layers by depositing or solidifying the required material. This makes it possible to realize highly complex designs that are primarily determined by the function and not by the manufacturing method [2]. While the shaping of polymers using available AM methods is already established, the adaptation to metallic and ceramic material in particular is subject to various challenges. For example, the manufacturing chain for ceramic components is not completed with the generation of a part geometry alone, as this is merely an intermediate step since thermal processing in the form of debinding and sintering follows in order to yield the final material properties.

Multi-Material Jetting (MMJ) is an innovative additive manufacturing process enabling the printing of ceramics and hard metals with the highest precision [3]. The functional principle of MMJ is based on the selective deposition of individual droplets of thermoplastic suspensions filled with ceramic or hard metal particles. Instead of volatile solvents, the powders in MMJ are dispersed in thermoplastic binders with selected additives, as known from low-pressure injection molding [4]. This implies advantages over the related additive methods. Depending on the choice of binder components used, processing is possible in a temperature range between 70 and 100 °C while maintaining low viscosity (<10 Pa-s at 100 s⁻¹) and comparatively high solid content levels (up to 67 vol%) [5,6,7]. Through the combination of different materials, property combinations such as electrically or thermally conductive–insulating or hard–tough can be realized within one component. Porosity and thus property gradients as Functional Graded Materials (FGMs) within one single part can also be realized by using thermoplastic ceramic suspensions through adding different proportions of pore-forming agents to the various suspensions and removing them during debinding, as has already been accomplished using multilayer technology, for instance [8,9,10].

By principle, MMJ as an additive manufacturing process allows the highest degree of freedom in component geometry along with the cost-efficient production of flexible quantities, but while its flexibility is a great advantage for innovative manufacturers, it also poses a challenge in preventing shortcomings in repeatability of part quality due to the inherent broader choice of built strategies. Typical quality shortcomings in additive manufacturing processes are insufficient shape and dimensional accuracy, undesirable porosity, and low process stability [11]. In this respect, quality control plays an important role in further development. Because manual quality control is inefficient, costly, and prone to errors due to subjective bias in evaluation, automated solutions are pursued [12].

In MMJ, subsequent to the printing process, the green parts still require debinding and sintering. Because the post-processing of sintered ceramic or hard metal components is very costly, it is intended that defects can already be detected and repaired on the green part, i.e., the condition before debinding. A potential optimization strategy may involve mapping the physical relationships of a process in a simulative model that calculates the quality and properties of a component as a function of its printing and material parameters. This approach is computationally intensive and requires a profound underlying understanding of the complex physical effects of the process in question. Often, the prerequisites for such simulations are not provided. In recent years, data-driven control strategies and, in particular, machine learning methods have gained application in AM process optimization, i.e., reducing costs and improving quality [13].

A survey of recent applications of machine learning in additive manufacturing reveals that most of the currently available solutions belong to the field of supervised learning, i.e., classification and regression [14]. Robust artificial neural networks, such as convolutional neural networks (CNNs), can extract basic features, such as anomalies in surfaces, in real time without human supervision [15,16]. Non-contact and non-destructive sensors such as cameras or laser line scanners are very commonly used [16,17]. Image-based surveillance belongs to the monitoring field of application. Image-based monitoring has become popular due to its comparatively easy integration into machines and facilities, which is typically due to its uncomplicated design and the use of inexpensive components [18,19]. The difference between camera- and laser line scanner-based monitoring is essentially the information content of the sensor data. Cameras provide image-only data, while laser line scanners can provide elevation/depth information or point clouds with precise geometric locations in addition to the images [17,20]. For instance, Wang et al. presented a classification approach for autonomous anomaly detection in fused deposition modeling by training a convolutional network that uses camera images to classify printed layers in situ into six different defect classes, e.g., cracks or air inclusions [13]. The various defects are detected with over 90 percent accuracy each. Similar investigations also exist in other additive technologies such as laser powder bed fusion [21] or laser metal deposition [12]. One application example of regression in AM is closed-loop control, which enables an adaptive printing process, proposed by Wang et al., documenting a design of an in situ control loop for liquid metal jet printing (LMJP). Random impacts in the process alter the shape and volume of the deposited droplets, leading to defects in the build layer [22]. In LMJP, the optimal process parameters change during printing, and the quantitative correlations for a specific adjustment of the throughput via the voltage of the piezo actuator of the dispensing system are unknown. The control concept includes a high-resolution camera that films the printing process and a controller that calculates the volume and velocity of the dosed droplets. In addition, droplet properties such as the occurrence of satellites are classified. Process data are used to train a neural network that learns how changes in drive voltage affect droplet characteristics. Using this knowledge, a PID controller can be used to measure the process stability in situ via the change in droplet characteristics and, if necessary, to regulate it to the optimum in a targeted loop via the jet voltage [22].

Due to complex physical interrelationships in the MMJ process, it is not obvious that a physical simulation model could be developed in an economically viable manner that would be sufficiently capable of predicting part quality based on process parameters. Therefore, developing data-driven control strategies and integrating them into an adaptive printing process is an important task in MMJ development. This publication deals with the automated detection of certain quality defects, essentially too little overlap, causing droplets to appear separated in the final printed surface, or too much overlap, causing the surface to bulge, on MMJ green parts based on in-line surface measurements and their processing with classical image processing combined with machine learning methods, in particular the training of convolutional neural networks (CNNs). The innovative core is the data processing of scans of multiple printed layers into an agglomerate of eight-channel image sections, which eventually enables us not only to classify the current process states in MMJ printing but also to gain predictions for the quality of subsequent printed layers. This ultimately provides a veritable basis for further process optimization in the MMJ process under consideration.

2. Multi-Material Jetting Principles

In 2014, with CerAM MMJ, formerly Thermoplastic 3D Printing, a technology for the additive manufacturing of ceramics and hard metals was developed at Fraunhofer IKTS in Dresden [5,6]. The MMJ process is based on the selective deposition of individual droplets from thermoplastic suspensions filled with particles [7]. A particular feature of MMJ is that even within a single layer several different materials can be printed, and thus multifunctional components (e.g., electrically insulating on the outside and electrically conductive on the inside) can be generated in a single process [23].

The MMJ process chain consists of three consecutive process steps: feedstock production, production of the green part (actual printing process), and thermal post-treatment. For the feedstock production, a ceramic powder and a thermoplastic binder system (polymer mixture) are dispersed and homogenized in a heatable disk stirrer at approx. 100 °C, depending on the material, and homogenized for several hours. The finished feedstock is solid at room temperature and is liquefied for the printing process by heating in the feed device (cartridge) of the microdispensing system.

The result of the printing process is the green part, which still contains thermoplastic binders that must be removed. Subsequently, the remaining ceramic must be sintered for the adjustment of the target properties. Accordingly, the thermal post-treatment is separated into debindering and sintering. Since the volume fraction of the ceramic powder within the suspension is typically around 50%, the green part will shrink during thermal post-treatment. Because complex diffusion processes occur during thermal post-treatment that cause material rearrangement processes, it is not possible to predict exactly how much shrinkage will occur in new part designs [24,25]. Shrinkage in the X and Y directions (layer plane) is usually in the range of 17–22%. Shrinkage in the Z direction can assume similar values depending on the material.

The print head consists of a Vermes microdispensing system (see Figure 1a), whose nozzle dispenses the droplets, and a laser profiler to scan the surface of the green part. The entire print head is attached to a CNC controlled three-axis kinematic system (X, Y, Z).

In the cartridge of the dosing system, the suspensions are melted at a suspension-specific temperature, typically around 100 °C, and fed to the microdispenser with the aid of compressed air. The micro-doser then dispenses the suspensions in the form of droplets at a defined frequency (up to 3000 Hz possible). For this purpose, a piezo-driven plunger moves up and down in a cylindrical chamber at the frequency set. When the plunger is at top, the suspension is pressed into the chamber via the static air pressure. The downward movement of the plunger shears off the material in the chamber and discharges it as droplets through a nozzle with a diameter of, e.g., 160 μm. For the movement of the plunger, five variables are defined for parameterization (see Figure 1b). The shape of the dispensed droplets (see Figure 1c) is typically described by their height, DHE (droplet height), and their maximum dimensions in the direction of travel of the printhead, DDA (droplet diameter aligned), and perpendicular to the direction of travel of the printhead, DDP (droplet diameter perpendicular). For a given set of droplet parameters and a given printhead speed, two parameters affect the quality of the printed layer: the overlap of droplets within a line, i.e., in the dispensing direction, OL_x, and the overlap of lines perpendicular to the dispensing direction, OL_y (see Figure 1c). Both overlapping factors describe the distance a_red by which adjacent droplets overlap in the dispensing direction or perpendicular to it, respectively.

Previous studies show that to obtain uniform line structures a value for OL_x in the range of 56% to 66% is to be aimed for [26]. If the overlap OL_x is too small, the line contour will become wavy, and if the overlap is too large, the contour will be smooth, but the lines will become excessive in height and width, ultimately allowing less fine geometries to be printed.

MMJ printed layers can have various process-specific defects or flaws which have a quality-reducing effect on the green part, and thus also on the sintered white part, and either require post-processing or even lead to scrapping. Some of the defects are random in nature, while others are systematic in nature and result from inadequate parameterizations. For the defects presented in this paper, it is assumed that parameters for reproducible round drops are available due to droplet shape analysis performed in advance. Given this, the defects that occur can be classified into three main groups:

• Dosing dropouts (machine caused; temporary and mostly for only one dispensing stroke);
• Parameterization errors (poorly chosen OL_x and OL_y);
• Defects at the boundary between the outline and infill (uneven transition with gaps or bumps, as print geometry is typically divided into the categories ‘outline’ and ‘infill’, discontinuities may occur at transitions between them, leading to defective prints, even if the parameterizations of ‘outline’ and ‘infill’ are well suited on their own).

The final component geometry results as an agglomerate of droplets, which add together as the smallest incremental volume elements in three spatial directions. The parameterization of the droplet overlaps (OL_x, OL_y) influences the quality of the printed layers and thus the quality of the component. First of all, the print settings of the infill are crucial, whereby the parameterization of the outline has to fit to that of the infill. If the overlaps OL_x and OL_y of the infill are chosen to be too little, the individual drops are separated from each other and hardly merged within a layer. This creates air pockets during printing, which can lead to porosity in the sintered component and scrap. If the overlaps OL_x and OL_y of the infill are chosen to be too large, excess material is formed due to which the layer bulges and appears bulbous. Furthermore, it can be seen that the outline is lower than the infill. With well-parameterized layers, no difference in height between infill and outline exists. A particular challenge in evaluating the suitability of the overlap parameterization for the layer quality is that if the unsuitability of the overlap parameterization is only minor, through error propagation, the unsuitability only becomes apparent after many layers have been printed.

3. Experimental Concept

The concept for the data-based evaluation of infill parameterizations in the MMJ printing process by a convolutional neural network to be trained is schematically illustrated in Figure 2. First of all, a droplet shape analysis is carried out for the selected material. For this purpose, droplets are printed with different dispensing parameters and are evaluated on the basis of specific target properties for the droplet shape, e.g., roundness and reproducibility. As a result, the characteristic dimensions of the droplets are known, which are required in the slicer for the adjustment of the overlaps of the droplets in the direction of the print head translation OL_x and perpendicular to it OL_y.

Subsequently, a test component is printed several times applying different overlap parameterizations with the determined optimum dispensing parameters, and the surface is scanned with a laser profilometer. The raw data from the scanner are processed and used as training data. Depending on the settings used for OL_x and OL_y, good and bad infills result. Since it is not known which overlap combinations will result in good or poor print quality, experiments must be iterative in order to ultimately have a balanced training data set, i.e., approximately equal numbers of good and poor print results.

Given the time-consuming printing and scanning process, 170 test components are printed and used for training and testing the CNN. Accordingly, the available raw data quantity is to be considered rather low compared to typical applications of artificial neural network training.

The raw data of the surface scan are available as a CSV file. This CSV file contains height data of the entire scanned area. In addition to the printed part, this scanned area also includes its environment, i.e., the inspection table, which is why the scan of the part must be separated from the environment scan. The inclination of the inspection table has an influence on the profilometer measurement, which has to be corrected, which is performed by means of software. Edge detection methods such as the Canny algorithm are used for this purpose [27,28]. Essentially, the raw data of each individual scan are prepared in such a way that at the end there are eight equally sized images of the infills of the eight layers of the test component, each of which is given the label good or bad for each layer.

Further, for the sake of processing time, the images are converted to grayscale because this significantly reduces the amount of data (approximately 30% data volume remaining), and the resulting image resolution with grayscale is still on par with the height resolution of the profilometer. The eight grayscale images are then arranged into a single grayscale image with eight channels and thus converted into a data format that the CNN is trained with. With relatively few data available in 170 images, it is a reasonable assumption that the predictive accuracy and robustness of the CNN can be improved with data augmentation methods. It is well known that the more data an ML algorithm has available, the more effective it is likely to be [29]. The problem with small data sets is that models using them do not generalize well from the data set. Consequently, these models suffer from the problem of overfitting [30]. Data augmentation is a way to reduce model overfitting by artificially increasing the variance and the amount of training data by using only information from the training data itself. Even if the augmented data are of lower quality, they still can improve the training as long as the relevant features are not distorted and the variance of the training data is increased. Instead of starting with an extremely large corpus of unstructured and unlabeled data, it is possible to take a small, curated corpus of structured data and expand it in a way that increases the performance of the models trained on it. Geometric distortions or deformations are commonly used to increase the number of samples for training deep neural models [31] and to compensate for the size of data sets [32], as well as to improve efficiency [33], often used as affine transformations for data expansion, but this is still the subject of research.

With geometric overlap, the droplets form the volume geometry intended to be produced by the printing process. The overlap in the feed direction of the print head OL_x and perpendicular to it OL_y are significant for the quality of the volume geometry manufactured.

The assumption is made that eight layers are sufficient to evaluate the parameterization unambiguously as good or bad. This assumption could be confirmed in preliminary tests, in which cuboids with good and poor parameterizations (OL_x and OL_y) were printed with varying numbers of layers. In order to be able to evaluate the quality of the infills with the human eye, the steps have a base area of 7 × 7 mm. With a stair structure as the demo component for data generation, eight images are available per printed component (one per layer or step). Each of these images must be labeled as good or bad. Basically, there are two different ways to label these eight images. In option 1, all stages are evaluated separately and individually. For example, the first three layers will receive the label good, while layers four to eight will be classified as bad. The images of the stages from all components will also be given to the CNN individually and separately. In this way, the CNN learns during training what a quality assessing person perceives as good or bad infill when assessing the layers individually. In option 2, on the other hand, all stages of a component, i.e., all images of a certain parameterization, receive the same label. This means that if, for example, the eighth layer shows an insufficient parameterization, not only is this last stage considered bad but also the previous seven layers, even if the poor parameterization is not apparent on those. As a result, the eight images of a component are not passed to the CNN separately but together, e.g., as multiple channels of an image. This is intended to aid the CNN as it learns to classify the development of the infill from level one to eight in training as good or bad. In this way, the CNN should not only abstract what individual bad infills look like, as in possibility 1, but also what insufficient infill parameterizations look like even at the lowest layers, where a quality assessing human cannot yet make such an evaluation. Accordingly, the CNN learns patterns that suggest a bad parameterization before it is visibly formed for humans. Option 2 of labeling introduces a higher degree of abstraction in the CNN’s learning and is also more interesting for practical purposes since it can be specifically studied how many layers a CNN needs compared to a human to make predictions about the goodness of the parameterization. In other words, it can be investigated how well the CNN learns patterns that constitute poor parameterizations for final parts but are hardly apparent to humans in early process layers.

4. Data Generation

The zirconia material applied is known to have poor infill for overlaps beyond the range of 25% to 60%. Accordingly, combinations of OL_x and OL_y are mainly investigated in that range [25%, 60%]. Droplets with the best dispensing parameters determined during the drop shape analysis were identified. Previous studies have already examined the weightings and interactions of the respective influencing variables, with the result that the parameters Falling Time, Needle Lift, Rising Time, and the feed air pressure are to be regarded as significant and sufficient for the optimization of the drop geometry [34]. Only the influences of the Falling Time (FT) and Needle Lift (NL) parameters are investigated in this study to find an appropriate droplet geometry. In general, there is a correlation between the flow behavior of the printing materials and the dispensing parameters (applied force over time and resulting shear and acceleration) and droplet properties. These general trends have already been derived through Process Data-Based Knowledge Discovery [34] by performing a series of tests on the material used in this publication, which yielded a range of parameters inferred from the data. Regarding the optimal dimensions for the DDA, DDP, and DHE, the main criterion is that the DDA shall be as equal as possible to the DDP, resulting in uniformity in print in terms of distribution of the material (droplets) relative to each other in a defined grid. The second criterion relates to the aspect ratio (ratio of height to diameter) of the droplets of approximately 1:6, which enables overlapping without later droplets bulging onto the previous droplet, but instead the later droplet slides off the previous droplet as it merges, appearing triangular in cross-section, with the lower sections of the triangles joining together while the height of the tips decreases to form a uniform layer. When the Needle Lift (NL) increases, the DDA, DDP, and DHE generally increase. However, the DHE can decrease again if the NL is too high. As the Falling Time increases, both the DDA and DDP decrease, while the DHE increases strongly but with diminishing effect, as with more FT less force accelerates the droplet towards the target and thus less spreading occurs [34].

As the aim of this study is not to investigate the exact interactions of many influencing parameters on the printed geometries but to demonstrate the applicability of machine learning methods for surface quality investigation, it is sufficient to approximate an optimum drop geometry with fewer parameters and to be able to qualitatively specify the effect of the influencing variables. The fixed parameters that will not be varied in the experiment are listed in

Table 1. Fixed experimental parameters in droplet shape analysis.

Parameter	Value
Nozzle diameter	100 µm
Cartridge temperature	285 K
Open time	0 ms
Rising Time	7 ms
Print head velocity	200 mm s−1
Air pressure	100 kPa

, which are common for the material zirconium oxide to be investigated. From empirical values for the material, it is known that acceptable drop geometries occur in the interval of Needle Lift (NL) = 40–50% and Falling Time (FT) = 0.25–0.4 ms. Within this range of values, all 12 combinations of values for FT and NL are examined.

For each parameter combination, several paths consisting of non-overlapping drops are printed and measured with the laser profile scanner. The raw data of this measurement, recorded in a CSV file, are transferred to a purpose-built Python code for automated data processing. In the first step, this software cleans the raw data for measurement artifacts and corrects both the inclination of the table on which measurements are taken and the difference between the different scanning speeds in the X and Y directions. The entire procedure of data preparation is described in detail in Section 5.

Subsequently, the software segments the individual droplets using the Von Otsu method [35,36] and distinguishes them from satellite particles. Through these preparatory steps and the segmentation, the image area of the scan associated with a droplet is available with the elevation data. Based on this, the software can calculate and output the characteristic values sought for the target properties. The best values found for Needle Lift (NL) and Falling Time (FT) are not to be published for the sake of competition. Droplets with the dispensing parameters determined during the drop shape analysis, which are printed subsequently in further experimental stages, show characteristic droplet dimensions of DDA = 706 μm and DDP = 627 μm.

During the labeling of the data, in order to assess as objectively and unbiasedly as possible, images of all printed structures are placed side by side without designation of the printing parameterization. In the evaluation of the infills, alongside obvious defects such as too little overlap (individual drops clearly visible) or too much overlap (surface appears fused and bulges), attention is paid to two criteria in particular:

• Visibility of individual drops, causing air pockets and porosity;
• Merging of infill lines in the edge area, resulting in the formation of localized protrusions and thus loss of dimensional accuracy.

Figure 3a,b show two examples of extreme infill parameterization. For Figure 3a, the overlap is set too low with OL_x = OL_y = 30%, while for Figure 3b, the overlap is set too high with OL_x = 65%, OL_y = 60%. Both components exhibit bad layers, which differ very clearly in their appearance. In the layers shown in Figure 3a, the individual droplets are clearly separated and hardly fused together within a layer. This means that there is a risk of air inclusions during printing, which in turn can lead to porosity in the sintered component and ultimately to scrap. On the other hand, excessive overlapping, as shown in Figure 3b, where the entire layer fuses and individual droplets are no longer recognizable, leads to a loss of dimensional accuracy. In many cases, the visible differences between different parameterizations are marginally small and can only be assessed fuzzily. The evaluation of the infills is therefore affected by subjective fuzziness. Assessing the infill parameterizations, the assumed correlation shown in Figure 3c can be confirmed in principle, since a cluster for OL_x and OL_y, within which good infills occur, truly appears, as shown in Figure 3d. It can be convenient to discretize uncertain data points by drawing a threshold curve around the cluster of good infill parameterizations in order to separate the two classes. All combinations on one side of the trend curve belong to the good class and all on the other side belong to the bad class. Supposing that all layers printed with the same settings always eventually yield identical print results, this will be a valid procedure. The aim of the approach for automated in-line monitoring is to cover all eventualities of influences that lead to different layer formations. Whatever may lead to differences in the quality of two layers despite having identical parameterization, i.e., the worst unknown scenario, will be taken fully under consideration by the model. In consequence, for parameterizations at the boundary of the clusters, two green parts parameterized equally but printed on different days can result in different assessments with regard to their appearance.

5. Data Preparation

For each of the 170 printed green parts, a CSV file is available containing raw scan data in 15,000 rows and 800 columns. In order to obtain eight-channel grayscale images to train CNN from these data, the following basic data preparation steps (see Figure 4) were performed: correction of the measurement artifacts and the shadowing effects, correction of the different scan point distances in the X and Y directions, correction of the measurement table unevenness, flexible cropping of the green part from the image background, cropping of the individual steps, and merging of the steps of a green part into an eight-channel grayscale image. Due to varying scan point distances of the laser scanner in the X and Y directions, the raw data are distorted in the X-Y plane. The measuring point distance in X direction depends on the adjustable variables travel speed and scan rate. For the selected scan settings, the measured height profile is downscaled from 15,000 × 800 to 7500 × 800 measuring points. Measurement artifacts are not-a-number values or values that exceed the measurement range (+/− 2.6 mm). Measurement artifacts are caused, among other factors, by shadowing effects at the edges of the scanned component. Within the data, these measurement artifacts can be located using filters. The correction is performed by assigning a new value to the pixel affected, which is formed from the trend of surrounding, valid pixels. Entire rows or columns consisting only of invalid values are deleted as a whole. The third flaw in the raw data to be corrected is the inclination of the measuring table. The measuring table of the laser scanner is the print bed of an MMJ system. As no reference measurement is available, the background must be recalculated for each scan. Each pixel carries a height value a_ij. The image background is approximated by calculating a straight line between two outer opposing pixels a_1i and a_ni. The value this straight line takes at the center of a pixel is assumed to be the background height for that pixel. This is repeated for all outer opposite pixel pairs. The result is the approximated uneven image background. This is now subtracted from the original image to correct the table inclination.

In order to generate images of the infills, the component is first segmented from the surroundings within the corrected scan. Once images of green parts of the same size in the base area are available, the infills can be cropped from the images with a fixed mask. Because the laser scanner used cannot always position the part exactly the same way within the scan, it must be possible to segment and cut it out flexibly and automatically. For this purpose, edge detection with the Canny algorithm, which yields a binary mask indicating edges, is utilized. Firstly, the height data are converted into grayscale values. Subsequently, the image is smoothed with a Gaussian filter to reduce noise. This removes excess details that can lead to unwanted edges in the course of the edge detection. The Canny algorithm then computes the magnitude and direction of the image gradient, and pixels that may not constitute the edge are removed by gradient magnitude thresholding. This is followed by the final step of hysteresis thresholding. Pixels with a gradient magnitude higher than T₂ are selected as sure-edge pixels, pixels with a lower gradient magnitude than T₁ are removed, and those with a gradient magnitude in between are only marked as an edge if they have at least one sure-edge pixel as a neighbor. The Canny feature from skimage is used for this purpose. It contains the three adjustable parameters width of the Gaussian signal as well as the lower and upper threshold value for hysteresis thresholding. Based on default values from the skimage documentation, iterative manual experimentation was performed using a Matplotlib graph window with sliders for the Canny parameters by visually inspecting the changes in edge detection results as a response to the manipulation of the Canny parameters. This led to the selection of a lower and upper hysteresis threshold of the gradient magnitude (T₁ < T₂) for the selection of the final edges from the set of potential edges. The goal is to adjust the parameters for Gaussian smoothing as well as the threshold values T₁ and T₂ in such a way that no edges outside the component are detected and the component contour distinctly emerges. Otherwise, the component cannot be correctly segmented. With Canny edge detection it is not possible to only detect the outermost component contour without the layer boundaries since the height difference between the layers corresponds to the height difference between the layer and the measuring table. The edges detected are then merged to form closed contours, of which the outermost contour is filled in, thus serving as a preliminary mask. This preliminary mask is approximated by a rectangle, which in turn is utilized to segment the part from the original scan. Experiments showed that two consecutive Gaussian filters with standard deviations of σ₁ = 15 and σ₂ = 50 visibly smooth the measured roughness of the component surface and the surrounding background. Subsequently, the values of T₁ and T₂ are varied until the target state is reached and only the outermost contour of the component and the layer boundaries on the component are detected. The final hysteresis thresholding values of T₁ = 0 and T₂ = 1.45 work properly for the desired purpose and all the 170 components with different printing overlaps. It should be noted that the parameters were identified manually and are only valid for the analyzed component and the selected overlaps.

The augmentation techniques, listed in

Table 2. Augmentation combinations examined.

Augmentation 1	Augmentation 2	Augmentation 3
Mirroring	Changing brightness	Mirroring
90° rotations	Sharpening or smoothing	90° rotations
		Changing brightness
		Sharpening or smoothing
		30° rotations

, were chosen so that the variations generated corresponded to expectable test data. This can prevent the CNN from learning to adapt to particular training data which do not correspond to reality. Since it is not possible to forecast which augmentation techniques will produce good results, three different augmentation combinations were examined. The augmentation settings randomly chosen for a training data are always applied equally to all eight channels of the original image, while the label is preserved. Augmentation 1 performs random flips and 90° rotations on the original images. Since MMJ layers are typically printed with a 90° offset from layer to layer, these transformations correspond to changes that do not occur in this test data set but do occur in daily use. Augmentation 2 randomly varies the brightness (using gamma transform) and/or sharpness of the images. In particular, changes in brightness represent variations that may be randomly included in the data if, for example, contaminants on the surface are also scanned. Consequently, due to the increase caused by the contamination, the rest of the infill is mapped darker because grayscale has a limited range of values. This then also reduces the resolution of the infill, which is close to a change in image sharpness. Augmentation 3 represents a combination of augmentation 1 and 2, whereby this is supplemented by random rotations of a maximum of +/− 30°. The rotation is always centered around the image center, and any resulting empty corners are filled via a reflection of the remaining image area. Rotations of the infill are to be expected if the component is not optimally aligned during the scanning process.

6. Model Training

The data set consisting of 170 images is split into 15% test data and 85% training data, with 25% of the training data used as validation data. The percentage ratio of the two classes good and bad is conserved within this division of the data set (44% good, 56% bad).

Applying neural networks for novel tasks, it is not known for which hyperparameters (totality of all parameters that determine the learning success for a given data set and basic net architecture) a minimum of the cost function or a maximum of the accuracy can be obtained. Since the number of possible hyperparameter combinations is infinite, basic assumptions are made that are based on standard CNN settings before a hyperparameter tuning in the form of a random search is performed: because the given task is binary image classification, the binary entropy is used as the loss function in combination with a sigmoid activation in the output layer. Furthermore, ReLU is set to be the activation function of the convolutional layers and the remaining dense layer. The Adam algorithm is ap-plied as an optimizer. The hyperparameter space examined in the random search is based on common CNN architectures [37,38,39] with few hidden layers (six to eight in total), in which the convolutional layers are typically followed by an activation in order to introduce non-linearity, enabling the network to learn more complex patterns. Subsequent pooling layers aggregate the information in respective regions, promoting translation invariance, enhancing, enhancing feature generalization, and reducing sensitivity to noise [37]. The generalized structure of the networks applied is shown in Figure 5. For the parameterization, the following notations were applied:

• The number of convolutional layers (L): [1, 3];
• The number of kernels per convolutional layer (CK): [16, 128];
• The kernel size of the convolutional layer (K): [(3,3), (5,5), (7,7)];
• The pooling size (P): [(2,2), (4,4), (6,6)];
• The number of units in the dense layer (D): [20, 160];
• The dropout rate in the dense layer (Dr): [0.0, 0.4].

Preliminary investigations with a grid search applying L = 2, K = (3, 3), P = (4, 4), and Dr = 0.0 as fixed hyperparameters, as well as CK = [32, 64], D = [60, 190], investigating the batch size in the range [16, 32, 64] and the learning rate in the range [0.1, 0.01, 0.001] as variable parameters show that particularly good results are obtained for a learning rate of 0.001 and a batch size of 32. Accordingly, these hyperparameters are not further varied. Simplifying, for a chosen hyperparameter set, all convolutional layers contain the same number of filters, and the pooling size also remains the same within the network. In addition, no padding is applied, meaning that the image decreases in size and that some information at the edge of the image will be lost with each convolution depending on the size of the kernel. The stride length of the convolution kernels is set to (1, 1), so that the spatial information is preserved. The random search is performed with 200 trials. For all six hyperparameters, the range of values is specified in a way that both relatively large CNNs with many parameters and relatively small CNNs with few parameters are supposed to be trained. Generally, large CNNs can absorb more information and achieve higher levels of abstraction. However, these advantages come with higher training costs and an increased risk of overfitting to the training data. Many hidden layers with numerous and large kernels per convolutional layer, small pooling sizes, numerous nodes in the dense layer, and low dropout rates result in a high number of parameters to train and vice versa. The individual trials are sorted by the maximum validation accuracy achieved. Additionally, an early stopping is implemented, which interrupts the training of a CNN if there is no improvement in a selected target variable after a selected number of epochs (patience). The target variable is the loss function of the validation data. The validation loss function can tend to decrease even when the validation accuracy is already stagnating, which implies that the network becomes more reliable in its decisions while its accuracy remains the same. The random search is performed for both non-augmented training data and the individual augmentation techniques. Since the required number of training epochs increases when augmentation techniques are applied, the maximum number of epochs and the patience of early stopping are adjusted individually for each random search.

In order to explore and efficiently analyze the general impact of the hyperparameters on the learning success of the CNN with large numbers of hyperparameters and large numbers of sets of experiments, the investigations are aided by plots of parallel coordinates (Figure 6). Each continuous line in the parallel coordinates plot represents a trained CNN, its hyperparameters, and its learning success by means of validation accuracy and validation loss.

Figure 6a shows a parallel coordinates diagram of the random search with all performed trials, their hyperparameters (pooling size, convolutional kernels, dense units, kernel size, number of convolutional layers, and dropout), and the validation results (accuracy and loss). From this display, a general overview of which hyperparameter combinations lead to different learning results can be derived. In Figure 6b,c,d, specific ranges of selected hyperparameters are highlighted. It can be clearly observed in Figure 6b that CNNs with relatively many parameters to be learned (small pooling size, multiple convolutional layers, many convolutional kernels and nodes in the dense layer) achieve poor learning results, represented by a poor accuracy and high loss. As illustrated in Figure 6c, CNNs with moderately few parameters (convolutional layers, convolutional kernels, dense units) to be trained achieve the best validation accuracies. As illustrated in Figure 6d, especially networks with two convolutional layers, few convolutional kernels, and a pooling size of six achieve reasonably high accuracies. The spectrum of hyperparameters studied indicates that the more complex models studied (more layers and more units) have a tendency to overfit. This is consistent with basic known properties from artificial neural network applications. In underfitting, the learning model is too simple and cannot learn the basic data relationships, while in overfitting, the model is complex and only remembers the training data with limited generalizability [40,41,42]. In both cases, underfitting and overfitting, the model cannot reliably assign previously unseen data. However, the fact that, for the spectrum of CNN hyperparameters studied, the simpler CNNs perform better suggests in principle that even more simple networks could have shown even better results, if necessary. This is opposed by the fact that the best performing models are already strongly reduced, and a further simplification is in principle no longer possible, which can be seen for example in the number of convolutional layers.

7. Results and Discussion

In addition to the models and their hyperparameterization, different methods of data input into the models are considered relevant for use in the predictive monitoring of the printing process. Therefore, a selection of the best models found is also used to investigate how the learning curve of the models changes when the given data contain less information. This reduction in information is carried out in two steps: First, images are generated that do not contain the edge area of the infill, which is crucial for the quality perception of humans. Subsequently, the data set of images without the edge area of the infill is limited to images of the first layer only.

Table 3. Model parameterization and data input variation.

Model	Number of Conv. Layers (L)	Kernels Conv. Layer (CK)	Kernel Size Conv. Layer (K)	Pooling Size (P)	Units in Dense Layer (D)	Dropout Rate Dense Layer (Dr)	All Layers as Input	Only First Layer as Input	Only Infill as Input	Augmentation
1	2	16	(7, 7)	(6, 6)	100	0.2	X			none
1.1	2	16	(7, 7)	(6, 6)	100	0.2		X		none
2	2	96	(3, 3)	(6, 6)	40	0.0	X			3
2.1	2	96	(3, 3)	(6, 6)	40	0.0		X		3
3	3	16	(5, 5)	(6, 6)	160	0.4	X			none
3.1	3	16	(5, 5)	(6, 6)	160	0.4		X		none
3.2	3	16	(5, 5)	(6, 6)	160	0.4			X	none
3.3	3	16	(5, 5)	(6, 6)	160	0.4		X	X	none

provides an overview of the models with their parameterization and data input discussed in the following.

Figure 7 shows the learning curves of the best model. The best model obtained, model 2, was trained with augmentation 3 (see Table 2 in Section 5). From epoch 250, the validation accuracy stagnates between 90% and 95%. As a final model, the learned parameters are stored at the epoch of least validation loss. This results in an accuracy of about 78% on the test data. Model 2 also has two convolutional layers and a pooling size of six. The trend that models with relatively few parameters to train tend to yield better accuracies can be confirmed by the coincidence searches performed with augmentation.

In Figure 7e, the predictions of model 2 on the validation and test data are plotted in the OL_x-OL_y diagram. As the training data, whose predictions are not plotted, always have a constant scale, the diameter of the validation and test data increases with the statistical prediction certainty of the CNN, although the diameter does not increase with the prediction certainty true to scale for display considerations. The confidence of the prediction originates from the value of the sigmoid activation function of the output layer. The model hits all predictions correctly for samples that are clearly in the bad or good range. The confidence with which these predictions are made is close to always high. In the range of very low values for OL_x and OL_y (20–30%) is a cluster of worse infill parameterizations. The predictions of these infills are correct, but significantly more uncertain than those of other worse parameterizations that are similarly far from the boundary line to the class good. The reason for this is presumably that the model was trained with only three data instances from this cluster, while other areas of the OL_x-OL_y diagram have a higher density of data instances.

The incorrect predictions of the CNN are all in the marginal region of both classes. This can be a consequence of the fact that the labels of the ratings were not carried out according to objective criteria but according to subjective evaluations. As a result, the labels include inconsistencies in the marginal region. A model can learn these inconsistencies in training data given sufficient training time by memorization, but the model thereby cannot abstract underlying patterns and predict the test data appropriately. Thus, as long as the inconsistencies exist in the evaluation of the infills, no model will be able to accurately predict this fringe region. The certainty of the predictions also tends to be smaller in the fringe band.

In general, a comparison of different augmentation techniques applied in the training experiments shows that augmentation does not significantly improve either the validation accuracy or the validation loss. Different augmentation techniques tested do not make the learning curves less volatile, which would imply a more stable learning process. However, the number of training epochs required increases significantly and is highest for augmentation 3 due to the strongest variance in the training data. The fact that no significant improvement in validation accuracy was observed with any of the augmentation combinations studied does not necessarily imply that the increased training time due to augmentation is useless. It is likely that variations learned through the augmented training images are not sufficiently included in the validation data applied to date. If validation data printed with other materials or other droplet parameterizations or containing different-sized image sections are used, the variations learned due to augmentation may indeed prove useful. However, the finding that, in the case of the specific application presented, none of the augmentation techniques applied can make a significant contribution to measurable model performance leaves room for further clarification. The contradiction to the existing consensus regarding the benefits of data augmentation in the scientific world of data science and machine learning remains. The observed results emphasize a need for more complex augmentation techniques over simple geometric operations such as rotation and distortion. These can be more sophisticated augmentation techniques, such as Style Transfer [43,44] and Generative Adversarial Networks [45,46] or combinations of traditional and advanced augmentation methods [47].

It is assumed that the hyperparameters of the best model found lead to good results even with less information, since many of the learned image features can also be contained in reduced data sets. To verify this, two models are trained with the same hyperparameters using only images from the first infill layer. The results of models 1.1 and 2.1 (see Table 3) on the validation data are good, but the accuracies of these models stagnate at a lower level compared to models 1 and 2. This shows that the hyperparameters found give good results even with less training information. For the exploration of learning success based on images not containing the edge region of the infill but all eight layers given, a random search is performed with the same search space as before without augmentation. The learning curves of the best model found, model 3.2, are shown in Figure 8c. Model 3.2 achieves similar values to models 1 and 2 for both validation accuracy and validation loss, which means that unlike humans, a CNN does not need visual information about the merges that occur at the edge of the infill for evaluation. This also confirms the trend that smaller models achieve higher accuracies. Although model 3 has the maximum possible number of 160 units in the dense layer, this is effectively greatly reduced by a high dropout rate of 0.4 in order to avoid overfitting. The hyperparameters of model 3.2 are used to train model 3.3, which is trained using only images of the first layer and excluding the edge of the infill. It is shown that model 3.3 is as accurate as model 3.2, and the training of model 3.3 converges to a validation accuracy of about 90%. Considering that humans need eight layers and the edge of the infill for classification, this is a very good and remarkable result. The result is particularly relevant in practice because it means that any section of the first printed layer is sufficient to evaluate the quality of the infill parameterization. Thus, the CNN must have learned patterns in the images that are unrecognizable to human domain experts. Hence, classification concepts turn into tools for the prediction of states that arise as a result of highly complex influence matrices with hidden cause–effect relationships. This is comparable to performances that Deep Learning has been able to demonstrate in medical predictions. Examples from the spectrum of medical prognostics include Deep Learning architectures for multi-label classification and intelligent prediction of common health risks [48], diagnostic classification and prognostic prediction using neuroimaging data in Alzheimer’s disease [49], and classification of cancer to improve prediction of patient outcomes [50].

In summary, the research conducted showed that neither augmentation techniques nor varying the amount of information per data sample have significant effects on learning success, i.e., the validation accuracy and validation loss. In practice, this means that a well-trained model can predict with 90% validation accuracy whether this layer is well or poorly parameterized after only one layer and based on any square section of the infill. This result could be achieved for one material and one droplet parameter set with 170 samples.

8. Conclusions

Due to complex physical interrelationships in the MMJ process, it is not obvious that a physical simulation model could be developed in an economically viable manner that would be sufficiently capable of predicting part quality based on process parameters. Therefore, the development of data-driven control strategies and their integration into an adaptive printing process constitute a fundamental element to further advance MMJ. For this purpose, concepts for the automated in-line quality assessment of specific defect patterns of the MMJ process were developed on the basis of height profile data of green parts. The investigated defect images address the parameterization of the infill. For automated detection, a novel approach of processing the in-line surface measurements by means of machine learning methods, in particular the training of convolutional neural networks, is developed. The tests were carried out with a feedstock consisting of 40% by volume of zirconium oxide and with an experimentally determined parameter capable of yielding appropriate droplets. A well-parameterized infill shows a flat and dense appearance, although no common standardized quantifiable rule exists for determining exactly what constitutes a good infill. The quality of an infill is assessed according to subjective standards and on the basis of several printed layers, so that any parameterization flaws may appear after a delay across several layers.

For the automated quality assessment of infill parameterizations, convolutional neural networks were trained on surface scans of printed parts to distinguish parameterizations into good and bad. To generate the training data, a demo part structure with eight layers was constructed and printed 170 times with different overlap settings, scanned, and labeled by process engineers according to subjective measures. A cluster of good parameterizations formed in the OL_x-OL_y diagram, with likely partially inconsistent evaluations in the periphery of the cluster. For data preparation, a pipeline imports the raw data from the scanner, corrects it for measurement errors, segments the green part from the image background, and generates grayscale images of the eight infills of the stages. Sequences of random searches were conducted to determine optimum hyperparameterizations of convolutional networks for training with and without augmentation.

The best model achieved validation accuracies of better than 90%. All misclassifications of the network were in the threshold band of positive and negative assessments, which is probably due to the fact that the underlying class division of the labels itself is uncertain in this range. The application of different augmentation techniques has little impact on the learning results of the models considering the validation data used. In particular, models with two convolutional layers and a pooling size of (6, 6) appeared to yield good accuracies. Models trained with images of the first layer only and without the infill edge obtained validation accuracies of 90%. Consequently, an arbitrary section of the first layer is sufficient to make a 90% accurate prediction about the quality of the final component after eight successive layers are printed. This exceeds human capabilities in this particular area; humans need subsequent print layers and the infill edge for comparable accuracies, which substantiates the potential of the approach.

Further work is needed to investigate the extent to which the results to date can be applied to other droplet parametrization sets and other materials. As this is proven to work, the parameterization of the outline can be classified in relation to that of the infill. Ideally, a future adaptive automation solution will not only be able to distinguish parameterizations into good and bad but also to suggest improved parameters and thus autonomously find suitable parameters for new materials.