Smart Cordless Power Drill - Detecting Fastener Driving States

[chris.rogers]

Smart Cordless Drill Application Demo

This application concept demonstrates how the common mechanical clutch found on nearly all consumer cordless drills might be replaced (or at least augmented) with a more intelligent motor control algorithm that governs motor speed and/or torque based on a specific task.

This might involve a mode selection where a user cycles through various drilling tasks and configurations such as:

Tasks

• Driving / removing fasteners
• Drilling holes

Materials

• Hardwood
• Softwood
• Ferrous Metals
• Non-Ferrous Metals
• Plastic
• Drywall

Based on the user’s task selection, the appropriate model and parameters for sensing job states can be invoked.

Our application involves reading vibration data from a 6DoF IMU (accel + gyro) as well as capturing motor voltage and trigger input (via voltage reading of the potentiometer for a variable speed drill trigger).

Building suitable models requires capturing and labeling event states where motor control intervention is necessary as well as recording various metadata that allows for model selection and configuration based on the user’s task and conditions. The illustration below shows a summary of these:

Building the Hardware

For this demo, we chose to use an inexpensive power drill commonly available through online resellers. Specifically, we chose the Black & Decker BDCDD12C 12V cordless drill as our test mule for this smart tool conversion demo.

Providing the intelligence, we selected the M5Stick C Plus based on the Espressif ESP32-PICO.

This device benefits from the following features that were well-suited to our demo:

• 240 MHz dual-core processor (capable MCU for running our inference models)
• 520KB SRAM (more than adequate to hold our ML model data)
• 4MB flash (plenty for our firmware application)
• Wi-Fi connectivity (sufficient wireless TpT to stream train/test data at 250Hz x 8 channels)
• Battery power
• Built-in 6 DoF IMU sensor (MPU6886)
• Small LCD screen (convenient to display UI for data collection and inference results)

All that was missing was a means to capture our motor voltage (0-12VDC) which was accomplished by adding an external I2C interfaced ADC based on the Microchip MCP3432.

|

|

|

Packaging everything up within the existing drill enclosure proved a bit challenging but manageable.

The M5Stick C Plus unit was then epoxy glued to the side of the case with a Grove connector providing the I2C interface to the internally mounted ADC for motor voltage.

Building Our First Model

To start, we focused on just one drill task for our model, driving Phillips head screw fasteners. If you’ve ever used a drill to drive a Phillips screw then you’ve probably encountered the issue of ‘cam out’ at some point. The tendency for the Phillips bit to slip within the head of the fastener if sufficient downforce is not applied while sinking the fastener into the material is known as ‘cam out’ and is damaging for both the bit and the screw head.

For our model, we will seek to detect four basic states for drill driving of fasteners:

1. Drill Off (excluding IMU sensor vibrations that do not involve active use of the drill)
2. Driving (sinking a screw into some material)
3. Removing (extracting a screw from some material)
4. Cam Out Fault (Phillips head bit slipping in fastener in either driving or removing states)

We’ll start by collecting some training data for each of the states of interest.

Capturing the ‘Off’ State:

This may sound trivial. Would could simply put the drill on the desk and capture data of nothing happening. The motor voltage, accelerometer, and gyro motion data would all be flatline signals and we’d gain not much information about this state. But we want a more robust model, so we’re going to capture a variety of common motions that the drill might be subject to without pulling the trigger to spin the drill. Things like:

• Pre-positioning the drill onto the fastener before actually turning it on
• Lifting the drill up and carrying to the project
• Changing a bit in the drill chuck
• Many other possible non-fastener turning activities

In the video below, you can see what types of ‘OFF’ activities we included for our model:

One thing we did not include but perhaps should consider for future versions is the exclusion of free spinning conditions. That is, simply pressing the trigger to spin the drill motor without an actual fastener being connected to the bit at the business end of the tool. The fact that we omitted this will become clear later when we show the resulting final model.

This highlights an important aspect of test design. It’s totally up to the developer to fully characterize what conditions should trigger various classifications and also what conditions should NOT trigger classifications. The AutoML algorithm is smart, but it cannot derive intent where example data was not collected.

Capturing Revelant Metadata

So we’ve collected one of our classes already. Before we continue onwards, we should touch on another detail key to getting good results: capturing relevant metadata.

In the above screenshot, our project metadata for the smart drill includes the following key context information for our data capture effort:

• Device
• Fastener
• Material
• Operator Handedness (i.e. left or right handed)

Each of these parameters describes context information that the application expert considers relevant to how the data might be influenced by outside factors. Will the motion pattern for driving of a screw be different for right versus left handed operators? Possibly. Until we know for sure, it’s probably wise to collect that information so allow us to filter or segment our models by this variable if it proves to be significant. Certainly this will be the case for different types of fasteners themselves as well as the base material for which the fastener is being driven into. So we’ll capture those parameters as relevant metadata too.

Collecting the Remaining Classes

We continue with collecting training data for the other classes in our model:

• Driving
• Removing
• Cam Out Fault

Since it is not practical to collect continuous long samples of each class, we utilize the capability of Data Studio to precisely define segments (regions in time) that represent each of these different classes.

The video clip below shows the result of a one-minute capture of repeated cycles of driving and removing screws from the wood block. The resulting data streaming along with synchronized video capture from a webcam allows for easy and precise segmentation of each of the different states of the drill. These are represented with the colored overlay bars on the time-series strip charts as well as the tabulated listing in the upper righthand ‘Segments’ window.

In total, we collect over 140,000 samples (equaling 9.3 minutes of data at 250Hz) of labeled data. To ensure good model accuracy, we seek to achieve balance across the classes to avoid biasing the results. Data Studio makes it easy to quickly see our overall label distribution statistics as shown below. We have somewhat more ‘Off’ data than the other labels, but overall a good distribution of labeled raw sensor data covering nearly 10 minutes of drill operation. Let’s see if that will be sufficient to generate a good working demo model.

Creating an ML Model From Labeled Sensor Training Data

Moving on to ML model construction, we will use SensiML’s AutoML engine to create a candidate ML firmware library for our target Espressif ESP32 device. This can be either using SensiML Analytics Studio, the turnkey subscription SaaS cloud service, or Piccolo AI, SensiML’s open-source variant, running on your own hardware.

In either instance, the workflow is similar. We simply following the steps as shown in the vertical menu bar on the left side of the UI progressing through the process. In the image below, we see the first step, Project Summary where ou can view/edit general overview information about the project.

To clarify a few of these:

• Captures: the number of sensor data capture files
  from Data Studio or imported CSV files from external tools

• Queries: the number of user-defined data queries
  offering flexibility to define various different slices of available data for model construction

• Pipelines: the number of user-configured ML pipelines
  allowing ML model configuration variants for test and validation of different approaches

• Models: the total number of models generated across pipelines
  each pipeline typically generates multiple candidate models from AutoML search to evaluate

We’ll skip the Data Manager tab as this simply provides another means within the Piccolo AI application to manipulate the training dataset we’ve already constructed in Data Studio.

Prepare Data - Defining the Model Query

The next step in the process is to define the subset of available train/test data that we wish to use for model building. The screen below under Prepare Data is where this gets defined.

For the drill demo, we’ll keep the query basic and just include everything we collected utilizing all source sensor channels except for the trigger channel. Trigger is a derived channel within our firmware that is simply a positive pulse output for all states where the motor voltage exceeds a threshold, so it offers no additional information beyond what is already in the motor voltage channel.

Build Model - Letting AutoML Work Its Magic

Now we’re ready to do some actual model building! We’ll start by selecting a generic AutoML template:

Next, we’ll establish the parameters for segmentation. This is an important attribute of our model and defines the region of data to be processed for classification. We will utilize a Windowing segmented that – as the name suggests – takes a window of data of fixed size and utilizes this for classification. The slide value is the step value between successive windowed segments to be classified. Since our data is collected at 250Hz sample rate, we’ll choose a 50 sample window size and a 50 sample slide value (for no overlap of successive classifications). This will give us a very responsive output with updated results every 200msec. For drilling, we want this fast result to detect cam out events the moment they occur. If we were to take the next step of using the output of the classifier as a motor control input, we’d want a fast responding model such as this to quickly react to faults.

The next decision within our AutoML modeling configuration is the choice of which feature generators to search across. Features are pre-classification data transformation algorithms that are chosen to accentuate differences in the various classes for the classification stage. For example, if the application was classifying which musical note is being played by a guitar, the most relevant feature from raw microphone sensor data would be a frequency feature (like FFT) to discern which note is being played. With over 80 feature extractors, it’s helpful to narrow down the list of features to search during the AutoML process so that we can arrive at candidate models more quickly. For our drill application, we’ll stick with basic statistical features.

With just these limited configurations, we’ll go ahead and run our pipeline and see what the AutoML process delivers for candidate models.

Exploring Candidate Models

After just a few minutes, we receive some candidate models for evaluation:

We learn a bit already from the summary table such as the overall accuracy, sensitivity, and F1 score for the models as well as the feature count. Generally models with higher feature count can be looked at suspiciously for possible overfitting, so we favor those that achieve good results with fewer features. Usually, the AutoML tool does a good job in ordering the candidates, so we’ll start by delving into the top model rank_0.

In the next tab Explore Model we can look at many details of each candidate model, from the confusion matrix (shown below) to feature plots, pipeline and model summaries, and the resulting knowledge pack.

We see from above that the rank_0 model has reasonably good performance with a slight tendency to falsely classify normal screw extraction and driving for cam out events. If we were unsatisfied with this level of accuracy, this would be our indicator to collect more data on cam out, driving, and removal segments to seek further improvement.

We can also look at the data at the feature extraction layer (before classification) to gain a sense for how well the features chosen are providing good class separation (clustering) of the data. We have 16 features in total and a limitation on our ability to graphically visualize in 16 dimensional space so we can choose to look at the any two features or better yet, choose a visualization technique based on the UMAP (Uniform Manifold Approximation and Projection) algorithm to reduce the dimensionality of our model for graphical visualization purposes (more on UMAP can be found here)

From the graph above, we can see good clustering of each of the classes with little overlap except the aforementioned confusion between Removing and Cam Out Fault states.

Testing Models

We can also test our data within the AutoML tool using either files we’ve excluded from the training process or with new files we subsequently capture and label with ground truth. The Test Model tab makes this process straightforward.

Shown above is a test run performed on our model showing the model confusion with Cam Out Fault and Driving and Removing events. With 84% accuracy, it appears we would benefit from further collection of model training data to build an ideal model that generalizes well. When collecting training data, we focused only on driving and removing operations under load (that is, with the bit engaged in a fastener being set or removed into a wood block.

HINT: What we did not collect was any free-running operation of the drill motor by itself not engaging a fastener. The result of failing to capture this common condition during the training data collection is the ML model delivering unexpected results during free-running conditions. This highlights an important aspect of training test design: Any condition you expect to encounter in predictive inference mode must be represented in the training data set itself.

Downloading Model Firmware

At this point, we’re getting close to having a working Smart Drill ML application. We need only to download the model library and integrate it into a functioning firmware application for the M5Stick ESP32 device.

To do so, we go to the Download Model tab and select Espressif ESP-IDF as our target.

In the following target configuration screen, we will leave all of the default settings except for changing the Application setting from AI Model Runner to Arduino Model Runner. This instructs SensiML’s model generator (Piccolo AI or Analytics Studio) to package the ML libraries in a manner that is readily imported into the Arduino IDE as we’ll see shortly.

Here it is also important to note the Class Map which informs us of how the specific classifications will be enumerated:

• Cam Out Fault = 1
• Driving = 2
• Off = 3
• Removing = 4

Not listed, but universally true across SensiML models is Unknown = 0

We’ll need this class map to define responses and behaviors of our application firmware which will call the model using kb_run_model and return the classification value from the class map above.

With the settings above and class map noted, we just need to click Download and await the downloaded .zip file containing the library code for our smart drill ML model. We’ll extract the files from this .zip archive to a folder on our PC. Within this folder we can see that it encapsulates yet another .zip archive called knowledgepack.zip. This .zip archive is formatted for easy inclusion into Arduino. If you examine its contents you will find a a directory called knowledgepack that contains a number of files including one called library.properties. This is a text file describing the library to the Arduino IDE and includes the following:

name=sensiml
version=1.0.0
author=SensiML
maintainer=SensiML <support@sensiml.com>
sentence=A library for SensiML ML Models
paragraph=Allows for Sensiml AI library in Arduino Projects. Sensiml provides the Library software subject to the licensing terms.  By using the Sensiml Library, you acknowledge and agree to the terms of the license contained at the link below.
category=Sensors
url=https://github.com/sensiml/
includes=kb.h,kb_output.h
architectures=*
dot_a_linkage=true
precompiled=full

For convenience, we can change the name to something more descriptive like name=SensiML Smart Drill ML Model so it’s easy to locate within the Arduino library listing. Just be sure to rezip any changes as the library must be in .zip archive form to import in Arduino.

Integrating the ML Model Library With Our Arduino Application

Integrating a SensiML ML model into Arduino code is quite straightforward and below is the process specific to the Smart Drill application. For reference we offer the sample code below which contains the application sketch file and necessary include files to compile the smart drill application:

Smart_Drill_M5StickC_PLUS.zip (2.5 MB)

Since we are targeting the M5Stack M5Stick C Plus device, it is necessary to load this board into the Arduino board manager. We’ll then configure to use this board, pick the right COM port to program it, and set the partition scheme to maximum available memory using Huge APP (3MB No OTA/1MB SPIFFS) since we’ll need enough memory to store our screen bitmaps.

Next we’ll import the SensiML Model library using the menu Sketch > Include Library > Add .ZIP Library…

And we’ll select the library we just imported to include in our Arduino Sketch. This will generate the #include statements in the top of our sketch for <kb.h> and <kb_output.h>.

To utilize our custom ML library, we need to minimally do three things:

1. Initialize the model before we start using it
2. Call kb_run_model to pass each successive sample array of all of our channels into the ML data processing ring buffer. This function call returns a negative value if there is no update in the inference result or the corresponding class number (see table given earlier) for a successful updated prediction value.
3. After each successful inference result, we call kb_reset_model(0) to reset the model

From there we can do what we wish with the results. For our sample application, we simply use the class index result to show a corresponding bitmap display on the M5Stick LCD that maps to the calculated result.

At the point we call kb_run_model, it’s vital that the pointer to the sensor array sample being passed matches the definition of sensor columns as given in the file model.json

Pulling It All Together - The End Result

With the integration of the model now done, we can test the result and see how well the model performs. In the video below you can see the output of the model as calculated directly on-device via the bitmap display on the M5Stick. You can also see the raw sensor data as streamed over WiFi to Data Studio which is running the model separately to provide overlay color bands representing the classifications of the model. Both are agreeing as expected.

Taking the concept further, we would likely do the following to improve and extend the application:

• Incorporate more training data covering the free running (no load) conditions where the drill is not actually driving or removing a fastener
• Add motor control circuitry and firmware logic to use the output of the model to intelligently control the drill’s motor and prevent cam out fault damage to the fastener and tool bit.