Introduction

We live in a world that’s keen on efficiency and on producing the maximum amount of any product, be it cars or food. Everyone involved is always looking for ways to make their work better, reduce mistakes, and do more in less time. Counting and sorting items accurately is a big part of being more efficient. In the past, this was a job that took a lot of time and had a high margin of error. But now, with technology like artificial intelligence (AI), counting and sorting things has become much easier and more accurate than ever before.

In this article, we’ll explore the importance of using AI techniques for counting and sorting tasks, dive into some of its applications, and discuss some future potentials and challenges. We will also try out a couple of hands-on code examples for counting and sorting using computer vision techniques. Let’s get right to it!

Understanding AI in Sorting and Counting

AI has changed the way many industries work by making machines do tasks even more precisely and quickly than humans. For tasks like counting and sorting, AI systems can use techniques like computer vision, machine learning, and natural language processing to accurately and efficiently identify, count, and sort different items. Let’s take a closer look at these methods.

Computer Vision

Computer vision is a branch of AI that is essential for AI-driven counting and sorting. It uses IoT cameras and sensors to capture detailed images of products, components, or any materials. Then, image processing techniques like object detection can be used to count related tasks. Once objects in an image are detected, we can extract the detected information and count them. For sorting-related tasks, we can use computer vision techniques like instance segmentation to detect and identify the masks of the objects and sort them based on the area of the mask.

Those are just two examples of how computer vision can be used for sorting and counting. Computer vision models can be trained to identify objects of different shapes, sizes, colours, and textures. This makes them suitable for various applications, from manufacturing automotive parts to food packaging.

For instance, conveyor systems are crucial in manufacturing as they keep products moving along the production line. When combined with computer vision techniques, they offer more benefits, such as customised sorting and real-time feedback. These applications increase overall productivity, making them far superior to traditional conveyor systems.

A conveyor system integrated with computer vision. Source: Blumenbecker

Machine Learning

We can also perform counting and sorting using machine learning methods. When dealing with numerical data, the process of counting begins by grouping similar data points together using algorithms such as K-means or DBSCAN. These algorithms help organise the data into clusters or categories, making it easier to identify and count each group. Once the data is grouped, the next step is to aggregate counts. You can count the number of data points in each cluster or category and understand the distribution of the data.

Sorting-related tasks, on the other hand, involve arranging the data based on specific criteria. This could be the magnitude of the data (e.g., sorting numbers from smallest to largest), the frequency of occurrence (e.g., sorting words by how often they appear), or a computed metric such as the mean or median value within each category.

Natural Language Processing (NLP)

It is also possible to perform counting and sorting through natural language processing. Counting related tasks using NLP can be done in several ways. Here’s one way:

• Named Entity Recognition (NER) can help identify and classify named entities in the text, such as people, organisations, and locations, and place them into predefined categories.

• Frequency analysis is performed on the identified entities or words to count how often each one appears. This helps understand the text’s content distribution.

• Then, we can apply topic modelling or categorisation techniques like Latent Dirichlet Allocation (LDA) to group the text into topics or categories. The number of documents or text snippets in each category or topic is then counted.

Relevant attributes, like word frequency or sentiment scores, are extracted from the text for sorting. These attributes are then used to establish the sorting criteria, such as sorting by sentiment score or the number of times a specific entity is mentioned. After that, we can implement a sorting algorithm based on that criteria. The choice of sorting algorithm can depend on the data and sorting needs. Whether you are sorting by numerical values, alphabetical order, or a custom ranking system depends on your sorting objective.

Now that we’ve understood how various methods are used to implement sorting and counting, let’s take a closer look at computer vision and YOLO Models and check out their benefits. We’ll be using both of these technologies for our coding tutorial later on.

Harnessing Computer Vision and YOLO Models

Computer vision allows computers to interpret and analyse the visual world, mimicking human vision. In a way, it gives sight to machines by using machine learning methods to recognise and categorise objects in images and videos. Various types of computer vision techniques exist, including image segmentation, object detection, and image classification.

The YOLO models are unique because their architecture supports real-time image analysis. Unlike traditional methods that separate object identification and classification, YOLO performs both tasks in one step, earning its name “You Only Look Once.” Recently, we’ve seen multiple versions of YOLO released, from YOLO to YOLOv9.

YOLOv8 has become widely popular for supporting multiple computer vision techniques, including instance segmentation, and performing very well on images. YOLO-World was recently released and was received with excitement by the AI community. It is a zero-shot object detection model that you can textually prompt. This means that without training YOLO-World, you can prompt it to detect different classes, as shown in the image below.

An example of what YOLO-World can do. Source: YOLO-World.cc

We’ll be using both YOLOv8 and YOLO-World for our counting and sorting coding in our coding examples. Before that, let’s take a quick look at the various benefits of using such AI-integrated systems for sorting and counting tasks to understand why the coding example you are about to walk through is important.

Benefits of Using AI for Counting and Sorting

Using AI technologies like computer vision for sorting and counting has many benefits. These systems can be incredibly accurate and precise while reducing the risk of errors. Accuracy is a vital parameter in industries where product quality and safety are of most importance, such as pharmaceuticals and aerospace.

These systems can work at high speeds and ensure that items are counted and sorted as quickly as possible. Increased efficiency translates to higher production rates and reduced lead times. Such systems can also reduce manual labour for counting and sorting tasks, which leads to cost savings in the long run.

Also, AI counting and sorting systems can generate valuable data that can be used for process optimisation and quality improvement. Manufacturers can use this data to gain insights into production trends, identify bottlenecks, and make data-driven decisions to improve production.

Applications of AI Sorting and Counting

Now that we’ve gained a solid understanding of the benefits of AI in sorting and counting tasks let’s dive into the real-world applications of AI sorting and counting systems.

Automotive Manufacturing

In the automotive industry, AI-powered counting and sorting systems are used to ensure the accurate use of fasteners like screws and bolts in vehicle assembly. Robots with machine vision and deep learning algorithms identify and differentiate fasteners, detecting defects and removing faulty ones with robotic arms. This enhances assembly precision and vehicle safety.

A robotic arm with machine vision monitoring fasteners. Source: Sciotex

Using pseudo-code, we can understand what the algorithm or method might look like for this type of AI sorting and counting system:

• Step 1: Equip the robotic arms with high-resolution cameras and sensors and ensure proper lighting for optimal image capture.

• Step 2: Continuously capture images of the fasteners on the assembly line.

• Step 3: For each captured image:

• Step 3.1: Use a Convolutional Neural Network (CNN) model like YOLO to detect fasteners in the image.

• Step 3.2: Count the number of detected fasteners.

• Step 3.3: Classify each detected fastener by type (e.g., screws, nuts, bolts) and check for defects. You can also apply a separate classification model or an integrated model that also performs defect detection.

• Step 4: Aggregate the counts and classifications for each type of fastener over a specified time frame. Maintain a separate count for the defective parts so they can be sorted out for proper quality control.

• Step 5: Process the aggregated data directly on the assembly line’s control system to minimise latency.

• Step 6: Transmit quality control alerts and inventory updates to the central manufacturing system.

Read more: AI is Reshaping the Automotive Industry

Traffic Management

AI counting and sorting systems can be easily implemented in traffic management using IoT Edge cameras and sensors. These systems are highly efficient when it comes to counting the number of running vehicles on the street at any given time. This system also uses an object detection model to detect vehicles. Once the counting and sorting results are processed locally by the edge devices, they are transferred to a server on the cloud where the traffic information is analysed and interactively visualised. Compared with traditional cloud paradigm systems, this approach significantly reduces the end-to-end latency and network resources.

Using AI to count and sort traffic. Source: Hindawi

Using pseudo-code, we can understand what the algorithm or method might look like for this type of AI sorting and counting system:

• Step 1: Place the IoT Edge cameras and sensors at strategic locations on the street, providing an excellent view of traffic.

• Step 2: Capture real-time video feed from the cameras.

• Step 3: For each video frame:

• Step 3.1: Use a real-time object detection model to detect vehicles in the frame

• Step 3.2: Count the number of vehicles detected in the frame.

• Step 3.3: Sort or classify the vehicles by type if necessary (e.g., cars, trucks, motorcycles).

• Step 4: Aggregate the counts and classification over a specified time window to form traffic data.

• Step 5: Process the aggregated traffic data locally to minimise latency using edge devices.

• Step 6: Transmit the processed traffic data to a server on the cloud.

• Step 7: On the cloud server:

• Step 7.1: Receive and store the incoming traffic data.

• Step 7.2: Analyse the traffic data for patterns, trends, or anomalies.

• Step 7.3: Generate interactive visualisations of the traffic information. This can include maps, graphs, and charts to represent traffic flow, density, and types of vehicles.

Read more: Exploring AI’s Role in Smart Solutions for Traffic & Transportation

Pharmaceutical Packaging

All pharmaceutical companies must follow strict quality control standards, and there is zero tolerance for error. AI counting and sorting systems can be used to count and sort medications into blister packs or bottles with incredible accuracy and precision. Such systems use 360-degree inspection technology that includes binocular cameras, online culling mechanisms, and micro-cameras for capturing and inspecting products. AI visual inspection platforms allow for easy monitoring of the process, minimising defects and ensuring a 100% qualification rate.

AI-powered Pill Counting and Sorting Machine. Source: Pharmapack

Using pseudo-code, we can understand what the algorithm or method might look like for this type of AI sorting and counting system:

• Step 1: Initialise the AI visual inspection platform fitted with visual inspection devices like 360-degree binocular cameras and micro cameras.

• Step 2: Start the conveyor belt to move products into the inspection zone

• Step 3: Capture images using 360-degree inspection technology

• Step 4: While analysing images with the AI system:

• Step 4.1: The AI system processes the captured images to identify the products.

• Step 4.2: Here, we obtain a proper count of the products entering the system, including the count of detected products.

• Step 4.3: We also maintain other quality control parameters set by the pharmaceutical standards.

• Step 5: Based on the AI analysis, determine whether each product meets the quality standards.

• Step 6: Sort products into categories: approved for packaging, needs re-inspection, or discarded due to defects.

• Step 6.1: Approved products are sorted into their respective blister packs or bottles.

• Step 6.2: Products needing re-inspection are moved to a separate area for further examination.

• Step 6.3: Defective products are discarded or marked for analysis to understand the defect source.

Read more: AI in Pharmaceutics: Automating Meds

Agriculture

Integrating AI into agricultural management is essential for a productive future, promising higher yields, improved animal welfare, and increased efficiency. AI-driven counting and sorting systems on aerial drones equipped with computer vision are transforming livestock management. These drones autonomously monitor animal populations over large areas, applying instance segmentation and deep learning to detect, identify, and categorise livestock by various characteristics. They also alert owners to any missing animals, providing constant surveillance.

Read more: Smart Farming: How AI is Transforming Livestock Management

Instance Segmentation on Livestock. Source: Labellerr

Using pseudo-code, we can understand what the algorithm or method might look like for this type of AI sorting and counting system:

• Step 1: Integrate a drone with computer vision capabilities

• Step 2: Fly the drone over farm areas that need to be monitored

• Step 3: Continuously capture high-resolution images and video footage as the drone flies.

• Step 3.1: Use deep learning models, such as segmentation, to process the visual data from the drone.

• Step 3.2: Identify and classify livestock based on size, colour, gender, etc.

• Step 3.3: Detect any unusual behaviour or signs of illness in animals. Track movement patterns to ensure all animals are within safe zones.

• Step 4: Automatically generate alerts if any livestock is missing or shows signs of distress. Provide comprehensive reports on the status, health, and distribution of livestock.

Code Walk-Through

The applications you went through above gave you a rough idea of how these problems are approached. Now, let’s go a step further and see how they are solved.

Food Processing Industry

Let’s take the food processing industry as an example. With the market size of AI in food and beverages expected to reach around $214 billion by 2033, many companies are considering adopting AI systems for food processing.

These companies can use AI to count and grade food products and monitor food quality. By improving transparency and traceability in food supply chains, AI systems have the potential to address challenges related to food safety, quality, and wastage.

Read more: How the Food Industry is Reconfigured by AI and Edge Computing

Let’s walk through some simple codes integrating AI features to count and sort apples. The counting will be done based on the quality of the apples, and sorting will be done based on their size in the image. Let’s get started.

Code for Sorting Apples Based on their Size

Our objective is to use YOLOv8’s ability to segment objects in an image to identify and segment apples in the image. As an output, we’ll get the mask of the identified apples. Then, we’ll calculate the area of the mask, sort the apples based on the area, and display only the larger apples. Let’s look at this step by step. Ensure to download an image of apples to use from the internet.

Step 1:

Install the required modules using Pip.

pip install ultralytics opencv-contrib-python

Step 2:

Import all the needed packages.

from ultralytics import YOLO 
import numpy as np
from pathlib import Path
import cv2

Step 3:

Load the YOLOv8 instance segmentation model. Since apples are a part of the COCO dataset classes, it is pre-trained to detect them.

model = YOLO('yolov8n-seg.pt')

Step 4:

Here, we set the conversion ratios. These ratios convert pixels to centimetres and square centimetres. The value can change depending on the resolution of your image or video frame.

RATIO_PIXEL_TO_CM = 78  # 78 pixels are 1cm
RATIO_PIXEL_TO_SQUARE_CM = 78 * 78

Step 5:

Next, we predict with the model. You can also print the detection results using ‘results[0].show()’. We’ll also create a list to store the area calculated. Make sure you have good input images like the ones shown below.

results = model.predict('path/to/image')
results[0].show()

Output Images of YOLOv8 Instance Segmentation.

Step 6:

In this step, we’ll iterate over the detection results and process each detected object. When we iterate over the object contour, we get its label. We’ll also create a binary mask for the contour and draw the contour on the mask.

area_list = []
# iterate detection results
for r in results:
    img = np.copy(r.orig_img)
    img_name = Path(r.path).stem

    # iterate each object contour
    for ci, c in enumerate(r):
        label = c.names[c.boxes.cls.tolist().pop()]

        b_mask = np.zeros(img.shape[:2], np.uint8)

        # Create contour mask
        contour = c.masks.xy.pop().astype(np.int32).reshape(-1, 1, 2)
        _ = cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)

Step 7:

Next, we calculate the area within the contour in square centimetres using the values from Step 4. Once we have the area, we append the values to the list we created in Step 5.

  # Detection crop
        x1, y1, x2, y2 = c.boxes.xyxy.cpu().numpy().squeeze().astype(np.int32)

        # Calculate area within the contour
        roi = img[y1:y2, x1:x2]
        grey = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
        _, threshold = cv2.threshold(grey, 150, 255, cv2.THRESH_BINARY)
        contours, _ = cv2.findContours(threshold, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        area_cm = 0
        for cnt in contours:
            area_cm += cv2.contourArea(cnt) / RATIO_PIXEL_TO_SQUARE_CM

        # Display the image with the area calculated
        cv2.putText(img, "Size: {}".format(round(area_cm, 2)), (x1, y1), cv2.FONT_HERSHEY_PLAIN, 1, (255, 0, 255), 2)

        area_list.append(round(area_cm, 2))

Step 8:

To see how the image looks with all the sizes calculated and displayed, add this line to the code.

cv2.imshow('All Sizes', img)

Calculate and Display Apples of All Sizes.

Step 9:

Sorting the list of areas and filtering out the largest 50% of the values. Store the filtered values on a new list.

test_list.sort(reverse=True)
length = len(test_list)
half_index = 1 if length == 1 else length // 2
largest_50_percent = test_list[:half_index]

Step 10:

Most of the code in this step is the same as that of Steps 6 and 7. Instead of displaying all the sizes, only display the largest 50%.

# Iterate detection results
for r in results:
    img2 = np.copy(r.orig_img)
    img_name = Path(r.path).stem

    # Iterate each object contour
    for ci, c in enumerate(r):
        label = c.names[c.boxes.cls.tolist().pop()]

        b_mask = np.zeros(img2.shape[:2], np.uint8)

        # Create contour mask
        contour = c.masks.xy.pop().astype(np.int32).reshape(-1, 1, 2)
        cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)

        # Detection crop
        x1, y1, x2, y2 = c.boxes.xyxy.cpu().numpy().squeeze().astype(np.int32)

        # Calculate area within the bounding box
        roi = img2[y1:y2, x1:x2]
        grey = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
        _, threshold = cv2.threshold(grey, 150, 255, cv2.THRESH_BINARY)
        contours, _ = cv2.findContours(threshold, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        area_cm2 = 0
        for cnt in contours:
            area_cm2 += cv2.contourArea(cnt) / RATIO_PIXEL_TO_SQUARE_CM

        # Display the image with the largest area calculated
        if round(area_cm2, 2) in largest_50_percent:
            cv2.putText(img2, "Size: {}".format(round(area_cm2, 2)), (x1, y1), cv2.FONT_HERSHEY_PLAIN, 1, (255, 0, 255), 2)

Step 11:

Code to display the final result and close the windows.

cv2.imshow('The Biggest Ones', img2)
cv2.waitKey(0)
cv2.destroyAllWindows()

Display the apples with the largest area calculated (largest 50%).

Great Job! Now, let’s look at counting apples.

Code for Counting Apples Based on Their Quality

Our objective is to use YOLO-World to detect apples in an image without training the model. We’ll detect red apples as ripe and green apples as unripe. Let’s go through the steps involved. Ensure to download an image of apples to use from the internet.

Step 1:

Install the required modules using Pip.

pip install ultralytics supervision opencv-contrib-python

Step 2:

Import all the needed packages.

from ultralytics import YOLOWorld
import supervision as sv
import cv2

Step 3:

Initialise the model and set the custom classes. It’s that easy.

# Initialise a YOLO-World model
model = YOLOWorld('yolov8s-world.pt')
# Define custom classes
model.set_classes(["Red Apple", "Green Apple"])

Step 4:

Here in this step, we read the input image and perform the predictions. Add (results[0].show()) to display the results of the prediction.

img_path = "Image.png"
img = cv2.imread(img_path)

# Execute prediction for specified categories on an image
results = model.predict(img)

# Display the prediction results
results[0].show()

The output of object detection using YOLO-World.

Step 5:

In this step, we’ll convert the ‘ultralytics’ detections to the ‘supervision’ format. Then, we’ll store the detected label data in a list.

detections = sv.Detections.from_ultralytics(results[0])
detection_list = detections.data['class_name']

Step 6:

Now, we initialise the counters for red apples (ripe) and green apples (unripe) and activate the counter by iterating over the detected label values from the list from step 5.

# Initialise counters
red_count = green_count = 0

# Count the number of each type of apple
for item in detection_list:
    if item == "Red Apple":
        red_count += 1
    elif item == "Green Apple":
        green_count += 1

Step 7:

Display the calculated counts on the image.

# Display the counts on the image
font = cv2.FONT_HERSHEY_SIMPLEX
cv2.putText(img, f'Ripe Apples: {red_count}', (10, 30), font, 1, (255, 255, 0), 2, cv2.LINE_AA)
cv2.putText(img, f'Unripe Apples: {green_count}', (10, 60), font, 1, (255, 255, 0), 2, cv2.LINE_AA)

Step 8:

Show the final results.

cv2.imshow('Apples', img)
cv2.waitKey(0)
cv2.destroyAllWindows()

The output displays the count of ripe and unripe apples.

The Future Implications and Challenges

Integrating AI into systems for counting and sorting marks a significant shift towards automation in various industries. We are going to start seeing this more and more in robotics. These robots, equipped with advanced optical sensors can track and analyse the quality of products in real-time and also enhance data collection on materials. This in turn, paves the way for continuous improvement and predictive maintenance.

However, adopting AI-driven counting and sorting systems presents challenges, including the initial high costs of machine vision technologies and the need for specialised training for employees. Also, data security and the necessity for regular system maintenance are crucial considerations. To fully harness AI benefits and mitigate these challenges, it’s essential to focus on long-term gains, ensure robust data protection, and design scalable systems capable of adapting to future needs.

What We Can Offer as TechnoLynx

At TechnoLynx, we specialise in delivering custom, innovative AI solutions specifically tailored to unique business challenges. We understand the difficulties of integrating AI into different industries and public sectors. Our expertise covers enhancing AI capabilities, ensuring efficiency, managing and analysing extensive data sets, and addressing ethical considerations.

We offer precise AI-driven software solutions designed to empower enhancements across many industries. Our expertise in computer vision, generative AI, GPU acceleration, and IoT edge computing can help you explore many possibilities. We aim to push the boundaries of innovation while ensuring adherence to vigorous safety standards. For more information, feel free to contact us.

Conclusion

AI improves various aspects of our lives when it comes to sorting and counting objects. From manufacturing systems to food production and processing, AI-based sorting and counting systems are used. However, the integration of AI into these sectors is not going to be easy and presents many challenges.

Addressing these challenges requires a collaborative approach among all stakeholders, including an AI solution provider that fully understands and addresses your concerns. At TechnoLynx, we specialise in providing customised AI solutions to navigate these challenges effectively, pushing the boundaries of innovation while ensuring safety and transparency.

Sources for images:

• Blumenbecker GmbH. (n.d.). Machine Vision: More than cameras

• Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., & Shan, Y. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection. arXiv preprint arXiv.

• Dinh, D.-L., Nguyen, H.-N., Thai, H.-T., & Le, K.-H. (2021). Towards AI-Based Traffic Counting System with Edge Computing. Journal of Advanced Transportation, 2021, Article ID 5551976.

• Kumari, P. (2023, October 16). Smart Animal Livestock Management Using Vision AI. Labellerr

• Pharmapack Technologies Corporation. (n.d.). 360 Filling & Sorting Machine

• Sciotex. (n.d.). Using AI for Counting and Sorting Products: Revolutionizing Efficiency in Quality Control

References:

• AltexSoft. (2023, November 1). Named Entity Recognition: The Mechanism, Methods, Use Cases, and Implementation Tips

• Bandyopadhyay, H. (2022) ‘The Definitive Guide to Instance Segmentation [+V7 Tutorial]’, V7 Blog

• Precedence Research. (n.d.). AI in Food and Beverages Market

• Skalski, P., & Gallagher, J. (2024, February 13). YOLO-World: Real-Time, Zero-Shot Object Detection. Roboflow Blog

• Ultralytics. (2024). Ultralytics YOLOv8 Docs