Primary Methods of Unsupervised Learning

There are a variety of ways to create a new machine learning model. Supervised learning is the simplest of these learning processes, but it requires human input and curated data sets. For a supervised learning process, you classify data with labels, then build a machine learning (ML) model around it. This ML model can then be used to classify new data in real time.

But what if you only have unclassified data (i.e data without any labels)? Is it possible to train a model with a data set like this? Can this be done without human curation?

Yes, leveraging unclassified data sets for model training is known as “unsupervised learning”.

What is Unsupervised Learning?

Unsupervised learning is also known as self-organization. It is a machine learning process that uses an algorithm for datasets which are neither classified nor labeled. In unsupervised learning, algorithms are allowed to act on data without guidance and they operate autonomously to discover interesting structures in the data based primarily on similarities and differences.

Let’s take a look at two of the most popular clustering and anomaly detection methods in use for unsupervised machine learning algorithms.

Types of Clustering 

1. K-means clustering

2. Hierarchical clustering

K-means Clustering

K-means clustering is a type of unsupervised learning, which is used when you have unlabeled data (data without defined categories or groups). The goal of this algorithm is to find groups in the data. It is intended to partition “N” objects into “K” clusters in which each object belongs to the cluster with the nearest mean.

Algorithm

The Κ-means clustering algorithm uses iterative refinement to produce a final result. The algorithm inputs are the number of clusters Κ, and the data set. The data set is a collection of features for each data point. The algorithm starts with initial estimates for the Κ centroids, which can either be randomly generated or randomly selected from the data set.

          Clustering data into K groups where K  is predefined

1. Select K points at random as cluster centers.
2. Assign objects to their closest cluster center according to the Euclidean distance function.
3. Calculate the centroid or mean of all objects in each cluster.
4. Repeat steps 2 and 3 until the same points are assigned to each cluster in consecutive rounds.

Choosing K

In general, there is no method for determining the exact value of K, but an estimate can be obtained by finding an “elbow point”. Increasing the number of clusters will always reduce the distance to data points, i.e. increasing K will always decrease this metric. This metric cannot be used as the sole target because when K is the same as the number of data points, then the metric approaches zero. Therefore, it is ideal to plot the mean distance to the centroid as a function of K. Then identify where the rate of decrease sharply shifts (i.e. the "elbow point"), and use this to determine K.

Hierarchical Clustering

Hierarchical clustering is an algorithm that groups similar objects into groups of clusters, where each cluster is distinct from each other cluster, and the objects within each cluster are broadly similar. For example, the organization of the files and folders on your personal computer is a hierarchy. Stepping into each of these folders will reveal more folders and files.

Working of Hierarchical Clustering

1. Start by assigning each observation as a separate cluster.
2. Find the clusters that are closest together.
3. Merge them into a single cluster, so that now you have one fewer cluster.
4. Repeat steps 2 and 3 until all items are clustered together.

Types of Hierarchical Clustering

a. Divisive

b. Agglomerative

a. Divisive

In divisive (top-down) clustering method we assign all of the observations to a single cluster and then partition the cluster into at least two similar clusters. We proceed recursively on each cluster until there is one cluster for each observation. Divisive clustering is conceptually more complex and thus, rarely used to solve real-life problems.

b. Agglomerative

Agglomerative hierarchical clustering (bottom-up), is a clustering method where we assign each observation to its own cluster. Agglomerative hierarchical clustering starts with every single object in a single cluster. Then, in each successive iteration, it agglomerates (merges) the closest pair of clusters by satisfying some similarity criteria, until all of the data converges in one cluster.

To determine the closest pair of clusters, the distance between each point is calculated using a distance function. These distances are generally called linkage between the clusters. There are three methods to determine the distance (linkage) between the clusters.

i. Single LinkageIn single linkage hierarchical clustering, the distance between two clusters is defined as the shortest distance between two points in each cluster.

ii. Complete LinkageIn complete linkage hierarchical clustering, the distance between two clusters is defined as the longest distance between two points in each cluster.

iii. Average Linkage

In average linkage hierarchical clustering, the distance between two clusters is defined as the average distance between each point in one cluster to every point in the other cluster.

Final Thoughts

Leveraging unsupervised learning to generate a machine learning model is now an accepted and feasible process to operate on unclassified data sets. While it’s more complex to set up and tune an unsupervised learning process, the benefit is that the source data does not have to be curated by a human curation team. This is a beneficial process when it’s not feasible or economical to curate the source learning data. In this article, we’ve outlined the core clustering and anomaly detection methods which are used to set up an unsupervised machine learning algorithm. We use unsupervised learning at IceCream Labs as one of the many machine learning processes for our Intelligent Data Mesh at the core of our solution.

Related e-commerce articles:

The Path to Upgrading Your Brick and Mortar Business 

Brick and mortar retail is often overlooked in the age of digital transformation. Many believed that the digital age would spell doom for physical stores. The “retail apocalypse” predicted the end of the brick and mortar retail model. However, this is not turning out to be the case, and a vast majority of shoppers still want to engage with retailers in a brick and mortar setting. A physical store gives customers the opportunity to get a feel for the products they are buying. It also presents retailers with a chance to provide customers with an unforgettable experience. Customers today want personalized shopping experiences.

Having a brick and mortar presence is also a good way to attract new customers. This is illustrated by the wave of innovative and trendy new retailers like Bonobos and Everlane who started off as online retailers but are looking to expand to physical stores across the country.

Omnichannel Experience

It is a mistake to believe that retailers are either only online or offline. Successful retailers today are operating across multiple different channels. The ability to deliver a highly engaging experience across all channels is the holy grail for retail success in the digital era. Brick and mortar form the cornerstone of this sophisticated omnichannel model of retail. However, physical stores must be able to provide customers with a multidimensional experience that touches all of their senses and enables them to connect with brands.

This retail experience begins the moment a customer enters the store for the first time. Customers intuitively react to the lighting, cleanliness, organization and flow of the store. Getting the physical design of the store right is crucial. So is the way products are arranged and displayed. Retailers today are also experimenting with technologies such as Augmented Reality(AR) and Virtual Reality(VR) to provide unforgettable in-store experiences to their customers.

Artificial Intelligence and Machine Learning

Artificial Intelligence(AI) and Machine Learning(ML) may sound like futuristic technologies, however, the reality is that these technologies are being widely applied in retail too. For example, AI and ML are already helping retailers make smarter choices when it comes to preventative maintenance. AI systems are used for product tagging and management, enabling retailers and employees to keep track of important products through a network of sensors. These technologies have changed the way retailers operate their business by enabling them to be able to understand what’s going on at all of their stores from an operations perspective.

There is no doubt that the brick and mortar retail model is here to stay. However, things are changing and the status quo is being disrupted. But this is only for the better. The importance of technology remains crucial for the success of any brick and mortar store. That is why you must understand and fully embrace the new technologies that can bring your business to a brighter future.

Related e-commerce articles:

Have you ever tried searching for a product on your favorite online shopping site, only to be disappointed when you couldn’t find the product that you’re looking for? Most product site search engines leverage accurate product categorization attributes to help narrow the search results for a user.

In this article we’re going to look at the impact that proper categorization has on search and how it’s now possible to automate product categorization with a machine learning model.

Categorization starts with a well designed product category taxonomy. The product taxonomy defines how each product type is related. The first couple levels of a product taxonomy contain broad category labels. For a grocery taxonomy, the top levels might be organized by departments within the store. It’s a logical representation of the way that a shopper would look for a given product in the physical store. A taxonomy is often referred to as a “Product tree”, with each product category referred to as a “branch” and each individual item referred to as a “leaf” on that branch.

Grocery taxonomy example:

1. Meat & Seafood

1. Fresh Meat

1. Ribs

2. Smoked Ham

3. Specialty Meat

4. Kosher Meat

5. ...

2. Fresh Seafood

3. Packaged Meat

4. Packaged Seafood

2. Produce

3. Deli

4. Bakery

5. Adult Beverages

6. Beverages

7. Floral

8. ...

For a new product to be put into the online product catalog, it first needs to be categorized appropriately into the correct level of the product taxonomy. This is easy enough for a human to complete the product categorization, however, when you have thousands and thousands of products, this can be a tedious process.

The science of search has evolved over the last two decades. Trying to determine the searchers intent from one or two words is not a simple process. We’re not going to dive into that in this article. However, in the specific use case of product search for an ecommerce website, most shoppers will generally include the object of their intent as part of the search input. In most cases this data can be used to quickly narrow the results set based on the product taxonomy. After all, the consumer isn’t looking for organic lettuce in the seafood section, nor would they be looking for seafood in the produce section. So one method to quickly close the search breadth is to narrow the search to specific sub-branch of the product taxonomy.

One downside to improper categorization is that improperly categorized products can become “lost”. When a product is mis-categorized on an improper branch of the taxonomy, the search engine may either (1) not find the product or (2) relegate the mis-categorized product to the bottom of the search results.

Don’t believe me? Try this: go to your favorite ecommerce provider, search for something, and then go to the last page of the search results. What do find there? Don’t let this happen to your product catalog.

In addition, the product category for a given catalog item can help define the product schema that should be employed to display the product information for the consumer on the product data page. The schema can also help define the meaning of generic product attributes, depending on the product type.

ATOM is the product categorization service from IceCream Labs. We developed ATOM as an API service which can be accessed automatically from your product information manager. ATOM takes a product title or description as an input and outputs the recommended product category for the item. ATOM is powered by a machine learning model that has been trained on millions of product records. It’s constantly learning as it processes new data.

With ATOM, you can properly categorize or validate a new product item before accepting it into your production product catalog.

To learn more about ATOM, or see a demo, contact our sales team: sales@icecreamlabs.com

PHOTO CREDIT: Photo by Min An from Pexels

Retail is changing at lightning speed and as we move towards the end of the year, as consumers begin anticipating what their shopping experience will look like in 2019. Retailers continue to evolve in a highly competitive world where delivery, customer experience, and convenience are the main factors that seal the fate of any store - forcing some into bankruptcy and propelling some into profits.

Here are five things to look forward to in retail next year, and most of them include technology:

More online grocery shopping

Despite having a small portion of consumers using online grocery shopping, industry experts expect digital sales to reach 20 percent of the total grocery market by 2025. Many retailers are partnering with third-party delivery companies such as Shipt and Instacart, enabling many consumers to order groceries from anywhere in a click or tap of a button. Soon, consumers will increasingly order online.

This includes both delivery and ordering online to pick up in store. It’s also expected that social media platforms like Instagram will continue discovering new ways to convince consumers to buy online.

Voice Retail

Experts say shoppers will increasingly pick up voice shopping through smartphones, Amazon devices, and vehicles.

Consumers with Alexa-enabled devices are already able to purchase their groceries, home goods, and gifts through Amazon and Whole Foods Market. But other retailers are starting to get in on the action.

Kroger recently announced plans to roll out voice ordering through Alexa-enabled devices and Amazon has released software that allows developers to integrate Alexa in vehicle infotainment systems.

More private labels

 Private labels have proven successful in the eyes of consumers this year. Dozens of retailers including Target, Kroger, Walmart, Aldi, and Amazon have expanded private label offerings this year.

Private labels are notorious for adding exclusivity that builds customer loyalty, all while keeping profit margins high without suppliers taking their cuts. Many of the retailers have passed the savings to the consumer with low-cost private labels that are increasingly growing in popularity.

Growth in artificial intelligence

Retailers have used artificial intelligence to learn consumer and market habits. The technology becomes increasingly beneficial for online retailers looking to upsell without a physical salesperson. Different subscription services like Stitch Fix and Kidbox have used AI to analyze subscriber data to recommend products that increase relevance and are more likely to be purchased.

Retailers are trying to use AI to expand holiday shopping earlier as well, learning what consumers will want most around the holidays as early in the year as possible. The intelligence can help spread out orders so delivery systems won’t become as congested close to the holidays.

More interactive aisles

As consumer shopping habits shift to favor experience, retailers are scrambling to find ways to draw crowds into stores. In 2019, augmented reality and virtual reality are likely to take a stronger foothold in all types of brick-and-mortar stores.

 For example, Kettering-based Marxent has developed augmented reality technology for Macy’s to show how furniture could look without having to purchase the items.

Leveraging AI and Machine Learning for Product Matching

There is a vast number of products sold online through various outlets all over the world. Identifying, matching and cross-checking products for purposes such as price comparison becomes a challenge as there are no global unique identifiers.

There are many situations where accurately identifying a product match is essential. For instance, stores may want to compare competitor prices for the same product they may offer. Similarly, customers may use comparison tools to get the best deals. Amazon allows different sellers to offer the same products only after ensuring that they are the same before listing the sellers in a single, unique product page.

Numerous products but no method to match them across different stores

Product titles/descriptions do not have a standardized format. Each store, as well as different sellers within a store, might have different titles and descriptions for the same products. Another challenge comes in with respect to attribute listings as different e-tailers follow different formats. The product images of the same product also differ across different e-tailers.

While there are standardized unique identifiers like UPC, MPN, GTIN, etc, they, however, may not be mentioned in the product page in all stores selling them. The attributes themselves may be described differently - for instance 9" and 9 inches. Images may be included but they can differ in perspective, clarity, tone, etc. The brand name may also be referred to in different ways like GE and General Electric.

<image>

It is an impossible task for a human to visit different product pages to ensure if they are matching the same products. Although, if the process is to be automated, how can it be ensured that the system makes sense of all the information. This is when AI and machine learning come into the picture. 

Machine Learning for Product Matching

In machine learning solutions for product matching, the solution provider must initially build a database with billions of products. This can be done by collecting information through web crawls and feeds. The system then has to come up with a universal taxonomy. This especially is a unique challenge as different retailers use different classifications for their products, and the same product might be listed in more than one category. For instance, a particular shoe model might be listed under casual shoes as well as dress shoes. The system first must design a standardized taxonomy, irrespective of how a particular store classifies its products.

There are standard classification models such as Google Taxonomy, GS1, and Amazon but a product match solution may devise its own taxonomy. The universal taxonomy is designed by identifying patterns and signals from titles, product descriptions and attributes, and from images.

Once a universal taxonomy is in place, the next step is making particular product matches. Here, there is a need for precise comparisons to ensure a particular product is indeed the same unique product, despite the differences in titles, images, descriptions, etc. First, there is a search for unique identifiers such as UPC or GTIN on the product page. Then, the product titles need to be compared. It needs to be noted that no two product titles are the same across different stores for the same product, for example:

Neural networks play a key role

Neural networks and deep learning techniques are extensively used to identify and learn from similarities, to identify and learn from differences, and produce word-level embedding to create a system of representation for common words. This involves teaching the system to recognize different references to a unique entity such as 'GE' and General Electric or 7" or 7 inches, to come up with one unique representation for each entity.

A product can be identified using its title, description, images and attributes or its specifications list. In many cases, the product title itself will yield a lot of information and the system needs to be trained to differentiate the product name (for instance, brand model) from the attributes.

<Phone model images>Samsung Galaxy Note 8 (US Version) Factory Unlocked Phone 64GB – Midnight Black (Certified Refurbished)Samsung Galaxy Note 8 is the phone model, and the title provides additional information like the memory size, US version, Factory Unlocked Refurbished, etc. 

Identifying and sorting product matches 

The information then needs to be extracted and sorted into the appropriate slots - Phone model, version, memory size, etc. Different techniques might be used to help the system learn to parse and sort the different sets of information. 

The next comparison comes in the form of more information about the product such as the title, description containing additional information and a specs table. These help add more knowledge about the product, and the machine will be better able to identify an exact product match or mismatch in the following comparison.

The standard identifying signals are similar results or positive matches for unique identification numbers (UPC or MPN), classification, brand, title, attributes, and image. For each comparison, the system follows a long procedure of checks or safety valves. The checks pass through a search for the unique identification number, a test for keyword similarities, brand normalization and match (for example, HP is the same as Hewlett Packard), attribute normalization and match ( 9 inches is the same as 9in, 9"), image matching, etc. There is also a check for variation in attributes such as:

For the best product match result, there has to be at least 99% of positive results. It will be considered a mismatch, even if it is a variation of what is essentially the same product. Different product match solutions employ different techniques and training methods, and it is a complicated process. Although, there is an advantage that neural networks and machine learning learn over time, and get better with each use.

E-commerce requires awesome product data to support successful search and conversion. Product data for the online grocery market is currently being created manually. Retailers are struggling to acquire the rich product data necessary to support their online needs. Brands are struggling to generate good data.

IceCream Labs CatalogIQ is designed to automatically extract attributes from product images. Using either label mechanicals or actual product images of the packaging, CatalogIQ can extract text from the labels. From there, the artificial intelligence in CatalogIQ understands what the text is and inserts it into the appropriate product attribute. The AI can also determine which images are the hero image, and front, back and side images.

CatalogIQ identifies brand names, sub brands and variants, normalizing the brand to the appropriate text. titles are generated from various attributes to create an SEO-rich title to optimize search. Other key attributes include feature/benefits, ingredients and nutrition facts.

CatalogIQ can score the data to help the merchandising and ecommerce teams understand which product records have been enhanced.

• Missing attributes
• Accuracy of attributes (are all of the attributes congruent?)
• How unique are the attributes?
• Is your product record SEO optimized?
• Do you have relevant search keywords?
• How well does your product data match up to customer site searches?

Grocery merchandising teams have the chore of uploading new catalogs from suppliers and manufacturers. Often this data arrives in the form of a spreadsheet. CatalogIQ can easily upload a new catalog file (in spreadsheet form) to quickly and easily complete the ingestion process.

Grocery retailer and channel partners expect high quality product data to list and sell your products online. Can you deliver the content?

CatalogIQ allows brand and product managers to auto-generate high quality product data directly from product label mechanicals and/or product images. If you're currently using manual processes to create product content and to check the accuracy of product data, then let CatalogIQ help you automate the creation process. You'll be able to complete the data creation process much faster than manual methods. CatalogIQ can also validate the content and ensure that it matches what is contained on all of the product labeling.

As a merchandising manager with a large product catalog, you know the difficulties of reviewing your product data and ensuring that everything in the catalog is ready to publish live to customers. There is always the nagging concern that something is inaccurate or missing when you push the “publish” button. Every time that you receive new data from your suppliers, it’s a chore to process the data. You have a long checklist to complete before you can publish data to the live catalog. Processing this checklist can consume all of your time.

People have increasingly become comfortable using technologies such as AI and machine learning in their day-to-day lives. Various companies also have increased their use of AI and machine learning into their product offerings as well as their processes. With the computer processing technology advancing increasingly, companies, institutions and even governments are gathering massive amounts of data as more consumer interactions move towards digital. This type of technology is already transforming the payments landscape in the following aspects -

Improving Efficiency

AI and machine learning have the potential to revolutionize the way payments are processed by enhancing operational efficiency and decreasing costs involved. For instance, AI enabled chatbots are reducing the load for customer service representatives.

With machine learning being incorporated into payments, learning algorithms play an important in helping speed along authorization of transactions and monitoring.
Furthermore, AI helps reduce the processing time for payments. It also helps eliminate human error and save time for correcting those mistakes.

For a business, processing large amounts of data to generate financial reports to satisfy regulatory and compliance requirements would involve a team that would perform repetitive data processing tasks. Leveraging AI would involve training the models to do these tasks, and the model can ensure completing the tasks faster and more accurately than humans. These technologies can improve efficiency while gathering important user insights in real-time.

Machine learning has already proved to be an invaluable part of fraud detection, but there are many opportunities that lie in product sales, customer care, and retention. Machine learning can draw vast amounts of available data and utilize it to profile customers to guess their product needs while offering new opportunities for upselling.

This model can also help identify which customers that companies are at risk for losing as well as halt customer loss before it happens. Machine learning can help automate many customer service interactions. This model which uses deep insights, cognitive engines and natural language processing is already widely available and the usage will only grow with time.

Fraud Prevention

​There are various methods for customers to make payments today. They are no longer limited to paying with cash or even cards. There are new payment methods on the rise such as card-not-present (CNP) transactions, but as it gets popular come new opportunities for fraud. AI and machine learning are at the forefront of not only detecting fraud but also preventing it before it happens.

These technologies already have the capability to uncover patterns and drive hidden insights and are working towards fine-tuning these insights. Companies are choosing to move away from a static model where the reliance is on supervised learning with input towards unsupervised learning wherein the deep belief neural network does not require a labeled training set, but continuously updates the model as new patterns emerge, allowing for a more robust and flexible fraud prevention detection tool.

As more commerce and payments move online, more data is accessible. This new robust algorithm uses machine learning to decrease the false positives and more agile detection of the actual frauds.

AI and machine learning have come a long way in the past decade. These technologies have already been adopted by many sectors and have transformed many aspects of traditional processes. Though exciting new technologies have been adopted by businesses to improve and enhance the payment process and customer experiences, the scope for future implementation is endless.

The retail experience of a shopper is the latest area that AI and machine learning are causing disruption. Most retailers recognizing the potential of these technologies have started aligning them into their business goals. Two crucial aspects - data and computing power have changed in the past few years in the space of AI, which has opened up new opportunities for retailers today.

Computing power is easy to see, with the advent and rise of smartphones which have phenomenal computing power when compared to the bulky phones and computers used decades ago. Businesses today have unlimited computing access to train their AI algorithms. Furthermore, the data available today is extremely rich and scalable. AI systems that leverage learning techniques such as Machine learning thrive on large, rich data sets. When fed appropriately, these systems discover patterns and correlations that would be otherwise difficult with a human intervention. These machine learning approaches automate data analysis, enabling users to create models that can then be used to make useful predictions about other similar data.

The speed at which AI can be deployed depends on specific critical factors. The first is the ability to test and measure. Retail giants can effectively deploy AI and test and measure consumer response. They can also leverage AI to measure the effect on their entire supply chain.

There is some innovative and interesting robot technology taking place in retail such as Grocery giant partnering with Nuro.AI to deliver groceries to the customers’ doorsteps. But most significant changes will come from the deployment of AI rather than the use of physical robots or autonomous vehicles.

Here are 3 AI-based scenarios that will transform the retail experience -

AI can detect underlying patterns in the shopping behavior of shoppers from the products that they buy and the method used to buy them. This could be a simple weekly purchase of groceries from the supermarket, the sporadic purchases of wine from the liquor store or the complex midnight icecream cravings from the local convenience store.

At a larger scale, analysis of the behavior of millions of consumers would enable supermarkets to predict the number of households that cook pasta every week. This would then inform the inventory management systems, and automatically optimize the stock of pasta. This information can also be shared with the suppliers, enabling more efficient inventory management and organized logistics.

The pricing challenge for supermarkets involves applying the right price and the right promotion to the right product. Retail pricing optimization requires data analysis at a granular level for each customer, product and transaction. To be effective, many factors need to be considered such as the impact of sales due to the changing price over time, seasonality, weather and competitors’ promotions.

A well-defined machine learning program can factor in all variations, including details such as purchase histories and product preferences to develop deep insights and pricing tailored to maximize revenue and ultimately, profit.

In the past, customer feedback was collected through forms and feedback cards that were filled out and placed in a suggestions box. The feedback had to be manually read and acted upon appropriately. With the rise of social media, the platforms were leveraged to express feedback publicly. Retailers subsequently engaged in social media scraping software to respond, resolve and engage with customers.

With the growing innovations, machine learning will play a larger role in this space. Machine learning and AI systems will be able to analyze unstructured data from multiple sources such as verbal comments or video content.  

The evolving retail experience

As a shopper moves through various stages in life, the circumstances and spending habits change. AI and machine learning models will adjust and be able to predict the needs of the consumer before the consumer even searches for a product.

This shift to predictive marketing would change the way shoppers purchase products, bringing in suggestions and recommendations that they would not have even considered. The possibilities would widen, all because of AI - for both consumers and retailers alike.