Some Things DO Change: Understanding Systems of Insight

Posted on Wed, Sep 7, 2016 @ 3:00 pm

The old expression, “Some things never change,” is one that I’ve lived by in my 40 years in the software business—until I learned it was wrong! I thought I was so smart, because I could see the patterns in new technology repeating over the years, and I got good at mapping the latest technology terms and tools back to the older ones I learned. This worked for the most part, since some principles of software and IT are universal and transcend specific technologies. For example, the notion of dividing your program into components makes sense for lots of reasons, whether they’re called subroutines, DLLs, objects or services. However, I have to say that recent technologies—those from 2010 and beyond—have defied my attempts at simple classifications.

Specifically, the term systems of insightapplies to a terrific set of new technologies designed to harvest the vast amount of data we now have on customers, then look for patterns and gain insight into their behaviors and desires based on those data patterns. These technologies, which include machine learning (ML), natural language processing (NLP), and descriptive and predictive data analytics, offer some amazing new properties through their associated reporting and graphing tools. As an old curmudgeon in this field, I thought these new tools were just rehashed, old concepts from the 80s and early 90s: artificial intelligence, expert systems, and report writers. While they do bear some resemblance to the old tools, these new technologies have come so far and so fast, they really are something altogether new.

What the old systems have in common with their newer counterparts is the ability to look at acquired data and see patterns, whether through sorting and filtering, then applying business rules and finally graphing the trends. However, the new tools I mentioned do this on a massively larger scale, at an incredible speed, and across multiple data source types, all while dealing with far more complex relationships. Their ability to handle trillions of bytes of data and make real-time decisions based on their insights goes so far beyond what’s been done in the past.

It’s not just the old, structured data about our customers that gets scanned (like purchase history and demographics), but much more personal information gathered from their cell phones and other connected devices. Today, we can know where our customers are physically located at any given moment. We can know if they’re in motion, their heart rate, what they’ve looked at online recently, what they’ve posted on social media, where they’ve traveled recently, and with whom they’ve connected on Facebook or LinkedIn.  We can know the temperatures of our customers’ homes, how they drive, their style of writing emails, the foods they enjoy at restaurants, the movies they like to watch, and so on.

The systems of insight tools, different from those used in systems of engagement or systems of record, seek to digest all this newfound customer data and create insights into a customer’s real motivations and what he or she is most likely to want or need next. Big data tools, such as Hadoop and Cassandra, give us the ability to capture and store all this diverse data, regardless of whether it’s text, pictures, sounds, documents, videos, etc.

ML tools, NLP tools and streaming analytics tools can then be used to apply advanced mathematical models and algorithms that enable us to quickly digest all this data, identify patterns, uncover hidden insights, and build ethnographic profiles of our customers to predict what they might need and—more importantly—even understand why they might need it.

ML tools such as Apache Spark, with its MLlib and GraphX graphics framework, can be connected quickly to a live stream of data from Twitter to analyze the tweets, determine their sentiment and graph the trend lines. We at OFS did this recently as a demonstration of the power of these tools. We were even able to build this in a day, and you can see a demo of it here. These ML and NLP tools are not just for social media or mobile data. You can hook up your traditional structured data stored in SQL databases by using a massively scalable and distributed pub/sub tool such as Apache Kafka to help you push a stream of data into ML tools.

Many companies are using these technologies with incredible effectiveness. For example, Stitch Fix is an online women’s apparel vendor that uses machine learning to predict which kind of clothes to send to its customers on a monthly basis. Not only does the software learn what a particular woman likes and dislikes (based on indicated preferences, social media analysis, and returns of unwanted merchandise), but the analytical model also can be improved by experts within the company who have unique insights into that particular woman from prior dialog with her. Stitch Fix sales have skyrocketed in the past three years as they have deployed these technologies.

These tools are modern weapons you must consider to stay relevant and competitive in your marketplace, because they allow you to get inside your customers’ heads to know what products and services you should offer to meet their hidden wants and needs. For those of you older types like me who think nothing ever changes, I invite you to take a look at systems of insight tools and see that things sometimes—or rather, always—DO change.


One thought on “Some Things DO Change: Understanding Systems of Insight

  1. Ron Pickle

    Data science has become really huge and has evolved by several notches since the past 8 years, when it used to be more of a concept. Earlier software applications were primarily database driven ones but now source of data is diverse and much varied and hence its interpretation has become tough leading to complex data analysis applications, leading to a completely different ecosystem of technologies and tools and hence gave birth to new lexicon.

Leave a Reply

Your email address will not be published. Required fields are marked *