Big Data in Technology Intelligence


These days, even the smallest company has recognized that processing large amounts of data is the way of the future and that the event landscape is teeming with events on Big Data, too. Today's challenges of Big Data and how to overcome them are especially apparent in the field of Technology Intelligence.

"640 K ought to be enough for anyone." This statement is attributed to none other than Microsoft founder, Bill Gates. Regardless of whether or not it's an urban legend, this quote shows that even proven experts aren't immune to the misjudgement of technological development.

Fortunately, today's computer are equipped with much more memory, thus enabling users to anticipate technological trends and analyze them. But in order to identify the main trends while facing a rapidly changing landscape of innovation and make reliable future decisions, enormous amounts of data need to be collected, processed, and interpreted in context. This is the task of Technology Intelligence (TI) or Technology Foresight.

Let's think about an everyday example: A high-tech entrepreneur would like to know which area of technology he should invest in. For that purpose, he needs reliable information showing him a number of relevant factors to consider: what is being researched and developed worldwide; how companies, researchers and developers act together; where the individual actors are located; who has which skills; and, in which direction the trend is shifting.


Heterogeneous and high-dimensional data

Some may argue that a simple Google search would suffice to identify relevant researchers. The results would provide a list of possibly millions of relevant documents that need to be sifted through.

In fact, today's computing power makes it fairly uncomplicated to scan even billions of documents with a full-text index for specific keywords. But it looks very different when network relationships between the individual data are revealed. With a network of only 100,000 actors interacting with each other, there are theoretically almost 5 billion pairs to be compared to uncover all collaborations.

In addition, you could allocate a variety of other features to all of the actors in this network such as a company, a university, a country of origin, the number of patents, etc. In practice, you have to deal with high-dimensional data that can pose a great challenge for statistical analysis.

Last but not least, the data is by no means always presented in a comparable form. If you want to evaluate patent data, you would be confronted with an unmanageable mass of more than 100 million patents and 200 million events regarding patenting, which are listed in more than 100 patent offices worldwide. The documents are written in different languages, refer to non-uniform structures in each country, and track a variety of patenting processes.

In working with big data, it is therefore important not only to master the sheer quantity and to count but also to link, evaluate, and put the individual information and dimensions in the right context - everything in real time. In our example, the high-tech entrepreneur is interested in how much money an institution is investing in a certain technology, who and how much experts are working on it, how competent and linked they are, how radical and innovative their new ideas are and how globally a competitor thinks and prepares himself.

In the Technology Intelligence branch, this high-dimensional and highly heterogeneous data is utilized with methods such as data scraping, data cleaning, and data mining. Thus, the search for correct and sufficient indicators is very important. Should the technology entrepreneur want to exploit patent data, he  needs to evaluate the impact of an invention or a portfolio of inventions and patents, their technological width, the degree of innovation, market coverage, and costs.


From Big Data to Smart Data

We must not forget: The greatest treasures of data are hardly worth anything without the human beings who enhance and interpret them. Humans play a central role at each step, from the selection of relevant data to choosing statistically significant indicators, and from using the appropriate methods of analysis to presenting the data in a form that enables other people to see the context and informational content.

Ultimately, the meaningful use of Big Data must always strive to give people the tools with which they can explore data easily and which are actually fun. If the enormous potential of Big Data should be rolled out for as many users as possible, delicate and complex scientific expert tools are rarely useful.

One path that the latest Technology Intelligence tools are following is the translation of complex data into intuitive visualizations that reveal the most important findings at a glance. Here, you can hardly see which enormous efforts are taking place "under the hood". In this manner, Big Data can become Smart Data and minimize the risks of miscalculation, such as the Bill Gates' inaccurate 640 K estimate.


Dr. Peter Walde is the founder and CEO of the Big Data and Visual Analytics company, mapegy GmbH. The Berlin-based company sees itself as the "Compass for the High Tech world". With their web-based analysis and visualization software, mapegy measures and monitors innovation, competition and technology trends for technology decision makers.

More information: www.mapegy.com

+49 (0)30 430 2212 0