NLU @ DataVal Analytics

DataVal Analytics has created a core engine that helps machines understand natural language. We believe that Natural Language Understanding (NLU) capability is of significant importance for building the next generation of applications. Our passion in this field led us to leave our safe havens to pursue our passion and reach the ultimate destination of NLU. DataVal Analytics, under the mentorship of Dr Sam Pitroda, embarked on this long and arduous journey to build a core engine that can understand natural language inputs.

We realized that most of the current global effort is focused towards a data driven approach based on statistical modelling. After deliberate research and trying multiple existing approaches we decided to take the road less taken. So we focused our effort towards understanding how humans processed Natural Language. Our core engine uses syntactic and semantic processing of textual inputs using knowledge of the real world to extract information. The extracted information is stored in a proprietary framework that enables context tracking of entities along with time and space reasoning.

Hypothesis

While a small child can understand and speak Natural Language, the most complex computer fails to achieve this simple task. We hypothesized that the main component of human understanding is to understand the objects and concepts in the world that humans learn through experience. There is also a need to know the cause and effect of actions performed by different objects and formalize the relationship between them. We also hypothesized that to use the information available, machines need the capability to carry out analysis and reasoning.

Approach

Hence we built our NLU capability on the basis of the Human Approach towards Understanding. We have set up an elaborate framework for information classification that provides universal knowledge representation. We also established an elaborate knowledge base of Concepts, Actions, Relations and Common Sense Information. Our system also has the unique capability to carry out Time and Space Reasoning based on the internal representation. The NLU engine is hence able to use inferencing to carry out Syntactic and Semantic processing of Natural Language Input.

Potential

By achieving Human level of Natural Language Understanding, machines can assist us in a totally new dimension. We feel that this technology can help us build Intelligent Virtual assistants that can think beyond just fetching weather information or ordering pizzas. It can be used in Knowledge based Search Engines which can understand what we want and give us answers instead of showing millions of results. They can help in building Expert Systems, which can assist professionals to do their job more efficiently. It can also paly a vital role in content validation for Social Networking Sites.

Facebook bAbI QA Challenge

Facebook’s Artificial Intelligence Research (FAIR) has hosted the bAbI QA tasks challenge since 2014 to encourage the research community to build machines that have the capability to process Natural Language. The challenge has been segregated into twenty distinct tasks that assess various distinct capabilities displayed by humans. The tests include complex tasks such as co-reference resolution, time and space reasoning, path navigation, size reasoning etc. DataVal is proud to claim that it has been able to build the core NLU engine that successfully completed all the tasks with 100% accuracy using a single common framework. DataVal NLU Core Engine is able to process the task without using large training datasets but by semantically training the vocabulary based on real world knowledge. This unique approach of DataVal provides the capability to build customised processing engines that can be used across different domains.

Our Approach

The DataVal NLU approach includes a comprehensive set of modules that carry out syntactic and semantic processing of data. It has the ability to process facts as a set of disparate events interconnected through time and space. The inbuilt intelligence enables the engine to process queries to provide a meaningful response. The broad capabilities built as part of the core engine is given in the image below:

DataVal NLU Architecture