Menu
Let’s first understand What is NLP (Natural Language Processing)? NLP is a component of AI(Artificial Intelligence). It is the ability of a computer program to understand human language as it is spoken. In layman terms, NLP is a way for computers to analyze human language & derive useful meaning from it.
What is natural language processing?
Natural language processing (NLP) is the branch of artificial intelligence (AI) that deals with communication: How can a computer be programmed to understand, process, and generate language just like a person?
While the term originally referred to a system’s ability to read, it’s since become a colloquialism for all computational linguistics. Subcategories include natural language generation (NLG) — a computer’s ability to create communication of its own — and natural language understanding (NLU) — the ability to understand slang, mispronunciations, misspellings, and other variants in language.
How natural language processing works
Natural language processing works through machine learning (ML). Machine learning systems store words and the ways they come together just like any other form of data. Phrases, sentences, and sometimes entire books are fed into ML engines where they’re processed based on grammatical rules, people’s real-life linguistic habits, or both. The computer then uses this data to find patterns and extrapolate what comes next. Take translation software, for example: In French, “I’m going to the park” is “Je vais au parc,” so machine learning predicts that “I’m going to the store” will also begin with “Je vais au.” All the computer needs after that is the word for “store.”
Common uses of natural language processing
Machine translation is one of the better NLP applications, but it’s not the most commonly used. Search is. Every time you look something up in Google or Bing, you're feeding data into the system. When you click on a search result, the system sees this as confirmation that the results it’s found are right and uses this information to better search in the future.
Chatbots work the same way: They integrate with Slack, Microsoft Messenger, and other chat programs where they read the language you use, then turn on when you type in a trigger phrase. Voice assistants like Siri and Alexa also kick into gear when they hear phrases like “Hey, Alexa.” That’s why critics say these programs are always listening: If they weren’t, they’d never know when you need them. Unless you turn an app on manually, natural language processing programs must operate in the background, waiting for that phrase.
Even if they are always there, NLP isn’t Big Brother. Natural language processing does more good for the world than bad. Just imagine your life without Google search. Or spellcheck, which uses NLP to compare the words you type to ones in the dictionary. Comparing the two data sets allows spellcheckers to identify what’s wrong and to offer suggestions.
Business benefits of natural language processing
Search and spellcheck are so commonplace, we often take them for granted, especially at work where NLP offers radical productivity gains. Want to know how many vacation days you have left? Don’t call HR. Save time and ask Talla, a chatbot that searches company policies for an answer. On the phone and need last quarter’s numbers? Mention them during your conversation and audio search startup SecondMind will show the answer on your screen. The company boasts its integrated search tool makes accounting and customer resource calls up to ten times shorter.
Natural language processing also helps job recruiters sort through resumes, attract diverse candidates, and hire more qualified workers. Spam detection uses NLP to keep unwanted email out of your inbox; programs like Outlook and Gmail use it to sort messages from certain people into folders you create.
Tools like sentiment analysis help companies quickly discern whether Tweets about them are good or bad so they can triage customer concerns. Sentiment analysis doesn’t just process words on social media, it breaks down the context in which they appear. Only 30 percent of English words are positive, says Skye Morét, data visualizer at analysis firm Periscopic — the rest are neutral or negative. So NLP helps businesses more fully understand a post: What’s the consumer emotion behind those neutral words?
Traditionally, corporations used natural language processing to classify feedback as positive or negative. But Ryan Smith, senior vice president of social and innovation at FleishmanHillard, says today’s tools identify more precise emotions, like sadness, anger, and fear.
Natural language processing for social good
In addition to helping companies process data, sentiment analysis also helps us understand society. Periscopic, for example, has paired NLP with visual recognition to create the Trump-Emoticoaster, a data engine that processes language and facial expressions in order to monitor President Donald Trump’s emotional state.
Similar tech could also prevent school shootings: At Columbia University, researchers have processed 2 million Tweets posted by 9,000 at-risk youth, looking for the answer to one question: How does language change as a teen comes closer and closer to getting violent?
“Problematic content can evolve over time,” says program director Dr. Desmond Patton. As at-risk youth grow closer to the brink, they reach out for help, using language. Natural language processing then flags problematic emotional states so that social workers can intervene.
Like Periscopic, Columbia pairs sentiment analysis with image recognition to improve accuracy. Patton says computer vision breaks down pictures attached to the Tweets, then machine learning processes them together with the language to tell “the actual emotionality of an image. Is this image about grief? Is this image about threats?...What else is happening in an image that helps us understand more complexly?” In addition to school shootings, the Columbia program hopes to also prevent gang violence.
Natural language processing for personal improvement
Natural language processing can also help you monitor your own emotional state. Woebot is an electronic therapist that connects with users via a Facebook Messenger chatbot or through a stand-alone app. There’s no high-level sentiment analysis here yet, though. Woebot essentially tracks only depression and anxiety, looking for words that may indicate users face an emergency situation.
The future of natural language processing
Woebot uses NLP to search for keyphrases, but the communication is so clunky, no one would ever confuse the app for a human being. The longer NLP is on the market, though, the better it gets, with some programs communicating so sophisticatedly we need tools like Botometer and BotOrNot to tell us if we’re talking to a real person.
As bot-driven accounts pop up on Twitter and Facebook, the next wave of tech may very well be NLP that detects NLP. Both Botometer and BotOrNot work by analyzing language for computer communication characteristics. Fortunately, we still live in an age where this can be accurately predicted. While advanced, today’s natural language processing is nowhere near perfect: Despite the fact that Woebot fully relies on it to function, CEO Alison Darcy said natural language understanding is the app’s biggest technical problem. “We're still at the very beginning in terms of this tech,” she told Inside AI’s Rob May.
Related artificial intelligence articles:
Next read this:
Powerful Insights
Take your understanding of unstructured data to a whole new level with a full suite of advanced text analytics features to extract entities, relationships, keywords, semantic roles and more.
Broad Language Coverage
Interpret text in thirteen different languages, with more on its way.
Domain Customization
Apply the knowledge of unique entities and relations in your industry or organization to your data.
Natural Language Understanding service overview
Start Building with Natural Language Understanding
Get started with Watson
Access Watson services on the IBM Cloud.
Developer tools
Everything you need to start building with Watson.
Lite
- – Analyze up to 30,000 NLU items per month
- – Use with any of the features
- – One free custom model
1 NLU item = 1 group of 10,000 characters x 1 feature
An NLU item is based on the number of data units enriched and the number of enrichment features applied. A data unit is 10,000 characters or less. For example: extracting Entities and Sentiment from 15,000 characters of text is (2 Data Units * 2 Enrichment Features) = 4 NLU Items.
Standard
- – 1-250,000 NLU items per month -$0.003 per NLU item
- – 250,001-5,000,000 NLU items per month -$0.001 per NLU item
- – 5,000,000 + NLU items per month -$0.0002 per NLU item
- – Custom model price per month -$800 per model
Premium
Contact Sales
Contact Sales
Watson Premium plans offer a higher level of security and isolation to help customers with sensitive data requirements.
Blogs
Videos
Code Patterns
Be part of the Community
Case study
Drafting high-quality litigation work in minutes
LegalMation developes a first-of-its-kind AI platform to automate routine litigation tasks, using IBM Watson. LegalMation uses Watson Discovery offerings to draft early phase response documents, which helped legal teams save time, drive down costs and shift strategic focus. See how Legalmation assembles a team of subject matter experts (SMEs) to use IBM Watson Knowledge Studio and IBM Watson Natural Language Understanding to create a domain-specific model focused on legal terminology and concepts.
Case study
Reinventing influencer marketing on social media
Influential leverages AI and IBM Watson to enable influencers to amplify their marketing messages through social media. Augmented intelligence through IBM Watson allows influencers to target campaigns towards strategic demographics. See how Influential uses IBM Watson Natural Language Understanding, IBM Watson Personality Insights and IBM Watson Tone Analyzer application programming interfaces (APIs) on the IBM Cloud Platform to improve social campaign performance.
Case study
Detecting citizen concerns within a community with high accuracy
Max Kelsen uses IBM Watson to build an insight engine for powering an AI platform that could provide insights into customer experience. The company collaborates with the local government to understand and query large amounts of private and public data. See how Max Kelsen uses IBM Watson Discovery, IBM Watson Natural Language Understanding, and IBM Watson Knowledge Studio to deliver insights on citizens interests.