Probabilistic Language Processing

Using logic to represent and reason we can represent knowledge about the world with

facts and rules, like the following ones:

bird(tweety).

fly(X) :- bird(X).

We can also use a theorem-prover to reason about the world and deduct new facts about

the world, for e.g.,

?- fly(tweety).

Yes

However, this often does not work outside of toy domains - non-tautologous certain

rules are hard to find.

A way to handle knowledge representation in real problems is to extend logic by using

certainty factors.

 In other words, replace

 IF condition THEN fact

with

IF condition with certainty x THEN fact with certainty f(x)

Unfortunately cannot really adapt logical inference to probabilistic inference, since the

latter is not context-free.

Replacing rules with conditional probabilities makes inferencing simpler.

Replace

smoking -> lung cancer

or

lotsofconditions, smoking -> lung cancer

with

P(lung cancer | smoking) = 0.6

Uncertainty is represented explicitly and quantitatively within probability theory, a

formalism that has been developed over centuries.

A probabilistic model describes the world in terms of a set S of possible states - the

sample space. We don’t know the true state of the world, so we (somehow) come up with

a probability distribution over S which gives the probability of any state being the true

one. The world usually described by a set of variables or attributes.

Consider the probabilistic model of a fictitious medical expert system. The ‘world’ is

described by 8 binary valued variables:

Review of Probability Theory

The primitives in probabilistic reasoning are random variables. Just like primitives in

Propositional Logic are propositions. A random variable is not in fact a variable, but a

function from a sample space S to another space, often the real numbers.

For example, let the random variable Sum (representing outcome of two die throws) be

defined thus: Sum(die1, die2) = die1 +die2

Each random variable has an associated probability distribution determined by the

underlying distribution on the sample space

Continuing our example : P(Sum = 2) = 1/36,

P(Sum = 3) = 2/36, . . . , P(Sum = 12) = 1/36

Conditional probability is defined as:

It means for any value x of A and any value y of B

Grammar Induction

All objects described are fixed or unique

"John is a student" student(john)

Here John refers to one unique person.

Objects described can be unique or variables to stand for a unique object

"All students are poor"

ForAll(S) [student(S) -> poor(S)]

Here S can be replaced by many different unique students.

This makes programs much more compact:

eg. ForAll(A,B)[brother(A,B) -> brother (B,A)] replaces half the possible statements about brothers

• Temporal => Represents truth over time.

• Modal => Represents doubt

• Higher order logics => Allows variable to represent many relations between objects

• Non-monotonic => Represents defaults

Given clauses C1 and C2, a clause C is a RESOLVENT of C1 and C2, if

1. There is a subset C1' = {A1, .., Am} of C1 of literals of the same sign, say

positive, and a subset C2' = {B1, .., Bn} of C2 of literals of the opposite sign, say

negative,

2. There are substitutions s1 and s2 that replace variables in C1' and C2' so as to

have new variables,

3. C2'' is obtained from C2 removing the negative signs from B1 .. Bn

4. There is an Most General Unifier s for the union of C1'.s1 and C2''.s2

and C is

((C1 - C1').s1 UNION (C2 - C2').s2).s

In symbols this Resolution inference rule becomes:

{C1, C2}

Let L be a list containing the initial state (L= the fringe)

Loop

if L is empty return failure

Node   select (L)

if Node is a goal

then return Node

(the path from initial state to Node)

else generate all successors of Node, and

merge the newly generated states into L

End Loop

Semantic Analysis

• Assigning meanings to the structures created by syntactic analysis.

• Mapping words and structures to particular domain objects in way consistent with

our knowledge of the world.

• Semantic can play an import role in selecting among competing syntactic analyses

and discarding illogical analyses.

– I robbed the bank -- bank is a river bank or a financial institution

• We have to decide the formalisms which will be used in the meaning

representation.

Ambiguity and Disambiguation

• Which knowledge representation will be used depends on the application --

Machine Translation, Database Query System.

• Requires the choice of representational framework, as well as the specific

meaning vocabulary (what are concepts and relationship between these concepts

-- ontology)

• Must be computationally effective.

• Common representational formalisms:

– first order predicate logic

– conceptual dependency graphs

– semantic networks

– Frame-based representations

Discourse Understanding

• Discourses are collection of coherent sentences (not arbitrary set of sentences)

• Discourses have also hierarchical structures (similar to sentences)

• anaphora resolution -- to resolve referring expression

– Mary bought a book for Kelly. She didn’t like it.

• She refers to Mary or Kelly. -- possibly Kelly

• It refers to what -- book.

– Mary had to lie for Kelly. She didn’t like it.

• Discourse structure may depend on application.

– Monologue

– Dialogue

– Human-Computer Interaction

Augmented Grammars

Parts-of-Speech (POS) Tagging

• Each word has a part-of-speech tag to describe its category.

• Part-of-speech tag of a word is one of major word groups (or its subgroups).

– open classes -- noun, verb, adjective, adverb

– closed classes -- prepositions, determiners, conjuctions, pronouns, particples

• POS Taggers try to find POS tags for the words.

•duck is a verb or noun? (morphological analyzer cannot make decision).

• A POS tagger may make that decision by looking the surrounding words.

– Duck! (verb)

– Duck is delicious for dinner. (noun)

Semantic Interpretation

Semantics: The meanings of the atomic symbols of the logic, and the rules for

determining the meanings of non-atomic expressions of the logic. It specifies what facts

in the world a sentence refers to. Hence, also specifies how you assign a truth value to a

sentence based on its meaning in the world. A fact is a claim about the world, and may be

true or false.

There are a number of logical systems with different syntax and semantics. We list below

a few of them.

Syntactic Processing

• Parsing -- converting a flat input sentence into a hierarchical structure that

corresponds to the units of meaning in the sentence.

• There are different parsing formalisms and algorithms.

• Most formalisms have two main components:

– grammar -- a declarative representation describing the syntactic structure

of sentences in the language.

– parser -- an algorithm that analyzes the input and outputs its structural

representation (its parse) consistent with the grammar specification.

• CFGs are in the center of many of the parsing mechanisms. But they are

complemented by some additional features that make the formalism more suitable

to handle natural languages.

Formal grammar for a fragment of English

Ambiguities are resolved using the following methods.

• models and algorithms are introduced to resolve ambiguities at different levels.

• part-of-speech tagging -- Deciding whether duck is verb or noun.

• word-sense disambiguation -- Deciding whether make is create or cook.

• lexical disambiguation -- Resolution of part-of-speech and word-sense ambiguities are two important kinds of lexical disambiguation.

• syntactic ambiguity -- her duck is an example of syntactic ambiguity, and can be addressed by probabilistic parsing.

Syntactic analysis

Syntax: The atomic symbols of the logical language, and the rules for constructing wellformed,

non-atomic expressions (symbol structures) of the logic. Syntax specifies the

symbols in the language and how they can be combined to form sentences. Hence facts

about the world are represented as sentences in logic.

Represent knowledge as formal logic:

All dogs have tails : dog(x) hasatail(x) Advantages:

• A set of strict rules.

o Can be used to derive more facts.

o Truths of new statements can be verified.

o Guaranteed correctness.

• Many inference procedures available to in implement standard rules of logic.

• Popular in AI systems. e.g Automated theorem proving.

Basic idea:

• Knowledge encoded in some procedures

o small programs that know how to do specific things, how to proceed.

o e.g a parser in a natural language understander has the knowledge that a noun phrase may contain articles, adjectives and nouns. It is represented by calls to routines that know how to process articles, adjectives and nouns.

Advantages:

• Heuristic or domain specific knowledge can be represented.

• Extended logical inferences, such as default reasoning facilitated.

• Side effects of actions may be modelled. Some rules may become false in time.

Keeping track of this in large systems may be tricky.

Disadvantages:

• Completeness -- not all cases may be represented.

• Consistency -- not all deductions may be correct.

e.g If we know that Fred is a bird we might deduce that Fred can fly. Later we might discover that Fred is an emu.

• Modularity is sacrificed. Changes in knowledge base might have far-reaching effects.

• Cumbersome control information.

The following properties should be possessed by a knowledge representation system.

Representational Adequacy

-- the ability to represent the required knowledge;

Inferential Adequacy

- the ability to manipulate the knowledge represented to produce new knowledge

corresponding to that inferred from the original;

• Representations of actions: programs that develop successor state descriptions which

represent actions.

• Representation of state: every state description is complete. This is because a

complete description of the initial state is given, and actions are represented by a program

that creates complete state descriptions.

• Representation of goals: a problem solving agent has only information about it's

goal, which is in terms of a goal test and the heuristic function.

• Representation of plans: in problem solving, the solution is a sequence of actions.

In a simple problem: "Get a quart of milk and a bunch of bananas and a variable speed

cordless drill" for a problem solving exercise we need to specify:

Initial State: the agent is at home without any objects that he is wanting.

Operator Set: everything the agent can do.

UNIT V - artificial Intelligence, Communication,

UNIT V

APPLICATIONS

Communication

Natural Language Processing (NLP) is the process of computer analysis of input provided

in a human language (natural language), and conversion of this input into a useful form of

representation.

The field of NLP is primarily concerned with getting computers to perform useful and

interesting tasks with human languages. The field of NLP is secondarily concerned with

helping us come to a better understanding of human language.

• The input/output of a NLP system can be:

– written text                                                               

– speech

The following language related information are useful in NLP:

• Phonology – concerns how words are related to the sounds that realize them.

• Morphology – concerns how words are constructed from more basic meaning units called morphemes. A morpheme is the primitive unit of meaning in a language.

• Syntax – concerns how can be put together to form correct sentences and determines what structural role each word plays in the sentence and what phrases are subparts of other phrases.

• Semantics – concerns what words mean and how these meaning combine in sentences to form sentence meaning. The study of context-independent meaning.

UNIT V

APPLICATIONS

Communication

Natural Language Processing (NLP) is the process of computer analysis of input provided

in a human language (natural language), and conversion of this input into a useful form of

representation.

The field of NLP is primarily concerned with getting computers to perform useful and

interesting tasks with human languages. The field of NLP is secondarily concerned with

helping us come to a better understanding of human language.

• The input/output of a NLP system can be:

– written text                                                               

– speech

The following language related information are useful in NLP:

• Phonology – concerns how words are related to the sounds that realize them.

• Morphology – concerns how words are constructed from more basic meaning units called morphemes. A morpheme is the primitive unit of meaning in a language.

• Syntax – concerns how can be put together to form correct sentences and determines what structural role each word plays in the sentence and what phrases are subparts of other phrases.

• Semantics – concerns what words mean and how these meaning combine in sentences to form sentence meaning. The study of context-independent meaning.

• Pragmatics – concerns how sentences are used in different situations and how use affects the interpretation of the sentence.

• Discourse – concerns how the immediately preceding sentences affect the interpretation of the next sentence. For example, interpreting pronouns and interpreting the temporal aspects of the information.

• World Knowledge – includes general knowledge about the world. What each language user must know about the other’s beliefs and goals.

Communication as action

• We will mostly concerned with written text (not speech).

• To process written text, we need:

– lexical, syntactic, semantic knowledge about the language

– discourse information, real world knowledge

• To process spoken language, we need everything required to process written text, plus the challenges of speech recognition and speech synthesis.

There are two components of NLP.

• Natural Language Understanding

– Mapping the given input in the natural language into a useful representation.

– Different level of analysis required

morphological analysis,

syntactic analysis,

semantic analysis,

discourse analysis.

• Natural Language Generation

– Producing output in the natural language from some internal representation.

– Different level of synthesis required,

deep planning (what to say),

syntactic generation

• NL Understanding is much harder than NL Generation. But, still both of them are hard.

The difficulty in NL understanding arises from the following facts:

• Natural language is extremely rich in form and structure, and very ambiguous.

– How to represent meaning,

– Which structures map to which meaning structures.

• One input can mean many different things. Ambiguity can be at different levels.

– Lexical (word level) ambiguity -- different meanings of words

– Syntactic ambiguity -- different ways to parse the sentence

– Interpreting partial information -- how to interpret pronouns

– Contextual information -- context of the sentence may affect the meaning of that sentence.

• Many input can mean the same thing.

• Interaction among components of the input is not clear.