Probabilistic Language Processing

Using logic to represent and reason we can represent knowledge about the world with

facts and rules, like the following ones:

bird(tweety).

fly(X) :- bird(X).

We can also use a theorem-prover to reason about the world and deduct new facts about

the world, for e.g.,

?- fly(tweety).

Yes

However, this often does not work outside of toy domains - non-tautologous certain

rules are hard to find.

A way to handle knowledge representation in real problems is to extend logic by using

certainty factors.

 In other words, replace

 IF condition THEN fact

with

IF condition with certainty x THEN fact with certainty f(x)

Unfortunately cannot really adapt logical inference to probabilistic inference, since the

latter is not context-free.

Replacing rules with conditional probabilities makes inferencing simpler.

Replace

smoking -> lung cancer

or

lotsofconditions, smoking -> lung cancer

with

P(lung cancer | smoking) = 0.6

Uncertainty is represented explicitly and quantitatively within probability theory, a

formalism that has been developed over centuries.

A probabilistic model describes the world in terms of a set S of possible states - the

sample space. We don’t know the true state of the world, so we (somehow) come up with

a probability distribution over S which gives the probability of any state being the true

one. The world usually described by a set of variables or attributes.

Consider the probabilistic model of a fictitious medical expert system. The ‘world’ is

described by 8 binary valued variables:

Review of Probability Theory

The primitives in probabilistic reasoning are random variables. Just like primitives in

Propositional Logic are propositions. A random variable is not in fact a variable, but a

function from a sample space S to another space, often the real numbers.

For example, let the random variable Sum (representing outcome of two die throws) be

defined thus: Sum(die1, die2) = die1 +die2

Each random variable has an associated probability distribution determined by the

underlying distribution on the sample space

Continuing our example : P(Sum = 2) = 1/36,

P(Sum = 3) = 2/36, . . . , P(Sum = 12) = 1/36

Conditional probability is defined as:

It means for any value x of A and any value y of B