Practical Text Mining

Practical Text Mining Ronen Feldman Information Systems Department HU School of Business Hebrew University of Jerusalem

[email protected]

Background • R Rapid id proliferation lif ti off information available in digital format • People have less time to absorb more information

The Information Landscape Problem Lack of tools to handle unstructured data

Unstructured (Textual)

80%

Structured St t d (Databases)

20%

Find Documents

Display Information

matching the Query

relevant to the Query

Actual information buried inside documents

Extract Information from within the documents

Long lists of documents

Aggregate over entire collection

Text Mining I Input t Documents

O t t Output Patterns Connections Profiles Trends

Seeing the Forest for the Trees

Let Text Mining Do the Legwork for You Text Mining

Find Material Read Understand Consolidate Absorb / Act

Document Types • Structured documents – Output from CGI

• Semi-structured documents – Seminar announcements – Job listings – Ads

• Free format documents – News – Scientific papers

Intelligent g Auto-Tagging gg g Finsbury Park Mosque (c) 2001, Chicago Tribune. Visit the Chicago Tribune on the Internet at http://www chicago tribune com/ http://www.chicago.tribune.com/ Distributed by Knight Ridder/Tribune Information Services. By Stephen J. Hedges and Cam Simpson

England

…….

United States

The Finsbury Park Mosque is the center of radical Muslim activism in England. Through its doors have passed at least three of the men no held on suspicion now s spicion of terrorist acti activity it in France, England and Belgium, as well as one Algerian man in prison in the United States. ``The mosque's chief cleric, Abu Hamza alMasri lost two hands fighting the Soviet Union in Afghanistan and he advocates the elimination of Western influence from Muslim countries. He was arrested in London in 1999 for his alleged involvement in a Yemen bomb plot, but was set free after Yemen failed to produce enough evidence to have him extradited. .'' ……

France England Belgium

Abu Hamza al-Masri Abu Hamza al-Masri chief cleric Finsbury Park Mosque

London Abu Hamza al-Masri London 1999 his alleged involvement in a Yemen bomb plot

Business Tagging Example BusinessNews T i B i N /T i

SAP Acquires Virsa for Compliance Capabilities

SAP

By Renee Boucher Ferguson

Virsa Systems

April 3, 3 2006 Honing its software compliance skills, SAP announced April 3 the acquisition of Virsa Systems, a privately held company that develops risk management software. software Terms of the deal were not disclosed. SAP has been strengthening its ties with Microsoft over the past year or so. The two software giants are working on a joint development project, Mendocino, which will integrate some MySAP ERP (enterprise resource planning) business processes with Microsoft Outlook. The first product is expected in 2007. 2007 "Companies are looking to adopt an integrated view of governance, risk and compliance instead of the current reactive and fragmented approach," said Shai Agassi, president of the Product and Technology Group and executive board member of SAP, in a statement. "We welcome Virsa employees, partners and customers to the SAP family."

risk management software SAP Virsa Systems known SAP Microsoft MySAP ERP Microsoft Outlook Shai Agassi SAP Shai P Sh i A Agassi i /P president of the Product and Technology Group and executive board member SAP

Professional: Name: Shai Agassi Company: SAP Position: President of the Product and Technology Group and executive board member

Acquisition: Acquirer:SAP Acquired: Virsa Systems

Company: SAP Company: Virsa Systems

Person: Shai Agassi

IndustryTerm: risk management software Company: Microsoft Product: Microsoft Outlook

Product: MySAP ERP

Tagging a Document Dynegy Inc Roger Hamilton BC-dynegy-enron-offer-update5 Dynegy May Offer at Least $8 Bln to Acquire Enron (Update5) By George Stein SOURCEc.2001 Bloomberg News BODY ……. ``Dynegy has to act fast,'' said Roger Hamilton a money manager with John Hamilton, Hancock Advisers Inc., which sold its Enron shares in recent weeks. ``If Enron can't get financing and its bonds go to junk, they lose counterparties and their marvelous business vanishes.'' Moody's Investors Service lowered its rating on Enron's bonds to ``Baa2'' and Standard & Poor's cut the debt to ``BBB.'' in the past two weeks. ……

John Hancock Advisers Inc. Roger Hamilton money manager John Hancock Advisers Inc.

Enron Corp

Moody's Investors Service Moody's Investors Service Enron Corp downgraded Baa2 bonds

Intelligence Article

Google’s Google s Article

Merger

Leveraging Content Investment Any type of content

• Unstructured textual content (current focus) • Structured data; audio; video (future) In any format

• Documents; PDFs; E-mails; articles; etc • “Raw” Raw or categorized • Formal; informal; combination From any source

• WWW; file systems; news feeds; etc. • Single source or combined sources

Running Example

Kamada and Kawai’s (KK) Method

A better Visualization

Summary Diagram

Runner’s Runner s World

The most connected models

Distance from the Kayano

Segmentation of the Shoes Market

Structural Equivalence