Content Mining: investigation of content information

Presentation Transcript

Content Mining: examination of content information Dunja Mladenić J.Stefan Institute, Ljubljana, Slovenia and Carnegie Mellon University, USA http://www-ai.ijs.si/DunjaMladenic/http://www.cs.cmu.edu/~dunja/

Web client profiling envision the client perusing the Web, more often than not by clicking hyperlinks objective: give assistance by highlighting the clicked hyperlinks (we expect that the client is tapping on fascinating hyperlinks) prompt a profile for every client independently the profile can be utilized to anticipate tapping on hyperlinks (for our situation), to gather intriguing Web-pages, to look at changed clients and share learning between them (community specialists)

Structure of the individual perusing right hand - Personal WebWatcher URL intermediary (counsel) page User The Web User profile altered page Personal WebWatcher

Personal WebWatcher in real life (1996) Highlight fascinating hyperlinks

Data Pyramid Wisdom Knowledge in addition to experience Knowledge Information in addition to tenets Information Data in addition to setting Data

What is Data Mining? Information mining (learning revelation in databases - KDD, business insight): finding intriguing (non-inconsequential , covered up, beforehand obscure and possibly helpful) regularities in huge datasets "Say something fascinating in regards to the information." "Decribe this information."

Data Mining: Potential use Market examination Risk investigation Fraud discovery Text Mining Web Mining ...

Why content investigation? The measure of content information on electronic media is developing day by day email, business reports, the Web, sorted out databases of documents,... There is a considerable measure of data contained in the content Available strategies and methodologies empowering taking care of fascinating and non-insignificant issues

Problem depiction (I) Text data separating Help with perusing the Web Generation and investigation of client profiles Automatic record classification and watchword task to reports Document bunching Document perception Document origin discovery Document replicating ID Language recognizable proof in content

Document order Document Classifier marked archives ??? report class (mark) unlabeled archive

Yahoo! page for one class

Automatic report order Problem: given is an arrangement of substance classifications loaded with records. The objective is: to naturally embed another record (allot at least one significant classes to another report). Content classifications can be organized (eg., Yahoo, Medline) or unstructured (eg., Reuters) The issue is like allocating watchwords to archives

Document to order: CFP for CoNLL-2000

Some anticipated classes

Our way to deal with record arrangement Data is acquired from the current accumulation of physically sorted reports, where the utilized substance classes are organized Using Text Mining techniques, we built a model that catches manual work of editors The model is utilized to naturally allot content classifications and the relating catchphrases to new, beforehand concealed reports

System engineering Feature development Web vectors of n-grams Subproblem definition Feature determination Classifier development named records (from Yahoo! progression) ?? Archive Classifier unlabeled report record class (name)

Summary of tests and results gaining from order chain of importance: considering just encouraging classes amid the characterization (5%-15% of classifications) developed report representation: new components for successions of two words include subset determination: Odds proportion utilizing 50-100 best elements (0.2%-5%)

More can be found at our venture page www.cs.cmu.edu/~TextLearning/pww/yplanet.html

Document initiation discovery Problem: in light of a database of reports and writers, allocate the most likely writer to another record Solution depends on the way that every writer utilizes a trademark recurrence circulation over words and expressions

Document replicating recognizable proof Problem: foresee likelihood that a given archive was duplicated (somewhat or totally) from some different document(s) from our database Algorithm utilizes complex ordering techniques on (various length) parts of records and looks at them against the given record

Natural dialect ID Text information investigation frameworks usually utilize some common dialect subordinate strategies Need for distinguishing proof of normal dialect the archive is composed in Problem: for a given content recognize the regular dialect it is composed in selecting among the predefined dialects

Algorithm for common dialect ID Basic calculations are straightforward: for every dialect construct a trademark recurrence table of sets and triples of letters that can be basically used to distinguish a report dialect (TextCat openly accessible framework, covers 60 dialects) Problem is with short records - for this situation we can utilize systems for dialect subordinate stop-words recognition (stop-words are visit in all dialects)

Problem portrayal (II) Topic ID and following in time arrangement of archives Document ordering in view of substance and not just catchphrases Content division of content Document synopsis Link examination Information extraction

Topic ID and following in time arrangement of records Problem: given is a period grouping of records (news) - in light of this report arrangement we need to: recognize report that presents new subject from the arrangement of new records recognize archives about existing subjects and associate them into a theme grouping

Text division in view of substance Problem: partition message that has no given structure (content table, sections, and so forth.) into fragments with comparative substance Example applications: theme following in news (talked news) ID of subjects in substantial, unstructured content databases

Algorithm for content division Algorithm: Divide content into sentences Represent every sentence with words and expressions it contains Calculate closeness between the sets of sentences Find a division (succession of delimiters), so that the likeness between the sentences inside a similar section is boosted and minimized between the fragments

Text Summarization Task: Given a content record make a rundown mirroring the record's substance Three fundamental stages: Analyzing the source content Determining its critical focuses Synthesizing a proper yield Most strategies embrace direct weighting model – every content unit (sentence) is surveyed by: Weight(U)=LocationInText(U)+CuePhrase(U)+Statistics(U)+AdditionalPresence(U) … yield comprises from highest content units (sentences)

I nformation extraction Collect an arrangement of Home pages from the Web and manufacture a "delicate" database of individuals (name, address, colleagues, inquire about zones and distributions, biography...) Collect electronic course declarations and concentrate area (room number), begin and end time, name of the speaker

Where are we now? Developing interest and requirement for taking care of extensive accumulations of content The zone is available in Slovenia for more than 5 years with solid global association joint R&D extend with: Microsoft Research, European and American research establishments, participation with Boeing Organization of universal occasions concentrated on Text Mining (ICML-99, KDD-2000, ICDM-2001)

Instead of conclusions... Content Mining empowers taking care of a few issues that are regularly not anticipated that would be tended to by PCs: archive origin location, distinguishing proof of related substance or finding "intriguing" individuals, record division and association, programmed accumulation of officer names for the chose part organizations, discovering specialists in some range, who is included with whom (finding interpersonal organizations), ...

To discover more data check: < http://www-personal.umich.edu/~wfan/text_mining.html > < http://ai.about.com/library/week after week/aa102899.htm > < http://extractor.iit.nrc.ca/catalogs/ml-connected to-ir.html > < http://www.content-analysis.de/> get explore papers at < http://www.researchindex.com > KDD-2000 Text Mining Workshop < http://www.cs.cmu.edu/~dunja/WshKDD2000.html > ECAI-2000 ML for Information Extraction < http://www.dcs.shef.ac.uk/~fabio/ecai-workshop.html > PRICAI-2000 Text and Web MiningWorkshop < http://textmining.krdl.org.sg/cfp.html > IJCAI-2001 Adaptive Text Extraction and Mining Workshop < http://www.smi.ucd.ie/ATEM2001/>, Text Learning: Beyond Supervision < http://www.cs.cmu.edu/~mccallum/textbeyond/> ICDM-2001 Text Mining Workshop < http://www-ai.ijs.si/DunjaMladenic/TextDM01/> ECML/PKDD-2001 Text Mining instructional exercise < http://www-ai.ijs.si/DunjaMladenic/TextDM01/Tutorial.ps>

Link Analysis Mechanisms for identifying which vertices in the diagram (pages on the web) are more vital on the premise of connection structure: Hits calculation (Hubs & Authorities) (Kleinberg 1998) PageRank (Page 1999) weighting (utilized by Google to better rank great pages)

Link examination on Amazon information We downloaded item pages from Amazon.com site: … items are associated with cross-offer connection ("clients who purchased this item additionally purchased taking after items… ") 130.000 books and 32.000 music CDs associated into chart Question: which items (books or CDs) are the most vital? … we utilized Hits calculation to figure the weights Harry Potter & Beatles won the test.

Popular books Harry Potter and the Goblet of Fire (Book 4): J K Rowling, Mary Grandpre The Beatles Anthology: The Beatles, Paul McCartney, Georg