Mining Large Samples of Web-Based Corpora

This paper presents a method to automatically mirror, process, and compare large samples of text corpora from Web-based information systems. The wealth of textual information contained in publicly available Web sites is converted into aggregated representations through textual analysis. The application of word lists, keyword analysis, term clustering, and correspondence analyses to identify and represent semantic relationships, including their longitudinal patterns, is illustrated through a case study that investigates the global coverage of solar power technologies in international media. The resulting graphs, indicators and tables describe complex relationships and developments that are hard to capture in traditional ways. As such they facilitate investigations about the nature and dynamics of Web content.