Skip to main content

Data, data, data ...

Data acquisition phase has started

As holds true for every project, I assume, the data that will be worked with is of vital importance. Our aim in the impresso project is to digitise and process various newspapers from the late 18th to the 20th century (200 years in total) so that historians can explore the data in previously unseen and unknown ways. As such, we will provide new tools to do research in history, and who knows, maybe we will also take a leading role in revolutionising how historical research will be done in a few years ;-).

But again, all this will not be able without any data. Already during the project application phase we contacted libraries in Switzerland and Luxembourg and asked them whether they have data available for our project. Many institutions gladly followed our call and committed themselves to assist us as good as they can. We are very happy that, for example, both the National Libraries of Luxembourg, as well as its Swiss counterpart are on board! Together with some smaller archives they have compiled a list of ca. 750 newspapers and periodicals which we need to consider.

If we were to include all the publications for our final tool, we would have to deal with roughly 56 billion words, and that is only for newspapers written in German. Although it would be tempting to just include all the newspapers we have in our source database, the question also is how valuable a resource is for a historian. This is why in a next phase, we need to decide on a set of publications we want to work with. In this undertaking the historians will have the final say. Their opinion is therefore very important to us.

Before the historians can decide on a selection of the most valuable newspapers, however, we need some statistics. After all, including only minor publications with a limited publication period is of now use to the historians. One of our visualising pros, Thijs van Beek, has taken the data the archives provided and built a small visualisation tool. With it you can examine all available sources, their sizes, languages, and origins. The data is not complete in all cases though, but the visualisation sums it up nicely, I think.

You find the interactive version here: http://impressostats.midasweb.lu/

Choosing from more than 750 publications will be hard. Since multiple institutions and partners collaborate in this project, everyone will have different needs and requirements. For these reasons, the data selection phase will take place at the first workshop next week, which will take place a the EPFL in Lausanne. All the collaborators will be present there and debate over a core set and an extended set. I'm looking forward to interesting discussions with computational linguists, archivists, visualisers, historians, and digital humanists. 

After the workshop, we will finally know what data we will be working with for the next three years. So, next week is when the work really begins ;-).

See you next time!



Labels:

Comments

Post a Comment

Popular posts from this blog

Extracting Text from PDFs

TET bindings for Python If you are about to extract text from digitised archives, you hope to receive clean XML files, because text extraction from XML files is an almost trivial undertaking. If you receive PDFs, on the other hand, things become a little more difficult. At least, OCR has been done already, so text recognition from pure images is done. For unknown reasons though, the output from the OCR has not been saved in an XML format, where you have the text in one file and the corresponding image (usually a tiff) in another file, but in a PDF where the text is saved in a stream alongside the image. Fortunately, there are libraries which allow you to extract this text stream from PDFs in an efficient manner. One such tool is PDFlib . Within the product range of PDFlib, you will find the Text and Image Extractor Toolkit, or TET for short. We have used the Python bindings to extract the text from the PDFs. A PDF of the first issue of the NZZ. But how to get the text out of
1 comment
Read more

From LSI to PLSI

Probabilistic Latent Semantic Indexing  (Hofmann 1999) (Note: Hofmann has a very concise writing style. This summary thus contains passages which are more or less directly copied from Hofmann's paper, simply because you could not summarise them any further. This is just to point out in all clarity that everything in this summary represents Hofmann's work (without making direct "quotations" explicit)). Problem Statement Hofmann proposes probabilistic latent semantic indexing, which builds on the method by  Deerwester et al. (1990) . The main difference is that Hofmann's method has a solid statistical foundation and that it performs significantly better on term matching tasks. The problem statement is similar: in human machine-interaction, the challenge is to retrieve relevant documents and present those to the user after he or she has formulated a request. These requests are often stated as a natural language query, within which the user enters some key
1 comment
Read more

Indexing by Latent Semantic Analysis (Deerwester et al., 1990)

Problem statement Deerwester et al. address a common issue in information retrieval, which is the often unsatisfying recall due to the differences how documents are indexed and with what terms users would like to find them.  As such, synonymous query terms fail to retrieve a smiliar document and thus have a serious impact on recall. What is more, polysemy might return documents which are of no interest to the user, which severes precision. The authors point out three factors which are responsible for recent systems to fail in such tasks: Incomplete identification of index words: documents usually never contain all terms a user might query the data with. There is no way of dealing with polysemeous words. Independency assumption of word types   Their assumption is that there is an underlying latent semantic structure (in which terms might either refer to the document they appear in, or to the overall topic, or b
Post a Comment
Read more