This is the podcast for the eleventh class, in which we see traditional and non-traditional methods of collecting data off the web. Traditional way is demonstrated using web crawling using wget, and non-traditional way is instantiated with YouTube harvesting.

INLS490-154W: Information Retrieval Systems Design and Implementation

Web crawling

Tools for organizing and accessing information have become indispensable. It is critical, therefore, to understand their design and operational foundations. In this course students will have an opportunity to learn about search engines, web crawling, and some Web 2.0 technologies based on hands-on experience and with a focus on techniques that can be used to access, retrieve, organize, and present information. Students will work with practical developmental tools and learn relevant concepts through experimentation. For instance, students will employ an open source search engine and learn about indexing, retrieving, and ranking techniques.

About

INLS490-154W: Information Retrieval Systems Design and Implementation