Thank you very much! Yes, I've used Python framework - Scrapy for obtaining the data for this project. It can be used for both small projects and also for building complex spiders. There are a lot of tools for web-crawling, but the ones that I find effective besides Scrapy, are Beautiful Soup, which is easy python library for pulling data from HTML/XML pages, and Selenium WebDriver, which provides Python API to use its powerful functionality. It was originally written for testing web pages, but is widely used for data crawling, especially in those cases where there are complex captchas and other barriers of accessing data.
