Introduction
=============

AsyncURLCrawler navigates through web pages concurrently by following hyperlinks to collect URLs. AsyncURLCrawler uses BFS algorithm. To make use of it check robots.txt of the domains first.

`Github Page <https://github.com/PouyaEsmaeili/AsyncURLCrawler>`_

`PyPi Page <https://pypi.org/project/AsyncURLCrawler/>`_