Web Development Basics
Web Scraping
Extracting data from websites ethically using requests and BeautifulSoup or Scrapy.
Interview: Data retrieval. Frequently tested on HTML DOM parsing, rate-limiting, proxy usage, and ethical scraping constraints.
Web Scraping is the technique of programmatically extracting information from websites. In Python, this is commonly done by downloading HTML using requests and parsing the DOM tree using BeautifulSoup.
Ethical Web Scraping
Scraping can consume significant server bandwidth. Always adhere to these scraping guidelines:
- Check robots.txt: Append
/robots.txtto the target domain to see which paths are disallowed for crawlers. - Rate Limiting: Insert delay intervals (e.g.
time.sleep(1)) between requests to avoid overloading target servers. - Identify User-Agent: Send a custom
User-Agentheader identifying your script and contact info.
BeautifulSoup DOM Traversal
BeautifulSoup parses raw HTML string and creates a parse tree. Elements can be searched using tag names, attributes (like class or id), or CSS selectors.
Use Cases
Data Aggregation — Fetching real estate pricing, jobs listing, or news headlines across multiple portals.
Competitive Intelligence — Monitoring competitor prices or reviews.
Academic Research — Gathering dataset corpuses from public directories.
Common Mistakes
Ignoring robots.txt — Crawling restricted directories, which can result in your IP getting blacklisted.
Hardcoding selectors — Relying on brittle nested selectors (like `div > table > tr > td[3]`) that break instantly on tiny UI layout changes.
Not handling timeouts — Making requests without timeout limits, causing crawler scripts to freeze indefinitely.