Python for Automation: Web Scraping, REST APIs and Everyday Scripts

Web scraping in Python is the process of sending an HTTP request to a webpage, receiving the HTML response, and extracting specific data programmatically. You use the requests library to fetch the page and BeautifulSoup to parse the HTML, pulling structured data like product prices or job listings from sites that do not offer a public API.

• Key Takeaway 1: requests + BeautifulSoup handles 80% of scraping jobs; use Selenium only when JavaScript rendering is required.
• Key Takeaway 2: Always check robots.txt and a site’s terms of service before scraping. Ignoring them carries legal risk.
• Key Takeaway 3: A REST API in Python is a web interface that sends and receives JSON. You can consume one with requests and build one in under 20 lines with Flask or FastAPI.
• Key Takeaway 4: Python’s built-in smtplib can send automated emails, including Excel reports, with no third-party dependency.
• Key Takeaway 5: Schedule any script with Linux cron or Windows Task Scheduler and it runs without you touching it again.

Automating Repetitive Work with Python

Think about the tasks that eat your Monday mornings: copying data from a government portal into a spreadsheet, emailing a weekly report to your manager, checking whether a competitor changed a price. Every one of those is a candidate for a Python automation script that runs while you’re having chai.

According to McKinsey Global Institute, roughly 60% of all occupations have at least 30% of activities that are technically automatable with current technology. Python is the most practical entry point to that automation because the language is readable, the ecosystem is huge, and the learning curve is gentle compared to alternatives.

The typical automation stack for an office professional looks like this:

Task	Python Tool	Approximate Time Saved/Week
Scraping pricing or tender data	requests + BeautifulSoup	2-3 hours
Pulling data from a third-party REST API	requests + json	1-2 hours
Generating and emailing Excel reports	openpyxl + smtplib	1-2 hours
Scheduling all of the above	cron / Task Scheduler	Ongoing

If you’re exploring where Python automation fits inside a broader data career, the Big Data Analytics notes on 3.0 University give useful context on how scripting connects to pipelines at scale.

Scheduling Scripts So They Run Themselves

Writing the script is only half the job. A script you run manually is just a fancier way of doing the same work. On Linux or macOS, open your crontab with crontab -e and add a line like 0 8 * * 1 python3 /home/user/report.py to run your script every Monday at 8 AM. Windows users get the same result through Task Scheduler, pointing it at your .py file and setting a weekly trigger.

Once your script is scheduled, the three hours you used to spend on data collection become three hours you can spend on analysis or client work.

Web Scraping in Python: How It Works and Legal Rules

What is web scraping in Python at a practical level? You fetch a page with requests.get(url), pass the response text to BeautifulSoup(html, 'html.parser'), and then call methods like .find() or .select() to pull out the data you need. A working Python web scraping script for a static page takes about 15 lines of code.

Some sites load content dynamically with JavaScript, which means requests only gets you a blank shell. That is when Selenium steps in. Selenium controls a real browser, waits for JavaScript to execute, and then lets you read the rendered HTML. It is slower and heavier, so use it only when a static approach fails.

Is Web Scraping in Python Legal? What You Must Check First

The short answer: it depends. Scraping publicly available data for personal research is generally tolerated, but scraping at scale, scraping data behind a login, or violating a site’s terms of service can lead to legal action. The Computer Fraud and Abuse Act in the US and India’s IT Act both have provisions that courts have applied to scraping disputes.

Before your Python web scraping script sends a single request, do three things. First, visit example.com/robots.txt and respect any Disallow directives. Second, read the site’s terms of service for any explicit scraping prohibition. Third, add a time.sleep(2) between requests as a basic rate limit so you do not hammer the server.

According to a 2023 report by Cloudflare, bot traffic accounted for 38% of all internet traffic, and aggressive scrapers are a primary reason sites deploy bot-detection tools. Being a responsible scraper, one who rate-limits, identifies their bot in the user-agent string, and stops when asked, keeps you on the right side of both ethics and the law.

A Practical Python Web Scraping Pattern for Indian Government Portals

Many Indian professionals need to pull tender notices from sites like the Central Public Procurement Portal (CPPP) or check SEBI filings. These are static HTML pages, which makes them ideal for the requests + BeautifulSoup pattern. Fetch the listing page, parse the table rows, write each row to a CSV with Python’s built-in csv module, and email the CSV every morning. That is a genuinely useful Python automation script you can build in an afternoon. Indian finance teams can apply the same approach to GST portal data, saving hours of manual copy-paste work each month.

Consuming and Building REST APIs in Python

An API (Application Programming Interface) in Python is a structured way for your code to talk to another service. A REST API specifically uses HTTP methods, GET to read data, POST to send data, PUT to update it, DELETE to remove it, and returns responses almost always formatted as JSON. When you ask what is a REST API in Python, the practical answer is: it is a URL you call with requests and a JSON object you get back.

Here is a minimal example consuming a currency exchange API:

import requests
response = requests.get("https://api.exchangerate-api.com/v4/latest/INR")
data = response.json()
print(data['rates']['USD'])

That four-line snippet pulls live INR-to-USD rates. Drop it into a scheduled script and you have a daily currency tracker that emails you the rate every morning, no browser needed.

How to Create an API in Python with Flask or FastAPI

Building your own API is simpler than most people expect. Flask and FastAPI are the two dominant choices. FastAPI is newer, generates automatic documentation, and is faster at runtime. Flask has a larger base of tutorials and is slightly easier for absolute beginners. Both let you create a working endpoint in under 20 lines.

Here is a five-line Flask endpoint that returns JSON:

from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/status')
def status(): return jsonify({"status": "ok", "version": "1.0"})
app.run(debug=True)

Run that file, visit http://127.0.0.1:5000/status in your browser, and you have a working REST API. From here you can add routes that query a database, process a form submission, or return scraped data on demand.

According to the Stack Overflow Developer Survey 2024, Flask ranks among the top five most-used web frameworks globally, and FastAPI has seen the fastest year-on-year growth of any Python web framework. If you are thinking about shifting your career from data science into AI and ML, knowing how to build and consume APIs is a non-negotiable skill in that transition.

Sending Automated Emails and Excel Reports with Python

Python’s standard library includes smtplib, which connects to any SMTP server and sends emails. Pair it with the email module to attach files, and you can send a formatted Excel report generated by openpyxl to your whole team without opening Outlook once.

The core pattern: create an EmailMessage object, set the From, To, and Subject fields, attach your file with msg.add_attachment(), then open an smtplib.SMTP_SSL connection to your mail server, login, and call send_message(). Gmail users need to generate an App Password from their Google Account settings since Google disabled plain-password access in 2022. For corporate environments using Microsoft 365, the SMTP host is smtp.office365.com on port 587.

If you want to know how Python automation fits into a longer career strategy, the guide on how to future-proof your career in the age of AI lays out which technical skills are holding their value right now.

Using Python for Excel Without Opening Excel

The openpyxl library reads and writes .xlsx files directly. You can load a workbook, update cells, apply formatting, add charts, and save it, all from a terminal on a server that does not even have Microsoft Office installed. That matters for teams running Python automation scripts on Linux servers or cloud VMs.

A realistic use case for Indian finance teams: pull monthly GST filing data from a portal using requests, write it into a pre-formatted Excel template with openpyxl, and email the finished file to the accounts team every 5th of the month via a cron job. Zero manual steps once it is set up.

The bootcamp training programs at 3.0 University cover exactly this kind of applied scripting, with hands-on labs that go from syntax basics to production-ready Python automation scripts.

Frequently Asked Questions

What is web scraping in Python?

Web scraping in Python is the process of programmatically fetching a web page with the requests library and extracting structured data from its HTML using BeautifulSoup or a similar parser. It is used to collect pricing data, job listings, news headlines, or any publicly visible information that is not available through a formal API.

What is an API in Python?

An API (Application Programming Interface) in Python is a set of rules that lets your code communicate with an external service or application. In practice, most APIs your Python scripts will call are REST APIs: you send an HTTP request to a URL, and the server returns a JSON response your code can read and process immediately.

How do I create an API in Python?

You can create a REST API in Python using Flask or FastAPI. Install Flask with pip install flask, define a function decorated with @app.route('/your-endpoint'), return a jsonify() response, and run the app. The whole thing takes under 20 lines of code. FastAPI adds automatic Swagger documentation and is preferred for production use.

How do I send an email using Python?

Use Python’s built-in smtplib module. Create an EmailMessage, set From, To, and Subject, optionally attach a file, then connect to your SMTP server with smtplib.SMTP_SSL(), call login() with your credentials, and call send_message(). Gmail requires an App Password generated from your Google Account settings since 2022.

Is web scraping legal?

Web scraping is not automatically illegal, but it can become so. Always check the site’s robots.txt file and terms of service before scraping. Scraping data behind a login, ignoring rate limits, or violating a site’s terms can expose you to legal action under India’s IT Act or equivalent laws in other jurisdictions. Scrape publicly available data, slowly, and ethically.

Can Python scrape JavaScript-rendered websites?

Yes. When requests returns only a blank shell because a site loads content via JavaScript, use Selenium or Playwright. These tools control a real browser, wait for JavaScript to execute, and then expose the fully rendered HTML for parsing with BeautifulSoup or similar libraries.

Python automation is one of those skills that pays back the learning time within weeks. You pick up the requests and BeautifulSoup pattern for web scraping in Python, build one scraper, schedule it, and suddenly a chunk of your week is free. Add a Flask endpoint or a smtplib email script and the scope of what you can automate keeps growing.

The next concrete steps: install Python 3.11+, run pip install requests beautifulsoup4 flask openpyxl, and pick one repetitive task from your actual job to automate this week. Even a script that saves you 30 minutes on Friday is proof of concept. From there, the jump to APIs and full automation pipelines is shorter than it looks.

Whether you are a student building your first portfolio, a working professional cutting down on manual reporting, or a career switcher looking for a practical edge, 3.0 University’s online certification courses in Cybersecurity, Ethical Hacking, Artificial Intelligence, Blockchain and Web3 give you structured, hands-on paths to industry-ready skills. Every course is built around real-world projects, not just theory. Join the REACH learner community to connect with peers who are building the same skills and ask questions as you go.

Last updated: August 2026. Reviewed by the 3University editorial team.