Instructor Notes
This is a placeholder file. Please add content here.
What is Web Scraping?
Instructor Note
Sign in
Setup
Introduce self
Code of conduct
-
Intro questions
- Do you have a specific web scraping application in mind?
- Is there anything in particular which you would like to learn in this session?
Slide - Definition of web scraping
-
Slide - What is web scraping?
- Extracting information from websites
- Manual - faster to automate
- Collect data in usable format, e.g. .cvsv
- Similar to web indexing - more targeted
Slides -
-
Example - Need to understand structure of a webpage
- Look at a couple of examples to give a flavour of what we might need to look at
Instructor Note
- Point out:
- Looks quite well ordered
- Refine search
- Export options
- Human can easily work out what data represents
- Computer needs more information
- Slide - HTML
- Can see reasonably well structured
- Looks like table of data on webpage - quite different in the code
- Could get computer to pick out specific information
Instructor Note
- Canadian MPs - well structured
- Slide - What if data isn’t organised in such an obvious way?
- Unstructured vs structured
- British MPs
- Similar data - no way to download
- Slide -> UK MPs HTML
- Data structured for viewing - table of cards
- Less easy to see how data would be gathered
Instructor Note
- Show UK MP tools
- Make sure really necessary to web scrape
- Slide - web scraping might not be necessary
Anatomy of a web page
Instructor Note
-
Slide - where HTML fits in
- even if using semi-automated tools still need to work out what to extract
HTML - uses tags to organise and format content.
-
Slide - ask what people think it will do
- Show file in editor
- open file from folder - opens in browser
- show address in browser is local folder location
Slides showing tags etc
structured doc, elements marked by tags
attributes - modify behaviour, appearance or functionality
list of tags in notes
Instructor Note
Leave out if short of time
CSS gives separation between display format and content
uses rules applied to HTML elements by selectors
can be useful for targeting elements when scraping
Slide - CSS added to head tag
Instructor Note
- Web pages tend to be generated by other programs
- Adds to the complexity
- Source of this page initially seems complex
- hunt for the tags
- Ctrl+f to search
- Slide - overall structure of page
Instructor Note
- Back to Canadian MPs page
- May need to expand elements
- Member data stored in table table, td and tr tags
- Hover over element in console to highlight on page
Instructor Note
- View Page Source shows HTML from when the page is loaded
- Selecting 2015 triggers the script
- Inspect then displays the rendered HTML
Manually scrape data using browser extensionsUsing the Chrome Web Scraper extension
Instructor Note
Web Scraper works with a Sitemap
Should see sitemap generated by previous exercise
May need to spend time inspecting code to decide on best selectors to use
-
Create new sitemap
- Can just select pages 1 and 2 OR
- Use search to select Lib Dem so that only 4 pages
-
Add new selector
- Show different selectors available
Instructor Note
- Information is on selected page
- Click on selector - breadcrumb changes
Instructor Note
- Slide - code for Diane Abbott email address
- 2nd contact-line
- only one for Jack Abbott
- on inspection can search for ‘Email Address’