Text · dataset · 2021
Evolution of Web search engine interfaces through SERP screenshots and HTML complete pages for 20 years
Listed in INESC TEC Research Data Repository
This dataset was extracted for a study on the evolution of Web search engine interfaces since their appearance.
Description
The well-known list of “10 blue links” has evolved into richer interfaces, often personalized to the search query, the user, and other aspects. We used the most searched queries by year to extract a representative sample of SERP from the Internet Archive.
The Internet Archive has been keeping snapshots and the respective HTML version of webpages over time and tts collection contains more than 50 billion webpages. We used Python and Selenium Webdriver, for browser automation, to visit each capture online, check if the capture is valid, save the HTML version, and generate a full screenshot. The dataset contains all the extracted captures.
Read the rest (4 more)
Each capture is represented by a screenshot, an HTML file, and a files' folder. We concatenate the initial of the search engine (G) with the capture's timestamp for file naming. The filename ends with a sequential integer "-N" if the timestamp is repeated.
For example, "G20070330145203-1" identifies a second capture from Google by March 30, 2007. The first is identified by "G20070330145203". Using this dataset, we analyzed how SERP evolved in terms of content, layout, design (e.g., color scheme, text styling, graphics), navigation, and file size.
We have registered the appearance of SERP features and analyzed the design patterns involved in each SERP component. We found that the number of elements in SERP has been rising over the years, demanding a more extensive interface area and larger files. This systematic analysis portrays evolution trends in search engine user interfaces and, more generally, web design.
We expect this work will trigger other, more specific studies that can take advantage of the dataset we provide here. This graphic represents the diversity of captures by year and search engine (Google and Bing).
Links
Get the data
- CapturesDataset rdm.inesctec.pt/dataset/4f8782a7-e9b0-46ae-beca-b8641f24c9e8/resource/a87e977f… ↗
download · from rdm inesctec pt
- ReadMe rdm.inesctec.pt/dataset/4f8782a7-e9b0-46ae-beca-b8641f24c9e8/resource/9bcbe426… ↗
download · from rdm inesctec pt
Where it is published
- INESC TEC Research Data Repository dataset page rdm.inesctec.pt/dataset/cs-2021-003 ↗
landing page · from rdm inesctec pt
Catalogue records · 1
- CKAN API rdm.inesctec.pt/api/3/action/package_show?id=4f8782a7-e9b0-46ae-beca-b8641f24c… ↗
metadata API · from rdm inesctec pt
Topics
- Stated by source
- CS: Computer Science
- From keywords
- Computer Science & AI · Engineering
- Inferred from text
- Text 75%
Provenance · 1 source records, 11 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| INESC TEC Research Data Repository | 4f8782a7-e9b0-46ae-beca-b8641f24c9e8 | 13 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| concepts[field].local:field:computer-science-ai | mapping · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| concepts[field].local:field:engineering | mapping · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| concepts[modality].local:modality:text | enrichment · rdm inesctec pt | keyword-concept-rules@1.0.0 | title+description (75%) |
| concepts[topic].ckan_group:rdm_inesctec_pt:cs-computer-science | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| created_date | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| description | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | /notes |
| license_text | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| publication_date | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | |
| title | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 | /title |
| updated_date | source · rdm inesctec pt | connector:rdm_inesctec_pt@1.0.0 |