Botflow is programming framework built for createing fast data pipeline. This tutorial is a sample for analyse Python book price distrubution with amazon book search result . And the tutorial dose not require extra tools and make you leave this page.
more information can be found in project home page:
https://github.com/kkyon/botflow
The package you need to install
pip3 instll -U botflow
update tonado to 4.5.3 because jupyter asyncio support issue.
pip3 install tornado==4.5.3
and you should have baic knowlege with html and beautiful soup.
All Steps¶
- Donwloads the serach result page (html) and get the book item html
- parse the price ,book title,images src
- filter out spam item
- create pandas dataframe from the prices .And draw the hist
Step 1. Create download pipe.¶
Make amazon book search with keyword Python in your chrome. You can get the search result page url :
https://www.amazon.com/s/ref=sr_pg_2?fst=p90x%3A1&rh=n%3A283155%2Ck%3Apython&page=2&d=1&keywords=python&ie=UTF8&qid=1536905092```
We can fingure the page no and keyword word in the url . In this tutorial we only replace page no with variable . You can try change the search keywords later.
```python
lambda p:f"https://www.amazon.com/s/ref=sr_pg_{p}?fst=p90x%3A1&page={p}&rh=n%3A283155%2Ck%3Apython&keywords=python&ie=UTF8&qid=1536500367",And with inspect the html ,we can find book item is under li tage . get all tag with soup api .find_all. It also return many unnecceary items .Don't worry about this .will clean it later.
Let's get start with only one page. We can easy fetch more page just change run param.
from botflow import *
p_download=Pipe(
lambda p:f"https://www.amazon.com/s/ref=sr_pg_{p}?fst=p90x%3A1&page={p}&rh=n%3A283155%2Ck%3Apython&keywords=python&ie=UTF8&qid=1536500367",
HttpLoader(), #download the page
lambda r:r.soup.find_all("li"),
Flat() #we don't need keep the hierarchy of page no.
)
lis=p_download.run(1) #get first page
#lis=p.run(range(1,10)) get top 10 pages items
print(len(lis))
We got many items now. let's pick one and see inside. It is actually a Html Tag . html code seem messy .How can first price infrom from it?
lis[1]
Step2: Parse book item (price,title,image)¶
Ok .Let's us play magic with jupyter book ,print out the html visually .and will inspect the code whitout leave this page.
from IPython.core.display import display, HTML
display(HTML(str(lis[1])))
Sponsored
Clicking an ad will take you to the product's page.Learn more about Sponsored Products.
[Sponsored]Matplotlib 2.x By Example: Multi-dimensional charts, graphs, and plots in Python
Aug 28, 2017With chrome inspect (right click menu > inspect ) the intresting element of the above Html block. it is easy to find out the tage anem with developer windows right side:
images : with tag imge ,inside A link
price : with css class span.a-offscreen and it is child of A link
title: with tag h2 ,inside A linkWe can build pipes for parsing price, image,title separatly . Remenber ,test with only item firstly .If you feel it is ok we can run it will full list.
p_get_img_src=Pipe(lambda t:t.select("a img"),lambda t:t['src'])
p_get_img_src.run(lis[1])
p_get_title=Pipe(lambda t:t.select("a h2"),lambda t:t.get_text())
p_get_title.run(lis[1])
p_get_price=Pipe(lambda t:t.select("a > span.a-offscreen"),lambda t:t.get_text())
p_get_price.run(lis[1])
p_get_price.Map(lambda x:float(x.replace("$",'')))
p_get_price.run(lis[1])
We created the imge,title,price parser pipes. now need to combine the results together. Zip operator is design for this purpose . it will combine muliti pipe resul to a list.
item_parse=Pipe(Zip(p_get_price,p_get_title,p_get_img_src))
item_parse.run(lis[1])
Now we run the p_item_parse with full list. Actully there is now differenc from runnning one item.
item_parse=Pipe(Zip(p_get_price,p_get_title,p_get_img_src))
item_parse.run(lis)
Step3: Filter the None item¶
We will use Filter operator to filter out the item with None price.
item_parse=Pipe(Zip(p_get_price,p_get_title,p_get_img_src)).Filter(lambda i : i[0])
r=item_parse.run(lis)
r
Step4: Draw hist with Pandas¶
import pandas as pd
headers=["price","title","image_src"]
df = pd.DataFrame(r, columns=headers)
df
There is some bad item with mulit prices .I just drop it out .You can do more complex convert with pandas.
price=pd.to_numeric(df['price'], errors='coerce')
price.hist()
Run with more data:¶
Get more item by set range of page no.¶
lis=p_download.run(range(1,10)
Change search keywords.¶
it is unable to create nest loop in pipes. we generate all combination with prodcut function.
from itertools import product
lis=p_download.run(product(range(1,10),['Python','Java','C++'])