Published By
Created On
30 Oct 2020 13:17:59 UTC
Transaction ID
Cost
Safe for Work
Free
Yes
How to Web Scrape Amazon using Python, Scrapy and MySQL | View output in phpmyadmin
Web scraping Amazon for 'Web Scraping Books' and storing the results to a MySQL (MariaDB) database *for academic purposes only* using Python and the Scrapy framework.
This video shows the 4 files you need to edit within the Scrapy framework, and a guided tour of the code including the database in phpmyadmin.
This builds upon the techniques and code shown in the previous videos so you may wish to watch those first.
Link : https://www.youtube.com/playlist?list=PLKMY3XNPiQ7sp76LfY81dgpZRdaNi4nnM
I have a website www.redandgreen.co.uk where you can contact me, and also read more about the code I've shown in this Amazon / SQL / Webscraping video series.
Feel free to view, clone and use my code for educational purposes, it's al on:
https://github.com/RGGH/Scrapy10/tree/DB-MySQL
Thanks, for watching,
See you around yeah?
Dr P.
...
https://www.youtube.com/watch?v=1BvWOdZ4S-E
This video shows the 4 files you need to edit within the Scrapy framework, and a guided tour of the code including the database in phpmyadmin.
This builds upon the techniques and code shown in the previous videos so you may wish to watch those first.
Link : https://www.youtube.com/playlist?list=PLKMY3XNPiQ7sp76LfY81dgpZRdaNi4nnM
I have a website www.redandgreen.co.uk where you can contact me, and also read more about the code I've shown in this Amazon / SQL / Webscraping video series.
Feel free to view, clone and use my code for educational purposes, it's al on:
https://github.com/RGGH/Scrapy10/tree/DB-MySQL
Thanks, for watching,
See you around yeah?
Dr P.
...
https://www.youtube.com/watch?v=1BvWOdZ4S-E
Author
Content Type
Unspecified
video/mp4
Language
English
More from the publisher
Controlling
VIDEO
LOOKI
looking-at-leetcode-debugging-in-vscode
Solving a problem on Leetcode, or rather taking inspiration from Leetcode and researching, learning new algorithms, and generally focusing on how the program works rather than the specific language it's written in.
If you're looking at learning another language maybe solve a problem on Leetcode in one language then write and test in another?
At the end I discuss EBAY API and ZSH shell customisation.
Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?
You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?
? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)
-------------------------------------------------------------------------
You can buy bitcoin here ???
? https://www.swanbitcoin.com/python360
+ get $10 in BTC when you join Swan !!
--------------------------------------------------------------------------
? Become a patron ?
? https://www.patreon.com/drpi
Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi
Thumbs up yeah? (cos Algos..)
#vscodedebugger #tutorials #leetcode
...
https://www.youtube.com/watch?v=uja4acDx9R8
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
PRICE
price-tracking-amazon-tutorial-python
How to web scrape Amazon and visualize the collected data for price tracking and trend monitoring.
This is a completed project based on web scraping the Amazon books category for: 'Web Scraping' books using Scrapy in Python. If you want to start collecting data now, so you can check if 'Black Friday' deals are really reduced then now is the time to start checking prices.
All of the files are available on GitHub. If you follow this you will learn how to web scrape Amazon to analyse entire categories of products and present the output in a human friendly plot (graph).
Warning : This is neither particularly 'quick' or 'easy' - there are plenty of tutorials that will use Selenium or BeautifulSoup to scrape a few products if that is what you need.
This project uses:
Python ✔️
Scrapy ✔️
MySQL ✔️
Pandas ✔️
Matplotlib ✔️
Raspberry Pi ✔️
ssh ✔️
CRON ✔️
~ CHAPTER TIMES ~
0:00 Intro
0:10 Network topology
0:18 Crontab CRON config
1:00 Run the spider manually
2:00 Viewing the output
4:38 The SQL query
6:18 The query result
10:50 Fixing remote access to MySQL database
12:12 bind-address = 0.0.0.0
14:17 Visualize the extracted data
19:18 Successful output
21:21 Adjusting the graph
Code : https://github.com/RGGH/Scrapy10/tree/DB-MySQL
This can be adapted and modified for price tracking and monitoring of any products on Amazon.
The entire Amazon Web Scraping playlist is here: https://www.youtube.com/playlist?list=PLKMY3XNPiQ7t8-1BT3XeM55nDLxtc0yoU
Disclaimer : This is for educational purposes only, if you run this code then you are responsible for it's use.
See you around yeah?
Dr. P
#python #amazon #blackfriday2020
...
https://www.youtube.com/watch?v=grkwcZmfeLA
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
WHY I
why-i-didn-t-use-selenium
Issues I had with trying to webscrape using Selenium.
Anybody got this working on current version of Raspbian on Pi / Pi Zero?
Check out the Minimalist online python IDE :
https://epyco.herokuapp.com/
https://epico.herokuapp.com/
Buy Dr Pi a Coffee...or Tea! : https://www.buymeacoffee.com/DrPi
...
https://www.youtube.com/watch?v=uw-wLjh8OQI
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
FIND
find-keywords-in-a-csv-file-based-on
Helping a subscriber Find Keywords in a CSV, here was his problem :
"Let's say we have a text file or csv file which contains specific keywords(related to social responsibility) and we have to find these keywords in all of our 469 files and have to conclude that these companies containing csr keywords are socially responsible"
This video features 'glob' and 'strip' along with 'list comprehensions' and 'readlines'
Helping a subscriber with his Machine Learning project!
Visit redandgreen blog for more Tutorials
=========================================
? http://redandgreen.co.uk/about/blog/
Subscribe to the YouTube Channel
=================================
? https://www.youtube.com/c/DrPiCode
Follow on Twitter - to get notified of new videos
=================================================
? https://twitter.com/RngWeb
? Become a patron ?
? https://www.patreon.com/drpi
Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi
Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?
You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?
? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)
Thumbs up yeah? (cos Algos..)
#keywords #csv #python
...
https://www.youtube.com/watch?v=SyHthrP7FQQ
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
YOUTU
youtube-data-api-part2-video-comments
Python code to extract metadata from YouTube videos for one entire channel. This is part 2, showing progress as I use the YouTube data API v3 to access resources from any chosen channel and format it to provide information as required.
*The YouTube Data API allows you to add a range of YouTube features to your application.
Using the API you can upload videos, manage playlists, and subscriptions, update channel settings, retrieve comments, descriptions and more. See what I have collected so far :
Intro
Code @ 1:06
Output @ 4:19
https://developers.google.com/youtube/v3/
The code so far is on my GitHub repo : https://github.com/RGGH/rng
1. Video title ✅
2. Subscribe count
3. Video id ✅
4. Video description ✅
5. Likes count
6. Dislikes count
7. Views count
8. Published date ✅
9. Comment count
10. Comment text ✅
11. Comment likes and dislikes
12. Comment author ✅
...
https://www.youtube.com/watch?v=PqS4B-8lzlA
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
PYTHO
python-github-workflow-poetry,-pytest,
Poetry, Pytest, GitHub Actions.
Automate your python project creation and use GitHub actions to automatically test your code with every push.
In this GitHub Actions worfklow Tutorial we cover the following topics:
First we look at the process of creating a GithHub Repository, cloning the repo,starting poetry, and then doing a push back to GitHub.
Also, we'll see how GitHub actions provide you with the green badge to show users your code has passed its checks.
After the overview with screenshots we run through the through a hands-on DEMO, where I show the syntax of poetry, and the github actions workflow file, and the entire CI/CD pipeline setup with my example Python project.
◼️ "Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you."
◼️ "GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD"
◼️ "pytest is a mature full-featured Python testing tool that helps you write better programs."
-- chapters --
00:00 Intro
01:45 Workflow
03:22 Overview with screenshots
12:41 VSCode demo of project
20:05 Altering import path / "Module not found"
25:52 Configure GitHub actions
------------------------------------------------------------------------------
https://python-poetry.org/
https://packaging.python.org/
-------------------------------------------------------------------------
You can buy bitcoin here ???
? https://www.swanbitcoin.com/python360
+ get $10 in BTC when you join Swan !!
--------------------------------------------------------------------------
? Become a patron ?
? https://www.patreon.com/drpi
Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi
Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?
You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?
? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)
Thumbs up yeah? (cos Algos..)
#githubactions #poetry #python
...
https://www.youtube.com/watch?v=HNhjWqZaFog
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
PYTHO
python-code-to-get-json-from-javascript
How to parse response.text from Scrapy, (or requests, or Beautiful Soup).
(No regular expressions needed, just split, strip, replace).
If you find that you are trying to scrape comments from a page, and they're inside javascript then you will need a way to get the values/text some other way. CSS or XPATH won't help you.
Here I show how to parse the javascript to pick out the JSON, and then load the JSON into a dictionary. Once the key:values are there, you can work with them very easily.
◉ 1:26 looking for JSON
◉ 2:19 view source
◉ 2:55 found JSON.parse
◉ 4:45 found comments inside JSON
◉ 5:29 print(response.text)
◉ 6:09 x = response.text.split('JSON.parse')[2].replace("\\u0022","\"").lstrip("\(").split(" ")[0].replace("}\");","\"").replace("\"{","\""})
◉ 8:43 json.loads
◉ 12:27 final output
Read about the process of extracting a dict from json from javascript on my redandgreen.co.uk website : https://redandgreen.co.uk/extracting-json-from-javascript-in-a-web-page/
RTFM ? :
https://docs.scrapy.org/en/latest/
https://docs.scrapy.org/en/latest/topics/shell.html
☕️☕️☕️ Buy Dr Pi a Coffee...or Tea! : https://www.buymeacoffee.com/DrPi ☕️☕️☕️
Thanks for watching.
See you around yeah?
Dr P.
...
https://www.youtube.com/watch?v=G_WY3XFuDXY
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
PYTHO
python-v-c-to-solve-same-problem
An easy to follow comparison between Python and C using the code to solve a factorial.
Visit redandgreen blog for more Tutorials
=========================================
? http://redandgreen.co.uk/about/blog/
Subscribe to the YouTube Channel
=================================
? https://www.youtube.com/c/DrPiCode
Follow on Twitter - to get notified of new videos
=================================================
? https://twitter.com/RngWeb
? Become a patron ?
? https://www.patreon.com/drpi
Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi
Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?
You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?
? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)
Thumbs up yeah? (cos Algos..)
#webscraping #tutorials #python
...
https://www.youtube.com/watch?v=zZgNSCEFelE
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English
Controlling
VIDEO
PYTHO
python-jobs-in-the-uk-march-2022
A look at Python jobs in the UK - March 2022
◼️ Remote work from home 'available'
◼️ How many specify AWS and Django?
◼️ "Support" v "Developer"
◼️ What is JIRA?
◼️ Check out the new 'Brave' Browser
https://unherd.com/2021/06/the-truth-about-the-great-reset/
"Standard recruitment costs tend to range between 15% and 20% of a candidate's first annual salary, but this can go as high as 30% for hard to fill positions."
Subscribe to the YouTube Channel
=================================
? https://www.youtube.com/c/DrPiCode
? Become a patron ?
? https://www.patreon.com/drpi
Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi
Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?
You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?
? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)
Thumbs up yeah? (cos Algos..)
#Jobs #WorkFromHome #python
...
https://www.youtube.com/watch?v=Sl9Txu38OAU
Transaction
Created
1 year ago
Content Type
Language
video/mp4
English