51,466 questions
0
votes
0
answers
3
views
ModuleNotFoundError: No module named 'scrapy_playwright.middleware'
I am trying out Scrapy Playwright for the first time and following examples online. I am getting the error "ModuleNotFoundError: No module named 'scrapy_playwright.middleware'" when I run ...
- 451
-6
votes
0
answers
40
views
How to fix URL prefix while scrapping [closed]
Whenever I am entering a URL to scrape data, at the backend the URL is added with a prefix, below are the 2 URLs
URL entered -
Boldfit-Thermal-Stylish-Branded-Bennies/product-reviews/B0BQQMLBYB/ref=...
0
votes
0
answers
17
views
Is it possible to scraping the addresses of all the buildings data in a targeted location e.g. mumbai from the google map?
I want to try scraping the addresses of all the buildings data in a targeted location (e.g. mumbai) from the google map using python?how it is possible to find out the all grey color boxes building in ...
- 841
0
votes
1
answer
25
views
I got a connection error when scraping a website using python
Am trying to scrape questions and answers from exammate to get topical questions using the requests library, BeautifulSoup , a regex and then download the images. I got a lot of this code from a ...
-1
votes
1
answer
19
views
Unable to access link to scrap data using Selenium
I am newbie in using Selenium. Recently, I am trying to scrap data from JODI data source. This link is workable in browser but when I use Selenium or bs4 to access this link (in code) it shows error (...
- 3
0
votes
1
answer
25
views
How to run a JavaScript function with playwright inside an iframe?
I am scraping data from a website, and at certain moments, it is necessary to solve a reCAPTCHA v2.
I am using the 2Captcha service to solve these CAPTCHAs. Upon receiving the API response, which is a ...
0
votes
0
answers
37
views
Website behaves differently during Selenium automation compared to manual browsing
I am trying to automate a task using selenium, the issue I am facing is on this link, when you press the button Vote for Asteria MU SEASON 20 PART 1-2 - 99999x and then you will be redirected to a new ...
-1
votes
0
answers
25
views
Adding DOI automatically to BibTex reference file
I have a file named ref.bib (txt file) that contains a list of articles in BibTeX format. For example, here is one article:
@article{campbell2020beyond,
title={Beyond zero trust: Trust is a ...
- 2,281
0
votes
0
answers
10
views
Despite using 'DrissionPage' library, Why I faced "This site can't be reached" issue
Can you please help me to solve this issue?
Why this code doesn't work. Despite using drissionpage library for scraping a website. I can't submit this form automatically though I can fill up the form. ...
0
votes
0
answers
30
views
Web scraping with Flutter
I'm trying to scrape a website using Flutter, the goal is to create an application that tracks trains from my city to Milan so I can use that instead of looking on the official apps which are awful. ...
-2
votes
0
answers
54
views
How to Scrape Data from the Zimyo Website Using Python? [closed]
I am assigned with a task to scrape data from this site. please provide python scraper code. the chrome extensions are found to be useless and i am left with no other issue than to try python code.
2
votes
2
answers
52
views
How to properly perform an Xpath English text search with R package rvest (doesn't seem to work currently)?
Learning rvest and would like to use it to query informational websites to determine if they contain (and then can extract) certain information. For instance, on the U.S. CDC main website:
https://www....
- 23
1
vote
1
answer
32
views
When I navigate to the URL and get the contents of the TABLE tag, it's empty
I am trying to scrape data from this website https://data.anbima.com.br/debentures/AALM11/agenda?page=1&size=100& and when I look at the DevTools > Elements, it has a TABLE tag with the ...
-2
votes
0
answers
38
views
Data scraping from a website with graphs; not able to understand the approach? [closed]
I want to scrape data from a website but I'm unsure of the approach. The website is https://worldhealthorg.shinyapps.io/mpx_global/#5_Detailed_case_data.
My desired output is something like this.
I ...
- 13
0
votes
0
answers
24
views
Why isnt page.evaluate() function code being executed within async function? - Puppeteer [closed]
const getLastCarData = async (page, make) => {
let sort = ascending;
let url = `https://www.xxxxxxxxxxxxx/xxxxxxxxx?make=${make}&postcode=XXXXXXX&sort=${sort}`;
// await page.goto(url,...
- 1
0
votes
2
answers
45
views
Unable to get the proper data of La Liga Points Table from the FC Barcelona website using BS4
Im trying to extract the points table data from the https://www.fcbarcelona.com/en/football/first-team/standings website. I want to extract the data of the current table (24-25) but it is extracting ...
1
vote
1
answer
55
views
Web scraping multiple tables on a page in R
I am trying to scrape all the data in multiple tables for the following link in R using rvest.
https://report-nle.dephub.go.id/dashboard/detail/27019186
When I try the following code, I am only able ...
- 661
0
votes
3
answers
63
views
Xpath - select all columns but the last
I am building a web scraper with Python and Selenium that scrapes the basketball reference website, and am in need of some fine-tuning of the Xpath statements that return the data I'm looking for. ...
- 11
-1
votes
1
answer
76
views
Python Web Scraping Selenium+BeautifulSoup with gspread and Google Sheets problem with big amounts of data
I am making a web scraper that scrapes all of the data from this page with around 1500 products.
I am extracting the name of the item, if it is a best seller, what kind of type of clothing it is, how ...
0
votes
1
answer
38
views
Webscraping a Website with a paginated table but no next button
I am trying to scrape the data from the first 3 pages of the paginated table here: https://www.fda.gov/safety/recalls-market-withdrawals-safety-alerts . So far I can only get data from the first page. ...
- 3
-1
votes
2
answers
37
views
how to write same 3 div class section for scrape
I have a problem with text scraping problem. The website that I am scraping has 3 same line of html but 3 different informations. Like this
'<div class="section">...</div>'
'<...
1
vote
1
answer
49
views
Can't get data from Companies House and HMRC API that have more than £2m in turnover data
I've been dealing with this issue for a day now and have troubles getting company data from HMRC and Companies House API. I mainly need companies that have more than £1m turnover. I have set ...
0
votes
2
answers
59
views
How to parse HTML hidden behind JS scripts
The FCC has a database with details about various broadcast licenses. Many of these licenses have pages like this one
Most of the data on these pages (and related ones) can be scraped very easily with ...
- 1
1
vote
1
answer
42
views
Get data points from a graph and close modal inside shadow root
I am trying to scrape the graph on the website: https://www.xe.com/currencycharts/?from=SGD&to=USD&view=5Y
First I tried to fetch the data via a network request: how to fetch data from dynamic ...
-3
votes
0
answers
40
views
Trying to finish scrape specific data from websites for my Solr search engine for my Django project
I am trying to finish the backend (Django) of my Solr search engine. I defined my fields in search_indexes.py, created my Solr core, configured my schema.xml and rebuilt my index.
A little backstory ...
0
votes
1
answer
29
views
Failed to extract all the image links linked to the floorplans using the requests module
I'm trying to get the image links associated with the floor plans located in the middle of the webpage using the requests module. The links are available in the page source, but I can't manage to ...
- 75
0
votes
1
answer
37
views
how can i use multiprocessing for web scraping in selenium python by calling one function in different driver duo to override issue
I have a sequence to do in web scraping but it took lot of time so instead of calling the single function one by one I want to pass it as different processes and if I do so I have to face conflict of ...
- 1
0
votes
0
answers
13
views
GoogleImageCrawler only downloads 1 image while BingImageCrawler works fine in iCrawler
I’m using iCrawler to create a CLI-based tool for downloading images from Google and Bing. While the BingImageCrawler works perfectly and downloads the requested number of images, the ...
- 1
0
votes
0
answers
27
views
Web scrapping profootballreference in JupyterLite
Hello I am trying to figure out how web scrape www.profootballreference.com using JupyterLite (browser version). I already know how to do it using pycharm and now want to learn how to web scrape using ...
- 13
0
votes
0
answers
38
views
Getting access denied error while scraping API data using Python and Selenium
I am trying to automate a process where I am trying to access the API of a website product list page and scrape product related information. When I use urls1[] array data, my code is giving proper ...
- 25
0
votes
0
answers
86
views
puppeteer Browser Not Getting Launch On Docker With Base Image 16-Buster
Trying To Use This Dockerfile
# Use an official Node.js image as the base
FROM node:16-buster
# Install necessary dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
...
-1
votes
2
answers
68
views
Unable to get a response after sending an HTTP request to a website
I'm trying to send an HTTP request to a website using the requests module, expecting to get a 200 status code. However, when I run the script, I always receive an error, which I will paste below. I've ...
- 142
0
votes
0
answers
41
views
Extracting tables through puppeeteer
I am working on my university project. I want to extract data from my university LMS; I cracked the login authentication, and now I want to extract HTML of my subjects attendance and sessional marks ...
0
votes
0
answers
30
views
Issues setting up Selenium server with PHP
I'm trying (as suggested by someone in my previous question) to set up Selenium Server standalone with php-webdriver, in order to get started with scraping a page behind a form
I'm following this ...
- 11
-1
votes
1
answer
38
views
Puppeteer - pass variable to selector
new to puppeteer and having trouble understanding how I can pass a variable to the browser inside the querySelector
I've got something to that goes and queries number of active users and returns the ...
- 53
0
votes
2
answers
56
views
Unable to download pictures with Python
I want to download four pictures from four Urls. Sometimes I manage to download properly two of them, sometimes one and sometimes none of them. The remaining pictures are downloaded but in a invalid ...
0
votes
2
answers
34
views
How to click and loop through html with Selenium Webdriver/Python with no hyperlink
I'm new to scraping and am trying to get the available apartments and their accompanying information from this website (https://sightmap.com/embed/rx1p8y12vd6). Logically, I'm trying to have selenium ...
- 17
0
votes
2
answers
60
views
Web Scraping Manchester City website using BS4 and python
So basically, I wanted to extract all the texts from the https://www.mancity.com/fixtures website regarding the dates of the upcoming matchs and put it into a list.
The issue is, im unable to extract ...
-4
votes
0
answers
51
views
Identify Dynamic Requests made by javascript for Purpose of Web Scraping
I would like to do web scraping to this webpage, and the following is what I have done trying to find the dynamic URL that is behind the javascipt of that page.
Open the page in Google Chrome.
"...
- 433
0
votes
1
answer
36
views
Python/Selenium: dynamically generated elements with same URL - html doesn't respond to clicking
I'm fairly new to web scraping and may have gone over my head on this, but I am trying to scrape apartment information from a dynamically generated website( https://noveatknox.com/floorplans/). I've ...
- 17
0
votes
1
answer
62
views
How can i handle the shadow-root (open) with selenium
Im working scrapping wallapop making a Second-hand cars listing. But Wallapop has a limit of 26 items. If you click "Ver más productos" (See more products) You can see more products on the ...
0
votes
1
answer
31
views
Why I get extra text when using bs4 and lxml
I have div object
<div class="body">
<div class="pull_right date details" title="21.11.2024 20:17:23 UTC+07:00">
20:17
</div>
<...
- 1
0
votes
0
answers
40
views
<img> Selector Returns Null in Headless Mode but Works in Non-Headless Mode
I'm working on a puppeteer-based scraper that extracts product details (image, title, and price) from a webpage. The scraper works perfectly in non-headless mode, but when I switch to headless mode, ...
- 31
0
votes
0
answers
51
views
Scrape website behind a form with dynamic action link
I'm trying to scrape a website page, but the page is hidden behind a form. I was trying to do this with some PHP and the simple_html_dom.php library. Unfortunately the action link of the form appears ...
- 11
0
votes
1
answer
62
views
How to use an existing browser for Playwright web scraping? [closed]
I am building a web scraper with Python and Playwright, and I’m facing two related challenges:
Playwright requires a separate browser (e.g., Chromium) to be included, which makes my packaged ...
- 77
0
votes
1
answer
106
views
Webscraping a dynamic page with rvest in R
I am wondering if it is possible to webscrape the following page
https://www.pbpstats.com/season-stats/nba?EntityType=Player&EntityId=1630169
Towards the top of the page there is a dropdown menu ...
0
votes
3
answers
83
views
How do I scrape the specific data I want?
I am taking my first crack at data scraping and am having trouble getting the specific data I want. Ultimately, I want to identify all players that were born and/or played high school baseball in ...
1
vote
1
answer
64
views
Scraping RotoWire Player Props and Returning Them as a DataFrame
I am trying to scrape https://www.rotowire.com/betting/nba/player-props.php and have the results returned in a dataframe. There are about a dozen tables on the page, and ideally I could loop through ...
- 23
1
vote
1
answer
40
views
Scraping data from fiba.basketball game overview page in R
FIBA used to have a page that was much more accessible to grab play-by-play and box score data from (example: scrape fiba stats box score).
They have a new game page that looks like this https://www....
- 47
0
votes
1
answer
35
views
Signature of method 'myspider. parse()' does not match signature of the base method in class 'Spider'
I'm trying to scrape a webpage (https://a-z-animals.com/animals/) to get all the animal names listed there.
I installed scrapy in my PyCharm project. Then, by using the terminal in PyCharm, created a ...
