This repository was archived by the owner on Jan 3, 2024. It is now read-only.
This repository was archived by the owner on Jan 3, 2024. It is now read-only.
Unable to get content of page from special characters (utf-8) URL #733
Open
Description
Hey,
I am currently in progress of switching from main Selenium to Selenium Wire to start using proxies for scraping activities.
Now everything works fine and as expected, except websites that have special characters in their addresses, like https://männimetsa.ee/. Here I have actually two issues.
- Website
page_sourceresults in empty content<html><head></head><body></body></html>. No matter what I do, if I implement waiting or not, this is what I get. No actual Exception in my try catch block is thrown. Just emptybody. - If using
request_interceptorI get this error:UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 1: ordinal not in range(128). Even if inside interceptor I just print out the request.path, its the error that gets thrown in my face. And printing request.path results in this text:https://männimetsa.ee/where thehttps://mpart is as active URL.
Not sure if issue 1 is related to issue 2, but both issues only happen with special character website URLs. So I have a feeling that both issues are caused by the same main reason.
If I switch back to Selenium, I can get the content of website with no problems.
Metadata
Metadata
Assignees
Labels
No labels