How to use _get_stderr_fileno method in Playwright Python

There is no direct way to achieve this, but there are workarounds you can use to achieve the same results using routing. Basically, after you no longer require the proxy, you route all requests playwright makes to a handler, which subsequently bypasses proxies and forwards the response back to playwright. However, this approach will only work with async version of playwright (otherwise executing a blocking call from a route handler will block further network traffic too). Consider the below code:

import aiohttp, asyncio
import playwright


def no_proxy_route(session):
    """
    Bypasses any proxy and routes request directly. It's a factory function to store value of session so we don't need
    to create a new one every time.
    """

    async def route_handler(route: playwright.async_api.Route):

        request = route.request

        try:
            body, status, headers = await fetch(session, request)

        # Abort route in case of error
        except (aiohttp.ClientConnectionError, asyncio.TimeoutError):
            await route.abort()
            return

        # If no error, fulfill route with the given body and headers
        await route.fulfill(status=status, headers=headers, body=body)

    return route_handler


async def fetch(session, request, timeout=5000):
    """
    Fetch the given request using aiohttp
    """

    assert timeout > 0, "timeout must be positive"
    try:
        async with session.request(request.method, request.url, headers=request.headers,
                                   data=request.post_data, timeout=timeout) as response:
            body = await response.read()
            status = response.status
            headers = response.headers

    except Exception:
        raise

    else:
        return body, status, headers

Now if you want to conditionally disable proxies for some requests, you simply create a matching pattern and route all those requests through no_proxy_route. An example is given below:

### All your previous code
# .
# .
# .
# .
### which required a proxy

# Create an aiohttp session with a dummy cookie jar. This is because we will be passing the cookies
# explicitly and don't want previous requests/responses cookies to persist and interfere
session = aiohttp.ClientSession(cookie_jar=aiohttp.DummyCookieJar())

# Create a route to our handler with an appropriate pattern. The below pattern will route ALL subsequent requests to 
# handler. Remember that you can use regex for patterns as well.
await page.route('**/*', no_proxy_route(session))

After this is done, all requests that match the pattern, with which you created the route, will not use a proxy, even if the context has one set. But again, this is only useful if you are using the async API for playwright.

I had the same issue as well. I ran this command, then restarted my command line and it started working.

playwright install-deps

I hope this helps.

google API block you to maximum 20 images

see here for more details

When you visit the page, playwright/chrome will need to send a network request to retrieve the PDF you're looking at. Just intercept that request, and grab the response body as a buffer.

If you are just navigating directly to the PDF, I think something like this would work:

response = page.goto(url)
print(response.status)
body = response.body()

# Open file in binary write mode
pdf_file = open('file.pdf', 'wb')

# Write bytes to file
pdf_file.write(body)

# Close file
pdf_file.close()

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

How about checking if the element is clickable: https://playwright.dev/docs/api/class-locator#locator-click

use trial option and if it returns true continue execution

  await page.click(First_Row_Download, , {trial:true})

You can pass both selectors separated by a comma. Ref.

new_selector = "text=Your are a winner, text=Better luck next time"
handle = query_selector(new_selector)

The following is one way to get the images on that page - apparently only 398, not 600+ - it's even specified in the top breadcrumb - Trail Running Shoes(398)). Solution is based on Selenium, you're welcome to functionalize it, OOP it, whatever, I'm only giving you the actual method of obtaining the images.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time as t

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("--headless")
chrome_options.add_argument("window-size=1280,1080")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
actions = ActionChains(browser)


url = 'https://www.tradeinn.com/runnerinn/en/mens-shoes-trail-running-shoes/10005/s#fq=id_familia=10002&sort=v30_sum;desc@tm10;asc&fe=&pf=id_subfamilia=10005@&start=0'
browser.get(url)
pbody = wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
for x in range(14):
    pbody.send_keys(Keys.PAGE_DOWN)
    print('scrolled')
    t.sleep(1)
t.sleep(5)
images = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'img[class="imagen_buscador"]')))
print(len(images))
for i in images:
    print(i.get_attribute('src'))

Result in terminal:

scrolled
scrolled
scrolled
[...]
398
https://www.tradeinn.com/h/13855/138556150/nike-juniper-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138552239/nike-react-pegasus-4-trail-running-shoes.jpg
https://www.tradeinn.com/h/13727/137276760/merrell-vapor-glove-3-shoes.jpg
https://www.tradeinn.com/h/13842/138429239/adidas-terrex-agravic-flow-2-goretex-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138556425/nike-wildhorse-7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13842/138429416/adidas-terrex-two-goretex-trail-running-shoes.jpg
https://www.tradeinn.com/h/13842/138429408/adidas-terrex-two-boa-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138556423/nike-wildhorse-7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574338/new-balance-410v7-all-terrain-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574792/new-balance-fresh-foam-x-hierro-v7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574340/new-balance-410v7-all-terrain-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137892027/adidas-terrex-two-boa-trail-running-shoes.jpg
https://www.tradeinn.com/h/13727/137276761/merrell-vapor-glove-3-shoes.jpg
https://www.tradeinn.com/h/13836/138368054/joma-trek-trail-running-shoes.jpg
https://www.tradeinn.com/h/13710/137107634/vibram-fivefingers-v-trail-2.0-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137891690/adidas-terrex-agravic-flow-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137892315/adidas-terrex-swift-r3-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138552241/nike-react-pegasus-4-trail-running-shoes.jpg
https://www.tradeinn.com/h/13803/138030718/nike-wildhorse-7-trail-running-shoes.jpg
[..]

For Selenium docs, see https://www.selenium.dev/documentation/

After investigating and trying several things, looks like the problem is the user_agent of the browser when is in headless mode, for some reason the default user agent does not like to that page, try with:

def extract_html(self):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36')
        page.goto('http://book.flygofirst.com/Flight/Select?inl=0&CHD=0&s=True&o1=BOM&d1=BLR&ADT=1&dd1=2022-12-10&gl=0&glo=0&cc=INR&mon=true')
        html = page.inner_html('#sectionBody')
        return html