How to use test_should_set_cookies_for_a_frame method in Playwright Python

The content function relies on document.documentElement.outerHTML, so you might get a formatted value. If the request returns a JSON you could grab the text from the response that goto returns:

response = page.goto("https://raw.githubusercontent.com/corysimmons/colors.json/master/colors.json")
jsonContent = response.text()
jsonResult = json.loads(jsonContent)
print(jsonResult)

If there is some processing in the middle, you could ask for the inner_text of the :root element:

jsonContent = page.inner_text(':root')
jsonResult = json.loads(jsonContent)
print(jsonResult)

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

There is no direct way to achieve this, but there are workarounds you can use to achieve the same results using routing. Basically, after you no longer require the proxy, you route all requests playwright makes to a handler, which subsequently bypasses proxies and forwards the response back to playwright. However, this approach will only work with async version of playwright (otherwise executing a blocking call from a route handler will block further network traffic too). Consider the below code:

import aiohttp, asyncio
import playwright


def no_proxy_route(session):
    """
    Bypasses any proxy and routes request directly. It's a factory function to store value of session so we don't need
    to create a new one every time.
    """

    async def route_handler(route: playwright.async_api.Route):

        request = route.request

        try:
            body, status, headers = await fetch(session, request)

        # Abort route in case of error
        except (aiohttp.ClientConnectionError, asyncio.TimeoutError):
            await route.abort()
            return

        # If no error, fulfill route with the given body and headers
        await route.fulfill(status=status, headers=headers, body=body)

    return route_handler


async def fetch(session, request, timeout=5000):
    """
    Fetch the given request using aiohttp
    """

    assert timeout > 0, "timeout must be positive"
    try:
        async with session.request(request.method, request.url, headers=request.headers,
                                   data=request.post_data, timeout=timeout) as response:
            body = await response.read()
            status = response.status
            headers = response.headers

    except Exception:
        raise

    else:
        return body, status, headers

Now if you want to conditionally disable proxies for some requests, you simply create a matching pattern and route all those requests through no_proxy_route. An example is given below:

### All your previous code
# .
# .
# .
# .
### which required a proxy

# Create an aiohttp session with a dummy cookie jar. This is because we will be passing the cookies
# explicitly and don't want previous requests/responses cookies to persist and interfere
session = aiohttp.ClientSession(cookie_jar=aiohttp.DummyCookieJar())

# Create a route to our handler with an appropriate pattern. The below pattern will route ALL subsequent requests to 
# handler. Remember that you can use regex for patterns as well.
await page.route('**/*', no_proxy_route(session))

After this is done, all requests that match the pattern, with which you created the route, will not use a proxy, even if the context has one set. But again, this is only useful if you are using the async API for playwright.

Playwright's default timer is 30 seconds. So in the try part, it tries for 30 seconds to get the series, but there is none, so after 30 seconds only, it goes to the exception.

You can set a custom timout when calling inner_text().

See: https://playwright.dev/python/docs/api/class-locator#locator-inner-text

It should be: await (await feedback.query_selector('[class="user-country"]')).inner_text(). But we discourage from using ElementHandles, see here: https://playwright.dev/docs/next/locators#locator-vs-elementhandle

The issue is the docker image you have selected (alpine) architecture tags dont match any that playwright supports.

python:3.9:alpine

/code/src # pip debug --verbose | grep "py3-"
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
  py3-none-musllinux_1_2_x86_64
  py3-none-musllinux_1_1_x86_64
  py3-none-musllinux_1_0_x86_64
  py3-none-linux_x86_64
  py3-none-any

playwright tags

/code/src # pip install playwright -vv | grep "Skipping link" | cut -f2 -d'(' | cut -f1 -d')' | grep "^py3" | sort -u
ERROR: Could not find a version that satisfies the requirement playwright (from versions: none)
ERROR: No matching distribution found for playwright
py3-none-macosx_10_13_x86_64
py3-none-macosx_11_0_arm64
py3-none-macosx_11_0_universal2
py3-none-manylinux1_x86_64
py3-none-manylinux2014_aarch64, py3-none-manylinux_2_17_aarch64
py3-none-manylinux_2_27_x86_64
py3-none-win32
py3-none-win_amd64

This is why pip is telling you there is no matching distribution because its checking all the builds for playwright against the builds alpine linux wants and there isnt a common one, so it cannot install. You will need to use a docker image with an architecture that playwright is built against and supports. You can see this in more detail if you run pip install playwright -vv in the alpine linux image

  Skipping link: none of the wheel's tags (py3-none-manylinux2014_aarch64, py3-none-manylinux_2_17_aarch64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/a6/c5/142449e50253fef
ff667584793ef303c1bb1ecf1cb849cd8b83cd9a653c1/playwright-1.27.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)
  Skipping link: none of the wheel's tags (py3-none-win32) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/79/54/cc0b824dd92507f4ed48e2a71424af2d0b4f91d72f3932ad4fefa7c68981/pla
ywright-1.27.1-py3-none-win32.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)
  Skipping link: none of the wheel's tags (py3-none-win_amd64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/5d/a0/e7009f47b7fef143617a2907fad14170a5ae765fe7bad8050402addd3cab
/playwright-1.27.1-py3-none-win_amd64.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)

Skipping link: not a file: https://pypi.org/simple/playwright/
Given no hashes to check 0 links for project 'playwright': discarding no candidates
ERROR: Could not find a version that satisfies the requirement playwright (from versions: none)
ERROR: No matching distribution found for playwright

You can see the architectures it builds against on there github https://github.com/microsoft/playwright-python/blob/release-1.27/setup.py#L79

google API block you to maximum 20 images

see here for more details

page.goto returns the Response of the page where its navigating to. So you can do something like that:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    response = page.goto("http://whatsmyuseragent.org/")
    assert response.ok

    browser.close()

I didn't use the selector correctly. This is working now correctly

def crawl_mainpage(self, response):
    driver = response.request.meta['driver']
    driver.find_element_by_xpath( "//span[@title='Company Profile']").click()
    sel = Selector(text=driver.page_source)
    driver.quit()

    yield {
    sel.xpath("//td[contains(text(), 'Year Established')]/following-sibling::td/div/div/div/text()").extract()
    }