Playwright Python

If you don't want to use context managers then you can start the browser without one using the .start() method:

from playwright.sync_api import sync_playwright

p = sync_playwright().start()
browser = p.chromium.launch(headless=False)
page = browser.new_page()
p.stop()

Remember to use the .stop() method once you are done.

You can use the select_options for selecting 30.

page.select_option('select#MainContent_ddlRecords', '30')

You can use the text selector and then click the checkboxes like this:

page.locator("text=Tabloul avocaţilor stagiari").click()

You should follow the Page Object Model Architecture: https://playwright.dev/python/docs/pom

Example of project

conftest.py
pom/
--login/
----login_page.py --> In this file we will have all selectors for login page and the basic functions for each element, like fill_user, fill_password, click_login_button etc.
--search/
----search_page.py --> In this file we will have all selectors for search page and the basic functions for each element, like fill_search_bar, click_search_button etc.
tests/
--login/
----test_login.py --> In this file we will use the methods that login_page.py contains like (fill_user, fill_password, click_login_button)
--search/
---- test_search.py --> In this file we will use the methods that login_page.py contains and methods that search_page.py contains (fill_search_bar, click_search_button)

The locator will help you find an element. But from there, if you don't have an element to pierce the Shadow DOM, you might need to do that manually.

text = page.locator('some-element').first.evaluate("node => node.shadowRoot.innerHTML")

Your code looks fine. Try the df.iloc[row_start:row_finish,column_start:column_finish] command. It works for me. It may also be a problem with the Jupyter notebook - style running of Playwright. Save this into a file.py and run from the terminal in the correct directory: python file.py

I used asyncio to run Playwright, as I had a template script available, but you should get it working without it too, if you want.

import pandas as pd
import asyncio
from playwright.async_api import Playwright, async_playwright
import pandas as pd

async def run(playwright: Playwright) -> None:
    browser = await playwright.chromium.launch(headless=False, slow_mo=50)
    page = await browser.new_page()
    ## Go to PAPA property address
    await page.goto("https://www.pbcgov.org/papa/Asps/PropertyDetail/PropertyDetail.aspx?parcel=73414434010020070")
    x = await page.content() #You're not using this, so you may as well remove this line.

    ## Click text=View Building Details
    await page.locator("text=View Building Details").click()
    #######################################

    ## Click text=Building 2
    await page.frame_locator("#MainContent_Iframe7").locator("text=Building 2").click()
    x2 = await page.frame_locator("#MainContent_Iframe7").locator("html").inner_html()

    ## Click text=Building 3
    await page.frame_locator("#MainContent_Iframe7").locator("text=Building 3").click()
    x3 = await page.frame_locator("#MainContent_Iframe7").locator("html").inner_html()

    ## Click text=Building 4
    await page.frame_locator("#MainContent_Iframe7").locator("text=Building 4").click()
    x4 = await page.frame_locator("#MainContent_Iframe7").locator("html").inner_html()

    x2s = pd.read_html(x2)
    x3s = pd.read_html(x3)
    x4s = pd.read_html(x4)

    print(x2s[3].iloc[0:,0:]) # When it works this is the table that I want
    print(x3s[3].iloc[0:,0:])
    print(x4s[3].iloc[0:,0:])

    year_built_building_1 = x2s[3].iloc[3][2]
    year_built_building_2 = x3s[3].iloc[3][2]
    year_built_building_3 = x4s[3].iloc[3][2]
    
    print(year_built_building_1,year_built_building_2,year_built_building_3)


async def main(debug=True) -> None:
    async with async_playwright() as playwright:
        await run(playwright)

asyncio.run(main())

On my side it outputs:

                                     1                                   2
0   Structural Element for  Building 1  Structural Element for  Building 1
1                                  NaN                                 NaN
2                      Exterior Wall 1               CONCRETE BLOCK STUCCO
3                           Year Built                                2018
4                  Air Condition Desc.                                 NaN
5                            Heat Type                                NONE
6                            Heat Fuel                                NONE
7                            Bed Rooms                                 NaN
8                           Full Baths                                 NaN
9                           Half Baths                                   1
10                      Roof Structure           WOOD JOISTS, WOOD/COMPOSI
11                          Roof Cover           ALUMINUM/STEEL PANEL ENML
12                     Interior Wall 1                              STABLE
13                        Floor Type 1                         RUBBER TILE
14                             Stories                                   1
                                     1                                   2
0   Structural Element for  Building 1  Structural Element for  Building 1
1                                  NaN                                 NaN
2                      Exterior Wall 1               CONCRETE BLOCK STUCCO
3                           Year Built                                2018
4                  Air Condition Desc.                                 NaN
5                            Heat Type                                NONE
6                            Heat Fuel                                NONE
7                            Bed Rooms                                 NaN
8                           Full Baths                                 NaN
9                           Half Baths                                   1
10                      Roof Structure           WOOD JOISTS, WOOD/COMPOSI
11                          Roof Cover           ALUMINUM/STEEL PANEL ENML
12                     Interior Wall 1                              STABLE
13                        Floor Type 1                         RUBBER TILE
14                             Stories                                   1
                                     1                                   2
0   Structural Element for  Building 1  Structural Element for  Building 1
1                                  NaN                                 NaN
2                      Exterior Wall 1               CONCRETE BLOCK STUCCO
3                           Year Built                                2018
4                  Air Condition Desc.                                 NaN
5                            Heat Type                                NONE
6                            Heat Fuel                                NONE
7                            Bed Rooms                                 NaN
8                           Full Baths                                 NaN
9                           Half Baths                                   1
10                      Roof Structure           WOOD JOISTS, WOOD/COMPOSI
11                          Roof Cover           ALUMINUM/STEEL PANEL ENML
12                     Interior Wall 1                              STABLE
13                        Floor Type 1                         RUBBER TILE
14                             Stories                                   1
2018 2018 2018

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''

Methods in python are snake case. context.add_cookies should do the work.

Version 1.22

iframe = page.frame_locator("YOUR_SELECTOR_HERE")

https://playwright.dev/python/docs/api/class-framelocator