I believe you want something like this:

for element in soup.find_all("a", {"class":"..."}):
    print(element['href'])

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

According to your question, to get the json response, You can follow my working solution as an example. Actually, the hidden api calls json response is here. The interesing matter is that the request method is post but it sends query string parameters & request payload/formdata and the next pages at the same time which type of response I face first time ever and I have to make both types of parameters to get desired json response. I've also made the pagination following json response and you can increase or decrease it according to json response maxpage.

import requests
import json
data= {
   
   "query":"query Browse( $query:String $page:Int $prg:Prg! $facet:String $sort:Sort $catId:String! $max_price:String $min_price:String $module_search:String $affinityOverride:AffinityOverride $ps:Int $ptss:String $beShelfId:String $fitmentFieldParams:JSON ={}$fitmentSearchParams:JSON ={}$rawFacet:String $seoPath:String $trsp:String $fetchMarquee:Boolean! $fetchSkyline:Boolean! $additionalQueryParams:JSON ={}){search( query:$query page:$page prg:$prg facet:$facet sort:$sort cat_id:$catId max_price:$max_price min_price:$min_price module_search:$module_search affinityOverride:$affinityOverride additionalQueryParams:$additionalQueryParams ps:$ps ptss:$ptss trsp:$trsp _be_shelf_id:$beShelfId ){query searchResult{...BrowseResultFragment}}contentLayout( channel:\"WWW\" pageType:\"BrowsePage\" tenant:\"WM_GLASS\" version:\"v1\" searchArgs:{query:$query cat_id:$catId _be_shelf_id:$beShelfId prg:$prg}){modules{...ModuleFragment configs{...on EnricherModuleConfigsV1{zoneV1}__typename...on _TempoWM_GLASSWWWSearchSortFilterModuleConfigs{facetsV1{...FacetFragment}}...on TempoWM_GLASSWWWPillsModuleConfigs{moduleSource pillsV2{...PillsModuleFragment}}...on TempoWM_GLASSWWWSearchFitmentModuleConfigs{fitments( fitmentSearchParams:$fitmentSearchParams fitmentFieldParams:$fitmentFieldParams ){...FitmentFragment sisFitmentResponse{...BrowseResultFragment}}}...on TempoWM_GLASSWWWStoreSelectionHeaderConfigs{fulfillmentMethodLabel storeDislayName}...on TempoWM_GLASSWWWBreadcrumbConfigs{_rawConfigs}...on TempoWM_GLASSWWWSponsoredProductCarouselConfigs{_rawConfigs}...PopularInModuleFragment...CopyBlockModuleFragment...BannerModuleFragment...HeroPOVModuleFragment...InlineSearchModuleFragment...MarqueeDisplayAdConfigsFragment @include(if:$fetchMarquee)...SkylineDisplayAdConfigsFragment @include(if:$fetchSkyline)...HorizontalChipModuleConfigsFragment}}...LayoutFragment pageMetadata{location{postalCode stateOrProvinceCode city storeId}pageContext}}seoBrowseMetaData( id:$catId facets:$rawFacet path:$seoPath facet_query_param:$facet _be_shelf_id:$beShelfId ){metaTitle metaDesc metaCanon h1}}fragment BrowseResultFragment on SearchInterface{title aggregatedCount...BreadCrumbFragment...DebugFragment...ItemStacksFragment...PageMetaDataFragment...PaginationFragment...RequestContextFragment...ErrorResponse modules{facetsV1{...FacetFragment}pills{...PillsModuleFragment}}}fragment ModuleFragment on TempoModule{name version type moduleId schedule{priority}matchedTrigger{zone}}fragment LayoutFragment on ContentLayout{layouts{id layout}}fragment BreadCrumbFragment on SearchInterface{breadCrumb{id name url}}fragment DebugFragment on SearchInterface{debug{sisUrl}}fragment ItemStacksFragment on SearchInterface{itemStacks{displayMessage meta{adsBeacon{adUuid moduleInfo max_ads}query stackId stackType title layoutEnum totalItemCount totalItemCountDisplay viewAllParams{query cat_id sort facet affinityOverride recall_set min_price max_price}}itemsV2{...ItemFragment...InGridMarqueeAdFragment}}}fragment ItemFragment on Product{__typename id usItemId fitmentLabel name checkStoreAvailabilityATC seeShippingEligibility brand type shortDescription imageInfo{...ProductImageInfoFragment}canonicalUrl externalInfo{url}category{path{name url}}badges{flags{...on BaseBadge{key text type id}}tags{...on BaseBadge{key text type}}}classType averageRating numberOfReviews esrb mediaRating salesUnitType sellerId sellerName hasSellerBadge availabilityStatusV2{display value}productLocation{displayValue aisle{zone aisle}}badge{type dynamicDisplayName}fulfillmentSpeed offerId preOrder{...PreorderFragment}priceInfo{...ProductPriceInfoFragment}variantCriteria{...VariantCriteriaFragment}fulfillmentBadge fulfillmentTitle fulfillmentType brand manufacturerName showAtc sponsoredProduct{spQs clickBeacon spTags}showOptions}fragment ProductImageInfoFragment on ProductImageInfo{thumbnailUrl}fragment ProductPriceInfoFragment on ProductPriceInfo{priceRange{minPrice maxPrice}currentPrice{...ProductPriceFragment}wasPrice{...ProductPriceFragment}unitPrice{...ProductPriceFragment}listPrice{...ProductPriceFragment}shipPrice{...ProductPriceFragment}subscriptionPrice{priceString subscriptionString}priceDisplayCodes{priceDisplayCondition finalCostByWeight}}fragment PreorderFragment on PreOrder{isPreOrder preOrderMessage preOrderStreetDateMessage}fragment ProductPriceFragment on ProductPrice{price priceString}fragment VariantCriteriaFragment on VariantCriterion{name type id isVariantTypeSwatch variantList{id images name rank swatchImageUrl availabilityStatus products selectedProduct{canonicalUrl usItemId}}}fragment InGridMarqueeAdFragment on MarqueePlaceholder{__typename type moduleLocation lazy}fragment PageMetaDataFragment on SearchInterface{pageMetadata{storeSelectionHeader{fulfillmentMethodLabel storeDislayName}title canonical description location{addressId}}}fragment PaginationFragment on SearchInterface{paginationV2{maxPage pageProperties}}fragment RequestContextFragment on SearchInterface{requestContext{vertical isFitmentFilterQueryApplied searchMatchType categories{id name}}}fragment ErrorResponse on SearchInterface{errorResponse{correlationId source errors{errorType statusCode statusMsg source}}}fragment PillsModuleFragment on PillsSearchInterface{title url image:imageV1{src alt}baseSeoURL}fragment BannerModuleFragment on TempoWM_GLASSWWWSearchBannerConfigs{moduleType viewConfig{title image imageAlt displayName description url urlAlt appStoreLink appStoreLinkAlt playStoreLink playStoreLinkAlt}}fragment PopularInModuleFragment on TempoWM_GLASSWWWPopularInBrowseConfigs{seoBrowseRelmData(id:$catId){relm{id name url}}}fragment CopyBlockModuleFragment on TempoWM_GLASSWWWCopyBlockConfigs{copyBlock(id:$catId){cwc}}fragment FacetFragment on Facet{name type layout min max selectedMin selectedMax unboundedMax stepSize values{id name description type itemCount isSelected baseSeoURL}}fragment FitmentFragment on Fitments{partTypeIDs result{status formId position quantityTitle extendedAttributes{...FitmentFieldFragment}labels{...LabelFragment}resultSubTitle}labels{...LabelFragment}savedVehicle{vehicleYear{...VehicleFieldFragment}vehicleMake{...VehicleFieldFragment}vehicleModel{...VehicleFieldFragment}additionalAttributes{...VehicleFieldFragment}}fitmentFields{...VehicleFieldFragment}fitmentForms{id fields{...FitmentFieldFragment}title labels{...LabelFragment}}}fragment LabelFragment on FitmentLabels{ctas{...FitmentLabelEntityFragment}messages{...FitmentLabelEntityFragment}links{...FitmentLabelEntityFragment}images{...FitmentLabelEntityFragment}}fragment FitmentLabelEntityFragment on FitmentLabelEntity{id label}fragment VehicleFieldFragment on FitmentVehicleField{id label value}fragment FitmentFieldFragment on FitmentField{id displayName value extended data{value label}dependsOn}fragment HeroPOVModuleFragment on TempoWM_GLASSWWWHeroPovConfigsV1{povCards{card{povStyle image{mobileImage{...TempoCommonImageFragment}desktopImage{...TempoCommonImageFragment}}heading{text textColor textSize}subheading{text textColor}detailsView{backgroundColor isTransparent}ctaButton{button{linkText clickThrough{value}}}logo{...TempoCommonImageFragment}links{link{linkText}}}}}fragment TempoCommonImageFragment on TempoCommonImage{src alt assetId uid clickThrough{value}}fragment InlineSearchModuleFragment on TempoWM_GLASSWWWInlineSearchConfigs{headingText placeholderText}fragment MarqueeDisplayAdConfigsFragment on TempoWM_GLASSWWWMarqueeDisplayAdConfigs{_rawConfigs ad{...DisplayAdFragment}}fragment DisplayAdFragment on Ad{...AdFragment adContent{type data{__typename...AdDataDisplayAdFragment}}}fragment AdFragment on Ad{status moduleType platform pageId pageType storeId stateCode zipCode pageContext moduleConfigs adsContext adRequestComposite}fragment AdDataDisplayAdFragment on AdData{...on DisplayAd{json status}}fragment SkylineDisplayAdConfigsFragment on TempoWM_GLASSWWWSkylineDisplayAdConfigs{_rawConfigs ad{...SkylineDisplayAdFragment}}fragment SkylineDisplayAdFragment on Ad{...SkylineAdFragment adContent{type data{__typename...SkylineAdDataDisplayAdFragment}}}fragment SkylineAdFragment on Ad{status moduleType platform pageId pageType storeId stateCode zipCode pageContext moduleConfigs adsContext adRequestComposite}fragment SkylineAdDataDisplayAdFragment on AdData{...on DisplayAd{json status}}fragment HorizontalChipModuleConfigsFragment on TempoWM_GLASSWWWHorizontalChipModuleConfigs{chipModuleSource:moduleSource chipModule{title url{linkText title clickThrough{type value}}}chipModuleWithImages{title url{linkText title clickThrough{type value}}image{alt clickThrough{type value}height src title width}}}",
   "variables":{
      "id":"",
      "affinityOverride":"default",
      "dealsId":"",
      "query":"",
      "page":1,
      "prg":"desktop",
      "catId":"3920",
      "facet":"",
      "sort":"best_seller",
      "rawFacet":"",
      "seoPath":"",
      "ps":40,
      "ptss":"",
      "trsp":"",
      "beShelfId":"",
      "recall_set":"",
      "module_search":"",
      "min_price":"",
      "max_price":"",
      "storeSlotBooked":"",
      "additionalQueryParams":None,
      "fitmentFieldParams":None,
      "fitmentSearchParams":{
         "id":"",
         "affinityOverride":"default",
         "dealsId":"",
         "query":"",
         "page":1,
         "prg":"desktop",
         "catId":"3920",
         "facet":"",
         "sort":"best_seller",
         "rawFacet":"",
         "seoPath":"",
         "ps":40,
         "ptss":"",
         "trsp":"",
         "beShelfId":"",
         "recall_set":"",
         "module_search":"",
         "min_price":"",
         "max_price":"",
         "storeSlotBooked":"",
         "additionalQueryParams":None,
         "cat_id":"3920",
         "_be_shelf_id":""
      },
      "fetchMarquee":True,
      "fetchSkyline":True,
      "fetchSbaTop":False
   }

}
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36',
    'content-type':'application/json',
    'wm_mp': 'true',
    'wm_page_url': 'https://www.walmart.com/browse/books/3920?sort=best_seller&affinityOverride=default',
    'wm_qos.correlation_id': 'FWpup9KEKUrLFOY68gppqfprABL16K6qE76g',
    'x-apollo-operation-name': 'Browse',
    'x-enable-server-timing': '1',
    'x-latency-trace': '1',
    'x-o-ccm': 'server',
    'x-o-correlation-id': 'FWpup9KEKUrLFOY68gppqfprABL16K6qE76g',
    'x-o-gql-query': 'query Browse',
    'x-o-market': 'us',
    'x-o-platform': 'rweb',
    'x-o-platform-version': 'main-176-e8acb5',
    'x-o-segment': 'oaoh'
    }


params= {
    "affinityOverride": "default",
    "page": "1",
    "prg": "desktop",
    "catId": "3920",
    "sort": "best_seller",
    "ps": "40",
    "fetchMarquee": "true",
    "fetchSkyline": "true",
    "fetchSbaTop": "false"}





for i in range(1,25,1):
    params['maxPage']=i
    api_url='https://www.walmart.com/orchestra/home/graphql/browse'
    resp = requests.post(api_url, data=json.dumps(data), headers=headers,params=params)
    r=resp.json()
    print(r)
    # items = r['data']['search']['searchResult']['itemStacks'][0]['itemsV2']
    # for item in items:
    #     price = item['priceInfo']['currentPrice']['price']
    #     print(price)

Output:

'https://i5.walmartimages.com/asr/deaaef8d-ca7f-4fc1-b556-1209c4c9000c.c6df15d971f1b9cddde27f80f17ae80b.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Sonic-The-Hedgehog-Coloring-Book-For-Kids-Girls-Adults-Toddlers-Kids-ages-2-8-Unofficial-25-high-quality-illustrations-Pages-8-5-x-11-Paperback-9781677024223/614029660?athbdg=L1600', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': [{'key': 'BESTSELLER', 'text': 'Best seller', 'type': 'LABEL', 'id': 'L1600'}], 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 'REGULAR', 'averageRating': 3.9, 'numberOfReviews': 8, 'esrb': None, 'mediaRating': None, 'salesUnitType': 'EACH', 'sellerId': 'F55CDC31AB754BB68FE0B39041159D63', 'sellerName': 'Walmart.com', 'hasSellerBadge': None, 'availabilityStatusV2': {'display': 'In stock', 
'value': 'IN_STOCK'}, 'productLocation': None, 'badge': [{'type': 'bestSeller', 'dynamicDisplayName': None}], 'fulfillmentSpeed': None, 'offerId': '27E4BA43A8704A1DABF0B37693611D16', 'preOrder': {'isPreOrder': False, 'preOrderMessage': None, 'preOrderStreetDateMessage': None}, 'priceInfo': {'priceRange': None, 'currentPrice': {'price': 6.99, 
'priceString': '$6.99'}, 'wasPrice': None, 'unitPrice': None, 'listPrice': None, 'shipPrice': None, 'subscriptionPrice': None, 'priceDisplayCodes': {'priceDisplayCondition': None, 'finalCostByWeight': None}}, 'variantCriteria': [], 'fulfillmentBadge': None, 
'fulfillmentTitle': 'title_shipToHome_not_available', 'fulfillmentType': 'FC', 'manufacturerName': None, 'showAtc': True, 'sponsoredProduct': None, 'showOptions': False}, {'__typename': 'Product', 'id': '16FA2JT4ZT52', 'usItemId': '491355610', 'fitmentLabel': None, 'name': 'Bible Word Search Books: Word Search Bible Puzzle Book - Extra Large 
Print: Bible Word Search Large Print Puzzles for Seniors and Adults - Beginners Edition (Large Print) (Paperback)', 'checkStoreAvailabilityATC': False, 'seeShippingEligibility': False, 'brand': None, 'type': 'REGULAR', 'shortDescription': '<li>Format:Paperback</li><li>Publication Date: 2019-11-22</li>', 'imageInfo': {'thumbnailUrl': 'https://i5.walmartimages.com/asr/46269778-a1bc-4a7a-aff2-75a825e35cf9.62e71231ecd42f6cda6d3701a3281b53.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Bible-Word-Search-Books-Puzzle-Book-Extra-Large-Print-Print-Puzzles-Seniors-Adults-Beginners-Edition-Large-Print-Paperback-9781710478792/491355610', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': None, 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 'REGULAR', 'averageRating': 4.7, 'numberOfReviews': 13, 'esrb': None, 'mediaRating': None, 'salesUnitType': 'EACH', 'sellerId': 'F55CDC31AB754BB68FE0B39041159D63', 'sellerName': 'Walmart.com', 'hasSellerBadge': None, 'availabilityStatusV2': {'display': 'In stock', 'value': 'IN_STOCK'}, 'productLocation': None, 'badge': None, 'fulfillmentSpeed': None, 'offerId': '0E3D37D69AE14435A4E83D3AE2789B7F', 'preOrder': {'isPreOrder': False, 'preOrderMessage': None, 'preOrderStreetDateMessage': None}, 'priceInfo': {'priceRange': None, 'currentPrice': {'price': 6.99, 'priceString': '$6.99'}, 'wasPrice': None, 'unitPrice': None, 'listPrice': None, 'shipPrice': None, 'subscriptionPrice': None, 'priceDisplayCodes': {'priceDisplayCondition': None, 'finalCostByWeight': None}}, 'variantCriteria': [], 'fulfillmentBadge': None, 'fulfillmentTitle': 'title_shipToHome_not_available', 'fulfillmentType': 'FC', 'manufacturerName': None, 'showAtc': True, 'sponsoredProduct': None, 'showOptions': False}, {'__typename': 'Product', 'id': '72BDRK2VT8QQ', 'usItemId': '599380007', 'fitmentLabel': None, 'name': 'Trace Letters and Numbers Workbook: Learn How to Write Alphabet Upper and Lower Case and Numbers (Series #2) (Paperback)', 'checkStoreAvailabilityATC': False, 'seeShippingEligibility': False, 'brand': None, 'type': 'REGULAR', 'shortDescription': '9781794540767', 'imageInfo': {'thumbnailUrl': 'https://i5.walmartimages.com/asr/535dff68-7946-4e20-899b-20d05015b05a_1.22a1f229111edd3725505c0db3fe1371.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Trace-Letters-and-Numbers-Workbook-Learn-How-to-Write-Alphabet-Upper-and-Lower-Case-and-Numbers-Series-2-Paperback/599380007?athbdg=L1600', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': [{'key': 'BESTSELLER', 'text': 'Best seller', 'type': 'LABEL', 'id': 'L1600'}], 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 'REGULAR', 'averageRating': 4.7, 'numberOfReviews': 37, 'esrb': None, 'mediaRating': 
None, 'salesUnitType': 'EACH', 'sellerId': 'F55CDC31AB754BB68FE0B39041159D63', 'sellerName': 'Walmart.com', 'hasSellerBadge': None, 'availabilityStatusV2': {'display': 'In 
stock', 'value': 'IN_STOCK'}, 'productLocation': None, 'badge': [{'type': 'bestSeller', 'dynamicDisplayName': None}], 'fulfillmentSpeed': None, 'offerId': 'B701DAA6361D4A97A599815F29FA450D', 'preOrder': {'isPreOrder': False, 'preOrderMessage': None, 'preOrderStreetDateMessage': None}, 'priceInfo': {'priceRange': None, 'currentPrice': {'price': 6.95, 'priceString': '$6.95'}, 'wasPrice': None, 'unitPrice': None, 'listPrice': None, 'shipPrice': None, 'subscriptionPrice': None, 'priceDisplayCodes': {'priceDisplayCondition': None, 'finalCostByWeight': None}}, 'variantCriteria': [], 'fulfillmentBadge': None, 'fulfillmentTitle': 'title_shipToHome_not_available', 'fulfillmentType': 'FC', 'manufacturerName': None, 'showAtc': True, 'sponsoredProduct': None, 'showOptions': False}, {'__typename': 'Product', 'id': '72DZILK2NY05', 'usItemId': '817841366', 'fitmentLabel': None, 'name': 'Toddler Coloring Book for Kids Age 1-3 : aby Activity Book Boys or Girls, Preschool coloring for Their Fun Early Learning of First Easy Number Shape and Color (Paperback)', 'checkStoreAvailabilityATC': False, 'seeShippingEligibility': False, 'brand': None, 'type': 'REGULAR', 'shortDescription': '<li>Format:Paperback</li><li>Publication Date: 2019-08-02</li>', 'imageInfo': {'thumbnailUrl': 'https://i5.walmartimages.com/asr/d2f8e8be-7fa1-4a25-a80c-e1741c6b2f6f.6392f3a67fccf0b396e1fb6ee2848b4b.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Toddler-Coloring-Book-Kids-Age-1-3-aby-Activity-Boys-Girls-Preschool-coloring-Their-Fun-Early-Learning-First-Easy-Number-Shape-Color-Paperback-9781086986501/817841366?athbdg=L1600', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': [{'key': 'BESTSELLER', 'text': 'Best seller', 'type': 'LABEL', 'id': 'L1600'}], 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 'REGULAR', 'averageRating': 5, 'numberOfReviews': 2, 'esrb': None, 'mediaRating': None, 'salesUnitType': 'EACH', 'sellerId': 'F55CDC31AB754BB68FE0B39041159D63', 'sellerName': 'Walmart.com', 'hasSellerBadge': None, 'availabilityStatusV2': {'display': 'In stock', 'value': 'IN_STOCK'}, 
'productLocation': None, 'badge': [{'type': 'bestSeller', 'dynamicDisplayName': None}], 'fulfillmentSpeed': None, 'offerId': 'D3D88022487D4BF19D540BED3742A75D', 'preOrder': {'isPreOrder': False, 'preOrderMessage': None, 'preOrderStreetDateMessage': None}, 'priceInfo': {'priceRange': None, 'currentPrice': {'price': 6.95, 'priceString': '$6.95'}, 'wasPrice': None, 'unitPrice': None, 'listPrice': None, 'shipPrice': None, 'subscriptionPrice': None, 'priceDisplayCodes': {'priceDisplayCondition': None, 'finalCostByWeight': None}}, 'variantCriteria': [], 'fulfillmentBadge': None, 'fulfillmentTitle': 'title_shipToHome_not_available', 'fulfillmentType': 'FC', 'manufacturerName': None, 'showAtc': True, 'sponsoredProduct': None, 'showOptions': False}, {'__typename': 'Product', 'id': '46CGMFA2PY1Y', 'usItemId': '56172624', 'fitmentLabel': None, 'name': 'Crystals for Beginners : The Guide to Get Started with the Healing Power of Crystals (Paperback)', 'checkStoreAvailabilityATC': False, 'seeShippingEligibility': False, 'brand': None, 'type': 'VARIANT', 'shortDescription': '<li>Format:Paperback</li><li>Publication 
Date: 2017-10-17</li>', 'imageInfo': {'thumbnailUrl': 'https://i5.walmartimages.com/asr/d2954574-c30c-48af-8297-900867a2458e_1.03867d2efc65af18a6fdbff418a68afa.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Crystals-for-Beginners-The-Guide-to-Get-Started-with-the-Healing-Power-of-Crystals-Paperback-9781623159917/56172624?athbdg=L1600', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': 
[{'key': 'BESTSELLER', 'text': 'Best seller', 'type': 'LABEL', 'id': 'L1600'}], 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 'VARIANT', 'averageRating': 5, 'numberOfReviews': 13, 'esrb': None, 'mediaRating': None, 'salesUnitType': 'EACH', 'sellerId': 'F55CDC31AB754BB68FE0B39041159D63', 'sellerName': 'Walmart.com', 'hasSellerBadge': None, 'availabilityStatusV2': {'display': 'In stock', 'value': 'IN_STOCK'}, 'productLocation': None, 'badge': [{'type': 'bestSeller', 'dynamicDisplayName': None}], 'fulfillmentSpeed': None, 'offerId': '5F59F4B1DE6945728E7F2EC9A3005472', 'preOrder': {'isPreOrder': False, 'preOrderMessage': None, 'preOrderStreetDateMessage': None}, 'priceInfo': {'priceRange': None, 'currentPrice': {'price': 8.99, 'priceString': '$8.99'}, 'wasPrice': None, 'unitPrice': None, 'listPrice': {'price': 14.99, 'priceString': '$14.99'}, 'shipPrice': None, 'subscriptionPrice': None, 'priceDisplayCodes': {'priceDisplayCondition': None, 'finalCostByWeight': None}}, 'variantCriteria': [], 'fulfillmentBadge': None, 'fulfillmentTitle': 'title_shipToHome_not_available', 'fulfillmentType': 'FC', 'manufacturerName': None, 'showAtc': True, 'sponsoredProduct': None, 'showOptions': False}, {'__typename': 'Product', 'id': '7FF8DA7PEPDT', 'usItemId': '136868031', 'fitmentLabel': None, 'name': 'Hack Learning: Hacking School Discipline : 9 Ways to Create a Culture of Empathy and Responsibility Using Restorative Justice (Series #22) (Paperback)', 'checkStoreAvailabilityATC': False, 'seeShippingEligibility': False, 'brand': None, 'type': 'REGULAR', 'shortDescription': '<li>Format:Paperback</li><li>Publication Date: 2019-03-12</li>', 'imageInfo': {'thumbnailUrl': 'https://i5.walmartimages.com/asr/4c639aa7-2580-4782-84ce-33a428cae000.571d547af78d39ffc35bdd28f988023f.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff'}, 'canonicalUrl': '/ip/Hack-Learning-Hacking-School-Discipline-9-Ways-Create-Culture-Empathy-Responsibility-Using-Restorative-Justice-Series-22-Paperback-9781948212137/136868031?athbdg=L1600', 'externalInfo': None, 'category': {'path': None}, 'badges': {'flags': [{'key': 'BESTSELLER', 'text': 'Best seller', 'type': 'LABEL', 'id': 'L1600'}], 'tags': [{'key': 'THREE_PLUS_DAY_SHIPPING', 'text': '3+ day shipping', 'type': 'LABEL'}, {'key': 'SAVE_WITH_W_PLUS', 'text': 'Save with', 'type': 'ICON'}]}, 'classType': 

Output: Extracted price as example:

22.99
1.22
6.99
9.95
5.95
9.81
5.99
13.17
4.52
6.99
4.99
7.99
6.79
5.99
6.5
6.95
6.99
5.99
4.99
5
4.99
11.93
5.99
4.99
6.99
6.99
6.95
6.95
8.99
14.81
5.13
7.29
3.95
5.99
5.5
5.99
16.88
6.99
6.99
1.99
22.99
1.22
6.99
9.95
5.95
9.81
5.99
13.17
4.52
6.99
4.99
7.99
6.7

This works for me, is your element maybe no textarea or prevents new lines with a custom onChange handler in e.g. React?

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch()
        page = browser.new_page()
        page.set_content("""<textarea id="foobar"></textarea>""")
        page.fill("#foobar", "foo\nbar")
        page.screenshot(path="example.png")
        browser.close()

The async context manager doesn't know anything about the asynchronous generator. Nothing in main knows about the asynchronous generator after you break, in fact. You've given yourself no way to wait for the generator's finalization.

If you want to wait for the generator to close, you need to handle closure explicitly:

async def main():
    async with async_context():
        gen = async_gen()
        try:
            async for _ in gen:
                break
        finally:
            await gen.aclose()

In Python 3.10, you'll be able to use contextlib.aclosing instead of the try/finally:

async def main():
    async with async_context():
        gen = async_gen()
        async with contextlib.aclosing(gen):
            async for _ in gen:
                break

The accepted answer does not work with the newer versions of Playwright. (Thanks @576i for pointing this out)

Here is the Python code that works with the newer versions (tested with version 1.5):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('http://whatsmyuseragent.org/')
    ua = page.query_selector(".user-agent");
    print(ua.inner_html())
    browser.close()

To get only the text, use the inner_text() function.

print(ua.inner_text())

There are many things you can improve there.

• First, you can read the text area value using page.locator('#startText').value()
• Then, you click on the check button. Don't wait using wait_for_load_state, because the page is not navigating.
• After the check action there is an SPAN element inside the "#inside" div. You can do page.locator('#inside SPAN').inner_html(). The inner_html function will wait for the element for you.

.NET syntax - this worked for me

await page.Locator(":nth-match(:text('your text'), 2)").ClickAsync();

If I understood correct, yes you can.

The scraper will close and you want to yield the results then instead of yielding after every iteration.

in your pipelines.py you need something like this

    def close_spider(self, spider):
        yield {"data": self.data}

make sure you defined the item_pipeline in your settings to make it work.

More information: https://docs.scrapy.org/en/latest/topics/item-pipeline.html