Switching from Playwright to Pydoll, need some help with features

[sadikhan918]

Hi, I recently came across this project and I like the customization depth of the browser (especially in regards to canvas fingerprinting and other fingerprinting techniques)

I have some questions about transitioning my project from Playwright to Pydoll though, as it seems more complex to do some things within Pydoll than in Playwright:

1. Persistent contexts -
   It seems that Pydoll is unable to launch actual persistent contexts, whereas in Playwright it creates a directory to store the browser profile which can be reopened after its creation. Pydoll has the ability to save and reuse the cookies from a session, but this doesn't include the full browser storage and can sometimes lead to different outcomes when comparing Playwright and Pydoll (Ex: Logging into a site with a persistent context in Playwright makes it so you don't have to login again to a site on the second run, Pydoll requires you to "soft sign-in" which just means hitting the login button and then the site automatically refreshes your session based on cookies). This isn't a huge deal but it would be nice to see if it was supported.

2. Network interception
   The docs show that this is definitely supported, but it seems quite difficult to understand when I want to intercept (view-only) some specific network requests. Playwright makes it easy like so:

with page.expect_response('api/someApiEndpoint') as response_info:
      page.click('#someElement')

This makes it so that when the browser clicks some element, it expects a network response and is able to receive the body of the response for other uses. Doing this in Pydoll seems a bit more difficult and I'm not sure if it's possible to do this feature at all (expecting a response when doing some specific action).

I appreciate any help and can provide more info/examples for clarification if needed.

[thalissonvs]

Hello @sadikhan918.

1. Yes, pydoll has support for that, but for now you have to do it kinda manually. I'll open an issue for that:

from pydoll.browser.options import ChromiumOptions
from pydoll.browser.chrome import Chrome

options = ChromiumOptions()
options.add_argument("--user-data-dir=/path/to/persistent/storage")

chrome = Chrome(options)
await chrome.start() # this will launch with the persistent context! 

For network interception, you need to enable the network_events. There's a entire section in the documentation just for that:
https://pydoll.tech/docs/features/network/monitoring/

I'll add this same pattern to pydoll eventually, but for now you can do this:

# you can just copy this context manager!
@asynccontextmanager
async def expect_response(tab, url_pattern: str, timeout: float = 10.0):
    event_fired = asyncio.Event()
    response_storage = {"data": None, "error": None}

    async def response_callback(event_details: ResponseReceivedEvent):
        response_url = event_details['params']['response']['url']
       
        if url_pattern in response_url:
            request_id = event_details['params']['requestId']
            try:
                body = await tab.get_response_body(request_id)
         
                response_storage['data'] = {
                    "url": response_url,
                    "status": event_details['params']['response']['status'],
                    "headers": event_details['params']['response']['headers'],
                    "body_json": body.json() if body.is_json else None,
                    "body_text": body.text,
                }
            except Exception as e:
                response_storage['error'] = f"Error: {e}"
            finally:
                event_fired.set()

    await tab.enable_network_events()
    callback_id = await tab.on(NetworkEvent.RESPONSE_RECEIVED, response_callback)
    try:
        yield response_storage
        await asyncio.wait_for(event_fired.wait(), timeout=timeout)
    except asyncio.TimeoutError:
        response_storage['error'] = f"Timeout{timeout}"
    finally:
        await tab.remove_callback(callback_id)


# now you can use like that:
async with expect_response(tab, 'api/someApiEndpoint', timeout=10) as response:
      await (await tab.find(id='someElement')).click()

Please note that you only need to copy this context manager. I'll add to pydoll soon. Thanks for the feedback!

[sadikhan918]

Thanks for the quick feedback, just wanted to check on these things:

1. Do Playwright user data directories work with Pydoll? I've used the command argument you sent with a valid directory that I used with playwright that already has sessions, but it doesn't seem like they transitioned over.

2. The code to intercept the network requests work, but this line was throwing an error for me:
   body = await tab.get_response_body(request_id)

With the error being that "Tab object does not have a method 'get_response_body'". This was fixed by changing it to "get_network_response_body", but from the documentation it should work with what was originally put. I'm unsure of why it isn't working. I'm on the latest version as well.

[thalissonvs]

I'm sorry, it was a typo. The correct method is get_network_response_body indeed. About the playwright directories, yes IF it's from the same browser

[sadikhan918]

Thanks, this is working for me.

One more question related to network interception:

When playwright looks for a response to intercept, it's able to specify by URL and method like so:

with page.expect_response(lambda req: 'api/someApiEndpoint' in req.url and req.request.method == 'POST') as response_info:
      page.click('#someElement')

This helps when the site sends an OPTION request along with a POST, so that you catch the POST explicitly. I haven't tested around enough with the current code to see if the issue happens with Pydoll, but I wanted to check if that was possible as well. This also highlights the ability to get the request info (including method and POST body) just from the response. I know that the post body can be obtained with get_request_post_data but this doesn't seem to be a function for the Tab object.