SoFunction
Updated on 2025-03-02

Solution to the problem that the web page cannot be moved when using Selenium in Python

Preface

If you encounter the problem of the <body> of the webpage when using Selenium, this is usually because the page's scrolling mechanism (for example, it may be using a container with a fixed height or a custom scroll bar) cannot be scrolled through simple JavaScript. This problem can be solved by the following methods.

1. Find scrollable elements

See if there are specific scrollable elements on the page instead of scrolling the entire <body> directly. For example, some web pages will use <div> or other containers to display content, and this element may have the overflow: auto or overflow: scroll attributes. You can try to locate that container and scroll it.

For example:

scrollable_div = driver.find_element(By.CSS_SELECTOR, '-container')  # Replace with the actual selectordriver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_div)

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_div)It is a way to use JavaScript in Selenium to perform page scrolling operations. Specifically, the function of this line of code is to place the specified element (named in the codescrollable_div) Scroll to the bottom.

  • arguments[0]: This is how to pass parameters in JavaScript. When you calldriver.execute_scriptWhen  , the second parameter passed (i.e.scrollable_div) Will act asarguments[0]Passed into JavaScript code.
  • scrollTop: This is an attribute of the DOM element, indicating the number of pixels currently scrolling vertically. By setting this value, you can control the scroll position of the element.
  • scrollHeight: This is an attribute of the DOM element that represents the total height of the element's content (including content that is not visible due to overflow).

The problem to be solved is scrolling to the bottom of a container element, not the entire page. Assume there is one on the pagedivContains a lot of content, and thisdivHave your own scrollbar. By turning thisdivofscrollTopSet as itscrollHeight, this can be implementeddivThe content of  scrolls to the bottom.

2. Scroll with JavaScript

in the case of<body>It is impossible to scroll, you can use JavaScript to try different scrolling methods, such as directly adjustingscrollTopValue. For example, move a certain pixel downward:

driver.execute_script("(0, 500);")  # Scroll down 500 pixels

3. Check page-load status

Make sure that the page is fully loaded before scrolling. Use explicit wait to confirm the status of the page. For example, useWebDriverWaitShould wait for the loading of an element:

from  import WebDriverWait
from  import expected_conditions as EC
 
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'your-element-class')))  # Replace with the actual class name

4. Looping

If you need to scroll the entire page, you can use a loop to continuously check the changes in scrolling height until you reach the bottom of the page. Here is a common way to deal with infinite scrolling lists:

last_height = driver.execute_script("return ")
 
while True:
    driver.execute_script("(0, );")
    (2)  # Wait for new content to load 
    new_height = driver.execute_script("return ")
    if new_height == last_height:
        break  # Arrive at the bottom and stop scrolling    last_height = new_height

5. Use Actions class to scroll

Using SeleniumActionChainsIt can simulate key presses or mouse events to perform more complex user interactions, such as pressing the "Down Arrow" key:

from .action_chains import ActionChains
 
actions = ActionChains(driver)
for _ in range(10):  # Example: Press the down arrow 10 times    actions.send_keys(Keys.ARROW_DOWN).perform()
    (0.5)  # Wait for a while，For content loading

Of course, you can also directly simulate the pull-down (i.e. scroll down) operation in the console, and use JavaScript to implement it. Here are the steps to implement the drop-down operation by executing JavaScript code in the console.

JavaScript code operation steps

1. Open the browser console

First, open the web page you want to operate and pressF12Key or right-click on the page and select "Check" to open the developer tool. Then switch to the "Console" tab in the developer tools.

2. Find scrollable elements

Suppose the element you want to scroll is adiv, and its class name is__vuescroll. You need to find this element first.

3. Write JavaScript code to pull down

Enter the following JavaScript code in the console, which will make the specifieddivScroll down:

// Find the div element you want to operatevar scrollableDiv = ('div.__vuescroll.hasVBar');
 
// If this div is foundif (scrollableDiv) {
    // Scroll the div to the bottom     = ;
} else {
    ('No scrollable div element found');
}

Code explanation

  • ('div.__vuescroll.hasVBar'): Find the class name through the CSS selector__vuescrollAnd there ishasVBarClassicdivElements.
  • scrollTop = : Put the element'sscrollTopThe attribute is set to the elementscrollHeight(Total height of element content), which will cause the element to scroll to the bottom.

4. Execute the code

Enter or paste the above code in the console and pressEnterKey. This will make the specifieddivThe element scrolls to the bottom.

Scroll gradually

If you want to scroll step by step instead of one-time to the bottom, you can use the following code:

// Find the div element you want to operatevar scrollableDiv = ('div.__vuescroll.hasVBar');
 
// If this div is foundif (scrollableDiv) {
    // The current scrollTop value    var currentScrollTop = ;
    
    // Add 100 pixels each time and gradually scroll to the bottom    var scrollInterval = setInterval(function() {
        // Add 100 pixels each time        currentScrollTop += 100;
        
        // Set a new scrollTop value         = currentScrollTop;
        
        // If it has already scrolled to the bottom, stop the timer        if (currentScrollTop &gt;= ) {
            clearInterval(scrollInterval);
        }
    }, 100); // Execute every 100 milliseconds} else {
    ('No scrollable div element found');
}

Code explanation

  • setInterval: Perform a scrolling operation every 100 milliseconds.
  • currentScrollTop += 100: Add 100 pixels each time.
  • clearInterval(scrollInterval): When scrolling to the bottom, clear the timer and stop scrolling.

By entering JavaScript code into the browser's console, you can directly simulate scrolling down. As needed, you can scroll to the bottom at once, or gradually increasescrollTopThe value of  to achieve smooth scrolling effect.

The above is the detailed content of the solution to the problem that cannot be broken when Python uses Selenium. For more information about Python's Selenium web pages, please pay attention to my other related articles!