I'm using a htmlunit (version 3) to scrapping a webpage.
HtmlPage p = webClient.getPage(URL);
HtmlElement b = p.getFirstByXPath("xpath selector1");
HtmlPage a = b.click();
a.executeJavaScript("window.scrollTo(0, 6000);");
webClient.waitForBackgroundJavaScript(2000);
My requirement is when open a page I have to do a click on a filtering button. The data is updated but the page is lazy loading, so I need to simulate scrolling to the bottom of the page to get more items. When a scroll is done a background request is executed by the server and retrieve more data.
I have a WebConnectionWrapper where catch the background request/response.
But an 400 Bad Request for URL error appear when I add this line on my code because I need to wait for a javascript scroll execution to listening the request that response with new data.
webClient.waitForBackgroundJavaScript(2000);
And without this line the request is done without any problem.
What could be the problem when add a waitForBackgroundJavaScript() ?
I'm using a htmlunit (version 3) to scrapping a webpage.
My requirement is when open a page I have to do a click on a filtering button. The data is updated but the page is lazy loading, so I need to simulate scrolling to the bottom of the page to get more items. When a scroll is done a background request is executed by the server and retrieve more data.
I have a
WebConnectionWrapperwhere catch the background request/response.But an
400 Bad Request for URLerror appear when I add this line on my code because I need to wait for a javascript scroll execution to listening the request that response with new data.webClient.waitForBackgroundJavaScript(2000);And without this line the request is done without any problem.
What could be the problem when add a
waitForBackgroundJavaScript()?