I am trying to download the dataset from this website but the website only allows me to download the folders one at a time by clicking on the download all files.
Below is my login ID and password that I try with wget but it still does not work.
wget --user kong --password helpmedownloadpls 1 Answer
--user and --password parameters to wget will not work here.
They are intended to use only with HTTP Basic authentication method which almost no website uses today.
This website (and hundreds of others) uses authentication based on cookies. When you fill in the login and password on the login form, a session cookie is generated and sent to your browser, and the browser sends this cookie in the subsequent requests as a proof that you are authenticated.
You have to write a script that first fills in the login form using wget with --post-data parameter to send the login and password and --save-cookies and --keep-session-cookies parameters to store the cookies you received upon successful login, and then downloads files using wget with --load-cookies parameter to use previously stored cookies.
