Repository navigation
[bug] fread does not work with https url behind a proxy #1686
Description
Activity
According to curl#1 starting from R 3.2.2 it guess proxy on windows, so curl probably isn't required for https any more. But for older versions of R we would still need curl.
Not sure but maybe addingdownload.methodargument to fread would manage https better,download.filecan usedownload.method="libcurl"(or"curl"), then usingcurl::curldownloadshould not be even needed.@jangorecki I thought of adding the argument for #1668, but wasn't sure how often it would come up... more evidence in favor here
@jangorecki, with
download.fileandmethod = "libcurl", it it explained that we could usehttps_proxyenvironment variable with the form[user:password@]machine[:port]and it is working.As now
httpsis working by default withdownload.file, it could be a good idea to adddownload.fileargs and not depend oncurl. For example, inremotespackage, changes have been made to use this behaviour and work withdownload.filemethods and its proxy config.About this issue specifically, setting correctly the environnement variable seems to work:
Sys.setenv(HTTPS_PROXY=http://[user:password@]proxy:port), I now manage to usefread.I discussed this solution in
curlrepo.Example
Sys.getenv("HTTPS_PROXY") #> [1] "http://myuser:mypassword@proxyIp:proxyPort" url <- "https://d37djvu3ytnwxt.cloudfront.net/asset-v1:MITx+15.071x_3+1T2016+type@asset+block/songs.csv" DT <- data.table::fread(url, verbose = T) #> Input contains no \n. Taking this to be a filename to open #> File opened, filesize is 0.002159 GB. #> Memory mapping ... ok #> Detected eol as \r only (no \n or \r afterwards). An old Mac 9 standard, discontinued in 2002 according to Wikipedia. #> Positioned on line 1 after skip or autostart #> This line is the autostart and not blank so searching up for the last non-blank ... line 1 #> Detecting sep ... ',' #> Detected 39 columns. Longest stretch was from line 1 to line 30 #> Starting data input on line 1 (either column names or first row of data). First 10 characters: year,songt #> All the fields on line 1 are character fields. Treating as the column names. #> Count of eol: 7574 (including 0 at the end) #> Count of sep: 287812 #> nrow = MIN( nsep [287812] / (ncol [39] -1), neol [7574] - endblanks [0] ) = 7574 #> Type codes (point 0): 144441333313333333333333333333333333331 #> Type codes (point 1): 144441333313333333333333333333333333331 #> Type codes (point 2): 144441333313333333333333333333333333331 #> Type codes (point 3): 144441333313333333333333333333333333331 #> Type codes (point 4): 144441333313333333333333333333333333331 #> Type codes (point 5): 144441333313333333333333333333333333331 #> Type codes (point 6): 144441333313333333333333333333333333331 #> Type codes (point 7): 144441333313333333333333333333333333331 #> Type codes (point 8): 144441333313333333333333333333333333331 #> Type codes (point 9): 144441333313333333333333333333333333331 #> Type codes (point 10): 144441333313333333333333333333333333331 #> Type codes: 144441333313333333333333333333333333331 (after applying colClasses and integer64) #> Type codes: 144441333313333333333333333333333333331 (after applying drop or select (if supplied) #> Allocating 39 column slots (39 - 0 dropped) #> Read 7574 rows. Exactly what was estimated and allocated up front #> 0.010s ( 3%) Memory map (rerun may be quicker) #> 0.000s ( 0%) sep and header detection #> 0.005s ( 1%) Count rows (wc -l) #> 0.045s ( 11%) Column type detection (100 rows at 10 points) #> 0.000s ( 0%) Allocation of 7574x39 result (xMB) in RAM #> 0.337s ( 85%) Reading data #> 0.000s ( 0%) Allocation for type bumps (if any), including gc time if triggered #> 0.000s ( 0%) Coercing data already read in type bumps (if any) #> 0.000s ( 0%) Changing na.strings to NA #> 0.397s Total
Hi,
I recently encounter a problem with https url to read a file with
fread. I had the following error#> Error in curl::curl_download(input, tt, mode = "wb", quiet = !showProgress): Timeout was reachedThe problem is I am behind a proxy and
curl_downloaddoes not seem to import current proxy setting on windows for IE. And I do not find a way to configurecurlproxy setting beforefreadHowever, with
download.filefunction, downloading the file with https url works perfectly without configuring anything.Looking at
fread,curl::curl_downloadis used for secure url whereas for non secure,download.fileis used ?Could it be possible that now R handles https url with
download.file, the suggestedcurllibrary is no longer need and would solves this problem?For those who could try to reproduce behin a proxy :
and my
sessionInfo()