Timout reached when downloading data

[dominicroye]

Hi,

Recently, I've been experiencing issues downloading data. I'm using three API keys, and it still can't do it.

[dieghernan]

Just today I was planning to look into it

[dominicroye]

Have you any news? Our MACE app is experiencing issues due to API problems. Sometimes, even aemet_stations() isn't working. I guess it's more of a problem on AEMET's site, although I just tried out directly the OpenData API and it seems to work fine.

[dieghernan]

I couldn’t reproduce the issue, it seems to be transient. It seems to me that it’s in the API provider side. Sorry not bringing better news 😔but I already tried a lot of things in order to overcome this, if you have any suggestions happy to give it a try Have a nice day! El El vie, 13 jun 2025 a las 8:48, Dominic Royé ***@***.***> escribió:

…

*dominicroye* left a comment (rOpenSpain/climaemet#74) <#74 (comment)> Have you any news? Our MACE app is experiencing issues due to API problems. Sometimes, even aemet_stations() isn't working. I guess it's more of a problem on AEMET's site. — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGDX32KIADQ7GJV4MTZDGNT3DJX2NAVCNFSM6AAAAAB6RVZOC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRZGI3DGMJUGA> . You are receiving this because you commented.Message ID: ***@***.***>

[dominicroye]

At this moment I have the following issue:

> stations <- aemet_stations() 
Error en dms2decdegrees(stations$longitud): Input can't be missing
Además: Aviso:
In aemet_api_call(newapientry, verbose, data_call = TRUE, apikey = apikey) :
  HTTP 200 OK: Se han encontrado 947 estaciones

Station data downloading seems to work now.

[dieghernan]

I think this may be another issue, the response of the API is

>  response
<httr2_response>
GET https://opendata.aemet.es/opendata/sh/1d633062
Status: 200 OK
Content-Type: text/plain
Body: In memory (172371 bytes)

The issue here is that Content-Type used to be text/json (and indeed the result is formatted as a JSON object) but because the header provides wrong information the httr2 package does not parse the object as it should be.

[dominicroye]

There is something more ongoing, see:

> aemet_daily_clim(c("0076", "0229I", "0201D"), 
+                  start = "2025-06-01", end = today())
# A tibble: 0 × 0
Avisos:
1: In aemet_api_call(newapientry, verbose, data_call = TRUE, apikey = apikey) :
  HTTP 200 OK: Se han encontrado 10 climatologias diarias
2: In aemet_api_call(apidest, verbose, apikey = apikey) :
  HTTP 404: No hay datos que satisfagan esos criterios
3: In aemet_api_call(newapientry, verbose, data_call = TRUE, apikey = apikey) :
  HTTP 200 OK: Se han encontrado 10 climatologias diarias

If I make the same request directly, it returns data, but maybe they've made changes to the JSON?

[dominicroye]

All functions seem to be affected:

> aemet_daily_clim(c("9434", "3195"))
# A tibble: 0 × 0
Avisos:
1: In aemet_api_call(newapientry, verbose, data_call = TRUE, apikey = apikey) :
  HTTP 200 OK: Se han encontrado 4 climatologias diarias
2: In aemet_api_call(newapientry, verbose, data_call = TRUE, apikey = apikey) :
  HTTP 200 OK: Se han encontrado 4 climatologias diarias

[jaimegutierrezh]

Hello everyone and thank you very much for your work. I am having this very same issue in my scripts. Like Dominic, I also tried to get the data via API request and it worked.

Do you think you’d find any solution soon?

[dominicroye]

I am experiencing issues again, but I'm thinking, what if the download saves data and restarts from this point? I am not sure if it is possible.

[Aimonius]

Same problem here:

HTTP 429: Límite de peticiones o caudal por minuto excedido para este usuario. Espere al siguiente minuto. Retrying...
HTTP 429: Límite de peticiones o caudal por minuto excedido para este usuario. Espere al siguiente minuto. Retrying...
Error in `httr2::req_perform()`:
! Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [opendata.aemet.es]:
Operation timed out after 20002 milliseconds with 0 bytes received
Run `rlang::last_trace()` to see where the error occurred.
🕓  AEMET API (41/60) | ■■■■■■■■■■■■■■■■■■■■■□□□□□□□□□□   68%  | ETA: 1m [1m 51.3s]

[jaimegutierrezh]

I am also having the same problem. Did you find any solution?

[dominicroye]

One solution is a workaround with a new function that saves already downloaded data and restarts where it left off.

library(climaemet)
library(purrr)
library(dplyr)

# Set your AEMET API key
aemet_api_key("YOUR_API_KEY")

# Robust download function with retries, resume, and temporary file saving
robust_climate_download <- function(
    stations,
    start_date,
    end_date,
    interval_days = 15,
    max_retries = 3,
    output_file = "climate_data_final.rds",
    temp_dir = "temp_chunks"
) {
  # Validaciones
  if (interval_days > 15) {
    warning("El intervalo máximo permitido por la API es de 15 días. Usando 15 días.")
    interval_days <- 15
  }
  stopifnot(is.character(stations), length(stations) > 0)
  
  dir.create(temp_dir, showWarnings = FALSE)
  start_date <- as.Date(start_date)
  end_date <- as.Date(end_date)
  date_seq <- seq(start_date, end_date, by = interval_days)
  date_seq <- unique(c(date_seq, end_date))  # Asegura que el último día esté incluido
  
  failed_stations <- c()
  error_log <- list()
  
  normalize_chunk <- function(df) {
    df %>%
      mutate(across(everything(), ~as.character(.x))) %>%
      mutate(across(where(is.character), ~trimws(.)))
  }
  
  download_chunks <- function(station) {
    for (i in seq_along(date_seq)[-length(date_seq)]) {
      chunk_key <- paste0(station, "_", date_seq[i])
      chunk_file <- file.path(temp_dir, paste0("chunk_", chunk_key, ".rds"))
      
      if (file.exists(chunk_file)) {
        message("Ya descargado: ", chunk_key)
        next
      }
      
      attempt <- 1
      success <- FALSE
      while (attempt <= max_retries && !success) {
        tryCatch({
          message("Descargando: ", station, " de ", date_seq[i], " a ", date_seq[i + 1] - 1)
          data <- aemet_daily_clim(
            station = station,
            start = date_seq[i],
            end = date_seq[i + 1] - 1,
            verbose = FALSE
          )
          if (nrow(data) > 0) {
            saveRDS(data, chunk_file)
            success <- TRUE
          } else {
            message("Sin datos intento ", attempt)
            attempt <- attempt + 1
            Sys.sleep(2)
          }
        }, error = function(e) {
          message("Error en estación ", station, " intento ", attempt, ": ", e$message)
          error_log[[station]] <- c(error_log[[station]], paste("Intento", attempt, ":", e$message))
          attempt <- attempt + 1
          Sys.sleep(2)
        })
      }
      
      if (!success) {
        return(FALSE)
      }
    }
    return(TRUE)
  }
  
  # Primer intento
  for (station in stations) {
    message("Estación: ", station)
    success <- download_chunks(station)
    if (!success) {
      failed_stations <- c(failed_stations, station)
    }
  }
  
  # Segundo intento para estaciones fallidas
  if (length(failed_stations) > 0) {
    message("Reintentando estaciones fallidas...")
    for (station in unique(failed_stations)) {
      message("Reintento estación: ", station)
      success <- download_chunks(station)
      if (success) {
        failed_stations <- setdiff(failed_stations, station)
      }
    }
  }
  
  # Combinar todos los chunks descargados
  all_chunks <- list.files(temp_dir, full.names = TRUE)
  combined_data <- map_dfr(all_chunks, ~normalize_chunk(readRDS(.)))
  saveRDS(combined_data, output_file)
  
  # Mensaje final
  message("Resumen:")
  message("Estaciones exitosas: ", length(stations) - length(failed_stations))
  message("Estaciones fallidas: ", length(failed_stations))
  message("Chunks descargados: ", length(all_chunks))
  
  if (length(failed_stations) > 0) {
    message("Estaciones fallidas definitivas: ", paste(unique(failed_stations), collapse = ", "))
  }
  
  return(combined_data)
}