Commit graph

114 commits

Author SHA1 Message Date
WebStreamr
2a47d97d5c
chore(fetcher): switch back to undici fetch 🤡 2025-09-12 13:20:59 +00:00
WebStreamr
7ee87ceb9e
chore(fetcher): remove interceptors and retries 2025-09-12 09:50:07 +00:00
WebStreamr
b513a8926b
feat: add support for proxy per host using minimatch 2025-09-12 08:13:18 +00:00
WebStreamr
2f370d1b9c
refactor: create dispatcher per request 2025-09-12 07:30:27 +00:00
WebStreamr
87893819b0
chore(fetcher): switch back to native fetch 🤡 2025-09-11 14:34:26 +00:00
WebStreamr
00bd11495f
chore(fetcher): try to remove the proxy headers (again) 2025-09-10 11:25:16 +00:00
renovate[bot]
8b5588936e
fix(deps): update dependency cacheable to v2 (#336)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: WebStreamr <210764791+webstreamr@users.noreply.github.com>
2025-09-10 10:11:52 +00:00
WebStreamr
211babec37
chore(fetcher): wait out short rate limits and retry 2025-09-05 08:52:36 +00:00
WebStreamr
e846900384
fix(fetcher): fix TTL for rate limit cache 2025-09-05 07:35:06 +00:00
WebStreamr
756cd01ab9
chore: decrease fetcher and extractor memory cache sizes 2025-09-03 14:39:27 +00:00
WebStreamr
fd0900e053
feat: add stats endpoint with cache stats 2025-09-03 13:00:07 +00:00
WebStreamr
83021fb773
chore(fetcher): decrease timeout to 5 seconds 2025-09-03 10:28:48 +00:00
WebStreamr
dc093f7109
refactor: consistently use cacheable for caching 2025-09-02 13:33:51 +00:00
WebStreamr
4d05ac859f
chore: handle NotFoundError / 404 inside source and cache it also 2025-09-01 10:53:59 +00:00
WebStreamr
c47f48c64b
chore(fetcher): log retries 2025-09-01 08:17:59 +00:00
WebStreamr
0fc7675560
chore(fetcher): retry timeouts once 2025-08-31 20:12:04 +00:00
WebStreamr
ef613f2742
chore: double HTTP cache, quadruple extractor cache 2025-08-16 16:40:15 +00:00
WebStreamr
fe5b3af887
chore(fetcher): allow to customize minimum cache ttl 2025-08-15 08:54:14 +00:00
WebStreamr
2a5284d555
refactor(fetcher): sort custom request init props 2025-08-15 08:50:38 +00:00
WebStreamr
a12ae7aaa5
chore(fetcher): retry 5xx errors up to 3 times 2025-08-13 10:49:36 +00:00
WebStreamr
e28de7d4d9
revert: "fix(fetcher): remove weird proxy headers in request"
This reverts commit ee09e10e6e.
2025-08-12 15:15:03 +00:00
WebStreamr
9a019396c8
fix(fetcher): simplify header and ttl determination, fix edge case bug 2025-08-11 15:47:04 +00:00
WebStreamr
ee09e10e6e
fix(fetcher): remove weird proxy headers in request
looks like this is finally causing weird issues and leading to some
Cloudflare blocks. we might be better of without it, let's try..
2025-08-11 12:09:46 +00:00
WebStreamr
0efdfa4a6f
fix(fetcher): better detect inline Cloudflare Turnstile 2025-08-11 11:40:58 +00:00
WebStreamr
7d700103c4
chore(fetcher): decrease default timeout back to 10s 2025-08-11 08:38:56 +00:00
WebStreamr
78ced84774
chore(fetcher): block a host temporarily after 30 times timing out instead of 10 2025-08-08 08:27:40 +00:00
WebStreamr
009bb930b5
chore(fetcher): increase default timeout to 15s
in most of the cases we're still very fast anyway
but in some of the small cases let's avoid timeout blocks by waiting
longer
2025-08-05 10:01:30 +00:00
WebStreamr
51b9104f07
chore: handle cloudflare 451 censor block, show block reasons 2025-07-21 19:35:52 +00:00
WebStreamr
674eb7ca6d
fix(fetcher): support Basic Auth 2025-07-20 18:44:37 +00:00
WebStreamr
de8e0bdd94
fix: do not use cache and proxy headers for health requests 2025-07-18 09:23:45 +00:00
WebStreamr
8f7039bbb3
feat(source): add PrimeWire 2025-07-15 21:46:54 +00:00
WebStreamr
e03621db0b
fix(fetcher): import correct Headers 2025-07-15 20:08:45 +00:00
WebStreamr
2c210941aa
refactor(fetcher): switch to undici fetch 2025-07-11 16:43:31 +00:00
WebStreamr
5c7b3fb7f2
refactor: more consistent import style 2025-07-02 09:28:10 +00:00
WebStreamr
acaa7221dd
chore(fetcher): use gzip to compress HTTP cache 2025-06-28 18:23:21 +00:00
WebStreamr
f5121bb1fb
chore(fetcher): use LRUCache for HTTP responses 2025-06-28 17:18:10 +00:00
WebStreamr
881fed53a4
chore(fetcher): configure max for all TTLCaches 2025-06-28 17:14:51 +00:00
WebStreamr
aee0fd8373
revert: Revert "fix(fetcher): remove 200 minimum cache"
This reverts commit 80e4f4edc4.
2025-06-27 19:18:56 +00:00
WebStreamr
80e4f4edc4
fix(fetcher): remove 200 minimum cache
causes problems with session-scoped responses which should not be cached
2025-06-25 20:05:31 +00:00
WebStreamr
e895077a7b
chore: handle invalid MediaFlow Proxy password better 2025-06-25 18:16:12 +00:00
WebStreamr
f90a53da68
feat(fetcher): add proxy support 2025-06-21 12:03:34 +00:00
WebStreamr
138e92afc5
chore: make ip in Context optional 2025-06-19 11:54:13 +00:00
WebStreamr
f7042b94ef
chore(fetcher): remember recent timeouts and throw if count is too high
This should avoid issues with longer outages of hosters like we had with
DoodStream now twice recently.
2025-06-17 19:09:22 +00:00
WebStreamr
e1f5192f4b
refactor: avoid leaking fetch internals by introducing TimeoutError 2025-06-17 15:42:47 +00:00
WebStreamr
d5ee89db9b
refactor(fetcher): use Semaphore with timeout for queuing 2025-06-16 09:03:35 +00:00
WebStreamr
2a8e67ca59
refactor(fetcher): introduce constants for defaults 2025-06-16 07:29:34 +00:00
WebStreamr
6cf31f60de
chore(fetcher): increase general timeouts 2025-06-15 11:34:09 +00:00
WebStreamr
6541782541
fix(fetcher): consider request body for cache key 2025-06-14 20:24:02 +00:00
WebStreamr
605e88fef4
chore: implement rate-limit detection 2025-06-14 19:51:00 +02:00
WebStreamr
f11c591dc5
chore: do not pass referer through context 2025-06-14 13:44:32 +02:00