WebStreamr
|
5ac359f1eb
|
chore(fetcher): retry unexpected exceptions
|
2025-10-02 15:07:57 +00:00 |
|
WebStreamr
|
5466472b21
|
chore(fetcher): add generic fetch also returning response url
|
2025-09-30 14:45:47 +00:00 |
|
WebStreamr
|
fcced2e122
|
chore(fetcher): add referer to fetch logging
|
2025-09-28 20:15:15 +00:00 |
|
WebStreamr
|
a96d015a0c
|
refactor(fetcher): introduce json method
|
2025-09-24 18:34:53 +00:00 |
|
WebStreamr
|
4697d150cf
|
chore(fetcher): double queue limit and timeouts
|
2025-09-24 18:04:00 +00:00 |
|
WebStreamr
|
0a0bedacb4
|
chore(fetcher): complete proxy headers
|
2025-09-23 08:50:56 +00:00 |
|
WebStreamr
|
7e771c2644
|
revert: Revert "chore(fetcher): try to remove the proxy headers (again)"
This reverts commit 00bd11495f.
|
2025-09-23 08:32:59 +00:00 |
|
WebStreamr
|
60f44c9603
|
revert: Revert "fix: remove turnstyle detection to check if it causes false-positives"
This reverts commit c10ca29fda.
|
2025-09-16 19:41:59 +00:00 |
|
WebStreamr
|
d90cd8f984
|
fix(fetcher): only retry 503 and 504
|
2025-09-16 14:49:13 +00:00 |
|
WebStreamr
|
c10ca29fda
|
fix: remove turnstyle detection to check if it causes false-positives
|
2025-09-16 13:05:03 +00:00 |
|
WebStreamr
|
ae32376758
|
fix(fetcher): re-use proxy agents
|
2025-09-14 18:18:09 +00:00 |
|
WebStreamr
|
8acbd37df3
|
revert: Revert recent reverts again
This reverts commit 78db179957.
|
2025-09-14 17:38:46 +00:00 |
|
WebStreamr
|
78db179957
|
revert: Revert "chore(fetcher): log more infos about responses"
This reverts commit 731bd33470.
|
2025-09-14 13:37:17 +00:00 |
|
WebStreamr
|
9a279b53a4
|
revert: Revert "feat: allow to disable cache"
This reverts commit 4078f0c404.
|
2025-09-14 13:37:14 +00:00 |
|
WebStreamr
|
4ca87f0ff1
|
revert: Revert "chore(fetcher): log proxy usage, allow to disable proxy via config"
This reverts commit b67e1b85d6.
|
2025-09-14 13:37:08 +00:00 |
|
WebStreamr
|
b67e1b85d6
|
chore(fetcher): log proxy usage, allow to disable proxy via config
|
2025-09-13 18:27:17 +00:00 |
|
WebStreamr
|
4078f0c404
|
feat: allow to disable cache
|
2025-09-13 17:51:12 +00:00 |
|
WebStreamr
|
731bd33470
|
chore(fetcher): log more infos about responses
|
2025-09-13 17:41:28 +00:00 |
|
WebStreamr
|
a729dc85e0
|
chore: add url info to blocked error
|
2025-09-12 18:35:22 +00:00 |
|
WebStreamr
|
d087318286
|
chore(fetcher): cache only 2xx and 404 responses
|
2025-09-12 18:24:19 +00:00 |
|
WebStreamr
|
69da82b139
|
chore(fetcher): bring back error retry
|
2025-09-12 17:53:37 +00:00 |
|
WebStreamr
|
2a47d97d5c
|
chore(fetcher): switch back to undici fetch 🤡
|
2025-09-12 13:20:59 +00:00 |
|
WebStreamr
|
7ee87ceb9e
|
chore(fetcher): remove interceptors and retries
|
2025-09-12 09:50:07 +00:00 |
|
WebStreamr
|
b513a8926b
|
feat: add support for proxy per host using minimatch
|
2025-09-12 08:13:18 +00:00 |
|
WebStreamr
|
2f370d1b9c
|
refactor: create dispatcher per request
|
2025-09-12 07:30:27 +00:00 |
|
WebStreamr
|
87893819b0
|
chore(fetcher): switch back to native fetch 🤡
|
2025-09-11 14:34:26 +00:00 |
|
WebStreamr
|
00bd11495f
|
chore(fetcher): try to remove the proxy headers (again)
|
2025-09-10 11:25:16 +00:00 |
|
renovate[bot]
|
8b5588936e
|
fix(deps): update dependency cacheable to v2 (#336)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: WebStreamr <210764791+webstreamr@users.noreply.github.com>
|
2025-09-10 10:11:52 +00:00 |
|
WebStreamr
|
211babec37
|
chore(fetcher): wait out short rate limits and retry
|
2025-09-05 08:52:36 +00:00 |
|
WebStreamr
|
e846900384
|
fix(fetcher): fix TTL for rate limit cache
|
2025-09-05 07:35:06 +00:00 |
|
WebStreamr
|
756cd01ab9
|
chore: decrease fetcher and extractor memory cache sizes
|
2025-09-03 14:39:27 +00:00 |
|
WebStreamr
|
fd0900e053
|
feat: add stats endpoint with cache stats
|
2025-09-03 13:00:07 +00:00 |
|
WebStreamr
|
83021fb773
|
chore(fetcher): decrease timeout to 5 seconds
|
2025-09-03 10:28:48 +00:00 |
|
WebStreamr
|
dc093f7109
|
refactor: consistently use cacheable for caching
|
2025-09-02 13:33:51 +00:00 |
|
WebStreamr
|
4d05ac859f
|
chore: handle NotFoundError / 404 inside source and cache it also
|
2025-09-01 10:53:59 +00:00 |
|
WebStreamr
|
c47f48c64b
|
chore(fetcher): log retries
|
2025-09-01 08:17:59 +00:00 |
|
WebStreamr
|
0fc7675560
|
chore(fetcher): retry timeouts once
|
2025-08-31 20:12:04 +00:00 |
|
WebStreamr
|
ef613f2742
|
chore: double HTTP cache, quadruple extractor cache
|
2025-08-16 16:40:15 +00:00 |
|
WebStreamr
|
fe5b3af887
|
chore(fetcher): allow to customize minimum cache ttl
|
2025-08-15 08:54:14 +00:00 |
|
WebStreamr
|
2a5284d555
|
refactor(fetcher): sort custom request init props
|
2025-08-15 08:50:38 +00:00 |
|
WebStreamr
|
a12ae7aaa5
|
chore(fetcher): retry 5xx errors up to 3 times
|
2025-08-13 10:49:36 +00:00 |
|
WebStreamr
|
e28de7d4d9
|
revert: "fix(fetcher): remove weird proxy headers in request"
This reverts commit ee09e10e6e.
|
2025-08-12 15:15:03 +00:00 |
|
WebStreamr
|
9a019396c8
|
fix(fetcher): simplify header and ttl determination, fix edge case bug
|
2025-08-11 15:47:04 +00:00 |
|
WebStreamr
|
ee09e10e6e
|
fix(fetcher): remove weird proxy headers in request
looks like this is finally causing weird issues and leading to some
Cloudflare blocks. we might be better of without it, let's try..
|
2025-08-11 12:09:46 +00:00 |
|
WebStreamr
|
0efdfa4a6f
|
fix(fetcher): better detect inline Cloudflare Turnstile
|
2025-08-11 11:40:58 +00:00 |
|
WebStreamr
|
7d700103c4
|
chore(fetcher): decrease default timeout back to 10s
|
2025-08-11 08:38:56 +00:00 |
|
WebStreamr
|
78ced84774
|
chore(fetcher): block a host temporarily after 30 times timing out instead of 10
|
2025-08-08 08:27:40 +00:00 |
|
WebStreamr
|
009bb930b5
|
chore(fetcher): increase default timeout to 15s
in most of the cases we're still very fast anyway
but in some of the small cases let's avoid timeout blocks by waiting
longer
|
2025-08-05 10:01:30 +00:00 |
|
WebStreamr
|
51b9104f07
|
chore: handle cloudflare 451 censor block, show block reasons
|
2025-07-21 19:35:52 +00:00 |
|
WebStreamr
|
674eb7ca6d
|
fix(fetcher): support Basic Auth
|
2025-07-20 18:44:37 +00:00 |
|