Selfhosted search speeds slower than public instance #774

New issue

Open

opened 2026-03-13 00:38:13 +00:00 by derrickb · 2 comments

derrickb commented

2026-03-13 00:38:13 +00:00

(Migrated from github.com)

This very well may not be an "issue" per se but thought it would be helpful to have this documented here.

I noticed that when using the public instance of WebStreamr, it returns results lightning fast. My selfhosted instance takes on average 700ms-1s longer to return the same results.

In this screenshot from aiostreams stats, the top entry is the public instance, bottom is my selfhosted one.

Any tips or ideas on how to optimize search times? Does the public instance have a more persistent cache it can check quickly?

This very well may not be an "issue" per se but thought it would be helpful to have this documented here. I noticed that when using the public instance of WebStreamr, it returns results lightning fast. My selfhosted instance takes on average 700ms-1s longer to return the same results. In this screenshot from aiostreams stats, the top entry is the public instance, bottom is my selfhosted one. <img width="387" height="208" alt="Image" src="https://github.com/user-attachments/assets/b3e3e8af-487a-4c84-b69c-39074214509a" /> Any tips or ideas on how to optimize search times? Does the public instance have a more persistent cache it can check quickly?

webstreamr commented

2026-03-13 08:49:16 +00:00

(Migrated from github.com)

this case is still fast for web scraper IMO :D but you're right, popular content is heavily cached on the public instance and that makes it almost return instantly.

I'm thinking about a cache pulling feature from the public instance. let's use this issue for this maybe, I need to do a bit of research and testing to figure out if it's even feasible if the cache is big. but in the best case it might be possible to either regularly pull the full cache or only the results needed for the current search.

this case is still fast for web scraper IMO :D but you're right, popular content is heavily cached on the public instance and that makes it almost return instantly. I'm thinking about a cache pulling feature from the public instance. let's use this issue for this maybe, I need to do a bit of research and testing to figure out if it's even feasible if the cache is big. but in the best case it might be possible to either regularly pull the full cache or only the results needed for the current search.

derrickb commented

2026-03-13 10:54:28 +00:00

(Migrated from github.com)

Yeah it's still very fast! Always get results in 500ms-1s. Just looking to optimize if possible. That makes sense, I assumed the public instance had much more cached than my local one.

Cache pulling feature would be awesome. Let me know if I can do any testing with my instance. Happy to help.

Yeah it's still very fast! Always get results in 500ms-1s. Just looking to optimize if possible. That makes sense, I assumed the public instance had much more cached than my local one. Cache pulling feature would be awesome. Let me know if I can do any testing with my instance. Happy to help.