SN release: semantic search \ stacker news ~meta

pull down to refresh

5760 sats \ 33 comments \ @k00b 16 Jan 2024 meta

zaps forwarded to @elvismercury (100%)

This is a subtle but (hopefully) significant improvement to search and related posts.

Before: search and related posts relied on term matches After: search ranks on term matches and semantic similarity while related posts relies exclusively on semantic similarity

This should improve both evergreen-ness of content and rabbit-holing. IMO related posts have improved an incredible amount, but search is also much better at finding relevant matches.

For months, @elvismercury hollered about text embeddings, but I didn't know what text embeddings were. This change uses on them by mapping a search query or item to a vector of meaning and looking for items with similar vectors.

view all related items

271 sats \ 1 reply \ @Undisciplined 16 Jan 2024

I've been neglecting "related posts". I'll attempt to change my ways.

70 sats \ 0 replies \ @kr 16 Jan 2024

me too, i’m optimistic these posts can help people discover entirely new parts of stacker news than they would otherwise find by browsing top/recent/hot

200 sats \ 0 replies \ @elvismercury fwd 16 Jan 2024

Whoo! Nice job @k00b, this is going to change everything!

151 sats \ 2 replies \ @WeAreAllSatoshi 16 Jan 2024

Yep I understood some of those words :P

30 sats \ 0 replies \ @siggy47 16 Jan 2024

5 more than me.

0 sats \ 0 replies \ @art 16 Jan 2024

I got "the" and that is all

369 sats \ 8 replies \ @ek 16 Jan 2024

Searching for what are you working on

Before:

https://m.stacker.news/12550

Now:

https://m.stacker.news/12551

Was able to immediately find the recent What are you working on this week? post now

42 sats \ 6 replies \ @OneOneSeven 16 Jan 2024

Nice.

I often use quotation marks when I'm searching for a exact phrase with Google and know the exact wording. Not a coder, is that easy to implement?

99 sats \ 0 replies \ @k00b OP 16 Jan 2024

It should be fairly trivial to add, but we don't support that currently.

21 sats \ 4 replies \ @0fje0 16 Jan 2024

I often use quotation marks when I'm searching

So do I. Implementing this would be awesome.

1138 sats \ 3 replies \ @k00b OP 17 Jan 2024

Done. fyi

215 sats \ 2 replies \ @OneOneSeven 17 Jan 2024

https://m.stacker.news/12775

Oh shiiiiiiit. thank you k00b!

217 sats \ 1 reply \ @0fje0 17 Jan 2024

https://m.stacker.news/12779

Indeed, thanks @k00b!

(Am I doing this right ???)

21 sats \ 0 replies \ @OneOneSeven 17 Jan 2024

LOL

21 sats \ 0 replies \ @BitcoinIsTheFuture 16 Jan 2024

Thanks, I’m a visual learner!

121 sats \ 0 replies \ @OriginalSize 16 Jan 2024

Very cool. I found learning about ML tech is helpful even if not implementing, just to understand how so many things are changing. Looking forward to searching on SN more.

121 sats \ 5 replies \ @SimpleStacker 16 Jan 2024 freebie

This is great! SN search functionality was pretty poor before, so hopefully this makes old posts way more searchable!

Just curious, what language model are you using to get the semantic embeddings?

499 sats \ 4 replies \ @k00b OP 16 Jan 2024

this sentence transformer

10 sats \ 3 replies \ @shyfire 16 Jan 2024

deleted by author

0 sats \ 2 replies \ @k00b OP 16 Jan 2024

We like using OpenSearch for search stuff so we store them there. I’m not sure if Postgres supports k-nn in arbitrary dimensions.

10k sats \ 1 reply \ @shyfire 16 Jan 2024

deleted by author

0 sats \ 0 replies \ @k00b OP 16 Jan 2024

You're right! Thanks for bringing it to my attention. I wasn't even aware of that extension.

I've been disappointed by pg's fts in the past so I didn't consider using pg for this, but we could make better use of the embeddings if they were in pg.

420 sats \ 1 reply \ @Signal312 31 Jan 2024

Here's some things in Search you may want to look at. Basically it seems like URLs aren't returned in search results when you'd think they would be.

I would expect this post Need feedback on product - Speed ( www.tryspeed.com ) to show up whenever I search for tryspeed. But here are my results:

https://stacker.news/search?q=tryspeed (tryspeed without quotes, lots shows up. The first search result that comes up does NOT appear to have tryspeed in the text anywhere that I could find. The closest I could find was the word speech. The second result has my targeted post. https://stacker.news/search?q=%22tryspeed%22 (tryspeed in quotes, nothing shows up) https://stacker.news/search?q=%22tryspeed.com%22 (tryspeed.com in quotes, nothing shows up) https://stacker.news/search?q=%22www.tryspeed.com%22 (www.tryspeed.com in quotes, my targeted post shows up)

Similar thing here: https://stacker.news/search?q=discoverpraxis without quotes, doesn't find target post https://stacker.news/search?q=%22discoverpraxis%22 with quotes, doesn't find target post https://stacker.news/search?q=%22discoverpraxis.com%22 with quotes, with .com, DOES find the posts I thought it should.

158 sats \ 0 replies \ @k00b OP 31 Jan 2024

https://stacker.news/search?q=url%3Awww.tryspeed.com

We do support url searching but it has to be explicit right now. Good feedback.

420 sats \ 1 reply \ @Signal312 30 Jan 2024

I'm really loving the quoted search, which allows you to search for the EXACT term. It makes Search a lot more useful.

Thank you!

Interestingly, I only discovered it by chance, just playing around with search. I know that on Google search, quoting a term makes it required, so I tried it out, and it worked.

Embarrassingly - I did NOT find the ability to do this by looking at the search screen, even though I search a lot. Usually on a search screen, I'm accustomed to just looking for the "advanced search" pop-up, which gives you all the keywords and tricks you can use.

Now I see that the quoted search ability IS now mentioned on the search screen. It's on the bottom, under "More Filters" in the unusual font. Which apparently was something that I just was not able to see.

Here's an idea - to make this more discoverable, how about putting a little "advanced search" link next to the magnifying glass, that pops up the advanced search options?

Also another thing - what about making the default ranking NOT be zaprank? Or at least, having an option to have it be "best match"? A lot of times when you search, it shows up some random posts, that have the same root word as what you're looking for, that happen to be heavily zapped. Many of these aren't useful, because the longer a post is, the more likely it is that it'll have the root of the word you're looking for.

30 sats \ 0 replies \ @k00b OP 30 Jan 2024

Great feedback.

what about making the default ranking NOT be zaprank?

We could give it less weight. Currently it's applied linearly, ie match_score*zaprank but it might make more sense to do match_score*log10(zaprank)

100 sats \ 0 replies \ @DiedOnTitan 16 Jan 2024

Google has proven that improving search is not a subtle upgrade, but a game changer. Well done SN. For the record, I am not a Google fan nor a surveillance company fan, but their business model relied on focusing on making search fast and relevant and it paid off hugely for them.

15 sats \ 0 replies \ @grayruby 16 Jan 2024

Good news.

10 sats \ 1 reply \ @cryptofolyo 16 Jan 2024

I think the search box should be more visible like the google home page

0 sats \ 0 replies \ @k00b OP 16 Jan 2024

Thanks! We agree. Our navigation bar needs a makeover.

0 sats \ 0 replies \ @riberet19 19 Jan 2024

everything that improves stacker.news is good news

0 sats \ 0 replies \ @yfaming 16 Jan 2024

Great Job!👍

0 sats \ 0 replies \ @030e0dca83 16 Jan 2024

It's great if you'll make search engine better than Reddit, course if I want to find something on Reddit I should go the google and search there "myAnswer + reddit" querry