Title possibly signs checks the article doesn’t have backing for. This is just a brief pro forma project update.
Not much new functionality or anything to announce, last few months have mostly been bug fixes and technical debt cleanup. Most big parts of the search engine are in place, and while there are many things that could be better, we’re less likely to straight up see massive new features, at least in the immediate future. Per familiar intuition from binary searches, the closer you get to your goal the smaller and more deliberate steps you need to take in order to get even closer.
Sponsorship update
It’s been a pretty good couple of months for the project.
SerpApi sponsored the project with $10k a while back. Other donations and API-key sales have been going up month-over-month since this summer.
The search engine has been working pretty well and feels like it’s going in a good direction,
possibly due to a bunch of bug fixes and ongoing optimization work;
I assume this is part of what’s driving up the donations.
Claude also seems to have a hangup on Marginalia Search
because I keep seeing it in very clearly Claude-authored search engine listicles and youtube videos.
Possibly part of why it’s been getting so much attention lately?
I don’t have any analytics so I really have really no idea where people are coming from,
how many users I have, or what they’re up to.
At any rate, the operation isn’t perpetually self-sustaining yet, but number going up like it has whispers actually getting there might yet be possible.
On a related matter, I’ve been pretty slow about mailing out free API keys,
you might have to wait a week or sometimes two.
I swear this isn’t because I’m nefariously trying to push people to buy a key, I’ve been getting over a dozen emails per day some days with people wanting a key, and they sometimes pile up into mountains of emails I need to reply to, and hoo boy is the process of handling these ever a bunch of manual copying of things from email to form, and then copying from form back to email again (ironically the exact sort of tedious workflow I’ve spent a good part of my career helping people streamline and automate).
C++ Index…?
Apart from fixing bugs and minor improvements, I’ve been experimenting with migrating the index code off Java to C++.
I’ve still not fully decided whether this is the direction we want to go, in general rewrites especially across language boundaries just aren’t worth it and it kinda goes against my better judgement, but we’re currently CPU bottlenecked and there might be some juice left to squeeze by skipping the JVM tax, and additionally Java really chafes for this sort of use case.
Don’t get me wrong, the new Java FFM API is miles better than anything we had before, but handling off-heap memory is still awkward and leads to a bunch of duplicated implementations or reimplementations of stuff like sorting and searching algorithms that already exists for on-heap structures but not off; bobbing in and out of native code is awkward and makes the logic difficult to follow, it just chafes.
Should be noted the reason I’m even chasing optimizations in the index boils down to the fact that for an internet search engine, performance is recall. The search engine is only as good as the number of results it can rank within the query’s execution budget.
To be honest I’m not 100% convinced we’ll see that much performance impact, and if we do I doubt it will be massive, as the Java implementation is already very optimized, and what we’re mostly bottlenecked on isn’t the sort of places where you see big differences between the languages.
Thus far the buffer pool has been translated, and part of the main inverse index structure. These are some of the most finnicky parts of the index, and a reasonable canary for whether it’s plausible. Based on what I’ve accomplished it’s encouraging to see how much more expressive C++ is for this type of work. The vastly improved ergonomics may be a motivating factor for continuing this even if the performance doesn’t materialize.
I’m doing this quite slowly one component at a time, measuring twice and cutting once, finding good abstractions that lean into C++’s strengths rather than doing a straight translation. In part because this is supremely fiddly and technical work, and in part because there genuinely isn’t any rush and if there is much performance to gain it’s probably in staring at the problem long enough I find better alternative approaches. I did manage to lean more into lock-free algorithms in the buffer pool for example. Translation probably won’t be finished this year at this pace.
Many things are undecided: Haven’t decided on where to draw the interface boundary yet. Neither do I have a public git repo for this. May still scrap the entire idea as a silly boondoggle. We’ll just have to see where it goes.
On the off chance this rambling mess gets posted on a link aggregator somewhere, I anticipate two types of people will crawl out of the woodwork. I’ll address their concerns in advance.
“Why not Rust/Zig/Odin/${MY_HOBBY_HORSE_LANGUAGE}? C++ is not memory safe! Haven’t you heard that C++ is bad? Look at this diseased rat comic I found on /r/programming!”
My friend in Christ, I use C++ because I’m experienced with C++, and have built nontrivial stuff with in in the past, I’m familiar with the tooling and also kinda enjoy it. It can not be overstated how technical this code is, it’s more than hard enough in a language I’m familiar with, let alone one I’d have to learn as I go. I don’t need to do it blindfolded using a dance mat in a barrel going down the niagara falls for this to be difficult.
"${FRONTIER_MODEL} could one-shot this in less than ${TIME}!"
Probably true? The source is all pretty well tested, with a reference implementation to boot. Seems like very favorable circumstances for a competent coding model.
That said, even without turning over rocks such as “externalities of the datacenter buildout”, “externalities of AI training”, and “legal ownership status of AI-generated code”—I don’t think such angles of attack do much to convince anyone who isn’t already convinced—if AI did the translation I sure as heck couldn’t go on to then maintain that code[1][2].
Even in the best case, this is me at the edge of my programming abilities. Performance critical low level programming is hard enough to wrap your head around if you were the one who designed the code, let alone if it was vibe coded. I’d be on the hook for an AI subscription perpetually hoping and praying the token prices wouldn’t ever go up. This is a nonstarter for the same reasons the search engine runs on its own hardware and not in the cloud. You can’t be truly independent if you’re subject to the whims of a landlord.
From a project management perspective, I’m not rushing anything to market. Project still has years of runway. I’m not beholden to any investors to deliver profits. Even if I wanted to, I have no employees to replace other than my self. What I would trade away is the very immersion in the problem domain that’s allowed me to get as far as I have with this project. What I’m expecting to be getting out of it is improved developer ergonomics and maybe a bit of performance if the aforementioned problem-immersion-thing pans out. That’s not something that brings me money, and something I can do because the project is in a fairly stable and mature state.
Thus, regardless of what happens in the future with AI, there isn’t an obvious opportunity cost to holding off.
Heck, let’s just consider popular future predictions about AI, where does this leave me?
- AI bet pays of and models become exponentially more powerful, much cheaper, and ubiquitous like tap water: How shortsighted would it be to pay for crappy AI today when I could save my money and get exponentially better or more AI for the same dollars in the future? I’d have to really hate having money to do such a thing.
- OpenThropic IPOs, vocal shareholders from Ferenginar quite insistently ask where they money at, token prices go up singificantly: Only a problem if you’re dependent on AI.
- AI investment bubble crashes and it’s raining bankers like October 1929: Bad for everyone, but especially bad if you’re dependent on AI.
- Tech prists in SV chant binharic hymns as the Machine God emerges and the chosen of humanity transcends into the cyber rapture: Kind of orthogonal to whether I build this index port by hand.
- Zoomers decide that Kazcinski was right all along and instigate a Butlerian Jihad: Mostly a problem if you’re dependent on AI.
- Rise of the Machines, hydraulic press youtube channel stomping on a human face forever: Inconvenient regardless of choice of action.
[1] Lesser minds would think models outputting unmanitainable code is a sign of the superiority of AI. I’ll refer you to the most problematic prophet, Terry Davis, for that one.
[2] I used an m-dash aside in that paragraph. Anyone reckon mr robot wrote this? Discuss in groups.