Uncertain Future For Marginalia Search

Posted: 2022-04-28

I found myself effectively without a job on short notice.

I’m not at all worried about finding another one, I have savings, and I have experience, and I have demonstrable skill. What I am concerned about is finding a source of income that’s compatible with putting some time on my personal projects.

Last bunch of years, I’ve been working 32 hour weeks, which is a pretty sweet deal especially combined with the zero hour commute you get working from home during the pandemic. Not every employer is fine with that, and while I do have options, I’m in a worse bargaining position than I have been before.

In the short term I’m going to take a few months’ hiatus and work full time on the search engine. I have a lot of ideas that I haven’t had the time to thoroughly explore that will hopefully produce decent improvements to some of the areas that are most lacking.

Perhaps the biggest thing is looking over the design a bit in terms of affordances, coming from something like Google, it can be difficult to find the sort of queries that produce interesting and worthwhile results. I think this can give a bad first impression.

Another thing that needs fixing is the summary text for each search result. It’s bad more often than it is good as things stand right now, and often makes good search results look like bad hits.

Finally, I’m working on changing the crawler model to be more aligned with WARCs, the format used by internet archive and various other crawlers such as commons crawl. This is half-done already, but hopefully should make the data more portable and open for collaborating with other crawling efforts.

Ideally I would love to be able to work on this for a longer time, but search engines just aren’t all that profitable, less still with my general lack of business sense. I’m incredibly grateful that I have patreon supporters and so on, but right now they just about pay the cost of running the hardware. But who knows, maybe I’ll build something so impressive in the next months as to open up some alternatives.

Not a likely turn of events, but still.

Reaching for the stars may not let us grasp them, we do all the same stand taller through the effort.

See Also