search-engine on marginalia.nu

An NSFW filter for Marginalia Search

Mon, 30 Mar 2026 00:00:00 +0000

… optional, that is.

I’ve been working on an NSFW filter for Marginalia Search, as that is something some people have asked for, primarily API consumers.

The search engine has had some domain based filtering for a while, based on the UT1 lists, but that isn’t a very comprehensive approach.

We’ll land on a single hidden layer neural network approach, implemented from scratch, but before landing on that, many other things were tried along the way.

Index Compression, Query Execution Improvements

Fri, 13 Feb 2026 00:00:00 +0000

The Marginalia Search index has recently seen some design tweaks to make it perform better, primarily the introduction of postings list compression.

Last year, the index was partially re-implemented with SSDs in mind. This was largely a success, but left some lingering issues with tail latencies that sometimes weren’t what they needed to be.

To ensure predictable execution times, the query execution is provided a timeout value, after which it will wrap up and return the best results it’s found. Query execution was so flaky that the actual timeout used when terminating the execution used to be something like 50ms lower than the provided value. This is obviously not a fantastic state of affairs.

Trust in Ranking

Sat, 31 Jan 2026 00:00:00 +0000

The Marginalia Search default ranking algorithm recently saw a fairly radical improvement, due to a new domain trust system that drastically reduces the number of content farm results, as long as there are human results it usually finds them across all the usual test queries.

Recently fixing a few bugs that made the search engine work more correctly had the unexpected and undesired side-effect of also making it surface more search engine spam and content farm-type results.

New Search Filtering in Web and API

Mon, 08 Dec 2025 00:00:00 +0000

The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI.

This was originally going to be an incredibly chaotic update, both annuncing the new features and doing a technical walkthrough of the changes but that ambition turned out a bit too chaotic, so let’s split them up and focus on the feature announcement bit today.

New Search Filtering GUI

It’s now possible to define a custom filter in the GUI, on the marginalia-search.com version of the website!

Language Support for Marginalia Search

Mon, 06 Oct 2025 00:00:00 +0000

One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish.

These changes are now live for testing, but with an extremely small corpus of documents.

As the search engine has been up to this point built with English in mind, some anglo-centric assumptions made it into its code. A lot of the research on search engines generally seems to embed similar assumptions.

Faster Index I/O with NVMe SSDs

Sat, 16 Aug 2025 00:00:00 +0000

The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes.

The index is already fairly large, but can sometimes feel smaller than it is, and paradoxically, query performance is a big part of why. If each query has a budget of 100-250ms, a design that finds and ranks results faster in that time period will produce better search results. There are other limitations as well, query understanding is still somewhat limited, where only minor changes to a query can unearth dozens of new related results.

Finding Dead Websites

Tue, 17 Jun 2025 00:00:00 +0000

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change.

This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking.

Profiling Websites

Thu, 29 May 2025 00:00:00 +0000

The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements.

The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code.

It turns out to be somewhat difficult to determine what a website that has non-trivial javascript will look like based its source code alone, as this would require us to among other things solve the halting problem.

PDF to Text, a challenging problem

Tue, 13 May 2025 00:00:00 +0000

The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months.

Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format.

It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”. These glyphs may be rotated, overlap, and appear out of order, with very little semantic information attached to them.

Crawl Order and Disorder

Thu, 27 Mar 2025 00:00:00 +0000

A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains.

This has been actualized recently, since the migration to slop crawl data has dropped memory requirements of the crawler by something like 80%, and as such I’ve been able to increase the number of crawling tasks, which has led to a bizarre case where 99.9% of the crawling is done in 4 days, and the remaining 0.1% takes a week.

Marginalia Search receives second nlnet grant

Tue, 25 Mar 2025 00:00:00 +0000

I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant.

All the details are not yet finalized, but tentatively the grant will go toward addressing most of the items in the project roadmap for 2025.

I’ve already been working full time on the project since summer 2023, and this grant secures additional development time, and extends the runway to a comfortable degree.

Will post more details as they are finalized.

Marginalia Search: 4 Years

Mon, 03 Mar 2025 00:00:00 +0000

This update is a few days late, the canonical birth date of the project is Feb 26.

It has been another year of Marginalia Search. The project is still ongoing, still my full time job, although the project is entering a somewhat more mature phase of development, most of the big pieces are in place and do a decent job at what they do.

The roadmap for the project is available on GitHub.

RSS Feeds and Real Time Crawling

Thu, 26 Dec 2024 00:00:00 +0000

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed.

The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any given point in time. To help compensate for this, a new crawler and index partition has been added that focuses on recently updated content.

Notes on binary soup

Tue, 05 Nov 2024 00:00:00 +0000

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before.

This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

Release Notes v2024.10.0

Mon, 14 Oct 2024 00:00:00 +0000

This is a new major release of marginalia search, mostly leaning toward the technical side.

Emphasis has been on ensuring the search engine has the technical capabilities to serve more types of queries, especially longer queries which it previously did not handle very well.

Effort has also been put toward making sure it’s possible to install and run outside of docker. There is still some work to be done to streamline the installation process, but we’re getting there.

Phrase Matching in Marginalia Search

Mon, 30 Sep 2024 00:00:00 +0000

Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query.

This is a write-up about implementing this change. This is going to be a relatively long post, as it represents about 4 months of work.

I’m also happy and grateful to announce that the nlnet people reached out after the run of the grant was over and asked me if I had more work in the pipe, and agreed to fund this change as well!

One year of solo dev, wrapping up the grant-funded work

Tue, 18 Jun 2024 00:00:00 +0000

A year ago I walked out of the office for the last time. I handed in my corpo laptop, said some good-byes, and since then I have been my own boss.

This first year has been funded by an NLnet grant, which I’m in the midst of wrapping up. As of now, the work is all done, the final request for payment has been sent.

There’s a similar last-day-of-school levity to both these events.

Experiment in Java native calls

Thu, 16 May 2024 00:00:00 +0000

I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search.

The Marginalia Search engine is mainly written in Java, which is language that is good at many things, but not particularly pleasant to work with when it comes to low level systems programming.

Unfortunately, a part of building an internet search engine involves database-adjacent low level programming.

Query Parsing and Understanding

Wed, 17 Apr 2024 00:00:00 +0000

Been working on improving Marginalia Search query parsing and understanding. This is going to be a pretty long update, as it’s a few months’ work.

Apart from cleaning up the somewhat messy query parsing code, a problem I’m trying to address is that the search engine is currently only good at dealing with fairly focused queries, they don’t need to be short, but if you try to qualify a search that is too broad by adding more terms, it often doesn’t produce anything useful.

Deep Bug

Wed, 10 Apr 2024 00:00:00 +0000

The project has been haunted by a mysterious bug since sometime February. It relates to the code that constructs the index, particularly the code that merges partial indices.

In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement.

You can conceptualize the revese index itself as two files, one with offset pointers into another file, which has sorted numbers. This code runs after each partition finishes crawling and processing its data, and has a run time of about 4 hours.

The Yak Shave

Wed, 28 Feb 2024 00:00:00 +0000

I set out a little over a week ago to add a service registry to Marginalia Search, primarily to reduce its dependence on docker. I would like it to be able to run on bare metal as well, which poses a problem since configuring the application manually is a bit of a headache with dozens of ports that need to be set up. It would also be desirable to be able to run multiple instances of important services in order elliminate downtime during upgrades.

Marginalia: 3 Years

Sun, 25 Feb 2024 00:00:00 +0000

It’s been three years since the inception of Marginalia Search, then a dinky experiment to find where the heck the cool Internet has gone, now my full time job.

While there’s always things that can be improved, it’s fair to say the search engine has never worked as well as it does right now.

A great number of milestones have been reached, perhaps biggest of all the search engine has moved out of my living room and into a proper enterprise server.

Best SEO spam 2024 reddit

Wed, 07 Feb 2024 00:00:00 +0000

One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time.

I’ve been noticing the search engine spam strategy of adding ‘reddit’ to page titles for a few years now, but it feels like it’s been growing a lot recently. I don’t think it’s actually working, but it’s so cute that they are trying.

Release Notes v2024.01.0

Wed, 24 Jan 2024 00:00:00 +0000

This is a major new release of the search engine software, corresponding to nearly four months of changes. In these months, the state of the code hasn’t been stable enough for a new release, but it’s now been brought to a stable point.

Release Highlights:

The installation procedure has been cleaned up.
It’s now possible to run the search engine in a white label/bare-bones mode, without any of the Marginalia Search branding or logic.
The Marginalia Search web interface has been overhauled. The site-info page has especially been given a large upgrade.
The search engine can use anchor texts to supplement keywords.
The search engine can use multiple index shards.
The operations GUI has been overhauled.
An operations manual has been written.
The crawler can now resume crawls in process due to intermediate WARCs.
The search engine can import several formats without external pre-processing.
The Academia filter has been improved
The Recipe filter has been improved
The system now penalizes documents that have obvious hallmarks of being written by ChatGPT in its quality assessment.

Other technical changes:

A Frivolous Feature

Fri, 22 Dec 2023 00:00:00 +0000

Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS. At a glance this seems like a somewhat frivolous feature, but it has interesting effects.

Autonomous Systems are part of the Internet’s routing infrastructure. If your mental model of an IP number is that they are the phone number of the computer, this is something akin to a postal code. Digging much deeper than that into BGP and autonomous systems is not really in the scope of this article, but Wikipedia has a relatively lucid article on this.

WARC'in the crawler

Wed, 20 Dec 2023 00:00:00 +0000

The Marginalia Crawler has seen improvements! A long term problem with the crawler design is that if for whatever reason the crawler shuts down, then it needs to re-start fetching whatever domains it was currently traversing during the termination from zero.

This isn’t fantastic, since not only does crawling a website take a fair bit of time, it’s a nuisance for the server admins to re-crawl stuff that was already fetched, and a real liability for ending up in robots.txt or some iptables ruleset.

Anchor Tags

Tue, 07 Nov 2023 00:00:00 +0000

I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage.

The problem I’m attempting to address is that many websites don’t really describe themselves particularly well. As Steve Ballmer’s stage performance once illustrated, merely repeating a word doesn’t on its own make what you’re saying relevant to the term.

Another good example of how it falls short is PuTTY’s website, which will be used as a pilot case to improve.

Partitioning The Index

Mon, 30 Oct 2023 00:00:00 +0000

So a bit of an update on what I’ve been working on. This will be adapted into release notes in a while, but I haven’t quite wrapped a bow on the change set yet.

Still, it has certainly been a few weeks. Didn’t quite land how busy I’ve been until I set down to draft this post. Them’s some changes, and I’m skipping a few to keep this meandering post at a sane length.

Moving Marginalia to a New Server

Sat, 07 Oct 2023 00:00:00 +0000

So the search engine is moving to a new server soon, thanks to the generous grant mentioned recently.

If you visit search.marginalia.nu now, it may or may not use the old or new server. It’ll be like this for a while, since I need them both for testing and maintenance type work.

I’ll also apologize if this post is a bit chaotic. It is a reflection of a very chaotic couple of weeks that apart from setting up this migration also involved a very short notice invitation for a presentation at ossym23.

Release Notes v2023.10.0

Sat, 07 Oct 2023 00:00:00 +0000

This is a mostly technical release. It takes the index from 106M to 164M documents.

Zero Downtime Upgrades and halved memory consumption

The initial focus of the release was to address the sometimes lengthy downtimes that have plagued the project when loading a new index.

There is a somewhat lengthy write-up about this here; but the short version is that this was very successful and a drastic optimization, removed not only the needed downtime, but added neat new features and slashed the RAM requirements in half!

Marginalia Search receives FUTO Grant

Fri, 15 Sep 2023 00:00:00 +0000

I’m happy to announce that the generous people at FUTO have granted the project $15,000 with no strings attached to help the search engine out with some more server power.

FUTO is a young Austin, TX-based organization “dedicated to developing, both through in-house engineering and investment, technologies that frustrate centralization and industry consolidation”. It’s one to keep an eye on, I believe their heart is in the right place and they have every possibility of making a real difference.

Absurd Success

Wed, 30 Aug 2023 00:00:00 +0000

So… I’ve had the most unreal week of coding. Zero exaggeration, I’ve halved the RAM requirements of the search engine, removed the need to take the system offline during an upgrade, removed hard limits on how many documents can be indexed, and quadrupled soft limits on how many keywords can be in the corpus.

It’s been a long term goal to keep it possible to run and operate the system on low-powered hardware, and so far improvements have been made, to the point where my 32 Gb RAM developer machine feels spacey rather than cramped, but this set of changes takes it several notches further.

Release Notes v2023.08.0

Tue, 22 Aug 2023 00:00:00 +0000

This release mainly aims to improve the operational side of the search engine, with an emphasis of automating tedious manual processes and optimizing crawling and data processing to use fewer resources.

Conventionally I try to link to relevant commits in these notes, but some of the changes were so sweeping and protracted it was hard to narrow it down to individual commits; in those cases I’ll link to the relevant code instead.

Message Queues, State Machines, Actors, UI

Sat, 12 Aug 2023 00:00:00 +0000

This is a bit of an what I’ve been working on style of post. It’s also a bit of a complement for the release notes of the upcoming release which should be dropping in a week or so. There’s some spit and polish still missing from these things, but if I don’t write about them now too much will have been ejected from the cache to make a well written post about it.

Release Notes v2023.06.0

Thu, 29 Jun 2023 00:00:00 +0000

New Features

Generator keywords

To provide additional ways of selecting search results, a synthetic keyword has been added for the <meta name="generator" content="..."> tag. This is basically a vanity tag that is used by some HTML generators to advertise themselves, and it’s also common for hand-edited HTML to include this tag with a string like “vim” or “myself”, as a wink to human readers of the code.

The generator keywords have the form generator:value. For example, to search for websites made with Hugo, you can use generator:hugo. Generator categories have also been added as searchable keywords, for example generator:wiki, generator:forum, generator:docs.

Full Time

Fri, 16 Jun 2023 00:00:00 +0000

I’m working on Marginalia Search full time.

I left the office for the last time today, and it’s the strangest feeling. I’ve quit jobs, taken time off work, been laid off, but this is different from any of those things. This is deliberate.

There’s a note of relief. I’ve essentially been working two pretty demanding jobs; one for pay and one for passion and the joy of making a difference.

Release Notes v2023.03.2

Thu, 25 May 2023 00:00:00 +0000

This is primarily a bugfix release that primarily addresses some issues with a metadata corruption that was introduced in the previous release.

New Features

File keywords

To provide more tools for navigating the web, the converter now generates synthetic keywords for documents that link to files on the same server based on their file ending.

If the file contains a link such as

<a href="file.zip">Download</a>

then he document will be tagged with the keyword file:zip as well as file:archive.

Going Github

Sat, 25 Mar 2023 00:00:00 +0000

I’ve moved Marginalia’s sources to Github. Can’t pick every battle.

The main reason is I’m kind of tired of the amount of spam bots that keep signing up to my Gitea. The juice of self-hosting a public-access git forge, even locked down to prevent arbitrary repo creation, that juice just isn’t worth the squeeze.

This is not without some consideration.

To be blunt, I don’t like Github. Their use of dark patterns leaves a real nasty after-taste. I’m also old enough to remember the Microsoft of the early 2000s very vividly.

Search Result Quality For Multiple Terms

Thu, 23 Mar 2023 00:00:00 +0000

This is a bit of a follow up to the previous post.

The Grand Code Restructuring [ 2023-03-17 ]

Marginalia’s search result quality has, for a long while, been pretty good as long as your search query is a single term, but for multiple search terms it’s been a bit hit-and-miss. Marginalia was never great at this, but the quality of results in this usage pattern has taken a bit of a dive recently due to a re-write of the index last fall.

The Grand Code Restructuring

Fri, 17 Mar 2023 00:00:00 +0000

In general I don’t like to fuss over code, but this is exactly what I’ve been doing in preparation of the NLnet funded work. I’ve spent the last month restructuring Marginalia’s code base. It’s not completely done, but I’ve made great headway.

Things got the way they got because in general for experimental solo-development projects, I think it makes sense to be fairly tolerant of technical debt.

Since refactoring is something that is extremely difficult to break up into parallel tracks or do in small iterations, the cost of refactoring is effectively multiplied by the number of people that could be working on the code.

Marginalia Search: 2 years, big news

Sun, 26 Feb 2023 00:00:00 +0000

No time like the project’s two year anniversary to drop this particular bomb…

Marginalia’s gotten an NLNet grant. This means I’ll be able to work full time on this project at least a year.

https://nlnet.nl/project/Marginalia/

This grant is essentially the best-case scenario for funding this project. It’ll be able to remain independent, open-source, and non-profit.

I won’t start in earnest for a few months as I’ve got loose ends to tie up before I can devote that sort of time. More details to come, but I’ll say as much as the first step is a tidying up of the sources and a move off my self-hosted git instance to an external git host yet to be decided.

Faster Index Joins

Tue, 03 Jan 2023 00:00:00 +0000

The most common (and most costly) operation of the marginalia search engine’s index is something like given a set of documents containing one keyword, find each documents containing another keyword.

The naive approach is to just iterate over each document identifier in the first set and do a membership test in the b-tree containing the second. This is an O(m log n)-operation, which on paper is pretty fast.

It turns out it can be made faster.

Creepy Website Similarity

Mon, 26 Dec 2022 00:00:00 +0000

This is a write-up about an experiment from a few months ago, in how to find websites that are similar to each other. Website similarity is useful for many things, including discovering new websites to crawl, as well as suggesting similar websites in the Marginalia Search random exploration mode.

A link to a slapdash interface for exploring the experimental data.

The approach chosen was to use the link graph look for websites that are linked to from the same websites. This turned out to work remarkably well.

About Marginalia Search

Fri, 23 Dec 2022 00:00:00 +0000

This information is outdated

The marginalia search project info now lives on about.marginalia-search.com.

Ever feel like the Internet has gotten a bit… I don’t know, samey? There’s funny images scrolling by and you blow some air through your nose and keep scrolling and then someone has done something upsetting and you write an angry comment and then you scroll some more.

Remember when you used to explore the Internet, when you used to discover cool little websites made by people and it wasn’t just a bunch of low effort content mill listicles and blog spam?

Carbon Dating HTML

Thu, 27 Oct 2022 00:00:00 +0000

One of the more common feature requests I’ve gotten for Marginalia Search is the ability to search by date. I’ve been a bit reluctant because this has the smell of a a surprisingly hard problem. Or rather, a surprisingly large number of easy problems.

The initial hurdle we’ll encounter is that among structured data, pubDate in available in RDFa, OpenGraph, JSON+LD, and Microdata.

A few examples:

<meta property="datePublished" content="2022-08-24" />
<meta itemprop="datePublished" content="2022-08-24" />
<meta property="article:published_time" content="2022-08-24T14:39:14Z" />
<script type="application/ld+json">
{"datePublished":"2022-08-24T14:39:14Z"}
</script>

So far not so that bad. This is at least a case where the web site tells you that here is the pub-date, the exact format of the date may vary, but this is solvable.

Marginalia's Index Reaches 100,000,000 Documents

Fri, 21 Oct 2022 00:00:00 +0000

A very brief note to announce reaching a long term goal and major milestone for marginalia search.

The search engine now indexes 106,857,244 documents!

The previous record was a bit south of seventy million. A hundred million has been a pie-in-the-sky goal for a very long time. It’s seemed borderline impossible to index a that many documents on a PC. Turns out it’s not. It’s more than possible.

Twice this may even be technically doable, but is way past the pain point of sheer logistics. It’s already a real headache to deal with this much data.

Fragments of the Old Web

Thu, 15 Sep 2022 00:00:00 +0000

The following is a list of curiosities I’ve found while crawling the internet looking for websites. A sort of greatest hits from my search engine.

There is no real system or theme to the links, other than the fact that they have made me go “huh, that’s neat” while visiting them.

Most of these are effectively impossible to find on Google, since they don’t use HTTPS, and aren’t optimized for mobile, and aren’t plastered in ads and tracking scripts.

Donate To This Project

Mon, 05 Sep 2022 00:00:00 +0000

This information is outdated

The marginalia search project info now lives on about.marginalia-search.com.

I’m just one guy building all of this on my own. I’d like to expand the search engine and make it more useful. My hope is that it will bring value to its users and enable a thriving independent Internet.

The search engine doesn’t have any secret sauce, all the source code is publicly available and as far as is legally and logistically possible, the data is also available.

The Evolution of Marginalia's crawling

Tue, 23 Aug 2022 00:00:00 +0000

In the primordial days of Marginalia Search, it used a dynamic approach to crawling the Internet.

It ran a number of crawler threads, 32 or 64 or some such, that fetched jobs from a director service, that grabbed them straight out of the URL database, these jobs were batches of 100 or so documents that needed to be crawled.

Crawling was not planned ahead of time, but rather decided through a combination of how much of a website had been visited, and the quality score of that website determined where to go next. It also promoted crawling websites adjacent to high quality websites.

Marginaliacoin, and hidden forums

Thu, 18 Aug 2022 00:00:00 +0000

I discovered someone has made a cryptocurrency called “Memex Marginalia Inu”. It appears to have been created February 23, which is around when the entry “I Have No Capslock And I Must Scream” went absurdly viral to the point where Elon Musk tweeted a link to it.

I Have No Capslock…

Mr Musk’s twitter orbit is exceptionally strange. The tweet was followed by a deluge of bizarre activity, strange emails with calls about stonk canine lunar expeditions, and apparently also a cryptocurrency land-grab of sorts. I can’t claim to understand why, but many of the emails got after the tweet were on the theme “what does this mean?”, almost as though Elon’s tweet was some sort of prophetic omen.

Fun with Anchor Text Keywords

Thu, 23 Jun 2022 00:00:00 +0000

Anchor texts are a very useful source of keywords for a search engine, and in an older version of the search engine, it used the text of such hyperlinks as a supplemental source for keywords, but due to a few redesigns, this feature has fallen off.

Last few days has been spent working on trying to re-implement it in a new and more powerful fashion. This has largely been enabled by a crawler re-design from a few months ago, which offers the crawled data in a lot more useful fashion and allows a lot more flexible post-processing.

marginalia.nu goes open source

Fri, 27 May 2022 00:00:00 +0000

After a bit of soul searching with regards to the future of the website, I’ve decided to open source the code for marginalia.nu, all of its services, including the search engine, encyclopedia, memex, etc.

A motivating factor is the search engine has sort of grown to a scale where it’s becoming increasingly difficult to productively work on as a personal solo project. It needs more structure. What’s kept me from open sourcing it so far has also been the need for more structure. The needs of the marginalia project, and the needs of an open source project have effectively aligned.

Uncertain Future For Marginalia Search

Thu, 28 Apr 2022 00:00:00 +0000

I found myself effectively without a job on short notice.

I’m not at all worried about finding another one, I have savings, and I have experience, and I have demonstrable skill. What I am concerned about is finding a source of income that’s compatible with putting some time on my personal projects.

Last bunch of years, I’ve been working 32 hour weeks, which is a pretty sweet deal especially combined with the zero hour commute you get working from home during the pandemic. Not every employer is fine with that, and while I do have options, I’m in a worse bargaining position than I have been before.

Lexicon Architectural Rubberducking

Mon, 11 Apr 2022 00:00:00 +0000

I’m going to think out loud for a moment about a problem I’m considering.

RAM is a precious resource on any server. Look at VPS servers, and you’ll be hard pressed to find one with much more than 32 Gb. Look at leasing a dedicated server, and it’s the RAM that really drives up the price. My server has 128 Gb, and it it’s so full it needs to unbutton its pants to sit down comfortably. Anything I can offload to disk is great.

The Bargain Bin B-Tree

Thu, 07 Apr 2022 00:00:00 +0000

I’ve been working lately on a bit of an overhaul of how the search engine does indexing. How it indexes its indices. “Index” is a bit of an overloaded term here, and it’s not the first that will crop up.

Let’s start from the beginning and build up and examine the problem of searching for a number in a list of numbers. You have a long list of numbers, let’s sort them because why not.

Growing Pains

Wed, 23 Mar 2022 00:00:00 +0000

The search engine index has grown quite considerably the last few weeks. It’s actually surpassed 50 million documents, which is quite some milestone. In February it was sitting at 27-28 million or so.

About 80% of this is side-loading all of stackoverflow and stackexchange, and part of it is additional crawling.

The crawler has to date fetched 91 million URLs, but only about a third of what is fetched actually qualifies for indexing for various reasons, some links may be dead, some may be redirects, some may just have too much javascript and cruft to qualify.

Marginalia Search: 1 year

Fri, 25 Feb 2022 00:00:00 +0000

I’ve caught some bug and don’t have the energy to write more than a brief note.

I want to commemorate the fact that work on the Marginalia search engine started one year ago. The first commit was on February 26th 2021, and contained a sketch for a website crawler and some data models.

In many ways, the paint is barely dry, yet it feels like this project has been around for a long while.

The Anatomy of Search Engine Spam

Mon, 07 Feb 2022 00:00:00 +0000

Black hat SEO is endlessly fascinating phenomenon to study. This post is about some tactics they use to make their sites rank higher.

The goal of blackhat SEO is to boost the search engine ranking of a page nobody particularly wants to see, usually ePharma, escort services, online casinos, shitcoins, hotel bookings; the bermuda pentagon of shady websites.

The theory behind most modern search engines is that if you get links from a high ranking domain, then your domain gets a higher ranking as well, which increases the traffic. The reality is a little more complicated than that, but this is a sufficient mental model to understand the basic how-to.

Can we unfuck internet discoverability?

Fri, 04 Feb 2022 00:00:00 +0000

I’ve been thinking a lot about how difficult it has become to discover quality content on the Internet, not because it isn’t there, but because the signal to noise ratio is really bad, and most venues of discovery don’t seem to be able to handle it.

Recommendation algorithms seem to work almost too well, to the point where it’s all kind of just showing you things you already like, rarely anything new that you might like. It’s an absolute tragedy both for small websites and for their potential audience.

Discovery and Design Considerations

Tue, 18 Jan 2022 00:00:00 +0000

It’s been a productive several weeks. I’ve got the feature pulling updates from RSS working, as mentioned earlier.

I’ve spent the last weeks designing the search engine’s web design, and did the MEMEX too for good measure.

It needed to be done as the blog theme that previously made the foundation for the design off had several problems, including loading a bunch of unnecessary fonts, and not using the screen space of desktop browsers well at all.

Search Result Relevance

Fri, 10 Dec 2021 00:00:00 +0000

This entry is about a few problems the search engine has been struggling with lately, and how I’ve been attempting to remedy them.

Before the article starts, I wanted to share an amusing new thing in the world of Internet spam.

For a while, people have been adding things like “reddit” to the end of their Google queries to get less blog spam. Well, guess what? The blog spammers are adding “reddit” to the end of their titles now.

Old and New

Fri, 12 Nov 2021 00:00:00 +0000

I’ve been thinking recently about the emphasis put on “new”, specifically for search engines, but the discussion has some merit even in a wider context. I will start wide and narrow down.

It is common to conflate new with good, and most being young sometime between 1950-2000 will indeed have seen marvellous improvements in quality of life and technology with each passing year. In the light of that, it’s at least easy to explain how one might confuse the two.

A Jaunt Through Keyword Extraction

Thu, 11 Nov 2021 00:00:00 +0000

Search results are only as good as the search engine’s ability to figure out what a page is about. Sure a keyword may appear in a page, but is it the topic of the page, or just some off-hand mention?

I didn’t really know anything about data mining or keyword extraction starting out, so I’ve had to learn on the fly. I’m just going to briefly list some of my first naive attempts at keyword extraction, just to give a context.

Shaking N-gram needles from large haystacks

Fri, 22 Oct 2021 00:00:00 +0000

A recurring problem when searching for text is identifying which parts of the text are in some sense useful. A first order solution is to just extract every word from the text, and match documents against whether they contain those words. This works really well if you don’t have a lot of documents to search through, but as the corpus of documents grows, so does the number of matches.

It’s possible to bucket the words based on where they appear in the document, but this is not something I’m doing at the moment and not something I will implement in the foreseeable future.

The Mystery of the Ceaseless Botnet DDoS

Sun, 10 Oct 2021 00:00:00 +0000

I’ve been dealing with a botnet for the last few days, that’s been sending junk search queries at an increasingly aggressive rate. They were reasonably easy to flag and block but just kept increasing the rate until that stopped working.

Long story short, my patience ran out and put my website behind cloudflare. I didn’t want to have to do this, because it does introduce a literal man in the middle and that kinda undermines the whole point of HTTPS, but I just don’t see any way around it. I just can’t spend every waking hour playing whac-a-mole with thousands of compromised servers flooding me with 50,000 search requests an hour. That’s five-six times more than when I was on the front page of HackerNews, and the attempts only increased.

Web Browsing

Sat, 09 Oct 2021 00:00:00 +0000

An idea I’ve had for a long time with regards to navigating the web is to find a way to browse it.

“Browse” a difficult word to use, because it has a newer connotation of just using a web browser, I mean it in the old pre-Internet sense, browse like when you flip through a magazine, or peruse an antiques shop, not really looking for anything in particular just sort of seeing if anything catches your eye.

Getting with the times

Wed, 06 Oct 2021 00:00:00 +0000

Since my search engine has expanded its scope to include blogs as well as primordial text documents, I’ve done some thinking about how to keep up with newer websites that actually grow and see updates.

Otherwise, as the crawl goes on, it tends to find fewer and fewer interesting web pages, and as the interesting pages are inevitably crawled to exhaustion, accumulate an ever growing amount of junk.

Re-visiting each page and looking for new links in previously visited pages is probably off the table, that’s something I can maybe do once a month.

Experimenting with Personalized PageRank

Sat, 02 Oct 2021 00:00:00 +0000

The last few days I’ve felt like my first attempt at a ranking algorithm for the search engine was pretty good, like it was producing some pretty interesting results. It felt close to what I wanted to accomplish.

The first ranking algorithm was a simple link-counting algorithm that did some weighting to promote pages that look in a certain fashion. It did seem to keep the page quality up, but also seemed to as a strange side-effect promote very “1996”-looking websites. This isn’t quite what I wanted to accomplish, I wanted to promote new sites as well as long as they were rich in content.

Astrolabe - The October Update

Fri, 01 Oct 2021 00:00:00 +0000

https://search.marginalia.nu

The October Update is live. It introduced drastically improved topic identification and an actual ranking algorithm; and the result is interesting to say the least. What’s striking is how much it’s beginning to feel like a search engine. When it fails to find stuff, you can kinda see how.

I’ve played with it for a while now and it does seem to produce relevant results for a lot of topics. A trade down in whimsical results but a big step up if you are looking for something specific, at least within the domain of topics where there are results to find.

Against the Flood

Sun, 19 Sep 2021 00:00:00 +0000

So hacker news apparently discovered my search engine, and really took a liking to the idea. Actually that’s a bit of an understatement, the thread has gotten 3.3k points and lingered on the front page for half a week. And I wasn’t planning for it to go quite that public yet. It has quietly been online for a while, but it was only very recently it started to feel like it was really coming together. It wasn’t perfect, there was still a lot of jankiness and limitations that could have been fixed with more time. The index was half the size it should have been. Someone discovered it and shared it. It took off like a rocket, and I’m still at a loss for words at the reception it’s gotten. I have received so many encouraging comments, emails, offers of collaboration, a few have even joined the patreon. I’ve been working through all the messages and I aim to reply to them all, but it takes time. I’m very grateful for all of this, since I half thought I was alone in this.

New Solutions Creating Old Problems

Tue, 14 Sep 2021 00:00:00 +0000

I’ve spent some time the last week optimizing how the search engine identifies appropriate search results, putting far more consideration into where and how the search terms appear in the page when determining the order they are presented.

Search-result relevance is a pretty difficult problem, but I do think the changes has brought the search engine in a very good direction.

A bit simplified, I’m building tiered indices, ranging from

The Curious Case of the Dot-Com Link Farms

Thu, 09 Sep 2021 00:00:00 +0000

I spent some time today weeding out yet more link-farms from my search engine’s index.

Typically what I would do is just block the subnet assigned to the VPS provider they’re on, and that does seem to work rather well. The cloud providers that don’t police what they host is almost always home to quite a lot of this stuff, so I don’t particularly mind scorching some earth in the name of a clean index.

The Small Website Discoverability Crisis

Wed, 08 Sep 2021 00:00:00 +0000

There are a lot of small websites on the Internet: Interesting websites, beautiful websites, unique websites.

Unfortunately they are incredibly hard to find. You cannot find them on Google or Reddit, and while you can stumble onto them with my search engine, it is not in a very directed fashion.

It is an unfortunate state of affairs. Even if you do not particularly care for becoming the next big thing, it’s still discouraging to put work into a website and get next to no traffic beyond the usual bots.

Soaring High

Thu, 02 Sep 2021 00:00:00 +0000

I’m currently indexing with my search engine. This isn’t an always-on sort of an affair, but rather something I turn on and off as it tends to require at least some degree of babysitting.

I’ve also been knocked out by the side-effects of the vaccine shot I got the other day, so it’s been mostly hands-off “parenting”.

What I’m trying to figure out just how far I can take it. I really don’t know. I took some backups and just let it do its thing relatively unmonitored.

The Astrolabe Part II: The Magic Power of Sampling Bias

Tue, 03 Aug 2021 00:00:00 +0000

As I have mentioned earlier, perhaps the biggest enemy of PageRank is the hegemony of PageRank-style algorithms. Once an algorithm like that becomes not only dominant, but known, it also creates a market for leveraging its design particulars.

Homogenous ecosystems are almost universally bad. It doesn’t really matter if it’s every computer running Windows XP, or every farmer planting genetically identical barley, what you get is extreme susceptibility to exploitation.

Index Optimizations

Fri, 23 Jul 2021 00:00:00 +0000

Don’t chase small optimizations

Said some smart person at some particular time, probably. If not, he ought to have; if worse comes to worst, I’m declaring it now. The cost of 2% here and 0.5% there is high, and the benefits are (by definition) low.

I have been optimizing Astrolabe, my search engine. The different kind of Search Engine Optimization. I’ve spent a lot of time recently doing soft optimization, improving the quality and relevance of search results, to great results. I’ll write about that later.

On Link Farms

Wed, 14 Jul 2021 00:00:00 +0000

I’m in the midst of rebuilding the index of my search engine to allow for better search results, and I’ve yet again found need to revisit how I handle link farms. It’s an ongoing arms race between search engines and link farmers to adjust (and circumvent) the detection algorithms. Detection and mitigation of link farms is something I’ve found I need to modify very frequently, as they are constantly evolving to look more like real websites.

The Astrolabe Part I: Lenscraft

Wed, 07 Jul 2021 00:00:00 +0000

Something you probably know, but may not have thought about a lot is that the Internet is large. It is unbelievably vast beyond any human comprehension. What you think of as “The Internet” is a tiny fraction of that vast space with its billions upon billions of websites.

We use various technologies, such as link aggregators and search engines to find our way and make sense of it all. Our choices in navigational aides also shapes the experience we have of the Internet. These convey a warped sense of what the Internet truly is. There is no way of not doing that. Since nothing can communicate the raw reality of the internet to a human mind, concessions need to be made. Some content needs to be promoted, other needs to be de-emphasized. An objective rendering is a pipe dream, even a fair random sample is a noisy incomprehensible mess.

search-engine on marginalia.nu

An NSFW filter for Marginalia Search

Index Compression, Query Execution Improvements

Trust in Ranking

New Search Filtering in Web and API

New Search Filtering GUI

Language Support for Marginalia Search

Faster Index I/O with NVMe SSDs

Finding Dead Websites

Table Of Contents

Profiling Websites

PDF to Text, a challenging problem

Crawl Order and Disorder

Marginalia Search receives second nlnet grant

Marginalia Search: 4 Years

RSS Feeds and Real Time Crawling

Notes on binary soup

Release Notes v2024.10.0

Phrase Matching in Marginalia Search

One year of solo dev, wrapping up the grant-funded work

Experiment in Java native calls

Query Parsing and Understanding

Deep Bug

The Yak Shave

Marginalia: 3 Years

Best SEO spam 2024 reddit

Release Notes v2024.01.0

A Frivolous Feature

WARC'in the crawler

Anchor Tags

Partitioning The Index

Moving Marginalia to a New Server

Release Notes v2023.10.0

Zero Downtime Upgrades and halved memory consumption

Marginalia Search receives FUTO Grant

Absurd Success

Release Notes v2023.08.0

Message Queues, State Machines, Actors, UI

Release Notes v2023.06.0

New Features

Generator keywords

Full Time

Release Notes v2023.03.2

New Features

File keywords

Going Github

Search Result Quality For Multiple Terms

The Grand Code Restructuring

Marginalia Search: 2 years, big news

Faster Index Joins

Creepy Website Similarity

About Marginalia Search

Carbon Dating HTML

Marginalia's Index Reaches 100,000,000 Documents

Fragments of the Old Web

Donate To This Project

The Evolution of Marginalia's crawling

Marginaliacoin, and hidden forums

Fun with Anchor Text Keywords

marginalia.nu goes open source

Uncertain Future For Marginalia Search

Lexicon Architectural Rubberducking

The Bargain Bin B-Tree

Growing Pains

Marginalia Search: 1 year

The Anatomy of Search Engine Spam

Can we unfuck internet discoverability?

Discovery and Design Considerations

Search Result Relevance

Old and New

A Jaunt Through Keyword Extraction

Shaking N-gram needles from large haystacks

The Mystery of the Ceaseless Botnet DDoS

Web Browsing

Getting with the times

Experimenting with Personalized PageRank

Astrolabe - The October Update

Against the Flood

New Solutions Creating Old Problems

The Curious Case of the Dot-Com Link Farms