Base64 would cause an increase in data being sent, which is something we are against doing if possible. There are also ad filters that block Base64 encoded images from being displayed on our site. I believe the numbers were 23% increase in data being sent the last I looked at it.

Let's imagine for instance, a user has 3000 favorites, we load them on one page. We include user, score, rating, and tags for filtering. We would serve an additional 23% of bandwidth to them for each image, not including the text information. or about 2.3KB. Multiply that by 3k. That's 6.7MB EVERY visit to their favorites page. And that's just the increase in bandwidth. This is assuming the base64 can not be cached on their browser, which I believe is still the case. If we include the text, it would be an additional 200B-1KB of text, per image on average. So let's just go with 200B. An additional 585KB of text data. The original file size of the data would be ~29MB of data. So, it would be highly unreasonable to implement that option.

If this were accessed on a mobile device, or a computer with low memory, RIP browser. RIP mobile bandwidth limitations, which we attempted to mitigate in the first place. RIP user experience. Javascript is fast, but I'm not certain it would be fast enough to process quickly on mobile, or even on a desktop to filter images in real-time in such a large quantity with a real world set of unique tags.

I wonder if a hidden tag system would work.

Basically a second set of tags for an image that is only accessed by the pages that call for them. Every time I favorite a post a fav_jedi1357 tag is added to it and this tag is removed if/when I un-favorate it.

The favorites page would be unique for each user and is basically a search for this tag using the regular tag search page. It can be filtered the same as any other tag search.

lozertuser,

Jerl already explained to me why the favorites idea (handling it all in client) wouldn't be feasible. Memory concerns, etc. It's clear now that it would have to be implemented server side.

When I mentioned inlining thumbnails as base64, it was in reference to the way main page paginated browsing currently works, so like 42 being inlined to the page at a time, not 3000. It would be up to you guys what hurts more... 23% more bandwidth usage or having 43 requests per page view (page then one per thumbnail) versus just 1 (page with inlined thumbnails). You guys know your system more than I do.

I was completely unaware that some ad blockers block base64 images. :( That's unfortunate. But then again, at least you have a warning on the page that ad blockers should be disabled anyways because of the other problems they cause on this site. I already have Gelbooru whitelisted.

jedi1357,

The ability to filter favorites by tag is already possible the way the data is arranged in the database. The fact that the user has favorited it is already saved in the database. Implementing it would just involve copy and pasting the query used to find images right now and adding a bit to it to filter these results down to just the user's favorites.

I'm making a few assumptions here about how the devs have organized the database but I have worked on enough systems before to know pretty confidently how the database tables are likely arranged and how this would be coded. Such a query would be a bit more resource intensive on the database server, so it would be up to the devs if they wanted that feature added.

I was told that the searcher does not use the DB but has its own index separate from the main DB. The API doesn't currently use the search function when populating a user's favorites page.

This is correct. We do not use a SQL query at all for searches. We use Apache Solr to handle searches, which only stores post information. Favorites are stored in their own table, which Solr does not cache.

I'm starting to see now why adding this on the backend was considered so difficult.

I'm not familiar with Apache Solr but I am a bit familiar with Elasticsearch. I know an interesting feature it has is to separate data out into distinct indexes. So, a user's favorites posts could be copied into an index dynamically created just for that user, storing just their favorites. This would then be searchable the same way the main posts are searchable. However, this would less to duplication of data and all the problems that entails. I can't think of any solution that doesn't have scalability problems.

Were the performance benefits of moving the posts into a separate data store really worth it compared to implementing searching in SQL? I thought technology like Apache Solr and Elasticsearch were meant for text search, not tag filtering like Gelbooru's search functionality.

Tag filtering on Gelbooru is text search. A post's tags are stored as a string in a single cell in their database row, and likewise in Solr as a single field.

The performance benefits are phenomenal. If we weren't using Solr, the site would be incredibly slow right now.

So I don't really know anything about coding, but this wouldn't even be possible/easy to do in the general search bar, similar to how you can filter for score or rating?

No, because apparently the posts' tag data is split up from the users' favorites data. That means it's trickier to get them to work together. Jerl states that they did this to increase performance for the main part of the site. So this is an example of a situation where the needs of the many outweigh a minority of users needing a niche feature.

Jerl, again I'm not familiar with Apache Solr's query syntax, so I'm sorry if I sound ignorant here, but is there no way to include a set of post IDs in the Solr query to filter the results down to? I'm drawing parallels to MongoDB's aggregation pipeline here... Aka first grab the set of IDs from the database storing user data, then use those as part of a query to Solr saying "find posts that match these tags BUT limited to IDs which are among the set X", without a need to do this filtering outside of Solr itself (which would obviously have memory problems).

I don't know much about how Solr works to know if that'd be possible or easy. I've only seen now it's currently implemented in our backend.

Either way, we'd need someone who knows a lot about Solr for that.