Rendered at 23:09:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mryijum 5 hours ago [-]
2-4 weeks feels so absurd that I reread the alert twice thinking maybe they meant something different. Database problem into discovery of failed backups, perhaps?
It'll be interesting to see the postmortem. I don't know much about their infrastructure.
tzs 2 hours ago [-]
Well that's annoying. I was literally planning later today to go through the list of active stories I'm following at Fanfiction.net, see which ones are also posted on AO3, and switch to reading those on AO3 because FFN is falling apart.
As someone who is just a reader the main annoyance at FFN is that when a story gets a new chapter it is often hard to get it to actually show the chapter. Sometimes you can only get it if you try to go directly to it, sometime you can only get it if you go to it from the previous chapter. Sometimes you can't get it at all.
Also sometimes if you try to reread the next-to-last chapter to remind yourself of context before reading the new chapter it say the next-to-last chapter was not found. As with the last chapter sometimes you can get it if you approach from the right direction.
Sometimes it is even the last several chapters that get temporarily lost this way.
So far this eventually has cleared up on a given story but sometimes it takes weeks.
From what I've seen authors mention there are also several issues with its support for authors making them as frustrated and annoyed as us pure readers.
Recently I've also noticed even the browsing story categories is falling apart. E.g., go there and from the "Browse" menu pick books. On the books page pick "Harry Potter". Then when it lists the HP stories click the "Crossovers" link to see the HP crossovers. That gives an error, and has done so for quite a while.
peri-cl 5 hours ago [-]
Wonder if it's related to the Gitlab crisis (the regexp vulnerability in Ruby that hit them first). Ao3 seems to be a Ruby on Rails app.
mschuster91 5 hours ago [-]
Per reddit [1] the data is safe.
Apparently they run bare metal that they own [2] so I'd assume their servers got compromised hard and they'll need to buy entirely new servers.
The data's *always* been safe. /r/DataHoarders would never overlook this.
Nvidia will serve you a free, high-speed half-terabyte download right now[0]. It's one with a cutoff date of 2020—but given that GPT-3 was released mid-2020, that's probably the one you actually want, for ML research.
There are later dumps, but they aren't quite public
anonymous908213 5 hours ago [-]
> In necessity, we could even switch to the second server with very little downtime
So much for that...
mschuster91 5 hours ago [-]
When the threat model is "a fire takes out the DC", sure. But "the servers are root-level compromised", whoops, that's worth rebuilding from scratch. No way to make sure whatever AI agent did the compromise didn't go and embed malware in some flash ROM.
4 hours ago [-]
rafram 4 hours ago [-]
You seem to be jumping to a lot of conclusions here. Who said anything about any server being compromised? Much less root-level compromised? And an AI agent being involved?
mschuster91 4 hours ago [-]
> You seem to be jumping to a lot of conclusions here.
There aren't that many scenarios which make a restoration time of weeks inevitable, even after taking into account the AO3 people are volunteers - given the large intersection between nerds and fanfiction or a loooot of furries working in IT, I'd assume you'll find the most competent people working on it, so I doubt the cause is incompetence.
They say the data is safe, which implies an offsite backup with a different architecture (or a managed provider). They said in the past they had a secondary site which they could switch over to if needed, which implies that the issue is not related to hardware or datacenter problems (e.g. your usual backhoe related issue or a physical destruction of the datacenter).
And that leaves only two options: a) a former staff member went fully rogue and everything needs to be rebuilt because everything might be backdoored, but there is no indication of that, or b) they got compromised really hard by one of the multitude of the vulnerabilities that cropped up the last days and chose to rebuild from scratch for a similar reason.
(And as far as I know AO3, like FFN, is limited to text and stuff like cover art, so the dataset should be small and easy enough to restore from a backup)
Hamuko 4 hours ago [-]
Was it mentioned somewhere that they got compromised? And that it's not for example a hardware failure? I imagine lead times for getting brand new hardware would be in the 2–4 week range these days.
mschuster91 4 hours ago [-]
> And that it's not for example a hardware failure?
They say they have a second site, the chance of hardware failure taking out both sites at the exact same time is near zero.
Avamander 4 hours ago [-]
As I understand it, the second site will allow them to resume operating at a limited capacity sooner than 2-4 weeks?
Other than that 2-4 does sound more like hardware lead time rather than time it takes to resolve a security incident.
wmf 4 hours ago [-]
Imagine if there was an API to rent servers within minutes or someday even seconds.
Avamander 4 hours ago [-]
That speed comes with its own cost though. Might be simply too unreasonable at their scale.
altairprime 4 hours ago [-]
> Imagine if there was an API to rent servers within minutes or someday even seconds.
Imagine if everyone offering said API were capricious, inhumane, and uncaring about customer support, service terminations, and customer wind-down processes.
Now that you’ve pictured AWS, GCP, DigitalOcean, et al. more fully, that rules them out as available options for a long-term fanfic-centric erotica archive site that has specific and unusual legal concerns and threats to its existence incompatible with skittish Redbox-style compute providers.
Was there a particular cloud provider you were thinking of, that is not a ‘good night, sleep well, perhaps we’ll kill you in the morning’ service provider incompatible with AO3?
wmf 3 hours ago [-]
There are a few "Nazi" hosting providers that could probably handle AO3. Maybe AO3 couldn't handle them.
altairprime 3 hours ago [-]
> There are a few "Nazi" hosting providers that could probably handle AO3.
I mean, you’re welcome to supply ammunition against your own argument here — ‘every hosting provider with a server-provisioning API is either capricious or Nazi-welcoming’ is a pretty damning condemnation of your own point! But .. if that’s not your intent, then I don’t really understand what point you’re trying to make in support of these APIs, and pithy sentence replies aren’t doing you any favors in conveying an interpretable opinion about AO3, server provisioning APIs and checks notes Nazis(?).
Are you suggesting that AO3 is akin to Nazi content? Are you suggesting that non-skittish hosting providers are uniformly also unprincipled? Are you equating erotica and fascism? I’d prefer to hear your explanation in your own words rather than be forced to infer bad faith participation here.
wmf 2 hours ago [-]
non-skittish hosting providers are uniformly also unprincipled
Yeah, this is what I'm saying (although absolute free speech is itself a principle). I wish it weren't so, but you're 99% right that the cloud isn't really usable for anything outside the Overton window, whether it's far left, far right, erotica, pro-suicide, extreme trolling, etc.
However if you're using Cloudflare you can "hide" your backend hosting provider and they usually won't kick you out because they don't take any reputation hit. So I still suspect a mainstream cloud (with good off-cloud backups) could work for AO3.
altairprime 2 hours ago [-]
> I wish it weren't so
Same :(
mschuster91 2 hours ago [-]
> Are you suggesting that AO3 is akin to Nazi content? Are you suggesting that non-skittish hosting providers are uniformly also unprincipled? Are you equating erotica and fascism? I’d prefer to hear your explanation in your own words rather than be forced to infer bad faith participation here.
I think the point they're trying to make is that "normal" hosting providers like AWS, Azure or Google won't accept AO3 as a customer in the first place or yeet them the day the first DMCA email arrives or some Evangelicals yap about furries and they realize what their customer is doing. "Nazi" hosters (aka wherever Gab, Truth Social et al are hosted) or your average cybercrime "bulletproof hoster" are being used to deal with a lot of bullshit, or they just pipe any complaints straight to /dev/null - and they are used to dealing with cyberattacks of all kinds as well because they are a very, very juicy target for anyone from antifascist activists over cybercriminals wanting to cash in on Trump's Truth Social privileges to other nations' secret services that might want to sow chaos by manipulating Trump's Truth Social feed.
So... AO3 is nowhere near Nazi crap in terms of content, but a very, very similar threat model (exhaustion of customer support goodwill).
It'll be interesting to see the postmortem. I don't know much about their infrastructure.
As someone who is just a reader the main annoyance at FFN is that when a story gets a new chapter it is often hard to get it to actually show the chapter. Sometimes you can only get it if you try to go directly to it, sometime you can only get it if you go to it from the previous chapter. Sometimes you can't get it at all.
Also sometimes if you try to reread the next-to-last chapter to remind yourself of context before reading the new chapter it say the next-to-last chapter was not found. As with the last chapter sometimes you can get it if you approach from the right direction.
Sometimes it is even the last several chapters that get temporarily lost this way.
So far this eventually has cleared up on a given story but sometimes it takes weeks.
From what I've seen authors mention there are also several issues with its support for authors making them as frustrated and annoyed as us pure readers.
Recently I've also noticed even the browsing story categories is falling apart. E.g., go there and from the "Browse" menu pick books. On the books page pick "Harry Potter". Then when it lists the HP stories click the "Crossovers" link to see the HP crossovers. That gives an error, and has done so for quite a while.
Apparently they run bare metal that they own [2] so I'd assume their servers got compromised hard and they'll need to buy entirely new servers.
[1] https://www.reddit.com/r/AO3/comments/1xir586/comment/19rubn...
[2] https://www.reddit.com/r/AO3/comments/1xir586/comment/19ruc1...
Nvidia will serve you a free, high-speed half-terabyte download right now[0]. It's one with a cutoff date of 2020—but given that GPT-3 was released mid-2020, that's probably the one you actually want, for ML research.
[0] https://huggingface.co/datasets/ray0rf1re/AO3-2020
So much for that...
There aren't that many scenarios which make a restoration time of weeks inevitable, even after taking into account the AO3 people are volunteers - given the large intersection between nerds and fanfiction or a loooot of furries working in IT, I'd assume you'll find the most competent people working on it, so I doubt the cause is incompetence.
They say the data is safe, which implies an offsite backup with a different architecture (or a managed provider). They said in the past they had a secondary site which they could switch over to if needed, which implies that the issue is not related to hardware or datacenter problems (e.g. your usual backhoe related issue or a physical destruction of the datacenter).
And that leaves only two options: a) a former staff member went fully rogue and everything needs to be rebuilt because everything might be backdoored, but there is no indication of that, or b) they got compromised really hard by one of the multitude of the vulnerabilities that cropped up the last days and chose to rebuild from scratch for a similar reason.
(And as far as I know AO3, like FFN, is limited to text and stuff like cover art, so the dataset should be small and easy enough to restore from a backup)
They say they have a second site, the chance of hardware failure taking out both sites at the exact same time is near zero.
Other than that 2-4 does sound more like hardware lead time rather than time it takes to resolve a security incident.
Imagine if everyone offering said API were capricious, inhumane, and uncaring about customer support, service terminations, and customer wind-down processes.
Now that you’ve pictured AWS, GCP, DigitalOcean, et al. more fully, that rules them out as available options for a long-term fanfic-centric erotica archive site that has specific and unusual legal concerns and threats to its existence incompatible with skittish Redbox-style compute providers.
Was there a particular cloud provider you were thinking of, that is not a ‘good night, sleep well, perhaps we’ll kill you in the morning’ service provider incompatible with AO3?
I mean, you’re welcome to supply ammunition against your own argument here — ‘every hosting provider with a server-provisioning API is either capricious or Nazi-welcoming’ is a pretty damning condemnation of your own point! But .. if that’s not your intent, then I don’t really understand what point you’re trying to make in support of these APIs, and pithy sentence replies aren’t doing you any favors in conveying an interpretable opinion about AO3, server provisioning APIs and checks notes Nazis(?).
Are you suggesting that AO3 is akin to Nazi content? Are you suggesting that non-skittish hosting providers are uniformly also unprincipled? Are you equating erotica and fascism? I’d prefer to hear your explanation in your own words rather than be forced to infer bad faith participation here.
Yeah, this is what I'm saying (although absolute free speech is itself a principle). I wish it weren't so, but you're 99% right that the cloud isn't really usable for anything outside the Overton window, whether it's far left, far right, erotica, pro-suicide, extreme trolling, etc.
However if you're using Cloudflare you can "hide" your backend hosting provider and they usually won't kick you out because they don't take any reputation hit. So I still suspect a mainstream cloud (with good off-cloud backups) could work for AO3.
Same :(
I think the point they're trying to make is that "normal" hosting providers like AWS, Azure or Google won't accept AO3 as a customer in the first place or yeet them the day the first DMCA email arrives or some Evangelicals yap about furries and they realize what their customer is doing. "Nazi" hosters (aka wherever Gab, Truth Social et al are hosted) or your average cybercrime "bulletproof hoster" are being used to deal with a lot of bullshit, or they just pipe any complaints straight to /dev/null - and they are used to dealing with cyberattacks of all kinds as well because they are a very, very juicy target for anyone from antifascist activists over cybercriminals wanting to cash in on Trump's Truth Social privileges to other nations' secret services that might want to sow chaos by manipulating Trump's Truth Social feed.
So... AO3 is nowhere near Nazi crap in terms of content, but a very, very similar threat model (exhaustion of customer support goodwill).
Their response is here and provides the desired clarification: https://news.ycombinator.com/item?id=50295919
Related:
Archive of Our Own
https://news.ycombinator.com/item?id=50128197
Fans are better than tech at organizing information online (2019)
https://news.ycombinator.com/item?id=43137627