Rendered at 22:33:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
badsectoracula 8 hours ago [-]
> Amodei, meanwhile, outlined some very specific things he wants from Washington: Stop selling Nvidia chips to China so its AI companies can’t build better models, and a crackdown on model distillation.
You have to be the embodiment of hubris to believe the only reason a technologically advanced country of 1.4 billion people with several AI labs and government support can make competitive models is because they all distill yours.
amluto 8 hours ago [-]
It’s not only hubris — I think it’s actively counterproductive. The US could choose to:
(a) Sell many billions of dollars of fancy chips to China, thus bringing in billions of dollars of money and billions of dollars of trade balance improvement. And let China continue to build models quickly when they seem oddly happy to export the trained weights for free.
Or
(b) Decline to do so, thus reducing exports and very very strongly encouraging China to do everything in their power to avoid needing to depend on our chips.
In exchange for (b), what do we gain? A temporary advantage in model training and availability?
ygjb 13 minutes ago [-]
Don't forget that when they increase export restrictions under the current US regime they signal further that economic ties are on their terms not fair terms. The rest of the world will continue to steer towards alternatives to US tech.
superxpro12 7 hours ago [-]
This admin is super in favor of isolationist policies right now. Whatever can bring in the most bribes. To the detriment of 99% of the public.
toasty228 8 hours ago [-]
They have such a weird and outdated view of China, it's almost comical. It's like America's answer to everything bad coming to them is protectionism. They still think China is a third world country with a bunch of factories and polluting industries not able to do anything else other than manufacturing products for foreign companies
rglover 8 hours ago [-]
China played the smart long game and took advantage of America's ignorance and short-sighted thinking. They already "won" in the sense that they outmaneuvered the U.S. because they relied on flaws in human nature (specifically, greed). Their moves around AI are just more of the same clever behavior.
I don't agree with all of their means for getting here, but to accomplish what they have in ~30 or so years [1] should blow minds far more than it does in the West. And they did it by convincing us to give them control of our manufacturing and selling us (literal) boatloads of cheap junk. Historians will look back on this era as one of the greatest demonstrations of "winning without firing a single shot."
This whole insular POV is tired, ignorant, and frankly just another indicator that America has lost the plot, shit-faced on its own arrogance.
China deliberately adopted western style market institutions. They joined the WTO. No, they aren't a fully westernized market, but they are closer to a competitive marketplace than the state dictating all of what will or will not be built.
I'm confused, what has America and the west lost the plot on? You would have preferred the west to exploit China more?
This framing of a zero sum game is counterproductive.
rglover 7 hours ago [-]
> This framing of a zero sum game is counterproductive.
I agree but that's not at all what I was getting at. Quite the opposite. The mistake America made, what made it vulnerable, was playing everything like a zero sum game. We're in the early days of that behavior coming to roost.
altcognito 7 hours ago [-]
No, that is exactly what you were getting at:
> China played the smart long game and took advantage of America's...
Zero sum games imply that if there is a winner, there has to be a loser. That one country takes advantage of another.
rglover 6 hours ago [-]
That's not my framing, that's literally what happened/is happening.
black6 8 hours ago [-]
Unrestricted Warfare by Colonels Xiangsui and Liang is a must read. China's been on the path for quite a while.
lioeters 6 hours ago [-]
> The English translation of the book was then published by a previously unknown Panamanian publisher, with the subtitle "China's Master Plan to Destroy America" and a picture of the burning World Trade Center on the cover.
Hmm, sounds awfully sensationalist, designed to fan the flames of Sinophobia and thinly veiled racism.
superxpro12 7 hours ago [-]
its the natural conclusion of raw unregulated capitalism. Without guiderails, we offshored all production (because it was cheaper), and we suffered a complete loss of manufacturing workforce. Now that Trump has gone full protectionist, we have nothing with which to manufacture from. It's beautifully ironic in a way, in that our greed will be our downfall.
Of course china isnt innocent in all of this. Their 996 work culture is inhuman, and basically indentured service. But, their government is able to leverage that workforce.
cucumber3732842 6 hours ago [-]
>we offshored all production (because it was cheaper),
And why was it cheaper? Shipping things from the literal other side of the world isn't free. Cultural and language barriers to efficient communication aren't free. Etc. etc.
rayiner 8 hours ago [-]
Your first point is correct, but your second point doesn't follow. Lack of protectionism is based on the notion that the U.S. will always be technologically one step ahead of China because China can only copy. That's what has gotten us to where we are. Note that China itself is a protectionist economy. So in some sense protectionism is recognizing that China has a pretty good industrial policy and copying it.
overfeed 6 hours ago [-]
> Note that China itself is a protectionist economy. So in some sense protectionism is recognizing that China has a pretty good industrial policy and copying it.
China's protectionism is emplaced to support long-term goals. Amodei's policy suggestion is explicitly in support of short term goals (1-5 years), a time window he claims to be "critical" without evidence. A cynic may link that window to forthcoming AI IPO's, and allowing current CEOs to secure ridiculous golden parachutes before leaving the mess of how to deal with the unpaced Chinese frontier models to their successors and future US legislature.
Letting AI companies set your foreign policy (voluntarily ceding global primacy to maximize domestic[1] profits) may have awful higher-order effects for the US. What's hilarious is the shameless about-face: for years, the same companies have been declaring that it is vital American AI innovation not be encumbered by new laws. Until the Chinese models glided to the frontier.
1. Maybe they hope to browbeat Europe too, like how they pushed out Huawei equipment from Euro telecoms. An alternative reading is that the loss of primacy is involuntary, so they may as well spin the reality, and salvage a win by colluding to eliminate better, open-weight models from the markets in "democracies" with the excuse that they are unpaced and therefore dangerous. What will predictibly happen is there will be `Qwen-8.1` for RoW, and a lobotomized `Qwen-8.1-paced` for Americans.
rayiner 4 hours ago [-]
Yeah, I agree with you on that point.
scottLobster 8 hours ago [-]
The alternative to protectionism is Chinese-style industrial policy. Which necessitates Chinese-style "we don't care if you're a billionaire, you will do as we say or men with guns will make you" levels of governmental control. Which I'm not sure any of us really want, as satisfying as it would to watch these CEOs get publicly re-educated.
They aren't patriotic and they don't really care about competing with China, they fancy themselves princes and are playing on the silents/boomers in charge being irreparably stuck in the Cold War.
oceanplexian 8 hours ago [-]
It goes both ways. It has been almost 10 years since I visited the PRC but the air and water were heavily polluted with industrial waste, outside the tier 1 cities people were living in makeshift subsistence tin housing, and it wasn’t uncommon to see farmers using oxcarts.
There’s a reason China has a GDP per capita somewhere around that of Argentina.
Of course, they also have futuristic cities, maglev, and plenty of brilliant engineers and scientists, therefore it makes sense to be pragmatic. But there are a lot people talking up China because they hate the US and are bought into a massive propaganda / influencer campaign.
toasty228 7 hours ago [-]
> outside the tier 1 cities people were living in makeshift subsistence tin housing
Having visited the US a few times it felt that way too
rayiner 8 hours ago [-]
Underestimating China's capability to make homegrown technology is a mistake I could understand 20 years ago. Now it's just a sign of poor judgment.
sschueller 8 hours ago [-]
Also, distillation means paying US AI firms api fees. Do they want to loose that source of income as well?
LiquidSky 6 hours ago [-]
It's funny because this is EXACTLY how it played out with Japan in the mid-20th century. "Japan can only make cheap copies of our stuff!" They even joke about it in Back to the Future.
lioeters 6 hours ago [-]
People joke how we live in Idiocracy now, but Back to the Future really nailed it with the dystopian world ruled by Biff the billionaire bully with no class or taste.
bpodgursky 8 hours ago [-]
Nobody thinks a distillation crackdown is a long term plan, but it does delay Chinese labs for the couple years while they are compute-starved. Of course they could make competitive models if they were drowning in NVIDIA chips but they aren't, and the native ones aren't at scale production for at least a year.
toasty228 8 hours ago [-]
If anything it'll speed up their progress by forcing local solutions.
> According to Bloomberg, supply bottlenecks for high-end memory and other key components could limit chip production this year. Since Huawei also has to serve other customers, DeepSeek might have to wait over a year for the full delivery, estimates Bloomberg.
Like I said, it will delay the Chinese labs for a couple years. These are not even top-line chips.
Frankly, China was not going to allow them to depend on NVIDIA forever, I don't think this motivates domestic production that much over the counterfactual. If NVIDIA didn't have export controls, China was going to set up formal import controls. They want to own their entire supply chain.
toasty228 7 hours ago [-]
They'll build five data centers by the time that 1m wide pot hole on your local highway ramp gets a (half assed) fix.
bpodgursky 7 hours ago [-]
They will build data centers but not have the chips to put in them until 2028. I don't know why you're arguing about things I'm not saying.
lightedman 8 hours ago [-]
"Like I said, it will delay the Chinese labs for a couple years."
Not even that long. China can simply order its industries to stop supplying externally and go full-domestic. It has happened before in others parts of Chinese industry; it will happen again. As it stands China can easily build computational nodes at scale, and their LineShine supercomputer holds top place in the Supercomputer Top 500 at almost 2.2 exaflops. They have zero issues building performant hardware.
Couple of years? Their current technology level can make that less than a month.
benxh 8 hours ago [-]
At the current pace of iteration from Deepseek I think this is a moot point, they'll keep building their own RL environments, and just keep RL on top of whatever flavor of model they can train/host on their Huawei SuperPods
giantrobot 8 hours ago [-]
But they're not compute starved. NVidia's kit is convenient for compute but not absolutely necessary. Chinese labs have been training with Huawei chips already. They also have a huge supply of existing Nvidia gear to use. Just because it's not the absolute latest generation isn't really important.
Also as we have seen, the Chinese labs will just buy their kit through cut outs or do training in locations where there's no embargo.
dfxm12 8 hours ago [-]
I don't think Amodei is dumb. He just doesn't want to compete for the chips. Sinophobia sells right now, so this is the angle he's going with.
DrProtic 7 hours ago [-]
I was on edge for last few months, but made a decision recently to not give another dollar to Anthropic.
It’s so obvious they’re playing safety card to get politicians to reign down on individual’s freedoms.
password54321 6 hours ago [-]
If it was that serious, we would be seeing a lot more action than a tweet and a blog post. I'm not even sure why the public needs to know about how "scary" AI is as we can't do much, don't know any of the technical details and Anthropic have already decided to punish anyone involved in anti-ai protests.
"Anthropic Is Building a Huge Surveillance System to Spy on Anti-AI Activists and Predict Their Activities"
All these companies are suddenly crying about "slowing down" after insisting on having no guardrails nor restrictions?
All at once in lockstep?
Who paid them to say this? These are all lies.
Arcuru 5 hours ago [-]
I can't find the reference right now, but I'm certain that Anthropic has been asking for a coordinated slowdown for at least 4 or 5 months. I think it was around their Mythos/Fable releases this spring.
bix6 9 hours ago [-]
Can anyone recommend me a viable alternative to these clowns? I would like to move my work (excel and light coding) off the big guys. My computer isn’t very powerful though so I need to upgrade or find hosted models (host my own?) that won’t steal my sensitive data?
Edit: miss me with the downvotes. These guys are clowns. I want alternatives. Thank you to everyone offering help!
marcuskaz 9 hours ago [-]
OpenRouter or Fireworks are two of many that allow you to run numerous different models. You can use open code harnesses like Pi or Opencode that connect easily to these services.
Thank you! Is there one you’d recommend in particular when working with sensitive data?
threecheese 9 minutes ago [-]
Depending on what your threat model is, and how much you trust OpenRouter and their upstream providers, they offer zero data retention (googleable: ZDR) APIs which come at a higher cost. You can find other zdr providers as well.
tolugenius 9 hours ago [-]
You might want to consider a local setup? Personally I do model + llama.cpp + Pi + tmux, that's as barebones as you can get and everything is great. I also use opencode for bigger task, I could probably hook that to a local model but I'm not sure (still new to this). But if you work with sensitive data you need a local setup purely.
bix6 8 hours ago [-]
I am interested in a local setup but whenever I read about it, it seems the money is better spent on cloud inference since even with thousands of dollars of my own hardware I wouldn’t have frontier like capability? I am bad at choosing models though (I just throw everything at Opus atm).
tolugenius 8 hours ago [-]
Well yes you won't have frontier capability but that doesn't mean you'd have nothing, I'm not sure what you have but I know there are tools that let you see what models (and their quant variants) you can safely run, I'd look into that first and see what options you have and what you're willing to delegate to a smaller model vs use opus for. Maybe opus to plan vs smaller models for focused implementation, or what works for you workflow and needs.
andy99 6 hours ago [-]
Openrouter lets you pick your provider and see their policy re retention and training. You can pick US providers e.g. digitalocean that don’t retain or train on your data. Openrouter is also now owned by stripe, so I don’t see any reason why using them with a trusted provider is riskier than using the big two (especially in light of some of the whole Navier Stokes thing)
If it’s really sensitive then don’t use a cloud provider.
jmtulloss 9 hours ago [-]
I’m biased (my wife works there) but Fireworks or another US-based inference provider is going to be a better fit than OpenRouter. Fireworks has zero data retention by default[1] and has tools to make it easy to run your existing harness against open models [2]
OpenRouter isn’t a provider, they route to other providers, so you would need to specify which ones you’re comfortable with anyway.
I think this would depend upon the level of sensitive data.
For OpenRouter, you can setup an API key and limit it to only models that claim to not train on data, but that is just a claim. You can then use trust to judge which providers will honor that claim.
But if the data is really sensitive, you might want either a local model or a business subscription with some big name in the US that legally promises no data training.
So are we talking some app idea you are playing around with, or files filled with PHI/PII that you have legal mandates to safeguard? If the latter, I would stick to only provider with enterprise agreements to not store/train on the data. Even the ones who promise no training are likely storing the data for monitoring for abuse or such short term.
bix6 8 hours ago [-]
For anything sensitive I have Claude business (team?) which (claims) no training but I would like to find an alternative for them. I’m just really tired of supporting Anthropic each month (same feelings for all the major providers).
Roark66 8 hours ago [-]
Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models.
I'm very happy with Deepinfra. Less model coverage, but good prices and quite fast.
However I have to caution you about one thing.
No one will give you as many input tokens for so little money as Claude Max x5 (maybe x20 too, I use x5).
I tend to use 1.1B to 1.4B a week about 0.8-1B cached. Even with cache were talking thousands of $ in API prices a week. Hundreds if we're talking cheap cloud like Deepinfra.
However, local AI well setup is actually a good alternative for this if Claude Max was unavailable.
For example my system a ryzen 7950x 192GB ram, 5x rtx3090 plus an rtx5060 ti 16gb. (3 rtx3090 cards via usb4 egpu dock). Let's me run Qwen3.8-Flash-Next with 3slots (no rtx5060 used) at 55tok/s decode dropping to 50 at the end of a 260k context, 1200tok/s refill dropping to 950 at the end of context.
With RAM and ssd cashing and 80% cache were talking on the order of 4B a week could be ingested by this setup (roughly) if it was running 24/7. I found 6 interactive cloud code sessions are fairly pleasant with this 3 user setup.
If I include the rtx5060 in the mix I can bump to 5 users, but it slows down by about 15% (note the speeds are give are for one active user, multiple users at once see maybe 70% of tgat per user so aggregate is much higher in multi user setup).
So in theory I should be able to run 10 cloud code sessions. Although I'm testing CC alternative now (pi with own plugins) because this model, while multimodal has only 260k context 30k of which CC eats on the getgo.
Many people say local AI makes no sense financially. But in the event you process huge inputs that are often cached it does make sense.
Ciantic 8 hours ago [-]
> Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models.
OpenRouter allows to make an API key locked to certain provider, I do that myself.
bix6 8 hours ago [-]
I haven’t heard of Deepinfra so I’ll check them out!
I think your local setup would be a bit much for me capability / price wise. But maybe I can find a scaled down version. I don’t need insane tok/s. Oftentimes I just let things run and come back later.
cmiles8 9 hours ago [-]
A lot of companies are starting to run towards open models on walled off infra. There might be a moat to be had in really good infra-as-a-service for hosting models, but there’s zero moat to be had in the models themselves. Which is a problem for those trying to convince people a proprietary model company should be worth trillions.
bix6 8 hours ago [-]
I am interested in running my own infra but the cost doesn’t seem worth it? Have you setup something local that’s viable?
dgellow 9 hours ago [-]
That’s an advantage, not a moat
myaccountonhn 8 hours ago [-]
Opencode go for 10$ a month is fantastic. Just use it with Glm5.2-flash.
akazantsev 8 hours ago [-]
It's better to buy models directly from providers that guarantee the quality of their quantization. Otherwise, model quality is a gamble. You can pay the same $10 on Novita or with any other provider.
This is crazy but honestly very tempting at this point.
emptybits 7 hours ago [-]
Four ostensibly competing billionaires agree with each other and want worldwide government regulations around their moneymaking? o_O
Chilling and suspicious. Of course, this will lead to a modicum of safety in some provable way to justify (in uncritical minds) the rest of the implications: fewer freedoms and more surveillance for everyone else.
lioeters 5 hours ago [-]
Alignment for the good of humanity has completely failed. Now one of the most powerful technology of our time - and the reins of the (arguably) most powerful nation - is in the hands of sociopaths who want to rule the world and decide the evolutionary course of future generations.
jens_tlb 9 hours ago [-]
[dead]
bbor 9 hours ago [-]
[dead]
mcshicks 8 hours ago [-]
I think there is an obvious example in the FAA of a successful safety regulation of another potentially dangerous technology industry. A similar agency would seem to me like a logical starting point. I did some minor work a long time ago in commercial avionics software and while it was a very tedious process I did have confidence in it. Obviously a lot of things would have to be different but the basic structure of development and testing standards, record keeping, transparency, and government incident investigation seems like the place to start.
lopsotronic 26 minutes ago [-]
"Giant Metal Things Hucked Through the Air Across Ocean" are of a quantitatively higher risk profile than "New Computer Thing", no matter how you slice that particular salami.
If you tossed the aviation regulatory framework out the window, you have maybe ten years before aviation reverts to high end hobbyists[1], due to the escalating risk profile.
[1] or proletarian desperation, of course.
ultrarunner 8 hours ago [-]
And what has the FAA done to both costs and innovation?
mcshicks 7 hours ago [-]
I'm getting on a plane tomorrow. I think I would prefer a world with the FAA than one without it.
LiquidSky 6 hours ago [-]
Move fast and break passengers
superxpro12 7 hours ago [-]
Not enough if you ask those 600 people Boeing killed from failing to disclose MCAS.
You have to be the embodiment of hubris to believe the only reason a technologically advanced country of 1.4 billion people with several AI labs and government support can make competitive models is because they all distill yours.
(a) Sell many billions of dollars of fancy chips to China, thus bringing in billions of dollars of money and billions of dollars of trade balance improvement. And let China continue to build models quickly when they seem oddly happy to export the trained weights for free.
Or
(b) Decline to do so, thus reducing exports and very very strongly encouraging China to do everything in their power to avoid needing to depend on our chips.
In exchange for (b), what do we gain? A temporary advantage in model training and availability?
I don't agree with all of their means for getting here, but to accomplish what they have in ~30 or so years [1] should blow minds far more than it does in the West. And they did it by convincing us to give them control of our manufacturing and selling us (literal) boatloads of cheap junk. Historians will look back on this era as one of the greatest demonstrations of "winning without firing a single shot."
This whole insular POV is tired, ignorant, and frankly just another indicator that America has lost the plot, shit-faced on its own arrogance.
[1] https://en.wikipedia.org/wiki/Historical_GDP_of_China
I'm confused, what has America and the west lost the plot on? You would have preferred the west to exploit China more?
This framing of a zero sum game is counterproductive.
I agree but that's not at all what I was getting at. Quite the opposite. The mistake America made, what made it vulnerable, was playing everything like a zero sum game. We're in the early days of that behavior coming to roost.
> China played the smart long game and took advantage of America's...
Zero sum games imply that if there is a winner, there has to be a loser. That one country takes advantage of another.
Hmm, sounds awfully sensationalist, designed to fan the flames of Sinophobia and thinly veiled racism.
Of course china isnt innocent in all of this. Their 996 work culture is inhuman, and basically indentured service. But, their government is able to leverage that workforce.
And why was it cheaper? Shipping things from the literal other side of the world isn't free. Cultural and language barriers to efficient communication aren't free. Etc. etc.
China's protectionism is emplaced to support long-term goals. Amodei's policy suggestion is explicitly in support of short term goals (1-5 years), a time window he claims to be "critical" without evidence. A cynic may link that window to forthcoming AI IPO's, and allowing current CEOs to secure ridiculous golden parachutes before leaving the mess of how to deal with the unpaced Chinese frontier models to their successors and future US legislature.
Letting AI companies set your foreign policy (voluntarily ceding global primacy to maximize domestic[1] profits) may have awful higher-order effects for the US. What's hilarious is the shameless about-face: for years, the same companies have been declaring that it is vital American AI innovation not be encumbered by new laws. Until the Chinese models glided to the frontier.
1. Maybe they hope to browbeat Europe too, like how they pushed out Huawei equipment from Euro telecoms. An alternative reading is that the loss of primacy is involuntary, so they may as well spin the reality, and salvage a win by colluding to eliminate better, open-weight models from the markets in "democracies" with the excuse that they are unpaced and therefore dangerous. What will predictibly happen is there will be `Qwen-8.1` for RoW, and a lobotomized `Qwen-8.1-paced` for Americans.
They aren't patriotic and they don't really care about competing with China, they fancy themselves princes and are playing on the silents/boomers in charge being irreparably stuck in the Cold War.
There’s a reason China has a GDP per capita somewhere around that of Argentina.
Of course, they also have futuristic cities, maglev, and plenty of brilliant engineers and scientists, therefore it makes sense to be pragmatic. But there are a lot people talking up China because they hate the US and are bought into a massive propaganda / influencer campaign.
Having visited the US a few times it felt that way too
https://www.heise.de/en/news/DeepSeek-orders-160-000-Huawei-...
Like I said, it will delay the Chinese labs for a couple years. These are not even top-line chips.
Frankly, China was not going to allow them to depend on NVIDIA forever, I don't think this motivates domestic production that much over the counterfactual. If NVIDIA didn't have export controls, China was going to set up formal import controls. They want to own their entire supply chain.
Not even that long. China can simply order its industries to stop supplying externally and go full-domestic. It has happened before in others parts of Chinese industry; it will happen again. As it stands China can easily build computational nodes at scale, and their LineShine supercomputer holds top place in the Supercomputer Top 500 at almost 2.2 exaflops. They have zero issues building performant hardware.
Couple of years? Their current technology level can make that less than a month.
Also as we have seen, the Chinese labs will just buy their kit through cut outs or do training in locations where there's no embargo.
It’s so obvious they’re playing safety card to get politicians to reign down on individual’s freedoms.
"Anthropic Is Building a Huge Surveillance System to Spy on Anti-AI Activists and Predict Their Activities"
https://futurism.com/artificial-intelligence/anthropic-surve...
All at once in lockstep?
Who paid them to say this? These are all lies.
Edit: miss me with the downvotes. These guys are clowns. I want alternatives. Thank you to everyone offering help!
https://openrouter.ai/
https://fireworks.ai/
http://pi.dev/
https://opencode.ai/
If it’s really sensitive then don’t use a cloud provider.
OpenRouter isn’t a provider, they route to other providers, so you would need to specify which ones you’re comfortable with anyway.
[1]:https://docs.fireworks.ai/guides/security_compliance/data_ha...
[2]:https://fireworks.ai/nexus
For OpenRouter, you can setup an API key and limit it to only models that claim to not train on data, but that is just a claim. You can then use trust to judge which providers will honor that claim.
But if the data is really sensitive, you might want either a local model or a business subscription with some big name in the US that legally promises no data training.
So are we talking some app idea you are playing around with, or files filled with PHI/PII that you have legal mandates to safeguard? If the latter, I would stick to only provider with enterprise agreements to not store/train on the data. Even the ones who promise no training are likely storing the data for monitoring for abuse or such short term.
I'm very happy with Deepinfra. Less model coverage, but good prices and quite fast.
However I have to caution you about one thing.
No one will give you as many input tokens for so little money as Claude Max x5 (maybe x20 too, I use x5).
I tend to use 1.1B to 1.4B a week about 0.8-1B cached. Even with cache were talking thousands of $ in API prices a week. Hundreds if we're talking cheap cloud like Deepinfra.
However, local AI well setup is actually a good alternative for this if Claude Max was unavailable.
For example my system a ryzen 7950x 192GB ram, 5x rtx3090 plus an rtx5060 ti 16gb. (3 rtx3090 cards via usb4 egpu dock). Let's me run Qwen3.8-Flash-Next with 3slots (no rtx5060 used) at 55tok/s decode dropping to 50 at the end of a 260k context, 1200tok/s refill dropping to 950 at the end of context.
With RAM and ssd cashing and 80% cache were talking on the order of 4B a week could be ingested by this setup (roughly) if it was running 24/7. I found 6 interactive cloud code sessions are fairly pleasant with this 3 user setup.
If I include the rtx5060 in the mix I can bump to 5 users, but it slows down by about 15% (note the speeds are give are for one active user, multiple users at once see maybe 70% of tgat per user so aggregate is much higher in multi user setup).
So in theory I should be able to run 10 cloud code sessions. Although I'm testing CC alternative now (pi with own plugins) because this model, while multimodal has only 260k context 30k of which CC eats on the getgo.
Many people say local AI makes no sense financially. But in the event you process huge inputs that are often cached it does make sense.
OpenRouter allows to make an API key locked to certain provider, I do that myself.
I think your local setup would be a bit much for me capability / price wise. But maybe I can find a scaled down version. I don’t need insane tok/s. Oftentimes I just let things run and come back later.
This is crazy but honestly very tempting at this point.
Chilling and suspicious. Of course, this will lead to a modicum of safety in some provable way to justify (in uncritical minds) the rest of the implications: fewer freedoms and more surveillance for everyone else.
If you tossed the aviation regulatory framework out the window, you have maybe ten years before aviation reverts to high end hobbyists[1], due to the escalating risk profile.
[1] or proletarian desperation, of course.