How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.
The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
So here’s a little bit of lemmy lore for you. You’re instance lemmy.ml is considered to be a tankie instance by some users. lemmy.dbzer0.com is an anarchist instance. The user you responded to probably made a generalisation based on this and assumed you were familiar with anarchist/communist/socialist/leftist discourse. From this comment I assume they were wrong.
Also a little bit about authority: people use authority to mean many things (this is even bought up in the video I linked above). But as far as anarchists are concerned (in general (no specific statement can be made about a group so vast)) authority is the act of coercing people to follow orders or commit involuntary acts. You’re boss can coerce you to neglect your health by threatening to fire you. Your government can force you to obey gender roles by threatening to jail you. A rich person can make you do whatever demeaning thing they want by dangling money in front of you (for reference see mrbeast) because otherwise your landlord will kick you out. This is authority and it is wrong. Those in authority can make mistakes, become greedy and start to think they have the power to do whatever they want (mostly because they can). This leads to suffering. My meaning of life is to minimise suffering. Anarchy is the belief that no-one should hold power over others. That all leadership should be scrutinised. It rejects blind faith in single people and encourages to think for yourself so no-one can do you wrong. And if you can’t be bothered, it encourages you to find people who genuinely care about you and let them stand up for you.
Thanks, I don’t really care for generalizing instances and didn’t really have a choice when I made my account.
But also your definition is impossibly broad as you well and I’m pretty sure not the general consensus. The video doesn’t define it as such either.
For one thing, by your definition we can have absolutely no meaningful human relationships. I can explain this more later when I have time if you don’t see what I mean
My explanation of authority wasn’t meant as a definition but rather a brief summarisation of a complex concept, Andrew does a better job actually explaining it. Like pointing out that Authority is confused with a lot of different concepts like respect or influence. Which I’m starting to suspect is happening here. meaningful human relationships are based on mutual respect. This is not authority as it is voluntary, reciprocated and revoked as soon as the other party steps over the line. This is what I believe is the basis of society and what we need to return to in order to live a truly free life. In modern society in most interactions respect has been replaced with authority. People in positions of power even use them synonymously.
That’s not really relevant here. This is more of a “genie is out of the bottle and now we have to learn how to deal with it situation”. The idea and technology of bots and AI training already exists. There’s no socioeconomic system that is going to magically make that go away.
I think the point you’re missing is that without the monetary incentive that arises under capitalism, there would be very little drive for anyone to build these wasteful AI systems. It’s difficult to imagine a group of people voluntarily amassing and then using the resources necessary for “AI” absent the desire to cash in on their investment. So you’re correct that an alternative economic system won’t “magically” make LLMs go away. I think it unlikely, however, that such wasteful nonsense would be used on any meaningful scale absent the perverse incentives of capitalism.
It’s difficult to imagine a group of people voluntarily amassing and then using the resources necessary for “AI” absent the desire to cash in on their investment.
He is not wrong. Unless people start to take steps , the dependency of tech will be used to chain most of us. Granted, these chains will be the kindest and gentlest chains seen in a long time.
Social revolution lives on in decentralized services, like this; the true battles will be later though. This year is a mild warm up. I can’t imagine the challenges that await many
Especially since the solution I cooked up for my site works just fine and took a lot less work. This is simply to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them. This is considerably simpler, and doesn’t require an entire nuclear plant powered AI to combat the opposition’s nuclear plant powered AI.
In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.
Fuck 'em. Route those motherfuckers right to /dev/null.
and try to slam your site with like 200+ requests per second
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
It’s worked alright for me. Your mileage may vary.
If someone is scraping my site at a low crawl rate I honestly don’t care so long as it doesn’t impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don’t. So I don’t.
If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn’t, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don’t want a chance to accidentally block one.
It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.
this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hogging the bandwidth.
Removed by mod
The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
Removed by mod
How can authority not exist? That’s staggeringly broad
Removed by mod
Not who you responded to but yeah I want to hear a drug fuelled rant I don’t even care what topic
Not the one you responded to but how about the tiredness-fuelled rant that I replied to the other person with.
Removed by mod
This is the most boring drug fuelled rant I’ve heard in my life
Removed by mod
I… Don’t think I’ve heard anyone claim authority shouldn’t exist.
So here’s a little bit of lemmy lore for you. You’re instance lemmy.ml is considered to be a tankie instance by some users. lemmy.dbzer0.com is an anarchist instance. The user you responded to probably made a generalisation based on this and assumed you were familiar with anarchist/communist/socialist/leftist discourse. From this comment I assume they were wrong.
So on behalf of no-one but myself: Hello! Welcome to Anarchism! The belief that authority should not exist. This belief comes from a lot of different places and wears a lot of different faces. Most short explanations aren’t sufficient and long explanation bore most. If you don’t mind a little learning here is a link: https://anarchistfaq.org/afaq/sectionA.html#seca1 and another one https://crimethinc.com/2016/09/28/feature-the-secret-is-to-begin-getting-started-further-resources-frequently-asked-questions#faq or if you like videos: https://youtu.be/lrTzjaXskUU.
Also a little bit about authority: people use authority to mean many things (this is even bought up in the video I linked above). But as far as anarchists are concerned (in general (no specific statement can be made about a group so vast)) authority is the act of coercing people to follow orders or commit involuntary acts. You’re boss can coerce you to neglect your health by threatening to fire you. Your government can force you to obey gender roles by threatening to jail you. A rich person can make you do whatever demeaning thing they want by dangling money in front of you (for reference see mrbeast) because otherwise your landlord will kick you out. This is authority and it is wrong. Those in authority can make mistakes, become greedy and start to think they have the power to do whatever they want (mostly because they can). This leads to suffering. My meaning of life is to minimise suffering. Anarchy is the belief that no-one should hold power over others. That all leadership should be scrutinised. It rejects blind faith in single people and encourages to think for yourself so no-one can do you wrong. And if you can’t be bothered, it encourages you to find people who genuinely care about you and let them stand up for you.
Thanks, I don’t really care for generalizing instances and didn’t really have a choice when I made my account.
But also your definition is impossibly broad as you well and I’m pretty sure not the general consensus. The video doesn’t define it as such either.
For one thing, by your definition we can have absolutely no meaningful human relationships. I can explain this more later when I have time if you don’t see what I mean
My explanation of authority wasn’t meant as a definition but rather a brief summarisation of a complex concept, Andrew does a better job actually explaining it. Like pointing out that Authority is confused with a lot of different concepts like respect or influence. Which I’m starting to suspect is happening here. meaningful human relationships are based on mutual respect. This is not authority as it is voluntary, reciprocated and revoked as soon as the other party steps over the line. This is what I believe is the basis of society and what we need to return to in order to live a truly free life. In modern society in most interactions respect has been replaced with authority. People in positions of power even use them synonymously.
Removed by mod
Removed by mod
That’s not really relevant here. This is more of a “genie is out of the bottle and now we have to learn how to deal with it situation”. The idea and technology of bots and AI training already exists. There’s no socioeconomic system that is going to magically make that go away.
I think the point you’re missing is that without the monetary incentive that arises under capitalism, there would be very little drive for anyone to build these wasteful AI systems. It’s difficult to imagine a group of people voluntarily amassing and then using the resources necessary for “AI” absent the desire to cash in on their investment. So you’re correct that an alternative economic system won’t “magically” make LLMs go away. I think it unlikely, however, that such wasteful nonsense would be used on any meaningful scale absent the perverse incentives of capitalism.
No imagination necessary.
I mean Dmitry Pospelov was arguing for AI control in the Soviet Union clear back in the 70s.
Just another way the state capitalist soviet union was closer to capitalism than socialism.
It is called regulation in sane parts of the world.
Sadly, those areas seem to be diminishing rapidly until more people enter the Find Out phase.
Removed by mod
everyone remembers tomogatchi, they were like a digital houseplant.
Removed by mod
You can still buy new ones. Take a nap.
Removed by mod
the used market
Removed by mod
Lost on Lemmy?
He is not wrong. Unless people start to take steps , the dependency of tech will be used to chain most of us. Granted, these chains will be the kindest and gentlest chains seen in a long time.
Social revolution lives on in decentralized services, like this; the true battles will be later though. This year is a mild warm up. I can’t imagine the challenges that await many
Removed by mod
Especially since the solution I cooked up for my site works just fine and took a lot less work. This is simply to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them. This is considerably simpler, and doesn’t require an entire nuclear plant powered AI to combat the opposition’s nuclear plant powered AI.
In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.
Fuck 'em. Route those motherfuckers right to /dev/null.
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
Yep. After you ban all the easy to spot ones you’re still left with far too many hard to ID bots. At least if your site is popular and large.
It’s worked alright for me. Your mileage may vary.
If someone is scraping my site at a low crawl rate I honestly don’t care so long as it doesn’t impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don’t. So I don’t.
If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn’t, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.
Geez, that’s a lot of requests!
It sure is. Needless to say, I noticed it happening.
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don’t want a chance to accidentally block one.
This is fair in those applications. I only run an ecommerce web site, though, so that doesn’t come into play.
Cloudflare offers that too, but you can’t always tell
It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.
Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.