Any gitea api rate limits?

Topic: Any gitea api rate limits? (Read 1539 times) previous topic - next topic

0 Members and 2 Guests are viewing this topic.

Any gitea api rate limits?

25 November 2023, 11:04:53

Hi all!

As an ARMtix developer, i have to periodically synchronize my local copies of PKGBUILDs repos with upstream gitea ones. However, since old monorepos are now gone, i have to pull each repo individually and it seems my build system sometimes fails to some of the repos. The logic behind this automatization only pulls/clones those repos which have updated_at field later than timestamp of the last sync. As such, if some repo failed to pull, it can only be pulled again when it is updated.
At this moment, i sometimes run manually a script which gets all repos updated_at field and compares with timestamp of the latest local commit. After that i pull repos which were marked as outdated by this script. I do it in parallel with 10 repos per iteration but at some iteration pull fails with a message like:

Code: [Select]

fatal: unable to access 'https://gitea.artixlinux.org/packages/kwallet.git/': Failed to connect to gitea.artixlinux.org port 443 after 130269 ms: Couldn't connect to server

What is current rate limit for git pull? Also, could there be any finer ways to make such a sync?

Additionally, it seems that updated_at is also changed with some other actions except commits, since i have a lot of repos marked as outdated by git pull reports that they are up-to-date.

Re: Any gitea api rate limits?

Reply #1 – 02 August 2024, 11:33:16

I resurrect this topic since from yesterday i mostly get timeouts from gitea http/https (though ssh works). Isn't there any firewall at gitea side which could block me if make too much requests?

Re: Any gitea api rate limits?

Reply #2 – 02 August 2024, 16:36:57

Yeah, fail2ban is running rampant right now. nous is dialing it back little by little.

Re: Any gitea api rate limits?

Reply #3 – 02 August 2024, 17:09:00

Quote from: corysanin – on 02 August 2024, 16:36:57

Yeah, fail2ban is running rampant right now. nous is dialing it back little by little.

Could someone put a note here once it's done? ARMtix build is stuck since i rely on http api to query repos + http git clone

Re: Any gitea api rate limits?

Reply #4 – 05 August 2024, 11:16:58

Quote from: phoenix_king_rus – on 02 August 2024, 17:09:00

Quote from: corysanin – on 02 August 2024, 16:36:57
Yeah, fail2ban is running rampant right now. nous is dialing it back little by little.
Could someone put a note here once it's done? ARMtix build is stuck since i rely on http api to query repos + http git clone

If you're on a static (or not-frequently-changing) IP, please send it in PM or better my email. I need to check the logs and see how often it triggers the filter.

Re: Any gitea api rate limits?

Reply #5 – 05 August 2024, 16:14:02

Before I start I'll just say I have no right to anything and maybe my use case is part of what you want to block ?

For a long time now (over a year) I've been maintaining a github repository of all the stable Artix package build files and another for the stable Arch package build files
https://github.com/gripped/artixpkgbuilds
https://github.com/gripped/archpkgbuilds

The purpose of this is I have another script which compares all the Artix PKGBUILD's with the Arch versions and creates a local repo based on the results. This is my idea of fun! (I should get out more)

It seems that for the time being, or maybe for good, this is unfeasible.
I was already rate limiting my script to one package processed every 1 seconds (3 for Arch because of their rate limiting) but have extended this to 5 seconds, which at a rough calculation would take 27 hours to process all the packages,
Each package processed involves a git pull, or a git clone for new packages

Even with the 5 second delay I get blocked fairly soon.

When I get blocked my whole ISP gets blocked. I can get a new IP fairly easily by disabling then enabling WAN on the router.
The ISP is huge. Even when the first octet is new I remain blocked for some time. eg last ip adresss was 86.x.x.x . Current is 109.x.x.x .
But the block remains. So that's why it appears the entire ISP is blocked somehow ?

Again if I can't do what I've been doing any more so be it. But I'm just letting you know in the hope that this is a case of unforeseen consequences ?

As an aside one way to make all the build files available to scripts, but have no load on your server would be to mirror the lot on Github.
A la https://github.com/archlinux/aur (Each individual AUR package repo is a branch on the actual repo)
As another aside the above repo is a way you can still get the the build files for packages which have been deleted from the AUR

Re: Any gitea api rate limits?

Reply #6 – 07 August 2024, 12:15:17

Quote from: gripped – on 05 August 2024, 16:14:02

For a long time now (over a year) I've been maintaining a github repository of all the stable Artix package build files and another for the stable Arch package build files

Relaxed fail2ban limits considerably. Please let us know how it goes.

Re: Any gitea api rate limits?

Reply #7 – 07 August 2024, 20:20:44

Quote from: nous – on 07 August 2024, 12:15:17

Please let us know how it goes.

No joy I'm sorry to say.

I just started the script again for the first time since my last post.
It was worse.
Two packages processed and then blocked.

Log
Spoiler (click to show/hide)

Code: [Select]

Already up to date.



Processing acl from system
-26140.788797617
Checkout master acl
Your branch is up to date with 'origin/master'.

Already on 'master'

Already up to date.



Processing amd-ucode from system
Split package, processing pkgbase linux-firmware
4.729791342000681
Checkout master linux-firmware
Your branch is up to date with 'origin/master'.

Already on 'master'

Already up to date.



Processing argon2 from system
4.735482882999349
Checkout master argon2
Your branch is up to date with 'origin/master'.

Already on 'master'


fatal: unable to access 'https://gitea.artixlinux.org/packages/argon2/': Failed to connect to gitea.artixlinux.org port 443: Connection refused

what happened?
Delete

Processing artix-cgroups from system
4.914999771997827
Checkout master artix-cgroups
Your branch is up to date with 'origin/master'.

Already on 'master'


fatal: unable to access 'https://gitea.artixlinux.org/packages/artix-cgroups/': Failed to connect to gitea.artixlinux.org port 443: Connection refused

what happened?
Delete

The number is the delay time to keep it at 1 request every 5 seconds.

On the bright side it is now just a IP based block. Or I got lucky when I changed my IP address ?

No worries on my part.

Re: Any gitea api rate limits?

Reply #8 – 08 August 2024, 11:57:47

Whitelisted /packages/.

Re: Any gitea api rate limits?

Reply #9 – 08 August 2024, 15:12:12

Quote from: nous – on 08 August 2024, 11:57:47

Whitelisted /packages/.

Still not working.

Blocked after three packages.

It's git http access that's affected at the least.

I tried a couple of things.

1. Altered the script to iterate the requests though a couple of my remote servers, and my home ip, with proxychains.
All blocked in short order

2. Adapted a function from one of my other Python scripts to hammer a few of the Gitea package pages with curl.
Spoiler (click to show/hide)

Code: [Select]

#!/usr/bin/python -u

import subprocess
from threading import Thread


def hammer():
    def run_curl(url, mirror_url):
        output = run_command(f'curl -s -w "%{{http_code}}" {url} -o /dev/null')
        print(f"{output}: {url}")
    pkgs = ["iso-profiles", "sssd", "nodejs", "netavark", "samba", "fwupd", "dns-over-https"]
    pkgs = pkgs + pkgs + pkgs + pkgs + pkgs + pkgs + pkgs + pkgs + pkgs
    thread_list = []
    for pkg in pkgs:
        url = f"https://gitea.artixlinux.org/packages/{pkg}"
        thread = Thread(target=run_curl, args=(url, pkg))
        thread_list.append(thread)
        thread.start()
    for thread in thread_list:
        thread.join()


def run_command(cmd):
    out = subprocess.check_output(cmd.split(" "), text=True)
    return out

while True:
    hammer()

I didn't run it for that long but it was clear that the server would happily process many requests per second via curl.
But it blocks very quickly when doing just one git pull every 5 seconds.

I have never used fail2ban so I don't know it's intricacies.

Just feedback again. It doesn't matter much to me but might more to @phoenix_king_rus ?

Re: Any gitea api rate limits?

Reply #10 – 08 August 2024, 15:36:33

Quote from: gripped – on 08 August 2024, 15:12:12

It doesn't matter much to me but might more to @phoenix_king_rus ?

It does, i still can't pull repos for ARMtix.
We also discuss this problem with nous in PM.
Btw, @nous could you share current fail2ban config? Maybe i could also try to experiment with it using my local gitea instance.

Re: Any gitea api rate limits?

Reply #11 – 09 August 2024, 13:53:48

Two filter files, one for GET reqs and one for POST:

Code: [Select]

# fail2ban filter configuration for nginx GET denied accesses
[INCLUDES]
before = nginx-error-common.conf
[Definition]
failregex = ^.*\[crit\].* Permission denied\), client: <HOST>.*request: "GET
ignoreregex = .*/public/api.*
              .*/artix/.*
              .*/packages/.*

Code: [Select]

# fail2ban filter configuration for nginx POST denied accesses
[INCLUDES]
before = nginx-error-common.conf
[Definition]
failregex = ^.*\[crit\].* Permission denied\), client: <HOST>.*request: "POST
ignoreregex = .*/artix/.*

Limits set in /etc/fail2ban/jail.local:

Code: [Select]

[nginx-denied-get]
findtime  = 1m
maxretry = 120
port     = http,https
logpath  = %(nginx_error_log)s
enabled = true

[nginx-denied-post]
findtime  = 1m
maxretry = 90
port     = http,https
logpath  = %(nginx_error_log)s
enabled = true

120 failed GETs/min is a lot, it befuddles me how you still earn bans.

Re: Any gitea api rate limits?

Reply #12 – 09 August 2024, 14:00:29

Oh, grepping for Ban, there was a forgotten nginx-denied filter in the default rules file (jail.conf) which banned lots of IPs. Let's see how it goes without it.

Re: Any gitea api rate limits?

Reply #13 – 09 August 2024, 16:16:28

So far so good.

I was probably overly hopeful at first. I changed the script back to 1 git pull per second. (I have no idea how many actual https requests one git pull results in)

It got further. I'd say about 20-30 packages then blocked

(I meant to count but was trigger happy closing the terminal)

If it's any help to you 86.147.37.212 was the IP address on that attempt.

I then changed it to 5 seconds per git pull again and all of 'system' is processed.
I'll be back in about 26 hours to let you know if it finished. Fairly confident.

If it does I'll experiment with lowering the delay to get an idea of a reliable rate limit from the user end.

Thanks.

Re: Any gitea api rate limits?

Reply #14 – 09 August 2024, 20:11:37

Quote from: gripped – on 09 August 2024, 16:16:28

It got further. I'd say about 20-30 packages then blocked (I meant to count but was trigger happy closing the terminal)

This created 180 requests (90 GET + 90 POST) in 44 seconds, so I guess it triggered the POST filter. Raised both limits to 180/min.