can not clone repository over ssh , and it is work over http

Gitea version (or commit ref): 1.10.1
Git version:  2.24.1
Operating system:  centos 7.7
Database (use [x]):  tidb 3.0  ok for mysql

[ ] PostgreSQL
[x ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[x ] No
[ ] Not relevant

Log gist:
can not clone repository over ssh , and it is work over http

Description

and I find some different detail between gogs and gitea when I installing :
1,the authorized_keys file show the line between command and ..app.ini in red color in gitea but not in gogs
2,when I try :  ssh -T git@ip .gogs respone : Hi there, You've successfully authenticated, but Gogs does not provide shell access ....      but gitea without this words
when I use :  git clone git@ip:pay-group/coocat-module-pay.git "  get error: " Could not read from remote repository.Please make sure you have the correct access rights and the repository exists"

down I cut 4 pic ,   the 1 , 2 is for gitea   and   the 3 ,4 is for gogs.

Screenshots

![图片](https://user-images.githubusercontent.com/33255974/71391823-bbdff300-2640-11ea-970f-06e61b6b56b8.png)
![图片](https://user-images.githubusercontent.com/33255974/71391847-d6b26780-2640-11ea-916e-e2887cd6ea0e.png)
![图片](https://user-images.githubusercontent.com/33255974/71391864-e336c000-2640-11ea-8a6b-ca7308103f92.png)

@zzps Perhaps at what repo you install Git?
And can you provide full log for this issue?

@zzps Perhaps at what repo you install Git?
And can you provide full log for this issue?

ok ,I have said that when gitea is not work, I do that use gogs with same way on the same machine . and clone work fine both over ssh and http. so  it can prove git is ok.
and you ask more log   ,i show  some.
app.ini   :
APP_NAME = 智脸猫
RUN_USER = git
RUN_MODE = prod
[oauth2]
JWT_SECRET = iY6hiKXXTNojeIufes_60zWdXGdA_Vdfg6cPHxmjYIs
[security]
INTERNAL_TOKEN = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE1NzcwOTU3Nzh9.WPuyTLb3sG43w0nbczkPO_qvlN7eCgeROJQwWLKhfQk
INSTALL_LOCK   = true
SECRET_KEY     = HYG2JjZsVeYq71FWiHYj7Favq4lEjYnV1nIQYlbhSNMjElqDqhsr0pTjdWq97e9K
[database]
DB_TYPE  = mysql
HOST     = <IP_ADDRESS>:4000
NAME     = gitea
USER     = root
PASSWD   = coolcat@zzps@081341203
SSL_MODE = disable
CHARSET  = utf8mb4
PATH     = /usr/local/gitea/data/gitea.db
[repository]
ROOT = /home/git/repository
[server]
SSH_DOMAIN       = <IP_ADDRESS>
DOMAIN           = <IP_ADDRESS>
HTTP_PORT        = 3000
ROOT_URL         = http://<IP_ADDRESS>:3000/
DISABLE_SSH      = false
SSH_PORT         = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /usr/local/gitea/data/lfs
LFS_JWT_SECRET   = M2-guVL-gKbsZHPXb6Ge-icZnuzgClt06FxwPQIMZbQ
OFFLINE_MODE     = false
BUILTIN_SSH_SERVER_USER = git
#START_SSH_SERVER = true
[mailer]
ENABLED = false
[service]
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = false
DISABLE_REGISTRATION              = false
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
ENABLE_CAPTCHA                    = false
REQUIRE_SIGNIN_VIEW               = false
DEFAULT_KEEP_EMAIL_PRIVATE        = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING       = true
NO_REPLY_ADDRESS                  = noreply.example.org
[picture]
DISABLE_GRAVATAR        = false
ENABLE_FEDERATED_AVATAR = true
[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false
[session]
PROVIDER = file
[log]
MODE      = file
LEVEL     = info
ROOT_PATH = /usr/local/gitea/log
and I try :   ssh -vT<EMAIL_ADDRESS> .the response is
$ ssh -vT<EMAIL_ADDRESS>OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to <IP_ADDRESS> [<IP_ADDRESS>] port 22.
debug1: Connection established.
debug1: identity file /c/Users/HZQ/.ssh/id_rsa type 0
debug1: identity file /c/Users/HZQ/.ssh/id_rsa-cert type -1
debug1: identity file /c/Users/HZQ/.ssh/id_dsa type -1
debug1: identity file /c/Users/HZQ/.ssh/id_dsa-cert type -1
debug1: identity file /c/Users/HZQ/.ssh/id_ecdsa type -1
debug1: identity file /c/Users/HZQ/.ssh/id_ecdsa-cert type -1
debug1: identity file /c/Users/HZQ/.ssh/id_ed25519 type -1
debug1: identity file /c/Users/HZQ/.ssh/id_ed25519-cert type -1
debug1: identity file /c/Users/HZQ/.ssh/id_xmss type -1
debug1: identity file /c/Users/HZQ/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to <IP_ADDRESS>:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher<EMAIL_ADDRESS>MAC:  compression: none
debug1: kex: client->server cipher<EMAIL_ADDRESS>MAC:  compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:tb7qobfbP/lYnboDAQIZ0WUIhfxNc63cnQkMesnG+9o
debug1: Host '<IP_ADDRESS>' is known and matches the ECDSA host key.
debug1: Found key in /c/Users/HZQ/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /c/Users/HZQ/.ssh/id_rsa RSA SHA256:WIbTyJ25Y8s4rc5RKcIOQuFnLBImTfpbsJyIwztJLIA
debug1: Will attempt key: /c/Users/HZQ/.ssh/id_dsa
debug1: Will attempt key: /c/Users/HZQ/.ssh/id_ecdsa
debug1: Will attempt key: /c/Users/HZQ/.ssh/id_ed25519
debug1: Will attempt key: /c/Users/HZQ/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/HZQ/.ssh/id_rsa RSA SHA256:WIbTyJ25Y8s4rc5RKcIOQuFnLBImTfpbsJyIwztJLIA
debug1: Server accepts key: /c/Users/HZQ/.ssh/id_rsa RSA SHA256:WIbTyJ25Y8s4rc5RKcIOQuFnLBImTfpbsJyIwztJLIA
debug1: Authentication succeeded (publickey).
Authenticated to <IP_ADDRESS> ([<IP_ADDRESS>]:22).
debug1: channel 0: new [client-session]
debug1: Requesting<EMAIL_ADDRESS>debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype<EMAIL_ADDRESS>want_reply 0
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: PTY allocation disabled.
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: PTY allocation disabled.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype<EMAIL_ADDRESS>reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3532, received 3412 bytes, in 0.2 seconds
Bytes per second: sent 14537.5, received 14043.6
debug1: Exit status 1

@zzps Perhaps at what repo you install Git?
And can you provide full log for this issue?

more  information!  when I use gitea inernal ssh server by add
START_SSH_SERVER = true
SSH_PORT  = 22222
it works!!!
so why it can't work with openssh???why?

You cannot share the git account with other git service. I think that's the reason why ssh not work.

You cannot share the git account with other git service. I think that's the reason why ssh not work.

I create user git just for run gitea，no others。by the way   I have said gogs works fine,  and use gitea built ssh server works fine

You can go to gitea ui -> admin panel and run Update the '.ssh/authorized_keys' file with Gitea SSH keys. (Not needed for the built-in SSH server.) and Resynchronize pre-receive, update and post-receive hooks of all repositories.. That will overwrite other git services' public key and hooks.

The key you use to connect to Gitea must be controlled by Gitea. That is, it must have been added to Gitea's db and must have been added to the authorized_keys file by Gitea.

You can go to gitea ui -> admin panel and run Update the '.ssh/authorized_keys' file with Gitea SSH keys. (Not needed for the built-in SSH server.) and Resynchronize pre-receive, update and post-receive hooks of all repositories.. That will overwrite other git services' public key and hooks.

i have done all that steps , no work!

The key you use to connect to Gitea must be controlled by Gitea. That is, it must have been added to Gitea's db and must have been added to the authorized_keys file by Gitea.

yes! I actualy do all these operation, but no work!   but gogs works well !

Are you running gogs and Gitea with the same user? Your description seems to imply that. That won't work and I'd ask you to think about why because the reason is simple.
Remove all of the gogs lines from authorized_keys.

The key you use to connect to Gitea must be controlled by Gitea. That is, it must have been added to Gitea's db and must have been added to the authorized_keys file by Gitea.

I do the gitea install strictly by the document, I see the  key be added to the db and authorized_keys file   all things be done ok, and it is fine in gogs but can not work in gitea

Are you running gogs and Gitea with the same user? Your description seems to imply that. That won't work and I'd ask you to think about why because the reason is simple.
Remove all of the gogs lines from authorized_keys.
yes ,  I run gogs and Gitea with the same user "git", before I change gogs to gitea, I have remove all the gogs key from authorized_keys . and it  still not work

If Gitea works on the internal SSH server then the problem lies at authorized_keys.
Let us begin again.

What user is running Gitea?
What config and what environment are you using to run Gitea and where?
Do the lines in .ssh/authorized_keys look correct for the above information? Give us an example - you can cut the key off after the first few letters.
Ensure that Gitea is now no longer running with the internal SSH server running otherwise the next step won't work.
Delete or better move the authorized_keys file, and regenerate it from the running Gitea process. Check how that differs.
When things fail are you even getting logs on the Gitea server?

I assure you this works - I use open sshd with Gitea myself - so the problem is with your configuration.
For simplicities sake don't try pushing just try SSHing in - that way we know if your key is being properly matched with an authorized key line.

If Gitea works on the internal SSH server then the problem lies at authorized_keys.
Let us begin again.
1. What user is running Gitea?

2. What config and what environment are you using to run Gitea and where?

3. Do the lines in .ssh/authorized_keys look correct for the above information? Give us an example - you can cut the key off after the first few letters.

4. Ensure that Gitea is now no longer running with the internal SSH server running otherwise the next step won't work.

5. Delete or better move the authorized_keys file, and regenerate it from the running Gitea process. Check how that differs.

6. When things fail are you even getting logs on the Gitea server?

I assure you this works - I use open sshd with Gitea myself - so the problem is with your configuration.
For simplicities sake don't try pushing just try SSHing in - that way we know if your key is being properly matched with an authorized key line.

ok, now I answer your question one by one:
1, I use user git run gitea , and only run gitea
2,  I do not know your point about the config and environment, I could say that  app.ini is above,and git version ,system too
3, I copy the key from authorized_keys here :
gitea public key
command="/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-14",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDkeFfDMzT355/gLB2OooglszXmLcYlOz/wrDjR4aJNV2aPBq7uKwktdZb+7ECZTT2DK6/BaC5cnbOmlqCt78dNWEMKx7jSgAOpEQSqY6EBGSsSkoQsXwjOwrfOPAfG/tQAXIKDhv51ih71MzHZFxcvw/eIOq+h3YvduBxXlrplJ+OfU68ecD26mCkrLhQCdB51ZZnSpGWJyEGucnXOrxuRooUlPZJnNSiRNb6nUmNKpsNqACygBJduxOIvVcEhizy6SSwXT0H+TQdE3kkfuCYHH7P9pShlPR5R5nyXRg/BZdA0tjPiD3CES5xnrc4tuiGBl/TpT4/t+RLPPzTgZSw3ayYXHzZMrLsoaMS5gpCK6TaJIc5OyS7oDNuDCRqPyJphQjISFG3kk+y2/Dnoy00TgAwJq5vaDBFiJwmpF/RQ3w9Dqb9PNNuCdhchCaPHnzOZqRvP82uQV2Nzv+0rBYAt3NQFAP0cZdXN+Td7kFvkrvFgaI3nkrxjilVnQPIUmywosmjcU1Rf+wN0hMSzJhOcOatef40vgM9MKUYQIRfaWpIPy4TTPs0LqPwPNIeSO8/TeS2VG0kmZAvlzEzH6AuXDTrEt1QmESD8w/iqrVDj8V9rZLKWDqBkV0TmGDgALyx/O7SLzdJLNU1ud5rbGGWZVty9tR5rQyAvZobraAA11w== HZQ@DESKTOP-PC28
4, I am sure I remove "START_SSH_SERVER = true" ,so it run with system ssh
5, every time I will manual delete authorized_keys  and use Update the '.ssh/authorized_keys' on the gitea web ui for a new one
6, no more log I can find ,only gitea.log with a lot of sql, no  valuable message

What happens if you SSH directly without going through git? You should get told you have authenticated by Gitea does not provide a shell or thereabouts

What happens if you SSH directly without going through git? You should get told you have authenticated by Gitea does not provide a shell or thereabouts

ssh<EMAIL_ADDRESS>The authenticity of host '<IP_ADDRESS> (<IP_ADDRESS>)' can't be established.
ECDSA key fingerprint is SHA256:tb7qobfbP/lYnboDAQIZ0WUIhfxNc63cnQkMesnG+9o.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
PTY allocation request failed on channel 0
Connection to <IP_ADDRESS> closed.

Well that's interesting.
From a command line on the server run:
/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-14

/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-14

/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-17
Hi there, zzps! You've successfully authenticated with the key named 本地pc, but Gitea does not provide shell access.
If this is unexpected, please log in with password and setup Gitea under another user.

Did you deliberately change the key number? I'm going to assume that the key number still matches authorized keys.
So now the question comes as to why SSH can't run or it fails for that command.
There's an interesting bit where SSH states Pty allocation failed. Now that's weird because we explicitly say don't give a Pty in our authorized_keys
So I wonder, if you remove the no-pty from the authorised_keys line if that helps things. Did Gogs have that too?
Another thing is if your key having a non ASCII name if that is a cause of problems - it shouldn't be - but it might be worth a try to see if things work if you change it something like pi-key.

because I have change the key before run ,so that I should replace serv-14 to serv-17, and  your command is serv-14 and I  run serv-17,it doesn't matter.
abount the pty question , I can test it , I have environment

Any progress on testing the pty things?

You just need to remove the no-pty, from the authorized_keys and try SSHing again

after remove no-pty in the authorised_keys
it shows:
$ ssh<EMAIL_ADDRESS>Connection to <IP_ADDRESS> closed.

You just need to remove the no-pty, from the authorized_keys and try SSHing again

it still can not clone over ssh

You just need to remove the no-pty, from the authorized_keys and try SSHing again

I think I should show you a strange discovery, the gitea generate authorized_keys and the line
"/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-1" shows with  red color .
and the gogs
generate authorized_keys and the line
"/usr/local/gogs/gogs serv key-3 --config='/usr/local/gogs/custom/conf/app.ini'" shows with white color .
!!!may be it is the clue!!!!!

So for some reason it appears SSH cannot run the command. I guess check the permissions for everything? Is there a difference with gogs?

So for some reason it appears SSH cannot run the command. I guess check the permissions for everything? Is there a difference with gogs?

I have check everything permissions , no difference with gogs.

Have you tried run ssh -T<EMAIL_ADDRESS>on your local machine?

done that and no help . and I move the environment to machine :<IP_ADDRESS>, and do all the step as document, it still doesn't work !

Is it possible that the serv command cannot talk to the Gitea server and that's the problem here?
(I've checked if the Chinese ideograms were a problem and they're not on my server)
Do you have some super locked down SSHd configuration?

It should return
Hi there, " + user.Name + "! You've successfully authenticated with the key named " + key.Name + ", but Gitea does not provide shell access.

Before this, there is no any internal http request.
So /usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-1 hasn't been invoked.
Maybe you could confirm that authorized_keys owned by git and has a permission mode 600.

@lunny
https://github.com/go-gitea/gitea/blob/f88715e29c8cd6865b69eb8e692d051d977e93a5/cmd/serv.go#L105
Makes a call to the Gitea server.
If authorized keys wasn't working SSH would refuse to allow them to authenticate and they wouldn't get the Pty refused message.
Looking again at cmd/serv.go the only way this could fail silently is if during setup setting.NewContext calls a fatal
https://github.com/go-gitea/gitea/blob/f88715e29c8cd6865b69eb8e692d051d977e93a5/cmd/serv.go#L49
So...
@zzps what are the permissions - (and I don't just want they're fine as a response here I want the actual permissions) of the Gitea conf file /usr/local/gitea/custom/conf/app.ini and each of its parent directories in turn. Whilst you're at it give me the permissions of /usr/local/gitea/gitea too.

Is it possible that the serv command cannot talk to the Gitea server and that's the problem here?
(I've checked if the Chinese ideograms were a problem and they're not on my server)
Do you have some super locked down SSHd configuration?

I have change Chinese to english, no   help too !

Maybe you could confirm tha

run  /usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' serv key-1  could return  what you said,  and I can show you the permission to prove all things config is ok.

@lunny
https://github.com/go-gitea/gitea/blob/f88715e29c8cd6865b69eb8e692d051d977e93a5/cmd/serv.go#L105
Makes a call to the Gitea server.
If authorized keys wasn't working SSH would refuse to allow them to authenticate and they wouldn't get the Pty refused message.
Looking again at cmd/serv.go the only way this could fail silently is if during setup setting.NewContext calls a fatal
https://github.com/go-gitea/gitea/blob/f88715e29c8cd6865b69eb8e692d051d977e93a5/cmd/serv.go#L49
So...
@zzps what are the permissions - (and I don't just want they're fine as a response here I want the actual permissions) of the Gitea conf file /usr/local/gitea/custom/conf/app.ini and each of its parent directories in turn. Whilst you're at it give me the permissions of /usr/local/gitea/gitea too.

I'm a bit stuck as to what could be going on here. I suspect the issue is that NewContext() is throwing a Fatal and that's killing things. Now rather distressingly, macaron.init will cause a fatal if it is started from a non readable directory - but why would gogs work in that case?!
OK is it possible that the home directory (according to /etc/passwd) for the git user is /usr/local/git or worse some other unreadable directory?
We could test the environment that is passed to ssh authorized_keys command by creating a script
#!/bin/sh
echo $PWD
ls -ld $PWD
/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' --help
/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' "$@"

say as /usr/local/gitea/gitea.shadow.sh
then replace the command in authorized_keys with it e.g.:
command="/usr/local/gitea/gitea.shadow.sh serv key-14",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDkeFfDMzT355/gLB2OooglszXmLcYlOz/wrDjR4aJNV2aPBq7uKwktdZb+7ECZTT2DK6/BaC5cnbOmlqCt78dNWEMKx7jSgAOpEQSqY6EBGSsSkoQsXwjOwrfOPAfG/tQAXIKDhv51ih71MzHZFxcvw/eIOq+h3YvduBxXlrplJ+OfU68ecD26mCkrLhQCdB51ZZnSpGWJyEGucnXOrxuRooUlPZJnNSiRNb6nUmNKpsNqACygBJduxOIvVcEhizy6SSwXT0H+TQdE3kkfuCYHH7P9pShlPR5R5nyXRg/BZdA0tjPiD3CES5xnrc4tuiGBl/TpT4/t+RLPPzTgZSw3ayYXHzZMrLsoaMS5gpCK6TaJIc5OyS7oDNuDCRqPyJphQjISFG3kk+y2/Dnoy00TgAwJq5vaDBFiJwmpF/RQ3w9Dqb9PNNuCdhchCaPHnzOZqRvP82uQV2Nzv+0rBYAt3NQFAP0cZdXN+Td7kFvkrvFgaI3nkrxjilVnQPIUmywosmjcU1Rf+wN0hMSzJhOcOatef40vgM9MKUYQIRfaWpIPy4TTPs0LqPwPNIeSO8/TeS2VG0kmZAvlzEzH6AuXDTrEt1QmESD8w/iqrVDj8V9rZLKWDqBkV0TmGDgALyx/O7SLzdJLNU1ud5rbGGWZVty9tR5rQyAvZobraAA11w== HZQ@DESKTOP-PC28

then try sshing directly to the server as the git user again.

Once again I'm sorry this is turning in to such a rigmarole. There is something very weird happening and replacing gogs with gitea is usually just drop-in at least at this point.

I'm a bit stuck as to what could be going on here. I suspect the issue is that NewContext() is throwing a Fatal and that's killing things. Now rather distressingly, macaron.init will cause a fatal if it is started from a non readable directory - but why would gogs work in that case?!
OK is it possible that the home directory (according to /etc/passwd) for the git user is /usr/local/git or worse some other unreadable directory?
We could test the environment that is passed to ssh authorized_keys command by creating a script
#!/bin/sh
echo $PWD
ls -ld $PWD
/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' --help
/usr/local/gitea/gitea --config='/usr/local/gitea/custom/conf/app.ini' "$@"

say as /usr/local/gitea/gitea.shadow.sh
then replace the command in authorized_keys with it e.g.:
command="/usr/local/gitea/gitea.shadow.sh serv key-14",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDkeFfDMzT355/gLB2OooglszXmLcYlOz/wrDjR4aJNV2aPBq7uKwktdZb+7ECZTT2DK6/BaC5cnbOmlqCt78dNWEMKx7jSgAOpEQSqY6EBGSsSkoQsXwjOwrfOPAfG/tQAXIKDhv51ih71MzHZFxcvw/eIOq+h3YvduBxXlrplJ+OfU68ecD26mCkrLhQCdB51ZZnSpGWJyEGucnXOrxuRooUlPZJnNSiRNb6nUmNKpsNqACygBJduxOIvVcEhizy6SSwXT0H+TQdE3kkfuCYHH7P9pShlPR5R5nyXRg/BZdA0tjPiD3CES5xnrc4tuiGBl/TpT4/t+RLPPzTgZSw3ayYXHzZMrLsoaMS5gpCK6TaJIc5OyS7oDNuDCRqPyJphQjISFG3kk+y2/Dnoy00TgAwJq5vaDBFiJwmpF/RQ3w9Dqb9PNNuCdhchCaPHnzOZqRvP82uQV2Nzv+0rBYAt3NQFAP0cZdXN+Td7kFvkrvFgaI3nkrxjilVnQPIUmywosmjcU1Rf+wN0hMSzJhOcOatef40vgM9MKUYQIRfaWpIPy4TTPs0LqPwPNIeSO8/TeS2VG0kmZAvlzEzH6AuXDTrEt1QmESD8w/iqrVDj8V9rZLKWDqBkV0TmGDgALyx/O7SLzdJLNU1ud5rbGGWZVty9tR5rQyAvZobraAA11w== HZQ@DESKTOP-PC28

then try sshing directly to the server as the git user again.
That should then cause a bunch of things to get printed before it fails.

now  I have done as your steps above, and this is the pic:

Once again I'm sorry this is turning in to such a rigmarole. There is something very weird happening and replacing gogs with gitea is usually just drop-in at least at this point.

I should give you my best thanks!

Could you add an echo $USER too?

echo $USER

Damn it's not that then... I'm quite stumped.
Ok on master I just added an extra option to serv: --debug would it be possible to grab master and change the authorized_keys line to include --debug ?

Damn it's not that then... I'm quite stumped.
Ok on master I just added an extra option to serv: --debug would it be possible to grab master and change the authorized_keys line to include --debug ?

how should I do? be  more detail please

I had similar issue before, but it might be not relevant
The gitea in my environment is deployed on top of Kube with Istio gateway and metallb lb.
I didn't configure the ingress for tcp 22, leading to the traffic not correctly went into gitea. after added a new rule for that, the issue fixed.
So, need to make sure the ssh traffic went to the gitea server other than the linux server.
when you try ssh -T xxx,  ideally you should see any logs in /var/log/auth.log

I run into the same situation. After some experiments, it seems to me that the cause should be GITEA_WORK_DIR.
If the default directory layout is used (AppPath=./gitea/gitea, AppWorkPath=./gitea, Custom path=./gitea/custom), everything just works fine. But with some customizations such as AppPath=/opt/bin/gitea, AppWorkPath=/srv/gitea, Custom path=/etc/gitea, ssh clone dies silently (while http clone is OK).
For reference, here are the commands which fails in ssh cloning:
sudo su - git
export GITEA_WORK_DIR=/srv/gitea
export GITEA_CUSTOM=/etc/gitea
/opt/bin/gitea web

After some (re)searching, the problem should lie in .ssh/authorized_keys. A simple change of the line
command="/opt/bin/gitea --config='/etc/gitea/conf/app.ini' serv key-*"

to
command="/opt/bin/gitea --work-path='/srv/gitea' --config='/etc/gitea/conf/app.ini' serv key-*"

did do the trick.
Hope it helps.

By the way, I have no idea whether the git hooks are also demanding --work-path. Just haven't test it.

@sunbing81 are you compiling your own gitea? If you are I suggest you look at https://docs.gitea.io/en-us/install-from-source/#changing-the-default-custompath-customconf-and-appworkpath which will allow you to just build gitea with the correct paths builtin

Hi @zeripath, I am using the official binary gitea-1.10.0.
Per the doc at install-from-binary, I thought custom directory layout should work out-of-the-box.
Or am I missing anything?

no you're not missing anything - just that if you were building from source you could simplify your codes to make everything a bit easier.
Anyway am I reading your comment correctly that you had to manually add a work-path to authorized_keys - if so that's a bug and we should fix that.

Yes, I have to manually add --work-path to .ssh/authorized_keys and all the git hooks ( pre-receive, update and post-receive).

I discovered yesterday that if the git user does not have a working shell (I had it set to /bin/true), I get the "Could not read from remote repository. Please make sure you have the correct access rights and the repository exists" error.  I changed the git users's shell to /bin/sh, and everything started working. Might this be your problem?

after i add workpath  it still clone fail over ssh!
why gogs work and gitea fail and they config all same!

@zzps sorry we still haven't been able to work this out. Generally this is very easy.
Gogs may work simply because it doesn't attempt to communicate with the running Gogs server.
I still don't understand where things are failing for you.
There's something odd about your set-up that is breaking things - I also can't understand why people would ever need the work-path setting up either.
I'm going to take a look at the work-path now.

I still can't work out why you would ever need to set the --work-path  - except if your config option is incorrect...
None of this makes any sense.

I still can't work out why you would ever need to set the --work-path - except if your config option is incorrect...
None of this makes any sense.

maybe  you chould check my compute , i could give you ip and account over secret channel

you can message me on discord as @zeripath

you can message me on discord as @zeripath

how? email or?

after i add workpath it still clone fail over ssh!
why gogs work and gitea fail and they config all same!

Per the comment, I have to add --work-path to not only .ssh/authorized_key, but also all the hooks under each repository (*.git/hooks/{pre-receive,update,post-receive}).
FYI, @zeripath @zzps

@sunbing81 - so I had a cursory look at this yesterday - I still can't work out why you need to do this.
I wonder if something is trying to create a directory. Do you have ROOT_PATH set in [log]?

Ah... what do you have set for LFS_CONTENT_PATH in [server] ? Is it an absolute path?

@sunbing81 - so I had a cursory look at this yesterday - I still can't work out why you need to do this.
I wonder if something is trying to create a directory. Do you have ROOT_PATH set in [log]?

Yes, I have a ROOT_PATH=/var/log/gitea in [log]

Ah... what do you have set for LFS_CONTENT_PATH in [server] ? Is it an absolute path?

I have a LFS_CONTENT_PATH = ./data/lfs in [server].

you can message me on discord as @zeripath

could you give me an email address?

@sunbing81 - so your problem should have been fixed by #11136
@zzps - Go to https://discord.gg/Gitea


Error after trying to install MGCB Editor

I clicked on "Install" option and got this cryptic error. I have plugin version 222.0.0 installed from Marketplace

Hey @capsey, thanks for the report. This will be fixed by #14, but, unfortunately, I don't know when I will have time to implement it :(
But contributions are welcome!


add: OGRE weighted pool to allowlist array

Description
Please include a summary of the change and if relevant which issue is fixed. Please also include relevant motivation and context.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Dependency changes
[ ] Code refactor / cleanup
[ ] Documentation or wording changes
[x] Other

How should this be tested?
Please provide instructions so we can test. Please also list any relevant details for your test configuration.

[ ] Test A
[ ] Test B

Visual context
Please provide any relevant visual context for UI changes or additions. This could be static screenshots or a loom screencast.
Checklist:

[x] I have performed a self-review of my own code
[x] I have requested at least 1 review (If the PR is significant enough, use best judgement here)
[x] I have commented my code where relevant, particularly in hard-to-understand areas
[x] If package-lock.json has changes, it was intentional.
[x] The base of this PR is master if hotfix, develop if not

Replaced by #3888

@keccakdog hey there.  Sorry it took us so long to get to this.  Your pool should be available for deposits soon.  Hope you still have interest in providing some liquidty on Balancer.
Feel free to get in touch on telegram @tritium_vlk if you have questions/need support in the future with pool creation/setup.


intro grammar fixes :)

@kwikness Unfortunately we had to remove the dev branch after your fixes (see https://github.com/impress/impress.js/issues/521#issuecomment-198642846), can you please open a Pull Request to the master branch instead? Thanks.


add resource requests and limits

quay-operator: add resource requests and limits (PROJQUAY-2011)

adds resource and requests and limits to all Quay components to help placement
this also enables cluster auto-scaling
prevent resource starving by setting set sane limits

@dmesser Could you fix the commit message to conform to our standard when you get the chance?

@alecmerdler I thought I already fixed it but I am not sure the check was re-triggered.  In any case, do we want to proceed with requests for now and talk about limits later? requests are needed otherwhise the HPAs won't trigger.

I removed the limits for the postgres instance for now. It's a more conservative approach since we wouldn't want the database to be killed unnecessarily when the host still has resources. It would be OOM'd by the host at some point.


The user should have the possibility to navigate/expand/popup the same section(shared information) across different pages

What needs to be solved?
Some forms require that the user shares  same set of information across the form (link to different object spread across different pages) which means that the same information needs to be reused across pages/sections/groups in the form.
F.ex A : List of phone numbers b:List of persons c: List of address. List A is shared across different pages and linked to different object. F.ex The user fills in the form and page 1 he fills the person details and he links phone 1 to person 1,3,5. Then he navigates to page 2 and fills in all addresses and now he has the need to link address1 ,7,9 to phone 2.
In such a case, we want the best possible user experience. In the real life scenario, the phone object is a complex object with f.ex 30 fields and with many repeating groups nested inside.  So how can we best solve this user experience issue?
How do you want it solved?
I think that a popup with many pages can be a possible option for making it easy for the user. So the user can click when he needs a new phone object, he can click add on page 2, person 2 section or he can add when required f.ex when adding information for person10. I would like to present this problem to the UX and agree on a best possible user experience
Alternative solutions

Additional context
This  issue is raised in relation to a specific requirement to MT-3 form Altinn/apps-backlog#5
F.ex in the picture below, the user needs to link a persons information with an activity/card/account etc. The user needs to have the possibility to add a new person if the list does not contain the person he needs to link to.  So the person list must be available across different sections/pages in the form without creating multiple copies of the same information.

It's a bit hazy to me what exactly is needed here, but as this issue is aging it might be worth looking into these features to see if they could be relevant for solving the problem:

https://docs.altinn.studio/app/development/data/options/#options-based-on-repeating-groups-from-the-data-model
https://docs.altinn.studio/app/development/ux/fields/grouping/alternatives/#add-element-from-separate-repeating-group


[Bug] Ordering by date doesn't work well on paginated table

Steps to reproduce the bug

Create a paginated pluto grid with a PlutoColumnType.date(format: 'dd/MM/yyyy')
Try to order the table by the date column

Expected results
Correct ordering
Actual results
If you don't "cache" the full table by entering on each page, the order will be incorrect
Code sample
filtering.zip
Execution Environment
Flutter version
Channel stable, 3.3.9
PlutoGrid version
PlutoGrid - v5.4.6
OS
Web

I will fix the bug and release it.
Thanks for reporting the issue.

@bosskmk maybe if ordering is done using the originalValue getter (#670) instead of value, this can be solved.
I haven't seen the ordering code but this can be a solution.

DateFormat (intl) conversion process is performed for sorting date types.
This converts the format once and saves it if the value is not changed.
If you access (getter) the value of the cell, the value for sorting is saved once at the beginning, but this problem occurs because the values of other pages that are not opened in pagination are not accessed.
This will be fixed soon.

Fixed in PlutoGrid version 5.4.7.
This problem only seems to occur when the Date format is dd/MM/yyyy.
It doesn't happen with yyyy/MM/dd.

this is the default format here on Brasil, so for everyone using the package to display data it'll be a nice update. Tks!


MIssing data for 27th of December (American market)

I have upgraded to the latest version. The data from 27th of January seem to be missing for most of american stocks (e.g., AAPL).
It goes directly from 2022-01-26 to 2022-01-28.
European market works fine.

I believe that is a Yahoo Finance issue and not a package issue. If you go to the Yahoo Finance website, the data is missing in the Historical Data tab of every ticker I have checked.

If you refresh the page, it will sometimes be there and sometimes not. I would recommend trying to pull the data later today and see if the issue is fixed.


Document how to create an action in any language

What is the issue?
From feedback on twitter:

Oh and apparently you can write Dagger actions in "Python, Go, TypeScript, Ruby, Java, and shell scripts".
@dagger_io how do I do this? I cannot find any docs on your site on how to write Dagger actions in those language. My first impression was you MUST use Go for it.

Though certainly nothing to consider this complete, there's a small example on the homepage next to the "Create actions with any programming language" section:

Adding that as an actual example to the documentation would be a great starting point to clue people in.


MBTiles cache seems to be corrupt

During tests with an MBTiles cache I discovered that the produced MBTiles file is malformed. At first I thought that the problem is connected to the reprojection that I had in my example, but I was able to reproduce the problem without reprojection (all projections in web mercator).
Here is a minimal config (adapted from the included mapproxy.yaml) that reproduces the problem:
services:
  demo:
  wmts:
    md:
      title: MapProxy WMS Proxy
      abstract: This is a minimal MapProxy example.

layers:
  - name: osm
    title: Omniscale OSM WMS - osm.omniscale.net
    sources: [osm_cache_mbtiles]

caches:
  osm_cache_mbtiles:
    grids: [webmercator]
    sources: [osm_wms]
    cache:
      type: mbtiles

sources:
  osm_wms:
    type: wms
    req:
      # use of this source is only permitted for testing
      url: http://osm.omniscale.net/proxy/service?
      layers: osm

grids:
    webmercator:
        base: GLOBAL_WEBMERCATOR

Note that running this example works when using mapproxy as tile server, so the demo application shows a correct map. However, loading the mbtiles file in LocusMap on Android or in QGIS produces a corrupted map:

I also tried to load the mbtiles file with MapTiler. This doesn't work at all, MapTiler displays the following error message:

MBTiles file is malformed. Missing metadata table, missing bounds value in metadata

I even tried it with an explicit bbox on the grid, but without luck. I'm using MapProxy 1.8.2 on Windows.
Any help would be appreciated.

Can you try GLOBAL_MERCATOR as the grid? MBTiles counts tiles from bottom to top (origin: sw), AFAIK.

MapTiler still shows the error that the file is malformed, but in LocusMap and QGIS the file works now. Thanks a lot.

Great.
For MapTiler: This is more or less a missing feature in MapProxy. The metadata table is not complete, as the actual data boundary is not known for MapProxy in advance.

I am using mapproxy-seed and get the same issue.   Is there some way to update the metadata table?


Unicode appear in topic.messages instead of Big Decimal

Is your proposal related to a problem?

when i define a schema in scheam registry where i need to push a key of type byte and
the values are presented inside the ui as unicode
for example:
"bytes": "\u0002"
where i am expecting:
000920011700048000480004800050
asa bigDecimal
Describe the solution you'd like

decerialize all /u prefix values into decimal values
Describe alternatives you've considered

i didn't tried to resolve the issue yet
but i will try to have a look into the source code of the project and try to add another decode level
depending on environment variable which i will call
KAFKA_CLUSTERS_0_DECODE_UNICODE: true
Additional context

i would be happy if i get small guidance on which src files to have a look where those are the files realted
to adding this feature

Hey, thanks for reaching out.
We appreciate the wish to help, but we'll need to take a look at this ourselves first, since I'm not sure if it's more of a frontend or the backend thing.

@nemerna, could you please share an example schema you have for this kind of purpose?

we are using v0.4.0 and facing the same with bigdecimal in schemaregistry, displayed values are like "bytes": "#òoÁ\u0000\u0000", what is the plan to release the fix for bigdecimal support?

@nitin1677 this might get resolved within #2125. Can you provide an example schema to reproduce this behavior?

i tried to deploy from master branch, which was deployed from the following commit:
https://github.com/provectus/kafka-ui/commit/a9c31e6
still getting the same issue, field value is displayed as below:
"field1": {
"bytes": "#òoÁ\u0000\u0000"
},
Schema definition for the field is as below:
{
"name": "field1",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 24,
"scale": 16
}
],
"doc": "test description.",
"default": null
}

this stuff is still happening in v0.7.1

@pgomes-lgc raise a new issue


Iptables fails when trying to use kuma service mesh

Describe the bug
Dear team,
As we evaluate Docker Desktop replacements, I've encountered an issue with Orbstack that currently prevents us from using it.
The problem
Our init container utilizes iptables to set up a transparent proxy within the raw table. However, applying certain rules to this table results in failure.
Investigation
While debugging, I discovered a similar issue reported in the Kubernetes SIG KIND project: https://github.com/kubernetes-sigs/kind/issues/3171. Interestingly, your comment suggests a missing kernel module might be the culprit: https://github.com/kubernetes-sigs/kind/issues/3171#issuecomment-1513510001.
Request for assistance
To effectively evaluate Orbstack and make informed recommendations to the team, I kindly request your assistance in resolving this issue. Any insights or troubleshooting steps regarding the missing kernel module or alternative solutions would be greatly appreciated.
To Reproduce

Create k3d cluster: k3d cluster create
Download kuma curl -L https://kuma.io/installer.sh | VERSION=2.6.1 sh -
Install kuma ./kuma-2.6.1/bin/kumactl install control-plane | kubectl apply -f-
Install kuma demo ./kuma-2.6.1/bin/kumactl install demo --without-gateway | kubectl apply -f-

Expected behavior
Applying iptables rules succeeds
Diagnostic report (REQUIRED)
OrbStack info:
Version: 1.4.3
Commit: 75d38a2c2d9a2b42d64378a8f50f31d4d566df10 (v1.4.3)
System info:
macOS: 14.3.1 (23D60)
CPU: arm64, 14 cores
CPU model: Apple M3 Max
Model: Mac15,11
Memory: 36 GiB
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2024-02-21T06-59-34.888626Z.zip

Screenshots and additional context (optional)
# kumactl is about to apply the iptables rules that will enable transparent proxying on the machine. The SSH connection may drop. If that happens, just reconnect again.
# writing following contents to rules file:  /tmp/iptables-rules-1708433360951651873.txt145072776
* raw

# Rules:
--append PREROUTING --protocol udp --source-port 53 --jump CT --zone 1
--append OUTPUT --protocol udp --destination-port 53 --match owner --uid-owner 5678 --jump CT --zone 1
--append OUTPUT --protocol udp --source-port 15053 --match owner --uid-owner 5678 --jump CT --zone 2
--append OUTPUT --protocol udp --destination-port 53 --jump CT --zone 2

COMMIT

* nat

# Custom Chains:
--new-chain KUMA_MESH_INBOUND
--new-chain KUMA_MESH_OUTBOUND
--new-chain KUMA_MESH_INBOUND_REDIRECT
--new-chain KUMA_MESH_OUTBOUND_REDIRECT

# Rules:
--append PREROUTING --protocol tcp --jump KUMA_MESH_INBOUND
--insert OUTPUT 1 --protocol udp --destination-port 53 --match owner --uid-owner 5678 --jump RETURN
--insert OUTPUT 2 --protocol udp --destination-port 53 --jump REDIRECT --to-ports 15053
--append OUTPUT --protocol tcp --jump KUMA_MESH_OUTBOUND
--append KUMA_MESH_INBOUND --protocol tcp --jump KUMA_MESH_INBOUND_REDIRECT
--append KUMA_MESH_OUTBOUND --source <IP_ADDRESS>/32 --out-interface lo --jump RETURN
--append KUMA_MESH_OUTBOUND --protocol tcp ! --destination-port 53 --out-interface lo ! --destination <IP_ADDRESS>/32 --match owner --uid-owner 5678 --jump KUMA_MESH_INBOUND_REDIRECT
--append KUMA_MESH_OUTBOUND --protocol tcp ! --destination-port 53 --out-interface lo --match owner ! --uid-owner 5678 --jump RETURN
--append KUMA_MESH_OUTBOUND --match owner --uid-owner 5678 --jump RETURN
--append KUMA_MESH_OUTBOUND --protocol tcp --destination-port 53 --jump REDIRECT --to-ports 15053
--append KUMA_MESH_OUTBOUND --destination <IP_ADDRESS>/32 --jump RETURN
--append KUMA_MESH_OUTBOUND --jump KUMA_MESH_OUTBOUND_REDIRECT
--append KUMA_MESH_INBOUND_REDIRECT --protocol tcp --jump REDIRECT --to-ports 15006
--append KUMA_MESH_OUTBOUND_REDIRECT --protocol tcp --jump REDIRECT --to-ports 15001

COMMIT

# [1/5] iptables-restore --wait=5 --noflush /tmp/iptables-rules-1708433360951651873.txt145072776 returned error: "executing command failed: exit status 1 (with output: \"iptables-restore: line 9 failed\\n\")" will try again in 2s
# [2/5] iptables-restore --wait=5 --noflush /tmp/iptables-rules-1708433360951651873.txt145072776 returned error: "executing command failed: exit status 1 (with output: \"iptables-restore: line 9 failed\\n\")" will try again in 2s
# [3/5] iptables-restore --wait=5 --noflush /tmp/iptables-rules-1708433360951651873.txt145072776 returned error: "executing command failed: exit status 1 (with output: \"iptables-restore: line 9 failed\\n\")" will try again in 2s
# [4/5] iptables-restore --wait=5 --noflush /tmp/iptables-rules-1708433360951651873.txt145072776 returned error: "executing command failed: exit status 1 (with output: \"iptables-restore: line 9 failed\\n\")" will try again in 2s
# [5/5] iptables-restore --wait=5 --noflush /tmp/iptables-rules-1708433360951651873.txt145072776 returned error: "executing command failed: exit status 1 (with output: \"iptables-restore: line 9 failed\\n\")"

Error: failed to setup transparent proxy: cannot restore ipv4 iptable rules: iptables-restore failed

Hi @slinorb, thank you for looking into it. We don't have a strict timeline, so we are able to adjust if needed.

Added for the next version.

@kdrag0n when do you plan to release v1.5.0, so I could test it?

Released in v1.5.0 Canary 1. To switch to Canary: Settings > Update channel
New feature: Debug Shell provides useful commands & tools, making it easy to debug any container (even if minimal).

Released in stable v1.5.0.

New feature: OrbStack Debug Shell provides useful commands & tools, making it easy to debug any container (even if minimal).
"It's 12 AM and your Docker container isn't working, but debugging it with docker exec is impossible because it has no shell or tools. OrbStack Debug Shell provides a nice zsh setup with auto-complete, editors, common tools, and over 80,000 packages, on top of any container."
Read more: blog post


Failed to reproduce results on Imagenet.

Thanks for your great work.
We try to reproduce reults of KD (without CTKD) using run_imagenet_distill.sh but t it seems that something is wrong.
python3 train_student.py --path-t ./save/models/ResNet34/resnet34.pth \
        --distill kd \
        --batch_size 256 --epochs 120 --dataset imagenet \
        --gpu_id 0,1,2,3,4,5,6,7 --num_workers 32 \
        --learning_rate 0.1 --lr_decay_epochs 30,60,90 --weight_decay 1e-4 \
        --model_s ResNet18 -r 1 -a 1 -b 0 --kd_T 4 \
        ...

RN34
RN18
KD

Paper
73.96
70.26
70.83

Ours
73.79
70.34
70.27, 70.48

I would like to know whether the cause is experimental error or parameter error.

This code was originally used for training on CIAFR-100. For training on ImageNet, I forgot to add the code to average the results of eight GPUs. I have updated my code. Please refer to helper/loops.py line 249-258. Add these code to your codebase and try again.

We used only one GPU for training. So maybe the results are not affected by this issue?

I didn't try to use one GPU for training on ImageNet. I always use 8* RTX 2080Ti for ImageNet. I guess this may be the reason for the different accuracy?

I didn't try to use one GPU for training on ImageNet. I always use 8* RTX 2080Ti for ImageNet. I guess this may be the reason for the different accuracy?

I guess this is an important reason.
Thanks for your reply.

I didn't try to use one GPU for training on ImageNet. I always use 8* RTX 2080Ti for ImageNet. I guess this may be the reason for the different accuracy?

How long is the training?

It will takes about one day for 120 epoch training


to_sql_where: consider replacing 'ST_GeomFromWKB' with 'ST_GeomFromWKT'

I want to use the result converted from to_sql_where in PostGIS, as follows:
ast = parse("INTERSECTS(geom, LINESTRING(0 0, 1 1))")
where = to_sql_where(ast, FIELD_MAPPING)

The result is as follows:
ST_GeomFromWKB(x'01020000000200000000000000000000000000000000000000000000000000F03F000000000000F03F')

But it cannot be executed. It should be like this below in PostGIS:
ST_GeomFromWKB('\x01020000000200000000000000000000000000000000000000000000000000F03F000000000000F03F')

So, would you consider replacing 'ST_GeomFromWKB' with 'ST_GeomFromWKT' to avoid differences in different dialect?

@huanglii, you can write your own evaluate.py file to avoid this issue like I have in this project :smile:
https://github.com/mkeller3/FastCollections/blob/main/api/filter/evaluate.py#L155


Intermittent SSL handshake error when calling S3

Current Behavior
We are experiencing intermittent errors ("SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN") when interacting with the S3 client from ECS in one of our applications. The error seems random and occasional, and it may occur while loading or getting objects from S3.
This is an example of an exception we got while getting an object from S3, right after loading it:
{
    "message": "The SSL connection could not be established, see inner exception.",
    "data": {},
    "innerException": {
        "ClassName": "System.Security.Authentication.AuthenticationException",
        "Message": "Authentication failed, see inner exception.",
        "Data": null,
        "InnerException": {
            "message": "SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.",
            "data": {},
            "stackTrace": "   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
            \n   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)",
            "source": "System.Net.Security",
            "hResult": -2146233088
        },
        "HelpURL": null,
        "StackTraceString": "   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
        \n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
        \n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)",
        "RemoteStackTraceString": null,
        "RemoteStackIndex": 0,
        "ExceptionMethod": null,
        "HResult": -2146233087,
        "Source": "System.Private.CoreLib",
        "WatsonBuckets": null
    },
    "stackTrace": "   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
    \n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
    \n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
    \n   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
    \n   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at MyNameSpace.AwsS3Context.GetContentVersionId(Guid id) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.AwsS3Context.PutEncryptedContentAsync(Guid id, Stream content) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.ManagerClass.SaveContentAsync(Guid id, Stream content) in /filePath/ManagerClass.cs:line XX
    \n   at MyNameSpace.ControllerClass.CreateAsync() in /filePath/ControllerClass.cs:line XX
    \n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
    \n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
    \n   at MyNameSpace.APIClass.Startup.<Configure>b__8_1(HttpContext context, Func`1 next) in /filePath/Startup.cs:line XX
    \n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)",
    "source": "System.Net.Http",
    "hResult": -2146233087
}

S3 client is configured as below:
// At startup
services.AddAWSService<IAmazonS3>();

// S3 client class
public class AwsS3Context : IAwsS3Context
{
    private readonly IAmazonS3 awsS3Client;

    public AwsS3Context(IAmazonS3 awsS3Client)
    {
        this.awsS3Client = awsS3Client;
    }

    public async Task<string> GetContentVersionId(Guid id)
    {
        var getS3Request = new GetObjectRequest
                  {
            BucketName = "bucket_name",
            Key = id.ToString()
        };

        using (var s3Item = await awsS3Client.GetObjectAsync(getS3Request).ConfigureAwait(false))
        {
            return s3Item.VersionId;
        }
    }

    public async Task<string> PutEncryptedContentAsync(Guid id, Stream content)
    {
        var config = new TransferUtilityConfig();

        try
        {
            using (var awsTransferUtility = new TransferUtility(awsS3Client, config))
            {
                var request = new TransferUtilityUploadRequest
                {
                    BucketName = "bucket_name",
                    Key = id.ToString(),
                    InputStream = content,
                    ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256
                };

                await awsTransferUtility.UploadAsync(request, CancellationToken.None).ConfigureAwait(false);
                return await GetContentVersionId(Guid);
            }
        }
        finally
        {
            content?.Dispose();
        }
    }
}

Expected Behavior
Intermittent SSL handshake errors when calling S3 are not expected.
Steps to Reproduce (for bugs)
We are not able to reproduce this error consistently. In order to test, we tried isolating the application in a dev environment provisioning a single container and sending it a regular load for some days, adjusting the request rate and size. After this we got the mentioned exception a couple of times but root cause or conditions that produce the error are not evident.
Context
This is a simple API implemented with ECS that receive different types of files to internally save them in an S3 bucket, providing a resource id to its clients.
Your Environment

AWSSDK.Core version used: AWSSDK.Core (<IP_ADDRESS>)
Service assembly and version used: AWSSDK.S3 (<IP_ADDRESS>)
Targeted .NET platform: .Net Core 2.1 (Microsoft.NETCore.App 2.1.0)
Visual Studio version: VS 2019 (16.4.1)
Operating System and version: Alpine 3.11 (From image mcr.microsoft.com/dotnet/core/aspnet:2.1-alpine)

.NET Core Info

.NET Core version used for development: 2.1
.NET Core version installed in the environment where application runs: 2.1

Seeing the same problem intermittently while reading data from S3

Have the same type of issue when working with Amazon.S3.Transfer.TransferUtility of AWSSDK.S3, v.<IP_ADDRESS>

Getting the same error intermittently. Running on Lambda.
Exception Message: SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.
Stack Trace:
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
  at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
  at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
  at internal method...

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.
UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,
Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?
Thanks,
Ashish

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.
UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,
Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?
Thanks,
Ashish

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.
UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,
Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?
Thanks,
Ashish

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.
Let me know, if you need more information.

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.
UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,
Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?
Thanks,
Ashish

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.
Let me know, if you need more information.

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.
Let me know, if you need more information.

Hi @samiechan,
I tried to reproduce the issue using the t2.micro Ubuntu instance by download the files repeatedly for 3 minutes (to simulate 80 files download) using the code below:
using System;
using System.IO;
using System.Reflection;
using Amazon.S3;
using Amazon.S3.Transfer;
using NLog;
using NLog.AWS.Logger;
using NLog.Config;
using NLog.Targets;

namespace SSLTestConsole
{
    class Program
    {
        static readonly string bucketName = "<<bucket-name>>";
        static readonly string downloadDirectory = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "DownloadedFiles");

        static void Main(string[] args)
        {
            ConfigureNLog();

            Logger logger = LogManager.GetCurrentClassLogger();

            logger.Info($"Checking if local download directory '{downloadDirectory}' exists.");
            if (!Directory.Exists(downloadDirectory))
            {
                logger.Info($"Creating local download directory '{downloadDirectory}'");
                Directory.CreateDirectory(downloadDirectory);
            }

            AmazonS3Client amazonS3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast2);

            logger.Info($"Executing ListObjectsV2Async on bucket '{bucketName}'");
            var listObjevctsV2Response = amazonS3Client.ListObjectsV2Async(new Amazon.S3.Model.ListObjectsV2Request() {
                BucketName = bucketName
            }).Result;
            logger.Info($"Found {listObjevctsV2Response.KeyCount} objects.");

            DateTime endDateTime = DateTime.Now.AddMinutes(3);
            while (DateTime.Now < endDateTime)
            {
                foreach (var s3Object in listObjevctsV2Response.S3Objects)
                {
                    string fileName = Path.GetFileNameWithoutExtension(s3Object.Key).Replace(" ", "_") + "_" + DateTime.Now.ToFileTime() + "." + Path.GetExtension(s3Object.Key);
                    logger.Info($"Downloading '{s3Object.Key}' to '{downloadDirectory}' as '{fileName}'  using TransferUtility");

                    TransferUtility transferUtility = new TransferUtility(new TransferUtilityConfig());
                    transferUtility.DownloadAsync(new TransferUtilityDownloadRequest()
                    {
                        BucketName = bucketName,
                        Key = s3Object.Key,
                        FilePath = Path.Combine(downloadDirectory, fileName)
                    }).GetAwaiter().GetResult();
                }
            }
        }

        static void ConfigureNLog()
        {
            var config = new LoggingConfiguration();

            var consoleTarget = new ColoredConsoleTarget();
            config.AddTarget("console", consoleTarget);

            var awsTarget = new AWSTarget()
            {
                LogGroup = "NLog.ProgrammaticConfigurationExample",
                Region = "us-east-2"
            };
            config.AddTarget("aws", awsTarget);

            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, consoleTarget));
            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, awsTarget));

            LogManager.Configuration = config;
        }
    }
}

.csproj file
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWS.Logger.NLog" Version="2.0.1" />
    <PackageReference Include="AWSSDK.S3" Version="<IP_ADDRESS>" />
  </ItemGroup>
</Project>

But somehow the issue is not reproducible. I have the maximum file size of 200MB, not sure if it would make a difference.
Would it be possible for you to share the sample code for reproduction?
Thanks,
Ashish

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.
Let me know, if you need more information.

Hi @samiechan,
I tried to reproduce the issue using the t2.micro Ubuntu instance by download the files repeatedly for 3 minutes (to simulate 80 files download) using the code below:
using System;
using System.IO;
using System.Reflection;
using Amazon.S3;
using Amazon.S3.Transfer;
using NLog;
using NLog.AWS.Logger;
using NLog.Config;
using NLog.Targets;

namespace SSLTestConsole
{
    class Program
    {
        static readonly string bucketName = "<<bucket-name>>";
        static readonly string downloadDirectory = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "DownloadedFiles");

        static void Main(string[] args)
        {
            ConfigureNLog();

            Logger logger = LogManager.GetCurrentClassLogger();

            logger.Info($"Checking if local download directory '{downloadDirectory}' exists.");
            if (!Directory.Exists(downloadDirectory))
            {
                logger.Info($"Creating local download directory '{downloadDirectory}'");
                Directory.CreateDirectory(downloadDirectory);
            }

            AmazonS3Client amazonS3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast2);

            logger.Info($"Executing ListObjectsV2Async on bucket '{bucketName}'");
            var listObjevctsV2Response = amazonS3Client.ListObjectsV2Async(new Amazon.S3.Model.ListObjectsV2Request() {
                BucketName = bucketName
            }).Result;
            logger.Info($"Found {listObjevctsV2Response.KeyCount} objects.");

            DateTime endDateTime = DateTime.Now.AddMinutes(3);
            while (DateTime.Now < endDateTime)
            {
                foreach (var s3Object in listObjevctsV2Response.S3Objects)
                {
                    string fileName = Path.GetFileNameWithoutExtension(s3Object.Key).Replace(" ", "_") + "_" + DateTime.Now.ToFileTime() + "." + Path.GetExtension(s3Object.Key);
                    logger.Info($"Downloading '{s3Object.Key}' to '{downloadDirectory}' as '{fileName}'  using TransferUtility");

                    TransferUtility transferUtility = new TransferUtility(new TransferUtilityConfig());
                    transferUtility.DownloadAsync(new TransferUtilityDownloadRequest()
                    {
                        BucketName = bucketName,
                        Key = s3Object.Key,
                        FilePath = Path.Combine(downloadDirectory, fileName)
                    }).GetAwaiter().GetResult();
                }
            }
        }

        static void ConfigureNLog()
        {
            var config = new LoggingConfiguration();

            var consoleTarget = new ColoredConsoleTarget();
            config.AddTarget("console", consoleTarget);

            var awsTarget = new AWSTarget()
            {
                LogGroup = "NLog.ProgrammaticConfigurationExample",
                Region = "us-east-2"
            };
            config.AddTarget("aws", awsTarget);

            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, consoleTarget));
            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, awsTarget));

            LogManager.Configuration = config;
        }
    }
}

.csproj file
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWS.Logger.NLog" Version="2.0.1" />
    <PackageReference Include="AWSSDK.S3" Version="<IP_ADDRESS>" />
  </ItemGroup>
</Project>

But somehow the issue is not reproducible. I have the maximum file size of 200MB, not sure if it would make a difference.
Would it be possible for you to share the sample code for reproduction?
Thanks,
Ashish

There was a retry logic added for intermittent SSL_ERROR_ZERO_RETURN issue in AWSSDK.Core <IP_ADDRESS>. Closing this issue.

There was a retry logic added for intermittent SSL_ERROR_ZERO_RETURN issue in AWSSDK.Core <IP_ADDRESS>. Closing this issue.

Why is "retry" a solution?


C implementation of hyperswarm? (IoT / embedded / ESP32)

I think hyperswarm would be great match for IoT devices.
Currently you usualy need cloud to control your smart light bulbs and/or domestic heating.
But with hyperswarm it would be possible to control my home/lab directly without middleman.
If we would be able to run hyperswarm directly on wifi-enabled microcontroller (like ESP32), we would be able to handle all communication through p2p network. But usualy it is only feasible to program such tiny devices in C.
Having hyperswarm ported to C would help a lot.
Also there is already LwIP network stack available on embedded devices, which is relatively advanced and should provide all needed APIs for sockets and raw packets.

I second this :) what would it take, moving some more of hyperswarm to native libs? Wouldn't that be really good for the node hyperswarm in the long run too? It doesn't have to be a separate implementation, it can be an optimization for the main package

I completely agree with your thoughts. The use of hyperswarm for IoT devices could indeed revolutionize the way we interact with our smart devices. The idea of eliminating the need for a cloud-based middleman and enabling direct control over devices through a peer-to-peer network is very appealing.
The potential of running hyperswarm on a wifi-enabled microcontroller like ESP32 opens up a whole new realm of possibilities. It would make the communication process more efficient and direct, which is a significant advantage in the world of IoT.
Porting hyperswarm to C would indeed be beneficial, considering that most microcontrollers are programmed in C due to its efficiency and control over hardware. This would make hyperswarm more accessible and usable in the realm of embedded systems.
The availability of the LwIP network stack on embedded devices is another advantage, as it provides all the necessary APIs for sockets and raw packets. This would make the implementation of hyperswarm on such devices much smoother.
In conclusion, your idea holds a lot of promise and could potentially lead to significant advancements in the field of IoT. It’s an exciting prospect to consider!

For reference sake, here is someone that compiled a realtime PC/MAC library for ESP32
https://github.com/torsoelectronics/link-esp
This is the ableton link library, it's a UDP metronome sync between ocmputers using Kalman filters

There are also some javascript projects for ESP32, that might be a lead if we can somehow make them work with the same native libraries

I feel that some of the work is already done here... I see there's https://www.moddable.com/
There will also be other js implementations, can we rip the beating heart of hyperswarm out of node and put it in one of these other platforms?

This sounds like exactly what we need for hyperswarm on esp @mafintosh do you think this sounds feasible?

The more I think about this idea the more it makes sense, imagine pairing a little usb hotspot with a esp device, and programming it to rove around, you could have remote controlled drone vehicles and extend their range anywhere where there's internet signal


Incorrect Username or Password

I get this error after any clean installation on my CentOS 6.6 x64 with PHP 5.3.3
I've reinstalled Codiad between 15 to 20 times trying any kind of workaround:

Disabling APC cache
Bypassing the sha1|md5 encryption for the password in /components/user/class.user.php
Deleting several times all files in /data
Trying different and longer username and password
Disabling any php extension
Removing all the php session files from /var/lib/php/session
And other 10-15 tries so on...

but I had absolutely no luck.
Every time i reinstall it from git Codiad lets me create the 1st project, but when I try to login after such creation it states ALWAYS "Incorrect Username or Password".
It's 3 days I'm trying to find a fix for this but no way.

Hmm, hard to say. Could you check your browers console to see what cames back from the request?

I've found the problem: looks like it works only with php until 4.3.x while stops working with php 4.4.x or newer.

I currently cant test it with 4.4.x cause I am using 5.6.x

Agree: I tested on one of my customers' vps with a 5.5.x and it works smoothly.
It has problems with 4.4.x for sure. Can't say which problem tho...

Maybe you coudl write it in the Readme, so everybody now it.

It would be more interesing to fix this, then to add it to the readme ;-)
Is there any error thrown in the browser console during the login? Please check the http requests and their response.

This is not a minor bug, seems quite major because php7 has the same issue.
The issue is that the config.php gets emptied and the user.php etc in the data folder also is compleatly empty.
But I do not know why it does that, still makes me unable to install Codiad on my php7 server.

I got the same issue. Cannot login after setup. I am using php 5.4

I've got this issue recently and that's what I see in console:
Uncaught TypeError: Cannot read property 'resize' of undefined at system.js:5 at dispatch (jquery.min.js:3) at i (jquery.min.js:3)
Tried on PHP 5.6 and PHP7


Remote buttons too small

Hi, since version 2.2.0 the remote buttons turned from this:

to this:

I think it started with https://github.com/xbmc/Kore/issues/299. That guy and I share a similar screen resolution, but with my 54 more vertical pixel I never had overlapping problems. Can this be somehow fine-tuned? Right now I'm on 2.1.0 because I never get the correct button with the small version.

I believe the way it is now is as it was originally designed. I wouldn't mind increasing the remote size. However, I'm afraid everyone else who was always using the correct size will start complaining...

It's also my opinion that the PR that changed this changed it for the better, and it should be kept. It is also worth pointing out that, even at the smaller resolutions, the remote buttons are bigger (52dp) than what Google recomends as the minimum (48dp).
Nevertheless, maybe they can be tweaked somewhat in smaller resolutions. Currently they are 52dp, we could try adding 4-8 dp.

I made a side by side comparison:

To me it seems that 2.1.0 is way more similar to the Google Play screenshot.
I also made a comparison of the remote with my thumb on the screen (I only have my old netbook webcam...):

As you can see, with 2.2.0 my thumb almost covers all the remote buttons.
I'm not saying this should be reverted, because button overlapping is in fact a problem. Maybe vertically center the buttons could be enough? Oh well, I made another comparison :)

I think that vertically centering the buttons would be worse, as it would make the buttons further away from the natural position of the thumb.
What device are you using and what dp does it have?

I have no idea what dp is, but resolution is 480x854, size 4.5". The phone is an ASUS Zenfone 4, A450CG. More info here: http://www.gsmarena.com/asus_zenfone_4_a450cg-6428.php

@poisdeux This looks like it's falling on the sw320dp bucket. Do you think we can increase the 58 dp (and maybe increase the 360dp to the same as 400dp)?
I'm only testing on 400dp and it looks good. Did you test it on other devices?

I'm sorry, I understand I explained myself poorly. I know nothing about Android development and I have no idea what "dp" stands for.
Unfortunately I have no other smartphone, could Genymotion be of any help?

I tested on other devices using virtual machines.
Below a screenshot on a device resembling the Samsung Galaxy S2 plus
And here a screenshot on a device resembling the ASUS Zenfone 4

Now for the buckets they fall into:
ASUS Zenfone 4: sqrt(480^2 + 854^2)/4.5 = 217 dpi  ==>  scale factor = 217/160 = 1.35 ==> sw = 480/1.35 = 355  matching on our sw360 dimensions.
Samsung G2 plus: sqrt(480^2 + 800^2)/4.3 = 217 dpi  ==>  scale factor = 217/160 = 1.35 ==> sw = 480/1.35 = 355
So they both fall into the sw360 definition and both are sized the same as you can see from the screenshots.
Unfortunately, as can be seen from the Galaxy S2 screenshot there isn't much room to increase the remote size. I think the only way we can solve this is by refactoring the remote code and make it responsive. I haven't thought this through, but making the background use a nine-patch image we should be able to scale it depending on the available width and height.

@poisdeux beware that ASUS phones don't use on-screen navigation bar as you can see in my screenshots, so there is much more room.

@shiba89 Yup, I see. On your phone there is MUCH more room. We need to redesign the remote to be able to dynamically use as much space as available. It is now sized statically based on a few definitions that depend on the smallest width of the device. In your case the smallest width is 480 and that matches on Kore's smallest width definition for 360 (matched on actual pixel size).

Looking at the screenshot i suspect there's something else going on...
@shiba89 Which Asus zenfone 4 do you have? You linked one with a 4.5" screen and 480x854 resolution, but there's this one:
https://www.asus.com/Phone/ZenFone-A400CG/specifications/
which has a 4" screen and 480x800 resolution.
@poisdeux This last one as a dpi of 233 and dp 330 which probably falls in the sw320 definition. Looking at the initial screenshots that shiba posted it's more consistent with this hypotheses.

@SyncedSynapse A450CG, 4.5". I chose it over A400CG because IPS vs TFT screen, so I'm perfectly sure of the model.

I implemented a scalable control pad (remote buttons) in PR #428. Hopefully this will work for everyone.
Below a screenshot from a Android Virtual Device resembling a Samsung Galaxy S2.

And a screenshot from a AVD resembling an Asus Zenfone 4.

Update: we tried to release a fix for this in 2.3.0 but unfortunately that didn't work out very well.
See #444 Hopefully we get it right the next time.

Hi, I have a Samsung Galaxy Core Prime (480x800 res) and the update caused me the same issue. Is there a safe place I can get an apk for 2.1.0?  Thanks!

#477 should fix the issue and will be included in the next release.
The fix is already available in the nightly build http://mirrors.kodi.tv/tools/kore/nightlies/

Thanks for the link! The new build does fix the size issue, but ultimately I opted for 2.1.0 because it has a more centered control pad. The new one is too close to the shortcuts below, at least on my phone.

Could you upload a screenshot? We could add some extra margin to make it more useable.


On What to Watch page (maybe other pages), YouTubeCenter for Chrome consumes excessive CPU.

I am currently running YouTube Center Developer Build 529, installed using TamperMonkey, on Chrome 43.0.2357.134, running on Kubuntu Linux 15.04.
I habitually keep a Youtube tab open and pinned, usually on the What To Watch page, or the home page for my Youtube account. Usually, what happens is that Youtube Center seems to work correctly at first, and I can surf, click around, watch videos.
However, if I just let the page sit, say if I go browse another tab, and then come back some time later, I notice that my machine's fans are running fast, and I check using a CPU meter app, and find that one of my CPU cores is at 100% usage. When I go into Chrome, I just hit Shift-Esc to summon Chrome's task manager, and I find my Youtube page is inexplicably eating one of my CPU cores. Surfing to the page, I find it extremely laggy, if it's not hung entirely. Closing the tab and re-opening Youtube fixes the problem, temporarily...
I get the impression that somewhere, Youtube Center is caught in an infinite loop. Maybe it's spinning trying to find a resource that was available, but is no longer, say because the contents of What To Watch got changed (which Google does all the time.)
How to reproduce this error? Surf to your Youtube home page, and go to What To Watch, then go surf on another tab, and come back after a while. Try using the page, especially scrolling down, which induces it to load more videos. I've found that a combination of letting the page sit for a long time, then trying to scroll down, or use Youtube Center features, say by mousing over videos to show their resolutions, & votes, will often trigger this bug.
Also. Try leaving a What To Watch page open for a while, then click on a video, watch the video, then click Back to return to the What To Watch page, which returns you to cached content, rather than fresh content.

I have a similar problem, upon loading the What to Watch page, the page will load fine for a few seconds, then freeze for another few seconds, before it is fully loaded. This only occurs on the What to Watch page. I haven't left it on long enough to notice any lag or cpu consumption after it's fully loaded though.

caeldom - I'm noticing that too. The What To Watch page loads, then the page freezes for a while, presumably while Youtube Center is processing stuff, then eventually unfreezes

Yes, there are several pages that just freeze up with YT Center now. You aren't alone with this and the persistent freezing, with nothing being done by the developer to sort it, forced me to stop using the script. The only time the script didn't freeze on page loads, was the short time after an update had been performed, which didn't last very long. I have no idea if it's a Tampermonkey problem, or it's actually a problem with the script itself.
I don't see the issue of excessive CPU / RAM usage being solved any time soon.

Good to know others are experiencing the same issue!


package installed but at run time getting error Plugin project :capturesdk_flutter not found. Please update settings.gradle.

package integrate error:
Plugin project :capturesdk_flutter not found. Please update settings.gradle.
FAILURE: Build failed with an exception.

Where:
Build file 'D:\CaptainChef_pos\android\build.gradle' line: 26

What went wrong:
A problem occurred evaluating root project 'android'.

A problem occurred configuring project ':app'.
Cannot invoke method afterEvaluate() on null object

@socketPM We're getting the same error following the steps on the developer portal. I get the build error in a fresh Flutter test project with the following steps:

flutter --version outputs:Flutter 3.19.6 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 54e66469a9 (3 weeks ago) • 2024-04-17 13:08:03 -0700
Engine • revision c4cd48e186
Tools • Dart 3.3.4 • DevTools 2.31.1

flutter create vanilla
cd vanilla
Edit pubspec.yaml as described in the developer portal using version ^1.5.0 (intentionally not reproducing the contents here since they contain credentials)
flutter build apk

Output:
Plugin project :capturesdk_flutter not found. Please update settings.gradle.

FAILURE: Build failed with an exception.

* Where:
Build file '[REDACTED]/vanilla/android/build.gradle' line: 13

* What went wrong:
A problem occurred evaluating root project 'android'.
> A problem occurred configuring project ':app'.
   > Cannot invoke method afterEvaluate() on null object

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

@dbravender-snappy these two seem odd, have you tried compiling our sample app?

Can you please share the integration process for socket developer?

@Raz4492 I'd assume you have gone through the documentation, but just in case, you can find it here: https://docs.socketmobile.com/captureflutter/en/latest/gettingStarted.html#sdk-installation .
Also, do you mind sharing your podfile (you can dact the none relevant parts) and send to developers[at]socketmobile.com. Also, what is the minimum version you are targeting? This Flutter version requires iOS 12.

@Raz4492 Just in case you haven't gone through the documentation, you can find it here: https://docs.socketmobile.com/captureflutter/en/latest/gettingStarted.html#sdk-installation

have you tried compiling our sample app?

Yes, I think the difference might be the Flutter version. I can't get it to compile using Flutter 3.19.6.

@dbravender-snappy Please try the following first and see if it works? And for followup discussions, let's move to developers[at]socketmobile.com.
dependencies:
flutter:
sdk: flutter
...
capturesdk_flutter:
git:
url<EMAIL_ADDRESS>version: ^1.5.11
...

Our new version 1.5.41 of the Flutter CaptureSDK is back on Github so you don't need the personal access token anymore and you can use directly the package as this in your pubspec.yaml:
dependencies:
flutter:
sdk: flutter
...
capturesdk_flutter: ^1.5.41
...
Let us know if that works for you now


Issue #32 still not fixed.

Stop creating hundreds of empty issues, thank you

Stop making bs comment that don't add anything to the subject.

Stop making bs comment that doesn't add anything to the subject. Edit: The issues aren't empty.

I see you are very polite, good on you. I'm not on tight terms with the creator but that only makes me wish less that Sergei actually spends his time and effort on the mod only to release it for people like you.
Does your comment adds anything to the subject? If you're so worried about efficient dialogue, go to C:DDA devcord or something similiar.
LDYOM is a small project currently maintained by one single person. There is literally no serious reason to solve everything exclusively through Github as you can just contact him on directly on Telegram & Discord, platforms where he actually appears online more.
Alternatively, you can organize everything into one big issue that I can unsubscribe from freely instead of creating heaps of issues that clutter up my e-mail and e-mails of any other person subscribed to the repository.
 Secondly, I ain't forcing your husband, Sergei to fix these issues. Third, didn't ask for your advice on what to join or not & Fourth, I won't be making accounts just 'cause you, an irrelevant person, doesn't like it.
Oh, Its cluttering your inbox? Well boo-hoo maybe you should like turn off notifications for GitHub altogether or beg your husband to remove you as a contributor, you have hardly contributed anything to the "project". On a legit note, if you really wanted to not receive notifs via email you could have turned it off, but instead you decided to ping me & cry here.
Have a hard stool ;)

Sergei, it's me. And I think we should end this discussion. I must admit that so many issues are not very informative. It is better to write the problem exactly in the description, just as you have made checkboxes also problems. Because there are a lot of issues and I'm starting to get confused myself.

Do you really know this clown? He hasn't really helped with anything in the past 2 YEARS and only now keeps crying about his inbox getting filled. Questioned my "PoLiTeNEsS" and isn't any better himself.

I must admit that so many issues are not very informative.

Why not a single time have you mentioned this before to me? @GeTechG If you have any problem with me talk directly to me.

Why not a single time have you mentioned this before to me?

I didn't ask anyone to write. I respond to you as I get comfortable. It wasn't that important before, but now I'm telling you this.

He hasn't really helped with anything in the past 2 YEARS
I don't want no other irrelevant person, who hasn't contributed to anything in over 2yrs, tell me bs.

Why do you keep bringing it up? I contributed to repo where I could, when I could, and if there isn't anything I can help with, it's better to stay aside. I've been in touch with GTG ever since through the platforms you dismiss (linked at the end of README!). I tried warming up some hype at a GTA:UG discord and in Russian DYOM community at VK. I provided feedback beyond Git and we all discussed, as of latest for example, how to pull off menu scaling as for end-user. The only reason I'm "irrelevant" is because you flag me as that, someone who came down 3 weeks ago and spent less time and work for the project.
I don't want to sound like some elitist and I hope I don't. My initial comment might sound rude as it's brief and sounds subjective but mutual rudeness only births war. That's not productive. Us moving on, me learning C++ for new ways to contribute, and you providing feedback in a better format would be much more productive.
I'm not pushing this piss-poor argument further. I hope you got what I meant.

Ok. So you want to bs...
Lets go over this "piss-poor" reply.

I contributed to repo where I could, when I could , and if there isn't anything I can help with, it's better to stay aside.

I don't see any useful ldyom-related activity on your profile in over 2 years, except this... this is the most active you have ever been in over 2 years & that too just to cry over GitHub email notifs; Could have reported bugs in preview builds, but no.

I've been in touch with GTG ever since through the platforms...

...& what am I supposed to do? Clap for you? Staying in touch and not doing nothing, doesn't contribute to anything.

I tried warming up some hype at a GTA:UG discord and in Russian DYOM community at VK.

Oh yeah, yeah, I can see it went really well. There's so much hype here.

I provided feedback beyond Git and we all discussed, as of latest for example, how to pull off menu scaling as for end-user.

Yes, the menu scaling. You could have contributed by coding it yourself i.e. if you really knew "how to pull off menu scaling" at all, of which I highly doubt you know anything.

The only reason I'm "irrelevant" is because you flag me as that, someone who came down 3 weeks ago and spent less time and work for the project.

Ah, so you are crying because I called you "irrelevant", which you obviously are, but is just in denial, so my words do affect you to a degree where you would make an entire paragraph explaining yourself to randos on the internet, who joined just 3 weeks ago and have contributed to the project more than you ever have in the last 2 years. I made 2 beta preview releases happen with various bug fixes. This is the funniest point, ngl xD

I don't want to sound like some elitist and I hope I don't. My initial comment might sound rude as it's brief and sounds subjective but mutual rudeness only births war. That's not productive.

Us moving on, me learning C++ for new ways to contribute, and you providing feedback in a better format would be much more productive.

 Also, yeah, you don't need to tell me what I have to do.

I'm not pushing this piss-poor argument further. I hope you got what I meant.

I mean... Yeah, you definitely, totally won't be continuing this "piss poor" argument any further after this 3rd comment you posted 6 hours later on a post that was created yesterday. Totally. @cherdaq

I must admit that your current behavior is also unpleasant. I'll repeat myself: I won't tolerate hatred towards anyone. Therefore, I'm forced to close this commit to stop further insults. @cherdaq tried to apologize for their initial comment and made an effort to meet you halfway, which you did not reciprocate. I have no connection to him; I act solely based on my own considerations. As for discussing the contributions made by him and you, I don't want to engage in that. I appreciate any help as long as it's not based on a competition of who has done more. I don't want assistance that involves measuring, excuse my language, "d*cks." Everyone has the right to contribute if they want to, and they do it out of personal enthusiasm. If you don't like something, that's your right; you don't have to participate.