Add support for keyfiles instead of username/password?

Can you add support for keyfiles instead of username/password?
See https://github.com/snowflakedb/snowflake-connector-net (possibly skip private_key_pwd)
conn.ConnectionString = "account=testaccount;authenticator=snowflake_jwt;user=testuser;private_key_file={pathToThePrivateKeyFile};private_key_pwd={passwordForDecryptingThePrivateKey};db=testdb;schema=testschema";

Originally posted by @Ekus in https://github.com/fixer-m/snowflake-db-net-client/issues/1#issuecomment-969158440

It will be great to have this. Any planned release date? @fixer-m

I created the following class to generate the necessary JWT, based on the following: https://docs.snowflake.com/en/developer-guide/sql-api/authenticating
Once you have the token, you would just add these headers to the request:
Authorization: Bearer <jwt>
X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT

This is .NET6 code that uses the System.IdentityModel.Tokens.Jwt package by Microsoft.  It was a little bit of a battle to find all of the right parameters, but it works for both plain and encrypted keys generated based on the Snowflake docs here: https://docs.snowflake.com/en/user-guide/key-pair-auth
NOTE: the PrivateKeyPem input expects to have the BEGIN and END tokens from the files - like these:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIE6TAbBgkqhkiG9w0BBQMwDgQILYPyCppzOwECAggABIIEyLiGSpeeGSe3xHP1
wHLjfCYycUPennlX2bd8yX8xOxGSGfvB+99+PmSlex0FmY9ov1J8H1H9Y3lMWXbL
...
-----END ENCRYPTED PRIVATE KEY-----

public class SnowflakeJwt
{
    public string Account { get; set; }
    public string User { get; set; }
    public string PrivateKeyPem { get; set; }
    public string PassPhrase { get; set; }

    public string QualifiedUsername
    {
        get { return $"{Account}.{User}".ToUpper(); }
    }

    public SnowflakeJwt(string account, string user, string privateKeyPem, string passPhrase = null)
    {
        Account = account;
        User = user;
        PrivateKeyPem = privateKeyPem;
        PassPhrase = passPhrase;
    }

    public string GetToken(TimeSpan lifetime)
    {
        string publicKeyFingerprint;

        using RSA rsa = RSA.Create();
        if (string.IsNullOrEmpty(PassPhrase))
        {
            // non-encrypted PEM
            rsa.ImportFromPem(PrivateKeyPem);
        }
        else
        {
            // encrypted PEM
            rsa.ImportFromEncryptedPem(PrivateKeyPem, Encoding.UTF8.GetBytes(PassPhrase));
        }

        // generate the public key fingerprint
        using (SHA256 sha256 = SHA256.Create())
        {
            publicKeyFingerprint = "SHA256:" + Convert.ToBase64String(
                sha256.ComputeHash(
                    rsa.ExportSubjectPublicKeyInfo()
                    )
                );
        }

        // define the token
        DateTime now = DateTime.UtcNow;

        var tokenHandler = new JwtSecurityTokenHandler();
        var tokenDescriptor = new SecurityTokenDescriptor
        {
            IssuedAt = now,
            Expires = now + lifetime,
            Issuer = QualifiedUsername + "." + publicKeyFingerprint,
            Subject = new ClaimsIdentity(new List<Claim> { new Claim("sub", QualifiedUsername) }),
            SigningCredentials = new SigningCredentials(new RsaSecurityKey(rsa), SecurityAlgorithms.RsaSha256)
            {
                CryptoProviderFactory = new CryptoProviderFactory { CacheSignatureProviders = false }
            }
        };

        // generate the token
        var token = tokenHandler.CreateToken(tokenDescriptor);

        return tokenHandler.WriteToken(token);
    }
}


Update hash of gstreamer linux deps

This change is

@bors-servo r=jdm

:pushpin: Commit 4ec6031 has been approved by jdm

:hourglass: Testing commit 4ec6031c9e76fb4d4606a80a1a9c4556243fb49b with merge 15337c56de0385a9e88a3bf0350db130aa5bddf9...

:sunny: Test successful - status-travis
Approved by: jdm
Pushing 15337c56de0385a9e88a3bf0350db130aa5bddf9 to master...


Choose directory before execution

adjust the execute_workspace function to prepend the directory before the command. #103

Thanks @Snoup97
I think it doesn't fully work since && will not work if it starts with a comment...
Since I already had a diff underway, I merged #123 this time!

Hi @AntonOsika
Thanks for the feedback.


add hb checkins for cron

Why was this change made? 🤔
Fixes #2869 -- add HB checkins for cron jobs.  Modeled on sul-dlss/preservation_catalog#2075
Config update for production in sul-dlss/shared_configs#1876

@peetucket I'm not entirely sure this is what I'm seeing, since I'm new to HB check-ins, but looking at https://app.honeybadger.io/projects/77112/check_ins it appears that H2's check-ins may not be reporting in, and it looks like only prod has been wired up in shared_configs (no stage or QA) unlike with our prescat HB setup. (I'm poking around here because I picked up the ticket to add check-ins to DSA.)

I believe the reason we have seen no check-ins yet for H2 is because this code was merged to main after we split the production branch off (currently running in prod).  That branch has no check-ins yet: https://github.com/sul-dlss/happy-heron/blob/production/config/schedule.rb
I tested one of the check-ins manually by curling from the H2 prod server and HB did pick it up so I think it should work when we deploy main of H2 to prod.
We can discuss if we want H2 stage and QA to check-in as well.  My recollection is that we didn't think it was that useful, but I could be wrong.

@peetucket OK, thank you!


Add assignment hooks for global variables

Relates to #1551
This is a quick and dirty PoC to hook a function into the assignment of globals. This PoC only adds a validation, but we can extend this to make it set other globals too, or use the same idea to get the globals.

Deleting this experiment in favour of  #1853.


Security bug with Commons FileUpload package in pom.xml

"The MultipartStream class in Apache Commons Fileupload before 1.3.2, as used in Apache Tomcat 7.x before 7.0.70, 8.x before 8.0.36, 8.5.x before 8.5.3, and 9.x before 9.0.0.M7 and other products, allows remote attackers to cause a denial of service (CPU consumption) via a long boundary string."
Recommendation is to upgrade to version Apache Commons Fileupload - 1.3.2
Location
ng-file-upload/demo/pom.xml
Sources
https://bugzilla.redhat.com/show_bug.cgi?id=1349468
https://www.debian.org/security/2016/dsa-3614

Or even better if demo folder doesn't ship in npm package.


Previously working feeds now only returning null with HTTP code 200

For example, Justin's own feed https://www.justinscarpetti.com/projects/amazon-wish-lister/api/?id=37XI10RRD17X2 currently only returns "null" in a 200 "OK" response. This broke within an hour after 2021-07-30 20:05 US/Pacific time.

Seems to have resolved just before 2021-07-31, 22:05 US/Pacific.


Bugfix: fail to capture rowset for version xxx

Issue number #227
The Tablet::capture_consistent_rowset() method is used in
Reader::prepare() method to get rowsets of specified version.
Tablet::capture_consistent_rowset() will check the _stale_rs_meta_map,
so rowsets that have been compacted but not yet been deleted can still be found.
This PR also splits the original init function into two functions: prepare and open.
The prepare function is used to fetch specific version of rowsets, while the open
function is used to open those rowsets.
The prepare function is usually called before query processing begins, while the open
function is called during query processing.
A single query may involve multiple tablets and may take a long time to execute. By
opening all the rowsets of the tablets sequentially before the query execution, it is
ensured that these rowsets will not be deleted during the execution, while the open
function can be called by multiple threads during the query execution to reduce the
latency of rowset opening.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@chaoyli PTAL

Good Job


Unable to find repo details from the local repository

Hello,
I am getting the error "Unable to find repo details from the local repository" but there is in fact a file .sastscanrc in /app.
I can see that a file findsecbugs-report.xml was generated but there is just one finding.
1. Why there is hust one finding? it seems the scanners stopped prematurely.
2. Also, how can I get findsecbugs-report.xml exported to a SARIF?
docker run --rm -e "WORKSPACE=%cd%" -v "C:\maven\apache-maven-3.6.3\bin" -v "C:\source_code\app":/app appthreat/sast-scan scan --src /app  --type java

/ _ \          |_   | |                  | |
/ /\ _ __  _ __ | | | |__  _ __ ___  __ | |
|  _  | '_ | '_ | | | '_ | '/ _ / ` | __|
| | | | |) | |) | | | | | | | |  __/ (| | |_
_| |_/ ./| ./_/ || ||_|  _|_,|_|
| |   | |
||   |_|
INFO [2020-04-01 13:26:44,486] Scanning /app using scan plugins ['credscan', 'java']
INFO [2020-04-01 13:26:44,494] ================================================================================
INFO [2020-04-01 13:26:44,496] ⚡ Executing "gitleaks --depth=2 --repo-path=/app --redact --timeout=2m --report=/app/reports/credscan-report.json --report-format=json"
INFO[2020-04-01T13:26:44Z] no leaks found, skipping writing report
INFO[2020-04-01T13:26:44Z] No leaks detected. 0 commits audited in 353 milliseconds 67 microseconds
INFO [2020-04-01 13:26:44,896] ================================================================================
INFO [2020-04-01 13:26:44,897] ⚡ Executing "java -jar /opt/spotbugs/lib/spotbugs.jar -textui -include /usr/local/src/spotbugs/include.xml -exclude /usr/local/src/spotbugs/exclude.xml -noClassOk -auxclasspathFromFile /tmp/tmpl891l6xd -sourcepath /app -quiet -medium -xml:withMessages -effort:max -nested:false -output /app/reports/findsecbugs-report.xml /app"WARNING [2020-04-01 13:27:02,662] Unable to find repo details from the local repository. Consider adding a local .sastscanrc file with the url details.
WARNING [2020-04-01 13:27:02,680] Project type is not supported: java
Thanks,

Hello @prabhu
I figure it out myself how to continue on error using the flag: --no-error, so I now have a SARIF file.
Still not sure why there is only one finding from find-sec-bugs and nothing else. So I still believe teh scan process is prematurely.
On a side note, I noticed 3 files are created, incl a JSON, a XML, a HTML and a SARIF file.
But I did not see anything with regards to Grafeas.
I know sasts-scan uses internally scan-reports which generates data in Grafeas format.
I can see in requirements.txt that scan-reports gets installed but I can't figure it where (dir) is located in the container filesystem to use it. I inspecte dthe container file system and inside lib there are only python files, while one level up there are system files (incl. cpp debug, eclipse, games, gcc
gems, golang, java, jvm, jvm-commmon, jvm-private, locale, modprobe.d, modules
node_modules, python3.6, rpm, sse2, swidtag, sysimage, system, tmpfiles.d and udev) but I could not see scan-reports.
Let me re-cap the questions as per update:
1. Why there is hust one finding? it seems the scanners stopped prematurely.
2. How to get data in Grafeas using scan-reports?
Thanks,
Javier Caceres

Hi @jacace,
For java projects, we must compile the project first before invoking the scan. The warning message you are seeing is to indicate the fact that the tool could not find the repository related information possibly because the folder is not a git repository? You can ignore the warning.
Grafeas is part of dependency scan. To enable it simply pass depscan along with type
Eg: --type credscan,java,depscan . You can also ignore the type parameter to enable automatic detection. With depscan the vulnerability information will appear in grafeas format. To improve vulnerability scanning you can pass GITHUB_TOKEN as an environment variable.
You can refer to some examples such as: https://github.com/AppThreat/WebGoat/blob/develop/azure-pipelines.yml#L23
https://github.com/AppThreat/sast-scan/blob/master/.github/workflows/pythonapp.yml#L34
Hope this helps.

To answer your other question regarding scan-reports it is installed via pip install to /usr/local/lib/python3.6/site-packages/
docker run --rm -e "WORKSPACE=${PWD}" -v $PWD:/app appthreat/sast-scan python3 -c "import sys; print(sys.path)"
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages']

docker run --rm -e "WORKSPACE=${PWD}" -v $PWD:/app appthreat/sast-scan python3 -c "import reporter; print(reporter.__file__)"
/usr/local/lib/python3.6/site-packages/reporter/__init__.py

Also, you might be having quite an old build for sast-scan. The new build no longer has that particular message - https://github.com/AppThreat/sast-scan/blob/master/lib/context.py#L85
A simple docker pull should resolve the issue.

thanks @prabhu !
I just pulled latest version but still seeing same error and unable to get a grafeas file as per detailed error log below:
Please note:

This is a indeed a (private) GitHub repo
The compiled source code is in: app/target/webapi-1.3.jar (i.e.: C:\app_source_code\target\webapi-1.3.jar).
The latest docker image version is reading correctly the file .sastscanrc

This is my comman dto run the image:
docker run --rm -e "WORKSPACE=%cd%" -v "C:\apache-maven-3.6.3\bin" -v "C:\app_source_code":/app appthreat/sast-scan scan --src /app --type credscan,java,depscan
Am I doing anything wrong? Thanks.
Scan log Below.

/ _ \          |_   | |                  | |
/ /\ _ __  _ __ | | | |__  _ __ ___  __ | |
|  _  | '_ | '_ | | | '_ | '/ _ / ` | __|
| | | | |) | |) | | | | | | | |  __/ (| | |_
_| |_/ ./| ./_/ || ||_|  _|_,|_|
| |   | |
||   |_|
INFO [2020-04-01 17:50:46,735] Scanning /app using scan plugins ['credscan', 'java', 'depscan']
INFO [2020-04-01 17:50:46,738] ================================================================================
INFO [2020-04-01 17:50:46,739] ⚡ Executing "gitleaks --depth=2 --repo-path=/app --redact --timeout=2m --report=/app/reports/credscan-report.json --report-format=json"
INFO[2020-04-01T17:50:47Z] no leaks found, skipping writing report
INFO[2020-04-01T17:50:47Z] No leaks detected. 0 commits audited in 265 milliseconds 279 microseconds
INFO [2020-04-01 17:50:47,045] ================================================================================
INFO [2020-04-01 17:50:47,045] ⚡ Executing "java -jar /opt/spotbugs/lib/spotbugs.jar -textui -include /usr/local/src/spotbugs/include.xml -exclude /usr/local/src/spotbugs/exclude.xml -noClassOk -auxclasspathFromFile /tmp/tmpjdfyjkyw -sourcepath /app -quiet -medium -xml:withMessages -effort:max -nested:false -output /app/reports/findsecbugs-report.xml /app"
WARNING [2020-04-01 17:51:01,967] Unable to find repo details from the local repository. Consider adding a local .sastscanrc file with the url details.
WARNING [2020-04-01 17:51:01,974] Project type is not supported: java
INFO [2020-04-01 17:51:01,975] ================================================================================
INFO [2020-04-01 17:51:01,975] ⚡ Executing "/usr/local/bin/depscan --no-banner --src /app --report_file /app/reports/depscan-report.json"
INFO [2020-04-01 17:51:02,880] ================================================================================
INFO [2020-04-01 17:51:02,893] ⚡ Executing "cdxgen -r -t java -o /app/reports/bom-java.xml /app"
WARNING [2020-04-01 17:51:04,506] Unable to parse /app/reports/bom-java.xml no element found: line 1, column 0
WARNING [2020-04-01 17:51:04,507] No packages found in the project!
INFO [2020-04-01 17:51:04,845] To use GitHub advisory source please set the environment variable GITHUB_TOKEN!
Thanks,

Can we do a zoom meeting since something is not looking right on windows? Please email me: prabhu @ appthreat.com

thanks @prabhu
Just FYI: I tried using the repository URL and github token too but got same result.
Assuming no git clone is required (since the code is already in C:\app_source_code) then the question is:
What's the structure expected in the source code local folder?
In my case I have pretty standard folders e.g.:
C:\app_source_code.github\workflows\main.yml
C:\app_source_code.mvn\wrapper
C:\app_source_code.settings
C:\app_source_code\target -> here the jar
C:\app_source_code\src\main\java\com -> here folders mapping the package structure
C:\app_source_code.sastscanrc
C:\app_source_code\pom.xml
etc...
docker run -v "C:\apache-maven-3.6.3\bin" -v "C:\app_source_code":/app -e "REPOSITORY_URL=https://github.com/myuser/myreoi.git" -e "GITHUB_TOKEN=" appthreat/sast-scan:latest scan --src /app

I have so far tested with .class files so invoking mvn compile before running the tool definitely works. The command you are using is fine (although mapping apache maven is not required).
Can you try with this image quay.io/appthreat/sast-scan:feature_mvn instead of appthreat/sast-scan:latest. I have added some debugs now.
docker run -v "C:\app_source_code":/app -e "REPOSITORY_URL=https://github.com/myuser/myrepo" -e "GITHUB_TOKEN=" quay.io/appthreat/sast-scan:feature_mvn scan --src /app

Hello @prabhu
I just tried the with the image suggested but deepscan neither BOM worked -  In can see it in the log.
fyi: I am on a Windows 10 laptop wunning Linux containers, not sue if this makes a differende.
docker run -v "C:\app_source_code_root":/app -e "REPOSITORY_URL=https://github.com/myuser/fruvyappApi" -e "GITHUB_TOKEN=" quay.io/appthreat/sast-scan:feature_mvn scan --src /app --no-error

/ _ \          |_   | |                  | |
/ /\ _ __  _ __ | | | |__  _ __ ___  __ | |
|  _  | '_ | '_ | | | '_ | '/ _ / ` | __|
| | | | |) | |) | | | | | | | |  __/ (| | |_
_| |_/ ./| ./_/ || ||_|  _|_,|_|
| |   | |
||   |_|
INFO [2020-04-01 19:07:54,690] Scanning /app using scan plugins ['credscan', 'java', 'depscan']
INFO [2020-04-01 19:07:54,702] ================================================================================
INFO [2020-04-01 19:07:54,702] ⚡ Executing "gitleaks --depth=2 --repo-path=/app --redact --timeout=2m --report=/app/reports/credscan-report.json --report-format=json"
INFO[2020-04-01T19:07:54Z] no leaks found, skipping writing report
INFO[2020-04-01T19:07:54Z] No leaks detected. 0 commits audited in 81 milliseconds 733 microseconds
INFO [2020-04-01 19:07:54,999] ================================================================================
INFO [2020-04-01 19:07:55,000] ⚡ Executing "java -jar /opt/spotbugs/lib/spotbugs.jar -textui -include /usr/local/src/spotbugs/include.xml -exclude /usr/local/src/spotbugs/exclude.xml -noClassOk -auxclasspathFromFile /tmp/tmpj6kfxca2 -sourcepath /app -quiet -medium -xml:withMessages -effort:max -nested:false -output /app/reports/findsecbugs-report.xml /app"
INFO [2020-04-01 19:08:16,775] ================================================================================
INFO [2020-04-01 19:08:16,776] ⚡ Executing "/usr/local/bin/depscan --no-banner --src /app --report_file /app/reports/depscan-report.json"
INFO [2020-04-01 19:08:18,394] ================================================================================
Traceback (most recent call last):
File "/usr/local/bin/depscan", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/depscan/cli.py", line 134, in main
create_bom(project_type, bom_file, args.src_dir)
File "/usr/local/lib/python3.6/site-packages/depscan/lib/bom.py", line 132, in create_bom
with open(bom_file, mode="w") as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/app/reports/bom-java.xml'
tool         description                             critical    high    medium    low  status

findsecbugs  Security audit by Find Security Bugs           1       0         0      0  ❌

Hello @prabhu
I just tested in an Ubuntu VM (previously I was running Linux containers in Windows 10) and got a bit further. deepscan still failing. Now BOM scan partially working. Do you see anything missing?
Also, still surprised to get just one finding.
$ docker run -v "/home/ubuntu/app_source_code":/app -e "REPOSITORY_URL=https://github.com/myuser/myrepo.git" -e "GITHUB_TOKEN=" quay.io/appthreat/sast-scan:feature_mvn scan --src /app

/ _ \          |_   | |                  | |
/ /\ _ __  _ __ | | | |__  _ __ ___  __ | |
|  _  | '_ | '_ | | | '_ | '/ _ / ` | __|
| | | | |) | |) | | | | | | | |  __/ (| | |_
_| |_/ ./| ./_/ || ||_|  _|_,|_|
| |   | |
||   |_|
INFO [2020-04-02 08:29:06,464] Scanning /app using scan plugins ['credscan', 'java', 'depscan']
INFO [2020-04-02 08:29:06,465] ================================================================================
INFO [2020-04-02 08:29:06,465] ⚡ Executing "gitleaks --depth=2 --repo-path=/app --redact --timeout=2m --report=/app/reports/credscan-report.json --report-format=json"
WARN[2020-04-02T08:29:06Z] Exceeded depth limit (2)
INFO[2020-04-02T08:29:06Z] no leaks found, skipping writing report
INFO[2020-04-02T08:29:06Z] No leaks detected. 2 commits audited in 24 milliseconds 510 microseconds
INFO [2020-04-02 08:29:06,549] ================================================================================
INFO [2020-04-02 08:29:06,549] ⚡ Executing "java -jar /opt/spotbugs/lib/spotbugs.jar -textui -include /usr/local/src/spotbugs/include.xml -exclude /usr/local/src/spotbugs/exclude.xml -noClassOk -auxclasspathFromFile /tmp/tmp33oqp9w6 -sourcepath /app -quiet -medium -xml:withMessages -effort:max -nested:false -output /app/reports/findsecbugs-report.xml /app"
INFO [2020-04-02 08:29:17,595] ================================================================================
INFO [2020-04-02 08:29:17,595] ⚡ Executing "/usr/local/bin/depscan --no-banner --src /app --report_file /app/reports/depscan-report.json"
INFO [2020-04-02 08:29:17,815] ================================================================================
INFO [2020-04-02 08:29:17,816] ⚡ Executing "cdxgen -r -t java -o /app/reports/bom-java.xml /app"
===License scan findings===
+-------------------------------------------+-----------+--------------+--------------------------------------------------+
| Package                                   | Version   | License Id   | License conditions                               |
+===========================================+===========+==============+==================================================+
| ch.qos.logback:logback-classic            | 1.2.3     | EPL-1.0      | disclose-source, include-copyright, same-license |
+-------------------------------------------+-----------+--------------+--------------------------------------------------+
| ch.qos.logback:logback-core               | 1.2.3     | EPL-1.0      | disclose-source, include-copyright, same-license |
+-------------------------------------------+-----------+--------------+--------------------------------------------------+
| jakarta.annotation:jakarta.annotation-api | 1.3.5     | EPL-2.0      | disclose-source, include-copyright, same-license |
+-------------------------------------------+-----------+--------------+--------------------------------------------------+
**Traceback (most recent call last):
File "/usr/local/bin/depscan", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/depscan/cli.py", line 171, in main
s.refresh()
File "/usr/local/lib/python3.6/site-packages/vdb/lib/nvd.py", line 93, in refresh
return self.download_all()
File "/usr/local/lib/python3.6/site-packages/vdb/lib/nvd.py", line 50, in download_all
self.store(data)
File "/usr/local/lib/python3.6/site-packages/vdb/lib/nvd.py", line 98, in store
docs = dbLib.store(db, data)
File "/usr/local/lib/python3.6/site-packages/vdb/lib/db.py", line 51, in store
docs = storage.store(datas, db_file=db["db_file"], index_file=db["index_file"])
File "/usr/local/lib/python3.6/site-packages/vdb/lib/storage.py", line 29, in store
packed_obj = msgpack.packb(data_list, use_bin_type=True)
File "/usr/local/lib64/python3.6/site-packages/msgpack/init.py", line 35, in packb
return Packer(kwargs).pack(o)
File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 296, in msgpack._cmsgpack.Packer.pack
MemoryError
tool         description                             critical    high    medium    low  status

findsecbugs  Security audit by Find Security Bugs           1       0         0      0  ❌
$ cd reports
$ ls -l
-rw-r--r-- 1 root root    2823 Apr  2 08:31 all-b3c5c8f8-417f-4c55-a349-b27036bae31f-report.json
-rw-r--r-- 1 root root 3051167 Apr  2 08:30 bom-java.xml
-rw-r--r-- 1 root root    9393 Apr  2 08:29 findsecbugs-report.html
-rw-r--r-- 1 root root    4486 Apr  2 08:29 findsecbugs-report.sarif
-rw-r--r-- 1 root root   28046 Apr  2 08:29 findsecbugs-report.xml
-rw-r--r-- 1 root root     503 Apr  2 08:30 license-java.json
Thanks,
Javier caceres

On Linux, the depscan is failing because it is running out of memory. Running the docker container with over 2GB of RAM (4GB is recommended) should help. Windows seems to be some kind of file write issue with WSL-Windows cross-over somewhere. Let me investigate.
What java version are you using to compile your project and can you do a find of all class files?
find . -name *.class
java -version

thanks @prabhu
just switched from a t2.micro to a t2.medium instance and the BOM and depscan scans worked fine.
One last question: does the grafeas json containes vulnerability occurrences? If so, what would be the anchor note?

Glad to hear this. For the note there are two models:
Model 1:

You can periodically create notes using vulnerability-db and the grafeas createNote api
While storing the data from depscan json you can refer to these notes via the name

In model 2:

Just create occurrence without an associated note.

This approach would require the VulnerabilityDetails to be present in each occurrence. This is currently not possible with depscan since it only passes the search result in occurrence format and keeps the details only in the database.
Hope this helps.

thanks for your help @prabhu !
I will mark this question as closed now.
I will review the bundled tools and reports and will come back if any question.
regards,


Invalid PNG buffer

We get this error every now and then.
The error is raised by lwip when the PNG signature is not valid.
I wonder if this is Mojang sending bogus images, or if node-canvas or even lwip itself output invalid images. It's also possible that we don't forgot to check if there actually is an image before passing it to lwip or that we're not waiting until it's 'ready'.
Either way, this error should be detected before it's raised.

We also get PNG decoding error sometimes

I see these errors in connection to texure servers returning 503.

Not happening that much anymore, but it's still happening.

Skin for user 5f9fa6d2-dc48-420e-b7c8-e8c82c817282
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 1832
Connection: keep-alive

����JFIF��XExifMM�i&��@� ��C
...

So that image/png is in fact a JPEG. Great, Mojang ;_;

Filed bug at WEB-124.

Filed yet another bug at WEB-134.
Both of these issues are caused by Mojang and it's nothing we can fix.


[WFCORE-634] Intermittent failures in SSLMasterSlaveOneWayTestCase fixed

Linux Build 1500 is now running using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca

Windows Build 1155 is now running using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca

Core - Full Integration Build 756 is now running using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca

Linux Build 1500 outcome was SUCCESS using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca
Summary: Tests passed: 3080, ignored: 66 Build time: 0:21:37

Windows Build 1155 outcome was SUCCESS using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca
Summary: Tests passed: 3080, ignored: 66 Build time: 0:21:37

Core - Full Integration Build 756 outcome was SUCCESS using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca
Summary: Tests passed: 2842, ignored: 416 Build time: 0:57:28

Core - Full Integration JDK8 Build 8065 outcome was FAILURE using a merge of 72378c2035c74a2776d0ee5aafce8cc02dc874ca
Summary: Compilation error: Compiler (new) Build time: 00:03:04


[Dropdown] Request for fixed position variant

Describe the bug
Dropdown used inside Shidoka Table when clicked and opened, the dropdown options are opening on top which is not scrollable.
Expected behavior
The dropdown list option should open at the bottom only.
Screenshots

Closing this as all feature requests must go through AHA now.


Use blog type as well as post type

@lxndrblz not sure if this is the correct way of adding another type to be matched; I'm not really familiar w/ Go or Hugo code so please let me know if there is a different approach!
This change allows the comments and post read time to show up for posts that are under the blog/ subdirectory.

@ericswpark It would be better if we made this adjustments based on the mainSections  parameter. As not every user calls their directory blog.

@lxndrblz hmm, but my mainSections parameter is blank because I don't want my posts showing up on the main page. Would it be possible to check it using a different parameter? Maybe we could create a new parameter, like postSectionName or something like that.

@ericswpark Sounds like a decent idea. Do you want to work on it?

@lxndrblz I'm not sure if I did it correctly but the new commit is now up on #172 . Please let me know if I messed anything up. Will close this PR in favor of the linked one :)


need port details for verify internet access section

The 'verify internet access' section refers to needing 2 ip ranges opened:
<IP_ADDRESS>/18
<IP_ADDRESS>/18
Please include which destination port(s) are needed on these ranges in this document as it is standard practice to lock down access to specific tcp/udp port #s or ranges.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c04356b4-e2a0-c095-5f47-b7fc1d2b574f
Version Independent ID: 72900c81-5858-bb03-06fb-4af272317f20
Content: Back up Windows machines with the MARS agent - Azure Backup
Content Source: articles/backup/backup-configure-vault.md
Service: backup
GitHub Login: @dcurwin
Microsoft Alias: dacurwin

assign:saurabhsensharma

fyi, i found a clarifying statement in a different doc. If it is accurate maybe you could add it here as well or link to the other doc so the details are only in one place?
https://docs.microsoft.com/en-us/azure/backup/backup-support-matrix-mars-agent
"Access to all of the URLs and IP addresses listed above uses the HTTPS protocol on port 443."

Thanks for your comment. The feedback has been shared with the content owner for further review. Thanks for your patience.

#reassign:dcurwin

Thanks for the suggestion. The document has been updated.
#please-close


Post url using protractor

Does protractor allow post as well as get. My app requires a post to my api however I can not find a post that allows me to specify the url and params.
I can do this
ptor.get('http://localhost:22222/pathtoapi?params=1');
but I can not do
var params = 1;
ptor.post('http://localhost:22222/', params,);
is this something that maybe added in the future or is it already possible via another method.

I'm facing the same issue here. My use case is that my webapp is normally accessed via an external portal that calls us with a POST request. This external portal is only used from the production and similar ones are used on UAT environments. In development, we directly access the webapp through a test page making POST requests emulating the portal.
I can't use the portal because it would not target the right platform, neither can't I go to the test page from protractor because of security reasons. I simply want to navigate to my webapp via a POST request.
So far, I started my tests with browser.get(), passing all params in the URL, but that is not consistent with what actually happens in real life, so I'd like to perform an actual browser POST. Isn't this at all possible?


decimal point

is that possible to make the following decimal values work as  expected below?
input       =     output(orientdb)
3            =     3.000000
3.1667   =    3.166700

@martingg88 the problem is not clear. Do you want help on writing the function "output" that does what you want?

yes.. i wana the output to be with the exact decimal point.
for example... i wana 3 to be 3.00 if for 2 decimal points.

Ok, use the format function. Example:
select amount.format("%.2f") from salary

Thanks.
But. it there possible to fix certain decimal points (etc: 2/more decimal points) without rounding?

What do you mean for fix?

sorry for type error...i mean to fit decimal points with rounding.
for example as follows:
3.567 will be 3.56 instead of 3.57.

Try to work at string level. Not tested but try this:
select amount.asString().substring( 0, eval( "$current.amount.indexOf('.') + 3 ") ) from salary

yes. thanks.  this work for select query.
how about insert query?
I have tried the following, but it still appear as 101.7 in orientdb table
insert into place set lat = '101.700000'.asString().substring( 0, eval( "101.700000.indexOf('.') + 7 ") )


Keyboard suggestions does not work

Software Versions

Python: 3.8
OS: android
Kivy: 2.0.0
p4a: develop
Kivy installation method: pip

The standard row that displays keyboard predictions does not appear on the keyboard.  It does not work with either the default keyboard or Microsoft Swift, the bug does not depend on the api version.  Below are screenshots showing which line with the word prediction I mean and how it looks in kivy
from kivy.lang import Builder
from kivy.app import App

KV = '''
BoxLayout:
    padding: "10dp"
    TextInput:
        keyboard_suggestions: True
'''

class Test(App):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.screen = Builder.load_string(KV)

    def build(self):
        return self.screen

Test().run()

https://user-images.githubusercontent.com/40869738/110184043-3dc0d400-7e21-11eb-8db9-bdf141f5f6df.MP4

It should work with master. But also see https://github.com/kivy/kivy/issues/7337 for additional issues.

@matham But in the master thread it does not work correctly https://github.com/kivy/kivy/issues/7415

Ok, but this specific issue is fixed in master. You can try back-porting the fix to 2.0.0 if you can't use master because of the linked issue.

Any progress here?

No


feat(router): plug router as a middleware

This PR introduces the concept of middleware in InstantSearch. It's leveraged internally to plug the routing system.
Why
When building tools on top of InstantSearch, you had no way to subscribe and detect changes.
Since Federated Search came in v4, you can build more complex apps, but they're sometimes hard to debug. You can now dynamically add and remove indices, which impacts the UI state and the search parameters sent to Algolia. To help debug apps in this situation, I plan to build a dev tool to visualize the InstantSearch tree.
This visualizer could be plugged as a Storybook addon (I built a POC based on this PR already) or as a browser extension (think like React Devtools).
Building such integrations was not possible before this middleware system.
Middleware definition
Middleware can be attached to InstantSearch with the use method.
function router({ instantSearchInstance }) {
  return {
    onStateChange({ state }) {
      // write in the history
    },
    subscribe() {},
    unsubscribe() {},
  };
}

function logger({ instantSearchInstance }) {
  return {
    onStateChange({ state }) {
      // log changes
    },
    subscribe() {},
    unsubscribe() {},
  };
}

const search = instantsearch({
  /* ... */
});

search.use(router, logger).start();

Middleware can also be attached after the search starts.

I wonder if there also should be either a parameter on InstantSearch or a widget-like interface for Vue InstantSearch etc.

I think all framework-based flavors (Vue InstantSearch, React InstantSearch, Angular InstantSearch) will have their dedicated APIs but this flexible top-level InstantSearch.js should enable all those to use middleware.

The renamed the method to hook middleware EXPERIMENTAL_use. This is meant to discourage users to use for now, as it's marked experimental.


Vault secrets not being replaced

Describe the bug
I attempted to create a KeycloakClient CR and a KeycloakUser CR with secrets read from the vault. My understanding was that the vault reference would be replaced with the contents of the file mounted at the designated location. In both cases, the secrets were stored literally, instead of being read from the vault.
The keycloak secrets appear to be mounted at the expected path in the container, and I believe, follow the required naming convention.
Version
Keycloak operator: 20.0.1, Realm operator: 1.0.0
Expected behavior
Secret '${vault.myclientsecret}' is replaced by the secret from the filesystem path.
Actual behavior
Secret is stored as a literal string: '${vault.myclientsecret}'.
How to Reproduce?
The keycloak CR looks as follows:
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: mykeycloak
  namespace: keycloak
spec:
  ...
  additionalOptions:
    - name: vault
      value: file
    - name: vault-dir
      value: /opt/keycloak/secrets
    - name: spi-vault-file-key-resolvers
      value: REALM_UNDERSCORE_KEY
  unsupported:
    podTemplate:
      spec:
        containers:
          - volumeMounts:
              - name: vault
                mountPath: /opt/keycloak/secrets
        volumes:
          - name: vault
            secret:
              secretName: keycloak-vault

I verified that the secrets were mounted in the container.
The client CR:
apiVersion: legacy.k8s.keycloak.org/v1alpha1
kind: KeycloakClient
metadata:
  name: myclient
  namespace: keycloak
spec:
  client:
    ...
    secret: '${vault.myclientsecret}'

Anything else?
I'm assuming the vault extension is enabled by default in the keycloak 20.0.1 operator. Is there a way to check?

$ /opt/keycloak/bin/kc.sh show-config

Current Mode: production
Current Configuration:
	kc.cache =  ispn (KcEnvVarConfigSource)
        ...
	kc.spi.vault.file.key.resolvers =  REALM_UNDERSCORE_KEY (KcEnvVarConfigSource)
	kc.vault =  file (KcEnvVarConfigSource)
	kc.vault-dir =  /opt/keycloak/secrets (KcEnvVarConfigSource)
	kc.vault.dir =  /opt/keycloak/secrets (KcEnvVarConfigSource)
	kc.version =  20.0.1 (SysPropConfigSource)

@nuwang Thanks for the report! Unfortunately, Vault is currently not supported for Client credentials, see the docs.


copy files into docker

copy files into docker

Hi @jinfagang, thanks for opening an issue about this 👍 . I've thought about this before too, but I'm not entirely sure whether to implement it.

I see containers as immutable deployment artifacts, which should not be manually modified at runtime. For example, the container digest specifies a specific version of the image contents, and modifying it would negate the value of using a digest. So copying files inside the container at runtime feels like a bit of anti-pattern.
dcp is somewhat limited in what it can do -- most k8s distributions use either containerd or crio as the container runtime, and dcp cannot interface with those. So it wouldn't be able to copy data into a live container running on k8s for example, or extract data out.

That being said, if you have a specific use-case in mind, we can discuss further and potentially look at implementing something like this.

hi, you are right, seems docker are imutable.

Closing for now.


NextJS ReferenceError

When trying to build with Vercel, issue being caused
 ⨯ node_modules/csv-import-react/build/index.esm.js (44363:0) @ persist.name
 ⨯ ReferenceError: localStorage is not defined

File: csv-import-react/build/index.esm.js
Temp Fix:
theme: typeof window !== 'undefined' ? localStorage.getItem(STORAGE_KEY) : 'light',

@civan Amazing


feat: added reduce min onnx import

Checklist

[X] Confirmed that run-checks all script has been executed.
[X] Made sure the book is up to date with changes in this PR.

Related Issues/PRs
Helps with #1714
Changes
Reduce min import based on the reduce max
Testing
Manual
cargo test -p burn-import --color=always -- --color=always
cargo test -p onnx-tests --color=always

Seems pretty similar to ReduceMax 😄

It was a great source of inspiration 😄


Feedback for “SQL Select”

In the docs, when selecting with count it says that the count is going to return a number, but it actually returns a string.

This confused me because I was trying to do some comparisons, and it was failing on me :disappointed:
The correct way was documented in the .mapWith example that was .mapWith(Number). We should update it to make sure others won't be confused.

Unfortunately count(*) is driver dependent. Were you using Mysql2? Having said that, I agree, that the docs should be updated.

I was using postgres.js.

The docs were updated and we now have a count() function that will return number.


[DevOps]: Integration test regeneration and tests/runtime/test_*.py stall

Is there an existing issue for the same bug?

[X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
[X] I have checked the existing issues.

Describe the bug
I'm trying to regenerate integration tests for this PR and in my environment when I run the regeneration script:
git fetch origin fix-pypi-package
git checkout fix-pypi-package
TEST_RUNTIME=eventstream ./tests/integration/regenerate.sh

Alternatively, even on main I see that the following command fails.
poetry run pytest tests/runtime/test_*.py

it stalls and then eventually times out.
Current OpenHands version
b72599379d6ce3e6abdb5cd28c6b091cfe8b36b2

(the final commit on the above PR)

Installation and Configuration
make build

Model and Agent
No response
Operating System
MacOS
Reproduction Steps
No response
Logs, Errors, Screenshots, and Additional Context
Full log
$ TEST_RUNTIME=eventstream ./tests/integration/regenerate.sh
Current working directory: /Users/gneubig/work/OpenHands
SCRIPT_DIR: /Users/gneubig/work/OpenHands/tests/integration
PROJECT_ROOT: /Users/gneubig/work/OpenHands
WORKSPACE_BASE: /Users/gneubig/work/OpenHands/_test_workspace
WORKSPACE_MOUNT_PATH: /Users/gneubig/work/OpenHands/_test_workspace
TEST_RUNTIME: eventstream

========STEP 1: Running test_edits for DelegatorAgent========

/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================= test session starts ==============================
platform darwin -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 -- /Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/bin/python
cachedir: .pytest_cache
rootdir: /Users/gneubig/work/OpenHands
configfile: pytest.ini
plugins: asyncio-0.24.0, cov-5.0.0, Faker-27.4.0, anyio-4.4.0, forked-1.6.0
asyncio: mode=Mode.STRICT, default_loop_scope=None
collecting ...
Paths used:
workspace_base: /Users/gneubig/work/OpenHands/_test_workspace
workspace_mount_path: /Users/gneubig/work/OpenHands/_test_workspace
workspace_mount_path_in_sandbox: /workspace
CONFIG: AppConfig(llms={'llm': LLMConfig(model='claude-3-5-sonnet-20240620', api_key='******', base_url=None, api_version=None, embedding_model='local', embedding_base_url=None, embedding_deployment_name=None, aws_access_key_id=None, aws_secret_access_key=None, aws_region_name=None, num_retries=10, retry_multiplier=2, retry_min_wait=3, retry_max_wait=300, timeout=None, max_message_chars=10000, temperature=0, top_p=0.5, custom_llm_provider=None, max_input_tokens=None, max_output_tokens=None, input_cost_per_token=None, output_cost_per_token=None, ollama_base_url=None, drop_params=None)}, agents={'agent': AgentConfig(memory_enabled=False, memory_max_threads=2, llm_config=None)}, default_agent='DelegatorAgent', sandbox=SandboxConfig(api_hostname='localhost', container_image='nikolaik/python-nodejs:python3.11-nodejs22', user_id=501, timeout=120, enable_auto_lint=False, use_host_network=True, initialize_plugins=True, runtime_extra_deps=None, runtime_startup_env_vars={}, browsergym_eval_env=None), security=SecurityConfig(confirmation_mode=False, security_analyzer=None), runtime='eventstream', file_store='memory', file_store_path='/tmp/file_store', workspace_base='/Users/gneubig/work/OpenHands/_test_workspace', workspace_mount_path='/Users/gneubig/work/OpenHands/_test_workspace', workspace_mount_path_in_sandbox='/workspace', workspace_mount_rewrite=None, cache_dir='/tmp/cache', run_as_openhands=True, max_iterations=15, max_budget_per_task=15, e2b_api_key=None, disable_color=False, jwt_secret='******', debug=False, enable_cli_session=False, file_uploads_max_file_size_mb=0, file_uploads_restrict_file_types=False, file_uploads_allowed_extensions=['.*']
collected 1 item

tests/integration/test_agent.py::test_edits 22:39:49 - openhands:INFO: main.py:75 - Initializing runtime: <class 'openhands.runtime.client.runtime.EventStreamRuntime'>
22:39:49 - openhands:INFO: runtime_build.py:41 - Using project root: /Users/gneubig/work/OpenHands
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - poetry-core
* Getting build dependencies for sdist...
* Building sdist...
Successfully built openhands_ai-0.9.0.tar.gz
22:39:52 - openhands:INFO: runtime_build.py:59 - Source distribution created at /Users/gneubig/work/OpenHands/dist/openhands_ai-0.9.0.tar.gz
22:39:52 - openhands:INFO: runtime_build.py:80 - Source distribution moved to /var/folders/12/vk4h8csj4ts7rql173zqn0sm0000gn/T/tmp7h1i2iot/project.tar.gz
22:39:52 - openhands:INFO: runtime_build.py:90 - Unpacked source code directory: /var/folders/12/vk4h8csj4ts7rql173zqn0sm0000gn/T/tmp7h1i2iot/code
22:39:52 - openhands:INFO: runtime_build.py:161 - Input base image: nikolaik/python-nodejs:python3.11-nodejs22
Skip init: False
Extra deps: None
Hash for docker build directory [/var/folders/12/vk4h8csj4ts7rql173zqn0sm0000gn/T/tmp7h1i2iot] (contents: ['Dockerfile', 'code']): 163ba92f78d4107a4e3fa9b7ce6b2e4b

22:39:52 - openhands:INFO: runtime_build.py:247 - Image [ghcr.io/all-hands-ai/runtime:163ba92f78d4107a4e3fa9b7ce6b2e4b] already exists so we will reuse it.
22:39:52 - openhands:INFO: runtime.py:183 - Starting container with image: ghcr.io/all-hands-ai/runtime:163ba92f78d4107a4e3fa9b7ce6b2e4b and name: openhands-sandbox-1622d1b4-8b0a-4f71-b9b2-32ed7c89d8ba_47f6bb569a34e4ed_d8f59e45-3116-4191-8ed0-59d983db1219
22:39:52 - openhands:WARNING: runtime.py:196 - Using host network mode. If you are using MacOS, please make sure you have the latest version of Docker Desktop and enabled host network feature: https://docs.docker.com/network/drivers/host/#docker-desktop
22:39:52 - openhands:INFO: runtime.py:204 - Mount dir: /workspace
22:39:52 - openhands:INFO: runtime.py:238 - Container started. Server url: http://localhost:63193
22:39:52 - openhands:INFO: runtime.py:157 - Container initialized with plugins: []
22:39:52 - openhands:INFO: runtime.py:160 - Container initialized with env vars: None
22:39:52 - openhands:INFO: main.py:151 - Agent Controller Initialized: Running agent DelegatorAgent, model claude-3-5-sonnet-20240620, with task: "Fix typos in bad.txt. Do not ask me for confirmation at any point."
22:39:52 - openhands:INFO: agent_controller.py:150 - [Agent Controller default] Starting step loop...
22:39:52 - USER_ACTION
**MessageAction** (source=EventSource.USER)
CONTENT: Fix typos in bad.txt. Do not ask me for confirmation at any point.

==============
DelegatorAgent LEVEL 0 LOCAL STEP 0 GLOBAL STEP 0

22:39:53 - ACTION
AgentDelegateAction(agent='StudyRepoForTaskAgent', inputs={'task': 'Fix typos in bad.txt. Do not ask me for confirmation at any point.'}, thought='', action='delegate')
22:39:53 - openhands:INFO: agent_controller.py:330 - [Agent Controller default]: start delegate, creating agent StudyRepoForTaskAgent using LLM LLM(model=claude-3-5-sonnet-20240620)

==============
StudyRepoForTaskAgent LEVEL 1 LOCAL STEP 0 GLOBAL STEP 1

22:39:53 - openhands:INFO: llm.py:431 - Cost: 1.00 USD | Accumulated Cost: 1.00 USD
22:39:53 - ACTION
**CmdRunAction (source=EventSource.AGENT)**
COMMAND:
ls
22:39:53 - openhands:INFO: runtime.py:325 - Awaiting session
22:40:03 - openhands:INFO: runtime.py:265 -
------------------------------Container logs:------------------------------
    |Starting action execution API on port 63193
    |02:39:56 - openhands:INFO: client.py:689 - Starting action execution API on port 63193
    |INFO:     Started server process [15]
    |INFO:     Waiting for application startup.
    |02:39:57 - openhands:INFO: browser_env.py:58 - Starting browser env...
    |02:40:00 - openhands:INFO: browser_env.py:101 - Browser env started.
    |02:40:00 - openhands:INFO: client.py:197 - Initializing by running 3 bash commands...
    |02:40:00 - openhands:INFO: client.py:208 - Bash init commands completed
    |02:40:00 - openhands:INFO: client.py:482 - Runtime client initialized.
    |INFO:     Application startup complete.
    |INFO:     Uvicorn running on http://<IP_ADDRESS>:63193 (Press CTRL+C to quit)
------------------------------------------------------------------------------------------
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-22' coro=<Runtime.on_event() done, defined at /Users/gneubig/work/OpenHands/openhands/runtime/runtime.py:129> exception=RetryError(<Future at 0x139cfa9d0 state=finished raised ClientConnectorError>)>
Traceback (most recent call last):
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 1073, in _wrap_create_connection
    sock = await aiohappyeyeballs.start_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohappyeyeballs/impl.py", line 122, in start_connection
    raise OSError(first_errno, msg)
ConnectionRefusedError: [Errno 61] Multiple exceptions: [Errno 61] Connect call failed ('::1', 64777, 0, 0), [Errno 61] Connect call failed ('<IP_ADDRESS>', 64777)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/work/OpenHands/openhands/runtime/client/runtime.py", line 276, in _wait_until_alive
    async with session.get(f'{self.api_url}/alive') as response:
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/client.py", line 1353, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/client.py", line 657, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 564, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 975, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 1350, in _create_direct_connection
    raise last_exc
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 1319, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/aiohttp/connector.py", line 1088, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host localhost:64777 ssl:default [Multiple exceptions: [Errno 61] Connect call failed ('::1', 64777, 0, 0), [Errno 61] Connect call failed ('<IP_ADDRESS>', 64777)]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gneubig/work/OpenHands/openhands/runtime/runtime.py", line 135, in on_event
    observation = await self.run_action(event)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/work/OpenHands/openhands/runtime/client/runtime.py", line 327, in run_action
    await self._wait_until_alive()
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 419, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x139cfa9d0 state=finished raised ClientConnectorError>]

Maybe IPv6 is an issue as the error mentions ::1

If this is set to False it works: https://github.com/All-Hands-AI/OpenHands/blob/653bc4ef6d8ed9dd864caf11697651c58fb5d299/tests/integration/test_agent.py#L32


Using custom properties / field codes

Hi !
In a Word document, it's possible to add field codes, with ctrl+F9 or via QuickPart, and the content { DOCPROPERTY prop_name }. Those custom properties are manageable and this allows to use a docx as a template where variables can be easily replaced.
In a pptx file, it's not possible to insert such fields, even if it's possible to set properties, that are stored in docProps/custom.xml. They don't seem to be accessible from python-pptx.
Do you see a simple way to insert custom variable placeholders in a powerpoint, that python-pptx will be able to update easily without messing up formatting and position ?
For the moment, we iterate over all paragraphs in all shapes, and replace {{variable name}} that the author put especially there with the appropriate value, but it destroys the formatting when this variable is inside a sentence with multiple formattings (example: some words are underlined).
Thanks !

I would look into CT_TextField (toward the bottom of this page http://python-pptx.readthedocs.io/en/latest/dev/analysis/txt-text.html), which is a legitimate occupant of a paragraph, roughly a special run. It has formatting elements so its content could be changed while retaining its formatting. I don't know what tooling support there is for it. I expect it's used for slide number fields and date fields, so those might give some ready examples.
I like the idea of using moustache placeholders for content, mostly because it's "no tools required". It's trickier than just replacing all the text in a paragraph or text frame though. You'd need code that assembles all the runs and concatenates all their content while remembering where formatting began and ended, and then replace the specified contents without losing the formatting. The .text attribute on paragraph, and text_frame nukes all that in the interest of utility and general purpose.

@renejsum added custom properties for docx and it's working flawlessy. I have tested this. The link for python-docx is here.
https://gist.github.com/renejsum/0ea621aabd0e4640391ddb3e361f6c02
Maybe this method would work for pptx as well.


feat: support context API

I think we can support context API to listen an action which usinguseReducer by attaching custom middleware.
const initialState = {count: 0};

function reducer(state, action) {
 // add middleware here
 middleware(state, action)
  switch (action.type) {
    case 'increment':
      return {count: state.count + 1};
    case 'decrement':
      return {count: state.count - 1};
    default:
      throw new Error();
  }

I think we should change usage when use context API
  const actions = {
    increase: () => {
      middleware({ type: "INCREASE", payload: 1 });
      return dispatch({ type: "INCREASE", payload: 1 });
    },
    substract: () => {
      middleware({ type: "SUB", payload: 1 });
      return dispatch({ type: "SUB", payload: 1 });
    }
  };

https://codesandbox.io/s/simple-react-context-example-2-forked-elys2?file=/src/index.js:749-1039

I think we should change usage when use context API
  const actions = {
    increase: () => {
      middleware({ type: "INCREASE", payload: 1 });
      return dispatch({ type: "INCREASE", payload: 1 });
    },
    substract: () => {
      middleware({ type: "SUB", payload: 1 });
      return dispatch({ type: "SUB", payload: 1 });
    }
  };

https://codesandbox.io/s/simple-react-context-example-2-forked-elys2?file=/src/index.js:749-1039

Fun fact redux is ok to dispatching an action in action, but Context API is not.

Fun fact redux is ok to dispatching an action in action, but Context API is not.


flynn env set fails on restored cluster

19:20:44.405927 cluster-monitor cluster-monitor'
+ echo 'Bootstrap succeeded'
Bootstrap succeeded
+ slept=0
+ add_local_flynn_config
++ cat controller.auth_pin.txt
++ cat controller.auth_key.txt
+ flynn cluster add -p a6x8x6NmzsUE+jcv76ZJi64xy6p01ZABaU8sL3TOIoM= qa-us-east-1-cluster-2016-03-16-13-41-56 qa-us-east-1.flynn.nexus.bazaarvoice.com 62b07a2c7f6e9d12928b81a3a9726649
Cluster "qa-us-east-1-cluster-2016-03-16-13-41-56" added and set as default.
+ [[ <IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS> == <IP_ADDRESS>* ]]
+ [[ 2016-03-16-17-17-01 != \n\o\n\e ]]
+ sleep 300
++ cat ssh_client_key
++ cat ssh_client_hosts
+ flynn -a gitreceive env set 'SSH_CLIENT_KEY=-----BEGIN RSA PRIVATE KEY-----
XXX
-----END RSA PRIVATE KEY-----' 'SSH_CLIENT_HOSTS=# github.com SSH-2.0-libssh-0.7.0
github.com ssh-rsa XXX'
unknown_error: Something went wrong

This is after restore from a rather large backup. https://gist.github.com/cc9031fb32da9dd462d1 https://gist.github.com/6bd8d4b4f753e20f7f89 and https://gist.github.com/15e140c432d1b627c5b4 are the debug-info.

Same error on a new cluster: https://gist.github.com/645dd02581fd5bf52419 https://gist.github.com/8783d07c8979beedd633 https://gist.github.com/fe296ccea45af2125d85 . Repeated tries did not improve things.

Same failure location, slightly different output: "GET https://controller.prod-us-east-1.flynn.nexus.bazaarvoice.com/apps/gitreceive/release: httpclient: raw req: unexpected status 503"
collect-debug-info: https://gist.github.com/959c55aa4a39d13167a8 https://gist.github.com/e3af35c71aa8a7a011f0 https://gist.github.com/db5dd7282fd67c8c9ae6

I think that between #2618 and #2590 the known causes of this should be mitigated. Please open a new issue if you see it again.


add github actions build test

GitHub  Actions only support two kinds of OS:  macOS  &  Ubuntu Now

@KimBioInfoStudio seems that this PR breaks CI due to the libdeflate, can you please try to fix it?


Test: Disable to-entities test

Skip test on to-entities due errors.
Issue to track the new changes needed #5791
Signed-off-by: Eloy Coto<EMAIL_ADDRESS>

This change is

test-me-please

test-me-please

@eloycoto this will fail the CI in K8s-1.12 until #5790 is merged

test-me-please


Added TMX map parsing.

I can rebase this down to one commit if we want. I was going through the diffs and noticed a couple things I missed.

Please condence down to one commit and update read me with any required frameworks that now need to be imported.

Updated


testing: t.Fatal does not fatal tests if defers do not complete

What version of Go are you using (go version)?
go version go1.9.2 darwin/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/zwass/dev/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.9.2/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.9.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4k/2q3ctn7d5_xb03_zsb3v6lgr0000gn/T/go-build085156185=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?
test_test.go
package test

import (
	"sync"
	"testing"
	"time"
)

func TestTest(t *testing.T) {
	var m sync.Mutex
	defer m.Lock()
	go func() {
		for {
			time.Sleep(1 * time.Second)
		}
	}()

	m.Lock()
	t.Fatal("foobar")
}

What did you expect to see?
go test -v
=== RUN   TestTest
--- FAIL: TestTest (0.00s)
	test_test.go:19: foobar
FAIL
exit status 1
FAIL	test	0.007s

What did you see instead?
go test -v
=== RUN   TestTest
panic: test timed out after 10m 0s

goroutine 17 [running]:
testing.startAlarm.func1()
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:1145 +0xf9
created by time.goFunc
	/usr/local/Cellar/go/1.9.2/libexec/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420094000, 0x1135230, 0x8, 0x113d640, 0x10688b6)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc420094000)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc420094000, 0xc420043de0)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42000a060, 0x11db860, 0x1, 0x1, 0xc42004c3b0)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420043f18, 0xc420043f70)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:921 +0x111
main.main()
	test/_test/_testmain.go:44 +0xdb

goroutine 6 [semacquire]:
sync.runtime_SemacquireMutex(0xc4200140ec, 0xc420048e00)
	/usr/local/Cellar/go/1.9.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4200140e8)
	/usr/local/Cellar/go/1.9.2/libexec/src/sync/mutex.go:134 +0xee
runtime.Goexit()
	/usr/local/Cellar/go/1.9.2/libexec/src/runtime/panic.go:366 +0x147
testing.(*common).FailNow(0xc4200940f0)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:544 +0x39
testing.(*common).Fatal(0xc4200940f0, 0xc420030788, 0x1, 0x1)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:582 +0x6f
test.TestTest(0xc4200940f0)
	/Users/zwass/dev/go/src/test/test_test.go:19 +0xdc
testing.tRunner(0xc4200940f0, 0x113d640)
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/Cellar/go/1.9.2/libexec/src/testing/testing.go:789 +0x2de

goroutine 7 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/Cellar/go/1.9.2/libexec/src/runtime/time.go:65 +0x130
test.TestTest.func1()
	/Users/zwass/dev/go/src/test/test_test.go:14 +0x2a
created by test.TestTest
	/Users/zwass/dev/go/src/test/test_test.go:12 +0x78
exit status 2
FAIL	test	600.010s

The above example is obviously broken, but this hanging behavior from the test runner causes quite a confusing experience when there are more subtle issues going on.
It's also worth noting that there is no way to determine from the dumped goroutines what is causing the hang.
From what I can tell, this is due to t.FailNow() using runtime.GoExit() which waits for defers to complete.
The docs for t.FailNow() say the following:

FailNow marks the function as having failed and stops its execution. Execution will continue at the next test or benchmark. FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test. Calling FailNow does not stop those other goroutines.

I'm marking this as a documentation issue.  If someone can figure out how to fix it reasonably, that's fine with me.  But in practice I think we should just document this and move on.

This is working as intended. You want those deferred calls to run and finish before the test is torn down. If one is buggy and blocks forever, well, sorry. Sent a CL.


[BugFix][TVMScript]fix var capturing order error.

This PR try to fix the following bug:
def test_var_capturing_order():
    b = 2

    @T.prim_func
    def test_case():
        k: T.int32 = b

if __name__ == "__main__":
    b = 1

In the prim func test_case, the vaule of b should be 2, rather than 1. The parser wrongly uses global vars to shadow the value of nonlocal vars, which should be reversed.

cc @junrushao

@tvm-bot rerun


Expose extension and vars from KingfisherOptionsInfo extension

In response to #505

Current coverage is 74.20% (diff: 93.75%)

Merging #506 into master will not change coverage

@@             master       #506   diff @@
==========================================
  Files            21         21
  Lines          2229       2229
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1654       1654
  Misses          575        575
  Partials          0          0

Powered by Codecov. Last update 61d09f3...a736572

@ManueGE Great! I will add some documentation for them before we could release an update.
Thanks!


Couldn't add persistent volume claims

The headless worked great. But when trying the stateful service, the persistent volumes were created successfully, but the persistent volume claim keeps showing pending, and never bound successfully.

What does it show when you execute this command: kubectl get pv,pvc

'get pv' shows all the 3 volumes created successfully, but 'get pvc' shows the claim is pending (after a long time"

here is a screenshot:

It looks like you are using a paid cluster on Bluemix Container service. The pvc will be provisioned dynamically which means you wouldn't need to create Persistent Volumes. The storage class (ibmc-file-bronze) will create that for you whenever a persistent volume claim is created. It should get provisioned in less than 5 minutes

I see. Should I just delete all the local volumes, and then rerun to create the statefulset? Thanks.

Just deleted all the local volumes, statefulset, and then rerun to create the statefulset. Didn't see any PV created automatically. The PVC still shows pending.

Can you do bx cs cluster-get YOUR_CLUSTER_NAME
It should show the addons in your cluster:
...
Addons
Name                   Enabled
customer-storage-pod   true
basic-ingress          true
storage-watcher-pod    true

It doesn't show the addon. How can I install it? Here is my screenshot:

Try updating the bx cli bx plugin update container-service -r Bluemix

Thanks. After updating the addon, I now do see the following. But I have tried to redeploy everything and still not working

Try doing kubectl get pods -n kube-system and let me know if there is a pod named ibm-storage-watcher-[xxxxx]

Yes, I do have a pod ibm-storage-watcher-2152805327-ccln6 running. Also every time, I run the statefulset creation, I did get a notification that a new storage order was approved, but not sure why I couldn't see the actual storage.

The problem might be on the Softlayer side. You cloud try manually creating a Persistent Volume that uses that file storage in softlayer. https://console.bluemix.net/docs/containers/cs_cluster.html#cs_cluster_volume_create
--- or ---
You can specify to not use the storage class (by setting annotation of storage class to empty) and use the local-volumes.yaml
You could modify the cassandra-statefulset.yaml
...
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
      annotations:
        volume.beta.kubernetes.io/storage-class: ""
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

That should use the volumes created through local-volumes.yaml (It's not gonna be using the file storage in Softlayer)

Thanks. It did work. But when I scaled it to 4 replica sets, the other 3 crashed, not sure why. See screenshot below:

Any logs on the cassandra-1,2,3? kubectl logs cassandra-1. Also check for the cassandra service kubectl get svc

Thanks. The log files did show the error from the cassandra seed variable. If I comment it out, it works fine. Not sure if there is a better way to do it. Thanks.


DbSet query with DateTime.Now parse CURRENT_TIMESTAMP

Steps to reproduce
Dbset query with DateTime.Now property with the definition of a DateTime.Now variable, generating sql is not the same, whether there are related configuration items can close the transfer related database function
DateTime dateTime = DateTime.Now;             ActivityCourse activityCourse = _activityCourse.Query(d => d.CourseID == courseID             && d.StartTime <= dateTime             && d.EndTime > dateTime             && d.IsDel == false             ).FirstOrDefault();             var data= activityCourse = _activityCourse.Query(d => d.CourseID == courseID              && d.StartTime <= DateTime.Now              && d.EndTime > DateTime.Now              && d.IsDel == false             ).FirstOrDefault();

Further technical details
EF Core version: 2.2.6
Database provider: Pomelo.EntityFrameworkCore.MySql
Target framework: .NET Core 2.2
Operating system:windows 10
IDE: (e.g. Visual Studio 2019 16.3)

I think it's not bug but by design. If you want to use specific DateTime for query, you should save the value in a variable first and pass it as a parameter.

@hez2010 Seems to have it right.

Yes,When I set it to a variable, he was right.Whether this can provide the relevant configuration item switch, or provide the relevant translation function instructions in the EF document, it is easy to write this DateTime.Now, the result of the mysql time zone inconsistent when embarrassed


Allow passing null for SqlParameter value

Is your feature request related to a problem? Please describe.
Currently, if you pass null to the Value in a SqlParameter constructor or use the AddWithValue method, you receive a rather cryptic exception message::
e.g.
using var conn = new SqlConnection(...);
await conn.OpenAsync();
using var cmd = new SqlCommand("select @", conn);
cmd.Parameters.Add(new("@", null) { DbType = DbType.Int32 });
var res = await cmd.ExecuteScalarAsync();

Microsoft.Data.SqlClient.SqlException : The parameterized query '(@ int)select @' expects the parameter '@', which was not supplied.

The message is entirely unclear, and makes you have to google it, only to discover that the correct resolution of this problem is to replace null with DbNull.Value.
There are many, many threads about this on stackoverflow and various forums.
Describe the solution you'd like
When you create a parameter with null, it internally stores down DbNull.Value.
This is technically a breaking change, but I'm not sure who actually benefits from this obtuse behavior where null is an exception rather than doing the logical thing.
Describe alternatives you've considered
Additional context

I'm not sure who actually benefits from this obtuse behavior where null is an exception rather than doing the logical thing.

You, the developer, do. By extension your users do.
Language null and sql null aren't conceptually the same thing and they don't directly map onto one another so you are required to indicate what should happen if a parameter is null:

is it the result of a progamming error in your .net code so that a reference type (usually string ime) which shouldn't have been a string has somehow made it to the insert stage without a value
is it an appropriate value for the parameter?
Without you making a specific choice it can be either but if it's the first you're going to get an error which will consume io, and go slowly as well as possibly rolling back any transaction you have open. All around expensive.

This situation would be different if generics and nullable type annotations had existed at the time that the api was designed, there would probably be no need for DBNull at all and mapping would be easier.

I don't buy it. At minimum the error could at least say something like...

If you meant to pass null, use DBNull.Value

Of course, this goes back to the problem of, if the library knew what I was doing why is it making me work for it?
This is a sqlserver error the same one you get with sp_executeSql without supplying a parameter:
exec sp_executesql N'select @',N'@ int'

The parameterized query '(@ int)select @' expects the parameter '@', which was not supplied.
So it appears that the driver sends the metadata, but then doesn't bother sending the actual value. The error sucks from sql server, but here I can at least see that oh hey, I never did specify a value for @, and if I truly never want to avoid sending a value, I can specify in the metadata declaration that @ int=null or whatever default I want.

I almost always end up using a tool like Dapper, or efcore which will magically wrap away this behavior. e.g.
await conn.ExecuteScalarAsync<int?>("select @foo",new {foo=new int?()}), will do the right thing and not throw an exception at me. I generally forget this happens when I try to not use an abstraction layer. But there are reasons to avoid using an abstraction layer, and the behavior shouldn't be so awful for working with the raw database operations.

is it the result of a progamming error in your .net code so that a reference type (usually string ime) which shouldn't have been a null has somehow made it to the insert stage without a value
In my example, and my real code that I needed to do this I was using a nullable int.

is it an appropriate value for the parameter? indicating you want to insert NULL
Without you making a specific choice it can be either but if it's the first you're going to get an error which will consume io, and go slowly as well as possibly rolling back any transaction you have open. All around expensive.

Null is an entirely valid for sql queries and for sql databases, so the questioning is disingenuous. E.g. you very well might want to leave address_line2.
e.g.
insert into dbo.address(user_id,address_line,address_line2,city,state,zip)
values(@userId,@addressLine,@addressLine2,@city,@state,@zip);

And similarly, you may want to query data for keys that contains null. Like finding all running instances of a particular job type for a date that have not completed yet.
select batch_id
from dbo.batch_run
where exists (
    select job_type,as_of_date,completion_time
    intersect
    select @jobType,@asOfDate,@completionTime
)

I am unsure what a more recently written driver like npgsql does, but I'm going to guess that sending null with a dbtype of int isn't an error with an obtuse message.

Of course, this goes back to the problem of, if the library knew what I was doing why is it making me work for it?

It doesn't. You didn't tell it. SqlParameter.Value is of object type and you can put anything you want in there. You want to pass in nothing and have it infer that language null means database null but that may not be the case because it could equally be an error of input data. You, the programmer, are supposed to give it unambiguous instructions which is what DBNull is for. DBNull is a value which indicates to the library that yes you really do mean to send null to the database.

I am unsure what a more recently written driver like npgsql does,

I'm pretty sure that because it uses generically typed parameter classes it is able to make safer assumptions and if you provide a language null to a field which is declared as being able to hold a null it will send it as database null. SqlClient uses object for the value container because it predates generics and so DBNull is needed as a sentinel value. /cc @roji

Npgsql does have a generic parameter API (e.g. to avoid boxing), but this isn't standard ADO.NET (see https://github.com/dotnet/runtime/issues/17446), and Npgsql also supports the standard non-generic API. One tricky point with the generic API, is that it's not possible to use it to write nulls for value types.
I've never quite investigated the entire question of ADO.NET, .NET null and database null, or how feasible it would be to allow .NET null to be interpreted as database null. IIRC the difference is significant in some places, e.g. when inserting a new row, .NET null means "use the column's default value" whereas database null means database null (but I'm not 100% sure on this anymore).
I do know that it's not something we can just consider doing without a lot of careful analysis. IMO this is also something that needs to be considered from a cross-database perspective, and not as a pure SqlClient change.

I guess the question then if I use a standard DbParameter in npgsql, does it consider null to be DbNull or does it throw an obscure message?
It seems most client abstraction libraries like EFCore and Dapper will hide the difference for the user when using parameterized queries.
Now I suppose in the case of a store procedure call, a parameter that has no value will invoke the default. (analogous to doing the following in sql.
exec sp_executesql N'select @',N'@ int=null'
-- is equivalent to the following
exec sp_executesql N'select @',N'@ int=null',@=default

Of course, if this is the desired nonsense behavior than perhaps it could be solved with a boolean on SqlParameter like IsDefault or whatever.
The crux I keep hearing is that sqlclient was written in .net 1.0 times, and all of it's cruft and unwieldiness is an unfortunate consequence of its legacy.

Hi @mburbea
Yes I agree driver is generating: exec sp_executesql N'select @',N'@ int',@=default here.
I tried with null and that works so maybe it can be fixed:
exec sp_executesql N'select @',N'@ int',@=null

@mburbea an important here, is that if you change an ADO.NET driver to suddenly treat language null as database null (instead of default), you're potentially breaking a whole lot of programs out there which rely on this behavior (which has been around since forever).
Of course, if there's a reliable way to detect the error situation and error with a better message, that's always a good idea (but I'm not sure that's the case here).

Just out of interest, with generic parameter types if you add new Parameter<int>() to a command is the default value for the type used if the parameter value isn't directly set after it's created?

I'm going to say that the behavior is safe to treat null as DbNull when the CommandType property is set to
CommandType.Text (the default) for SqlParameters.
I am also not suggesting to make this change in the DataReader or in ExecuteScalar.  I still expect those to unfortunately require a call to IsDbNull{Async}
Since procedures, may want this behavior where unsupplied parameters are treated as default, and I have no idea what the CommandType.Table option does, we can safely make this change.
I suppose if anyone was testing for this exception, they would break. Like I acknowledged earlier, it's a breaking change, but outside of testing for an exception, I can't imagine anyone relying on this behavior.

Like I acknowledged earlier, it's a breaking change, but outside of testing for an exception, I can't imagine anyone relying on this behavior.

The problem with being part of the runtime for so long is that I can be certain that someone somewhere has relied on the current behaviour and a change will break it and absolutely ruin some poorly paid maintenance developers day/week/month. Much like the BCL behavioural changes are extremely hard to justify in old apis, subtle behavioural changes in this library even though it's optional are hard to justify. The upgrade path from System to Microsoft versions needs to be smooth to allow adoption in LOB software.
Now, when we get generic parameters like postgres has that'll be all new surface area with no need to use DBNull and then we can do things more appropriate for the language and runtime we have today.

The runtime has already made some breaking changes to allow behavior that has always thrown an exception.
e.g.
var dict = new Dictionary<int, int> { [0] = 0, [1] = 1 };
foreach (var kvp in dict)
{
     if (kvp.Key == 0) dict.Remove(kvp.Key);
}

In .net framework 4.8 this throws an exception ( Collection was modified; enumeration operation may not execute). In .net core 3/.net 5 it does not. Again, since this is only an exception, and it is very unlikely to be caught in anything but a unit test testing for an exception they still made the change.
I think that being so cautious means that the library can never evolve and we are simply stuck with crappy behavior that nobody uses, and basically everyone is required to adopt an abstraction layer.

FYI,
You can solve this issue by passing parameter as below:
cmd.Parameters.Add(new SqlParameter("@", null) { SqlValue = DBNull.Value });

The runtime has already made some breaking changes to allow behavior that has always thrown an exception.
e.g.

I know, I made that change :grin: https://github.com/dotnet/coreclr/pull/18854
I'm also sure that it will have broken some code being ported from netfx. going from netcore to netfx is a porting process and that particular break was documented and called out in release notes iirc.
Going from System to Microsoft versions of this library really needs not to contain breaking changes if at all possible. I'm not saying it won't (I know cancellation exceptions are going to change soon) but the bar for approval of those changes is pretty high.
In context I don't think there is a lot that can be done with the legacy parameter mechanism without high risk of breaking someone's code, even if they wrote bad code I still don't want to break it if I can help it.

@Wraith2, ha didn't realize you made that change. That's pretty funny as that was the first breaking change that came to my mind. (and a welcome one).
@cheenamalhotra, I'm aware that works.
Also nit:
If I don't specify dbtype the inferred type by the library is nvarchar(4000), where as sqlservers inferred default type is int. Like issue #255, I can assume this incongruity is in the hell freezing over category before this ever gets fixed.

#255 I can probably fix quite easily because of the suggested appcontext switch it just wasn't on my radar.

I'm going to say that the behavior is safe to treat null as DbNull when the CommandType property is set to
CommandType.Text (the default) for SqlParameters.
I am also not suggesting to make this change in the DataReader or in ExecuteScalar. I still expect those to unfortunately require a call to IsDbNull{Async}

I don't see how CommandType is related, or how it's safe. Unless I'm missing something, if you create a SqlCommand today with CommandType.Text, set text to INSERT INTO table (column) VALUES (@p) and have null as the parameter value, you will get a row with the default value defined on the column. The change you propose will insert database NULL instead, which is a pretty major breaking change.
IsDbNull{Async} is for the reading side, which is a different discussion - up to now we've been discussing the writing side.

The runtime has already made some breaking changes to allow behavior that has always thrown an exception.

Changing behavior to not throw is by definition not considered a breaking change. What your proposing would change the meaning of existing, working code to do something entirely different.
To be honest, I don't really see the point of this discussion... There current way things work don't block anyone - as @cheenamalhotra mentioned above, users simply need to pass DBNull.Value instead of null; regardless of what we think of it, it is what it is and that's how ADO.NET works. If users accidentally use null in the wrong way, it's unfortunate that they get a cryptic message, but that certainly doesn't seem worth a potentially huge breaking change. That isn't to say things shouldn't be improved if they can be, but it's better to concentrate on more valuable things.

@roji, the CommandType matters because default does not work as you described in this case. If I try this...
using (var cmd = new SqlCommand("insert into dbo.t(c)values(@)", conn))
{
    cmd.Parameters.AddWithValue("@", null);
    await cmd.ExecuteNonQueryAsync();
}

I get that cryptic exception

Microsoft.Data.SqlClient.SqlException (0x80131904): The parameterized query '(@ nvarchar(4000))insert into dbo.t(c)values(@)' expects the parameter '@', which was not supplied.

However, if I use a store procedure which does have a default value supplied for a parameter like::
create or alter procedure dbo.p(@ nvarchar(4000)='horse')
as
insert into t(c) values(@);

Then the following works as you describe (note that I have to set the mode to procedure).
 using (var cmd = new SqlCommand("dbo.p", conn) { CommandType = CommandType.StoredProcedure })
{
    cmd.Parameters.AddWithValue("@", null);
    await cmd.ExecuteNonQueryAsync();
}

This is why I mentioned to keep the behavior of null meaning default for procedures. For text where it can ONLY be an error, I think we can unblock the change safely.
I say it's an error and an odd design because lots of people run into this problem. (Search for the exception on stackoverflow if you think i'm alone). I disagree with the viewpoint that it doesn't block people so it shouldn't be fixed. Ultimately, many libraries like efcore, dapper, and now my internal layer now has code akin to new("@{paramName}", value ?? DbNull.Value). I don't think just because the workaround is simple, doesn't mean we shouldn't correct the problem.

I can't think of any case where calling AddWithValue with a null value is ever the right thing to have done and I think I'd have said it should throw an exception. We can't change that either.
If the exception message needs to be changed to better inform people about the problem then that can probably be done but changing the behaviour will cause a massive compatibility problem.

@Wraith2, I'm still trying to find this case where the behavior works as you describe such that it is a breaking change. How does this impact compatability?
If we make the change only occur when CommandType = CommandType.Text then the only people who would be impacted are those who were intentionally trying to get this exception.

Also to add, when trying to mock this in SSMS, it seems like setting default keyword as value to a SQL Variable is just not allowed, and that's how we could pass parameter in SSMS queries, if I'm not wrong. So if driver sets this as value, in an RPC call, it could be possible it's internally doing the same thing and that's why does not accept it and fails?
exec sp_executesql N'insert into t1 values (@1, @2)',N'@1 nvarchar(4000),@2 int',@1=default,@2=NULL
-- throws: The parameterized query '(@ nvarchar(4000))select @' expects the parameter '@', which was not supplied.

It's also different than passing value directly in the query in a non-parameterized way or via another procedure internally (like explained by @mburbea above), and that's out of context for null parameter value.

The stored procedure calls exec dbo.p @=default on server, but leads to inserting "Horse" in the table as the parameter value default is not considered a valid value.
So I'm also confused where does default as parameter even work right now?
If we can repro a working scenario, we can get a better understanding.

@mburbea

FWIW I agree it's odd/inconsistent that language null is accepted and interpreted as default with CommandType.StoredProcedure, but not with CommandType.CommandText.
However, treating language null as database null for CommandType.Text would introduce a subtle change between the two; a command converted from StoredProcedure to CommandText would suddenly start sending database NULL instead of default, whereas today it at least throws - which seems much better (users are made aware that things won't work in the same way as before).
The discussion above is only based on the observed behavior of SqlClient. Have we checked what the Odbc/Ole drivers do? How about other database drivers such as MySQL, Oracle, PostgreSQL? It's easy enough to propose a behavior change for SqlClient, .NET has a database abstraction (ADO.NET), and we want to keep behavior consistent across drivers.
There are already other places ADO.NET where language null and database NULL are distinct - notably on the read side. ExecuteScalar, for example, returns DBNull.Value for database null, and language null if no resultset was provided; this proposal would introduce more inconsistency.

I feel like I'm repeating myself, but I still am not seeing the big problem here. Yes - the API might not be ideal, and new users to ADO.NET are likely to bump into this. However, the error message seems very explicit to me: The parameterized query '(@p nvarchar(4000))INSERT INTO foo (bar) VALUES (@p)' expects the parameter '@p', which was not supplied. - the specific parameter name is mentioned, and the user has all the info to successfully understand what they need to do. The workaround is extremely simple (replace null with DBNull.Value), and after bumping into this once, users aren't likely to do this again.
To summarize, I really am not against improving things in general, and even think that the occasional breaking change makes sense if it serves a very valuable purpose. IHMO that simply isn't the case here - the value of this change seems very small weighed against the larger potential impact.

@roji, to answer a few of your points::
I have not tested in other providers. My suggestion was to this repository, SqlClient. I imagine you can answer the postgres question better than anyone else.
And I don't think it logically makes sense to change the read side of the equation. Nor do I see the problem. The sqlclient is already inconsistent in how it treats language null vs db null. If you use a DataTable  as a source for a SqlBulkCopy, it will treat language null as db null. Simmilarly, a tvp using a DataTable source will also treat null values as dbNull. (Using DataTable.CreateReader() for both also works without it vomiting).
var dt = new DataTable { Columns = { { "C", typeof(string) } }, Rows = { { new object[] { null } } } };
await bulkCopy.WriteToServerAsync(dt);
using (var cmd = new SqlCommand(@"insert into t select * from @",conn))
{
    cmd.Parameters.Add(new("@", dt) { SqlDbType = SqlDbType.Structured, TypeName = "dbo.tvp" });
    await cmd.ExecuteNonQueryAsync();
}

I think the message is unclear. I supplied the parameter @. It makes it seem like I forgot a parameter when calling the proc. The message I think at minimum in text mode, should say something akin to "If you meant to use null send DbNull.Value"

I do agree that adding some text to help guide users in the direction of DBNull.Value makes sense, along the lines of the message you propose. But I'd be careful about doing a behavioral change over this point without some thorough research into the other providers and general null handling in ADO.NET.

FYI this bug still exists as of Jan 2023.
For anyone who ended up on this GitHub issue from a web search like I did, I solved it by writing a little wrapper:
public static async AdHocNonQuery(string query, params KeyValuePair<string, object>[] parameters)
{
    using var connection = new SqlConnection(YourConnectionStringHere);
    connection.Open();
    using var sqlCommand = new SqlCommand(query, connection)
    {
        CommandType = CommandType.Text
    };
    foreach (var parameter in parameters)
    {
        if(parameter.Value == null)
        {
            sqlCommand.Parameters.AddWithValue(parameter.Key, DBNull.Value);
        }
        else
        {
            sqlCommand.Parameters.AddWithValue(parameter.Key, parameter.Value);
        }
    }
    await sqlCommand.ExecuteNonQueryAsync().ConfigureAwait(false);
}

private static KeyValuePair<string, object> Param(string paramName, object paramValue)
{
    return new KeyValuePair<string, object>(paramName, paramValue);
}

Then you can call it like so:
await AdHocNonQuery("UPDATE [User] SET [DateOfBirth]=@DOB WHERE [UserID]=@UserID",
 Param("@DOB", null), Param("@UserID", 42));

@ScottRFrost this is not a bug; this is how the ADO.NET API was designed to work.
Of course, that doesn't mean it can't be changed at some point.

@Wraith2 wrote:

You want to pass in nothing and have it infer that language null means database null but that may not be the case because it could equally be an error of input data.

Why are we making this distinction between "language null" and "database null", but not between:

"language int" and "database int"
"language string" and "database string (varchar/text/etc)"
"language bool" and "database bool (bit)"
"language datetime" and "database datetime"
and so on

Developers can pass in all these other values as-is. If a language didn't have a null token, then something would be required to indicate that the developer intends to pass a null instead of, say, an empty string. But when a language has a null token, as is the case in C#, "language null" and "database null" are the same thing and they mean "no value" for the variable or column.
The error message saying a parameter wasn't provided sounds like it wasn't provided at all, not with any value including the non-value null, which in "language" terms should mean AddWithValue wasn't even called for that parameter.
I understand the point about not changing the API and breaking things for people, so I have these suggestions to contribute:

If AddWithValue requires a non-null value, it should be throwing an exception there and not even making a call to the database. The exception message can then make it clear that a value is required and if the developer intends to send null, the developer should pass DBNull.Value.  An exception is already being thrown in this case but this one would be more useful.

Add a new method AddWithNull("@ParameterName") which would be a thin wrapper around AddWithValue("@ParameterName", DBNull.Value) and hopefully will show up for new developers with auto-complete and help them avoid this issue altogether.

If AddWithValue requires a non-null value, it should be throwing an exception there and not even making a call to the database.

It is legal and common to call AddWithValue to create the parameter and then later, often in a loop, set the value to a non-null. value. This is because it's often awkward to call a different parameter ctor or Add overload, AddWithValue is quick common and easy. Changing to an exception now would break code working code at runtime, not a fun experience for users.

Add a new method AddWithNull("@ParameterName") which would be a thin wrapper around AddWithValue("@ParameterName", DBNull.Value) and hopefully will show up for new developers with auto-complete and help them avoid this issue altogether.

This could work. It's something that should probably be added to the DbCommand with a default implementation that can be overloaded by each provider rather than something we do specifically in SqlClient. What do you think @roji ?


Update SHFB docs

Update https://nlog-project.org/documentation/
for NLog 4.6

i want to help with this issue. Can you guide me?

hi!
Thanks for the help!
The https://github.com/NLog/NLog/blob/dev/src/Docs/NLog.shfbproj should be run (maybe tweak some bit), with all updated dlls (also need maybe some more dlls).
Then we need a PR to https://github.com/NLog/NLog.github.io/tree/master/documentation
Any experience with SHFB?

I'm wrote you gitter.im

updated here: https://nlog-project.org/documentation/v4.6.0/
Still need to change the index.html (https://nlog-project.org/documentation)


Use env to avoid committing any private tokens

I thought about this already. I could make the config-Module a peer-dependency and define the name of the Environment-variables here.
Then I had to thought, that this actually should be configured by the main-App not my module. Thats why I want to implement a forRootAsync Method. By doing this the developer can use his own env-variable or hardcode it or whatever he wants.
What do you think about that?

oh yes, good idea! You are right 🤓 . The main app should take care of this and pass it through 👍

This got implemented with the forRootAsync-Method


fix: reproduce transfer

Relates to #2960

Summary
Reproduction for transfer tests.
Breaking Changes
Checklist

[ ] All changes are covered by tests (or not applicable)
[ ] All changes are documented (or not applicable)
[ ] I reviewed the entire PR myself (preferably, on GH UI)
[ ] I described all Breaking Changes (or there's none)

this can be close due to https://github.com/FuelLabs/fuels-ts/issues/2960