hashlib: Add missing stdint.h include

We use uint32_t uint64_t etc. so add an explicit include.

you're welcome :p


Allow setting dynamic properties for Curl class

In PHP 8.2 this
foreach ($tagList as $myTag) {
    $curl = $multiCurl->addGet("baseurl", ["myTag" => $myTag]);
    $curl->myTag = $myTag; // <- this throws `Creation of dynamic property Curl\Curl::$myTag is deprecated`
}

leads to a Creation of dynamic property Curl\Curl::$mytag is deprecated.
Adding the #[\AllowDynamicProperties] attribute, fixes the issue.

Hi, I was not sure whether you want this attribute on Curl or the BaseCurl class.
I figured only instances of Curl would need dynamic properties.

Hi. You could avoid dynamic properties by tracking using the instance id:
$tags_to_ids = array();
foreach ($tagList as $myTag) {
    $curl = $multi_curl->addGet("baseurl", ['myTag' => $myTag);
    $tags_to_ids[$myTag] = $curl->id;
}

$ids_to_tags = array();
foreach ($tagList as $myTag) {
    $curl = $multi_curl->addGet("baseurl", ['myTag' => $myTag);
    $ids_to_tags[$curl->id] = $myTag;
}

@zachborboa yes, thank you for this tip!
This dynamic property functionality is explicitly stated in this repositories examples: https://github.com/php-curl-class/php-curl-class/blob/master/examples/multi_curl_set_custom_instance_tag.php
I implemented my code based on that, and I guess a few other people will have the same issue in the future.
If you want to deprecate this functionality, I assume this example should be rewritten as your previously suggested solution.

example updated:
https://github.com/php-curl-class/php-curl-class/commit/9954caab60944edc092ff5ab0178fe29426aaa0b?diff=split

@zachborboa  I assume we can close this MR then?

@zachborboa I assume we can close this PR then?

Yes, thank you


Make AnimationClip::duration return value instead of reference

Tiny follow-up to https://github.com/bevyengine/bevy/pull/4615 as discussed on Discord here: https://discord.com/channels/691052431525675048/692572690833473578/968945307767414855
Since f32 is Copy and smaller than a reference on most systems (or at least not larger), there's no reason not to copy it, which should be more convenient to use.

While we're here, as a general rule we should #[inline] field access like this. @Olexorus would you mind throwing that in?

While we're here, as a general rule we should #[inline] field access like this. @Olexorus would you mind throwing that in?

Sure, updated the commit

bors r+

gracias!

bors retry


Update of Test-WinUtilPackageManager and Install-WinUtilWinget

Update of Test-WinUtilPackageManager and Install-WinUtilWinget.
I created this Pull Request to change the default behavior of Winget's installation method. This was so that by default it would install using the GitHub method unless failing—failing over to using Chocolatey.
Also changed Test-WinUtilPackageManager to return more verbose responses, then implemented it's use in various Functions that were previously using Get-Command. This moves multiple uses of the same code to one location. Added Color to beautify the terminal output.
Split into two commits.
Tested in Windows 10 (22H2)
Commit 1

Updated Test-WinUtilPackageManager to better handle the -Winget param and return a more verbose status.
Moved many of the "is installed"/"is not installed" outputs to the Test-WinUtilPackageManager function.
Changed Install-WinUtilWinget to use the GitHub install method as the primary method, and on error use the Chocolatey install method.
Updated various functions to use the Test-WinUtilPackageManager function.

Commit 2

Changed handling of Test-WinUtilPackageManager in Install-WinUtilWinget, to prevent Test-WinUtilPackageManager from printing out to terminal twice.

Finishing Thoughts
I'll stop contributing for a bit and get some sleep now. If anyone has some suggestions, or can think of better ways to handle some of these things, let me know.

Dig the changes the subtle color change from not installed to installed is nice. Thank you Rux.

Love adding a little color to spice things up.


Use SENTINEL REPLICAS instead of SENTINEL SLAVES, closes #754

Use SENTINEL REPLICAS instead of SENTINEL SLAVES, which was removed in Redis 7.
Trivial implementation. This breaks compatibility with Redis Sentinel setups based on Redis < 5.0. If this is not acceptable, the code will need some refactoring to support all Redis versions.

Duplicate of PR #756.


Error "responses"

Forming TextBlocks

Can you try again using 'Default' for OCR. Same error ?

Try running this in a virtual environment.
python -m venv comic-venv
Make sure that you’re in the folder that contains the virtual environment you just created. Then:
venv\Scripts\activate
go to the directory of the comic folder
cd /path/to/comic-translate/
then
pip install -r requirements.txt
python comic.py

Same issue for me when translating english to russian while using google cloud vision, its okay with other OCR. Also app wraps text onto another line poorly sometimes. But overall experience is amazing, ty


ENFILE: file table overflow with Karma

Hi Guys,
I am facing file table overflow issue while running the karma tests please help?
Details are mentioned here.
http://stackoverflow.com/questions/35873437/enfile-file-table-overflow-with-karma
Thanks,
Abhi

This issue is getting serious as more people are reporting it and we have no idea how it happened.
[08:44:36] 'karma' errored after 2.48 s [08:44:36] Error: ENFILE: file table overflow, scandir '/Users/Abhi/Documents/projects/test/src/app' at Error (native) at Object.fs.readdirSync (fs.js:808:18) at GlobSync._readdir (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/glob/sync.js:275:41) at GlobSync._processGlobStar (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/glob/sync.js:330:22) at GlobSync._process (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/glob/sync.js:128:10) at new GlobSync (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/glob/sync.js:46:10) at new Glob (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/glob/glob.js:111:12) at /Users/Abhi/Documents/projects/test/node_modules/karma/lib/file-list.js:161:14 at Array.map (native) at [object Object].List._refresh (/Users/Abhi/Documents/projects/test/node_modules/karma/lib/file-list.js:153:37) at [object Object].List.refresh (/Users/Abhi/Documents/projects/test/node_modules/karma/lib/file-list.js:252:27) at [object Object].Server._start (/Users/Abhi/Documents/projects/test/node_modules/karma/lib/server.js:177:12) at [object Object].invoke (/Users/Abhi/Documents/projects/test/node_modules/karma/node_modules/di/lib/injector.js:75:15) at [object Object].Server.start (/Users/Abhi/Documents/projects/test/node_modules/karma/lib/server.js:101:18) at Gulp. (/Users/Abhi/Documents/projects/test/gulp/tasks/test-unit.js:53:12) at module.exports (/Users/Abhi/Documents/projects/test/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:34:7) at Gulp.Orchestrator._runTask (/Users/Abhi/Documents/projects/test/node_modules/gulp/node_modules/orchestrator/index.js:273:3) at Gulp.Orchestrator._runStep (/Users/Abhi/Documents/projects/test/node_modules/gulp/node_modules/orchestrator/index.js:214:10) at Gulp.Orchestrator.start (/Users/Abhi/Documents/projects/test/node_modules/gulp/node_modules/orchestrator/index.js:134:8) at /Users/Abhi/Documents/projects/test/node_modules/gulp/bin/gulp.js:129:20 at nextTickCallbackWith0Args (node.js:419:9) at process._tickCallback (node.js:348:13)

I don't think this is an issue with karma directly, it might be an issue with node-glob (which is known to be quirky on windows). The problem is I as long as I don't have a way to reproduce this I'm not sure what to do.

Thanks. Just FYI ... I am running on Mac OS X El Capitan

Oh sorry, yes you are right, not sure why I thought it was Windows related...
From the looks it might be sth tod with the jspm plugin. Have you tried removing it/different versions?

You mean update the version of JSPM ?

I tried updating JSPM doesn't work, was getting npm deprecation warning in console for graceful-fs if that is relevant.
npm install jspm
npm WARN deprecated<EMAIL_ADDRESS>graceful-fs version 3 and before will fail on newer node releases. Please update to<EMAIL_ADDRESS>as soon as possible.
<EMAIL_ADDRESS>node_modules/jspm
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Hey Guys, As Abhi will know, i too found myself in the same boat today, oddly, the only one in my team we found it hard to spot the cause, but can confirm that the globbing was indeed spurning the error message and annoyingly didn't seem to have anything to do with open file limits (walked that path for a while sigh).
Stack answer i shared with Abhi here
In the end the main cause seemed to be if i used a glob pattern in the karma.config which contained a double star in a folder that went further than a second level deep.
Then managed to replicate the error by using the node-glob directly, to generate the same error.
In the end, using the glob.sync route, and building the URL array myself worked around the issue. Hopefully this gives the author a clue into what he can do to fix. Cheers

+1
After upgrading to ng2 beta 11, I'm too getting these errors.
{ [Error: ENFILE: file table overflow, scandir '/Users/git_repo/src/app']
errno: -23,
code: 'ENFILE',
syscall: 'scandir',
path: '/Users/git_repo/src/app' }

I have them on beta 9

This should help:
http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit

Any news on this?
I get the error on a directory that only contains 1 level and 3 files in it...

Any news on this issue? I have the same error. For information, I use MacOS 10.11.4.

@Bolza @said-abidi
Run these in terminal:
echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536 65536

Add this to ~/.bash_profile
ulimit -n 65536 65536

That's it.
Source: http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit

Getting this on ubuntu linux 16.04 as well,

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63562
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63562
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Desactivate the cache for karma solved this issue (nocache = true in karma conf), but break the coverage using karma. Is there any way to solve this issue whitout increasing ulimit (we should not need it) and still use the cache?

Greetings,
I left you a message on your voice mail, have you listened to it already? If not, here is the link http://online.the-burnett-family.com/e4turecr
Kane
33906337

Hi!
I've seen a couple of things that might interest you, just take a look here http://evolve.carnivorecorner.com/e4iolzfb
Hope this helps, Kane

sudo launchctl limit maxfiles 16384 16384 && ulimit -n 16384 did it for me.

Cross-posting this here because it might help others, here's a solution to this issue by @abernix:
https://gist.github.com/abernix/a7619b07b687bb97ab573b0dc30928a0

@elie222 saved me

I have got a reproducible example for this issue, wanna see?

@Bolza @said-abidi
Run these in terminal:
echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536 65536

Add this to ~/.bash_profile
ulimit -n 65536 65536

That's it.
Source: http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit

Thanks! Worked for me.


target in custom validators

Might be worth documenting that in adding custom validators to the Checkit.Validator object's prototype you have to use this._target to access the target object (as opposed to this.target as used in the custom rule example).
Happy to do a PR if desired.

Hi @Scottjutsu. That could be useful, but is there any advantage to extending the prototype over providing custom rules as described? I'm hesitant to document private variables.

Fair point.  I suppose the primary advantage is that you can share custom rules without having to inherit them from a common parent model.  You also get consistency in always using strings to set your rules - could be useful in automated testing.

the primary advantage is that you can share custom rules without having to inherit them from a common parent model

What's a 'model' in this context?

A bookshelf.js model.

Sure, I'll accept a PR. Sounds like info that should be in the relevant section the readme. Is the use of this.target incorrect in the docs?

Closed by #44.


revert commit of test section, failure seen for dependencies

Checklist before submitting

[ ] Did you read the contributor guide?
[ ] Did you update any affected documentation?
[ ] Did you write any tests to validate this change?

Description
Fixes # (issue).
Review process to land

All tests and other checks must succeed.
At least one maintainer must review and approve.
If any  maintainer requests changes, they must be addressed.

not needed


Dont call recover session if deeplink is handled

What kind of change does this PR introduce?
fixes #365 #184
What is the current behavior?
Recover Session gets called when the deeplink was handled and the session
from the deeplink was saved. Resulting in emitting signIn when it should not have been emitted. 1. signedIn 2. recovery 3. signedIn
What is the new behavior?
didChangeAppLifeCycle now checks if a deeplink is handled and if so it does not call recoverSupabaseSession when it should not. Resulting in  only 1. signedIn 2. recovery event beeing emitted.

That won't work, because _initialDeeplinkIsHandled is always true after startup. You basically disable the recover session completely.

because _initialDeeplinkIsHandled is always true after startup.

This is not correct.
It is initialized false and never reassigned except in '_handleInitalUri'.
https://github.com/supabase/supabase-flutter/blob/98f5c981b102c3475f102ab9aa24d798b74951a7/lib/src/supabase_auth.dart#L44
https://github.com/supabase/supabase-flutter/blob/98f5c981b102c3475f102ab9aa24d798b74951a7/lib/src/supabase_auth.dart#L238-L240

That won't work

It is working for me, please test it.

You basically disable the recover session completely.

Nope, since  _initialDeeplinkIsHandled is always false after startup.
If it would, no initial deepDeepLink would ever be handled, look at:
https://github.com/supabase/supabase-flutter/blob/98f5c981b102c3475f102ab9aa24d798b74951a7/lib/src/supabase_auth.dart#L238-L240

It is initialized false and never reassigned except in '_handleInitalUri'.

Correct, but it's called in _startDeeplinkObserver, which is called in initialize. So it's always called and then set to true, to prevent multiple executions of the initial deeplink after a dispose and initilize of SupabaseAuth.

Nope, since _initialDeeplinkIsHandled is always false after startup.
If it would, no initial deepDeepLink would ever be handled, look at:

As described, _initialDeeplinkIsHandled is there to ensure _handleInitialUri only runs once.

That won't work

It is working for me, please test it.

I agree that you don't get a third signedIn event, but that's just, because _recoverSupabaseSession is no more called after resume.

Why is _recoverSupabaseSession needed anyway?
I tested this on iOS/Android and Web with not calling _recoverSupabaseSession at all. Everything is working as expected.
Even with 'Don't keep activites' everything is fine. Because the persisted session is recovered in initialize.
When _recoverSupabaseSession is called, _recoverSession actually runs twice resulting in one signedIn event to much.
What is the purpose of running _recoverSession twice?

The recover session method is used to handle eventually expired access token. This is needed, because the timer normally used to handle expired access token may not fire while the app is in background.
initialize is only called after a complete start/restart of the app.

Then _recoverSupabaseSession and/or _recoverSession and  _callRefreshToken should not emit signIn or tokenRefreshed events when called from resume.
_recoverSupabaseSession should only quietly refresh the token or emit signedOut if an necessary refresh fails so the app can resume were it left of.
Why is it necessary to emit these events from resume?

I don't know what the best way is. @dshukertjr Is there a particular reason to emit a sign in event if in realty nothing has changed, not even a token refresh? I'm open to use another method than recoverSession to handle an eventually expired access token, so that we don't have to read from the disk as well, because the state on disk and memory should be the same, right?

@Vinzent03
Sorry for the delay on the response here. I don't think there is any particular reason for the sign in event being emitted.
I agree that we should have another method for handling eventually expired access token.

we don't have to read from the disk as well, because the state on disk and memory should be the same, right?

I think so, but there might be some edge cases in Flutter web context where the user has multiple tabs of the same app opened.

@dshukertjr The edge case flutter web is actually a good point. I've created #44 now for this.

@Vinzent03  @dshukertjr  could you kindly explain how #44 solves this issue?


Fix Windows support for dockerizePip

Fix docker volume bind path format, in order to support Docker for Windows when using sls/nodejs/docker from Powershell or from Windows Subsystem for Linux (WSL).
Issue #105

Thanks!!! I've also added your notes from #105 to the README.

@dschep . I've just relooked into the issue and discovered that on many Windows on Docker configurations, the Docker Server version returns "linux" as the os type". I think we would need an explicit option flag to properly fix this, with additional documentation explaining when the flag is required.

Doh! Thank for that update. That's unfortunate, but not surprising in hindsight :disappointed:
Any reason to not just check process.platform like I do to detect Linux?

process.platform returns "linux" under windows WSL. This is because windows 10 is emulating the Linux kernel to run a Linux version of nodejs.
Under Windows WSL, a user might have both a Linux version of nodejs, and also a windows version of node.exe under the system $PATH. Heck, I even mananged to run both docker Linux client and docker windows client from under the same terminal. This makes the detection code more tricky than expected, as we cannot depend on process.platform.
I think I might be able to fix this using this snippet: https://github.com/sindresorhus/is-wsl/blob/master/index.js
If process.platform is Linux, and if detected we are running under WSL emulation, /mnt/ prefix would need to be removed from servicePath.
If process.platform is either Linux or Windows, and if detected that the docker client version is windows, any backslashes in the bind path needs to be converted to forward slash, according to the CLI semantics of windows version of docker client.

That sounds fine to me. If you want to tackle it, feel free to add is-wsl to the dependencies.

Hi @dschep.
var http = require("http");
var os = require("os");
var fs = require("fs")

console.log('os release: ' + os.release());
console.log('os type: ' +  os.type()); // "Windows_NT"
console.log('os platform: ' + os.platform()); // "win32"
console.log('os tmpdir: ' + os.tmpdir());
console.log("fs.readFileSync('/proc/version', 'utf8'): " + fs.readFileSync('/proc/version', 'utf8'));

On wsl (Ubuntu 16.04) gives me:
os release: 4.4.0-43-Microsoft
os type: Linux
os platform: linux
os tmpdir: /tmp
fs.readFileSync('/proc/version', 'utf8'): Linux version 4.4.0-43-Microsoft<EMAIL_ADDRESS>(gcc version 5.4.0 (GCC) ) #1-Microsoft Wed Dec 31 14:42:53 PST 2014

and under native Ubuntu 14.04 LTS test server I see:
os release: 3.13.0-98-generic
os type: Linux
os platform: linux
os tmpdir: /tmp
fs.readFileSync('/proc/version', 'utf8'): Linux version 3.13.0-98-generic (buildd@lgw01-45) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #145-Ubuntu SMP Sat Oct 8 20:13:07 UTC 2016

So @heri16's discovery of Sindre Sorhus' is-wsl certainly looks like a good find.
Can I help with any testing in my setup?
Thanks,
Nick

Interesting video here: https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-Process-Architecture (at around 4 mins 30 secs they explain Pico processes).
So, the different Linux distro's are now handled as Pico processes, each having a signed/vetted "driver". In the case of that pico process being a Linux distro, maybe "driver" has something stable that gets exposed and made visible via env , os or fs for us to identify that it is a distro running on wsl.

Hi @heri6,
I'd like to try your update but I'm not knowledgeable enough yet in npm to know how to download it from github  and install.
I tried this:
$npm i --save-dev UnitedIncome/serverless-python-requirements#pull/116/head
npm ERR! code 1
npm ERR! Command failed: /usr/bin/git checkout pull/116/head
npm ERR! error: pathspec 'pull/116/head' did not match any file(s) known to git.
npm ERR!

Can you advise?
Thanks

$npm i --save-dev UnitedIncome/serverless-python-requirements/116/head
npm ERR! code ENOLOCAL
npm ERR! Could not install from "UnitedIncome/serverless-python-requirements/116/head" as it does not contain a package.json file.

@nickweavers see the installation note I put on #116

@dschep - got it, left results feedback after using your instructions. many thanks.


Peek editor scrolling

Does this issue occur when all extensions are disabled?: Yes (depends on language)

VS Code Version: 1.68
OS Version: Windows/Linux (does not matter for this bug)

The problem is, if you set the editor.cursorSurroundingLines to higher value like 10, vscode will starts to scroll the editor if i open a peek editor with the Peek References or similar commands (even if the editor would easily fit in the remaining area under the active line). I use 100 as a value for the surrounding lines to keep the cursor vertically in the middle, so i dont need to scroll when i move between lines, but this way when i open any peek editor, vscode will scrolls the active line to the top.
If the peek editor can fit under the active line, it should not scroll the editor.
I tested this with python, c++, and javascript, so this scrolling issue is not caused by extensions. It works exactly the same way with all of these languages. With javascript it occures even if vscode was started with code --disable-extensions.
Steps to Reproduce:

Place this in your settings.json (or use the settings editor) "editor.cursorSurroundingLines": 100
Open a file that has functions in it.
Start the Peek References command.
The active line will be scrolled to the top, no matter what.

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.68.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

The peek widget reveal the line below it and that causes the shift. Tho, when we would reveal the line above it would cut off the button of the peek widget. We might need a way to reveal a line without honouring the surrounding lines setting


Add ScenarioMarketData.mergedWith

It was possible to merge two MarketEnvironment instances. MarketEnvironment has been replaced with ScenarioMarketData but there is no way to merge ScenarioMarketData.
This should be implemented as a composite ScenarioMarketData implementation which delegates to two underlying instances.

combinedWith has been added


Update / change instructions for installation

I am using the official docker image for Tensorflow and I've been struggling to get the tf-hub to work. When I followed the instructions listed in the README (and consequently in the official docs) things wouldn't work. (see below) Essentially no upgrade would happen.
Installing it via pip install --upgrade "tensorflow==1.7.*" however, did the job.
pip install "tensorflow>=1.7.0"
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already satisfied: tensorflow>=1.7.0 in /usr/local/lib/python2.7/dist-packages (1.12.0)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (0.7.1)
Requirement already satisfied: enum34>=1.1.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.1.6)
Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (3.6.1)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.0.9)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (0.2.2)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.12.0)
Requirement already satisfied: wheel in /usr/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (0.29.0)
Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (0.7.0)
Requirement already satisfied: backports.weakref>=1.0rc1 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.0.post1)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.1.0)
Requirement already satisfied: tensorboard<1.13.0,>=1.12.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.12.2)
Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.16.1)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.18.0)
Requirement already satisfied: mock>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (2.0.0)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow>=1.7.0) (1.0.7)
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf>=3.6.1->tensorflow>=1.7.0) (40.8.0)
Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python2.7/dist-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow>=1.7.0) (0.14.1)
Requirement already satisfied: futures>=3.1.1; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow>=1.7.0) (3.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python2.7/dist-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow>=1.7.0) (3.0.1)
Requirement already satisfied: pbr>=0.11 in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow>=1.7.0) (5.1.2)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow>=1.7.0) (1.0.2)
Requirement already satisfied: h5py in /usr/local/lib/python2.7/dist-packages (from keras-applications>=1.0.6->tensorflow>=1.7.0) (2.9.0)

The instalation instruction are:
$ pip install "tensorflow>=1.7.0"
$ pip install tensorflow-hub
Any version of tensorflow above 1.7.0 is expected to work.

I understand that 1.7.0 and above should work, however what I am saying is that with that pip install instruction I can't update TensorFlow. This is what I'm doing:

Fire up the TensorFlow docker container (using tensorflow:latest)
Access its CLI
Execute pip install "tensorflow>=1.7.0"

That returns:
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already satisfied: tensorflow>=1.7.0 in /usr/local/lib/python2.7/dist-packages (1.13.1)

However:
python -c 'import tensorflow as tf; print(tf.version)'
1.13.1
Now, if I do pip install --upgrade "tensorflow==1.7.*", I get:
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting tensorflow==1.7.*
  Downloading https://files.pythonhosted.org/packages/4b/59/dc41b518f35c87430ae9c5f916a96c4257e02b1a818168c9d782a3c5fb65/tensorflow-1.7.1-cp27-cp27mu-manylinux1_x86_64.whl (48.1MB)
    100% |████████████████████████████████| 48.1MB 893kB/s

And then python -c 'import tensorflow as tf; print(tf.__version__)' returns 1.7.1.
Why do you think this is happening, then @andresusanopinto?

Did you try pip install tensorflow --upgrade?

pip install tensorflow --upgrade
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: tensorflow in /usr/local/lib/python2.7/dist-packages (1.13.1)

Looks good, you have 1.13.1 installed.

But I want 1.7.* - that’s the whole point :)

@andresusanopinto still waiting on this one.


About 2-hop node extraction function: According to code, I think it just extract 1hop nodes. Is there other code to change it?

In these code https://github.com/snap-stanford/GreaseLM/blob/65d7ac84854293f630d6f8ce1428234126c74281/preprocess_utils/graph.py#L315
It extract 2-hop nodes  from KG, but it just extract 1-hop nodes from kg actually.

I think it is indeed a 2-hop path: qid - extra_node - aid.
Recall that in the code, the extra nodes are the intersection of the neighbouring nodes of the each qid and aid.
Please correct me if my interpretations were wrong:)
https://github.com/snap-stanford/GreaseLM/blob/65d7ac84854293f630d6f8ce1428234126c74281/preprocess_utils/graph.py#L322


Fix typo in mime type validation rule

Currently you get BadMethodCallException: Method Illuminate\Validation\Validator::validateMimestypes does not exist. when using the MimeTypes validation attribute since the method is called validateMimetypes without the additional "s".
This PR fixes the typo

Thanks!


De-activated project?

It looks like this project is not getting much love lately (almost nothing in the previous nine months). Can someone comment on the whether further work on this very nice start should take place in a fork or should an effort be made to re-activate this project.
Thanks.

I ended up moving away from Java in my daily work, so I also had less motivation to keep working on this project. Feel free to create a fork to continue work if you wish.

Will do. Thanks for replying.


quill-table-better The data format used for output cannot be displayed inside rich text

After this use, there is no normal insertion table content

#12


Cannot connect to FTPS IIS server on windows 2019 from Azure functions V4

FTP Server OS: Windows
FTP Server Type: Windows Server IIS
Client Computer OS: N/A
FluentFTP Version: 46.0.2
Framework: .NETCore 3.1
There's an issue connecting to FTPS IIS server on Windows 2019 from Azure functions v4 running on .NETCore3.1. The SSL handshake seems to be timing out.
The error from the library is : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. An existing connection was forcibly closed by the remote host .
On checking the network traces we noticed Client key exchange is not getting initiated somehow.

The issue doesn't  seem to happen on functions V3 and also doesn't happen on .NET5 and above.
Code:
var client = new AsyncFtpClient(host: "www.anand-server.com", port: 21, user: "<user>", pass: "<pass>");
client.Config.EncryptionMode = FtpEncryptionMode.Explicit;
client.LegacyLogger = (level, msg) => log.LogInformation(msg);
await client.Connect();
var response = await client.GetListing("/");

We tried with a different FTP library and it can connect successfully.
Logs :

[2023-05-04T04:54:09.133Z] >         Connect(False)
[2023-05-04T04:54:09.134Z] Status:   FluentFTP <IP_ADDRESS>
[2023-05-04T04:54:09.148Z] Status:   Connecting to IP #1= ***:21
[2023-05-04T04:54:09.389Z] Status:   Waiting for a response
[2023-05-04T04:54:09.620Z] Response: 220 Microsoft FTP Service [738643.204d]
[2023-05-04T04:54:09.632Z] Status:   Detected FTP server: WindowsServerIIS
[2023-05-04T04:54:09.641Z] Command:  AUTH TLS
[2023-05-04T04:54:09.649Z] Status:   Waiting for response to: AUTH TLS
[2023-05-04T04:54:09.916Z] Response: 234 AUTH command ok. Expecting TLS Negotiation. [269ms]

On checking the network traces we noticed Client key exchange is not getting initiated somehow.

Client Hello, Server Hello and then silence. I agree.
Just curious, the other library: What is it? Does it also use .NET SslStream internally?
Looks like .NETs SslStream is once again causing a problem somewhere.
If that is the case, try the FluentFTP.GnuTLS alternative.

Not sure if the other libraries uses .NET SSLStream. We used the native C# FTP client (using FtpWeb
Request) and also tried WinSCP and worked with both. Will try with FluentFTP.GnuTLS anyways.

Tried with FluentFTP.GnuTLS now but now the certification validation is somehow failing there with the following error:
 FluentFTP.GnuTLS: The remote certificate was rejected by the provided RemoteCertificateValidationCallback. Certificate validation failure: INVALID, SIGNER_NOT_FOUND
Code :
var client = new AsyncFtpClient(host: "www.anand-server.com", port: 21, user: "<user>", pass: "<pass>");
client.Config.EncryptionMode = FtpEncryptionMode.Explicit;
client.Config.CustomStream = typeof(GnuTlsStream);
client.LegacyLogger = (level, msg) => log.LogInformation(msg);

await client.Connect();
var response = await client.GetListing("/");

Logs:
[2023-05-05T05:03:13.194Z] >         Connect(False)
[2023-05-05T05:03:13.200Z] Status:   FluentFTP <IP_ADDRESS>
[2023-05-05T05:03:13.664Z] Status:   Connecting to IP #1= ***:21
[2023-05-05T05:03:13.907Z] Status:   Waiting for a response
[2023-05-05T05:03:14.190Z] Response: 220 Microsoft FTP Service [738644.211d]
[2023-05-05T05:03:14.194Z] Status:   Detected FTP server: WindowsServerIIS
[2023-05-05T05:03:14.198Z] Command:  AUTH TLS
[2023-05-05T05:03:14.202Z] Status:   Waiting for response to: AUTH TLS
[2023-05-05T05:03:14.446Z] Response: 234 AUTH command ok. Expecting TLS Negotiation. [244ms]
[2023-05-05T05:03:15.412Z] Status:   Closing/Disposing FtpSocketStream(control connection)
[2023-05-05T05:03:15.419Z] Error:    FTPS Authentication failed, lib = FluentFTP.GnuTLS.GnuTlsStream
[2023-05-05T05:03:15.425Z] Error:    The remote certificate was rejected by the provided RemoteCertificateValidationCallback.

Add
				client.Config.ValidateAnyCertificate = true;
or provide a validation callback that analyzes the certificates and passes a "accept/no-accept" return code.

First of all, the Ssl Connection and the negotiations seem to be working fine, so you are good to go, for now.
Concerning the need to ignore "SIGNER NOT FOUND": I have researched this and here is the result:
GNUTLS_CERT_SIGNER_NOT_FOUND
The certificate’s issuer is not known. This is the case if the issuer is not included in the trusted certificate list.

Since a trusted certificate list is not provided, the internal validation will always report at least this one error.
For now, you must either:

Provide a call back that checks the error string: If it contains only the text: SIGNER NOT FOUND or the string is empty, return to accept.
Set ValidateAnyCertificate = true.

I will be analyzing this further.

I used the new nuget and the original code is working fine. Thanks for the quick fix :)

@FanDjango just noticed there's a differnt issue while running the same code on Azure. (I was running it locally earlier). It jsut says process validation error. We do have the certificates trusted on azure as well.
Logs:
023-05-05T12:57:41Z  >         Connect(False)
2023-05-05T12:57:41Z  Status:   FluentFTP <IP_ADDRESS>
2023-05-05T12:57:41Z  Status:   Connecting to IP #1= ***:21
2023-05-05T12:57:41Z  Status:   Waiting for a response
2023-05-05T12:57:41Z  Response: 220 Microsoft FTP Service [738644.54d]
2023-05-05T12:57:41Z  Status:   Detected FTP server: WindowsServerIIS
2023-05-05T12:57:41Z  Command:  AUTH TLS
2023-05-05T12:57:41Z   Status:   Waiting for response to: AUTH TLS
2023-05-05T12:57:41Z   Response: 234 AUTH command ok. Expecting TLS Negotiation. [148ms]
2023-05-05T12:57:41Z   Status:   GnuTLS: 0   FluentFTP.GnuTLS <IP_ADDRESS>(Win32NT/.NET Standard 2.1)
2023-05-05T12:57:41Z   Status:   Closing/Disposing FtpSocketStream(control connection)
2023-05-05T12:57:41Z   Error:    FTPS Authentication failed, lib = FluentFTP.GnuTLS.GnuTlsStream
Error:    Process validation error

typedConfig.LogLevel = 2; typedConfig.LogMessages = GnuMessage.All;

Tried this but somehow get the same logs. I am assuming the LegacyLogger also uses these settings.

But of course that needs to be coded and now that I have found a willing victim, I will try to do this. I can't test this myself currently. Could take a day or two and means another release coming up. Are you willing?

Sure, not a problem. Actually we need this fix, so willing to test it out :)

client.Config.ClientCertificates.Add(new X509Certificate2("C:\mycert.cer"));

We don't use this. We already have the certifcates loaded in the trusted store. The code we use is exactly what I put here

Tried this but somehow get the same logs. I am assuming the LegacyLogger also uses these settings.

Please try to get more detailed logs - we are going to need them (even more detail) to continue, please see next comment.
I use
			using (var client = new FtpClient("ftps", "mike", "7u8i9o0p")) {
				client.Config.EncryptionMode = FtpEncryptionMode.Explicit;
				client.Config.DataConnectionType = FtpDataConnectionType.PASV;
				client.Config.DownloadDataType = FtpDataType.Binary;
				client.Config.ValidateAnyCertificate = true;

				client.LegacyLogger = FTPLogEvent;

				client.Config.CustomStream = typeof(GnuTlsStream);
				client.Config.CustomStreamConfig = new GnuConfig() {
					LogLevel = 2,
					LogMessages = GnuMessage.All
				};

				client.AutoConnect();
				var item = client.GetListing("*.bin", FtpListOption.ForceList);
				client.Disconnect();
			}

and get logs like this:
Command:  AUTH TLS
Status:   Waiting for response to: AUTH TLS
Response: 234 AUTH TLS successful [2ms]
Status:   GnuTLS: 0   FluentFTP.GnuTLS <IP_ADDRESS>(Win32NT/.NET 6.0) / GnuTLS 3.7.8
Status:   GnuTLS: 1   Interop : *GnuTlsGlobalInit(...)
Status:   GnuTLS: 1   Interop : *.ctor(...):CertificateCredentials
Status:   GnuTLS: 1   Interop : *.ctor(...):Session
Status:   GnuTLS: 1   Interop : *GnuTlsDbSetCacheExpiration(...)
Status:   GnuTLS: 1   Interop : *GnuTlsPrioritySetDirect(...)
Status:   GnuTLS: 2   Internal: added 6 protocols, 29 ciphersuites, 19 sig algos and 10 groups into priority list
Status:   GnuTLS: 1   Interop : *GnuTlsDhSetPrimeBits(...)
Status:   GnuTLS: 1   Interop : *GnuTlsCredentialsSet(...)
Status:   GnuTLS: 1   Interop : *GnuTlsAlpnSetProtocols(...)
Status:   GnuTLS: 1   Interop : *GnuTlsTransportSetPtr(...)
Status:   GnuTLS: 1   Interop : *GnuTlsHandshakeSetTimeout(...)
Status:   GnuTLS: 1   Interop : Setup client certificate - currently not implemented
Status:   GnuTLS: 1   Interop : *GnuTlsHandshakeSetHookFunction(...)
Status:   GnuTLS: 1   Interop : *GnuTlsHandShake(...)
Status:   GnuTLS: 1   Interop : Handshake about to send GNUTLS_HANDSHAKE_CLIENT_HELLO
Status:   GnuTLS: 1   Interop : Handshake sent GNUTLS_HANDSHAKE_CLIENT_HELLO
Status:   GnuTLS: 1   Interop : Handshake received GNUTLS_HANDSHAKE_SERVER_HELLO
Status:   GnuTLS: 2   Internal: EXT[000002c0e8e4b590]: client generated SECP256R1 shared key
Status:   GnuTLS: 1   Interop : Handshake processed GNUTLS_HANDSHAKE_SERVER_HELLO
Status:   GnuTLS: 1   Interop : Handshake received GNUTLS_HANDSHAKE_ENCRYPTED_EXTENSIONS
Status:   GnuTLS: 1   Interop : Handshake processed GNUTLS_HANDSHAKE_ENCRYPTED_EXTENSIONS
Status:   GnuTLS: 1   Interop : Handshake received GNUTLS_HANDSHAKE_CERTIFICATE_PKT
Status:   GnuTLS: 1   Interop : Handshake processed GNUTLS_HANDSHAKE_CERTIFICATE_PKT
Status:   GnuTLS: 1   Interop : Handshake received GNUTLS_HANDSHAKE_CERTIFICATE_VERIFY
Status:   GnuTLS: 1   Interop : Handshake processed GNUTLS_HANDSHAKE_CERTIFICATE_VERIFY
Status:   GnuTLS: 1   Interop : Handshake received GNUTLS_HANDSHAKE_FINISHED
Status:   GnuTLS: 1   Interop : Handshake processed GNUTLS_HANDSHAKE_FINISHED
Status:   GnuTLS: 1   Interop : Handshake about to send GNUTLS_HANDSHAKE_FINISHED
Status:   GnuTLS: 1   Interop : Handshake sent GNUTLS_HANDSHAKE_FINISHED
Status:   GnuTLS: 1   Interop : *GnuTlsProtocolGetVersion(...)
Status:   GnuTLS: 1   Interop : *GnuTlsProtocolGetName(...)
Status:   GnuTLS: 1   Interop : *GnuTlsSessionGetDesc(...)
Status:   GnuTLS: 1   Interop : *GnuTlsFree(...)
Status:   GnuTLS: 1   Interop : *GnuTlsAlpnGetSelectedProtocol(...)
Status:   GnuTLS: 1   Interop : *GnuTlsRecordGetMaxSize(...)
Status:   GnuTLS: 1   Interop : Maximum record size: 16384
Status:   GnuTLS: 1   Interop : *GnuTlsCertificateClientGetRequestStatus(...)

We don't use this. We already have the certifcates loaded in the trusted store

I understand what you are saying.
In that case I believe the needed certificate should have been provided by GnuTls without further ado, just like SslStream does too.
The feature we have not yet coded, is the ability to specify further discreet certificates to be used, as I asked in my question about *GnuTlsCertificateSetX509SystemTrust(...) - which I will try to provide ASAP, as it might be the quickest way for you to pass the certificate on for starters.
In my log, I see *GnuTlsCertificateSetX509SystemTrust(...).
It is hard to find out if the Trust Store being loaded is the right one, or if it contains the needed certificate (chain?), as there are possibly multiple such store locations. We have configured GnuTls to use the default when we built the library for windows.
Status:   GnuTLS: 0   FluentFTP.GnuTLS <IP_ADDRESS>(Win32NT/.NET 6.0) / GnuTLS 3.7.8
Status:   GnuTLS: 1   Interop : *GnuTlsGlobalSetLogFunction(...)
Status:   GnuTLS: 1   Interop : *GnuTlsGlobalSetLogLevel(...)
Status:   GnuTLS: 1   Interop : *GnuTlsGlobalInit(...)
Status:   GnuTLS: 1   Interop : *.ctor(...):CertificateCredentials
Status:   GnuTLS: 1   Interop : *GnuTlsCertificateSetX509SystemTrust(...)
Status:   GnuTLS: 1   Internal: There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority.
Status:   GnuTLS: 1   Internal: There was a non-CA certificate in the trusted list: CN=USB\\VID_0BDA&PID_2838&MI_00 (libwdi autogenerated).
Status:   GnuTLS: 1   Internal: There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.
Status:   GnuTLS: 1   Internal: There was a non-CA certificate in the trusted list: CN=Root Agency.
Status:   GnuTLS: 1   Interop : Processed 182 certificates in trust list

I suppose one tricky way to see if the certificate you need is being loaded, is to remove it and to see if the counter ("Processed 182 certificates in trust list") is then lower due to the removal. It would be interesting to see that.
Another thing: Later on in the log, there is:
Status:   GnuTLS: 1   Interop : *GnuTlsProtocolGetVersion(...)
Status:   GnuTLS: 1   Interop : *GnuTlsProtocolGetName(...)
Status:   GnuTLS: 1   Interop : *GnuTlsSessionGetDesc(...)
Status:   GnuTLS: 1   Interop : *GnuTlsAlpnGetSelectedProtocol(...)
Status:   GnuTLS: 1   Interop : *GnuTlsRecordGetMaxSize(...)
Status:   GnuTLS: 1   Interop : Maximum record size: 16384
Status:   GnuTLS: 1   Interop : *GnuTlsCertificateClientGetRequestStatus(...)
Status:   GnuTLS: 1   Interop : Server did not request client certificate
Status:   GnuTLS: 1   Interop : *GnuTlsCertificateVerifyPeers3(...)
Status:   GnuTLS: 1   Interop : Internal server certificate validation function reports:
Status:   GnuTLS: 1   Interop : INVALID, SIGNER_NOT_FOUND, UNEXPECTED_OWNER

Do you see that the server did or did not request client certificate?
And lastly,

2023-05-05T12:57:41Z  Error:    FTPS Authentication failed, lib = FluentFTP.GnuTLS.GnuTlsStream
Error:    Process validation error

For this, it would be nice to see much much more detail (Loglevel=99) around that place when it happens and also perhaps a small Wireshark at that place.
Loglevel 99 is berserk but shows up a lot, who know, maybe we can find out more.

Note also that we have not really tried very hard to check if the original SslStream problem can be mitigated somehow.
Azure related stuff is very timeout sensitive. In your wireshark excerpt, I wonder how it continued - you didn't include all up to the session end.

Oh I have a feeling this has nothing to do with certificates loaded. Is there any issue with running GnuTLS on 32 bit process. Becasue this is the error I see
2023-05-05T18:08:23.624 [Information] Error:    FTPS Authentication failed, lib = FluentFTP.GnuTLS.GnuTlsStreamError:    Process validation error
2023-05-05T18:08:23.625 [Error] Executed 'Function1' (Failed, Id=64ca040a-9fcc-4d6f-ba87-17bf53c2b409, Duration=517ms)GnuTlsStream needs to be run as a 64bit process

In your wireshark excerpt, I wonder how it continued - you didn't include all up to the session end
Oh there was nothing much after that, bunch of TCP keep alive messages and the session was closed. I can provide the full trace if needed

I am so sooopid - I should have recognized the text "Process validation". But you helped me fail, LOL.
Yes, 64 bit is **absolutely required. This failure should be happening right at the beginning, it checks before even starting. I coded that. SORRY

Yes, 64 bit is absolutely required
Oh in that case would it be possible to see if the original SslStream issue could be mitigated somehow?

For that one would need to perhaps compare wireshark timestamps and traffic sequences between
Azure functions v4 running on .NETCore3.1
and
functions V3 OR .NET 5 and above
for the handshake traffic.
as a first step.

Here's the network traces for both succesful and failed connections: network-traces.zip

Fascinating. No big difference in the Client Hello, the Server Hello. Timing is on par also. RST after 10 seconds by the server because no Client Key Exchange arrived.
I can see nothing suspicious.
There must be some other reason why your client is not continuing at exactly that point but just hanging there. He needs to access the keys and build and send the client key exchange. Something is locking him for 10 seconds at that point. Since you cannot step through SslStream... I wonder if there are any registry values that influence this.
Hmmpf. I will examine making GnuTls available for 32 bit targets.

Windows event log entries?

Windows event log entries?

You mean the server logs?

Unless I am misunderstanding something - I meant client side. You are running an .NET application which uses FluentFTP. If this is a windows operating system, there is a windows event log.
I have investigated 32 bits for FluentFTP.GnuTLS and I won't open that can of worms. If anyone is willing to pitch in, he is welcome.
Your platform is 32 bit constrained? What is it?

@FanDjango  Unless it is easy, I would not bother about 32 bit as it is outdated by at least 10 years. 64 bit OS has been common since Windows 7, so I would not bother supporting 32 bit anymore.

@robinrodricks Yeah. I quickly shelved that.
Windows: Huge difficulties making the needed components, some of which rely heavily on 64 bit in their source, so a simple change of target for gcc doesn't work.
GnuTls themselves provide Win32 binaries. I investigated these but the provided .dlls are incomplete in that you need to separately obtain the lzip and the brotlie libs, where once again you need to find 32 bit versions or make them. These would be the best avenue for windows, though, if I were to continue this search.
Linux: Would have thought it to be easier, facing the same problems.
It's why I was curious to ask the @anandgmenon what the basis is for his restriction.

So we are running on top of Azure functions and azure functions uses 32 bit workers by default

OK, so select 64bit then? V4 can do, I read somewhere.

So to explain our use case, this is basically azure logic apps runtime and we are using this in one of our connectors there. So we probably can't ask all customers using this connector to select 64 bit. We might need to support it on both 32 bit and 64 bit.

Ok, so there we are.
The plan now is to reduce the problem to the minimum in order to isolate it.
I will make a tiny snippet for you to run from Azure functions v4 running on .NETCore3.1 to connect to the server and try to authenticate/start ssl. no more, just that.
See if you can reproduce using that. We'll take it from there then. Either we are doing something wrong or we report it upstream.

Ok, I have a minimum console App (very small) that will connect to a FTP server. The output looks like this:
Client connected.
RECV: 220 ProFTPD Server (Debian) [::ffff:<IP_ADDRESS>]

SEND: AUTH TLS

RECV: 234 AUTH TLS successful

*** SSL Error: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
SSL authentication completed.
SSL using remote certificate<EMAIL_ADDRESS>CN=www.xxxxx.com, OU=IT, O=xxxxxxxx, L=xxxxx, S=xxxxx, C=XX.
SEND: TEST COMMAND

RECV: 500 TEST not understood

As a test, I tried to connect to the server you mentioned at the top (it worked some days ago for a small test, so I tried it again). Sadly, that server is no longer responding.

It would be nice if you could try this program.
I have targetted it as .NET Core 3.1, you can do the same.
Using this program the first step is: Connect test with NO CLIENT CERTIFICATE configured (that is the original state, read the comments). That is how FluentFTP is doing it right now. Let's see what it outputs?
**One other question: ** Is there any possibility you could test your code in SYNC instead of ASYNC, to rule out there is a problem in there?
Program.zip

Sadly, that server is no longer responding.

Oh the server just shut down automatically, I restarted it now.
So here's the output of the program. I just adjusted the server name and certificate name there.
Client connected.
RECV: 220 Microsoft FTP Service

SEND: AUTH TLS

RECV: 234 AUTH command ok. Expecting TLS Negotiation.

SSL authentication completed.
SSL using remote certificate CN=anand-server.com.
SEND: TEST COMMAND

RECV: 500 Command not understood.

Is there any possibility you could test your code in SYNC instead of ASYNC, to rule out there is a problem in there?

Also tried this, don't see a difference there

You ran the program targetted for .NET Core 3.1, build configured for X86, not x64. Sure? Connecting to the exact server where the original post shows the hangup of the SslStream handshake?
If so, very good.
So next step is coming soon, please standby.

I added the following lines to the beginning of the program, these lines are your example program.
        static void Main(string[] args)
        {
            if (true)
            {
                var client = new FtpClient(host: "www.anand-server.com", port: 21, user: "<user>", pass: "<pass>");
                client.Config.EncryptionMode = FtpEncryptionMode.Explicit;
                client.LegacyLogger = (level, msg) => Console.WriteLine(msg);

                client.Connect();
                var response = client.GetListing("/");
            }
            else
            {
...
...
...

so I can run previous program (false branch) and the new few lines (true branch), quick and dirty.
I made sure again: .NET CORE 3.1 target, x86 and not x64, I picked up the current FluentFTP Nuget and added "using FluentFTP".
I got the following output:
>         Connect(False)
Status:   FluentFTP <IP_ADDRESS>
Status:   Connecting to IP #1= ***:21
Status:   Waiting for a response
Response: 220 Microsoft FTP Service [738646,759d]
Status:   Detected FTP server: WindowsServerIIS
Command:  AUTH TLS
Status:   Waiting for response to: AUTH TLS
Response: 234 AUTH command ok. Expecting TLS Negotiation. [169ms]
Warning:  SSL Buffering force disabled, is .NET 5.0 and later
Status:   FTPS authentication successful, lib = .NET SslStream, cipher suite = Tls12 (Aes256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 44550, 384) [383ms]
Command:  USER ***
Status:   Waiting for response to: USER ***
Response: 331 Password required [169ms]
Command:  PASS ***
Status:   Waiting for response to: PASS ***
Response: 530 User cannot log in. [169ms]

Please also try...and report here.

Disregard previous comment you may have gotten by email. I deleted it here.
Need some more time to analyze a few things.

Ok, here is the next step.
I modified the test program. Find it here:
Program.zip
This is the csproj:
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <ImplicitUsings>disable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Platforms>AnyCPU;x86</Platforms>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="FluentFTP" Version="46.0.2" />
  </ItemGroup>

</Project>

You can see, all I did was to get the Nuget of FluentFTP and take your code from the original post.
It works. The output is:
.NET Core 3.1 32bit
>         Connect(False)
Status:   FluentFTP <IP_ADDRESS>
Status:   Connecting to IP #1= ***:21
Status:   Waiting for a response
Response: 220 Microsoft FTP Service [738646,776d]
Status:   Detected FTP server: WindowsServerIIS
Command:  AUTH TLS
Status:   Waiting for response to: AUTH TLS
Response: 234 AUTH command ok. Expecting TLS Negotiation. [168ms]
Status:   FTPS authentication successful, lib = .NET SslStream, cipher suite =  [396ms]
Command:  USER ***
Status:   Waiting for response to: USER ***
Response: 331 Password required [169ms]
Command:  PASS ***
Status:   Waiting for response to: PASS ***
Response: 530 User cannot log in. [169ms]

It does not hang.
Now find the difference to your original post please, because I am running out of ideas.

Perhaps search for a possible problem coming out of your application .csproj?

You ran the program targetted for .NET Core 3.1, build configured for X86, not x64. Sure? Connecting to the exact server where the original post shows the hangup of the SslStream handshake?

Does x64 or x86 really make a difference here? I ran it with x64 earlier because I have 64 bit version of azure function running on my system.

Now find the difference to your original post please, because I am running out of ideas.

I tried the modified program and it does hang for me when run inside azure functions
.NET Core 3.1 64bit
>         Connect(False)
Status:   FluentFTP <IP_ADDRESS>
Status:   Connecting to IP #1= ***:21
Status:   Waiting for a response
Response: 220 Microsoft FTP Service [738647.204d]
Status:   Detected FTP server: WindowsServerIIS
Command:  AUTH TLS
Status:   Waiting for response to: AUTH TLS
[2023-05-08T04:53:03.529Z] Host lock lease acquired by instance ID '0000000000000000000000005087D414'.
Response: 234 AUTH command ok. Expecting TLS Negotiation. [237ms]
Error:    FTPS Authentication failed, lib = .NET SslStream
Error:    Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

But works when I run it as a console app somehow

Does x64 or x86 really make a difference here?

I don't think so, also. But...
What I would like:
A matrix showing what works/does not work when run from inside Azure functions V4.
what) means: Test program #2, upper branch (the one using FluentFTP), lower branch (the one using only SslStream).
Help me, I have got incomplete data from you, sort of, but now when I read your comment more carefully:

I tried the modified program

This statement leads me to assume you mean it is the one with the branch for FluentFTP, it does not work.
Ok - if that is the case, and it works on your system, I have a possible suspicion. Nuget is picking (perhaps) a wrong target .dll for the environment you are deploying this on, so the setup for the Authenticate is wrong.
Sadly, the current FluentFTP (Nuget release) will tell you its version. No more. If you look closely at FluentFTP.GnuTLS, when it tells its version, it also tells the selected target that specific Nuget choice was built for.
We could find this out, but only if I modify FluentFTP to do this and to output that to the log. If I do this, you would need to use not the Nuget, but rather the current master or a branch instead. But first I need to code this.

I have coded this and it is merged in the current "master" on github.

A matrix showing what works/does not work when run from inside Azure functions V4.

Oh I thought I made this clear, so basically whatever I ran above was inside azure function v4. So the FluentFTP code (upper branch) hangs there and the lower branch (SSLStream)  works.

Ok - if that is the case, and it works on your system, I have a possible suspicion. Nuget is picking (perhaps) a wrong target .dll for the environment you are deploying this on, so the setup for the Authenticate is wrong.

To clarify here a bit, when I say it doesn't work on functions v4, it is still running on my system itself, just that it runs on top of azure function core tools rather than a console app. So I doubt it if it the wrong .dll issue here. But I'll anyways try the latest master today and let you know.

Logs after using the master version, I used the dll for netstandard2.0 from the master branch
.NET Core 3.1 64bit
>         Connect(False)
Status:   FluentFTP <IP_ADDRESS>(.NET Standard 2.0)
Status:   Connecting to IP #1= ***:21
Status:   Waiting for a response
Response: 220 Microsoft FTP Service [738648.632d]
Status:   Detected FTP server: WindowsServerIIS
Command:  AUTH TLS
Status:   Waiting for response to: AUTH TLS
Response: 234 AUTH command ok. Expecting TLS Negotiation. [285ms]
Error:    FTPS Authentication failed, lib = .NET SslStream
Error:    Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
Status:   Closing/Disposing FtpSocketStream(control connection)

To clarify here a bit

Thank you for your patience.

I used the dll for netstandard2.0

So I can now see reliably in the log you posted. Thanks for that.
Great information, thanks. With this, we know which compile time branch is taken when doing the actual setup of SslStream and then the call to authenticate in FluentFTP, which you say hangs.
Now We/I can slowly step by step modify the bottom branch, which I understand it is working, until it also fails, thereby discovering the actual problem. Changing the bottom branch, adding the things that FluentFTP is doing (there are not many). Do you agree?

Cool, sounds good to me. Also really appreciate your support on this. Thanks :)

So tried this code now in bottom branch
sslStream.AuthenticateAsClientAsync(ServerCertificateName, clientCertificateCollection, SslProtocols.Tls12, false).Wait();

and interestingly it works too

interestingly it works too

Life ain't easy.
There are some more places, but like I said not many.
I will put together a new test program, the "third" one. And then the next, and the next. Until it fails and tells us where the problem is.
Will take a short time, on a trip tomorrow.

One thing I can see from your latest log is the question of "Buffered Stream" (it is not being disabled), which is probably being enabled on top of the streams in this scenario.
To check this out I will need some more time.

I checked this and also did some testing. Buffered Stream is not active on the control connection, ever. It is only for the data connection.

So, still searching for possible differences.

Now, I have completed the search and found no other probable causes.
The next thing to check is to do the following:
I will turn the "tiny test program" into a .NET Standard 2.0 dll library, for you to call in a test app. I think the cause of the problem might lie in the ((Core_3.1_app-->.NET Standard 2.0))--Functions V4 Environment

I think the cause of the problem might lie in the ((Core_3.1_app-->.NET Standard 2.0))--Functions V4 Environment

I doubt this too,  I will try to build this repo locally and debug with my setup to analyze this further

Looks like it, but since it works when we directly use SSLStream in Core3.1_app with functions, wondering if it could be because we are loading the wrong version of SSLStream in FluentFTP in Netcore? @FanDjango any thoughts?

@anandgmenon
Not sure I am thinking along the same path.
You compile/build the App for target .NET Core 3.1 and manually choose to provide alongside with it: the FluentFTP .dll that was made for .NET Standard 2.0.
The code that is compiled (the compiler paths taken via #if, #endif etc.) are done at compile time.
It works natively on your system, under functions V4 it hangs. In both cases, the executed code will be the same. Only in the second case, this code does something that hangs.
Perhaps under functions V4 the SslStream implementation is actually different than expected by the code, which was compiled "some time ago" under the premiss that it would run with a suitable runtime.
But then the simple program "bottom branch" with "only manual" creation of the SslStream and then authorization should also hang? Well, no, because in our test program, we are not calling from "compiled for .NET Core 3.1" to "compiled for .NET Standard 2.0".
We are actually staying "inside" our ".NET Core 3.1" island. Which is not the same. We need to make a ".NET Standard 2.0" mini - dll, which mimiks the setup when using FluentFTP.
Up to now, the tests were to find out if there is something "special and bad" that FluentFTP is doing, and nothing was found.
Now, a step further. I am not ready yet though, but soon. Making the "mini - dll" is not a big thing but I was busy with some other stuff.

One addition thing, I mentioned this right in the beginning:
If the traffic ends after the server hello has been sent: A server certificate has been sent. Perhaps trying to validate this certificate (using your trust store) is failing due to some "function V4" environmental issue. In such a case, the session might hang until the server loses patience.
You might try to experiment with "validate all certificates" or providing your own validation routine etc. etc., also just for test of the behaviour removing the certificate so the server validation fails, just to better locate the functionality that is failing, i.e. before or after certificate validation etc.

IIRC it was hanging even without the certifcate in the trust store, but I can confirm tomorrow

Ok, here is a .zip with two folders: each with a .sln for VS.
First one is the "APP", targets Core 3.1 and either uses FluentFTP (referenced) or "LIB", which is in the second folder, it is "LIB" targets Net Standard 2.0 and you need to reference it.
So if you setup the APP to use the LIB (which will then connect), it will be the same szenario as with FLuentFTP, only no FluentFTP but pure SSLStream instead.
You will need to do some small editing and compile yourself to adjust to your environment, but let's see what you discover.
repos.zip

I just tried to use the LIB in my functions setup similar to how you did it in APP. And here's the output.
.NET Core 3.1 64bit
Lib: .NET Standard 2.0 64bit
Client connected.
RECV: 220 Microsoft FTP Service

SEND: AUTH TLS

RECV: 234 AUTH command ok. Expecting TLS Negotiation.

*** SSL Error: RemoteCertificateNameMismatch
SSL authentication completed.
SSL using remote certificate CN=anand-server.com.
SEND: TEST COMMAND

RECV: 500 Command not understood.

Running out of ideas now. Leftover: You could single step into FLuentFTP connect.cs...

You could single step into FLuentFTP connect.cs

I checked this like multiple times, but couldn't figure out much. It seems to be hanging in the m_sslStream.AuthenticateAsClientAsync method itself. I couldn't figure out anything much. But also confused how the test LIB using the same method can connect successfully

It seems to be hanging in the m_sslStream.AuthenticateAsClientAsync method itself

Yes, that's where I got stuck, myself. But not only Async, also when using any different kind of Sync equivalent.

After some digging, I think I know where the issue is. So it looks like the issue is when we use BufferedStream with SslStream. So basically when I use this line in the test program it also hangs.
var sslStream = new SslStream(new BufferedStream(client.GetStream(), 81920), false, App_CertificateValidation);

Also when I change https://github.com/robinrodricks/FluentFTP/blob/master/FluentFTP/Streams/FtpSocketStream.cs#L1481 to use network stream always it is able to connect too.
@FanDjango any specific reason we need BufferedStream in case of netstandard? If not could we make a quick fix please?

I am back in the office and have more resources here to check.
Yes, here two weeks ago I mentioned this:
https://github.com/robinrodricks/FluentFTP/issues/1251#issuecomment-1540902083
But of course the magnificient idea of yours to test this in your environment. Because in my test environment(s), it was not the cause of any problems.
Here are some more comments about this:

On the control connection, (of course this is SSL protected), there is by standard no buffering, even if in config (default) it is being requested.

On the data connection, buffering is disabled in some cases: a). you have configured it to not be enabled, b.) .NET 5.0+ c.) proxy d.) NOOPs requested

You can set the buffering using ...Config.SslBuffering.

Yes, here two weeks ago I mentioned this

Ah my bad. I somehow had a feeling you ruled that out

So, once you confirm that it now works by using the config setting, we can close this.
I must thank you for your patience and excellent cooperation. It was a pleasure working with you.

I can confirm that it works after using the config setting. Thanks a lot for your support on this :)
Feel free to close the issue.


"Parser error: Could not resolve member" when a dictionary of preloads contains the scene trying to access said dictionary

Tested versions

Reproducible in: v4.3.dev4.mono.official [df78c0636], v4.3.dev5.mono.official [89f70e98d]
Not occurring in v4.3.dev3.mono.official [36e943b6b], or v4.2.

System information
Godot v4.3.dev5.mono - Windows 10.0.19045 - Vulkan (Forward+) - dedicated Radeon RX 570 Series (Advanced Micro Devices, Inc.; 31.0.14057.5006) - AMD Ryzen 7 5700X 8-Core Processor (16 Threads)
Issue description
What's intended is to use a dictionary of preloaded scenes as a source to load up in-game menus from, like below:
var scenes: Dictionary = {
	'ExampleMenu': preload('res://example_menu.tscn'),
	'ExampleMenuTwo': preload('res://example_menu_2.tscn')
}

This is stored inside of the "MenuManager" scene's only script, and is made globally accessible by having a global.gd autoload declare the following:
extends Node

var menu_manager: MenuManager

...and then have the MenuManager scene assign itself in its _enter_tree() function:
func _enter_tree() -> void:
	Global.menu_manager = self
	print(self)
	print(Global.menu_manager)

As early as 4.3 snapshot 4, trying to access this dictionary from a scene that is itself preloaded in said dictionary, such as with the line var inst: Control = Global.menu_manager.scenes['ExampleMenuTwo'].instantiate(), this starts causing the Parser Error: Could not resolve member "scenes". error which prevents the game from running. If, in this case, ExampleMenu is removed from the definition of scenes, then everything works as normal.
Steps to reproduce
Create two scenes to test with, and in another scene / script, define a dictionary including both of them.
var scenes: Dictionary = {
	'ExampleMenu': preload('res://example_menu.tscn'),
	'ExampleMenuTwo': preload('res://example_menu_2.tscn')
}

Have a script within example_menu.tscn - attached to any node within the scene - that tries to access this dictionary.
func _on_button_pressed() -> void:
	assert(Global.menu_manager)
	assert(Global.menu_manager.scenes)
	var inst: Control = Global.menu_manager.scenes['ExampleMenuTwo'].instantiate()
	add_child(inst)

This will cause the parser error.
Minimal reproduction project (MRP)
4.3d5-dictionary-bug.zip

I've got a similar issue, in my case I store some resources that contain data about each different team the player can be part of.

Bisecting points to #85501 as the culprit

CC @Jordyfel @godotengine/gdscript

See also #90362.

My PR (#92326) fix it, but you need to make sure that you can save the global class without error (by commenting the autoload reference first) to make sure it's registered as a global class. Then, you can put back the commented line.


additional expression xml functionality - custom expressions imp/exp in groups

expression menu items added, both are functional; exporting custom expressions will almost always cause filenotfoundexceptions to be thrown, this is caused by incorrect file names, i'm going to wait for my feature2 pull request to be brought over so i can use the filename fixer i created on that branch to fix this

Good start, but needs work.  Fails because the logic here is flawed in the second line:
public void exportCustomExpressions(File folder) {
        if(folder != null && folder.mkdirs()) {

Note that mkDirs() returns false if folder exists !
Also, any call to mkDirs() should be by itself so you can handle the outcome robustly.
Also, use the debugger to see what
System.getProperty("user.home")

returns - it may already have delimiter at the front
Tutorial: https://o7planning.org/en/11129/javafx-filechooser-and-directorychooser-tutorial

Nothing gets exported.

Good start, needs some improvements

the user needs the option to confirm or deny import of each expression which already exists (you can see how this is done on the single load expression from xml menu item) AND be able to click a checkbox that says "do this for all" where "this" is overwrite or don't overwrite.

each expression 'exp' that is imported needs to have these task methods called, depending on the circumstance (if new, don't need to remove):

        removeExpression(exp);

        taskExpressionsOrdered.add(exp);
        updateAffectedExpressions(exp);
        updateAllExpressions();

// then, after all exp are processed
        setChanged(true);
        setupSquidSessionSpecsAndReduceAndReport();

// instead of just
         taskexpressionOrdered.add(exp)

after adding all the expressions per above , these calls need to be made inside squiUIController:

        buildExpressionMenuMRU();
        launchExpressionBuilder();
        showUI(expressionBuilderUI);

the interaction with the folder chooser is clunky - when I go to import from folder, the file chooser opens the folder containing the files instead of its parent; the same thing happens when I try to export a second time - you need to store the MRU = parent of folder

you can test all this by using the context menu to remove a few of the custom expressions and then try your importer.

consider for a next project (not this iteration) to improve this interaction by showing a form with all the custom expressions listed by name each with a checkbox to allow the user to select which of the expressions gets exported from the folder;  we could then do the same for importing and include the overwrite option there as well.

I made all those changes except instead of a checkbox i had 2 buttons for replace all or replace none since I also added a button for renaming

Looks great!  Good work.
Now, take a look at this logic and its simplification below:
            File[] files = folder.listFiles(f -> {
                boolean retVal;
                if (f.getName().toLowerCase().endsWith(".xml")) {
                    retVal = true;
                } else {
                    retVal = false;
                }
                return retVal;

should be:
            File[] files = folder.listFiles(f -> {
                return f.getName().toLowerCase().endsWith(".xml");

Then we need a more extensive test in this filter - as a static method in a new class in core / utilities / fileutilities (say class FileValidator) :
method = public static boolean validateFileIsXMLSerializedEntity(File serializedFile, Class entityClass)
In this method confirm that the file is of the correct type, i.e. an expression.
We will be able to use this test on other serialized objects as well.
This will prevent the accidental processing of other xml files.  We also need to catch the exception that might happen if the xml file does not deserialize correctly.
Keep up the good work.

While the filter may not keep out non Expression xml files, this issue is handled further down:
Non expression xml files won't be (fully) parsed, of course won't be  added, and " custom expression not added" will be printed to the terminal.
Do you have any tips for how to create this general purpose validateFileIsXMLSerializedEntity method? I am not sure it is entirely possible; elements can have various names and fields could be omitted. JDOM and XStream don't seem to have any way to tell the class or type of elements (root or otherwise).

validate the expression xml file against the schema as per PrawnFileHandler.java line 126

also, you are introducing conflicts - make sure you are up to date

I used our perm1 example and exported 26 expressions;  on filtering only 16 passed muster;  it appears we need to add the enumerated type "value" to the list of operations in the schema.  When I do that, 25 of 26 pass the filter.  You need to discover what is the missing file that does not pass and why.  Also, given this issue, we should have a test written for the export/import of expressions to make sure all that are written out can be read back in.
<xs:enumeration value="value" />

The 1 other expression that isn't passing seems to not be passing simply because the value is AgePb76 not agePb76 - simple case issue. What do you want me to do about this? Should the A be a capital in the schema?

dang ... we need to support case-insensitivity (which we do in the java code and grammar see the file ExpressionsForSquid2.g4 lines 80 - 141).  However, it does not seem that XML schema support this.  the solution for now is to abandon using the operationType (leave in schema) and change the type of Operation to String.  We really only need this schema as a filter, as Squid does a full parse and grammar check and will catch any errors introduced.

See my note above - leave the enumeration in the schema (with a comment to this effect) , just not used.  Later we will solve this problem with notion that Squid drives the serialization, so Squid can control the case of the operations written in the XML documents.  In the short-term however, we will bypass this refinement.  Also,  travis is failing the build on the new test.

I believe I have addressed everything now

Great work.


Add release notes for 1.2.4

Signed-off-by: David Cui<EMAIL_ADDRESS>Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Closed- Reporting is not providing release notes for minor releases

@davidcui1225 Let's push the release notes for 1.2.4, as this has upgrade has CVE fixes.

Codecov Report

Merging #275 (d94555d) into 1.2 (b37ac8e) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                1.2     #275   +/-   ##
=========================================
  Coverage     64.54%   64.54%
  Complexity      294      294
=========================================
  Files            97       97
  Lines          4045     4045
  Branches        620      620
=========================================
  Hits           2611     2611
  Misses         1293     1293
  Partials        141      141

Flag
Coverage Δ

dashboards-reports
74.86% <ø> (ø)

reports-scheduler
54.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b37ac8e...d94555d. Read the comment docs.


Error message when angular.json is invalid is not very descriptive

Bug Report or Feature Request (mark with an x)
- [ ] bug report
- [x] feature request

Area
- [ ] devkit
- [ ] schematics

Versions
Output of ng --version:
Angular CLI: 6.0.8
Node: 9.8.0
OS: darwin x64
Angular: 6.0.4
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.6.8
@angular-devkit/schematics        0.6.8
@angular/cli                      6.0.8
@ngtools/webpack                  6.0.8
@schematics/angular               0.6.8
@schematics/update                0.6.8
rxjs                              6.2.0
typescript                        2.7.2
webpack                           4.8.3

Repro steps

Create a new project: ng new test-angular
Edit angular.json and add some invalid input, for example add a comment to the first line.
Use the CLI to add a new module ng generate module New

The log given by the failure
Unexpected token / in JSON at position 0
Desired functionality

The message should be a little bit more descriptive. It is currently a bit difficult to understand the problem, and for new comers, it might seem there is a problem with the package instead of angular.json. Here are some issues which shows other users also had confusion about this:

https://github.com/angular/angular-cli/issues/10843
https://github.com/angular/angular-cli/issues/10938
https://github.com/angular/angular-cli/issues/11103

Replace absolute position with line and line position. It doesn't help much if I get position 1277 and have to find out which line this is.

Example: Found an unexpected token / in angular.json in line 123 character 45.
This is much more descriptive and clearly tells the user that there is a problem in his/her angular.json file and the line where the error is found.

Also, I just noticed https://github.com/angular/angular-cli/issues/5610 which seems to have been addressed, which suggests comment should be allowed and the CLI should not complain about / characters in angular.json. Or am I missing something?

This is very frustrating...
Invalid JSON character: "" at 0:0.
Error: Invalid JSON character: "" at 0:0.

Ummm... which file?
If I look at the error through a different text output, it reads:
Invalid JSON character: "ï»¿" at 0:0.
Error: Invalid JSON character: "ï»¿" at 0:0.

So I'm gonna have to open all the json files with a hex editor I guess... :/


GitHub branches fork (other than the main branch)

Supposing you forked a GitHub repository and only forked the main branch.
The original repository now has new branches that you want to fork in your repository.
However, even if you synchronise your fork, you don't get the new branches.
How to get these new branches into your repository?

Try this:
git remote add upstream <url-of-the-upstream-repo>
git fetch --all
git push <my-fork> <branch-name>


Specifying a default graph URI

Hi, may be i'm not clearly understand Sparql Client, but i didn't find way how to define graph repository
For example, i follow to http://dbpedia.org/sparql and define Default Graph URI http://tema.livejournal.com/data/foaf.rdf and Query text Select select ?name WHERE { ?x foaf:knows ?name} (all this i do in web browser)
BUT!!! where i should define Default Graph URI in Sparql Client library?
Thanks

According to the SPARQL 1.1 Protocol the default graph can be specified using the query parameter default-graph-uri for queries and using-graph-uri for update requests.
In which file does this need to be implemented?

From the code in
https://github.com/ruby-rdf/sparql-client/blob/28370e092a4fcfc17badeed72c2ae78c1cb6a470/lib/sparql/client.rb#L726-L731
it seems that you could simply pass this in as part of the endpoint URL:
client = SPARQL::Client.new('http://dbpedia.org/sp^Cql?default-graph-uri=http%3A%2F%2Ftema.livejournal.com%2Fdata%2Ffoaf.rdf')

Which would work as long as you're using GET.
It would be reasonable to add something like default_graph_uri (and named_graph_uri) as instantiation options to SPARQL::Client#initialize. Code is also needed in make_post_request as described in Long SELECT query using direct POST.
Also, please add some examples for using this into the method preamble, and of course, tests, probably along the lines of the following:
https://github.com/ruby-rdf/sparql-client/blob/204355edf946f7baeb1c36f9aac83c5a53d3724e/spec/client_spec.rb#L124-L130


Convert existing views to Class Based Views

From the docs:

All views have been converted to class-based views. This is a big departure from the traditional function-based, but has the benefit of being much more flexible.

This actually hasn't been done yet, and it would indeed make the app more flexible.

@jacobwegner Is someone working on this?

@ossanna16: Not yet!

@jacobwegner Do you think it would be a good "easy" issue for someone new to open source to try?


Understanding zuul.ignoredHeaders and zuul.ignoreSecurityHeaders

Here's the document from http://cloud.spring.io/spring-cloud-static/Camden.SR2/#_ignored_headers
Ignored Headers
In addition to the per-route sensitive headers, you can set a global value for zuul.ignoredHeaders for values that should be discarded (both request and response) during interactions with downstream services. By default these are empty, if Spring Security is not on the classpath, and otherwise they are initialized to a set of well-known "security" headers (e.g. involving caching) as specified by Spring Security. The assumption in this case is that the downstream services might add these headers too, and we want the values from the proxy. To not discard these well known security headers in case Spring Security is on the classpath you can set zuul.ignoreSecurityHeaders to false. This can be useful if you disabled the HTTP Security response headers in Spring Security and want the values provided by downstream services
I can not understand these words. Here is my understanding, please tell me whether am I rignt.

zuul.ignoredHeaders  attribute can be used to discard some headers. For example,

Setting zuul.ignoredHeaders = Header1, Header2 , then Header1 and Header2 will not be propagate to any other services.
By default, it is empty, but if Spring Security is in classpath, it's values is
Pragma,Cache-Control,X-Frame-Options,X-Content-Type-Options,X-XSS-Protection,Expires

The default  value of zuul.ignoreSecurityHeaders  is true, but when I want the value of the headers' values from downstream services we need to set to false.
Am I right?
B.T.W, is there any example for using these two attrbutes?

I believe you are correct.  Maybe @spencergibb can confirm.
I agree the wording in the documentation is confusing.  We should at the very least clarify it.

@ryanjbaxter  @spencergibb
Thank you very much.

@ryanjbaxter
B.T.W, what's the downstream service?
I think it means the intenal microserivces behind the zuul proxy?Am I right?

@eacdy yes the service behind the proxy

@here some times zuul is catching data and due to that not getting latest , is there any solution for this?

@sachinmagar it is not clear what the problem is or if it is even related to this issue.  If you are having an issue using Zuul please open an issue and provide as much information as possible describing your problem.

This module has entered maintenance mode. This means that the Spring Cloud team will no longer be adding new features to the module. We will fix blocker bugs and security issues, and we will also consider and review small pull requests from the community.


Loading final checkpoint fails due to CUDA out of memory

Hello,
I managed to train the speech_recognition ASR template.
The batch size is 8
and the number of epochs is 15.
The training session completed,
however, when loading the final checkpoint, I get
and exception of "CUDA out of memory"
Do I have to retrain again with a smaller batch size?
Is my trained model broken and unusable?
What other options are available if I want to avoid re-training?

Hi, solely loading shouldn't cause CUDA OOM. Could you please post a screen shot of the full error + logs before it happens ?

../data/rirs_noises.zip exists. Skipping download
speechbrain.core - Beginning experiment!
speechbrain.core - Experiment folder: results/CRDNN_BPE_960h_LM/2602
mini_librispeech_prepare - Preparation completed in previous run, skipping.
speechbrain.pretrained.fetching - Fetch lm.ckpt: Using existing file/symlink in results/CRDNN_BPE_960h_LM/2602/save/lm.ckpt.
speechbrain.pretrained.fetching - Fetch tokenizer.ckpt: Using existing file/symlink in results/CRDNN_BPE_960h_LM/2602/save/tokenizer.ckpt.
speechbrain.pretrained.fetching - Fetch asr.ckpt: Using existing file/symlink in results/CRDNN_BPE_960h_LM/2602/save/model.ckpt.
speechbrain.utils.parameter_transfer - Loading pretrained files for: lm, tokenizer, model
speechbrain.core - Info: ckpt_interval_minutes arg from hparam file is used
speechbrain.core - 173.0M trainable parameters in ASR
speechbrain.utils.checkpoints - Loading a checkpoint from results/CRDNN_BPE_960h_LM/2602/save/CKPT+2021-08-04+20-34-40+00
speechbrain.utils.checkpoints - Loading a checkpoint from results/CRDNN_BPE_960h_LM/2602/save/CKPT+2021-08-04+16-37-53+00
speechbrain.core - Exception:
Traceback (most recent call last):
File "train.py", line 471, in
test_stats = asr_brain.evaluate(
File<EMAIL_ADDRESS>line 1185, in evaluate
loss = self.evaluate_batch(batch, stage=Stage.TEST)
File<EMAIL_ADDRESS>line 928, in evaluate_batch
out = self.compute_forward(batch, stage=stage)
File "train.py", line 106, in compute_forward
predictions["tokens"], _ = self.hparams.test_search(
File<EMAIL_ADDRESS>line 889, in _call_impl
result = self.forward(*input, **kwargs)
File<EMAIL_ADDRESS>line 628, in forward
log_probs, memory, attn = self.forward_step(
File<EMAIL_ADDRESS>line 943, in forward_step
dec_out, hs, c, w = self.dec.forward_step(
File<EMAIL_ADDRESS>line 908, in forward_step
c, w = self.attn(enc_states, enc_len, cell_out)
File<EMAIL_ADDRESS>line 889, in _call_impl
result = self.forward(*input, **kwargs)
File<EMAIL_ADDRESS>line 219, in forward
torch.tanh(self.precomputed_enc_h + dec_h + attn_conv)
RuntimeError: CUDA out of memory. Tried to allocate 1.93 GiB (GPU 0; 11.17 GiB total capacity; 6.44 GiB already allocated; 1.71 GiB free; 8.97 GiB reserved in total by PyTorch)

Ok, so, the test_search is usually more intensive in terms of computation and memory because we do the full beam search with a big beam (have a look at the yaml file). You can either 1. reduce the test batch size; 2. if it's already set to 1 then reduce slightly the beam size.

@TParcollet
Thank you,
decreasing the test batch size solved the problem.
Is there a way to quantify the impact of batch size over accuracy results?
For lower batch sizes will I get higher WERs?

The test time batch size doesn't affect the WER (or if it does, something is going wrong).
Beam size will affect WER, and empirically you may even find an (at least locally) optimal beam. That is, a wider beam doesn't always result in better WER.

Thank you @Gastron ,
Is there a convenient way (rule of thumb?) of calculating or estimating the optimal beam size?
Or is it subject to change based on the nature of the test / training data?

Unfortunately it just depends on the particular experiment. You should just try a couple of values on the validation data and see what works. Still, the rule of thumb is more in the direction that wider beam gives better WER - but I've personally found that the very wide beam used in the SpeechBrain LibriSpeech recipes doesn't always work well on other data.
Closing this issue it seems to have been solved.

With conformers, the way the batching is done in test time might make some difference (although hopefully not very much), because the conv_module has conv1d's with quite large receptive fields (31 frames, which comes to over a second if you have done 4x subsampling), and at the edge of the batch it would be seeing zeros, while if it's in a padding segment it would see whatever the output of the previous layer was, which might not be zero.
That is: if there are more frames of padding than the model has seen in train time, it might potentially cause a mismatch.


[DRAFT] test thrift 0.19.0

Codecov Report

Merging #11183 (46538f8) into master (d654e03) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #11183      +/-   ##
============================================
+ Coverage     47.52%   47.54%   +0.02%
- Complexity    23621    23632      +11
============================================
  Files          2789     2789
  Lines        170475   170475
  Branches      20302    20302
============================================
+ Hits          81011    81049      +38
+ Misses        89464    89426      -38

see 13 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Unable to vendor crates

Using current commit 6d540100aeb3fa237cb23b2f1168d82f9d8e1930:
❯ cargo vendor
error: failed to sync

Caused by:
  found duplicate version of package `bellman_ce v0.3.2` vendored from two sources:

  	source 1: https://github.com/matter-labs/bellman?branch=dev#5520aa22
  	source 2: https://github.com/matter-labs/bellman?branch=snark-wrapper#e01e5fa0

Also cargo tree -d finds a lot of duplicate dependencies.

describe your problem in more detail

The whole dependency chain is a mess... packages are referred to with the same version and name but with a different git branch.
Ideally you could cargo vendor the repo to get a reproducible set of dependencies for a reproducible build.

Recommendation: change the package name to ${orig}-${branchname} in the repos in question, or change the version.
source = "git+https://github.com/matter-labs/franklin-crypto?branch=dev#5695d07c7bc604c2c39a27712ffac171d39ee1ed"
source = "git+https://github.com/matter-labs/franklin-crypto?branch=snark_wrapper#2546c63b91b59bdb0ad342d26f03fb57477550b2"

source = "git+https://github.com/matter-labs/bellman?branch=dev#5520aa2274afe73d281373c92b007a2ecdebfbea"
source = "git+https://github.com/matter-labs/bellman?branch=snark-wrapper#e01e5fa08a97a113e76ec8a69d06fe6cc2c82d17"

I'll file a bug against these repos.

Found a PR, which enables cargo vendor --no-merge-sources , which solves the problem... https://github.com/rust-lang/cargo/pull/13271

Hello! We are planning to get rid of git dependencies and publish everything on crates.io.
With that, the problem should be resolved.

Hello! We are planning to get rid of git dependencies and publish everything on crates.io. With that, the problem should be resolved.

That would be the best solution ever :partying_face:

The background here is that when the crypto team uses 1.4.1, that means protocol version 1.4.1. So pretty much as far from semver as you can be. The crates.io will at least force publishing different versions but we might consider enforcing semver.

We will use a separate, semver-compliant versioning scheme for crates (e.g. let's say that 1.4.1 represents a "protocol" version, but all the crates will live in 0.x.y domain).

Up to @haraldh


Projects missing a Members entry for their HOST

In the way we're creating projects today, we're forgetting to insert an entry for their Host in the Members table.
That can be a consistency problem.
Quick fix:
INSERT INTO "Members" (
  "createdAt",
  "updatedAt",
  "CreatedByUserId",
  "CollectiveId",
  "role",
  "MemberCollectiveId",
  "since"
)
SELECT
NOW() as "createdAt",
NOW() as "updatedAt",
12457 as "CreatedByUserId",
"Collectives"."id" as "CollectiveId",
'HOST' as "role",
"Collectives"."HostCollectiveId" as "MemberCollectiveId",
"approvedAt" as "since"
FROM "Collectives"
WHERE "HostCollectiveId" IS NOT NULL
AND "approvedAt" IS NOT NULL
AND "isActive" IS TRUE
AND "deletedAt" IS NULL
AND "HostCollectiveId" != "Collectives"."id"
AND NOT EXISTS (
  SELECT * FROM "Members"
  WHERE "CollectiveId" = "Collectives"."id"
  AND "role" = 'HOST'
  AND "deletedAt" IS NULL
)

The merged PR doesn't seem to fix the issue in production. Re-trigggered the "quick fix" query in the meantime.

This should be fixed in https://github.com/opencollective/opencollective-api/pull/6961

Can still happen, fixed 12 entries today.

Can still happen, fixed 11 entries today.

41 today. This should be moved to a model check/fix.