v0.10.0 finds errors in react-native example

I don't know if this is a bug or expected, but I couldn't find anything in the documentation. This is related to this issue on react-native.
To reproduce this issue:

create a new react-native project using react-native init testflow
cd testflow
flow --json

Output:
flow is outdated, going to launch a new one.
The flow server will be ready in a moment
Flow server launched for /Users/djpfahler/localhost/testflow
Spawned flow server (child pid=12001)
Logs will go to /var/folders/mh/lh59l19x601d96hkhgxk406h0000gn/T/flow/zSUserszSdjpfahlerzSlocalhostzStestflow.log
The flow server will be ready in a moment.
{"passed":false,"errors":[{"message":[{"descr":"undefined\nThis type is incompatible with","code":0,"path":"/Users/djpfahler/localhost/testflow/node_modules/react-native/Libraries/Components/TabBarIOS/TabBarItemIOS.ios.js","line":70,"endline":70,"start":15,"end":34},{"descr":"boolean","code":0,"path":"/Users/djpfahler/localhost/testflow/node_modules/react-native/Libraries/Components/TabBarIOS/TabBarItemIOS.ios.js","line":91,"endline":91,"start":62,"end":68}]},{"message":[{"descr":"undefined\nThis type is incompatible with","code":0,"path":"/Users/djpfahler/localhost/testflow/node_modules/react-native/Libraries/ReactIOS/ReactIOSMount.js","line":197,"endline":197,"start":7,"end":13},{"descr":"boolean","code":0,"path":"/Users/djpfahler/localhost/testflow/node_modules/react-native/Libraries/ReactIOS/ReactIOSMount.js","line":194,"endline":194,"start":59,"end":62}]}],"version":" Apr 24 2015 18:33:55"}

This was not the case with<EMAIL_ADDRESS>as can be reproduced by downloading the 0.9.2 executable and then run ~/Downloads/flow-0.9.2/flow --json in the testflow directory which will output:
flow is outdated, going to launch a new one.
The flow server will be ready in a moment
Flow server launched for /Users/djpfahler/localhost/testflow
Spawned flow server (child pid=12240)
Logs will go to /var/folders/mh/lh59l19x601d96hkhgxk406h0000gn/T/flow/zSUserszSdjpfahlerzSlocalhostzStestflow.log
The flow server will be ready in a moment.
{"passed":true,"errors":[],"version":" Apr 17 2015 16:01:38"}an take some time. Retrying... -

Please make sure you intentionally "broke" this. I would also appreciate if you could comment on https://github.com/facebook/react-native/issues/938.
Thank you a lot. :)

Hey @davidpfahler, thanks a lot of the report. I'll make sure to comment on https://github.com/facebook/react-native/issues/938 as well.
Almost every release of Flow includes some breaking changes, as we make Flow smarter and it notices things it missed before. It seems that the version of react-native on GitHub still expects to work with Flow v0.9.2. Flow v0.10.0 complains about a few more things, and react-native needs a few small updates in order to work without any Flow errors. We've made those changes, but it may take a few days for those changes to make it to GitHub. In the meantime, Flow v0.9.2 should still work.
What we clearly have missed here is a good way of tying Flow's versions to those expected by react-native. If Flow was just a npm package, we would specify the specific Flow version expected in the package.json and be done with it. We're talking about this internally to figure out the best thing to do moving forward. Sorry for the trouble and thanks for the report!

react-native 0.4.1 has been pushed to npm, and it includes the fixes for Flow v0.10.0. This week we'll invest in making sure this is much smoother for future deploys. Thanks again for reporting!


What does joint_camlidar and joint_camlidar_equivalent_extr do ?

Hello,
thanks for the great project and for releasing the code!
I have a doubt regarding two flags (for the waymo dataset, used also in StreetSurf).
What does joint_camlidar and joint_camlidar_equivalent_extr do ?
In waymo_dataset.py, I see the description:
joint_camlidar=True, # Joint all cameras and lidars; Set this to [true] if you needs to calibrate cam/lidar extrjoint_camlidar_equivalent_extr=True, # Set to [false] if you needs to calibrate cam/lidar extr
Does that mean that if we set joint_camlidar to true, it will refine the extrinsics of the cameras and for the lidars?
That would be great, but not sure where this would happen, as the only difference that I see when setting the joint_camlidar=True or False is how you create the new_odict ( transform=c2v @ dpose versus transform=c2w )
Thanks for your time and availability!

It's not about pose refinement. When joint_camlidar_equivalent_extr is turned on, it means we acknowledge that c2w when the image is captured is different from the c2w derived from the ego vehicle's pose, due to the non-zero ego-motion and the lag between pose timestamp and image timestamp. Therefore, it takes into consideration the ego motion and uses the ego-compensated pose, i.e. c2v @ dpose, to construct the dataset. Otherwise, we ignore this and simply use the c2w when the image is captured.

Thanks for the answer @Geniussh !
Could you please point me on where in the code the poses are being wrapped as pytorch variables?

It's in the same learnable_params.py I linked above. And you can track it down to the lowest level which leads to nr3d_lib.models.attributes. You can find how the forward pass is defined for the custom TransformRT class, which is what I meant by "wrap" but it's certainly an oversimplified saying.
In my past projects I just initialized a torch variable with length 6, and with requires_grad=True. 3 for translation and 3 for rotation. I did not impose any constraint on the rotation during training and it also worked. IMHO I guess the author defined such custom libraries here mainly in order to impose SO3 constrain during training.  Maybe I'm wrong.

Thanks for the prompt reply and availability @Geniussh . Really appreciated!
I guess I can close this issue


Add wwwroot to .gitignore

Fixes #9184

This version of .gitignore had wwwroot

@danroth27 	can we merge this? We've had wwwroot listed in .gitignore in the past. If it's turns out to be a problem we'll update it.

@Rick-Anderson No, I wouldn't. This .gitignore file change seems way too aggressive. I'm sure we will have some sample apps that need content wwwroot or will want to include JavaScript or CSS files. I would instead focus our efforts on simplifying the existing samples

Last pitch. It seems aggressive but it's actually not. We've had wwwroot in for extended periods with no problems. We've almost always had wwwroot/lib/ in .gitignore.
For the 95% of samples that don't need wwwroot, it's alot of work to remove  wwwroot and it's typically not done. It's impossible to police keeping out unneeded wwwroot assets without .gitignore.
For the 5% that need wwwroot , it's trivial to add.
Should we go back to what we had for most of our history? see #4532
**/sample/**/wwwroot/lib/
**/samples/**/wwwroot/lib/

I'm OK with wwwroot/lib/ - that's 97% of wwwroot.
Keep in mind samples that currently contain wwwroot assets don't loose them when wwwroot is added back and the project is updated.

Should we go back to what we had for most of our history? see #4532

I think that's certainly better than excluding all static files support from every sample.


Update guardian_db to be compatible with guardian base

Right now guardian is at 0.10.1 while guardian_db works only with 0.9.0

Thanks. Updated.

FYI: https://hex.pm/packages/guardian_db
Is still saying Guardian 0.9 is what has to be installed to use Guardian_DB

@hassox  I don't see that updated. 0.4.0 still requires Guardian 0.9 as @combsco says.
Am I doing something wrong?


Wip lc gtlt

shoot, i didn't realize this was mis-targeted at the ceph-master branch. i pushed these commits onto master instead, and force-pushed ceph-master to remove them


Enable versioning for jenkins shared libraries

Problem:
Since the jenkinsFile for individual client or any component repositories will be living in the respective GitHub repository, the share jenkins library stored in opensearch-build repo will act as upstream library for all actions.
If not versioned, a breaking change can be introduced causing the non-exercised release workflows in remote repositories to break.
Acceptance Criteria:

Versioned jenkins shared libraries are published somewhere
All jenkins files use a particular version of shared library and not just point to main or current commit

Researched a bit on pulling different versions for remote jenkins shared libraries, and found this plugin https://plugins.jenkins.io/workflow-cps-global-lib-http/
Discussed with @prudhvigodithi, we can publish the zips to maven or any other artifact repository (s3 works as well) for different versions and accordingly pull the required version.
Few things to keep in mind:

Jenkins Shared Library versions will be independent of opensearch and other product's version. (should we separate this out into new repository?)
Version release notes, change log needs to be maintained to let the consumer of this library know what changed and if the next version can cause any breaking changes.
Versioning should follow semver

+1 on separating the library into a new repository and automating away builds/releases

This is even possible with git, separating out is good, if so then we should consider separating out the python workflows as well.
What is propose is?
Upon every merge to main or when we call for a date of release (Not linked to the Os and OSD release).
git clone -b main https://github.com/opensearch-project/opensearch-build.git --depth 1
git archive --format zip --output buildsetup-1.0.0.zip HEAD src/ vars/ tests/
(Optional with tag: git archive --format zip --output buildsetup-1.0.0.zip v1.2.1 vars/ tests/, where v1.2.1 is the gittag)
This will archive the shared libraries and and python code base workflows to a zip file.
Publish this zip file to maven org.opensearch.buildcode.buildsetup.1.0.0 or to any other location where it can be later imported and extracted to get the shared libraries and and python code to execute the builds.
Now for every Jenkinsfile (distribution builds or client releases), the 1st stage would always be getting the zip and extracting it, so that the code base is now on local workspace which facilities the next stages of execution.
With this versioning is enabled for the main code (shared libraries and and python code), which eliminates the branching if organized properly what can go into the zip file.
Jenkinsfiles are like constants will always be on the main branch, but its shared libraries are versioned.

I think we are complicating this by mixing 2 things, python build code and jenkins shared library.
I agree they are connected today in someway but that won't be the case in the long run. For example, python code receives frequent contributions which might need frequent releases compared to jenkins shared libraries which won't be that often.
This version history might be confusing for external consumers who do not care about python build code.
WDYT @bbarani ?

All I'm thinking is versioning the entire code and inside that code we can choose if we want shared libraries or normal workflows, I dont deny its easy to separate the shared libraries and version it, but its always a hanging piece to maintain, from rest of the code base, with this I'm open for thoughts.
@dblock @bbarani

I would recommend moving forward with separate repo for Jenkins shared library for now to unblock the progress and piggy back for other items soon after that as required. @prudhvigodithi @gaiksaya

Sounds good. In that case I don't think we need to publish at all as git tags can be used by remote repositories to use a particular version of shared libraries using dynamic retrieval method https://www.jenkins.io/doc/book/pipeline/shared-libraries/#dynamic-retrieval
As for python code we have issue #2490 opened to figure out that problem.
Thanks!

Listing down the files/folders that need to be moved to new repository:

https://github.com/opensearch-project/opensearch-build/vars
https://github.com/opensearch-project/opensearch-build/tree/main/src/jenkins
https://github.com/opensearch-project/opensearch-build/tree/main/tests/jenkins
https://github.com/opensearch-project/opensearch-build/tree/main/publish

We need to make sure that the code inside this libraries are not cloning main branches of build repo else versioning is irrelevant.
Also we have been confusing these jenkins shared libraries as methods which is not right. Libraries should be generic.
All other code bases should go in the python code base and not groovy.
Once the libraries are moved, the current jenkins file will use this libraries as below:
library identifier: 'jenkins@<tag>', retriever: modernSCM(
  [$class: 'GitSCMSource',
    remote: 'https://github.com/opensearch-project/opensearch-build-shared-libraries.git'])

Resources to move this libraries while preserving the history: https://medium.com/@ayushya/move-directory-from-one-repository-to-another-preserving-git-history-d210fa049d4b
cc: @opensearch-project/engineering-effectiveness

[Offline discussion] Apart from shared libraries, we would still need branching or code separation for build side of things. Example: OpenSearch makes some huge change in upcoming 3.x that changes the rpm spec file or tar-install.sh script. In that case the changes to build code should be only concerned to 3.x and not 2.x. We do not have a mechanism today to separate that out.
As for shared libraries, few jenkins libraries like rpmDistValidation needs to move out of the groovy code and converted to python code to make it easier for the any user to even run it locally which is not possible today.
Only CI related libraries need to be part of jenkins code base as stated before.
We will be moving ahead with separating the code base into new GitHub repo called opensearch-build-libraries.
Need to define the release process for the repo.
Some good examples that already exist out there: https://github.com/cloudogu/ces-build-lib
https://github.com/cloudogu/gitops-build-lib

Jenkins shared libraries are now separate into new repository: https://github.com/opensearch-project/opensearch-build-libraries
We will be using the above repo for adding new libraries. How we manage release cycle for this new repo can be discussed in the issue opened here: https://github.com/opensearch-project/opensearch-build-libraries/issues/5
Also the current jenkins files in this repo and other repos need to consume the new repo as remote library which can be tracked here https://github.com/opensearch-project/opensearch-build/issues/2646
Closing this issue!
Thanks!


Sap-Pancake balance

About the PR
Change the conversion rate from 12:1 to 2:1
Why / Balance
12 to 1 syrup is quite abysmal for a conversion rate for a flavor-only liquid, also you don't want to kill-murder a Diona for your pancake.

As the person who originally added this, you'll need to make sure the amount of water vapor produced is still consistent with how much is boiled away (just stick a distiller in a closed room, make syrup in that room, and see how much water it produces), and also change the reagents you get from centrifuging sap to line up as well.

i would be happy to do it, but i am no where near familiar with the files and it was already a nightmare to find these 2 silly lines to change

Your changelog section isn't formatted correctly and should be more direct, please check the PR guidelines again. It should be something like:
Changelog
:cl:

tweak: Making syrup requires less sap now.

As the person who originally added this, you'll need to make sure the amount of water vapor produced is still consistent with how much is boiled away (just stick a distiller in a closed room, make syrup in that room, and see how much water it produces), and also change the reagents you get from centrifuging sap to line up as well.

Realistically this would mean making the ratio of water and sugar about 6:4, and making less water vapor come out, correct?


Set column(s) in CSV file as primary key for table create

No. Unnecessary. User should define table, then load CSV into it.


kwazthis should handle destination dirty buffers.  [phab:T52]

Created by amshali at 2015-07-10 22:57:53:
Given a local_repo, it is ideal for us in editor land if kwasthis actually computed the right offset when the destination file(which defines the element) is dirty.
Thank you!

schroederc wrote at 2015-07-10 23:00:02:
As we've discussed offline, the xrefs service doesn't currently support dirty destinations.  We would have to either add a new API call or do the mapping locally.
fromberger, do you have any strong opinions on this front?

fromberger wrote at 2015-07-13 13:41:47:
I think it makes sense to have the cross-reference service support this—at least for file decorations, as it'd be a lot of fairly subtle position-munging logic for the client to carry around.
We should think about how we might want to do it, though; e.g., does the client send the whole buffer, or just the diff itself?

amshali wrote at 2015-07-13 18:04:55:
This would be awesome for us.
Cody added this local_repo flag which can be used to look up any dirty buffer by a path.

schroederc wrote at 2015-07-21 17:22:33:
fromberger, do you agree that this should be a new API method then?

fromberger wrote at 2015-07-21 17:42:02:

! In (earlier comment), schroederc wrote:
fromberger, do you agree that this should be a new API method then?

Do you think it should be a separate method? I don't object, but it seems like it's an aspect rather than a separate method.

schroederc wrote at 2015-07-21 18:16:22:
amshali, how would like a service interface to look for this kind of use case?
Do you want to send the server all of the known dirty files in the FileDecorationsRequest?  Or do you want to make an additional call to patch anchors given a dirty buffer (as a new API call)?  Or something else?

amshali wrote at 2015-07-21 18:18:15:

! In (earlier comment), schroederc wrote:
amshali, how would like a service interface to look for this kind of use case?

Do you want to send the server all of the known dirty files in the FileDecorationsRequest?  Or do you want to make an additional call to patch anchors given a dirty buffer (as a new API call)?  Or something else?

No I don't want to do that. I just wish to send the dirty destination one.

fromberger wrote at 2015-07-21 18:21:22:

! In (earlier comment), amshali wrote:
No I don't want to do that. I just wish to send the dirty destination one.

I think there are two related issues: One is how to patch references in a dirty file, so that they move.
The other is how to patch the targets of references, that point to other dirty buffers in your client view.
Right now, the latter is something we do not support at all (even in the internal implementation). But if you're expecting a client view to link to other modified files in the same view, that's something we'd need to figure out. In the limit, the client view could have a lot of modified files. I expect in general it'll be a fairly small number, though.

amshali wrote at 2015-07-21 18:32:45:

! In (earlier comment), fromberger wrote:

! In (earlier comment), amshali wrote:
No I don't want to do that. I just wish to send the dirty destination one.

I think there are two related issues: One is how to patch references in a dirty file, so that they move.
The other is how to patch the targets of references, that point to other dirty buffers in your client view.
Right now, the latter is something we do not support at all (even in the internal implementation). But if you're expecting a client view to link to other modified files in the same view, that's something we'd need to figure out. In the limit, the client view could have a lot of modified files. I expect in general it'll be a fairly small number, though.

I don't want to send all the dirty buffers to the server. But if we assume that there are just a few of those, maybe it's ok to give them to server. On the other hand when we are doing jump to definition, there is only one target file. If that is dirty I want the grok service to be able to remap the locations in that file given a dirty buffer which I can provide.

fromberger wrote at 2015-07-21 20:36:34:

! In (earlier comment), amshali wrote:

! In (earlier comment), fromberger wrote:
I think there are two related issues: One is how to patch references in a dirty file, so that they move.
The other is how to patch the targets of references, that point to other dirty buffers in your client view.

Right now, the latter is something we do not support at all (even in the internal implementation). But if you're expecting a client view to link to other modified files in the same view, that's something we'd need to figure out. In the limit, the client view could have a lot of modified files. I expect in general it'll be a fairly small number, though.

I don't want to send all the dirty buffers to the server. But if we assume that there are just a few of those, maybe it's ok to give them to server. On the other hand when we are doing jump to definition, there is only one target file. If that is dirty I want the grok service to be able to remap the locations in that file given a dirty buffer which I can provide.

I think there's an open question of what to send to the server. The contents of dirty files is certainly the easiest to understand, though it could potentially be a lot of data. Compression could help if we went that route. But it could also just be a unified diff, for example—which would be smaller and easier for the client to cache.  I'm not sure what makes most sense.
If possible, I'd like to avoid the server having to keep state. It could maybe make sense to have per-session state, but I'd prefer we explore the stateless options first.

schroederc wrote at 2015-07-21 20:37:28:
I definitely don't want to have state either.

schroederc wrote at 2015-07-30 18:03:56:
amshali, what do you think of a proxy server that exposes the same API surface as the http_server and transparently does patching on all API calls (including Nodes) based on a given local repository?  This would allow for destination patching as well as better performance since the proxy can attempt some caching.

amshali wrote at 2015-07-30 18:11:00:
I would love that. Is it possible to run multiple proxies? Because we might have multiple project(repos) open.

schroederc wrote at 2015-07-30 18:14:52:
What's your current setup for multiple projects?  Do you have multiple servers running?
Running the proxy shouldn't be any different than running the http_server other than the additional flag(s) to determine where to find the dirty files.

amshali wrote at 2015-07-30 18:19:36:
Can proxy accept multiple
local_repos
```?
Our current setup relies on calling the command line tool with a new local_repo everytime. I suppose I will do the same thing with proxy and it will just figure out the rest?

schroederc wrote at 2015-07-30 18:21:54:
Is there 1 http_server running per project?

amshali wrote at 2015-07-30 21:08:38:
No. 1 for all


Adds procps package to container

The procps package is responsible for the ps command on linux. Without ps we can't start the Spark history-server, unfortunately.
Signed-off-by: DylanGuedes<EMAIL_ADDRESS>

He didn't add procps in his MR, how is this a duplicate?

Without ps we can't start the Spark history-server

I was pointing out the fact that the history server implementation is being worked on


train the multi-modality-perceiver

How to train the model? I tried but it could not converge. Thank you very much if you can tell the concrete details.

Hi, @HuiqinWu
I'm trying to build my own perceiver based on this repo. But I couldn't find where to start the training process. Could you help me with it?
Many thanks!

Were you guys able to figure it out? I am also facing same problem


updated README and spec.json

updated README and specs.json file. Now resolvable url should be available for companion guide.

@TallTed, while we haven't had the chance to meet in person, I've noticed your ongoing efforts in refining the specifications by addressing grammar and other aspects. Would you be interested in being added to the list of contributors? We'd welcome you on the list if you're open to being included.

open for review

[@andorsk] Would you be interested in being added to the list of contributors? We'd welcome you on the list if you're open to being included.

Sure. You can add me anywhere that's a recognition of work I've already done.


✨ [Feature] Support for Automatic dependency Injection

Before feedback (在反馈之前)

[X] There are no features that I want to submit in Issues (当前 Issues 中没有我将要提交的新功能)

Describe Feature (描述功能)
Any plans to support automatic dependency injection much like uber/fx using reflection?

We don't have a plan to support the feature for now, but I think it's easy to use it in the framework directly.

Close, please reopen if necessary.


Add a clangd.fallbackCommand or extend compile_commands.json to support more complex scenarios.

If clangd can't find a compilation command in the compilation database, as is common for, say, header files included in your source files, it will fallback to calling the clang executable along with any flags you define in clangd.fallbackFlags. However, this does not support cross compilation scenarios. I need to be able to specify the clangd.fallbackCommand.
I think another, better, way to address this would be to modify compile_commands.json to allow "fallback" commands that work for all files and other commands can extend. This would actually simplify compile_commands.json as well, such that you don't have to repeat arguments to each file you would like to compile.
Example:
Before:
[
  {
    "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.3\\out\\src",
    "arguments": [
      "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++.exe",
      "-mcpu=cortex-m0plus",
      "-mthumb",
      "-c",
      "-g",
      "-Os",
      "-w",
      "-std=gnu++11",
      "-ffunction-sections",
      "-fdata-sections",
      "-fno-threadsafe-statics",
      "-nostdlib",
      "--param",
      "max-inline-insns-single=500",
      "-fno-rtti",
      "-fno-exceptions",
      "-MMD",
      "-DF_CPU=48000000L",
      "-DARDUINO=10607",
      "-DARDUINO_SAMD_MKRWIFI1010",
      "-DARDUINO_ARCH_SAMD",
      "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
      "-D__SAMD21G18A__",
      "-DUSB_VID=0x2341",
      "-DUSB_PID=0x8054",
      "-DUSBCON",
      "-DUSB_MANUFACTURER=\"Arduino LLC\"",
      "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
      "-DUSE_BQ24195L_PMIC",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0/CMSIS/Include/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0/CMSIS/Device/ATMEL/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated-avr-comp",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
      "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp",
      "-o",
      "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp.o"
    ],
    "file": "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp"
  },
  {
    "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.3\\out\\src",
    "arguments": [
      "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++.exe",
      "-mcpu=cortex-m0plus",
      "-mthumb",
      "-c",
      "-g",
      "-Os",
      "-w",
      "-std=gnu++11",
      "-ffunction-sections",
      "-fdata-sections",
      "-fno-threadsafe-statics",
      "-nostdlib",
      "--param",
      "max-inline-insns-single=500",
      "-fno-rtti",
      "-fno-exceptions",
      "-MMD",
      "-DF_CPU=48000000L",
      "-DARDUINO=10607",
      "-DARDUINO_SAMD_MKRWIFI1010",
      "-DARDUINO_ARCH_SAMD",
      "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
      "-D__SAMD21G18A__",
      "-DUSB_VID=0x2341",
      "-DUSB_PID=0x8054",
      "-DUSBCON",
      "-DUSB_MANUFACTURER=\"Arduino LLC\"",
      "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
      "-DUSE_BQ24195L_PMIC",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0/CMSIS/Include/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0/CMSIS/Device/ATMEL/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated-avr-comp",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
      "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp",
      "-o",
      "c:\\source\\repos\\clangd-arduino\\build\\core\\variant.cpp.o"
    ],
    "file": "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp"
  }
]

After:
[
  {
    "name": "fallback",
    "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.3\\out\\src",
    "command": "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++.exe",
    "arguments": [
      "-mcpu=cortex-m0plus",
      "-mthumb",
      "-g",
      "-Os",
      "-w",
      "-std=gnu++11",
      "-ffunction-sections",
      "-fdata-sections",
      "-fno-threadsafe-statics",
      "-nostdlib",
      "--param",
      "max-inline-insns-single=500",
      "-fno-rtti",
      "-fno-exceptions",
      "-MMD",
      "-DF_CPU=48000000L",
      "-DARDUINO=10607",
      "-DARDUINO_SAMD_MKRWIFI1010",
      "-DARDUINO_ARCH_SAMD",
      "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
      "-D__SAMD21G18A__",
      "-DUSB_VID=0x2341",
      "-DUSB_PID=0x8054",
      "-DUSBCON",
      "-DUSB_MANUFACTURER=\"Arduino LLC\"",
      "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
      "-DUSE_BQ24195L_PMIC",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0/CMSIS/Include/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0/CMSIS/Device/ATMEL/",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated-avr-comp",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
      "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010"
    ]
  },
  {
    "name": "clangd_arduino",
    "command": "${fallback.command}",
    "arguments": [
      "${fallback.arguments}",
      "-c",
      "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp",
      "-o",
      "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp.o"
    ],
    "file": "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp"
  },
  {
    "name": "variant",
    "command": "${fallback.command}",
    "arguments": [
      "${fallback.arguments}",
      "-c",
      "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp",
      "-o",
      "c:\\source\\repos\\clangd-arduino\\build\\core\\variant.cpp.o"
    ],
    "file": "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp"
  }
]

Granted this example was contrived rather quickly.
Logs
I[18:58:14.272] Failed to find compilation database for c:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\Arduino.h
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\Llvm\bin\clang -xobjective-c++-header c:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\Arduino.h -fsyntax-only -resource-dir=c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0

System information
PS C:\source\repos\clangd-arduino> clangd --version
clangd version 11.0.0
Editor/LSP plugin: VS Code
Operating system: Windows 10

Just saw this thread.
I think adding the --compile-commands-dir= partially solves the issue, however, when analyzing a cpp file that includes a header for a compilation unit not in your compilation database, clangd seems to still try to compile it with clang and fallback arguments as the errors that are produced seem consistent with that.

i believe the piece missing here is the command line generated by clangd is not executed as-is it is used to create a clang compiler instance. hence the need for deleting --driver-mode or --resource-dir when invoking the command as-is, because these won't work with gcc.
as mentioned in a separate topic the diagnostic for anon bit field qualifiers is a real error for C++. You can compare https://en.cppreference.com/w/cpp/language/bit_field vs https://en.cppreference.com/w/c/language/bit_field (I know cppreference is not authoritative for the spec but it is close enough), basically this is allowed in C, but not in C++.
so unless there are some builtin macros clang is failing to infer (which is a shortcoming of custom toolchain support, if a compiler has magical builtin macros clang(d) can't really know about them and there's no good way to communicate those today), this must be an extension of gcc. We can diagnose this better if you can provide the contents from the ac.h around line 241 (as indicated by clang's error).

i believe the piece missing here is the command line generated by clangd is not executed as-is it is used to create a clang compiler instance. hence the need for deleting --driver-mode or --resource-dir when invoking the command as-is, because these won't work with gcc.

I see. I couldn't find those commands on the GCC documentation, so that makes sense.

as mentioned in a separate topic the diagnostic for anon bit field qualifiers is a real error for C++. You can compare https://en.cppreference.com/w/cpp/language/bit_field vs https://en.cppreference.com/w/c/language/bit_field (I know cppreference is not authoritative for the spec but it is close enough), basically this is allowed in C, but not in C++.

Yes, I realize it is a real error. Obviously though, it compiles fine using gcc-arm-eabi so there is some bug with clang where the cross compilation is not working correctly.

so unless there are some builtin macros clang is failing to infer (which is a shortcoming of custom toolchain support, if a compiler has magical builtin macros clang(d) can't really know about them and there's no good way to communicate those today), this must be an extension of gcc. We can diagnose this better if you can provide the contents from the ac.h around line 241 (as indicated by clang's error).

I am not an expert on this low-level stuff, to be honest, so I really appreciate all the help you could give, this is alwo why I am using Arduino and not developing down to the metal for SAMD 😁. But the lines that clangd-arduino.ino are complaining about are ac.h(241, 18):

ac.h
  ```
  235 /* -------- AC_INTFLAG : (AC Offset: 0x06) (R/W  8) Interrupt Flag Status and Clear -------- */
  236 #if !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__))
  237 typedef union { // __I to avoid read-modify-write on write-to-clear register
  238   struct {
  239     __I uint8_t  COMP0:1;          /*!< bit:      0  Comparator 0                       */
  240     __I uint8_t  COMP1:1;          /*!< bit:      1  Comparator 1                       */
  241     __I uint8_t  :2;               /*!< bit:  2.. 3  Reserved                           */
  242     __I uint8_t  WIN0:1;           /*!< bit:      4  Window 0                           */
  243     __I uint8_t  :3;               /*!< bit:  5.. 7  Reserved                           */
  244   } bit;                       /*!< Structure used for bit  access                  */
  245   struct {
  246     __I uint8_t  COMP:2;           /*!< bit:  0.. 1  Comparator x                       */
  247     __I uint8_t  :2;               /*!< bit:  2.. 3  Reserved                           */
  248     __I uint8_t  WIN:1;            /*!< bit:      4  Window x                           */
  249     __I uint8_t  :3;               /*!< bit:  5.. 7  Reserved                           */
  250   } vec;                       /*!< Structure used for vec  access                  */
  251   uint8_t reg;                 /*!< Type      used for register access              */
  252 } AC_INTFLAG_Type;
  253 #endif /* !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__)) */

It is interesting to note that when I open ac.h, intellisense changes and does not provide an error about ac.h(241, 18), rather the following lines:

ac.h
  |   Lines        |   Intellisense Error                                                                                |
  |   1              |   Too many errors emitted, stopping nowclang(fatal_too_many_errors)   |
  |   60-66       |   Unknown type name 'uint8_t'clang(unknown_typename)                      |
  |   88-96       |   Unknown type name 'uint8_t'clang(unknown_typename)                      |
  |   116-122   |   Unknown type name 'uint16_t'clang(unknown_typename)                    |

ac.h:1
  ```cpp
  /**
  ```

ac.h:56-68
  ```cpp
  /* -------- AC_CTRLA : (AC Offset: 0x00) (R/W  8) Control A -------- */
  #if !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__))
  typedef union {
    struct {
      uint8_t  SWRST:1;          /*!< bit:      0  Software Reset                     */
      uint8_t  ENABLE:1;         /*!< bit:      1  Enable                             */
      uint8_t  RUNSTDBY:1;       /*!< bit:      2  Run in Standby                     */
      uint8_t  :4;               /*!< bit:  3.. 6  Reserved                           */
      uint8_t  LPMUX:1;          /*!< bit:      7  Low-Power Mux                      */
    } bit;                       /*!< Structure used for bit  access                  */
    uint8_t reg;                 /*!< Type      used for register access              */
  } AC_CTRLA_Type;
  #endif /* !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__)) */
  ```

ac.h:84-98
  ```cpp
  /* -------- AC_CTRLB : (AC Offset: 0x01) ( /W  8) Control B -------- */
  #if !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__))
  typedef union {
  struct {
      uint8_t  START0:1;         /*!< bit:      0  Comparator 0 Start Comparison      */
      uint8_t  START1:1;         /*!< bit:      1  Comparator 1 Start Comparison      */
      uint8_t  :6;               /*!< bit:  2.. 7  Reserved                           */
    } bit;                       /*!< Structure used for bit  access                  */
    struct {
      uint8_t  START:2;          /*!< bit:  0.. 1  Comparator x Start Comparison      */
      uint8_t  :6;               /*!< bit:  2.. 7  Reserved                           */
    } vec;                       /*!< Structure used for vec  access                  */
    uint8_t reg;                 /*!< Type      used for register access              */
  } AC_CTRLB_Type;
  #endif /* !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__)) */
  ```

ac.h:112-135
  ```cpp
  /* -------- AC_EVCTRL : (AC Offset: 0x02) (R/W 16) Event Control -------- */
  #if !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__))
  typedef union {
    struct {
      uint16_t COMPEO0:1;        /*!< bit:      0  Comparator 0 Event Output Enable   */
      uint16_t COMPEO1:1;        /*!< bit:      1  Comparator 1 Event Output Enable   */
      uint16_t :2;               /*!< bit:  2.. 3  Reserved                           */
      uint16_t WINEO0:1;         /*!< bit:      4  Window 0 Event Output Enable       */
      uint16_t :3;               /*!< bit:  5.. 7  Reserved                           */
      uint16_t COMPEI0:1;        /*!< bit:      8  Comparator 0 Event Input           */
      uint16_t COMPEI1:1;        /*!< bit:      9  Comparator 1 Event Input           */
      uint16_t :6;               /*!< bit: 10..15  Reserved                           */
    } bit;                       /*!< Structure used for bit  access                  */
    struct {
      uint16_t COMPEO:2;         /*!< bit:  0.. 1  Comparator x Event Output Enable   */
      uint16_t :2;               /*!< bit:  2.. 3  Reserved                           */
      uint16_t WINEO:1;          /*!< bit:      4  Window x Event Output Enable       */
      uint16_t :3;               /*!< bit:  5.. 7  Reserved                           */
      uint16_t COMPEI:2;         /*!< bit:  8.. 9  Comparator x Event Input           */
      uint16_t :6;               /*!< bit: 10..15  Reserved                           */
    } vec;                       /*!< Structure used for vec  access                  */
    uint16_t reg;                /*!< Type      used for register access              */
  } AC_EVCTRL_Type;
  #endif /* !(defined(__ASSEMBLY__) || defined(__IAR_SYSTEMS_ASM__)) */
  ```

It seems to me like the basic types for SAMD are somehow missing. Or this could be a false positive.

This is probably about the definition of __I macro. You can check where it is defined to see if there's anything interesting.

This is probably about the definition of __I macro. You can check where it is defined to see if there's anything interesting.

It sounds like you may have a good idea of what is going on. I did notice that this ac.h file never includes stdint.h but that shouldn't be a probem, right? I have been trying to find where this __I is defined and have had no luck. Any hints?

Sorry feels like this is getting messier and away from land of clangd. First of all, clangd would provide really poor results on those internal headers without a proper project setup for those. It is fine to include them in another project as they can usually be compiled as part of your project with good compile flags clangd has access to, but if you open them directly story changes a lot since the flags are usually from far away directories and even more those headers are not necessarily self-contained (e.g. they are only meaningful as part of a bigger translation unit that includes a bunch of other code before that particular header).
I was suggesting the location of __I but don't see anything around that in your last message. It is probably the case that __I is defined with a cv-qualifier in some configs and without in others (by depending on some macros). I was suggesting you figure out which branch clangd should use and find out why it is not using that one.

Sorry feels like this is getting messier and away from land of clangd. First of all, clangd would provide really poor results on those internal headers without a proper project setup for those. It is fine to include them in another project as they can usually be compiled as part of your project with good compile flags clangd has access to, but if you open them directly story changes a lot since the flags are usually from far away directories and even more those headers are not necessarily self-contained (e.g. they are only meaningful as part of a bigger translation unit that includes a bunch of other code before that particular header).

Right, I don't specifically care about errors from clangd when I open non-project files, but unfortunately these errors spill over to any of my code and cause all sorts of false positives on basic stuff like initializing strings. It is pretty much unusable. Again I don't understand if the project files compile fine with GCC why should it be any different with clangd intellisense? Does clangd aggressively try to compile files that are not part of the project or included, even when I don't open them? It makes no sense.

I was suggesting the location of __I but don't see anything around that in your last message. It is probably the case that __I is defined with a cv-qualifier in some configs and without in others (by depending on some macros). I was suggesting you figure out which branch clangd should use and find out why it is not using that one.

Does clangd aggressively try to compile files that are not part of the project or included, even when I don't open them?

I'm sure you're aware that C and C++ compilers operate at the level of "translation units". That is to say, while a compilation command may only be given one input file, any #include-ed files need to be read as well, and the declarations inside them processed by the compiler. Clangd is no different: it needs to process the declarations in headers to be able to correctly process the declarations in the main file.

but unfortunately these errors spill over to any of my code and cause all sorts of false positives on basic stuff like initializing strings.

That's just it: even if you suppress the errors coming from the headers, not being able to correctly parse the declarations in the headers will mean not being able to correctly handle code in the main file which uses those declarations.


Added collapsible GA and Preview in each artifact.

Will do some clean up on the js file.

I was thinking something like:
azure-core
- <link to latest ga>
- <link to latest preview>
- collapsible group for other versions.


Make Basic Questions Opening Screen Text 3-4 Words Max

Make the Basic Questions Opening Screen Text 3-4 Words Max - (ie “MA Notice of Appearance”)
Let the next screen get more in depth

Change the initial text to: "Complete a ‘Notice of Appearance’ Form:"

Edited code on line 75 to:
code: |
interview_short_title = "Complete a 'Notice of Appearance’ Form"


Change SFTP base directory for non-Windows connections

🗣 Description
This pull request changes the SFTP base directory for non-Windows connections.
💭 Motivation and context
This is necessary for the Guacamole file browser to correct interpret the symlink to the EFS file share that resides on the VNC user's desktop in the GoPhish instance type.  The same change was already made by @mcdonnnj for Windows connections.
🧪 Testing
I created a new guacscanner AMI for our COOL staging environment with these changes and verified that the file share symlink is now interpreted correctly.
✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Finalize version.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 75.342%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
212

Relevant Lines:
262

💛  - Coveralls

This should resolve the last remaining piece of cisagov/cool-system-internal#68, correct?

Well, along with cisagov/guacscanner-docker#6, cisagov/guacamole-composition#33, cisagov/ansible-role-guacamole#41, and cisagov/guacamole-packer#57 it does.


chore(main): release 1.1.2

:robot: I have created a release beep boop
1.1.2 (2022-11-07)
🧹 Miscellaneous

use npm_config__authtoken for publishing (b356665)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/apollo-server-integrations/apollo-server-integration-h3/releases/tag/v1.1.2 :sunflower:


Clear the servers to free memory in auto deployment

Because @tedim52 doesn't remember to do this even when I ask him to

hAha, LGTM


add option to install GDAL on win with cgoehlkes binaries

don't know if this is ok, helps me on windows though.
@relativityhd please have a look

Sieht gut aus 👍🏻


No record with "visual_input": "2"

As titled.

Thanks for your question!
visual_input=0 - no visual
visual_input=1 - with original visual
visual_input=2 - with modified visual (non-common sense)


Can't add a supposedly healthy ml backend to LabelStudio using docker compose.

Hello,
I am using docker-compose to run Label Studio and my custom ml backend. Containers seem to run fine, and my backend passes "health" check - if i type in browser http://localhost:9090/health I am getting status "UP". However, when trying to "Validate and Save" model in Label Studio I am getting error:

app_1    | {"asctime": "09/Apr/2023:14:20:41 +0000", "name": "urllib3.connectionpool", "funcName": "urlopen", "lineno": 812, "levelname": "WARNING", "user_id": null, "message": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6577060f40>: Failed to establish a new connection: [Errno 111] Connection refused')': /health", "request_id": null}
app_1    |
app_1    | {"asctime": "09/Apr/2023:14:20:41 +0000", "name": "urllib3.connectionpool", "funcName": "urlopen", "lineno": 812, "levelname": "WARNING", "user_id": null, "message": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f65770610f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /health", "request_id": null}
app_1    |
app_1    | {"asctime": "09/Apr/2023:14:20:41 +0000", "name": "core.utils.common", "funcName": "custom_exception_handler", "lineno": 81, "levelname": "ERROR", "user_id": null, "message": "0a91039c-66b8-46b7-871a-d6945e773544 {'non_field_errors': [ErrorDetail(string='Can\\'t connect to ML backend http://localhost:9090, health check failed. Make sure it is up and your firewall is properly configured. Learn more about how to set up an ML backend. Additional info:HTTPConnectionPool(host=\\'localhost\\', port=9090): Max retries exceeded with url: /health (Caused by NewConnectionError(\\'<urllib3.connection.HTTPConnection object at 0x7f6577061420>: Failed to establish a new connection: [Errno 111] Connection refused\\'))', code='invalid')]}", "exc_info": "Traceback (most recent call last):
  File \"/usr/local/lib/python3.10/dist-packages/rest_framework/views.py\", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File \"/usr/local/lib/python3.10/dist-packages/django/utils/decorators.py\", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File \"/usr/local/lib/python3.10/dist-packages/rest_framework/generics.py\", line 242, in post
    return self.create(request, *args, **kwargs)
  File \"/usr/local/lib/python3.10/dist-packages/rest_framework/mixins.py\", line 18, in create
    serializer.is_valid(raise_exception=True)
  File \"/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py\", line 235, in is_valid
    raise ValidationError(self.errors)
rest_framework.exceptions.ValidationError: {'non_field_errors': [ErrorDetail(string='Can\\'t connect to ML backend http://localhost:9090, health check failed. Make sure it is up and your firewall is properly configured. Learn more about how to set up an ML backend. Additional info:HTTPConnectionPool(host=\\'localhost\\', port=9090): Max retries exceeded with url: /health (Caused by NewConnectionError(\\'<urllib3.connection.HTTPConnection object at 0x7f6577061420>: Failed to establish a new connection: [Errno 111] Connection refused\\'))', code='invalid')]}", "request_id": null}
app_1    |
app_1    | {"asctime": "09/Apr/2023:14:20:41 +0000", "name": "django.request", "funcName": "log_response", "lineno": 224, "levelname": "WARNING", "user_id": null, "message": "Bad Request: /api/ml/", "status_code": 400, "request": "<WSGIRequest: POST '/api/ml/'>", "request_id": null}
app_1    |
app_1    | {"asctime": "09/Apr/2023:14:20:41 +0000", "name": "django.request", "funcName": "log_response", "lineno": 224, "levelname": "WARNING", "user_id": null, "message": "Bad Request: /api/ml/", "status_code": 400, "request": "<WSGIRequest: POST '/api/ml/'>", "request_id": null}
app_1    |
nginx_1  | {"timestamp":"1681050041050","http":{"method":"POST","request_id":"ab5b5c1ae881f29c3d131e0a5b2db9f3","status_code":400,"content_type":"application/json","useragent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0","referrer":"http://localhost:8080/projects/1/settings/ml","x_forwarded_for":"","url":"/api/ml","version":"HTTP/1.1","connection":"148","connection_requests":"10"},"network":{"bytes_written":1488,"bytes_read":1064,"client":{"ip":"<IP_ADDRESS>","port":50188},"destination":{"ip":"<IP_ADDRESS>","port":8085},"nginx":{"request_time":"0.031","upstream_connect_time":"0.000","upstream_response_time":"0.030","upstream_header_time":"0.030"}}}

I don't know what exactly validation does, maybe it tries to load my model and fails there but the exception is not propagated? How to debug further?
ml backend server shows no issues:
redis   | 1:C 09 Apr 2023 14:18:17.718 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis   | 1:C 09 Apr 2023 14:18:17.718 # Redis version=7.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
redis   | 1:C 09 Apr 2023 14:18:17.718 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis   | 1:M 09 Apr 2023 14:18:17.718 * monotonic clock: POSIX clock_gettime
redis   | 1:M 09 Apr 2023 14:18:17.719 * Running mode=standalone, port=6379.
redis   | 1:M 09 Apr 2023 14:18:17.719 # Server initialized
redis   | 1:M 09 Apr 2023 14:18:17.719 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis   | 1:M 09 Apr 2023 14:18:17.719 * Loading RDB produced by version 7.0.10
redis   | 1:M 09 Apr 2023 14:18:17.719 * RDB age 1218 seconds
redis   | 1:M 09 Apr 2023 14:18:17.719 * RDB memory usage when created 0.85 Mb
redis   | 1:M 09 Apr 2023 14:18:17.719 * Done loading RDB, keys loaded: 0, keys expired: 0.
redis   | 1:M 09 Apr 2023 14:18:17.719 * DB loaded from disk: 0.000 seconds
redis   | 1:M 09 Apr 2023 14:18:17.719 * Ready to accept connections

I did test this also on one of the examples "dummy_model" and the behaviour is the same.

Tried command curl -X GET http://localhost:9090/health from inside of Label Studio container and got "curl: (7) Failed to connect to localhost port 9090 after 0 ms: Connection refused"

Fixed it by creating an external network in docker-compose.yml of dummy_model. That network is label studio network and then I can reach the container by using its direct ip, not localhost. I don't think this is the correct solution.
What is interesting is that when I am inside label studio container now I can curl -X GET http://localhost:9090/health but this address is still unreachable when typed in web interface for adding a model.

There is an easier fix. You can specify name of the container instead of the ip. That way the model can link up with the Label studio. In principle you need to give the runtime information about internal network.

Leaving a comment to add that I'm having the same issue.


Fixes #16 - mega model running out of memory

Prior to this fix the mega model will (more often than not) fail when running in a loop.
Clearing the cache seems to fix the issue.

A side note on this fix: depending on your video card it is still possible to run out of memory even with this code in place while running mega (if you do anything intensive on the GPU, that includes multiple tabs in firefox or a video on youtube).

Hmm right now my solution is adding this at the cell after
display(model.generate_image(text, seed, grid_size))
torch.cuda.empty_cache()

Thanks for this. They are having issues on replicate too where they have to restart the server every 10 minutes. I wonder if clearing cuda cache would fix that problem

Thanks for this. They are having issues on replicate too where they have to restart the server every 10 minutes. I wonder if clearing cuda cache would fix that problem

oh interesting, i'd love to find out if this helps. there are probably solutions that can be combined with this surrounding PYTORCH_CUDA_ALLOC_CONF which is mentioned in the error messages that pop up when running out of memory


Interested?

I sent you a dm on Discord and pinged you about the status of this plugin, let me know if that would be okay.

added as collaborator :+1:


fix #2225 by changing namespace for UseHub

fixes #2225

Declined by Shiproom and we can't take it in dev.


Fix remaining known issues on Python 3.5.2 in Beam 2.11.0 release branch.

This change cherry-picks commits from #7873.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

cc: @aaltay @charlesccychen
We also plan to have a follow-up change to master to address remaining cleanup comments on #7873, if that happens before RC1 is created I'll try to add them to this PR and to cherry-pick them to 2.11.0.

Run Python PreCommit

Run Python PreCommit

I plan to CP commit(s) from https://github.com/apache/beam/pull/7902 to this PR once https://github.com/apache/beam/pull/7902 is merged.

Please send a different PR for cp'ing #7902.

Sent out https://github.com/apache/beam/pull/7909 to cherry-pick #7902.


threading on linux with clang

Summary:
Threading on linux + clang is broken in most linux distributions, but can be fixed.
Description:
Most linux + clang configurations will not be able to turn on threading in stan-math as they run into this issue. It would be great to automatically detect whenever the solution mentioned on stackoverflow works and if so link it in automatically. Very likely this is linked to the libc++ version used. If libc++ >= 4.0 is used, then things should work out of the box, but this needs to be verified.
Reproducible Steps:
On linux with a clang compiler do
0. use clang on linux

enable threading, see here
./runTests.py test/unit/math/rev/mat/functor/gradient_test.cpp

Current Output:
This won't link
Expected Output:
Describe what you expect the output to be. Knowing the correct behavior is also very useful.
Additional Information:
Provide any additional information here.
Current Version:
develop, 7th April 2018

This issue is mis-named but would be good to get this fixed.  The configuration actually works if you change make/detect_default_cc (as below).  So the issue is linux + clang + libc++ (some versions!).  On my Ubuntu installing the default packaged libc++ actually fails to build our tests unless I modify some headers in libc++ (to stop pretending like libc++ has xlocale suport and a cxxabi compatability header).
I know there are "reasons" why -stdlib=libc++ was added for clang++ but they've never been specified and we don't know if they're still relevant so I'd say fix this by dumping libc++ from the default config.
.. @syclik @seantalts do you recall the reason why this -stdlib switch is so broad?  If the real reason is lost to history can we try changing it and see who complains? It's irritating to have to keep hacking things to make Ubuntu + clang work with Stan.
krzysztof@fit:~/packages/stan-math-threads$ git diff
diff --git a/make/detect_cc b/make/detect_cc
index fd899cd1b4..104ce0a987 100644
--- a/make/detect_cc
+++ b/make/detect_cc
@@ -10,7 +10,6 @@ ifneq (,$(findstring clang++,$(CC)))
   # To deal with that we force it to use llvm's bundled libc++ replacement for
   # gcc's stdlibc++ here. Plus it's better to rely on just one compiler if we
   # can.
-  CXXFLAGS += -stdlib=libc++
 endif
 ifneq (,$(findstring mingw32-g++,$(CC)))
   CC_TYPE ?= mingw32-g++

Oh right, the comment does reference Travis, but Travis is not something I can fix.  Tell me what fails on Travis and we have a real to work on.  Or put a make/local as part of the Travis config and stop having an old Travis bug drive our development.

I think you are right in that we should not enforce to use libc++ whenever clang++ is used - rather we should use whatever is the default of the clang++ we are using.
I haven't set this up, but I am guessing that the rationale behind the setting is that on Travis we need to force clang++ to use libc++ via this option. The reason to push clang++ into the direction of libc++ is probably that libc++ is recommended over libstdc++ for use with clang++. Now, that is obviously not a super good recommendation as we are discovering.
I would argue that we should not specify the -stdlib option for clang++ such that it will use the default used to build the compiler (whatever that is). This should give us the best outcome in terms of system compatibility, etc.
@seantalts or @syclik Any thoughts?

I am assuming this is irrelevant with TBB now. No issues on Linux + clang or g++. Closing.


Update doc for data_specs in Dataset.iterator()

@goodfeli Could you have a look? Could you merge if it is clear enough, or let me know if it is not?

@goodfeli Updated.

I think all your comments should be addressed, unless something is still not clear about the use of CompositeSpace, in which case, please let me know.


when HTTPClientAndScheme returns a nil client, handle it

when HTTPClientAndScheme returns a nil client, return an error
fixed #2098
Signed-off-by: Sun Hongliang<EMAIL_ADDRESS>

LGTM

@allencloud we want to do an RC release shortly, so we had to make some changes and open a different PR. Sorry about that, it was slightly urgent.

@allencloud Thanks for submitting the PR! We planned to release v1.2.0-rc3 today so we take #2102 instead. Sorry about the condition. We really appreciate your effort.

That is all right. @nishanttotla @dongluochen


Dedup methods

Hi team!
It seems Exact Dedup, MinHash, Suffix Array, and Bloom Filter are studied in the dedup ablation experiments. I have a couple of questions related to these:

Where can I find the code for the Suffix Array?
Are there any examples of how to use MinHash and Bloom Filter? I see that their code is uploaded 2 weeks ago, but I'm unsure how to integrate them into the process code.

Thank you!

Hi @ch-shin ,
Apologies for the late response!

Our suffix array deduplication is based on the implementation from Lee et al., you can find the original implementation here: https://github.com/google-research/deduplicate-text-datasets
Currently we recommend running deduplication as a separate stage from the process code, by following the respective instructions e.g. for BFF:  https://github.com/mlfoundations/dclm/tree/main/dedup/bff

Let us know if the above are helpful!


Force Encryption by default with option to disable.

Could this be possible?

See mafintosh/torrent-stream#55


Multiple cursors on next reference?

Hello,
I think this feature was included in previous versions of vscode, but just after discover it, I think it disapear, the feature is the option to not create multiple cursors for every references of the selected text, The think is sometimes I've got a copy pasted from other code, or some example, and I wish to change the references of a variable or something, and I don't wish to break all my code, I think I used that 3 or 4 times some month ago, but now, I don't found this feature.
Can it will be included?
Thanks.

cmd+d / ctrl+d?

Yes, sorry, I was searching it for various dais. Sorry.


Back pressure for thread pool task queue

Hi,
starting from the basic inference example, how would you advise to implement a simple back pressure mechanism? If I understand correctly, the task queue of the thread pool implementation that Tensorrt lab uses, does not have an upper size limit. Thus, if for example, my data ingest is much faster than inference, the program will eventually run out of memory, as the input tensors have to be captured by the lambdas in the task queue. It would be nice to have an optional behavior, where enqueue becomes blocking once the task queue reaches a certain size.

Great question.
The nvRPC examples, similar to TRTIS, have several limits on the depth of the work queues.

the number of gRPC contexts registered with the executor only allow for XXX gRPC messages to be pullled off the wire
executor->RegisterContexts(rpcCompute, rpcResources, XXX);
if you are using the c++ or the python [the tensorrt runtime], InferRunner, has limits of the queue depth enforced by the InferenceManager.   We provide a canonical example of what's happening in the InferRunner which demonstrates where the a call to the runtime might block.
auto buffers = GetResources()->GetBuffers();

if you use the offload variant of the Infer method, then yes, you could get into a situation where your queue depth becomes unbounded and you run out of resources.
if you use the direct variant which operates on a Bindings object, then you've had to acquire a Bindings object, which similar to the canonical example means you've been limited by the InferenceManager and the call may have blocked

Does this help?
The second part of your question on recognizing and reacting to queue depth.
Both TRTIS and the some of the TRTLAB examples expose Prometheus metrics.  The load ratio = request_time / compute_time is a nice way to gauge queue depth.   If you just measure queue depth, that doesn't help you distinguish the type of model in the queue.  A large queue depth of models that don't take long to compute means something different than a queue depth of a model that takes a long time to compute.  Load Ratio help normalize that with respect to the time the model takes to compute.  You can use the load ratio and gpu energy consumption metrics to trigger horizontal auto scaling.  Follow Issue #20 to track progress on this.
One thing we don't do in either the LAB or TRTIS is check the gRPC deadline.  We should do this.  This is one reactive way we can deal with queue depth is to simply start canceling requests on the server side.   If you control the client, then this is a good feedback mechanism that you need to grow the number of workers.  Follow Issue #21 to track progress.

Thanks! Yes, that helps a lot. I will use a fixed size ressource pool of buffers, into which my IO threads write the input blobs.

Let me know if you run into any problems or if we can improve the experience in any way.


fix: register vectorize utils even if not running remotely

it feels more intuitive to do:
const vectorize = hubVectorize()
vectorize?.upsert(/* */)

rather than
const vectorize = typeof hubVectorize !== 'undefined' ? hubVectorize() : null
vectorize?.upsert(/* */)

added to which the latter is a type error, if you run nuxi prepare without the --remote flag.

for me, I want to be able to develop my site even if offline so I would suggest it doesn't throw an error if called without --remote - letting the user handle it is IMO a better DX


Allow passthrough of req, res, ctx, next by having x-passthrough-middleware: true

Currently, we already support if we have specific path with specific method.
Let's explore if it's possible or not to have an all-method-operation (e.g. for proxy purposes).

I think it's not possible. Swagger is for "exact" operations, and then we may have different paths, with different query parameters, with different body, etc. etc.
Let's mark this as won't fix.


Feature/423

Provide segment standardizing, make db query for ui easier.

Coverage decreased (-0.4%) to 18.163% when pulling 5ae51d2fc37ae53aa56e450fe85104451eadf919 on feature/423 into 17410e665b4285c8132d365860760776837b33c6 on master.

Coverage decreased (-0.4%) to 18.163% when pulling 60f94213df5d8dd542251b150130488cba31b393 on feature/423 into 17410e665b4285c8132d365860760776837b33c6 on master.

Coverage decreased (-0.4%) to 18.163% when pulling 6208b9cb44a0fc5c5c66d6ce5b6aef7870d85511 on feature/423 into 17410e665b4285c8132d365860760776837b33c6 on master.


增加对intent的action的设置支持

在postcard里增加对intent的action的设置支持

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: zhi1ong:x: lucifinilsuYou have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: zhi1ong:x: lucifinilsuYou have signed the CLA already but the status is still pending? Let us recheck it.

拉取 develop 最新的代码，修改之后合并到 develop，请勿直接合并到 master


Bug: n/no-missing-imports doesn't work correctly for workspace modules

Environment
Node version: 20
yarn version: 4
ESLint version: 8.56
eslint-plugin-n version: 16.6.2
Operating System: MacOS
What rule do you want to report?
no-missing-imports
Link to Minimal Reproducible Example
https://github.com/JeremyLoy/hidden-module-import-issue-example/tree/main
What did you expect to happen?
In this monorepo using yarn workspaces, I have a local package named @myorg/math. It is encapsulating an implementation, and only exposing one module via it's package.json exports property.
All imports from this package are getting reported as missing by eslint-plugin-n, when only the private module implementation.js should be getting reported as missing. See the test file for comments.
Participation

[X] I am willing to submit a pull request for this issue.

Additional comments
I'm willing to help contribute if you point me in the correct direction!

I will take a look at your example at the end of the day (UK time)
I do know that in eslint v17 the resolution mechanisms are completely overhauled (#139).
You could try using version 17 (yarn add -D eslint-plugin-n@next)

That works, thanks!


return 0 in ukernels

Context: https://github.com/openxla/iree/pull/13600#issuecomment-1546129017
We always return 0 in these ukernels because validation is assertions - the idea was that these ukernels are called internally by compiler-generated code, and it the compiler generates an invalid call, that's a bug, so it's fair to assert. We can revisit if we believe the sociology of who's calling ukernels is changing, but keep in mind that validation-as-asserts was a nice code-size win in release builds, and conveniently freed us from having to think about validation overhead.

Agree that asserts are best!


Fix upgrade-scenario-01

This PR moves bid adding to after the nodes that are bidding have been synced. This way the nodes will not become validators before they are ready, and not get ejected.

Good point. They shouldn't be needed, let me remove them :+1:


The command log sorts inconsistent

Issuing watson log outputs all events in a certain time range. It groups them by day and prints the days from most recent to the oldest day in that range. However, within these days, it sorts the events from oldest to most recent.
I consider this to be a bit inconsistent. It hinders to read the log from bottom to top or vice versa.
Is this behaviour on purpose or could/should it be changed?

It is a feature, not a bug! I like to see the latest day on top, and, for each day, the daily flow from top to bottom.
But I understand that you may find this inconsistent.


new development environment secureauth credentials

Work with secureauth team to get new credentials

@mheadd I tried to work through the service now ticket, then my brain melted. Can we pair on this when you have a chance?

Issue opened in Servicenow


Image props

I tried my best at exposing as much as made sense to me, but... There are a few filters that are used exclusively on text to force it to mimic the look of --EDI-text. Optimally, there would be just a simple color: parameter instead of a filter, but I can't make it work no matter how hard I bash my head into it. If you want to give it a try the filters are on lines 840 (footer of summary pages) and 1148 (kana chart in reviews).

There are a few filters that are used exclusively on text to force it to mimic the look of --EDI-text. Optimally, there would be just a simple color: parameter instead of a filter, but I can't make it work no matter how hard I bash my head into it. If you want to give it a try the filters are on lines 840 (footer of summary pages) and 1148 (kana chart in reviews).

For line 840, there was a filter from the dashboard that was applied to the parent footer element in addition to the filter in that specific rule (so normal text colors were still getting filtered).
The solution is to make the dashboard rule only work on the dashboard. On the kana pages, the footer element is nested fairly deeply, but on the dashboard it's a direct child of the body, so this rule now only applies on the dashboard:
body > footer {
  filter: var(--ED-footer-filter);
}

I can't reproduce the issue on line 1148 (kana chart in reviews). Just using --ED-text or --EDI-text-color without the filter works for me.
Apologies, but prettier made some additional formatting changes. I'll add a .prettierrc in another PR so we can be more consistent with our formatting.
Also: what editor are you using? Almost the whole world has switched to VScode these days (even me, and I'm an old BSD Unix vi die hard). You're kinda fighting agains the grain if you don't use VScode, fwiw. I've not used it on Linux, but I know it's at least possible. VScode with emmet, prettier, snippets, etc. makes any coding about a million times easier, fwiw.
I noticed some other indentation issues, but I'll fix them in a separate commit to keep things cleaner.

I added one more rule for the self-study quiz. Doesn't belong in this PR, but just wanted to make things as easy as possible.
It fixes the styling in the following screenshot (without the fix, the "Failed Last Review" text next to the checkbox was white on white).

I'm okay to merge this. Passing back to you for final review.

Also: what editor are you using? Almost the whole world has switched to VScode these days (even me, and I'm an old BSD Unix vi die hard). You're kinda fighting agains the grain if you don't use VScode, fwiw.

For most of my work I use VScodium, but using a dedicated text editor for this project seemed unnecessary. I have been using the default one stylus provides. It's hard to compete with out of the box live previews.

I can't reproduce the issue on line 1148 (kana chart in reviews). Just using --ED-text or --EDI-text-color without the filter works for me.

I will check if it's not a browser issue and will remove the filter accordingly.

I will check if it's not a browser issue and will remove the filter accordingly.

I see you already did so. And what's more, it now appears to be working as intended on my side as well. I do not dare to ask frivolous questions such as why or how for I am just a humble servant of the code who will greatfully receive this blessing and courteously merge it.

It's hard to compete with out of the box live previews.

Agreed. But for what it's worth, I use a compromise workflow (because I distrust the in-browser editor for things like indenting, linting, and formatting of entire files):

I do most of my heavy editing in VScode.

To test, I just ctrl-a ctrl-c in my editor, then ctrl-a ctrl-v in the stylus editor, clicking the button to overwrite the entire file, then ctrl-s to save in stylus for good measure. Note that pasting the entire file in either stylus window (the top , untargeted comment header window, or the bottom www.wanikani.com targeted window) will overwrite BOTH sections.

I perform any tinkering/tweaking in stylus so that I get live updates, but I try hard to only change a rule or two at a time.

When I'm satisfied with my tweaks, I'll select just the rule(s) I'm interested in, ctrl-c, then paste them into my editor.

Once everything seems to be working in stylus, I do one final whole-file copy-and-paste from my editor to stylus to ensure I've captured everything.

Basically, my rules are:

Whole file cut and paste only FROM my editor to stylus.
Only individual rules/selectors/property-assignments flow from stylus to my editor.

The song-and-dance is a little bit of a pain, but it works well for me, and getting prettier-on-save is worth it to me.

By the way: I've started deleting merged branches (turns out they can be resurrected anyway. Things tend to get a bit cluttered with too many branches in the pulldown.

By the way: I've started deleting merged branches (turns out they can be resurrected anyway

Oh thats good news, I will get rid of the rest of them (or at least the old ones), so they don't display as active branches on the main page anymore.

I will try to work an ide into my workflow based on your example, but can't promisse the result will be any better. Im super unfamiliar with formatters in general, it might take some time for me to get used to them.


feat: Add Android support

Add support for Android. Since there is no native API for passkeys available on Android a possible workaround would be to use a web view and authenticate with the server through it.

Update: Initial Library for Android Passkeys will now be release "soon". I will start working on it once its being released (probably early next year).

@f-23 This would be awesome! No rush, but what is the current status of this?

@andrew-levy The Android team has pushed back its plans for a native passkey API on Android, so right now it is unclear when it'll be released.
From the FAQ link above:

For Android apps, we aim to have an initial library for passkey support soon.

I'd rather wait for this native API than implement some kind of workaround which would become obsolete within a few months.

@f-23 Makes sense, thanks for the update. Any thoughts on adding web support to make this truly cross platform? If you're interested, I might have time to help implement that.

@andrew-levy I think that would definitely make a good addition! I've created #7 so we can track and discuss the details on that over there.

Great news! Android team has release an alpha version of credentials library. Do you any help with integration?

@zolbooo Sounds good! Contributions are always welcome, so if you wanna work on this you're welcome to.

Status update:
I was able to take a closer look at the new androidx credentials library and managed to implement a somewhat working version into react-native-passkey.
Since the native android library is still in a very early alpha/preview phase I would prefer to wait though until a stable version is being released.
Even with the testing I did I've encountered a handful of bugs and issues. As soon as those are resolved the full implementation should not take to long based on what I've already managed to do.

Checking this article from late March: https://medium.com/androiddevelopers/bringing-seamless-authentication-to-your-apps-using-credential-manager-api-b3f0d09e0093. Are there any updates on the state of those bugs and how Android team is addressing them?
And thanks for working on it! 🙏🏼

It's looking a lot better with the newer alpha versions.
Nevertheless I've found another issue preventing me from fully implementing this yet. Created an issue on the android credentials repo for this, will have to wait and see what the Android team says.
I'm gonna keep you updated!

Probably they will announce it on Google I/O.

Android support is now available!
Bear in mind that the native androidx credentials library is still in alpha. I'm gonna use this ticket to track any issues that might occur.

beta version is live, would be possible updating?

Android support is now available!
Bear in mind that the native androidx credentials library is still in alpha. I'm gonna use this ticket to track any issues that might occur.

@f-23  what version of native androidx credentials is compatible with react native passkeys at the moment,
and when would you be updating using the new beta version..?

RN Passkeys is using androidx credentials 1.0.0-alpha08 right now.
I'm working on updating to the latest beta version.

react-native-passkey has been updated to use androidx credentials 1.2.0-beta03.
Please update to 2.1.0.

great!
thanks


Populate more properties in the distributed cache project file

Looks like we're still missing some XML docs. Will add those in another commit.

Closing in favour of #14.


pkg/nspawntool: rewrite nspawntool to validate srcdir

Rewrite RunNode() in nspawntool to check errors in a structured way.

Introduce a structure bindOption that represents a single bind option string, e.g. --bind=/src/file:/dst/file. Each instance has 3 entries, bindPrefix, srcMount, and dstMount, which would be --bind=, /src/file, and /dst/file repectively in the example.

Initially a list of bind options is composed of type []bindOption, which will be afterwards transformed into []string that can be directly passed to systemd-nspawn. Those conversions are done by buildDefaultBindsList() as well as buildK8sBindsList().

For each srcMount, check if the path exists. If not, error out. This is necessary for avoiding such cases where systemd-nspawn gets stuck suddenly without printing any error, whenever a non-existent path is passed as srcMount.

Fixes https://github.com/kinvolk/kube-spawn/issues/120

Rebased.


Fix relative image paths

I have a project with a non-flat directory tree, similar to this:
basedir/images/.png
basedir/content/.adoc
basedir/docs/master_doc.adoc
Assume source.adoc defines the following attributes:
:imagesdir: ../../images
:imagesoutdir: {imagesdir}/equations/generated_images

I found that asciidoctor-pdf could not retrieve the asciidoctor-mathematical generated images if :imagesdir: was set, if :imagesoutdir: was not set, or both not set.
This PR allows any combination of :imagesdir: and :imagesoutdir: attributes to work even if they are empty or not in a flat directory tree. The attributes can be relative or absolute paths. If the attributes are left empty, the images are created in the same directory as the AsciiDoc source file, which I believe is the default behaviour for asciidoctor-pdf if :imagesdir: is not set.

Thank you for your contribution.

This didn't fix my issue in #83 but it did move the location of where the HTML processor was writing images to.  It was putting them in the images folder of my project directory, now is it putting them in the /images folder at the root of my filesystem.
I still have the same problem where the image src attribute is wrong unless I put the generated images in the wrong location on my filesystem and depend on the PDF extension to put them in the correct spot for me.

@bwklein I just watched your video from #83, and a lot it is beyond me because I'm not familiar with web stuff. But I think the fundamental problem you're having is that this extension (asciidoctor-mathematical) is not meant to be called by the html5 backend, nor is it needed. The html5 backend handles latex/stem support beautifully by itself using something else, maybe MathJax or similar. You should only call this extension when running the PDF backend. I'll comment more specifically on #83.

I use mathematical for HTML because I want optimized images (PNG) for the equations and do not want the poor page load experience when MathJax does it's thing after page load.


Upgrade keyfunc to new stable release v1.0.0

What this PR does:
Upgrades the github.com/MicahParks/keyfunc to the latest stable release v1.0.0.
Which issue(s) this PR fixes:
None.
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

Codecov Report

Merging #318 (fd0757d) into develop (aa3a55c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #318   +/-   ##
========================================
  Coverage    36.52%   36.52%
========================================
  Files           54       54
  Lines         3253     3253
========================================
  Hits          1188     1188
  Misses        1945     1945
  Partials       120      120

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa3a55c...fd0757d. Read the comment docs.


Added 'writable' parameter to the Expandable attribute

Sometimes it's convenient to have expandable scripatable object as read only

I prefer to make the Expandable attribute to work with the Readonly attribute. I've created an issue - #272


VPCs in multiple regions?

As far as i can see, it's not possible to create one VPC in one region, and another VPC in another region.
I know this could be solved by having a provider per region, but that does not seem to be supported by this module.
Any ideas?

You are right, this module does not support region name as an argument because the module does not have provider block. You should specify it outside of the module like this:
provider "aws" {
  region = "eu-west-1"
}

module "vpc" {
  # ...
}

Works perfectly, thank you.
provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  alias = "eu-central-1"
  region = "eu-central-1"
}

module "vpc_ireland" {
  ...
}

module "vpc_frankfurt" {
  providers {
    aws = "aws.eu-central-1"
  }
  ...
}

@AzCii  when I tried to implement your code snippet I get an error from terraform that is unable to get a valid cidr module.vpc.aws_vpc.this: expected "cidr_block" to contain a network CIDR with between 16 and 28 significant bits, got: 0
my code snippet
provider ...

module "vpc_eu_central1" {
   source = "../modules/vpc_blueprint"
   providers { aws = "aws.eu-central-1"}
   cidr = "<IP_ADDRESS>/24"
}

... modules/vpc_blueprint
module "vpc" {
   ...
   cidr = "${var.cidr}"
   ...
}

Reading the error message, I think your var.cidr might be empty or at least not containing an CIDR block

I agree with @AzCii , variable cidr should be set to a valid CIDR.

Also, <IP_ADDRESS>/24 is a valid value for cidr.

Works perfectly, thank you.
provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  alias = "eu-central-1"
  region = "eu-central-1"
}

module "vpc_ireland" {
  ...
}

module "vpc_frankfurt" {
  providers {
    aws = "aws.eu-central-1"
  }
  ...
}

If you like me are ending up here and are on Terraform >=12, don't forget it should be:
  providers = {
    aws = "aws.<alias>"
  }

Note the = sign after providers, added with TF12

Do you get the following warning in the latest terraform versions?
Warning: Provider aws is undefined
│
│   on vpc_mods.tf line 3, in module "vpc_frankfurt":
│    3:     aws = "aws.eu-central-1"
│
│ Module module.vpc_frankfurt does not declare a provider named aws.
│ If you wish to specify a provider configuration for the module, add an entry for aws in the required_providers block within the module.

@dtakis Try to write without double quotes. aws = "aws.eu-central-1" => aws = aws.eu-central-1

@antonbabenko thank you! You are correct, this is the correct syntax now.

should be
providers = {     aws = aws.<alias>   }


Use functions for constraints in order to unify expression language

As discussed last week, here is a proposal for how it could look.
Three concepts are introduced:

validation keyname (instead of constraints) that is simply a boolean expression. It must evaluate to true for the value to be valid. Any expression can be used with any function with any degree of nesting.
A new VAL special name that refers to the current value, which can be used as the first argument in $get_property like SELF, TARGET, etc.
And $VAL as a shortcut for { $get_property: [ VAL ] }

Examples:
data_types:

  # "validation" is a boolean expression that must evaluate to true for the value to be valid
  # If not specified, it's as if it was "validation: true"
  # The special name "VAL" can be used with "$get_property" to refer to the value
  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_property: [ VAL ] }, 0 ] }

  # Because the above is cumbersome, let's introduce "$VAL" (escapable) as a shortcut
  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $VAL, 0 ] }

  # Function calls can be nested with boolean logic functions, such as "$or" and "$and"
  # and really any function can be called
  Host:
    derived_from: string
    validation:
      $or:
      - { $pattern: [ $VAL, ^((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)(\.(?!$)|$)){4}$ ] } # IPv4
      - $and:
        - { $equals: [ $VAL, localhost ] }
        - { $get_input: [ allow-names ] } # boolean input type

  # Powerful validations are possible across properties in a complex type
  # (impossible with the existing constraint system)
  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $get_property: [ VAL, high ] }, { $get_property: [ VAL, low ] } ]

  # The following type refers to SELF, which means it can only be used in contexts in which SELF
  # exists and has the right property (see "Compute" node type below)
  MaxRAM:
    derived_from: scalar-unit.size
    validation:
      $greater_or_equal: [ $VAL, { $get_property: [ SELF, min-ram ] } ]

node_types:

  Compute:
    properties:
      min-ram:
        type: scalar-unit.size
      max-ram:
        type: MaxRAM # this will work because we have "SELF.min-ram"

Feedback:

"VAL" is confusing, sounds like "validation". Perhaps it should be "VALUE"?
The call site within data types does not include SELF or an ability to call get_input. A call site with those would be assigned only when used in a node or relationship context. TOSCA processors may not all be happy with referring to SELF in data types.

Instead of reusing $get_property, we can introduce a new TOSCA-path-aware function, $get_value (and then we don't need the special name "VAL"):
  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_value: [] }, 0 ] }

  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $VALUE, 0 ] }

  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $get_value: [ high ] }, { $get_value: [ low ] } ]

If we do decide that the shortcut is desireable then rather that it being '$VALUE' we should consider
using '$THIS_VALUE'. Although this is more verbose it makes it clearer that it means get_value with an empty set, i.e. without any path.
If you want it to be really short AND discourage further syntax extensions of this kind then it could be '$_' although I don't like that much.
I'm not totally convinced the short cut is worth it. Would the following be valid and equivalent to the first example ?
Count1:
derived_from: integer
validation: { $greater_or_equal: [ { $get_value: [Count1] }, 0 ] }

We should consider making our function syntax more flexible, and adopt the following:

Any string that starts with a $ is interpreted to be a function name. This makes $ a reserved character in TOSCA. String names that start with $ and that are not functions must be escaped.
Functions that don't take any arguments don't need to use map syntax. For example

Count:
    derived_from: integer
    validation: { $greater_or_equal: [ $get_value, 0] }

is the same as
Count:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_value: [] }, 0] }

Functions that take only one argument don't need to use the 'list' syntax for arguments. For example

NonZero:
    derived_from: integer
    validation: { $not: { $equal: [ {$get_value: [] }, 0 ] }

is the same as
NonZero:
    derived_from: integer
    validation: { $not: [ { $equal: [ {$get_value: [] }, 0 ] ] }

I think this would simplify function syntax without introducing additional parsing challenges.

The propsal from @lauwers looks good but I have one question: If I were writing a custom function which took only one argument would I have to write it so that had the capability to accept
a) the argument
and also as an alternative
b) a list with one entry which contained the argument?
if so then not so great, it would better if the TOSCA parser did that for me and passed only one of those possibilities, in which case we would have to specifiy which one.

Some input regarding what functions would be useful for both conditions and constraints and have them as supported by default (all of them boolean):

Functions that could work for both lists (seq) and strings:
is_suffix (string - string, value/list - list)
is_prefix (string - string, value/list - list)
is_contained (string - string, value/list - list)

Functions to check if is a value or key in a list/map
values_of (value/list - list, value/list - map)
keys_of (value/list - map)

Boolean logic functions:
and, or, not, xor, eq, imp

forall and exists:
Functions that apply another function (given as the 1st argument) to each of the elements of the list (2nd argument) and a value (3rd argument).
e.g.:
$forall: [equal, [el1, el2, ... eln], value]]
$exists: [equal, [el1, el2, ... eln], value]]
Note1: we do not write equal with $ since we do not want to evaluate equal before evaluating forall
Note2: arg1 must be a string that resolves to a boolean function name, arg2 must be a list.

In today's ad hoc with proposed a general syntax for magic words in TOSCA, see issue #133.
With that in mind we proposed that the function to use to retrieve the current value in constraint contexts be $value. Examples based on the original post above:
data_types:

  # Full syntax with no arguments
  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $value: [] }, 0 ] }

  # Magic word
  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $value, 0 ] }

  # Full syntax with arguments
  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $value: [ high ] }, { $value: [ low ] } ]

As I understand it a custom function could in principle also be called value. At this point in time, we are at liberty to define reserved TOSCA magic words like value, but if after TOSCA 2.0 has been out for some time, and in TOSCA 2.1 we want to introduce a new magic word hello, then we risk that there are already solutions out-there that defined a custom function named hello.
So in my experience, it is a good idea to reserve a namespace for TOSCA reserved words before anyone is using the custom functions feature.
For example we could say that all reserved magic words must end in two underscores $value__ or $hello__. That way we can later safely introduce new magic words without risking backwards compatibility issues.

I believe the current spec makes it impossible to define a custom function with the same name as a built-in function, since such a function would be interpreted as a refinement/augmentation of the built-in function, not an entirely new function in its own right. My understanding is that refinements can only add implementations to existing function signatures, or append new function signatures.

This is an important issue for us to consider -- what if you import two profiles in which each defines a function with the same name? @pmbruun has a point that perhaps functions should be namespaced, perhaps with the same ":" notation we use for types. e.g. something like $aws:server_name

Yes, I thought we had previously decided that function names are namespaced. Can you think of reasons why they shouldn't?

This proposal has been accepted and is documented in Section 5.4.6 of https://docs.oasis-open.org/tosca/TOSCA/v2.0/csd05/TOSCA-v2.0-csd05.html. That section shows the use of the $value function when defining constraints/validations.


suggestion: it would be really useful to include mempool total value (utxo totals), and total value xfer rate

For mempool value (utxo total sat in mempool): I would try to include this first by adding a second y axis for mempool utxo totals.  If that screws up chart visually, it could be added as additional chart instead.
This would reveal if mempool spikes are likely spam, or looks like real value is pushing up the mempool.
A related metric would be total value xfer rate, i.e. utxo total sat / kB.   If the fee rate is spiking, but the total value xfer rate is crashing, this would be suspicious, or at any rate evidence that there are plenty of low value transactions that could wait their turn without making more block space available.
If both fee rate and total value xfer rate are spiking, that means there are high values of bitcoin trying to move around, that are getting close to being crowded out.  Which would be evidence that big blockers are right, and the time to 2mb-fork is closer than the small blockers think.

Sounds good, except perhaps we should call "small blockers" ... "never forkers" :-)

0.12 version eliminates what's called "priority transaction" sub-pool, i.e. upto 50,000 bytes in each block that was allocated to transactions with large amount of days destroyed, etc ... after the size of this "priority area" is changed to 0 by default, no transaction with under the dust relay minimium will propagate at all. At this point, one can argue that there is no more spam left bc if a transaction is paying more than dust relay minimum, then it's not spam (according to Bitcoin's working definition).

My definition of spam is suspect uneconomic transactions: transactions created specifically to manipulate statistics and public perception.  It's impossible to prove definitely where tx statistics manipulation is taking place, but my purpose with this suggestion is to make it as difficulat as possible, by providing public metrics that are increasingly difficult/expensive to convincingly fake.

Thanks for the suggestions.
It's indeed useful, but outside the scope of this project, which is simply to estimate the required tx fee needed for a given wait time.
I believe tradeblock does show the data that you are looking for (under "recent mempool").