WIP: 566 System Capability Manager

Adding System Capability Manager per #566

Codecov Report

Merging #588 into develop will decrease coverage by 1.37%.
The diff coverage is 60.29%.

@@              Coverage Diff              @@
##             develop     #588      +/-   ##
=============================================
- Coverage      39.96%   38.58%   -1.38%
- Complexity      2301     2329      +28
=============================================
  Files            322      323       +1
  Lines          14011    14877     +866
  Branches        1458     1731     +273
=============================================
+ Hits            5599     5740     +141
- Misses          8211     8909     +698
- Partials         201      228      +27

Impacted Files
Coverage Δ
Complexity Δ

...n/java/com/smartdevicelink/proxy/SdlProxyBase.java
6.39% <0%> (-0.03%)
6 <0> (ø)

...vicelink/proxy/rpc/enums/SystemCapabilityType.java
87.5% <100%> (+16.07%)
2 <0> (ø)
:arrow_down:

...om/smartdevicelink/proxy/rpc/SystemCapability.java
42.85% <22.22%> (+7.56%)
6 <1> (+2)
:arrow_up:

...smartdevicelink/proxy/SystemCapabilityManager.java
76.31% <76.31%> (ø)
5 <5> (?)

...om/smartdevicelink/transport/SdlRouterService.java
6.94% <0%> (+0.78%)
8% <0%> (+3%)
:arrow_up:

...icelink/transport/MultiplexBluetoothTransport.java
9.32% <0%> (+2.61%)
19% <0%> (+8%)
:arrow_up:

...celink/proxy/rpc/RegisterAppInterfaceResponse.java
81.03% <0%> (+5.17%)
31% <0%> (+2%)
:arrow_up:

...in/java/com/smartdevicelink/proxy/RPCResponse.java
32.25% <0%> (+9.67%)
6% <0%> (+2%)
:arrow_up:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f7da95...7ed0735. Read the comment docs.


I don't see changes when rendering Pendulum + Atari question

Awesome repo and paper. One question, when I render multiple initializations of "SunblazePendulumRandomExtreme-v0", I can't see the changes in the bar length. This doesn't happen in the rest of the environments. Maybe the changes are very small, or is there a problem with the code?
Another question: is there an easy way to implement the Breakout environment (e.g. with gym function env="SunblazeBreakutSomething").
Thank you!

I've got the same issue with CartPole :(


Re-introduce heartbeat metrics in a new periodic function since pollQueue is going away

Alternatively, we can consult Amazon's built-in cloudwatch metrics to see what's happening.

Let's investigate getting the metrics out and putting them in our dashboard.

We don't have access to cloudwatch, but we're going to try to use datadog to see if it will work for this.

I think we can:

Emit datadog metrics from accept/process where it makes sense.
Create a cron'd lambda that checks any other resources and publish those to datadog.

Not sure if the cron makes sense, it depends on what metrics are needed. Some/All might be available directly from cloudwatch, which are automatically sent to datadog.

After looking at https://docs.datadoghq.com/integrations/amazon_lambda/ I see they recommend the log product, which we don't have, for lambda metrics.
Let's just publish metrics to Cloudwatch. They should eventually make it to datadog and we'll go from there.

If I'm going to add a function to report the metrics, we might as well send them to Ping Centre using the heartbeat ping like originally planned. Not sure what we get by adding datadog to the mix


Absolute image paths should be prefixed with "." not the current directory

Absolute image paths should be prefixed with "." not the current directory

. denotes the current directory, so prefixing absolute paths with it would be pretty strange. What is the problem exactly?

Because absolute paths won’t work when loaded into E+, since the module is actually under /module/<id>/.
But when using an absolute path for proper rendering in VSCode, the root is the root of the module.


Throw an exception if factory parameters are not an array

We are running into an issue in production where we randomly get a Error: Unsupported operand types exception about this line:
https://github.com/predis/predis/blob/4c1aada5aebf0afde266b512ed069f822148c3a2/src/Connection/Factory.php#L151
The exception added by this PR will hopefully help us determine the cause of $parameters not being an array.

What version are you running in production?

We are running 1.1.7

Okay, please re-submit this PR to the v1.1 branch.


ew202350 GT-FluFNP submission

There are no issues with your submission. I'd just like to confirm that you intend to submit forecasts generated prior to today's truth data update. Is this correct? - Alex


Update FAQ page

Update the answer to the question: What if I need to customize what fit() does?

Thanks for the review!


Pass on more headers (List-*, OpenPGP).

Some more headers to pass on.
Would make #24 obsolete.

Is there any reason beyond lack of time not to merge this?

I finally merged this, thanks!


The status reported by AppManager may be wrong

On some devices, when the AppManager reports the status of the app to ood-daemon by accident, there may be a timing problem. For example, the two consecutive reports in the log below should be starting->running to be correct, so should to confirm that the report in the AppManager is Serialization or not, especially when rapid state changes result in two consecutive reports
Logs

[2023-03-29 12:09:01.467420 +00:00] INFO [ThreadId(2)] [service/ood-daemon/src/status/status.rs:169] recv server status report: service=9tGpLNnBAfLV9kPnVwAfcWZEQCFPSV9LBUyWJc419MHP, valuje={"auto_update":true,"id":"9tGpLNnBAfLV9kPnVwAfcWZEQCFPSV9LBUyWJc419MHP","name":"CYFS DSG Service","process_state":"Running","version":"1.0.4"}
[2023-03-29 12:09:01.475972 +00:00] INFO [ThreadId(2)] [service/ood-daemon/src/status/status.rs:169] recv server status report: service=9tGpLNnBAfLV9kPnVwAfcWZEQCFPSV9LBUyWJc419MHP, valuje={"auto_update":true,"id":"9tGpLNnBAfLV9kPnVwAfcWZEQCFPSV9LBUyWJc419MHP","name":"CYFS DSG Service","process_state":"Starting","version":"1.0.4"}

fixed at 0fdf644b


Minor update to support adobe/aio-cli-config

Description
This minor update:

adds configuration support using the https://github.com/adobe/aio-cli-config module.
adds edit command
adds list command
adds flags to existing commands to save to global or project configuration storage

Related Issue
This supports the new config module https://github.com/adobe/aio-cli-config

How Has This Been Tested?
100% jest coverage

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

oops. lets maybe wait for he module to be on npm.


Support Matrix and Supported Configurations in Public Docs

Today: We do have very limited information on our public docs about supported configurations and installation requirements. The information that exists is distributed across several pages.
Expectations: A central place that customers can access and find all information about supported configurations and installation requirements.
Examples:
Terraform Enterprise https://developer.hashicorp.com/terraform/enterprise/flexible-deployments/install
JBoss EAP: https://access.redhat.com/articles/2026253
Example link: Like oracle, we can simply note that certain configurations are untested yet.
Compatibility Matrix

IMO this needs to be partially owned by the SA/SMLE team along with Eng & Docs (like how Marc had owned the last significant update). This is not something that consultants can or should do.

I'm removing this from consultants queue.


java ci should capture full stack traces in the test run logs

To see the full stack trace of the errors, re-run Maven with the -e switch.

It would be nice if the CI pipelines were configured to run mvn.cmd with -e for full stack traces in the output logs:
https://dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_apis/build/builds/122129/logs/85
2019-10-01T22:36:06.1361408Z org.spockframework.runtime.WrongExceptionThrownError: Expected exception of type 'com.azure.storage.blob.models.StorageException', but got 'java.lang.IllegalStateException'
2019-10-01T22:36:06.1367760Z 	at com.azure.storage.blob.specialized.BlockBlobAPITest.Buffered upload network error(BlockBlobAPITest.groovy:1073)
2019-10-01T22:36:06.2114125Z Caused by: java.lang.IllegalStateException: BufferedUpload thread interrupted. Thread:16
2019-10-01T22:36:06.2120190Z
2019-10-01T22:36:11.6082231Z [ERROR] Failures:
2019-10-01T22:36:11.6083914Z [ERROR]   BlockBlobAPITest.Buffered upload network error:1073->SpecInternals.thrownImpl:67->SpecInternals.checkExceptionThrown:80 Expected exception of type 'com.azure.storage.blob.models.StorageException', but got 'java.lang.IllegalStateException'
2019-10-01T22:36:11.6084928Z [ERROR] Tests run: 1062, Failures: 1, Errors: 0, Skipped: 16
2019-10-01T22:36:46.3978058Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project azure-storage-blob: There are test failures.
2019-10-01T22:36:46.3984044Z [ERROR]
2019-10-01T22:36:46.3985221Z [ERROR] Please refer to D:\a\1\s\sdk\storage\azure-storage-blob\target\surefire-reports for the individual test results.
2019-10-01T22:36:46.3986159Z [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
2019-10-01T22:36:46.3986755Z [ERROR] -> [Help 1]
2019-10-01T22:36:46.3987238Z [ERROR]
2019-10-01T22:36:46.3987792Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2019-10-01T22:36:46.3988288Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2019-10-01T22:36:46.3989328Z [ERROR]
2019-10-01T22:36:46.3989895Z [ERROR] For more information about the errors and possible solutions, please read the following articles:
2019-10-01T22:36:46.3990566Z [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
2019-10-01T22:36:46.3991095Z [ERROR]
2019-10-01T22:36:46.3991687Z [ERROR] After correcting the problems, you can resume the build with the command
2019-10-01T22:36:46.3992224Z [ERROR]   mvn <goals> -rf :azure-storage-blob
2019-10-01T22:36:46.5514746Z The process 'C:\ProgramData\chocolatey\lib\maven\apache-maven-3.6.1\bin\mvn.cmd' failed with exit code 1

@mitchdenny is this something you could assist with?

This would be mitigated by implementing https://github.com/Azure/azure-sdk-for-java/issues/5441 with AZURE_LOG_LEVEL = 3 or 2.
The console output may not output it, but the surefire logs will have all the spew, environment variables used, etc.

Yep -- this is something that we can do. I was thinking of trying to standardize where we put diagnostic logs like this, so if we have a path where files can go that guarantees that those files will be lifted into an artifact for each job leg.
We need to be a little bit careful about accidentally leaking credentials - we try to be sensible but you never know what crazy stuff some tools put in their log files.

This is very similar to https://github.com/Azure/azure-sdk/issues/658.  I think the thing we want here is to produce verbose logs as a build artifact (not pump them into the console output of the build).

Josh - it looks like the full stack is being uploaded into Azure Pipelines:

Going to close this since it is capturing the detail in the test results, which are then published into pipelines. Let me know if this is not what you are seeing.


Add more logging when setting headers.

Thanks! I'd prefer separate PRs for separate things (like the pep8 fixes) in the future.

I'll figure out how to do separate pull requests in future - I didn't realise the subsequent commits would be added to one puu request.  Thanks :)


[CadlValidation CI] Require package-lock.json be uploaded to TypeSpec PRs

Require package-lock.json to be uploaded to have a list of packages and their deps per service specification

@catalinaperalta: Can you please add more details?  Is this about adding a new rule to the "Cadl Validation" step to show an error if a folder is missing a package-lock.json?  Or is this about adding package-lock.json files to the repo to pin package versions?

The initial ask behind this issue was to add an error if a folder is missing the package-lock.json so that we can have a source of consistent dependencies for folks checking in work now.

@catalinaperalta: I believe this issue is obsoleted by the new design which uses a package.json and package-lock.json in the repo root.  Please re-open if you disagree.
CC: @weshaggard


Increase contrast for logo in dark mode

right now it's pretty much unreadbable 😕
a white or gray bg would get great!
I'm willing to open a pr ❤ just want to make sure it wont be stuck in limbo 😅

Thanks for the suggestion, I've actually remade the logo and the splash image so that it fixes the contrast and just haven't updated the main branch yet.


Custom Logo

Any thought on being able to upload/change the default bar logo? Perhaps under Settings > Profile > #3 Bar ??  Be nice to show a more customized icon.

This is now available when editing a bar, let me know if there are any issues


Add debug error test to demonstrate connection closed

Description
Closes #2394.

The test fails either one of following error. This demonstrates connection is closed by internal listener.
 setup.go:246:
        	Error Trace:	/home/chi/go/src/github.com/chilagrow/FerretDB/integration/setup/setup.go:246
        	            				/usr/local/go/src/testing/testing.go:1150
        	            				/usr/local/go/src/testing/testing.go:1328
        	            				/usr/local/go/src/testing/testing.go:1570
        	Error:      	Received unexpected error:
        	            	connection(<IP_ADDRESS>:44693[-7]) socket was unexpectedly closed: EOF
        	Test:       	TestDebug

Or it may fail if partial message is read like below.
 setup.go:246:
        	Error Trace:	/home/chi/go/src/github.com/chilagrow/FerretDB/integration/setup/setup.go:246
        	            				/usr/local/go/src/testing/testing.go:1150
        	            				/usr/local/go/src/testing/testing.go:1328
        	            				/usr/local/go/src/testing/testing.go:1570
        	Error:      	Received unexpected error:
        	            	connection(<IP_ADDRESS>:44799[-7]) incomplete read of message header: read tcp <IP_ADDRESS>:40116-><IP_ADDRESS>:44799: read: connection reset by peer
        	Test:       	TestDebug

What happens:

It finds unrecoverable error https://github.com/FerretDB/FerretDB/blob/main/internal/clientconn/conn.go#L443
It stops connection https://github.com/FerretDB/FerretDB/blob/main/internal/clientconn/conn.go#L361
the connection is no longer available to test client https://github.com/FerretDB/FerretDB/blob/main/internal/clientconn/listener.go#L309
Then test client tries to do clean up database etc using the closed connection. So it fails.

Readiness checklist

[ ] I added/updated unit tests.
[ ] I added/updated integration/compatibility tests.
[ ] I added/updated comments and checked rendering.
[ ] I made spot refactorings.
[ ] I updated user documentation.
[ ] I ran task all, and it passed.
[ ] I ensured that PR title is good enough for the changelog.
[ ] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
[ ] I marked all done items in this checklist.

Duplicate of #2394

Duplicate of #2564


Change healthcheck URL to do method + path + version + host header

Server - build from master on Apr 7
In "Add Load Balancer" page , "Check URL" has "http://example.com/health-check" as helper text.
The URI passed should be of the format - "http method" "uri" . Example of the health check uri i used - GET /hostname.html.
Also "Check URL" should be renamed to "Check URI"

Vince, if HTTP method is not specified, its defaulted to OPTION on the haproxy side. And if option requests are rejected by the target web server, the request is always gonna fail. We might wanna show the dropdown with available methods to an end user - OPTION/GET/.. and construct the URI parameter as  .

Ok that's kind of weird to call the field URI if it's really method+path+http version+http headers...
But anyway no normal human understands URL vs URI (and this isn't it), so it was intentional to label it URL.
Changing it to a drop down of methods and labeling it as a path + whatever other options are available sounds good, I can assemble them into the format haproxy wants.  I think the field should be called httpRequest or something in the API though.

@vincent99 I directly translated the naming from haproxy..wasn't a good idea. What about "heathcheckRequest" for the name? Or would it be a tautology considering its the field of the object called healthCheck

Are you doing any special processing or do you just do option httpchk ${uri}? The full form is several different properties for haproxy:
option httpchk <method> <uri> <version>
In the HTTP spec it's called the "Request-Line":
Request-Line   = Method SP Request-URI SP HTTP-Version CRLF
So how about calling it requestLine, and as far as naming is concerned we'll ignore the part where <version> might also contain CRLF and some HTTP headers.

@vincent99 ok, lets call it request line as I just put "httpcheck " to the haproxy config

UI changed to produce method + uri + version + optional Host header.  The field in the API isn't renamed yet.

v0.16.2 (UI / latest)
Verified Fixed

Repoening to reverify because the field was renamed from uri -> requestLine...

v0.18.1
Verified Fixed


voltage difference after first step between coreneuron.cell_permute 0 and 1

Describe the issue
Puzzling, though trivial, difference in voltage between coreneuron.cell_permute = 0 and = 1 after single step for a ringtest model.
I.e, the single voltage difference is
66c66
< 18 -63.9930201641485
---
> 18 -63.9930201641486

To Reproduce
NEURON 8.0a-657-g464541abb
cmake .. -G Ninja -DCMAKE_INSTALL_PREFIX=install -DPYTHON_EXECUTABLE=`pyenv which python` -DNRN_ENABLE_RX3D=OFF -DCMAKE_BUILD_TYPE=Debug -DNRN_ENABLE_CORENEURON=ON -DNRN_ENABLE_TESTS=ON -DCMAKE_C_COMPILER=gcc-9 -DCMAKE_CXX_COMPILER=g++-9

ninja install

Modify external/tests/ringtest as follows
diff --git a/ringtest.py b/ringtest.py
index 0c87f5d..04abf92 100644
--- a/ringtest.py
+++ b/ringtest.py
@@ -91,6 +91,7 @@ def prun(tstop):

     # run simulation
     pc.psolve(tstop)
+    pc.prcellstate(10, "dt")

     # calculate various time statistics for profiling/debugging

@@ -151,6 +152,8 @@ if __name__ == '__main__':
         coreneuron.enable = True
         coreneuron.file_mode = coreneuron_file_mode
         coreneuron.gpu = coreneuron_gpu
+        coreneuron.prcellstate = 10
+        coreneuron.cell_permute = 0

         if args.multisplit is True:
             print("Error: multi-split is not supported with CoreNEURON\n")
@@ -174,7 +177,13 @@ if __name__ == '__main__':
     ## Initialize ##

     pc.set_maxstep(10)
-    h.stdinit()
+    rdm = h.Random()
+    rdm.Random123(5,5,5)
+    rdm.uniform(-70., -60.)
+    for sec in h.allsec():
+        for seg in sec.allseg():
+            seg.v = rdm.repick()
+    h.finitialize()
     timeit("initialized", settings.rank)

Run twice with coreneuron.cell_permute = 1 and coreneuron.cell_permute = 0
python ringtest.py  -rparm -tstop 0.025 -npt 1 -compart 1 3  -coreneuron

After the first run using permute 1, move corenrn files to new folder p1
After second run with permute 0, view the voltage difference with
diff 10_cpu_t0.025000.corenrn p1/10_cpu_t0.025000.corenrn|more

Expected behavior
No differences at all between the files. Because cell_permute 0 and 1 should not affect the gaussian elimination order of operations within an individual cell.
System (please complete the following information)

OS: Ubuntu 18.4
Compiler: gcc version 9.3.0
Version: NEURON 8.0a-657-g464541abb and CoreNEURON HEAD detached at 2b3e746
Backend: x86_64

Additional context
Add any other context about the problem here.

Note no difference in
hines@hines-T7500:~/neuron/cellperm1/external/tests/ringtest$ diff 10_cpu_t0.000000.corenrn p1/10_cpu_t0.000000.corenrn

increasing the prcellstate resolution
hines@hines-T7500:~/neuron/cellperm1/external/coreneuron/coreneuron/io$ git diff
diff --git a/coreneuron/io/prcellstate.cpp b/coreneuron/io/prcellstate.cpp
index c397dab..b60f37d 100644
--- a/coreneuron/io/prcellstate.cpp
+++ b/coreneuron/io/prcellstate.cpp
@@ -18,7 +18,7 @@
 #include "coreneuron/coreneuron.hpp"
 #include "coreneuron/utils/nrnoc_aux.hpp"

-#define precision 15
+#define precision 18
 namespace coreneuron {

and doing the tests gives two location differences for voltage.
hines@hines-T7500:~/neuron/cellperm1/external/tests/ringtest$ diff 10_cpu_t0.025000.corenrn p1/10_cpu_t0.025000.corenrn|more
66c66
< 18 -63.9930201641485468
---
> 18 -63.9930201641485539
81c81
< 33 -63.7966426029935008
---
> 33 -63.7966426029935079

It turns out that child order for cell_permute=1 can differ from the child order for cell_permute=0. This means that rhs and d can differ because of the statements in coreneuron/sim/treeset_core.cpp
vec_rhs[parent_index[i]]` += vec_a[i] * dv; and vec_d[parent_index[i]] -= vec_a[i];. Because of the order difference, further round off errors will accumulate differently during gaussian elimination.

Ok. This should be closed then? i.e. no changes needed?

Here is an example of the first substantive difference in calculation of rhs that child ordering differences can have. Note it is sandwiched between neighbors that are the same in final value to %.18g. The integer lines are the unpermuted parent index followed by the unpermuted child indices. The floating point lines are the value of rhs prior to the addition of the contribution from the first child, followed by contribution and result of addition of the child. For cell_permute 0 and 1 the contributions of each child are the same but the final result of addition of all children can add up differently.
< 403 788 790
<   0.0545115931425236705 + -0.769440542653670012 = -0.714928949511146383 + -0.427811064428813337 = -1.14274001393995972
---
> 403 790 788
>   0.0545115931425236705 + -0.427811064428813337 = -0.373299471286289652 + -0.769440542653670012 = -1.14274001393995972
811,812c811,812
< 405 792 795
<   0.0312458529786176602 + 0.404870216714824094 = 0.436116069693441744 + 0.627417826098806808 = 1.06353389579224844
---
> 405 795 792
>   0.0312458529786176602 + 0.627417826098806808 = 0.658663679077424513 + 0.404870216714824094 = 1.06353389579224866
829,830c829,830
< 414 807 809
<   0.016770234989009921 + -0.212156687676731598 = -0.195386452687721673 + 0.313066693714535493 = 0.11768024102681382
---
> 414 809 807
>   0.016770234989009921 + 0.313066693714535493 = 0.329836928703545418 + -0.212156687676731598 = 0.11768024102681382

Also note that the calculation of permuted NrnThread._v_parent_index has been verified by storing the original unpermuted vector transferred from NEURON to int* orig_parent  and showing that
int* p = nt._permute;
int* ip = inverse_permute(p, nt.end);
for (int i=0; i < nt.end; ++i) {
  if (i >= nt.ncell) {
    assert(nt._v_parent_index[i] == p[orig_parent[ip[i]]]);
  }else{
    assert(nt._v_parent_index[i] == -1);
  }
}

It is a surprise to me that the interleaved cell_permute=1 calculation resulted in a permutation that can reverse the order of children. I think it is worthwhile to trace the cause of this and, if possible, keep the node index ordering of each cell the same for
cell_permute=0 and cell_permute=1.  It is also worthwhile to understand the large first step discrepancies (in the 6th or 7th decimal place) which initiated this investigation and resulted in merging neuronsimulator/nrn#1477


"COPY FROM" is bypassing type validation

CrateDB version:
4.1.7
Environment description:

local CrateDB instance
Number of nodes: 1

Problem description:
By inserting data with COPY FROM ... from S3 it is possible to write string values (tested with empty strings) into boolean columns. By querying such data SQLActionException[SQLParseException: Can't convert "" to boolean] is returned. I haven't tested this issue with different types or different values.

Steps to reproduce:
# run crate
docker run -ti --rm -p 4200:4200 crate:4.1.7

# create a table with a boolean column
create table x (y boolean);

# create a json file to be imported inside the container. Note the string value
echo '{"y": ""}' > /tmp/x.json

# import that file
COPY x FROM 'file:///tmp/x.json';

# Try to query the data
select * from x

This is currently documented behavior

CrateDB does not check if the column’s data types match the types from the import file. It does not cast the types but will always import the data as in the source file. Furthermore CrateDB will only check for primary key duplicates but not for other Column Constraints like NOT NULL.

But I guess we should consider changing that behavior to avoid issues like this.


[SPARK-31116][SQL] Fix nested schema case-sensitivity in ParquetRowConverter

What changes were proposed in this pull request?
This PR (SPARK-31116) add caseSensitive parameter to ParquetRowConverter so that it handle materialize parquet properly with respect to case sensitivity
Why are the changes needed?
From spark 3.0.0, below statement throws IllegalArgumentException in caseInsensitive mode because of explicit field index searching in ParquetRowConverter. As we already constructed parquet requested schema and catalyst requested schema during schema clipping in ParquetReadSupport, just follow these behavior.
val path = "/some/temp/path"

spark
  .range(1L)
  .selectExpr("NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS StructColumn")
  .write.parquet(path)

val caseInsensitiveSchema = new StructType()
  .add(
    "StructColumn",
    new StructType()
      .add("LowerCase", LongType)
      .add("camelcase", LongType))

spark.read.schema(caseInsensitiveSchema).parquet(path).show()

Does this PR introduce any user-facing change?
No. The changes are only in unreleased branches (master and branch-3.0).
How was this patch tested?
Passed new test cases that check parquet column selection with respect to schemas and case sensitivities

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #119711 has started for PR 27888 at commit 0de8d20.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24440/
Test PASSed.

Thank you for making your first contribution, @kimtkyeom .

cc @cloud-fan , @gengliangwang and @yhuai

cc @rxin as a release manager for 3.0.0. (Also, cc @gatorsmile )

Is this a regression in 3.0? If it is, do you know which commit/PR caused it?

@cloud-fan After #22880, IllegalArgumentException is thrown.

is it possible to normalize column names before entering these low-level parquet classes?

Maybe possible in 'ParquetReadSupport' similar to clipping parquet requested schema then generate normalized caralystRequestedSchema I think. But i could not found clean way to normalize it, espacially normalization through Array type is quite complicated.

Test build #119711 has finished for PR 27888 at commit 0de8d20.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119711/
Test PASSed.

For SPARK-25407, the PR will be #24307 . I took over @mallman 's PR (#22880) at that time by #24307 .
cc @dbtsai , too.

Test build #119734 has started for PR 27888 at commit d6cbb67.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24463/
Test PASSed.

@kimtkyeom, SPARK-4502 is in 2.4.0. Does that mean this regression exists in Spark 2.4.x too?

@HyukjinKwon I'm not deeply investigated on 2.4.x, but i'm not experienced this issue in 2.4.x.
We currently use 2.4.3 and 2.4.4 with spark.sql.optimizer.nestedSchemaPruning.enabled to true, but this exception is not thrown.

Thanks for confirmation, @kimtkyeom

Test build #119735 has started for PR 27888 at commit 9816f66.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24464/
Test PASSed.

Test build #119742 has started for PR 27888 at commit 5e6dd75.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24471/
Test PASSed.

Test build #119734 has finished for PR 27888 at commit d6cbb67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119734/
Test PASSed.

Test build #119735 has finished for PR 27888 at commit 9816f66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119735/
Test PASSed.

Test build #119742 has finished for PR 27888 at commit 5e6dd75.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119742/
Test FAILed.

Retest this please.

Test build #119773 has started for PR 27888 at commit 5e6dd75.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24503/
Test PASSed.

Test build #119773 has finished for PR 27888 at commit 5e6dd75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119773/
Test PASSed.

Hi, @kimtkyeom . Could you revise once more?

Hi, @kimtkyeom . Could you revise once more?

@dongjoon-hyun OK. I'll revise soon.

Test build #119779 has started for PR 27888 at commit fb4880b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24509/
Test PASSed.

Test build #119779 has finished for PR 27888 at commit fb4880b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119779/
Test FAILed.

The fix seems fine except the comments above. @dbtsai, WDYT about disabling nested pruning by default for Spark 3.0 (see https://github.com/apache/spark/pull/27888#pullrequestreview-373986169)?

Test build #119838 has started for PR 27888 at commit 4bac37f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24568/
Test PASSed.

Test build #119838 has finished for PR 27888 at commit 4bac37f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119838/
Test FAILed.

Test build #119845 has started for PR 27888 at commit 6055d22.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24575/
Test PASSed.

Test build #119846 has started for PR 27888 at commit 9709e96.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24577/
Test PASSed.

For your removed test code, if you think that is missed in the existing test coverage, you can make an independent [TESTS] PR . That is also a helpful.
For the bug-fix PR, we should focus the exact bug only. That's the reason why I recommend you to remove the irrelevant ones.

For @HyukjinKwon 's comment (https://github.com/apache/spark/pull/27888#issuecomment-599354017), yes, I agree with your concern. We can do that before 3.0.0 release. For now, shall we keep it until RC1 in order to get more active feedback? BTW, I'm not sure we can have RC1 in this month.

Sure, +1 more feedback during the RC and for the advices about tests as well.

Test build #119852 has started for PR 27888 at commit ca07f74.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24582/
Test PASSed.

BTW, @kimtkyeom . While reviewing this PR again, the failure report ORC looks wrongs.

https://github.com/apache/spark/pull/27888#discussion_r391521516

You wrote like the following, but it was JSON (Relation[StructColumn#329] json). Could you confirm your failure report again?
 Could you confirm your failure report again?
ORC passed case insensitive test cases, but it failed case sensitive manner.

[info] - [SPARK-31116](https://issues.apache.org/jira/browse/SPARK-31116): Select nested columns correctly in case sensitive manner *** FAILED *** (871 milliseconds)
[info]   Results do not match for query:
[info]   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
[info]   Timezone Env:
[info]
[info]   == Parsed Logical Plan ==
[info]   Relation[StructColumn#329] json
[info]
[info]   == Analyzed Logical Plan ==
[info]   StructColumn: struc

@dongjoon-hyun Ah, sorry I mis-pasted test result. ORC also shows same result as following whatever value of 'spark.sql.optimizer.nestedSchemaPruning.enabled`
[info] - SPARK-31116: Select nested columns correctly in case sensitive manner *** FAILED *** (905 milliseconds)
[info]   Results do not match for query:
[info]   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
[info]   Timezone Env:
[info]
[info]   == Parsed Logical Plan ==
[info]   Relation[StructColumn#331] orc
[info]
[info]   == Analyzed Logical Plan ==
[info]   StructColumn: struct<LowerCase:bigint,camelcase:bigint>
[info]   Relation[StructColumn#331] orc
[info]
[info]   == Optimized Logical Plan ==
[info]   Relation[StructColumn#331] orc
[info]
[info]   == Physical Plan ==
[info]   FileScan orc [StructColumn#331] Batched: false, DataFilters: [], Format: ORC, Location: InMemoryFileIndex[file:/Users/kimtkyeom/Dev/spark_devel/target/tmp/spark-f1fb325e-9ff3-4945-81c7-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>>
[info]
[info]   == Results ==
[info]
[info]   == Results ==
[info]   !== Correct Answer - 1 ==   == Spark Answer - 1 ==
[info]   !struct<>                   struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>>
[info]   ![null]                     [[null,null]] (QueryTest.scala:248)

That's a correct behavior in case-sensitive mode, isn't it? Given the original schema NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS StructColumn, StructColumn is matched and LowerCase and camelcase is not matched. So, the result is a struct with two null fields.

That's a correct behavior in case-sensitive mode, isn't it? Given the original schema NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS StructColumn, StructColumn is matched and LowerCase and camelcase is not matched. So, the result is a struct with two null fields.

@dongjoon-hyun Ah got it. There is no failure except above case. (BTW, I think materialization of rows with nested column where non-matched column would be consistent regardless of file format, but that is out of this PR scope.)

Test build #119863 has started for PR 27888 at commit 44a9e72.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24593/
Test PASSed.

Test build #119846 has finished for PR 27888 at commit 9709e96.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119846/
Test FAILed.

Test build #119845 has finished for PR 27888 at commit 6055d22.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119845/
Test FAILed.

Test build #119865 has started for PR 27888 at commit a0d9a19.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24595/
Test PASSed.

Test build #119852 has finished for PR 27888 at commit ca07f74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119852/
Test PASSed.

Test build #119863 has finished for PR 27888 at commit 44a9e72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119863/
Test PASSed.

Test build #119865 has finished for PR 27888 at commit a0d9a19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master/3.0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119865/
Test PASSed.

Thank you so much for your first contribution, @kimtkyeom . I added you kimtkyeom (JIRA ID) to Apache Spark Contributor Group and assigned you SPARK-31116. We are looking forward to see more contributions from you and DevSisters!

Thanks for all reviewers with all generous review & comments! :)

+1, LGTM.


Unable to assign values when passing an array of variants to a method.

I am unable to assign values when passing an array of variants to a method.
When I assign an arrayvalue to a field, I get the following error:
Typeconversion of the expression Microsoft.Dynamics.Nav.Runtime.NavIndirectValue to a Microsoft.Dynamics.Nav.Runtime.NavText-value is not possible.

AL Extension Version: 0.16.19969
procedure ArrayVarTest();
var
VariantArray : array [3] of Variant;
begin
VariantArray[1] := 1;                //Option(integer)
VariantArray[2] := 2.99;           //Decimal
VariantArray[3] := 'TestText';   //Text
PassArrayToMethod(VariantArray);
end;
procedure PassArrayToMethod(VariantArray : array [3] of Variant);
var
OptionField : Option;
DecimalField : Decimal;
TextField : Text;
begin
OptionField := VariantArray[1];
DecimalField := VariantArray[2];
TextField := VariantArray[3];
end;

Is this being investigated? Or should I try and think of a workaround?

This is a potential bug and on our investigation backlog. We have a lot of cases right now so can't comment on the resolution time.

Is this working with C/AL?

Yes this should work in C/AL.

Hi @DavyVerbeek! Thanks for reporting this issue. With VS Code version 1.28.2, AL Extension version 2.0.48254 and the latest cloud sandbox, I can't reproduce your issue. Here is my AL file:
pageextension 50150 MyExtension extends "Customer List"
{
    trigger OnOpenPage();
    var
        VariantArray: array[3] of Variant;
    begin
        VariantArray[1] := 1; //Option(integer)
        VariantArray[2] := 2.99; //Decimal
        VariantArray[3] := 'TestText'; //Text
        PassArrayToMethod(VariantArray);
    end;

    procedure PassArrayToMethod(VariantArray: array[3] of Variant);
    var
        OptionField: Option;
        DecimalField: Decimal;
        TextField: Text;
    begin
        OptionField := VariantArray[1];
        DecimalField := VariantArray[2];
        TextField := VariantArray[3];

        Message(Format(OptionField));
        Message(Format(DecimalField));
        Message(TextField);
    end;
}

The values are printed correctly here. Can you try this out again? If you are running another version of BC / NAV, please contact the regular support channel:
You can open Support Request to CSS through PartnerSource portal or contact your Service Account Manager (SAM) in the local subsidiary to understand what is included in your contract as of support incident and PAH (Partner Advisory Hours). Your SAM might also step by step direct you how to open a support request or how to get credentials if this is the first time for you / your company. For more details please read:

https://blogs.msdn.microsoft.com/nav/2018/05/03/al-developer-previews-multiple-releases-and-github/
https://github.com/Microsoft/AL/wiki/Frequently-Asked-Questions#i-found-an-issue-on-nav-2018-what-do-i-do

Thanks again for reporting this issue.

Closing due to the fact that we couldn't reproduce the issue and for lack of additional input.


error in README instructions for cloud-reader

instructions read: "a sibling epub_content/ folder is expected to contain exploded or zipped EPUBs"
actually, the cloud-reader folder is the parent folder, and Readium looks for the epub_content folder within the cloud-reader folder (not a sibling folder).
so, would expect to see: "a child epub_content/ folder is expected to contain exploded or zipped EPUBs", or similar.

Thanks for the bug report!
Reference:
https://github.com/readium/readium-js-viewer/blob/develop/README.md#cloud-reader-deployment
"
The contents of the cloud-reader distribution folder (see section above) can be uploaded to an HTTP server as-is (either in the http://domain.com/ root, or any subfolder path http://domain.com/reader/), and a sibling epub_content/ folder is expected to contain exploded or zipped EPUBs
"

Corrected and clarified (I hope).  551ba2


Preocts 20220612 pyproject

Round two of pyproject.toml upgrade. This time we are using flit as the build engine as it seems smoother with the toml config than setuptools.
This change moves all configs into pyproject.toml to see what all the hype is about.

flake8 args added to pre-commit-config because flake8 doesn't support pyproject.toml
point of entry script added, this now installs with pip install . (that's neat)
Cleaned up readme and added notes

closes #3
closes #5


Feature Request: GPT2 fine tuning

Yes, feel free to open a PR if you want.
It's just a regular PyTorch model so all the standard ways of training a PyTorch model work.

Is is possible to fine-tune GPT2 on downstream tasks currently?

same questions


Uploading files to GCS via WriteChannel cause the memory to increase - memory not released

Hi
I am uploading files to GCS bucket using WriteChannel
WriteChannel writeChannel =  storage.writer(blobInfo);
writeChannel.write(byteBuffer);
writeChannel.close();
After many files being uploaded the memory increased and doesn't released.
Thanks,
Itay

Hi @ihudedi,
When creating a WriteChannel an internal buffer is created to perform chunking of the upload to GCS.
By default, the buffer size is 16MiB. Depending on which JVM you're using along with which garbage collector, the allocated buffers can sometimes be immediately placed into the Old Generation of the heap.  If you're app is not generating enough garbage to cause the Old Generation to need a full collection often the JVM won't.
If the objects you are uploading are smaller than 16MiB, it probably makes sense to reduce that size by doing
try (WriteChannel writeChannel = storage.writer(info)) {
  writeChannel.setChunkSize(4 * 1024 * 1024); // 4MiB instead of the default of 16MiB
  writeChannel.write(byteBuffer);
}

If you're able, could you share which JVM, version and GC settings you're using?

Hi @BenWhitehead
java version "11.0.10" 2021-01-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)
-XX:+UseParallelGC -XX:-TieredCompilation -Xmx2048m -Xshare:off -XX:MaxDirectMemorySize=512m -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError

Can you try adding -Djdk.nio.maxCachedBufferSize=33554432 to your command line? I just learned that in java11 there is a jdk built in ByteBuffer cache which attempts to help, but since storage uses large buffers by java standards this could be causing the resident memory of the process to be inflated. Setting this system property tells the jvm to limit it's cache to 32MiB ( you could also choose a smaller number if it makes sense for your application).

Closing this due to lack of response. Please open a new issue if you are still experiencing this issue.


too large sidebar under the star icon

The star button on the main map pulls up a new sidebar. This sidebar is too big for a small ipad and the top portion is unreachable as this doesn’t scroll.

Fixed with commit 105ff8df2e50335ff20dc266ab05bd1bc0c4d16d.
@JEAndrew let me know if this works alright for you.


Dependency requirements do not allow for integration with Phoenix v1.3.0-rc

When running mix deps.get (Phoenix 1.3.0-rc.2):
Failed to use "phoenix" (version 1.3.0-rc.2) because
  terraform (version 1.0.0) requires >= 1.2.0 and < 2.0.0 *
  mix.lock specifies 1.3.0-rc.2

* This requirement does not match pre-releases. To match pre-releases include a pre-release in the requirement, such as: "~> 2.0-beta".

Perhaps change {:phoenix, ">= 1.2.0 and < 2.0.0"} to {:phoenix, ">= 1.2.0 and < 2.0.0 or ~> 1.3.0-rc"}

Thanks! Would you like to open a PR for it?


TODO item: Implement other VCS: Hg, Bzr, Svn

If we decide to support these VCSs.

I think each of these should be broken out into their own issue.


Auto-hide sidebar

VSCode Version: 1.3.0-insider
OS Version: Win 10

Currently, when you open a side panel and then click back in the editor area, the side panel remains open.
It would be rather handy to add a button (pin or something) which changes this behavior for each panel, allowing to either keep the current way of working or making the side panel auto-collapse when you click outside.

cc @bpasero @stevencl @bgashler1

For the git-panel, it could be useful to have something like "auto-hide if there is nothing to show", or even better yet "go back to last side-panel if there is nothing to show".
I'm often finding myself switching from explorer-panel to git-panel, after my commit I am being left with an empty git panel, and have to click the explorer panel again to get back into my workflow.
This would be a lot more comfortable if the git-panel would yield automatically to the explorer-panel.
I realize this is rather unconventional from the UX perspective, even more so since there isnt (at least for me) any other panel apart from git-panel where this would make sense.
Just throwing the idea out there, maybe an UX expert can make something out of it.

https://github.com/Microsoft/vscode/issues/3742

Well, I'm refrencing more to the Visual Studio panels 'auto-hide' : you click on it  - it shows, you click outside - it goes ... simple as that. Hovering/timeouts etc. are not really necessary, but anyway - #3742 is similar, I must admit.
Btw., the GIT sidepannel must be totally hidden (no icon on the bar at all) when git is disabled in settings, I think.

Went ahead and made an extension for this: https://marketplace.visualstudio.com/items?itemName=venryx.vscode-vtools
It's not perfect -- it only closes the side-bar when the user changes the selection (eg, clicking or dragging at a new location) -- but it achieves the basic function.
Built-in support is still ideal, but this should help until then.


ctr run --detach causes --rm to have no effect

Description
When doing ctr run --detach --rm ..., the --rm has no effect because the ctr process has ended after detaching from the launched container, before the container exits and would have been removed.
Steps to reproduce the issue

ctr run --rm --detach docker.io/library/hello-world:latest test
ctr container ls

Describe the results you received and expected
The test container still exists. It should have been deleted after finishing.
This is probably not possible, in which case specifying --rm --detach should at least produce a warning, possibly an error.
What version of containerd are you using?
containerd github.com/containerd/containerd v1.6.13 78f51771157abb6c9ed224c22013cdf09962315d
Any other relevant information
No response
Show configuration if it is related to CRI plugin.
No response

Yes, this isn't really possible unless we add some new service that can monitor containers and clean things up.

I think it could be consistent with nerdctl and return an errors.New("flag -d and --rm cannot be specified together")
/assign


Several errors occurred in a few plugins

When I execute mac_apt_mounted_sys_data.py (which was cloned on Feb. 3, 2021), QUICKLOOK plugin and SAVEDSTATE plugin.
QUICKLOOK plugin error:

2021-02-04 15:51:20|MAIN|INFO|Running plugin QUICKLOOK
2021-02-04 15:51:20|MAIN.QUICKLOOK|INFO|QuickLook Cache data found!
2021-02-04 15:51:20|MAIN|ERROR|An exception occurred while running plugin - QUICKLOOK
Traceback (most recent call last):
File "./mac_apt_mounted_sys_data.py", line 217, in
plugin.Plugin_Start(mac_info)
File "/Users/macforensics/Documents/GitHub/mac_apt/plugins/quicklook.py", line 472, in Plugin_Start
parseDbNew(c, quicklook_array, quicklook_db_path, thumbnail_file, mac_info, user)
File "/Users/macforensics/Documents/GitHub/mac_apt/plugins/quicklook.py", line 372, in parseDbNew
apfs_c = export.apfs_db.conn.cursor()
AttributeError: 'MountedMacInfoSeperateSysData' object has no attribute 'apfs_db'

SAVEDSTATE plugin error:

2021-02-04 15:51:20|MAIN|INFO|Running plugin SAVEDSTATE
2021-02-04 15:51:20|MAIN.HELPERS.MACINFO|ERROR|Exception in _GetFileSizeNoPathMod() : [Errno 2] No such file or directory: '/Volumes/Macintosh HD - Data/Users/macforensics/Library/Saved Application State/com.apple.calculator.savedState'
2021-02-04 15:51:20|MAIN.HELPERS.MACINFO|ERROR|Exception in _GetFileSizeNoPathMod() : [Errno 2] No such file or directory: '/Volumes/Macintosh HD - Data/Users/macforensics/Library/Saved Application State/com.apple.AppStore.savedState'
2021-02-04 15:51:20|MAIN.HELPERS.MACINFO|ERROR|Exception in _GetFileSizeNoPathMod() : [Errno 2] No such file or directory: '/Volumes/Macintosh HD - Data/Users/macforensics/Library/Saved Application State/com.apple.iMovieApp.savedState'
2021-02-04 15:51:20|MAIN.HELPERS.MACINFO|ERROR|Exception in _GetFileSizeNoPathMod() : [Errno 2] No such file or directory: '/Volumes/Macintosh HD - Data/Users/macforensics/Library/Saved Application State/com.skitch.skitch.savedState'

These .savedState files above are symlink files, but their target files don't exist.
Check .savedState file:
(mac_apt) macforensics@ermacforensics mac_apt % ls -al /Volumes/Macintosh\ HD\ -\ Data/Users/macforensics/Library/Saved\ Application\ State
(snip)
lrwxr-xr-x   1 macforensics  staff   128  9  5  2019 com.apple.calculator.savedState -> /Users/macforensics/Library/Containers/com.apple.calculator/Data/Library/Saved Application State/com.apple.calculator.savedState
(snip)

Check symlink target file (It doesn't exist):
(mac_apt) macforensics@ermacforensics mac_apt % ls -al /Users/macforensics/Library/Containers/com.apple.calculator/Data/Library/Saved\ Application\ State/com.apple.calculator.savedState
ls: /Users/macforensics/Library/Containers/com.apple.calculator/Data/Library/Saved Application State/com.apple.calculator.savedState: No such file or directory

Finally, an exception occurred like below.

Exception ignored in: <function SqliteWriter.del at 0x7f9d781657b8>
Traceback (most recent call last):
File "/Users/macforensics/Documents/GitHub/mac_apt/plugins/helpers/writer.py", line 402, in del
ValueError: SqliteWriter destructor, Dear coder, you forgot to close db.

I have attached a log file too.
Log.20210204-155117.txt

I will take a look and let you know what the actual problem is, however this is what I can tell immediately.
The QUICKLOOK plugin will not work with mounted images because the quicklook database identifies files via inode numbers. If you run it over a disk image, mac_apt builds its own apfs database and can map file paths to inode numbers. This cannot be done on live/mounted volumes. I should add some exception handling and warnings in the code.
It looks like there is a problem with symlinks referencing files that don't exist, and I need to add some more exception handling for that too.

Fixed the exceptions from QUICKLOOK. This should now run and export the files, but won't be able to tell you its original name/path (when run in MOUNTED mode).


Confusing build failed message (time out and memory consumption)

Recently I have seen some reports of build failure like https://github.com/rtfd/readthedocs.org/issues/4443, where the failed build display two different messages as the failure reason.
Here you can see an example https://readthedocs.org/projects/oapackage/builds/7559891/. The error message is Build exited due to time out, but the last command that was executed failed with  Command killed due to excessive memory consumption.

This is because the best guess we have when docker kills something is looking for the 137 exit code, which usually indicates that memory was an issue. Unfortunately, we don't have a better way to determine this, so we're likely to get false positives. What happened here is project timed out, killing the process inside docker, reporting the exit code as 137.
Not sure if there is a lot we can do here, or if there is, if there is much benefit.

Not sure if there is a lot we can do here, or if there is, if there is much benefit.

mmm, I haven't see the code, but we could try to get the error from the latest command (I think we already inspect the output), so both error messages match. Right now is confusing for the users, and gives a wrong hint about the problem.


Split ${CROSS_CFLAGS} into ${CFLAGS} and ${CROSS_CFLAGS} (#9)

This separates the native compile options that use the system
installations of zlib and expat from the local cross-compiling options
that use the included vendored libraries.

#9


Adding the '.' Prefix

I worked out a couple of ways to fix this and was planning on submitting a slightly more complex PR but when I went to create the PR I noticed you had done the string file fix using this convention. So I am submitting this PR using your chosen convention. Hope it is to your liking!

Sure!
Before

After

It looks like I got a little overzealous and I didn't need to fix tuples but Dicts, Lists, and Sets all have the same issue as Strings with the '.' prefix.
Hope this helps

I am so sorry. I mistakenly thought you had already noticed this problem and merged a fix on the string snippets. So I gave very little context in my original PR thinking I was just fixing 'the rest of the issue'. I can now see that the solution posted to string snippets is only present in my fork of your repo. In any case, the editor doesn't recognize the snippet until after the '.' prefix as shown above. Thus, when you accept completion it doubles the dot operator. Adding the '.' to the prefix was one of a few ways I found to work this out. I didn't want to break any compatibility so I was going to add the solution as a list like so:

If you think this is a better way of handling the issue, I am happy to implement this in the PR. Please forgive my stupidity here, this is my first public opensource contribution and I got a bit nervous about it.

Hey, hope I am not pestering you. Just thought I should check on this, I am happy to cancel this PR and file a cleaner one if you prefer. Can you just let me know what the preferred convention would be for the fix? (i.e. Would you prefer the '.' simply be appended, or should I add the change as a list as shown above?). I am very excited to help! :)

This might be silly, but you just made my day! Thank you!


TEST: enable xml doctree tracking

This PR adds in doctree xml testing

Codecov Report

Merging #39 (9367977) into master (4a28f71) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #39   +/-   ##
=======================================
  Coverage   95.83%   95.83%
=======================================
  Files           4        4
  Lines         384      384
=======================================
  Hits          368      368
  Misses         16       16

Flag
Coverage Δ

pytests
95.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a28f71...9367977. Read the comment docs.

@AakashGfude I am going ahead and merging this. It is just adding in xml doctree tracking to make future development easier -- particularly when working on the transforms


More descriptive error message instead of 'Bad data'

This message should only be returned when providing malicious input to Keystatic's API endpoints which is why it's not detailed. the Keystatic UI should never cause this error to occur, did you find a case where it does?


Fix fabric remote install/instantiate bug

Codecov Report

Merging #64 into release-rc4 will increase coverage by 0.06%.
The diff coverage is 82.92%.

@@                Coverage Diff                @@
##             release-rc4      #64      +/-   ##
=================================================
+ Coverage          59.52%   59.58%   +0.06%
- Complexity           312      315       +3
=================================================
  Files                 35       35
  Lines               2809     2821      +12
  Branches             204      206       +2
=================================================
+ Hits                1672     1681       +9
- Misses               993      994       +1
- Partials             144      146       +2

Impacted Files
Coverage Δ
Complexity Δ

...ss/stub/fabric/proxy/ProxyChaincodeDeployment.java
44.27% <0.00%> (-0.35%)
4.00 <0.00> (ø)

.../webank/wecross/stub/fabric/TransactionParams.java
87.50% <75.00%> (-2.50%)
13.00 <2.00> (+2.00)
:arrow_down:

...m/webank/wecross/stub/fabric/FabricConnection.java
55.73% <83.33%> (+0.17%)
47.00 <4.00> (+1.00)

...c/main/java/com/webank/wecross/utils/TarUtils.java
91.17% <85.71%> (+3.29%)
5.00 <3.00> (ø)

...nk/wecross/stub/fabric/EndorserRequestFactory.java
89.78% <86.66%> (-0.51%)
17.00 <0.00> (ø)

.../wecross/stub/fabric/ChaincodeResourceManager.java
81.72% <100.00%> (ø)
23.00 <3.00> (ø)

...a/com/webank/wecross/stub/fabric/FabricDriver.java
60.74% <100.00%> (-0.54%)
31.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a3aef9...c738f53. Read the comment docs.


More explicit error handling

Add an exceptions module with one custom exception: UnsupportedModelError
Replace each instance of a bare Exception with a more explicit exception, either ValueError, RuntimeError, TypeError or UnsupportedModelError

Any exceptions raised in a commented out block of code were left alone
closes https://github.com/slundberg/shap/issues/2033

Thank you so much for this! For some reason it got marked "read" and I didn't see it. Now there is also another PR that adds even a bit more detail than this one, so I am going to merge that one and leave my thanks here.


[DUOS-490][risk=no] Filter pending cases for uniqueness

Addresses
https://broadinstitute.atlassian.net/browse/DUOS-490
Changes

Ensure uniqueness for pending cases.

Coverage decreased (-0.04%) to 60.204% when pulling 40451dd592a6236c4118dc546220647ca284708a on DUOS-490 into ab318a523e60ae61c41740fe778ac3cd894604e6 on develop.


Create chainId limit/validation

Description
trezor has/had a 32 bit chain id limit, and therefore has difficulty interacting with chain that have a chain id >32 bit.  Trezor is working on supporting 64 bit chain id's at present.  Devops on palm has asked for besu to implement a constraint on chain Id so that id does not exceed 64 bit.  In light of conversation, this should probably be a warning rather than an explicit constraint
Acceptance Criteria

warning when chain id exceeds 64 bit

The EVM opcode (CHAINID) allows to return a value for chainid up to 256bits.

Good point, it seems like a warning might be more appropriate.
I think the issue here is that hardware wallets, specifically trezor were not able to work with the palm chain id.  a 256 bit chain id is a bit absurd really.  I doubt anybody would have the patience to type in a chain id in excess of 64 bits.  $0.02.

Please don't link to private resources. Please provide context. No idea who Josh is.

There are chains that use more than 64 bits, so a warning instead of a block would preserve compatibility.

@drinkcoffee is working on cross chain operations with the EEA and has devised new ways to present chain IDs. See also https://github.com/ChainAgnostic/CAIPs/pull/49

The limit on that standard (32 digits) limits the space to 109/110 bits.  The ideas I've seen are to take some common string "Acme Widget Chain 2021", hash it, and that's the chainid.  You would want 256 for some hash algorithms.  64 is a bit short for a hash.

@garyschulte @jflo to unstick the trezor wallet addresses perhaps Besu could be configured to accept multiple ChainIDs.  The evm opcode only reports one but multiple are accepted in protected transactions.  Not sure how to advertise it to the evm, but the json rpcs could add an extra field listing an array of all accepted ChainIDs.

Interested in contributing to this issue.

I think there might be a bit more going on than emitting the warning, removing good first issue label.

It would be good if the chain id was 256 bits. My reasoning is:

This allows ephemeral blockchain ids to be generated on the fly randomly, without having to worry about collisions.
Additionally, as @atoulme pointed out, the Enterprise Ethereum Alliance is in the midst of standardising a 256 bit chain id to be used for crosschain.

Close as current 256 bit value is fine, warning can be handled client side.


Emu pk

Enable passing linear and nonlinear pk emulators to the cosmology class (baccoemu and euclid emulator 2 implemented for now, generic emulator class introduced extendable to other emulators). Added baccoemu baryonic effect emulator to baryons class.

@matteozennaro flake8 is failing (from experience, best run this on your machine before pushing). Do flake8 pyccl/ and flake8 benchmarks/ and solve all the issues that pop up.

The tests are failing because they can't import baccoemu. Try getting it to install by adding it here: https://github.com/LSSTDESC/CCL/blob/master/.github/environment.yml

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

163 of 221 (73.76%)  changed or added relevant lines in 9 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-1.0%) to 96.583%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pyccl/emulators/emu_base.py
10
11
90.91%

pyccl/cosmology.py
16
18
88.89%

pyccl/emulators/baccoemu_linear_pk.py
41
48
85.42%

pyccl/emulators/baccoemu_nonlinear_pk.py
41
48
85.42%

pyccl/emulators/ee2_nonlinear_pk.py
11
30
36.67%

pyccl/baryons/baccoemu_baryons.py
38
60
63.33%

Files with Coverage Reduction
New Missed Lines
%

pyccl/cosmology.py
1
98.98%

Totals

Change from base Build<PHONE_NUMBER>:
-1.0%

Covered Lines:
5286

Relevant Lines:
5473

💛  - Coveralls

It seems that classy requires pyproject-toml.

May be related https://github.com/lesgourg/class_public/issues/416


Removed repeated sentence and extra space

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/node-howto-e2e.md
:white_check_mark:Succeeded
View (azure-node-latest)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Save CLI history to file

This is a useful feature.

I clobbered my working directory a while ago and eventually forgot that I wrote this. I just went to write it again but discovered I had already written it. Anyway the commit is rebased if you want to consider merging it still.

Merged to ptr branch. It will be merged to master in a few days, so this PR can still remain open.


Upstream memory allocation bug to MFEM

Not part of this PR, but this should be upstreamed to MFEM.
Originally posted by @jamiebramwell in https://github.com/LLNL/serac/pull/523#r689906850

This might be closable. I believe this bug disappeared after moving to mfem 4.2, although I don't think we ever found an explanation for the problem.


Calyptos: Support for static route deployment (instead of bgp)

Could have an attribute such as default_attributes.midokura.static_route: true
This would kick off multiple resources to allow access to instances over their public interfaces based on the default_attributes.eucalyptus.network.config-json.PublicIps string

Hi @dmccue,
I began testing some different approaches to this last week.
I also discussed this briefly with @mbacchi last week. We are looking at larger changes in regards to how Calyptos handles system level items (such as network interfaces) in general. This may add to the potential size of the environment document, but should also allow that document to be more transparent and flexible.  I think this refactor will cover this use case.
In the meantime...
We could also add a recipe to the Midokura-Cookbook that can be used directly via chef-client on the GW's to see if that addresses the problem/need?  ie: "chef-client -z -o midokura::static-gateway" ?
Thanks!
-M

The midokura::static-gateway using chef-solo should work temporarily, let me know when to start testing


Merge master into development branch

The aim of this patch is to keep the stage of development branch same with master, so that we can open code development of next release.

Coverage increased (+0.07%) to 60.745% when pulling d1daeef3cc74e8eb4e955de932422ee2ea0910f5 on master into d44f8287226bacd0627cb5d33d43b7ecf25445dd on development.


xunit 2.4.2 - tests not showing in Revit

I've written a few tests and tried to run them in Revit but they didn't show.
After some testing, I've found out that this was caused by the xunit 2.4.2. After downgrading to 2.4.1 all works fine.

Fixed, thanks for the report!


Fix truthiness values in molecule

I don't why this is failing https://github.com/ansible-community/ansible-consul/actions/runs/3868407775/jobs/6593825338#step:4:446

@bbaassssiiee @brianshumate Could someone please hop in and help fixing the molecule-tests?
I'm not that deep into these tests and I don't know what and why it is breaking here (and apparently all other runs).

The issues seem container related, and not specific to molecule.

I disabled a few scenarios and enabled newer distros. Debian containers seem broken.

@ppuschmann Please rebase your changes, as we've just made some changes to CI in this repository. I'll have a look at this PR afterwards.

I created another PR with the same intent, to replace this one.


实现任务延迟固定时间后重试，在server实现还是worker实现比较合适？要适用于工作流的某个任务，任务重试时整个工作流等待重试完成后再继续，是否能实现？

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

建议 server 端实现。
类似需求正在规划中。


Localisation - Label formatting incorrect

I’m a developer from germarny and use your simple waterfall chart in some projects. My customers often have the requirement to format numbers. Here in Germany we format numbers with a dot as thousand separator. Is it possible to change that somehow? I can format my measures as I like, it always shows the thousand separator with comma.

The visual now includes "Localisation" for the purpose of formatting numbers and dates.

Request by Christopher Hähnel

Delivered in version <IP_ADDRESS>


use exceptions_dict

Short description:
use exceptions_dict instead of old exceptions args.
More details:
Following PR https://github.com/RedHatQE/openshift-python-wrapper/pull/24 that was merged, all ocp_resources-related code should be adjusted accordingly.
What this PR does / why we need it:
i see warnings in my metrics tests and discovered that it was due to using one of the resource.py methods (wait_for_condition).
Which issue(s) this PR fixes:

avoiding faillure on October 12
make sure the warnings do not appear in the pytest run summary.

Special notes for reviewer:
Bug:

the metrics test i mentioned in the PR description passed and warnings did not appear.

/verified

/verify

/verify

/verified

@myakove , please see the terminal buffer I sent on the relevant gerrit PR for verification of this PR.
/verified

/verified

/assign @myakove

/verified

@myakove , please see my verification for this on my test (gerrit PR).

/approve

@myakove , please see my verification for this on the test gerrit PR.


3D WorldEnvironment-CubeMap rotation

It would be very nice feature if CubeMap (of 3D WorldEnvironment node) could have ability to become rotated.
This way it could be very easy to make a moving sky / stars / clouds effect just simply rotating the CubeMap along Y axis. (throught animationplayer / from code / timer ect...)

seems implemented in 3.1


Error while building node-sass | can not download file

Hi,
I'm getting error while running our project. It cannot download the binaries, and the binaries are not available with url:  https://raw.githubusercontent.com/sass/node-sass-binaries/v2.0.1/linux-x64-3.14.5/binding.node
How can I solve this problem?
Thanks
04-Mar-2015 10:14:00	Can not download file from https://raw.githubusercontent.com/sass/node-sass-binaries/v2.0.1/linux-x64-3.14.5/binding.node
04-Mar-2015 10:14:00
04-Mar-2015 10:14:00	><EMAIL_ADDRESS>postinstall /var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass
04-Mar-2015 10:14:00	> node scripts/build.js
04-Mar-2015 10:14:00
04-Mar-2015 10:14:00	Building: /usr/bin/node node_modules/pangyp/bin/node-gyp rebuild
04-Mar-2015 10:14:01	make: Entering directory `/var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass/build'
04-Mar-2015 10:14:01	  CXX(target) Release/obj.target/binding/src/binding.o
04-Mar-2015 10:14:01	In file included from ../src/binding.cpp:3:
04-Mar-2015 10:14:01	../src/sass_context_wrapper.h:4:34: error: libsass/sass_context.h: No such file or directory
04-Mar-2015 10:14:01	In file included from ../src/binding.cpp:3:
04-Mar-2015 10:14:01	../src/sass_context_wrapper.h:26: error: ISO C++ forbids declaration of ‘Sass_Import’ with no type
04-Mar-2015 10:14:01	../src/sass_context_wrapper.h:26: error: expected ‘;’ before ‘*’ token
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘void prepare_import_results(v8::Local<v8::Value>, sass_context_wrapper*)’:
04-Mar-2015 10:14:01	../src/binding.cpp:24: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:24: error: ‘sass_make_import_list’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:36: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:36: error: ‘sass_make_import_entry’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:40: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:40: error: ‘sass_make_import_list’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:45: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:45: error: ‘sass_make_import_entry’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:48: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:48: error: ‘sass_make_import_list’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:49: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp:49: error: ‘sass_make_import_entry’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘Sass_Import** sass_importer(const char*, const char*, void*)’:
04-Mar-2015 10:14:01	../src/binding.cpp:106: error: ‘struct sass_context_wrapper’ has no member named ‘imports’
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘void extract_options(v8::Local<v8::Object>, void*, sass_context_wrapper*, bool, bool)’:
04-Mar-2015 10:14:01	../src/binding.cpp:118: error: ‘sass_file_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:122: error: ‘sass_data_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:125: error: ‘sass_context_get_options’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:146: error: ‘sass_make_importer’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:146: error: ‘sass_option_set_importer’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:150: error: ‘sass_option_set_input_path’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:153: error: ‘sass_option_set_output_path’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:154: error: ‘Sass_Output_Style’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:154: error: ‘sass_option_set_output_style’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:155: error: ‘sass_option_set_is_indented_syntax_src’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:156: error: ‘sass_option_set_source_comments’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:157: error: ‘sass_option_set_omit_source_map_url’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:158: error: ‘sass_option_set_source_map_embed’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:159: error: ‘sass_option_set_source_map_contents’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:160: error: ‘sass_option_set_source_map_file’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:161: error: ‘sass_option_set_include_path’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:162: error: ‘sass_option_set_precision’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: At global scope:
04-Mar-2015 10:14:01	../src/binding.cpp:165: error: ‘Sass_Context’ has not been declared
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘void get_stats(sass_context_wrapper*, int*)’:
04-Mar-2015 10:14:01	../src/binding.cpp:168: error: ‘sass_context_get_included_files’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:172: error: ‘nullptr’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: At global scope:
04-Mar-2015 10:14:01	../src/binding.cpp:180: error: ‘Sass_Context’ has not been declared
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘int get_result(sass_context_wrapper*, int*, bool)’:
04-Mar-2015 10:14:01	../src/binding.cpp:183: error: ‘sass_context_get_error_status’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:186: error: ‘sass_context_get_output_string’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:187: error: ‘sass_context_get_source_map_string’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:198: error: ‘sass_context_get_error_json’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘void make_callback(uv_work_t*)’:
04-Mar-2015 10:14:01	../src/binding.cpp:212: error: ‘sass_data_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:215: error: ‘sass_file_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:218: error: cannot convert ‘Sass_Context*’ to ‘int*’ for argument ‘2’ to ‘int get_result(sass_context_wrapper*, int*, bool)’
04-Mar-2015 10:14:01	../src/binding.cpp:226: error: ‘sass_context_get_error_json’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘v8::Handle<v8::Value> render(const v8::Arguments&)’:
04-Mar-2015 10:14:01	../src/binding.cpp:248: error: ‘sass_make_data_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘v8::Handle<v8::Value> render_sync(const v8::Arguments&)’:
04-Mar-2015 10:14:01	../src/binding.cpp:265: error: ‘sass_make_data_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:266: error: ‘sass_data_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:273: error: cannot convert ‘Sass_Context*’ to ‘int*’ for argument ‘2’ to ‘int get_result(sass_context_wrapper*, int*, bool)’
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘v8::Handle<v8::Value> render_file(const v8::Arguments&)’:
04-Mar-2015 10:14:01	../src/binding.cpp:285: error: ‘sass_make_file_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp: In function ‘v8::Handle<v8::Value> render_file_sync(const v8::Arguments&)’:
04-Mar-2015 10:14:01	../src/binding.cpp:302: error: ‘sass_make_file_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:303: error: ‘sass_file_context_get_context’ was not declared in this scope
04-Mar-2015 10:14:01	../src/binding.cpp:309: error: cannot convert ‘Sass_Context*’ to ‘int*’ for argument ‘2’ to ‘int get_result(sass_context_wrapper*, int*, bool)’
04-Mar-2015 10:14:01	make: Leaving directory `/var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass/build'
04-Mar-2015 10:14:01	make: *** [Release/obj.target/binding/src/binding.o] Error 1
04-Mar-2015 10:14:01	gyp ERR! build error
04-Mar-2015 10:14:01	gyp ERR! stack Error: `make` failed with exit code: 2
04-Mar-2015 10:14:01	gyp ERR! stack     at ChildProcess.onExit (/var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass/node_modules/pangyp/lib/build.js:271:23)
04-Mar-2015 10:14:01	gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
04-Mar-2015 10:14:01	gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:810:12)
04-Mar-2015 10:14:01	gyp ERR! System Linux 2.6.32-431.17.1.el6.x86_64
04-Mar-2015 10:14:01	gyp ERR! command "/usr/bin/node" "/var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass/node_modules/pangyp/bin/node-gyp" "rebuild"
04-Mar-2015 10:14:01	gyp ERR! cwd /var/opt/bamboo/xml-data/build-dir/WEB-ND-COMP/design/node_modules/grunt-kss/node_modules/kss/node_modules/node-sass
04-Mar-2015 10:14:01	gyp ERR! node -v v0.10.33
04-Mar-2015 10:14:01	gyp ERR! pangyp -v v2.1.0
04-Mar-2015 10:14:01	gyp ERR! not ok
04-Mar-2015 10:14:01	Build failed

Hi @Yobikap, how are you installing node-sass? If looks like you're using an unreleased/unstable version.

You can close this issue, you were right, I was using a wrong version.


Web Portal for Receiving Orders

Sprint Review on 2022-10-18 indicated this may instead mean integration with web portals that other organization support (e.g. LIMS, labs)


Bump setuptools from 67.3.3 to 67.4.0 (#5626)

Pull request opened by the merge tool on behalf of #5626

Merged in 6c29672e5b075cba080b66ef766eaff47a4f8ecf


ci: refactor pre-release ci

Codecov Report

Merging #112 (99e709f) into master (2addb99) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #112   +/-   ##
=======================================
  Coverage   97.79%   97.79%
=======================================
  Files          18       18
  Lines         272      272
  Branches       39       39
=======================================
  Hits          266      266
  Misses          4        4
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2addb99...99e709f. Read the comment docs.


bug: limit_by_labels in <=1.0.1 replaced by limit_by_label in provider and not fixed here

See for details https://github.com/lacework/terraform-provider-lacework/issues/421

Hi @MaxymVlasov, I can't believe we missed this. I am really sorry for the delay, I am personally fixing this right now.

@MaxymVlasov actually, looking at this change https://github.com/lacework/terraform-aws-ecr/pull/39 it seems that we have fixed this module. 🤔 What is the problem you are experiencing?

🤔 I'll double check and return

Please do and let us know if you are having issues. We acknowledge we really messed up with that change in the provider so if there is anything we can do to unblock you, or anyone else, we will do it right away.

lacework/lacework 1.0.1, lacework/ecr/aws 0.7.2 conf
terraform {
  required_version = "~> 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.38"
    }

    lacework = {
      source  = "lacework/lacework"
      version = "~> 1.0.1"
    }
  }
}

module "ecr" {
  source  = "lacework/ecr/aws"
  version = "0.7.2"

  count = module.this.enabled ? 1 : 0

  lacework_integration_name = module.this.tags.Name
  iam_role_name             = module.lacework_iam_role_name.id
}

tf plan: No changes. Your infrastructure matches the configuration.
 lacework/lacework 1.1.0 (same for 1.4.0), lacework/ecr/aws 0.7.3 conf
terraform {
  required_version = "~> 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.38"
    }

    lacework = {
      source  = "lacework/lacework"
      version = "~> 1.1.0"
    }
  }
}

module "ecr" {
  source  = "lacework/ecr/aws"
  version = "0.7.3"

  count = module.this.enabled ? 1 : 0

  lacework_integration_name = module.this.tags.Name
  iam_role_name             = module.lacework_iam_role_name.id
}

tf plan:
╷
│ Error: Failed to decode resource from state
│
│ Error decoding "module.ecr[0].lacework_integration_ecr.iam_role" from previous state:
│ missing expected [
╵
╷
│ Error: missing expected [
│
│   with module.ecr[0].lacework_integration_ecr.iam_role,
│   on .terraform/modules/ecr/main.tf line 36, in resource "lacework_integration_ecr" "iam_role":
│   36: resource "lacework_integration_ecr" "iam_role" {
│
╵
Releasing state lock. This may take a few moments...
exit status 1

I dunno why it expects [ here
https://github.com/lacework/terraform-aws-ecr/blob/a330b97e661bd553a69b23929cb6eda936e1386b/main.tf#L36-L55

Maybe it relates count arg, but...

@afiune Happy Friday! Ping you in case you miss my previous msg :)

We're also encountering this, though with a slightly different error message.
Version 1.0.1 of the provider and 0.7.2 of the module reports "No changes", but upgrading the provider to 1.4.0 and the module to 0.7.3 gives us:
╷
│ Error: Failed to decode resource from state
│
│ Error decoding "module.lacework_ecr.lacework_integration_ecr.iam_role" from previous state: unsupported attribute "limit_by_labels"
╵

@afiune ?

@sarahhodne @MaxymVlasov apologies for the delayed response.
Let me take a quick look at this thread again. 👀

Essentially change https://github.com/lacework/terraform-aws-ecr/pull/39 was a breaking change since we changed the variable limit_by_labels from map(string) to list(object()) which should have been released as v1.0.0 instead.
With this breaking change you are going to have to update your Terraform code for the limit_by_labels variable and use the new format.
If you are using it like this:
  limit_by_labels       = {example: "example"}

You will have to update it to:
limit_by_labels = [{
    key   = "example"
    value = "test"
    }]

We apologize for the inconveniences. 😞

@afiune  we don't use limit_by_labels anymore, but we still get the next error:
https://github.com/lacework/terraform-aws-ecr/issues/41#issuecomment-1402004804

I tried to replicate both of your problems @MaxymVlasov @sarahhodne but I am unable to do so.
I created a new ECR integration with the versions you have, ran terraform apply and then upgraded but I don't seem to have the same issues you have.
Code:
terraform {
  required_providers {
    lacework = {
      source  = "lacework/lacework"
      version = "= 1.0.1"
    }
  }
}

provider "aws" {
  region  = "us-west-2"
  profile = "customerdemo"
}

module "lacework_ecr" {
  source                    = "lacework/ecr/aws"
  version                   = "= 0.7.2"
  lacework_integration_name = "Bug 41"

  limit_by_labels = {
    example : "example"
    example2 : "example"
  }
}

Ran init:
➜  terraform init -upgrade
Upgrading modules...
Downloading registry.terraform.io/lacework/ecr/aws 0.7.2 for lacework_ecr...
- lacework_ecr in .terraform/modules/lacework_ecr
Downloading registry.terraform.io/lacework/iam-role/aws 0.3.1 for lacework_ecr.lacework_ecr_iam_role...
- lacework_ecr.lacework_ecr_iam_role in .terraform/modules/lacework_ecr.lacework_ecr_iam_role

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/random versions matching ">= 2.1.0"...
- Finding lacework/lacework versions matching "~> 1.0, 1.0.1"...
- Finding hashicorp/aws versions matching ">= 3.0.0, < 5.0.0"...
- Finding hashicorp/time versions matching "~> 0.6"...
- Using previously-installed hashicorp/random v3.5.1
- Installing lacework/lacework v1.0.1...
- Installed lacework/lacework v1.0.1 (signed by a HashiCorp partner, key ID 51A8966FFFC33F87)
- Using previously-installed hashicorp/aws v4.64.0
- Using previously-installed hashicorp/time v0.9.1

Terraform has been successfully initialized!

 Resources: 6 added, 0 changed, 0 destroyed.
➜
➜  lacework cr show TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE
                  CONTAINER REGISTRY GUID                     NAME       TYPE       STATUS    STATE
-----------------------------------------------------------+--------+-------------+---------+--------
  TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE   Bug 41   ContVulnCfg   Enabled   Ok

                                    DETAILS
--------------------------------------------------------------------------------
    AWS AUTH TYPE           AWS_IAM
    EXTERNAL ID             .0A7mc0@uae-3yTd
    LAST SUCCESSFUL STATE   2023-04-21T12:53:59-07:00
    LIMIT BY LABEL          example:example,example2:example

I upgraded and tried to re apply.
New Code:
terraform {
  required_providers {
    lacework = {
      source = "lacework/lacework"
      # version = "= 1.0.1"
    }
  }
}

provider "aws" {
  region  = "us-west-2"
  profile = "customerdemo"
}

module "lacework_ecr" {
  source = "lacework/ecr/aws"
  # version = "= 0.7.2"

  lacework_integration_name = "Bug 41"

  limit_by_labels = [
    {
      key   = "example"
      value = "example"
    },
    {
      key   = "example2"
      value = "example"
    }
  ]
}

Upgraded:
➜  terraform init -upgrade
Upgrading modules...
Downloading registry.terraform.io/lacework/ecr/aws 0.7.3 for lacework_ecr...
- lacework_ecr in .terraform/modules/lacework_ecr
Downloading registry.terraform.io/lacework/iam-role/aws 0.3.1 for lacework_ecr.lacework_ecr_iam_role...
- lacework_ecr.lacework_ecr_iam_role in .terraform/modules/lacework_ecr.lacework_ecr_iam_role

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/random versions matching ">= 2.1.0"...
- Finding lacework/lacework versions matching "~> 1.1"...
- Finding hashicorp/aws versions matching ">= 3.0.0, < 5.0.0"...
- Finding hashicorp/time versions matching "~> 0.6"...
- Installing lacework/lacework v1.6.0...
- Installed lacework/lacework v1.6.0 (signed by a HashiCorp partner, key ID 51A8966FFFC33F87)
- Using previously-installed hashicorp/aws v4.64.0
- Using previously-installed hashicorp/time v0.9.1
- Using previously-installed hashicorp/random v3.5.1

Terraform has been successfully initialized!

Ran plan and apply:
➜  terraform apply
module.lacework_ecr.module.lacework_ecr_iam_role.random_string.external_id[0]: Refreshing state... [id=.0A7mc0@uae-3yTd]
module.lacework_ecr.module.lacework_ecr_iam_role.random_id.uniq: Refreshing state... [id=XU476g]
module.lacework_ecr.module.lacework_ecr_iam_role.aws_iam_role.lacework_iam_role[0]: Refreshing state... [id=lw-iam-5d4e3bea]
module.lacework_ecr.aws_iam_role_policy_attachment.ecr_read_only_policy_attachment: Refreshing state... [id=lw-iam-5d4e3bea-20230421194621934300000001]
module.lacework_ecr.time_sleep.wait_time: Refreshing state... [id=2023-04-21T19:46:37Z]
module.lacework_ecr.lacework_integration_ecr.iam_role: Refreshing state... [id=TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.lacework_ecr.lacework_integration_ecr.iam_role has changed
  ~ resource "lacework_integration_ecr" "iam_role" {
        id                      = "TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE"
        name                    = "Bug 41"
        # (12 unchanged attributes hidden)

      + limit_by_label {
          + key   = "example2"
          + value = "example"
        }
        # (1 unchanged block hidden)
    }

  # module.lacework_ecr.module.lacework_ecr_iam_role.aws_iam_role.lacework_iam_role[0] has changed
  ~ resource "aws_iam_role" "lacework_iam_role" {
        id                    = "lw-iam-5d4e3bea"
        name                  = "lw-iam-5d4e3bea"
      ~ role_last_used        = [
          ~ {
              ~ last_used_date = "" -> "2023-04-21T19:49:49Z"
              ~ region         = "" -> "us-west-2"
            },
        ]
        tags                  = {}
        # (9 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may
include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.lacework_ecr.lacework_integration_ecr.iam_role will be updated in-place
  ~ resource "lacework_integration_ecr" "iam_role" {
        id                      = "TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE"
        name                    = "Bug 41"
        # (12 unchanged attributes hidden)

      + limit_by_label {
          + key   = "example"
          + value = "example"
        }
        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.lacework_ecr.lacework_integration_ecr.iam_role: Modifying... [id=TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE]
module.lacework_ecr.lacework_integration_ecr.iam_role: Modifications complete after 2s [id=TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Verified that the labels are still there with the Lacework CLI:
➜  lacework cr show TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE
                  CONTAINER REGISTRY GUID                     NAME       TYPE       STATUS    STATE
-----------------------------------------------------------+--------+-------------+---------+--------
  TECHALLY_1933D090578A352EF9908A3965FA76AD4E161C839F59FAE   Bug 41   ContVulnCfg   Enabled   Ok

                                    DETAILS
--------------------------------------------------------------------------------
    AWS AUTH TYPE           AWS_IAM
    EXTERNAL ID             .0A7mc0@uae-3yTd
    LAST SUCCESSFUL STATE   2023-04-21T12:58:14-07:00
    LIMIT BY LABEL          example2:example,example:example

You have a few options here.

Destroy the module and resources from the previous version that works, then upgrade and re-deploy
You can manually manipulate the Terraform state and remove the limit_by_labels variable from the affected resource

For the second option, you could try running something similar to:
$ terraform state pull > state.json

Edit state.json and remove any line with limit_by_labels, then increment the serial attribute (ex "serial": 21, -> "serial": 22,) save and push it to the new state with:
$ terraform state push state.json
$ terraform plan      # hopefully this should work again

Finally, remove the temp state:
$ rm state.json

We recreated ECR, because it was not disruptive change for us


Keyword/Phrase Context Persistence via Model LLM

Idea

Currently, the bot can change it's capacity based on the User Preference. So, instead of tossing (dequeuing) a message and throwing it away, we can Grab necessary information from the chat history and keep it as "necessary" information for the bot to know.
For example, if we prompted to bot to act like a pirate, eventually it will lose the memory of "acting as a pirate".
Essentially we want to have this Patient 0 of messages that will hold "extra" context that it should always remember for a conversation!

Questions to answer

What are the key facts I should keep about this message?
Out of the message, what is the most important? (asking an LLM)
Does this go in a second queue? A different private variable in queue.ts?
Do we use a Priority queue or sort when asking a model to prioritize facts in a message.

// queue.ts
...

export class Queue<T> implements IQueue<T> {
    private storage: T[] = []
    private necessaryContext = [] // **list of necessary information for the bot to have if something gets dequeued

    /**
     * Set up Queue
     * @param capacity max length of queue
     */
    constructor(public capacity: number = 5) {}

    ...

We could either have a list of "necessary context" or maybe just 1 and we keep appending to it.
Another idea is to keep it as it's own class to handle this.
Essentially when we dequeue something from the storage, we want to run it through some general model to pick out what is relevant and necessary from the context we are removing from the history.

// messageCreate.ts
import { ChatResponse } from 'ollama'

...

let response: ChatResponse

// check if we can push, if not, remove oldest
if (msgHist.size() === msgHist.capacity) msgHist.dequeue()

// push user response before ollama query
msgHist.enqueue({
    role: 'user',
    content: message.content
})

...

The code above still needs to be modified to handle capacity change from #39

Essentially in the while loop that is to be implemented, it would run through and dequeue, then find the best relevancy from it or none at all based on the current context already.

Thoughts:

When implementing this, likely we will just feed the old output to the model (or maybe a specific model that is good at truncating and summarizing text?).
When a new generation is created, have some kind of context Queue? This queue can be longer as the messages being stored here are (should be) significantly smaller.
An issue that might arise with this is how are we going to feed this context when a query is made? How is this going to be stored and what really should be the limit for this?

cc: @JT2M0L3Y

This feature could be completed prior to release. It can be a bit complex as it tries to save certain "contexts" while chatting with a user.
Another Idea with this is to think about how ChatGPT does this on their website. If the user prompts something that is noteworthy for it to remember, it should remember it.
There are many possibilities but we can make the bot summarize what it needs to remember and have some kind of storage for it for the user. If we do base it off of ChatGPT, then this memery will persist between chats.
cc: @JT2M0L3Y