Replace return json_error by raise JsonableError

This is a follow-up for #19066 that completes the migration by implementing dedicated subclasses of JsonableError to fully cover all the cases where we use return json_error.

This is great, thanks for working on this @PIG208!  I just pushed back to this PR after merging the series ending with 6a04648fd75a5067ad2a3ba4abc989aba6a1d278 and semi-implementing a different approach for the authentication failures.  Several of my new commits fail tests (since I didn't update the error strings in tests first, and then later didn't update the HTTP 401 changes).  Can you fix that?
I'm also starting an "api design" thread about the naming questions; we'll want to do an API changelog entry for this once that settles.

See https://chat.zulip.org/#narrow/stream/378-api-design/topic/authentication.20error.20handling/near/1225193 for the chat.zulip.org thread on making decisions around this.

The CI should be passing after this.

Great!  I merged 43cca07b4bb2db09da6fb4dbd8caba6e9afc6a7f.  I'll post a few comments, and wait for the API conversation to settle; after that, we can add API changelog entries and merge.

I merged all but the 403=>401 commit as the series ending with 3fdfbb27f9f36a66688fe130edce22914f5f2c34.  Huge thanks for doing this migration @PIG208!
The last bit of changing 403 to 401 is still pending more discussion on whether we want to make that change.

Merged, after extending the commit with API documentation changelog notes, thanks @PIG208!


Allow building vscode with Python 3 (instead of Python 2)

The "How to Contribute" site says that for building vscode, Python >= 2.7 is required and "version 3 is not supported".
On many systems used as a build environment, it get's increasingly difficult to accept Python 2 to be installed and/or used, due to its already elapsed end of life, which implies considerable security concerns.
The request is to change the build tooling in a way to use Python 3 instead of Python 2.
(Sorry if this is already discussed somewhere else, I searched and wasn't able to find anything.)

I believe Python 2.7 was required only for node-gyp (to compile node native addons)
However node-gyp has added support for Python 3.x long time ago. I think I only have Python 3.x but I can still build Code from source.
So maybe only this documentation is outdated.

Give 3 a try. If it works, fantastic. If it doesn't, let us know what the issue is exactly. Our build and dev machines are more than happy to keep running 2.

Thanks, I went ahead and was able to successfully build vscode with solely Python 3.8 on my system (macOS 11).
Before that, I tried it with solely Python 3.9 and got an error during the "node-gyp rebuild" command (as indicated by @yume-chan, node-gyp seems to be the reason for the Python version restriction).
These experiences seem to correspond with the stated node-gyp requirements ("Python v2.7, v3.5, v3.6, v3.7, or v3.8").
Important notice in order not to get "deluded": after the successful build with Python 3.8, a build with solely Python 3.9 was also successful. However, this was most probably only the case because node-gyp has already been built before. I tried it on a second machine and confirmed this.
I would suggest to change the "How to Contribute" site in a way that it does not state any Python versions; instead there should be a reference to the node-gyp requirements for allowing the reader to determine what are the current supported Python versions.

That's fantastic. Could I ask you to try the same on Windows? Just so we have a clear conscience making this switch.

Ok, I now tested it also on Windows 10 with Python 3.7 and Python 3.9. Both vscode builds were successful.
(The pull request for the docu update is https://github.com/microsoft/vscode-wiki/pull/163.)


Show command line arguments for project resource types

Description
Shows command line arguments and executable in the source column for projects, if both are present. Additionally, improves the text visualizer view to show the entire command for both projects and executables (it currently only shows the executable name)

Fixes #5570
Checklist

Is this feature complete?

[X] Yes. Ready to ship.
[ ] No. Follow-up changes expected.

Are you including unit tests for the changes and scenario tests if relevant?

[ ] Yes
[X] No

Did you add public API?

[ ] Yes

If yes, did you have an API Review for it?

[ ] Yes
[ ] No

Did you add <remarks /> and <code /> elements on your triple slash comments?

[ ] Yes
[ ] No

[X] No

Does the change make any security assumptions or guarantees?

[ ] Yes

If yes, have you done a threat model and had a security review?

[ ] Yes
[ ] No

[X] No

Does the change require an update in our Aspire docs?

[ ] Yes

Link to aspire-docs issue:

[X] No

Microsoft Reviewers: Open in CodeFlow

@davidfowl @DamianEdwards Container arguments are treated as a sensitive value in resource details. It seems inconsistent that executable and project command line arguments aren't sensitive.

@JamesNK yeah agreed. We could be clever about it I think for projects and not hide known arguments though.

I don't think we can be clever and try to understand arguments. There are tons of different formats and there is no way for the dashboard to reliably parse them all. As far as we're concerned arguments are an opaque string.
What I think we should do:

On the main grid either:

Add masking to the main grid command line arguments. If there are arguments then the mask/unmask button is available to click on the column to show all values.
Or simply not show command line arguments for projects and containers and expect people to view the resource details.

Ensure that command line arguments are visible in resource properties like they are for containers, and they're marked as sensitive.

I don't think we can be clever and try to understand arguments. There are tons of different formats and there is no way for the dashboard to reliably parse them all. As far as we're concerned arguments are an opaque string.

We can revisit based on how this looks once implemented, but I think specifically for project resources we can be clever enough to not hide the standard args we pass as part of orchestrating projects into an Aspire launch, e.g. --no-build, --project, -c, etc., and specifically I'm talking about the arg names, not their values.

I think we can be clever and hide arguments with secrets. We have that knowledge and we can force people to model secrets as secret parameters to get this feature. What we do now is a bit aggressive assuming all arguments are secrets.

For now, lets make both executable and container arguments behave the same way. Let treat both as non-secret for now and file an issue to detect secrets used in arguments and hide them if there are secrets in there.

We can even get clever and redact the secrets 😄

Looks like a real test failure cc @radical

[xUnit.net 00:07:03.16]     Aspire.Workload.Tests.BuildAndRunTemplateTests.StarterTemplateNewAndRunWithoutExplicitBuild(config: "Release") [FAIL]
[xUnit.net 00:07:03.16]       [Source for apiservice]
Expected: aspire_starter_run_Release_a10gjci0_yq5.ApiService.csproj
Actual: dotnet run --no-build --project C:\h\w\AAD0096A\t\testroot\aspire_starter_run_Release_a10gjci0_yq5\aspire_starter_run_Release_a10gjci0_yq5.ApiService\aspire_starter_run_Release_a10gjci0_yq5.ApiService.csproj -c Release --no-launch-profile

Yeah, the test needs to be updated to track changes in this PR. @adamint

StarterTemplateNewAndRunWithoutExplicitBuild

I keep forgetting about these workload tests 😄

@JamesNK  @davidfowl for approval

@JamesNK for review.

I added some args to a project in TestShop:
var catalogDbApp = builder.AddProject<Projects.CatalogDb>("catalogdbapp")
                          .WithReference(catalogDb)
                          .WithArgs(["one", "two"]);

This is what I see:

Is that what's indented? It seems like CatalogDb.csproj one two would be more what is expected.
Also, unit tests should be added here. GetSourceColumnValueAndTooltip could easily be moved out of the component and tested with various ResourceViewModel instances.

Do we erase the project name when we show the args?

Do we erase the project name when we show the args?

Considering the project name is included in default args, yes. I have changed that now, so it will show up twice, but the executable (dotnet) does not show up in the source column when there is a project:

We could hide common arguments by default in a follow-up?

What I see now:

And visualizer:

It's not perfect, but it seems good enough. @davidfowl is this what you imagined?

Yea, I think that's better


Hud10837/route tracker

@TADraeseke this should all be working fine and implements the correct behavior with the new route tracker location data source. The Readme may need some more work, though. Could you take a look? Thanks

Thanks @TADraeseke !


i used your code. but i encounter this problem. i don't know why

File "C:/Users/kevin/Anaconda2/node-webkit/launcher/NLP.py", line 202, in
val.append(svo.process_parse_tree(root_tree[0]))
TypeError: 'listiterator' object has no attribute 'getitem'

Hm strange, I will try to set it up and figure out why, might be a new version of NLTK that messes things up.

thanks!! ithought that the nltk's libraries and functions may change.

Did a quick fix, it should now work.


EntityQueryModelVisitor may be too powerful!!!

Review when making breaking changes; consider number of dependencies.

At least investigate for 3.0.


Update to 1.18.1

New gradle, new fabric loader, and new Minecraft version. The code of the mod is still the same, but the dev env is less broken.

Also please change your commit messages to something more formal (more specifically the last one)


[Question]: How do I prevent vectors from OpenSearch to be included in the payload during querying?

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
After doing some analysis using the LlamaDebugHandler, I noticed that all the vectors + metadata is sent as context to the llm when using query engine. This causes unnecessary noise and additional charges to our account which can degrade the performance of our applications. How do I restrict the payload to only send the document text?
from llama_index.core.callbacks import (
    CallbackManager,
    LlamaDebugHandler,
    CBEventType,
)
from llama_index.core import Settings
llama_debug = LlamaDebugHandler(print_trace_on_end=True)
callback_manager = CallbackManager([llama_debug])
Settings.callback_manager = callback_manager

os_kwargs = dict(
    endpoint=OPENSEARCH_URL,
    index=COLLECTION_NAME,
    dim=EMBEDDING_DIM,
    embedding_field="vector_field",
    text_field="page_text",
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    engine="faiss"
)

os_vector_client = OpensearchVectorClient(**os_kwargs)
vector_store = OpensearchVectorStore(os_vector_client)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

as_query_engine_kwargs = {
    "response_mode": "tree_summarize",
    "llm": Settings.llm,
    "similarity_top_k": 2,
}

retriever = index.as_query_engine(**as_query_engine_kwargs)
response = retriever.query("<some general query>")
print(response)
.......

event_pairs = llama_debug.get_llm_inputs_outputs()
print(event_pairs[0][1].payload['messages'][1])

With the output as
user: Context information from multiple sources is below.
---------------------
metadata: {'project_metadata': {'...}, 'document_metadata': {...}}
page_text:  ...

vector_field: [0.002726590959355235, ..., -0.006757325027137995,
---------------------
Given the information from multiple sources and not prior knowledge, answer the query.
Query: <query question>
Answer:

@dblilienthal Did you put the embedding into the node metadata? I'm unable to reproduce this with default settings
For example:
docker run -d -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<some password>" -e "plugins.security.disabled=true" opensearchproject/opensearch:latest

And then
from os import getenv
from llama_index.core import SimpleDirectoryReader
from llama_index.vector_stores.opensearch import (
    OpensearchVectorStore,
    OpensearchVectorClient,
)
from llama_index.core import VectorStoreIndex, StorageContext

# http endpoint for your cluster (opensearch required for vector index usage)
endpoint = getenv("OPENSEARCH_ENDPOINT", "http://<IP_ADDRESS>:9200")

# index to demonstrate the VectorStore impl
idx = getenv("OPENSEARCH_INDEX", "llama-index-demo")

# load some sample data
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# OpensearchVectorClient stores text in this field by default
text_field = "content"
# OpensearchVectorClient stores embeddings in this field by default
embedding_field = "embedding"
# OpensearchVectorClient encapsulates logic for a
# single opensearch index with vector search enabled
client = OpensearchVectorClient(
    endpoint, idx, 1536, embedding_field=embedding_field, text_field=text_field
)
# initialize vector store
vector_store = OpensearchVectorStore(client)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# initialize an index using our sample data and the client we just created
index = VectorStoreIndex.from_documents(
    documents=documents, storage_context=storage_context
)

# print the node format that the llm will get
nodes = index.as_retriever().retrieve("What did the author do growing up?")
print(nodes[0].node.get_content(metadata_mode="llm"))

tbh I'm a little lost at what you changed. Its also weird that I wasn't able to reproduce.
Are you using an opensearch db that was populated without using llama-index? Or how did you initially create the index?
When using llama-index to populate the db, it just serializes the node into the db, so the embedding field should not be included (as evidenced by my own code above)


IP_FAMILY not backwards compatible?

What happened:
Running newest image produces following error which stops kubelet from booting
+ [[ -z '' ]]
/etc/eks/bootstrap.sh: line 384: DESCRIBE_CLUSTER_RESULT: unbound variable
++ awk '{print $2}'
+ IP_FAMILY=
++ err_report 384

What you expected to happen:
kubelet would boot
How to reproduce it (as minimally and precisely as possible):
A new parameter was introduced a little less that 2 weeks ago; --ip-family. It is not mentioned in any changelog. It defaults to nothing; IP_FAMILY="${IP_FAMILY:-}"
If it's not set, which it isn't be deult, this conditional is encountered
if [[ -z "${IP_FAMILY}" ]]; then
      IP_FAMILY=$(cat $DESCRIBE_CLUSTER_RESULT | awk '{print $2}')
fi

The variable referenced is set only if the conditional above the previous is met
if [[ -z "${B64_CLUSTER_CA}" ]] || [[ -z "${APISERVER_ENDPOINT}" ]]; then
    DESCRIBE_CLUSTER_RESULT="/tmp/describe_cluster_result.txt"

For ages (at least since 1.16), we've been bootstrapping our clusters with
/etc/eks/bootstrap.sh --b64-cluster-ca 'LSTiB...S0tLQo=' --apiserver-endpoint 'https://2462...019.yl4.us-east-1.eks.amazonaws.com'  --kubelet-extra-args "" 'cluster1'

Ergo, we do not meet the conditional, and therefore the IP_FAMILY attempts to cat the output of a file which doesnt exist.
Anything else we need to know?:
Environment:

AWS Region: us-east-1 and us-west-2, but its unimportant
Instance Type(s): c5.4xlarge
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): "eks.6"
Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): "1.19"
AMI Version:
Kernel (e.g. uname -a): Linux ip-10-11-171-109.ec2.internal 5.4.149-73.259.amzn2.x86_64 #1 SMP Mon Sep 27 12:48:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-0fc6bb86ee7c7948e"
BUILD_TIME="Sat Oct 23 18:45:37 UTC 2021"
BUILD_KERNEL="5.4.149-73.259.amzn2.x86_64"
ARCH="x86_64"

Hi  caphrim007@,
Are you seeing this error with EKS optimized AMI with latest bootstrap.sh? I tried reproducing this issue locally but couldn't do it. This is the configuration I used:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=LS0t******==
API_SERVER_URL=https://11111******.gr7.us-west-2.eks.amazonaws.com
clusterName=dev

/etc/eks/bootstrap.sh $clusterName --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
--//--

Please let me know if I am not following the steps correctly.

@ravisinha0506 i think you're right. i must have snipe'd the release before that follow-up fix i see in the bootstrap file went in. the current file looks correct. the file that i have in my ami's is incorrect. will update.


Challenge #28 Unable to escape a backslash in a string, not working correctly

The instructions are to "Encode the following sequence, separated by spaces:
backslash tab tab carriage-return new-line and assign it to myStr", however when I write the code:
var myStr = "\"; it returns the answer: myStr = "\" instead of printing one backslash it prints two.
I've tried typing it several different ways with several different combinations with different words and none of them work.
Escape Sequences in Strings
http://precourse.devmountain.com/challenges/escape-sequences-in-strings#?solution=
var myStr %3D "First Line\\ \n Second Line"%3B %2F%2F Change this line

Issue Description
Whenever I try to escape backslashes
Browser Information

Google Chrome

Browser Name, Version: Google Crome
Operating System: Windows
Mobile, Desktop, or Tablet: Desktop

Your Code

 my Str = " \\ ";

Screenshot

@actualchandler freeCodeCamp does not support issues for devmountain. Also, you have an error in your code.


RenderAnimatedOpacityMixin causes child pixel snapping when fully opaque

RenderAnimatedOpacityMixin has the following paint method, which avoids painting both the child widget and the opacity later when fully transparent. When fully opaque, the opacity layer is elided while the child is painted. Unfortunately this last optimization causes obvious pixel snapping on low DPR screens.
RenderAnimatedOpacityMixin.paint
 @override
  void paint(PaintingContext context, Offset offset) {
    if (child != null) {
      if (_alpha == 0) {
        // No need to keep the layer. We'll create a new one if necessary.
        layer = null;
        return;
      }
      if (_alpha == 255) {
        // No need to keep the layer. We'll create a new one if necessary.
        layer = null;
        context.paintChild(child!, offset);
        return;
      }
      assert(needsCompositing);
      layer = context.pushOpacity(offset, _alpha!, super.paint, oldLayer: layer as OpacityLayer?);
    }
  }

This can be seen fairly easily when a mouse hover first reveals a tooltip, such as in the "example app" section of the flutter gallery. The last frame visible shifts - fairly dramatically compared to the previous animation. Though the video below uses 10x time dilation to make it more obvious, it is still hard to see in the low res/low framerate GIF.

Keeping the opacity layer even when at fully opaque removes this jump.

More of an engine question, but how much does eliding the opacity layer buy us?

https://user-images.githubusercontent.com/8975114/107298876-16960300-6a2b-11eb-961d-0590c93b7fef.mp4
Github now supports mp4s, yay!

The likely cause of this is that when the opacity layer is rendering its save layer to which to apply its opacity, the entire picture there is being rendered independently from the surrounding picture data. We've essentially introduced a break in the scene rendering at that saveLayer.
When you remove the layer and thus remove is saveLayer, the picture is rendered into the rest of the scene as one big picture. The slight shift is due to "pixel snapping" of each picture independently. When it is a separate picture, it gets individually pixel snapped based on its local bounds. When it is part of the bigger picture, the picture snapping happens on a larger group and then the individual widget is rendered relative to that non-local pixel snapping decision.

There may be a simpler solution than keeping the widget when it isn't needed. Basically just ensure that the Picture objects are not glued together with or across this widget by careful manipulation of the current recording - with the potential addition of adding a RepaintBoundary mechanism to prevent opacity changes from forcing repaints of the surrounding content.
See #83145 (comment)

We can fix this this pretty easily by just putting the 255 child in an offsetlayer instead of using the offset parameter.

Or, as @flar mentioned on the PR, just wrapping the child in a repaint boundary, which will mean it always gets an OffsetLayer.

Or, endRecordingIfNeeded before and after painting the child? But that would also happen, possibly more organically, by using a layer instead.


Heap Use-After-Free issue in chttp2 reported by fuzzer

Sanitizer report:
=================================================================
==498769==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000008a20 at pc 0x5647193214e4 bp 0x7ffcff1965a0 sp 0x7ffcff196598
READ of size 8 at 0x629000008a20 thread T0
    #0 0x5647193214e3 in gpr_mpscq_pop_and_check_end(gpr_mpscq*, bool*) third_party/grpc/src/core/lib/gpr/mpscq.cc:49:43
    #1 0x564719321237 in gpr_mpscq_pop(gpr_mpscq*) third_party/grpc/src/core/lib/gpr/mpscq.cc:44:10
    #2 0x5647192d93b7 in grpc_combiner_continue_exec_ctx() third_party/grpc/src/core/lib/iomgr/combiner.cc:244:25
    #3 0x5647192e12e3 in grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #4 0x564718f2a89e in grpc_core::ExecCtx::~ExecCtx() third_party/grpc/src/core/lib/iomgr/exec_ctx.h:85:5
    #5 0x56471928e470 in grpc_server_shutdown_and_notify third_party/grpc/src/core/lib/surface/server.cc:1219:1
    #6 0x564718f24a39 in LLVMFuzzerTestOneInput third_party/grpc/test/core/end2end/fuzzers/api_fuzzer.cc:892:11
    #7 0x564719336823 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:515:13
    #8 0x5647193255c4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:280:6
    #9 0x56471932c271 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:703:9
    #10 0x564719341d62 in main third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #11 0x7f3aff398bbc in __libc_start_main (/usr/grte/v4/lib64/libc.so.6+0x38bbc)
    #12 0x564718e525c8 in _start /usr/grte/v4/debug-src/glibc-2.19/csu/../sysdeps/x86_64/start.S:108

0x629000008a20 is located 14368 bytes inside of 16520-byte region [0x629000005200,0x629000009288)
freed by thread T0 here:
    #0 0x564718f0c442 in __interceptor_free third_party/llvm/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:68:3
    #1 0x564719228d2b in destruct_transport(grpc_chttp2_transport*) third_party/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:204:3
    #2 0x56471923edb7 in destroy_transport_locked(void*, grpc_error*) third_party/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:572:3
    #3 0x5647192d9491 in grpc_combiner_continue_exec_ctx() third_party/grpc/src/core/lib/iomgr/combiner.cc:261:5
    #4 0x5647192e12e3 in grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #5 0x564718f2a89e in grpc_core::ExecCtx::~ExecCtx() third_party/grpc/src/core/lib/iomgr/exec_ctx.h:85:5
    #6 0x56471928e470 in grpc_server_shutdown_and_notify third_party/grpc/src/core/lib/surface/server.cc:1219:1
    #7 0x564718f24a39 in LLVMFuzzerTestOneInput third_party/grpc/test/core/end2end/fuzzers/api_fuzzer.cc:892:11
    #8 0x564719336823 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:515:13
    #9 0x5647193255c4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:280:6
    #10 0x56471932c271 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:703:9
    #11 0x564719341d62 in main third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #12 0x7f3aff398bbc in __libc_start_main (/usr/grte/v4/lib64/libc.so.6+0x38bbc)
    #13 0x564718e525c8 in _start /usr/grte/v4/debug-src/glibc-2.19/csu/../sysdeps/x86_64/start.S:108

previously allocated by thread T0 here:
    #0 0x564718f0c997 in __interceptor_calloc third_party/llvm/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:97:3
    #1 0x56471931aa15 in gpr_zalloc third_party/grpc/src/core/lib/gpr/alloc.cc:68:7
    #2 0x564719232985 in grpc_create_chttp2_transport(grpc_channel_args const*, grpc_endpoint*, bool) third_party/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:3121:31
    #3 0x564718f97620 in on_handshake_done(void*, grpc_error*) third_party/grpc/src/core/ext/transport/chttp2/client/chttp2_connector.cc:117:9
    #4 0x5647192e15e3 in exec_ctx_run(grpc_closure*, grpc_error*) third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #5 0x5647192e1370 in grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #6 0x564718f2a89e in grpc_core::ExecCtx::~ExecCtx() third_party/grpc/src/core/lib/iomgr/exec_ctx.h:85:5
    #7 0x5647192d0581 in grpc_timer_manager_tick() third_party/grpc/src/core/lib/iomgr/timer_manager.cc:104:1
    #8 0x564718f24647 in LLVMFuzzerTestOneInput third_party/grpc/test/core/end2end/fuzzers/api_fuzzer.cc:808:5
    #9 0x564719336823 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:515:13
    #10 0x5647193255c4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:280:6
    #11 0x56471932c271 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:703:9
    #12 0x564719341d62 in main third_party/llvm/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #13 0x7f3aff398bbc in __libc_start_main (/usr/grte/v4/lib64/libc.so.6+0x38bbc)
    #14 0x564718e525c8 in _start /usr/grte/v4/debug-src/glibc-2.19/csu/../sysdeps/x86_64/start.S:108

SUMMARY: AddressSanitizer: heap-use-after-free third_party/grpc/src/core/lib/gpr/mpscq.cc:49:43 in gpr_mpscq_pop_and_check_end(gpr_mpscq*, bool*)
Shadow bytes around the buggy address:
  0x0c527fff90f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c527fff9140: fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9150: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9160: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9170: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c527fff9190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==498769==ABORTING

The issue is that -

A timer is being set with a callback to &t->retry_initiate_ping_locked in maybe_initiate_ping. source
A ref on transport is not being taken when we initialize the timer.
This means that if the timer fires after the transport is destroyed then it will result in a heap use-after-free resulting in possible SEGFAULTs

To fix this, #14739 takes a ref before initializing the timer and unrefs it at the end of the function retry_initiate_ping_locked


Can`t trade on DEX with TW

Hi!
Recently, I decided to use DEX in Trust Wallet, but found one problem...
When I try to open an order in TrustWallet via DEX, I get the message "Account not found". (Screenshot attached)
Dear developers, how to solve this issue?

duplicated with https://github.com/trustwallet/wallet-core/discussions/1287


How to remove JFXDialog content ?

Hello,
is there a method to remove the content from a JFXDialog ?
I couldn't find any, so everytime I do dialog.setContent(content), it adds one more content on foreground. Am I doing it the wrong way ?

Hello,
well the implementation of setContent should call the method setAll instead of add, will fix it soon and push the changes.
Thanks for pointing out the issue 👍
Regards,

The fix will be pushed to JFoenix 1.6 by the end of the month, meanwhile you can get the fix by downloading the latest JFoenix jar.
Regards,


Make Iterant's ++ take a lazy (by-name) parameter

Currently the signature for ++ for Iterant is this:
sealed abstract class Iterant[F[_], A] {
  //...

  def ++[B >: A](rhs: Iterant[F, B])(implicit F: Applicative[F]): Iterant[F, B]
}

We need to change it to take a by-name parameter:
def ++[B >: A](rhs: => Iterant[F, B])(implicit F: Applicative[F]): Iterant[F, B]

Note that via this change we might slow down operators that depend on ++, so we need to provide a strict alternative. Might as well provide a function on the companion object:
object Iterant {
  // ...

  def concat[F[_], A](xs: Iterant[F, A]*): Iterant[F, A] = ???
}

This needs to be implemented in the most efficient way possible, so it should not be in terms of flatMap, but in terms of the current ++ implementation.
Also search and replace in the code-base any usage of ++ with concat (I don't think there are any, but should be double checked).

Just commenting that one of the Scala Spree participants is working on this issue. :)

Resolved in #656


New query param required in vimeo url.

Hello. I need to pass a new parameter to the url, some sort of security token. I have an url like https://player.vimeo.com/video/${id}?h=${token}.
Thanks.

https://www.npmjs.com/package/@conquerplus/react-native-web-video

Hey, @jfbn we will be publishing the new version (1.1.0) of the library these days that add this feature.
Regards, Philip


Allow ad-hoc inserts using tuples

This provides a more lightweight API for inserts that mirrors what is
possible with .set for updates. The implementation provided ensures
that all elements within the tuple are valid for the same table, and
that there's no funkiness like sticking an upsert value in there.
(Secret way to break Diesel -- It will totally allow you to stick a
Vec in there, and will generate invalid SQL as a result. We should fix
that one eventually, but it's pretty low priority)
One thing I would also like to allow is to allow mixing ad-hoc values
and Insertable structs. (See 10374a5026bcab2202f7f67f057493c4636d547b
for rationale). I tried a naive approach where I moved the parens out
from the InsertValues impl on tuples, and into InsertStatement, but
that breaks PG upsert, so I'll need to spend more time on it. I've left
the test cases for that in place, but I will defer actually making them
pass for a later PR in order to keep the PR size and scope reasonable.
I expected to need more tests/compile-tests, but what's included here
felt like it pretty exhaustively tests everything.
Fixes #789

Going to mark the failing test as #[should_panic], as its solution is the same as the other 2 mentioned in the PR description (the issue is that we're currently generating VALUES (('foo'), ('bar')) which is valid with most expressions, but invalid with DEFAULT.


Feat: 771 remove unnecessary security concerns

fixes #771

feat: reduce privileges granted through SCC for OpenShift
feat: set mounted volumes as read only as much as possible
feat: set pod hostPID to true to avoid collisions/unwanted behavior in host PID namespace
feat: remove unnecessary mount of /sys/kernel/debug

Same improvements were already merged in the kepler-operator: https://github.com/sustainable-computing-io/kepler-operator/pull/185
And the removal of the /sys/kernel/debug was discussed here: https://github.com/sustainable-computing-io/kepler-model-server/pull/158#discussion_r1323177090

@BGrasnick would you mind if I took a stab at addressing the comments and submit another PR?

Hey @sthaha! Sorry for the delay! I was away for a while but now I am back and ready to jump back in :)

Thank you @BGrasnick , the PR is quite useful :)


AccordionItem styles for container are not applied

The container of the AccordeonItem cannot be styles via data-option-styles
Step to reproduce
Given this code, which is 95% a copy from the exemple.
When open/closed the custom btn class are applied, but not the container ones
{% embed<EMAIL_ADDRESS>    with {
      items: items,
      item_attr: {
        data_option_styles: {
          btn: {
            open: 'bg-grey-light rounded-t'
          },
          container: {
            open: 'bg-red',
            active: '',
            closed: 'rounded'
          }
        }
      }
    }
  %}
    {% block title %}

          {{ item.title }}

    {% endblock %}
    {% block content %}

        {{ item.content }}

    {% endblock %}
  {% endembed %}

Possible solution:
It seems to be missing some object spreading on the container transition ?
  async open() {
    // ...
    const { container, ...otherStyles } = this.$options.styles;

    /** @type {AccordionItemRefs} */
    const refs = this.$refs;

    await Promise.all([
      transition(refs.container, {
        from: { height: 0 }, // Missing ...container.closed
        active: container.active,
        to: { height: `${refs.content.offsetHeight}px` }, // Missing ...container.open
      }).then(() => {
        return Promise.resolve();
      }),
      ...Object.entries(otherStyles)
        .filter(([refName]) => refs[refName])
        .map(([refName, { open, active, closed } = { open: '', active: '', closed: '' }]) =>
          // ...
        ),
    ]);
  }

This was intended in the first draft of the component to avoid overriding the required height transition. The content ref should be used instead to define custom styles for the closed and open states.
We could find a workaround to allow custom styles on the container ref if it is really needed.

It is not mandatory to be able to style the container, workarounds with content & btn refs are okay.
But the source code is maybe missleading in AccordionItem.js with this declaration:
styles: {
  type: Object,
  /**
   * @return {AccordionItemStylesOption}
   */
  default: () => ({
    container: {
      open: '',
      active: '',
      closed: '',
    },
  }),
},

Because if you override it, it does nothing (for the ref container)

This is now possible to have a fluid transition using grid-template-rows
cf: https://codepen.io/DHawku/pen/vYzLNVK?editors=1100
Shall we implement it as the default behavior ?

oh yeah !


Issue with building after install

I'm getting a Build Error (Funnel) issue after installing the addon + option 2 of the installation options (we have to serve the sprite sheets locally for offline use). The error reads:
ENOENT: no such file or directory, lstat 'undefined/emojione-defs'
If there's any other information I can supply to help resolve this I'm happy to give it. Thanks

Hey @Houndsto0th!
I've just tried installing the addon to a fresh Ember CLI 3.1.3 app. I see some deprecation warnings, but no build errors.
What's the ember-cli version of your app? What's your ember-cli-build.js config?

my ember-cli-build.js is:
/* global require, module */
var EmberAddon = require('ember-cli/lib/broccoli/ember-addon');

module.exports = function (defaults) {
    var app = new EmberAddon(defaults, {
        // Add options here
        'ember-emojione': {
            // Did you install individual files or the full package?
            separatePackages: true,
            // Do you want EmojiOne CSS to be included in your app?
            shouldImportCss: false,
            // Are you going to use components that insert emoji?
            shouldImportDefs: false,
            // Whether to use a sprite sheet or individual images
            spriteSheet: true,
            // Enable one of these options if you want to include
            // EmojiOne assets into your app's distro.
            shouldIncludePngSprite: false,
            shouldIncludeSvgSprite: true,
            shouldIncludePngImages: false,
            shouldIncludeSvgImages: false,
            pngImagesKind: 'png',
            svgImagesKind: 'svg',
            packageNameMain:        'emojione',
            packageNameJs:          'emojione-js',
            packageNameSvgSprite:   'emojione-svg',
            shouldIncludeComponents: false,
          },
    });

    /*
     This build file specifies the options for the dummy test app of this
     addon, located in `/tests/dummy`
     This build file does *not* influence how the addon or the app using it
     behave. You most likely want to be modifying `./index.js` or app's build file
     */

    app.import('bower_components/moment/moment.js');
    return app.toTree();
};

and I'm on "ember-cli": "~2.18.2",

Please also post the relevant part of your bower.json.

This might be caused by a different problem that I'm working through right now. I'll either comment back or close this based on the next hour or so of troubleshooting. I appreciate the quick responses

So, after pulling a fresh branch I'm still getting the error with:
Build failed.
Build Error (Funnel)

ENOENT: no such file or directory, lstat 'bower_components/emojione-defs'

This happened after following the install steps:
yarn add -D ember-emojione
then
bower install -S emojione

So, after following the other install step I'm able to get further it seems but I still run into an issue with the svg spritesheet
ENOENT: no such file or directory, lstat 'bower_components/emojione-svg/assets/sprites'
 "dependencies": {
    "emojione-js": "https://raw.githubusercontent.com/Ranks/emojione/v2.2.7/lib/js/emojione.js",
    "emojione-css": "https://raw.githubusercontent.com/Ranks/emojione/v2.2.7/assets/css/emojione.css",
    "emojione-svg": "https://raw.githubusercontent.com/Ranks/emojione/v2.2.7/assets/sprites/emojione.sprites.svg",
    "emojione-defs": "https://raw.githubusercontent.com/Ranks/emojione/v2.2.7/emoji.json"
  }

is what my bower.json looks like

One more thing to note, it looks like if I use the png sprite sheet that works just fine but using the svg sheet seems broken

I confirm this is a bug.
I have attempted a fix in the fix-51-svg-sprite-import-path branch. Can you please try it via:
yarn add -D Deveo/ember-emojione#fix-51-svg-sprite-import-path

I'll release it after you confirm it resolves the issue.

I can confirm switching to this branch allowed me to run tests with my ember-cli-build.js having the svg import set to true and the png set to false.

I'm not sure this is related but it looks like the svg sprite sheet is being brought in as bower_components/emojione-svg/index.svg instead of the emojione.sprites.svg file and that's interfering with the _generateTreeForSvgSprite method in your index.js

I'm not sure this is related but it looks like the svg sprite sheet is being brought in as bower_components/emojione-svg/index.svg

This is expected when you choose not install the whole emojione package and instead go for
bower install -S emojione-svg=https://raw.githubusercontent.com/Ranks/emojione/v2.2.7/assets/sprites/emojione.sprites.svg

The problem you were seeing was caused by the addon using the whole-package path to the SVG file instead of the individual import path.
I'm gonna release the fix later. Meanwhile, please use the addon from the branch.

So how can I fix the file import for the svg sprite sheet?

Oh, I thought using the fix-51-svg-sprite-import-path branch fixed the issue for you. 😦
Can you please tell me what error you see on that branch?

it looks like the svg sprite sheet file is called index.svg but the addon looks for it to be named emojione.sprites.svg

Maybe it's that the installation instructions aren't comprehensive for using the svg sprite sheet?

That may be true, but please describe what's happening for you on the branch. Any specific error message?

I just get a 404 on the localhost:4200/assets/sprites/emojione.sprites.svg

Any update on this? I've just added an app.import statement pointing to the index.svg file for now


cmd/go: go.mod file got overridden after go build

What version of Go are you using (go version)?

$ go version
go version go1.12.5 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/svilgelm/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/svilgelm/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/svilgelm/Library/Mobile Documents/com~apple~CloudDocs/workspace/k8ss/iam_api/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9x/_3ch6yp158dcsmh8b_4f17tr0000gn/T/go-build142036667=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
I'm trying to add gocloak module to mine as a dependency but something strange happens. Can you please explain to me what I'm doing wrong and why this is the result?
add the import in my go file like this
import "github.com/Nerzal/gocloak"
add this to my go.mod file
require github.com/Nerzal/gocloak v2.0.0
I build with go build and my go.mod file got overridden like this
require github.com/Nerzal/gocloak v0.0.0-20190525151100-dfd1e50572f8

What did you expect to see?
require github.com/Nerzal/gocloak v2.0.0
What did you see instead?
require github.com/Nerzal/gocloak v0.0.0-20190525151100-dfd1e50572f8

the github.com/Nerzal/gocloak repo has v2.0.0 tag, but the go build/test always override the version in go.mod file to v0.0.0-20190525151100-dfd1e50572f8

If you do not wish to for your go.mod to be updated, you can do a build using the -mod=readonly flag, i.e.:
go build -mod=readonly .

Thanks, but why the versions of other packages are not changed? Build changes only the line with gocloak?
require (
	github.com/DATA-DOG/godog v0.7.13
	github.com/Nerzal/gocloak v0.0.0-20190525151100-dfd1e50572f8
	github.com/dgrijalva/jwt-go v3.2.0+incompatible
	github.com/golang/mock v1.3.1
	github.com/golang/protobuf v1.3.1
	github.com/google/uuid v1.1.1
	github.com/grpc-ecosystem/go-grpc-middleware v1.0.0
	github.com/inconshreveable/mousetrap v1.0.0 // indirect
	github.com/mattn/go-runewidth v0.0.4 // indirect
	github.com/mitchellh/go-homedir v1.1.0
	github.com/olekukonko/tablewriter v0.0.1
	github.com/spf13/cobra v0.0.3
	github.com/spf13/pflag v1.0.3
	github.com/spf13/viper v1.3.2
	golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f
	golang.org/x/lint v0.0.0-20190409202823-959b441ac422 // indirect
	golang.org/x/net v0.0.0-20190522155817-f3200d17e092
	golang.org/x/sys v0.0.0-20190529130038-5219a1e1c5f8 // indirect
	golang.org/x/text v0.3.2 // indirect
	golang.org/x/tools v0.0.0-20190529010454-aa71c3f32488 // indirect
	google.golang.org/grpc v1.20.1
	gopkg.in/resty.v1 v1.12.0
	gopkg.in/yaml.v2 v2.2.2
)

On possible reason could be because you have gocloak set to v2 in go.mod and your import statement does not have a v2 suffix then go build assumes uses the latest commit that is a <2 version and updates the go.mod to point to that commit and version, one solution would be to change your imports from:
import "github.com/Nerzal/gocloak"

to
import "github.com/Nerzal/gocloak/v2"

This is documented here:

Go Modules - Semantic Import Versioning

build iam/pkg/cmd/iam_api: cannot load github.com/Nerzal/gocloak/v2: cannot find module providing package github.com/Nerzal/gocloak/v2

Why build doesn't override the github.com/DATA-DOG/godog v0.7.13or github.com/dgrijalva/jwt-go v3.2.0+incompatible lines? We import these packages without v0 or v3 suffixes

Update your go.mod file package to also have v2 suffix, when the package version is below 2 then package suffixes are not needed, i.e., in go.mod:
require github.com/Nerzal/gocloak/v2 v2.0.0

May I recommend opening a help topic in the golang forums?, it seems like this is not an issue with the go project.
I'll follow up there when time permitting, the forums can be located here:

https://forums.golangbridge.org/

I think the reason this is happening is because github.com/Nerzal/gocloak is a v2 module, but is missing the required /v2 at the end of its module statement in its go.mod:
https://github.com/Nerzal/gocloak/blob/v2.0.1/go.mod#L1
I'd suggest opening an issue on that repository.
#31543 and other issues cover making this a bit clearer. (See "Factor 2" section in #31543 for a bit more).
CC @tbpg @jayconrod @bcmills

Why build doesn't override the github.com/DATA-DOG/godog v0.7.13or github.com/dgrijalva/jwt-go v3.2.0+incompatible lines? We import these packages without v0 or v3 suffixes

v0 and v1 packages do not require the major version suffix to import/use them. However, once you get beyond v2, you have to include the major version.
This is a new requirement, so in order to facilitate backwards compatibility with repos that already had v2+ tags, Go supports +incompatible on the version. This makes it so you can use a v2+ tag even if the repository doesn't follow semantic import versioning. Once you create a module for the repo, it should follow the semantic import versioning (SIV) rule, or you end up with the pseudo-version you see in this issue.
The gocloak repo is not following this rule right now. Go should make it easier to understand, find, and prevent this error, as @thepudds pointed out.
I think we can close this issue in favor of #31543.

Guys, thank you so much for the help, it's little bit unclear


feat: add configurable JSON-LD namespaces/context for each scope

What this PR changes/adds
Add configurable JSON-LD namespaces/prefix according to the DR #3509
Why it does that
Flexibility and clean API
Linked Issue(s)
Implements #3508
Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.

Codecov Report
Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (ffcdfb6) 72.03% compared to head (12d94cc) 72.11%.
Report is 5 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3516      +/-   ##
==========================================
+ Coverage   72.03%   72.11%   +0.07%
==========================================
  Files         834      833       -1
  Lines       16896    16974      +78
  Branches      948      956       +8
==========================================
+ Hits        12171    12240      +69
- Misses       4320     4329       +9
  Partials      405      405

Files
Coverage Δ

...edc/web/jersey/jsonld/JerseyJsonLdInterceptor.java
92.59% <100.00%> (+0.28%)
:arrow_up:

...pi/configuration/DspApiConfigurationExtension.java
97.82% <100.00%> (+0.20%)
:arrow_up:

...rialization/JsonLdRemoteMessageSerializerImpl.java
86.66% <100.00%> (+0.95%)
:arrow_up:

...iguration/ManagementApiConfigurationExtension.java
70.83% <100.00%> (+1.26%)
:arrow_up:

...n/java/org/eclipse/edc/jsonld/JsonLdExtension.java
75.75% <ø> (-2.63%)
:arrow_down:

...eclipse/edc/protocol/dsp/DspHttpCoreExtension.java
92.59% <0.00%> (ø)

...in/java/org/eclipse/edc/jsonld/TitaniumJsonLd.java
90.90% <92.00%> (-0.48%)
:arrow_down:

...c/main/java/org/eclipse/edc/jsonld/spi/JsonLd.java
0.00% <0.00%> (ø)

... and 23 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


MGMT-9524 Persist domain

Open questions:

Should we block the UI until the changes get effective?
Should we handle more componentRoutes in the ingress/cluster resource?
How to handle certificates of remaining routes (just spec.host of Route resources are patched)?

Description
Persist domain name change. Implemented on the backend to reduce risks with network during the complex flow.
Fixes # (issue)
Type of change
Please select the appropriate options:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] This change is a documentation update

Testing
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Configuration:

Versions:
Hardware:

Checklist

[ ] I have performed a self-review of my own code
[ ] If a change is adding a feature, it should require a change to the README.md and the review should catch this.
[ ] If the change is a fix, it should have an issue. The review should make sure the comments state the issue (not just the number) and it should use the keywords that will close the issue on merge.
[ ] A change should not be merged unless it passes CI or there is a comment/update saying what testing was passed.
[ ] PRs should not be merged unless positively reviewed.

:tada: This PR is included in version 1.1.12 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


to_csv doesn't download all rows

No error was thrown.  269 of 399 rows were downloaded. Code is right out of the example at https://pypi.org/project/gsheets/:
 s.sheets[0].to_csv('members.csv', encoding='utf-8', dialect='excel')

The workbook contains multiple sheets.  I haven't tried deleting all the other sheets (have to go through the OAuth pain), so that could be it (although the 4 other sheets are small).  There is one cross-page link from the sheet I want to another sheet.
Is there a way I can get the second half (specify a row range)?

I created a single sheet workbook. Results were identical. I am having better luck with pygsheets so won't be pursuing this further.

Thanks. Can you maybe provide a minimal reproducible example or share a workbook with some (e.g. dummy) data that allows to reproduce this?

Closing as not reproducible for now.


How to set context in AWS Lambda handler?

My code works fine with this code:
const {dialogflow} = require('actions-on-google');
const app = dialogflow();
// bunch of app.intent()
exports.handler = app;

Now I want to set context.callbackWaitsForEmptyEventLoop in AWS Lambda handler function in order to make my backend code work, and I did some search and tried this:
const {dialogflow} = require('actions-on-google');
const app = dialogflow();
// bunch of app.intent()
exports.handler = async function(event, context) {
    logger.info("got event: " + event)
    context.callbackWaitsForEmptyEventLoop = false;
    try {
        let body = JSON.parse(event.body);
        let headers = event.headers;
        let response = await app.handler(body, headers)
        logger.info(response)
        return response
    } catch (err) {
        logger.info("err: " + err)
    }
};

The code above provides a good response, but dialogflow received the following:

Webhook call failed. Error: 502 Bad Gateway

The documents only give the line exports.fulfillment = app which is not informative about how to set the content inside.
Can someone please help me with this? Any help is appreciated.

Found the solution in #219 :
const app = dialogflow()
app.frameworks.lambda.responder = (event, context, callback, result) => {
  if (result.status >= 200 && result.status < 300) {
    callback(null, result.body)
  } else {
    callback(JSON.stringify(result))
  }
}
// ...
exports.handler = app

Closing this issue.

Hi @luomanke currently that code isn't implemented yet in the library. You would have to use something like
exports.handler = async (event, context, callback) => {
  const body = event.body || event
  const headers = event.headers || context.request.headers || []
  const result = await app.handler(body, headers);
  if (result.status >= 200 && result.status < 300) {
    callback(null, result.body);
  } else {
    callback(JSON.stringify(result));
  }
}

The code in the documentation is for AWS Lambda API Gateway/HTTP Proxy integration.

Hi @Canain , I just found out that my solution is not correct, this part of code is not working (the program is still using the line exports.handler = app ):
app.frameworks.lambda.responder = (event, context, callback, result) => {
  if (result.status >= 200 && result.status < 300) {
    callback(null, result.body)
  } else {
    callback(JSON.stringify(result))
  }
}
exports.handler = app

Than I tried the code you provided as well, and it reported and error: 'No intent was provided and fallback handler is not defined.'
Do you have any idea what might be the issue here? Thanks in advance.

Your app can't do anything unless you define either at least one intent or a fallback handler. This is described in the documentation.

Hi @igilham , thanks for the response.  Can you tell me something else?`)
})

exports.handler = async (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;
  const body = event.body || event;
  const headers = event.headers || context.request.headers || [];
  const result = await app.handler(body, headers);
  if (result.status >= 200 && result.status < 300) {
    callback(null, result.body);
  } else {
    callback(JSON.stringify(result));
  }
}

If I use exports.handler = app, it works perfectly. But somehow in my case app.handler did not get these app.intent() above.

Found the solution in this link. Basically it's because the line const result = await app.handler(JSON.parse(body), headers); provided a JSON object that the AWS API Gateway cannot understand.
The result generated in my code gives:
{
        "status": 200,
        "headers": {
            "content-type": "application/json;charset=utf-8"
        },
        "body": {
            "followupEventInput": {
                "name": "ask-something",
                "languageCode": "en"
            }
        }
}

But what the API Gateway needs is something like this:
{
        "statusCode": 200,
        "headers": {
            "content-type": "application/json;charset=utf-8"
        },
        "body": "{\"payload\":{\"google\":{\"expectUserResponse\":true,\"systemIntent\":{\"intent\":\"actions.intent.SIGN_IN\",\"data\":{\"@type\":\"type.googleapis.com/google.actions.v2.SignInValueSpec\"}}}}}",
        "isBase64Encoded": false
}

3 fields went wrong here:

"status" should change to "statusCode".
"body" is followed by a JSON string instead of an object.
"isBase64Encoded" is not defined.

So the working code is like this:
exports.handler = async (event, context) => {
    context.callbackWaitsForEmptyEventLoop = false;
    const body = event.body || event;
    const headers = event.headers || context.request.headers || [];
    const result = await app.handler(JSON.parse(body), headers);

    if (result.status >= 200 && result.status < 300) {
        let response = {
            "statusCode": result.status,
            "headers": result.headers,
            "body": JSON.stringify(result.body),
            "isBase64Encoded": false
        }
        return response
    }
}

Can someone fix this issue in the library, so that the response generated from app.handler() function can be used in AWS API Gateway directly without reformatting.

@luomanke The code snippet there is just for people not using the AWS API Gateway mode.
Since you are using that, you should be able to just do
exports.handler = app

Which will respond in the format you described. When you are using app.handler, you are actually bypassing the lambda response generation and instead get the standard format for a generic web framework and thus need to manually transform the format.

@Canain , you are right, but in my case, I need to set
context.callbackWaitsForEmptyEventLoop = false;

in order to make sequelizejs to work properly in the AWS lambda function. If I use
exports.handler = app

there is no clue how to set context, so I have to construct a customized handler.

Try to see if this works:
app.middleware((conv, framework) => {
  if (framework.lambda) {
    framework.lambda.context.callbackWaitsForEmptyEventLoop = false
  }
})

@Canain This works as well. Thanks for your help!


Add BUILD.bazel files for drake_ros_viz and drake_ros_tf2

This will enable both drake-ros packages to be built in a bazel environment.
Note: When adding these package's BUILD.bazel files to drake-ros, inside the cc_library function, we should investigate using a recursive glob (and the exclude arg) to grab src files cleanly.

Merged in #200 and #202. Closing as completed.


Deployment

Hey you should deploy the project so people can access it, you can do it using netlify in a few clicks its super simple.
https://www.netlify.com/

Hey you should deploy the project so people can access it, you can do it using netlify in a few clicks its super simple.
https://www.netlify.com/

Hey totally forgot to do this thank you for raising this!
https://tanviroy-forkify.netlify.app/


no example provided on how to import from another cosmos db

Hi
Under "Select data source" these is no sample for below topics. the links go no were. Could you please provide information on how to import from other cosmos db?
Azure Cosmos DB collections
Azure Cosmos DB bulk import
Azure Cosmos DB sequential record import

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9edf1df2-588e-01ff-6d1c-234b8e51d550
Version Independent ID: 27873c08-b6cc-a52f-f277-ff9818a5e4d2
Content: Database migration tool for Azure Cosmos DB
Content Source: articles/cosmos-db/import-data.md
Service: cosmos-db
GitHub Login: @deborahc
Microsoft Alias: dech

@tkumark Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

Reference: The Azure Cosmos DB collections link does reference an example but is not very complete. The other two links simply reference the same page.

@tkumark the content to bulk import, and sequential record import is present but the links are broken. I fixed them now. Import from Azure Cosmos DB section has command line examples as well as UI walk through on how to select source and destination values. Hope it helps.
#Migrate data from one Azure Cosmos DB collection to another Azure Cosmos DB collections
dt.exe /s:DocumentDB /s.ConnectionString:"AccountEndpoint=;AccountKey=;Database=;" /s.Collection:TEColl /t:DocumentDBBulk /t.ConnectionString:" AccountEndpoint=;AccountKey=;Database=;" /t.Collection:TESessions /t.CollectionThroughput:2500
#Migrate data from multiple Azure Cosmos DB collections to a single Azure Cosmos DB collection
dt.exe /s:DocumentDB /s.ConnectionString:"AccountEndpoint=;AccountKey=;Database=;" /s.Collection:comp1|comp2|comp3|comp4 /t:DocumentDBBulk /t.ConnectionString:"AccountEndpoint=;AccountKey=;Database=;" /t.Collection:singleCollection /t.CollectionThroughput:2500
#Export an Azure Cosmos DB collection to a JSON file
dt.exe /s:DocumentDB /s.ConnectionString:"AccountEndpoint=;AccountKey=;Database=;" /s.Collection:StoresSub /t:JsonFile /t.File:StoresExport.json /t.Overwrite /t.CollectionThroughput:2500

#please-close

@tkumark We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Objects are only loaded, never saved.

Not implemented yet.

Began work on this with b5918ac8faad8441f4f63abe227df84c2ff3ab66.

Getting close, guilds still don't save at all and changes to players do not save.


How to restart CI?

@ocefpaf sorry to bother you again. The dependencies are built, but I am unsure how to restart the CIs for this feedstock. At least travis is not yet initialized apparently, so it asks me for a commit. Is there a way to trigger it without committing something senseless to the repo?

Solved. I figured out that I can use conda-smithy to do this.

@ocefpaf sorry to bother you again.

No problem. But sometimes it will be faster for you to reach me via gitter: https://gitter.im/conda-forge/conda-forge.github.io
Glad you figure it out!

Is there a way to trigger it without committing something senseless to the repo?

You should be able to restart Travis-CI via Travis web interface. But it may take a while for the repo to appear in your list. You can force it to sync though.


Some questions about Stem1D

Thank you for your outstanding work. I am trying to understand the structure of Stem1D in your paper, and I have a few questions. Could you please answer them?
If the input is an image of size 2242243, after the first Pointwise Convolution it becomes 22422464. I want to know how the height, width, and number of channels of the image change when passing through the following structure of Stem1D, which is indicated by the question mark in the uploaded image. It seems that you have divided all the channels into 8 groups, each with a different angle, and the output height and width are also different. For example, when the angle is 0, how does the height, width, and number of channels of the image change?It might even be impossible to get integers due to str = 2 according to your Definition1 calculation.
In various Blocks, the parameters of Depthwise Oriented Conv1d are str = 1, pad = kernel_size // 2. Why in Stem1D str = 2 and pad = (kernel_size - stride) // 2?

Hi, thank you for your interest in our paper!
The formula we chose for $P$ and $Q$ is: $P = \frac{H}{str}$ and $Q=\frac{W}{str}$ so in your notation each strided convolution takes as input a tensor of shape $H \times W  \times C$ and outputs a tensor of shape $\frac{H}{str} \times  \frac{W}{str} \times C$. We implicitly assume that $H$ and $W$ are divisible by the stride (for simplicity).
To go back to your example the 1st depthwise convolution takes as input a tensor of shape 224x224x64 and outputs a tensor of shape 112x112x64.
After the second depthwise convolution, the output is of shape 56x56x64.
The result of both convolutions leads to a $4\times$ reduction of $(H,W)$ which is what we would obtain by applying a standard stride=4 2d convolution.
We took inspiration from the paper Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs, Ding et al, https://arxiv.org/abs/2203.06717 to factor the stem this way.
For your second question, pad is used only to instantiate the right shape of weight from nn.Conv2d. It is not the same as $pad$ that you see in formula 1. Sorry for the confusion, I did not refactor the code properly before release.
To compute pad I used the standard formula for calculating the shape of the output activation in the case of strided convolutions with padding and took pad such that the output shape is exactly $\frac{H}{str} \times  \frac{W}{str} \times C$. You can check online for details about this formula for example: https://stackoverflow.com/questions/56450969/how-to-calculate-output-sizes-after-a-convolution-layer-in-a-configuration-file.
The formula I used works for stride=1 and stride=2 and should also work for other strides if I remember correctly but it's been a while and I'm not too sure anymore.
In practice, $pad$ which you see in formula 1 is always set to $floor(\frac{K}{2})$ because we want to center the convolution.
To reiterate, pad and $pad$ are independent. pad is used to compute the shape of weight and $pad$ is used in formula 1.
I hope this helps. Let me know if you have any other questions!

Thank you very much for your response. I would like to confirm if I understand correctly.

I should not use the formula in Definition 1 to calculate the output shape, but rather use the standard formula for calculation.

In Stem1D, padding = kernel_size // 2, instead of padding = (kernel_size - stride) // 2 as written in the code. Just like in the image I recently uploaded, if the kernel_size is 1×31, then padding = (0, 14).

I am not quite clear on how the angle is determined. For an angle of 0°, it represents the horizontal direction, for 45° it represents the anti-diagonal direction, and for 90° it represents the vertical direction. But what does a 22.5° convolutional kernel look like? Due to the limitations of the kernel size, can arbitrary angles not be rotated?

Additionally, in the PDF file I uploaded, I drew the structural diagrams of Stem, Block, ConvNeXt1D, etc. from your paper.  I would like to confirm if they are correct.

Oriented1D ConvNeXt.pdf
Once again, thank you for your response!

Hi, thank you for your interest in our paper! The formula we chose for P and Q is: P=Hstr and Q=Wstr so in your notation each strided convolution takes as input a tensor of shape H×W×C and outputs a tensor of shape Hstr×Wstr×C. We implicitly assume that H and W are divisible by the stride (for simplicity).
To go back to your example the 1st depthwise convolution takes as input a tensor of shape 224x224x64 and outputs a tensor of shape 112x112x64. After the second depthwise convolution, the output is of shape 56x56x64.
The result of both convolutions leads to a 4× reduction of (H,W) which is what we would obtain by applying a standard stride=4 2d convolution. We took inspiration from the paper Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs, Ding et al, https://arxiv.org/abs/2203.06717 to factor the stem this way.
For your second question, pad is used only to instantiate the right shape of weight from nn.Conv2d. It is not the same as pad that you see in formula 1. Sorry for the confusion, I did not refactor the code properly before release.
To compute pad I used the standard formula for calculating the shape of the output activation in the case of strided convolutions with padding and took pad such that the output shape is exactly Hstr×Wstr×C. You can check online for details about this formula for example: https://stackoverflow.com/questions/56450969/how-to-calculate-output-sizes-after-a-convolution-layer-in-a-configuration-file. This formula works for stride=1 and stride=2 and should work for all strides if I'm not mistaken.
In practice, pad which you see in formula 1 is always set to floor(K2) because we want to center the convolution. Note that we assume that the kernel size K is odd.
To reiterate, pad and pad are independent. pad is used to compute the shape of weight and pad is used in formula 1.
I hope this helps. Let me know if you have any other questions!

Thank you very much for your response. I would like to confirm if I understand correctly.

I should not use the formula in Definition 1 to calculate the output shape, but rather use the standard formula for calculation.

In Stem1D, padding = kernel_size // 2, instead of padding = (kernel_size - stride) // 2 as written in the code. Just like in the image I recently uploaded, if the kernel_size is 1×31, then padding = (0, 14).

I am not quite clear on how the angle is determined. For an angle of 0°, it represents the horizontal direction, for 45° it represents the anti-diagonal direction, and for 90° it represents the vertical direction. But what does a 22.5° convolutional kernel look like? Due to the limitations of the kernel size, can arbitrary angles not be rotated?

Additionally, in the PDF file I uploaded, I drew the structural diagrams of Stem, Block, ConvNeXt1D, etc. from your paper.  I would like to confirm if they are correct.

Oriented1D ConvNeXt.pdf
Once again, thank you for your response!

Yes that's correct
That's right (although I would argue that it is exactly as written in the code but in this file: https://github.com/princeton-vl/Oriented1D/blob/master/models/dwoconv1d/dwoconv1d.py#L133)
Definition 1 determines exactly the form of the kernel.
Basically the set of coordinates $(-(k-pad)\sin \theta, (k-pad) \cos \theta)$ define the coordinates of the kernel with respect to the center of the output pixel. When you map them out on a 2D grid you get something similar to figure 1 of the paper.
For example when $\theta = \pi / 4$, $\sin \theta = \cos \theta = \sqrt{2}/2$ so you get coordinates $((k-K/2) \sqrt{2}/2, (k-K/2) \sqrt{2}/2)$ which is a diagonal. More explicitly, say $K=7$, then $pad=3$ and $k=0, ..., 6$ so you get coordinates: $\sqrt{2}/2(-3, 3), \sqrt{2}/2(-2, 2), \sqrt{2}/2(-1, 1), \sqrt{2}/2(0, 0), \sqrt{2}/2(1,-1), \sqrt{2}/2(2, -2), \sqrt{2}/2(3, -3)$ which gives you a diagonal when you round to integer coordinates.
For $\theta = \pi / 8$, you get coordinates of $\approx (0.38*(k-K/2), 0.92*(k-K/2))$ which is a $\approx 22.5^\circ$ angle.
The idea behind this is to represent the points on a rotated line (=the kernel) through the equation that these points verify e.g. $x \cos \theta +  y \sin \theta = 0$ (= definition 1). This is usually called the implicit equation of a line and you can find some explanations on https://guzintamath.com/textsavvy/2018/05/04/implicit-equations-for-lines/ and https://stackoverflow.com/questions/27442437/rotate-a-line-by-a-given-angle for example. You can also check appendix D.1 and rotation matrices for more math details. I hope this explanation helps.
For speed we round down the coordinates (you could consider bilinear interpolation instead for example for full precision) so we cannot represent all angles due to this discretization. In particular small variations in angle for small kernels may not be noticeable. This might be part of the reason why having more than 8 directions is not useful.
We explore these kinds of questions in the appendix - if you're interested you can refer to appendices E.2 and D.2 of the paper.
I haven't checked too hard but it looks correct to me.

Thank you very much for patiently providing detailed explanations, which have been of great help to me and helped me understand how padding and angles are determined.

In the code here (https://github.com/princeton-vl/Oriented1D/blob/master/models/convnext.py#L150), padding = (kernel_size - stride) // 2, which has left me a bit confused.

Additionally, I attempted to use your code to run on other datasets. In an image classification task, I used a dataset for classifying five types of flowers https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz , and achieved an accuracy of 97%, which is fantastic!

However, in a semantic segmentation task, I tried to perform semantic segmentation on a crack dataset from Kaggle https://www.kaggle.com/datasets/lakshaymiddha/crack-segmentation-dataset/data , but the results weren't very good. It seems that all pixels are being classified as background, just like the image I uploaded. Is it because the crack pixels are too few in the dataset? Or is it because using the ImageNet classification Checkpoint as pre-trained weights is causing the poor performance? Or should I adjust the image preprocessing method or other parameters? Or is it possible that this network is not suitable for binary semantic segmentation of cracks and background?

If possible, could you please provide me with some advice? In any case, thank you very much!
I modified unilm/beit/semantic_segmentation/configs/base/datasets/ade20k.py into crack.py
dataset settings
dataset_type = 'CrackSegmentationDataset'
data_root = 'data/crack_segmentation'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
crop_size = (512, 512)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', reduce_zero_label=True),
# dict(type='Resize', img_scale=(2048, 512), ratio_range=(0.5, 2.0)),
dict(type='Resize', img_scale=(448, 448), ratio_range=(0.5, 2.0)),
dict(type='RandomCrop', crop_size=crop_size, cat_max_ratio=0.75),
dict(type='RandomFlip', prob=0.5),
dict(type='PhotoMetricDistortion'),
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size=crop_size, pad_val=0, seg_pad_val=255),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_semantic_seg']),
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(448, 448),
# img_scale=(2048, 512),
# img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(type='Normalize', **img_norm_cfg),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img']),
])
]
data = dict(
samples_per_gpu=4,
workers_per_gpu=4,
train=dict(
type=dataset_type,
data_root=data_root,
img_dir='images/training',
ann_dir='annotations/training',
pipeline=train_pipeline),
val=dict(
type=dataset_type,
data_root=data_root,
img_dir='images/validation',
ann_dir='annotations/validation',
pipeline=test_pipeline),
test=dict(
type=dataset_type,
data_root=data_root,
img_dir='images/validation',
ann_dir='annotations/validation',
pipeline=test_pipeline))
I modified unilm/beit/semantic_segmentation/configs/base/models/upernet_convnext.py
model
norm_cfg = dict(type='SyncBN', requires_grad=True)
model = dict(
type='EncoderDecoder',
pretrained=None,
backbone=dict(
type='ConvNeXt',
variant='1d',
in_chans=3,
depths=[3, 3, 9, 3],
dims=[96, 192, 384, 768],
drop_path_rate=0.2,
layer_scale_init_value=1.0,
out_indices=[0, 1, 2, 3],
),
decode_head=dict(
type='UPerHead',
in_channels=[128, 256, 512, 1024],
in_index=[0, 1, 2, 3],
pool_scales=(1, 2, 3, 6),
channels=512,
dropout_ratio=0.1,
num_classes=2,    # background and crack
norm_cfg=norm_cfg,
align_corners=False,
loss_decode=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)),  # default: False
auxiliary_head=dict(
type='FCNHead',
in_channels=384,
in_index=2,
channels=256,
num_convs=1,
concat_input=False,
dropout_ratio=0.1,
num_classes=2,   # background and crack
norm_cfg=norm_cfg,
align_corners=False,
loss_decode=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=0.4)),   # default: False
# model training and testing settings
train_cfg=dict(),
test_cfg=dict(mode='whole'))

Yes I agree that my code is not very clear, I apologize for that, I should overhaul everything at some point.
That's great thanks for sharing! :) Out of curiosity, how does it compare versus other models? (if you've run the comparisons)
Yes you need to prune the weights first to remove all uninitialized weights (see README section "Conversion") if you haven't done so.
I apologize for this, it is due to a mistake I made: I initially trained my model with kernel sizes [31,31,31,31] instead of the correct [31, 31, 27, 13] - I did not realize it would affect performance on downstream tasks. I stuck with my mistake because the [31, 31, 31, 31] model is the one I used to get my numbers and I did not have enough time to retrain the models before paper submission.
The issue is this: when the image size is 7x7, a kernel size of 31 is too big so some of the weights are never optimized and stay  random. This does not matter for image classification because the input sizes you see at training are the same as in evaluation but it impacts semantic segmentation because the segmentation image size is much bigger.
The hacky solution is to extract a model which only uses the weights from [31, 31, 27, 13]. I provide a script to do that.
The better solution would be to train the model directly with [31, 31, 27, 13] (you can also finetune from a pretrained [31, 31, 27, 13] backbone which I provide in the download links and was obtained from the hacky solution).
If you want to go with the second solution, you should change https://github.com/princeton-vl/Oriented1D/blob/master/models/convnext.py#L488 to reflect the new kernel sizes.
The convnext model defined for semantic segmentation uses those kernel sizes: https://github.com/princeton-vl/Oriented1D/blob/master/semantic_segmentation/backbone/convnext.py#L319

If you have more questions (about part 3 for instance) please let me know

Yes I agree that my code is not very clear, I apologize for that, I should overhaul everything at some point.
That's great thanks for sharing! :) Out of curiosity, how does it compare versus other models if you've run the comparisons?
I'm not too sure, do you have the same issue with a different model e.g. 2d convnext? Have you checked that the data is passed correctly? Data imbalance could definitively be an issue: https://mmsegmentation.readthedocs.io/en/latest/advanced_guides/training_tricks.html#class-balanced-loss presents a solution against that. Could also be that the data is very out-of-distribution in which case self-supervised learning methods would be better suited here (e.g. DINO)

Thank you very much for your two suggestions. Here are some of my thoughts:

I haven't precisely run other models, but I've roughly experimented with networks like VGG16, GoogLeNet, ResNet, etc. I will share the accurate results with you later. In general, Oriented1D ConvNeXt approaches the accuracy of ConvNeXt, which can achieve 100% accuracy.

Regarding the first suggestion, I tried 2dConvNeXt with convolution kernel sizes of [7,7,7,7], but its semantic segmentation performance on the crack dataset was not good. So, I don't think the issue lies with the 1d convolution kernel size of [31,31,31,31].

For your second suggestion, I believe the data is being correctly passed because the model's accuracy in background segmentation for the crack dataset gradually increased to 100%, while the accuracy for crack segmentation remained close to 0. I also attempted to use the method of adding weights as mentioned on the website you provided, but it didn't have much effect. I tried using U2Net for semantic segmentation on the crack dataset, and a rough run yielded a maxF1 of at least 0.69, indicating acceptable prediction performance. In the future, I may try to improve other models like U2Net with 1d convolution, although I'm concerned about my coding ability. I'll also attempt semantic segmentation on other datasets to assess their effectiveness.

I discovered a new problem. For θ=45°, sin θ≈0.70, cos θ≈0.70. This yields coordinates (-0.70(k-pad), 0.70(k-pad)) for K=7, pad=3, k=0,1,2,…,6: (2.1, -2.1), (1.4,-1.4), (0.7, -0.7), (0, 0), (-0.7, 0.7), (-1.4,1.4), (-2.1, 2.1). Rounding these coordinates to the nearest integer, we get (2,-2),(1,-1),(1,-1),(0,0),(-1,1),(-1,1),(-2,2). If we round down, we get (2,-2),(1,-1),(0,0),(0,0),(0,0),(-1,1),(-2,2). Should the paper use rounding these coordinates to the nearest integer or rounding down? It seems we cannot obtain the diagonal, and due to rounding, different values of k may yield the same coordinates, potentially transforming the K=7 convolution kernel into a K=5 convolution kernel.

Ok I see that's interesting thank you for sharing!

Hmmm I'm not too sure what could be wrong. Can you try using focal loss instead (to help with class imbalance)? Using smaller scale images (to virtually increase dataset size)? One thing you could do also is check the differences in setup between U2Net and ConvNeXt. Try to match modeling choices. if there's only architecture differences I guess you could try tuning the ConvNeXt architecture (adding more stages for larger receptive field, changing depth to make it easier to train or more expressivd, changing expansion factor, changing stem channels, ...) or playing with hyperparameters (learning rate, batch size, ...).

Yes you are correct. As you mention the current schema leads to repetition of certain weights due to rounding. I introduced bilinear interpolation and shearing to fix this issue but these approaches were not successful. I decided it would be easier to stick with this rounding as the goal was to prove the concept of oriented convolutions and in my opinion (you might disagree) this issue is not fundamental to oriented kernels but is an artifact of discretization.
For more details: 1) the bilinear interpolation was too costly to run so impractical. 2) my experiments suggested that there was no difference between oriented vs sheared kernels on ImageNet and it was more complicated to run and explain so I stuck with oriented convolutions. I might not have tested enough though and could have made a hasty conclusion.
As a side note, I guess this means that we can further optimize oriented kernels in terms of speed / accuracy?
As a random note, one practical fix would be to add a fixed-at-init (dx,dy) offset for every kernel index, aggregate it into a tensor and add support for it in implementation. It shouldn't affect performance too much. We could for example init those such that no kernel index overlap after rounding or init them at random following a uniform [-1,1] distribution.


Add computed voltage and current to aggregates from TEDAPI status data

v0.10.9- TEDAPI Voltage & Current

Add computed voltage and current to /api/meters/aggregates using system data from TEDAPI status payload.

Proxy test: jasonacox/pypowerwall:0.10.9t63-beta

@jasonacox - I think I've run into a wrinkle with this - at least assuming this is used for system vitals. I've only just upgraded to 4.4.6 (been busy) and while I can see the system vitals, the voltages being reported for solar, home and grid are around the 415V level, while the powerwall reports around 235V. I'm guessing the 415V is being calculated assuming 120V/phase? I'd expect all of the voltages to be around 230-240V locally.
I did look around both here and in Powerwall-Dashboard, but didn't see any issues addressing this. If there  is a live discussion, could you point me at it. If not, would you prefer an issue raised here or under Powerwall-Dashboard?

Yes, let's open an issue. Where do you see the bad voltages?  When you get a chance, send me the raw data you see for /api/meters/aggregates (e.g. http://localhost:8675/api/meters/aggregates)


Update TPLs on the RZ

Description
Resolves #19120

I updated run-build-visit to include rztopaz.
I then ran it and regenerated the TPLs under /usr/workspace/visit/visit/thirdparty_shared/3.4.0/toss4
I then updated the config site file for rztopaz to use the new library versions.

Type of change

[ ] Bug fix
[x] New feature~~
[ ] Documentation update
[ ] Other

How Has This Been Tested?

I built visit using the new config site file on the RZ. I ran it and visualized curv3d.silo.
Reminders:

Please follow the style guidelines of this project.
Please perform a self-review of your code before submitting a PR and asking others to review it.
Please assign reviewers (see VisIt's PR procedures for more information).

Checklist:

[ ] I have commented my code where applicable.
[ ] I have updated the release notes.
[ ] I have made corresponding changes to the documentation.
[ ] I have added debugging support to my changes.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have confirmed new and existing unit tests pass locally with my changes.
[ ] I have added new baselines for any new tests to the repo.
[x] I have NOT made any changes to protocol or public interfaces in an RC branch.~~

Merge w/ develop in #19133


Merge ContextConnection handling

Relates to #2838, #1261.
This is partially to fix the documentation and the obsolescence message on SqlConnectionStringBuilder.ContextConnection. It goes a little further though: setting ContextConnection to true will now always throw an ArgumentException, and manually specifying it in the connection string will now throw an explicit error highlighting that Microsoft.Data.SqlClient doesn't support it.
Doing this also meant merging the handling of it between .NET Core and Framework. Once that was done, I've reintroduced a number of places where references to SqlConnectionString.ContextConnection had been removed from the .NET Core code. This is a prerequisite to merge SqlDataReaderSmi et al.
One point of note here is that SqlConnectionString.ContextConnection (line 763 of SqlConnectionString.cs) is hardcoded to return false. This is so that the .NET JIT can optimise away any evaluation - example here.
This changes the public API surface slightly: it introduces the SqlConnectionStringBuilder.ContextConnection property to .NET Core, and changes the message on its Obsolete attribute on both platforms.

@edwardneal Please resolve conflicts so we can continue with reviews.
Thanks!

Thanks @cheenamalhotra - now resolved, pending CI.

I would vote for removing the property from .NET Framework too instead of adding and making it obsolete in .NET Core.. This property was first marked obsolete in v3.0.Does that sound acceptable @David-Engel ?

Thanks @cheenamalhotra. Part of the reason I picked this approach is to make a specific statement that SqlClient doesn't support connecting to SQL Server's context connection, so if the documentation is updated to make that lack of support clear then I don't have any objections to removing the property from .NET Framework (besides the API compatibility concerns - but a v6.0 release is probably our last opportunity to make this kind of breaking change for a while.)
The SqlConnectionStringBuilder.ContextConnection property also backs a Context Connection key in the connection string. If we decide to remove the property, I'd prefer to keep treating this as a valid key and to throw a specific exception if its value parses to true, since it's a behavioural change from S.D.S.

I would vote for removing the property from .NET Framework too instead of adding and making it obsolete in .NET Core.. This property was first marked obsolete in v3.0.
Does that sound acceptable @David-Engel ?

@saurabh500 - Do you think MDS will ever get into the CLR? If not, I agree we should just remove the ContextConnection property.

@David-Engel @cheenamalhotra dont know about forever. 😀
But this hasn't been on the radar or been tested in the past 5 years.
I will vote for removal from netfx.

@edwardneal Could you please update the PR as per comments from David and Saurabh?

Thanks all - I've just rolled back the related additions to .NET Core's codebase and removed the ContextConnection property from both codebases. There's one piece of new functionality: specifying Context Connection=true in the connection string will now throw a new exception which explicitly states that this behaviour isn't supported.
SQLCLR casts quite a long shadow here. I've handled the API surface in this PR, and will remove the various Smi classes and code paths once 6.0 is released. Once those are gone, I'll close my original issue 2838.

It's in the works, but there's a lot to review. An early draft comes to +1152, -8142 lines and I don't want to drop it in a single PR a month before v6.0. I'm thinking of introducing a set of four PRs post-release:

Remove SqlCommand/SqlDataReader Smi code paths and files
Remove SqlConnection Smi code paths and files
With all referencing code now removed, the SmiEventSink references are potentially no longer relevant. Remove these code paths and files if not
Remove any references within SqlUtil and the resource strings

@edwardneal
I will proceed to merge this as we're in a major version bump for 6.0, so this change is acceptable. Remaining cleanup I hope doesn't impact public surface area any further, so we can continue to track with #2838.

Thanks @cheenamalhotra. I've kept the 6.0 milestone in mind, and have front-loaded all of the surface area changes that my cleanup needs. The rest of the cleanup can wait until later.
I'm conscious that it's pretty unusual to completely delete part of the API surface, and if that's too disruptive for the release, I can easily submit changes which add it back.


Enable/Disable call speaker phone

WEBRTC-XXX - Ticket Description.

:older_man: :baby: Behaviors
Before changes
No implementation to control the call output from speaker phone to earpiece speaker.
After changes
// What is after
Added implementation in Call class to enable/disable the speaker phone.
✋ Manual testing

Create call or accept call invitation
execute enableSpeakerphone method in Call class from the TelnyxClient class

Thanks for the Community PR @icodelifee ! I'll test this and do the necessary version bumps to release on pub dev.

@icodelifee
0.0.6 released: https://pub.dev/packages/telnyx_webrtc


New package: ExtendableSparse v0.1.0

Registering package: ExtendableSparse
Repository: https://github.com/j-fu/ExtendableSparse.jl
Created by: @j-fu
Version: v0.1.0
Commit: e7f5335ab28aeb77c6f428117f7d197e583695c1
Git reference: master

You should have come compat entries for your dependencies.

You should have come compat entries for your dependencies.

Done. Thanks for the hint.


Calculation of projected graph edge 'weight' in network analysis is duplicative of 'length', may not be required

The introduction of an all pairs Dijkstra's shortest path function to support neighbourhood analysis in 2020 (https://github.com/global-healthy-liveable-cities/global-indicators/commit/b00df984256e4d445c1f4af65598e354ed85bbf2#), included the calculation of a weight attribute for edges.  This recast the existing length attribute associated with the projected network graph to an integer.  A comment suggested that the floating point value was recorded in a string, hence the recasting.  The sacrifice of precision, rounding to nearest meter, would have been intended to further speed up processing for larger cities.
However, re-approaching this step I can confirm that the current length attribute is a float data type; I'm also not convinced that the casting to integer at that point in the analysis is necessary or makes things quicker.
I believe these lines could be removed:
https://github.com/global-healthy-liveable-cities/global-indicators/blob/b0c5fd80d0c4200d83925bf6725595ef3eded39e/process/subprocesses/_11_neighbourhood_analysis.py#L107-L117
while this line could be amended to use 'length' instead:
https://github.com/global-healthy-liveable-cities/global-indicators/blob/b0c5fd80d0c4200d83925bf6725595ef3eded39e/process/subprocesses/_11_neighbourhood_analysis.py#L128-L130

Just did a quick A/B comparison of the two approaches for a reasonably large-ish network (Baltimore), that suggests there isn't much benefit from the simplification --- took about the same amount of time for this operation (~3 minutes and 50 seconds) using either approach:
Using the 'weight' integer measure:
>>> all_pairs_d = pd.DataFrame(
...             [
...                 (k, v.keys())
...                 for k, v in tqdm(
...                     nx.all_pairs_dijkstra_path_length(
...                         G_proj, neighbourhood_distance, 'weight',
...                     ),
...                     total=total_nodes,
...                     unit='nodes',
...                     desc=' ' * 18,
...                 )
...             ],
...             columns=['osmid', 'nodes'],
...         ).set_index('osmid')
                  : 78038nodes [03:47, 343.75nodes/s]

Using the orgiinal 'length' float measure:
>>> all_pairs_d = pd.DataFrame(
...             [
...                 (k, v.keys())
...                 for k, v in tqdm(
...                     nx.all_pairs_dijkstra_path_length(
...                         G_proj, neighbourhood_distance, 'length',
...                     ),
...                     total=total_nodes,
...                     unit='nodes',
...                     desc=' ' * 18,
...                 )
...             ],
...             columns=['osmid', 'nodes'],
...         ).set_index('osmid')
                  : 78038nodes [03:49, 340.71nodes/s]

For a particular node (49399486), the respective results from these were:
using weight:
>>> grid.loc[gdf_nodes.loc[all_pairs_d.loc[49399486].nodes, 'grid_id'].dropna().unique()].mean()
<stdin>:1: FutureWarning: The default value of numeric_only in GeoDataFrame.mean is deprecated. In a future version, it will default to False. In addition, specifying 'numeric_only=None' is deprecated. Select only valid columns or specify the value of numeric_only to silence this warning.
pop_est                     70.318841
area_sqkm                    0.016533
pop_per_sqkm              4253.351317
intersection_count           4.239130
intersections_per_sqkm     256.410812
dtype: float64

using length:
>>> grid.loc[gdf_nodes.loc[all_pairs_d.loc[49399486].nodes, 'grid_id'].dropna().unique()].mean()
<stdin>:1: FutureWarning: The default value of numeric_only in GeoDataFrame.mean is deprecated. In a future version, it will default to False. In addition, specifying 'numeric_only=None' is deprecated. Select only valid columns or specify the value of numeric_only to silence this warning.
pop_est                     69.007407
area_sqkm                    0.016533
pop_per_sqkm              4174.027114
intersection_count           4.237037
intersections_per_sqkm     256.284189
dtype: float64

So - no important difference in results (that was rationale for using the integer simplification, under assumption it improved analysis performance presumably) or time to process (in which case, there is no need to simplify to integer).

just ran the analysis with the above returning to using 'length' as cost (floating point) rather than 'weight' (integer, derived from length) for Las Palmas, with the following marginal results that indicate in addition to no practical difference in analysis time, there is no practical difference in results (and if there were, its a problem with the weight variable anyway).
tldr, there's no harm but also effectively no benefit to using the integer-simplified weight variable when evaluating neighbourhood distance to justify the additional code complexity for doing that

using weight (integer)
using length(float; from which 'weight' was   derived)
Absolute difference (length-weight)
Relative difference (length-weight)/weight

Area (sqkm)
103.3
103.3
0.000
0.0000

Population estimate
366501
366501
0.000
0.0000

Population per sqkm
3547
3547
0.000
0.0000

Intersections
6629
6629
0.000
0.0000

Intersections per sqkm
64.2
64.2
0.000
0.0000

urban_sample_point_count
37526
37526
0.000
0.0000

pop_pct_access_500m_fresh_food_market_score
48.81
48.81
0.000
0.0000

pop_pct_access_500m_convenience_score
51.02
51.02
0.000
0.0000

pop_pct_access_500m_pt_osm_any_score
82.91
82.91
0.000
0.0000

pop_pct_access_500m_public_open_space_any_score
70.43
70.43
0.000
0.0000

pop_pct_access_500m_public_open_space_large_score
43.67
43.67
0.000
0.0000

pop_pct_access_500m_pt_gtfs_any_score
78.54
78.54
0.000
0.0000

pop_pct_access_500m_pt_gtfs_freq_30_score
73.46
73.46
0.000
0.0000

pop_pct_access_500m_pt_gtfs_freq_20_score
65.86
65.86
0.000
0.0000

pop_pct_access_500m_pt_any_score
90.07
90.07
0.000
0.0000

pop_nh_pop_density
9354.00
9362.18
8.188
0.0009

pop_nh_intersection_density
176.13
176.17
0.042
0.0002

pop_daily_living
1.90
1.90
0.000
0.0000

pop_walkability
0.65
0.65
0.002
0.0024

local_nh_population_density
7089.62
7088.12
-1.508
-0.0002

local_nh_intersection_density
153.58
153.53
-0.044
-0.0003

local_daily_living
1.43
1.43
0.000
0.0000

local_walkability
-0.81
-0.81
0.000
0.0003