Configuration bearer component with azure ad results in dapr sidecar crash

Expected Behavior
Bearer component middleware with azure ad should work
Actual Behavior
Bearer component middleware component configuration with azure ad seems to crash dapr sidecar
Steps to Reproduce the Problem
dapr configuration
dapr status -k
  NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE  CREATED
  dapr-sidecar-injector  dapr-system  True     Running  1         1.1.2    2d   2021-04-26 09:53.23
  dapr-placement-server  dapr-system  True     Running  1         1.1.2    2d   2021-04-26 09:53.23
  dapr-sentry            dapr-system  True     Running  1         1.1.2    2d   2021-04-26 09:53.23
  dapr-operator          dapr-system  True     Running  1         1.1.2    2d   2021-04-26 09:53.23
  dapr-dashboard         dapr-system  True     Running  1         0.6.0    2d   2021-04-26 09:53.23

create azure ad app
API_APP_NAME=dapr-demo-app1
TENANT_ID=$(az account show --query tenantId -o tsv)
API_APP_URI_ID=https://$TENANT_NAME/$API_APP_NAME
API_APP_ID=$(az ad app create --display-name $API_APP_NAME --homepage http://localhost --identifier-uris $API_APP_URI_ID  -o json | jq -r '.appId')
APP_NAMESPACE="bearer"
kubectl create ns $APP_NAMESPACE

created bearer component
cat <<EOF | kubectl apply -f -
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: bearer-token
  namespace: $APP_NAMESPACE
spec:
  type: middleware.http.bearer
  version: v1
  metadata:
  - name: clientId
    value: "$API_APP_ID"
  - name: issuerURL
    value: "https://sts.windows.net/$TENANT_ID/"
EOF

created bearer configuration
cat <<EOF | kubectl apply -f -
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: bearer-config
  namespace: $APP_NAMESPACE
spec:
  httpPipeline:
    handlers:
    - name: bearer-token
      type: middleware.http.bearer
EOF

created deployment using the configuration
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-logger-bearer
  namespace: $APP_NAMESPACE
spec:
  replicas: 1
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: dummy-logger
  template:
    metadata:
      labels:
        app: dummy-logger
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "dummy-logger-bearer"
        dapr.io/app-port: "80"
        dapr.io/config: "bearer-config"
        dapr.io/log-as-json: "true"
        dapr.io/log-level: "debug"
    spec:
      containers:
      - name: dummy-logger
        image: denniszielke/dummy-logger:latest
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

container starts crash looping and this is the log of the dapr sidecar
kubectl logs dummy-logger-bearer-5c6d54ff95-hllw5  -n bearer daprd
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"starting Dapr Runtime -- version 1.1.2 -- commit 3148f36","scope":"dapr.runtime","time":"2021-04-28T16:02:20.308622204Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"log level set to: debug","scope":"dapr.runtime","time":"2021-04-28T16:02:20.308713607Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"metrics server started on :9090/","scope":"dapr.metrics","time":"2021-04-28T16:02:20.308936414Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"No Access control policy specified","scope":"dapr.configuration","time":"2021-04-28T16:02:20.352296594Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"kubernetes mode configured","scope":"dapr.runtime","time":"2021-04-28T16:02:20.352509101Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"app id: dummy-logger-bearer","scope":"dapr.runtime","time":"2021-04-28T16:02:20.352635406Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"mTLS enabled. creating sidecar authenticator","scope":"dapr.runtime","time":"2021-04-28T16:02:20.352946016Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"trust anchors and cert chain extracted successfully","scope":"dapr.runtime.security","time":"2021-04-28T16:02:20.353128122Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"authenticator created","scope":"dapr.runtime","time":"2021-04-28T16:02:20.353154223Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"Initialized name resolution to kubernetes","scope":"dapr.runtime","time":"2021-04-28T16:02:20.353559337Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"loading component. name: kubernetes, type: secretstores.kubernetes/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.353758244Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"component loaded. name: kubernetes, type: secretstores.kubernetes/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.355744412Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"found component. name: bearer-token, type: middleware.http.bearer/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362596946Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"found component. name: oauth2, type: middleware.http.oauth2/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362626047Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"found component. name: my-oauth-component, type: middleware.http.oauth2clientcredentials/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362637147Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"found component. name: my-oauth-component, type: middleware.http.oauth2clientcredentials/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362689149Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"found component. name: bearer-token, type: middleware.http.bearer/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362700449Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"waiting for all outstanding components to be processed","scope":"dapr.runtime","time":"2021-04-28T16:02:20.36271395Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"debug","msg":"loading component. name: bearer-token, type: middleware.http.bearer/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362770552Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"component loaded. name: bearer-token, type: middleware.http.bearer/v1","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362811753Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"all outstanding components processed","scope":"dapr.runtime","time":"2021-04-28T16:02:20.362906756Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"enabled middleware.http.bearer/ http middleware","scope":"dapr.runtime","time":"2021-04-28T16:02:20.711610857Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"enabled gRPC tracing middleware","scope":"dapr.runtime.grpc.api","time":"2021-04-28T16:02:20.711723661Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"enabled gRPC metrics middleware","scope":"dapr.runtime.grpc.api","time":"2021-04-28T16:02:20.711761462Z","type":"log","ver":"1.1.2"}
{"app_id":"dummy-logger-bearer","instance":"dummy-logger-bearer-5c6d54ff95-hllw5","level":"info","msg":"API gRPC server is running on port 50001","scope":"dapr.runtime","time":"2021-04-28T16:02:20.711817564Z","type":"log","ver":"1.1.2"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16bde9a]

goroutine 1 [running]:
github.com/dapr/dapr/pkg/middleware/http.Pipeline.Apply(0xc0004a2048, 0x1, 0x1, 0xc00020c900, 0x1)
	/home/runner/work/dapr/dapr/pkg/middleware/http/http_pipeline.go:26 +0x3a
github.com/dapr/dapr/pkg/http.(*server).useComponents(...)
	/home/runner/work/dapr/dapr/pkg/http/server.go:106
github.com/dapr/dapr/pkg/http.(*server).StartNonBlocking(0xc0003ca0b0)
	/home/runner/work/dapr/dapr/pkg/http/server.go:60 +0x67
github.com/dapr/dapr/pkg/runtime.(*DaprRuntime).startHTTPServer(0xc0001dcd80, 0xdac, 0x1e61, 0x3b4d100, 0x1, 0xc0004a2048, 0x1, 0x1)
	/home/runner/work/dapr/dapr/pkg/runtime/runtime.go:720 +0x4bc
github.com/dapr/dapr/pkg/runtime.(*DaprRuntime).initRuntime(0xc0001dcd80, 0xc000324160, 0xa, 0xc000320790)
	/home/runner/work/dapr/dapr/pkg/runtime/runtime.go:324 +0x626
github.com/dapr/dapr/pkg/runtime.(*DaprRuntime).Run(0xc0001dcd80, 0xc0005dff40, 0x7, 0x7, 0x5f, 0xc000102000)
	/home/runner/work/dapr/dapr/pkg/runtime/runtime.go:204 +0x25e
main.main()
	/home/runner/work/dapr/dapr/cmd/daprd/main.go:134 +0xf9f

Release Note

RELEASE NOTE:

Hi, I'm having exactly the same issue with Dapr 1.5. @denniszielke, Have you managed to find a workaround?

Unfortunately no.


Generate json for the supported weight formats

As we discussed during our last meeting, it would be great if we have can get a list of meta data about our supported formats.
I propose that we generate a weight_formats_spec.json file from the CI:
{
  "consumers": {
     "deepimagej": {
       "name": "DeepImageJ",
       "icon": "....",
       ....
     }
      ...
  },
  "weight_formats": {
    "keras_hdf5": {
       "name": "Keras HDF5",
       "description": "Weight formats for models generated from Keras in HDF5 format",
       "consumers": ["imjoy", "deepimagej"]
    }
    ...
  }
}

@FynnBe could you also take a look at this?

I made a weights formats spec:
https://github.com/bioimage-io/spec-bioimage-io/blob/gh-pages/weight_formats_spec.json

where is the code that generated it?

I wrote it by hand ;) the packager now use it on bioimage.io
There are two parts, the consumers we need to make a list, then fetch the detailed information from: https://github.com/bioimage-io/bioimage-io-models/blob/gh-pages/manifest.bioimage.io.json
For the formats, we may need to construct it by hand, I would perhaps read from a source json file with all the hand configured settings, then automatically expand it into the full version and place it under gh-pages branch. Any better idea?

There are two parts, the consumers we need to make a list, then fetch the detailed information from: https://github.com/bioimage-io/bioimage-io-models/blob/gh-pages/manifest.bioimage.io.json

Why don't we add 'supported_weight_formats' to an application RDF (and maybe a generic_consumer flag to filter out specialized applications? Or we add RDF type: consumer?). Then discover all applications through manifest.bioimage.io.json  and create the weight formats overview?
This would allow application/consumer maintainers to directly 'sign up' for a weights format.
Also (eventually) the resulted weight formats overview should indicate if tests fail or pass for each consumer respectively...

It's a good idea to use RDF for describing the consumers, but we don't really need an additinal type, the application is already covers consumer software. Maybe we just need to add supported formats to the 'config' of the existing applications. We can also use that to sign up new formats.

One downside though, if some consumer decided to change an existing format, and that might cause issue for other consumer who use that, and we cannot easily prevent that. For now I would prefer a centralized approach to define new formats. Anyone who want to signup new formats need to send PR to the spec repo.

It would be great to validate the weight formats an application signs up for, though, no?

One downside though, if some consumer decided to change an existing format, and that might cause issue for other consumer who use that, and we cannot easily prevent that. For now I would prefer a centralized approach to define new formats. Anyone who want to signup new formats need to send PR to the spec repo.

That's another good reason to keep the weight formats themselves centralized and validatable...

I'll start with a manual mapping of {consumer: [supported weight formats]}, but I would prefer to incorporate this into the application spec soon...

Why not get it from the application RDF already? I think don't need a dedicated spec for application at the moment, we can just go for the generic config field. Can you add that already to the Ilastik application config? I can change the compilation script to verify whether a configured formats exists.
Then we only need a weights definition here.

Actually we could avoid the 'supported_weights_formats' field if we would just check every weights entry of every model with every application (given that we have full dynamic tests)...
then we have a 'supported by' table for every weight format of every model...

But why not add supported_weights_formats to the RDF (for type: application only/in a derived "Application RDF")?
Otherwise we enforce rules onto the config field or

Well, we can, but it’s might be better to have a explicit list, because
that’s a property of the application which should not depend on the models
it linked to. An application can support a new weight format even though
there is no model use that yet.
On Wed 23 Jun 2021 at 11:17, FynnBe @.***> wrote:

Actually we could avoid the 'supported_weights_formats' field if we would
just check every weights entry of every model with every application (given
that we have full dynamic tests)...
then we have a 'supported by' table for every weight format of every
model...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bioimage-io/spec-bioimage-io/issues/130#issuecomment-866673733,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADU3S6737PYKFM4C2IDHLTTUGRB3ANCNFSM47BKFCPQ
.

But adding supported_weights_formats to the (Application) RDF is an explicit list.  And this explicit list could of course contain a new weight format that no model uses yet as long as the validator knows this weight format. But adding a new weight format will always need a PR here to adapt the validator anyhow.

I'll make a draft PR to show you what I mean...

But why not add supported_weights_formats to the RDF (for type: application only/in a derived "Application RDF")?
Otherwise we enforce rules onto the config field or
I suppose we can make it "official" later on as well and use 'config' for now... it seems like we'll end up doing that anyway though...

I would prefer that we test the idea with config and then upgrade it later to a spec. Define spec before actually testing it often result in much less stable spec. It is fine at the beginning, but we should really try to avoid it as we have more users.
I would rather focus on the model RDF spec and generic RDF spec at the moment, branching out for other RDF types is not necessary at the moment and are likely complicate the development and maintenance.
But don't get me wrong, we do need to make things more standardize, but let's take it easy and test out ideas before upgrading to a spec.

alright, but we'll make it official in 0.4.0 (or 0.3.3 👀 ) 🚀 👍

config:  {supported_weights_formats: ["wf1", "wf2"]}

it is for now... (with 3 's')

config:  {supported_weights_formats: ["wf1", "wf2"]}

it is for now... (with 3 's')

Looks good! Can you then compille it from the manifest.bioimage.io.json and generate the weight_formats_spec.json in the current format? Just to be clear, I can potentially do this dynamically in the packager, but this will result in downloading the entire manifest.bioimage.io.json everytime in the packager.

alright, but we'll make it official in 0.4.0 (or 0.3.3 👀 )

I would leave it in config until we sort out other RDF types, there are datasets, and the new workflows may also need special keys that need to be taken into account.

Just to be clear, I can potentially do this dynamically in the packager, but this will result in downloading the entire manifest.bioimage.io.json everytime in the packager.

Also note that 0.4.0 is the model RDF version, the changes we are making here won't affect anything but the website and the packager.

sure, I just wanted to indicate that I would like to make this official rather sooner than later. also note that https://github.com/bioimage-io/spec-bioimage-io/pull/125 unifies the general and the model RDF format versions (to the model one)

Looks good! Can you then compille it from the manifest.bioimage.io.json and generate the weight_formats_spec.json in the current format?

will do, do you need the 'consumers' section in the model format as well or is that redundant?

also note that #125 unifies the general and the model RDF format versions (to the model one)

Let's discuss that here: https://github.com/bioimage-io/spec-bioimage-io/issues/140

will do, do you need the 'consumers' section in the model format as well or is that redundant?

Thanks! Yes, I will need that too.

@FynnBe To make the packager working, I had to add this lines in the CI: https://github.com/bioimage-io/spec-bioimage-io/blob/2eb5a4630c8bd2a374aa0c20d440a4219898bbf0/.github/workflows/build.yml#L72-L73
Please remove that when you can generate that file.

@FynnBe @constantinpape  Noticed another thing, when a model contains multiple formats that are supported by the same consumer, for example, this model contains two formats that are supported by ilastik, I am thinking that maybe we can make the supported_weight_formats ordered by priority. So we only propose to download the first matched format and consider that as the preferred format for the consumer. What  do you think?

Yes, although for download we might want to prefer formats that are the most cross-compatible, like onnx, no?
Regardless of what we offer as download, ordered priority for weights formats makes sense 👍

This one is fixed, closing now.


You are trying to offload the whole model to the disk. Please use the disk_offload function instead.

System Info
`Accelerate` version: 0.29.3
- Platform: Windows-10-10.0.19045-SP0
- `accelerate` bash location: c:\Users\Sardar Moazzam\AppData\Roaming\Python\Python310\Scripts\accelerate.exe
- Python version: 3.10.4
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.3.0+cpu (False)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- System RAM: 15.89 GB
- `Accelerate` default config:
	Not found

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[X] My own task or dataset (give details below)

Reproduction
import transformers
from transformers import AutoModelForCausalLM
import torch
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.float16, low_cpu_mem_usage = True).cpu()
from accelerate import disk_offload
disk_offload(model=model, offload_dir="alpha")
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
prompt = pipeline.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
terminators = [
pipeline.tokenizer.eos_token_id,
pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
outputs = pipeline(
prompt,
max_new_tokens=256,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])
Expected behavior
I am running the following code and it gives the following error:
""You are trying to offload the whole model to the disk. Please use the disk_offload function instead.""
for solving the error i added the following code (so the above given code is after the addition to solve the previous issue):
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.float16, low_cpu_mem_usage = True).cpu()
from accelerate import disk_offload
disk_offload(model=model, offload_dir="alpha")
but when i add this code in the my original one and try to run it that after some time of running  then the jupyter kernel crashes.  can you please also tell me that what is this ""offload_dir="alpha"". Can you please explain it to me

cc @SunMarc

Hi @Moazzamnamal, you are trying to load the model twice when calling pipeline and  AutoModelForCausalLM.from_pretrained. Just do :
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)

and it should load your model on cpu and disk since you don't have a GPU.

@SunMarc thanks let me try it

You should give "low_cpu_mem_usage" : "True" in order to prevent kernel crash while loading model to memory..
 model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct" pipe = pipeline(     "text-generation",     model=model_id,     model_kwargs={"torch_dtype": torch.bfloat16,                   "low_cpu_mem_usage" : "True",                   },     device="cpu", )


add str and repr to classes

[ ] Project
[ ] Service (don't include secrets)
[ ] Release
[ ] PR
[ ] Issue

* Service (**don't** include secrets)

Most of them have implemented __str__, do you want it to be used for __repr__?
Is the first and last letter of token ok? We already discuss that in #336

oh, so we probably just need repr then; reference:
In [24]: user.get_projects()
Out[24]:
[<ogr.services.pagure.project.PagureProject at 0x7f1c6c550850>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c550690>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c6f1410>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c6f1190>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c6f16d0>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c6f1310>,
 <ogr.services.pagure.project.PagureProject at 0x7f1c6c6f1250>]

not too helpful :D

* Is the first and last letter of `token` ok? We already discuss that in #336

sounds good

bump (needed a lot in tests)


Component Name shown as "Anonymous Component"

6.0.0-beta.2
Browser and OS info
Chromium Version 86.0.4240.111 (Official Build) Arch Linux (64-bit)
Steps to reproduce
Create Basic Vue 3 App and Component:
const app = Vue.createApp({
    data() {
        return {
            comments: [
                {
                    username: "alice",
                    content: "First Comment!"
                },
                {
                    username: "bob",
                    content: "Hello World!"
                }
            ]
        }
    }
});

app.component("comment", {
    props: {
        comment: {
            type: Object,
            required: true
        }
    },
    template: `

{{ comment.username }}

{{ comment.content }}

    `
});

const mountedApp = app.mount('#app');

        <comment v-for="comment in comments" :comment="comment">
        </comment>

What is expected?
I expect to see the component name (comment) in Vue Devtools. under Root
What is actually happening?
Vue Devtools returns "Anonymous Component" for each component instance

try to use named component.

Same problem, the name has been declared when registering the component

try to use named component.

app.component("comment", {
name: 'comment',  // component name declared in here.
props: {
comment: {
type: Object,
required: true
}
},
template:

{{ comment.username }}

{{ comment.content }}

});

Hi both, and thanks for your answers.
@TimRChen , am I missing something in v3?
Shouldn't this be optional at best, as the name is already declared as the first argument to app.component ?
https://v3.vuejs.org/guide/component-registration.html#component-names

Hi both, and thanks for your answers.
@TimRChen , am I missing something in v3?
Shouldn't this be optional at best, as the name is already declared as the first argument to app.component ?
https://v3.vuejs.org/guide/component-registration.html#component-names

That's the point

Hi, there. I found this method: https://github.com/vuejs/vue-devtools/blob/6d8fee4d058716fe72825c9ae22cf831ef8f5172/packages/shared-utils/src/util.js#L260-L269
For reference only

@TimRChen So if registration automatically sets the components name why does it need an additional name property to show up in dev tools as something other than Anonymous Component? Why doesn't dev tools show the name that the component was automatically given?

@TimRChen So if registration automatically sets the components name why does it need an additional name property to show up in dev tools as something other than Anonymous Component? Why doesn't dev tools show the name that the component was automatically given?

I don't know. but you can find anser in source code.

Please provide a runnable reproduction.

I'm getting this issue with the Home component of the Hello World project created by Vue CLI v4.5.11 (with settings: Vue 3, TypeScript, class-based components, router enabled).

I got this issue when using vue-styled-components. fixed it like this
import styled from "vue3-styled-components";
const Flexbox = styled.div`
   position: absolute;
   top: 0;
   left: 0;
   right: 0;
   bottom: 0;
 `;
// this line is important

Flexbox.name = Flexbox;

/*VUEJS*/
export default{
    components:{
        Flexbox
    }
}

I got this problem with the Icon component imported from @iconify/vue and setup script.
The fix is to register this component with a name in main.ts:
import { createApp } from 'vue'
import { Icon } from '@iconify/vue'
import App from './App.vue'

createApp(App).component('icon', Icon)

这是一封自动回复邮件。已经收到您的来信，我会尽快回复。

I have the same problem in 2022.
Latest stable devtools version (6.1.3) installed. Using Vue 3 with Single File Components and script setup.


Where is the details about  Prior Knowledge?

Great work!
But I have an important question. How to get the embedding vector of the schema Sc in Section "Schemata"?
So, I wonder if you would consider sending me the details about the Schema Representations.
Thank you very much! :)

Hi @clgx00, the embedding vectors s_c are trained jointly with the rest of the network as described in the paper. The code here includes all the required details. I'm not sure which other specific details are you looking for?
Maybe our follow-up paper Improving Scene Graph Classification by Exploiting Knowledge from Texts can also be of help.

您好，我已经收到您的来信，祝好


Update changefeed docs for protected timestamps

Fixes DOC-9241
This PR updates the changefeed docs to include some updates to how (by default) changefeeds interact with PTS.

New default cluster setting section
Deprecation of protect_data_from_gc_on_pause
Add the upgrade to v23.2 breaking change to the release notes.

Most new content rendered: Protect Changefeed Data
Updates to docs include moving the previous Garbage Collection and changefeeds section to its own page since this is a critical part of changefeed usage that customers need to understand.

TFTRs!


Problems encountered in parallel testing

Description
When I tried to run two UI automation test case in parallel, I encounter problems. Only one test case can run successful and another always failed.
Failed test case shows "Can't find some element , time out.". But if I run failed test alone, it will pass.
Can someone help solve this problem?
Environment
MSTest V2 1.3.2
OS: Windows 10
VS: Visual Enterprise 2017 Version 15.8.7

@pvlakshm I had reading the article. It seems only run in Parallel test with different .dll ?

Here's an extract from what @pvlakshm commented,
Partition tests in terms of a taxonomy as follows:
(1) “Pure Unit Tests” (can run in parallel)
(2) Functional Tests that can run in parallel with some modifications (e.g.: 2 functional tests trying to create/delete the same folder can be ‘fixed’ to remove the assumption that they have exclusive use to the folder).
(3) Functional Tests that cannot be modified to run in parallel (e.g.: 2 Coded UI tests doing mouse actions on the desktop, 2 functional tests writing to the Bluetooth/IR port, etc.)
You should not try to run tests in parallel unless the tests are controlling different application processes.


BrowserStack bug

Hi, I Just grabbed the latest build of your Key Checker and I was testing BrowserStack when I saw this:
Choose checkers: 7

You choose BrowserStack Access Key checker
Input your BrowserStack Username:
xxxxx

Input your BrowserStack Key:
xxxxx

Branch.IO Key and Secret invalid

Thank you.

@geeknik Thank you for reporting this issue!, I have fixed the bug

Thanks for the quick fix, however, GitHub is doing the same thing. When I enter GitHub details, it tells me that my Heroku API key is invalid.

The SauceLabs option tells me that my SendGrid API key is invalid as well.


use utility belt.

Ebert has finished reviewing this Pull Request and has found:

9 possible new issues (including those that may have been commented here).

You can see more details about this review at https://ebertapp.io/github/tyrchen/ex_pre_commit_hook/pulls/3.


added some bold text, full stop and some other stuff.

I hope all my changes make sense.

Thanks panquesito

Hi, would you mind going over the suggested changes by Panquesito7 again?


actions: bump setup-node

Applied mkpr on .github/workflows/*.yml
sed s/setup-node<EMAIL_ADDRESS>

:tada: This PR is included in version 6.11.30 :tada:
The release is available on:

npm package (@latest dist-tag)
v6.11.30
GitHub release

Your semantic-release bot :package::rocket:


I did some cleaning up and added some comments

I cleaned things up a bit in the blog.module.js file.
Also added a lot of comments in the postlist.service.js file to make all of those functions more useable.
Did a few other small things to improve my own code and get things running.

Changes in the postlist.service look more flexible now. :shipit:


WFK2-833 Fix phone validation flags

In the contacts-mobile-basic QS, the country flags in the phone validation did not display the correct county all the time.

Updated the plugin to bring in fixes.
Set the plugin to be destroyed before data retrieval and then initialize after data retrieval.
Removed the old plugin on a separate commit since they were listed as a "rename" on one commit.

I still can see the wrong flag in phone number in Firefox 35, even Chrome browser works as wanted. Is that js library browser/javascript version sensitive?

@joshuawilson hmm, I cleared cache and it started to work, but it seems to me a weird if it depends on cleared cache. WDYT?


Set up CI with Azure Pipelines

[skip ci]

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: 10132321 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Interactive mode is not teaching the bot in normal run mode

Rasa Core version:
0.82
Python version:
3.6.4
Operating system (windows, osx, ...):
windows
Issue:
As base I use stories which build a standard conversation flow now I try to train interactive exceptions and try to improve the bot in this way.
in the interactive mode the bot change the  weights and gets better results.
If I start the bot in the "productive mode" the bot don't contains any improvement.
Even if I use the export function and add the story one by one to the stories.md and train it new i don't get the same results as in the interactive mode.
I don't use the standard Keras Policy but the same in both cases.
Content of domain file (if used & relevant):

I'll close this for now, please let us know if you have any further questions.


WSL Causes Massive ammount of memory usage when running with docker, related to  #4166

Environment
My Windows build number: [run cmd.exe /c ver]:
'\\wsl$\Ubuntu\home\lav'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
Microsoft Windows [Version 10.0.19042.630]

My Distribution version: [On Debian or Ubuntu run lsb_release -r in WSL] :
Release:        20.04

Whether the issue is on WSL 2 and/or WSL 1: [run cat /proc/version in WSL]
Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Tue Jun 23 12:58:10 UTC 2020

Steps to reproduce

start wsl and docker desktop
run a docker compose and pull up any image
close the docker desktop from windows
memory usage still remains

WSL logs:
Expected behavior
After Closing docker, windows should claim the memory back from WSL
Actual behavior
vmmem still uses massive amount of ram

Workaround:

in a PowerShell, do
wsl --shutdown

Docker shows a popup "The WSL 2 backend was shutdown unexpectedly ..." Click "Yes".

If you missed the popup, just restart Docker.

Workaround:

in a PowerShell, do
wsl --shutdown

Docker shows a popup "The WSL 2 backend was shutdown unexpectedly ..." Click "Yes".

If you missed the popup, just restart Docker.


makePythonPoetryEnvironment

Problem to solve
makePythonPypiEnvironment internally uses poetry to determine the full dependencies tree and outputs a source-able template and generates its own lock format.
But this brings some drawbacks:

the dependencies format is transformed from poetry into a custom format https://github.com/fluidattacks/makes/issues/1149
the output is incompatible with nix standard python env and pkgs derivations making difficult to integrate https://github.com/fluidattacks/makes/issues/1161
pypi envs cannot be merged together into 1 python env. This causes that vscode settings util cannot be used to setup the full env plus brings some python env integration issues (same pkgs with diferent versions enabled)

Intended users
Python developers
Permissions and Security
Proposal

Support makePythonPoetryEnvironment that receives a common poetry-lock (solving #1149) outputs a nix python env derivation (like makePyprojectPackage, and solving #1161 and env integration)
try https://github.com/nix-community/poetry2nix
this util will replace makePythonPypiEnvironment

Test plan
What does success look like, and how can we measure that?
makePythonPypiEnvironment
Links / references
/label ~"type::feature"

I will address making makePythonPypiEnvironment use poetry standard files on this issue

New builtin makePythonEnvironment generates an error: home directory '/homeless-shelter' exists; please remove it to assure purity of builds without sandboxing on jobs using the arch amd64.
https://us-east-1.console.aws.amazon.com/batch/home?region=us-east-1#jobs/ec2/detail/aa714a44-0ef6-4419-a082-070f76eaeac1

@jmesa85 This is related to Nix not being able to run in sandbox mode on non-privileged containers


Update README URLs based on HTTP redirects

Created with https://github.com/dkhamsing/frankenstein
GitHub Corrected URLs

Was
Now

https://github.com/tcrayford/shoutout
https://github.com/yeller/shoutout

HTTPS Corrected URLs

Was
Now

http://github.com/asenchi/proclaim
https://github.com/asenchi/proclaim

http://github.com/jamesgolick/degrade
https://github.com/jamesgolick/degrade

Nice!

:smile:

@reneklacan for president!


Not sure if an issue or not

Per https://github.com/webmin/webmin/issues/498 Freebsd CPU info should be showing.
I don't see it listed on System Information page.
Provide the following information:

What is your server-side OS and its version?
FreeBSD 11
What is your client-side OS, browser and its versions?
Mac OS 10.12.4
What are the steps to reproduce the issue
Load system info page.

Thanks,
Wes

What's your Webmin version?

1.840

Interesting. Does it show in old themes?

I switched to the gray theme and it did not show

Hm, ok. I'll come back to you after I have a chance to test it.

Thanks. Like I said, not sure if it's an issue or not... :)

Can you give me the link to FreeBSD 11 iso? I'll install it first thing tomorrow and try to reproduce the bug.

https://www.freebsd.org/where.html

Funny, I couldn't find it last time I was looking for it. Thanks.

Installed FreeBSD 11 and latest Webmin.
I don't see CPU info in Authentic Theme just as well as in Gray Theme.
My best guess that installation requires some library as a dependency.
WebminCore does take care about displaying it, this should be addressed to Jamie I think. He might know why or fix it in case it's a bug.


Missing packages from search

First, thanks very much for this package!
I was just kicking the tires on its "discovery" capabilities for unregistered packages, and noticed the following apparent misses among my own packages (which are obviously natural candidates for me to try):

ArrayIteration.jl is missing from lookup("iteration") or lookup("sparse")
KernelTools.jl is missing from lookup("kernel") or lookup("halide")
ZMQancient.jl is missing from lookup("zmq")
Nothing shows up for lookup("hemireal")

@timholy Thanks for reporting, let me check.

@timholy I've checked the database and the packages have not been imported. The issue being that:

they are not official packages. That is they're not part of METADATA. These are included by default.

a GitHub search is performed, looking for Julia repos and looking for the text "Pkg." in the README. The GitHub search is a bit naive at the moment. Basically, I needed a way to somehow separate "installable" packages from "regular" Julia scripts. Hence I assumed that there will be some installation instructions, either in the form of Pkg.add(...) or Pkg.clone(...). Your packages READMEs don't include either.

Do you have any ideas on how to extend the search to differentiate packages from plain scripts? I'd be happy to extend the search to include it.

Another naive way of performing the search, but possibly better, would be to search for all Julia repos but only keep the ones ending in ".jl"?

I like the idea of keeping the ".jl" ones. (Speaking of which, this repo should be renamed PkgSearch.jl.)

You could also check for repos that have a test folder?

The test we use on Travis for language: julia is if the repo has a file at src/PkgName.jl, where PkgName is the repo name, minus the trailing .jl if present. Not sure if github search is capable of expressing this easily.

I've updated the GitHub search to look for Julia repos named with ".jl" - this seems to be a convention that's applied consistently. Yes, good point, I've renamed my packages as well 👍
So it definitely indexed a lot more this time, some problems still present, I'll keep digging.

ArrayIteration.jl is missing from lookup("iteration") or lookup("sparse")
a. julia> PkgSearch.lookup("iteration") (5th result, 1st page)
b. a search by "sparse" also finds it but there are so many packages.If you filter more, by
julia> PkgSearch.lookup("sparse array") it's the 2nd

KernelTools.jl is missing from lookup("kernel") or lookup("halide")
a. for lookup("kernel") shows up on page 5 now
b. lookup("halide") shows up, there's just one result

ZMQancient.jl is missing from lookup("zmq") - still missing, must check.

Nothing shows up for lookup("hemireal") - still nothing, will keep digging.

@tkelman Good one thanks, I'll give it a try to see if it works.


Articles not appearing on 10.15.7

I may be wrong but it seemed like this started happening after updating to 10.15.7. NNW will show a badge with unread articles, but when I click on the aggregate feeds, the badge disappears and no articles appear. Clicking on individual feeds will show old articles but nothing new.

What kind of account are you using?  Can you recreate this consistently?

No account. Just reading feeds on my local machine.

Are you using keyboard shortcuts to navigate any?

Same result as clicking. Unread badge disappears but no articles.
Also, a slight correction. When clicking on "Today", the badge remains (still no articles). The other ones will clear the badge.

Would it be possible for you to get a screen recording of this happening so that I can recreate the scenario on my computer?  I have to be able to recreate it to debug the problem.  (You need to compress the video into a zip file to attach it to this ticket.)

I have a screen recording along with logs. Do you have a way I could send this info to you in private?

Sure, just email it to<EMAIL_ADDRESS> I really appreciate the extra effort.

Sent. Thanks.

Ok, so as you mentioned in your email, we might have a database corruption problem.  This is a new one for me.
Let's try this:

Export your OPML file from the On My Mac account from the File menu.
Deactivate your On My Mac account in the Preferences.
Create a new On My Mac account and name it what ever you want.
Import your previously exported OPML file into the new account.

This will create a new database for you and we should be able to get closer to the root of the problem.  It could be a disk corruption issue, something in a feed that is corrupting your database, or some scenario where NNW is corrupting it.

That seemed to work. Let me know what information you need now.

I don't think we need anything more right now.  I think this was a one-time disk corruption issue that may or may not have had something to do with the upgrade to 10.15.7.
If it happens again, just comment on this ticket and @brentsimmons or I will see it and reopen the issue.


action: Add a support for large GHA runners

The newly introduced large runners support nested virtualisation [1].
To use the LVH action on it, we need to tweak a few things:

Install qemu/kvm dependencies
sudo /dev/kvm

[1]: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners
Signed-off-by: Martynas Pumputis<EMAIL_ADDRESS>

It works - https://github.com/cilium/cilium/actions/runs/3714104526/jobs/6297570169

After #28 we probably also want to use sudo tee in the DNS setup step rather than a plain echo.

After https://github.com/cilium/little-vm-helper/pull/28 we probably also want to use sudo tee in the DNS setup step rather than a plain echo.

The echo resolver happens inside an LVH VM, while the sudoes in this PR are needed for a hypervisor host.

Disregard my comment, I need more coffee


fix(deps): update module github.com/gruntwork-io/terratest to v0.41.18

This PR contains the following updates:

Package
Type
Update
Change

github.com/gruntwork-io/terratest
require
patch
v0.41.17 -> v0.41.18

Release Notes

gruntwork-io/terratest
v0.41.18
Compare Source
What's Changed

Feat: Add ability to get a random region for a specific service by @MoonMoon1919 in https://github.com/gruntwork-io/terratest/pull/1269

New Contributors

@MoonMoon1919 made their first contribution in https://github.com/gruntwork-io/terratest/pull/1269

Full Changelog: https://github.com/gruntwork-io/terratest/compare/v0.41.17...v0.41.18

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.8.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


No access tier column when opening a blob in a new tab

Storage Explorer Version: 1.1.0 RC
Platform: Linux
Architecture: x64
Build Number: 2018501.1
Commit: 754b96f7
Regression From: N/A
Steps to Reproduce:

Open a blob container which has files with access tiers
Open the blob container again via "Open in new tab"

Expected Experience:
The new editor has an access tier column
Actual Experience:
There is no access tier column

Out for PR for 1.1.0


Add dirty count for repo

What i mean by that, is to have a number following or preceding the symbol for a dirty repo. Kinda like tide does it

I really love hydro but this is one of the feature missing for me :(

Maybe a flag to show or hide this number?

Sure ! would love it!

I've submitted PR #44 for this issue. I didn't make the show/hide configurable since the ahead/behind numbers aren't configurable either.

No feature flag is a deal breaker for me as I don't intend to use this myself.

Fair enough. I've pushed another commit to the PR adding the feature flag. Does this suffice?

@jorgebucaran is there anything more you need for this PR?

See you at #44.


Datepicker , disable dynamically dates.

hi,
i need to disable specific dates into the datepicker.
The problem is that the list date to disable is dynamic and directly depends on user action.
In my scenario i'm working on a room reservation app, the user chose a room and then i  receive a list of 'already booked date' tha i need to disable in the datepicker.
the disabled date only appears if i change the month (that force the datepicker to rcall adin the disalbing function). Is it possible to manually trigger the datepicker to check again dates ?

+1 datepicker-popup solution.

+1

still actual

+1 datepicker-popup

Please do not only add +1 to issues without adding anything further to the discussion.  All that does is serve to clutter up the thread.  Thanks.

Here is my solution (I had to enable/disable holidays selection + colouring depending on checkbox state):

add $scope.$watch() to watch variable bound to checkbox
in $watch function write:

$scope.date = angular.copy( $scope.date ); // $scope.date is ng-model variable of datepicker
In this way the datepicker is forced to re-render itself.


DiskLruCache2

Goals

Read while writing.
Avoid I/O in synchronized blocks.
Only one file per entry.

API sketch
class DiskLruCache2 {
  Entry get(String key);
  Entry put(String key, ByteString metadata, Source upstream);
  // ... plus methods for sizing, trimming, iterating, etc.

  static class Entry {
    ByteString metadata();
    Source newSource();
  }
}

Implementation sketch
Each entry that is being written has these sources and sinks:

The upstream source passed into put(). This has a logical cursor starting at 0, representing the next byte to read.
The file sink. Data read from upstream is written to the file sink. Its cursor starts at 0 and is incremented when a flush completes successfully. Its cursor represents that last byte that is safe to read from the file.
The sources created by calls to newSource(). On each attempt to read it will read from the file (if its cursor is lower than the file cursor) or from the network otherwise. If it reads from the network it is responsible for writing to the file.

Readers cooperate with synchronization, with only one reading from upstream at a time. We should also add a buffer to contain up to N bytes preceding the upstream source’s cursor. Readers may prefer to read from this buffer instead of from the file for better performance. Also this makes it possible to reduce flushes to the file system.
Notes

What happens when a FileInputStream reads the last byte of a file? Does it eagerly EOF? Or does it continue to read bytes if further bytes are later written to the file?
The current DiskLruCache has CLEAN and DIRTY entries in its journal. We should do likewise. An entry can be made clean after the upstream source on a put reaches EOF. This requires a last flush to the file.
We want to be able to put() a new entry when an existing entry is already in place. This is necessary when a conditional GET misses.
We may also want to be able to put() new metadata without updating the rest of the cache. This might motivate putting the metadata ByteString at the end of the file.

Since 2 only comes once, I wanted to chime in with  high level  requests that would help a data loading library we are currently working on.
I'd like to be able to construct a DataStore<Request,Response>  The store would have the following interface:
Response get(request)//return from memory if present else disk if present else network
Response fresh(request)//return from network
notify()//listen for all responses (not just 1 key)

I'm not sure if it makes sense to encapsulate the memory cache inside of OKHTTP since you would want to cache the inflated models for speed of retrieval. This can't be done without passing a parser into the client
Also, ideally the disk cache policy would be configurable the way a Guava Cache is rather than forcing cache control on server (or spoofed in an interceptor):
diskLRUCache = CacheBuilder.newBuilder()
                .maximumSize(getCacheSize())
                .expireAfterAccess(getCacheTTL(), TimeUnit.MILLISECONDS)
                .build();

Comparing old to new data is something that would be helpful as well.  Say I have a data call that returns ID + timestamps of correction for all articles from today.  I want to be able to fetch that data from network and compare it to the last entry in the cache.  This could be solved by either allowing custom keys or having a way to recreate a cache key from request params.
Finally cache hits when dealing with timestamps is difficult. I had a project where every request sent needed to include current time. Maybe it would make sense to have a a way to mark request params as ignored when it comes to generating a cache key.
Thanks for tracking more flexible cache.

Here’s a tentative plan on the structure of the new cache’s index. This plan focuses on what happens on the file system; there’s also a model of the index in memory.
As always, I’m open to suggestions & ideas!
Cache Sequence ID
The cache is built around a long counter that’s incremented on every insert and on every access. This is what’ll power LRU. It’ll be called the sequence ID. It starts at 1.
Index
The index is a single file like index.ok. Unlike OkHttp’s current cache journal this file is edited in-place. There’s no rebuilding.
The index is:
64 byte header
N × 32 byte entry slots
The header is:
16 byte prefix like OkHttp index v1\n.
8 byte created at timestamp
8 byte entry slot count; should equal (file length - header length) / entry length
8 byte next ID to assign (initially 1)
8 byte request count
8 byte hit count
8 byte unused (for 64 byte alignment)
Each entry is:
16 byte SHA-256 hash of URL (truncated from 256 bits to 128 bits)
8 byte length (content + metadata), or the placeholder 0xffffffffffffffff if unknown
8 byte sequence ID
Creates
To insert an entry named by its hash:

Take a sequence number
Find an empty entry slot, creating one at the end if necessary
Write the entry the slot
Flush the index to disk
Create a file named by the hash in hex<PHONE_NUMBER>3344445555666677778888.ok
Later, once the entry is committed, update the entry’s length

Reads
To get an entry named by its hash, we:

Increment the request count
Look for a matching hash. If found and the file is readable:

Increment the hit count.
Assign the entry’s sequence number to a new sequence number

Updates
To update an entry already in the cache:

Assign the entry’s sequence number to a new sequence number
Assign the entry’s length to the placeholder length
Later, once the entry is committed, update the entry’s length

Deletes
To remove an entry from the cache:

Remove its file from the file system if it exists
Zero out its entry

Trims
While the cache size is too large, repeatedly remove the entry with the lowest sequence ID.
Loading the Cache
The most difficult part of index loading is that the index may be incomplete. We don’t force a flush after every change, and so it’s possible that several changes are missing from the index. This is expected.
As a rule of thumb, the index’s set of files is allowed to be a superset of the file system’s files. We never want a file on the file system that isn’t known by the index. But it’s okay if the index knows about a file that hasn’t been created yet, or a file that has already been deleted.
If the index header is malformed (no prefix or fails to satisfy other invariants) we are on a hostile filesystem. Refuse to cache anything.
Otherwise iterate the entries. Any entry with an placeholder length is a partial entry. Remove that entry.

@swankjesse Is there any intention to make an OkHttp Cache that is only in-memory or make the persisted data encrypted so as to better handle using the cache with secure data?

@ScottPierce it's a feature I’d like to do eventually. Maybe just a mechanism to let you transform the stream.

Hello guys, any updates on the issue?

Hi @swankjesse, are there any plans to provide an in-memory OkHttp Cache? That would be really useful for apps that don't want to (or are not allowed to) store any sensitive data on a filesystem.

As of 5.0 (currently Alpha) you can provide your own FileSystem (for example https://github.com/square/okio/blob/master/okio-fakefilesystem/src/commonMain/kotlin/okio/fakefilesystem/FakeFileSystem.kt) and pass it to the public Cache constructor.
https://github.com/square/okhttp/pull/6550/files

Yep. Our FakeFileSystem is a good in-memory file system for that.
But you may also consider doing an in-memory cache of the parsed response body (a model object) rather than it's encoded bytes. For example Picasso and Coil cache bitmap objects in memory, not their byte arrays. The drawback of this is it’s more work if your models aren't supported by a library like these.

What a fantastic benefit of the FakeFileSystem. Thanks for the suggestion!

Closing as low priority. I think tolerance to interrupts might be the main improvement we currently need.
Given Coil has reimplemented a Cache instead of OkHttp, it's likely we should look at a common pattern if we do.
@swankjesse re-open if this is still in scope.


SyncBot: Improve the solution to the infinite loop problem

https://www.flowdock.com/app/rulemotion/resin-propaganda/threads/2-k4y7dkMhxmVmpSLc4AHFIz-de
The metadata as a footer solution is not suitable for resin, and needs reconsidering.  The thread above contains some thoughts.

Raised this in architecture.


As a User I want methods to generate UUIDs

created by @dimus at https://gitlab.com/gogna/gnparser/-/issues/87

UUID is created by gnlib now

UUID is created by gnlib now


PHP Compatibility

Implementing PHP Compatibility test. Thanks @grappler for fixing composer issue :smile:

Looks good. What do you think about this layout?


feat: support pull request workflows

The devops-vars jobs are "push" focused. This change allows them to run on pull requests and determine values based on a workflow for pull requests.
Example of a Deployment

Example of a Teardown

Nice work on this!


Task - New descriptor

As a user I want a very basic form for creating new descriptors, augmented with the radial annotator at multiple points.
JSON endpoints (some of these are available from matrix_row_coder likely):

[x] Descriptor GET
[x] Descriptor POST
[x] CharacterState GET
[x] CharacterState POST
[x] GeneAttribute GET
[x] GeneAttribute POST

@mjy what about those two checkbox left?

@jlpereira Tricky.  You see the Gene attributes section?  We need client side composition of some logic that gets posted.  It's non trivial, but the idea is that we need to select sequence IDs, and then combine them in logical notitation (that's the and/or bit).  I can't remember the exact thing to post, but it's something like (1|2&(3|4) where '|' == or and '&' == and.  Still NOT a high priority.  We're also missing sequence smart selector.

Already done. Closing


Publish chart workflow is not pushing latest chart to gh-pages branch

This workflow is expected to publish the latest charts to the default  branch 'gh-pages' on release.
https://github.com/deislabs/ratify/blob/main/.github/workflows/publish-charts.yml
However the workflow runs without the error, but the branch doesn't seem to have the latest chart.
The workaround to is to merge changes from main into 'gh-pages' branch manually until this issue is resolved.

@dtzar , i saw there is a discussion to start a patch release to resolve #264. I noticed with the last release the charts were not automatically updated, this gh-pages will need to be manually updated until this issue gets fixed.

Yes, Was hoping to have Eric's PR merged and then just releasing 1.1.0-alpha.1 to resolve things.

This is resolved now with the latest release.


CodeCoverage is always enabled when configuring it with a .runsettings file, independend of /EnableCodeCoverage

Description
I have a project with a .runsettings to configure code coverage
<DataCollectionRunSettings>
  <DataCollectors>
    <!-- https://docs.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis -->
    <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0"
                   assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
      <Configuration>
        <CodeCoverage>
          <ModulePaths>
            <Exclude>
              <ModulePath>.*FakeItEasy.*</ModulePath>
              <ModulePath>.*GongSolutions.WPF.DragDrop.*</ModulePath>
            </Exclude>
          </ModulePaths>
        </CodeCoverage>
      </Configuration>
    </DataCollector>
  </DataCollectors>
</DataCollectionRunSettings>

In VSTS I have set up a Gated build and a CI build (which runs after Gated)
Gated will run only the impacted tests, CI all
I noticed that even though I have set my Gated build not to run Code Coverage, I still get CC at the end of the Build.
Looking through the code I figured that each collector has a flag IsEnabled https://github.com/Microsoft/vstest/blob/master/src/Microsoft.TestPlatform.ObjectModel/DataCollector/DataCollectorSettings.cs#L46
and if any is, that data collection will happen https://github.com/Microsoft/vstest/blob/master/src/Microsoft.TestPlatform.ObjectModel/DataCollector/DataCollectionRunSettings.cs#L71
This leads to vstest.console.exe ignoring the /EnableCodeCoverage parameter.
I tried disabling the DataCollector via enabled=false, which leads to no code coverage independent of /EnableCodeCoverage
Steps to reproduce
See description
Expected behavior
/EnableCodeCoverage should enable/disable the CC data collector or there should be some means of configuring the data collector without automatically enabling it
Actual behavior
Data collection can not be configured without specifically enabling/disabling it.
Environment
Visual Studio 2017 15.4.2
.NET Framework 4.6.1 project

@Mertsch you can use enabled="false" attribute in .runsettings to disable a CodeCoverage DataCollector by default. You can enable it by using /EnableCodeCoverage or `/collect:"Code Coverage" command line arguments.
More info here.

I used the below runsettings
<RunSettings>
  <DataCollectionRunSettings>
  <DataCollectors>
    <!-- https://docs.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis -->
    <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0"
                   assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" enabled="false">
      <Configuration>
        <CodeCoverage>
          <ModulePaths>
            <Exclude>
              <ModulePath>.*FakeItEasy.*</ModulePath>
              <ModulePath>.*GongSolutions.WPF.DragDrop.*</ModulePath>
            </Exclude>
          </ModulePaths>
        </CodeCoverage>
      </Configuration>
    </DataCollector>
  </DataCollectors>
</DataCollectionRunSettings>
</RunSettings>

And this command, I don't get any CodeCoverage report.

C:\code\github\harshjain2\vstest>"C:\code\github\harshjain2\vstest\artifacts\Debug\packages\Microsoft.TestPlatform.15.6.0-dev\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" .\test\TestAssets\SimpleTestProject2\bin\Debug\net451\SimpleTestProject2.dll /settings:C:\Users\harjain\Desktop\rs.runsettings

But when I use this command I get CodeCoverage report.

C:\code\github\harshjain2\vstest>"C:\code\github\harshjain2\vstest\artifacts\Debug\packages\Microsoft.TestPlatform.15.6.0-dev\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" .\test\TestAssets\SimpleTestProject2\bin\Debug\net451\SimpleTestProject2.dll /settings:C:\Users\harjain\Desktop\rs.runsettings /EnableCodeCoverage

@harshjain2 I did the same thing and there is no code coverage
2017-11-11T16:35:50.5001676Z ##[section]Starting: Test Assemblies
2017-11-11T16:35:50.5001676Z ==============================================================================
2017-11-11T16:35:50.5001676Z Task         : Visual Studio Test
2017-11-11T16:35:50.5001676Z Description  : Run tests with Visual Studio test runner
2017-11-11T16:35:50.5001676Z Version      : 2.2.3
2017-11-11T16:35:50.5001676Z Author       : Microsoft Corporation
2017-11-11T16:35:50.5001676Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2017-11-11T16:35:50.5001676Z ==============================================================================
2017-11-11T16:35:52.6064502Z Run the tests locally using vstest.console.exe
2017-11-11T16:35:52.6064502Z ========================================================
2017-11-11T16:35:52.6064502Z Test selector : Test assemblies
2017-11-11T16:35:52.6064502Z Test assemblies : **\*tests.dll,!**\obj\**,!**\packages\**
2017-11-11T16:35:52.6064502Z Test filter criteria : null
2017-11-11T16:35:52.6064502Z Search folder : C:\Build\_work\7\s
2017-11-11T16:35:52.6064502Z Run settings file : C:\Build\_work\7\s\Default.runsettings
2017-11-11T16:35:52.6064502Z Run in parallel : false
2017-11-11T16:35:52.6064502Z Run in isolation : true
2017-11-11T16:35:52.6064502Z Path to custom adapters : null
2017-11-11T16:35:52.6064502Z Other console options : null
2017-11-11T16:35:52.6064502Z Code coverage enabled : true
2017-11-11T16:35:52.6064502Z VisualStudio version selected for test execution : 15.0
2017-11-11T16:35:53.6242932Z ========================================================
2017-11-11T16:35:56.3898627Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @C:\Build\_work\_temp\624a5181-c6fe-11e7-8b4c-61552c82644e.txt
2017-11-11T16:35:56.6623766Z Microsoft (R) Test Execution Command Line Tool Version 15.0.26929.2
2017-11-11T16:35:56.6623766Z Copyright (c) Microsoft Corporation.  All rights reserved.
2017-11-11T16:35:56.6623766Z
2017-11-11T16:35:56.6623766Z vstest.console.exe
2017-11-11T16:35:56.6623766Z "C:\Build\_work\7\s\Project.PDF.Tests\bin\Release\Project.PDF.Tests.dll"
2017-11-11T16:35:56.6623766Z "C:\Build\_work\7\s\Project.Tests\bin\Release\Project.Tests.dll"
2017-11-11T16:35:56.6623766Z /Settings:"C:\Build\_work\7\s\Default.runsettings"
2017-11-11T16:35:56.6623766Z /EnableCodeCoverage
2017-11-11T16:35:56.6623766Z /InIsolation
2017-11-11T16:35:56.6623766Z /logger:"trx"
2017-11-11T16:35:56.6623766Z /TestAdapterPath:"C:\Build\_work\7\s"
2017-11-11T16:35:57.2429794Z Starting test execution, please wait...
2017-11-11T16:36:03.3779983Z Passed   [...]
2017-11-11T16:36:40.8104176Z Results File: C:\Build\_work\7\s\TestResults\VSTSBuildAgent_BUILDSERVER 2017-11-11 16_36_03.trx
2017-11-11T16:36:40.8104176Z
2017-11-11T16:36:40.8260419Z Total tests: 30. Passed: 30. Failed: 0. Skipped: 0.
2017-11-11T16:36:40.8260419Z Test Run Successful.
2017-11-11T16:36:40.8260419Z Test execution time: 43.3347 Seconds
2017-11-11T16:36:41.5209118Z ##[section]Async Command Start: Publish test results
2017-11-11T16:36:41.6416167Z Publishing test results to test run '7386'
2017-11-11T16:36:41.6416167Z Test results remaining: 30. Test run id: 7386
2017-11-11T16:36:44.0154675Z Published Test Run : https://account.visualstudio.com/Project/_TestManagement/Runs#runId=7386&_a=runCharts
2017-11-11T16:36:44.0154675Z ##[section]Async Command End: Publish test results
2017-11-11T16:36:44.0154675Z ##[section]Finishing: Test Assemblies

using
<package id="MSTest.TestAdapter" version="1.2.0" targetFramework="net461" />
<package id="MSTest.TestFramework" version="1.2.0" targetFramework="net461" />

@Mertsch I suppose the issue is resolved for you, can you please confirm.

@mayankbansal018 Did I miss something? There was no update and enabled="false" + /EnableCodeCoverage still leads to no code coverage and is no solution.
If there is no CC in .runsettings everything works fine.

@Mertsch behind the scenes /EnableCodeCoverage flag simply tries to add some default CodeCoverage configuration to runsettings if there is nothing in runsettings which would enable CodeCoverage.
In your case the runsettings file already contains the CodeCoverage section, so /EnableCodeCoverage flag will do nothing, it honors whatever setting user has mentioned in runsettings file, which in this case says that do not do code coverage as you have marked enabled = false.
I think the name /EnableCodeCoverage is misleading here causing users to assume the if it is set then codecoverage will always happen, which is not true in this specific case.
So for this case the solution would be that you only provide runsettings which contain code coverage settings only when you want it.

@mayankbansal018 I understand, would be nice to see this behavior changed.
I guess it would be a simple change to detect an existing CC setting and set the IsEnabled flag to true.
Having to maintain two different .runsettings files with (almost) identical content seems a rather bad solution.
Also your colleague said in https://github.com/Microsoft/vstest/issues/1278#issuecomment-343663679 that it works for him. I see he is using some kind of preview version. Maybe it will be resolved in VS 2017 15.5 / 15.6 ?

@Mertsch I just tried out what mycolleague said in #1278 (comment) , & it works fine in VS 2017.5, the issue should get resolved for you once you move to latest VS 2017.5
I'll close this issue, please do reopen if you still face the problem after upgrading to latest VS

@mayankbansal018 Thank you for the clarification, I will Upgrade to 15.5 as soon as it's released and try again.


Interop with Besu

Address following issues:
Setting up the Besu network
Asset Swap (Besu vs. Besu)
Asset Swap (Besu vs. Fabric)

Closing as this is duplicate of #31 .


Base64 coded Mails

Why are some rules failing in base64 coded Mails?
The documentation does not really explain encoded, converted and stripped of the content types:
The FAQ says that messages wil be encoded in UTF-8.
Choosing a different content filter may help?

Version 3.8.0 might fix this problem.


Improve rendering of story links in our docs

There were several broken links, caused by links being misrendered. I added several comments and removed unused parameters in the process, but the important bit is the chaneg to rendering of plan/test links, usualy pointed to by verified-by links. No longer adding suffixes or composing URLs, we rely on web_link completely, and if a link points to multiple objects, like a hierarchy of multiple tests with the same prefix, we render all of them. It turns out to be hard to get web link correctly for such a root node, and it provides nice list of all related links to the user.
Related to https://github.com/teemtee/tmt/issues/2868
Pull Request Checklist

[x] implement the feature

Just wonder, if we could provide (in the future) a better way to report a broken link. Currently make docs ends up with:
  File "/home/psss/git/tmt/tmt/utils.py", line 7021, in render_template
    return environment.from_string(template).render(**variables).strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 214, in top-level template code
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 113, in template
ZeroDivisionError: division by zero

Which does not give any hint about which link is broken. But, I guess, that would be material for a separate pull request.

Just wonder, if we could provide (in the future) a better way to report a broken link. Currently make docs ends up with:
  File "/home/psss/git/tmt/tmt/utils.py", line 7021, in render_template
    return environment.from_string(template).render(**variables).strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 214, in top-level template code
  File "/home/psss/.local/share/hatch/env/virtual/tmt/v_yBwOy-/docs/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 113, in template
ZeroDivisionError: division by zero

Which does not give any hint about which link is broken. But, I guess, that would be material for a separate pull request.

That’s the plan, a helper to provide nice exception with context and filenamed and stuff.

No need to wait for functional tests, only documentation changed.

@psss FTR, here's the followup: https://github.com/teemtee/tmt/pull/3009


Cleanup license.txt

Github does not currently recognize the license for this project.  A project I help maintain had the same problem a long time ago, and it was explained that GH license check system looks for the full license text.  Merely including the notice blurb in LICENSE or LICENSE.txt is insufficient.  This PR includes the full license text in a way that should be recognized by GH

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


The network takes a lot more disk space than expected

Hello, I am confused when it comes to the size of the network saved on the hdd. It takes up 2.4 GB on disk, when I would expect it to take only 810 MB.
When I initialize the model and save it to disk before learning, it really takes up 810 MB but once I train it, it takes up 2.4 GB what are the two additional parameters stored per weight and is it possible to save the model in the basic representation?
I am using Theano backend. Also, this network barely fits into my 6 GB VRAM, I have to use batch size of 48 so that it doesn't overflow. How many variables are there per weight? I am using Adam optimizer, so I would expect 4 variables: the weight/bias, the first momentum, the second momentum, the cumulative gradient. Then a single parameter per neuron for storing the current gradient w.r.t. its inputs but that's only a few. So that still adds up to 4 * 810 = 3240 MB but in reality, it takes 1.3 GB more VRAM. What is the overhead?
Thanks in advance!

Okay, so I have looked into the model save file and found out that the extra parameters are indeed the first and second momentum for each weight from the Adam optimizer. I presume that when one wants to save just the network for production, doing model.optimizer = None would be sufficient.
Concerning the VRAM memory, I am still not aware of any way how to predict how much space will the whole learning actually require. I guess it is a hard question to answer because different types of layers demand a different amount of scratch space for the intermediate calculations, so it probably depends also on the structure of the network not just on the number of parameters. Is that a correct assumption?

Use model.save_weights instead of model.save to save just the weights.


Potentially unreliable feature extraction on ELF sample

When running below rule on 294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc vivisect sometimes extract the localtime api feature and sometimes it doesn't. Is there a bug in the extractor, vivisect, or elsewhere?
rule:
  meta:
    name: get system time on Linux
    namespace: host-interaction/os/info
    authors:
      - "@ramen0x3f"
    scope: function
    examples:
      - 294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc:0x404970
  features:
    - and:
      - os: linux
      - or:
        - api: localtime

ref: https://github.com/mandiant/capa-rules/pull/654

agree that the feature is not extracted:

viv doesn't seem to be finding the surrounding function (sub_404970):

viv doesnt recognize the reference to function sub_40BB10 (that calls sub_404970)

prolog of that function is quite standard:

i haven't seen it work on my machine, but if it's an issue of timing or something, that's maybe not surprising. if you can try a few more times and confirm that it sometimes works then i'll dive more into viv.

it seems like the emucode analyzer might work here. alternatively, since the prolog is so clean, a simple prolog analyzer could work well (but viv doesn't have one yet, i dont think).

Interesting! No match on new workspace every time:
> while true; do rm ../tests/data/294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.elf_.viv; ~/venv/bin/python ../scripts/show-features.py ../tests/data/294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.elf_ | grep "api(localtime)"; done

Match, when resusing existing workspace!?
> while true; do ~/venv/bin/python ../scripts/show-features.py ../tests/data/294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.elf_ | grep -a "api(localtime)"; done
  insn: 0x404A45: api(localtime)
  insn: 0x404A45: api(localtime)
  insn: 0x404A45: api(localtime)

All exported features for a run that includes api(localtime):
feats.txt

emucode looks at pointers that point to a location:
https://github.com/vivisect/vivisect/blob/ff87de9947a5e91d11082ca1730fe9aa899200b1/vivisect/analysis/generic/emucode.py#L116
however, viv doesn't seem to create a location at 0x40BB10, so it never gets emulated. looking into this now.

root cause reported upstream. we'll be blocked on viv here, unless we want to introduce our own viv passes with capa to better handle things like this.


about the picture

Thanks for this wonderful benchmark.
I'm a bit confused, are these two parts written in reverse?

Thanks for catching that... it is indeed a typo.


Rearrange folder structures (#45)

This revision rearranges folder structures to make C# API.

Change Includes/Core, Sources/Core to Cores
Change Includes/API, Sources/API to APIs

Codecov Report

Merging #46 into master will not change coverage.
The diff coverage is 77.72%.

@@           Coverage Diff           @@
##           master      #46   +/-   ##
=======================================
  Coverage   77.46%   77.46%
=======================================
  Files         361      361
  Lines       16853    16853
=======================================
  Hits        13056    13056
  Misses       3797     3797

Impacted Files
Coverage Δ

Cores/Field/VectorField2.cpp
22.22% <ø> (ø)

Cores/Field/ConstantScalarField2.h
100% <ø> (ø)

Cores/Field/CustomVectorField3.cpp
23.61% <ø> (ø)

Cores/Emitter/VolumeParticleEmitter3.h
100% <ø> (ø)

Cores/Field/ScalarField2.cpp
22.22% <ø> (ø)

Cores/FDM/FDMMGLinearSystem3-Impl.h
73.33% <ø> (ø)

Cores/Emitter/VolumeGridEmitter2.h
100% <ø> (ø)

Cores/FDM/FDMMGLinearSystem2.cpp
87.69% <ø> (ø)

Cores/BoundingBox/BoundingBox2-Impl.h
100% <ø> (ø)

Cores/FDM/FDMUtils.cpp
100% <ø> (ø)

... and 773 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1230b9...860da65. Read the comment docs.


[BUG] explain on Dataproc does not show process running on the GPU, even though the UI does

When trying to run explain on the example join operation on Dataproc, the output shows the operations are not running on the GPU.  However the UI shows the operators on the GPU.
sraheja@sr-test5-m:~$ spark-shell
spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/01/21 18:04:24 WARN com.nvidia.spark.rapids.SQLExecPlugin: Installing extensions to enable rapids GPU SQL support. To disable GPU support set `spark.rapids.sql.enabled` to false
21/01/21 18:04:24 WARN com.nvidia.spark.udf.Plugin: Installing rapids UDF compiler extensions to Spark. The compiler is disabled by default. To enable it, set `spark.rapids.sql.udfCompiler.enabled` to true
Spark context Web UI available at http://sr-test5-m.c.rapids-spark.internal:38447
Spark context available as 'sc' (master = yarn, app id = application_1611251592859_0001).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/

Using Scala version 2.12.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_275)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val df = sc.makeRDD(1 to 10000000, 6).toDF
df: org.apache.spark.sql.DataFrame = [value: int]
scala> val df2 = sc.makeRDD(1 to 10000000, 6).toDF
df2: org.apache.spark.sql.DataFrame = [value: int]
scala> df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").count
res0: Long = 10000000
scala> df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").explain
df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").explain
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [a#40], [b#42], Inner
   :- Sort [a#40 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(a#40, 48), true, [id=#324]
   :     +- Project [value#2 AS a#40]
   :        +- SerializeFromObject [input[0, int, false] AS value#2]
   :           +- Scan[obj#1]
   +- Sort [b#42 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(b#42, 48), true, [id=#325]
         +- Project [value#8 AS b#42]
            +- SerializeFromObject [input[0, int, false] AS value#8]
               +- Scan[obj#7]

The Dataproc UI shows the operators running on the GPU:

eventLogs-application_1611251592859_0001.zip
And the Physical Plan in the details for the query are also shown to be on the GPU:
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- GpuColumnarToRow false
   +- GpuHashAggregate(keys=[], functions=[gpucount(1)], output=[count#25L])
      +- GpuShuffleCoalesce<PHONE_NUMBER>
         +- ShuffleQueryStage 3
            +- GpuColumnarExchange gpusinglepartitioning(), false, [id=#283]
               +- GpuHashAggregate(keys=[], functions=[partial_gpucount(1)], output=[count#28L])
                  +- GpuProject
                     +- GpuBroadcastHashJoin [a#12], [b#14], Inner, BuildLeft
                        :- BroadcastQueryStage 2
                        :  +- GpuBroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))), [id=#200]
                        :     +- GpuShuffleCoalesce<PHONE_NUMBER>
                        :        +- GpuCustomShuffleReader local
                        :           +- ShuffleQueryStage 0
                        :              +- GpuColumnarExchange gpuhashpartitioning(a#12, 48), true, [id=#105]
                        :                 +- GpuProject [value#2 AS a#12]
                        :                    +- GpuRowToColumnar TargetSize(2147483647)
                        :                       +- *(1) SerializeFromObject [input[0, int, false] AS value#2]
                        :                          +- Scan[obj#1]
                        +- GpuShuffleCoalesce<PHONE_NUMBER>
                           +- GpuCustomShuffleReader local
                              +- ShuffleQueryStage 1
                                 +- GpuColumnarExchange gpuhashpartitioning(b#14, 48), true, [id=#130]
                                    +- GpuProject [value#8 AS b#14]
                                       +- GpuRowToColumnar TargetSize(2147483647)
                                          +- *(2) SerializeFromObject [input[0, int, false] AS value#8]
                                             +- Scan[obj#7]

Running the same on Apache Spark 3.0.1 gives:
scala> df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").explain
== Physical Plan ==
GpuColumnarToRow false
+- GpuShuffledHashJoin [a#31], [b#33], Inner, BuildRight, false
   :- GpuShuffleCoalesce<PHONE_NUMBER>
   :  +- GpuColumnarExchange gpuhashpartitioning(a#31, 10), true, [id=#221]
   :     +- GpuProject [value#2 AS a#31]
   :        +- GpuRowToColumnar TargetSize(2147483647)
   :           +- *(1) SerializeFromObject [input[0, int, false] AS value#2]
   :              +- Scan[obj#1]
   +- GpuCoalesceBatches RequireSingleBatch
      +- GpuShuffleCoalesce<PHONE_NUMBER>
         +- GpuColumnarExchange gpuhashpartitioning(b#33, 10), true, [id=#228]
            +- GpuProject [value#8 AS b#33]
               +- GpuRowToColumnar TargetSize(2147483647)
                  +- *(2) SerializeFromObject [input[0, int, false] AS value#8]
                     +- Scan[obj#7]

Why would the explain output on GCP not show the operation on the GPU?
Sample command to spin up a dataproc cluster with spark-rapids 0.3.0 (replace YOUR-BUCKET with the name of a valid gcs bucket):
gcloud dataproc clusters create test-cluster \
--region europe-west4 \
--zone europe-west4-c \
--image-version=2.0.0-RC22-ubuntu18 \
--master-machine-type n1-standard-4 \
--num-workers 2 \
--worker-accelerator type=nvidia-tesla-t4,count=1 \
--worker-machine-type n1-standard-8 \
--num-worker-local-ssds 4 \
--initialization-actions gs://goog-dataproc-initialization-actions-europe-west4/gpu/install_gpu_driver.sh,gs://goog-dataproc-initialization-actions-europe-west4/rapids/rapids.sh \
--optional-components=JUPYTER,ZEPPELIN \
--metadata gpu-driver-provider=NVIDIA \
--metadata rapids-runtime=SPARK,cuda-version=11.0
--bucket {YOUR_GCS_BUCKET} \
--enable-component-gateway \
--properties=^#^spark:spark.yarn.unmanagedAM.enabled=false

Seem like the physical plan print out will be consistent if we turn dataproc's AQE off: by set spark.sql.adaptive.enabled=false in /usr/lib/spark/conf/spark-defaults.conf

Seem like the physical plan print out will be consistent if we turn dataproc's AQE off: by set spark.sql.adaptive.enabled=false in /usr/lib/spark/conf/spark-defaults.conf

This is not datatproc specifc, this problem appears when AQE is on.

This is not datatproc specifc, this problem appears when AQE is on.


⚠ BCR Bot has degraded performance

In ad16c79, BCR Bot (https://bcr.alternativasinteligentes.com/BCRApi/web/chatbot) experienced degraded performance:

HTTP code: 200
Response time: 1085 ms

Resolved: BCR Bot performance has improved in 5065916.


Using Markdown to create a list of items to complete

[ ] Turn on GitHub Pages
[ ] Outline my portfolio
[ ] Introduce myself to the world

Markup

[ ] prabhat

prabhat kumar

ok


Application error is overridden with MQTT::NotConnectedException

When some error (eg. JSON::ParserError) is raised in an application, Ruby reports MQTT::NotConnectedException instead of the actual cause.
Example
require 'em/mqtt'
require 'json'

EventMachine.run do
  c = ClientConnection.connect('test.mosquitto.org')
  JSON.parse("!$%&")  # raises json parse error
end

Expected Result
JSON::ParserError is shown
Actual Result
/Users/yhara/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/em-mqtt-0.0.4/lib/em/mqtt/client_connection.rb:109:in `unbind': Connection to server lost (MQTT::NotConnectedException)
        from /Users/yhara/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1465:in `event_callback'
        from /Users/yhara/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:202:in `release_machine'
        from /Users/yhara/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:202:in `ensure in run'
        from /Users/yhara/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:226:in `run'
        from c.rb:16:in `<main>'

Possible fix
You can check $! to know why #unbind is called. ($! holds the last exception raised and not yet handled by rescue)
 # class EventMachine::MQTT::ClientConnection
   def unbind
     timer.cancel if timer
     unless state == :disconnecting
-      raise MQTT::NotConnectedException.new("Connection to server lost")
+      if $!
+        # Connection is lost because of an application error
+      else
+        raise MQTT::NotConnectedException.new("Connection to server lost")
+      end
     end
     @state = :disconnected
   end

I have struggled with this problem a lot - it is one of the reasons why I haven't done much more development on it.
I thought that the problem was a mixture of exceptions and Deferrables.
Looking at your code - I think the exception should be caught before unbind - rather than just giving a better error message when the connection is dropped?


[flow-router-ssr] Is  still necessary when using Flow Router SSR?

It's not needed. I'm using it and it's adding that statement automatically.


The search Feature should also consider the Opendata.swiss Keywords entered via the Cube Creator

The search Feature should also consider the Opendata.swiss Keywords entered via the Cube Creator
Originally posted by @ortnever in https://github.com/visualize-admin/visualization-tool/issues/105#issuecomment-934355316

@ortnever here you mean the themes and organizations for example, or is there other keywords ?

Hello Patrick, I mean the opendata.swiss.Keywords in the metadata of the cube (not mandatory but quite useful). Or maybe these keywords are already taken into account by the search? I haven't really tested it.

Thanks for the feedback véronique.
No, right now those keywords are not included in the search. Since those keywords are not displayed anywhere in visualize.admin, we would also have to show them in the results and in the dataset metadata if the search works on them.
@AnninaWalker for information. Here is how opendata.swiss deal with keywords: https://opendata.swiss/de/dataset?q=budget


[Bug]: Failed to load plugin

Contact Details
<EMAIL_ADDRESS>What happened?
Hi, I installed Database folder for the first time. However, after installation, the plugin cannot work properly.  Although I may have fixed by myself, I would like to report it in case someone meet the same problem.
Plugin can be found in settings- Community plugins - installed plugins, but the button of DB Folder is grey. When I tried to turn it on, it shows me a callout on the left-top of the Obsidian, saying "failed to load plugin dbfolder"
I tried to install it on another computer (Both of them are Win11) & reinstall Obsidian, and it comes to be the same.
Finally, I found a clue in Devtools, there shows a bug that "main.js" couldn't be found in the plugin folder. After downloading this file manually, the plugin works well.
I downloaded the plugin from the "Community Plugins" in Obsidian, and there was no attention from any firewall software. Could it be because I put my Obsidian vault in Onedrive? Or because there are some Chinese characters in my path? (but other files like "style.css" are downloaded successfully)
Hope you can give me some help. Thank you
What platform were you using?
Desktop
Version of the plugin
3.3.0
Relevant log output
app.js:1 Plugin failure: dbfolder Error: ENOENT: no such file or directory, open 'C:\Users\yodaliu\OneDrive - tongji.edu.cn\小刘搞学术\.obsidian\plugins\dbfolder\main.js'

SYSTEM INFO:
	Obsidian version: v1.1.9
	Installer version: v1.1.9
	Operating system: Windows 10 Pro 10.0.22621 # ?? My laptop is really Win 11
	Login status: not logged in
	Insider build toggle: off
	Live preview: on
	Legacy editor: off
	Base theme: adapt to system
	Community theme: Blue Topaz
	Snippets enabled: 0
	Restricted mode: off
	Plugins installed: 5
	Plugins enabled: 5
		1: proxy github v1.15.0
		2: Activity History v0.1.4
		3: Folder Note v0.7.3
		4: Style Settings v0.4.11
		5: DB Folder v3.3.0

Are you using the latest version of the plugin in your Obsidian vault?

[X] I have verified that I am on the latest version

Are you check if there is a similar issue?

[X] I have verified that there are not similar issues

Sorry, I found it might be a bug of Obsidian. As I downloaded another plugin, and it missed the main.js as well.


Fix url warning on Recovery page

https://discord.com/channels/976052458205892608/979439846030467093/985832528747311145

Hey @followdarko ! Thanks for the PR. This should be fixed in #2732 , you can test in the PR preview by going to https://my-near-wallet-pr-2732.onrender.com/recover-account for example or other PR previews. The code changes are very similar as well: https://github.com/near/near-wallet/pull/2732/files#diff-34c6b74d79716de434a853069d50cbacf40087168bb849145d174107a1d29aaf.
Feel free to reopen if you have any further suggestions! 😄


Fix compilation when complex.h is included

complex.h defines an I macro, which conflicts with libtomcrypt.c.
Thanks for this awesome library, by the way!

The #undef I part works, but redefining it like that won't work properly. Before that #undef sequence, I expands to (__extension__ 1.0iF); afterwards, it's just TLS_I_MACRO, which isn't likely to be an existing token.
Instead, consider doing this. https://gcc.gnu.org/onlinedocs/gcc/Push_002fPop-Macro-Pragmas.html
Alternatively, rename it in libtomcrypt.

Ah, you're right.

Instead, consider doing this. https://gcc.gnu.org/onlinedocs/gcc/Push_002fPop-Macro-Pragmas.html

I think this might cause trouble with some C compilers like TCC (which I'm currently trying to get to work with TLSe).

Alternatively, rename it in libtomcrypt

That worked, but the #undef solution seemed cleaner to me at first sight.
@eduardsui Do you think we should go with #pragma push_macro or editing libtomcrypt.c?

It looks like TCC supports #pragma push_macro too - I guess I will push that approach now (we can still change it before merging though if anyone might spot a problem).

https://github.com/eduardsui/tlse/issues/68#issuecomment-1206583489
Can you check whether an updated ltc would help? I've fixed those duplicate symbols problems already in the branch mentioned in the comment. But I'm not sure whether the version you bundled is still 100% compatible, so maybe it also breaks somewhere else ...

should go with #pragma push_macro or editing libtomcrypt.c?

Hello, and thanks for the fix. I think it's ok, but I would test for the compiler (I'm not sure that #pragma pop/push_macro is standard).
Thanks again,
Eduard


feature request: allow plot request over network

Dear Maintainers,
I would love to setup a visdom plotting server, and have all of my cluster machines access the plotting server over the network using an access token.
Would this be considered for the roadmap?

Hey @episodeyang !
By 'access the plotting server', do you mean plotting remotely over a network? Yes, this kind of support is implicitly built in (we actually use it this way already), though the access token authentication hasn't really been implemented yet (#85). Did you mean something more elaborate?
Allan

@ajabri I just found out about this in the source code:) Closing since this feature is already implemented.


[iOS] Notification play not working

🔙 Regression
On iOS, when we press the play/pause button, the icon changes but the player is still playing
Old (and correct) behavior
When press pause button, the song should pause.
When press play butoon, the song should play where it was paused
Current behavior
The player keeps playing the song
Reproduction steps
Configuration
Version: 0.6.7
Platform:

[x] :iphone: iOS
[ ] :robot: Android
[ ] :checkered_flag: WPF
[ ] :earth_americas: UWP
[ ] :apple: MacOS
[ ] :tv: tvOS
[ ] :monkey: Xamarin.Forms

Can you try with 0.6.8? It seems to work for me. It is out on nuget now.

@martijn00 , I tried with the latest version and it's still not working on iOS. Here a repo for reproduce https://github.com/valentin-debris/XamarinMediaDebug/tree/0.6.8-stop_player

@martijn00 , after debugging the Media Player, I found own to make the notif working.
I have to set CrossMediaManager.Current.NotificationManager.ShowPlayPauseControls = true.
It's confusing because without it, the buttons are visible and clickable, but there are no callbacks assign I guess. And for Android, it works without adding this line.

I set CrossMediaManager.Current.NotificationManager.ShowPlayPauseControls = true when I started playing audio, but it didn't work for me. Waiting for the changes merged in #535 to hopefully help with this issue.

@valentin-debris can you check again with 0.7.4?

Verified that 0.7.4 worked for me without the additional line. 👍

I can also verify, this feature works properly on my side.

Seems OK.

For anyone else having issues on 0.7.3, setting ShowPlayPauseControls to true worked on iOS 12.4
Thanks


feat(sound playback): allow simultaneous playback

Implements the simultaneous playback feature requested in #6
I'm not sure, if there should be some form of rate limit built in to prevent spamming long playing sounds and maybe overload the server. I guess in small friendgroups/servers this shouldn't be too much of a problem. Also, there always is the stop button to abort all sounds.

LGTM! 👍 I've changed the the target from main to development. I'll merge everything into main once I make a new release. You can already use the changes by switching the Docker tag to development.


Face Tracking Using Arduino

Python does the image processing, Arduino controls the servo motors.
An awesome IOT implementation.

Hi, what are we suppose to do in this? Like provide you with a code or what?
Thanks

Yes all implementation

As per my knowledge regarding python and arduino, then there will be one file having python code and other arduino code. I can include software requirements and libraries used. Is there anything that i am missing?

Yes correct
Try to make a good readme showing all procedures+ output

I got your points. How am i suppose to show output?I have no hardwares right now. I will be going to try for this issue so can you assign this to me please? Thank you

Hi, I have seen project mentors giving "dwoc-accepted" label. I have no idea about this but i guess that its necessary for dwoc participants. Kindly check for that. Thank you for giving me a chance

Let me ask from organizer.

@smriti1313 can you please tell me which repo had dwoc-accepted tag?

@smriti1313 can you please tell me which repo had dwoc-accepted tag?

Yeah. check this