refactor(logging): improve LoggingConfig & deprecate LoggingConfig.propagate

Description
I started by refactoring the tests. It might be easier to review this PR commit by commit.
This PR will:

Fix minor bugs like the one described in https://github.com/orgs/litestar-org/discussions/3482

What's a bit strange is that I have to provide a "standard" formatter otherwise I get a KeyError:

Only pass valid values to the dictConfig classes
Deprecate propagate at the root dictConfig level (this value is only supported at the Logger level, it was passed to dictConfig() but has no effect AFAIK)

Closes

Pyright reports issues like this one:
  /home/runner/work/litestar/litestar/tests/unit/test_logging/test_logging_config.py:155:22 - error: Cannot access member "handlers" for type "Logger"

If I add type: ignore[attr-defined], then mypy reports issues like error: Unused "type: ignore" comment  [unused-ignore].
So I end up with type: ignore[attr-defined,unused-ignore] to get both mypy & pyright happy.
If anyone has a better idea, let me know. Thank 🙏

So I end up with type: ignore[attr-defined,unused-ignore] to get both mypy & pyright happy.

Finally updated to pyright: ignore[reportGeneralTypeIssues]

@jderrien Can you update the branch so this can be merged? :)

@provinzkraut I rebased the branch and pushed it, but the auto-merge got disabled in the process. Could you merge it please?
What should I have done to keep the auto-merge working?

What should I have done to keep the auto-merge working? Do it via the GitHub UI?

GitHub tells me that this only works if you have given maintainers write access to the branch in your fork.


Support View usage for PIVOT operator

Description
Support create/select/drop view on stmt with pivot operator
Issues Resolved
Task: BABEL-4673
Test Scenarios Covered

Use case based -

VIEW OF 2 COLUMN SRC TABLE PIVOT
VIEW OF 3 COLUMN SRC TABLE PIVOT
VIEW OF 4 COLUMN SRC TABLE PIVOT
VIEW OF TOP PIVOT
VIEW OF DISTINCT PIVOT
VIEW OF UNION PIVOT
VIEW OF PIVOT IN SUBQUERY
VIEW OF PIVOT FUNCTION
VIEW OF PIVOT FUNCTION
VIEW OF CTE PIVOT AS SRC TABLE
VIEW OF CTE PIVOT AS SRC TABLE JOIN ANOTHER PIVOT
VIEW OF CTE TABLE AS PIVOT SRC TABLE
VIEW OF JOIN PIVOT
VIEW OF INNER JOIN PIVOT
VIEW OF LEFT JOIN PIVOT
VIEW OF RIGHT JOIN PIVOT
VIEW OF FULL JOIN PIVOT
VIEW OF CROSS JOIN PIVOT
VIEW OF COMMA JOIN PIVOT
VIEW OF COMMA CROSS JOIN PIVOT

Minor version upgrade tests -

metadata consistency check before and after upgrade

Major version upgrade tests -

metadata consistency check before and after upgrade

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

182 of 188 (96.81%)  changed or added relevant lines in 4 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.06%) to 73.59%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

contrib/babelfishpg_tsql/runtime/functions.c
39
42
92.86%

contrib/babelfishpg_tsql/src/hooks.c
46
49
93.88%

Files with Coverage Reduction
New Missed Lines
%

contrib/babelfishpg_tsql/src/tsqlUnsupportedFeatureHandler.cpp
1
85.87%

Totals

Change from base Build<PHONE_NUMBER>1:
0.06%

Covered Lines:
43935

Relevant Lines:
59702

💛  - Coveralls

15.8 related upgrade tests will get passed after BABEL3_X_DEV commit merged


Binary Insertion Sort

Submission Title
Binary Insertion Sort
Submission Type
Algorithm
Submission Description
Implementing Insertion Sort but using binary search to find the insertion point to improve time complexity.
Folder Related To
Algorithms => Sorting_Algorithms => Insertion_Sort
Contributors

[ ]  C++ -
[ ]  C -
[x]  Python -
[ ] Java -
[ ] JavaScript -
[ ] TypeScript -
[ ] Golang -
[ ] Ruby -
[ ] php -
[ ] Swift -
[ ] C# -

I would like to work on this issue using Python. Kindly assign it to me.

@ahampriyanshu  Could you please assign the task for Javascript.

Hi @kinshukdua , please make a new folder Binary_Insertion_Sort in the Sorting_Algorithms sub folder

Hi, @ahampriyanshu. Could I be assigned the task for Java?

Hi @kinshukdua , please make a new folder Binary_Insertion_Sort in the Sorting_Algorithms sub folder

Yes sure.

Hi, @ahampriyanshu  can you assign me on this. I'll do this task in C++.

Can you please assign this task to me , I would implement it in C

Hi @ahampriyanshu looks like Java has already been completed. I can take Swift instead. Could you reassign please?

Hi, @ahampriyanshu can you assign me on this. I'll do this task in C++.

@ahampriyanshu  Merge the PR.

Hi @juneadkhan , go ahead with swift

thanks @everyone :v:


Fixed continuous 20% CPU load (tested on iPad 3)

Hi Rob,
Without this fix I have continuously 20% CPU load on iPad 3. With fix 0%. Could you check please.
Thanks

@Uniprog Can you replicate this issue with the example app included in the repo? (e.g. check cpu load on device only, not on the simulator)

I tested on iPad 3 iOS 9.3.5(13G36) real device. And without fix I had 20-22% CPU load. This because animation continuously calling draw rect method. But with fix I had 0% CPU and no  continuously calling draw rect.

@Uniprog Looks like this is failing a test. I have not had time to review w/ the example app on a device, so I'd advise just using your fork for the immediate future as this seems like a significant change.

Closing since I have not heard back


503 - UnexpectedGetChannelsError  err:   message: 14 UNAVAILABLE: Name resolution failed for target dns:xxxxx6xxf6ivybctid.onion:10009

Been trying to troubleshoot this error that seemed to happen randomly, as I use BOS daily. But today it suddenly began giving me this error. I tried to reinstall nodejs but no resolution... Thanks for any help. I am not using docker, it's all on a local machine.
`bacon@baconmachine:~$ bos rebalance
err:

503
UnexpectedGetChannelsError
err:
message: 14 UNAVAILABLE: Name resolution failed for target dns:2r2efomsgg77wxmduzjevnwjo377h2y64faulnurosa6xxf6ivybctid.onion:10009
stack:
- Error: 14 UNAVAILABLE: Name resolution failed for target dns:2r2efoxxxx.onion:10009
-     at Object.callErrorFromStatus (/usr/local/lib/node_modules/balanceofsatoshis/node_modules/lightning/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
-     at Object.onReceiveStatus (/usr/local/lib/node_modules/balanceofsatoshis/node_modules/lightning/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
-     at Object.onReceiveStatus (/usr/local/lib/node_modules/balanceofsatoshis/node_modules/lightning/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:334:141)
-     at Object.onReceiveStatus (/usr/local/lib/node_modules/balanceofsatoshis/node_modules/lightning/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:297:181)
-     at /usr/local/lib/node_modules/balanceofsatoshis/node_modules/lightning/node_modules/@grpc/grpc-js/build/src/call-stream.js:129:78
-     at processTicksAndRejections (internal/process/task_queues.js:75:11)

bacon@baconmachine:~$ bos --version
7.6.0
bacon@baconmachine:~$ node -v
v14.15.0
bacon@baconmachine:~$ npm -v
6.14.8
`

try adding the node via credentials.json

Hey Alex can I get a bit more details implementing that resolution? Sorry.

You can set credentials.json for a saved node to make sure it's using the correct credentials
instructions here:
https://github.com/alexbosworth/balanceofsatoshis#saved-nodes

Thank you, Alex. That worked. Much appreciated as usual!

There are also instructions to set that node as the default if you don't want to specify it


embeddings.index Truncation RuntimeError: The size of tensor a (889) must match the size of tensor b (512) at non-singleton dimension 1

Hello, when I try to run the indexing step, I get this error.
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-6e863ca8aecc> in <module>
----> 1 embeddings.index(to_index)

~\Anaconda3\envs\bert2\lib\site-packages\txtai\embeddings.py in index(self, documents)
     80
     81         # Transform documents to embeddings vectors
---> 82         ids, dimensions, stream = self.model.index(documents)
     83
     84         # Load streamed embeddings back to memory

~\Anaconda3\envs\bert2\lib\site-packages\txtai\vectors.py in index(self, documents)
    245                 if len(batch) == 500:
    246                     # Convert batch to embeddings
--> 247                     uids, dimensions = self.batch(batch, output)
    248                     ids.extend(uids)
    249

~\Anaconda3\envs\bert2\lib\site-packages\txtai\vectors.py in batch(self, documents, output)
    279
    280         # Build embeddings
--> 281         embeddings = self.model.encode(documents, show_progress_bar=False)
    282         for embedding in embeddings:
    283             if not dimensions:

~\Anaconda3\envs\bert2\lib\site-packages\sentence_transformers\SentenceTransformer.py in encode(self, sentences, batch_size, show_progress_bar, output_value, convert_to_numpy, convert_to_tensor, device, normalize_embeddings)
    192
    193             with torch.no_grad():
--> 194                 out_features = self.forward(features)
    195
    196                 if output_value == 'token_embeddings':

~\Anaconda3\envs\bert2\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    117     def forward(self, input):
    118         for module in self:
--> 119             input = module(input)
    120         return input
    121

~\Anaconda3\envs\bert2\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Anaconda3\envs\bert2\lib\site-packages\sentence_transformers\models\Transformer.py in forward(self, features)
     36             trans_features['token_type_ids'] = features['token_type_ids']
     37
---> 38         output_states = self.auto_model(**trans_features, return_dict=False)
     39         output_tokens = output_states[0]
     40

~\Anaconda3\envs\bert2\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Anaconda3\envs\bert2\lib\site-packages\transformers\models\bert\modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    962         head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
    963
--> 964         embedding_output = self.embeddings(
    965             input_ids=input_ids,
    966             position_ids=position_ids,

~\Anaconda3\envs\bert2\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Anaconda3\envs\bert2\lib\site-packages\transformers\models\bert\modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)
    205         if self.position_embedding_type == "absolute":
    206             position_embeddings = self.position_embeddings(position_ids)
--> 207             embeddings += position_embeddings
    208         embeddings = self.LayerNorm(embeddings)
    209         embeddings = self.dropout(embeddings)

RuntimeError: The size of tensor a (889) must match the size of tensor b (512) at non-singleton dimension 1

Where to_index =
 [('0015023cc06b5362d332b3baf348d11567ca2fbb',
  'The RNA pseudoknots in foot-and-mouth disease virus are dispensable for genome replication but essential for the production of infectious virus. 2 3\nword count: 194 22 Text word count: 5168 23 24 25 author/funder. All rights reserved. No reuse allowed without permission. Abstract 27 The positive stranded RNA genomes of picornaviruses comprise a single large open reading 28 frame flanked by 5′ and 3′ untranslated regions (UTRs). Foot-and-mouth disease virus (FMDV) 29 has an unusually large 5′ UTR (1.3 kb) containing five structural domains. These include the 30 internal ribosome entry site (IRES), which facilitates initiation of translation, and the cis-acting 31 replication element (cre). Less well characterised structures are a 5′ terminal 360 nucleotide 32 stem-loop, a variable length poly-C-tract of approximately 100-200 nucleotides and a series of 33 two to four tandemly repeated pseudoknots (PKs). We investigated the structures of the PKs 34 by selective 2′ hydroxyl acetylation analysed by primer extension (SHAPE) analysis and 35 determined their contribution to genome replication by mutation and deletion experiments. 36 SHAPE and mutation experiments confirmed the importance of the previously predicted PK 37 structures for their function. Deletion experiments showed that although PKs are not essential 38',
  None),
 ('00340eea543336d54adda18236424de6a5e91c9d',
  'Analysis Title: Regaining perspective on SARS-CoV-2 molecular tracing and its implications\nDuring the past three months, a new coronavirus (SARS-CoV-2) epidemic has been growing exponentially, affecting over 100 thousand people worldwide, and causing enormous distress to economies and societies of affected countries. A plethora of analyses based on viral sequences has already been published, in scientific journals as well as through non-peer reviewed channels, to investigate SARS-CoV-2 genetic heterogeneity and spatiotemporal dissemination. We examined all full genome sequences currently available to assess the presence of sufficient information for reliable phylogenetic and phylogeographic studies. Our analysis clearly shows severe limitations in the present data, in light of which any finding should be considered, at the very best, preliminary and hypothesis-generating. Hence the need for avoiding stigmatization based on partial information, and for continuing concerted efforts to increase number and quality of the sequences required for robust tracing of the epidemic.',
  None),
 ('004f0f8bb66cf446678dc13cf2701feec4f36d76',
  'Healthcare-resource-adjusted vulnerabilities towards the 2019-nCoV epidemic across China\n',
  None), ...]

How do I fix this? I don't see anywhere in the documentation about this. I assume the error message:
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length.
Default to no truncation.

is related, and I need to set a max_length to 512 so that any documents that are larger than 512 get truncated to 512 tokens, but I don't see anywhere how to do that...

Thank you for reporting this issue!
Can you share the transformer model you are using for this test?

That is using
embeddings = Embeddings({"method": "transformers", "path": "sentence-transformers/bert-base-nli-mean-tokens", "scoring": "bm25"})

Thank you. It looks like sentence-transformers 1.x no longer defaults the tokenizer.max_seq_length, which is fine for models that have a model_max_length set. In the case of the model here, it does not, hence the error.
I will add an additional config option to optionally set the tokenizer.max_seq_length to address this issue.
In the meantime you have a couple different options.

Use a different model that has a model_max_length, for example "sentence-transformers/stsb-distilbert-base"
Downgrade to sentence-transformers==0.4.1
Wait for the next version of txtai to use this model and set maxlength=512 manually

After the fix it will look something like this:
embeddings = Embeddings({"method": "transformers", "path": "sentence-transformers/bert-base-nli-mean-tokens", "maxlength": 512})

Thanks you, you're absolutely right! Downgrading sentence-transformers to 0.4.1 did the trick for now!

Just committed a fix for this. You can now also try installing the latest master branch pip install git+https://github.com/neuml/txtai
Otherwise, it will be in txtai 3.0

Just committed a fix for this. You can now also try installing the latest master branch pip install git+https://github.com/neuml/txtai
Otherwise, it will be in txtai 3.0

Actually pip install git+https://github.com/neuml/txtai does not work for me, whether or not I used pip install sentence-transformers==0.4.1 or used the most recent version of sentence-transformers. The only combination that has worked for me was the pypi version of txtai with sentence-transformers version 0.4.1

Did you run it like this?
embeddings = Embeddings({"method": "transformers", "path": "sentence-transformers/bert-base-nli-mean-tokens", "maxlength": 512})

The sentence-tranformers change is actually an improvement as defaulting to 128 tokens isn't the best strategy. But it does require an extra parameter if the model doesn't have the model_max_length parameter set.

You're right, it does work if I include it like that.
However, it doesn't work for the Similarity pipeline
if I use Similarity("gsarti/scibert-nli") or Similarity("bionlp/bluebert_pubmed_uncased_L-24_H-1024_A-16")
then I get the same error

Thank you for confirming this works for Embeddings.
This Similarity issue is similar but unrelated to this issue. Each pipeline needs to add a kwargs of {truncation:true} when otherwise not specified. I've added #79 to address this.

Any pointers on how to do this? I've noticed that strangely Similarity("gsarti/covidbert-nli") works fine in my code but not Similarity("gsarti/scibert-nli") despite both being BERT models with "max_position_embeddings": 512, in their config.json

I had tried to copy what you did in the Summary pipeline to labels.py in my fork:
kwargs = {"truncation": True, "max_length": 512, "max_seq_length": 512}
# Run ZSL pipeline
results = self.pipeline(text, labels, multi_label=multilabel, **kwargs)

to seemingly no change in result

It's really about the Tokenizer.
>>> tokenizer = AutoTokenizer.from_pretrained("gsarti/scibert-nli")
>>> tokenizer.model_max_length
1000000000000000019884624838656

Something that may work for your local testing:
kwargs = {"truncation": True,}
# Run ZSL pipeline
self.pipeline.tokenizer.model_max_length = 512
results = self.pipeline(text, labels, multi_label=multilabel, **kwargs)

I think transformers is assuming this field is always set with the tokenizer. Currently, max_length can't be overridden in calls to the tokenizer for hf pipelines. May be a worthwhile change upstream with transformers to allow that field to be set.
https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/base.py#L638

Going to reopen this issue. I think I can make this less manual. Plan to remove the maxlength parameter I just added and attempt to copy the max_position_embeddings config parameter if the tokenizer doesn't have a max_model_length set.

Just committed a fix that should address both embeddings and pipelines. The maxlength parameter is no longer needed, it will take the max_position_embeddings config parameter when it's not detected in the tokenizer.

I can confirm that as of the latest commit, my code using a pretrained Bluebert model works without running into the issue! Thanks for the fix!

Great, glad to hear it, appreciate your help on this!


RHTAP: Add inputRefs to CloudWatch log forwarder

Add inputRefs to CloudWatch log forwarder as they were prematurely, as we still don't have those audit logs in Splunk.
The intension here is to have also infrastructure and audit logs in CloudWatch for the time being.

LGTM


karpenter printing relaxing soft constraints every minute

Version

Karpenter Version: v0.27.1

Kubernetes Version: v1.25
Expected Behavior
I would expect karpenter to warn about relaxing soft constraints less frequently
Actual Behavior
karpenter is logging DEBUG	controller.deprovisioning	relaxing soft constraints for pod since it previously failed to schedule, removing: spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0]={"weight":1,"podAffinityTerm":{"topologyKey":"[topology.kubernetes.io/zone](http://topology.kubernetes.io/zone)"}} about every 60 seconds.
Steps to Reproduce the Problem
enable node consolidation for a provisioner and deploy an application that has anti affinity set
Resource Specs and Logs
provisioner
resource "kubernetes_manifest" "provisioner_default" {   computed_fields = ["spec.requirements"]   manifest = {     "apiVersion" = "karpenter.sh/v1alpha5"     "kind"       = "Provisioner"     "metadata" = {       "name" = "default"     }     "spec" = {       "providerRef" = {         "name" = "default"       }       "kubeletConfiguration" = {         "clusterDNS" = [           "<IP_ADDRESS>" ## NODE LOCAL         ]       }       "requirements" = [         {           "key"      = "karpenter.sh/capacity-type"           "operator" = "In"           "values" = [             "spot",           ]         },         {           "key"      = "node.kubernetes.io/instance-type"           "operator" = "In"           "values"   = var.karpenter_default_allowed_spot_instance_types         },         {           "key"      = "topology.kubernetes.io/zone"           "operator" = "In"           "values"   = ["${local.region}a", "${local.region}b", "${local.region}c"]         },       ]       "consolidation" = {         "enabled" = "true"       }     }   }
pod anti affinity spec for an application on the cluster:
spec {         affinity {           pod_anti_affinity {             preferred_during_scheduling_ignored_during_execution {               weight = 1               pod_affinity_term {                 topology_key = "topology.kubernetes.io/zone"               }             }           }         }
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I'm seeing the same thing in my environments.  Karpenter has worked pretty well for us as a default install.  Now we're trying to get fancier and split compute workloads with with a nodeSelector/label combination.  Each node selector has it's own nodepool and even though the labeling is correct I get the "relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints" notice in debug logs.
I'm not seeing anything related to this in the karpenter troubleshooting guide.  Are there any other docs that can guide me to a better/working spec.topologySpreadConstraint ?


Social share prev

@lukestokes can you add a nice description in the description meta tags?

Yeah, I'll take a look. Thanks!


我是小白 想问一下如何编译

我使用idea cloin编译找不到导入的东西
compilation terminated.
CMakeFiles/drizzleDumper_master.dir/build.make:62: recipe for target 'CMakeFiles/drizzleDumper_master.dir/drizzleDumper.c.o' failed
make[3]: *** [CMakeFiles/drizzleDumper_master.dir/drizzleDumper.c.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/drizzleDumper_master.dir/all' failed
make[2]: *** [CMakeFiles/drizzleDumper_master.dir/all] Error 2
CMakeFiles/Makefile2:79: recipe for target 'CMakeFiles/drizzleDumper_master.dir/rule' failed
make[1]: *** [CMakeFiles/drizzleDumper_master.dir/rule] Error 2
Makefile:118: recipe for target 'drizzleDumper_master' failed
make: *** [drizzleDumper_master] Error 2

[ 50%] Building C object CMakeFiles/drizzleDumper_master.dir/drizzleDumper.c.o
In file included from /cygdrive/f/src/c_compiler_path/drizzleDumper-master1/drizzleDumper-master/drizzleDumper.c:6:0:
/cygdrive/f/src/c_compiler_path/drizzleDumper-master1/drizzleDumper-master/drizzleDumper.h:14:24: fatal error: sys/ptrace.h: No such file or directory
#include <sys/ptrace.h>

NDK直接编译


Storage bandwidth is wasted during template uploads/imports

At the moment when a template is being imported (via url or upload) there is a phase called "Installing template" or something similar. From what I noticed this phase calculates a hash and saves it to the database by reading the downloaded file over the storage network.
During this phase the template is not usable and I believe that should not be the case. I understand that the hash has a purpose and I am not saying it should be removed but I believe this should be an optional task that should be done in background.
In the following examples I do not consider disk speeds, just network speed/bandwidth.
For normal templates this is not necessarily noticeable. Consider this scenario (best case):
Template size: 4GB
Ingress bandwidth: 100 mb/s
Storage bandwidth: 1 gb/s
Download time required: 5.45 seconds
Installing template time required: ~0.6 seconds
Most templates (not ISOs) however are considerably larger, some could be even up to 500 GB (I do have a few templates that I have to import with very large sizes).
In such a case, installing template time required would be about 75 seconds (best case).
During this time (installing template time) the whole bandwidth available for the storage network (if it even is on a separate NIC) would be used up by this process resulting in bad performance for the cluster.
Ways to fix this would be:

either compute the hash as the transfer is happening.
make it optional (maybe even opt-in) and do it in background only (maybe even limit the bandwidth used for this)

Thanks @alexandru-bagu - the installing template phase is also extracting the downloaded template (for compressed templates) once it has finished downloading it which takes time as well. The data exchanged during the time of installation between the management server and SSVM are commands and answers to simply check if the installation is finished. Unfortunately I don't think this task can be set as optional - what do you think @rohityadavcloud @DaanHoogland?

The optimisation you mention could be done, if you wish But I would definately opt for opt-out (maybe to be reversersed by a global setting.
It sounds strange to me that after downloading the network would still be fully occupied by the status query thaat checks the status of the install. That should be minimal.

The optimisation you mention could be done, if you wish But I would definately opt for opt-out (maybe to be reversersed by a global setting. It sounds strange to me that after downloading the network would still be fully occupied by the status query thaat checks the status of the install. That should be minimal.

I doubt it's the status check that uses the network. My thinking was that if I use shared storage for templates which is required afaik then whenever the SSVM does an operation network bandwidth has to be used. Meaning

when downloading the template traffic should look like this: [public net] -download-> [ssvm] -save-> [storage] (ssvm downloads from public net and writes to storage)
when download is complete and template is being extracted traffic is probably this: [storage] -read-> [ssvm] -process-> [storage] (ssvm reads from storage and writes to storage)
when hash is being computed traffic is probably this: [storage] -read-> [ssvm] and then when it finishes reading and computing the hash push it to cloudstack

If hash is computed separate from the download then additional bandwidth is going to be used to read the file and compute it.
Additionally I hope that in case where the template is not compressed the file is just renamed and not copied into the proper name and the other one removed.
*For more context, we are currently looking for a way to import a template that has about 4 TB. Importing such a template alone would take a long time even with a 10GB connection. To have the system waste more bandwidth to compute a hash that is not even going to be validated any other time is not useful.

*For more context, we are currently looking for a way to import a template that has about 4 TB. Importing such a template alone would take a long time even with a 10GB connection. To have the system waste more bandwidth to compute a hash that is not even going to be validated any other time is not useful.

makes sense. The checksum will be used on initial download if you supply an excepted checksum. The chacksum of the extracted file will later be used on store to store (e.g. could be cross zone) copying of the template.

The registerTemplate API has a checksum arg for this - could this facilitate this? However, when this is provided we would want the downloaded template file from url (or uploaded) to pass this checksum, either way we may not be able to skip the checksum calculation on the template


Fix bat file

Add reactNativePath to cli.js

Summary
Add reactNativePath to cli.js
Changelog
[GENERAL] [FIXED] - Internal - Update packagers to the latest CLI
Test Plan

Thanks for the fix!


Translate actual messages from Python

Currently, the answer to what() contains two parts:
SomeError: message from Python

Explanation of what SomeError means

The first part is not considered for translation; the second part is. It might be useful to consider making the original message translatable. This would likely affect how explain() works in order to have the translation of the message from Python appear, but ensuring that it does not appear twice when the message cannot be translated (either because English is selected or because a translation has not been created).

Hi! I just attended your lightning talk on this. Where do we modify the actual text? I have 2 more languages I can contribute for explanations. Thank you!

i will do the spanish part

Hi @sudosoph The translations are done using standard "gettext" tools.  You can find an explanation in the documentation: https://aroberge.github.io/friendly-traceback-docs/docs/html/translation_notes.html
Note that there are MANY strings to translate, and that this could be overwhelming to do by a single person - especially since I keep adding new content. However, it can be done little by little: the way that it works is that, if it cannot find a translation, it uses the English default (no exception raised).
Feel free to contact me at any time with questions or comments, either through github issues, or via email.

@ralle123: see what I just wrote as a comment above.

Moved to new project: https://github.com/friendly-traceback/friendly-traceback/issues/49


add SM2 pub key.

SM2 is a set of cryptographic algorithms based on elliptic curve cryptography, including a digital signature, public key encryption and key exchange scheme.

After actual testing, we can use the same algorithm as
https://stackoverflow.com/questions/17171542/algorithm-for-elliptic-curve-point-compression
to compress the SM2 public key and recover it successfully.
In our case, the compressed sm2 public key zEPJcWxWVMvrRvSLt4WsPekiqt89YVFRYxSkHbNRjaPfZWTPs will be uncompressed as follows:
{
            "kty": "EC",
            "crv": "SM2",
            "x": "4_1l5yuoHlvhmCsYzfYCLJXFU8-uv5464mWmC-tdyx4",
            "y": "mJoxCEVBRMrgL4B544mhN85kpjXkwXY_HEb_SltPWh0"
}

The algorithm code used for recovery is as follows:
const bigInt = require("big-integer");

// Consts for SM2 curve.
const two = new bigInt(2),
    prime = new bigInt('FFFFFFFEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000000FFFFFFFFFFFFFFFF', 16),
    b = new bigInt('28E9FA9E9D9F5E344D5A9E4BCF6509A7F39789F515AB8F92DDBCBD414D940E93', 16),
    // Pre-computed value, or literal
    pIdent = prime.add(1).divide(4);

function pad_with_zeroes(number, length) {
    var retval = '' + number;
    while (retval.length < length) {
        retval = '0' + retval;
    }
    return retval;
}

/**
 * Point decompress SM2 curve
 * @param {string} Compressed representation in hex string
 * @return {string} Uncompressed representation in hex string
 */
function ECPointDecompress(comp) {
    var signY = new Number(comp[1]) - 2;
    var x = new bigInt(comp.substring(2), 16);
    // y^2 = x^3 - 3x + b
    var y = x.pow(3).subtract(x.multiply(3)).add(b).modPow(pIdent, prime);
    // If the parity doesn't match it's the *other* root
    if (y.mod(2).toJSNumber() !== signY) {
        // y = prime - y
        y = prime.subtract(y);
    }
    return '04' + pad_with_zeroes(x.toString(16), 64) + pad_with_zeroes(y.toString(16), 64);
}

Examples:
ECPointDecompress('03e3fd65e72ba81e5be1982b18cdf6022c95c553cfaebf9e3ae265a60beb5dcb1e')

returns:
04e3fd65e72ba81e5be1982b18cdf6022c95c553cfaebf9e3ae265a60beb5dcb1e989a3108454144cae02f8079e389a137ce64a635e4c1763f1c46ff4a5b4f5a1d

seems reasonable to me, I'll leave this open till tomorrow in case others have input but will then merge it for you; this thread should serve as background for anyone that comes along wanting to know more about the entry

This request was not merged. :)

that's a bit weird, doesn't say who closed it, was that you @xicilion? I'm happy to merge this if it's still wanted? sorry for being slow on getting to it.

Didn't you close it? I'm also confused about this. This request is still expected to be merged, thank you very much.

Nope, I didn't close it, it doesn't say who or why, it's just closed! I'll sort it out though.

thank you very much.


Feature: SoapySDR support

Support for SoapySDR lib would bring support for many SDR devices instead of adding specific support for each one.
It would make usable SDRPlay, Limesdr, hackrf and other.

SoapySDR support added in 2114d31b374313e0c3a1803a4429567f4c3ce500 and previous commits. Implementation was only tested with RTL-SDR and Airspy Mini, volunteers for testing with other supported HW are welcome. Please report bugs and problems.
SoapySDR is disabled by default, enable it by (requires SoapySDR installation):
cmake .. -DSOAPYSDR=ON

Thank you for adding this feature. I will make a try.

Tried with SDRPlay RSP1, works fine.
Thank you for the great work.

I am happy it works, thank you for your feedback!


v0.32.0: huge AVIF output images cannot be opened in older software

images cannot be open

i'm using default config
files:
https://drive.google.com/file/d/1sLXwxH0cdjnkwQ7higs_qHTUc3IFxgQI/view?usp=sharing

The prebuilt binaries provided by sharp v0.32.x include libaom v3.6.0. It looks like it has a bug relating to encoding very large AVIF images where the output is valid but uses new compression features ("level 7 and 8") that are not yet widely supported in older decoders, including Chrome.
https://aomedia.googlesource.com/aom/+/7f006ee7f3da90c86d1b05696a5feac4f4162ac9^!/
<EMAIL_ADDRESS>This has been fixed in libaom v3.6.1, which a future version will include in prebuilt binaries.

The prebuilt binaries provided by sharp v0.32.4 include libaom v3.6.1 with the fix for this.


update to version 0.10.10

Codecov Report

Merging #1773 (153c2fa) into main (758a4ec) will decrease coverage by 5.01%.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main    #1773      +/-   ##
==========================================
- Coverage   54.02%   49.02%   -5.01%
==========================================
  Files          47       47
  Lines        7135     7135
==========================================
- Hits         3855     3498     -357
- Misses       3280     3637     +357

see 14 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Added Alien Shooting Game Closes #3149

Alien Shooting and Issue number : #3149

Close #3149

Checklist:

[X] I have mentioned the issue number in my Pull Request.
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have created a helpful and easy to understand README.md

@MohitGupta121 please review and merge it and also add appropriate labels.


Add CI and changelog for the project

[ ] Add CI.
[x] Add CHANGELOG.md

Blocked by #19.

See https://github.com/dic-iit/kindyn-vio/pull/20.


Pre-commit hook fails with unrecognized arguments: --root

Here is my pre-commit config file
default_language_version:
  python: python3.12
repos:
  # Check dependencies between apps
  - repo: https://github.com/Never-Over/tach
    rev: "v0.2.0"
    hooks:
      - id: tach

When I try to commit, I get:
tach.....................................................................Failed
- hook id: tach
- exit code: 2
usage: tach [-h] {pkg,check,install,sync,clean} ...
tach: error: unrecognized arguments: --root

Seems like it was added in https://github.com/gauge-sh/tach/pull/62
I don't see any mention of the --root option in the help of tach check:
 ➜ tach check --help
usage: tach check [-h] [--strict] [-e file_or_path,...]

Check existing boundaries against your dependencies and package interfaces

options:
  -h, --help            show this help message and exit
  --strict              Raise errors if any dependency constraints are unused.
  -e file_or_path,..., --exclude file_or_path,...
                        Comma separated path list to exclude. tests/, ci/, etc.

What am I missing?

Thanks for raising this issue! This feature was broken when using the pre-commit framework, sorry for the inconvenience. We just published a new version 0.2.1 which addresses this! You can check out #66 for more details.
If you adjust the rev in your posted config to v0.2.1, your config should work as-is.

Amazing, that worked! Thanks 🎉


Error 500 Network Associations

Describe the bug
Using the API to associate a Network to a Block - error Internal Server Error
To Reproduce
Associate a virtual network to a block via the API

Thanks for bringing this to my attention @positivism1 . I've found the issue and will have a fix pushed out for it in the morning along with instructions on how to update the images in your Private ACR.

Ok @positivism1, I believe this should now be resolved with #161. Please pull the latest code from main and then push a new version of the Engine container to your Private ACR using the below process:
https://azure.github.io/ipam/#/contributing/README?id=building-production-containers-images-using-a-private-acr
Please let me know if you have any issues, hopefully this resolves the issue for you!

Thanks! It's working now. 👏

Great to hear, thanks for the great teamwork!


Better random colors

Updated newIrowake() to implement a process that better avoids drawing very similar colors. It's fairly heuristic at the moment and could potentially benefit from additional tweaking, but works well enough at the moment. The general idea is to draw colors from equally-spaced points along a circle in the color plane of CIE LAB space and with monotonically varying brightness so that the colors are all far apart, then to shuffle them to decrease the chance that nearby colors are similar.
Creating a better system for randomly assigning colors as new elements are created is probably done as well as it can be right now without re-assigning colors to all lines every time a new line is created, since there's no way of knowing how many lines there will be.
I'm not sure what's going on with the additions to files other than GraphBase.js, these seem like problems with sitting on the branch for a couple of weeks while I figured this out

I manually fixed the files that were erroneously different, and don't really understand the codecov errors that are occurring that seem to be a result of this

Replaced by https://github.com/robx/pzprjs/pull/278


Add Assignment for Provider convertibles

Expected Behavior
Currently if you try to use the Kotlin DSL assignment operator with a provider convertible, it fails because it only supports providers
Current Behavior (optional)
No response
Context
this can be caused by the following configuration
[versions]
version = "1"
version-b = "2"

interface Extension {
  val foo: Property<String>
}

val theExtension by extensions.creating(Extension::class)

theExtension.foo = libs.version.b // works as "b" is a Provider<String>
theExtension.foo = libs.version // Fails as "version" is a ProviderConvertible<String>

The issue is in the backlog of the relevant team, but the existence of a workaround means it might take a while before a fix is made.

The (ugly) workaround is to use .asProvider() to convert from ProviderConvertible<String> to Provider.
Probably the ProviderConvertible should delegate to Provider more transparently.

When we were implementing Property.assign functionality we decided we will support only assignments for types that can be set to Property via Property.set() method. We decided for that since we want to keep Kotlin DSL very explicit and we want to avoid falling in a trap of implicit conversions that can sometimes feel like magic.
From that I don't think we want to support this case. Also ProviderConvertible is very niche thing used for dependencies. I would avoid adding that. Using .asProvider() is the way to go.

Whilst I do understand your point of avoiding implicit conversions, this causes weird behavior when accessing version catalogs, since ProviderConvertible is only used, if the name is a prefix and an actual reference, which might be confusing for people unfamiliar with this behavior


Disable deprecation checks in src distribution test

Deprecations and warnings are in our face already and this causes
flakiness when new deprecations are introduced.

@bot-gradle test QF

OK, I've already triggered QuickFeedback build for you.

@bot-gradle test and merge

OK, I've already triggered a build for you.


Replace doesn't work in order

When I replace the sprites they are placed from highest to lowest.
I am using version 0.5.4, I read that they had fixed that, but I don't know if that's what they mean.

A demo video:
https://user-images.githubusercontent.com/91706615/233479652-1a44c5b9-452a-4ee3-855b-940d20fb8461.mp4

One solution would be to rename the files in my case when I export the spr, the files come with the name (item_1245-860v2-v2), I realized that apparently the OBJ has a problem with the Hyphen (-).
I saw a way to make this easier.
The following code that they have to execute in Powershell.exe:
ls | %{Rename-Item $_ -NewName ("Item_{0}.obd" -f $nr++)}
https://user-images.githubusercontent.com/91706615/233517180-c939c12f-77d1-4b83-904c-bc5597e56dee.mp4
Now the files will be renamed in Item_0, when you replace they will now be placed in order.


No Gradio/local tunnel link being created.

I have tried pretty much everything but no matter what i try, even with everything running no gradio link is created id like no know what should i do in this case.

Copy the error/output here

Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 12, in
from modules import import_hook, errors
ImportError: cannot import name 'import_hook' from 'modules' (unknown location)

Use the latest colab, I added an update to fix that

now i got this: Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 12, in
from modules import devices, sd_samplers, upscaler
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers.py", line 10, in
from modules import prompt_parser, devices, processing, images
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 14, in
import modules.sd_hijack
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_hijack.py", line 10, in
import modules.textual_inversion.textual_inversion
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 13, in
from modules import shared, devices, sd_hijack, processing, sd_models, images
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/shared.py", line 14, in
import modules.sd_models
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_models.py", line 17, in
from modules import shared, modelloader, devices, script_callbacks, sd_vae, sd_disable_initialization, errors, hashes
ImportError: cannot import name 'sd_vae' from 'modules' (unknown location)

stop then rerun the colab by skipping the first cell and try to reproduce the error

now it works

now i got this: Traceback (most recent call last): File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 12, in from modules import devices, sd_samplers, upscaler File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers.py", line 10, in from modules import prompt_parser, devices, processing, images File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 14, in import modules.sd_hijack File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_hijack.py", line 10, in import modules.textual_inversion.textual_inversion File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 13, in from modules import shared, devices, sd_hijack, processing, sd_models, images File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/shared.py", line 14, in import modules.sd_models File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_models.py", line 17, in from modules import shared, modelloader, devices, script_callbacks, sd_vae, sd_disable_initialization, errors, hashes ImportError: cannot import name 'sd_vae' from 'modules' (unknown location)

from google.colab import drive
drive.mount('/content/gdrive')

first cell, change '/content/gdrive' to 'content/drive'

@jirat-sumitanun  why ?


Restrict minimum typical stay of Events to 15m (EXPOSUREAPP-6764)

While creating a new event minimum stay should be at least 15m
close #2950
Ticket

The ticket says : 'When the owner of the "Event" specified no "Typical Stay" value (which is currently the default), then the guest CWA shall use the "End" time of the event for automatic check out.' You have changed the default duration to 15 min, which seems not to be what the ticket asked for, or was there some further discussion?

Please follow the discussion on the Github issue linked above specially https://github.com/corona-warn-app/cwa-app-android/issues/2950#issuecomment-826147376
on iOS this seem to be the behaviour


config.yaml not generated after first use

#Ref readme.md

The configuration of the script is done in ~/.config/sw-cli-tools/config.yaml. If the file does not exist on your system, it is created after the first usage of the script.

I tried to generate test data with sw:generate. This command doesn't work without config.yaml setting for Datagenerator/locale
After that I created a plugin with sw:plugin:create and no config.yaml was generated

which OS do you use? where do you execute the cli-tools?

ddev describe
NAME       TYPE  LOCATION            URL                          STATUS
shopware5  php   ~/htdocs/shopware5  https://shopware5.ddev.site  running

Project Information
-------------------
PHP version:   	7.2
MariaDB version
MySQL version

URLs
----
http://shopware5.ddev.site
http://<IP_ADDRESS>:32795
https://shopware5.ddev.site
https://<IP_ADDRESS>:32794

MySQL/MariaDB Credentials
-------------------------
Username: "db", Password: "db", Default database: "db"

or use root credentials when needed: Username: "root", Password: "root"

Database hostname and port INSIDE container: db:3306
To connect to db server inside container or in project settings files:
mysql --host=db --user=db --password=db --database=db
Database hostname and port from HOST: <IP_ADDRESS>:32782
To connect to mysql from your host machine,
mysql --host=<IP_ADDRESS> --port=32782 --user=db --password=db --database=db

I installed (and self-updated) sw.phar to ~/bin/sw (chmod +x ) via wget http://shopwarelabs.github.io/sw-cli-tools/sw.phar && mv sw.phar ~/bin/sw

if you create the file by yourself, it be read by the tool?

Yes. I copy & paste config.yaml.dist to ~/.config/sw-cli-tools/config.yaml changed Dategenerator settings and this worked for me

ok, nice to here that :+1:
I will leave this issue open, so that one could have a look, why it is not auto generated in the first place

From my point of view, there's no code, which should save a config file  - maybe i overlooked this part


[Extensions] Pass through the allowExtensionTargets flag from the Workspace to PackageGraph

Motivation:
This lets clients of libSwiftPM enable the experimental support for package extensions.

@swift-ci please smoke test


Display when no Things

When a person has not yet created any Thing objects, go ahead and have some sort of default display so it is not just a blank gray screen.

Added a comment that prompts the user to create a new Thing via the add button. Hidden when atleast one thing is in existence.


Add link to example webhook receiver in documentation

Adds a link to an example of a webhook receiver (https://github.com/mrexodia/go-gitea-webhook)

Codecov Report

Merging #3517 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3517      +/-   ##
==========================================
- Coverage   35.69%   35.69%   -0.01%
==========================================
  Files         281      281
  Lines       40701    40701
==========================================
- Hits        14529    14527       -2
- Misses      24026    24027       +1
- Partials     2146     2147       +1

Impacted Files
Coverage Δ

models/repo_list.go
65.62% <0%> (-1.57%)
:arrow_down:

models/repo_indexer.go
48.3% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b104ab3...a1f11ac. Read the comment docs.

I think an unknown external link could not be accepted.

Yeah that makes sense, however I think it’s important to show examples in the documentation. Perhaps I could transfer the repository to the gitea organization after a code review?

@mrexodia yes, you can do that. Please send a PR and put the example in a suitable sub folder.

@lunny what would be a suitable location? I think it would be easier to transfer the repository to the go-gitea organisation. Should I do that?

@mrexodia I think we ever have a repository named examples.

Closing per @lunny's comment.

My transfer request for the repository was never accepted, so I'm confused why this has been closed. I will create a new pull request though, which simply includes the code in the documentation so there is no problem with external links...


cell optimization handler

Initial attempt

I will close the PR as according to our current understanding, restarting the simulation wouldn't change anything. See here for more details.

Sure. That's make sense, but please don't forget that there also was fix
for structure calculations (previously during restart plugin used old
structure from database to restart while it had to use the last obtained.
On Tue, 21 Sep 2021, 13:47 Aliaksandr Yakutovich, @.***>
wrote:

I will close the PR as according to our current understanding, restarting
the simulation wouldn't change anything. See here
https://groups.google.com/g/cp2k/c/_1K-srPeWE4/m/uePL3MZ2AQAJ for more
details.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aiidateam/aiida-cp2k/pull/143#issuecomment-923903783,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARGJTSCGVNCB3STWWLXYZJLUDBWEZANCNFSM5AJKMIQA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Sure. That's make sense, but please don't forget that there also was fix for structure calculations (previously during restart plugin used old structure from database to restart while it had to use the last obtained.

Right, thanks for reminding me. Can you make another PR with that change only?

Dear Aliaksandr,
I'm sorry, I forgot to send you pull request. I will do this today, when I
return home or max on Wednesday.
Sorry for delay.
Best wishes,
Vasyl
On Fri, 24 Sep 2021, 00:10 Aliaksandr Yakutovich, @.***>
wrote:

Sure. That's make sense, but please don't forget that there also was fix
for structure calculations (previously during restart plugin used old
structure from database to restart while it had to use the last obtained.
Right, thanks for reminding me. Can you make another PR with that change
only?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aiidateam/aiida-cp2k/pull/143#issuecomment-926198027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARGJTSAIRBNCAKZKB26U6N3UDOQUZANCNFSM5AJKMIQA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Clarify parts of the "Configuration" documentation.

Closes #34

@henryiii would you be able to take a look at this?
I think I addressed all of your comments.

I need to print on my comments  to trade add a trading bot on my api
On Thu, 08 Jul 2021, 14:22 Henry Schreiner, @.***>
wrote:

@.**** approved this pull request.
I think this looks like a solid improvement, thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/repo-helper/whey/pull/38#pullrequestreview-701992796,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUEVABCKAPEEALXVALUXCDDTWWJ73ANCNFSM47PQWHVQ
.

Thanks, I'll merge this then.
Do let me know if you have any more suggestions.


Source column description from DBT is missing in Datahub

Describe the bug
Source column description from DBT is not populated to Datahub
To Reproduce

Create sample DBT model (I used this one: https://github.com/dbt-labs/trino-dbt-tpch-demo)
Set up description field in sources.yml
Run dbt
Ingest dbt json files into datahub

Expected behavior
I want to see column descriptions from DBT in Datahub
Screenshots
Datahub

DBT Docs

Hi @maver1ck ! We are investigating this at present. If you click on the "dbt" source in the right side panel, do you see the descriptions (based on what you've posted I'm assuming not)
Cheers
John

We should be extracting the "comments" from DBT sources. I can confirm we are seeing for sources.
Can you please confirm that when navigating to the DBT dedicated source page (right side) you cannot see it? If yes, then I think we've found a real bug in the rendering layer.
Cheers
John

I'm sending screenshot from dbt source. There is no columns at all.
Sources are from this file:
https://github.com/dbt-labs/trino-dbt-tpch-demo/blob/main/models/staging/tpch/tpch_sources.yml
This is ingestion config
source:
  type: "dbt"
  config:
    # Coordinates
    # To use this as-is, set the environment variable DBT_PROJECT_ROOT to the root folder of your dbt project
    manifest_path: "/Users/maverick/src/ezdihar/trino-dbt-tpch-demo/target/manifest.json"
    catalog_path: "/Users/maverick/src/ezdihar/trino-dbt-tpch-demo/target/catalog.json"
    sources_path: "/Users/maverick/src/ezdihar/trino-dbt-tpch-demo/target/sources.json" # optional for freshness
    test_results_path: "/Users/maverick/src/ezdihar/trino-dbt-tpch-demo/target/run_results.json" # optional for recording dbt test results after running dbt test
    platform: "trino"
    target_platform: "trino" # e.g. bigquery/postgres/etc.
    query_tag_mapping:
      tag:
          match: ".*"
          operation": "add_tag"
          config:
            tag: "{{ $match }}"

# sink configs

sink:
  type: "datahub-rest"
  config:
    # IP of the pod on k8s
    server: "http://<IP_ADDRESS>:8080"

It looks like it's not able to extract anything from dbt. My guess is that it's because there are no "columns" defined for that source in your schema.yml file within the dbt project - see here for an example https://github.com/hsheth2/sample-dbt/blob/318c0d22d37983c60b7e60291d8520e631efaac5/models/base.yml#L12-L30
dbt doesn't actually have any information about the columns/descriptions of your tables unless you explicitly define it. DataHub relies on metadata from dbt, which means we have the same limitation.
Most folks use macros like dbt-codegen to create these yaml files, which will in turn enable our integration to work too.

@maver1ck your setup seems pretty reasonable and seems like it should work. Given that this looks like a demo environment, would it be possible for you to upload your catalog/manifest/sources.json files so I can take a peek?


cc-wrapper: fix on darwin

The ld-wrapper.sh script calls readlink in some circumstances. We need
to ensure that this is the readlink from the coreutils package so
that flag support is as expected.
Without doing this, the following happens with a trivial main.c:
nix-env -f "<nixpkgs>" -iA pkgs.clang
$ clang main.c -L /nix/../nix/store/2ankvagznq062x1gifpxwkk7fp3xwy63-xnu-2422.115.4/Library -o a.out
readlink: illegal option -- f
usage: readlink [-n] [file ...]

The key element is the .. in the path supplied to the linker via a
-L flag. With this patch, the above invocation works correctly on
darwin, whose native /usr/bin/readlink does not support the -f flag.
Fixes #6447

Ah, so the use case here is for interactive invocations of clang/ld, rather than via Nix scripts, right?

Yes, that is correct!

Given that it's a mass rebuild, this should probably go into staging rather than master.

Ah, okay. Can you move it or shall I?

I don't think GitHub allows one to repoint a PR. Someone could just merge the patch manually to staging, but let's get some feedback first. As I mentioned in the linked issue, this will resolve the particular instance but I'm wondering if there are more robust ways to do this more broadly that might apply here too.

Yes, what you said sounded reasonable. I figured that since nothing has happened for several months, fixing this instance now is expedient, and perhaps the lack of broader outcry indicates the problem doesn't arise as frequently as it might. It's a pretty substantial nuisance, as it makes development tools installed via nix much less useful than they otherwise ought to be.

There were multiple impure references in cc-wrapper scripts. I wonder if it's better to hardcode the paths directly 093f58b2c8c4854886acb694081ce90d1e1b2045 or just specify PATH='@coreutils@/bin' at the beginning of a script. (IIRC that shouldn't affect $PATH value exported by the script, but I'm not certain.)

Setting PATH explicitly (and not including its previous value) at the top of our scripts is actually pretty appealing. Advantages (if we make it a policy):

We don't accidentally inherit stuff from the user's shell
We don't have to adorn every invocation of a utility with its full path
We get a loud failure if we forget a dependency, because the utility won't be found
We could in theory verify semi-automatically that our scripts follow the policy (by looking for PATH= in the first few lines, and ensuring that PATH isn't referenced on the RHS)

:+1: from me on that approach

That sounds like the right approach.

This is actually a bit trickier than I anticipated. First, the straightforward idea of just using, e.g., @coreutils@ in the shell scripts to work with substituteAll doesn't work as cc-wrapper has a nativeTools parameter that can cause that variable to not be defined. So I tweaked that a bit as you can see in the new patches, but it still doesn't work; here's the error I get after a while
building path(s) ‘/nix/store/i8lvwn55nbynswx6wdsvw4bvj8aw0scg-stdenv-darwin’
output ‘/nix/store/i8lvwn55nbynswx6wdsvw4bvj8aw0scg-stdenv-darwin’ is not allowed to refer to path ‘/nix/store/7l8l3sp3wk0idx0191qhn4hvdvx0wznq-bootstrap-tools’
error: build of ‘/nix/store/fxsahfv1352licgp95mhwgip109cvj27-stdenv-darwin.drv’ failed

Perhaps one of the more experienced darwin folks (e.g. @copumpkin @shlevy @gridaphobe @pikajude ) might suggest a fix. I can work at it, but it's a slog.

The problem is that the final version of cc-wrapper is referencing the bootstrap coreutils, not the final version. One potential fix would be to rewrap the final clang after building the final coreutils.
But is there any reason we need to use identical wrappers for builds and for users? We already have bashInteractive for example.

I'm not familiar enough with the bring-up of the whole environment to make architectural changes. I had hoped that the nativeTools flag would sufficiently twiddle the configuration between bootstrapping and final builds to avoid issues like this. It seems like adding a ccInteractive would be a fairly intrusive change if it would be necessary to refer to it when building the stdenv.

The whole point is that it wouldn't be necessary to refer to it when building stdenv, right? Since cc-wrapper works fine in nix builds, it only fails for the user?

Oh, I'll have to re-run the test I guess. The build error made me think my attempted rebuild of things broke while building stdenv. These builds take a long time to get back to a development environment I can work with, so I usually kick off a downstream build and check back in a day to read the error messages. I'll try again some other time and only build one thing at a time as that error message is apparently pretty confusing.

You'll have to revert your change. The build error is because the final stdenv contains a reference to the bootstrap tools. If you have a separate ccInteractive, it will refer to the newly-built coreutils, not the bootstrap coreutils.

The linked issue for this PR was the problem being exposed in GHC through its use of the linker. So, if I'm understanding the suggestion, it would be to leave cc-wrapper as it was, but change all downstream dev tools to use a different ccInteractive package?

Ah, yeah. I'm not sure what the best solution is there really.
Hmm, I wonder if the better solution is to include coreutils in nix-env environments by default?

Part of my question that I perhaps didn't make suitably clear is why I'm running into this problem with this change: I don't understand what this change does that introduces the illegal stdenv reference. Perhaps that could suggest an easy way to adjust this approach of explicitly setting PATH as it seems like rather a good idea in general.
It seems like the right fix should be adjacent to the current state of the PR given that my original attempt at just using an explicit '@coreutils@path in thereadlink` call addressed the motivating symptom without any noticeable adverse consequences.

Hmm, are you sure that your original attempt worked? Based on a modern nixpkgs? The allowed-references stuff is somewhat recent.

Yes, I'm sure. I haven't come across allowed-references, though, so perhaps we're still not on the same page.

Sorry, it's allowedReferences and allowedRequisites. The error message above comes from https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/pure-darwin/default.nix#L296-L302

Oh, right, I should have recognized that. I'm actually not sure which specific reference in stdenv is causing the trouble. I've ag'ed through the stdenv store path in the error message, but there's no bootstrap-tools path there, so I'm guessing it's a transitive dep but I'm not sure of an easy(-ish) way of tracking it down.

You can do nix-store -q --tree to see what's bringing in the reference to bootstrap-tools

Well, that's quite a large tree, but the bootstrap-tools references are all over the place: bash, bison, perl, etc. I feel like this is related to the stageFun definition that sets dontPatchShebangs=1 to avoid retaining dependencies. The logic there does a path comparison with the bootstrapTools derivation to control the patching behavior. That seems super fragile, so I'm not sure how to leverage it to fix other path substitution issues. It seems like hack-upon-hack, but one option might be to use that same check to cripple the @coreutils@ substitution.

I misunderstood the suggestion to look atnix-store -q output. Without this PR, bootstrap-tools is still in the nix-store -q --tree output for things like bash, so I don't know what to look at for the build that errors out. I've pasted the tree here, so if anyone can identify the problem, please share. All I know how to do is twiddle things and rebuild, which takes a long time.

You can also get something far clearer with --graph, since --tree duplicates common parts of the graph and makes the whole thing look far larger than it is. You can just feed --graph into graphviz :smile:

@acowley You looked at the graph f or the .drv file. We need to see the graph for the build output, i.e. the runtime dependencies graph.

I looked at the .drv because,
$ nix-store -q --tree /nix/store/i8lvwn55nbynswx6wdsvw4bvj8aw0scg-stdenv-darwin
/nix/store/i8lvwn55nbynswx6wdsvw4bvj8aw0scg-stdenv-darwin
error: path ‘/nix/store/i8lvwn55nbynswx6wdsvw4bvj8aw0scg-stdenv-darwin’ is not valid

I can read the out paths from the mentioned derivations and query them, but is there a faster way of doing so?

Ugh, right, because the build failed. Can you check the dependency tree of the cc-wrapper that's closest to stdenv?

@edolstra How difficult would it be to have nix store the dependency info of a build that failed due to disallowed dependencies (either because of allowedReferences or circular reference in multiple outputs) somewhere for investigations like this?

I don't quite follow what you mean by "closest". I looked at the out path of /nix/store/9zf63nb5x5dy7lznf2xdfh5ba0fdki8g-clang-wrapper-3.6.2.drv (mentioned in the stdenv derivation's tree), and bootstrap-tools didn't show up there, but I"m not sure if that's the same thing.

How difficult would it be to have nix store the dependency info of a build that failed due to disallowed dependencies (either because of allowedReferences or circular reference in multiple outputs) somewhere for investigations like this?

The output paths of the failed build remain in the /nix/store (at least for me). You can grep -R through them...

Small update: I tried taking out the PR lines that clear PATH in gnatlink-wrapper.sh, ld-solaris-wrapper.sh (probably not relevant), and setup-hook.sh. This let me get much further. However, I run into an error building the gnum4 package (needed for building GHC). My suspicion is that I was unsetting a PATH that was being relied on by something like patchShebangs, and the bootstrapTools shell was leaking in as per the warning in the pure-darwin stdenv definition, but that's just a guess.
I dug into the m4 build failure a bit since it's a smaller thing to look at, and it's kind of interesting. The build failures are due to various checks in m4's build system to accommodate Apple's SDK. Since we still pull a version of LibSystem in, we run into trouble with things like redefinitions of sigismember and sigaction (from signal.h in m4's lib directory). I tweaked the C pre-processor guards around a few things to nudge it further along, but then ran into a problem with clock resolution. I stopped digging at this point, but can keep going if anyone thinks it's helpful.
Alternatively, if I take the unpacked nixpkgs m4 package, and ./configure && make in a regular OS X shell, it builds just fine with the normal OS X CLTools.
So either explicitly setting the PATH as suggested is still doing unwanted violence, or we previously had unintended dependencies on the PATH not being pruned down so extensively.
To summarize the current situation, the only working version of this PR -- "working" in that the parts of nixpkgs I use all build, and compilers (clang and ghc) can be used outside of a nix shell -- is the one where only the call to readlink is changed to @coreutils@/bin/readlink. Every other variation of this fix has run aground somewhere.

I finally got this working. The last holdup was that cc-wrapper.sh also calls grep which we should have been ensuring was the grep from nix.
I've only built this on darwin so far (and that was before the last rebase, so I'll build again tonight), so other stdenvs likely need adjustment to satisfy the explicit gnugrep dependency. Does this approach seem okay to folks?

Ah, right, my assumption in https://github.com/NixOS/nixpkgs/pull/11798#issuecomment-165705219 was wrong – apparently it is necessary to do some kind of save-restore dance. While in these cases it's not too bad due to only having a single external exec at the end, it will probably be too cumbersome as a general technique (at least with bash).
I assume you have tested this well on darwin? Linux doesn't evaluate ATM, but I'll have a look into it.
How bash's environment seems to work
If one makes a change to a variable that is exported, that change is propagated to sub-processes. Even if one marks it local, commands called within the function inherit that (new) value.

Yes, I tried to avoid the dance, but couldn't make that work, either. It's been a long, boring voyage :)
I did a new build last night, so I'll work it over today to make sure everything is ship shape on Darwin. It seems fine from poking around clang and GHC, but I'll update if I find a problem.

Everything looks good on Darwin. Can anyone help me navigate getting Linux and NixOS (and WIndows?) validation? I can wrangle VMs, but if there's existing infrastructure I'd rather use that.

I spun up a NixOS VM on EC2, set nixpkgs to a checkout of this branch, and did a nixos-rebuild --switch without any trouble. Installing clang also works fine. Not sure how much that evidence is worth. I'll try a non-NixOS linux when I have some time.

The last commit seems to get Linux up and running.
The trick there is that, as far as I can see, Linux was working because the grep executable came along with the bootstrap tools. Explicitly requiring gnugrep required a bit more plumbing.
Any help getting other stdenvs in order would be much appreciated!

There are rather a lot of arguably problematic shell scripts that make use of utilities. write-mirror-list.sh uses grep, and setup.sh from the generic stdenv uses tar, as examples.
Perhaps this kind of explicit dependency tracking isn't realistic for nixpkgs, and people would rather have these things white listed in some way. It takes hours of compiling to get to a point where these errors come out, and I'm not sure how much more of it I can reasonably do as this seems like perhaps a deliberate choice in the way nixpkgs is designed.
Non-Linux platforms could alternately be supported by patching on an as-needed basis.

Staged after some minor fixups: 8f48a97 and 98c7e70.


Add job performance to max price pdf downloads

Hitas Pull Request
Description
Makes downloading confirmed/unconfirmed max price PDFs create job performance objects.

Frontend changes to include request_date on these endpoints are necessary to make this work!

Pull request checklist
Check the boxes for each DoD item that has been completed:

Testing

[x] Changes have been tested
[x] Automatic tests have been added

Database

[ ] Database migrations will work in the DEV & TEST environments
[ ] initial.json has been updated to work with migrations
[ ] Oracle migration has been updated

Documentation

[ ] Tooltips have been added in the frontend for all new fields
[x] OpenAPI definitions have been updated
[ ] Test instructions have been written for the customer in the appropriate ticket in Jira
[ ] Terminology page in Confluence has been updated

Test plan

[x] Automated tests
[ ] Download max price PDFs (manually before frontend is updated) and observe the job performance objects in the database

Tickets
This pull request resolves all or part of the following ticket(s): HT-514, HT-515, HT-567

@vincit-matu frontend changes can be included in this PR, I think? Otherwise, ninja will break for PDFs when this PR is merged.

We didn't talk about limiting the creation of job performances in any way, so I left it like that. Might be work talking to the customer about it before deciding anything.


Code with union of types is not compiled by TSC 3.1.1

The following code is not compiled by TSC 3.1.1 throwing error: Property 'address' does not exist on type 'string | AddressInfo'. Property 'address' does not exist on type 'string'.
    httpServer.on("listening", () => {
      console.log(
        `Listening on http://${httpServer!.address().address}` +
        ` env: ${process.env.NODE_ENV}`);
    });

corresponding method declared inside Server class as:
    class Server extends events.EventEmitter {
        ...
        address(): AddressInfo | string;

and AddressInfo has field with name address:
    interface AddressInfo {
        address: string;
        family: string;
        port: number;
    }

The code was compiled by previous versions of TS without any errors.

If the value is AddressInfo | string, it's possibly a string. string doesn't have an address property, so the code can possibly try to access a nonexistent property, printing "Listening on http://undefined". It sounds like TypeScript is doing the right thing and probably worked the same in previous versions -- maybe the httpServer library you're using changed?

Ok. Will close.


Tech Toolkits Page

This is in response to the issue #286
https://github.com/user-attachments/assets/57f74c85-ad28-4c49-b74d-fe2af093e9cb

This was the original page

https://github.com/user-attachments/assets/9aaf78a1-bf8a-4a54-8d68-045987a98493


Reduce memory consumption in Microsoft.Net.Http.Client, update tooling

Hi! I looked into the code for this project and noticed there were a few places where performance could be improved. I'll try to break down the changes I made here:

Cache the bytes written in ChunkedWriteStream.EndContentAsync
Use a pooled array from System.Buffers.ArrayPool in BufferedReadStream (I added System.Buffers as a dependency)
Call base.Dispose in HttpConnectionResponseContent (since HttpContent.Dispose actually does work)

I also made a few other changes with the tooling, to make it easier to build/setup the project:

Convert the DNX .xproj files to use the CLI
Update the global.json to use the RTM .NET Core SDK (preview1 is RC2, preview2 is RTM)

Awesome PR. Thanks for the help. I commented inline on one of them but given the release of .NET Core 1.0 a few days back you can either update to that in this PR or I can do it in an other.
Thanks!

@anonpenguin Thanks for this contribution. We just moved this project over to Microsoft ownership, which means there's now a CLA process. Would you mind ensuring that you've signed the Microsoft CLA at https://cla.microsoft.com?

@jstarks Sure, no problem; I've just signed the CLA.

@jterry75 Thanks for responding. I've added another bit to the PR, to also use ArrayPool when renting arrays in MultiplexedStream, so you might want to review that as well.

LGTM

LGTM. Thanks again for making this change.


Labels fail to print correctly with newlines

Hello, I am trying to use ariadne for error reporting for a crate. I am getting a problem with printing Reports that span multiple lines.
Here is some example code:
#[test]
fn ariadne() {
    let source = "(\r\n\r\n}";

    Report::build(ReportKind::Error, (), 0)
        .with_config(Config::default().with_char_set(CharSet::Ascii))
        .with_label(Label::new(0..1).with_message("opens here"))
        .with_label(Label::new(5..6).with_message("closes here"))
        .finish()
        .eprint(Source::from(source))
        .unwrap();
}

and it produces this:
Error:
   ,-[<unknown>:1:1]
   |
 1 | (
   : |
   : `-- opens here
---'

What should happen: The start and end lines are displayed along with both labels
What actually happens: Only the start line is displayed; the second label is completely absent.

I cannot figure out what causes it to go wrong but my hunch is that the span from 5..6 is the problem because 6 is "out of bounds" of the string (even though it's used as the exclusive endpoint).

I should note that it displays perfectly fine if there aren't carriage returns (even if the range's end is equal to the length of the string.) Replacing the \r in the string with r makes it print totally fine.

I will try to find the time to look into this tomorrow. I think it should probably be quite a simple fix.

I have done some more investigating and this appears to only happen with carriage returns in the source code. Here's an example:
#[test]
fn ariadne_borked() {
    let test_str = r#"(my
(complex
    lisp 123
    with
    (a bad token here))!
(oh the horror)"#;

    let lf = test_str.replace("\r\n", "\n");
    let crlf = lf.replace("\n", "\r\n");

    for (test_name, source) in [("lf", lf), ("crlf", crlf)] {
        let bad_pos = source.as_str().find('!').unwrap();
        println!("{}:", test_name);
        let error = Report::build(ReportKind::Error, (), bad_pos)
            .with_label(Label::new(bad_pos..bad_pos + 1).with_message("this token is invalid"))
            .finish();
        error.eprint(Source::from(source)).unwrap();
    }
}

and here's its output:
lf:
Error:
   ╭─[<unknown>:5:24]
   │
 5 │     (a bad token here))!
   ·                        ┬
   ·                        ╰── this token is invalid
───╯
crlf:
Error:
   ╭─[<unknown>:6:3]
   │
 6 │ (oh the horror)
   ·   ┬
   ·   ╰── this token is invalid
───╯

Thanks for the further investigation. Unfortunately, I'm away for the next 10 days so am unable to fix this (unless I can find time to do so). Is this a high-priority issue for you? If so, you could temporarily resolve it by replacing carriage return characters in your source files with spaces or some other invisible character. I believe that this should work.

Yes .replace('\r', ""); is working great for me.

Great, although it's worth mentioning that removing the character entirely will result in the wrong character offset being used for later lines! You'll probably need to replace it with an actual character.

Sorry I meant i am stripping it out of the text before I parse it at all. my parser/error handler never gets a string with an \r in it at all.


fixed error with non-constant mu vector in dcar_proper sampler

Fixes a bug in MCMC sampling of dcar_proper distributions, when the mean vector mu is non-constant.
The original (buggy) version correctly implemented the equations in the GeoBUGS User Manual, Version 1.2 (September 2004), which contains a typo in the conditional specification of the proper CAR distribution.
Fixes #1157

Noting this PR might not initially pass testing, since testing may well include posterior results for proper CAR models.

Thanks, @danielturek . Would you like to fix the equation for the conditional density for the CAR in the manual or should I?

Done.  Nice eye @paciorek

I have an efficiency request here. Can you call getParam for mu once (instead of twice) and then index into it for the target and for the neighbors. I did a quick check and changing to that looks like it might save 10% of the MCMC time on Connor's example.

@paciorek I just made these changes.  Only one call to getParam now.  See what you think.

@danielturek did you forget to push to github?

@paciorek Sorry.  Done.  Pushed just now.

Three further thoughts (sorry to belabor things)

So the 'plusOne' business seems hard to read. Is there a reason not to assign the full vector of means to a variable and then index that as needed to get the targetMu and neighborMus? I don't think there will be extra cost since there presumably is at least temporary vector created by the output of getParam, regardless of whether we create one explicitly. Sorry to back-seat drive, but given I'm looking at it, that's my reaction.
Can we take this opportunity to remove the for loop in the setup code and use %in%:

neighborIndices[1, ] <- which(targetDCARscalarComponents %in% neighborNodes)

Also, do you know why we are using 1-row 2-d arrays instead of a vector for neighborsC, and neighborIndices?

No worries about pushing back on these design/implementation choices.

Logic is that the total number of means could be huge, but largest number of neighbors comparatively small.  I think this makes sense.
Could use %in%, but trying to be defensive against non-consecutive declarations of CAR process nodes.  Maybe that's not possible, but this is explicit, and it's in setup code, so this seemed safer to me.  Also noting, the code you drafted would not work, would be incorrect, due to LHS indexing - so more care and code would be necessary regardless.
Option is between creating vectors of at least +2 length extra (to avoid scalar vs. vector ambiguity, while also accounting for the num neighbors = 0 case) or otherwise to use arrays, as was done here.  If using vectors is known to be much better for performance reasons, then this design choice could be changed.

Ok, thanks for responses.
Regarding item 1, I don't see how your approach saves anything. In this code
targetNeighborMus <- model$getParam(targetDCAR, 'mu')[targetNeighborIndices[1,1:numNeighborsPlusOne]]

based on looking at the generated C++, an intermediate variable of length N is getting created based on the call to getParam, so doing the indexing within the RHS is no different than assigning the result of getParam to an explicit variable and then grabbing the subsets out of the full vector.

Side note: I was curious if it mattered that we call getParam on every individual target, so I did some monkeying around (I needed to create a version of $run that took an input parameter, which required creating sampler_BASE2) and if we instead call getParam before looping through the componentSamplerFunctions and pass the mean vector into the individual samplers it does save a bit of time on this example (9.2 seconds vs. 9.8). Not sure how that would scale as N changes. Not advocating we do this now, but something to consider more at some point.

@paciorek Thanks for explaining this.  I understand your comments, about each component sampler getting the entire mean vector, into a length-N vector, then subsetting as necessary.
I agree, this could be avoided by one call to getParam in the encompassing CAR sampler function, then passing the relevant neighbor mean components to each component (scalar) sampler function.  Given the small amount of testing you did, my inclination is to leave this as-is - which is working, simpler, and very similar in your speed test.
How do you feel about that?  Or, is there another suggestion that you made, which I missed?

I'm happy to skip using getParam once outside of the individual sampler functions, but I would still vote that we use getParam inside the individual sampler function run code to extract the entire mean vector to a local variable in the run code and then subset that to get the target and the neighbor means, avoiding the "plusOne" approach of concatenating the target and neighbor indices, which I think makes the code harder to read. As I mentioned in either your approach or my suggestion, there is a temporary N-long vector being created in the C++, so using getParam and then extracting the target+neighbor values in one line of code doesn't actually save us anything.

@paciorek Ok, that makes sense.  Are you able to make this change in the CAR component sampler code?

@danielturek Yes, I'll make that change.

Ok, we now set a variable that contains the entire vector of mu values in the run code and then index that.
I also realized that we could use match in place of the for loop in setup code and it should achieve the same robustness but much faster. I do think that with a for loop over target elements and then the for loop in setup code that we could have non-negligible time spent on that loop and the use of match should speed things up by at least an order of magnitude if not more.
@danielturek feel free to take a look.

Just pushed a minor change of variable name.
(The previous name targetNeighborIndices was meant to suggest "target index and neighbor indices together".  Changed this name back to neighborIndices)

Good catch. Thx.


InputRequired validator doesn't work for FieldList

The InputRequired validator checks the raw_data attribute of the field to determine if input was submitted, but this attribute is never populated for FieldLists. This causes validation to fail even when at least one valid list item is submitted.
This appears to be the same problem mentioned in #220.
I think the validator could easily be modified to conditionally check the entries attribute for FieldLists, but I'm not sure if that is the preferred solution for this bug.

If you need to validate that all fields in a ListField are filled, that's a good case for a custom validator. And who's to say that "required" means "all fields have data" rather than "at least one field has data"? I don't want to add behavior for specific fields to generic validators.
However, you'd probably be better served by adding the validator to the wrapped field instead. (This is off the top of my head, might need adjustment.)
items = FieldList(StringField(validators=[DataRequired()]))

I only want to validate that there is at least one entry in the list. In my opinion, that is the only reasonable interpretation of "required" when it comes to a FieldList.
It should at least be clarified somewhere that the InputRequired validator does not work for FieldLists because I've seen it mistakenly used in a few places now, including in the mongoengine model-to-form converter in Flask-Admin.

Just to chime in, my interpretation of required for FieldList is that an empty list was provided.


Spring JdbcTemplate batch update causing leak detection trigger

using mysql and Spring jdbc template the line is triggering the error is in spring code
Connection con = dataSource.getConnection();
2015-10-28 15:20:00,209 [Hikari Housekeeping Timer (pool oeDatabase)]
WARN  com.zaxxer.hikari.util.LeakTask- Connection leak detection triggered, stack trace follows
java.lang.Exception
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:658)
at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:946)
at com.outboundengine.dba.base.BaseEntity.batchUpdate(BaseEntity.java:184)

You might try asking this on the Spring forum or stackoverflow.  It looks like someone isn't closing the connection -- or the connection is in use for longer than the leak detection threshold.  Either it is a real leak, or you need to extend the leakDetectionThreshold to a longer period.

ok thanks
On Sat, Oct 31, 2015 at 10:58 PM, Brett Wooldridge<EMAIL_ADDRESS>
wrote:

You might try asking this on the Spring forum or stackoverflow. It looks
like someone isn't closing the connection -- or the connection is in use
for longer than the leak detection threshold. Either it is a real leak, or
you need to extend the leakDetectionThreshold to a longer period.
—
Reply to this email directly or view it on GitHub
https://github.com/brettwooldridge/HikariCP/issues/475#issuecomment-152792151
.


Shap features add support to model pipelines

Problem Description
Currently, when passing the model pipeline to any shap feature, we use SHAP explainer to explain it directly, which causes an error.
Desired Outcome
All shap features e.g. ShapRFECV should support using sklearn and imblearn pipelines.
Solution Outline
Before passing the model to shap.explainer in probatus.uitils.shap_helpers.shap_calccheck if the model is a model pipeline and if so, extract the last step from it.
The support for the pipelines will be quite limited. It will only work if the previous steps of the pipeline do not perturb the feature names or number of features. For instance, if the user uses PCA to reduce number of features,  this will cause that the model is not analysable using SHAP anymore.

I am considering implementing this issue. However, I see a couple of problems. Pipeline might apply different steps before running the model. It can be extracting features, rescaling the data, over or undersampling and many others. If you want to apply SHAP to analyse a pipeline you have to first apply all transformers of the pipeline on the input data, and then apply the estimator. So there are following side effects:

Data rescaling and simple transformations - different values of the features compared to the initial values. This might cause that the summary plot of the model or analysing specific samples will be significantly harder for the user, since we apply the transformations right before passing the model to SHAP
Feature extraction, selection,  PCA, missing indicators etc - Error because we have new features, and we don't have names for these features. I also think we should not try to find the names for these features,  because this will be very hard to implement inside ShapRFECV
Some features dropped in the pipeline - Error

Having in mind that we are able to support the pipelines partially for SHAP features, and when we do, the results might be misleading for the user, I think the user should apply all the transformations before passing it to e.g. ShapRFECV, Model interpret or ShapResemblance model.
This has certain drawbacks in case the user wants to use methods that require passing X1 and X2 or X_train and X_test, because then the user has to apply the transformers correctly on these sets. We might want to add a docs page, in which we explain how to handle model pipelines for different use cases. And throw an informative error if the model pipeline is passed to Shap.Explainer
@timvink @sbjelogr what do you think?

You have a good point, @Matgrb.
Indeed shap values are well defined for a model. The transformers might mess-up the interpretation quite a bit.
In the first point (Data rescaling and simple transformations), as long as the transformations are linear (say MinMaxScaler), it's easy to translate the interpretations (even with a summary plot).
However, in this category you might have some other transformers (like the imputers, FunctionTransormers, or any categorical encoder that encodes the target information (Example WoeEncoders), that are not so easy to interpret.
In addition, depending on how you use imblearn transfromers on a training and validation set, you might have very misleading shap interpretations (recall that shap always calculates the marginal contribution of a feature relative to the average target distribution on your sample, and imblearn transformers are probably gonna mess up the average distribution of the sample, hence affecting the shap interpretations)
I would suggest to have a warning being raised by default in the first case (that might be turned off by a verbosity flag for example), something like
def func_name(...,verbosity=1,):

  if verbosity >1:
      warnings.warn("Shap interpretation might not be well defined with a pipeline. The recommended approach is to first apply the transformers, and then use the function with the last step of the pipeline"

In addition, all the three points you mention  definitely an issue when you need to use shap for interpretation purposes.
However, in the context of feature selection,  with ShapRFECV, probably you might be able to have looser requirements on which transformers you reject by raising an error)

I think for now it will be safer to just not support model pipelines and ask users to apply it beforehand.
Indeed ShapRFECV could have looser requirements, let's see if users of the package make more issues about it, and how they want to use it in their example code.
For now I added an error with a message explaining that we don't support it and they need to apply the transformers beforehand.