Change Embedding model to Jina Embeddings

Right now it uses OpenAI embedding models, change it to use https://docs.llamaindex.ai/en/stable/examples/embeddings/jinaai_embeddings/

can you check if I am going anywhere with this ??
I'd like to get off this repo


4688 Header ending with -bin need buffer values

For metadata entries where the name ends with -bin the values must be
Buffers instead of strings.
The solution checks if the metadata entry should be a buffer and, when
yes, constructs a Buffer from the value. The value is split into the
encoding part and the real value part. If no encoding is specified the
Buffer will be constructed with the default encoding.
The default encoding when none is provided to the Buffer.from call is utf8.
The solution accepts empty values, as well as values without the encoding part.
The encoding part is not restricted to the current encoding values, so it should
be ok if in future new values are added. It is restricted though to some characters:

a-z
A-Z
0-9
underscore (_)
hyphen (-)

Closes #4688.

Is there anyway to get some more traction on this fix ?

:thinking: Is there anything I can do to improve the chances of pushing this forward?

Hi @jevooq thanks for contributing this one!
We're going to close this for now and tackle the -bin issue when we work on gRPC later on as part of our roadmap.
Feel free to report any issues you find on gRPC as those will be useful for us, similar to the #4688 issue.


【Feedback】UX improvement_Fix position when scrolling

Can we fix the position of address bar when scrolling?
It is convenient for users to perform some operations （delegate, send, withdraw rewards） while scrolling to view account details.
https://www.figma.com/proto/w7LpoB8DejNVumjJPDrp7j/Forbole-X-_web-app_1.0-2021-04?page-id=192%3A2&node-id=2586%3A200026&viewport=241%2C48%2C0.18&scaling=scale-down&starting-point-node-id=2586%3A200026

Please add your planning poker estimate with ZenHub @rachelhox


Enumeration Screen Redirects Incorrectly

From Brenda: When you add an enumeration screen, it takes you to the last screen in the program instead of the new created repeater screen).
Screenshot attached.

This occurs since the block creation handler always redirects to the last block definition in the program (code). Non-enumerator blocks are always added to the end of the list, so this is the correct behavior in that case. For enumerators though, the newly added repeated screen would be added as the last block for the enumerator's parent block.
Presently, the updated program is returned from ProgramService.addRepeatedBlockToProgram ProgramService.addBlockToProgram. This path is the only current callsite. To fix this, we can return a tuple of the Program as well as the newly added block. Then when creating the redirect URL, the new block can be used instead. There will need to be some adjustments in the error case, since this also displays errors by rendering a form using the last block in the program (code).


Permission denied and "GET / HTTP/1.1" 404 using sam local with examples

Hello,
I am wanting to use the aws-lambda-web-adapter with the native Python lambda runtime, not Flask or FastAPI. Since there isn't one in the examples, to iterate faster, I wanted to first get sam local working with one of the examples but I am having a tough time getting sam local start-api or sam local invoke-lambda working.
These are the steps I followed:

Cloned the repo
cd examples/expressjs-zip since it is mentioned https://github.com/awslabs/aws-lambda-web-adapter#lambda-functions-packaged-as-zip-package-for-aws-managed-runtimes
Modify 8080 port in app.js to 8081 so it doesn't conflict with Emulator
sam build --use-container
sam local start-api
This brings up the following:

❯ sam local start-api
Mounting HelloWorldFunction at http://<IP_ADDRESS>:3000/ [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
Mounting HelloWorldFunction at http://<IP_ADDRESS>:3000/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you used sam build before
running local commands, you will need to re-run sam build for the changes to be picked up. You only need to restart SAM CLI if you update your AWS SAM template
2023-09-27 21:00:44 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://<IP_ADDRESS>:3000
2023-09-27 21:00:44 Press CTRL+C to quit

When I make the request:
curl --location 'http://<IP_ADDRESS>:3000/' I get a 404 and see the following in the logs:
Invoking run.sh (nodejs14.x)
arn:aws:lambda:us-east-1:753240598075:layer:LambdaAdapterLayerX86:17 is already cached. Skipping download
Local image is up-to-date
Using local image: samcli/lambda-nodejs:14-x86_64-34d8303825d2c9fd2d540071c.

Mounting /Users/kentonparton/code/aws-lambda-web-adapter/examples/expressjs-zip/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 0d6216dd-715c-433d-be3f-839bf33552d5 Version: $LATEST
END RequestId: 0d6216dd-715c-433d-be3f-839bf33552d5
REPORT RequestId: 0d6216dd-715c-433d-be3f-839bf33552d5  Init Duration: 0.95 ms  Duration: 176.81 ms     Billed Duration: 177 ms Memory Size: 1024 MB    Max Memory Used: 1024 MB
/opt/bootstrap: /var/task/run.sh: /bin/bash: bad interpreter: Permission denied
/opt/bootstrap: line 3: /var/task/run.sh: Success
2023-09-27 21:01:57 <IP_ADDRESS> - - [27/Sep/2023 21:01:57] "GET / HTTP/1.1" 404 -

I have tried the other examples and run into the same issue. Am I missing a step? Thanks!

Hi Kenton,
Thanks for raising this issue. From the logs, the issue is about the execution permission with run.sh shell.
/opt/bootstrap: /var/task/run.sh: /bin/bash: bad interpreter: Permission denied

To fix it, run the following command.
chmod +x hello-world/run.sh

We're closer! I'm just getting the 404 now:
Invoking run.sh (nodejs14.x)
arn:aws:lambda:us-east-1:753240598075:layer:LambdaAdapterLayerX86:17 is already cached. Skipping download
Local image is up-to-date
Using local image: samcli/lambda-nodejs:14-x86_64-34d8303825d2c9fd2d540071c.

Mounting /Users/kentonparton/code/aws-lambda-web-adapter/examples/expressjs-zip/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: c3f6a5d8-5d2d-4fdc-b645-2c7859763863 Version: $LATEST
END RequestId: c3f6a5d8-5d2d-4fdc-b645-2c7859763863
REPORT RequestId: c3f6a5d8-5d2d-4fdc-b645-2c7859763863  Init Duration: 2.19 ms  Duration: 216.45 ms     Billed Duration: 217 ms Memory Size: 1024 MB    Max Memory Used: 1024 MB
2023-09-28 07:08:13 <IP_ADDRESS> - - [28/Sep/2023 07:08:13] "GET / HTTP/1.1" 404 -
Invoking run.sh (nodejs14.x)
arn:aws:lambda:us-east-1:753240598075:layer:LambdaAdapterLayerX86:17 is already cached. Skipping download
Using local image: samcli/lambda-nodejs:14-x86_64-34d8303825d2c9fd2d540071c.

Mounting /Users/kentonparton/code/aws-lambda-web-adapter/examples/expressjs-zip/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: e9f96a37-3d81-4654-8e7d-f31eb5379685 Version: $LATEST
END RequestId: e9f96a37-3d81-4654-8e7d-f31eb5379685
REPORT RequestId: e9f96a37-3d81-4654-8e7d-f31eb5379685  Init Duration: 1.76 ms  Duration: 172.46 ms     Billed Duration: 173 ms Memory Size: 1024 MB    Max Memory Used: 1024 MB
2023-09-28 07:08:49 <IP_ADDRESS> - - [28/Sep/2023 07:08:49] "GET /hello HTTP/1.1" 404 -

I tried:

curl --location 'http://<IP_ADDRESS>:3000/'
curl --location 'http://<IP_ADDRESS>:3000/hello'

P.S. I had tried chmod +x hello-world/run.sh but I never re-built afterwards.

Did you add the environment variable 'PATH=8081'?

Ahh shoot, I updated the PORT in app.js instead of as a ENV variable.
On expressjs-zip, I am getting the following, this is more of an FYI. It is working with the fastapi-zip
Mounting /Users/kentonparton/code/aws-lambda-web-adapter/examples/expressjs-zip/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: d8bf0672-e345-466a-9a1e-7c13ec3f5112 Version: $LATEST
Function 'HelloWorldFunction' timed out after 3 seconds
Example app listening at http://localhost:8081
No response from invoke container for HelloWorldFunction
Invalid lambda response received: Lambda response must be valid json
2023-09-28 12:20:33 <IP_ADDRESS> - - [28/Sep/2023 12:20:33] "GET / HTTP/1.1" 502 -

Personally, I am not wanting to run FastAPI or Flask but rather invoke a Python handler like it's done natively:
handler.py
import json

def lambda_handler(event, context):
    print(event)
    print(context)
    return {
        "status_code": 200,
        "body": json.dumps({"message": "hello world"})
    }

The reason for using aws-lambda-web-adapter is so that I can leverage response streaming.
With this is mind, is there a way to use Handler: app.handler instead of Handler: run.sh and would I expect event and context to be present or would I access it another way?

The official python managed runtime does not support response streaming yet. So we have to use LWA and start a web server to work around that.

Did you checkout FastAPI response streaming example?

Yeah, I've given the FastAPI response streaming example a try. It works perfectly.
I was hoping to avoid the additional packages. Also, all my code is already boken out into separate lambda endpoints. I was hoping there was a way to invoke a handler function like a regular lambda, strip out the Starlette StreamingResponse implementation, and stream that way. Not possible?

No, it's not possible until the official Python Runtime supporting it.


Add embedding related methods in numpy version

Description
Create embedding related methods in 'gluonnlp.embedding':
embed_loader.list_sources: Get valid token embedding names and their pre-trained file names.
embed_loader.load_embeddings: Load pretrained embedding file to build an embedding matrix for a given Vocab.
evaluation.CosineSimilarity: a function to compute the cosine similarity.
evaluation.HyperbolicCosineSimilarity: a function to compute the cosine similarity in the Hyperbolic space.
evaluation.ThreeCosAdd: a Class for 3CosAdd analogy.
evaluation.ThreeCosMul: a Class for 3CosMul analogy.
About evaluation
Currently the implementations of embedding.evaluation are not very satisfactory. Suggestions are welcome.
Checklist
Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

Is it possible to get the embedding of words in raw text if it’s a HybridBlock? We may want to calculate the embedding from raw text or out-of-vocabulary words. which is the purpose of FastText.
Get Outlook for iOShttps://aka.ms/o0ukef

From: acphile<EMAIL_ADDRESS>Sent: Saturday, July 18, 2020 2:13:41 AM
To: dmlc/gluon-nlp<EMAIL_ADDRESS>Cc: Xingjian SHI<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [dmlc/gluon-nlp] Add embedding related methods in numpy version (#1263)
@acphile commented on this pull request.

In src/gluonnlp/embedding/embed_loader.pyhttps://github.com/dmlc/gluon-nlp/pull/1263#discussion_r456769686:

for cls_name, embedding_cls in text_embedding_reg.items():

   if pretrained_name_or_dir in embedding_cls:

       source = pretrained_name_or_dir

       embedding_dir = os.path.join(root_path, cls_name)

       file_name, file_hash = embedding_cls[source]

       url = _get_file_url(cls_name, file_name)

       file_path = os.path.join(embedding_dir, file_name)

       if not os.path.exists(file_path) or not check_sha1(file_path, file_hash):

           logging.info('Embedding file {} is not found. Downloading from Gluon Repository. '

                        'This may take some time.'.format(file_name))

           download(url, file_path, sha1_hash=file_hash)

       return file_path

return None

+def load_embeddings(vocab, pretrained_name_or_dir='glove.6B.50d', unknown='',
I think we can set a base class Class EmbeddingModel(HybridBlock) to serve as the base class of embedding models. And we can attach some evaluation functions to this class. For just loading a embedding matrix, we can simply use the current load_embeddings for users to manually set_data, or have Class WordEmbedding(EmbeddingModel) and move the functionality of load_embeddings to this class. And complex embedding models like FastText, character-level CNN can be further implemented based on EmbeddingModel. This embedding models may be implemented in models/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/pull/1263#discussion_r456769686, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3QCI7C27JYYSU5VEKLR4FRULANCNFSM4O2EOW6Q.

Is it possible to get the embedding of words in raw text if it’s a HybridBlock? We may want to calculate the embedding from raw text or out-of-vocabulary words. which is the purpose of FastText. Get Outlook for iOShttps://aka.ms/o0ukef
________________________________ From: acphile<EMAIL_ADDRESS>Sent: Saturday, July 18, 2020 2:13:41 AM To: dmlc/gluon-nlp<EMAIL_ADDRESS>Cc: Xingjian SHI<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [dmlc/gluon-nlp] Add embedding related methods in numpy version (#1263) @acphile commented on this pull request.
________________________________ In src/gluonnlp/embedding/embed_loader.py<#1263 (comment)>:

for cls_name, embedding_cls in text_embedding_reg.items():
if pretrained_name_or_dir in embedding_cls: + source = pretrained_name_or_dir + embedding_dir = os.path.join(root_path, cls_name) + file_name, file_hash = embedding_cls[source] + url = _get_file_url(cls_name, file_name) + file_path = os.path.join(embedding_dir, file_name) + if not os.path.exists(file_path) or not check_sha1(file_path, file_hash): + logging.info('Embedding file {} is not found. Downloading from Gluon Repository. ' + 'This may take some time.'.format(file_name)) + download(url, file_path, sha1_hash=file_hash) + return file_path + + return None + +def load_embeddings(vocab, pretrained_name_or_dir='glove.6B.50d', unknown='', I think we can set a base class Class EmbeddingModel(HybridBlock) to serve as the base class of embedding models. And we can attach some evaluation functions to this class. For just loading a embedding matrix, we can simply use the current load_embeddings for users to manually set_data, or have Class WordEmbedding(EmbeddingModel) and move the functionality of load_embeddings to this class. And complex embedding models like FastText, character-level CNN can be further implemented based on EmbeddingModel. This embedding models may be implemented in models/ — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1263 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3QCI7C27JYYSU5VEKLR4FRULANCNFSM4O2EOW6Q.

For getting the embeddings for unknown words, there are the following situations:
1. words in the vocabulary but not in the embedding file
The default method is to sample from normal distribution. And users can use unk_method to define your own way. I made some simple tests that fasttext.cc actually computes faster than the original gluon approach in v0.9.x, so you simply can do as follows:
fast = fasttext.load_model('model.bin')
def ngram(words):
     return np.array([fast[word] for word in words])
embedding_matrix = load_embeddings(vocab, source, unk_method=ngram)

In this case, we get an embedding matrix for a given vocabulary.
Now I have added the feature that use the embedding file as the vocabulary:
embedding_matrix, vocab = load_embeddings(vocab=None, pretrained_name_or_dir=source)

2. words not in the vocabulary
I think generally we just use the embedding of <unk> for these OOV words. Of course we can still generate some initial embedding vectors by FastText, but since they are not updated during training, I don't think it is very useful.
To further maintain the information from these words, in practice we may use some character-level NN. For example, we may use a character-level CNN to compute the embedding of a word and the parameters are learnable during training. That's why I think we should create a base class EmbeddingModel(HybridBlock) and have some embedding models (in the other words, part of the Neural Network) so that it would be easier for users to build their NLP models.
EmbeddingModel (and its children class) can serve as a black box: Inside there can be only a simple embedding lookup or some NN layers. What we want is to input List[word index] or List[List[character index] (maybe both or others) and get the word representations which can even be contextualized representations. So we can set the embedding evaluation functions as the class methods in a more general way.

The advantage of fasttext is that there is no need to care about OOV words. Thus, you may need to still offer this functionality.

@acphile This is the advantage of using subwords, there will be no/less OOV words if you are using a subword repsentation. For example, GPT-2/GPT-3 chose to use the byte-based BPE encoding because there will never be OOV words. Also, you may check Section 2.1 of https://arxiv.org/pdf/1911.03688.pdf to see how different models may adopt different strategies for dealing with the OOV problem.

@acphile This is the advantage of using subwords. Basically, there will be no/less number of OOV words if you are using a subword repsentation. For example, GPT-2/GPT-3 chose to use the byte-based BPE encoding because there will never be OOV words. Also, you may check Section 2.1 of https://arxiv.org/pdf/1911.03688.pdf to see how different models may adopt different strategies for dealing with the OOV problem.

I understand that and in my context, vocabulary not only refers to vocabulary of a word ('vocabulary') but also refers to a lookup dict which records the different tokens for a certain input. For example, a vocabulary of trigram records the most trigrams which occurs in the dataset. Since raw text can be transformed to several different types of inputs (like List[word], List[List[ngram]],List[BPE]) by tokenizer and further transformed to List[int] or List[List[int]] by vocab. For embedding part, we only need to handle with integers and that's why previous I suggest using EmbeddingModel(HybridBlock). Of course FastText is very useful and I think it is better to implement as Class FastText(EmbeddingModel) instead of Class FastText:. It is somewhat like https://github.com/dmlc/gluon-nlp/blob/v0.9.x/src/gluonnlp/model/train/embedding.py#L175 but I think we could using vocab and tokenizer instead or just separate subword_function and token_to_idx from this class directly

@acphile The problem is that it will be inefficient to have the tokenizer output all the ngram combinations. Instead, you ask the tokenizer to output a list of tokens and each token will be converted to the embedding.

@acphile Is it possible to also refer to the implementation in gensim https://radimrehurek.com/gensim/models/fasttext.html#module-gensim.models.fasttext?

@acphile The problem is that it will be inefficient to have the tokenizer output all the ngram combinations. Instead, you ask the tokenizer to output a list of tokens and each token will be converted to the embedding.

For each token, gensim still output all ngrams to compute the corresponding embeddings: https://github.com/RaRe-Technologies/gensim/blob/c0e0169565116854993b22efef29e3c402ec6c69/gensim/models/fasttext_inner.pyx#L672
And they use hash buckets for converting ngrams to indexes: https://github.com/RaRe-Technologies/gensim/blob/c0e0169565116854993b22efef29e3c402ec6c69/gensim/models/fasttext.py#L1289
I think maybe we can make a supplement for hash lookup in vocab.

I think it’s better to do it internally and the user may not need to care about it. Basically, we need a way to map raw text tokens to embedding vectors.
Get Outlook for iOShttps://aka.ms/o0ukef

From: phile<EMAIL_ADDRESS>Sent: Tuesday, July 21, 2020 4:06:26 AM
To: dmlc/gluon-nlp<EMAIL_ADDRESS>Cc: Xingjian SHI<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [dmlc/gluon-nlp] Add embedding related methods in numpy version (#1263)
@acphilehttps://github.com/acphile The problem is that it will be inefficient to have the tokenizer output all the ngram combinations. Instead, you ask the tokenizer to output a list of tokens and each token will be converted to the embedding.
For each token, gensim still output all ngrams to compute the corresponding embeddings: https://github.com/RaRe-Technologies/gensim/blob/c0e0169565116854993b22efef29e3c402ec6c69/gensim/models/fasttext_inner.pyx#L672
And they use hash buckets for converting ngrams to indexes: https://github.com/RaRe-Technologies/gensim/blob/c0e0169565116854993b22efef29e3c402ec6c69/gensim/models/fasttext.py#L1289
I think maybe we can make a supplement for hash lookup in vocab.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/pull/1263#issuecomment-661790310, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3RJXLSXID2G23UIHPDR4VZDFANCNFSM4O2EOW6Q.

Can you add some tests in https://github.com/dmlc/gluon-nlp/tree/numpy/tests?


install-node: better error handling

don't use set -e, read http://mywiki.wooledge.org/BashFAQ/105#Exercises
don't mask wget stderr output with -q
add -o pipefail so wget | bash fails if wget fails

What's this PR for?
better error handling in install-node
Testing Plan:
run the script. The semantics haven't changed.

@andersk what are your thoughts on this?

don't use set -e, read http://mywiki.wooledge.org/BashFAQ/105#Exercises

So Bash is a crappy language, yes—but that doesn’t mean we should go around deleting all our error handling! set -e works just fine for a simple script like this, and is certainly better than nothing.

So Bash is a crappy language, yes—but that doesn’t mean we should go around deleting all our error handling! set -e works just fine for a simple script like this, and is certainly better than nothing.

My main concern with -e is that it gives a false sense of security regarding error handling. This is a pet peeve of mine so pardon the longer reply. Consider these simple scenarios, similar to ones in the link posted. Try to guess what the script does before checking the results.
#!/usr/bin/env bash
set -e
false && echo no
echo yes

resultThis echoes `yes`
Now we want to move the first line into a function:
#!/usr/bin/env bash
set -e
function f { false && echo no; }
f
echo yes

resultThis doesn't echo anything
This means a minor refactor changed the error handling of your code! This is non-trivial to follow.
What happens if we move the whole thing into a function?
#!/usr/bin/env bash
set -e
function f {
    false && echo no
    echo yes
}
f

resultThis echoes yes again
Seems ok. What if we change the first function line to just false?
#!/usr/bin/env bash
set -e
function f {
    false
    echo yes
}
f

resultit doesn't print anything as expected
What if we later need to capture the output of this function?
#!/usr/bin/env bash
set -e
function f {
    false
    echo yes
}
x=$(f)
echo $x

resultThis again echoes yes!
That's because the $(f) subshell doesn't inherit set -e (there's a new option in bash that does propagate it with shopt -s inherit_errexit, not in xenial yet). So now it's not just a refactor that can cause a change but also the type of invocation of our functions! If you want it to behave the same you have to put set -e in every function.
If I can't refactor the code as-is without changing the error handling semantics I feel really unsafe. If a function call behaves differently when called directly as opposed to being called in a subshell I don't feel safe either.
If you can wrap your head around all these "rules" then you're safe. I really can't though and never use set -e for that reason.
Opinions welcome :)

With #10162 I can close this and create a new PR that only adds pipefail and wget -nv to solve the most important issue. If you find my "don't use set -e" agreeable I can open a new issue to discuss this separately.

I'm pretty comfortable with the shell rules around error handling, and I think integrating shellcheck will help folks new to the codebase with avoiding risky patterns in our shell codebase (which I don't expect to grow a lot over time; most of the stuff that changes often is in Python anyway).  And regardless, the set -e model of making things fail by default in most places is definitely better than the alternative of having to do explicit error handling checks after ever line :).

I'm pretty comfortable with the shell rules around error handling

If by that you mean you knew the answer to all my examples then I tip my hat.

the set -e model of making things fail by default in most places is definitely better than the alternative of having to do explicit error handling checks after ever line :).

I disagree, but that's OK.


Flux Rift size should scale (a bit) with node aspect count

For example, if I have drained a node completely so that it has zero points of anything left, or even just one or two points, it shouldn't make a rift when broken.
Conversely, if I have built or found a node with thousands or tens of thousands of aspect points (it's happened), it should make a much bigger rift, although there should be an upper cap, of course, to prevent world-eating disasters.
Perhaps a config option for this behaviour vs just the same size every time.

Rifts for nos scale from node breaker's warp. Guess I'll rework that.

Ah, that explains why they were all maximum size in my testing world, lol.


[bug] julich v29 does not support fsaverage

siibra version
as of v0.2a51 (commit 5143f40a20b5d66f91ee26ef9afb3b2779ed4c40)
how to reproduce:
import siibra
atlas=siibra.atlases['human']
fsaverage=atlas.spaces['fsaverage']
julich290=atlas.parcellations['julich 2 9']
julich290.supports_space(fsaverage) # expects True, actual return False

it does show the following warning message:
[siibra:INFO] A GiftiSurfaceLabeling object was registered, but this type is not yet explicitly supported.

Why is this an issue?
siibra-api uses the supports call to populate availableParcellations key, which is in turn used by siibra-explorer to relate template spaces with parcellations.
should julich v29 not support fsaverage, siibra-explorer will not be able to show julich v29 in fsaverage
other
possibly related to #13

The problem is in core/concept.py:
    @property
    def supported_spaces(self):
        """
        The list of spaces for which volumetric datasets are registered.
        """
        return list({v.space for v in self.volumes})

At the time of cec273e17547ab0bb10bb12e12c9406e5961a9ff, this is no longer an issue.


KAFKA-6370: KafkaMetricsGroup.toScope should filter out tags null value

KafkaMetricsGroup.toScope should filter out tags with value of null to avoid NullPointerException thrown.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@tedyu Please review this minor patch. Thanks.

Lgtm

@rajinisivaram Please review this minor change.Thanks.

@rajinisivaram Agreed. Closed this pull request due to it's a non-reproducible issue.


Change interpolation algorithm so that it preserves the integrated area

There have been a few cases where Skyline's chromatogram interpolation algorithm produces unexpected results.
Most recently, there was this support request:
https://skyline.ms/announcements/home/support/thread.view?rowId=59209
Here is what the raw chromatogram looks like:

and here is what the current implementation of the interpolation algorithm looks like:

and here is what the interpolated chromatogram looks like with this pull request:

The current interpolation algorithm sometimes produces unexpected results if the interpolated time values happen to fall in strange places in between the existing points.
The new algorithm is guaranteed to preserve the integrated area over all ranges where the y-values of the end points are unchanged.

Have you tested the new algorithm for performance? My memory of a prior change to the algorithm to preserve areas was that it was considerably slower.

I made a new pull request in my own fork: https://github.com/nickshulman/pwiz/pull/2


Metrics are not updated during graceful shutdown

Do you want to request a feature or report a bug?

Bug

What did you do?
Gracefully shutdown Traefik, query prometheus stats, see that traefik_entrypoint_open_connections and traefik_service_open_connections metrics do not decrease.

What did you expect to see?
I expect to see traefik_entrypoint_open_connections and traefik_service_open_connections metrics being updated event if Traefik is shutting down.
What did you see instead?
I see no update in traefik_entrypoint_open_connections and traefik_service_open_connections metrics as queries never end.
Output of traefik version: (What version of Traefik are you using?)

2.1.3

What is your environment & configuration (arguments, toml, provider, platform, ...)?
[metrics]
  [metrics.prometheus]
[ping]
  entryPoint = "ping"
[api]
  dashboard = true

[entryPoints]
  [entryPoints.ping]
    address = ":8081"
    [entryPoints.ping.transport]
      [entryPoints.ping.transport.lifeCycle]
        requestAcceptGraceTimeout = "60s"
        graceTimeOut = "2h"

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)
traefik_entrypoint_open_connections{entrypoint="https",method="OPTIONS",protocol="http"} 14

But there is no more open connections so it should not be 14 but 0 instead.

Hi @L3o-pold
Thanks for your contribution and your interest for Traefik !
I tried to reproduce the bug you described, but had the metrics updated, and i think it's a configuration issue.
For prometheus, traefik exposes the metrics endpoint (/metrics) that serves the metrics.
In your configuration, the metrics endpoint cannot respond after the requestAcceptGraceTimeout period is exceeded, because you've set it on your "ping" entrypoint.
To have your metrics updated (the metrics endpoint still responding), you need to configure it on a specific entrypoint with a requestAcceptGraceTimeout that exceeds the overall durations (requestAcceptGraceTimeout + graceTimeout) sets on all other entrypoints. (In your configuration 2h61sor so).
Is that solving your issue ?

Hi @rtribotte
Sorry I didn't pass the good configuration. Indead, my entrypoint have the right requestAcceptGraceTimeout I think.
[ping]
    [api]
      dashboard = true
    [metrics]
      [metrics.prometheus]
    [providers]
      [providers.file]
        directory = "/rules"

    [entryPoints]
      [entryPoints.traefik]
        address = ":8080"
        [entryPoints.traefik.transport]
          [entryPoints.traefik.transport.lifeCycle]
            requestAcceptGraceTimeout = "2h"
            graceTimeOut = "2h"

This configuration should be ok for fetching metrics during 2 hours.
In prometheus, I see that it can still fetch metrics, but the values remaining the same and are not updated.

Ok, if the metrics are configured on treafik entrypoint in your configuration, then, it must accept the requests during 2 hours after a graceful shutdown, and while others entrypoints are in the graceTimeout if the first config you submitted is still valid.
But if you can provide a full configuration, it will be easier to understand what's going wrong.
I'll share with you a configuration that works and makes me believe that it is working as expected.
I used this docker compose file:
version: '3.5'

services:
  traefik:
    image: traefik:2.1.3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.web.transport.lifecycle.requestAcceptGraceTimeout=1s
      - --entrypoints.web.transport.lifecycle.graceTimeOut=10s
      - --entrypoints.metrics.address=:8282
      - --entrypoints.metrics.transport.lifecycle.requestAcceptGraceTimeout=30s
      - --entrypoints.metrics.transport.lifecycle.graceTimeOut=5s
      - --log.level=DEBUG
      - --providers.docker
      - --providers.docker.exposedByDefault=false
      - --metrics=true
      - --metrics.prometheus=true
      - --metrics.prometheus.entryPoint=metrics
    stop_grace_period: 35s
    ports:
      - '80:80'
      - '8282:8282'

  api1:
    image: containous/whoami
    labels:
      traefik.enable: 'true'
      traefik.http.routers.router0.entrypoints: web
      traefik.http.routers.router0.rule: Host(`whoami`)
    ports:
      - '8001:80'

Observing the metrics endpoint : watch 'curl --raw localhost:8282/metrics -s | grep "^[^#].*open_connections"'
Using the following scenario:
docker-compose up -d
--> connection on metrics entrypoint is showing
curl http://whoami/\?wait\=10s
--> connection on web entrypoint and on service
docker-compose stop traefik
--> the connection is still up, and when the connection finally end, i can see the metrics updated and showing zero opened connections until the stop grace period expires.
WDYT ?

@rtribotte by default prometheus metrics will be exposed on the traefik entrypoint.
A more detailed configuration file:
[accessLog]
    [ping]
    [api]
      dashboard = true
    [metrics]
      [metrics.prometheus]
    [providers]
      [providers.file]
        directory = "/rules"

    [entryPoints]
      [entryPoints.traefik]
        address = ":8080"
        [entryPoints.traefik.transport]
          [entryPoints.traefik.transport.lifeCycle]
            requestAcceptGraceTimeout = "2h"
            graceTimeOut = "2h"

      [entryPoints.http]
        address = ":80"
        [entryPoints.http.forwardedHeaders]
          insecure = true
        [entryPoints.http.transport]
          [entryPoints.http.transport.lifeCycle]
            requestAcceptGraceTimeout = "60s"
            graceTimeOut = "2h"

[http.routers.varnish]
        rule = "Host(`****`)"
        service = "varnish"
        entryPoints = ["http"]

I'll try to reproduce on a docker-compose environment.

Hi @L3o-pold,
I share with you the configuration i used to test it outside docker-compose, and with the metrics on traefik entrypoint.
Launch a whoami service : docker run -d -P --name wai -p 8001:80 containous/whoami
Run ./traefik with this config :
#static conf
[entryPoints]
  [entryPoints.traefik]
    address = ":8080"
    [entryPoints.traefik.transport]
      [entryPoints.traefik.transport.lifeCycle]
        requestAcceptGraceTimeout = "30s"
        graceTimeOut = "5s"
  [entryPoints.web]
    address = ":80"
    [entryPoints.web.transport]
      [entryPoints.web.transport.lifeCycle]
        requestAcceptGraceTimeout = "1s"
        graceTimeOut = "10s"

[metrics]
  [metrics.prometheus]

[log]
  level = "DEBUG"

[providers]
  [providers.file]
    filename = "dynamic.yml"

#dynamic conf
http:
  routers:
    rt:
      entryPoints:
        - "web"
      rule: "Host(`whoami`)"
      service: wai

  services:
    wai:
      loadBalancer:
        servers:
          - url: "http://<IP_ADDRESS>:8001"

I cannot reproduce your issue, traefik metrics are served and updated accordingly to the configuration.

@rtribotte cannot reproduce the issue with docker-compose either. I'll try to reproduce with Kubernetes.

Hi! I'm Træfiker :robot: the bot in charge of tidying up the issues.
I have to close this one because of its lack of activity :disappointed:
Feel free to re-open it or join our Community Forum.


Required Fields in additional requirements not easily identifiable

Does not support smooth data entry

@mccullochk @calebissharp "This is on my think it's a quick win -- pull from it if you're looking for something" list.

@LisaMoore1, I assume this is one of the ones you were referring to?

@mccullochk Yes. I'll look for the list as soon as I can.

Required Fields:

[ ] Category
[ ] Name
[ ] Description

Addressed in #390

@mccullochk Same sizing request for this one then I'll close it.


Unauthenticated access to gs://kfserving-samples has been blocked

/kind bug
It looks like unauthenticated access to the kfserving-samples GCS bucket has been removed.
While trying to run KServe examples I'm now receiving 401 errors from the storage URI.
What steps did you take and what happened:

Try to create an InferenceService from a notebook using the kfserving SDK, using Kubeflow v1.5.0
Pod fails to initialize
Check pod logs

kubectl logs sklearn-iris-predictor-default-00001-deployment-985cb595f-gmx9c storage-initializer
[I 220606 19:58:37 initializer-entrypoint:13] Initializing, args: src_uri [gs://kfserving-samples/models/sklearn/iris] dest_path[ [/mnt/models]
[I 220606 19:58:37 storage:50] Copying contents of gs://kfserving-samples/models/sklearn/iris to local
[W 220606 19:58:37 _default:285] Authentication failed using Compute Engine authentication due to unavailable metadata server.
[I 220606 19:58:37 requests:499] Refreshing credentials due to a 401 response. Attempt 1/2.
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in download
    Storage._download_gcs(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 142, in _download_gcs
    for blob in blobs:
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 208, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 244, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 373, in _next_page
    response = self._get_next_page_response()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 433, in _get_next_page_response
    method=self._HTTP_METHOD, path=self.path, query_params=params
  File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 78, in api_request
    return call()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 475, in api_request
    timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 333, in _make_request
    method, url, headers, data, target_object, timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 371, in _do_request
    url=url, method=method, headers=headers, data=data, timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/google/auth/transport/requests.py", line 511, in request
    self.credentials.refresh(auth_request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/credentials.py", line 170, in refresh
    raise ValueError("Anonymous credentials cannot be refreshed.")
ValueError: Anonymous credentials cannot be refreshed.

Similarly, I'm unable to see any of the examples from a browser.
What did you expect to happen:
Example models should allow unauthenticated access so that they can be used.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

Istio Version: v1.11.4
Knative Version: v0.22.1
KFServing Version: v0.6.1
Kubeflow version: v1.5.0
Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
Minikube version: n/a
Kubernetes version: (use kubectl version): v1.19.16-eks-25803e
OS (e.g. from /etc/os-release): n/a

kfserving-samples gcs bucket is deprecated, please use kfserving-examples instead and we have updated all docs to reflect that.

Thanks for the correction here - hadn't seen the updated docs.
I'll close this accordingly.


Icon/Logo Proposal

If the project needs logo, I am willing to give my best design for this. Thanks PO !

How about this sir ?
I am open for any suggestions

The concept of the design :

Shield is the main common equipment in blocking
The helmet will serve as the guard of the app

Can you show the Android logo, such as a robot, because the design of OkSocket is a Socket communication framework used by Android developers.

i will add it sir,

Thank you very much.I am very glad to receive your Logo support.


Fix Telemetry Service SQL Server pipeline flaky test

What this PR does / why we need it:
fix Telemetry Service SQL Server pipeline flaky test
Will this PR make the community happier?
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
How is this PR tested

[ ] unit test
[ ] e2e test
[x] other (please specify)

Special notes for your reviewer:
Release note:

This version has successfully run five times. It should be OK..


problem with flashing wr3e

I have curtains motor switch with wr3e and trying to flash it.
When I pull tx2 pin on ground when starting, I see diode on the board doesn't light. I think I switch chip to download board. But I can't connect to tx0 rx0 to flash firmware. I'm sure about hardware wiring. Maybe does it have something special to open port? I tried to pull tx2 permanently and pull about 1 sec at start.
In normal mode I see in logs on rx2 tx2 strings but in incorrect codepage.
Do I need to unsolder chip before flashing it?

Hi
What does it mean you "can't connect to tx0 rx0"?
UART2 is used for flashing, read here: https://docs.libretuya.ml/docs/platform/realtek-ambz/flashing/
You don't need UART0 at all, it's not used for flashing, nor for log outputs.
Pull TX2 pin to GND, meanwhile power up the board. The chip then enters "download mode" and waits for flashing commands. You have to disconnect TX2 from GND before the actual flashing process, but without powering the board off.
You can check if it is in download mode by opening up a terminal (PuTTY, or other) and setting any baud rate. It should print something (not readable) few times a second.

Thanks!
It was very simple. I read that manual many times:) I don't know why I wanted to flash with tx0 rx0. I thought to see here the same logic as bk7231t
ps: I flashed over the air esphome with web, but it doesn't boot. I will check later it. If I don't mistake somewhere again, I will write you
Big thanks again!:)

I noticed unstable ota update from console (esphome upload). very frequently firmware doesn't boot after upload ota and reboot.

Try using OTA through web server, it works much better. The changelog mostly applied to BK7231, since I didn't notice such problems on Realtek, but it might fix some stuff on RTL as well.
You can update LT, compile ESPHome and try uploading it in any way.


Fix date sorting

The search allowed for years using 2 or 4 digits, while the sorting considered only 2. The reformatting is in ISO format.

The change to support 4 digit years sounds good I'm not sure this PR should adjust the formatting in parseData as well though. What scenario were you running into where this failed as is?

I have  a table dumped from a MSSQL query where the dates are formatted as 27/07/2018
When I order the data I saw that they weren't ordered correctly, firstly because of the regex, but also I think due to new Date() not understanding the date only in dd/mm/yyyy format. Reformatting as an ISO date (yyyy-mm-dd) allows the correct order. I tried on chrome and internet explorer.

👍


dataPrep()

Need to assign criteria & generate table first:

[ ] Perform unit conversions
[ ] Multiple records per day?
[ ] Activity types (check that native == target)
[ ] Fraction types (check that native == target)

Final pre-assessment data prep steps:

[ ] Dissolved > total fraction
[ ] River/streams samples w/ associated depths (reject depth >= 4 m)
[ ] Flow (potential visual review?)

Potential flags:

[ ] Value based flags/rejections (i.e. CRAZY VALUES!!!!)
[ ] Sample vs. lab dates/times? Holding times?
[ ] Estimated vs. calculated

Any others?

See dataPrep() function outline in data-prep branch.

 Final data cleanup

Jake, do we need to add a case-insensitive grepl function for isolating dissolved and total records (e.g. someone types in DISSOLVED v. dissolved), or does WQP constrain this part of the data entry process?


Update respond-machine-alerts.md

Added another note(Disclaimer) about devices that are behind a full VPN tunnel.

@JohanFreelancer9 Please copy review

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/4797

@mjcaparas Please merge this PR. Thanks.

@mjcaparas Gentle reminder, please merge this PR. Thanks


Multi-location updates

Does polymefire support multi-location updates as explained in this blog https://firebase.googleblog.com/2015/09/introducing-multi-location-updates-and_86.html? I've searched through the code base but could not find any calls to ref.update() method. Is multi-location update something that's in the road map? Thanks.

I'm also struggling to get this to work. Did you manage to find a solution? Thanks.

@pascalblij I ended up doing something like this
  <firebase-document
      id="document"
      app-name="myApp">
  </firebase-document>

  var firebaseDb = this.$.document.db;
  var firebaseDbRef = firebaseDb.ref();
  firebaseDbRef.update(multiLocationData).then(function() {
        //on success
   })
;

You only really need the firebase db reference and this is but one way to get it. Alternatively, you could use their firebase javascript sdk directly. Hope it helps.


cilium-dbg: implement bpf frag list command

Implement cilium-dbg bpf frag list command to list IPV4 datagram fragments.
Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] If your commit description contains a Fixes: <commit-id> tag, then
please add the commit author[s] as reviewer[s] to this issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[ ] Are you a user of Cilium? Please add yourself to the Users doc
[x] Thanks for contributing!

Implement cilium-dbg bpf frag list command to list IPv4 datagram fragments.
Fixes: #34348
Implement `cilium-dbg bpf frag list` command to list IPV4 datagram fragments.

Example output:

/test

Thanks!
I don't have a strong preference between the two options. If I had to pick one, I'd maybe select B), because I find it easier to associate the right port with the right IP (with option A, we have SRC DEST SRC DEST, and the source port ends up next to the destination address so we've got to visually “jump” over a column to find the port corresponding to an address).
I don't remember how we print addresses/ports with other commands though. Consistency is always good to have.

But i am not sure why the address and port are separated in two structs FragmentKey and FragmentValue, what's the consideration behind this?

But i am not sure why the address and port are separated in two structs FragmentKey and FragmentValue, what's the consideration behind this?

For IPv4 fragmentation, we don't reassemble datagrams while packets are in transit. What we need to support fragmentation is a way to retrieve the src/dest port for each fragment, so that we can correctly apply L3/L4 policy rules on the datagram; and then we leave reassembly to the destination endpoint. So we have an eBPF map where we add a new entry each time we see the first fragment for a datagram, and then we use this entry for follow-up fragments to retrieve the port numbers. More details in https://github.com/cilium/cilium/pull/10264#issue-568369766. Does this answer the question?

Updated.

/test

/test

/test

See some errors failing CI failures related to Envoy versions, there's been commits merged in the past couple days that seem related.  @Huweicai Maybe rebase and rerun tests?

OK, I messed up :facepalm:. I tried to rebase your PR for you, since we need someone with permissions to run the /test anyway.
I did git push -f Huweicai main, when I should have used git push -f Huweicai/main:main. So instead of pushing the rebased branch, I pushed Cilium's main onto your branch. As a consequence, the PR closed, because there's no difference between your updated branch and Cilium's main. Oopsies. I tried to update your branch again pushing the correct reference, but I can't update it anymore, I think because the PR is closed. And I can't reopen the PR, because there's no new commit on the branch. Oopsies again.
So I've kept a backup of your branch before rebasing, and I pushed it at https://github.com/qmonnet/cilium/tree/Huweicai/main_backup, which you should also have locally. I also pushed my rebased version at https://github.com/qmonnet/cilium/tree/Huweicai/main_rebased, but we probably don't need it given that main will keep moving. The way to fix my mistake, I believe, would be for your to rebase and push your changes to your own main branch again. After that, we should be able to re-open the PR.
Sincere apologies for the trouble :disappointed:.

OK, I messed up 🤦. I tried to rebase your PR for you, since we need someone with permissions to run the /test anyway.
I did git push -f Huweicai main, when I should have used git push -f Huweicai/main:main. So instead of pushing the rebased branch, I pushed Cilium's main onto your branch. As a consequence, the PR closed, because there's no difference between your updated branch and Cilium's main. Oopsies. I tried to update your branch again pushing the correct reference, but I can't update it anymore, I think because the PR is closed. And I can't reopen the PR, because there's no new commit on the branch. Oopsies again.
So I've kept a backup of your branch before rebasing, and I pushed it at https://github.com/qmonnet/cilium/tree/Huweicai/main_backup, which you should also have locally. I also pushed my rebased version at https://github.com/qmonnet/cilium/tree/Huweicai/main_rebased, but we probably don't need it given that main will keep moving. The way to fix my mistake, I believe, would be for your to rebase and push your changes to your own main branch again. After that, we should be able to re-open the PR.
Sincere apologies for the trouble 😞.

No problem, i have code in my local machine, i'll rebase and push it.

I re-pushed the code based on the latest commit, but it seems that there's no reopen button.
Should I create a new one?
https://github.com/Huweicai/cilium

/test

I could reopen now that you've updated your branch. OK thanks! I won't touch your branch anymore :sweat_smile:


Folder Support

Thanks for building this tool to access macOS services from within Alfred, it's very useful to me. However, it doesn't seem to work if I select a folder instead of a file. For example, I use iTerm2, which provides a system service to open a new iTerm2 window at the given folder. To use this from finder, right click on a folder, then click Services, and then I see a list of services, including one called "New iTerm2 Window Here".
When I find a folder in Alfred, then invoice the "macOS Services" action provided by this workflow, the list of services available to me in Alfred does not include the "New iTerm2 Window Here" service.
I code a bit, but not in Objective-C or Go. I poked around in the files included in this workflow, and I can see in PathsToPasteboard.js that you set the content type of the pasteboard to public.file-url, which I don't think includes folders. In an attempt to make this work myself, I tried editing PathsToPasteboard.js so the content type was public.folder but that still didn't make the embedded alfred-services binary return services that apply to folders (like "New iTerm2 Window Here". I looked at the alfred debugging log, and poked around in main.go, and I'm not smart enough to figure out where the problem is.

PathsToPasteboard.js is only run from the Alfred File Action, where there isn't much information about the files available (Alfred throws most of it away).
Try using the "current selection" (or "clipboard contents") Hotkey instead of the File Action. They should put a lot more information on the clipboard, and a lot more macOS services should be available as a result.


NoData should be an option in the plotty renderer

Currently, the nodata value is set to -9999.
It would be useful for this to be an option in the renderer options.


Painting_the_fence

Hello sir,
I have added the files, in Images folder I have added the screenshots and also added the code.
Pls kindly accept my pr @avinash201199 sir.

@manognyaa
Hey, no need to add add images , you need to add everything on a single file so please remove the images.
I'm closing the pr , please do the requested changes and then make new pull request with issue number, will accept your pr :)


add new blogs

This PR updates the blog with new external blog posts

@xmulligan all good now, let's ship


Feature request: ignore patterns for files or directories

Currently, bump-pydantic considers all files ending in .py when scanning and applying modifications. In a codebase where source files are located at the project root, and a virtual environment is present, bump-pydantic takes a very long time to go through everything in the virtual environment's site-packages in addition to the actual project source files.
Allowing the user to specify patterns to ignore when scanning, using a command-line argument, will solve this problem.
Further, some default patterns may be put in place, for example:

Ignoring .venv as it is a common name for virtual environment directories, or
Ignoring everything in .gitignore if it is present, if not falling back to the previous point

If the user wishes to suppress default ignore patterns, they might do so using --ignore="" or something similar.
These implementation details are up to the maintainers to decide.
If this feature is approved, I would like to work on it.

PR welcome. 👍


Figure out how to handle IAsyncDisposable types in DI

IAsyncDisposable is coming to in C# 8  https://github.com/dotnet/corefx/issues/32640. We need to figure out what happens when services implement IAsyncDisposable. We need to figure out what to prefer if both are on a type and we also need to implement the interface on the ServiceProvider type.

And on IServiceScope. It would be a breaking change but a lot of containers would need to add support for IAsyncDisposables so maybe coordinated breaking change is fine.


Read from remote root file

I try to use root_numpy for reading remote root file
filename = 'root://eospublic.cern.ch//eos/opendata/cms/Run2010B/Mu/AOD/Apr21ReReco-v1/0000/00459D48-EB70-E011-AF09-90E6BA19A252.root'
root_numpy.list_trees(filename)
And I got the error:
ERROR:ROOT.TUnixSystem.DynamicPathName:Netx[.so | .dll | .dylib | .sl | .dl | .a] does not exist in /usr/local/lib/root:.:/usr/local/lib/root::/usr/local/lib/root/cint/cint/stl
Is it possible read it? TChain correctly read the root file.

Thanks for reporting. I get the same error, and this also fails:
>>> from ROOT import TFile
>>> f = TFile.Open(filename)
Error in <TUnixSystem::FindDynamicLibrary>: NetxNG[.so | .dll | .dylib | .sl | .dl | .a] does not exist in ...

Are you sure that TChain is reading your file? I still get this:
>>> c = TChain()
>>> c.Add(filename)
1
>>> c.GetEntries()
Error in <TUnixSystem::FindDynamicLibrary>: NetxNG[.so | .dll | .dylib | .sl | .dl | .a] does not exist ...

For some reason, libNetxNG.so is not being built when compiling ROOT. This looks to be the same issue: https://github.com/Homebrew/homebrew-science/issues/1130
I'd expect this to work fine in root_numpy as long as ROOT has the appropriate libraries available.

Thank you for answer!
Indeed the problem was in libNetxNG.so. I followed your link and now it is ok.
By the way, root_numpy can read, for example, EventAuxiliary.id_.luminosityBlock_ (similar to https://root.cern.ch/phpBB3/viewtopic.php?f=3&t=15868) without any edm::EventAuxiliary class and without CMS virtual machine (http://opendata.cern.ch/VM/CMS, http://opendata.cern.ch/getting-started/CMS). Keep it up!

@tata-antares - as far as reading that, it's just more or less a branch. As long as PyROOT, in general, can read and access the branches -- so can root_numpy. The only problems I've seen are when you have branches of pointers instead which is happening with a lot of the new EDM changes, and that makes things a little more difficult.

The initial problem was to read this branch direct EventAuxiliary.id_.luminosityBlock_.
For example, reading recoCaloJets_ak5CaloJets__RECO.obj.pt_ is simple. TTree contains branch recoCaloJets_ak5CaloJets__RECO., so it is possible:
tree.GetEntry(n)
tree.GetBranch('recoCaloJets_ak5CaloJets__RECO.obj.pt_').GetLeaf(
               'recoCaloJets_ak5CaloJets__RECO.obj.pt_').GetValue()

But for EventAuxiliary it doesn't work. TTree contains EventAuxiliary.
tree.GetBranch('EventAuxiliary').GetLeaf('EventAuxiliary').GetValue()

which return edm::EventAuxiliary structure.
Do you know, how to read EventAuxiliary.id_.luminosityBlock_ using PyROOT?

Well, I'm ATLAS-side. I don't know if I can get access to your file, but it honestly sounds like CMS isn't providing backwards-compatibility when they store these kinds of custom objects in the file (at least not as much as how ATLAS is doing it, which is expected). If you're able to retrieve an edm::EventAuxiliary structure using PyROOT, that implies PyROOT knows the type, otherwise it would've have thrown an error (I think).

@tata-antares great to hear it works now! I'll make this issue as a ROOT issue and will close this.
I'll have a look at that opendata CMS file and will try to read EventAuxiliary.id_.luminosityBlock_

Thanks!
I tried to read root by PyROOT in the same way as you are doing in root_numpy, but can't reproduce it. That is why for me very interesting how to read without root_numpy using standard ROOT (without CMS virtual machine, in which necessary classes exist)

Nice:
>>> arr = root2array("root://eospublic.cern.ch//eos/opendata/cms/Run2010B/MuOnia/AOD/Apr21ReReco-v1/0001/002041B8-FF77-E011-8A94-00215E21EC4A.root", "Events", branches="EventAuxiliary.id_.luminosityBlock_")
>>> arr
array([ 425.,  425.,  425., ...,  434.,  434.,  434.])
>>> len(arr)
12453

root_numpy is able to do this because any branch names that don't match up with actual names of branches in the tree are automatically considered as TTreeFormula expressions. So anything that TTreeFormula understands, root_numpy will convert to a column of doubles.


Add support for debian upgrades

Mostly what we had already for Ubuntu as they both use apt.
Some testing
Before running host-upgrader:
$ uname -a
Linux debian-9 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux

Run host-upgrader:
root@debian-9:/vagrant# docker run --rm --name host-upgrades --privileged   -v $PWD/config:/etc/host-upgrades   -v /run/host-upgrades:/run/host-upgrades   -v /var/run/dbus:/var/run/dbus   -v /run/log/journal:/run/log/journal   kontena/pharos-host-upgrades:dev /usr/local/bin/pharos-host-upgrades -v 9 -reboot
2018/10/02 12:14:04 pharos-host-upgrades version 0.2.0 (Go go1.10.2)
2018/10/02 12:14:04 Load config from --config-path=/etc/host-upgrades
2018/10/02 12:14:04 Copying configs to --host-mount=/run/host-upgrades
2018/10/02 12:14:04 hosts/ubuntu probe mismatch: Debian GNU/Linux 9 (stretch)
2018/10/02 12:14:04 hosts/centos probe mismatch: Debian GNU/Linux 9 (stretch)
2018/10/02 12:14:04 hosts/debian boot time: 2018-10-02 11:54:14 +0000 UTC
2018/10/02 12:14:04 hosts/debian probe success: hosts.Info{OperatingSystem:"Debian", OperatingSystemRelease:"GNU/Linux", Kernel:"Linux", KernelRelease:"4.9.0-7-amd64", BootTime:time.Time{wall:0x0, ext:63674078054, loc:(*time.Location)(0x1813020)}}
2018/10/02 12:14:04 Probed host: Debian GNU/Linux
2018/10/02 12:14:04 hosts/debian: using host path /run/host-upgrades for output files
2018/10/02 12:14:04 hosts/debian: using copied unattended-upgrades.conf at /run/host-upgrades/unattended-upgrades.conf
2018/10/02 12:14:04 hosts/debian: using generated host-upgrades.sh at /run/host-upgrades/host-upgrades.sh
2018/10/02 12:14:04 No --schedule given, will run once
2018/10/02 12:14:04 No --kube configuration
2018/10/02 12:14:04 Using --reboot, will reboot host after upgrades if required
2018/10/02 12:14:04 Skip kube locking
2018/10/02 12:14:04 Running host upgrades...
2018/10/02 12:14:04 hosts/debian upgrade...
2018/10/02 12:14:04 systemd/exec host-upgrades.service: systemd.ExecOptions{Unit:"host-upgrades.service", Cmd:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, Env:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}
2018/10/02 12:14:04 systemd/exec host-upgrades.service: reset
2018/10/02 12:14:04 systemd/exec host-upgrades.service: start []dbus.Property{dbus.Property{Name:"Type", Value:dbus.Variant{sig:dbus.Signature{str:"s"}, value:"oneshot"}}, dbus.Property{Name:"Environment", Value:dbus.Variant{sig:dbus.Signature{str:"as"}, value:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}}, dbus.Property{Name:"ExecStart", Value:dbus.Variant{sig:dbus.Signature{str:"a(sasb)"}, value:[]dbus.execStart{dbus.execStart{Path:"/bin/sh", Args:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, UncleanIsFailure:false}}}}}
2018/10/02 12:14:04 systemd/exec host-upgrades.service: wait...
2018/10/02 12:14:11 systemd/exec host-upgrades.service: done, status=done
2018/10/02 12:14:11 systemd/journal: read journal...
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.473651 +0000 UTC + set -ue
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.473651 +0000 UTC + apt-get update
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.637302 +0000 UTC Ign:1 http://deb.debian.org/debian stretch InRelease
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.661622 +0000 UTC Hit:2 http://deb.debian.org/debian stretch Release
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.790722 +0000 UTC Hit:3 http://security.debian.org/debian-security stretch/updates InRelease
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:04.9678 +0000 UTC Hit:5 https://download.docker.com/linux/debian stretch InRelease
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:05.341929 +0000 UTC Hit:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:06.098891 +0000 UTC Reading package lists...
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:06.126974 +0000 UTC + unattended-upgrade -v
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.512769 +0000 UTC debug: running /usr/lib/os-probes/50mounted-tests on /dev/sda2
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.522965 +0000 UTC debug: /dev/sda2 type not recognised; skipping
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.524933 +0000 UTC debug: os detected by /usr/lib/os-probes/50mounted-tests
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.530142 +0000 UTC debug: running /usr/lib/os-probes/50mounted-tests on /dev/sda5
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.536049 +0000 UTC debug: /dev/sda5 is a swap partition; skipping
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.537881 +0000 UTC debug: os detected by /usr/lib/os-probes/50mounted-tests
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.858785 +0000 UTC + [ -e /run/reboot-required ]
2018/10/02 12:14:11 systemd/journal: 2018-10-02 12:14:11.858785 +0000 UTC + cp -a /run/reboot-required /run/host-upgrades/reboot-required
2018/10/02 12:14:11 systemd/exec host-upgrades.service: done
2018/10/02 12:14:11 Skip updating kube node condition
2018/10/02 12:14:11 Reboot required, rebooting without draining kube node...
2018/10/02 12:14:11 hosts/debian reboot...
Connection to <IP_ADDRESS> closed by remote host.
Connection to <IP_ADDRESS> closed.

After reboot:
root@debian-9:~# uname -a
Linux debian-9 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) x86_64 GNU/Linux

Same instructions as with Ubuntu :) Added to README


the height of the container is not fixed.

There is a live demo: https://codepen.io/hronro/details/bWdjqe

You're using the autoHeight parameter in the example you've linked, so the height grows.
You're providing a callback for function height- so an estimated height is used until a row is actually rendered. The default estimated height is much smaller than the example you've linked to so the initial height estimated is much smaller than what it gets refined to as you scroll down. You can use the estimatedRowSize prop to specify a more accurate default.


Incorrect language

can you fix this, some packages with english not correct luci-app-nextdns luci-app-dnscrypt-proxy luci-app-attendedsysupgrade and many others you can find yourself, thank you

Upstream problem. via: https://github.com/openwrt/luci/commit/62ee3fbfbf91e41d1260ea7023145fbdd4eb0d86
Please report it in the upsteam repo. https://github.com/openwrt/luci/issues


Multiple Audience Support

Hello,
It seems there is a bug / issue / lack of feature with the audience validation logic.  If you pass a list of audiences, it must be in all of them, not just one.  I believe it's not "standard" to allow multiple audiences as is done with the Issuer claim.
Please see the logic on line 415
https://github.com/auth0/java-jwt/blob/dd6bc25d9a7c2857bae29757a9198b798ac5734f/lib/src/main/java/com/auth0/jwt/JWTVerifier.java#L415
vs the logic on line 421
https://github.com/auth0/java-jwt/blob/dd6bc25d9a7c2857bae29757a9198b798ac5734f/lib/src/main/java/com/auth0/jwt/JWTVerifier.java#L421
Would it be possible to give us the option to validate all or at least one?  Its unclear to me why we have a list for aud, and a single value for issuer, but thats neither here nor there, can we simply check if one exists?  Not sure if this would need a broader vetting, or perhaps a new api to check for at least one vs all?
Could we simply tweak like 415 with Collections.disjoint(audience, value); instead of !audience.containsAll(value);

When validating a JWT, you can specify the audience(s), and the expectation is that all of the expected audiences are contained in the JWT's aud claim. The JWT may contain additional audiences than those specified to validate, but it must contain all the audiences specified.
For example, consider a JWT with an aud claim value of [https://domain.auth0.com/userinfo, https://api-id.com]. When validating the JWT, you can pass [https://domain.auth0.com/userinfo] to ensure the token is valid for the userinfo API, or [https://api-id.com] to ensure the token is valid for some custom API, or pass both to ensure the token is valid to be used for both APIs.

When validating a JWT, you can specify the audience(s), and the expectation is that all of the expected audiences are contained in the JWT's aud claim. The JWT may contain additional audiences than those specified to validate, but it must contain all the audiences specified.
For example, consider a JWT with an aud claim value of [https://domain.auth0.com/userinfo, https://api-id.com]. When validating the JWT, you can pass [https://domain.auth0.com/userinfo] to ensure the token is valid for the userinfo API, or [https://api-id.com] to ensure the token is valid for some custom API, or pass both to ensure the token is valid to be used for both APIs.

@jimmyjames  Agreed that is how it works today, but in our use case we do not want to require that they all are there, rather we want to require that at least one of the list is there.  We did work around this by doing a multiple pass validation logic but it seems a little ugly.

@jimmyjames  Agreed that is how it works today, but in our use case we do not want to require that they all are there, rather we want to require that at least one of the list is there.  We did work around this by doing a multiple pass validation logic but it seems a little ugly.

@PvanHengel when you say you do a multiple pass validation as a workaround, do you mean you are running multiple verifications until one passes with one of the expected audiences?
It might be cleaner to just skip audience validation in this scenario, and instead validate your expectations from the decoded JWT claim.
Regarding updating the SDK to support validating at least one of the expected audience is present, instead of all of them, I think it's something we should do. I don't know if it's something we can do really soon, as changing the current behavior would be a breaking change, so would need to be done in the next major release. Perhaps there's a non-breaking way to enable this functionality, and if so we can do that.

@PvanHengel when you say you do a multiple pass validation as a workaround, do you mean you are running multiple verifications until one passes with one of the expected audiences?
It might be cleaner to just skip audience validation in this scenario, and instead validate your expectations from the decoded JWT claim.
Regarding updating the SDK to support validating at least one of the expected audience is present, instead of all of them, I think it's something we should do. I don't know if it's something we can do really soon, as changing the current behavior would be a breaking change, so would need to be done in the next major release. Perhaps there's a non-breaking way to enable this functionality, and if so we can do that.

@PvanHengel does the changes in #472 look like they will meet your requirements?


php is not necessary

Just wanted to say that I agree with this PR.
The php prefix is actually causing problems for me (on OS X), because php refers to /usr/bin/php instead of to /usr/bin/env php (i.e., /usr/local/bin/php).
In any case, the leading php being removed leaves this up to the shebang line in php-cs-fixer, which is what should occur IMO :-)

Looks good to me. Thanks!

Published to v0.23.2


Allow classes derived from LeafNode to optionally skip reading and writing the ValueMask

No ABI or file format changes.

Closing this pull request - it's more than two years old and I believe I've found a better way of getting around this issue when looking into delay-loading.


App is not working when pairing with a Ledger USB device.

Stuck at the "looking for devices" page, Ledger is unlocked.
Pixel 6 / LineageOS 20
If you need further Infos please let me know.

I am having similar issues with my HNT wallet.  Set up with a ledger but ledger refuses to recognize the wallet.  Says "Looking for devices" when ledger is unlocked.  Not sure how to proceed.

Hi @Vincent432
visit the official Ledger page to initiate a chat with our support team on the live chat to report your errors for review and for further assistance via ledger support

You should use a different wallet until #262 is done


[SPARK-28198][PYTHON][FOLLOW-UP] Add a separate evaluation type for mapPartitionsInPandas

What changes were proposed in this pull request?
This PR proposes to add a separate evaluation type for mapPartitionsInPandas case.
Had an offline discussion with @rxin, @mengxr and @cloud-fan
The reason is basically:

SCALAR_ITER doesn't make sense with mapPartitionsInPandas.
It cannot share the same Pandas UDF, for instance, at select and mapPartitionsInPandas unlike GROUPED_AGG because iterator's return type is different.

Renaming SCALAR_ITER as MAP_ITER is abandoned due to 2. reason.
For XXX_ITER, it might have to have a different interface in the future if we happen to add other versions of them. But this is an orthogonal topic with mapPartitionsInPandas.
How was this patch tested?
Existing tests should cover.

@BryanCutler, @icexelloss, @viirya too

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12351/
Test PASSed.

Test build #107182 has started for PR 25044 at commit 9f2181e.

LGTM, it's much cleaner!

+1 for this change.

Test build #107182 has finished for PR 25044 at commit 9f2181e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107182/
Test FAILed.

Retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12372/
Test PASSed.

Test build #107203 has started for PR 25044 at commit 9f2181e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107203/
Test FAILed.

I'd change the name of the function from mapPartitionsInPandas to mapInPandas. This removes the "partition" physical concept, which usually don't exist in the DataFrame API. It'd open up more flexibility in the future for optimizations, e.g. we might run multiple threads for a single partition.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12375/
Test PASSed.

Test build #107207 has started for PR 25044 at commit 3dd374e.

Yup, sounds good. Seems like we can also potentially let the function accept DataFrame too instead of iterator of DataFrames in the future and mapInPandas` can avoid to add multiple APIs for every single case.

For documentation, I am going to update later at https://github.com/apache/spark/pull/25025 together.

Test build #107209 has started for PR 25044 at commit fd70990.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12378/
Test PASSed.

Test build #107207 has finished for PR 25044 at commit 3dd374e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107207/
Test PASSed.

Test build #107209 has finished for PR 25044 at commit fd70990.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107209/
Test PASSed.

Merged to master.
Thanks all for reviewing this.


KAFKA-10199: Separate state updater from old restore

Separates the code path for the new state updater from
the code path of the old restoration.
Ensures that with the state updater tasks are processed
before all tasks are running.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Call for review: @wcarlson5

LGTM! Thanks @cadonna . No more major comments.

Build failures are unrelated:
Build / JDK 17 and Scala 2.13 / testLegacyAlterConfigs() – kafka.server.KRaftClusterTest
Build / JDK 11 and Scala 2.13 / shouldNotMakeStoreAvailableUntilAllStoresAvailable – org.apache.kafka.streams.integration.QueryableStateIntegrationTest


Add support for Albumentations and Imgaug Transforms

Instead of reinventing the wheel, we should allow transforms from Albumentations and Imgaug libraries to be callable within an augraphy pipeline.  In general, our pipeline process should be familiar to a user of Albumentations, so in theory this should be an easy adaptation to support their transforms in an Augraphy pipeline.
In general, we should provide basic transforms which may be defined in these libraries which are needed for the common document-oriented augmentations.  So, we'll have some minimal overlap; however, we should not require these other libraries for the default and most common augmentation needs.
Note that Albumentations currently seems to be well maintained while Imgaug has not been updated in a little over a year as of this moment.  So, we should prioritize Albumentations over Imgaug.  It appears that Imgaug has additional blur transforms, which can be useful to vary since deblurring of basic gaussian blur is fairly easily learnable using deep learning methods; but we should not invest too much effort in supporting a library that is not being maintained.

From a quick glance at Albumentations , those should be related to digital side of the augmentation and should be more low level if we compare it to the existing augmentations. If we want to support them in the current work, i think the best way is to enable the usage of those libraries in each phase augmentation sequence pipeline, for example:
ink_phase = AugmentationSequence([
MedianBlur(),
RandomFog()])
But there's some issues need to be resolved since they are from difference base classes, or is there a better way to do this?

But there's some issues need to be resolved since they are from different base classes, or is there a better way to do this?

We should be able to check the type info on the transforms supplied to the pipeline and handle that based on compliance with the interfaces that those methods expose, all without actually depending on those libraries within our requirements.txt.

I'm cleaning things up for a PR now, but I'll test this afterwards. I expect it already works as intended, since all of our augmentations treat image data as numpy.array objects, and so does every other package I've seen in this space.
I'll drop an example here in a little bit.

This was very easy. We can't directly use Albumentations or Imgaug augmentations within our pipeline, but we can expose simple wrapper classes which convert them to our own, and these can be added directly to our pipeline.
For example, using Albumentations:
`class AlbumentationToAugraphyAugmentation(Augmentation):
"""A wrapper for Albumentations to build an Augraphy Augmentation.""
"
def __init__(self, layer, albumentationAugmentation, p=0.5):
    self.layer = layer
    self.augmentation = albumentationAugmentation
    super()__init__(p=p)

def __call__(self, data):
    image = data[self.layer][-1].result
    result = self.augmentation(image = image)["image"]
    data[self.layer].append(AugmentationResult(self, result)   `

I think it'd be great to add a module for wrappers like this; future feature requests to support interoperability with other augmentation libraries would be as simple as adding a new wrapper class, and would not require us to formally depend on these other libraries.

I did it. The latest commit here adds a module providing a wrapper class ForeignAugmentation which can be used in our pipeline. This does not require any additional dependencies.
You use this just like any other Augraphy augmentation, but pass in the constructor for the Albumentations/imgaug augmentation like so (here's a piece of a modified default pipeline):
import albumentations as A
from imgaug import augmenters as iaa
...

    post_phase = AugmentationSequence([DirtyRollersAugmentation(p=0.0),
                                       OneOf([
                                           LightingGradientAugmentation(),
                                           BrightnessAugmentation('post')
                                       ]),
                                       SubtleNoiseAugmentation(),
                                       ForeignAugmentation("post", A.RandomGamma(), p=0.5),
                                       ForeignAugmentation("post", iaa.Fliplr(p=0.5),  p=1),
                                       JpegAugmentation()
                                       ])

This worked out great!
If we ever need to have any explicit optional reference to an external library, I came across this code from Albumentations that seems to handle that cleverly:
try:
    import imgaug

    imgaug_available = True
except ImportError:
    imgaug_available = False

skipif_imgaug = pytest.mark.skipif(imgaug_available, reason="The test was skipped because imgaug is installed")

Then with some tests (just as an example of the concept), this was used to control whether external references were accessed:
from tests.conftest import skipif_no_imgaug, skipif_imgaug

@skipif_no_imgaug
def test_iaaperpective_augmentation_imported_when_imgaug_is_installed():
    from albumentations.imgaug.transforms import IAAPerspective

    t = IAAPerspective()
    assert isinstance(t, A.DualTransform)


Support using constructor property types only, instead of @Inject

I wrote a wrapper using your connect, inside my own store.js file...
...
import Connector from 'ng2-redux/lib/components/connector'

export function Store(reduxStore) {
  const __connector = new Connector(reduxStore);
  return {
    ...reduxStore,
    connect: __connector.connect
  };
}
...
export function createStore() {
  ...
  return Store(createStoreWithMiddleware(reducer, state));
}

Then in my index.js ...
import {Store, createStore} from './store';
...
bootstrap(App, [
  provide(Store, { useValue: createStore() }),
  ...
]);

This way, in my components, I only need constructor(store:Store) to get the store reference, which works with babel/es6 as well as the preferred use for ng2.
Yes, it's explicit and one step removed from the provider, however, it allows for easier integration at the component level... not sure if the provider itself could be changed to support a type reference, but it would be nicer than poluting my constructors with @Inject

Maybe if you separated Connector, and its' dependencies into an ng2-redux-connector module, so it can be used more easily?  Either with the provider method this module uses, or the more explicit method I used.

+1
Michael J. Ryan<EMAIL_ADDRESS>2016年3月8日 週二，下午12:00寫道：

I wrote a wrapper using your connect, inside by store.js file...
...
import Connector from 'ng2-redux/lib/components/connector'
export function Store(reduxStore) {
const __connector = new Connector(reduxStore);
return {
...reduxStore,
connect: __connector.connect
};
}
...
export function createStore() {
...
const store = Store(createStoreWithMiddleware(reducer, state));
...
return Store(store);
}
Then in my index.js ...
import {Store, createStore} from './store';
...
bootstrap(App, [
provide(Store, { useValue: createStore() }),
...
]);
This way, in my components, I only need constructor(store:Store) to get
the store reference, which works with babel/es6 as well as the preferred
use for ng2.
Yes, it's explicit and one step removed from the provider, however, it
allows for easier integration at the component level... not sure if the
provider itself could be changed to support a type reference, but it would
be nicer than poluting my constructors with @Inject
https://github.com/Inject, which iirc, doesn't work via babel
extensions outside typescript.
—
Reply to this email directly or view it on GitHub
https://github.com/wbuchwalter/ng2-redux/issues/26.

--
Thanks,
Stanislaus

@tracker1 We are working on some changes for the next major release, and we will include something like this. Thanks for the input/code - much appreciated.

can also use an OpaqueToken

@tracker1 getting ready to have a new release available soon that will support property injection instead of needing @Inject, sorry for the delay on getting this out.

@e-schultz is this fixed by 2.2.3?

As of release 2.2.3 you can now do:
class FooComponent {
  constructor(private ngRedux: NgRedux) {}
}

Cool... no longer working on the app in question, but nice to see the feature just the same.  Really do like working with Redux over the "angular way" of services, etc...


exec user process caused "operation not permitted" due to OPTIONS='--selinux-enabled'

Did a clean install of Fn Project based upon the documentation provided by the Fn project. Using the Oracle provided ol73 vagrant image.
Invoking a function provides the following error:
container failed to initialize, please ensure you are using the latest fdk / format and check the logs.
The fn logs show the following:
[time="2019-05-04T12:20:29Z" level=info msg="hot function terminated" app_id=01DA1A2VPENG8G00GZJ0000001 container_id=01DA1CKPXRNG8G00GZJ000000Q cpus= error="container exit code 1" fn_id=01DA1A5YC6NG8G00GZJ0000002 idle_timeout=30 image="fndemouser/app1:0.0.3" memory=256
time="2019-05-04T12:20:29Z" level=error msg="api error" action="server.handleHTTPTriggerCall)-fm" app_name=app1 code=502 error="container failed to initialize, please ensure you are using the latest fdk / format and check the logs" trigger_source=/app1]

Using a remote syslog server "papertrailapp.com" I was able to capture the following error:
May 04 14:44:02 fnproject app_id=01DA1A2VPENG8G00GZJ0000001,fn_id=01DA1A5YC6NG8G00GZJ0000002: standard_init_linux.go:190: exec user process caused "operation not permitted"
After searching I found the following workaround, after applying it the functions are working as expected.

review /etc/sysconfig/docker
remove --selinux-enabled from OPTIONS
restart Docker and FN
test function (should work now)

Even though it resolves the issue at hand , it feels like a workaround at the best. Either, the documentation should mention this as a required step (to check) or we should find a way to prevent this situation from happening.
background version info:
[root@fnproject app1]# fn version
Client version is latest version: 0.5.76
Server version:  0.3.702

[root@fnproject app1]# docker --version
Docker version 18.09.1-ol, build c3ab8a8

[root@fnproject app1]# uname -a
Linux fnproject 4.1.12-61.1.28.el7uek.x86_64 #2 SMP Thu Feb 23 19:55:12 PST 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@fnproject app1]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      29

Got same problem while following python tutorial.
Official output from fn invoke is:
fn invoke pythonapp pythonfn
Error invoking function. status: 502 message: container failed to initialize, please ensure you are using the latest fdk / format and check the logs
however with papertrail syslog i get:
Jun 22 22:10:40 fuffa app_id=01DE0BH5PNNG8G00GZJ0000001,fn_id=01DE0BJZTNNG8G00GZJ0000002: standard_init_linux.go:190: exec user process caused "operation not permitted"
this using docker from ubunt 18.04 with snap.
if i use docker-ce with normal installation no problems at all

@samueletonon ,
did you try the workaround I proposed in the original post? Even though it is a workaround it can solve the issue and can help you move forward.

review /etc/sysconfig/docker
remove --selinux-enabled from OPTIONS
restart Docker and FN

hello,
no i can't use your workaround as snap install everything in a different way.
for example in the machine i am using selinux has been disabled, thus if i install normal docker through apt instead of snap the issue is not present
Sent from my Android device with K-9 Mail. Please excuse my brevity.

@samueletonon interesting..... that would / could mean that the way the installer is setting some config differs per package. No real surprise, however, it would be good to figure out what exactly. I did all the installs with yum (no apt/snap).

I just removed the snap docker and reinstall with apt get and that resolved the issue.


Inline the exercise README insert

Now that we're operating with a track-wide template for generating exercise
READMEs we no longer need a separate markdown document to use for hints
and instructions that apply to all the exercises in a track. These instructions
can be defined directly in the template.
The first commit regenerates the exercise READMEs to pick up tweaks to problem descriptions, which have been updated with various formatting changes, whitespace changes, and clarifications to the description.
That gives us a clean diff when inlining the template and regenerating.
Ref: https://github.com/exercism/meta/issues/94

We'd like to keep the top in the phone number README

We'd like to keep the top in the phone number README

Ok, in that case that section needs to live in a separate hints.md file. I'll make it so.

OK, I've rebased this onto master, and made the phone-number change as a separate commit. This should be good to go.


⚠ Bofrost.de has degraded performance

In 859f982, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1495 ms

Resolved: Bofrost.de performance has improved in d366cae.


SQI randomization code changes

@vrathee @renujhamtani Please review

Merged in
https://github.com/redhataccess/redhat_access_angular_ui_common/pull/26

Not merging this as Vikas merged Keyur's changes.


Icon Map copy icon not working

Prerequisites

[X] I have searched for duplicate or closed issues
[X] I have read the contributing guidelines

Describe the issue
Copy icon from the IconMap when used in a data table row action renders as a square

Reduced test cases
No response
What operating system(s) are you seeing the problem on?
Windows
What browser(s) are you seeing the problem on?
Chrome
What is the issue regarding ?
@trimble-oss/modus-web-components
What version of npm package are you using ?
0.1.52
Priority
Medium
What product/project are you using Modus Components for ?
Work Center
What is your team/division name ?
Trimble Viewpoint
Are you willing to contribute ?
None
Are you using Modus Web Components in production ?
No response

The copy icon is appearing testing it in the latest version (0.5.1)

The copy icon is appearing testing it in the latest version (0.5.1)


[ENH] option to keep column names in Lag

This PR adds a parameter keep_column_names (default=False) to the Lag transformer, which enables Lag to return the same column names in transform as in the input. This is for avoiding to produce column names lag_x__varname1, lag_x__varname2, etc, if only one lag x is input.
The default is False and not True according to the reasoning in the recent https://github.com/alan-turing-institute/sktime/pull/3261 which made the defaults consistent.
The use case for not having too long variable names is in cases where we lag twice, or the lagging happens internally in another estimator, such as the reduction prototypes: https://github.com/alan-turing-institute/sktime/pull/3333
FYI, what do you think, @KishManani?

Looks good to me!


deleteByQuery is missing

Please make sure that in the case the distinct is enabled on the index (with distinct > 1), we  distinct=false at query time, so the number of hits won't be > 1000. Otherwise, it will returns the following error:
{
  "message": "invalid distinct value, hitsPerPage * distinct should be <= 1000",
  "status": 400
}

The Objective-C API client is now deprecated in favor of the recently released Swift client v3. The Objective-C client will not receive any new features. It will, however, continue to receive bug fixes.


Set up a dedicated development device in the IoT Lab

Should we set up a device in the IoT lab that is dedicated specifically to development? This device would serve as a place for team members to work on enhancements to the Netrics code as well as a first stage to test out fixes for bugs. It would be accessible to our team and collaborators only.

@ggmartins does the IoT lab device already somewhat fulfill this purpose?

I think so


MVT keep Z

Hi,
I have some 3D geometries in PostGIS. Is it possible to return tiles with Z coordinate? Now all the tiles are only lat/lon.
Thanks!

See https://github.com/mapbox/vector-tile-spec/issues/111 - MVT doesn't support 3d at the moment to my knowledge


Company internal URL

Provide internal URL e.g. link to wiki page with company details

Fixed in 1df1bf2c07fc85e02a88dd6667ee66c175268705


Add check for SBOM output

This is complicated as there are many ways to produce SBOMs, I need to think through how we can check for this but for now wanted to keep this feature request in the backlog

example of CNCF project that publishes an SBOM
https://fluxcd.io/blog/2022/02/security-the-value-of-sboms/

Hi @caniszczyk
We could start by checking:

If the latest release on Github contains an asset which name contains sbom (or other relevant patterns). This would cover cases like Flux, as for each version they publish the corresponding SBOM asset (i.e. flux_0.28.2_sbom.spdx.json).

If the README file contains a SBOM section that explains where they are published to, format used, etc.

That sounds like a good start to me imho
On Fri, Mar 25, 2022 at 12:32 PM Sergio C. Arteaga @.***>
wrote:

Hi @caniszczyk https://github.com/caniszczyk
We could start by checking:

If the latest release on Github contains an asset which name contains
sbom (or other relevant patterns). This would cover cases like Flux,
as for each version they publish the corresponding SBOM asset (i.e.
flux_0.28.2_sbom.spdx.json).
If the README file contains a SBOM section that explains where they
are published to, format used, etc.
—
Reply to this email directly, view it on GitHub
https://github.com/cncf/clomonitor/issues/36#issuecomment-1078932373,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAPSIM6DWZIFOKLOWK3DHDVBWP6NANCNFSM5N64I3PA
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Cheers,
Chris Aniszczyk
https://aniszczyk.org


Serve combined GeoJSON files?

Wondering if it makes sense to also serve, eg, a single GeoJSON file for all state boundaries together.
(This is just a use-case I've bumped up against time and again, and while GitHub seems like a great platform for geo hosting, I can't seem to find a canonical repo for this. Is this it??? :smile: )

PR #6 includes a topojson file for all states in context of 114th congress.
(see: https://github.com/bradoyler/districts/blob/topojson-make/us/114th-congress.json)

Merged files would be great. Especially with party affiliation.

If there is still interest, please take a look at: https://github.com/Andrew-Chen-Wang/district-autoupdate It'll generate your merged file, but this GitHub actions only applies to the cds folder (the one applicable thing for my use case).


Caching tensorflow model between inferences

I'm trying to serve a tensorflow model (~3MB as a pickle or .pb file), and it is being reloaded into a MemoryDataSet every time the inference pipeline is called.
Is there a way of caching the model between executions to speed up this process?
I've tried using a CachedDataSet;
  type: CacheDataSet
  versioned: true
  dataset:
    type: tensorflow.TensorFlowModelDataset
    filepath: data/06_models/classifier
    layer: model

But this causes an error.
In our API, the mlflow model containing our preprocessing+tensorflow pipeline is loaded as follows
pipeline = mlflow.pyfunc.load_model(logged_model)
and called using
pipeline.predict(data)

Hi @mb1069, nice to see you found the solution!
Just in case, you can use
kpm_kwargs={
    'copy_mode': {"my_dataset_name": "assign"}
}

if you want to change only the copy for the "my_dataset_name" entry in your catalog.
A bit off topic, but I'd be glad to have feedback on pipeline_ml_factory: do you find it easy to use and understand? Is it convenient / useful?

Hi Yolan,
I've found it very easy to use and adapt to my needs. The only feature I wish existed is an option to exclude nodes using a tag, but I think that's more of an issue for Kedro?

Sorry, I did not see your answer earlier, I don't know why. Glad to see you find it easy, I know many people struggle to understand its purpose.
You can do something like my_pipeline.only_nodes_with_tags("my_tag"), but AFAIK there is no my_pipeline.except_nodes_with_tags("my_tag") method in kedro. As you say, this is entirely related to kedro and not kedro-mlflow.


"Address" question type: define UI/UX/functionality

We have some a mockup for this question type, but we need to get more detail on the final UI/UX. In Miro, please help us define:

What the error text for each possible type of error looks like, and where on/near the widget it's shown
How the widget looks when it's in a default state (nothing entered), a focused state, a valid state (correct input entered), and an error state.
Suggested pixel dimensions for all input field(s) in questions of this type.
How the custom functionality, if any, should work.
Anything else needed to define how this question type works.

Ideally, include 2-3 mockups showing what the question looks like in several different states (e.g. you've inputted half the answer, you've inputted all the answer but there's an error...)
See also #398, the central issue for this question type.
Feel free to only take on part of this task, by the way.

See #398 for mockups

Sorry, the Miro link from above broke. See tiny.cc/cvf-flow.


add functionality to filter Cards

This is not necessary any more, filter will be through cardtag relation.