Add support for optional cloud (aws, gcs, etc.) credentials for s3 for ingestion

Description
Cloud InputSource (such as S3, etc.) should be able to support taking in credentials optionally in order to be able to read from multiple buckets/location. This is in order to use different credentials than the default (i.e. the one provided in the common runtime property) and also to be able to use different credentials for each ingestion.
Motivation
The source of raw data (input data) can be stored in a different bucket/location which have different credentials than the deep storage location. For example, we might want to consume from s3 and also use s3 as a deep storage, however, the raw data s3 and deep storage s3 are different buckets with different credentials.
We may also want to ingest from multiple bucket/location. For example, we might want to consume from multiple s3 buckets each with its own credentials. We should not have to go and change the druid.s3.accesskey to read raw data from different bucket for every ingestion.
Proposed changes
Adding new fields to the inputSource spec. These fields are 'accessKeyId' and 'secretAccessKey' inside the (new) 'properties' map under the "inputSource" section
These fields are optional.
...
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": "123455",
          "secretAccessKey": "masdjlaksjdlakjsd"
        }
      }
      "inputFormat": {
        "type": "json"
      },
      ...
    },
...

If 'properties' map is not present then the normal cloud client/configuration will be used (current behavior). If 'properties' map is present but none or only one field inside is present (not both 'accessKeyId' and 'secretAccessKey') then the task will failed. If both fields are given, a new client will be constructed for this ingestion task. The new client will use the access key and secret key given with the rest of the other configurations (i.e. region, etc) from the one currently used. (In the future we might support other s3/cloud configuration in the 'properties' by allowing other keys such as 'endpoint.signingRegion' in the 'properties' map)
For example, we will create a new s3 client for inputSource type s3 with overrideAccessKeyId and overrideSecretAccessKey given with something similar to below:
BasicAWSCredentials awsCreds = new BasicAWSCredentials("access_key_id", "secret_key_id");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                        .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
                        .build();

This new client will only be use for reading the input data. Deep storage will still be handle with the default client/configurations.
Passing in the key/secret will be handle with the passwordProvider class. passwordProvider is an interface which can be backed by different ways to (optionally securely) access secrets. (https://druid.apache.org/docs/latest/operations/password-provider.html#docsNav)
For example with the environment variable password provider:
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": {
              "type": "environment",
              "variable": "<ENV.KEY.NAME.CONTAINING.ACCESS.KEY>"
           },
          "secretAccessKey": {
              "type": "environment",
              "variable": "<ENV.KEY.NAME.CONTAINING.SECRET.ACCESS.KEY>"
           },
        }
      }

We can also add new Password Provider implementation (future improvement):

We can make the new Password Provider takes in a file path. The file path will contains the access key and secret key. The file will be on the machine or on cloud (using default cloud credential)
We can make the new Password Provider takes in an encrypted access key and secret key. The decryption key will be an environment variable that is set beforehand. Customer will have to encrypt the access key and secret key themself before putting it in the ingestionSpec
(This can be implement later if we find useful and in addition, can be use in other places that uses the Password Provider).

Rationale

Thought about adding a map to the inputSource section in the ingestionSpec.The map will be able to specify the association between credentials and prefixes/uris for cases where there are multiple prefixes/uris using multiple different credentials. However, this is very verbose (repeating prefixes/uris) and cumbersome. With the proposed change, we can still achieve the same result by using the the "Multi" input source idea (future work). Each InputSource that requires different credentials (even if it is the same type) will requires a separate "inputSource" entry, each with its own overrideAccessKeyId and overrideSecretAccessKey fields. (Note that the proposed 'properties' map is configs for a SINGLE s3/cloud configuration that will be apply to all the URIs/Prefixs/Buckets in that "inputSource")
Thought about the needs for encrypt/decrypt keys in the ingestionSpec. Since the ingestionSpec is very visible (log in many places), it is very important to make sure the credentials are not plain text in the ingestionSpec. The proposed solution addressed this issue. The proposed solution also make sure that the ingestionSpec config values can be reuse in Druid for subsequent ingestion.

Operational impact

Creating new cloud client for every ingestion that uses this new fields
No concern with backward compatibility since these new fields are optional.

Test plan

New integration tests will be added.
Will test in a real Druid cluster.

Future work

The "Multi" input source. This is to support ingestion from multiple different cloud credentials (such as multiple s3 buckets or mixture of s3 buckets and gcs buckets where all of them have different credentials) in the same ingestionSpec.
New Password Provider implementations described above.

Thanks for proposing this @maytasm3, I think this would be a great addition. Just to confirm, if an ingestion spec was loaded with the same accessKey and secretAccessKey as another previous ingestions spec we would keep the old client around in this case and reuse it, or would we create a new client on the fly? It sounds like we already have a mechanism in place for encrypting fields in the ingestion spec, is that right?

@zachjsh

The client instantiation is done inside the peon process which is forked from the middleManager. It might be difficult to keep the old client around and reuse it. We can do that as a further optimization if required. Are you worried about the cost for initializing the client?
We have PasswordProvider which abstract out the underlying mechanism to retrieve the password. There is additional work to implement file path password and env variable as key for encrypt/decrypt password.
This solution does not support multiple cloud storage providers at one time. Supporting multiple cloud storage providers in single ingestion can be done using the the "Multi" input source idea. Out of scope for this proposal

Hi @maytasm3, thank you for the nice proposal! This feature will be super useful. I have two comments on the new fields.

I think the new fields are pretty straightforward and the prefix "override" is not necessary. It's also confusing when you read from s3 but use some other storage type as deep storage than s3.
I think we may need more input-source-specific configurations such as endpoint.url, endpoint.signingRegion, and so on in the future. What do you think about adding a new field containing "accessKeyId" and "secretAccessKey" fields as below?

      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": "123455",
          "secretAccessKey": "masdjlaksjdlakjsd"
        }
      }

@jihoonson Thanks for the feedbacks.

If I understand correctly, passwordProvider is an interface which can be backed by different ways to (optionally securely) access secrets. An existing implementation we have that I am planning to use (for first version of this feature) is EnvironmentVariablePasswordProvider. This will retrieve the password using the supplied environment key provided (the environment key names will be provided in the ingestion spec). We can use this for now since it is already implemented and is use to deal with passing password in other configs. Other ideas I have that can implement the passwordProvider to provide alternative ways of passing passwords are:

We can make the ingestionSpec takes in a file path. The file path will contains the access key and secret key. The file will be on the machine or on cloud (using default cloud credential)
We can make the ingestionSpec takes in an encrypted access key and secret key. The decryption key will be an environment variable that is set beforehand. Customer will have to encrypt the  access key and secret key themself before putting it in the ingestionSpec
(This can be implement later if we find useful and in addition, can be use in other places that requires passing password though config/api/insecure channel.

Good point. I agree. We can omit the "override" prefix. Since these new keys is part of the "inputSource", I do not think people will confuse it with other usage of s3 (since it is in the "inputSource" section). Anyhow, the "override" is not needed.
That's a good idea. I agree that in the future it might be necessary to support other properties of s3/cloud to make the ingestion more flexible (and since we already support taking in optional credentials, there is no reason architecturally not to support other properties).

@jihoonson
Sorry for the confusion around passwordProvider. I think we can reuse the existing Password Providers we have for now (no need to implement new ones). We will treat the accessKeyId as one Password Provider and the secretAccessKey as another Password Provider. Similar to how AWSCredentialsConfig (where you define a separate Password Providers for the .accessKey and the .secretKey).
For example if user decide to use plain text:
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": {
              "type": "default",
              "password": "abcdef"
           },
          "secretAccessKey": {
              "type": "default",
              "password": "asdasdadsasd"
           },
        }
      }

Or they can use Environment variable password provider:
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": {
              "type": "environment",
              "variable": "<ENV.KEY.NAME.CONTAINING.ACCESS.KEY>"
           },
          "secretAccessKey": {
              "type": "environment",
              "variable": "<ENV.KEY.NAME.CONTAINING.SECRET.ACCESS.KEY>"
           },
        }
      }

Or they can have a mix:
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://foo/bar", "s3://bar/foo"],
        "properties": {
          "accessKeyId": {
              "type": "default",
              "variable": "asdasdasdasd"
           },
          "secretAccessKey": {
              "type": "environment",
              "variable": "<ENV.KEY.NAME.CONTAINING.SECRET.ACCESS.KEY>"
           },
        }
      }

@maytasm3 thanks, sounds good 👍


Add Spanish language

Thank you very much for your work! I added the Spanish language, it's the least I can do for your project :)

gracias @Haikuno !


⚠ Bofrost.be has degraded performance

In 2e0b0e0, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 877 ms

Resolved: Bofrost.be performance has improved in 157e53b after 6 minutes.


📜 Add v9.0.0 commands documentation version

📜 Documentation update
🎉 A new version of axone-protocol/axoned is available.
This PR add this new v9.0.0 version into the documentation.

@amimart @ccamel I've added fix for broken link since root path of commands tabs is now axoned.md instead of okp4.md


Error running npm install in github-action/: "ENOTDIR: not a directory, open '/workspaces/devcontainer-build-run/github-action/node_modules/devcontainer-build-run-common/npm-shrinkwrap.json'"

I am encountering an error when running npm install in the directory github-action, in the Dev Container:
vscode ➜ /workspaces/devcontainer-build-run (feat/support-giving-extended-privileges-to-container ✗) $ cd github-action/
vscode ➜ /workspaces/devcontainer-build-run/github-action (feat/support-giving-extended-privileges-to-container ✗) $ ls
dist  jest.config.js  lib  Makefile  node_modules  package.json  package-lock.json  src  tsconfig.json
vscode ➜ /workspaces/devcontainer-build-run/github-action (feat/support-giving-extended-privileges-to-container ✗) $ npm install
npm ERR! code ENOTDIR
npm ERR! syscall open
npm ERR! path /workspaces/devcontainer-build-run/github-action/node_modules/devcontainer-build-run-common/npm-shrinkwrap.json
npm ERR! errno -20
npm ERR! ENOTDIR: not a directory, open '/workspaces/devcontainer-build-run/github-action/node_modules/devcontainer-build-run-common/npm-shrinkwrap.json'
npm ERR!  Embarrassingly, I missed it among the other issues I'd created and spotted during some housekeeping :-(
Is this related to #100?

@stuartleeks, it's all good, thanks!
I ran into this issue around the same time as that pull request, but this is a different issue.
I encountered this issue after implementing the change in #100 when working on a change to enable dev containers configured with docker-in-docker (commit here). I haven't been able to resolve this issue, but the docker-in-docker support ended up not being a high priority addition to me


Update travis badge to new travis link

Hey, chaning links in the readme even increased coverage. Awesome!

On a serious note: Do we have any tests relying on random numbers without a fixed seed that could explain why this creates differences in coverage?


Upgrade sqlite3 version to 3.41.2

Steps taken to upgrade:
cd ./upgrade
go run ./upgrade.go
cd ..
go test ./... -v
git add sqlite3*

The upgrade tool makes this pretty easy :)

Any update this?  Would love to see this go in.

Seems https://github.com/mattn/go-sqlite3/pull/1166 replaces this.


access log formatters: cleanup of parsing and syntax validating

Commit Message:
General cleanup of substitution formatter
Additional Description:
Part of #20389. Some refactoring of parsing and syntax validation. There are no user-facing changes.
Summary of changes:

moved almost all parsing and command syntax checking to regex. Individual formatters do not need to parse command individually and check the syntax. Syntax checking is done in general way and use CommandSyntaxChecker flags which describe if command should have params and length.
moved remaining formatters to use lookup table (similar to PR #19411) instead of long if ... else .. if ... else
added speed test to measure time to create formatters from format string. There is about 2% reduction in time required to create formatters. On my dev machine times are

     old code: BM_AccessLogFormatterSetup          1114274 ns      1113990 ns          631
     new code: BM_AccessLogFormatterSetup          1106752 ns      1106569 ns          644

Some details about how parsing changed.
Previously entire string COMMAND(SUBCOMMAND):LENGTH was passed to a formatter and the formatter scanned the string extracting each part: COMMAND, SUBCOMMAND and LENGTH. The string was passed to many formatters in form of if ... else ... if.. else ... until one formatter accepted the string.
The new code uses regex to break COMMAND(SUBCOMMAND):LENGTH into COMMAND, SUBCOMMAND and LENGTH and uses lookup table to find COMMAND flags which tell if the formatter needs SUBCOMMAND and LENGTH to be present. This is generic code. If syntax matches the flags, SUBCOMMAND and LENGTH are passed to the formatter's constructor.
Risk Level: Med.
Testing: Added few unit tests. All previous unit test pass.
Docs Changes: No. No user facing changes.
Release Notes: No

/retest

/assign-from envoyproxy/envoy-maintainers

Ping @dio or @lizan for a look, as codeowners?

@rgs1 FYI - please look at these changes. I know you use some of the internal parsing routines and the changes in this PR may affect you.

@cpakulski thanks for the heads-up. AFAIK it did not break our build cc: @JuniorHsu

It change the signature so we need to adjust it. but happy to see this change to the error-less setup. Thanks


Implement Hexadecimal conversion model(convert from base16 to base10 and vice versa)

Write functions to convert between hexadecimal(base 16) and decimal(base 10)
The model is located here "./functions/src/model/hexadecimal.ts"

Algorithm to convert from base16 to base10
Algorithm to convert from base10 to base16

To contribute, read the contribution docs Contribution docs
Be sure to write the actual algorithms and not use built in conversion functions.

Hi, I ended up doing this,

 let result: string = '';
        let r = 0;
        while (decimal != 0) {

            // r = decimal % 16 + r;
            r = (decimal ) & 0xF;
            if (r >= 10 && r <= 15) {
                result += String.fromCharCode(65 + (r - 10));
                console.log('hex:', result);
            }else {
                result += String.fromCharCode(48 + r);
                console.log('dec:', result);
            }
            decimal >>= 4;
        }

//result should be reversed
        return result;

The problem is the result will contained hexadecimal value in reverse and I am not sure how to solve that without using
builtin functions.. I am not familiar with math advanced functions.. and found that as a challenge.. any help ?

Hey @0xWaleed  , What I meant was you arent allowed to use built in Converter functions such as :
hexString = yourNumber.toString(16);

however, regular typescript/javascript functionality such as 'reversing a string' like:
function reverse(s){
    return s.split("").reverse().join("");
}

is allowed.

I hope that makes sense. But to give you a quick solution for your problem; you could just loop through the string backwards and add each charector into a new array... Then return that new array.
However, you are free to do anything you want / any method you want 👍🏾

Thanks, I solved as fast as it works. my point is to get familiar with Github and the contribution system.
usually I solve problem by covering it in test and keep the refactor later. but hope my last commit satisfy the requirement.
🤷🏻

@0xWaleed , Thanks, yes I tested it myself and it  works. Thanks for also writing the Test.
It has been merged. 🚀


ow.ch: Implement a MVS channel type

Implement a channel type using H2 MVStore with options to be persisted in memory or disk, map with a single name or based on a function, optional map housekeeping with a function, compression options, etc.
Then probably some ow.ch.utils functions to provide map names and housekeeping based on date and housekeeping based on number of records. This would be useful for a new output on nattrmon.

Done in https://github.com/OpenAF/openaf/commit/be6d258bf3249eb074de95071ef895f653f1b5de


[WIP]High performance agent

This is a big pull request, which includes: #236 #206 #241 , specially #236 provided multi features, check the issues for further understanding and discussion.
The core module at agent-side will refactor.
cc @pengys5 @ascrutae @bai-yang . If [WIP] is still in pull request title, means I am still working on this. Why sending this pull request before I finish? For helping your guys to find and understand what I am doing.

Support #206 #241 #253 to provide more features for plugins.


[Gally]: master <- dev

Automatically created by Git-Ally

:tada: This PR is included in version 2.18.13 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Issue running seqvec on example file in Google Colab

I've tried getting seqvec to install on google colab, but have encountered some errors.
I run the following command
!pip install seqvec

ERROR: en-core-web-sm 2.2.5 has requirement spacy>=2.2.2, but you'll have spacy 2.1.9 which is incompatible.
ERROR: botocore 1.20.70 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.
ERROR: responses 0.13.3 has requirement urllib3>=1.25.10, but you'll have urllib3 1.24.3 which is incompatible.

At the end it appears to have finished installing?? An gave this final message
Successfully installed allennlp-0.9.0 blis-0.2.4 boto3-1.17.70 botocore-1.20.70 conllu-1.3.1 flaky-3.7.0 flask-cors-3.0.10 ftfy-6.0.1 gevent-1.4.0 jmespath-0.10.0 jsonnet-0.17.0 jsonpickle-2.0.0 numpydoc-1.1.0 overrides-6.1.0 parsimonious-0.8.1 plac-0.9.6 preshed-2.0.1 pytorch-pretrained-bert-0.6.2 pytorch-transformers-1.1.0 responses-0.13.3 s3transfer-0.4.2 sentencepiece-0.1.95 seqvec-0.4.1 spacy-2.1.9 tensorboardX-2.2 thinc-7.0.8 typing-utils-0.0.3 unidecode-1.2.0 word2number-1.1

I downloaded the test fasta file with this command:
!wget https://raw.githubusercontent.com/Rostlab/SeqVec/master/test-data/sequences.fasta

Then, I ran the follwoing command with seqvec:
!seqvec -i sequences.fasta -o embeddings.npz

And I get all of these glorious errors
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
Traceback (most recent call last):
  File "/usr/local/bin/seqvec", line 5, in <module>
    from seqvec.seqvec import main
  File "/usr/local/lib/python3.7/dist-packages/seqvec/seqvec.py", line 15, in <module>
    from allennlp.commands.elmo import ElmoEmbedder
  File "/usr/local/lib/python3.7/dist-packages/allennlp/commands/__init__.py", line 8, in <module>
    from allennlp.commands.configure import Configure
  File "/usr/local/lib/python3.7/dist-packages/allennlp/commands/configure.py", line 26, in <module>
    from allennlp.service.config_explorer import make_app
  File "/usr/local/lib/python3.7/dist-packages/allennlp/service/config_explorer.py", line 24, in <module>
    from allennlp.common.configuration import configure, choices
  File "/usr/local/lib/python3.7/dist-packages/allennlp/common/configuration.py", line 17, in <module>
    from allennlp.data.dataset_readers import DatasetReader
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/__init__.py", line 1, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/dataset_readers/__init__.py", line 10, in <module>
    from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in <module>
    from allennlp.data.instance import Instance
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/instance.py", line 3, in <module>
    from allennlp.data.fields.field import DataArray, Field
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/fields/__init__.py", line 7, in <module>
    from allennlp.data.fields.array_field import ArrayField
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/fields/array_field.py", line 10, in <module>
    class ArrayField(Field[numpy.ndarray]):
  File "/usr/local/lib/python3.7/dist-packages/allennlp/data/fields/array_field.py", line 50, in ArrayField
    @overrides
  File "/usr/local/lib/python3.7/dist-packages/overrides/overrides.py", line 88, in overrides
    return _overrides(method, check_signature, check_at_runtime)
  File "/usr/local/lib/python3.7/dist-packages/overrides/overrides.py", line 114, in _overrides
    _validate_method(method, super_class, check_signature)
  File "/usr/local/lib/python3.7/dist-packages/overrides/overrides.py", line 135, in _validate_method
    ensure_signature_is_compatible(super_method, method, is_static)
  File "/usr/local/lib/python3.7/dist-packages/overrides/signature.py", line 93, in ensure_signature_is_compatible
    ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)
  File "/usr/local/lib/python3.7/dist-packages/overrides/signature.py", line 288, in ensure_return_type_compatibility
    f"{method_name}: return type `{sub_return}` is not a `{super_return}`."
TypeError: ArrayField.empty_field: return type `None` is not a `<class 'allennlp.data.fields.field.Field'>`.

Any thoughts on how I can get it working in Google colab?
Here is a link to the notebook with the various things I have tried already to solve this.
https://colab.research.google.com/drive/1KB6KYB20LXwvV2wnj8rdcV-x_fxXYYO9?usp=sharing

I had the same issue. I created new conda environment, installed python=3.7 and seqvec, and downgraded overrides to 3.1.0 as clarified by @girum89.
conda create -n seqvec python=3.7
conda activate seqvec
pip install seqvec
pip install overrides==3.1.0
I guess the issue can be closed.

I'm grateful for y'all's interest in SeqVec, but I'd like to again point you to bio_embeddings, where I'm actually going to fix those kind of problems


Integrate auto-shutdown configuration in ARM template deployments for Azure VMs

Is it possible to enable the VM Auto Shutdown configuration from Template Switch?
Here is a like with some ARM snippets to do this:
https://www.danielstechblog.info/integrate-auto-shutdown-configuration-arm-template-deployments-azure-vms/

Good idea - will implement this

Implemented


Fix incorrect loss_weights error message, clarify docs

Summary
loss_weights argument to model.fit only accepts python lists, so no numpy arrays. clarify docs for that. Also the error message was incorrect, asking for a "list of dicts" instead of a "list or dict" as the docs ask for.
PR Overview
No special requirements, minor change to docs, error message

n This PR requires new unit tests [y/n] (make sure tests are included)
? This PR requires to update the documentation [y/n] (make sure the docs are up-to-date)
n This PR is backwards compatible [y/n]
n This PR changes the current API [y/n] (all API changes need to be approved by fchollet)

Edit:
Force-pushed to reflow text to fit in 85 cols

"list OR dict" is very different from "list OF dictS"
It's not obvious at all given that most methods take numpy "lists" (arrays).

Superseded by https://github.com/tensorflow/tensorflow/pull/41859


Zambia 2019 retrain

Retrained Zambia model https://github.com/nasaharvest/crop-mask/issues/221

I see some double code here with #249.
Here are the two PRs:

Zambia_2019 -> master
Uganda_2019 -> master
What I usually do to make it cleaner is:
Zambia_2019 -> master
Uganda_2019 -> Zambia_2019

Then merge in Zambia_2019 first and Uganda_2019 will be updated to go into master.


Suggestion on improved #warning text

@tobozo: This library was just what I was looking for - thank you!  (PS. works on the ESP32-S3 too.)
PS. You might consider modifying "pfs.c" to something like:
#if defined BOARD_HAS_PSRAM
  #warning "Will use PSRAM or heap"
#elif !defined CONFIG_SPIRAM_SUPPORT
  #warning "No SPIRAM detected, will use heap"
#endif

Also, it would be good if one could override the #define FPSRAM_PARTITION_SIZE 0.5 line (in "PSRamFS.h") by pre-defining the variable to say #define FPSRAM_PARTITION_SIZE 0.25 in my main sketch, rather than having to modify the "PSRamFS.h" file.  However I can't find a way of defining the variable in my sketch so that is visible to the "PSRamFS.h" definition?!

hi, thanks for your feedback :+1:
psram and spiram may be different things depending on the device and sdk version so I've merged both conditions and modified the detection logic.
you can now call this before PsRamFS.begin() to adjust the partition size:
PSRamFS.setPartitionSize( ESP.getFreePsram()/2 );


A number_of_units argument could simplify the creation of large testing grounds

If we could use a 'multiplier' for a technology, network component or household (base-load), it would be fairly easy to build very large testing ground with very little overhead in work en computational effort.
I don't know what criterion to use for letting ETLoader decide whether to 'group' a number of the same technologies/households yet. One possible option would be to make it dependent on the number of end-points the topology contains:

If the number of end-points >= number of households: no need to group
If the number of end-points <= number of households: group households (and their technologies) per end-point using a number_of_units argument to quantify.

NOTE: Currently, the functionality to 'group' things is implicitly there: if you multiply the demand (for households) or the capacity (for technologies) with a factor, that factor effectively is the number_of_units.

UPDATE: the 'units' attribute has been added to ETLoader.
To do:

[ ] Make it possible to scale back a scenario with 'units' to national size
[ ] When technologies and households are coupled to a topology, the units should be assigned automatically and intelligently to maximise grouping (and thus minimise overhead)

Assigning @grdw

@grdw could this be easily be abstracted into a pull request? Perhaps this is a feature that we could deploy and test separately from the total work-flow update?


Code Sign error for cocoapods

I followed the guide to setup fastlane with hockey app. The error only happened to cocoapods libraries. Is there any settings I need to fine tune?
▸ Building Pods/CocoaLumberjack [(Release)]
▸ Check Dependencies

❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution:XXXX XXX” were found.

Here is my fastfile
fastlane_version "1.95.0"

default_platform :ios

platform :ios do
  # Run this before doing anything else
  before_all do
    # If the IT status is not clean, abort. We won't want to include junk in the build
    ensure_git_status_clean
  end

  private_lane :build_app do |options|

    # This part is done only when the app is not for the "production" environment
    if not options[:release]
      # Modulate the color of the icons
      color_icon(
        modulation: "#{options[:modulation]}"
      )
      # Add the build number to the icon
      build_number_icon
    end

    # Update the app name
    app_name(
      plist_path: "#{options[:project_name]}/Info.plist",
      app_name: options[:app_name]
    )

    # Update the app identifier
    update_app_identifier(
      xcodeproj: "#{options[:project_name]}.xcodeproj",
      plist_path: "#{options[:project_name]}/Info.plist",
      app_identifier: options[:app_identifier]
    )

    # Install the certificate
    import_certificate(
      certificate_path: options[:certificate_path],
      certificate_password: options[:certificate_password],
      keychain_name: "login.keychain"
    )

    # Install the provisioning profile
    update_project_provisioning(
      xcodeproj: "#{options[:project_name]}.xcodeproj",
      profile: options[:profile]
    )

    # Version bump
    increment_build_number(
      build_number: options[:build_number]
    )

    # Build the app
    gym(
      scheme: "#{options[:scheme]}",
      configuration: options[:configuration],
      provisioning_profile_path: options[:profile],
      codesigning_identity: options[:codesigning_identity],
      export_method: options[:export_method]
    )

  end

  # Publish to Hockeyapp
  private_lane :publish_hockey do |options|

    # Generate a changelog with GIT since the last successful build in Jenkins
    changelog = sh("git log --graph --pretty=format:'%h -%d %s <%an>' --abbrev-commit #{ENV['GIT_PREVIOUS_SUCCESSFUL_COMMIT'] || 'HEAD^^^^^'}..HEAD")

    # Send the app to Hockeyapp (fill in your API token!)
    hockey(
      api_token: "[Hockey app key]",
      notes: "#{changelog.to_s}",
      release_type: options[:release_type]
    )
  end

  # Publish to the AppStore
  private_lane :publish_appstore do |options|
    deliver(force: true)
  end

  # Build and publish the Beta version to Hockeyapp
lane :beta_hockeyapp do
# Build
build_app(
  # Not a production release, so add build number and do the color modulation of the icons
  release:false,
  # Modulate the colors of the icons by these degrees
  modulation:66.6,
  # Change the app name
  app_name:"<App Name>",
  # Set the app id
  app_identifier:"com.company.<appname>",
  # Set the path to the certificate to use in building
  certificate_path:"./Certificates/Company/<cert>.p12",
  # Set the password of the p12 certificate file
  certificate_password:"<password>",
  # Set the path to the provisioning profile to use (change this!)
  profile:"./Certificates/Company/<profile>.mobileprovision",
  # What configuration to use, usefull for keeping different API keys etc between environments
  configuration:"[INT] <project_name>",
  # Use this codesigning identity (this is the name of the certificate in your keychain)
  codesigning_identity:"Project Distribution",
  # Export an enterprise app
  export_method:"enterprise",
  # the projectname, this is the name of the .xcodeproj file and the folder containing your code in the project
  project_name:<Project>,
  # the scheme to build
  scheme:"Project-Internal",
  # the build number to use, we use the build number from Jenkins
  build_number: ENV["BUILD_NUMBER"]
)
# Push to Hockeyapp as Beta release
publish_hockey(release_type: "0")
  end

  # After all the steps have completed succesfully, run this.
  after_all do |lane|

    # Remove all build artifacts, but keep mobileprovisioning profiles since they are stored in GIT
    clean_build_artifacts(
      exclude_pattern: ".*\.mobileprovision"
    )

    # Reset all changes to the git checkout
    reset_git_repo(
      force: true
    )
  end

  # If there was an error, run this
  error do |lane, exception|
    # Remove all build artifacts, but keep mobileprovisioning profiles since they are stored in GIT
    clean_build_artifacts(
      exclude_pattern: ".*\.mobileprovision"
    )

    # Reset all changes to the git checkout
    reset_git_repo(
      force: true
    )
  end
end

Can you please post the full log? Thanks!

We recommend following the official code signing guide 👍

Hi Krause,
Unfortunately I don't have paid account in github. I wish to know if I can settle this problem without the match.
[09:59:11]: -------------------------------------------------
[09:59:11]: --- Step: Verifying required fastlane version ---
[09:59:11]: -------------------------------------------------
[09:59:11]: fastlane version valid
[09:59:11]: ------------------------------
[09:59:11]: --- Step: default_platform ---
[09:59:11]: ------------------------------
[09:59:11]: Driving the lane 'ios beta_hockeyapp' 🚀
[09:59:11]: -------------------------------------
[09:59:11]: --- Step: ensure_git_status_clean ---
[09:59:11]: -------------------------------------
[09:59:11]: Git status is clean, all good! 💪
[09:59:11]: ------------------------------------------
[09:59:11]: --- Step: Switch to ios build_app lane ---
[09:59:11]: ------------------------------------------
[09:59:11]: Cruising over to lane 'ios build_app' 🚖
[09:59:11]: ----------------------
[09:59:11]: --- Step: app_name ---
[09:59:11]: ----------------------
[09:59:12]: Helper.log is deprecated. Use `UI` class instead
[09:59:12]: Updated project/Info.plist 💾.
[09:59:12]: -----------------------------------
[09:59:12]: --- Step: update_app_identifier ---
[09:59:12]: -----------------------------------
[09:59:12]: Updated project.xcodeproj 💾.
[09:59:12]: --------------------------------
[09:59:12]: --- Step: import_certificate ---
[09:59:12]: --------------------------------
[09:59:12]: -----------------------------------------
[09:59:12]: --- Step: update_project_provisioning ---
[09:59:12]: -----------------------------------------
[09:59:12]: You’re updating provisioning profiles directly in your project, but have you considered easier ways to do code signing?
[09:59:12]: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/CodeSigning.md
[09:59:12]: Parsing mobile provisioning profile from './fastlane/certificates/XXXXX_Distribution.mobileprovision'
[09:59:12]: Going to update project 'project.xcodeproj' with UUID
[09:59:12]: Updating target project...
[09:59:12]: Updating configuration Debug...
[09:59:12]: Updating configuration Release...
[09:59:12]: Updating target project...
[09:59:12]: Updating configuration Debug...
[09:59:12]: Updating configuration Release...
[09:59:12]: Successfully updated project settings in'project.xcodeproj'
[09:59:12]: ------------------------------------
[09:59:12]: --- Step: increment_build_number ---
[09:59:12]: ------------------------------------
[09:59:12]: $ cd /Users/<project>/project && agvtool next-version -all && cd -
[09:59:12]: ▸ Setting version of project project to:
[09:59:12]: ▸ 2.
[09:59:12]: ▸ Also setting CFBundleVersion key (assuming it exists)
[09:59:12]: ▸ Updating CFBundleVersion in Info.plist(s)...
[09:59:12]: ▸ Updated CFBundleVersion in "project.xcodeproj/../project copy-Info.plist" to 2
[09:59:12]: ▸ Updated CFBundleVersion in "project.xcodeproj/../project/Info.plist" to 2
[09:59:12]: ▸ /Users/<path>/project
[09:59:13]: -----------------
[09:59:13]: --- Step: gym ---
[09:59:13]: -----------------
[09:59:13]: Installing provisioning profile...
[09:59:13]: xcrun xcodebuild -list -workspace ./project.xcworkspace

+---------------------------+----------------------------------------------------------------------+
|                                      Summary for gym 1.6.3                                       |
+---------------------------+----------------------------------------------------------------------+
| scheme                    | project-TS-Internal                                               |
| configuration             | [INT] Web Wrapper                                                    |
| provisioning_profile_path | ./fastlane/certificates/XXXXXX_Distribution.mobileprovision |
| codesigning_identity      | iPhone Distribution: XXXXXX                              |
| export_method             | enterprise                                                           |
| workspace                 | ./project.xcworkspace                                             |
| destination               | generic/platform=iOS                                                 |
| output_name               | [INT] Web Wrapper                                                    |
| clean                     | false                                                                |
| output_directory          | .                                                                    |
| silent                    | false                                                                |
| use_legacy_build_api      | false                                                                |
| buildlog_path             | ~/Library/Logs/gym                                                   |
+---------------------------+----------------------------------------------------------------------+

[09:59:14]: $ set -o pipefail && xcodebuild -workspace ./project.xcworkspace -scheme project-TS-Internal -configuration '[INT] Web Wrapper' -destination 'generic/platform=iOS' -archivePath '/Users/XXXX/Library/Developer/Xcode/Archives/2016-06-20/[INT] Web Wrapper 2016-06-20 09.59.14.xcarchive' archive CODE_SIGN_IDENTITY='iPhone Distribution: XXXXX' | tee /Users/<path>/Logs/gym/\[INT\]\ XXXX-XXX-Internal.log | xcpretty
[09:59:15]: ▸ Building Pods/SwiftKeychainWrapper [(Release)]
[09:59:15]: ▸ Check Dependencies
[09:59:15]: ▸ ❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.
[09:59:15]: ▸ Building Pods/CocoaLumberjack [(Release)]
[09:59:15]: ▸ Check Dependencies
[09:59:15]: ▸ ❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.
[09:59:15]: ▸ Building Pods/DZNWebViewController [(Release)]
[09:59:15]: ▸ Check Dependencies
[09:59:15]: ▸ ❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.
[09:59:15]: ▸ ** ARCHIVE FAILED **
[09:59:15]: ▸ The following build commands failed:
[09:59:15]: ▸ Check dependencies
[09:59:15]: ▸ Check dependencies
[09:59:15]: ▸ Check dependencies
[09:59:15]: ▸ (3 failures)
▸ Building Pods/SwiftKeychainWrapper [(Release)]
▸ Check Dependencies

❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.

▸ Building Pods/CocoaLumberjack [(Release)]
▸ Check Dependencies

❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.

▸ Building Pods/DZNWebViewController [(Release)]
▸ Check Dependencies

❌  Code Sign error: No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution: XXXXXX” were found.

** ARCHIVE FAILED **

The following build commands failed:
Check dependencies
Check dependencies
Check dependencies
(3 failures)
[09:59:15]: Exit status: 65
[09:59:15]: -----------------------------------
[09:59:15]: --- Step: clean_build_artifacts ---
[09:59:15]: -----------------------------------
[09:59:15]: Cleaned up build artifacts 🐙
[09:59:15]: ----------------------------
[09:59:15]: --- Step: reset_git_repo ---
[09:59:15]: ----------------------------
[09:59:15]: $ git reset --hard HEAD
[09:59:15]: ▸ HEAD is now at 0957b3e Changed certficates path
[09:59:15]: $ git clean -qfdx
[09:59:15]: Git repo was reset and cleaned back to a pristine state.
[09:59:15]: Variable Dump:
[09:59:15]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios beta_hockeyapp", :GIT_REPO_WAS_CLEAN_ON_START=>true, :BUILD_NUMBER=>"2"}
[09:59:15]: Error building the application - see the log above

+------+-------------------------------------+-------------+
|                     fastlane summary                     |
+------+-------------------------------------+-------------+
| Step | Action                              | Time (in s) |
+------+-------------------------------------+-------------+
| 1    | Verifying required fastlane version | 0           |
| 2    | default_platform                    | 0           |
| 3    | ensure_git_status_clean             | 0           |
| 4    | Switch to ios build_app lane        | 0           |
| 5    | app_name                            | 0           |
| 6    | update_app_identifier               | 0           |
| 7    | import_certificate                  | 0           |
| 8    | update_project_provisioning         | 0           |
| 9    | increment_build_number              | 0           |
| 10   | gym                                 | 2           |
| 11   | clean_build_artifacts               | 0           |
| 12   | reset_git_repo                      | 0           |
+------+-------------------------------------+-------------+

[09:59:15]: fastlane finished with errors

Try calling cert and sigh before gym.

As the error message says
No matching codesigning identity found: No codesigning identities (i.e. certificate and private key pairs) matching “iPhone Distribution:XXXX XXX” were found.

The code signing identity is not installed locally, so the provisioning profile can't be used

@KrauseFx I have installed certs and provisioning profiles properly as it can be installed on my real device.
@vpolouchkine sorry, I'm not sure which part is sigh.

You can find out more about sigh here.

Have you seen our new Code Signing Troubleshooting Guide? It will help you resolve the most common code signing issues :+1:

I had the same issue yesterday.
The build would succeed locally but fail on the Mac Mini running Jenkins.
I use match to share certificates and provisioning between machines.
Also, the project I'm working on is Swift only.
Yesterday I added 3rd party dependencies to the project for the first time through Cocoapods when the build failures started happening.
After doing some research I found  this post  that was helpful.
After adding this at the end of the Podfile the build started working again:
post_install do |installer|
    installer.pods_project.targets.each do |target|
        target.build_configurations.each do |config|
            config.build_settings['EXPANDED_CODE_SIGN_IDENTITY'] = ""
            config.build_settings['CODE_SIGNING_REQUIRED'] = "NO"
            config.build_settings['CODE_SIGNING_ALLOWED'] = "NO"
        end
    end
end

If anyone ends up here, I found my solution was to go delete all my local provisioning profiles, and let match do its thing from there.  (All profiles are in ~/Library/MobileDevice/Provisioning Profiles)
Seems like maybe I had an old pre-match one that didn't use the current certificate, and even though I had the environment variable in the PROJECT for sigh/match, cocoa pods didn't like that.  Or something, I dunno.   ¯_(ツ)_/¯

There's been a number of new comments on https://github.com/CocoaPods/CocoaPods/issues/4331. We have been able to build, then resign and submit an app to iTC using CocoaPods and dynamic frameworks.
Important thing to look at is the bundle IDs of those frameworks.
AFAIK, with latest Xcode 8 beta Apple finally removed the requirement for code signing frameworks. So our lives should become much easier soon :)


Pipeline Broken by Ubuntu 18.04 (20220710 update) PR #5882

Description
Our YAML pipeline broke following your Ubuntu 18.04 agent update 20220710 in PR #5882.
Without any changes to our YAML pipeline definition for over thirty days, following our last successful execution of a YAML pipeline that sets a pipeline variable using the AzurePowerShell@5 task on 7/11/2022 at 9:47am EDT, all subsequent executions are failing as of 7/14/2022 at 12:57pm EDT and thereafter.
Note: Repointing our YAML to Ubuntu 20.04 fixes this newly introduced 18.04 issue
Here is the task definition:

     - task: AzurePowerShell@5
       name: setImageTagScript
       env:
         buildId: $(Build.BuildId)
       inputs:
       pwsh: true
         azurePowerShellVersion: latestVersion
         FailOnStandardError: true
         scriptType: inlineScript
         azureSubscription: '$(AzDO_Service_Connection_Prod)'
         inline: |
           Write-Output "Setting output variable 'imageTag' to: $env:buildId"
           Write-Host "##vso[task.setvariable variable=imageTag;isOutput=true]$env:buildId"

Here are the relevant lines from the log:

##[debug]Processed: ##vso[task.setvariable variable=imageTag;isOutput=true]5064
##[debug]Exit code 0 received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
##[debug]task result: Failed
##[error]PowerShell wrote one or more lines to the standard error stream.
##[debug]Processed: ##vso[task.issue type=error;]PowerShell wrote one or more lines to the standard error stream.
##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell wrote one or more lines to the standard error stream.

Even though the Exit Code from the AzurePowerShell@5 task is zero (0) which should indicate successful execution, it is acting as if it failed.
Platforms affected

[X] Azure DevOps
[ ] GitHub Actions

Virtual environments affected

[X] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 10.15
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
20220710.1
Is it regression?
Yes - Version 20220626.1 worked
Expected behavior
Setting a pipeline variable using AzurePowerShell@5 task in Azure DevOps used to and should continue to succeed without resulting in a PowerShell error upon Exit Code 0 (success).
The following used to work in 20220626.1:

Write-Host "##vso[task.setvariable variable=imageTag;isOutput=true]$env:buildId"

It no longer works as of 20220710.1
Actual behavior
The following diagnostic pipeline execution output reveals the current behavior:

##[debug]Processed: ##vso[task.setvariable variable=imageTag;isOutput=true]5064
##[debug]Exit code 0 received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
##[debug]task result: Failed
##[error]PowerShell wrote one or more lines to the standard error stream.
##[debug]Processed: ##vso[task.issue type=error;]PowerShell wrote one or more lines to the standard error stream.
##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell wrote one or more lines to the standard error stream.

Repro steps
Create an AzurePowerShell@5 task running under the hosted agent ubuntu-18.04 as follows.  Note: We have an environment variable - we do not know if this matters or not.

    - task: AzurePowerShell@5
       name: setImageTagScript
       env:
         buildId: $(Build.BuildId)
       inputs:
       pwsh: true
         azurePowerShellVersion: latestVersion
         FailOnStandardError: true
         scriptType: inlineScript
         azureSubscription: '$(AzDO_Service_Connection_Prod)'
         inline: |
           Write-Output "Setting output variable 'imageTag' to: $env:buildId"
           Write-Host "##vso[task.setvariable variable=imageTag;isOutput=true]$env:buildId"

This should fail.
Retargeting this to ubuntu-20.04 should succeed.

@al-cheb While I appreciate your referring to the external bug, our issue does not relate to MSAL calls, but to a simple "Write-Host" statement within an Azure DevOps Pipeline to set a pipeline variable via the '##vsto' expression.  Therefore, I have to disagree the cause being the same.  That said, if the latest Azure PowerShell module upgrade to the Ubuntu 18.04 vm image rendered numerous Azure PowerShell related activities useless, it should be reverted.

@dariusonsched, in that case you should create a new issue in  - azure-pipelines-tasks repository


Icon

Hi there, may i know how u add the icons ? Like living room, bedroom icon?
im still newbie in uploading images into m5stack so can you help me?

Take a look at the data subfolder and make sure to name your images accordingly to the others. Then you can just use then as an icon inside the config.json.
Images and configuration get copied into the filesystem as part of the make spiffs_spiffs_bin build command. Make sure to flash after :)

thank you for your help.
btw, i have already found the filesystem folder, but may i know how to flash it? is it by running the file from command prompt?

You simply run make spiffs_spiffs_bin before make flash.
make spiffs_spiffs_bin generates the .bin file which represents a separate partition which will be flashed during make flash. If you want to learn more about SPIFFS and flashing I recommend the ESP-IDF documentation :)

is the ESP-IDF documentation is this link?
https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/storage/spiffs.html

the make spiffs_spiffs_bin is located in folder? or i have to install any package first?

There are some detailed build instructions in the README file of this repo :) If you follow those you can run make spiffs_spiffs_bin... spiffs_spiffs_bin is just a command for the makefile.

oh okay2 thank you very much for your help sieren. Im really new to this. Will try my best

No worries. Try googling for some cmake and makefile tutorials to maybe get a better understanding of how a build-system works. I think that might help :) I'll close this for now.


Update _settings.scss

bug is fixed

This was already fixed in a recent commit. Thank you anyway.


Vagrant: install wrapt

latest devlib depend on wrapt.  The vagrant file should pip install wrapt.  I'd rather do this once #113 is merged rather than adding even more stuff there.

Might it be easier to just use each lib's setup.py?

Yes, probably we could just pip install bart, trappy and devlib... @derkling what do you think?

If we are going to pip install everything, then we should get rid of submodules.

Tough  decision... one more time.
If it simplify deployment on VMs, perhaps it's better to pip install everything... however, we must be aware that the LISA shell by default uses "DEVMOVE"... thus it expects local installed submodules...

OK, then let's just install dependencies when needed. I'll send a PR for solving this issue.

On 19-Jul 06:09, Michele Di Giorgio wrote:

OK, then let's just install dependencies when needed. I'll send a PR
for solving this issue.

Just to be clear, I'm open also to the other solution... only worried
for the changes required. We spent a lot of time to integrate
submodules... we just need to have a tought about what can be the
possible issues.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/ARM-software/lisa/issues/145#issuecomment-233626911

--
#include <best/regards.h>
Patrick Bellasi

Adding things is simple. IMHO it is probably better to solve this by using tags and provide users with a stable release (that has no dependency or other issues). This way, dependency issues will hopefully only appear in development versions.


fix parsing small relocation entries

It seems like some versions of LLVM generate relocation entries with only offset and info present.
Change the parser to require at least 16 byte entries (to prevent a panic) and skip the entry size
instead of hardcoding a length.

LGTM


net/http: Incorrect comment for MethodPatch in method.go

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go 1.6.2

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

N/A

What did you expect to see?

incorrect comment for MethodPatch in net/http/method.go: "// RFC 5741" should be "// RFC 5789".
see rfc5789

What did you see instead?

N/A

This is a duplicate of #15546.


Add to documentation how to get the tgt value

The documentation has no where specific mentioning the retrival of the tgt value. Add this code under the https://pycai.gitbook.io/welcome/api/values "tgt" section which retrives the tgt value:
PyCai("token").chat.get_chat("char")["participants"][0]["user"]["username"] if not PyCai("token").chat.get_chat("char")[
    "participants"][0]["is_human"] else PyCai("token").chat.get_chat("char")["participants"][1]["user"]["username"]

Thank you, that's great solution! It works fine!
But here is a rewritten version that seems to be easier to understand:
token = "HERE IS ACCOUNT TOKEN" # TODO
char_id = "HERE IS CHARACTER ID" # TODO

client = PyCAI("token")

tgt = client.chat.get_chat(char_id)["participants"][0]["user"]["username"] if not client.chat.get_chat(char_id)[
    "participants"][0]["is_human"] else client.chat.get_chat(char_id)["participants"][1]["user"]["username"]

print(tgt)

Now it does not mix that "user" is just a JSON field but "char" is a parameter needed to be changed.
And yes, then I run this code, I got a string with internal_id:here:is-some-hex-digits-and-delimiters, that's okay that it has internal_id at beginning.

👍 Good idea

Added! https://docs.kram.cat/qna.html#what-s-tgt


[reactivsocket-cpp/build] Remove unnecessary boost dependency

boost is not used used anywhere in the code

we need to fix travis first


"manually" installing setuptools

I am trying to use setuptools inside world of tanks (a game), but it has its own python (2.7.X) and doesn't allow pip, so is it possible to manually install it. So actually not installing it but just calling the needed files independently?

Manually installing setuptools can be done from sources using python bootstrap.py and python setup.py install ...  Another alternative would be to add setuptools' source directory to the Python path (after generating the metadata with python bootstrap.py). But without more information about world of tanks' embedded Python and how it's managed, there's no way to know if that will work... Sorry, you'll have to investigate yourself.


pysam version issue?

When run "python nanocorrect.py" I got the following error:
Traceback (most recent call last):
File "nanocorrect.py", line 187, in
(seq, n_reads) = run_poa_and_consensus(overlaps, read_idx)
File "nanocorrect.py", line 146, in run_poa_and_consensus
(in_fn, n_reads) = write_poa_input(overlaps, read_idx)
File "nanocorrect.py", line 62, in write_poa_input
read_id1 = ref.references[read_idx]
AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'
python 2.7.3
pysam: 0.7.5

Can you provide the complete set of commands that you ran?

Thanks.
Here are the commands tested (within the installation path of nanocorrect):
make -f nanocorrect-overlap.make INPUT=reads.fasta NAME=nc
python nanocorrect.py nc > corrected.fasta
where reads.fasta are minion data (pass filter).

You need to provide a range of reads to correct:
python nanocorrect.py nc 1:50 > corrected.fasta

This will correct the first 50 reads in the file.

Sorry. I tried different range setting. and got the same error.
Many thanks in advance

nc.db:
files =         1
411 nc.pp m000000_000000_00000_c1898213712391273
blocks =         1
size =        50 cutoff =         0 all = 0
0         0
411       411
python nanocorrect.py nc 2:20 > corrected.fasta
Traceback (most recent call last):
File "nanocorrect.py", line 187, in
(seq, n_reads) = run_poa_and_consensus(overlaps, read_idx)
File "nanocorrect.py", line 146, in run_poa_and_consensus
(in_fn, n_reads) = write_poa_input(overlaps, read_idx)
File "nanocorrect.py", line 62, in write_poa_input
read_id1 = ref.references[read_idx]
AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'

What version of pysam do you have?

That output is in the file reads.pp.fasta.fai, right?

yes :-)

Did you have a chance to try pysam 0.8.3?

Sorry for my late response ...
I tried to set up a virtual machine within nanocorrect installation dir. What I did was the following:
virtualenv --no-site-packages test_vm
source test_vm/bin/activate
pip install pysam
source test_vm/bin/activate
which python
/misc/ngseq7/src/nanocorrect/test_vm/bin/python
more test_vm/lib/python2.7/site-packages/pysam/version.py
pysam versioning information
version = "0.8.3"
samtools_version = "1.2"
htslib_version = "1.2.1"
but still the same error showed up when nanocorrect was run:
(test_vm)$ make -f nanocorrect-overlap.make INPUT=reads.fasta NAME=nc3
nanocorrect-preprocess.pl reads.fasta > nc3.pp.fasta
fasta2DB nc3 nc3.pp.fasta
DBsplit -s50 nc3
DBdust nc3
HPCdaligner -t5 -mdust nc3 > HPCcommands.txt
/bin/bash HPCcommands.txt
LAcat nc3 > nc3.las
rm nc3.*.las
(test_vm)$ python nanocorrect.py nc3 1:411 > corrected.fasta
Traceback (most recent call last):
File "nanocorrect.py", line 187, in
(seq, n_reads) = run_poa_and_consensus(overlaps, read_idx)
File "nanocorrect.py", line 146, in run_poa_and_consensus
(in_fn, n_reads) = write_poa_input(overlaps, read_idx)
File "nanocorrect.py", line 62, in write_poa_input
read_id1 = ref.references[read_idx]
AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'
Many thanks.

I haven't forgotten about this but I'm a bit stumped. @alexjironkin had a similar problem today but after upgrading to 0.8.3 he reports that nanocorrect runs properly

In python console if you do:
pysam.__version__

Do you also get 0.8.3?

Thanks, Jared and Alex.
Alex you are right and it is also what has puzzled me. In python console (started within the VM), the version of pysam is still 0.7.5. You know the fix to this? Sorry that I am not very familiar with Python

Hmm, that's really strange. It looks like there are 2 versions of pysamtools on your system.
To see where it is do the following in Python console:
import pysam, os
print os.path.abspath(pysam.__file__)

This should show you absolute path to the pyc file with pysam 0.7.5. Depending where it is you may be able to uninstall it using
pip uninstall pysam
What OS/Python is this?

I am using Debian GNU/Linux 7.8 (wheezy). It is a shared server with multi-user access.
I found the path to pysam 0.7.5 following your instructions. But when I did pip uninstall pysam, either within or outside VM, the path did not show up in the list (as shown below):
/usr/local/ngseq/src/nanocorrect$ source test_vm/bin/activate
(test_vm):/usr/local/ngseq/src/nanocorrect$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pysam, os
print os.path.abspath(pysam.file)
/usr/local/ngseq/lib/python/pysam/init.pyc
quit()
(test_vm):/usr/local/ngseq/src/nanocorrect$ deactivate
/usr/local/ngseq/src/nanocorrect$ pip uninstall pysam
Uninstalling pysam:
/usr/local/lib/python2.7/dist-packages/pysam
/usr/local/lib/python2.7/dist-packages/pysam-0.8.0.egg-info
Proceed (y/n)? n
/usr/local/ngseq/src/nanocorrect$ source test_vm/bin/activate
(test_vm):/usr/local/ngseq/src/nanocorrect$ pip uninstall pysam
Uninstalling pysam:
/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam
/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam-0.8.3-py2.7.egg-info
Proceed (y/n)? n

What's the output of the following in Python console both for virtualenv and none:
import os, sys

print os.environ['PYTHONPATH']
print sys.path

within VM:

print os.environ['PYTHONPATH']
:/usr/local/ngseq/lib/python/:/usr/local/ngseq/lib/python2.7:/usr/local/ngseq/lib/python2.7/site-packages:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules:/usr/local/ngseq/stow/qiime-r1487/lib/:/usr/local/ngseq/stow/PyNAST-1.1/lib/:/usr/local/ngseq/src/Denoiser_0.851/:/usr/local/ngseq/lib/python:/usr/local/ngseq/lib/python2.6/site-packages
print sys.path
['', '/usr/local/ngseq/lib/python/keyring-3.3-py2.7.egg', '/usr/local/ngseq/lib/python/requests-2.1.0-py2.7.egg', '/usr/local/ngseq/lib/python/distribute-0.6.28-py2.7.egg', '/usr/local/ngseq/lib/python/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg', '/usr/local/ngseq/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/misc/ngseq7/src/nanocorrect', '/usr/local/ngseq/lib/python', '/usr/local/ngseq/lib/python2.7', '/usr/local/ngseq/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/plat-linux2', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-tk', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-old', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages']

outside VM:

print os.environ['PYTHONPATH']
:/usr/local/ngseq/lib/python/:/usr/local/ngseq/lib/python2.7:/usr/local/ngseq/lib/python2.7/site-packages:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules:/usr/local/ngseq/stow/qiime-r1487/lib/:/usr/local/ngseq/stow/PyNAST-1.1/lib/:/usr/local/ngseq/src/Denoiser_0.851/:/usr/local/ngseq/lib/python:/usr/local/ngseq/lib/python2.6/site-packages
print sys.path
['', '/usr/local/ngseq/lib/python/keyring-3.3-py2.7.egg', '/usr/local/ngseq/lib/python/requests-2.1.0-py2.7.egg', '/usr/local/ngseq/lib/python/distribute-0.6.28-py2.7.egg', '/usr/local/ngseq/lib/python/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg', '/usr/local/ngseq/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/h5py-2.3.1-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/poretools-0.3.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/rpy2-2.4.3-py2.7-linux-x86_64.egg', '/misc/ngseq7/src/nanocorrect', '/usr/local/ngseq/lib/python', '/usr/local/ngseq/lib/python2.7', '/usr/local/ngseq/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages']

Have a nice weekend

It looks like your '/usr/local/ngseq/lib/python2.7' is before '/misc/ngseq7/src/nanocorrect/test_vm/lib'
I don't know how that happened. If there is no system requirement to have pysam 8.0 I would suggest upgrading to 8.3.
As a quick hack puting
PYTHONPATH=/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam:$PYTHONPATH python  blah blah blah
inside your VM should temporarily change the search paths and your pysam will appear firstin the search list.It is a little mystery how the PYTHONPATH got tangled.

As this seems to be a python install issue rather than a nanocorrect problem I'm closing for now. Please reopen it if persists after fixing your paths as suggested by @alexjironkin.


Account Unlinking

Closes CNCT-2627

PR-Codex overview
This PR introduces the UnlinkAccount functionality across various wallet classes, implementing methods to unlink accounts from the current wallet while handling exceptions for unsupported operations.
Detailed summary

Added UnlinkAccount method in PrivateKeyWallet with an exception for unsupported operation.
Implemented UnlinkAccountAsync in EmbeddedWallet to interact with the server.
Enhanced UnlinkAccount in SmartWallet to check wallet type before unlinking.
Added detailed documentation for UnlinkAccount in IThirdwebWallet.
Created UnlinkAccountAsync in ServerBase to handle server requests for unlinking accounts.
Updated EcosystemWallet to check for connection status before unlinking accounts.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Codecov Report
Attention: Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Project coverage is 77.96%. Comparing base (429187d) to head (95673ad).

Files with missing lines
Patch %
Lines

...irdweb/Thirdweb.Wallets/SmartWallet/SmartWallet.cs
0.00%
7 Missing :warning:

...rdweb.Wallets/PrivateKeyWallet/PrivateKeyWallet.cs
0.00%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #107      +/-   ##
==========================================
- Coverage   78.23%   77.96%   -0.27%
==========================================
  Files          28       28
  Lines        4488     4497       +9
  Branches      619      620       +1
==========================================
- Hits         3511     3506       -5
- Misses        812      826      +14
  Partials      165      165

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


How to early return from pipeline definition?

I have a pipeline similar to:
Buildkite::Builder.pipeline do
  if build_skipped?
    command do
      label "skip"
      command "echo 'Build skipped'"
    end

    # break
  end

  # rest of pipeline
end

But due to this code, there isn't a clean way to exit early:
https://github.com/Gusto/buildkite-builder/blob/deedcfb4d036b5a51d25ba813a9acc6bc917dd1a/lib/buildkite/builder/pipeline.rb#L75
I'm not sure if throw/catch would be a good solution here, but otherwise the workaround is to wrap the whole rest of the pipeline in an else which should be avoidable.

So funny that I encountered a need for this very recently! I haven't had a chance to try it out yet, but does next work for you?
Buildkite::Builder.pipeline do
  if build_skipped?
    command do
      label "skip"
      command "echo 'Build skipped'"
    end

    next
  end

  # rest of pipeline
end

I tried everything but next :joy: that worked!
Thank you!! :bow:


typo Tennant

https://github.com/ovrclk/akash/blob/37fbd963a636b5adde21962e647183253b8eec27/manifest/auth.go#L83

Thanks so much @azhang! 😍


Await both the heartbeat request and shutdown channel in the heartbeat task

Motivation
We don't detect cancellations when we're waiting for a response from a peer, so this could cause hangs.
https://github.com/ZcashFoundation/zebra/blob/852c5d63bba22ed9547722fdc9bc9b56081dd116/zebra-network/src/peer/handshake.rs#L986-L999
Specifications

Designs

Related Work

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @jvff @oxarbitrage @teor2345 @upbqdn


Antd version needs to match project antd version

Currently the package specifies the antd version in the dependencies as ^5.3.0. When using yarn as package manager, when upgrading the antd dependency in a project, yarn wants to keep the version previously installed for each dependency and copies the previously installed version to node_modules inside @ant-design/static-style-extract directory in node_modules. This causes two different versions of antd to be installed, which makes the themes not work when extracting the css. Instead of the current way of specifying the antd dependency, it would be better to put it inside peerDependencies, as is suggested in the documentation.

We're looking forward to that, too.

Thanks for the feedback. Let me follow up on this.

Completed in<EMAIL_ADDRESS>


Allow Xcode project Info.plist to be moved

Description
Developers are able to change the location of Xcode project Info.plists in Xcode build settings, and can have different Info.plists for different schemes or configurations.
The Info.plist is only used to cheaply figure out a bundle identifier without parsing the build settings.

Check the default Info.plist location, but fall back to the build settings if it's not found.
Never parse the bundle identifier from the Xcode project file.  This is unsafe since there can be multiple bundle identifiers if the user adds another bundle to a project, like their own framework or a watch app, etc.

Related Issues
Fixes https://github.com/flutter/flutter/issues/46730
See https://github.com/flutter/flutter/issues/28895#issuecomment-552409176
Tests
I added the following tests:
project_test
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I signed the [CLA].
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read [Handling breaking changes].

[x] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

Bleeeeh parsing the project file is the only way to get the bundle identifier when not on macOS.
I guess we can fall back to that if all the other methods fail since I don't think it's actually used for anything on other platforms except display/parsing organization names.


vscode terminal error

Type: Bug
I used kotlin in vscode some time ago and everything was working fine and the code ran successfully. But today I opened vscode again to run the kotlin code and got the error
kotlinc : The term 'kotlinc' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I run the code using command in external cmd window and it runs successfully. But it always gives an error in vscode

I tried rolling back the vscode version to version 1.69.2 which was working fine some time ago, but the error persists.
After trying many times, I updated vscode version to 1.70 again. The code can run successfully again.
What is the problem?
VS Code version: Code 1.70.0 (da76f93349a72022ca4670c1b84860304616aaa2, 2022-08-04T04:38:16.462Z)
OS version: Windows_NT x64 10.0.17763
Modes:

System Info

Item
Value

CPUs
Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz (8 x 2600)

GPU Status
2d_canvas: unavailable_softwarecanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: disabled_softwaremultiple_raster_threads: enabled_onopengl: disabled_offrasterization: disabled_softwareraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: disabled_softwarevideo_encode: disabled_softwarevulkan: disabled_offwebgl: unavailable_softwarewebgl2: unavailable_software

Load (avg)
undefined

Memory (System)
16.00GB (2.62GB free)

Process Argv
--crash-reporter-id eff41e00-9cca-49eb-98d7-445f3a050d2c

Screen Reader
no

VM
0%

Extensions (32)

Extension
Author (truncated)
Version

vscode-markdownlint
Dav
0.47.0

code-runner
for
0.11.8

kotlin
fwc
0.2.26

vscode-sshfs
Kel
1.25.0

Kotlin
mat
1.7.1

dotenv
mik
1.0.1

vscode-docker
ms-
1.22.1

vscode-language-pack-zh-hans
MS-
1.70.8030922

python
ms-
2022.12.0

vscode-pylance
ms-
2022.8.10

jupyter
ms-
2022.7.1002181843

jupyter-keymap
ms-
1.0.0

jupyter-renderers
ms-
1.0.9

powershell
ms-
2022.7.2

indent-rainbow
ode
8.3.1

vscode-nuget-gallery
pat
0.0.24

material-icon-theme
PKi
4.19.0

fabric8-analytics
red
0.3.6

java
red
1.9.0

vscode-commons
red
0.0.6

vscode-xml
red
0.21.2022062916

my-code-actions
rio
1.0.0

vscode-pets
ton
1.11.0

vscodeintellicode
Vis
1.2.22

vscode-gradle
vsc
3.12.3

vscode-java-debug
vsc
0.43.0

vscode-java-dependency
vsc
0.21.0

vscode-java-pack
vsc
0.25.0

vscode-java-test
vsc
0.36.0

vscode-maven
vsc
0.37.0

vscode-spring-initializr
vsc
0.10.1

vscode-icons
vsc
11.15.0

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
vscscmwlcmt:30465135
cppdebug:30492333
pylanb8912cf:30529770
vsclangdc:30486549
c4g48928:30535728
hb751961:30541207
dsvsc012:30540252

This type of issue usually means that program was missing from your PATH. You can verify this if you can reproduce it again by echoing the value of that

Of course I know it's a problem with the path. But I can run successfully in external cmd window and fail in vscode. Does this have nothing to do with vscode?


[zh] fix contenst for zh-cn/2023-01-05-retroactive-default-storage-class.md

Like en fix docs.
raise a separate PR for the 'cn'..

/retitle fix contenst for zh-cn/2023-01-05-retroactive-default-storage-class.md

/retitle [zh] fix contenst for zh-cn/2023-01-05-retroactive-default-storage-class.md

mistake the md 's review..
Its not the view on github , but on the online-docs web .
close it..


Hiding the password details when we execute spark-sql

Hi Support,
I'm using a shell script to execute spark-sql
It look like this
spark-shell --num-executors 50 --master yarn --queue SVOC_ELT --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-<IP_ADDRESS>.1.5.0-152.jar --conf spark.security.credentials.hiveserver2.enabled=false --conf spark.sql.hive.hiveserver2.jdbc.url='jdbc:hive2://xxxx:2181,yyy:2181,zzz:2181/;ssl=true;serviceDiscoveryMode=zooKeeperHA;zooKeeperNamespace=hiveserver2-interactive;user=USER;password=PW'
once this get executed  it prompt for scala>
from
scala> spark.conf.get( "spark.sql.hive.hiveserver2.jdbc.url")
That return the jdbc url and it contain my password details , which is not correct.
Is there is any method to pass this credentials to jdbc url when I'm calling the script.
or hide this details
Tharanga

anyone can help on this ?

@tharanga2006  Please help with the link of document on which above document feedback request is created.
In case your query in not related to document content, open a new forum thread in MSDN forum or Stack Overflow as it could be better place to get help on your scenario. These forum community members could provide their expert guidance on your scenario based on their experience. Thanks.

@tharanga2006  Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.


Crashing luarocks-hererocks

When running :PackerInstall/Sync the luarocks-hererocks step fails due to a compilation error. It turns out that you must set the MACOSX_DEPLOYMENT_TARGET environment variable on macOS now in order for builds to go through for luajit. In my case I'm on macOS 12.4 and so I added export MACOSX_DEPLOYMENT_TARGET="12.4" to my .zshrc.  It would probably be nice if lvim set this env var or at least output something helpful when it fails to build.
Thanks!

See this issue https://github.com/LuaJIT/LuaJIT/issues/764

I'm not using MacOS.
I can add vim.fn.setenv("MACOSX_DEPLOYMENT_TARGET", "version")
When I run :lua vim.notify(vim.inspect(vim.loop.os_uname())) I  get info
{
  machine = "x86_64",
  release = "5.18.2-1-default",
  sysname = "Linux",
  version = "#1 SMP PREEMPT_DYNAMIC Tue Jun 7 06:08:54 UTC 2022 (695cfee)"
}

Can You help me? Run this command and tell me: release == 12.4 (your version)?
Thanks in advance!

Hello, I have Macbook Air M1, I get
{
  machine = "arm64",
  release = "21.6.0",
  sysname = "Darwin",
  version = "Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:35 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T8101"
}

How to install lvim? it's stuck at installing hererocks.

See comment above.

See comment above.

Mine says:
{
  machine = "x86_64",
  release = "21.6.0",
  sysname = "Darwin",
  version = "Darwin Kernel Version 21.6.0: Wed Aug 10 14:25:27 PDT 2022; root:xnu-8020.141.5~2/RELEASE_X86_64"
}

This is relevant on macos:
❯ sw_vers -productVersion
12.5.1

and even more on the nose on macos:
❯ MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion | awk -F \. {'print $1 "." $2}') echo $MACOSX_DEPLOYMENT_TARGET
12.5

export MACOSX_DEPLOYMENT_TARGET="12.4"
export MACOSX_DEPLOYMENT_TARGET="12.5.1" still doesn't help.

@aasutossh , try install neovim from source:
git clone https://github.com/neovim/neovim
cd neovim || return
make CMAKE_BUILD_TYPE=RelWithDebInfo
sudo make install

building neovim from source doesn't have any effect. it's same as before. stuck at installing hererocks...

Also, how do you clean (uninstall) neovim after being installed with sudo make install?


Integrate provider 'Encrypt' file info API

@claudiaHash Please integrate the new changes from Provider and Aquarius


Planned Maintenance Spec

Choose a PR Template
Switch to "Preview" on this description then select one of the choices below.
Click here to open a PR for a Data Plane API.
Click here to open a PR for a Control Plane (ARM) API.

Add a description using a template so that you can select the purpose of this pr and have the due diligence list section

If this is for a new vesrion then the first commit needs to be an exact copy of the previous API version. All new changes should only be added in the subsequent commits.
This allows the reviewer to get a clear understanding of the actual changes being introduced. With the way the PR is raised now, it is not possible for the reviewer to tell what the changes are. Please either abandon the PR and raise another one with the recommendation or create a new set of commits on this PR following the recommendation. If you are doing the later option please indicate which commit is the exact copy of the previous version.


Upgrade jzlib version

Motivation:
A new version of jzlib was released some time ago.
Modifications:
Upgrade to the newest jzlib version.
Result:
Using latest release.

@normanmaurer - Ship it.

Cherry-picked into 4.0 (08b4c7d6b5132d8da05fc8c3b470c219b09348cb). 4.1 and master are already using this version


[editor]: Support zip in preview

Please add support for .zip files to the editor preview.
Generated by adamraichu.zip-viewer

Duplicate of #46.

Sorry, but I don't plan to support previewing zip files inside of zip files. Instead, use the selective extraction feature to extract the zip you want to preview, then open it normally.


empty responses

Hi,
I use the API from an Azure Function (similar to AWS Lambda) to provide my travelling colleagues with some flight weather info before they drive to the airport.
However, like 4 out of 10 times the API comes back with an empty response and only after retrying several times I get an actual response with weather info.
As this is used as a Slackbot, retrying multiple times is obviously not great as it destroys user experience.
Invoke-RestMethod -Method Get -Uri "http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info" -Verbose
VERBOSE: GET http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info with 0-byte payload
VERBOSE: received 1114-byte response of content type application/json

This is a successful response, but then like 40% of all times I get a 0-byte response back.
Any idea as to why this happens or what I can do to prevent this?

The code infrastructure I made around the API was not designed to respond to 7K requests per minute. I have plans on how to fix it so that the number of timeout errors decrease. I am currently unable to apply them, but Azure support is currently working on it.

Cool, thanks. For now I'll go straight to https://aviationweather.gov then.
This morning (10PM UTC time) it took around 15 tries to get a valid response back.

Hi there. I have some good news. I've finally built and deployed a test
version of the new AVWX API. I'd like to hear what you think before going
through the process of replacing the old version.
It's currently sitting at http://avwx-api.azurewebsites.net/
The API is the only functioning part of the test. I've changed the API
structure slightly, but I hope it makes more sense than the previous one.
METAR with station:
http://avwx-api.azurewebsites.net/metar/KLEX
METAR with lat/lon:
http://avwx-api.azurewebsites.net/api/metar/23.45,-15.65
TAF with station:
http://avwx-api.azurewebsites.net/api/metar/KLEX
TAF with lat/lon
http://avwx-api.azurewebsites.net/api/taf/23.45,-15.65
"format" and "options" values can be URL parameters OR request headers. New
format defaults to JSON.
Additionally, I made it so that the original .php endpoints will still work
via routing even though there is no PHP code on the server anymore. It's
pure Python.
I've also open-sourced the code of the API itself on GitHub
https://github.com/flyinactor91/AVWX-API
Michael duPont
On Tue, Aug 2, 2016 at 9:33 PM davidobrien1985<EMAIL_ADDRESS>wrote:

Cool, thanks. For now I'll go straight to https://aviationweather.gov
then.
This morning (10PM UTC time) it took around 15 tries to get a valid
response back.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/flyinactor91/AVWX-Engine/issues/2#issuecomment-237105811,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACedjQFb1dTqtxZMmwhkssxwPse-phqrks5qb-_tgaJpZM4JaPo8
.

--
Michael duPont
mdupont.com

@flyinactor91 does that test API still work? I just saw your feedback and tried this:
Invoke-RestMethod -Method Get -Uri "http://avwx-api.azurewebsites.net/metar/eddk" -verbose
VERBOSE: GET http://avwx-api.azurewebsites.net/metar/eddk with 0-byte payload
Invoke-RestMethod :
404 Not Found
Not Found
The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri "http://avwx-api.azurewebsites.net ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Sorry. I forgot to put api in that specific link. This will work as do the
rest.
http://avwx-api.azurewebsites.net/api/metar/eddk
On Tue, Aug 23, 2016 at 8:22 PM davidobrien1985<EMAIL_ADDRESS>wrote:

@flyinactor91 https://github.com/flyinactor91 does that test API still
work? I just saw your feedback and tried this:
Invoke-RestMethod -Method Get -Uri "http://avwx-api.azurewebsites.net/metar/eddk" -verbose
VERBOSE: GET http://avwx-api.azurewebsites.net/metar/eddk with 0-byte payload
Invoke-RestMethod :
404 Not Found
Not Found
The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.
At line:1 char:1

Invoke-RestMethod -Method Get -Uri "http://avwx-api.azurewebsites.net ...

  + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/flyinactor91/AVWX-Engine/issues/2#issuecomment-241922872,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACedjWFpXpNHAXm7aHeHVAXu8rUJmMRCks5qi47WgaJpZM4JaPo8
.

--
Michael duPont
mdupont.com

Alright, it works. Thanks very much.
You could also maybe try running it as an Azure Function? Might make it a bit faster even?!
I've so far not seen any timeouts. :+1:

That’s something I’ve also considered (and may test in the future). For now, I like that Falsk’s routing let’s me serve the new API design while also satisfying the old php endpoints with the same code. The Azure API App setup let’s more test potential load, so I’ll switch it over once I know it can handle the load. This should scale out (rather than up) far more gracefully than the old one.

On Aug 23, 2016, at 8:54 PM, davidobrien1985<EMAIL_ADDRESS>wrote:
Alright, it works. Thanks very much.
You could also maybe try running it as an Azure Function? Might make it a bit faster even?!
I've so far not seen any timeouts. 👍
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/flyinactor91/AVWX-Engine/issues/2#issuecomment-241927393, or mute the thread https://github.com/notifications/unsubscribe-auth/ACedjTLG0DptbGoighnIBqo5LLe7XFKkks5qi5YygaJpZM4JaPo8.

The API has been updated to the new architecture. I've also released the full API code in a separate repository https://github.com/flyinactor91/AVWX-API


Upgrade to 0.63

Currently the app is built against RNW version 62.  Upgrade to 63.
Known issues:
-Will need to remove the manual linking of @react-native-communit/checkbox and letting auto-linking do its thing

github is weird.  closing


Prevent animation on first card click

Currently it animates to show the main card when you click on it after loading the page, even though it's already showing the main card.

That was really anoying, thanks!

I mean, I'd say that's better than a random animation happening out of nowhere 😛
Maybe it could be done better with transitions and transform properties rather than keyframes?


Feat: Implementing Firebase Analytics

Description:
Implementing firebase / analytics on ios and android, I created an event called Location and it will be logged every time a user press the location button in the settings, this event will have the following values:
{
    eventAction: 'press'
    region: 'button'
    value: true|false
}

Linked issues:

Feat: 265
How to test:

git pull && git checkout Feat/implementing-firebase-analytic
Run yarn install

Android

You have to download the google-service.json file in firebase or ask me for it
Place the file in the root of android/app/
Run cd android && ./gradlew assembleDebug && ./gradlew clean (OPTIONALLY) this is to clean gradlew
Run adb shell setprop debug.firebase.analytics.app **package_name** to  enable debug view and delete the period of time to log events (IF YOU DON'T DO THAT IT WILL LOG EVENTS EVERY 24 HOURS)
Run the app, and head over to settings and hit the location button
Go to firebase console and in analytics section hit debugView
You should see all the logs of the events and its value

Ios

You have to download the googleService-info.plist file in firebase or ask me for it
Place the file in the root of ios/
In Xcode, select Product -> Scheme -> EditScheme -> Run -> Select Arguments tab add this argument -FIRDebugEnabled to  enable debug view and delete the period of time to log events (IF YOU DON'T DO THAT IT WILL LOG EVENTS EVERY 24 HOURS)
Run the app, and head over to settings and hit the location button
Go to firebase console and in analytics section hit debugView
You should see all the logs of the events and its value

SOME TEST ARE GOING TO FAIL BECAUSE IT NEEDS TWO PRIVATE FILES THAT I CANNOT UPLOAD


Create Ian.md

Notes for Reviewers
This PR fixes #
Signed commits

[ ] Yes, I signed my commits.

🚀 Preview for commit 40eceeada6a9de3ea9daa2145dea2b4eeeef2e60 at: https://66ec254d1f336daf318a0b31--meshery-docs-preview.netlify.app

It's nice to have you participating on the Meshery project and in the Layer5 community. 🙂 We encourage all newcomers to star the Meshery repo. Have you added your star? ⭐

@marblom007 Yes, I have! thank you !


Not using set-cookie for refresh token

Is it intended or a bug that the JWT refresh token isn't being set using set-cookie, and instead just being sent with the response to /login (or refresh)? If it's intended use, I assume it would have to be managed by the client, which seems insecure.

In the LoginView class of this project you can see in the method get_response(self) that the refresh token is set as a cookie.
https://github.com/jazzband/dj-rest-auth/blob/06eeac95e0499640bf10f30a13fabd5ae33f73d8/dj_rest_auth/views.py#L123-L132
Of course you need to be sure to use correctly the settings of the package. In your case put REST_USE_JWT param to True, should work.
https://github.com/jazzband/dj-rest-auth/blob/06eeac95e0499640bf10f30a13fabd5ae33f73d8/dj_rest_auth/views.py#L105

In the LoginView class of this project you can see in the method get_response(self) that the refresh token is set as a cookie.
https://github.com/jazzband/dj-rest-auth/blob/06eeac95e0499640bf10f30a13fabd5ae33f73d8/dj_rest_auth/views.py#L123-L132
Of course you need to be sure to use correctly the settings of the package. In your case put REST_USE_JWT param to True, should work.
https://github.com/jazzband/dj-rest-auth/blob/06eeac95e0499640bf10f30a13fabd5ae33f73d8/dj_rest_auth/views.py#L105

@SpicyPaper I just looked into it again after some time. Obviously I had set REST_USE_JWT to true, but after looking into the code you mentioned I noticed the setting JWT_AUTH_REFRESH_COOKIE which doesn't seem to be mentioned in the docs, but is set to None when you don't set it. Since it wasn't mentioned in the docs, I hadn't set it, it was set to None and the refresh token wasn't set.