Added endpoint to render html content of an NDC file

Endpoint at: /ndcs/{country code}
uses redcarpet to transform markdown to html, with a customised renderer to ensure links to media files point to S3.
What is needed:

S3 configuration in .env
some country NDC to test with, which needs to be imported from S3 as described here #1

Looks good to me!
We can render this easily in the react app using dangerouslySetInnerHTML; I don't really know what steps we can take to make sure the markup is sane and avoid nasty injections. This might not be a concern at all if this data comes from a trusted source.

Data comes from scraped pdf files, so I guess it's safe from injections. Because the html is produced by a markdown parser, hopefully no strange html either.

Hi guys, whats the original format we get before turning it into html? html not really the best format to work with, as @ticklemynausea said we can use dangerouslySetInnerHTML which is discouraged, hence the name ^^

If you have a markdown already that would be a better format for the frontend.

I totally agree with @vectorsize, I prefer to get the .md and parse in the frontend using for example https://github.com/rexxars/react-markdown#notes (what doesn't use dangerouslySetInnerHTML) this makes the thing a lot more flexible.

We have the NDC content in both Markdown and HTML. They are static and won't be changing, and are from a trusted source. So the risk is virtually zero.
But we can discuss next week, it wouldn't be a big change here.

So apparently some of the content won't be available in .md, so it's going to have to be html sadly. I will simplify this endpoint so that it goes straight for the .html version from S3.

Multiple changes resulting from recent discussions & developments:

branch rebased with develop
path is now ndcs/:code/full
serves html straight from files provided from the client, since we won't have .md versions for all of the countries
uses Nokogiri to amend image paths (images in S3, previously that was done by the markdown renderer)
introduces more consistent naming for the respective controller, importer and rake task, to avoid confusion with NDC content a.k.a. NCD overview
also renamed content -> full_text in database table
importing rake task is now: rake ndcs:full:import

agreed, will fix - thanks!


Correction to Role Name

Question: How do I add a user to the Administrators group?
The Role is now called: "Api Management Service Contributor" instead of "Api Management Contributor".

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b45736d2-b0f4-05c3-8e2f-fde88f53b61a
Version Independent ID: f6c4d89a-1cc5-3e2a-6a3c-3b292cecead7
Content: Azure API Management FAQ
Content Source: articles/api-management/api-management-faq.md
Service: api-management
GitHub Login: @vladvino
Microsoft Alias: apimpm

@manuelmeyer1 Thanks for catching that!
@Potapy4 Thanks for working up the PR!
We will assign this issue to the content author for review and approving the PR.


Update api.rst

idk rst very well but i need to know where i can edit and make descriptions for the payloads and stuff
Summary
editing rst so i can autoclass this later and add onto the attributetable

Checklist

[ ] If code changes were made then they have been tested.

[ ] I have updated the documentation to reflect the changes.

[ ] This PR fixes an issue.
[ ] This PR adds something new (e.g. new method or parameters).
[ ] This PR is a breaking change (e.g. methods or parameters removed/renamed)
[x] This PR is not a code change (e.g. documentation, README, …)

That's not how you do that.


fix(tests): For DNS discovery only use a local DNS server

Description
Because the Node builder by default configures DNS discovery the tests
all do a lookup.  This went to the staging DNS server, but even so, we
hit 429 Too Many Requests on this sometimes.
Instead this configures a domain under .test..  This domain is
reserved for tests and any DNS server should return an emtpy response
without going through upstream DNS servers - unless the records were
directly added to the DNS server receiving the query in which case it
will respond.
So now all our default DNS discovery just doesn't discover anything,
which is fine as all tests have other mechanisms to find the nodes.
The tests that do use DNS discovery run their own local DNS server for
this, which is also switched to .test., because that's the right
domain to use for this.
Breaking Changes
None
Notes & open questions
I feel like we should also try this with the relay server.  That will
probably be a bit harder and it is not an issue for me right now.
I actually need this for the quinn11 branch, but the fix is so generic
that it makes more sense to do on main and merge it back.
Change checklist

[x] Self-review.
[x] Documentation updates following the style guide, if relevant.
[x] Tests if relevant.
[x] All breaking changes documented.

Okay. Yeah I think I see the reasoning.
The tests would just start sending requests to <node_id>.staging-dns.iroh.link, even though none of the tests would use a PkarrPublisher, so they would never actually be able to find anything there.
The only tests that were using a PkarrPublisher were running the publisher locally, using test_utils::dns_and_pkarr_servers::DnsPkarrServer, and were configuring it to be at "dns.example", even though ideally it'd be hosted on a .test. TLD, such that if we end up actually asking a real DNS server, it'd know not to recurse.
And the tests that do use the PkarrDnsServer would use that server directly for DNS resolving, instead of the current router/system/<IP_ADDRESS> or whatever.

Even if we don't depend on DNS discovery for our unit tests, wouldn't it be possible that someone who's developing something that uses iroh as a library, as well as a PkarrPublisher, end up suddenly not being able to connect?
You'd need to update the usages of N0_DNS_PKARR_RELAY_STAGING, too, right?

Right now we have tools (public!) in iroh_net::test_utils which let you easily spin up a test-local DNS/Pkarr server (and relay server).  We use this in the tests that do need this functionality.
I guess it is fair to point out that anyone else using this in tests will now by default black-hole this.  That needs to be documented clearly indeed.

The goal is to not hit any public infra for discovery during tests.  Unless on purpose.
@Arqu I'm not sure I fully understand your comment, but maybe the DnsPkarrServer from test_utils is what you're after?
I've added a bunch of documentation that was dearly missing anyway.  I hope this makes it clearer to users what is happening by default and how this changes during tests.
PTAL


Retrieval Example: Failed load from saved model

Hello,
I was going through the retrieval example and torch.load() throws an error in two_tower_retrieval.py:
RuntimeError: Need to initialize default process group using "init_process_group" before loading ShardedTensor
I initialize the process group as the error indicates and run into the following error on dmp.load_state_dict(retrieval_sd):
AttributeError: 'Tensor' object has no attribute 'local_shards'
To Reproduce
a. execute two_tower_train.py using command 2 in the retrieval example's README to get the retrieval model.
b. execute two_tower_retrieval.py using command 3.
Any help is appreciated!

Hey, the error may stem from trying to load an FSDP checkpoint from a non-FSDP model. I don't know why the second script would not initialize a distributed model since both the train and retrieval script seem to assume a distributed setup; maybe it's related to the example not using torchx to launch command 3.
I.e., maybe this would work as command 3:
torchx run -s local_cwd dist.ddp -j 1x2 --gpu 2 --script two_tower_retrieval.py --load_dir <path_to_save_shards>


Add New Paper

See a new paper posted on ASIA CCS' 2022, "MAB-Malware: A Reinforcement Learning Framework for Blackbox Generation of Adversarial Malware".
And it's code on https://github.com/weisong-ucr/MAB-malware

sorry, I see it has been posted on arxiv since 2020, and you have listed.

Thanks for the information!


set (legacy) swarm NetworkMode

We run into the problem that docker-machine configured swarm images did run within the bridge network (no specific mode was configured via the API call).
Usually you want those images to run within the host network.
To fix our situation the attached diff helped.
Would it make sense to make a option or similar to get it working for general use?
diff -U 6  a/machine/libmachine/provision/configure_swarm.go configure_swarm.go
--- b/machine/libmachine/provision/configure_swarm.go	2016-07-21 13:13:07.000000000 +0200
+++ configure_swarm.go	2016-08-03 11:21:57.000000000 +0200
@@ -90,12 +90,13 @@
 					{
 						HostIp:   "<IP_ADDRESS>",
 						HostPort: port,
 					},
 				},
 			},
+                        NetworkMode: "host",
 		}

 		swarmMasterConfig := &dockerclient.ContainerConfig{
 			Image: swarmOptions.Image,
 			Env:   swarmOptions.Env,
 			ExposedPorts: map[string]struct{}{
@@ -115,12 +116,13 @@
 	if swarmOptions.Agent {
 		workerHostConfig := dockerclient.HostConfig{
 			RestartPolicy: dockerclient.RestartPolicy{
 				Name:              "always",
 				MaximumRetryCount: 0,
 			},
+                        NetworkMode: "host",
 		}

 		cmdWorker := []string{
 			"join",
 			"--advertise",
 			advertiseInfo,

Why do you want them to run with --net host?

Why do you want them to run with --net host?

To run on localhost, like consul as swarm KV.


example/raspberrypi/relay /main.py - no state change

Hi
I have tried the example file as is
https://github.com/Wappsto/python-wappsto-iot/blob/main/example/raspberrypi/relay/main.py.
It do not, return any errors, but it do not change the state of the relay on/off. It do not change on the dashboard either.
The pin number is corret.
What am I missing in this file?

Hey @toms77.
It could be many things, so we can start with the basic.

Have you make sure that there are a ca.crt, client.key and a `client.crt´ in your config folder?
Are your relay_control function triggered when you toggle the switch on wappsto?

If the relay_control function is triggered, it will be something to do with the GPIO library/setup.
IF the relay_control function is not triggered, it is the WappstoIoT library/config, and we will need a log output. (Use the default python logging module. Can be a bit spamy, so using the RichHandler. Examples shown below.)

import logging
from rich.logging import RichHandler

# With Rich Handler.
logging.basicConfig(
  level=logging.DEBUG,
  format="%(asctime)s - %(name)s - %(message)s",
  handlers=[RichHandler()],
)
# Without Rich Handler.
logging.basicConfig(
  level=logging.DEBUG,
  format="%(asctime)s - %(name)s - %(message)s",
)

When you have setup the logging, started up the script and letting it connect & setup all the values, (the Spamy part). When it have settled, toggle the switch on Wappsto and see if there is received some data.
**(Before you post any log output, please remove your Devices IP (found in the start of the file), and replace all uuids with '***', can be done with: sed -E 's/([a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12})/***/')

Hey @tacozMacleo
Thanks, for the help with the logging, that went smooth. 👍
The problem seems to fall under part 2) I'm getting a value error.
ERROR    2024-01-03 11:12:17,723 - wappstoiot.modules.value -     value.py:563                              OnChange callback error.                              Traceback (most recent call last):                                File                              "/home/toms/.local/lib/python3.9/site-packages/wappstoio                              t/modules/value.py", line 561, in _cb_float                                  callback(self, data)                                File "/home/toms/FlowerPower/main.py", line 211, in                              relay_control                                  GPIO.output(water_pump_pin, new_value)                              ValueError: Value must be an integer/boolean or a                              list/tuple of integers/booleans
The template settings: (as the example)
permission=wappstoiot.PermissionType.READWRITE,
        value_template=wappstoiot.ValueTemplate.BOOLEAN_ONOFF

The pin name is just changed:
water_pump_pin = 5
Any sugestings to why it rapports a ValueError??

Aaarh.
The RPi.GPIO says that it need a integer or a boolean. The new_value are a float or a string. So it need to be converted to a int. So replace: GPIO.output(relay_pin, new_value) with: GPIO.output(relay_pin, int(new_value)) and it should work.
Do you mind add: print(type(new_value) in the relay_control and give me that print line, to see if it is WappstoIoT that give you wrong type, or it is RPi.GPIO that is pedantic about integer or float?
Thanks.

Sure thing
new_value  = <class 'float'>
This must come from the mapping in valuetemplate?

Thanks, It seems that it is RPi.GPIO that is more pedantic about it. I will fix the example code. And make it as resolved.
Yes, the Wappsto.ValueTemplate.BOOLEAN_ONOFF given the config for a number value, so that is set up behind the screens.
Wappsto.ValueTemplate have a long list of default templates. If none of them fits your needs, you can always setup your own with: device.createStringValue or device.createNumberValue.
If your IntelliSense do not show you them, you can find a list in: wappsto/iot/modules/template.py

Thanks, I have acutely been looking for an example off how to create your own valuetemplate and been looking at the template fil  a lot. It would have been nice with the "make you own template" recipe in the comments of that template file.
Thanks for the help.


[Protected Request] page 6550 "Whse. Item Tracking Lines"

Hello,
Is it possible to get these variables as protected var in page 6550 "Whse. Item Tracking Lines"?
    ColorOfQuantityArray: array[2] of Integer;
    UndefinedQtyArray: array[2] of Decimal;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Release 2442 distribution

Why
In order to deploy new features, improvements and bug fixes, we need to release a new distribution.
What
Create and release a new distribution 2442

[!IMPORTANT]

[x] Ship new era
[x] :fire: Breaking-change in client for Mithril stake distribution protocol message computation
[x] Make client for Cardano transactions stable
[x]  #2003
[ ] Activate Cardano stake distribution certification:

[ ] in pre-release-preview
[ ] in release-preprod
[ ] in release-mainnet

[ ] Upgrade to Cardano node 9.2.1:

[ ] in pre-release-preview
[ ] in release-preprod
[ ] in release-mainnet

Distribution +1:

[!NOTE]

[ ] Make client for Cardano stake distribution stable

How
Phase 1: Prepare distribution

[x] Test backward compatibility with previous signer version(s)
[x] Upgrade repository dependencies (with runbook)
[x] Check that all minor versions of all crates have been bumped for breaking change (if needed)
[x] Check if versions of repository linked crates are correct for publication to crates.io(mithril-common, mithril-client)
[x] Check if Cardano minimum supported versions in networks.json must be updated following the runbook
[x] Merge a PR for rotating the CHANGELOG.md (set the distribution version)
[x] Create a draft PR for rotating the documentation

Phase 2: Rollout pre-release distribution

[x] Update the configuration of pre-release-preview:

[x] Upgrade the Cardano node version to 9.2.1
[x] Activate the certification of the Cardano Stake Distribution

[x] Tag and deploy a 2442.0-pre pre-release distribution on the pre-release-preview network
[x] Fill 'Highlights' and release notes section in the pre-release page on GitHub
[x] Make sure pre-release-preview is working as expected
[x] Make sure libraries version next have been published on npm registry
[ ] Run an all green client multi-platform test in the CI
[x] Run an all green aggregator stress test in the CI
[ ] Run a Cardano transaction prover benchmark:

[ ] Store results in the issue
[ ] Make sure that there is no performance drop since last release

[ ] Announce the pre-release on discord channel
[x] Announce any breaking change in:

[x] #cardano-scaling slack channel for pre-release
[x] #mithril-lace slack channel for pre-release
[x] #blockfrost-scaling slack channel for pre-release

[ ] Wait for at least 3 signers running the new version (could be more if era change is tested) over at least 2 epochs

Phase 3: Rollout release distribution

[ ] Tag and deploy a 2442.0 release distribution on the pre-release-preview network
[ ] Fill highlights and release notes section in the release page on GitHub
[ ] Make sure pre-release-preview is working as expected
[ ] Update networks compatibility table in release notes
[ ] Update the configuration of release-preprod:

[ ] Upgrade the Cardano node version to 9.2.1 in release-preprod
[ ] Activate the certification of the Cardano Stake Distribution

[ ] Update the configuration of release-mainnet:

[ ] Upgrade the Cardano node version to 9.2.1 in release-mainnet

[ ] Release the 2442.0 distribution
[ ] Make sure release-mainnet is working as expected
[ ] Make sure release-preprod is working as expected
[ ] Make sure libraries have been published on crates.io
[ ] Make sure libraries version latest have been published on npm registry
[ ] Merge the PR for rotating the documentation
[ ] Merge a PR for rotating the CHANGELOG.md (updating the release date of the distribution)
[ ] Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
[ ] Announce any breaking/new feature change in:

[ ] #cardano-scaling slack channel for release
[ ] #mithril-lace slack channel for release
[ ] #blockfrost-scaling slack channel for release

[ ] Dev blog post:

[ ] Activation of the Cardano Stake Distribution certification in pre-release-preview and release-preprod

Phase 4: Activate features

[ ] Cardano stake distribution:

[ ] When signer release >= 0.2.182 (2437.1) is run by > 95% of the stake (within 2 weeks, or abort),
[ ] Activate the certification of Cardano stake distribution in release-mainnet
[ ] Restart the release-mainnet deployment in the CI
[ ] Make sure release-mainnet is working as expected
[ ] Announce the release on discord channel
[ ] Dev blog post: Activation of Cardano stake distribution in release-mainnet

Cardano transactions prover benchmark

[!NOTE]
run on GCP with e2-highmem-2 machine with 2 vCPUs and 15GB of RAM
not during Cardano database snapshotting

Benchmark results:

total_requests
concurrency
transactions/request
requests/s

500
50
1
56.11

500
50
10
41.44

500
50
20
33.54

500
50
30
30.91

500
50
40
27.23

500
50
50
24.67

vs previous results

total_requests
concurrency
transactions/request
requests/s

500
50
1
58.71

500
50
10
49.19

500
50
20
39.17

500
50
30
34.11

500
50
40
29.21

500
50
50
24.39

Command run:

# Aggregator endpoint
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator

# Transactions file to prove
export TRANSACTIONS_FILE=transactions-preview.txt

# Transactions proved per request range definition
export TRANSACTIONS_PER_REQUEST_MIN=0
export TRANSACTIONS_PER_REQUEST_MAX=50
export TRANSACTIONS_PER_REQUEST_STEP=10

# Apache benchmark total request sent per benchmark
export AB_TOTAL_REQUESTS=500

# Apache benchmark concurrency level range definition
export AB_CONCURRENCY_MIN=50
export AB_CONCURRENCY_MAX=50
export AB_CONCURRENCY_STEP=50

# Run the benchmark:
./benchmark-aggregator-prover.sh

Command result:

MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK

>> Aggregator endpoint: https://aggregator.pre-release-preview.api.mithril.network/aggregator
>> Aggregator route: /proof/cardano-transaction
>> Transactions file: transactions-preview.txt
>> Transactions available: [100]
>> Transactions per request range: [0 10 20 30 40 50]
>> AB concurrency range: [50]
>> AB total requests per run: [500]
>> AB total runs: 6
>> Output file: benchmark.csv

>> [#1/6] Running stress test with 500 requests with 1 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (56.11 requests/s)

>> [#2/6] Running stress test with 500 requests with 10 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (41.44 requests/s)

>> [#3/6] Running stress test with 500 requests with 20 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (33.54 requests/s)

>> [#4/6] Running stress test with 500 requests with 30 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (30.91 requests/s)

>> [#5/6] Running stress test with 500 requests with 40 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (27.23 requests/s)

>> [#6/6] Running stress test with 500 requests with 50 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (24.67 requests/s)

>> Benchmark completed:

total_requests,concurrency,transactions/request,requests/s
500,50,1,56.11
500,50,10,41.44
500,50,20,33.54
500,50,30,30.91
500,50,40,27.23
500,50,50,24.67


Add new Detection as Code blog

Good content included in the blog added by David French on how to create a new detection as code pipeline using common or free tooling.

Thanks for the add @tuckner. This is a great blog post btw.


TypeError: LoraConfig.init() got an unexpected keyword argument 'eva_config'

System Info
Name: Peft
Who can help?
No response
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder
[x] My own task or dataset (give details below)

Reproduction
1 model_name = "Mohan-08/math-dataset-deepmind-FT"
2 tokenizer = AutoTokenizer.from_pretrained(model_name)
----> 3 model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", trust_remote_code=True)
Expected behavior
Facing this error:
TypeError: LoraConfig.init() got an unexpected keyword argument 'eva_config'

This means that the model was trained with a more recent PEFT version than what you're using. Either upgrade to the latest PEFT version or manually edit the adapter_config.json and remove the entry:
https://huggingface.co/Mohan-08/math-dataset-deepmind-FT/blob/main/adapter_config.json#L6

This is my fine turned model:
model_name = "Mohan-08/math-dataset-deepmind-FT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device)

but this is model is behaving like the base model like gemma. what was the problem with this.
I have files like this

when i load my model also, it is loading the gemma weights!

Could you show me the code you used that results in the same results as the base model? When I tested it, I get different logits from the base model vs the fine tuned model.

My doubt was, Fine Turned model also loads the same weights? but the output seems different

The fine-tuned model will load the base weights and then add the LoRA weights on top, so this is expected. As you mentioned, the outputs are different, so I think everything works as expected.

This may be because of you trying to use the old version of peft library with new version chechpoint. I solved that with just removing the eva config
There how you can do that
import shutil
import os
import json
from peft import LoraConfig

# Define the path to the adapter_config.json file
adapter_config_path = f"{cfg.lora_dir}/adapter_config.json"

# Step 1: Read the adapter_config.json file
with open(adapter_config_path, 'r') as file:
    adapter_config = json.load(file)

# Step 2: Remove the eva_config key if it exists
adapter_config.pop('eva_config', None)
adapter_config.pop('exclude_modules', None)
adapter_config.pop('lora_bias', None)

# Step 3: Create the checkpoint_save_dir folder if it doesn't exist
checkpoint_save_dir = cfg.lora_dir.split("/")[-1]
if not os.path.exists(checkpoint_save_dir):
    os.makedirs(checkpoint_save_dir)

# Step 4: Define the path to save the modified adapter_config.json
checkpoint_config_save_path = f'{checkpoint_save_dir}/adapter_config.json'

# Step 5: Save the changes back to the adapter_config.json file
with open(checkpoint_config_save_path, 'w') as file:
    json.dump(adapter_config, file, indent=4)

for file_name in os.listdir(cfg.lora_dir):
    source_file_path = os.path.join(cfg.lora_dir, file_name)
    destination_file_path = os.path.join(checkpoint_save_dir, file_name)

    # Skip copying adapter_config.json since we already modified and saved it
    if file_name != "adapter_config.json":
        if os.path.isfile(source_file_path):
            shutil.copy2(source_file_path, destination_file_path)
        elif os.path.isdir(source_file_path):
            shutil.copytree(source_file_path, destination_file_path)

# Step 6: Load the LoraConfig using the modified configuration
lora_config = LoraConfig.from_pretrained(checkpoint_save_dir)
print(lora_config)

model_0 = PeftModel.from_pretrained(model_0, checkpoint_save_dir)
model_1 = PeftModel.from_pretrained(model_1, checkpoint_save_dir)

This means that the model was trained with a more recent PEFT version than what you're using. Either upgrade to the latest PEFT version or manually edit the adapter_config.json and remove the entry:
https://huggingface.co/Mohan-08/math-dataset-deepmind-FT/blob/main/adapter_config.json#L6

awesome man~ I merged original model weights with adapter successfully.

Thank you, Issue fixed.


[SPARK-17672] Spark 2.0 history server web Ui takes too long for a single application

What changes were proposed in this pull request?
Added a new API getApplicationInfo(appId: String) in class ApplicationHistoryProvider and class SparkUI to get app info. In this change, FsHistoryProvider can directly fetch one app info in O(1) time complexity compared to O(n) before the change which used an Iterator.find() interface.
Both ApplicationCache and OneApplicationResource classes adopt this new api.
How was this patch tested?
manual tests

Can one of the admins verify this patch?

LGTM

ok to test

Test build #66004 has started for PR 15247 at commit 9feb812.

Test build #66004 has finished for PR 15247 at commit 9feb812.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66004/
Test FAILed.

Test build #66013 has started for PR 15247 at commit 7e437a1.

Test build #66013 has finished for PR 15247 at commit 7e437a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66013/
Test PASSed.

One minor comment otherwise looks ok.

Test build #66050 has started for PR 15247 at commit 67ea2de.

Test build #66050 has finished for PR 15247 at commit 67ea2de.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66050/
Test FAILed.

Merged build finished. Test FAILed.

Anyone knows why the last test failed in the sql module? My change has nothing to do with it.

Jenkins, retest this please

Test build #66059 has started for PR 15247 at commit 67ea2de.

Test build #66059 has finished for PR 15247 at commit 67ea2de.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66059/
Test FAILed.

Another test failed in the mllib: org.apache.spark.mllib.classification.NaiveBayesSuite.Naive Bayes Multinomial
@ajbozarth  Have you seen this kind of unrelated failure before? Do I have the permission to trigger the test? Thanks!

Jenkins, retest this please

retest this please

Test build #66110 has started for PR 15247 at commit 67ea2de.

Test build #66110 has finished for PR 15247 at commit 67ea2de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66110/
Test PASSed.

LGTM merging into master 2.0, thanks.


Upload endpoint

Upload simulation via POST /simulation
Generates nod.xml, edg.xml, con.xml and rou.xml from the network request body
Runs netconvert to generate net.xml file
Creates sumocfg file

Docs: http://localhost:8080/swagger-ui.html (start the server first!)

Also improved README and fix missing library in docker


Tag filter for ~e621

The fork my Discord server uses has a tag filter for the e621 command that blocks anything with tags that are against the rules or do not show up in the embed, like flash files.
This adds filtering without adding to the tags that the bot has to search, as its limited to only six by the site.
A feature request probably needs to be filed for this, but it would be nice to see someone improve and expand this to include more NSFW searches and have the tags configurable. As of right now I'm just keeping the fork up to date with public NadekoBot.

Thanks for showing interest in contributing <3
However, huge changes to the code are currently occurring in the 1.4 branch which are not yet pushed as I'm fixing the last few things. Another thing is that if this was done, it would have to be configurable on the module level (for all commands), not just for a single one, and not hard-coded but stored in the database.
This has been requested multiple times, and I'll consider it sometime, but atm it's very low priority.


Change to account for initial storage units and capacity energy storage in the multi-year feature

Pull request details
Describe the changes made in this pull request
These changes account for the modifications of the input data files to include the investment in energy storage as part of the multi-year investment feature.
List of related issues or pull requests
Part of #806
Collaboration confirmation
As a contributor I confirm

[ ] I read and followed the instructions in README.dev.md
[ ] The documentation is up to date with the changes introduced in this Pull Request (or NA)
[ ] Tests are passing
[ ] Lint is passing

@gnawin @abelsiqueira These changes are only part of modifying the input data to make the code changes accordingly with the multi-year investment and storage. I prefer to do it step by step so it is easy to review. After this change in the files, I will send another PR with the changes for the multi-year.


Recommended way to pass/use parameters to DataTable class?

What is the recommended way to pass parameters to a class that extends DataTable?
For example, let's say I want to add where to the query.
Controller:
public function index(UsersDataTable $dataTable)
{
    return $dataTable->render('users.index');
}

DataTable class:
class UsersDataTable extends DataTable
{
    public function dataTable($query)
    {
        return datatables()
            ->eloquent($query)
            ->addColumn('action', 'users.action');
    }

    public function query(User $model)
    {
        return $model->newQuery();
    }

Where would I add, say, ->where('role', 'Admin') to the call inside the controller, and how do I implement it properly in the class?

Several ways:
If role is from request, I usually add it on the query.
    public function query(User $model)
    {
        $query = $model->newQuery()
        if (request('role')) {
            $query->where('role',  request('role'));
        }

        return $query;
    }

If it's a fixed scope and you want the users dataTable to be reusable, you can use Scope class. See https://yajrabox.com/docs/laravel-datatables/master/buttons-console for ref but sorry since it's not updated but should work.
public function index(UsersDataTable $dataTable)
{
    return $dataTable->addScope(new AdminRoleScope)->render('users.index');
}


Spacy V3 decorator string name

Hello guys,
With the V3 update when I run the example code it complains:
ValueError: [E966] `nlp.add_pipe` now takes the string name of the registered component factory, not a callable component. Expected string, but got <spacy_cld.spacy_cld.LanguageDetector object at 0x7fb8d9051ed0> (name: 'None').

- If you created your component with `nlp.create_pipe('name')`: remove nlp.create_pipe and call `nlp.add_pipe('name')` instead.

- If you passed in a component like `TextCategorizer()`: call `nlp.add_pipe` with the string name instead, e.g. `nlp.add_pipe('textcat')`.

- If you're using a custom component: Add the decorator<EMAIL_ADDRESS>(for function components) or<EMAIL_ADDRESS>(for class components / factories) to your custom component and assign it a name, e.g. `@Language.component('your_name')`. You can then run `nlp.add_pipe('your_name')` to add it to the pipeline.

I figured out that now we have to pass the string name, to nlp.add_pipe but how?
I've tried nlp.add_pipe("langdetect"), nlp.add_pipe("LanguageDetector"),nlp.add_pipe("languagedetector") and none of them seems to work.
Can you help me with this ?

Hi,
Since I'm new to SpaCy and Python, I'm not sure if this is the correct way to implement it. For Python 3.9 with SpaCy 3.0.3 the following worked for me:
import spacy
from spacy.language import Language
from spacy_langdetect import LanguageDetector

# Add LanguageDetector and assign it a string name
@Language.factory("language_detector")
def create_language_detector(nlp, name):
    return LanguageDetector(language_detection_function=None)

# Load the model
nlp = spacy.load("en_core_web_sm")

# Add components using their string names
nlp.add_pipe("language_detector")

# Analyze components and their attributes
text = "This is an English text."
doc = nlp(text)

# Document level language detection.
print(doc._.language)

# See what happened to the pipes
nlp.analyze_pipes(pretty=True)`

Is this the right way to use it with SpaCy3?
Regards,
Cusard

same problem

Hello everybody!
Thanks to @Cusard I got the example code to work with the current spacy version.
import spacy
from spacy.language import Language
from spacy_langdetect import LanguageDetector

@Language.factory("language_detector")
def create_language_detector(nlp, name):
    return LanguageDetector(language_detection_function=None)

nlp = spacy.load("en_core_web_sm")

nlp.add_pipe('language_detector')
text = 'This is an english text.'
doc = nlp(text)
# document level language detection. Think of it like average language of the document!
print(doc._.language)
# sentence level language detection
for sent in doc.sents:
   print(sent, sent._.language)

The output looks like this:
{'language': 'en', 'score': 0.9999983570159962}
This is an english text. {'language': 'en', 'score': 0.9999956329695125}

Thanks for sharing the solution. It worked for me too.
It will be nice if the project home page had the example update: https://spacy.io/universe/project/spacy-langdetect

The example provided by @FelixSiegfriedRiedel  works for me with v3.3.
I've also raised an issue about updating the documentation:


During training setup: Error. "sample_training_data/" does not contain files

Repro steps, all per training/README.md:

Clone the repo
cd training
pip install -r requirements.txt
./setup_max_model_training max-text-classifier-training-config.yaml
Follow instructions to step, "Bucket is empty. Enter 'Y' to upload data from local directory or press 'ENTER' if you don't want to upload:"
Press y, <enter> (to use sample_training_data/)
'[MESSAGE] Error. "sample_training_data/" does not contain files' appears

> cd sample_training_data
> ll
total 88K
-rw-r--r-- 1 randy randy 83K Nov  9 12:19 ibm_claim_stance_dataset_reformatted.tsv
-rw-r--r-- 1 randy randy 797 Nov  9 12:19 README.md

This is with python 3.6.8 on Ubuntu 18.04.
I'm guessing that I can manually upload the data later with the same effect (haven't tried yet), but this is obviously not intended behavior. Additionally, note that it breaks the setup script, which is at this point stuck in a loop prompting for a data directory path but not accepting any paths as valid.

@thisisrandy We'll publish a fix later today. Late last week we've pushed a change to the training framework package that expects all data files to reside in subdirectory named  data/. If you move ibm_claim_stance_dataset_reformatted.tsv

The issue should now  be resolved. If you clone this repository again and install the training/requirements.txt model training should work as expected. The latest version of the max-training-framework package expects data files to reside in a subdirectory named data. This model was still using the old directory layout and setup therefore couldn't find the tsv file.

Confirmed fixed. Thanks for the quick reply!


I got error that is mean "too large to enable inline code ...."

Hello~
I got an issue when I upload Node.js release(latest version) file
The deployment package of your Lambda function "FirstLambda" is too large to enable inline code editing. However, you can still invoke your function.
I did all in the tutorial video but I got an error
Please help me~~

Hi, I got the same error, did you find a solution? @josephmjjo ?


Pause Stablequant

After economic issues, StableQuant was unable to continue the project. There is no security risk to funds to warrant a panic, but deposits should be paused.

😢


Open pdf view into a new tab

In Resources, when a teacher clicks on the Download Resource open the preview page in a new tab, rather than in the existing one as they may not know how to navigate back to the CSUnplugged site.

I need to look into https://docs.djangoproject.com/en/1.8/ref/request-response/#fileresponse-objects


DIGITS/examples/semantic-segmentation/ net output

Hi,
After training the net from the example (DIGITS/examples/semantic-segmentation) and running a test on an image in DIGITS resulting in some segmentation result, I'm trying to visualize the network output in python to prepare the framework for net deployment but the labels I'm getting are all mixed up spatially.
Am I reading the network output incorrectly?
Currently using:
# **im** is a preloaded image
# prepare input
net.blobs['data'].reshape(1, 3, im.shape[0], im.shape[1])
net.reshape()
transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
transformer.set_transpose('data', (2,0,1))
transformer.set_channel_swap('data', (2,1,0))
transformer.set_raw_scale('data', 255.0)
# classify
out = net.forward(data=np.asarray([transformer.preprocess('data', im)]))
# figure out label from score layer
label_raw = out['score'].argmax(axis=1)
# color output
lut = np.zeros((256, 1, 3), dtype=np.uint8)
lut[1,0,0] = 255
lut[1,0,1] = 255
lut[1,0,2] = 255

lut[2,0,0] = 0
lut[2,0,1] = 0
lut[2,0,2] = 255

lut[3,0,0] = 0
lut[3,0,1] = 255
lut[3,0,2] = 0

lut[4,0,0] = 0
lut[4,0,1] = 255
lut[4,0,2] = 255

label_raw = cv2.cvtColor(label_raw, cv2.COLOR_GRAY2BGR);
label = cv2.LUT(label_raw, lut)
# save image
cv2.imwrite(os.path.join(OUT_DIR,str(project_id) + "_" + str(image_id) + "_img" + ".png"), label)

Protip: use "```py" for syntax highlighting (I've updated your post)

Figured it out.
the following lines were not needed in my case:
    transformer.set_channel_swap('data', (2,1,0))
    transformer.set_raw_scale('data', 255.0)


Updated interactive data wrangling with Spark doc.

Added a step for session configuration that was missing in the previous documentation.

Learn Build status updates of commit b47e1c3:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: GitBranchDeletedOrForcePushed]

Cannot sync git repo to specified commit because ref +refs/pull/101215/merge has been deleted or has been force pushed.
 If this is pull request, please ensure it is mergeable (no merge conflict) and open (not closed).

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Learn Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Do not catch errors during callback dispatching

Here you throw the error is something went wrong. But actually error is being caught and process will keep going and fail later.
I encountered with it when tried to run install command on package.json with links on GitHub. I expected to get a proper error message, but just got:
TypeError: Cannot read property 'dist' of null
    at expose (/Users/just-boris/coding/ied/lib/expose.js:48:31)
    at /Users/just-boris/coding/ied/node_modules/async/lib/async.js:356:13

With this fix, it will be ok
Error: No satisfying target found for redux-router@github:just-boris/redux-router
    at /Users/just-boris/coding/ied/lib/resolve.js:84:17
    at resolvePending (/Users/just-boris/coding/ied/lib/resolve.js:15:25)

Yup. Yup. Looks good.


LPC845-BRK: Use hardware SPI and TMS RLE mode as JTAG output

Use #4 so that we can bit bash TMS, while TDI and TCK are controlled over hardware SPI to achieve high speeds (probably faster than the serial port can read in, to be honest).

Technically, since the LPC845 has dual SPI ports, I could actually just put one into master mode, use SCLK (as TCK) and MOSI (as TDI), feed SCLK into the second SPI port, and have its MISO (as TMS), and have TDO fed into MISO of the first SPI port.  It would be hilarious.

So manually toggled a GPIO pin to act as TMS, and did a simple RLE detection to use LPC845's SPI hardware to output TCK/TDI and clock in TDO.
It worked - but then I found that it would seem that transferring the data back via the virtual serial port sometimes drops data.  Slowing down the data rate (either by manually adding a 10-20ms pause between successive bytes while keeping SPI at max speed, or slowing the SPI port down dramatically) makes the com port send data back.
Now, is this because Win32 has a small receive buffer, and we need to increase the receive buffer size, or the CMSIS-DAP debug probe drops it?
I don't know.  But I think that I won't chase this much further - going to try develop on the STM32 board that I've got.  If this also happens on that board, then we'll look into increasing buffer sizes (as part of #2).


Change progress bar logger to print all eval metrics

What does this PR do?
Make the progress bar logger print all eval metrics instead of just accuracy.
What issue(s) does this change relate to?
CO-1857

sample progress bar output:


Hash

@jouni-kantola does this PR preplace https://github.com/davidxcheng/ruin/pull/4 ?

Yeah, I guess. PR #5 has one more commit. I haven't tried what happens if they are merged one after another. I guess it's worth trying to find out :information_source:

Ok, I'll merge both


duplicate uniquekey not support

how to build sql :on duplicate key update ....

It's recommended to use Build to implement it.
ib := sqlbuilder.NewInsertBuilder()
ib.InsertInto("demo.user")
ib.Cols("id", "name", "status", "created_at")
ib.Values(1, "Huan Du", 1, Raw("UNIX_TIMESTAMP(NOW())"))
ib.Values(2, "Charmy Liu", 1,<PHONE_NUMBER>)

builder := sqlbuilder.Build("%? ON DUPLICATED KEY UPDATE status = $?", ib, 1)
sql, args := builder.Build()
// use sql and args.

Feel free to reopen this issue if you still have any question.


Cross-framework communication?

Describe the bug
Our code base is evolving in a way where the part which acts as a client with regard to H.Ipc is built against .Net 8.0, while the part acting as server is on .Net Framework 4.8. This upgrade to .Net 8 on one side seems to affect the ability to send messages to the server, i.e. the client seems to go into a non-responsive state, and the server never seems to receive the message.
The processes are able to connect (on the server side we get the ClientConnected event, but no messages.
Any idea if this kind of "cross-framework" communication should be expected to work? Or what could be possible causes of this "lack of communication"?
Steps to reproduce the bug

Create a solution with three projects: Client, Protocol, and Server
Client should target .Net8.0, Protocol should target .NetStandard2.0, Server should target .Net Framework 4.8
Define message interface and implementations in Protocol project
Create Server code to handle messages in Server project
Create Client code to create and send messages in Client project
Run each of the Server and Client projects in separate terminal windows, and notice that Server never receives messages sent from Client.
Make Server project target .Net 8.0, build & run, and observe that messages are passed as expected.

Expected behavior
Ability to pass messages from a client targeting .Net 8 to a server targeting .Net Framework 4.8 - and vice versa.
Screenshots
No response
NuGet package version
H.Ipc 1.0.1
Platform
WPF, Console
IDE
Other
Additional context
No response

Yes, this problem has already arisen and I did not find a quick solution. Internally the library uses System.IO.Pipes.NamedPipeServerStream, and here different things are possible:

.Net Framework and .Net 8 are binary incompatible and there is no way to fix it
If we play with the settings, maybe we can make it compatible

But it requires time for testing/setting up the environment. If you can do it, I could implement it in the library.
Or I am available on a paid basis on weekends with a rate of $50 (only for Open-Source projects)

I looked a bit further into the pipes being set up for my client and server, respectively, as I noticed that there are some conditional formatter project/package references on the underlying H.Pipes project (depending on target framework).
By default it seems that the pipe on my client (net8.0) uses MessagePackFormatter, while my server (net48) uses BinaryFormatter.
I thought I would look into configuring the pipes to use the same type of formatter, if possible. But your response does not raise my hopes of success too much... 😄

Excellent! It's strange that I remembered that there was no solution here.

Yeah, I could easily imagine that a binary formatter could be incompatible across frameworks. Good thing that you made alternatives available here. 👍


DRAFT: Predicted and generalized explosives

There are still some issues, we should setup projectile prediction, and then setup some structure for a generalized explosion since there is a lot of repeated code for mines & dynamite otherwise. Let's chat about this internally.

We'll do prefab based instead of this


Registry API: Return the payload data directly

Currently, we return a RegistryPayload where the signature is always init.
This was a necessary change from the original design as zone transfer over
DNS does not allow to send those custom records.
We can just return a RegistryPayloadData directly and save ourselve some trouble.

That'd be even better ! I was just writing some tests for a bug fix and got annoyed, figured the fix was easy and non-controversial.


Create minimal working example.

It would be great to add a reference to a minimal working example.
Originally posted by @matthiasgeihs in https://github.com/perun-network/erdstall-ts-sdk/pull/11#r665165749

Done in tutorial.


Hot Exit: promise handling is often bad

I found many places where a method is called that actually returns a promise. Without adding a done handler you swallow possible errors and also make it very hard to understand that a method call is a long running operation.
A typical pattern we use is method().done(null, errors.onUnhandledException).
Examples: untitledEditorModel

@Tyriar fyi only found some in backupModelTracker.ts.

Thanks, I had a look at this but wasn't sure which were relevant now. I reviewed the commit for future reference :smile:


Static imports to the same package gets removed by CodeGen

If you import a method statically from a class located in the same package, it gets removed by CodeGen.

Might be related to #95 and #96.


Action to wipe all buffers but selected?

I am attempting to write an action for the buffers provider to delete all buffers except those selected. Kinda like the inverse of CTRL+X. My action is provided with the selected buffers, which I can get bufnrs from. Is there a way for me to also get all of the buffers displayed by fzf-lua? fzf-lua is doing a lot of filtering that I'd rather not repeat.
I've been digging through the source code, and i can't quite figure out how to get to this programatically.
Thanks for your incredible work, I love the plugin!

Update to https://github.com/ibhagwan/fzf-lua/commit/6363b14efe02d2fbb42858132606b24e15b5a78e first.
Requires fzf version 0.36 and above
require("fzf-lua").setup({
  buffers = {
    actions = {
      ["ctrl-x"] = { fn = require("fzf-lua").actions.buf_del, reload = true, prefix = "toggle-all+" }
    },
  },
})

The prefix toggle-all+ is taken from fzf available actions, to see the full list man fzf and search for AVAILABLE ACTIONS.

Thanks for your incredible work, I love the plugin!

Ty <3

🤯  Ooh, really nice solution!  Worked perfectly.

🤯 Ooh, really nice solution! Invert the selection, and then call the built-in buf_del. Worked perfectly.

Ty @tradiff!
Edited the title so it's easier for other people to find, since this is yet another obscure undocumented option :-)

My full config for this if anyone else wants to do something similar:
        buffers    = {
          no_header = true,
          fzf_opts = {
            ['--header'] = [['<ctrl-x> delete unselected']],
          },
          actions = {
            -- delete the unselected buffers
            ['ctrl-x'] = { fn = fzflua.actions.buf_del, reload = true, prefix = 'toggle-all+' }
          }
        }

Permalink
https://github.com/tradiff/dotfiles/commit/a6229ffc4da0c618f6a62ed109e5b3c575780ed4


Make table view default

For all project report make table view as default and card view as go to option.The rationale is from public and campus user perspective the table view allows people to search and export the data that they need.The card view does not have the option to search or export.
The ideal scenario will be to to remove card view to avoid switching between views and have simple design for the page.

In Progress

Included in pull request SU23 CEPI Issue list fix_2 #2644

Deployed in production


Link from README to CLDF metadata is wrong

The link from README.md to
https://github.com/cldf-datasets/gata/blob/main/cldf/Generic-metadata.json
is broken.
Not sure, why this might have happened - maybe you just need to recreate the README.

I changed this manually to refer to the correct StructureDataset metadata


Mags-agm-3rd-option-chanhe-aug1

updates to 3rd agm option per ticket #285

/gcbrun

Anyone available to do the PR pretty please? Looking to have this deployed to prod asap (after reviewing the temp URL, of course). Plus, looks like this one didn't make it into dev:https://github.com/bcgov/business-ar/pull/276 - can we lump that one in?

Temporary Url for review: https://business-ar-dev--pr-296-m4t4fzpj.web.app

This is what I get when I click on that URL:

This is what I get when I click on that URL:

It looks like this when input nano id:

Great! What is the next step to get this and https://github.com/bcgov/business-ar/pull/276 deployed? Goal is to get it to prod before Monday, when the next set of reminders go out.

I will reach out to Thor and see when he can take a look at this


Create project logo

Set it in README and in Godot project settings.

I´ve just read this, got inspired and made some quick n dirty design.

GOAT.zip

Wow! I didn't expect anyone to design a logo so fast, thank you very much! Your idea is interesting, and completely different than what I had in mind :D I was thinking about something similar to Godot robot's head, but with a goatee and horns ;) Assuming, of course, that Godot maintainers would be ok with that.
But I like your idea too and I'm open for discussion ;) Do you think you could design something a bit simpler, that would look good as a miniature? And would it be possible to release your work under Creative Commons/public domain?

I checked my mails every half an hour after posting it and... missed your reaction hours later xD.
Glad you like it too. Of course I experiment with it tomorrow and yeah I don’t care which license. CC0 is probably the best Option or CCby[GOAT] if ya like.
I am all legion for this. ;)

GOATZ.zip
License: CC0

Thank you for your amazing work @WesBlanko ! I have decided, however, to use another, more simple logo. It is already added to README and docs.
I'm really glad that you've found the time to work on your version and I'm sorry that it was not used, but I hope it will not discourage you from engaging in the project's development ;)


Central dashboard (Aggregated details of all tests) part 2

User story

As a user
I want all the information in an aggregated state (the screenshot is only from an internal massive firmware scan that shows some numbers) using app stack charts.

Acceptance criteria

[ ]        Top 10 CVEs (CVE 10 upwards) with a short description -> Klick on a CVE to get an overview of which Firmware it is affected

[x]          Top 10 Linux kernel versions -> klick on a version to get access to all Reports with the kernel version

[ ]         Top 10 Vendors-> klick on a Vendor to get access to all Reports with the kernel version

[ ]        Notifications of new firmware uploaded for testing and some state info like uploaded started, finished

[x]       Top 10 strcpy binaries (highest number of stripes)

with the severity depiction in a pie chart.
Definition of done

Code review from peers has been completed and passed

Code is fully implemented and commented
Adheres to coding guidelines
Commits are not squashed
Code is checked into the repository

Unit tests for backend have been written and are passing
Manual performance tests on the browser are passing
No critical bugs are open

closed on 07.07.2021


⚠ Bofrost.be has degraded performance

In 499b21a, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2955 ms

Resolved: Bofrost.be performance has improved in cd9b62d.


Manufacturing broken because of gateway_mfr error

@shawaj @SebastianMaj @marvinmarnold
Manufacturing is using images sent to them on Dec 2, 2021. FIRMWARE_VERSION=2<IP_ADDRESS>. We still see the error gateway_mfr test finished with error. This means that we're not on-boarding any units until this gets cleared.
This is the screenshot of the output they get:

Also the full debug log from syslog.
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,735 INFO  [hpt] Decoded diagnostics report: {'diagnostics_passed': False, 'errors': ['ECC', 'BN', 'LOR'
, 'OK', 'PK', 'PF'], 'serial_number': '00000000b9c4a6f2', 'ECC': 'gateway_mfr test finished with error, {"result": "fail", "tests": [{"output":
"ok", "result": "pass", "test": "serial"}, {"output": "unlocked", "result": "fail", "test": "zone_locked(data)"}, {"output": "unlocked", "result
": "fail", "test": "zone_locked(config)"}, {"output": "invalid slot: 0", "result": "fail", "test": "slot_config(0..=15, ecc)"}, {"output": "inva
lid slot: 3", "result": "fail", "test": "key_config(0..=15, ecc)"}, {"output": "ecc608 error\\n\\nCaused by:\\n    ecc error ExecError", "result
": "fail", "test": "miner_key(0)"}, {"output": "ecc608 error\\n\\nCaused by:\\n    ecc error ExecError", "result": "fail", "test": "sign(0)"}, {
"output": "ecc608 error\\n\\nCaused by:\\n    ecc error ExecError", "result": "fail", "test": "ecdh(0)"}]}', 'E0': 'F0:4C:D5:5A:52:73', 'eth_mac
_address': 'F0:4C:D5:5A:52:73', 'W0': '34:C3:D2:37:E0:72', 'wifi_mac_address': '34:C3:D2:37:E0:72', 'BN': 'Env var BALENA_DEVICE_NAME_AT_INIT no
t set', 'BALENA_DEVICE_NAME_AT_INIT': 'Env var BALENA_DEVICE_NAME_AT_INIT not set', 'ID': '61e1191213e5ef678b71ec612c4128a0', 'BALENA_DEVICE_UUI
D': '61e1191213e5ef678b71ec612c4128a0', 'BA': 'HELIUM-OUTDOOR-915', 'BALENA_APP_NAME': 'HELIUM-OUTDOOR-915', 'FR': '915', 'FREQ': '915', 'FW': '
2<IP_ADDRESS>', 'FIRMWARE_VERSION': '2<IP_ADDRESS>', 'VA': 'NEBHNT-OUT1', 'VARIANT': 'NEBHNT-OUT1', 'BT': True, 'LTE': False, 'LOR': False, 'lora'
: False, 'OK': 'gateway_mfr exited with a non-zero status', 'onboarding_key': 'gateway_mfr exited with a non-zero status', 'PK': 'gateway_mfr ex
ited with a non-zero status', 'public_key': 'gateway_mfr exited with a non-zero status', 'PF': False, 'legacy_pass_fail': False}
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,779 DEBUG [hpt] Trying to get version information from /version.
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,805 INFO  [hpt] 🟢  Version Information:#012diagnostics_version: 72f141f#012firmware_version: 2<IP_ADDRESS>
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,820 INFO  [hpt] Processing miner using new hpt logic.
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,828 ERROR [hpt] Miner FAILED diagnostics because: {'ECC', 'LOR', 'PK', 'OK'}
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,841 DEBUG [hpt] initFile.txt returned but not all tests passed:#012ECC Error: gateway_mfr test finished
 with error, {"result": "fail", "tests": [{"output": "ok", "result": "pass", "test": "serial"}, {"output": "unlocked", "result": "fail", "test":
 "zone_locked(data)"}, {"output": "unlocked", "result": "fail", "test": "zone_locked(config)"}, {"output": "invalid slot: 0", "result": "fail",
"test": "slot_config(0..=15, ecc)"}, {"output": "invalid slot: 3", "result": "fail", "test": "key_config(0..=15, ecc)"}, {"output": "ecc608 erro
r\n\nCaused by:\n    ecc error ExecError", "result": "fail", "test": "miner_key(0)"}, {"output": "ecc608 error\n\nCaused by:\n    ecc error Exec
Error", "result": "fail", "test": "sign(0)"}, {"output": "ecc608 error\n\nCaused by:\n    ecc error ExecError", "result": "fail", "test": "ecdh(
0)"}]}#012BN Error: Env var BALENA_DEVICE_NAME_AT_INIT not set#012LOR Error: False#012OK Error: gateway_mfr exited with a non-zero status#012PK
Error: gateway_mfr exited with a non-zero status#012PF Error: False
Dec  3 02:02:47 hpt6 2021-12-03 02:02:47,842 WARNI [hpt] , retrying in 10 seconds...

My vote would be to return to using the hm-gwmfr container until this can be resolved @marvinmarnold @vpetersson because the new gateway-mfr-rs has caused nothing but trouble IMO.

@kashifpk I wonder if the issue could be something to do with this:
https://github.com/NebraLtd/hm-diag/blob/8430efdddc5710ccfb65d7b1d7a2bf3a4d212817/hw_diag/app.py#L22-L25

@kashifpk I wonder if the issue could be something to do with this:

@shawaj no idea TBH. I haven't touched the ECC related stuff yet. Will dive into that once I receive the miners (stuck at customs here). One thing that is strange it that on latest master of hm-pyhelper I have the provision_key function at line 74 and not at line 94.
Comparing https://github.com/NebraLtd/hm-pyhelper/blob/master/hm_pyhelper/miner_param.py
with attached version that I have the code is a bit different too.

Checking why the difference is there in master.

I'm confused as to what you are saying?
hm-pyhelper master has it at line 94 where I'm looking?

yes, this is crazy hm-pyhelper I have has commit from Nov 9 and doing a git pull origin master says everything is up to date. Any way, will just do a fresh clone of the repo.

@kashifpk looking at the error above, it seems to me as if the test is failing because it hasn't been programmed, but in provision_key() it should notice that and then provision the key.
I guess this logic is not working correctly for some reason

Fresh clone got me the latest changes. so we have the same code. I see the code. May be we need some way of returning the output of https://github.com/NebraLtd/hm-pyhelper/blob/8e13f79ea3daba04c250a4d94fc59d6388050e6c/hm_pyhelper/miner_param.py#L103 to see why it's actually failing?

To me it looks like initFile.txt is generating before provision_key even fires

To me it looks like initFile.txt is generating before provision_key even fires (perhaps it's bypassing it altogether)
The response in that file is what I would expect to see on an ECC that hasn't been provisioned

No, the provisioning is executed before the app server is even started. That said, we might need to validate that this logic in fact works properly.

we might need to validate that this logic in fact works properly

It definitely doesn't. I can tell you that for free 😉
The provisioning works if you manually run it but it doesn't if you just turn on a device. Same happens on RockPi. I thought it was RockPi specific but clearly not.

Update:
On production we have devices with the same firmware and hm-diag version:
{"diagnostics_version":"72f141f","firmware_version":"2<IP_ADDRESS>"}
that we have on the manufacturing miners. On testnet the versions are newer but that is just because of the new PR template merge.
{"diagnostics_version":"68d4984","firmware_version":"2<IP_ADDRESS>-1"}
And none of these give any errors. So this should help isolate the issue because these devices get updated while in manufacturing before on-boarding we do trying to generate keys etc?

@kashifpk I completely agree.  we have one of the devices on the testnet.  on a newer firmware version than in production.  i am trying to test on my miner.  but it looks like my flash card has finally died.  And I cannot now flash the miner for the penultimate version.

I think the reason may be gateway-mfr executable is built for target: arm-unknown-linux-gnueabihf. The previous target was target: aarch64-unknown-linux-musl. I will check this in more detail.

I think the reason may be gateway-mfr executable is built for target: arm-unknown-linux-gnueabihf. The previous target was target: aarch64-unknown-linux-musl. I will check this in more detail.

This is not the case.

@kashifpk if the key is already provisioned this error won't occur.
All the testnet devices we programmed using the hm-gwmfr container that we used to use and so will pass the tests because they have been correctly provisioned already.
The issue is that the provision_key() new method is never firing or the logic is wrong and therefore the ECC is never provisioned and fails the tests.
This will not be reproducible on a device that already has a provisioned key

@shawaj seems like we get this (or at least similar) error on some of the devices in production. @salmanfarisvp identified some devices and we do see this on those devices too. You can follow the discussion regarding that in slack.

As I said above, in my opinion we should revert to hm-gwmfr container until this can be solved. As this is now holding up production unnecessarily.

But in principle yes I agree that this is something related to code that runs when the device is not yet onboarded.

@shawaj seems like we get this (or at least similar) error on some of the devices in production. @salmanfarisvp identified some devices and we do see this on those devices too. You can follow the discussion regarding that in slack.

That's unrelated IMO. Perhaps a failed ECC key or similar.
But also this logic is totally untested and has caused nothing but issues. We had a working way with hm-gwmfr and we broke it for no apparent reason.
Let's revert to that and can solve this later.

@kashifpk do you have a link to the slack conversation you mentioned with  @salmanfarisvp ?

@shawaj seems like we get this (or at least similar) error on some of the devices in production. @salmanfarisvp identified some devices and we do see this on those devices too. You can follow the discussion regarding that in slack.

That's unrelated IMO. Perhaps a failed ECC key or similar.
But also this logic is totally untested and has caused nothing but issues. We had a working way with hm-gwmfr and we broke it for no apparent reason.
Let's revert to that and can solve this later.

Reverting this is far from trivial.

@kashifpk do you have a link to the slack conversation you mentioned with @salmanfarisvp ?

@shawaj
https://nebraltd.slack.com/archives/C024BNQ1Y6T/p1638532099453300

The only other thing I can think is, will this logic definitely work:
https://github.com/NebraLtd/hm-pyhelper/blob/8e13f79ea3daba04c250a4d94fc59d6388050e6c/hm_pyhelper/miner_param.py#L94-L109
Compared to what it was before:
https://github.com/NebraLtd/hm-pyhelper/blob/407402e9ca43c26e268eb24ea559d08a5b9dc080/hm_pyhelper/miner_param.py#L94-L120

Reverting this is far from trivial.

Why did we break it in the first place @vpetersson ?
We need to do end to end tests of stuff before winging in changes to manufacturing images.
In any case, it's pretty trivial.... We can just revert the container for diag to an old version as well as some back in gwmfr

No, because this would break hotspot production tool. But we just found something that looks promising.

We can't use the Erlang based version for RockPi regardless, so we need to sort out regardless. Looks like we have found the root cause as well.

Why can't we use the Erlang version for RockPi?
But in any case looks like it's sorted now. Fingers crossed.

@kashifpk maybe we should add in a log message of "ECC provisioning has run" or if it skips "ECC tests passed, skipping provisioning"

Potentially closed by https://github.com/NebraLtd/helium-miner-software/commit/9b9f232ea17511af8f50a4412c81c39869f59310

@kashifpk maybe we should add in a log message of "ECC provisioning has run" or if it skips "ECC tests passed, skipping provisioning" (in hm-pyhelper)

Sure will get this implemented next week.

Error from completely unprovisioned key:
ECC Error: gateway_mfr test finished
 with error,
{
  "result": "fail",
  "tests": [
    {
      "output": "ok", "result": "pass", "test": "serial"}, {"output": "unlocked", "result": "fail", "test":
 "zone_locked(data)"}, {"output": "unlocked", "result": "fail", "test": "zone_locked(config)"}, {"output": "invalid slot: 0", "result": "fail",
"test": "slot_config(0..=15, ecc)"}, {"output": "invalid slot: 3", "result": "fail", "test": "key_config(0..=15, ecc)"}, {"output": "ecc608 erro
r\n\nCaused by:\n    ecc error ExecError", "result": "fail", "test": "miner_key(0)"}, {"output": "ecc608 error\n\nCaused by:\n    ecc error Exec
Error", "result": "fail", "test": "sign(0)"}, {"output": "ecc608 error\n\nCaused by:\n    ecc error ExecError", "result": "fail", "test": "ecdh(
0)"}]}

Error from device with errored provisioning:
root@f2bb3b2936d7:/opt/python-dependencies/hm_pyhelper# ./gateway_mfr test
{
  "result": "fail",
  "tests": [
    {
      "output": "ok",
      "result": "pass",
      "test": "serial"
    },
    {
      "output": "ok",
      "result": "pass",
      "test": "zone_locked(data)"
    },
    {
      "output": "ok",
      "result": "pass",
      "test": "zone_locked(config)"
    },
    {
      "output": "ok",
      "result": "pass",
      "test": "slot_config(0..=15, ecc)"
    },
    {
      "output": "ok",
      "result": "pass",
      "test": "key_config(0..=15, ecc)"
    },
    {
      "output": "decode error\n\nCaused by:\n    not a compact key",
      "result": "fail",
      "test": "miner_key(0)"
    },
    {
      "output": "decode error\n\nCaused by:\n    not a compact key",
      "result": "fail",
      "test": "sign(0)"
    },
    {
      "output": "decode error\n\nCaused by:\n    not a compact key",
      "result": "fail",
      "test": "ecdh(0)"
    }
  ]
}

ref https://github.com/helium/gateway-mfr-rs/issues/8

This error is a timeout error - it can be caused either by an ECC module board that has come loose (on outdoor) or a loose CM3 module / daugterboard - or alternatively (in rare cases) an actual failed ECC
2021-12-04 01:06:24,714 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(41) - gateway_mfr response stdout: b'{\n  "result": "fail",\n  "tests": [\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "serial"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "zone_locked(data)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "zone_locked(config)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "slot_config(0..=15, ecc)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "key_config(0..=15, ecc)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "miner_key(0)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "sign(0)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "ecdh(0)"\n    }\n  ]\n}\n'
 diagnostics  INFO:hm_pyhelper.miner_param:gateway_mfr response stdout: b'{\n  "result": "fail",\n  "tests": [\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "serial"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "zone_locked(data)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "zone_locked(config)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "slot_config(0..=15, ecc)"\n    },\n    {\n      "output": "timeout/retry error",\n      "result": "fail",\n      "test": "key_config(0..=15, ecc)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "miner_key(0)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "sign(0)"\n    },\n    {\n      "output": "ecc608 error\\n\\nCaused by:\\n    timeout/retry error",\n      "result": "fail",\n      "test": "ecdh(0)"\n    }\n  ]\n}\n'
 diagnostics  2021-12-04 01:06:24,715 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(43) - gateway_mfr response stderr: b''
 diagnostics  INFO:hm_pyhelper.miner_param:gateway_mfr response stderr: b''
 diagnostics  2021-12-04 01:06:24,751 - [ERROR] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(47) - gateway_mfr exited with a non-zero status
 diagnostics  Traceback (most recent call last):
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 36, in run_gateway_mfr
 diagnostics      run_gateway_mfr_result = subprocess.run(
 diagnostics    File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
 diagnostics      raise CalledProcessError(retcode, process.args,
 diagnostics  subprocess.CalledProcessError: Command '['/opt/python-dependencies/hm_pyhelper/gateway_mfr', 'provision']' returned non-zero exit status 1.
 diagnostics  ERROR:hm_pyhelper.miner_param:gateway_mfr exited with a non-zero status
 diagnostics  Traceback (most recent call last):
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 36, in run_gateway_mfr
 diagnostics      run_gateway_mfr_result = subprocess.run(
 diagnostics    File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
 diagnostics      raise CalledProcessError(retcode, process.args,
 diagnostics  subprocess.CalledProcessError: Command '['/opt/python-dependencies/hm_pyhelper/gateway_mfr', 'provision']' returned non-zero exit status 1.
 diagnostics  [2021-12-04 01:06:24 +0000] [8] [ERROR] Exception in worker process
 diagnostics  Traceback (most recent call last):
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 36, in run_gateway_mfr
 diagnostics      run_gateway_mfr_result = subprocess.run(
 diagnostics    File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
 diagnostics      raise CalledProcessError(retcode, process.args,
 diagnostics  subprocess.CalledProcessError: Command '['/opt/python-dependencies/hm_pyhelper/gateway_mfr', 'provision']' returned non-zero exit status 1.
 diagnostics
 diagnostics  During handling of the above exception, another exception occurred:
 diagnostics
 diagnostics  Traceback (most recent call last):
 diagnostics    File "/opt/python-dependencies/gunicorn/arbiter.py", line 589, in spawn_worker
 diagnostics      worker.init_process()
 diagnostics    File "/opt/python-dependencies/gunicorn/workers/base.py", line 134, in init_process
 diagnostics      self.load_wsgi()
 diagnostics    File "/opt/python-dependencies/gunicorn/workers/base.py", line 146, in load_wsgi
 diagnostics      self.wsgi = self.app.wsgi()
 diagnostics    File "/opt/python-dependencies/gunicorn/app/base.py", line 67, in wsgi
 diagnostics      self.callable = self.load()
 diagnostics    File "/opt/python-dependencies/gunicorn/app/wsgiapp.py", line 58, in load
 diagnostics      return self.load_wsgiapp()
 diagnostics    File "/opt/python-dependencies/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
 diagnostics      return util.import_app(self.app_uri)
 diagnostics    File "/opt/python-dependencies/gunicorn/util.py", line 359, in import_app
 diagnostics      mod = importlib.import_module(module)
 diagnostics    File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
 diagnostics      return _bootstrap._gcd_import(name[level:], package, level)
 diagnostics    File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
 diagnostics    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
 diagnostics    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
 diagnostics    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
 diagnostics    File "<frozen importlib._bootstrap_external>", line 883, in exec_module
 diagnostics    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 diagnostics    File "/opt/python-dependencies/hw_diag/__init__.py", line 3, in <module>
 diagnostics      wsgi_app = get_app(__name__)
 diagnostics    File "/opt/python-dependencies/hw_diag/app.py", line 30, in get_app
 diagnostics      perform_key_provisioning()
 diagnostics    File "/opt/python-dependencies/decorator.py", line 232, in fun
 diagnostics      return caller(func, *(extras + args), **kw)
 diagnostics    File "/opt/python-dependencies/retry/api.py", line 73, in retry_decorator
 diagnostics      return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
 diagnostics    File "/opt/python-dependencies/retry/api.py", line 33, in __retry_internal
 diagnostics      return f()
 diagnostics    File "/opt/python-dependencies/hw_diag/app.py", line 24, in perform_key_provisioning
 diagnostics      if not provision_key():
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 103, in provision_key
 diagnostics      gateway_mfr_result = run_gateway_mfr(["provision"])
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/lock_singleton.py", line 71, in wrapper_lock_ecc
 diagnostics      raise ex
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/lock_singleton.py", line 60, in wrapper_lock_ecc
 diagnostics      raise ex
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/lock_singleton.py", line 57, in wrapper_lock_ecc
 diagnostics      value = func(*args, **kwargs)
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 48, in run_gateway_mfr
 diagnostics      raise ECCMalfunctionException(err_str).with_traceback(e.__traceback__)
 diagnostics    File "/opt/python-dependencies/hm_pyhelper/miner_param.py", line 36, in run_gateway_mfr
 diagnostics      run_gateway_mfr_result = subprocess.run(
 diagnostics    File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
 diagnostics      raise CalledProcessError(retcode, process.args,
 diagnostics  hm_pyhelper.exceptions.ECCMalfunctionException: gateway_mfr exited with a non-zero status
 diagnostics  [2021-12-04 01:06:24 +0000] [8] [INFO] Worker exiting (pid: 8)
 diagnostics  [2021-12-04 01:06:25 +0000] [1] [INFO] Shutting down: Master
 diagnostics  [2021-12-04 01:06:25 +0000] [1] [INFO] Reason: Worker failed to boot.

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc test
+--------------------+------+
|        name        |result|
+--------------------+------+
|     serial_num     |  ok  |
|{zone_locked,config}|  ok  |
| {zone_locked,data} |  ok  |
|    slot_config     |  ok  |
|     key_config     |  ok  |
|     miner_key      |  ok  |
+--------------------+------+

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc onboarding
112XcXrknVtvmQ3DMGcKdPqH9f8Yj3GswmJiUSR8YWNVkxfFNnsN

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc provision_onboard
11m3KCgNMEqdcT2cHQL1qQPC3ZC5d8fJJsXu4h6v2Za8Av4UbHL
root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc test
+--------------------+------+
|        name        |result|
+--------------------+------+
|     serial_num     |  ok  |
|{zone_locked,config}|  ok  |
| {zone_locked,data} |  ok  |
|    slot_config     |  ok  |
|     key_config     |  ok  |
|     miner_key      |  ok  |
+--------------------+------+

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc onboarding
11m3KCgNMEqdcT2cHQL1qQPC3ZC5d8fJJsXu4h6v2Za8Av4UbHL
root@9869a5b:/opt/gateway_mfr#

Re the "not a compact key" error... It seems this was a legacy issue with gateway-mfr-rs..
https://github.com/helium/gateway-mfr-rs/issues/8#issuecomment-986818080

Closing this issue as this is now running in manufacturing for a couple of days without problems.

this does work to rekey a miner fyi @vpetersson :
root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc test
+--------------------+------+
|        name        |result|
+--------------------+------+
|     serial_num     |  ok  |
|{zone_locked,config}|  ok  |
| {zone_locked,data} |  ok  |
|    slot_config     |  ok  |
|     key_config     |  ok  |
|     miner_key      |  ok  |
+--------------------+------+

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc onboarding
112XcXrknVtvmQ3DMGcKdPqH9f8Yj3GswmJiUSR8YWNVkxfFNnsN

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc provision_onboard
11m3KCgNMEqdcT2cHQL1qQPC3ZC5d8fJJsXu4h6v2Za8Av4UbHL
root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc test
+--------------------+------+
|        name        |result|
+--------------------+------+
|     serial_num     |  ok  |
|{zone_locked,config}|  ok  |
| {zone_locked,data} |  ok  |
|    slot_config     |  ok  |
|     key_config     |  ok  |
|     miner_key      |  ok  |
+--------------------+------+

root@9869a5b:/opt/gateway_mfr# /opt/gateway_mfr/bin/gateway_mfr ecc onboarding
11m3KCgNMEqdcT2cHQL1qQPC3ZC5d8fJJsXu4h6v2Za8Av4UbHL
root@9869a5b:/opt/gateway_mfr#

Can also rekey a miner with new gateway-mfr-rs using gateway_mfr key --generate <slot>

Good find. Ping @marvinmarnold / @robputt

Will add a doc in product-management repo so we don't lose this info


PERMANANENT DRAFT PR FOR DEMO ENV: DO NOT MERGE

DO NOT MERGE

Warnings

:warning:
Please add the JIRA issue key to the PR title (e.g. MB-123)

  Generated by :no_entry_sign: dangerJS against 8332a0f9e6b56365c965cd1cae69a73974334d2d


Attach system replication status badge on secondary node

A feedback from the customers says: The system replication is a characteristic of secondary nodes.
And here's the PR addressing it.

Great! Maybe we could leave the sap systems test in place and just check for 200 plus couple of strings without regex, wdyt?

Great! Maybe we could leave the sap systems test in place and just check for 200 plus couple of strings without regex, wdyt?

Ok to me :wink:


removeUnusedImports commands do not appear in the Command Palette

Does this issue occur when all extensions are disabled?: Yes/No

Steps to Reproduce:

Check the Command Palette for the new js/ts Remove Unused Imports commands (in a a ts file with imports, if that matters).

The commands do not show up in the Command Palette although they are in the Keyboard Shortcuts.
Version: 1.73.0-insider (user setup)
Commit: 8fa188b2b301d36553cbc9ce1b0a146ccb93351f
Date: 2022-11-01T15:30:14.729Z
Electron: 19.0.17
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.25231
Sandboxed: Yes

The command  requires using TS 4.9+. You can create a keybinding for the command, but it will only work on that typescript version. That's why we don't show it in the command palette


Intent 5 analysis

Developer intents: Provisioning intents

Intent 5
5SED
5MEE
EXRA
EXRC
EXCA
EXSM
GMEC

5. "I can ask the operator to provision my application server to the optimal MEC for a specific terminal, taking into account connectivity, resources (e.g. vCPU, Memory, network interfaces, storage, GPU) shortest network path, cost, network load, MEC platform load, application privacy considerations etc."
No
No
Yes
N/A
N/A
N/A
Yes

5MEE
Details
Comments

GET /mecplatforms\?UEIdentityType=[a-zA-Z0-9-]*\&UEIdentity=[a-zA-Z0-9.-]*?
Return the optimal MEC platform for a given UE identifier
Method may use these input parameters: region,zone,serviceProfileId,subscriberDensity, UEIdentityType,UEIdentity . This method does not cover provision, which API method can be invoked to trigger provision?

EXRA
Details
Comments

POST /api/v1/auth/ctrl/CreateApp
Create Application. Creates a definition for an application for Cloudlet deployment. It supports autoprovision policies triggered in combination with findcloudlet
It requires also autoprovision policy to trigger auto app instance.

POST /api/v1/auth/ctrl/CreateAppInst
Create Application Instance. Creates an instance of an App on a Cloudlet where it is defined by an App plus a ClusterInst key. Many of the fields here are inherited from the App definition
API user needs to specify the cloudlet manually where to instantiate the app, it may use external data to make that decision.

GMEC
Details
Comments

GET /application/lcm/{appId}
Instantiates an application on an OP zone. Details about application and zones where application instance should be created. Field 'operator' and 'opCountry' should be specified if the zone belongs to a partner OP.
This is the method to instantiate the app. Which method is used to identify the optimal MEC? Does it take into account all intent options?

The intent seems to assume that it is the client / the developer who actually takes care of provisioning an application to a cloudlet. This is not the philosophy of our platform and therefore also not our APIs where the application onboarding including declaring the provisioning policies and the actual provisioning and subsequently the client based discovery are two completely separate things. The platform takes care of the actual app provisioning (matching the client requests and the provisioning policies) dynamically. This helps the developers to abstract the complexity of a distributed edge infrastructure.
There are two ways to actually trigger the provisioning of an application instance to a specific cloudlet:
1.) Independent of the client / a specific location: CreateAppInst - Creates an instance of an App on a specific Cloudlet
2.) Automatically by the platform following the policy defined through CreateAutoProvPolicy API and taking into account the client requests (FindCloudlet) coming in. In that way the platform dynamically provisions (and unvprosions) application instances where and when they are needed.

The Application Instantiation APIs from Capgemini currently assumes that the valid application service locations in form of Availability Zone(s) (AZ or zones) has been provided in the Application Onboarding API. The App Instantiation API contains references to the zones used during onboarding and the implementations e.g. operator platform (OP) will use the cloudlets associated to these zones for instantiating the given application as identified by the appId.

5GFF MEC Exposure & Experience Management API does not support instantiation/deployment of the application on MEC platforms.
Focus is on helping developers identify the most optimal MEC sites for deploying their application and also help end users of the application to connect to the most optimal among the application instance running on multiple MEC platforms.

This are the flow diagrams for corresponding EdgeXR APIs
createApp (https://api.edgexr.org/#tag/App/operation/CreateApp)
sequenceDiagram
    participant developer
    participant operator
    Note over developer,operator: PRE App developer registered,
 authenticated and authorised
    developer->>operator: POST /CreateApp  <app manifest>
    operator->>developer: 200 OK

Manual app deployment
createAppInst ( https://api.edgexr.org/#tag/AppInst/operation/CreateAppInst)
sequenceDiagram
    participant developer
    participant operator
    Note over developer,operator: PRE App developer registered,
 authenticated and authorised
    Note over developer,operator: PRE App developer has created an app and has access to cloudlet(s)
    developer->>operator: POST /CreateAppInst  <app, cloudlet(s)>
    operator->>developer: 200 OK

Automatic app deployment triggered by app client requests issuing findCloudlet()
createAutoProvPolicy ( https://api.edgexr.org/#tag/AutoProvPolicy/operation/CreateAutoProvPolicy)
addAppAutoProvPolicy (https://api.edgexr.org/#tag/App/operation/AddAppAutoProvPolicy)
sequenceDiagram
    participant developer
    participant operator
    Note over developer,operator: PRE App developer registered,
 authenticated and authorised
    Note over developer,operator: PRE App developer has created an app and has access to cloudlet(s)
    developer->>operator: POST /CreateAutoProvPolicy  <cloudlet(s), thresholds>
    operator->>developer: 200 OK
    developer->>operator: POST /AddAppAutoProvPolicy  <app, autoprovpolicy>
    operator->>developer: 200 OK


feat: add support for multi dynamic key name

DEV-9056
https://app.clickup.com/t/2645698/DEV-9056
--
Closes #356

lgtm, though Im not sure I fully understand how the duplicateMultiTypeERC725SchemaEntry works. We create schemas for each possible typing, and progress as if each one was passed individually. How are we sure that the correct type is used for encoding/decoding? I may lack context

The correct type is given by the user. The user will use the lib like this:

And because of this "duplicate step" the lib will be able to encode/decode properly.


Support shadows with DynamicFonts

Currently, DynamicFonts can't cast any shadows, not even sharp ones. It would be nice if they could be set to cast a sharp shadow at least, with customizable X/Y offset and color.
Something like this (from another engine):

I think Label has the option for that
On Sat, Jun 11, 2016 at 3:25 PM, Hugo Locurcio<EMAIL_ADDRESS>wrote:

Currently, DynamicFonts can't cast any shadows, not even sharp ones. It
would be nice if they could be set to cast a sharp shadow at least, with
customizable X/Y offset and color.
Something like this (from another engine): [image: shot0001]
https://cloud.githubusercontent.com/assets/180032/15986801/83915e9e-3012-11e6-9e42-50f22093db80.jpg
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5159, or mute the thread
https://github.com/notifications/unsubscribe/AF-Z29H1GeQDRAqBYYySU4R5GOnQyWBMks5qKv1_gaJpZM4IznDv
.

I think Label has the option for that

I can confirm, it also has outline options. Closing.


sap_hana_preconfigure: SAP HANA 2.0 SPS08 needs compat-sap-c++-13.

See also SAP note 3449186.

Solved in #895.


latest release include 'v' prefix in release tag

All previous releases did not include the v prefix in the tag, v14.0.0 is the first and breaks the pattern of no prefix e.g. 13.3.0.

It is worth noting that this is not an issue with citgm as the above PR will fix it from our end but we don't want to make a change that you guys will then revert to with the next release

I'll fix this later today, by deleting the current tag and creating a new one, when I'm at my computer (unless someone else gets to it first).

thanks, please close this issue when this has been rectified

This is fixed now.


Added creation/deletion of registry keys to static HTML report summary

These changes allow for more detailed information about the registry to be displayed in the web gui. It does not differentiate between key addition and modification and so treats all writes the same. Writes have green text and are prefixed with 'w+', deletions have red text and are prefixed with '-'. An example is below:

Just checking on the status of this pull request. Is there something that I could do the help clean up behavior.py?

Is there a chance this will make it into the 1.2 release?

I need to work on it.
Firstly, whenever the results structure changes, it causes compatibility issues. Secondly, the HTML report is pretty much abandoned, so this should be ported to the Django interface.
Lastly, as it is replacing the Summary, at that point I'd rather replace it completely for the other sections as well.

Screenshot of changes made by most recent commit.

This functionality has been integrated as part of the monitor branch. Both the json report and the (deprecated) HTML report now show a summary with all these things in there, sorted by the type of event (file written, regkey created, etc). Integration for Django is still pending, but this is not the goal of this pull request anyway. So, closing this PR.


Allow custom CORS origins

I currently use a custom lambda function which handles OPTIONS requests so I can allow or disallow arbitrary domains for CORS. Is there a way I can do this with Claudia?

At the moment, CORS origins are hard-coded to * (all origins allowed). It should be easy to add some more flexibility there, eg allow people to specify an options handler or provide a list of domains supported.
What does your current function use to decide if the request should be allowed or not?

Right now I'm using it as an API that I only want my own domains to be able to call, so it echoes the domain name if the caller's domain is mine and doesn't return any origin header otherwise (see SO). I can imagine a case where I'd want to be able to dynamically allow others to use my API and so I'd write a lambda function that pulls a list of allowed origins from e.g. DynamoDB.

and is this restriction global for all your API calls, or is it on individual paths? is the way forward to specify an OPTIONS handler globally for the entire API, or do you need finer grained control where different options apply to different request paths?

I specify one globally for my whole API because I don't have a need for CORS control at the endpoint granularity, but I'd love to hear others' thoughts on this.

this is now supported, with API Builder 1.1.0 and Claudia 1.1.0; see https://github.com/claudiajs/claudia-api-builder#controlling-cross-origin-resource-sharing-headers for more information


Panic op should take an extra argument

Panic should be given an extra argument, whose type is specified by a parameter.
This is so that if we want to panic with some linear variables in scope, we have a place we can wire them to so we can avoid linearity errors.

Should the linear values be explicitly discarded first?

I think something in the prelude discard: forall T: Type(Any). T -> () would be better for hugr than a qubit specific one. If we add it, a tket2 "qubit discard" method could just be discard TypeApply'd to Qubit type

I've come to realize that I don't know what panic is. It seems odd for it to be a dataflow op with inputs and outputs, since it does not really have either (or at best, it has a single input message). According to the spec, "any operation may panic". This suggests that it is not actually a node of the graph, but a part of the semantics of all (operation) nodes.
In this view, there is no problem with linear values not being consumed by it, since it isn't part of the DAG.

This is true, but it is also true that panic is specified as a function in the standard library which always panics, intended to be called when you have no better option(e.g. a Case which can never happen, but which you are still required to provide). This function is not so useful: your graph will be invalid unless you deal with all linear values and send the right types to the output node.

Thanks, that clarifies the issue for me!


Prevent bringing Unity window to front during upload

Normally when an avatar build completes, the Unity window will be brought to front and take focus.
This is fine for building a single avatar, but is undesirable behavior when uploading multiple avatars in the background.

It looks bringing Unity to front is cased by VRCSDK and It looks will be fixed in VRCSDK 3.3.0.

Fixed in 0.2.0 with upgrading VRCSDK to 3.3.0


Added Algo Trading SMA Algorithm

@pankaj-bind Please Review

#1031

Move it to miscellaneous algorithm folder

Ok

Move it to miscellaneous algorithm folder

Done


Using functions in the ruleset

Do you have an example of using function in the workset
I  tried with the following ruleset and function but it fails to run
# Source: https://docs.stoplight.io/docs/spectral/4dec24461f3af-open-api-rules
extends:
 - spectral:oas
functionsDir: "./functions"
functions:
 - consistent-response-body
rules:
 info-contact: off
 no-$ref-siblings: off
 oas2-api-host: off
 oas2-api-schemes: off
 openapi-tags: off
 operation-description: off
 operation-tags: off
 operation-tag-defined: off

 # Note: The Spectral VSCode extension will not display "hint" messages, so
 # use "info" rather than "hint".
 # just a note here

 az-additional-properties-and-properties:
   description: Don't specify additionalProperties as a sibling of properties.
   severity: warn
   formats: ["oas2", "oas3"]
   given: $..[?(@object() && @.type === 'object' && @.properties)]
   then:
     field: additionalProperties
     function: falsy

 az-consistent-response-body:
   description: Ensure the get, put, and patch response body schemas are consistent.
   message: "{{error}}"
   severity: warn
   formats: ["oas2"]
   given: $.paths.*
   then:
     function: consistent-response-body

and function
// If put or patch is a create (returns 201), then verify that put, get, and patch response body
// schemas are consistent.

// path Item should be a [path item object](https://github.com/OAI/OpenAPI-Specification/blob/main/versions/2.0.md#pathItemObject).
// This function assumes it is running on a resolved doc.
module.exports = (pathItem, _opts, paths) => {
if (pathItem === null || typeof pathItem !== 'object') {
  return [];
}
const path = paths.path || paths.target || [];

const errors = [];

// resource schema is create operation response schema
const createResponseSchema = ((op) => op?.responses?.['201']?.schema);
const resourceSchema = createResponseSchema(pathItem.put) || createResponseSchema(pathItem.patch);
if (resourceSchema) {
  ['put', 'get', 'patch'].forEach((method) => {
    const responseSchema = pathItem[method]?.responses?.['200']?.schema;
    if (responseSchema && responseSchema !== resourceSchema) {
      errors.push({
        message: 'Response body schema does not match create response body schema.',
        path: [...path, method, 'responses', '200', 'schema'],
      });
    }
  });
}

return errors;
};

This is based upon the azure ruleset azure-api-style-guide
Ive also attached the azure fucntions logs
query_data.csv
log

@UniperMaster :
Thank you so much for reaching out.
Functions will work with the ruleset.
You are currently using the SimpleJS syntax and in order for our linter to work, it needs ES syntax. Our current recommendation is for you to transition from SimpleJS to ES syntax.
Transitioning from SimpleJS syntax (which is similar to CommonJS) to ES Modules (ESM) involves a few steps. Let’s explore how you can make this transition:

Understand the Differences:

CommonJS uses require() for importing modules and module.exports or exports for exporting.
ESM introduces the import and export syntax natively to JavaScript.
ESM offers better performance, compatibility, and static analysis benefits.

Update Your Code:

Start by converting your module syntax from require to import.
Rename files to use the .mjs extension (or adjust the type property in package.json if you prefer .js for ESM).

Package.json Configuration (optional):

Set "type": "module" in your package.json to enable ESM.
Ensure backward compatibility by specifying .js or .mjs extensions for ESM files.

For the future: we are currently having conversations about supporting this in the future. In the mean time, I will add a note to the readme file