eluent_introduction_category modeling broken in latest backmerge

@brynnz22 @anastasiyaprymolenna @kheal
I did a back-merge from nmdc-schema into berkeley-schema-fy24 this morning and accidentally pushed it into berkeley-schema-fy24 main instead of a branch
Everything seems fine except for anything related to eluent_introduction_category
Are we still going to use that after the remodeling @kheal did yesterday?
If so, I could use some help getting that to work again. I'll take responsibility for digging up the last good state of the two repos.

problematic example data files:

Database-interleaved.yaml
Database-mass-spectrometry.yaml
Database-mass_spectrometry_gc.yaml
MassSpectrometry-NOM.yaml

Actually, this seems unrelated to eluent_introduction_category

[ERROR] [src/data/problem/Database-mass-spectrometry.yaml/0] 'nmdc:dgms-99-zUCd5N' does not match '^(nmdc):(dgns|omprc)-([0-9][a-z]{0,6}[0-9])-([A-Za-z0-9]{1,})(\\.[A-Za-z0-9]{1,})*(_[A-Za-z0-9_\\.-]+)?$' in /data_generation_set/0/id
[ERROR] [src/data/problem/Database-mass-spectrometry.yaml/0] 'nmdc:dgms-99-zUCd5x' does not match '^(nmdc):(dgns|omprc)-([0-9][a-z]{0,6}[0-9])-([A-Za-z0-9]{1,})(\\.[A-Za-z0-9]{1,})*(_[A-Za-z0-9_\\.-]+)?$' in /data_generation_set/1/id

nmdc:dgms-99-zUCd5N and nmdc:dgms-99-zUCd5x are both asserted to be of type nmdc:MassSpectrometry
MassSpectrometry is_a DataGeneration and has
slot_usage:
  id:
    name: id
    domain_of:
    - NamedThing
    structured_pattern:
      syntax: '{id_nmdc_prefix}:(dgms|omprc)-{id_shoulder}-{id_blade}{id_version}{id_locus}'
      interpolated: true

I added this to DataGeneration

    slot_usage:
      id:
        pattern: "^nmdc:(omprc|dgms|dgns)-[0-9][a-z]{0,6}[0-9]-[A-Za-z0-9]{1,}(\\.[A-Za-z0-9]{1,})*(_[A-Za-z0-9_\\.-]+)?$" # better applied as a structured_pattern, but even that might get out of sync from the asserted Study structured_pattern

@brynnz22 @anastasiyaprymolenna @kheal
I did a back-merge from nmdc-schema into berkeley-schema-fy24 this morning and accidentally pushed it into berkeley-schema-fy24 main instead of a branch
Everything seems fine except for anything related to eluent_introduction_category
Are we still going to use that after the remodeling @kheal did yesterday?
If so, I could use some help getting that to work again. I'll take responsibility for digging up the last good state of the two repos.

The eluent_introduction_category is one of the slots I added during the remodel of the Chromatography Configurations, so yes we'll use that slot.  Looks like you figured it out re:IDs,  but let me know if I can help debug.


withState | access state value

I am using withState('value, 'setValue', '') to keep the state of an input field. When the input field is changed, I want to  update the state, and debounce a search action (after 300ms).
The issue I am running into, is that when I want to acces my state in the onChange handler on my input field, my state is not up to date (from calling setValue()).
I figured out that I should not be using the value right after setValue (with basically calls setState), since this is synchronously and you should not use this.state.value after setting state. So … I figure I should use a callback function: https://www.webpackbin.com/bins/-KvC7Ui2TNnxCqRJ6I6w
However, I always get the previous state returned, instead of the "new state". Is this a bug? Or how can I fix this? Thanks!

1 example - value is updated but you use previous definition of logResult with old value inside it's not because something async or sync it's just how javascript works.
2nd example is the same as first.
I can't get what the question?

1 example - value is updated but you use previous definition of logResult with old value inside it's not because something async or sync it's just how javascript works.

@istarkov How can I, in this example, log the current state instead of the previous state? How can I acces the new value?

log current
https://www.webpackbin.com/bins/-KvCDTdtiU_rKwbpgcUO
for your case, you don't need to access current state value, you can debounce in handler
see here https://github.com/deepsweet/hocs debounce hoc

BTW I would recommend you to read something about javascript, about variables closures, scopes etc

@istarkov Could you point me in any direction where I can read more about this? Do you have an example of this using a debounce handler? What I am trying to do is keep a state for input field, and only once per eg. 500ms trigger a search() action. For this example I instead used a logger, just so I can see the result.
Thanks for your help.

Using a class: https://www.webpackbin.com/bins/-KvCZF-K1MWOiFb3MobY
Using recompose: https://www.webpackbin.com/bins/-KvCZsOSaIXs9v29cQL1
Still trying to figure out the difference 🤓


Add links to GitHub

The aim of this change is to help people find the GitHub repository of this project, as it may not be obvious from the npm page.

@clarus :+1:
Thank you a lot, I totally forgot to add the homepage and repository metadata to the package.json.


Add Veritoken logo

[✓] Filename in the lowercase format
[✓] Image format: png
[✓] Image size: 256 by 256 px

Also, you can use these graphics to say that your ERC20 token fully compatible with Trust Wallet (https://trustwallet.com/logo-assets). Tokens will show up automatically once the user receives them.
Feel free to share this on social media and tag Trust Wallet:

Twitter: @trustwalletapp

Facebook: @trustwalletapp


Exclude the versioned requirements

Currently, pyup provides fixed dependency versions as requirements. We should exclude this from the master branch and move the testing of new dependency versions in a separate branch.

[x] create branch version-check
[x] adjust pyup to provide PRs of updates to the version-check branch
[x] exclude the fixed dependencies from master
[ ] document this in the development docs (sphinx)

I made the branch and changed the pyup branch. I noticed that one may also use pin: False to avoid versioned requirements from pyup.

It is probably also more important to check this file, as this contains the versioned dependencies of the default install:
https://github.com/inbo/niche_vlaanderen/blob/master/docs/niche_env.yml

@johanvdw if I understand this correctly, the pin: False would be a better solution compared to the separate branch (version_check which still need a hook to be in sync with master)?  So, let's use that one?
How would the niche_env.yml file be checked?


Is it possible to use the dimmy with the i2c connector and vanpi core?

Hi, i love your input and project. I got most thing working now in home assistant. But is there a possibility to use the dimmy?

Hi i think you are write with me on the https://forum.pekaway.de/ .  I’m not using the Dimmy because I’m going to do everything via W-LED but as far as I know it has MQTT because there’s an ESP on it, you could then simply integrate it via W-Lan or you would have to write a component for it.
If you follow my instructions, all i2c and UART connections are patched and you can access them and use them in your own custom_component.

You could try it with https://github.com/antonverburg/ha-pca9685.
Unfortunately, as I mentioned, I can't test it currently because I don't have a Dimmy.


Adding .SetFormatOutput() to Settings will break any return codes from called script.

There seems to have been a regression - as the Task("Failing-Powershell-File")  example in the test package does not work as expected.
The Test is this
`Task("Failing-Powershell-File")
.Description("An example showing how to handle errors from Powershell scripts")
.Does(() =>
{
Information("Calling failing Powershell File");
    var resultCollection = StartPowershellFile("./failingScript.ps1", new PowershellSettings()
        **.SetFormatOutput()**
        .SetLogOutput());

    var returnCode = int.Parse(resultCollection[0].BaseObject.ToString());
    if (returnCode != 0) {
        throw new ApplicationException("Script failed to execute");
    }
});`

This will now not work as expected until you remove the .SetFormatOutput() PowerShellSetting.  This is consistent to what we have been experiencing.
Also of note is that the Task above  is not actually wired into anything  and will only run as a custom target - I dont think that is expected.

Hopefully this is clearer?


Crash when reading classifier model FluidMLPClassifier from JSON file

Hello!
There's a crash when reading the classifier model FluidMLPClassifier from a JSON file. The SC post window returns:
Server 'localhost' exited with exit code 0.
server 'localhost' disconnected shared memory interface

To replicate the error, you can follow the FluidMLPClassifier Help file and add this code (you will need to change the paths):
~mlp.write("/Users/anna/Desktop/test/testMLP.json")

~mlp2 = FluidMLPClassifier(s,[3],activation:FluidMLPClassifier.sigmoid,maxIter:20,learnRate:0.01,batchSize:1,validation:0.1);

~mlp2.read("/Users/anna/Desktop/test/testMLP.json")

hmmm here it doesn't crash. Which version have you got?  Can you try the 1.0.6 release candidate? I am testing on :
Fluid Corpus Manipulation Toolkit: version 1.0.6+sha.8403b25.core.sha.4a7d9daa
I have the previous public version (1.0.5) which I will try right away and hope to reproduce.

ok I was able to reproduce in the previous public version (1.0.5+sha.5bb4245.core.sha.001df55a)

luckily, we fixed it in 1.0.6 it seems. Can you confirm?

I tried 1.0.6 (pre-release) and it works, thank you very much!

fixed in forthcoming 1.0.6

Super, thanks! already using it :)


switch KEEP_REPORTDIR to boolean type

fixes:
Scheduling project: [build-scripts » jobs » jdk17u » jdk17u-alpine-linux-aarch64-temurin_SmokeTests](https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-aarch64-temurin_SmokeTests/)
15:57:34  The parameter 'KEEP_REPORTDIR' did not have the type expected by [build-scripts » jobs » jdk17u » jdk17u-alpine-linux-aarch64-temurin_SmokeTests](https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-aarch64-temurin_SmokeTests/). Converting to Boolean Parameter.


UnknownVariables doesn't return unknown variables defined in files

Running scons will print {}, while running scons my_var="cli_value" will print {'my_var': 'cli_value'}
#custom.py
my_var="value"

import SCons
from SCons.Variables import *

env = Environment()

options = Variables(["custom.py"], ARGUMENTS)
options.Add("my_option", "", "")

options.Update(env)

print(options.UnknownVariables())

The current manpage wording suggests "unknown" variables from files should indeed be flagged:

Returns a dictionary containing any variables that were specified in the files and/or args parameters when Variables was called, but the object was not actually configured for.

I can confirm there's no processing of unknowns in the variables-file reading section. It just does an exec to add the assignments to the dict.


memory leak - unreachable log collation host

This memory leak is the result of configuring a log collation host in “logs_xml.config” and then having the host be down during the traffic run. We verified that this occurs with vanilla 6.2.1. We also noticed that there was a memory leak fix in this function patched in TS-4872, which is present in 6.2.x, 7.1.x, and master. So we tried the latest 6.2.x (current) from yesterday, but the memory leak still occurred. We have also seen that bringing up the log host during the test run stabilizes the memory allocation so we’re pretty sure that this is the trigger for the memory leak.
Some heap-check output from ATS compiled with tcmalloc --
Total: 3014.2 MB
2902.3  96.3%  96.3%   2902.3  96.3% ats_malloc
76.0   2.5%  98.8%     76.0   2.5% ats_memalign
35.8   1.2% 100.0%   2938.1  97.5% LogObject::_checkout_write
0.1   0.0% 100.0%      0.1   0.0% BaseLogFile::open_file
0.0   0.0% 100.0%      0.0   0.0% ResourceTracker::lookup (inline)

Confirmed this memory leak still occurs with 7.1.3.

Collation is gone, so closing this.


Would you kindly update Xlora to support Quantized Models?

to train xlora on free collab we need to load a quantized model but currently, xlora does not support the quantized model and layers are not swapping.
Please upgrade xlora for the quantized model, mostly uses BitsAndBytesConfig to load the model in 4-bit or 8bit in free collab, But the quantized model could not convert into xlora so please update xlora for quantized models.

@Abdullah-kwl , could you please paste the result of printing model?

PeftModelForCausalLM(
(base_model): LoraModel(
(model): MistralForCausalLM(
(model): MistralModel(
(embed_tokens): Embedding(32000, 4096, padding_idx=2)
(layers): ModuleList(
(0-31): 32 x MistralDecoderLayer(
(self_attn): MistralAttention(
(q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)
(k_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
(lora_dropout): ModuleDict(
(adapter_1): Dropout(p=0.1, inplace=False)
(adapter_2): Dropout(p=0.1, inplace=False)
(adapter_3): Dropout(p=0.1, inplace=False)
)
(lora_A): ModuleDict(
(adapter_1): Linear(in_features=4096, out_features=4, bias=False)
(adapter_2): Linear(in_features=4096, out_features=4, bias=False)
(adapter_3): Linear(in_features=4096, out_features=4, bias=False)
)
(lora_B): ModuleDict(
(adapter_1): Linear(in_features=4, out_features=1024, bias=False)
(adapter_2): Linear(in_features=4, out_features=1024, bias=False)
(adapter_3): Linear(in_features=4, out_features=1024, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
)
(v_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
(lora_dropout): ModuleDict(
(adapter_1): Dropout(p=0.1, inplace=False)
(adapter_2): Dropout(p=0.1, inplace=False)
(adapter_3): Dropout(p=0.1, inplace=False)
)
(lora_A): ModuleDict(
(adapter_1): Linear(in_features=4096, out_features=4, bias=False)
(adapter_2): Linear(in_features=4096, out_features=4, bias=False)
(adapter_3): Linear(in_features=4096, out_features=4, bias=False)
)
(lora_B): ModuleDict(
(adapter_1): Linear(in_features=4, out_features=1024, bias=False)
(adapter_2): Linear(in_features=4, out_features=1024, bias=False)
(adapter_3): Linear(in_features=4, out_features=1024, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
)
(o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)
(rotary_emb): MistralRotaryEmbedding()
)
(mlp): MistralMLP(
(gate_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)
(up_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)
(down_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=14336, out_features=4096, bias=False)
(lora_dropout): ModuleDict(
(adapter_1): Dropout(p=0.1, inplace=False)
(adapter_2): Dropout(p=0.1, inplace=False)
(adapter_3): Dropout(p=0.1, inplace=False)
)
(lora_A): ModuleDict(
(adapter_1): Linear(in_features=14336, out_features=4, bias=False)
(adapter_2): Linear(in_features=14336, out_features=4, bias=False)
(adapter_3): Linear(in_features=14336, out_features=4, bias=False)
)
(lora_B): ModuleDict(
(adapter_1): Linear(in_features=4, out_features=4096, bias=False)
(adapter_2): Linear(in_features=4, out_features=4096, bias=False)
(adapter_3): Linear(in_features=4, out_features=4096, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
)
(act_fn): SiLU()
)
(input_layernorm): MistralRMSNorm()
(post_attention_layernorm): MistralRMSNorm()
)
)
(norm): MistralRMSNorm()
)
(lm_head): CastOutputToFloat(
(0): Linear(in_features=4096, out_features=32000, bias=False)
)
)
)
(internal_xlora_classifier): xLoRAClassifier(
(softmax): TemperatureScaledSoftmax(
(softmax): Softmax(dim=-1)
)
(inner): ModuleList(
(0): Linear(in_features=4096, out_features=2048, bias=True)
(1-6): 6 x Linear(in_features=2048, out_features=2048, bias=True)
)
(last): Linear(in_features=2048, out_features=3, bias=True)
)
)

You can review my notebook at : https://drive.google.com/file/d/1_B1ualsMbRfYWy0gdjdMi9RSDU-qmPHf/view?usp=sharing

Thank you. I plan on working on this later today.

Also, Checkout this notebook : https://colab.research.google.com/drive/1Eyh-mBd0LpcJwyzBHjGKhwNLQ9R74eLl?usp=drive_open

What adjustments should we make if we wish to upgrade XLora for IA^3?

We have begun work here and it will be completed shortly.

Ok

Hi,
Just wanted to make sure that the current version supports Quantised models since I think some tests haven't been passed here, and the commit hasn't been merged to the main branch.


Fix AZP Linux packaging to build configure.BuildHash

The SimpleITK repository hash used be built was incorrectly set to the
same path/checkout as the CI scripts. The docker run command was
mounting the CI SimpleITK source directory overriding the
SIMPLEITK_GIT_TAG env varible. This mount is removed.

The linux packages in this build are from the wrong version:
https://dev.azure.com/SimpleITK-DevOps/SimpleITK/_build/results?buildId=3881&view=results
This PR should correct it.


Bad performance compared to llava?

Hi authors, I ran images in your demo but I got a much worse results compared to Llava. The captions are very short even if I follow the same prompt for detailed description in the paper and I used the same image for both llava and this. The higher resolution also doesn't capture the small text in the correct location. Is there something wrong with the demo? I cannot get results close to anything shown in the paper.
Can you explain how you added perceiver resampler too? Since perceiver resampler is used for videos, is the temporal dimension used for the number of images?
Thanks in advance.

@eddyrivers10
Dear Eddy,
Could you kindly provide some images for us? We need them to identify the potential causes of the issue we're facing. We've conducted numerous comparisons with LLaVA 1.5 and believe that our results should at least match its performance. Your assistance in sharing these images would be greatly appreciated.

I chose 5 images that occurred today for testing, which I uploaded for analysis. The first four are from today's 2023 League of Legends World Championship, and the fifth is a random shot taken for testing purposes. Based on the images I uploaded, I find that in most cases, Monkey testing performs better than Llava 1.5.

Hi, I just used the default image of the man ironing on the back of his car from the llava dem https://llava.hliu.cc/. When I asked for a detailed caption I get A man is ironing his shirt on the back of a car.
Second example I tried was the water image from the llava demo too. The caption received is A long wooden pier on a lake surrounded by trees..
The captions are very short compared to llava. However llava hallucinates more as the captions are longer. Does the demo use the same decoding length? I wonder how you can make Monkey give descriptive sentences without hallucinating. Thanks!

Hello. You should click the “Generate” button to get a detailed description of the image. For the VQA task, we utilize the following format: "[question] Answer: [answer]." For the detailed caption task, we employ the format: "Generate the detailed caption in English: [caption]."  If you provide the input 'Generate the detailed caption in English:' and click “submit” button, the model's input will be 'Generate the detailed caption in English: Answer:'. This input deviates from what we were trained on.

Hi authors, sorry if I used it wrongly, I haven't had the chance to check the demo again since the web traffic is high now. But it looks like I have clicked on the wrong option which is why the captions are so short.
For the captions above it looks like they are shorter but more factual. Do you know if there's a comparison on hallucination versus llava and other larger resolution models? Thank you!

@Yuliang-Liu 。In Monkey paper，  Is “llava1.5 pretrained with CC3M” which the Table 6  says same with original llava1.5？这个模型训练数据是跟原始的llava1.5-13b一样吗

Hello. Our experiments involved a comparative analysis where we pretrained LLaVA1.5 using the same original data from LLaVA, with the modification of replacing 427k text with our generated annotations. The instruction tuning data  were exclusively sourced from LLaVA1.5.

@echo840  Thanks for your reply.  when pretrained by llava data, mmbench score is 68.3 that higher than llav1.5 which is 67.7。 This is an amazing result。


Add href to host icons, possibly making them keyboard-focusable and screen reader-accessible

Sorry it took so long to get to this, we've been swamped with DockerCon.

Any idea when this will land in a release? Just tried a single-node install and it doesn't appear that these changes are there--that or something is very wrong.

We normally create a 0.WW-rc1 In DockerHub every Wednesday (WW being the week number of the year).  That gets looked at by our QA, some issues are maybe fixed with additional -rcX tags. When they're happy, usually Friday or Monday, the newest one gets promoted to latest.
This week I believe there was an update but we didn't change the UI tag it contains because I've been working on a large set of changes to update to Ember.js 1.13 and it wasn't stable yet.  So long story short, likely next Friday.
If you click on the Endpoint in the API & Keys page, then navigate up to /v1/settings/api.ui.index, that is the URL for the UI that the backend serves up. Clicking edit and replacing the version number with latest will always give you what's in the master branch of this repo, which would include your PR.   We use this internally and don't normally merge broken stuff to master but there's no strong guarantee that its stable at any given time.  (Hmm... Hopefully the API UI is accessible...)

Where is the API and keys page? I've poked around the UI a bit but can't
seem to find it.
Thanks.

Oh, looks like there may be more inaccessible UI. The Getting Started
page references an account icon which I can't find. Betting it's also
missing a href/alt/aria-label attribute.


Add .with_args feature to rust library

This features exists in bossy::Command here: https://docs.rs/bossy/latest/bossy/struct.Command.html#method.with_args
It is quite useful, allowing you to write this:
let kill_cmd = session.command("/root/.cargo/bin/nu").withargs([
  "-c",
  r##"ps | filter {|ps| $ps.name == "surreal"} | get pid | each {|pid| kill $pid }"##,
]);

... instead of this:
let mut kill_cmd = session.command("/root/.cargo/bin/nu");
kill_cmd.args([
  "-c",
  r##"ps | filter {|ps| $ps.name == "surreal"} | get pid | each {|pid| kill $pid }"##,
]);

This saves using the mut keyword and a new line, and can even be implemented trivially using an extension trait

That does sound nice to have!
We can also have a more general
fn with_command(self, f: impl FnOnce(&mut Self)) -> Self;

I would accept a PR for this.

@NobodyXu
I am checking out the codebase and have some basic understanding. The suggestion seems like a interesting addition of API. If were to implement the function signature you suggested, based on my understanding that would be an addition of something like
// in src/session.rs, I presumed?
// presuming Self == openssh::Session?
pub fn with_command(self, f: impl FnOnce(&mut Self)) -> Self { todo!() }

Could you give a concrete example of what would be a instance of function/closure of as the argument of the type: f: impl FnOnce(&mut Self) to call the new API?
I just started playing with the codebase and don't see what such instance would be like (presumably a block of code involving use of some of the existing APIs to do some business, but I just don't visualise how that might be the case)

@mingmamma the function will be added to openssh::OwningCommand, which has many functions taking a mutable reference, such as arg, stdin, stdout.


ModSecurity2 write LogsParts not allowed

Hi,
I removed section E and F from SecAuditLogParts:
SecAuditLogParts ABIJDHZ
But after a block in my wordpress page, modsecurity.log generate E Section (--a49cd13e-E--), according E6u.txt attached:
E6u.txt
But I made simple test, like /phpfilethatdonotexist.php?something=../../etc/passwd, and the Section "E" isn't written.
Server:

ModSecurity 2.9.5
WebServer: Apache HTTPD Server 2.4.53
OS: FreeBSD 13.1

Rule Set:

Last from GIT: https://github.com/coreruleset/coreruleset.git

Hello @devnull-hub-lab ,
Even though you have that setting for SecAuditLogParts, what you are seeing could be because of a "ctl:auditLogParts=+E" action for one of the rules triggered for that HTTP transaction.
In your case rule 953110 is one of the rules listed in the audit log that you attached. And in the version of that rule that I just checked at their github site, it does indeed include "ctl:auditLogParts=+E"

Hi @martinhsv,
Thanks for your reply, and sorry for any inconvenience.
I understand this, but I not agree with this "ctl:auditLogParts=+E" in the rules.
They are bypassing what was defined in modsecurity.conf, which should be mandatory for logging all rules.
Maybe if there was a directive in crs-setup.conf where it could be defined which LogParts is mandatory...
I removed "ctl:auditLogParts=+E" but don't worked. But there are several other places with this directive, it's probably going through another point. I'll do a fine analysis on the rules.
Thanks again, Martin!

I think it can be closed. I disagree with this directive in coreruleset, but it is not an issue in ModSecurity.
Thanks @martinhsv.


Update energy-overview-card.ts

SVG line was not responsive in smaller size, it had overflow on the left.
Removing height solved the issue, but aligned the svg lower to the bottom compared to the original solution, which seemed to be fixed by adding 3px padding to the bottom.
Before:

After:

Oh, sorry about that, I forgot to test it on greater screen, I will working on it.


Login should keep username

Login/Logout -> don't delete username or better: use username of last login... (maybe it was an autologin)

also show username if it's available after startup via login view. Currently no username is shown.


Optimizations for T0

I'm trying to replicate the T5 ONNX optimization notebook (the latest version, on the feat/t5_3b branch) but for T0_3B (which in itself is a derivative of T5, but with a slightly different config and no tie_word_embeddings.
I installed ONNX runtime from source as described in the notebook.
The only changes I made to the notebook are replacing "t5-3b" with "bigscience/T0_3B", and commenting out out_dec["last_hidden_state"] = out_dec["last_hidden_state"] * (pytorch_model.model_dim**-0.5) in the ExportT5 class, as T0 does not use tie word embeddings.
However, the notebook fails on dec_if_ort_model = create_model_for_provider(dec_if_model_path, "CUDAExecutionProvider", log_severity=3), with the error: Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from ./test-dec-if/model.onnx failed:This is an invalid model. Error: the graph is not acyclic.
Shouldn't T0 work because it is essentially T5? Your help would be greatly appreciated @pommedeterresautee. Thanks!

Hi @michaelroyzen ,
I run it on my side and it worked perfectly (with the same modifications than those you did).
Closest model to T0 is t5_v1-1 as it uses gated RELU instead of RELU (T5) in FF step.
I have also tried with google/t5-v1_1-small for a rapid test and it also worked perfectly.
Can you try with the small one?

I really meant onnx library not onnx runtime. It's the one used to manipulate computation graph and it may be key in the merge process.

Gotcha, we do use onnx 1.11.0. Just tried with google/t5-v1_1-small and got InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from ./test-dec-if/model.onnx failed:This is an invalid model. on dec_if_ort_model = create_model_for_provider(dec_if_model_path, "CUDAExecutionProvider", log_severity=3).
One thing to note is we are using the nvcr.io/nvidia/pytorch:22.04-py3 Docker container with CUDA 11.6 -- we tried building it with nvcr.io/nvidia/pytorch:21.07-py3 which has CUDA 11.4, but then we get import issues when trying to use it in the notebook.

I did my test locally, Ubuntu 22.04, CUDA 11.7 and any Nvidia dependency up to date basically.
Instinctively I think it's not related to CUDA drivers, etc. It's something related to the graph.
Would it be possible to set log severity to 0 and share logs?
If onnx is not the cause, may be Pytorch ?
I would test with NVC 22.02 which seem to be the last one to be built with Pytorch 1.11 (22.04 is based on a 1.12 nightly):
https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-04.html#rel_22-04

I got it to work just now actually using CUDA 11.4 from the 21.08 image and then manually bumping ONNX to 1.11.0 and PyTorch to 1.11. Thanks!
Now I'm dealing with CUDA OOM when trying to run the benchmarks (hardware: A10 GPU w/ 24GB VRAM), it seems that loading enc_fp16_onnx = create_model_for_provider(encoder_fp16_model_path, "CUDAExecutionProvider", log_severity=3) enc_fp16_onnx_binding: IOBinding = enc_fp16_onnx.io_binding() dec_onnx = create_model_for_provider(dec_if_fp16_model_path, "CUDAExecutionProvider", log_severity=3) dec_onnx_binding: IOBinding = dec_onnx.io_binding() takes up all of GPU memory.

The issue is with past states which of course become bigger and bigger with model size + beam search + long seq len.
With the code on the repo, the best you can do is 128 tokens with beam search 2 on 24Gb RAM (at least on a 3090RTX).
We are working on reducing memory footprint, right now by a quite simple modif of the T5 code in transformers (that we plan to apply through monkey patching) we are able to double the max beam search (or probably seq len, not tried :-)).
We think there is a way to go much further in RAM consumption for large models by leveraging EL-Attention:
https://arxiv.org/abs/2105.04779
Some code has been open sourced (fastseq https://github.com/microsoft/fastseq) and we have been able to verify the benchmarks of the paper (including on T5).
Unfortunately the open source code is written in such a way that it needs some refactoring to be used IRL (and//or on recent version of transformers lib).
Is it something you would be interested to work on? if yes we could share with you know how we think the code can be adapted.

Is the reason that the ONNX model is ~2x the memory size of the PyTorch one because of the IF node merging the decoders with and without cache? If so, I'd be interested in discarding the version without cache to save that memory.
Fastseq looks interesting, it'd be great if you could share how you think it could be adapted.

Hi, it seems that the official Pytorch 1.12 breaks GPT-2 unit test and probably T5.
We will dig into it.
Regarding the fastseq thing, after analyzing the few thousands of loc in the repo, it appeared that the modification to get most of the perf are quite light. For now, our approach is to monkey patch source code. For that we compare fastseq source code with transformers version 4.12.5.
To test the approach we have isolated a small (in LoC) optimization of T5 responsible of 50% of the whole speed-up.
It's in  #103 .
Next step is to apply the same approach for the cache deduplicating thing related to beam search.
Let us know if you need more info.

Running the new t5_bf16 notebook with T0 results in this error when exporting the encoder:
`RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_632/3381225355.py in
----> 1 convert_to_onnx(
2     model_pytorch=pytorch_model.encoder,
3     output_path=encoder_model_path,
4     inputs_pytorch={"input_ids": input_ids},
5     var_output_seq=True,
/opt/conda/lib/python3.8/site-packages/transformer_deploy/backends/pytorch_utils.py in convert_to_onnx(model_pytorch, output_path, inputs_pytorch, quantization, var_output_seq, output_names)
151         input_names = list(inputs_pytorch.keys())
152     with torch.no_grad():
--> 153         torch.onnx.export(
154             model_pytorch,  # model to optimize
155             args=tuple(inputs_pytorch.values()),  # tuple of multiple inputs
/opt/conda/lib/python3.8/site-packages/torch/onnx/init.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
303
304     from torch.onnx import utils
--> 305     return utils.export(model, args, f, export_params, verbose, training,
306                         input_names, output_names, aten,
307                         operator_export_type, opset_version, _retain_param_name,
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
85         else:
86             operator_export_type = OperatorExportTypes.ONNX
---> 87     _export(model, args, f, export_params, verbose, training, input_names, output_names,
88             operator_export_type=operator_export_type, opset_version=opset_version,
89             _retain_param_name=_retain_param_name, do_constant_folding=do_constant_folding,
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, _retain_param_name, do_constant_folding, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, enable_onnx_checker, use_external_data_format, onnx_shape_inference)
690
691             graph, params_dict, torch_out =
--> 692                 _model_to_graph(model, args, verbose, input_names,
693                                 output_names, operator_export_type,
694                                 example_outputs, _retain_param_name,
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in _model_to_graph(model, args, verbose, input_names, output_names, operator_export_type, example_outputs, _retain_param_name, do_constant_folding, _disable_torch_constant_prop, fixed_batch_size, training, dynamic_axes)
468     params_dict = _get_named_param_dict(graph, params)
469
--> 470     graph = _optimize_graph(graph, operator_export_type,
471                             _disable_torch_constant_prop=_disable_torch_constant_prop,
472                             fixed_batch_size=fixed_batch_size, params_dict=params_dict,
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in _optimize_graph(graph, operator_export_type, _disable_torch_constant_prop, fixed_batch_size, params_dict, dynamic_axes, input_names, module)
196         dynamic_axes = {} if dynamic_axes is None else dynamic_axes
197         torch._C._jit_pass_onnx_set_dynamic_input_shape(graph, dynamic_axes, input_names)
--> 198     graph = torch._C._jit_pass_onnx(graph, operator_export_type)
199     torch._C._jit_pass_lint(graph)
200
/opt/conda/lib/python3.8/site-packages/torch/onnx/init.py in _run_symbolic_function(*args, **kwargs)
346 def _run_symbolic_function(*args, **kwargs):
347     from torch.onnx import utils
--> 348     return utils._run_symbolic_function(*args, **kwargs)
349
350
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in _run_symbolic_function(g, block, n, inputs, env, operator_export_type)
995                     return None
996                 attrs = {k: n[k] for k in n.attributeNames()}
--> 997                 return symbolic_fn(g, *inputs, **attrs)
998
999         elif ns == "prim":
/opt/conda/lib/python3.8/site-packages/torch/onnx/symbolic_helper.py in wrapper(g, *args, **kwargs)
170             if len(kwargs) == 1:
171                 assert "_outputs" in kwargs
--> 172             return fn(g, *args, **kwargs)
173
174         return wrapper
/opt/conda/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py in embedding(g, weight, indices, padding_idx, scale_grad_by_freq, sparse)
485                       "ONNX does not support not updating the embedding vector at padding_idx during training.")
486
--> 487     return g.op("Gather", weight, indices)
488
489
/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py in _graph_op(g, opname, *raw_args, **kwargs)
891     if _onnx_shape_inference:
892         from torch.onnx.symbolic_helper import _export_onnx_opset_version as opset_version
--> 893         torch._C._jit_pass_onnx_node_shape_type_inference(n, _params_dict, opset_version)
894
895     if outputs == 1:
RuntimeError: unexpected tensor scalar type`


Unable to deploy as no mock.json from deploy/results

How do I generate the json files in results in order to deploy the contracts? Or am I doing sth out of the order?

You can put this into mock.json
[
    {
      "info": "",
      "queuedAt": ",
      "executedAt": "",
      "executionTransaction": "",
      "target": "",
      "value": "",
      "signature": "",
      "paramTypes": [],
      "params": [],
      "eta": "1630122300"
    }
]```


Call setupPlaceList() onResume in NearbyActivity

In our current master, there is an issue with Nearby Places not loading at all - the progress bar runs forever. After digging around a bit, I figured the reason is that setupPlaceList(this) is only being called in refreshView and not when the activity is loaded.
This fixes that problem in my tests, but there is a small issue where the very first time Nearby loads (on a newly started application), it says "No Nearby Places found", probably because the GPS hasn't had time to get a fix yet. The second time onwards, this works fine.
But would be great if anyone could build on this to fix that, perhaps by delaying the first load until a location is found?

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@9888a07). Click here to learn what that means.
The diff coverage is 100%.

@@           Coverage Diff            @@
##             master    #976   +/-   ##
========================================
  Coverage          ?   7.06%
========================================
  Files             ?     100
  Lines             ?    5240
  Branches          ?     485
========================================
  Hits              ?     370
  Misses            ?    4839
  Partials          ?      31

Impacted Files
Coverage Δ

...ava/fr/free/nrw/commons/nearby/NearbyActivity.java
42.26% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9888a07...7a50a7d. Read the comment docs.

Switched to using refreshView as per the review, it solves the initial loading issue as well. :)


How to undo or remove a fastBlur ?

I simply want to blur the whole screen / whole activity (as when adding a typical menu on the left).
I guess I should use
BlurKit.getInstance().fastBlur(activity_root_view , 5, 1f);
How to undo this when the menu goes away?  Thanks.

I would like to know this as well


wip: Gen events falco engine

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

If contributing rules or changes to rules, please make sure to also uncomment one of the following line:

/kind rule-update

/kind rule-create

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine
/area rules

/area tests

/area proposals

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

Refactor falco engine to be agnostic to specific event sources. This allows creating a "lite" version of falco that does not have explicit syscall support if we wish.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

5b56eed Squash w/ falco_formats only formats events now, no lua bindings

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

e02a7e6 Squash w/ tests commit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

18f60d2 TEMP: Use non-master branch of falcosecurity/libs

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Support native objc Gradle building

I don't use Xcode directly**, instead I prefer to use the Gradle wrapper for Xcode clang toolchain.
See https://docs.gradle.org/current/userguide/nativeBinaries.html
All functionality (that I need at least) of both j2objcc and Xcode pods can be done in Gradle directly.  So I'm basically using the j2objcTranslate functionality, and then doing the rest in gradle (without j2objcTest or xcode pods).  My current setup is at:
https://gist.github.com/advayDev1/636f34fa80164645e205
This sets up normal Gradle tasks for building my objc libraries, building test executables, and running tests.
I think folks might want to use Xcode pods or Gradle builds, and the core plugin probably should not enforce either.  Neither actually are very coupled to j2objc, which is purely translation.  I propose we have a tools/ subdirectory or similar that supports pods and gradle native builds, so users can choose what they want.
** For many reasons, most importantly:

Xcode doesn't handle non-flat source directories very well.
I prefer to use one build toolchain (and one IDE when needed, IDEA/Android Studio) without denormalization/duplication of configuration.
Using a single Gradle build ensures a single presubmit/build command builds and tests all platforms and all flavors of my projects.

This is very interesting Advay. We should discuss this in person when we
meet up.
On Tue, May 19, 2015 at 10:48 PM Advay Mengle<EMAIL_ADDRESS>wrote:

I don't use Xcode directly**, instead I prefer to use the Gradle wrapper
for Xcode clang toolchain.
See https://docs.gradle.org/current/userguide/nativeBinaries.html
All functionality (that I need at least) of both j2objcc and Xcode pods
can be done in Gradle directly. So I'm basically using the j2objcTranslate
functionality, and then doing the rest in gradle (without j2objcTest or
xcode pods). My current setup is at:
https://gist.github.com/advayDev1/636f34fa80164645e205
This sets up normal Gradle tasks for building my objc libraries, building
test executables, and running tests.
I think folks might want to use Xcode pods or Gradle builds, and the core
plugin probably should not enforce either. Neither actually are very
coupled to j2objc, which is purely translation. I propose we have a tools/
subdirectory or similar that supports pods and gradle native builds, so
users can choose what they want.
** For many reasons, most importantly:

Xcode doesn't handle non-flat source directories very well.
I prefer to use one build toolchain (and one IDE when needed,
IDEA/Android Studio) without denormalization/duplication of configuration.
Using a single Gradle build ensures a single presubmit/build command
builds and tests all platforms and all flavors of my projects.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76.

@brunobowden Yep will do.
@confile pointed out that perhaps we should be directly using pods on another issue, without the raw copy.  This issue addresses yet another alternative.  Maybe we can continue the discussion here.

btw, i upgraded one of my projects to gradle 2.4 today and using native
clang compiler is way faster/more parallel (and gets incremental build
correct) than exec-ing j2objcc (for the j2objcCompile task)...
On Wed, May 20, 2015 at 3:31 PM Bruno Bowden<EMAIL_ADDRESS>wrote:

This is very interesting Advay. We should discuss this in person when we
meet up.
On Tue, May 19, 2015 at 10:48 PM Advay Mengle<EMAIL_ADDRESS>wrote:

I don't use Xcode directly**, instead I prefer to use the Gradle wrapper
for Xcode clang toolchain.
See https://docs.gradle.org/current/userguide/nativeBinaries.html
All functionality (that I need at least) of both j2objcc and Xcode pods
can be done in Gradle directly. So I'm basically using the
j2objcTranslate
functionality, and then doing the rest in gradle (without j2objcTest or
xcode pods). My current setup is at:
https://gist.github.com/advayDev1/636f34fa80164645e205
This sets up normal Gradle tasks for building my objc libraries, building
test executables, and running tests.
I think folks might want to use Xcode pods or Gradle builds, and the core
plugin probably should not enforce either. Neither actually are very
coupled to j2objc, which is purely translation. I propose we have a
tools/
subdirectory or similar that supports pods and gradle native builds, so
users can choose what they want.
** For many reasons, most importantly:

Xcode doesn't handle non-flat source directories very well.
I prefer to use one build toolchain (and one IDE when needed,
IDEA/Android Studio) without denormalization/duplication of
configuration.
Using a single Gradle build ensures a single presubmit/build command
builds and tests all platforms and all flavors of my projects.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-104062500
.

Can you make this an option? Then we can consider switching it to the
default over time.
On Sun, May 31, 2015 at 5:12 PM Advay Mengle<EMAIL_ADDRESS>wrote:

btw, i upgraded one of my projects to gradle 2.4 today and using native
clang compiler is way faster/more parallel (and gets incremental build
correct) than exec-ing j2objcc (for the j2objcCompile task)...
On Wed, May 20, 2015 at 3:31 PM Bruno Bowden<EMAIL_ADDRESS>wrote:

This is very interesting Advay. We should discuss this in person when we
meet up.
On Tue, May 19, 2015 at 10:48 PM Advay Mengle<EMAIL_ADDRESS>wrote:

I don't use Xcode directly**, instead I prefer to use the Gradle
wrapper
for Xcode clang toolchain.
See https://docs.gradle.org/current/userguide/nativeBinaries.html
All functionality (that I need at least) of both j2objcc and Xcode pods
can be done in Gradle directly. So I'm basically using the
j2objcTranslate
functionality, and then doing the rest in gradle (without j2objcTest or
xcode pods). My current setup is at:
https://gist.github.com/advayDev1/636f34fa80164645e205
This sets up normal Gradle tasks for building my objc libraries,
building
test executables, and running tests.
I think folks might want to use Xcode pods or Gradle builds, and the
core
plugin probably should not enforce either. Neither actually are very
coupled to j2objc, which is purely translation. I propose we have a
tools/
subdirectory or similar that supports pods and gradle native builds, so
users can choose what they want.
** For many reasons, most importantly:

Xcode doesn't handle non-flat source directories very well.
I prefer to use one build toolchain (and one IDE when needed,
IDEA/Android Studio) without denormalization/duplication of
configuration.
Using a single Gradle build ensures a single presubmit/build command
builds and tests all platforms and all flavors of my projects.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76.

—
Reply to this email directly or view it on GitHub
<
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-104062500
.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-107261781
.

I'll document some stuff for our Tuesday get together and we can decide
whether it is the way to go.  SG?
On Sun, May 31, 2015, 5:17 PM Bruno Bowden<EMAIL_ADDRESS>wrote:

Can you make this an option? Then we can consider switching it to the
default over time.
On Sun, May 31, 2015 at 5:12 PM Advay Mengle<EMAIL_ADDRESS>wrote:

btw, i upgraded one of my projects to gradle 2.4 today and using native
clang compiler is way faster/more parallel (and gets incremental build
correct) than exec-ing j2objcc (for the j2objcCompile task)...
On Wed, May 20, 2015 at 3:31 PM Bruno Bowden<EMAIL_ADDRESS>wrote:

This is very interesting Advay. We should discuss this in person when
we
meet up.
On Tue, May 19, 2015 at 10:48 PM Advay Mengle <
<EMAIL_ADDRESS>wrote:

I don't use Xcode directly**, instead I prefer to use the Gradle
wrapper
for Xcode clang toolchain.
See https://docs.gradle.org/current/userguide/nativeBinaries.html
All functionality (that I need at least) of both j2objcc and Xcode
pods
can be done in Gradle directly. So I'm basically using the
j2objcTranslate
functionality, and then doing the rest in gradle (without j2objcTest
or
xcode pods). My current setup is at:
https://gist.github.com/advayDev1/636f34fa80164645e205
This sets up normal Gradle tasks for building my objc libraries,
building
test executables, and running tests.
I think folks might want to use Xcode pods or Gradle builds, and the
core
plugin probably should not enforce either. Neither actually are very
coupled to j2objc, which is purely translation. I propose we have a
tools/
subdirectory or similar that supports pods and gradle native builds,
so
users can choose what they want.
** For many reasons, most importantly:

Xcode doesn't handle non-flat source directories very well.
I prefer to use one build toolchain (and one IDE when needed,
IDEA/Android Studio) without denormalization/duplication of
configuration.
Using a single Gradle build ensures a single presubmit/build
command
builds and tests all platforms and all flavors of my projects.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76.

—
Reply to this email directly or view it on GitHub
<

https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-104062500

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-107261781
.

—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-107261916
.

@advayDev1 Not sure what you are talking about but I will have a look at your documentation.

For now take a look at the gist linked to from the first post.  It is close
to what I have running internally and should be self explanatory if you
also read the gradle native build link in the same post.
Re my last comment: In gradle 2.4 they made native compiles run in parallel
according to the number of cores you have.
On Sun, May 31, 2015, 5:35 PM Michael Gorski<EMAIL_ADDRESS>wrote:

@advayDev1 https://github.com/advayDev1 Not sure what you are talking
about but I will have a look at your documentation.
—
Reply to this email directly or view it on GitHub
https://github.com/brunobowden/j2objc-gradle/issues/76#issuecomment-107269859
.


ignore next fails for ES6 function in object literal

I have a repo case:
https://github.com/Pyrolistical/istanbul-ignore-shorthand-bug
It should be 100% coverage, but it fails to ignore ES6 function in object literals.
Here is a screenshot of the coverage report.

The function ignoreAboveShorthand fails to be ignored.  The workaround is to write it like the ignoreInFrontShorthand function or use longhand like ignoreAboveLongHand

For some reason the workaround doesn't work when using jest.  I had to use another workaround to put the /* istanbul ignore next*/ inside the body of the function

I'm encountering the same issue with Jest, any updates on this?

Does anyone know what can be done about this when using Jest?


Use utf8 for output file when downloading schema

Should fix https://github.com/rossumai/elisctl/issues/32 but it needs to be tested on windows yet.

Manually tested on windows. This PR solves th error.


Build Docker images

As part of our effort to support Kubernetes, we need to deploy the agent as a Docker image which can then be retrieved later as part of a container. It can also be used by any other system that is dependent upon Docker.
The image is built on a Linux Alpine base in order to produce an extremely small image.
CircleCI is set up to only build and deploy new versions of the image when a new version is published (by pushing a new tag starting with v).
Closes #22

Just FYI, compiled go programs can run in a scratch docker image (i.e. nothing but the underlying kernel). That'll get you an even more minimal image size.

@lukesteensen Thanks! I didn't even think of that.


chore(examples/web): use exported strings for attributes

Which problem is this PR solving?

Part Of #2025

Short description of the changes
On package examples/web:

Update @opentelemetry/semantic-conventions ^1.23.0
Use exported strings for Semantic Attributes.

/easycla


Responsive style improvements

Summary | Résumé
Improving the responsive rendering of form-builder.

Couple of small things
Could add some padding to the top here:

Headings end up too small (on smaller screens)

Maybe we can wrap the CSV button to drop under for smaller screens


GAS balance incorrect. negative claimable GAS

[TestNet] Progress: 1684947/1684947
Current behavior

GAS balance incorrect
available claims is negative

output of wallet command
"percent_synced": 100
...
"script_hash": "ALXncHucqcqrM3GocHrc7GkRToWN3zXyyV",
"frozen": false,
"votes": [],
"balances": {
    "0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7": "510.0"
}
...
"claims": {
    "available": "-4999994.04742",
    "unavailable": "0.0"
}

Expected behavior

GAS balance should be 0
claimable GAS in the wallet should be a positive number

See https://neoscan-testnet.io/address/ALXncHucqcqrM3GocHrc7GkRToWN3zXyyV
for transaction hisotry
How to reproduce

New install of neo-python 0.7.5 using pip
np-prompt to sync testnet
import wif (I can provide wif privately)
tried wallet rebuild using 0.7.5 and 0.7.6 but didn't help

Your environment
Let us know in what environment you're running into the issue:

OS: Ubuntu 16.04.5 LTS
neo-python version: 0.7.6
Python version: Python 3.6.6

Updated to 0.7.7.dev0 (development branch)
Purged my local testnet chain and re-synced. Also rebuilt the wallet.
[TestNet] Progress: 1691020/1691020
            "version": 0,
            "script_hash": "ALXncHucqcqrM3GocHrc7GkRToWN3zXyyV",
            "frozen": false,
            "votes": [],
            "balances": {
                "0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7": "510.0"
            },

...
    "claims": {
        "available": "2000011.22289",
        "unavailable": "0.0"
    }

The numbers are still incorrect.
The three addresses in the wallet are
ARBnLVCAMcZeDZnvQSfn9q5zZ7eEyQu9wi
ALXncHucqcqrM3GocHrc7GkRToWN3zXyyV
AKe5QcddN31d8BDjn29CEoFoi2jb3fzvJ9

I'm also seeing similar issues.

+1

closing due to inactivity and issue likely to be solved by #611


Energy data set generation

Hello, I tried to make my own data set to check the network training effect. The training effect of energy is very bad, which is three orders of magnitude lower than the original ethanol energy. The energy I used is the total energy produced by LAMMPS in each step of training water molecules.May I ask what is the specific energy of the ethanol data set provided by Wu Jie?

Hey @mukui123 - sorry to hear that the training isn't going well. My guess is that it has something to do with units. What are the units of energies, forces, and coordinates? The force units should be the energy units divided by the coordinate units, in order for training to work. One way you can test this hypothesis is by setting the energy loss coefficient to 0 and seeing if force training improves. (Because if you're only predicting one quantity then the units don't matter.)
As for Wujie's dataset, I believe that it's data from the MD17 benchmark.
Let me know if this helps!

Also, you said the energy is three orders of magnitude lower than the true result. On the graph they seem to be similar orders of magnitude? But if they are very different, I'd suggest subtracting the mean energy from the data before training. That can have a big impact on performance.

Also, you said the energy is three orders of magnitude lower than the true result. On the graph they seem to be similar orders of magnitude? But if they are very different, I'd suggest subtracting the mean energy from the data before training. That can have a big impact on performance.

Thanks for your advice. I'll try again！

No problem!


Issue crafting Petro Petunia (Dragonstone Use)

The recipe states to use a Dragonstone, however this can't be added to the petal apothecary.

The recipe doesn't use a dragonstone? Either way, Botania changed the petal apothecary to accept any of item.

Ah, I had the wrong branch on local. I now see the recipe uses a dragonstone, but I can confirm it works with latest botania.


C\error\TempConv.c

Find the errors in the code and correct them.

Hi @Taanvee please assign this to me.


try to stabilize the petrale model

look into bootstrapping petrale and getting rid of params on bounds. Does this make the model more stable/faster?
Currently, this is not included in the gh actions tests because it runs too slowly. On jenkins, it sometimes found a different solution, likely because there were params on bounds.

Petrale was removed because its features are already captured in other models in the test set


docs: add "why" section

Convince users (and myself TBH) by:

Adding design principles so users may agree with them
Add comparison to visually see why our lib from several viewpoints

Improves index by splitting between new users & existing users.
Improve get started next steps

Current dependencies on/for this PR:

#345  👈
main

This stack of pull requests is managed by Graphite.


fix when re-adding nft burning old nft

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.01% :tada:

Comparison is base (8dc3e74) 63.56% compared to head (f5e467b) 63.58%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #393      +/-   ##
==========================================
+ Coverage   63.56%   63.58%   +0.01%
==========================================
  Files          79       79
  Lines       22303    22314      +11
==========================================
+ Hits        14177    14188      +11
  Misses       8126     8126

Files Changed
Coverage Δ

pallets/adsc/src/lib.rs
69.80% <100.00%> (+0.29%)
:arrow_up:

pallets/adsc/src/tests.rs
96.96% <100.00%> (+0.15%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update Armenian keymaps

This adds a couple of mappings that were accidentally omitted from the previous version. Thank you for including these keymaps!

I'll include it, thanks.


Tag releases in Git(Hub)

There have been quite a few releases of sendgrid-django, as indicated by PyPI, but these releases don't appear to be tagged on GitHub. For those of us who don't upgrade as fast as everyone else, this makes it difficult to hunt down the source for our version of the package when we want to view it on GitHub.
Would it be possible for a tag to be created or a release to be created for future versions of sendgrid-django, to make it easier to track down versions of the package and compare them?

@kevin-brown It's good idea and practice.
I think we should add something to project like https://github.com/peritus/bumpversion
I can create PR :)
What do you think @elbuo8

@andriisoldatenko that looks interesting. Go for it 👍

👍
In addition, it'd be good to clearly explain backwards incompatible changes and the upgrade steps. I hadn't upgraded in about 9 months and had to carefully read the commit history in order to figure out that the SENDGRID_USER/SENDGRID_PASSWORD settings were removed in favor of SENDGRID_API_KEY.

+1 A CHANGELOG document would also be extremely useful. I'm at version 1.4.0 and trying to figure out if I can upgrade to 4.0.4 is quite a challenge.
But, even just tagging the releases would be extremely helpful!


Hunger Games prompts not displayed on some screen resolutions

@raphael0202 reports on a (French OS) (French app) (Logged in to OFF) that he does not see the Hunger Games prompts. That might have to do with the ratio of his screen, possibly.

We have only one layout, and it's here: https://github.com/openfoodfacts/openfoodfacts-androidapp/blob/develop/app/src/main/res/layout/activity_product_browsing_list.xml#L47


API Schema for the service("people") controlling the State

Proposed API Schema
=> also working with waypoint

GET /vehicles/{region}

   {
      "last_update": "timestamp",
      "vehicles": [
         { ... waypoint ... for every vehicle one }
      ]
   }

or more structured
   {
    "last_update": "timestamp",
      "vehicles": {
        "4": {
          "20": { last waypoint },
          "33" : { last waypoint }
        },
        "61": {
           "2": { last waypoint },
           "3": { last waypoint }
        }
      }
   }

Ideas of other features

Detailed Vehicle Information (would require saving data in memory for some time)

History of Delays
Path Took

Region Detailed Information

Average of Delays
Count of Vehicles that currently drive around (grouped by line)

I think the first version is better? Because I would convert the second one into the first one otherwise.

I'm not sure if we want to carry extra data around in the same response. Per @MarcelCoding just a list of waypoints is easier for frontend, so I would just go with it. For anything else, if we need it, I would just move it to separate endpoint (since merging this into just a list of waypoints will be a pain).

You mean don't send the coordinates directly? How should interpolation then be done?

Nope, I only mean "extra features" mentioned above

Ohh, I can't read, sorry xd

@MarcelCoding U mean that we also would send the interpolated data over the websocket ? And the frontend would be reduced to pure display minimal logic ?

Idk, at least that is how it is done currently. There would alps be other options. Idk, I think it does not realy matter how it is done.

Yeah thats a different discussion I would say

So I started building variant number 1. With the extra caveat that I will save in memory a history of time X.

@revol-xut maybe worth looking at memcached?

@oxapentane maybe worth for scalability but my worst calculation would result of holding approx a 1GB in Memory Assuming we get approx 16 Waypoints per Second (currently we get like 2)


Show hooks in devtools

Getting started...

[x] Make hooks panel show up for components with hooks
[ ] Display hook name
[ ] Display hook state
[ ] Make hook state editable
[ ] Show nested hooks

Fixes #1381 .

This is a lot easier with the next iteration of the react-devtools. It will probably be published soon which is why I don't think it's worth to spent more time with our existing integration.
Support for hooks will be added in #1549.


Custom quad metadata support and querying capability

Quadmetadata Proposal -
https://hackmd.io/BwdgZgJhCmBGwFoBMJpgQFhAQ2g20EAbAmPEgIwDMAnMDSEjUA==
Salient features in this pull request
Add a field called 'QuadMetadata' as another direction to quad.Quad.
QuadMetadata is a map[string]string and relies on the Value type 'QuadRef'.
Modify usage of quad getDirection in the Elastic backend to support this field.
Add Gizmo functions InE, OutE, and BothE to support metadata querying
Add time range querying and regex querying capability on quadmetadata to the elastic backend
Pass in elasticsearch index name as an option (through cayley_example.yml file) instead of hardcoding
Elastic implementation details:

The values in the map are used as unique identifiers for a quad (they are used to form the hash value for the quad). For example, if a given quad has metadata {"index1": "val1", "index2": "val2"}, and another quad has metadata {"index1": "val3", "index2": "val4"}, then these quads are unique.
QuadMetadata is not a node; it is associated only with the quad.
OutE function syntax (same for InE and BothE): path.OutE(predicate,{metadata_field: filter_expression})
Time range query example: g.V("SubjectNode").OutE("PredicateNode", {"field1":"value1", "timestamp":"2015-01-01=>2015-01-15"}).All()
The arrow symbol '=>' is used as a separator between the starting time and ending time. Any field that is set as type 'date' in its elastic mapping will be interpreted as a time range query.
Regex query example: g.V("SubjectNode").OutE("PredicateNode", {"field1":"item.", "field2":"v.", "timestamp":"2015-01-01=>2015-01-15"}).All()
Any field that is not set as type 'date' in its elastic mapping can be queried with a regex expression. In the example above, field1 and field2 are queried with regex.

This change is

This should wait until https://github.com/cayleygraph/cayley/pull/634 is finished and rebased on top of it.
Plus, we will probably have a more generic implementation of quad metadata from our side.

Review status: 0 of 12 files reviewed at latest revision, all discussions resolved, some commit checks broke.

Comments from Reviewable

@dennwc Agreed, #634 comes first. We had a discussion with Barak on the quad metadata implementation and came up with this proposal - https://hackmd.io/BwdgZgJhCmBGwFoBMJpgQFhAQ2g20EAbAmPEgIwDMAnMDSEjUA==
If you have thoughts on a more generic implementation, I'd be happy to take that on.

Reviewed 1 of 4 files at r5, 1 of 11 files at r8, 5 of 14 files at r9.
Review status: 4 of 15 files reviewed at latest revision, 7 unresolved discussions, some commit checks broke.

a discussion (no related file):
The problem with this PR is that in only allows writing metadata to Elastic and nothing else. We should also design a solution for other backends.

cayley_example.yml, line 10 at r9 (raw file):

  # backend-specific options
  options:
    index: "cayley"

bolt backend has no index option

data/testdata2.nq, line 1 at r9 (raw file):

<brad> <attime> "2015-01-01" .

I haven't seen  any tests using this file. Is it necessary?

quad/quad.go, line 51 at r8 (raw file):

// Make creates a quad with provided values.
func Make(subject, predicate, object, label interface{}, quadmetadata interface{}) (q Quad) {

This will break compatibility. This should be a separate method MakeWithMeta or something similar

quad/quad.go, line 122 at r8 (raw file):

	Object       Value `json:"object"`
	Label        Value `json:"label,omitempty"`
	QuadMetadata Value `json:"quadmetadata,omitempty"`

I guess it would be better to make a field Metadata map[string]Value or Metadata map[quad.Direction]Value and make quad.Direction a string/IRI. The point is - metadata should not be any different from Label (we will just leave it as a separate field for compatibility).

quad/value.go, line 137 at r9 (raw file):

// QuadRef is a map from string to any value for holding extra Quad metadata
type QuadRef map[string]string

The problem with declaring maps as Values is that user will be able to write quads with this map in any other direction: subject, object, predicate. And I'm not even sure what this should mean in RDF world. So please check comment about making Metadata a map.

query/gizmo/traversals.go, line 22 at r9 (raw file):

	"fmt"

	//"github.com/cayleygraph/cayley/vendor/github.com/dop251/goja"

remove this

Comments from Reviewable

Review status: 4 of 15 files reviewed at latest revision, 7 unresolved discussions, some commit checks broke.

a discussion (no related file):
Previously, dennwc (Denys Smirnov) wrote…
The problem with this PR is that in only allows writing metadata to Elastic and nothing else. We should also design a solution for other backends.

Yes, I guess this PR has two parts. First part is altering structure of Quad to support the Metadata field and adding Gizmo methods to query it - this would be applicable across all backends.
The second part is backend-specific changes to support the Metadata field - even if this part was not added immediately for other backends, the updated Quad structure would not break the existing implementations (they just wouldn't be dealing with the Metadata field for the time being).

cayley_example.yml, line 10 at r9 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
bolt backend has no index option

Done.

data/testdata2.nq, line 1 at r9 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
I haven't seen  any tests using this file. Is it necessary?

No it's not, will remove.

quad/quad.go, line 51 at r8 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
This will break compatibility. This should be a separate method MakeWithMeta or something similar

Why does it break compatibility? The metadata case is addressed in an if statement:
if q.QuadMetadata, ok = AsValue(quadmetadata); !ok {
		q.QuadMetadata = String(fmt.Sprint(quadmetadata))
	}

quad/quad.go, line 122 at r8 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
I guess it would be better to make a field Metadata map[string]Value or Metadata map[quad.Direction]Value and make quad.Direction a string/IRI. The point is - metadata should not be any different from Label (we will just leave it as a separate field for compatibility).

I agree on the name change. My initial thought was to make it type map[string]Value but a lot methods in quadstore.go and iterator.go that are used in queries rely on the quad.Value and graph.Value types. Some examples:
func (qs *QuadStore) ValueOf(s quad.Value) graph.Value {

func (qs *QuadStore) NameOf(v graph.Value) quad.Value {

func hashOf(s quad.Value) string {

func toQuadValue(d elasticString) quad.Value {

func NewIterator(qs *QuadStore, resultType string, d quad.Direction, val graph.Value) *Iterator {

Which is why I thought it would be better to make the entire map as a Value so that I could refer to the entire Metadata field as a Value type while performing queries and not write special cases to parse it.
With regard to map[quad.Direction]Value, I thought quad.Direction was meant to refer to the fields of a quad and not any generic String, as below:
// Direction specifies an edge's type.
type Direction byte

// List of the valid directions of a quad.
const (
	Any Direction = iota
	Subject
	Predicate
	Object
	Label
	QuadMetadata
)

var Directions = []Direction{Subject, Predicate, Object, Label, QuadMetadata}

quad/value.go, line 137 at r9 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
The problem with declaring maps as Values is that user will be able to write quads with this map in any other direction: subject, object, predicate. And I'm not even sure what this should mean in RDF world. So please check comment about making Metadata a map.

Making Metadata a map causes problems with a lot of the existing methods in quadstore.go and iterator.go that rely on quad.Value type (I gave more detail on the other comment). I don't think the user would be able to write maps in another direction though because the JSON is marshalled into a rawQuad struct, which declares the types explicitly (quad.go):
type rawQuad struct {
	Subject      string            `json:"subject"`
	Predicate    string            `json:"predicate"`
	Object       string            `json:"object"`
	Label        string            `json:"label,omitempty"`
	QuadMetadata map[string]string `json:"quadmetadata,omitempty"`
}

func (q Quad) MarshalJSON() ([]byte, error) {
	rq := rawQuad{
		Subject:   q.Subject.String(),
		Predicate: q.Predicate.String(),
		Object:    q.Object.String(),
	}
	if q.Label != nil {
		rq.Label = q.Label.String()
	}
	if q.QuadMetadata != nil {
		rq.QuadMetadata = q.QuadMetadata.Native().(map[string]string)
	}
	return json.Marshal(rq)
}

So it would cause a type conversion error if a user tried to write a map into Subject for example.

query/gizmo/traversals.go, line 22 at r9 (raw file):
Previously, dennwc (Denys Smirnov) wrote…
remove this

Done.

Comments from Reviewable

Reviewed 2 of 3 files at r10.
Review status: 2 of 13 files reviewed at latest revision, 5 unresolved discussions, some commit checks broke.

a discussion (no related file):
Previously, camazotz (Navalava) wrote…
Yes, I guess this PR has two parts. First part is altering structure of Quad to support the Metadata field and adding Gizmo methods to query it - this would be applicable across all backends.
The second part is backend-specific changes to support the Metadata field - even if this part was not added immediately for other backends, the updated Quad structure would not break the existing implementations (they just wouldn't be dealing with the Metadata field for the time being).

I ported Elastic to NoSQL and will soon merge it (#657). Please rebase all your changes on top of it. This will make sure that your changes will work for Mongo as well.
Note: you need to rebase, not merge. In other cases you will have many unrelated changes in this PR.

graph/shape/path.go, line 14 at r9 (raw file):

	switch s1 := s1.(type) {
	case AllNodes:
		fmt.Println("allnodes")

remove debug prints

quad/quad.go, line 51 at r8 (raw file):
Previously, camazotz (Navalava) wrote…
Why does it break compatibility? The metadata case is addressed in an if statement:
if q.QuadMetadata, ok = AsValue(quadmetadata); !ok {
		q.QuadMetadata = String(fmt.Sprint(quadmetadata))
	}

It break compatibility for existing users that use Make - it will now accept 5 args instead of 4.

Comments from Reviewable

Closing because of conflicts. Feel free to rebase and resubmit PR.


problems running angularjs e2e testing with protractor in phantomjs

When doing E2E tests with Protractor and PhantomJS and doing things like
expect(element(by.repeater('book in library') row(0).column('{{book.name}}')).getText()).toBe('Some Text');
fails only in PhantomJS, but not in Firefox or Chrome.
Html

{{book.name}}

Protractor E2E test case:
it('test', function () {
var bookName = element(by.repeater('book in library track by id').row(0).column('{{book.name}}'));
expect(bookName.getText()).toEqual('test');
});
I am using angularjs 1.2.14, protractor:1.0.0-rc4, phantomjs:1.9.7-14 on linux This test case runs well in Chrome and Firefox but fails in PhantomJS and the error I get is
Error:Expected '' to equal 'test'
When I do count on rows works fine but trying to fetch column value, I am getting the empty string.
// works fine:
expect(element.all(by.repeater('book in library track by id')).count()).toEqual(9);
// test fails:
expect(bookName.getText()).toEqual('test');
Is there way to fix this?

@svuppala , I had a similar problem and worked around it by replacing :
return element(by.className('navbar-title')).getText();

by
return element(by.css('.navbar-title span')).getWebElement().getInnerHtml();

for the given html:

Select a theme

We have the same problem and also made it work by changing getText() to getInnerHtml() (without getWebElement()). Would this issue be Protractor specific or should we file an issue at the PhantomJS project?

For posterity:
getText() returns the text in the visible part of the element, or an empty string if the element is hidden somehow. See http://angular.github.io/protractor/#/api?view=webdriver.WebElement.prototype.getText
In my particular case, the viewport of phantomjs was ridiculously small (300x400 if i remember correctly).
I have a mediaquery that hides the element when the screen is narrow, so getText() was actually right: there was no text displayed.
Setting your phantomjs viewport to a reasonable size could also have an impact on the result of getText().

Sounds like the issue was small viewport. Closing as answered.


Merge master into the TensorFlow branch

Brings the tensorflow branch up to date with master.

Can confirm: all tests passing locally on linux. PTAL @dan-zheng ?

I believe we can trigger CI on tensorflow-merge -> tensorflow PRs.
@swift-ci Please test tensorflow

I built a toolchain and ran swift test on swift-apis.
Compilation succeeds when running swift test, but I ran into the known linker error on macOS:
2019-06-11 11:35:48.398 xctest[98347:18759624] The bundle “TensorFlowPackageTests.xctest” couldn’t be loaded because it is damaged or missing necessary resources. Try reinstalling the bundle.
2019-06-11 11:35:48.399 xctest[98347:18759624] (dlopen_preflight(/Users/danielzheng/swift-merge/tensorflow-swift-apis/.build/x86_64-apple-macosx/debug/TensorFlowPackageTests.xctest/Contents/MacOS/TensorFlowPackageTests): Library not loaded: /usr/lib/swift/libswiftPython.dylib
  Referenced from: /Users/danielzheng/swift-merge/tensorflow-swift-apis/.build/x86_64-apple-macosx/debug/TensorFlowPackageTests.xctest/Contents/MacOS/TensorFlowPackageTests
  Reason: image not found)

I think merging the PR now is reasonable.


Add NEB workflow for VASP

Because a typical input for NEB can take multiple forms, this issue will outline the different options.
Helpful links:

Atomate NEB yaml
Atomate NEB py
pymatgen-diffusion neb
pymatgen-diffusion pathfinder
VASP's guide for NEB
Plug-in guide for CI-NEB

Nudged elastic band is composed of the following stages...

Relax the starting bulk structure

Identify all symmetrically unique pathways

*** and then the remaindering steps are done for each individual pathway ***
*** NOTE: this guide only looks at the FIRST diffusion pathway ***

Relax the start/end supercell images (or only one of these if they are equivalent)

Interpolate the start/end supercell images and empirically relax these using IDPP.

Relax all images using NEB

Input parameters
For diffusion analysis, a user may want to start at a different step -- for example, they could have a specific start/end structure that they want to provide rather than having automatically detection of unique pathways. Therefore, we break this workflow in multiple smaller ones. We must account for the following input scenarios:

neb_all_paths: a bulk crystal + diffusing species is given and full analysis of all paths is requested

neb_single_path: a bulk crystal + diffusing species + pathway index is given, where index gives the specific pathway to be grabbed from DistinctPathFinder

neb_from_indices: a bulk crystal + start/end index for diffusing ion is given and that specific pathway should be analyzed

neb_from_endpoints: two endpoint supercell structures are given and that specific pathway should be analyzed

neb_from_images: a series of images are given and that specific pathway should be analyzed

On top of these 5 scenarios, I should also consider...

vacancy vs. interstitial diffusion
plugins for CI-NEB

@laurenmm @scott-materials Is there a scenario that I overlooked in the "input parameters" section above? I'm trying to think of other inputs that our lab would want.

NEB will be added in #70

Yes, I think this captures all likely input scenarios.
Separately, from your description, I'm unsure how neb_single_path and neb_from_indices differ.  They both specify a path index and run a single NEB?

neb_single_path --> you would provide one index here that indicates which pathway to use from the DistinctPathFinder. For example,  DistinctPathFinder would find 5 symmetrically unique paths, and you say "I want to run the pathway that is 4th in this list (aka the 4th shortest pathway)"
neb_from_indices --> the indices given refer to start and end Wyckoff positions of the structure. For example, you could have a crystal structure with 5 unique Fluoride sites and you say "I want to run the pathway that connects F site number 1 (a tetrahedral position) to F site number 3 (an octahedral position)".

This looks good to me!  I did notice a typo in your 4th line of code. Remaindering should be remaining.

I was forgetting about symmetry and periodic boundary conditions before, so I switched neb_from_indicies to neb_from_startend_sites. Rather than giving site index, the user will give a PeriodicSite object. This let's the user specify unique hops between symmetrically equivalent sites. For example, this would allow a diffusion pathway that goes from a site at (0,0,0) to (1,1,1).


Map 1581 change non residential room type

This code enhance the transactions we are able to do with Location application and consume new backend api endpoints in order to change non residential location TYPE  as an example if the non-res location is OFFICE and we want to swap to TREATMENT ROOM.

Just one comment, other than that it looks good except for the failing integration test,
I re-run the pipeline and it was fine earlier, it was a glitch, I think (some Cypress server)
https://app.circleci.com/jobs/github/ministryofjustice/hmpps-locations-inside-prison/3494
all good now


Checkpoint optimizer states

Currently, mxnet only checkpoint the model. In practice, the computation states are not only model but also states of the optimizer. For SGD with momentum enabled, momentum is the state of the SGD optimizer, and it saves something like history of gradient. So, when checkpointing, the states of optimizer should be saved as well.
This PR saves optimizer states when checkpointing.

The change in function interface will break many functionalities. New arguments should have a default value, e.g. None.
There are multiple optimizer states for each weight when training with multiple gpus

Sorry for the late reply. Yes, you're right, in a mutli-gpu setting, the optimizer states are co-located with kvstore.

Hi @piiswrong, sorry for the late reply. I don't quite understand "multiple optimizer states for each weight". Do you mean "for each weight, there are many optimizer states replicas on each GPU" or "for each weight, there are many kinds of optimizer states".
Because, in the case of using kv-store, the optimizer states are partitioned among multiple GPUs. For a single weight, the optimizer state is not replicated, right? In my implementation, the optimizer states and model are checkpointed within the same file('%s-%04d.params' % (prefix, epoch), prefix contains kv.rank). I think it is correct. If I missed something, please let me know.

Solved by #4408


Object Oriented JavaScript - Confusing code exercise descriptions

There are two exercises dealing with JavaScript array functions that don't change the original array (.map and .filter).  It is very confusing to a learner whether to modify the existing array or to assign the results to a separate array.  Here's a snippet from http://www.freecodecamp.com/challenges/iterate-over-arrays-with-map
Use the map function to add 3 to every value in the variable oldArray.
var oldArray = [1,2,3,4,5];
var newArray = [];

// Only change code below this line.

newArray = oldArray;

The test assertions are all on newArray, meaning it doesn't matter if oldArray was changed at all.
I have two suggestions, either of which should make these exercises less confusing:

Remove all references of newArray and make the test assertions on oldArray

or

Reword the goal statement to : Use the map function to add 3 to every value in the variable oldArray, and save the results into variable newArray.  oldArray should not change.
Update the starter code to :

var oldArray = [1,2,3,4,5];
// Only change code below this line.

var newArray = oldArray;

This concept would apply to both:
http://www.freecodecamp.com/challenges/iterate-over-arrays-with-map
http://www.freecodecamp.com/challenges/filter-arrays-with-filter

I like suggestion number two :+1:


Refactor TagTemplate to use Tailwind classes

Remove all SCSS styles and replace with Tailwind classes

@ihsavru would like to solve this issue. Can I get assigned?


Create a company - Incorrect API URL

The POST URL "https://api.businesscentral.dynamics.com/v2.0/{environment name}/api/microsoft/automation/2.0/companies({companyId})/automationCompanies" is incorrect. The stated version is automation/2.0/ thus giving error 404.
The correct version is automation/v2.0/.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 036fbaa0-3f6b-e4f7-cd1e-c35b37cdee6e
Version Independent ID: 9efb41bd-bad8-597f-08e8-2b3c7e05f8e9
Content: Introduction to automation APIs - Business Central
Content Source: dev-itpro/administration/itpro-introduction-to-automation-apis.md
Service: dynamics365-business-central
GitHub Login: @henrikwh
Microsoft Alias: solsen

Thanks for catching this. The topic will be fixed with the next doc refresh.


Stack trace not showing in script

Just got this exception:
Driftwood 2D
Starting up...
[0] INFO: Path: appended: data/castle/
[0] INFO: Path: rebuilt
[0] INFO: Audio: initialized mixer output
[0] INFO: Audio: initialized mixer audio format support: ogg
[0] INFO: Script: loaded: Scripts/map_util.py
[0] INFO: Script: loaded: Entities/__init__.py
[0] ERROR: Script: __load: broken script: init.py:
Traceback (most recent call last):
TypeError: list indices must be integers or slices, not str
[0] ERROR: Script: no such module: init.py
Shutting down...

Not sure where the error is happening, which makes it hard to solve. I think the exception is in some engine code that the script is calling (code that I'm writing that's currently buggy).

Weird. See if 6fa0d13 fixed it. I also updated the error messages to be less rude.

It would help if I knew the crash conditions.

Oh, there's nothing special about the particular error I was getting. To replicate just introduce any error into the code that throws and exception. Or raise one manually for test purposes.
I'll test in a bit!

Looks like it's not fixed yet.
To reproduce
# init.py
Driftwood.script.register(None, None)

# scriptmanager.py

class ScriptManager
    def register(self, event, func):
        def f():
            o = []
            o[3]
        f()

What I see
[0] ERROR: Script: __load: error from script: init.py:
Traceback (most recent call last):
  File "./scriptmanager.py", line 303, in __load
IndexError: list index out of range
[0] ERROR: Script: no such module: init.py

What I expect
A traceback that includes the call to register and its own call to f. f is just there to test the stack trace. It should definitely show.

Hm. Maybe because it's failing inside ScriptManager it's tripping up that system. I'll poke around.

This reproduction isn't very helpful to me because it's new code. Can you make our engine crash this way and then give me the script that did it?

Sure thing!
Can you try this:
$ git fetch
$ git checkout pmer/artificial-error
$ make
$ bin/driftwood

It's a branch that contains the edits in my reproduction. Read 03c75f540ea540bc5c6244db80a5c4aa378706bc. It induces an artificial exception to test the behavior of our exception handling.

\o/


Add error handler for exporting review results

This PR added an error handler for exporting review results.

Nice! Code with the Blob looks more complex than I expected.


Add bounds detection to zooming in/out

Tiles in grid cannot be too small or large.

Now from the start, can only increase or decrease it 4 times in either direction. Need to replace hard-coded values with respect to the initial tile size.


Fix Returning Error on HelmRelease NotFound and Fix Security Vulnerability

Fixes the task erroring out when the HelmRelease that we are attempting to delete is NotFound
Fixes helmv3 < v.3.6.1 vulnerability

Fixes the task erroring out when the HelmRelease that we are attempting to delete is NotFound
Fixes helmv3 < v.3.6.1 vulnerability
Just for my knowledge, what is the exact vulnerability?

Fixes the task erroring out when the HelmRelease that we are attempting to delete is NotFound
Fixes helmv3 < v.3.6.1 vulnerability

Just for my knowledge, what is the exact vulnerability?

It's listed on the dependabot security vulnerability page: https://github.com/Azure/orkestra-workflow-executor/security/dependabot/executors/default/go.mod/helm.sh%2Fhelm%2Fv3/open


Set up project files and infrastructure

What is this PR introducing?
Tons of setup work for the project
Initializes the NPM project, adds all the configs and stuff for libraries like electron, eslint, jest, playwright etc., adds some automatic actions workflows, starts setting up the file architecture.
Worth mentioning that I installed an eslint config thats used internally by eslint because it was easier than building one ourselves. If it has some rules that end up being really annoying, we can remove it and make our own.
What components of our system does this change impact? Which core feature is this relevant to?
N/A
Does this change depend on any other unmerged PRs? Link them here if so.
N/A
Pre-commit to-do list

[ ] Have you written unit/integration tests for this?
[ ] Have you run this against the testing suite?
[ ] Can this be broken up into smaller features?
[X] Have you asked a teammate for review?
[X] Have you given a written explanation of the change?

Anything else important?
Haven't really tested if the GitHub Actions workflows work properly yet, will probably have to do some tinkering with the configs for everything as we start development.

Ended up removing the workflow for Playwright because something kept breaking with it trying to launch Electron. I googled it and the only thread I found said that its some weirdness with Ubuntu and didn't propose a fix, so I just decided to delete the workflow altogether. It's not too hard to run ITs manually before making a PR anyways


deadlock/hang when nesting jwalk within par_iter()

To start, here is a minimal example to reproduce the issue: https://github.com/Byron/reproduce-rayon-hang .
To reproduce, you can run the following (link to code):
git clone  https://github.com/Byron/reproduce-rayon-hang
cd reproduce-rayon-hang
cargo run -- break

I could workaround it by setting the pool size higher than the size of all inputs, which happens when invoking cargo run above (without the -- break argument).
Despite me CCing @jessegrosjean as the author of jwalk, I hope bringing it up here might help resolve why this interaction between jwalk and rayon fails despite jwalk implementing public rayon traits.
A related issue seems to be #690 , which involves a par_bridge() instead of par_iter().
The reason for me looking into this was the hang in https://github.com/starship/starship/issues/4251 . I think the reason was that these users ran on computers with 4 cores, while starship would want to use par_iter() on 5 inputs, with one of these causing jwalk to run within the par_iter() as part of gitoxide's git_repository::discover().
It would definitely be valuable if we could find and fix the root cause of this as I fear this issue might pop up more often - by default, gitoxide uses jwalk and if any user of gitoxide happens to open a repository within a par_iter() context, this hang could occour for the portion of end-users who don't have enough cores or whose workload is higher.
Thanks for your help.

I think in starship, the issue might also be that all available threads attempt to wait on the central OnceCell<context::Repo>, blocking all worker that jwalk attempts to spawn.

If code is blocking a rayon thread in a non-cooperative way, I don't know what rayon could do about it.
For example, with RAYON_NUM_THREADS=1, this spins forever:
use jwalk::WalkDir;

fn main() {
    rayon::scope(|_| {
        eprintln!("WalkDir…");
        for _entry in WalkDir::new(".") {}
        eprintln!("WalkDir completed");
    });
}

What I found interesting is that jwalk seems to continuously traverse the directory without ever finishing, like it can't make progress (in my example).
In the example right above, it's the same stacktrace.

This makes me hopeful that it's some interaction with the Bridge that jwalk is implementing that is subtly incorrect in these scenarios, so it can be fixed one way or another.
I'd be surprised if jwalk was able to continuously make up work for itself to keep blocking the one thread there is, maybe @jessegrosjean could chime in for clarification?
Or in other words, if jwalk would really be doing that, I'd expect it to happen without the outer rayon::scope(…) {}, but that seems to work fine even with only one thread. So at the very least, there must be some unhappy interaction between the two that should be fixable one way or another.

#997 should help with par_bridge deadlocking itself, but it looks like jwalk still goes wild here...

Sorry for not responding earlier, I'm using just Swift for the last year and haven't been using this code.
@cuviper Thanks for Rayon 1.6.1, it fixes the problem that I was running into that caused me to need the custom jwalk_par_bridge code. I've now removed that from jwalk main. That's good because it removes an unknown in jwalk, but as you note jwalk is still having a problem with the bug reported in this thread.
@Byron I have taken a quick look and can reproduce the problem you describe, but don't have any idea on solution yet. You would be amazed at how quickly jwalk, rayon, and rust all fall out of my understanding. I will try to load it all back in sometime next week and see if I have any ideas. Also if you feel like you could make more progress without me in the way I'm also happy to transfer jwalk to you for maintenance. I'm encouraged to see the various projects that you've built on it.

@jessegrosjean I am excited that you could reproduce the problem, and that you were able to reduce the surface that could cause this issue.

Also if you feel like you could make more progress without me in the way I'm also happy to transfer jwalk to you for maintenance. I'm encouraged to see the various projects that you've built on it.

The killer-feature really is its compatibility to walkdir and the built-in parallel sorting. Indeed it has been very important for my projects and being able to maintain it myself would be an honour I gladly accept. That should give me the push to finally understand how its machine works as well, the entire rayon interaction eludes me 😅. pdu has a parallel recursive implementation as well, but doesn't support sorting nor iteration, but I wonder if it was possible to implement it without rayon entirely to ease maintenance.

I wonder if it was possible to implement jwalk without rayon entirely to ease maintenance.

@Byron I can't say anything for sure, but I don't think so. jwalk is originally inspired by ignore which does implement the parallelism on its own. In my mind that part of the code gets messy, and I think the end result is slower they what rayon provides.
Maybe we do a zoom meeting sometime and I can explain any bits of the code (maybe I can anyway :)) you have questions on? I'm https://mastodon.social/@jessegrosjean or<EMAIL_ADDRESS>

That's great news! And indeed, thread-pool management is something that is probably a bit undervalued by many as they don't actually notice it exists, even though they would if it didn't 😅. Knowing that having its own thread-pool can also resolve the issue is great as well, it's something gitoxide could then manage somehow. If rayon threads in ThreadPools would automatically shutdown after a while it should be simple and good enough, even though auto-shutdown didn't seem to be present or I failed to find it in the docs.

…but it would also be nice for jwalk to always work when sharing the global rayon thread pool.

Maybe over time the issue will be solved here in rayon so waiting is a possible cause of action. I am happy to do that as well.

Maybe we do a zoom meeting sometime and I can explain any bits of the code (maybe I can anyway :)) you have questions on?

That's a fantastic idea 🎉🙏. Maybe it could be some sort of hand-over all-in-one, and I'd be looking forward to meet the fellow Rustacean upon which a lot of my software depends :D. I will send you an email with a calendly link so you can find a time that works for you.

Maybe over time the issue will be solved here in rayon so waiting is a possible cause of action. I am happy to do that as well.

Maybe... though I think it's much more likely that that problem is on jwalk end :)

Coincidentally I have found another issue in jwalk that could cause the iterator to block even though it would eventually finish. This certainly is unrelated though even though I hoped it would help.
This leaves us with a reproducible busy hang when running the crash.rs program (thanks @jessegrosjean for adding it).
Citing @cuviper followed by @jessegrosjean

If code is blocking a rayon thread in a non-cooperative way, I don't know what rayon could do about it.

Maybe... though I think it's much more likely that that problem is on jwalk end :)

I have a hunch that the busy hang is something that jwalk is causing itself, and that a preferred way of dealing with it would be to detect if such an issue could occour and abort with an error before we grind to a halt, or actually fix it somehow.
Thus I think it's fair to continue the quest on jwalk's side and thank everyone here for their support, it's much appreciated 🙏.

And I could validate that it is indeed jwalk that enters a yield loop, so I presume it's the logic here that should detect that no progress can be made somehow.
What's interesting is that the closure that produces results is never invoked, so I think I have a handle to fixing this now 🎉. After all, the threadpool is busy executing the task that tries to spawn another that it depends on, but cannot finish unless it gets results from a task that cannot be spawned. A clear dependency loop that has to be broken on jwalk's side.

And it's done: jwalk can now detect deadlocks and fail gracefully.