Looking for advice

Hi everyone! My name is Nolan. I recently was introduced to the Nightscout community by my family out of interest in the artificial pancreas system. My father is a T1 diabetic who among other things also suffers from celiacs disease. As soon as i read about the program and saw how it could effect his life i was instantly on board.
However, I dont have a major background in computer science or software engineering. I study Information Science as a junior at the University of Pittsburgh but this isnt entirely helpful for writing the algorithms and in-depth code implementation that this project requires. My experience includes taking two courses each in C++ and java programming and im ok with navigating Linux for the more basic things from working with pi in the past.  My question to all of you is do you think a person of my skill level could safely and effectively venture into this project?
I'm willing to learn and i realize it might take a large amount of time in doing so but its the least i could do for someone who has done so much for me throughout my life. I do want to be cautious however, i realize fiddling with a medical device is something that deserves the utmost care.
If you would like to email me personally you can do so at<EMAIL_ADDRESS>Thanks for any Help!
-Nolan

Sounds like you have more than enough technical experience to dive right
in.  We've had folks with absolutely no Linux experience get themselves up
and running successfully, albeit with a bit longer learning curve than
folks more familiar with how to get around a command line.  You really
don't need any particular programming or algorithms background: mostly
you'll just be assembling one-liners that collect and process various data
using tools that have already been created.  Some programming experience is
useful if you want to drill down into the code to see exactly how the
recommendations are generated, but that's not a requirement to successfully
build and safely test a loop.
Take a look at https://github.com/openaps/docs, and then hop into
https://gitter.im/nightscout/intend-to-bolus with any questions.  If you're
committed to a bit of learning, you'll be up and running before you know
it.  :-)
-Scott
On Tue, Jan 26, 2016 at 2:36 PM, nlewis93<EMAIL_ADDRESS>wrote:

Hi everyone! My name is Nolan I recently was introduced to the Nightscout
community by my family out of interest in the artificial pancreas system My
father is a T1 diabetic who among other things also suffers from celiacs
disease As soon as i read about the program and saw how it could effect his
life i was instantly on board
However, I dont have a major background in computer science or software
engineering I study Information Science as a junior at the University of
Pittsburgh but this isnt entirely helpful for writing the algorithms and
in-depth code implementation that this project requires My experience
includes taking two courses each in C++ and java programming and im ok with
navigating Linux for the more basic things from working with pi in the past
My question to all of you is do you think a person of my skill level could
safely and effectively venture into this project?
I'm willing to learn and i realize it might take a large amount of time in
doing so but its the least i could do for someone who has done so much for
me throughout my life I do want to be cautious however, i realize fiddling
with a medical device is something that deserves the utmost care
If you would like to email me personally you can do so at
Nlewis5525@gmailcom
Thanks for any Help!
-Nolan
—
Reply to this email directly or view it on GitHub
https://github.com/openaps/openaps/issues/78.

Scottleibrand, Hello and thanks for the quick reply!
I'm glad to hear that others have tried and been successful before me. However, I feel like i'm a bit confused with the systems overall. From what i've learned through a bit of research i have (in a broad scope) two ways to go. One being the use of an android to pull data from the cgm and pump to display onto other devices for a little more safety and overall access and the second being the implementation of a raspberry pi and multiple algorithms to fully automate the pump's processes. In addition to that I assumed I had to write all of those pieces of code to make it work. An example of what i mean is building the manual system in Phase two on the Open Artificial Pancreas System.
https://openaps.gitbooks.io/building-an-open-artificial-pancreas-system/content/docs/Build-manual-system/considerations.html
Again thanks so much for any help!
-Nolan

The former is what we call Nightscout.  It doesn't give you any control over the pump.  The latter is what we call OpenAPS.
Your assumption is no longer correct.  Read the rest of the docs: the oref0 toolset has all the algorithms and code you need.
And definitely hop into Gitter with questions.  There are dozens of people who can answer them.


ndjson message - Add hook type to message

Is your feature request related to a problem? Please describe.
The hook type (before, beforestep .... etc) is not available in the ndjson message. Need it in the reports to identify the hook type. Also for enabling or disabling display based on hook type.
Describe the solution you'd like
Available in the hook message. Will make porting reports based on json easier, in which this information is available.
Describe alternatives you've considered
Additional context

@mpkorstanje - I think you were working on something that would effectively close this as completed?

I was. I needed these to implement a message protocol based json formatter. But I'm blocked on until I solve other problems. If any one wants this, feel free to pick up #224.


Exception in thread "ModelLoaderThread" java.lang.OutOfMemoryError: Java heap space

I'm trying to run the parser as a library in Java. However, every time I'm trying to get an instance of parser final Parser parser = Parser.getInstance(); I got an OutOfMemoryError even despite changing heap space to 8GB.  Are models bigger than that? What it is that the code is doing when loading tokens like  backgroundBow: definierende = 2.0 ? Do you have any other suggestion apart from increasing heap space? I'm not completely sure if this will help me to solve this issue. Part of the error I get is:
10:58:53.047 [ModelLoaderThread] DEBUG o.a.scienceparse.ParserLMFeatures - backgroundBow: definierende = 2.0
10:58:58.121 [ModelLoaderThread] DEBUG o.a.scienceparse.ParserLMFeatures - backgroundBow: definierenden = 2.0
Exception in thread "ModelLoaderThread" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.(String.java:207)
at java.lang.String.substring(String.java:1969)
at ch.qos.logback.classic.pattern.TargetLengthBasedClassNameAbbreviator.abbreviate(TargetLengthBasedClassNameAbbreviator.java:58)

The way I solved this last issue was by adding the jar file of the common core dependency. There are some dependencies that are not up-to-date. I guess you'll update them soon.

org.allenai.common
common-core_2.11
1.4.8

If you manually mess with the dependency jars, you might sometimes get ClassNotFoundException. I would not recommend doing that.
8GB of heap should be enough, though more is better. Can you show me what exactly you're running to get that OutOfMemoryError? Depending on where you configure the memory settings, and the context in which you're running, memory settings might not apply properly.

I couldn't find the right dependencies by adding them into the pom file, so I went for this temporary solution while those dependencies are updated eventually.
I tried to run it on IntelliJ, set up the heap size for the compiler, which might not be applying memory settings as desired. I'm getting that OutOfMemoryError when running final Parser parser = Parser.getInstance(); within the main method

The heap size of the compiler does not matter in this case. This is about the heap size during runtime.
Do you have a GitHub repo I could look at, or maybe a small example project that shows the problem?

What do you run to make this fail?

Having the same issue. Could you get around this issue?


Reading data with queue is even slower than using feed_dict

It seems reading data with feed_dict is inefficient in tensorflow (I found it 3-4 times slower than a theano-based implementation with a totally same network structure), I turn to a queue-based method as official recommendation. However, my experiment result give a even worse performance. Is there anything wrong ?
Related issue
#3377 Moving data from CPU to GPU is slow
Environment info
Operating System: Ubuntu 16.04
CPU: Intel i4790-k
GPU: Nvidia GTX 1070 (8 GB)
Memory: 16 GB
Tensorflow version: 1.0
CUDA version: 8.0
cuDNN version: 5.0
Implementation Example

Using feed_dict

x, y = np.array(...)
in_x, in_y = tf.placeholder(...)
""" f() is some computation in the network, including embedding_lookup,
bidirectional_dynamic_rnn, dense """
train_op = f(in_x, in_y)
sess = tf.Session()
for _ in range(num_epochs): # num_epochs is 100 here
    sess.run(train_op, {in_x: x, in_y: y})

Using queue

x, y = np.array(...)
x, y = tf.convert_to_tensor(...)
in_x, in_y = tf.train.slice_input_producer([x, y], num_epochs=100)
in_x, in_y = tf.train.batch([in_x, in_y], batch_size=32)
train_op = f(in_x, in_y)
sess = tf.Session()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess, coord)
try:
    while not coord.should_stop():
        sess.run(train_op)
    except Exception as e:
        coord.request_stop(e)
    finally:
        coord.request_stop()
coord.join(threads)

I troubleshooted similar issue here and it was caused by Python thread scheduler choosing a bad strategy. Essentially Python would schedule computation thread that issues a single enqueue call, then this thread would block and have to be pre-empted. Python doesn't support parallel execution of Python code and pre-emption is slow so this part is a performance hit. Eventually it pre-empts dequeue thread to schedule enqueue thread, which does a single enqueue call before Python decides to give execution back to main thread. This back-and-forth dance introduced 10x slowdown in the pipeline.
One solution is to make sure that you never have queue starvation, for instance, by making enqueue part faster (ie by using enqueue many as here https://github.com/tensorflow/tensorflow/issues/3009 ) and by making dequeue part slower (ie, add more computation to train_op until it becomes slow enough to be bottleneck)

@yaroslavvb Thx for your advice.
However, I fail to fix the problem. I try replace tf.train.batch with tf.train.shuffle_batch and set min_after_dequeue=10000  to avoid queue starvation (I assume this parameter help fix it, right ?), but my program as slow as before.
What's more, I follow the link that you mentioned to generate timeline. It did take a long time to generate a timeline.json file (600 MB+, with correct .json file format), but failed to be visualized using chrome://tracing and give a blank output. Do you have any idea of what's going wrong ?

What I meant was -- make sure your dequeue part is slower than enqueue. Pre-loading the queue is not really a solution because if your computation runs faster than enqueueing, it'll consume all the elements on the queue and you'll be back to queue starvation. The reality is that queues are bad when your computation is extremely cheap because of the way Python multi-threading works, feed_dict may actually be better for those cases.

@yaroslavvb Thank you for reminding me of checking in TensorBoard.
I do some test and it seems that enqueue part is not a bottleneck of the program, it's full (and capacity is large enough) all the time. There is no problem of starvation.
And I find that dequeue operation really cause problems, according to the compute time statistics in TensorBoard, dequeue cost 6x compute time than my model, which result in low GPU-Util and even worse performance than using feed_dict.

You could narrow it down by making your program single threaded - first
preload the queue and then consume it in the same thread. If it's still
slow, then could narrow it down to a case that spends most time in dequeue
and is single threaded, at which point CPU profile can tell where the time
is spent
On Feb 27, 2017 2:43 AM, "akaitsuki-ii"<EMAIL_ADDRESS>wrote:

@yaroslavvb https://github.com/yaroslavvb Thank you for reminding me of
checking in TensorBoard.
I do some test and it seems that enqueue part is not a bottleneck of
the program, it's full (and capacity is large enough) all the time. There
is no problem of starvation.
And I find that dequeue operation really cause problems, according to
the compute time statistics in TensorBoard, dequeue cost 6x compute
time than my model, which result in low GPU-Util and even worse performance
than using feed_dict.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/7817#issuecomment-282686023,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABaHL5LVWVVBOMnyKTCJPSkiGVC56Ioks5rgqi3gaJpZM4MJ7ZR
.

BTW, here's an example of making reader pipeline single-threaded -- https://github.com/yaroslavvb/stuff/blob/master/ericyue-slowreader/benchmark-batch.py
Especially see the line with sess.run(a_queue_qr.enqueue_ops). Restructuring the problem in this way can help rule out some Python-threads related strangeness going on

@yaroslavvb Sorry for check it so late.
I did preload the queue full enough (using single thread sess.run(enqueue_ops)) this time before start running my model and consume the queue, there is no more enqueue operation after dequeue is start.
But it seems to be very slow for dequeue in this situation.
The following is the graph and node metadata showed in TensorBoard. You can see that compute time of dequeue operation is 24.2ms, which is much longer than that of my model, 1.36ms.

However, if I replace my model with much light operation like tf.square, dequeue will be much faster with same tensors output.

I can not figure out what the problem is...?

@yaroslavvb
My implementation rely on 3rd library tensorlayer and it takes me sometime to rewrite it using tensorflow only.
You can check and download the example .py file with this Dropbox link.
Please check different performance of dequeue operation between sess.run(train_op) and sess.run(light_op).

@akaitsuki-ii so I tried making a timeline for your stuff (https://github.com/yaroslavvb/stuff/commit/5d2feffd62fa50d71dd2dfd314ee405c21d58a25), and I couldn't because your execution graph is too large. There are 942,316 ops being executed in the first session.run call. The dump of stepstats is over a GB and timeline generation chokes. So I think that's the real problem, and the "enqueue" op being slow is a false positive since tensorboard couldn't parse all the ops

@yaroslavvb
Errr...In fact I did try to make timeline before and fail because of the same reason.
There may be a mistake that do you really mean that dequeue op being slow is false positive? As enqueue op is not contained in sess.run call.
But if dequeue op is not really the problem, why I get worse performance changing to queue-based implementation than feeding with feed_dict at each sess.run call ?
Is it just better using feed_dict instead of queue_based method in this situation ?

There are about 1 million ops being executed (grep node_stats stepstats-1.json | wc), and it takes 4.72 second on my 2014 MacBook, which seems fairly fast for 1 million ops. Such large computation graphs are outside of typical range of cases TensorFlow is tuned for, so there's significant overhead involved.
As to why I said it is a false positive, if you open stepstats-1.json generated by the commit I referenced, you will see the following
    node_stats {
      node_name: "random_shuffle_queue_DequeueMany/n"
      all_start_micros:<PHONE_NUMBER>948609
      op_start_rel_micros: 1
      op_end_rel_micros: 1
      all_end_rel_micros: 52
      memory {
        allocator_name: "cpu"

So the Dequeue op takes 51 microseconds.
As to why feed_dict is faster, it's an interesting question, and would require some digging to getto. One theory is that dequeue op requires a kernel launch, so when you already have a million ops being launched, perhaps the thread scheduling system is overloaded,  any new kernel launches are slow. Meanwhile, feed_dict approach does not add an additional op for reading data from queue, so no kernel launch.

I'm not sure I follow the explanation for feed_dict, @yaroslavvb , since feed_dict translates to adding a Recv op in the graph for each thing being fed, and those still have to run.
Dequeue on the other hand is always suspicious since it can be sometimes fast and sometimes slow (if the queue is often full but sometimes empty). It might make sense to aggregate the costs across many invocations before saying it's not the issue.

@alextp Good point about feed_dict, I can see an extra _Recv node in that graph.
BTW that script removes that queue variability -- there's no queue runners, things are first loaded onto the queue, and then consumed from the queue in the same thread.

Just want to add something here, I implemented a multiprocess-based data feeding pipeline for multi-task learning. avg. GPU utilization >90% and quad-core CPU utilization >95%. In case anyone interested: https://hanxiao.github.io/2017/07/07/Get-10x-Speedup-in-Tensorflow-Multi-Task-Learning-using-Python-Multiprocessing/

I don't know if it is still relevant, but reading through the discussion I do not find the reason or the solution to the problem. I have a very simple network (no operations almost) and I use:
tf.train.shuffle_batch(decoded, capacity=batch_size * 50,
batch_size=batch_size,
num_threads=6,
min_after_dequeue=batch_size * 10,
allow_smaller_final_batch=True)
As my performance drops as I increase the batch_size almost linearly, i.e., increasing batch_size 4 times, it takes 4 times as much.


Discussion for Scope being a list and not a string?

Was there ever any discussion as to why Scope on the OIDC middleware is a collection and not a string? ResponseType is still a string. It feels cumbersome. Anyone else feel the same way?

No discussion, but since [I'm the guy who replaced the Scope property by a list] (https://github.com/aspnet/Security/pull/276/files#r31257639), I can shed some light on the subject :smile:
The most important reason was for consistency: the OAuth2 generic middleware used a list of strings and the OIDC client middleware a single string.  Using a list hides how the scope is finally formatted by the specific implementation and make the way users deal with scope consistent across the social providers.

Well, screw all of that, IMO. The katana OAuth2 MW is dead. And you know exactly what the OIDC and OAuth2 specs say for the scope. If the aberrant implementations needs something different, then do it in their own implementation.
The job of frameworks is to make the life of the consumer easier, not the life of the implementer.

Well, screw all of that, IMO. The katana OAuth2 MW is dead.

There was no OAuth2 generic middleware in Katana, so not sure to understand what you mean.

And you know exactly what the OIDC and OAuth2 specs say for the scope.

Both specs only state that scope is space-separated string, they don't mention anything about how we should implement it internally.

If the aberrant implementations needs something different, then do it in their own implementation.

That's exactly what we do in the virtual FormatScope method. But it seems you totally missed my point. If we used a list, consumers would have to use different scopes patterns for Facebook and Google:
options.Scope = "scope1,scope2"; // Facebook, Reddit.
options.Scope = "scope1 scope2"; // Google (and the other standard providers)

Using a list abstracts the way the scope is formatted:
options.Scope.Add("scope1");
options.Scope.Add("scope2");

There was no OAuth2 generic middleware in Katana, so not sure to understand what you mean.

I was referring to the Katana OAuth2 authorization server middleware. Is that not what you mentioned above when you said that the point was for consistency? Perhaps we're not talking about the same thing.
Other complaints:

It's not obvious what the defaults are (probably something that is AAD focused).
It's not consistent with the other settings (again, response_type)
Most people that use the OIDC middleware don't implement the OAuth2 katana middleware, so the consistency there is not obvious. (this is my comment above so perhaps we're not talking about the same thing)
It's not consistent with how it was in Katana. I already noticed another issue on the issue tracker asking why Scope was no longer the way it used to be.

I know it's something small. But I teach this stuff to a lot of developers each year, and yet another little oddity like this requires yet another 5 minutes of explanation (or discovery on their part). Not time well spent (again for the consumer).
Again, I'm sure this doesn't matter in the end. I'm just raising the point that when I'm using this MW as a consumer I'm pretty sure that 99% of the time I will need to write this code:
options.Scope.Clear();
options.Scope.Add("openid");
options.Scope.Add("email");
options.Scope.Add("profile");
options.Scope.Add("api1");

So I know exactly what I'm getting. Otherwise the code looks like:
options.Scope.Add("email");
options.Scope.Add("api1");

And then i'm not really sure what I'm getting (or maybe the next dev on the project who inherits the code).
So my argument here is for clarity and simplicity for the consuming developer. I felt the string based approach in the Katana OIDC middleware achieved that.

I was referring to the Katana OAuth2 authorization server middleware. Is that not what you mentioned above when you said that the point was for consistency? Perhaps we're not talking about the same thing.

Hell no. By "OAuth2 generic middleware", I meant [this thing] (https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNet.Authentication.OAuth/OAuthHandler.cs). It's a whole new middleware, designed to support virtually any OAuth2 server. You can either subclass it or use the generic notifications. All the social providers inherit from OAuthMiddleware, including the 24 providers developed by the community.

As I said I already feel it's the aberrant implementation's fault if they don't follow the RFC/specs.  But I digress.
The reason for opening the issue stands: I think the design is more important for the consumer (for clarity and simplicity) rather than the implementer.

It's not consistent with the other settings (again, response_type)

One could argue that response_type has only predefined (standard) sets that allow no customization (unlike scope, whose values are totally implementations-specific).

It's not consistent with how it was in Katana. I already noticed another issue on the issue tracker asking why Scope was no longer the way it used to be.

There are many things that have been updated in vNext. This change is relatively explicit IMHO.

And then i'm not really sure what I'm getting (or maybe the next dev on the project who inherits the code). So my argument here is for clarity and simplicity for the consuming developer. I felt the string based approach in the Katana OIDC middleware achieved that.

True. But on the other hand, it allows customers to only focus on the provider-specific scopes, without having to re-specify the standard ones. IMHO, it's clearer and simpler.
You have valid points, but I don't think using a string is fundamentally better than using a list. I've reviewed ~20 providers for aspnet-contrib and many specify custom scopes in the options classes. Until now, I've never heard someone complaining about the scope thingy (well, except you, obviously :smile:).

Fair enough. I'd just like someone at Microsoft to spend some time thinking about it. I know it's late, and I know this is a nitpicky request, but I wanted some attention spent on it.

A) I prefer the List approach as it lets us worry about the formatting. It's even defined as a list in the spec https://tools.ietf.org/html/rfc6749#section-3.3.

The response_type comparison is interesting. It appears to be primarily single-value except for extensions. We only support code, id_token, and code id_token, and we use constants for those (OpenIdConnectResponseTypes).

B) I agree that the defaults need to be more transparent.

We have two different ways to do defaults right now.

Add them on the Options by default, then you need to clear them:
https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNet.Authentication.OpenIdConnect/OpenIdConnectOptions.cs#L163
vs
Only add them in the MW constructor if none were specified:
https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNet.Authentication.Google/GoogleMiddleware.cs#L72-L77

Which looks more useful?
Either way we need better doc comments that explain the defaults. Google, MSA, and OIDC have defaults. Facebook doesn't. I don't think it's relevant to Twitter.

Personally, option 1) has always been my favorite approach (https://github.com/aspnet/Security/pull/257#discussion_r32473302).

The only default that makes sense to me is "openid" (since it's the OIDC MW). I don't think it matters where -- either in options ctor, or in MW ctor.
Also, you should consider calling Distinct() on the scopes list before sending them to the OP -- currently they're duplicated in the authorization request if they're in the list multiple times. That's what apps will do when they don't know what the defaults are.

For the defaults we should do (1) where the defaults are initialized in the list, and it's up to the app developer to add/remove/clear/whatever.

@Eilon now that "delegate-style" middleware extensions have all been removed, clearing/removing a property list is really ugly :smile:
Before:
app.UseOpenIdConnectAuthentication(options => {
    options.Scope.Clear();
    options.Scope.Add("custom")
});

Now:
var openIdConnectOptions = new OpenIdConnectOptions();
openIdConnectOptions.Scope.Clear();
openIdConnectOptions.Scope.Add("custom");
app.UseOpenIdConnectAuthentication(openIdConnectOptions);

You could always add an extension method to simplify this case. Something like:
public static class OpenIdConnectOptionsExtensions
{
    public static OpenIdConnectOptions ResetScope(this OpenIdConnectOptions options, params string[] scopes) {
        options.Scope.Clear();
        foreach (var scope in scopes) {
            options.Scope.Add(scope);
        }
        return options;
    }
}

With usage:
app.UseOpenIdConnectAuthentication(new OpenIdConnectOptions().ResetScope("custom1", "custom2"));

For what it's worth, I unfortunately spent the last couple of days proving some of @brockallen 's points related to scopes and defaults.  The default in the OpenIDConnectOptions includes "profile" which pulls everything (at least in the case of Auth0).  Auth0 provides a lot of claims.  Enough claims to overwhelm Safari mobile's cookie domain size limits, which results in an obscure CookieChunkingManagerError for users using mobile browsers.

The default of "profile" definitely burned me in this case and it took a lot of digging to figure out how to change the scopes. Changing the scopes is clumsy as @PinpointTownes  pointed out (and not easily discoverable when you are trying to learn all the other new stuff).
I am not sure if there is anything actionable because removing "profile" would have its own set of issues, but at the very least, I think others can learn from my mistake.

@leastprivilege and I just did a workshop this week on this topic and it was somewhat painful to do scopes as items in the collection. The API today favors the middleware implementer and not the consumer. Even now post-RC2 I think it'd still be nice to have scope as a single string.


wav2vec 2.0 - provide example dataset line in data prep scripts

🚀 Feature Request
Provide an example line formatted in a way expected by the training/finetuning scripts.
Like:
file_a
file_a_line_format

file_b
file_b_line_format

Motivation
Right now you have to reason about the whole data preparation scripts to grasp what is the expected format, and they only work for LibriSpeech anyway: https://github.com/pytorch/fairseq/tree/master/examples/wav2vec.
I mean, I still don't get it what files are necessary to run the fine-tuning and how each file looks like.
Pitch
Adding like 3 lines of documentation in each preprocessing script or README is not something too hard I imagine.

yes is true, it's hard to create similar process, because no clue at all... to reproduce the similar process with another dataset.
already two weeks learn how to do it. still cannot figure all process...
what I know:

create a new dataset, just many wav file 16k 1 channel (mono), with lenght 2 - 15 s.
download from youtube, I use ytmp3 to download in my language (indonesian) then use python to convert and chunk it base silence with set the treshhold.
I add mozilla common voice in speech dataset (convert it to wav 16k, 1 channel).
but still not clear how many hours that needed to make good result.
for someone that do this for hobby, it's like not fun anymore i think to create 58k hours dataset.
use wav2vec_manifest.py to create tsv file as a direction file to point where the wav file exist.
train it...create pre-trained model,
finetuning step... here the problem start.
to finetuning, i got clue to create another tsv file from libri_labels.py, we still have a tutorial until here. but to train with kenlm 4-gram language model. that still unclear about:

in kenlm module we still can get clue how to create binary file, but... in finetuning we must use several file like:

a letter dictionary,  how to make dict.ltr.txt? because librispeech doesn't have my language...indonesian language...
a lexicon file, how to make lexicon file? it's looks like or similar with kaldi lexicon I think, did you use kaldi to create lexicon file?

start from here I'm stuck...
step by step tutorial, how to reproduce file that needed and the format of data like @kwasnydam said, it's needed...

Hey, you can refer to this page for dict.ltr.txt, lexicon and kenlm.
Note: In that page dict.ltr.txt is referred as token.txt.
And yeah you need a tsv file for both train and valid and create  .ltr and  .wrd files that are in format used by model to start fine-tuning your model. You can refer to libri_labels.py for that.

I mean, that's what I am talking about, instead of having the users read through the whole libri_labels.py script, which is made so that automatic generation of necessary files for that specific dataset works, just provide these few lines in the readme for wav2vec2.0 or scatter them across the relevant scripts (correct me if I got the format wrong, cause still, not everything has been working correctly for me yet).
To run the audio pretraining, prepare the `train.tsv` and `valid.txt` files, containing lines:
<path/to/the/audio/file/in/supported/format>\t<audio_len_in_samples>
You can use the wav2vec_manifest.py script to get tsv files for an arbitrary dataset.

To run the ctc fine-tuning, prepare the following files: dict.ltr.txt (check example here:<link>) {train,valid}.wrd, {train,valid}.ltr
.wrd file should contain transcriptions corresponding to the files specified in {train,valid}.tsv in the same order.
.ltr file should contain lines from .wrd file where each grapheme is divided by a space, words are divided by '|' and there is | at the end of the sentence

example .wrd file

i didn't know shepherds knew how to read said a girl's voice behind him
do i knock or something
it's extremely suspicious that there's no information about brains that didn't come from a brain
that was his work

example .ltr  file
i | d i d n ' t | k n o w | s h e p h e r d s | k n e w | h o w | t o | r e a d | s a i d | a | g i r l ' s | v o i c e | b e h i n d | h i m |
d o | i | k n o c k | o r | s o m e t h i n g |
i t ' s | e x t r e m e l y | s u s p i c i o u s | t h a t | t h e r e ' s | n o | i n f o r m a t i o n | a b o u t | b r a i n s | t h a t | d i d n ' t | c o m e | f r o m | a | b r a i n |
t h a t | w a s | h i s | w o r k |

To decode with LM, check: https://github.com/facebookresearch/wav2letter/wiki/Data-Preparation

Hey, you can refer to this page for dict.ltr.txt, lexicon and kenlm.
Note: In that page dict.ltr.txt is referred as token.txt.
And yeah you need a tsv file for both train and valid and create  .ltr and  .wrd files that are in format used by model to start fine-tuning your model. You can refer to libri_labels.py for that.

thank's bro..., never know that before...
by the way, the road is still long, did I should ask if I got problem in further step?

Hi,
This issue #2514 may help you :)

By the way, I kinda managed to put everything in place and I have working inference and training pipelines, maybe I will put some pull request with the relevant docs that would help the future users.
One new thing that bothers me is that the dict.ltr.txt is being looked for in the dataset directory also on the inference, which caused me 1 hour of debugging because my inference directory contained the dict file that was compatible with the previous version of the model, not the one I was testing (the order of labels was wrong). So first of all, during the inference examples/speech_recognition/infer.py, the code should search for the labels file in the --path directory, not in the dataset directory, and also the dict.ltr.txt should be copied to the model save directory right at the beginning of training, as it is inherently a part of the model and it won't work correctly unless a properly-ordered label file with all the labels is supplied. The second thing is, in case, as mentioned here: https://github.com/pytorch/fairseq/issues/2514, the count does not really matter, then I think it would be more intuitive to just provide the dictionary in ASCII/UTF-8 ordering, that way it will stay the same between different datasets instead of changing order when the counts are different.

By the way, I kinda managed to put everything in place and I have working inference and training pipelines, maybe I will put some pull request with the relevant docs that would help the future users.
One new thing that bothers me is that the dict.ltr.txt is being looked for in the dataset directory also on the inference, which caused me 1 hour of debugging because my inference directory contained the dict file that was compatible with the previous version of the model, not the one I was testing (the order of labels was wrong). So first of all, during the inference examples/speech_recognition/infer.py, the code should search for the labels file in the --path directory, not in the dataset directory, and also the dict.ltr.txt should be copied to the model save directory right at the beginning of training, as it is inherently a part of the model and it won't work correctly unless a properly-ordered label file with all the labels is supplied. The second thing is, in case, as mentioned here: #2514, the count does not really matter, then I think it would be more intuitive to just provide the dictionary in ASCII/UTF-8 ordering, that way it will stay the same between different datasets instead of changing order when the counts are different.

As a new researcher in this field, it would be great if you can put everything in one place. I'm struggle with this wav2vec for a couple of weeks now.
Thank you in advance.

By the way, I kinda managed to put everything in place and I have working inference and training pipelines, maybe I will put some pull request with the relevant docs that would help the future users.
One new thing that bothers me is that the dict.ltr.txt is being looked for in the dataset directory also on the inference, which caused me 1 hour of debugging because my inference directory contained the dict file that was compatible with the previous version of the model, not the one I was testing (the order of labels was wrong). So first of all, during the inference examples/speech_recognition/infer.py, the code should search for the labels file in the --path directory, not in the dataset directory, and also the dict.ltr.txt should be copied to the model save directory right at the beginning of training, as it is inherently a part of the model and it won't work correctly unless a properly-ordered label file with all the labels is supplied. The second thing is, in case, as mentioned here: #2514, the count does not really matter, then I think it would be more intuitive to just provide the dictionary in ASCII/UTF-8 ordering, that way it will stay the same between different datasets instead of changing order when the counts are different.

As a new researcher in this field, it would be great if you can put everything in one place. I'm struggle with this wav2vec for a couple of weeks now.
Thank you in advance.

Hey, for starters I think the guy in this answer got a lot of things right, especially when it comes to the proper dataset formatting so I think you could try starting here: https://github.com/pytorch/fairseq/issues/2493#issuecomment-719915281


Update requirements.txt

The current requirements are outdated and no longer work with the latest container. These new ones will work with a new installation

Thanks 👍


Calling GetStdHandle and then calling ctrl-c triggers error rather than clean exit

Windows Terminal version
1.15.2874.0
Windows build number
10.0.19044.0
Other Software
No response
Steps to reproduce
Create a rust project with the following code in main.rs (I guess the equivalent in C++ might also give the same output):
`use std::{thread, time::Duration};
use winapi::um::processenv::GetStdHandle;
use winapi::um::winbase::STD_OUTPUT_HANDLE;
fn main() {
let handle = unsafe {GetStdHandle(STD_OUTPUT_HANDLE)};
thread::sleep(Duration::from_secs(5));

}`
And then call cargo run. Before the program exits naturally after the 5 second sleep call ctr-c to manually terminate the program early.
Expected Behavior
Program should exit gracefully without error
Actual Behavior
Rather than failing gracefully the program fails with:
error: process didn't exit successfully: target\debug\your_program.exe (exit code: 0xc000013a, STATUS_CONTROL_C_EXIT)
Dropping or closing the handle also has no effect and the resulting error is the same.

Since you closed this bug, I'm guessing that you figured out the problem?


feat: send appName

About the changes

I was sending appName manually via properties on UnleashContext but this was broken by #20, so this fixes for those of us who needs to send appName.
It also changes how the query parameters are constructed to be more idiomatic for dart, and safer.
Discussion points

I didn't add it into UnleashContext because it's exposed to library users and it could be confusing because appName is already passed to UnleashClient, and it can't be changed there.
There might be other missing properties you should look into adding, like environment?

One of the CI things got cancelled so the checks failed @kwasniew

@andreasgangso will check it today

@andreasgangso This is a good finding. I'll merge it, add some tests and also enhance it with the environment.

@andreasgangso version 1.4.0 should have all the fixes.


What should I do to use the Soft max

Hi what should I do to update the leaf nodes to use a Soft Max instead of being nn.Linear ?
Thank you

If you mean that you want each leaf node to predict exactly one probability distribution, I think you should store predictions of leaf nodes using nn.Parameterlist() instead of nn.Linear. Furthermore, codes on the forward process should be slightly different (you may need to further use nn.functional.softmax()).
After thinking for a while, I think this job can still be done with some efforts. Good luck !


chore: added changesets

What I did

added changesets to track semver changes
added icon directories to package exports

I pushed a version which includes license comments and exports but couldn't publish it (no token)
Would you please

publish 1.0.1
rebase this PR to just add changesets

Should we add cicd here?


Remove particles js from docs page

Reading the docs page makes my laptop fan go bonkers because of particles effect in the header. Please remove it as it is highly annoying while reading the documentation.

Ok, I will remove it from subpages.


Multiline strings are not indented properly

I consider the following to be good:
        testdir.makeini("""
            [pytest]
            paths=blah.py""")

But currently it becomes:
        testdir.makeini("""
[pytest]
                        paths=blah.py""")

I've started working on fixing it.

I don't think then lines after """ should be indented.  It forces the user to run dedent on it.

I don't think then lines after """ should be indented. It forces the user to run dedent on it.

It would keep manually indented lines: so you would dedent manually after <cr> and then lines after it would also be indented in the same way, e.g.:
There could also be an option for this behavior.
We agree though that the current behavior is just wrong, right? :)

What behavior would you line to control?
Should be the indent after test("""foo controllable in a different way from test("""?
Would you like to have it being 0 for you in both cases?

Added a setting for this - previous question still remains.  Currently the setting gets used regardless of if there is something after the opening """.

yes i think there should be no difference.  """ are literal strings so I feel they should be as literal as possible by default. :)

you mean like:
foo = """
X
   abc
   Y
"""

enter after """, you end up at X
enter after c, you wind up at Y

?

@hynek
Yes, that behavior will be kept.
But I've meant that when using == etc on a non-empty line that is in a multiline string, the existing indentation won't be changed.

I've worked some more on #50 and think it should be good by now, please review it.


RGB_FILLED failed but depth successed

I tried RGB_PREFILLED, NORMAL, and SEMENTICS.
RGB_PREFILLED can display the correct content but it still has some wrong pixels, the results of NORMAL and SEMENTICS are both black.  But these all don't work.

Hello @rainprob ,
I recommend to reinstall the GibsonEnv clearly. You can try to install docker version instead of building source. Have a nice day.

@rainprob I observe similar issues when the gibson/core/render/render_cuda_f.so is not correctly compiled with nvcc. Can you check your nvcc --version is the same as nvidia-smi version?

@fxia22
You are right. My nvcc --version is 11.0, however nvidia-smi version is 11.2. So, how can I solve that? Make them both versions same？

@Berk035
Thanks for your reply.

@fxia22
You are right. My nvcc --version is 11.0, however nvidia-smi version is 11.2. So, how can I solve that? Make them both versions same？

In gibson docker container, nvcc --version is 9.0 and nvdia-smi version is 11.2.

It seems that you have multiple versions of cuda installed in /usr/local, you can expose the version used in nvidia-smi by soft linking cuda to it. For example:

@fxia22
I tried to download cuda11.2 for runtime that has the same version as cuda driver version. But the results shown in web_ui is still the same.

@rainprob did you build the code again? You would need to run ./build.sh build_local in GibsonEnv folder.

@fxia22 yes, but the buiding result shows some warnings. I am not sure if the warnings mean building failure and will lead a wrong camera output.

@fxia22

same here


BZ-1195980 - [MetaData - Versions] The count of number of versions is wrong

@wmedvede I'm not familiar with that area of our code base. @Rikkola  or @manstis  could you please take a look on it?

was merged

@wmedvede already merged, but this looks ok.


command manifest ignore draft PR and custom labels

TL;DR
    steps:
      - uses: google-github-actions/release-please-action@v3
        with:
          command: manifest
          draft-pull-request: "true"
          labels: "be-autorelease: pending"
          release-labels: "be-autorelease: tagged"

when I use this action for monorepo, the pull request is not a draft anymore and labels config not applied
However, when I am not specifying command manifest, the draft pull request config works and label works
Do I have to specify this inside the manifest?if yes how to specify it because at the moment it doesn't do anything if I put it inside the manifest as well
Expected behavior
draft pull request works and custom labels work
Observed behavior
No response
Action YAML
on:
  push:
    branches:
      - main
name: release-please
jobs:
  release-please:
    runs-on: ubuntu-latest
    steps:
      - uses: google-github-actions/release-please-action@v3
        with:
          command: manifest
          draft-pull-request: "true"
          labels: "be-autorelease: pending"
          release-labels: "be-autorelease: tagged"

Log output
No response
Additional information
No response

If you are using a manifest, all configs need to be in the manifest.

@chingor13 I tried to put it in manifest but it doesn't seems to work (?)
I believe this is the actual json reference that I should look up to
and this is the content of release-please-config.json
{
  "include-v-in-tag": false,
  "tag-separator": "/",
  "draft-pull-request": true,
  "packages": {
    "payment": {
      "release-type": "go",
      "changelog-path": "CHANGELOG.md",
      "package-name": "payment"
    },
    "wallet": {
      "release-type": "go",
      "changelog-path": "CHANGELOG.md",
      "package-name": "wallet"
    }
  }
}

The PR result is not in draft

the config of include-v-in-tag is working, so does the tag-separator setting.
So I'm wondering why the draft-pull-request doesn't work


Separate verified accounts with 1 Million followers/Affiliates from Twitter Blue Subscribers

Hi! Wanted to make a post on here because I absolutely LOVE and appreciate what you've been doing to modern day Twitter with the Control Panel. I have no clue how to do any of this stuff that goes to making it all work so well so I wanted to make a suggestion.
As far as I know, all accounts that get 1 million followers get Twitter Blue for free and accounts that are affiliated also have a Twitter Blue logo sometimes. If there is a way to let those accounts keep the old checkmark and have people that actually pay for Blue to have the logo, that'd be awesome.

Not planning to add this as it'd require doing a lot more work everywhere a check mark can appear to check the details of the account, however in the next release users with more than 1 million followers won't be hidden by default using the new "Hide Twitter Blue replies" feature #312


Clarify behavior of transformIgnorePatterns when including multiple patterns.

Summary
I recently bumped into an mystery working with transformIgnorePatterns, where I had two negative lookaheads for node_modules, like so:
{
  "transformIgnorePatterns": [
    "node_modules/(?!(foo)/)",
    "node_modules/(?!(bar)/)"
  ]
}

As I came to realize, this meant that both node_modules/foo and node_modules/bar folders were not transformed, because they would always match in the other pattern. I missed the significance of the line:

If the file path matches any of the patterns, it will not be transformed.

Maybe this is completely evident to other people, but it does seem like an easy detail to miss while working with it.
I figured I'd share these suggestions, and if the team thinks it's a worthwhile addition, I can add the changes in the other doc versions.
Test plan
Checked running website locally.

Codecov Report

Merging #11796 (685143e) into master (98f10e6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #11796   +/-   ##
=======================================
  Coverage   69.04%   69.04%
=======================================
  Files         312      312
  Lines       16366    16366
  Branches     4746     4746
=======================================
  Hits        11300    11300
  Misses       5039     5039
  Partials       27       27

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98f10e6...685143e. Read the comment docs.

@sigveio thanks for the thoughtful and thorough feedback! I'm glad you think the additions are useful.
I've pushed up some changes, let me know if they address your concerns. Once it looks good, I'll copy the changes to the versioned docs as well.

@sigveio Ready for another review! I copied the changes over, but the actual tweaks can be seen in 5b13a37.

Done! Thanks for the heads up @SimenB!


Ftu/hack domain id unsafe

@JabX @sebez What do you think of this ?
I need that because of Microsoft.Identity that requires the PK to be an int.
I'm not found of the name "UNSAFE" maybe "STRICT". But I want the devs to not use it except of that place.

Yup, can be a solution for that


use verbose option of index to display more information during training

verbose of faiss is super verbose, not sure if that's good to do

enable it only if debug logging is on


Reset HTTP connection state when connection is closed

A connection instance can be reused even if the server
has closed the associated socket.
When this happens, urllib3 will ".close()" the connection
which also closes and deletes the socket associated with
the connection.
However, the AWSHTTPConnection object is still reused
in the underlying connection pool.  The HTTP client will
notice that associated socket is None and will create
a new socket thereby establishing a new connection to the
AWS endpoint.
The bug on our end was that we were not properly resetting
our own instance state variables we use to keep track of
if we've sent the expect header, if we've seen a response,
etc.  We were assuming that a connection instance is removed
from the connection pool when the connection and underlying
socket is closed.
As a result, state from the previous (and generally unrelated)
response was leaking into the current response, specifically
in the scenario when the service responds with some other
status besides the "100 continue" response (i.e a fast fail).
This ended up with two things happening:

We were reusing the status code of the previous response
Because we thought we've already seen a response, we were
not reading the HTTP response headers.

Consequently, the HTTP headers were being sent to our
response parser, which would throw exceptions because it
was not receiving data in the format it was expecting.
We now ensure that we also reset our own state when the
parent class resets its state, which happens in the
".close()" method.
In addition to the associated test, I've also manually
verified this fix by writing a small http server that
returns the same sequence of HTTP responses that trigger
this error.
cc @kyleknap @mtdowling @rayluo @JordonPhillips

Nice. :ship:

:shipit:

Nice catch! :ship:


Register mappings for Artelys Crystal Super Grid 2022.1

Permissions added for Artelys Modelers to submit scenarios for the "Artelys Crystal Super Grid 2022.1" model to the ECEMF-internal instance.


[SUPPORT] Don't understand the result for bulk insert first and then upsert

First I run the bulk insert operation to write 4 records to a new COW table(the record key are A,B,C,D). Then I run another upsert opertion on this table with two records(the record key are A,C)
As known, bulk insert operation doesn't build index or do index loop during inserting, so when the first bulk insert is done, no index is created. When the second upsert operation begins,  the operation will assume that there is no A or C in the table because there is no index for A,C, that means ,the upsert operation will insert two records(record key are A and C),but I found that the upsert operation indeed update A and C records,
I don't understand how and why this happens, could you help explain? Thanks

package org.example

import org.apache.hudi.DataSourceReadOptions.{BEGIN_INSTANTTIME_OPT_KEY, END_INSTANTTIME_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL, QUERY_TYPE_OPT_KEY}
import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions.{BULK_INSERT_OPERATION_OPT_VAL, INSERT_OPERATION_OPT_VAL, UPSERT_OPERATION_OPT_VAL}
import org.apache.hudi.config.{HoodieIndexConfig, HoodieWriteConfig}
import org.apache.hudi.index.HoodieIndex
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import org.apache.spark.sql.{SaveMode, SparkSession}

case class OrderPart(
                      name: String,
                      price: String,
                      creation_date: String,
                      part: String
                    )

object Bulk_Insert_Then_Upsert {

  val step1 = Seq(
    OrderPart("A", "12.7", "2020-11-18 14:43:32", "A"),
    OrderPart("B", "13.2", "2020-11-18 14:42:21", "B"),
    OrderPart("C", "11.6", "2020-11-18 14:47:19", "C"),
    OrderPart("D", "10.4", "2020-11-18 14:46:50", "D")
  )

  //Update A and C
  val step2 = Seq(
    OrderPart("A", "112.7", "2020-11-18 14:43:33", "A"),
    OrderPart("C", "111.6", "2020-11-18 14:47:20", "C")
  )

  val hudi_table_name = "Bulk_Insert_Then_Upsert" + System.currentTimeMillis()
  val base_path = "file:///d:/data/hudi_demo/" + hudi_table_name

  def do_insert(dataCode: Int, saveMode: SaveMode, operation: String, spark:SparkSession): Unit = {

    import spark.implicits._

    val df = dataCode match {
      case 1 => spark.createDataset(step1)
      case 2 => spark.createDataset(step2)
      case _ => throw new IllegalArgumentException("Illegal argument")
    }

    df.write.format("hudi")

      .option(DataSourceWriteOptions.RECORDKEY_FIELD.key(), "name")
      .option(DataSourceWriteOptions.OPERATION.key(), operation)
      .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "creation_date")
      .option(HoodieIndexConfig.INDEX_TYPE_PROP, HoodieIndex.IndexType.GLOBAL_BLOOM.name())

      .option("hoodie.insert.shuffle.parallelism", "1")
      .option("hoodie.upsert.shuffle.parallelism", "1")

      .option(HoodieWriteConfig.TABLE_NAME, hudi_table_name)
      .option(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "part")
      .mode(saveMode)

      .save(base_path)
  }

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("Bulk_Insert_Then_Upsert").
      config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .enableHiveSupport()
      .master("local[1]")
      .getOrCreate()

    do_insert(1, SaveMode.Overwrite, BULK_INSERT_OPERATION_OPT_VAL, spark)
    Thread.sleep(10 * 1000)
    do_insert(2, SaveMode.Append, UPSERT_OPERATION_OPT_VAL, spark)
    query_table_snapshot(spark)
  }

  //The result is that A and C are updated,not appended as new records.
  def query_table_snapshot(spark:SparkSession): Unit = {

    val df = spark.read.format("hudi").load(base_path)
    df.createOrReplaceTempView("t")
    spark.sql(
      """
        select _hoodie_commit_time,_hoodie_commit_seqno,_hoodie_record_key,_hoodie_partition_path,name,price,creation_date,part
        from t
        order by  _hoodie_commit_time desc
        """).show(truncate = false)
  }
}

First of all, could you tell me which version of hudi you are using?

As known, bulk insert operation doesn't build index or do index loop during inserting.

In fact, this is not accurate. When writing a parquet file in Hudi, a Bloom filter based on the primary key is constructed and serialized into the footer.

First of all, could you tell me which version of hudi you are using?

As known, bulk insert operation doesn't build index or do index loop during inserting.

In fact, this is not accurate. When writing a parquet file in Hudi, a Bloom filter based on the primary key is constructed and serialized into the footer.

Thanks @BruceKellan for the help answer. I am using hudi 0.9.0，Spark 3.0.0
Looks I have misunderstood here, I got following two conclusions,  please correct me if I am wrong. Thanks,@BruceKellan

Per your answer,during bulk insert, indices are indeed created and saved in the parquet footer,  and the indices  will take effect for the later upsert operation
I tried another experiment, bulk insert records that have duplicate record keys, I find that these duplicate records are all written into the table, it looks that during bulk insert, indices are created but are not used for de-duplicate through record key loop during bulk insert

@bithw1 Bulk insert under bucket index will still shuffle the records by record keys into buckets/file groups with deterministic hashing algorithm, which is in line with the following-up upsert operation.

Thanks @danny0405 for the helpful reply.


Errors propagate through transaction

What is the goal of this PR?
We align with Client NodeJS version 2.6.1 (mostly the work in https://github.com/vaticle/typedb-client-nodejs/pull/197), which implements a better error propagation mechanism: when an exception occurs, we store it against all the transaction's active transmit queues to retrieve whenever the user tries to perform an operation in the transaction anywhere.
What are the changes implemented in this PR?

store errors received from gRPC against each receive queue
return a new exception type, transaction is closed with errors, which throws the errors from all queues (note that this can be duplicate if there are multiple open transmit queues that have been given the same error)
we clean up queues that are no longer needed, so we minimise the number of times the user sees the same exception

Review Summary:
We improve some terminology in the ResponseCollector - the collector presents itself as ONE collector. However, inside, the member variable is collectors -- so the question arises if the inner collectors are the actual collectors. To make this clearer, we rename the inner variable to be response_queues, since it represents a map to Queue objects, rather than further Collectors


Feat/get revoked block

Added getRevokedBlock function to retrieve the block number that the revocation took place in

Can consider using public variables instead.


Migrated the package to null safety

Closes #2

Please, accept the pull request!
It would be great to use the local_hero package in null safety enabled projects!

There were some issues with some tests, so I took the liberty to migrate it from my side. I'm closing this, since I uploaded a new version (0.2.0) with null safety.


Invoke-Command does not apply VerbosePreference to ScriptBlock

Steps to reproduce
PowerShell 6Beta2 still has the same gotcha as Windows PowerShell that one needs to manually pass and set the VerbosePreference into scriptblocks, even when using advanced functions. It would be nice if the VerbosePreference of the Invoke-Command Cmdlet would be set automatically in the executed scriptblock.
Invoke-Command -ScriptBlock {Write-Verbose "Hello World"} -Verbose
Invoke-Command -ScriptBlock {[CmdletBinding()]Param() Write-Verbose "Hello World"} -Verbose
Expected behavior
I would expect in both cases that the VerbosePreference gets applied automatically
VERBOSE: Hello World
Actual behavior
No verbose output.
One has to pass and set it manually to get it working:
 Invoke-Command -ScriptBlock {[CmdletBinding()]Param($VerbosePreference) Write-Verbose "Hello World"} -Verbose -ArgumentList ('Continue')
Environment data
Name                           Value
----                           -----
PSVersion                      6.0.0-beta
PSEdition                      Core
BuildVersion                   <IP_ADDRESS>
CLRVersion
GitCommitId                    v6.0.0-beta.2
OS                             Microsoft Windows 10.0.15063
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

@bergmeister I see the ask here, but how would you specify that you want -Verbose on the Invoke-Command, and not on the -ScriptBlock? You'd still have to provide a way to do that.

@pcgeek86 My impression was that with the general usage of -Verbose is that one either gets no verbose output or everything. For example consider when calling function A that calls function B internally, the VerbosePreference propagates from A to B by default without doing anything. That's why I thought that this would be the more intuitive default and in the special case if one really wanted the Verbose output from Invoke-Command only but not the ScriptBlock one could still pass the desired VerbosePreference to the ScriptBlock as Invoke-Command -ScriptBlock {[CmdletBinding()]Param($VerbosePreference) Write-Verbose "Hello World"} -Verbose -ArgumentList ('SilentlyContinue')
This is certainly a more philosophical discussion but to me it would make the behaviour more consistent and intuitive. Should the outcome of this discussion be a change, then one would most certainly also need to include other preferences such as InformationPreference (and maybe even the WhatIfPreference).

I too ran into the same problem with not understanding why my verbose statements within an Invoke-Command were not outputting.

I think having verbose output for both the Invoke-Command cmdlet and the command itself is sane default behaviour. If someone wants verbosity only from Invoke-Command itself and not the command, they can set $VerbosePreference or pass the -Verbose:$false switch to the commands explicitly in the command scope or by using a similar workaround explained above.

I have to support this idea as well: Invoke-Command should definitely pass it's own -Verbose parameter to the script it's executing. I've tried every workaround and they all suck. Can't see this breaking anything.

+1
This seems more like a bug than a feature. Users expect the CmdletBinding properties to propogate through an Invoike-Command - any chance they will in the near future?

Here is a related RFC.

I believe this affects also ForEach-Object -Parallel
E.g. try:
$VerbosePreference = 'Continue'
Write-Verbose 'this prints'
1..3 | ForEach-Object -Verbose -Parallel { [CmdLetBinding()]Param() Write-Verbose 'this won''t' }

and:
$VerbosePreference = 'Continue'
Write-Verbose 'this prints'
1..3 | ForEach-Object -Parallel { $VerbosePreference = $using:VerbosePreference; Write-Verbose 'but this will' }

I believe this affects also ForEach-Object -Parallel
E.g. try:
$VerbosePreference = 'Continue'
Write-Verbose 'this prints'
1..3 | ForEach-Object -Verbose -Parallel { [CmdLetBinding()]Param() Write-Verbose 'this won''t' }

and:
$VerbosePreference = 'Continue'
Write-Verbose 'this prints'
1..3 | ForEach-Object -Parallel { $VerbosePreference = $using:VerbosePreference; Write-Verbose 'but this will' }

I noticed there is no issue with $ErrorActionPreference, $WarningPreference or $InformationPreference, this applies only to $VerbosePreference and DebugPreference
Now I'm wondering if that's indeed so or is it by design that only Verbose and Debug are affected but not other preferences?
Except those 5 preferences I didn't test others.
It surprises me that I see information messages, errors and warning but not verbose and debug.


Re-Deriving Ensemble Mutations w/ different numpass parameter

Hey,
I've used ensemble mutation calling to call mutations which appear in 2/5 individual callers. I'm now wanting to go back and re-call those which appear in 3/5, 4/5, 5/5, and 1/5. Is there a way of doing this without doing the whole mutation-calling step again, but using the individual called VCF files as input? What would the yaml for this look like? I've attached the basic format of the one I'm using.
details:

algorithm:
aligner: bwa
mark_duplicates: true
recalibrate: gatk
realign: gatk
effects: vep
platform: illumina
variantcaller: [mutect, mutect2, freebayes, vardict, varscan]
ensemble:
numpass: 2
analysis: variant2
description: p145-normal
files:

p145_N_1.fastq.gz
p145_N_2.fastq.gz
genome_build: hg38-noalt
metadata:
batch: p145
phenotype: normal

algorithm:
aligner: bwa
mark_duplicates: true
recalibrate: gatk
realign: gatk
effects: vep
platform: illumina
variantcaller: [mutect, mutect2, freebayes, vardict, varscan]
ensemble:
numpass: 2
analysis: variant2
description: p145-tumor
files:

p145_T_1.fastq.gz
p145_T_2.fastq.gz
genome_build: hg38-noalt
metadata:
batch: p145
phenotype: tumor
fc_date: 'Jun-17'
fc_name: p145_T-N_Calling
upload:
dir: ./

Best wishes,
Nick

Nick;
Thanks for the question. We don't have a clean way to re-run the ensemble calling from within bcbio, unless you've saved the work directory. If you have the work directory, then you can manually remove the ensemble directory, adjust the YAML settings, and then re-run producing a new ensemble callset.
Alternatively, you could run with with the minimum numpass you want (1 in this case) and then do custom post-filtering for higher sets using the CALLERS attribute in the INFO field. That's probably the easiest way to avoid a lot of manual re-running and editing.
Hope this helps.

Hey,
Thanks a lot, I'll give it a go.


storage/reader: Avoid some copy in mvcc reader

Signed-off-by: MyonKeminta<EMAIL_ADDRESS>
What have you changed? (mandatory)
This PR tries to avoid some clones of keys in mvcc reader.
What are the type of the changes? (mandatory)

Improvement (change which is an improvement to an existing feature)

How has this PR been tested? (mandatory)
By unit tests
Does this PR affect documentation (docs) update? (mandatory)
No
Does this PR affect tidb-ansible update? (mandatory)
No

@breeswish @Connor1996 PTAL!

@MyonKeminta Do you have some interests to work on provide reference in Mvcc Scanners? According to my benchmarks there can be 7% improvements in the integration benchmark (using a MySQL client)

@breeswish Of course! But I think I need some time to think about how to do that 🤔

@MyonKeminta I once had an experiment over this. The code is pretty awful but the outcome is great. The main challenge (of writing elegant code) is that currently we use Key structure, which enforces type safety of encoded values or raw values, which is great, but itself only support owned values. There is no way to enforce type safety by only providing references. For example, our Cursor::seek and other friends accept &Key, which requires an owned Vec<..> being built (and then pass it as a reference). This fundamentally make it impossible to be zero copy (because what we receive from RocksDB is just a &[u8] ).

Sorry, why this involves me? I have no idea about these changes. Is that a misspell?

@morefreeze Yes it should be a misspell, nevermind :)

@siddontang Not yet. I didn't expect this PR to significantly improve the performance, but I just thought that it's never bad to remove the unnecessary copies.
@breeswish Thank you! I'll check it later. By the way, I'm considering whether we can let the Key able to represent both owned value and references by using unsafe code... 🤔

@myonkeminta How about just KeyRef for references and Key for owned values? (I don't know. just a brain storm)

@breeswish I don't know either before a try 😂


Grpc Timeout

BUG REPORT
**I am trying to run the marl benchmark example provided and facing the grpc timeout issue. *
SMARTS version
0.4.18
Previous associated issues
https://github.com/huawei-noah/SMARTS/issues/614
Steps to reproduce the bug
Perform the setup steps for smarts MARL Benchmark from readme  and then after installation run the below command
from baselines/marl_benchmark/
python3.7 run.py scenarios/intersections/4lane -f agents/ppo/baseline-lane-control.yaml
$ python3.7 run.py -f <config_file>
Resulting and expected behaviour
Should start the training process with out any errors
Error logs and screenshots

System information

Operating system & Environment [Please note if using Docker]: Ubuntu 18.04
Number of system cores available
Python version [Please note if using Jupyter] Python 3.7.7
Python packages [e.g. from pip freeze]  freeze.txt
SUMO version  1.10.0
Date when bug occurred 2021-11-21

*Impact [If known]

I have tested with master branch and release-4.18 branch as well and leading to the same issue. @Gamenot @Adaickalavan @RutvikGupta can you please kindly help me with the fix.

Use of grpc has been suppressed and should not cause any errors in the latest develop branch. We hope this addresses your question and thus this issue is being closed. Please feel free to reopen this issue if your problem persists or open a new issue for other questions.


Windows support

Currently, this project relies on the rlottie C library to be precompiled and pkg-config to be able to find it. This seems to be incompatible with windows.
I see two options going forward:

Somehow detect precompiled versions of rlottie on Windows. No idea if that's possible, and which tools are necessary to do this.
Add a vendor feature similar to other -sys crates in the ecosystem that downloads and compiles a version of rlottie and statically links it.

These options are not incompatible with each other, so both could be implemented if desired. As I don't have windows, I cannot implement this myself, but PRs are welcome. Any implementing PR needs to have a CI setup that tests everything on windows as I won't be able to test locally.

The vendor feature has the advantage that the user does not have to care, how to install rlottie.
As example rlottie is also not package for mac os brew.

I could run rlottie-rs on my windows.
Here is the process what I did:

install mingw and llvm, pkgconfiglite using chocolatey.

$ choco install mingw -y
$ choco install llvm -y
$ choco install pkgconfiglite -y

Install rlottie. ( ninja install may need administrator privillege )

$ git clone --depth=1 https://github.com/Samsung/rlottie
$ cd rlottie
$ meson build
$ ninja -C build
$ ninja install -C build

set C_INCLUDE_PATH, LIBRARY_PATH, PATH to each folder ( ninja install shows )

in my case, C_INCLUDE_PATH to C:¥include, LIBRARY_PATH to  C:¥lib, PATH  to C:¥bin

copy rlottie\example\win32Player\rlottie.lib and rlottie\example\win32Player\rlottie.dll to PATH folder (in my case, C:¥bin)

then finally try $ cargo install rlottie-rs


Crash when editing stream from persistent to on-demand

[ This is such a cool project!!!  Works really well. Thank you. ]
First bug report (not reproducible, but hopefully the stack is helpful.)
I edited an rtsp stream from persistent to on-demand..
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xbe18a9]
goroutine 7447 [running]:
main.(*MuxerHLS).SetFPS(...)
/data/dev/RTSPtoWeb/hlsMuxer.go:43
main.(*StorageST).HlsMuxerSetFPS(0xc00035a0e0, 0xc00003ec0d, 0x24, 0x1499730, 0x1, 0x19)
/data/dev/RTSPtoWeb/storageStreamChannel.go:353 +0x1e9
main.StreamServerRunStream(0xc00003ec0d, 0x24, 0x1499730, 0x1, 0xc000302420, 0x0, 0x0, 0x0)
/data/dev/RTSPtoWeb/streamCore.go:172 +0xd18
main.StreamServerRunStreamDo(0xc00003ec0d, 0x24, 0x1499730, 0x1)
/data/dev/RTSPtoWeb/streamCore.go:43 +0x4db
created by main.(*StorageST).StreamEdit
/data/dev/RTSPtoWeb/storageStream.go:65 +0x636
exit status 2

+1. Also I have noticed that

when it does not cause a crash, changing the state an rtsp stream from persistent to on-demand is not saved in config.json.

if the state is manually set to on-demand: true in the config.json,  this setting is changed when the program runs for a while,  to persistent again.

very strange behaviour.


[Bugs] Issues in new json settings editor

Environment

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.59.1
OS Version: ubuntu 20.04

Bug 1: nodes do not maintain collapsed/expanded state

open json settings
collapse or expand some nodes
go to other tab
go back to json tab - all nodes are expanded

It is inconvenient spending time finding a setting, switching away to refer to something, then switch back and losing your place because all the nodes are once again expanded.
Bug 2: nodes are not sorted

open json settings
nodes are shown in random order

It is inconvenient having nodes in random order - it's already hard to find settings. So for example, the "files" node should come before the "window" node.
Bug 3: first node is now redundant

open json settings
the first node is the remnant of the old json settings editor's "most recent" feature

The first node has no value anymore, and since it's not labelled as such, it gets confusing because there are double nodes for everything in that list, which show up in searches.
Justification for fixing these bugs
Those bugs have been around for a long time, but we managed. However now that the json settings editor was removed ([1], [2]), please at least meet us halfway and address them.
Without the advanced features of the old json settings editor, these bugs make the experience time consuming and frustrating.

Sorry, it's not clear which nodes you are talking about since you said "open json settings", maybe you can include a screenshot just to make sure I understand. Are you looking at the GUI editor?

Not the GUI editor but rather the new text-based editor:

ctrl + , to open user settings
or .vscode/settings.json to open workspace settings

Got it. If I fold a region of code, it stays folded when I switch tabs and switch back. Maybe you are closing the editor?
The comment about sort order is fair, it is indeed arbitrary. I would like to keep the "Commonly used" section since I think it makes the most common settings easier to discover.

Folding: works for me too, now... Weird wasn't working before.
Commonly used: I assumed it wasn't actually doing anything, didn't realize it was still operational.

It's just a hardcoded list of popular settings. Maybe we can sort the groups in a better way though


Prepare for the next stellar protocol upgrade

https://www.stellar.org/developers-blog/guide-to-protocol-14-prep

@zaibon I assume this was done in 046188aa3f9383fdb5e17f420e6d62f5fb4ba18a?

Right. I kept this open to make sure we do not forget to deploy on mainnet


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/980 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit 65e33a1:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-08-05-05-49-10 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Context length

Question
Is there any table/page with the context-len of each model?

Hello,
InternVL-Chat-V1-5 has a 4k context length. Mini-InternVL-Chat-2B/4B-V1-5 has an 8k context length. All models in the InternVL2 series have an 8k context length.


Expandable section breaks with specific content

I wanted to create an expendable section with the  and  tags but the docs crash locally when I put in certain kind of content. Here is the content I use for reproduction:
Expand if you want to view more

#### 1. Configure Prisma schema & database connection

First, ensure that database connection is configured properly. In your Prisma schema, set the `url` of the `datasource` block to the `POSTGRES_PRISMA_URL` and the `directUrl` to the `POSTGRES_URL_NON_POOLING` environment variable. You also need to enable the `driverAdapters` feature flag:

```prisma file=schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["driverAdapters"]
}

datasource db {
  provider  = "postgresql"
  url       = env("POSTGRES_PRISMA_URL") // uses connection pooling
  directUrl = env("POSTGRES_URL_NON_POOLING") // uses a direct connection
}
```

Next, you need to set the `POSTGRES_PRISMA_URL` and `POSTGRES_URL_NON_POOLING` environment variable to the values of your database connection.

If you ran `npx prisma init`, you can use the `.env` file that was created by this command to set these:

```bash file=.env
POSTGRES_PRISMA_URL="postgres://user:password@host-pooler.region.postgres.vercel-storage.com:5432/name?pgbouncer=true&connect_timeout=15"
POSTGRES_URL_NON_POOLING="postgres://user:password@host.region.postgres.vercel-storage.com:5432/name"
```

Here is what the file might look like with some sample values that look similar to the values for Vercel Postgres database:

```bash file=.env
POSTGRES_PRISMA_URL="postgres://default:3vtEpyNbn6pl@ep-red-dew-a2fqe2pt-pooler.eu-central-1.postgres.vercel-storage.com:5432/verceldb?pgbouncer=true&connect_timeout=15"
POSTGRES_URL_NON_POOLING="postgres://default:3vtEpyNbn6pl@ep-red-dew-a2fqe2pt.eu-central-1.postgres.vercel-storage.com:5432/verceldb"
```

Seems to be resolved if you put the Expand if you want to view more on a new line, e.g.:

Expand if you want to view more

#### 1. Configure Prisma schema & database connection

First, ensure that database connection is configured properly. In your Prisma schema, set the `url` of the `datasource` block to the `POSTGRES_PRISMA_URL` and the `directUrl` to the `POSTGRES_URL_NON_POOLING` environment variable. You also need to enable the `driverAdapters` feature flag:

```prisma file=schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["driverAdapters"]
}

datasource db {
  provider  = "postgresql"
  url       = env("POSTGRES_PRISMA_URL") // uses connection pooling
  directUrl = env("POSTGRES_URL_NON_POOLING") // uses a direct connection
}

Next, you need to set the POSTGRES_PRISMA_URL and POSTGRES_URL_NON_POOLING environment variable to the values of your database connection.
If you ran npx prisma init, you can use the .env file that was created by this command to set these:
POSTGRES_PRISMA_URL="postgres://user:password@host-pooler.region.postgres.vercel-storage.com:5432/name?pgbouncer=true&connect_timeout=15"
POSTGRES_URL_NON_POOLING="postgres://user:password@host.region.postgres.vercel-storage.com:5432/name"

Here is what the file might look like with some sample values that look similar to the values for Vercel Postgres database:
POSTGRES_PRISMA_URL="postgres://default:3vtEpyNbn6pl@ep-red-dew-a2fqe2pt-pooler.eu-central-1.postgres.vercel-storage.com:5432/verceldb?pgbouncer=true&connect_timeout=15"
POSTGRES_URL_NON_POOLING="postgres://default:3vtEpyNbn6pl@ep-red-dew-a2fqe2pt.eu-central-1.postgres.vercel-storage.com:5432/verceldb"

```


Cross-reference GDScript built-in rounding methods to ease discovery

This closes #19315.

Thanks!

Cherry-picked for 3.2.3.


[Monitor] az monitor app-insights api-key: Add warning to remove permission and example

Fix #1945
This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your PR is merged into master branch, a new PR will be created to update src/index.json automatically.
The precondition is to put your code inside this repo and upgrade the version in the PR but do not modify src/index.json.

add to S174


Fetch Request Crash

Hello I am running windows 11 and just attempted to use the application. for the first video it seems like it tried to download the video and then partialy through it crashed. Now when I attempt to fetch a video it just immediately crashes. Here is the error I am getting. I am running the CMD as administrator. I did change the download type to MP4 and changed the download location.After changing the video format from .MP4 back to any it is now working. I am successfully downloading videos in MP4 format even though the fomat radial drop down is set to Any. Hope this helps with future troubleshooting.
File "d:\Documents\yt-channel-downloader.venv\Lib\site-packages\pytube_main_.py", line 346, in title
raise exceptions.PytubeError(
pytube.exceptions.PytubeError: Exception while accessing title of https://youtube.com/watch?v=dhk2WU506Uc. Please file a bug report at https://github.com/pytube/pytube
(.venv) d:\Documents\yt-channel-downloader>

Hi,
Thank you for opening this issue. Recent changes on YouTube caused this problem, and there was also a bug with the "Any" format. I have fixed both issues.
You can use the updated app by cloning the repository. However, the fixed version is not yet available as an installer for Windows, as I'm planning to make a few more improvements and possibly use a different packaging method.
Please let me know if it works for you now.

Hi,
Thank you for opening this issue. Recent changes on YouTube caused this problem, and there was also a bug with the "Any" format. I have fixed both issues.
You can use the updated app by cloning the repository. However, the fixed version is not yet available as an installer for Windows, as I'm planning to make a few more improvements and possibly use a different packaging method.
Please let me know if it works for you now.

Version 0.4.2 is out and the installer for Windows is also available in Releases.


Remove styles when pasting text

Fixed by stripping styles when text is pasted.
Resolves #1
https://user-images.githubusercontent.com/47112778/229919856-5ed14c4c-b956-4039-b7ed-1fc9c8eb7f76.mp4

Perfect! Thank you


Have zipwrapper understand compressed tar files.

Is your feature request related to a problem?
Robert Casperson from LLNL called and was trying to open a compressed multipart silo file and this didn't work. He was compressing the files because he wanted to save disk space. The problem is that only the root file got decompressed and any files that were referenced by the root file weren't found. I e-mailed Mark about this and he suggested enhancing the zipwrapper reader to understand compressed tar files. This way uncompressing the file would uncrompress all the files and it would be in the proper form (assuming the files didn't have absolute paths, which would is generally a bad idea anyway).
The compressed tar file name would need to be the same as the root file name. For example, if the files were:
abc.root.silo
abc/abc.0.silo
abc/abc.1.silo
Then the compressed tar file name would be "abc.root.silo.tar.gz" consisting of "abc.root.silo" and the directory "abc".
Impact

[X] High - User productivity significantly improved

@casperson1 I'll be recording some notes here as I begin work. Since you are new to GitHub, I suggest having a quick look over our docs on managing notifications. NOTE: that link will be active only for a while because its part of a pull request. If it fails, try this one

If possible, I'd like to make the solution we develop generally useful and not require a special file naming convention. That said, one basic requirement is that each tarball (and companion root file) represents the file(s) needed for a single timestep. But, I think that can pan out in a few different ways...
Case 1A all files, including root file are in the tarball

User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
ZipWrapper uncompresses and untars foo_0000.tar.gz producing a directory, foo_0000 with a root file, foo_0000/foo.root, and domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ...,  in the directory

Case 1B all files, including root file are in the tarball but root expands at same level as tarball...

User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
ZipWrapper uncompresses and untars foo_0000.tar.gz producing a root file at the same level as the tarfile foo0000.root and a directory, foo_0000 with domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ...,  in the directory

Case 2 root file and tarball are companion files

User asks VisIt to open foo_0000.root.tar.gz
ZipWrapper uncompresses the root file
VisIt checks for companion foo_0000.tar.gz and if present VisIt untars and uncompresses the companion tarball.

From @casperson1

$ ls
xxxx00000.root
xxxx00000    <- silo files in here

$ zip -r xxxx00000.root.zip xxxx00000*
(Adds xxxx00000.root and all the files in the xxxx00000 directory)

Ideally when I open xxxx00000.root.zip in visit with ZipWrapper it would extract in the tmp directory, and know to open xxxx00000.root.  That file depends on the silo files in xxxx00000 which I believe would extract along with xxxx00000.root.zip.   It almost seems like visit knows to do this, but I may be misunderstanding.  The point I got stuck today is it didn’t actually unzip the file and seemed to try to do something with sh, leading to permissions errors.

Ideally when I open xxxx00000.root.zip in visit with ZipWrapper it would extract in the tmp directory, and know to open xxxx00000.root.

Ok, so this is captured in Case 1B...the user asks VisIt to open a compressed archive and the root file expands as the same level as the archive and all the domain files are in a subdir.

Case 1A all files, including root file are in the tarball

User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
ZipWrapper uncompresses and untars foo_0000.tar.gz producing a directory, foo_0000 with a root file, foo_0000/foo.root, and domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ...,  in the directory

In this case, the open (double clicking or whatever) of foo_0000.tar.gz turns into an open of foo_0000/foo.root.

Case 1B all files, including root file are in the tarball but root expands at same level as tarball...

User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
ZipWrapper uncompresses and untars foo_0000.tar.gz producing a root file at the same level as the tarfile foo_0000.root and a directory, foo_0000 with domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ...,  in the directory

In this case, the open of foo_0000.tar.gz turns into an open of foo_0000.root.

Case 2 root file and tarball are companion files

User asks VisIt to open foo_0000.root.gz
ZipWrapper uncompresses the root file
VisIt checks for companion foo_0000.tar.gz and if present VisIt untars and uncompresses the companion tarball.

In this case, the open of foo_0000.root.gz turns into an open of foo_0000.root which is exactly how currently designed (e.g. non-tar-enabled) ZipWrapper behaves.
In all cases, when VisIt is removing old uncompressed results, it will need to add -r to recursively removed directories containing untar'd contents.


Config

added the ability to define buttons layouts from the config.

I've tested your code and it works good.

Nice!
On Thu, 26 Oct 2023, 1:44 pm Aditya Garg, @.***> wrote:

Merged #3 https://github.com/AdityaGarg8/tiny-dfr/pull/3 into icon.
—
Reply to this email directly, view it on GitHub
https://github.com/AdityaGarg8/tiny-dfr/pull/3#event-10779419442, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEFKRHAN6TYC3ZFXQ2XVMQDYBJLMDAVCNFSM6AAAAAA6PXNB7SVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJQG43TSNBRHE2DIMQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


grid: only show base-line instead of all grid-lines

I can't seem to find a way to hide(red crosses) all but the base-line(blue arrow) on the x-axis.
I wan't an option under
grid: { xaxis:  {                                                    lines: {                                                            show: true ,                                                           startonly: true                                                            }                                                      }
Alternatively unhinge lines from tickAmount under yaxis. Setting tickAmount to 0 can show only
base-line .
My temporary solution is for the moment to hide grid, set sparkline: {enabled: false} and
use axisBorder with a manual offsetY so it fits into the 0-line-pos.

The manual offset position will not work correctly if the value in the yaxis changes.
So, here is what I suggest:

Use annotations and draw a line on 0 value.
Turn off all the xaxis and yaxis line

annotations: {
  yaxis: [
    {
      y: 0,
      strokeDashArray: 0,
      borderColor: "#333",
      fillColor: "#c2c2c2",
      opacity: 0.8,
      offsetX: 0,
      offsetY: 0
    }
  ]
},
xaxis: {
  axisBorder: {
    show: false
  },
  axisTicks: {
    show: false
  },
},
grid: {
  yaxis: {
    lines: {
      show: false
    }
  }
}

Here is a full codepen example

It will better if you hide the value of the y axis