Incremental processing

It seems like it is only possible to replace a dataset entirely and then re-run the analysis. Incremental processing would enable more efficient processing by avoiding recomputation. Here's how Pachyderm does it.

Hi @kskyten
Could you please give us a bit more details? Are you talking about deep learning style computations when a result of each epoch will be saved and then, possible, reused? If so, how do you save the results - as a single file or a set of files in an input directory (like in Pachyderm docs)?

Let's say you have a folder of images and you define a computationally intensive pipeline for them. After having defined this pipeline you acquire new images that you want to process. This corresponds to the inter-datum incrementality in Pachyderm, if I understood correctly.

We are going to implement Dependency to directory #154 which is a little bit related to this incremental feature. But it is not enough.
This is a good scenario. Let's keep this feature request - it will be implemented. We just need to decide a priority.

@dmpetrov , are we there yet or we should keep this open?

@mroutis Yep, this should work now by using dvc unprotect. https://dvc.org/doc/user-guide/update-tracked-file . Closing, please feel free to reopen.

Thanks, @dmpetrov

Hi, has this eventually been implemented ?

Hi @Pierre-Bartet, it have not been yet implemented. Would you find this feature useful? Could you descirbe your use case?

@pared Use cases equivalent to the one already described:

Scenario: I have a dir with images and I'd like to process each image with dwt_ll4.py to generate a spectrum matrixes. If new 23 images were added I'd like to run the script only for these 23 images.

In fact this is exactly at this point that data version control tools could become really useful because the situation is messier than in the 'vanilla' use case. AFAIK Pachyderm is the only tool with this kind of feature but it is super heavy. I think what is needed is to be able to

Add an empty collection of objects:
$ dvc add --collection  my_collection

Create a pipeline using this collection as a dependency
$ dvc run --d  my_collection computation.py

Later add new files to the collection:
$ dvc add new_file(s) --to my_collection

So that a dvc repro would only run on needed files.

could both mean

'apply computation.py separately on each object of the collection' -> If for example, we want to subtract the mean of each image from each pixel
'apply computation.py on the whole collection' -> If for example, we want to subtract the global mean of all the images from each pixel

I don't think that distinguishing whether to apply a given code on each image or all images should be DVC responsibility. DVC is an abstraction that takes care of tracking code and its dependencies.  But there are too many use cases of DVC (any data science project, so data can range from CSV to folder full of files).
This use case(apply on image vs dataset) assumes that DVC is aware of code logic and dependency structure, while it should not be.
Having said that I think that we could implement this feature (applying on "new" data) for directory dependencies, but it would need to be users' conscious decision to use it.
One option could be, for example, flag-like:
dvc run -d modified_folder -o spectrum_matrices_folder --incremental code.py
Implementation won't be that easy though, because we should probably create some temporary dir that would store "only new" dependency files, and apply code to this temp dir and append output to already existing output of this stage.
Some notes:

How can we be sure that output md5 is the same as it would be if we just run processing on the whole modified_folder? Does it even matter if the user made a conscious decision to use --incremental?
In the case of code using information about the whole dataset, this feature can produce wrong outputs.
(e.g subtract mean of all images, that case requires running the code on the whole dataset, in other case mean of all images won't be mean of all images, just batch mean)
(At least in the beginning) We would need verification that this option is used for dir dependency

After writing this, it seems to me that my proposition for this solution is wrong since it can easily be misused. Leaving this here for further discussion. Maybe we could implement this as advanced use case but that seems dangerously similar to --outs-preserve. Or maybe someone has a better idea of how it should be done.

@Pierre-Bartet @pared an interesting discussion. I also don't see a way how DVC being completely agnostic to the way code and files are organized now can solve this now. Thinking about the right abstraction for the job, I have first the question back:
Let's imaging we do:
$ dvc run --d my_collection computation.py
How does this script organized internally? How does it read the data from this my_collection?
I'm asking because right now like I mentioned we don't dictate any of that. But when we start doing things incrementally how do pass that delta information to the script? How will it know that it needs to read only certain files from the directory?
@Pierre-Bartet do you have some ideas that comes to your mind?

I've been thinking about it but I see no solution that would not totally change DVC's behaviour, mostly because of what you just said:

when we start doing things incrementally how do pass that delta information to the script?

The behaviour I had in mind was to have the dependencies given as arguments to the scripts (here computation.py) so that dvc run could either:

call repeteadly computation.py file_i
call a single time computation.py folder

But that is indeed a huge deviation from the current dvc behaviour

This is also a feature which I would like to see handled better with DVC if possible. Currently, the only way to avoid expensive re-computation on partial data sets when adding new data is to bypass DVC altogether, execute the processing on only the new files, and then run dvc commit on the results -- which seems rather to detract from the whole point of using DVC in the first place to track changes to datasets. And yet, it must be possible to avoid re-processing tens of thousands of files just because a single new file was added to an input directory.
@Pierre-Bartet, my thoughts for a possible behaviour were also in the same direction as yours, passing [delta] dependencies to the script being run.
A gentler approach along the same lines may be the following:

DVC could maintain an index of filepaths changed since the last commit, for example in a section of the .dvc file of the dependency in question
It's up to the script called by dvc run whether it wants to read and make use of this index to filter the input files, or to ignore the information and process everything as usual
After a successful run, the index is flushed and the hashes re-computed

This is for sure "advanced" behaviour and requires users to be aware of the existence of the index. But at least it doesn't mean DVC having to change its behaviour in regard to how it calls commands with dvc run.

I find this feature really useful :sweat_smile:
Not ML related but with ETL is pretty common to do such things.
Having a cache  is not useful if you are still recomputing files.
The simplest scenario that I can imagine is by giving the user a way to tell which files are new, for example:
mkdir data
for i in {1..3}; do echo $i > data/$i; done
dvc add data

When you add more files:
for i in {4..10}; do echo $i > data/i; done

Doing dvc ls --outs --new data.dvc would output:
4
5
6
7
8
9
10

that way the user would iterate over all of those files and run the respective command and then dvc commit data.
The implementation for ls --outs --new would be something like the following:
from dvc.repo import Repo
from dvc.stage import Stage

repo = Repo('.')
stage = Stage.load(repo, 'data.dvc')
out = stage.outs[0]
path = out.path_info

cached = set(path / info['relpath'] for info in out.dir_cache)
current = set(out.remote.walk_files(path))
new = cached - current

for path in new:
    print(new)

We could start with a simple implementation and add more features depending on users request, what do you think?

This is especially frustrating now that I tried to implement my own incremental-processing workaround: I simply set an argument to my computation.py which, if true, doesn't touch any input file that already has a corresponding output file in the output.
I realise that this breaks the ability for DVC to know that all outputs were created by the command being run (but I already can't use DVC properly in my case anyway, since the only solution I have to avoid reprocessing all files every time a single file changes in the input is to manually run computation.py and dvc commit the results, which has the same breaking effect).
So I executed my pseudo-incremental version of computation.py on my input, only to discover that DVC automatically removes anything in the output location every time dvc run is called (without warning! -- see https://github.com/iterative/dvc/issues/2027 ) 😭

Hi @markrowan !
DVC deletes those outputs automatically only if it knows that it already has them in the cache, so you would be able to dvc checkout to get them back. We have an unofficial hidden flag for dvc run, that tells dvc to not remove the particular output before running the command. It is called --outs-persist ain CLI and persist: true in dvc-file itself. We don't document it because we are not quite sure if we are okay with how hacky it is and are afraid of people misusing it and then getting unreproducible pipelines 🙂 Please be aware of that, but feel free to give it a try and if you do, please make sure to let us know what you think about it. 🙂

This is a great use-case. I even heard about this requirements for a couple of times outside of our issue tracker like "can DVC processes only new images?". It needs to be implemented as a part of DVC.
Challenges
There are some challenges with API and how to align the feature with the current DVC interface and which part goes to code:

"a huge deviation from the current dvc behaviour" @Pierre-Bartet
"[use case] ... assumes that DVC is aware of code logic and dependency structure" @pared
"... how DVC being completely agnostic to the way code and files are organized..." @shcheklein

Solutions
All of the above questions are great. Also, @Pierre-Bartet, @pared and @markrowan pointed to possible solutions. Some of the solutions might align well with DVC design.
Passing files as an argument (append to the args list) to the code is the best protocol between DVC and code that I see for now (I mean the first option with file_i):

The behaviour I had in mind was to have the dependencies given as arguments to the scripts (here computation.py) so that dvc run could either:

call repeteadly computation.py file_i
call a single time computation.py folder

How this can look from command line:
$ ls my_collection/
file_pig.jpg   file_cat.jpg   file_dog.jpg
$ dvc run --process-dependency my_collection/ --process-output processed/ python computation.py
# it runs the command for each of the file/dir with new/changed filename for each of the runs
#   python computation.py file_pig.jpg
#   python computation.py file_cat.jpg
#   python computation.py file_dog.jpg

What is important:

DVC needs to understand that it works in "streaming" mode and it needs to be signaled to the tool by a separate command (not dvc run) or by a special set of parameters like in the example above --process-dependency/--process-output.
DVC should create output dir for the first run when it was not yet created
DVC should not remove the outputs dir before running/reproducing the commands
Regular dependencies -d myscript.py -d libs/ might also be used
Regular outputs cannot be specified -o logs/ (why not? any useful use case?). I feel like here is the place with another "deviation" from DVC.
To make this command reproducible we need to make sure that DVC can get a checksum for each version of input dir. (how to get checksums for dirs? if not data dir? should we limit the command by data dirs?)
Inside the input dir we should be able to work with files, dirs or both.
Store all inputs and all outputs checksums (index from @markrowan comment). Probably the same way we do it now. Based on my understanding, we don't need to store mapping 'input_file_i --> processed_file_i' we just need a "checksum" of the entire input dir and each of the outputs.

Open design questions

Do we need a separate command for this like dvc process/stream instead of dvc run
Better option names instead of --process-dependency and --process-output. Related to the previous question.
--recursive option to process all files from the dir. This option is a reason to have a separate command since dvc run does not need it.

Presenting result to users
When the use case will be implemented we need to show a nice looking use case in the documentation or a separate blog post. It's great to have this use case even before the release - it will help driving requirements and test the implementation.
@Pierre-Bartet @markrowan @kskyten could you please help us to define the use-case? The major questions:

Dataset. Which dataset we should use? It should be open, and look natural, not like file_cat.jpg  file_dog.jpg.
Motivation. What kind of processing do we need to apply? If images, we should explain the motivation of not doing this processing on-flight or why the processing result should be stored. Heavy image processing - any examples of this?

If someone can participate or even drive this doc/blog-post activity it would be awesome.
Related issues
For a broader perspective, there are some issues that might be (or might not) related to this one. It would be great is we find some similarities in the scenarios and can reuse this.

#1018 "build matrix". Difference: the build matrix has a predefined set of runs (and run parameters) while this issue takes all the files as the set of parameters. In theory, build matrix might be extended to this use case.
#2825 pulling subset of files from a data-dir. It is kind of "interactive mode" of working with data folder. @Pierre-Bartet used this interaction-like trick in command $ dvc add new_file(s) --to my_collection
Also, there was a verbal request from a user who asked for the ability to add a file into data-dir without pulling data (for optimization purposes). It is an inversion/push-version of #2825.

Regular outputs cannot be specified -o logs/ (why not? any useful use case?). I feel like here is the place with another "deviation" from DVC.

Yes both the input and the output will be some kind of new 'collection' objects.

To make this command reproducible we need to make sure that DVC can get a checksum for each version of input dir. (how to get checksums for dirs? if not data dir? should we limit the command by data dirs?)

This can be done using hash lists or hash trees.

Inside the input dir we should be able to work with files, dirs or both.

Therefore hash trees. Also trees can help if you want to be able to support folder with a large number of files (I've faced several use cases involving millions of files).

Related issues

Isn't all this also related to parallel processing ? (not a rhetoric question)

Inside the input dir we should be able to work with files, dirs or both.

Therefore hash trees. Also trees can help if you want to be able to support folder with a large number of files (I've faced several use cases involving millions of files).

Right. We are already using some kind of hashing in DVC. It would be great if the current hashing mechanism would be reused. The existing one might not work for not-data files and dirs.

Related issues

Isn't all this also related to parallel processing ? (not a rhetoric question)

Absolutely! Updated.
It will require an additional command line parameter like --jobs 24 which is another argument for separating this into a separate command to not over pollute dvc run params list.

There is still a sub use case of training on top. For this you both need an old output (a pretrained model) and a set of added files (new inputs), which is essential to some deep learning, esp. images/nlp scenarios.
This, however, complicates this a lot:

and output is reused as a dep,
there is no checksum reproducibility, because the final model depends on how we chunked the data.

an output is reused as a dep,
Isn't it already the case ?

there is no checksum reproducibility, because the final model depends on how we chunked the data.

Why ? In the situation you describe, the model is build from all the images, so it is not incremental: the training script must be applied to the whole image folder, not image by image.

Isn't it already the case ?

Not if you in fact process them 1 by 1, so that f(collection(items)) = collection(f(items)).

Why ? In the situation you describe, the model is build from all the images, so it is not incremental: the training script must be applied to the whole image folder, not image by image.

You train a model on a set of images, then you add a new batch and you train on a new batch starting from the pretrained model, not from random weights. This will result in a model knowing about images of both batches, but this is not the same as training on a combined set form the start, so we loose checksum reproducibility. This might be, however, close enough and faster to compute.

We can think of the difference as "do our transform function have this property":
f(chunk1 + chunk2, start=base) = f(chunk2, start=f(chunk1, start=base))

And also is it a strict = or a good enough ~ for a user.

We should also consider raising the priority. What do you thing guys?

You train a model on a set of images, then you add a new batch and you train on a new batch starting from the pretrained model, not from random weights.

Ok I see, so the flow will be like that:

As long as a graph is a DAG there should be no checksum problem. Here maybe what is confusing is that both 'model 1' and 'model 2' could share the same key, for example 'model.pkl', but which one would you choose ? If you choose 'Model 1 + 2', then 'Model 1'  will be inaccessible. It's a problem of hiding an intermediate result, there is no checksum problem here.

By nature of dvc both models will share the same filename, they will go in different git revisions though. Batches also combine into a single dep, in current design out = f(dep), that's it, history of a dep does not affect out, with incremental training it might affect.

Ok now I understand, there is 'currently' no way to distinguish between:

and

But I am not sure to un derstand this 'currently':

Batches also combine into a single dep

What batches, are they already implemented ?
IMHO the use case you describe already works with DVC: you prepare 'model 1' with the data you have, then when you have more data you build a new 'model 1 + 2' from 'model 1' plus the new data.
But as you said, if there are lots of batches,  maybe some sugar can be added to hide 'model 1' and expose 'Model 1 + 2'.

What batches, are they already implemented ? Are we talking about the current DVC status or the possible implementation of incremental situations ?

You can add a dir with files into dvc, and then add some files to that dir and dvc add it again. It will have new checksum but same path, so you can rerun a pipeline with dvc repro on the new data, this will involve processing the whole thing of cause.

IMHO the use case you describe already works with DVC: you prepare 'model 1' with the data you have, then when you have more data you build a new 'model 1 + 2' from 'model 1' plus the new data.
But as you said, if there are lots of batches, maybe some sugar can be added to hide 'model 1' and expose 'Model 1 + 2'.

You mean add a new stage to produce a new model?

So you dup all outs and deps and stages. We can say that in dvc logic the data dep, might not be simply a dir of files, but a collection of batches (or a dir with some history), i.e. an incremental data artifact, then an input of a stage is also not a dir, but a collection of batches, then output will be reproducible, only to reproduce that you need to do:
f(chunkN, start=f(chunkN-1, start=...))

We will need probably an erase history command too.

I've never encountered the situation you describe: either the number of batch was small, and indeed I wanted to train Model 1 + 2 using Model 1 + Batch 2, with a tractable number of batch and a need for being able to access Model 1 and model 2 separately, or the number of batch was big enough so that I wanted to retrain on all the batches (not incremental). But indeed your use case exists and it would be a nice feature.
I don't know about the exact DVC internal mechanism, but it seems to me that you (the front user) could write a pipeline stage like this:
f(chunkN, start=f(chunkN-1, start=...))
But under the hood it would be in reality:
dataA -> prepared_dataA -> modelA -> metricsA (this might be longer)
                           \
dataB -> prepared_dataB -> modelB -> metricsB
                             \
dataC -> prepared_dataC -> modelC -> metricsC

...

.. you don't die somewhere here because it is just automatically done ...

@Suor @Pierre-Bartet either I'm missing something or the case of running the stage on all images + pre-trained model can be solved by https://github.com/iterative/dvc/issues/331#issuecomment-559254859 (it's already in DVC but  we don't merge docs for since don't like introducing "virtual" cycles into DAGs for now - so consider this semi-official).
The case from the ticket description is and a bunch of comments below is about actual Pachyderm-like incremental processing - when you run your script only on new files.
Just to clarify, which one those have you been discussing?

@shcheklein persist outs might help, we still need two things:

signaling to command which files are new
checking pretrained model existance and checksum as we do for deps

@shcheklein : Indeed what we've been discussing with @Suor is a different kind of 'incremental'

I think this depends on the what the DVC think itself to be?
For me it's a data tracking tool in training staging, and data tracking and metrics comparing help me a lot when I make some experiments to the model, after that I use other tools to deploy my model.
So I use DVC as a data tracking tool like git, not a computation engine .I don't know what kind of tool, you guys want it to be

One more case and request - https://discordapp.com/channels/485586884165107732/563406153334128681/718108510671601824

Hi! My current main issue with DVC is the fact that the pipelines are structured with a stage = few inputs -> a few outputs. That works great on some cases, but it fail when I need to apply the same operation on many files (tens of thousands in my case). Eg running a ML model for detecting the keypoints of a person. Basically there is no map operation. Do you know if that is on the roadmap, or if there is a github issue about that?

So we might have map-style incremental processing and reduce-style one. The map seems like a lot simpler should we separate it?

@Suor it seems to me that both map or reduce would require some general mechanism first - how to apply certain logic only to new files. The way that logic being applied and actual output is being produced seems to be a simpler question to solve (and we even have some hacks like persistent outputs). Unless I'm missing something. Could you clarify, why do you think they are inherently different?

I am the author of the comment above on Discord:

One more case and request - https://discordapp.com/channels/485586884165107732/563406153334128681/718108510671601824

I think it's also interesting to look at that use-case in relation with:

"dvc: consider introducing build matrix" #1018 (already mentioned)
"How to manage repetitive dvc run commands (like unpacking of many zip files) #1119
"Reconfigurable pipelines" #1462
"Taking a step back and looking at the big picture" #3549

A potential different workaround would be to allow pipelines to create pipelines, so a "master" pipeline could create N different pipelines (one per file to process), and those would be executed one after the other. This would also allow very flexible workflows with dynamic pipeline reconfiguration based on scripts. "ML experiments and hyperparameters tuning" #2799  could be related.  I can't find any issue related to that one, is it hidden or should I create it?

@MatthieuBizien Please feel free to create an issue for that :)

I hope this suggestion adds something to the above thread.
Say I have 1M files in the directory ./data/pre, and a python script process_dir.py which goes over each file in ./data/pre and processes it and creates a file in the same name in a directory ./data/post.  e.g. ./data/pre/123.txt --> ./data/post/123.txt
As we all know, if I define a pipeline:
dvc run -n process -d process_dir.py -d data/pre -o data/post python process_dir.py

then dvc repro will erase all content of /data/post and re-run the script from scratch if I add or remove even one file from /data/pre, which unfortunately means that those 1M files will unnecessarily be processed again - even though we keep cached copies of them.
suggestion: if we were able to define a rule that connects directly in the DAG pairs of data/pre/X.txt --> data/post/X.txt in the context of the process stage, then when can adjust the process stage in the pipeline as follows:

identify which file-pairs haven't changed and remove those files to a temp dir
run the process stage as you normally would.
move the file-pairs from the temp dir back to their original locations.

It's like we are enhancing the DAG to include more local dependencies (and independencies), between input files and output files.

@jonilaserson the discussion here circles around that in a way. Your use case is however is a subset of whatever we discuss. It also has a workaround employing an undocumented --outs-persist feature, see above.

I'm using DVC for a project which processes continually produced time-series data in a series of steps. I described my use case above a little, but maybe I'll mention how I'm using it now.
My current system wraps DVC in code which produces DVC jobs based on the input data: it scans the input directories, creates a DVC file describing a job for each file that it finds unless a DVC file is already present and then runs DVC repro to compute any steps which aren't yet present.
The benefits of this structure are that, once the wrapper step has run, the repository is a fully valid DVC repository which doesn't need any special tool beyond DVC itself to interact with. In order to make this work, I needed to bin my data into preset chunks and have a way of mapping from inputs to outputs (since those outputs are then used in subsequent steps). In my time-series application, I did this by mandating filenames like "20200720_stepA_1.csv".
To bring this functionality into DVC itself, I'd imagine a couple of changes.
Meta jobs
I found the generation of DVC jobs to work quite well. You could imagine a "meta-job" which defines how input data should be processed to output data, and can then be used in later steps. Meta jobs can form their own DAG, even if there's no input data yet.
Running a metajob creates normal DVC jobs for each valid combination of input chunks present. A chunk is present if either

it depends on raw files (identified somehow in the metajob config) which are present
it depends on outputs from previous metajobs in the metajob DAG, whose input chunks are all present

Metajobs can also depend on normal DVC jobs/files, in which case the dependency is passed to all generated jobs. All metajobs are run before normal jobs, to generate a set of normal jobs for all available input data. The normal jobs can then be run in the same way as they currently are.
Step to step mapping - inputs
The step-to-step mapping would need to be configurable. This would need to be done such that subsequent steps can use the outputs of previous ones. You'd need to decide "do I always map a single step output to a single step input?" I.e. if my input source (call it "stepA") has data like this:
20200718_stepA_0.csv
20200718_stepA_1.csv
20200719_stepA_0.csv
20200720_stepA_0.csv
20200720_stepA_1.csv
20200720_stepA_2.csv

And "stepB" depends on "stepA", should "stepB"  be called three times (for the three days, with all inputs for a given day passes as inputs to stepB), six times (for each file, passed individually) or once (easier implemented by passing the whole folder, as in the current version of DVC).
You might achieve this by e.g. defining a regex for inputs. For stepA above you might store the regex ^(\d{8})_stepA_\d.csv$ in your step config. This matches all the desired inputs.
For case 2 (stepB should be called six times), this would be enough.
For case 1 (stepB should be called three times, with grouped inputs) you might also specify
groupBy:
  - 1

where this tells DVC to group by the contents of capture group 1 in the regex (i.e. the timestamp).
Step to step mapping - outputs
The outputs would also need defining. "stepB" might only have a single output, with the form $1_stepB_0.csv. This is a regex replacement string, and should resolve to a specific, single filename for any of the inputs which match the input regex.
Input passing
Since your steps are being called with different inputs each time, you'll also need some way of passing these inputs to the command that's being called. That might involve a substitution token in the command, like python process_files.py %INPUT_1% or, for a variable-length list of input files, python process_files.py %INPUTS_1% with an additional field input_1_format: "-f %INPUT% ".
That's a bit ugly right now, I'm sure some thought could make it more elegant.
Summary
That's a bit of a brain-dump and I'm sorry if it's not totally coherent. I thought I'd document my experience with adding this kind of functionality on in case it informs the requirements for adding this into DVC itself.
I'd summarise my suggestion by:

Define metajobs which can generate DVC jobs for batches of available input data
Let these metajobs identify which input data they should be called with, and what output this step should produce in response
Have metajobs generate normal DVC jobs for all possible combinations of inputs (according to the metajobs' specs)
Process the resulting network of DVC jobs in the normal way, committing the both metajob DVC files and the generated DVC files to the repo (metajobs would contain a hash of all the input filenames / output dvc jobs).


Question about user identify

Hi team,
a question about the js api call regarding to user identity.
So we are using mixpanel.alias when user signup and mixpanel.identify for user login.
For some reason we have some incorrect unique id set to some user, (seeing from mixpanel live view dashbroad)
so we try to have fix the issue by adding the below code when our app starts.
const id = mixpanel.get_distinct_id();
if (INCORRECT_FORMAT) {
   mixpanel.people.set({
      '$email': `${CORRECT_EMAIL}`,
      '$name': ${CORRECT_NAME}`
  });
  mixpanel.identify(`${CORRECT_ID}`);
}

However, it doesnt seems to showing correct beacon on the live view (checking from browser console also mixpanel debug plugin that we are sending correct distinct_id.
Any suggestion?

This sounds like an implementation issue which our support team can help with. They've got a lot of experience helping correct id and aliasing issues, and can work with you personally to get your project straightened out. Just drop that text in an email to<EMAIL_ADDRESS>I'm going to close this issue out for now, but if it turns out to be a lib code issue that our engineering team can help with, will be happy to reopen.

thanks


The computational graph will be destroyed after feeding tensor/list of tensors into vmap

Hi,
I've got some troubles with vmap in my code. I wanted to use vmap to accelerate the element wise gradient computation but found vmap destroyed the computation graph inside. Below I show the minimal reproduction code. I would like to know whether it is something expected. Or is there any alternative way to compute element wise gradients efficiently? (A detailed post for my need can be viewed here (pytorch forum)).
to reproduce:
import torch
from functorch import vmap

# vmap on tensor
a = torch.rand(5).cuda()
a.requires_grad = True
b = a * 100

print(a.requires_grad, b.requires_grad)

def test1(a,b):
    return [torch.tensor(a.requires_grad), torch.tensor(b.requires_grad)]
    # return torch.autograd.grad(a, b, retain_graph=True, create_graph=True)[0]

print(vmap(test1)(b,a))

# vmap on list of tensor
a1 = list(a.chunk(5))
b1 = list(b.chunk(5))
for i in range(5):
    print(a1[i].requires_grad, b1[i].requires_grad)

def test2(a,b):
    return [torch.tensor(a[0].requires_grad), torch.tensor(b[0].requires_grad)]
    # return torch.autograd.grad(a[0], b[0], retain_graph=True, create_graph=True)[0]

print(vmap(test2)(b1,a1))

Thanks!

@kfxw what kind of model are you using? Depending on the model we might point you to different solutions.
To compute per-sample-gradients using vmap, we cannot call torch.autograd.grad inside of vmap, which I think is what you're trying to do. This is a limitation with how autograd works. Instead, we need to reframe the problem using functorch's grad transform.
There's an example on how to compute per-sample-gradients of a very simple model (that is just one nn.Linear) here: https://github.com/facebookresearch/functorch#working-with-nn-modules-make_functional-and-friends. I copied the example below:
import torch
from functorch import make_functional, vmap, grad

model = torch.nn.Linear(3, 3)
data = torch.randn(64, 3)
targets = torch.randn(64, 3)

func_model, params = make_functional(model)

def compute_loss(params, data, targets):
    preds = func_model(params, data)
    return torch.mean((preds - targets) ** 2)

per_sample_grads = vmap(grad(compute_loss), (None, 0, 0))(params, data, targets)

Note that we use functorch's grad operator to compute gradients and then use vmap to batch over them to produce per-sample-gradients.

Thanks for your reply!
I've taken a look at the grad example. I think using the combination of vmap and grad means that you want the same data division / parallelization for both forward and backward pass.
However, in my case this could be different. Briefly speaking, the forward pass is the conventional mini-batching, parallelizing on the batch dim. The backward pass is going to compute the diagonal of the jacobian (and hessian) matrix, parallelizing on more dims. grad is not capable to handle the different parallelizations. This kind of computation is usually seen for physics-related models. I can show you an unaccelerated example below.
# input data, the 1D coordinates X with the shape [batch_size, num_points]
#   e.g. X.shape --> [3, 500]
pred = model(X)     # parallelized on the 'batch_size' dim
sum_1st_grad = torch.zeros(3)
# how to speed-up this loop
for b in range(3):
    for i in range(500):
        sum_1st_grad[b] += torch.autograd.grad(pred[b,i], X[b,i])[0]     # d pred_i / d x_i

As for more information about the model, it can be a Conv1d or a graph conv imported from pytorch_geometric. So the non-diagonal values of the jacobian matrix d pred_i / d x_j may not be zero.
Thanks for your time!

Hi @kfxw, thank you for the clarification! You're right -- the example I provided above is different from what you're looking for.
Does something like the following reflect the semantics you want? (It uses nn.Linear, but you can plug in Conv1d or something else) It sounds like for each element in the batch (of size 500), you want the summed diagonal of the jacobian (or hessian)
import torch
from functorch import make_functional, vmap, jacrev

X = torch.randn(3, 500)
model = torch.nn.Linear(500, 500)

# Size [3, 500, 500]
jacs = vmap(jacrev(model))(X)

# Get each diagonal and sum them.
sum_1st_grad = jacs.diagonal(1, 2).sum(-1)

Actually, we might need one additional step. The item passed to vmap(jacrev assumes that the model isn't parallelized on the batch_size dim. We can resolve this by unsqueezing and squeezing a dimension:
import torch
from functorch import make_functional, vmap, jacrev

X = torch.randn(3, 500)
model = torch.nn.Linear(500, 500)

def unparallelized_model(x):
    return model(x.unsqueeze(0)).squeeze(0)

# Size [3, 500, 500]
jacs = vmap(jacrev(unparallelized_model))(X)

# Get each diagonal and sum them.
sum_1st_grad = jacs.diagonal(1, 2).sum(-1)


GitLens is crashing VSCode immediately after startup

GitLens Version: 11.0.1
Git Version: git version 2.22.0.windows.1

VSCode Version: 1.51.1
OS Version: Windows 10 (2004)

Steps to Reproduce:

Open VSCode with extension enabled.
"Extension host terminated unexpectedly" almost immediately.
Disable GitLens extension, restart extension host or reload the window.
Everything is fine.

When I open VSCode DevTools, I can see that extension process seems to OOM / can't GC:

Initially I thought this was a general VSCode issue, and it took a long time to pinpoint it to a specific extension:

Ughhh so in the end I had to disable all extensions (turns out, I've got lots of them over years) and then manually go through them one by one to re-enable.
In the end, turns out it was GitLens - the one I suspected the least...
[...]
Originally posted by @RReverser in https://github.com/microsoft/vscode/issues/107950#issuecomment-729680457

I've only noticed this started happening last 2 days, but given that 11.0.1 release 2 days ago doesn't seem that big, it could be that the issue actually started when extension got updated to 11.0.0.

Nevermind, I think it's still a general VSCode issue, it just takes longer to OOM now...


What is Lottie.vue in example code?

The example code references Lottie.vue, what is this? What would go in that component? I'm a little confused

I also have this doubt. Did you find any solution?

I also have this doubt. Did you find any solution?

Yes I found that this component file (Lottie.vue) is actually in the the vue-lottie folder in node_modules, I just copied it out of there and placed with my other components then linked to it... No idea if that's what officially needs to be done or why there is no mention of it in the docs though...


//WIP: Add a horizontal auto scaler and Metrics

This PR adds a Horizontal Auto-Scaler and Metrics to the kubenab Application.
The HPA (Horizontal Pod Auto-Scaler) can be configured to work with native K8s Metrics (CPU Utilization, etc.) or with Prometheus Metrics (Request per Second; Request Duration; etc.)
Closes #11 and Closes #13
TODO

[x] Add Prometheus Metrics

[x] Use the CRD's of the Prometheus-Operator to collect the Metrics

[ ] Add Metrics to Helm Chart

[ ] Document usable Metrics of HPA in the Helm-Chart values.yaml

[ ] Document Usage of the HPA

Changelog
Here is a Changelog, which can be later used when releasing a new version.

Helm-Chart: Add minimum Kubernetes Version (1.16+)
Raw Deployments: Update apiVersion to support deployment on a 1.16+ K8s-Cluster

@l0nax thanks for your PRs 👍
Could you please rebase this PR from master, as I did split main.go to two files:

main.go
admissions.go

Thanks

@rimusz master rebased over feature/11_add-a-horizontal-auto-scaler

Please move HPA code out from the master and put to addmissions.go, thanks

Done

But then we would have Problems with Deployments on 1.16+ K8s-Clusters.
Because the APIs has been marked as deprecated.
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#deprecations-and-removals

We could add a check for the Cluster Version and then change the apiVersion dynamically.

yes, the ApiVersion needs to be changed dynamically as e.g. cloud providers there is no k8s 1.16 support yet and it is very rear to see k8s 1.16 in prod yet

Added Support for K8s-Clusters with version <=1.12 and >=1.16

The custom Metric for Request per Second in combination with the HPA is implemented and works but the Request Duration/ Latency seems to be difficult to implement with the prometheus-adapter I think I will need some more time.

The HPA Feature works now perfectlly.
The HPA reacted after 10 seconds.
~] k describe hpa release-name-kubenab
Name:                                  release-name-kubenab
Namespace:                             monitoring
Labels:                                app.kubernetes.io/instance=release-name
                                       app.kubernetes.io/managed-by=Tiller
                                       app.kubernetes.io/name=kubenab
                                       app.kubernetes.io/version=0.3.2
                                       helm.sh/chart=kubenab-0.0.7
Annotations:                           kubectl.kubernetes.io/last-applied-configuration:
                                         {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"...
CreationTimestamp:                     Wed, 09 Oct 2019 21:48:42 +0200
Reference:                             Deployment/kubenab-deployment
Metrics:                               ( current / target )
  "http_requests_per_second" on pods:  568870m / 300
  "http_request_duration" on pods:     4952m / 5
Min replicas:                          1
Max replicas:                          10
Deployment pods:                       1 current / 2 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from pods metric http_requests_per_second
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  15s   horizontal-pod-autoscaler  New size: 2; reason: pods metric http_requests_per_second above target

The deployment get's scaled up:
[...]
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubenab-deployment-6468d895f5 (2/2 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  97s   deployment-controller  Scaled up replica set kubenab-deployment-6468d895f5 to 2

The PR is now ready to be reviewed.

/label enhancement


Increase of memory

Dear Developers:
I notice that after the last pandas-plink update, the memory needed to load the same amount of matrix significantly increase (during the process of lazy loading). I wonder could you resolve this issue

I met the same issue. When I tried to read chr1 plink file, I gave 60G and failed. but I increased to 128G and it worked

.

I will close it as it seems to be solved. Sorry for picking it up just now...


Does it not support tree data?

Does it not support tree data? If so, how should the support be implemented?Because I found that I was returning an empty array!

i need support tree data～


[OSD][2.7] Update observability plugin commit

Description
Update observability commit to include:
https://github.com/opensearch-project/dashboards-observability/commit/88bd10260316af2457cbcb45466ac89dbba08892
Issues Resolved
n/a
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #3444 (bbfafb3) into main (3cbb6ce) will decrease coverage by 0.29%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main    #3444      +/-   ##
==========================================
- Coverage   91.74%   91.46%   -0.29%
==========================================
  Files         172      172
  Lines        4991     4991
==========================================
- Hits         4579     4565      -14
- Misses        412      426      +14

see 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Interleave shell script with python code

So that the shell script could be more useful. Maybe create a template language, similar to Django's templating system.

It is unlikely to happen. Since it is very likely that the python code needs to call stash for executing shell commands (i.e. python is embedded in shell scripts but shell script is also embedded in python, multiple layer nesting), it makes little sense to have the python code interleaved in the first place. An example is as follows:
cd dir1
{%
for i in range(10):
    _('touch tmp%d' % i)
%}

The above code might as well just be a pure python script since the python code needs to call stash anyway, i.e.
_('cd dir')
for i in range(10):
    _('touch tmp%d' % i)

plus the first approach will lead some complications to sub-shell emulation.


removed base64coder dependency, using core lib Base64 instead.

removed base64coder dependency, using core lib Base64 instead.
like suggested in:
https://github.com/tsums/AndroidCookieJar/issues/3#issuecomment-144480318

Can you update the README file as well with this change?

Sure. I'll do it tomorrow.


claudia destroy throws error if I have a Custom Domain Name

Please use GitHub issues only to report bugs. To ask a general question or request assistance/support, please use the Claudia.js Gitter Chat instead.
To report a bug or a problem, please fill in the sections below. The more you provide, the better we'll be able to help.
I know this is expected. And the error message is helpful 😄
I think we could introduce a --force flag to also remove the Custom Domain Name. Or friendly print out a command to remove the Custom Domain Name.

Expected behaviour:

What actually happens:

Link to a minimal, executable project that demonstrates the problem:

Steps to install the project:

Steps to reproduce the problem:

this is not a bug. please use the gitter channel for feature ideas


MSProducts.xml System SKU mismatch

Hi,
Noticed this morning when downloading driver packages that the comments field is being populated with incorrect information for two Surface devices:
"Model SystemSKU="Surface_Book_2_1832, Surface_Book_2_1793" ProductName="SurfaceBook" DisplayName="Surface Book""
"Model SystemSKU="Surface_Pro_6_1796_Consumer, Surface_Pro_6_1796_Commercial" ProductName="SurfaceGo" DisplayName="Surface Go""
The Surface Go is an easy fix, but for the Surface Book I can't find any reference on Microsoft docs to what its system sku is due to it being retired, I believe. Also can't find any reference to "Surface_Book_2_1832" only "Surface_Book_1832" which related to the Surface Book 2.
Thanks
Phil

The skus for the Surface Laptop / Laptop 2 / Laptop 3 / Go models are mixed up as well.

All SKU's have been updated in the XML. The source content for the SKU values is - https://docs.microsoft.com/en-us/surface/surface-system-sku-reference


A couple of new client issues

A couple of clients got caught in the crossfire.
@bio-boris, @JamesJeffryes

a4353da has been merged and deployed on ci/appdev

Thanks.


Left align tabs when they wrap to second line

Is this something we should push back into patternfly instead?  @rhamilto

It works OK on the Patternfly page here: https://www.patternfly.org/widgets/#tabs. (Actually all of the secondary style tabs have a left padding there, which I don't like as well for our UI.)
It might be the way we mixin the pf styles because of our use of uib-tabset.

Is this something we should push back into patternfly instead?

Probably not because this style is really intended to be used as a secondary level of tabs, and removing the left padding on the secondary would mean the text labels of the first primary and secondary tabs wouldn't align, which looks kinda funky IMO.

I actually prefer to have the left padding in either context (i.e., do not remove the left padding).  Does it looks great?  No.  Does it look less awkward than removing the left padding?  Maybe.

This is how it would look if we kept the padding. (For what it's worth, we're not removing it ourselves. It's lost when we mix in the pf secondary tab styles to uib-tabset.)


Fix incorrect prop type (arrow)

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:

What is the current behavior?
According to docs, arrow supposedly can have the null value but it's type prevents that.
Issue Number: N/A
What is the new behavior?
Allow null for arrow prop
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

Awesome! thanks for the contribution too!


Remove trailing spaces

Related to issue #13.
Remove trailing spaces.

Rework needed as resolving conflicts.


Double check map updating

When using this stuff in JupyterLab, it looks like the map may need to be updated when the notebook is resized. I think we used to have the right logic for the file viewer part of it, but not sure if that got used in the output renderer.

Looks like there is no updating of the map view under resize events. I think the issue is that react.js doesn't know about any of that, so as long as we are using react.js it may be tough to propagate that down to react. We can talk about this and see if there is a way of getting it to work.

It appears to be working for me.

Now I see what you mean! Before, the map was not centering on resize. I am now passing width and height props to React (which are pushed down onResize) and now it looks much better:

Great!


Added validator('reset') to index.html

This is missing from documentation:
$(...).validator('reset')
I found this feature from searching PRs. This will be helpful for people using this library.

It's not meant to be part of the public API. It gets called automatically when the form is reset.

Hmm, the current v0.11.5 exposes this now. I'm using this library on my
angular project and I needed this to reset the validation state when
switching data entry objects.
For example, user1 is validated to have invalid field. Then, I switched to
user2. Without reset, user2 will show invalid fields.
Do you recommend another way to do this?
Thanks,
David
On Wed, Aug 17, 2016, 5:45 PM Cina Saffary<EMAIL_ADDRESS>wrote:

Closed #399 https://github.com/1000hz/bootstrap-validator/pull/399.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/1000hz/bootstrap-validator/pull/399#event-759146532,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADVIiyDWTgIF0n6NQWiqG6D9VOLG5mIdks5qg4D5gaJpZM4Jm6qi
.

It's on the prototype, so it's technically accessible by calling .validator('reset') but that doesn't mean it's necessarily part of the recommended public interface.
If you call $('#myUserForm').trigger('reset') or myFormElement.reset(). Let me know if resetting your form isn't an option and I'll consider elevating this to the public API. I'm not against it, but if it can be avoided I'd rather not increase the breadth of the API.

Yes, $('#myUserForm').trigger('reset') works. As a side note, I needed to call .validator('update') since not all elements are on the DOM due to how angular loads. So, it's natural for me to call reset and didn't think of using jquery's trigger.
Like I said, trigger works so I can use that. I'm wondering your reasoning for making an API public vs not.
Also, the way I'm mixing your validation versus my server validation, I'm wondering if there's a way to force the check mark to be a cross mark?  After submission (like email is already in use.), I can show the error but it's still checked. Technically, it's checked because the email is valid. What's the correct way to do this. Thanks again for your help!

I figure out how to force a cross mark using angular way. Thanks again.

I'm wondering your reasoning for making an API public vs not.

My goal is to make the plugin as simple as possible to use. The more options and methods the plugin has, the more daunting of a task it is for new users to understand it. I want as much of the plugin's ideal behavior to happen without the user having to think about it or write extra code for it.
In the case of Validator.prototype.reset, it was added because people were resetting their forms and finding that the validator state was still persisting. It made sense to just clear the validator state automatically as a result of the form being reset.


anthropic_version = packaging.version.parse(version("anthropic")) AttributeError: module 'packaging' has no attribute 'version'

System Info
When I initialise ChatAnthropic(), it got the error:
anthropic_version = packaging.version.parse(version("anthropic"))
AttributeError: module 'packaging' has no attribute 'version'
Who can help?
@hwchase17 @agola11
Information

[ ] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
from langchain.chat_models import ChatOpenAI, ChatAnthropic
llm = ChatAnthropic()
Expected behavior
As shown above.

I tried to run the same, and I am able to reproduce the same error (AttributeError: module 'packaging' has no attribute 'version')

What's working is the solution suggested by bot i.e.

from importlib.metadata import version
from packaging.version import parse

What might be going wrong in our case is that, package.version is getting confused with importlib.metadata.version and there is a mixup.

What we can do to resolve is:

from importlib.metadata import version
from packaging.version import parse

anthropic_version = parse(version("anthropic"))

out of curiosity, what python version are you getting issue with?

My version is 3.10.12


The Wbits=8, Abits=8 degrades serverely on CIFAR10 comparing to the 32bits version

I tested the cifar_train_eval.py with resenet20, lenet5 configuration. And I discovered that after the weights are quantized if activations are further quantized as well, the accuracy drops severely for both network.
And I also noticed that it was due to the activation quantization in quant_dorefa.py did not correctly cover the dynamic range in the activation (refer to pytorch static quantization).
I will make a PR later on.

Hi. I have another opinion。
This paper said that 'Here we assume the output of the previous layer has passed through a bounded activation function h, which ensures r ∈ [0, 1].
This can also seen in quant_dorefa.py. You can see 61st line torch.clamp(x, 0, 1)
So the dynamic range is [0, 1].

Hi. I have another opinion。
This paper said that 'Here we assume the output of the previous layer has passed through a bounded activation function h, which ensures r ∈ [0, 1].
This can also seen in quant_dorefa.py. You can see 61st line torch.clamp(x, 0, 1)
So the dynamic range is [0, 1].

The point is in the experiment it does not align with such an assumption. You could try to reproduce the accuracy drops issue I have mentioned with the same means.

Hi @HaFred , thanks for your comment, but currently I don't have GPU resource, so I'm afraid I can't check the experiments, feel free to make a PR if you can verify your idea.

Hi @HaFred , thanks for your comment, but currently I don't have GPU resource, so I'm afraid I can't check the experiments, feel free to make a PR if you can verify your idea.

No problem. I will make a PR when I am available. Cheers~


fix(users): Update assertion for duplicate username/email for newer v…

Update assertion for duplicate username/email for newer versions of mongodb.
When running a newer version of mongodb, users are reporting an issue when executing tests because the response from the mongo request has changed.  This change should support new/old versions of the test.
Resolves #1124

Thanks @rhutchison.
@ilanbiala I think the mongodb version comment is not needed there and actually toLowerCase is probably a common method we should employ for the rest of the tests.
The commit is already set correctly so let's just merge.

@lirantal I'm pretty sure the message has changed too because it says containEql instead of equal...so the comment is definitely necessary.

@ilanbiala I still don't see why we need the comments there. containEql is just a little more forgiving than eql, that's all. That test should work just fine with eql too I think, and if not because there is more test from the output then still don't see it as an issue.

@lirantal because we haven't really ever used containEql, we should specify why it's necessary in this case. Most people when writing new tests probably wouldn't pay attention if they switched MongoDB versions. The comment won't hurt, and it will probably help at least a few people if not us as well to remember why it is that way.

I think a comment there is just nit-picking. So we never used containEql before, big deal. Anyone can just open the shouldjs API docs and read what it does. If we had to comment every new method of some library that we used we'll have 50% ratio of code comments :)

@rhutchison can you clarify exactly which MongoDB versions changed this? Mongoose requires 2.1.6, so I'm not sure how you're using a newer version of MongoDB when we don't even have one in the package.json.

@ilanbiala I am running 3.2.0

so I'm not sure how you're using a newer version of MongoDB when we don't even have one in the package.json.

I went to https://www.mongodb.org/downloads#production and clicked download.

Oh the actual installation changed it, I thought you installed through npm a different version than Mongoose's dependency. In that case I think it's even more important to have the comment. Honestly it won't hurt. @rhutchison can you just add a comment in both places where you made the change saying which versions have which message and that the containEql catches both versions.

I'm struggling to see the importance of in-line comments for this change.
If someone cares that much, they can read the history, the original issue (#1124), or the manual for shouldjs.
You can merge the PR when you get off your high horse.

@rhutchison you may not see it because you'll remember why, but it's important for the thousands of users MEAN.js has and many people will not read the history, the issue, or the manual to know about the MongoDB versions. The comment isn't really for you as much as it is for the rest of the community.

@ilanbiala just let go this time :)
I don't think the in line comments there justify the case at hand.

@rhutchison there are also 2 tests that need to be fixed on the frontend I believe.
1) Users E2E Tests: Signup Validation Should report Email already exists
   Message:
     Expected '11000 duplicate key error collection: mean-test.users index: email already exists' to be 'Email already exists'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/Users/ilan/Desktop/mean/modules/users/tests/e2e/users.e2e.tests.js:275:62)
    at process._tickCallback (node.js:402:9)

  2) Users E2E Tests: Signup Validation Should report Username already exists
   Message:
     Expected '11000 duplicate key error collection: mean-test.users index: username already exists' to be 'Username already exists'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/Users/ilan/Desktop/mean/modules/users/tests/e2e/users.e2e.tests.js:294:62)
    at process._tickCallback (node.js:402:9)

@ilanbiala how is it possible that the build passes then?

These are for when MongoDB 3 is used, which I don't think we are using on Travis.

@rhutchison there are also 2 tests that need to be fixed on the frontend I believe.

I'll look it over - thanks.

@ilanbiala I think this has been addressed with #1246

Yep, closing.


Minvera MATH

why is MATH called minvera_math?
Minerva is a model that happened to use MATH, but it doesn't define the dataset. The current naming makes it hard to find.
https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/minerva_math

Hi! This was done because it uses the 4-shot prompt disclosed by Minerva and uses SymPy to check answer equivalence as described in the Minerva paper.
Will get the math_* variant added, to support both!

another question on this - is the exact-match metric now defined in v0.4.2 the same as the maj@1 metric reported in the llama papers?


define some "basic kubernetes" information to include in Keptn docs

Summary
We are finding that many of our new/potential users are fairly new to Kubernetes as well. Investigate whether we could provide a brief intro to some basic concepts and links to some getting-started-with-k8s courses and docs would be helpful.
We definitely DO NOT want to own any significant documenation about k8s basics, just something to help people getting started.
Provide a similar content to https://argo-cd.readthedocs.io/en/stable/understand_the_basics/
DoD

[ ] Analyze what other CNCF projects are doing.  For example, ArgoCD, Understand the Basics.
[ ] Propose what we should do for Keptn

This issue about providing Kubernetes basics for new users sounds really important!  I'm interested in exploring this and researching what other CNCF projects are doing. Can I get assigned to this issue?

Sure thing, thanks @UtkarshUmre !

Proposal: Simplifying Keptn Onboarding for New Users
Introduction:
Keptn's automation capabilities are powerful, but some new users face a hurdle: limited Kubernetes experience. This proposal outlines a plan to improve Keptn onboarding by providing essential Kubernetes basics and curated learning resources, without owning significant Kubernetes documentation.
Problem:
New users with limited Kubernetes knowledge struggle to grasp Keptn's full potential, leading to:

Slower adoption: Difficulty navigating Keptn's features due to underlying Kubernetes concepts.

Increased support requests: Users require additional assistance understanding Kubernetes fundamentals within Keptn.

Frustration and confusion: Learning curve hinders users from experiencing Keptn's benefits.

Proposed Solution:
1. Essential Kubernetes Concepts:

Concise overview: Briefly explain core Kubernetes concepts relevant to Keptn (e.g., Pods, Deployments, Services).

Focus on practical understanding: Avoid deep dives, prioritize practical application within Keptn.

Visual aids and diagrams: Enhance comprehension with clear visuals and simplified explanations.

2. Curated Learning Resources:

High-quality selection: Choose official CNCF projects (e.g., Kubernetes docs) and other trusted sources.

Categorized list: Organize resources by difficulty level and topic for easy navigation.

Short summaries: Briefly explain each resource's relevance and value within Keptn context.

Benefits:

Reduced onboarding friction: New users can grasp essential Kubernetes concepts quickly and easily.

Independent learning: Curated resources empower users to learn at their own pace without relying heavily on support.

Improved user satisfaction: Streamlined onboarding leads to a more positive Keptn experience.

Increased adoption: Simplified access attracts more users, expanding Keptn's reach and community.

Implementation:

Inventory existing resources: Identify relevant content within Keptn documentation and community forums.

Review and curate external resources: Select high-quality materials from CNCF projects and trusted sources.

Develop concise Kubernetes overview: Craft a clear and visual explanation of key concepts specific to Keptn.

Integrate resources and overview: Create a dedicated section or landing page within Keptn documentation.

Promote and gather feedback: Actively publicize the new resources and collect user feedback for continuous improvement.

Conclusion:
Investing in a user-friendly onboarding experience for new users with limited Kubernetes knowledge will significantly benefit Keptn's growth and adoption. By providing essential concepts and curated learning resources, we can empower users to unlock Keptn's full potential and contribute to a thriving community.

Proposal: Simplifying Keptn Onboarding for New Users
Introduction:
Keptn's automation capabilities are powerful, but some new users face a hurdle: limited Kubernetes experience. This proposal outlines a plan to improve Keptn onboarding by providing essential Kubernetes basics and curated learning resources, without owning significant Kubernetes documentation.
Problem:
New users with limited Kubernetes knowledge struggle to grasp Keptn's full potential, leading to:

Slower adoption: Difficulty navigating Keptn's features due to underlying Kubernetes concepts.

Increased support requests: Users require additional assistance understanding Kubernetes fundamentals within Keptn.

Frustration and confusion: Learning curve hinders users from experiencing Keptn's benefits.

Proposed Solution:
1. Essential Kubernetes Concepts:

Concise overview: Briefly explain core Kubernetes concepts relevant to Keptn (e.g., Pods, Deployments, Services).

Focus on practical understanding: Avoid deep dives, prioritize practical application within Keptn.

Visual aids and diagrams: Enhance comprehension with clear visuals and simplified explanations.

2. Curated Learning Resources:

High-quality selection: Choose official CNCF projects (e.g., Kubernetes docs) and other trusted sources.

Categorized list: Organize resources by difficulty level and topic for easy navigation.

Short summaries: Briefly explain each resource's relevance and value within Keptn context.

Benefits:

Reduced onboarding friction: New users can grasp essential Kubernetes concepts quickly and easily.

Independent learning: Curated resources empower users to learn at their own pace without relying heavily on support.

Improved user satisfaction: Streamlined onboarding leads to a more positive Keptn experience.

Increased adoption: Simplified access attracts more users, expanding Keptn's reach and community.

Implementation:

Inventory existing resources: Identify relevant content within Keptn documentation and community forums.

Review and curate external resources: Select high-quality materials from CNCF projects and trusted sources.

Develop concise Kubernetes overview: Craft a clear and visual explanation of key concepts specific to Keptn.

Integrate resources and overview: Create a dedicated section or landing page within Keptn documentation.

Promote and gather feedback: Actively publicize the new resources and collect user feedback for continuous improvement.

Conclusion:
Investing in a user-friendly onboarding experience for new users with limited Kubernetes knowledge will significantly benefit Keptn's growth and adoption. By providing essential concepts and curated learning resources, we can empower users to unlock Keptn's full potential and contribute to a thriving community.

@UtkarshUmre thank you for your suggestions! A few comments on my side that are probably not highlighted enough in the ticket:

the added documentation should not require hard effort to keep it up to date: this implies always preferring to link external resources instead of writing them ourselves. A good example would be  ArgoCD, Understand the Basics.

this content may be at the beginning of our core concepts page https://keptn.sh/latest/docs/core-concepts/ in a format like "Understanding Kubernetes"

Instead of explaining twice what is the correlation between keptn components and kubernetes primitives, we could link to the component section of the website, where this should/is already explained (https://keptn.sh/latest/docs/components/)

Very nice proposal, @UtkarshUmre !  For the sake of argument, let me offer a different argument.  Note, I am myself vacillating between the two opinions, not truly disagreeing with you.
Our general policy is to avoid documenting software that is not our own.  If we add this to our documenation, we have to maintain it, which means constantly reviewing it and updating it as Kubernetes evolves.  Is this effort warranted?
Your #2, a curated list of resources, may be enough to cover what you describe in #1 -- hopefully some good material is already available.  What might be warranted is a precise description of how the Keptn components relate to the Kubernetes objects and concepts.  We might put this into the "Components" section.  Perhaps add a "Keptn and Kubernetes" page to that section and then augment the descriptions of the different components with specific information about how that component integrates with Kubernetes, like what gets a new pod, perhaps pull in the informaton about namespaces that is currently in https://keptn.sh/latest/docs/installation/configuration/namespace/?h=namespace ...
What do you think?  Shall we tag this issue for refinement this week and discuss it?  Will you be attending this week's Community Meeting on Wednesday?

I think we can close this issue since there's no activity and we have this page in the contribution guide that already has the kubernetes fundamentals on it: https://keptn.sh/stable/docs/contribute/general/technologies/

The team decided to create a documentation issue to add and/or exchange some links on the technologies page and add a new "for beginners" section at the top with beginner resources.
the resources should include beginner material for k8s and containers.

The result of the research done here will be implemented in https://github.com/keptn/lifecycle-toolkit/issues/3097.


Warn on redirects

301 into 404 doesn't give me an error anymore.
This is the URL in question to give you an example: http://deckmaster.info/images/cards/BNG/-66703-hr.jpg
I'm using the GitHub Action of lychee.
With lychee v0.10.0 I got an error on that URL.
With v0.10.1 the same URL reports a success.
v0.10.0 /<EMAIL_ADDRESS>(I used to extract the links from the xml with XMLStarlet first, and wrote them to a text file)
--no-progress --require-https -- extracted_urls_from_original_file.txt

v0.10.1 /<EMAIL_ADDRESS>--no-progress --require-https --cache --max-cache-age 1h -- original_file.xml

The URL and its status codes can be inspected e.g. here, or in more detail on this site.

Should redirects be listed as warnings or hints to make users aware and give them the chance to adapt to the source directly? When moved permanently, there is a chance that the original link will break at one point, too.

Note: https://httpstat.us/ is a cool page that can help with easily checking certain status codes.

Not sure I understand. I think it works as expected for me?
❯❯❯ lychee --version                                                                                ✘
lychee 0.10.1

❯❯❯ echo 'http://deckmaster.info/images/cards/BNG/-66703-hr.jpg' | lychee -
Issues found in 1 input. Find details below.

[stdin]:
✗ [404] http://deckmaster.info/images/cards/BNG/-66703-hr.jpg | Network error: Not Found

🔍 1 Total ✅ 0 OK 🚫 1 Error (HTTP:1)

Testing it directly with HTTPS also works:
❯❯❯ echo 'https://deckmaster.info/images/cards/BNG/-66703-hr.jpg' | lychee -                                                                                ✘
Issues found in 1 input. Find details below.

[stdin]:
✗ [404] https://deckmaster.info/images/cards/BNG/-66703-hr.jpg | Network error: Not Found

🔍 1 Total ✅ 0 OK 🚫 1 Error (HTTP:1)

So maybe I'm missing something? Could it be --cache?

I think there was some problem on my side.

However, maybe this is more of a request then, that 301 should not be silently followed and report the status of the redirected to, new target. Those will most likely be successful and nobody realizes that there was a redirect in place which can lead to issues in the future.

Should redirects be listed as warnings or hints to make users aware and give them the chance to adapt to the source directly? When moved permanently, there is a chance that the original link will break at one point, too.

Links that got moved and have temporary redirects in place might be removed at one point. But when that happens, there is no way to look up the new location that the 301 was reporting before.
Is it possible to highlight such redirects to give maintainers the chance to fix those ahead of time and don't run into such issues?
Even reporting them as errors is probably fine and better than following the 301 and say everything works?

Even reporting them as errors is probably fine and better than following the 301 and say everything works?

Related: https://github.com/lycheeverse/lychee-action/issues/164
If we fix this we have a workaround for this case: setting --max-redirects to 0. In that case 301s would result in an error.

https://github.com/lycheeverse/lychee/pull/987 fixes --max-redirects behavior.
For example
 echo 'https://httpstat.us/301' | lychee --max-redirects 0 -

fails now as expected (with a build master, not with a released version yet).
Same for your URL.
echo 'http://deckmaster.info/images/cards/BNG/-66703-hr.jpg' | lychee --max-redirects 0 -

Can you double-check?

@tooomm, I assume this is fixed now and the fix got released with lychee 0.12.0. If this is not the case, feel free to reopen.


Admin role users in AWS unable to create budgets

Describe the bug
Clecio on the NRM Observability Metrics team is reporting unable to budgets in AWS
Additional context

See channel aws-tenant-requests for details (post on Aug 10)

Definition of done

Admin and Dev (per tech team review and discussion) to be able to create budgets in AWS

This was only reproducible through the AWS Cosole's UI, Budgets could be created through the CLI and through GitHub Actions/ Terraform. I let Clecio know, and will follow up with Ryan J on the UI bug.


Add Inputs examples

Subject
Inputs
Description
Add Inputs
Are you willing to participate in the development of this feature and create a pull request?
None

You may also refer to https://tailwindui.com/components/application-ui/forms/input-groups when making these examples.

I'll do it!

I'll do it!

This issue was closed.


Remove pedantic clippy lints from CI

They cause too much trouble with breaking CI.

@bors r+

:pushpin: Commit ea5f70d110f359e2076e72b6ab40c5855cc903a7 has been approved by Amanieu

:hourglass: Testing commit ea5f70d110f359e2076e72b6ab40c5855cc903a7 with merge 8ac672189e8ab5d5a1ebb39cd40028e17ffb3ebe...

:sunny: Test successful - checks-actions
Approved by: Amanieu
Pushing 8ac672189e8ab5d5a1ebb39cd40028e17ffb3ebe to master...


Enable Travis builds and add badge to README

Travis builds work, and badge has been added. However, the tests aren't run because of the Credentials.plist issue. Opening a new issue on that, and closing this one.


add testMode into IngestionParams

Summary
Add TestMode into Ingestion Param.
Usage: when the test mode is set to be true, the current ingestion will be persisted into {entity}_test and {relationship}_test table for data validation purposes.
More details see Model 2.0 Migration Validation
Testing Done
Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 68.06%. Comparing base (e3ea189) to head (99d8993).

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #402   +/-   ##
=========================================
  Coverage     68.06%   68.06%
  Complexity     1450     1450
=========================================
  Files           137      137
  Lines          5665     5665
  Branches        621      621
=========================================
  Hits           3856     3856
  Misses         1554     1554
  Partials        255      255

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Mazumder - Search CARD db for antibiotic resistance genes

Use CARD db to find antibiotic resistance genes for ARGOS orgs

The following google sheets documents encompass resistance gene data available from the CARD database. The sheet labeled "CARD Resistance Prevalence" reports genes that are considered to be antibiotic resistance determinants and the corresponding resistance drug class for a given pathogen. The Resistance Gene Identifier tool uses resistance prevalence data to predict the development of AMR in viral and bacterial pathogens based on nucleotide and amino acid sequence similarities. The sheet labeled "CARD Putative AMR Resistomes" contains the RGI tool analysis results for specific NCBI nucleotide accessions.
CARD Resistance Prevalence:
https://docs.google.com/spreadsheets/d/1ECIVS4Doom376Gn9u7Pvc6g9-N6067h1vwG-cyUWnHs/edit?usp=sharing
CARD Putative AMR Resistomes:
https://docs.google.com/spreadsheets/d/1OasikLEvQ2R3mNWMTf2FnEK-vUW9PTN4lVh_z2UdDV0/edit?usp=sharing


[Feature details] Big cleanup of feature details / bottom sheet impl

This includes several changes organized by commits, including a large rename commit, and a refactor and simplification of bottom sheet metrics. Sorry it's a big one, please feel free to review commits separately if it's useful; I've tried to make each commit clear and meaningful.
Fixes #463, #464.

@dturner gentle ping to PTAL once you're back. Thanks in advance!

Thanks for the nudge will try to review this today or tomorrow.

@dturner Thanks for the review! PTAL for reapproval?


Adding missing file ncdap_test/manyurls.h

re: Github Issue: https://github.com/Unidata/netcdf-c/issues/2085
The file ncdap_test/manyurls.h was not added to the repository.
It is controlled by the --enable-dap-long-tests option.
Solution: add the file.

Is nothing building in CI that should have caught that?

This is controlled by --enable-dap-long-tests, which is one of a number of
options unlikely to be used, and so not tested routinely. There are a number of others,
Perhaps we should have  a non-matrix'ized github action that is specifically to test
some of these outlier options.

BTW, the reason we don't add a row to the matrix for these is that the size of the matrix
is limited and we would soon exceed the allowed size for matrices.

I've merged this in to #2093 and am expanding the github actions to make sure that adding the missing file doesn't result in new/different failures.

@DennisHeimbigner I'm seeing failures when enabling enable-dap-long-tests.  I'll start debugging these, but you can see the details over at #2093


Zeros for area_swept and infinities for num_cpua

There are 22 records for survey GSL-N with area_swept = 0 (and num_cpua = Inf).
How can swept area be zero? Is this a bug (e.g., in matching swept areas onto catch records) or signature of invalid trawl(s) that is not included in current routines? (I am not familiar with the details of how GSL-N surveys are recorded.)
# Load data
load(url("https://github.com/AquaAuma/FishGlob_data/blob/main/outputs/Compiled_data/FishGlob_public_clean.RData?raw=true"))
# Identify area_swept = 0 records
data[which(data$area_swept == 0), c("survey", "haul_id", "area_swept", "accepted_name", "num", "num_cpue", "num_cpua")]

missing swept area estimates even after model fits:

@zoekitchel do you think you could check the GSL survey and probably replace 0 swept area values by NAs, or figure out why we have 0 values? Thanks

@jepa , this would be one of the North American fixes to implement. It would mean replacing the GSL-S and GSL-N (Gulf of St. Lawrence South and North) swept area=0 values with NA early enough in the data processing so that num_cpua also gets calculated as NA. Thank you!

Done, see my last push(es).  https://github.com/AquaAuma/FishGlob_data/commit/d688f21da7bb0adf794b65b9a0681bda4513f68c, da2d85ed2f4fd5858c6d2c807c83ec9d65478d4e and ebd7193c425600fa7b9c7932198d784abee32e07


IE11 breaks due to new CustomEvent()

This is on version 7.0.0. Results in SCRIPT445: Object doesn't support this action

Please look at previously closed issues before opening new ones. This has been answered numerous times: see #585, #536,  #578

My bad, I'd only searched open issues.

No worries 👍


Feature Suggestion: Filter Scanning by Quality (stars)

A nice feature would be the option to configure the scanner to only scan weapons/artifacts by star rating (example, minimum 4 stars).

This feature has been added in the v1.1.1 prerelease.


Replication of PostgreSQL db fails with malformed array literal error...

I am getting the following error when trying to replicate two PostgreSQL databases, This happens on every attempt to update the target db from the source db.  has anyone seen this before?  Looking here and elsewhere it appears to be related to copying an array but using the wrong brackets ('[...]' instead of '{...}').  Sync fails with the same error as the second one below; scan does not fail (but not surprising as it is not attempting to write to the target db):
"insert failed with ActiveRecord::JDBCError: org.postgresql.util.PSQLException: ERROR: malformed array literal: "---"
Another instance (seems to support the bracketing issue I read about online):
"insert failed with ActiveRecord::JDBCError: org.postgresql.util.PSQLException: ERROR: malformed array literal: "--- SE
Detail: Array value must start with "{" or dimension information.
Position: 361: insert... "
Please help, I love this tool and want to use it, but this is a deal-breaker currently.

This was a bug. Postgres specific fields like arrays and JSON were not written correctly.
I released version 2.0.2 with a bug fix. Please try with that version.
I am tentatively closing this issue.

Sorry to comment on this closed ticket, but does this also work for hstore columns? I was just working on a local fork of this to correct the behavior of both hstore and array values when copying them. You have specifically mentioned arrays, but I just wanted to check before I made a PR for the hstore changes.


There are no ammunition shown in ammunition tab. Just Enought items doesnt show its recipe.

Do you use Java 8?
Yes
What version of Minecraft do you use?
1.12.1
What version of MinecraftForge do you use?
forge-1.12.1-<IP_ADDRESS>4
What version of SanLib do you use?
SanLib-1.12-1.2.0
What version of Turret Mod Rebirth do you use?
TurretMod-1.12-4.0.0-beta.2
What is the issue / bug you have encountered? Give us a short description.
Ammunition isnt craftable. Ammo tab is empty on that turret crafting table. Just Enought Items doesnt show the recipe
How can it be reproduced?
Place down the turret crafting table and go to ammunition section. Or press Recipe on JEI.
Please post your log file on https://gist.github.com and provide the link:
Latest https://www.mediafire.com/file/yzy3yzpj5lb9428/latest.log
Latest FML https://www.mediafire.com/file/liwiaw5pexuzpei/fml-client-latest.log

12:59:07] [main/ERROR] [sapturretmod/sapturretmod]: Parsing error loading electrolyte generator recipe from /assets/sapturretmod/recipes_sapturretmod/assembly/group_upg.json
com.google.gson.JsonSyntaxException: Json cannot be null
	at de.sanandrew.mods.sanlib.lib.util.JsonUtils.getIntVal(JsonUtils.java:83) ~[JsonUtils.class:?]
	at de.sanandrew.mods.turretmod.registry.assembly.TurretAssemblyRecipes.registerJsonRecipes(TurretAssemblyRecipes.java:102) ~[TurretAssemblyRecipes.class:?]

hmmm... I'll look into it.

Just Enought items mod shows recipes for turrets, assembly tables etc, but not the ammunition. Ammunition can be obtained only from creative and thats it. And why i cant input stuff from top using hoppers into generator?

It seems to load the turrets first, then errors trying to load the ammo by invalidly loading a group as a recipe.
Also the generator only accepts items from the sides of the bottom part.

Okay, could you try out this release?
https://github.com/SanAndreasP/TurretModRebirth/releases/download/4.0.0-beta.2.1/TurretMod-1.12-4.0.0-beta.2.1.jar
Given I don't run into this issue myself, you'd have to test it. Please let me know if this fixes it, so I can release it to the public.

Yes. This fixed it. I dont have much time so i didnt checked if they do have crafting recipe, but they do appear in ammunition tab, so i think it will work. Thanks.

nice


Multiple set commands are sent in single packet to server - no pipeline in code - auto pipeline is also not enabled.

Please find below the code that sends 100 set commands in a for loop. For some execution each set command is sent as individual packet to redis server and each response ("ok") is received as individual packet.

On some executions multiple set commands are sent in one packet to redis server and the responses are grouped into one packet.

const Redis = require('ioredis');
let endpoints = new Array(1)
endpoints[0] = { host: "ip", port: 26379};
const redis = new Redis({
sentinels: endpoints,
password: '',
name: 'mymaster',
role: 'master',
});
console.log(redis)
redis.on('connect', function(err) {
if(err) {
console.log('error: ' + JSON.stringify(err));
return;
}
console.log('[Redis] up and running!');
return;
});
for (let i = 0; i < 100; i++) {
redis.set('hello' + i, 'test' + i, 'EX', 60, function(err,result) {
console.log('set command error response: ' + err)
console.log('set command result: ' + result)
});
}
When the volume of requests is high to the node.js microservice, less packets with more number of set commands are sent to redis sentinel host. The microservice memory keeps growing and crashes with OOM.
Trying to understand how the grouping of commands happens when pipeline is not used and also the auto pipeline is not enabled.
can this grouping behavior cause memory issues as observed in my case? Is there a way to force one command per request?
ioredis version - 4.11.2
Please advice. Any pointers will be of great help. Thank you

@TysonAndre You have mentioned about protocol pipeline (RESP) - https://github.com/redis/ioredis/issues/1377 Is this related?

@luin request you to look into this issue and share your thoughts.


test: fix test_system_backup_and_restore_volume_with_backingimage

Which issue(s) this PR fixes:

Issue https://github.com/longhorn/longhorn/issues/10052
What this PR does / why we need it:
fix test_system_backup_and_restore_volume_with_backingimage
Special notes for your reviewer:
Additional documentation or context

Test results: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/7964/


Use matcher branch for testing

Description
Testing matchers for github actions
Issues Resolved
List any existing issues this PR resolves
Check List

[ ] All tests pass. See TESTING.md for details.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable.

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Warning

:warning:
This Pull Request is probably missing tests.

  Generated by :no_entry_sign: Danger

Tests completed.


Route Groups

The ability to group a set of related routes together and then call the group... this will be useful for building a drop down menu and such....

completed


Check for existence of layers in cache before returning cached base image

In this PR:
Implementation of a proposed solution in #3733’s discussions, to check for presence of layer files in cache in addition to the manifest json before returning the cached base image(s). This would also address the unauthorized error described in #2220 and also reported in #2007’s comments.
Details:

Adds allLayersCached() method to Cache and CacheStorageReader that checks for existence of cached layers for all layers described in the image manifest.
Updates PullBaseImageStep.getCachedBaseImages() to invoke this check, and returns no cached images if layers are missing.

Two approaches for layer check:

Changes up to 755dcfc is a simpler check which verifies that the corresponding layer directory exists in the cache (but not that its contents are valid)
b231d79 checks layers by making a call to retrieve the layer, though the retrieved layer is unused and the later ObtainBaseImageLayerStep would retrieve it again

Open to suggestions on what the better approach might be here - the second option does redundant work, but is a more rigorous validation.

@emmileaf Took a first look at the changes! To make sure that I'm understanding the two approaches correctly:

Approach 1 checks for the existence of {cacheDirectory}/layers/{layerDigest} where the layer digests are read from the manifest.
Approach 2 actually tries to grab the cached layer given the digests and if this act results in some sort of an IO issue then the layer is deemed to not be present in the cached?

Both approaches are valid but I'm a little concerned about the performance overhead approach 2 might add especially since the change will result in the cached layer being retrieved twice. Approach 1 might be a sufficient check. In terms of readability it also makes it more explicit that we are checking for the presence of the layer. Catching exceptions also works but there is usually a risk involved with it if we're not as specific as possible since methods can throw an exception for a number of reasons and it is usually recommended to let exceptions propagate to the caller or be handled by surrounding frameworks (testing frameworks, for example). Lmk what you think.

@mpeddada1 Thanks for the review! Your understanding of the two approaches is spot on.
I also agree with the concerns here around approach 2’s redundant work overhead and relying on exceptions. The downside of approach 1's check I was trying to work around is that, CacheStorageWriter.writeCompressed() creates the {cacheDirectory}/layers/{layerDigest} directory first, so it is possible for this check to succeed but the layer partially written (or not valid for retrieval in some other way). Not sure how unlikely that race condition would be though.
Maybe there is some way to make this check more rigorous without fully retrieving the layer? I'll revert the last commit to switch back to approach 1, and see if I can make any improvements from there.

Just realized this statement isn't true and I had misunderstood this cache write logic earlier - it looks like a temp folder is created for writing the layer first, and then the contents get moved into {cacheDirectory}/layers/{layerDigest} (ref: #879 (comment)). This addresses the concern with partial layer directories, and I'll go ahead with the switch to approach 1 in this PR. Apologies for the confusion!

Ah This changes the write to only produce a layer directory at the intended location in full. This way, cache reads never see an incomplete layer directory. That is reassuring, thank you for verifying this!

@mpeddada1 @elefeint @chanseokoh Thank you so much for all the help here with this PR!

Thank you! Will update changelog and merge.

Hi @emmileaf @elefeint - thank you for this fix!
We'd love to get this available for quite a few of our builds that fail from this, so I was wondering if there is a release with this fix planned soon?

Hi @herder - we currently have some work in progress on compatibility in our release build environment, but can plan for a release later this week including this change.

@herder jib-core 0.23.0, jib-maven-plugin 3.3.1, and jib-gradle-plugin 3.3.1 have been released with this fix.


Can we add MSMQ server as supported data sources for on-premise data gateway?

Is it possible to use Azure LogicApp to connect to on-prem MSMQ server using on-prem data gateway?
If so then why is it not listed on the supported data sources list while Microsoft support IBM MQ series etc?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7892e678-a98f-73fa-7c54-1a8a921f35c7
Version Independent ID: 064c9db9-5e18-7266-1740-ada24a2086a1
Content: Access data sources on premises - Azure Logic Apps
Content Source: articles/logic-apps/logic-apps-gateway-connection.md
Service: logic-apps
GitHub Login: @ecfan
Microsoft Alias: estfan

Hi @Anand-Moghe - Thank you for your feedback! We will review and update as appropriate.

Hi @Anand-Moghe,
Thanks for your question! Based on the Connector reference list for Azure Logic Apps, no connector is available for Microsoft Message Queue (MS MQ).
If you'd like to submit a request to the product team, please post your request at the Azure Logic Apps product feedback page.

#assign:@ecfan