Legion: (runtime) leak of index partitions / index expressions

The following Legion program is adapted from the Legate code described in https://github.com/nv-legate/legate.core/issues/548, where I'm developing a sparse machine learning system. The Legion program is making a similar sequence of calls that the Legate program is making under the hood.
#include "legion.h"

using namespace Legion;

enum TASK_IDS {
  TID_TOP_LEVEL,
};

enum FIELD_IDS {
  FID_1,
};

void top_level_task(const Task* task, const std::vector<PhysicalRegion>& regions, Context ctx, Runtime* runtime) {
  auto n =<PHONE_NUMBER>;
  auto ispace = runtime->create_index_space(ctx, Rect<1>{0, n - 1});
  auto fspace = runtime->create_field_space(ctx);
  {
    auto alloc = runtime->create_field_allocator(ctx, fspace);
    alloc.allocate_field(sizeof(double), FID_1);
  }
  auto region = runtime->create_logical_region(ctx, ispace, fspace);
  runtime->fill_field(ctx, region, region, FID_1, double(0));

  int64_t i = 0;
  int64_t batchsize = 100;
  while (true) {
    if (i % 100 == 0) {
      runtime->issue_execution_fence(ctx).wait();
      std::cout << "i = " << i << std::endl;
    }

    auto lo = i * batchsize;
    auto hi = (i + 1) * batchsize;
    auto ipart = runtime->create_index_partition(ctx, ispace, Rect<1>{0, 0}, {
        {DomainPoint(Point<1>{0}), Domain({Rect<1>{lo, hi - 1}})}
    });
    auto lpart = runtime->get_logical_partition(ctx, region, ipart);
    auto subreg = runtime->get_logical_subregion_by_color(ctx, lpart, DomainPoint(Point<1>(0)));
    auto preg = runtime->map_region(ctx, RegionRequirement(
        subreg, LEGION_READ_WRITE, LEGION_EXCLUSIVE, region).add_field(FID_1));
    preg.wait_until_valid();
    runtime->unmap_region(ctx, preg);
    runtime->destroy_index_partition(ctx, ipart);
    i++;
  }
}

int main(int argc, char** argv) {
  Runtime::set_top_level_task_id(TID_TOP_LEVEL);
  {
    TaskVariantRegistrar registrar(TID_TOP_LEVEL, "top_level");
    registrar.add_constraint(ProcessorConstraint(Processor::LOC_PROC));
    registrar.set_replicable();
    Runtime::preregister_task_variant<top_level_task>(registrar, "top_level");
  }
  return Runtime::start(argc, argv);
}

When compiled against control_replication with -DLEGION_GC and ran with -ll:csize 4000 -dm:exact_region, the following data is reported by legion_gc.py -l after 300 and 600 iterations, respectively. To get this printout without a legion_gc.py error, the script needs to be modified so that report_references is a noop.
LEAK SUMMARY
  LEAKED FUTURES: 1
  Leaked Future Maps: 0
  LEAKED CONSTRAINTS: 3
  LEAKED MANAGERS: 307
  Pinned Managers: 0
  LEAKED VIEWS: 307
  LEAKED EQUIVALENCE SETS: 1
  LEAKED INDEX SPACES: 616
  LEAKED INDEX PARTITIONS: 307
  LEAKED INDEX EXPRESSIONS: 94507
  LEAKED FIELD SPACES: 1
  LEAKED REGIONS: 307
  LEAKED PARTITIONS: 307

LEAK SUMMARY
  LEAKED FUTURES: 1
  Leaked Future Maps: 0
  LEAKED CONSTRAINTS: 3
  LEAKED MANAGERS: 603
  Pinned Managers: 0
  LEAKED VIEWS: 603
  LEAKED EQUIVALENCE SETS: 1
  LEAKED INDEX SPACES: 1208
  LEAKED INDEX PARTITIONS: 603
  LEAKED INDEX EXPRESSIONS: 364188
  LEAKED FIELD SPACES: 1
  LEAKED REGIONS: 603
  LEAKED PARTITIONS: 603

As far as I can tell, the application is freeing runtime data structures as eagerly as possible -- the IndexPartition created at each iteration is deleted, and the logical subregion and logical partition are not data structures that can be destroyed with the Legion API. The batchsize here is small enough that Legate (rightfully so) does not want to create a single partition and index into it as the number of subregions would be too large. Is there something the application should be doing here, or is the runtime not collecting things that it should be?

Things I've tried (suggestions from @magnatelee)

Running under a tight memory limit so that batches cause GC to get invoked and collect instances
Running without the default mapper's cache enabled so physical instance references don't accumulate

Something seems a bit suspect, because even running this for 10 iterations (and killing it before clean exit) generates a legion gc log that legion_gc.py explodes on:
Traceback (most recent call last):
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 1335, in <module>
    main()
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 1331, in main
    state.check_for_leaks(assert_on_error, verbose)
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 1203, in check_for_leaks
    if not index_space.check_for_leaks(assert_on_error, verbose):
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 398, in check_for_leaks
    self.report_references(printer, RESOURCE_REF_KIND, verbose)
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 512, in report_references
    src.report_references(printer, kind, verbose)
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 512, in report_references
    src.report_references(printer, kind, verbose)
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 512, in report_references
    src.report_references(printer, kind, verbose)
  [Previous line repeated 988 more times]
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 510, in report_references
    printer.println('Nested Resource '+repr(src)+' (Refs='+str(refs)+')')
  File "/Users/rohany/Documents/research/legion/examples/slicing/../../tools/legion_gc.py", line 668, in __repr__
    return 'Partition '+str(self.did)+' (Node='+str(self.node)+') ('+str(self.index_partition)+','+str(self.field_space)+','+str(self.tree_id)+')'
RecursionError: maximum recursion depth exceeded while getting the str of an object

This isn't that surprising to me. The managers are still alive (notice their counts double too) and they are holding references to their index sub-spaces to keep them alive. You're not even close to filling up the system memory with these instances so there is no garbage collection to trigger manager deletion. Try running with -ll:csize 1 and see what happens once you hit that 1MB footprint. The instances should start being collected, which will clean up the managers, and thereby release references on the index sub-spaces. If that occurs then we can discuss how to more eagerly collect instances. (Note this experiment still requires the default mapper not to be holding any references to the managers either.) If the manager count stops doubling but the index space count keeps doubling then I'll agree there is an issue with the runtime; as long as they are correlated though then everything is currently still working as intended.

Something seems a bit suspect, because even running this for 10 iterations (and killing it before clean exit) generates a legion gc log that legion_gc.py explodes on

The hand-over-hand reference counting that Legion does will sometimes temporarily introduce cycles into the reference graph which are resolved when the new references are added and the old ones removed. If you happen to kill Legion in the middle of it's execution when once of these cycles is in existence, then Legion GC will go into an infinite loop around the cycle (unless you run with the -c option to check for cycles), which will eventually overflow the Python interpreter's stack.

The managers are still alive (notice their counts double too) and they are holding references to their index sub-spaces to keep them alive. You're not even close to filling up the system memory with these instances so there is no garbage collection to trigger manager deletion. Try running with -ll:csize 1 and see what happens once you hit that 1MB footprint.

I tried this -- increased the batch size to 10000 and ran with -ll:csize 1 and still saw the same pattern of growth in legion_gc.py of index spaces, but not managers.
LEAK SUMMARY
  LEAKED FUTURES: 189
  LEAKD FUTURE MAPS: 87
  LEAKED CONSTRAINTS: 3
  LEAKED MANAGERS: 26
  Pinned Managers: 0
  LEAKED VIEWS: 27
  LEAKED EQUIVALENCE SETS: 2
  LEAKED INDEX SPACES: 602
  LEAKED INDEX PARTITIONS: 300
  LEAKED INDEX EXPRESSIONS: 23305
  LEAKED FIELD SPACES: 1
  LEAKED REGIONS: 213
  LEAKED PARTITIONS: 300

LEAK SUMMARY
  LEAKED FUTURES: 194
  LEAKD FUTURE MAPS: 92
  LEAKED CONSTRAINTS: 3
  LEAKED MANAGERS: 26
  Pinned Managers: 0
  LEAKED VIEWS: 27
  LEAKED EQUIVALENCE SETS: 2
  LEAKED INDEX SPACES: 1002
  LEAKED INDEX PARTITIONS: 500
  LEAKED INDEX EXPRESSIONS: 84305
  LEAKED FIELD SPACES: 1
  LEAKED REGIONS: 408
  LEAKED PARTITIONS: 500

(Note this experiment still requires the default mapper not to be holding any references to the managers either.)

The main place I know of this is the cache of mappings the default mapper maintains, which I disabled.

The hand-over-hand reference counting that Legion does will sometimes temporarily introduce cycles into the reference graph which are resolved when the new references are added and the old ones removed. If you happen to kill Legion in the middle of it's execution when once of these cycles is in existence, then Legion GC will go into an infinite loop around the cycle (unless you run with the -c option to check for cycles), which will eventually overflow the Python interpreter's stack.

Ok, that makes sense then. It's annoying because I can't see what's holding the references, and legion cleans eveyrthing up successfully on a normal exit, so I have a kill it in the middle.

Provide a diff for the change that you made to the default mapper.

➜  slicing git:(control_replication) ✗ cat diff.txt                                                                                (base) 17:06:56
diff --git a/runtime/mappers/default_mapper.cc b/runtime/mappers/default_mapper.cc
index 624b5b374a..32ac8da69a 100644
--- a/runtime/mappers/default_mapper.cc
+++ b/runtime/mappers/default_mapper.cc
@@ -1692,7 +1692,7 @@ namespace Legion {
     //--------------------------------------------------------------------------
     {
       // Always cache task result.
-      return DEFAULT_CACHE_POLICY_ENABLE;
+      return DEFAULT_CACHE_POLICY_DISABLE;
     }

     //--------------------------------------------------------------------------
@@ -2394,6 +2394,7 @@ namespace Legion {
                                 bool meets_fill_constraints, bool reduction)
     //--------------------------------------------------------------------------
     {
+      return LEGION_GC_FIRST_PRIORITY;
       // Pretty simple: keep our big instances around
       // as long as possible, delete reduction instances
       // as soon as possible, otherwise we are ambivalent
       ```

Also, to get the GC behavior, the application must be modified slightly -- i've edited it in the post now.

I've dug into this program and the runtime is still behaving as expected. The reason the index spaces and index partitions are not being deleted is because they are still necessary to represent valid data of the root region. Consider the first iteration of the main loop which touches the points [0,99]. It then maps them, overwrites those subpoints with a fill, and then deletes the partition. However, that doesn't mean that the data in [0,99] are suddenly cleared. (Legion doesn't do value testing so it can't see that the fill to the subregion is the same value as the fill to the root at the beginning of the program; it thinks there is "dirty" fill data in [0,99].) If the program were to come along and ask to map the top-level region, Legion is still responsible for reporting the most recent updates to those points, which was done in that particular partition and subregion, so Legion keeps references to them around until they are no longer needed. It is precisely these references that are keeping the logical partition, it's logical subregion, and the corresponding index partition/subspace alive even after they have been deleted. You can test this hypothesis by changing the fill inside the loop to fill the root region again instead of just filling the subregion. Alternatively if you don't want to fill the whole root, you can create a new partition that aliases with the deleted one and do the fill there; it will have the same effect. If you do either of those, then the index spaces will still hang around until the managers are deleted by the garbage collector, but after that everything else gets cleaned up and we hit a steady state of memory usage.
You also had an additional leak of index spaces from using a deprecated index partitioning API. The color spaces for every single partition were leaking because the runtime was having to make them implicitly for the deprecated API and then the application wasn't cleaning them up.
With fixes for the two issues mentioned above, I can run the following program like this: -ll:csize 1 -dm:exact_region and see no memory growth.
#include "legion.h"

using namespace Legion;

enum TASK_IDS {
  TID_TOP_LEVEL,
};

enum FIELD_IDS {
  FID_1,
};

void top_level_task(const Task* task, const std::vector<PhysicalRegion>& regions, Context ctx, Runtime* runtime) {
  auto n =<PHONE_NUMBER>;
  auto ispace = runtime->create_index_space(ctx, Rect<1>{0, n - 1});
  auto fspace = runtime->create_field_space(ctx);
  {
    auto alloc = runtime->create_field_allocator(ctx, fspace);
    alloc.allocate_field(sizeof(double), FID_1);
  }
  auto region = runtime->create_logical_region(ctx, ispace, fspace);
  runtime->fill_field(ctx, region, region, FID_1, double(0));

  int64_t i = 0;
  int64_t batchsize = 10000;
  while (true) {
    if (i % 100 == 0) {
      runtime->issue_execution_fence(ctx).wait();
      std::cout << "i = " << i << std::endl;
    }

    auto lo = i * batchsize;
    auto hi = (i + 1) * batchsize;
    auto ipart = runtime->create_index_partition(ctx, ispace, Rect<1>{0, 0}, {
        {DomainPoint(Point<1>{0}), Domain({Rect<1>{lo, hi - 1}})}
    });
    auto lpart = runtime->get_logical_partition(ctx, region, ipart);
    auto subreg = runtime->get_logical_subregion_by_color(ctx, lpart, DomainPoint(Point<1>(0)));
    auto preg = runtime->map_region(ctx, RegionRequirement(
        subreg, LEGION_READ_ONLY, LEGION_EXCLUSIVE, region).add_field(FID_1));
    preg.wait_until_valid();
    runtime->unmap_region(ctx, preg);
    runtime->fill_field(ctx, region, region, FID_1, double(0));
    auto color_space = runtime->get_index_partition_color_space_name(ctx, ipart);
    runtime->destroy_index_partition(ctx, ipart);
    runtime->destroy_index_space(ctx, color_space);
    i++;
  }
}

int main(int argc, char** argv) {
  Runtime::set_top_level_task_id(TID_TOP_LEVEL);
  {
    TaskVariantRegistrar registrar(TID_TOP_LEVEL, "top_level");
    registrar.add_constraint(ProcessorConstraint(Processor::LOC_PROC));
    registrar.set_replicable();
    Runtime::preregister_task_variant<top_level_task>(registrar, "top_level");
  }
  return Runtime::start(argc, argv);
}

Alternatively if you don't want to fill the whole root, you can create a new partition that aliases with the deleted one and do the fill there; it will have the same effect.

Can you elaborate here?
Ok, you've convinced me that the batches here (as written) are necessary for maintaining valid copies of the data. As expected, if I remove the fill on the subregion, then I will OOM, as the valid instances that are created from the inline mapping can't be collected (because they are the only valid instances). This would be the case even if the original region had been mapping to an instance and I was slicing out subregions of it into separate instances, unless I used untrack_valid_regions (however, this doesn't have the semantics that I think we'd want for Legate right now).
So the real question I have is how do I effectively loop through a region and slice out subregions (for a sequence of read-only operations) of it without blowing up any sort of internal Legion data structure? I'm working on a machine learning application which does exactly this -- it creates a large NumPy array of the full data (therefore having some valid instances across the machine), then loops over the array in batches, slicing out each batch to train the network on. The instances created by the slice should be mostly ephemeral, as they only have a lifetime of one iteration in the training. I guess that I could have a task that writes to the main partition of the full data at each iteration, but this sounds wasteful. I can't have the accumulation of these index expressions, as I've observed non-trivial slowdowns in the runtime as training goes on.
Does FlexFlow have a similar problem like this? How do you guys manage batches @lockshaw @jiazhihao?

You also had an additional leak of index spaces from using a deprecated index partitioning API. The color spaces for every single partition were leaking because the runtime was having to make them implicitly for the deprecated API and then the application wasn't cleaning them up.

Good to know, I have to check if Legate is using the old API then.

Your insights were helpful in helping me translate some of this into my Legate application, and I think the question now comes back to eager collections. I tested in my application doing a write to the equivalent of region in this application, and see that my per-iteration times grow until a memory limit is hit, and then there's a collection, the iteration time goes back to normal then slowly grows again.
Since I'm doing a write to region, deleting the created index partitions (and using the new API now, so not leaking color spaces anymore), the references to the index spaces and index expressions are the physical instances that aren't getting collected until memory pressure kicks in. As I mentioned before, that doesn't work for me here because the constraint isn't memory usage, but runtime cost of managing these extra data structures slowing things down.
So the question is how do we think eager collection of physical instances in this case might work? The recently proposed discard API is kind of related but not what we want. Legate at a high level knows when slices get dropped, so it theoretically could issue deletions, though there are heuristics around here. I don't know if such a thing should be mapper controlled, because I'm not sure the Legate mapper at least has the information to drive such a decision.
cc @magnatelee @manopapad if you have had any thoughts about this.

Can you elaborate here?

If you made a different partition that overlapped with [0,99], say [0,199], and wrote to that instead, that would invalidate the data in the [0,99] partition and allow the reclamation of the index space and index partition.

So the real question I have is how do I effectively loop through a region and slice out subregions (for a sequence of read-only operations) of it without blowing up any sort of internal Legion data structure?

I suspect it will involve a combination of the the discard feature that we discussed in the Legion meeting, which is a way for the application to promise that it doesn't need that data anymore which will ensure that instances are invalidated, and then a separate set of features in the mapper API for the mapper to request early deletion of the instances.

Does FlexFlow have a similar problem like this? How do you guys manage batches

The FlexFlow guys can correct me if I'm wrong, but from my observation of their code they don't do what your ML application is doing which is making one big region, and then slicing for each of the batches. Instead they make a single logical region for doing training on a batch, and they map multiple batches onto that same logical region across top-level training iterations. They can do that because they know they are doing training.

I suspect it will involve a combination of the the discard feature that we discussed in the Legion meeting, which is a way for the application to promise that it doesn't need that data anymore which will ensure that instances are invalidated,

I don't see how discard could be used, because it also invalidates that subset of the data in the parent region. The numpy array needs to be valid at all indices, so we can't just discard slices as we GC them in Python.

I tested in my application doing a write to the equivalent of region in this application, and see that my per-iteration times grow until a memory limit is hit, and then there's a collection, the iteration time goes back to normal then slowly grows again.

That's what I would expect with the current implementation as well.

So the question is how do we think eager collection of physical instances in this case might work?

Right, that is the part that is missing (assuming we have discard to invalidate instances).

I don't see how discard could be used, because it also invalidates that subset of the data in the parent region. The numpy array needs to be valid at all indices, so we can't just discard slices as we GC them in Python.

If that is the case, then it is not about invalidating and deleting instances, but "spilling" data from instances in a "constrained" memory to a less-constrained memory. Note though that this will keep around the index spaces and partitions, so your memory problem might still be an issue. I guess I'm interested in why the user felt compelled to write it like this, rather than do it like the FlexFlow people do where they map multiple batches onto the same array across iterations.

Note though that this will keep around the index spaces and partitions, so your memory problem might still be an issue.

This seems to be the issue for me, not the actual memory. The runtime overhead increasing I see seems to be related to the number of index expressions that are leaked.

I guess I'm interested in why the user felt compelled to write it like this

I think that was just the natural "numpy" way of writing the application. See here: https://github.com/elibol/sparse-autograd/blob/main/train-movielens.py#L59. The application loads up a large sparse matrix, randomly sorts the coordinates, and then selects slices from the matrix to use as the training set for that particular epoch. The application originally did an indirect gather where it generated a random set of indices and then selected those indices from the sparse matrix, but we moved to this approach because of the slowness of having indirect copies on the hotpath. That approach would actually avoid this problem I think, but I don't see another way of doing this.

The FlexFlow guys can correct me if I'm wrong, but from my observation of their code they don't do what your ML application is doing which is making one big region, and then slicing for each of the batches. Instead they make a single logical region for doing training on a batch, and they map multiple batches onto that same logical region across top-level training iterations. They can do that because they know they are doing training.

This was a good hint though. I think I can make progress on this by mapping the "data" array onto a numpy array, rather than a cunumeric array, and then attach slices of the numpy array into cunumeric arrays, avoiding this problem. An initial test seems to be working out for me, though it's kind of a hack IMO.

The runtime overhead increasing I see seems to be related to the number of index expressions/spaces/instances that are "leaked".

We should profile that. I'm betting the issue comes with the congruence testing. If you have a bunch of index spaces that are all the same size, it's really hard for Legion to test for congruence to deduplicate index space expressions right now. I know where the acceleration data structure for that lookup needs to go, but I would want to confirm that that is the cause of the problems before I implement it.

The application loads up a large sparse matrix, randomly sorts the coordinates, and then selects slices from the matrix to use as the training set for that particular epoch.

In general Legion partitions are much better when they correspond to something associated with the physical machine rather than something associated with the properties of the data, but we can try to make it work better when we have tons of partitions.

It doesn't show up that noticeably in profiles (at least to me) -- http://sapling.stanford.edu/~rohany/sparseml-no-mapper-rec-inst. All I can tell is that there's an increase in utility processor utilization near the end of this profile.

In general Legion partitions are much better when they correspond to something associated with the physical machine rather than something associated with the properties of the data, but we can try to make it work better when we have tons of partitions.

Yeah, I agree. However, that's not something a NumPy user is thinking about when slicing up their data.

All I can tell is that there's an increase in utility processor utilization near the end of this profile.

That profile still looks pretty limited on the Python processor to me. The runtime overhead is ticking up very gradually, but I'm not sure that is a meaningful signal. It could be but I wouldn't bother worrying about it until it starts to dominate performance at which point it would be much more obvious what is going on.

Yeah, I agree. However, that's not something a NumPy user is thinking about when slicing up their data.

I agree, I just think that it's Legate's job to figure out the partitions and map them on to Legion better. A little bit of bounded lazy evaluation and analysis of slices of the program would go a long way here.

That profile still looks pretty limited on the Python processor to me.

it's true, but there's nothing I can do about that for this application -- the workload is fixed.

It could be but I wouldn't bother worrying about it until it starts to dominate performance at which point it would be much more obvious what is going on.

The performance difference does become noticeable though (I guess I can't take a long enough profile to see it), but this is ~3 epochs. After 10-15 training epochs the epoch time becomes double the initial time.

The performance difference does become noticeable though (I guess I can't take a long enough profile to see it), but this is ~3 epochs. After 10-15 training epochs the epoch time becomes double the initial time.

Take a large profile and then prune out all but the last 20 seconds or so of execution time.


Added IPv6 support

In this commit I've added IPv6 support and done some minor clean-up of the socket & address stack.

Still need to write unit tests.

The travis-CI tests failed on clang.  The failure might be a flake.
Please add the approrpriate "override" directives on your virtual methods.  (see the travis-ci logs).
When you submit the above change for re-eval, CI will run again.  If the test still fails, I'll attempt to find out why.
Thank you for your patience!

@dennisjenkins75 There is a flaky problem with Client with handling response that I have already been discovering for 3 weeks (one problem with handling I have  fixed).

I've still got more tests to write anyway, so don't worry too much about it yet. I just had five minutes last night so I got the bare minimum together.

Ok.  Cool.  Just ping this issue with an explicit request for code review + merge when you are ready.  Thank you very much for your contribution!

Hi @dennisjenkins75,
I've written all the tests I can think of for now, please review and merge.
Thanks,
Tim


Reconsider bundled polyfills

See my comment in https://github.com/facebookincubator/create-react-app/pull/3200#issuecomment-340219166. We should remove any polyfills that are only necessary for browsers that aren't supported out of the box by React 16 anyway. We should also document the most efficient way polyfill for the lowest possible supported browsers by React 16 (AFAIK IE9). This would be a breaking change.

I would love to see two js bundles being produced, one to be served via <script type="module" src="modern.js"> and one to be served via <script nomodule src="legacy.js">.
Justification

Pushing large bundles to evergreen browsers is wasteful. If they support type="module" then they need much less polyfilling.
Firefox will very soon support type="module".
Safari 10.1 downloads and runs both bundles, but safari 10.1 is dying out and a tiny bit of boilerplate can prevent both bundles from running.
IE<=11 may download both bundles, but only runs the legacy one. That said, IE<=11 is mostly used on desktop computers with good internet connections, so the dual download isn't a huge issue.
Mobile browsers that download both bundles are also dying out.

We're dropping IE 11 support with v2. See more instructions in the migration section.


[BUG] New-PnPTeamsTeam: Failed to find users with user principal name

Reporting an Issue or Missing Feature
This is a bug
Expected behavior
New-PnPTeamsTeam with -Members -and/or -Owner should not throw errors
Actual behavior
New-PnPTeamsTeam throws an error below when calling -Owners

New-PnPTeamsTeam: Failed to find users with user principal name

However, the team get created no problem:

Steps to reproduce behavior
'TeamName' | %{ New-PnPTeamsTeam -DisplayName $_ -MailNickName $_ -Owners @veeam.com }

What is the version of the Cmdlet module you are running?
Name    : PnP.PowerShell
Version : 1.9.47

Which operating system/environment are you running PnP PowerShell on?

[x] Windows
[ ] Linux
[ ] MacOS
[ ] Azure Cloud Shell
[ ] Azure Functions
[ ] Other : please specify

Reference
#1585

@YuriySamorodov - Not sure if this was at a point in time, but I'm not able to reproduce with 1.9.47, even adding owners & members

Can you please try again?

@KoenZomers I am getting a similar error, but I am using the current released version 1.9.0.
Resource<EMAIL_ADDRESS>does not exist or one of its queried reference-property objects are not present.
We just upgraded from the previous PnP PowerShell library (not cross platform). So, I tried this same call in that version and it worked.

@mdressel - can you try this with the latest nightly in your local machine or a VM and test it ?
@YuriySamorodov  - any update on this ? Not able to reproduce this at our end :(

If one of you being able to reproduce wants to privately share a Fiddler trace when this happens, that would be very useful in allowing us to try to figure out why it happens. On our side nobody can reproduce this. Please do not share Fiddler traces here, but share them privately.

@YuriySamorodov  or @mdressel - any chance you guys can reproduce this with latest nightly ?? We are approaching closer to a major release and would really like to fix this asap before that :)
This is a heavily used command, so don't want to ship it buggy 😊

Also, we have not had the issue again. But I think the issue may be related to the fact that the users that caused the issue were in a separate AD domain and no one has created a request for a new team since for that AD domain.  Just FYI.

Have just encountered a similar error when sending emails using PnP. In our scenario the user principal name in AD was different to the email address. Not sure if this helps investigation!

Have just encountered a similar error Resource<EMAIL_ADDRESS>does not exist or one of its queried reference-property objects are not present." when sending emails using PnP. In our scenario the user principal name in AD was different to the email address. Not sure if this helps investigation!

It sure does, because I've been fiddling with that. Can you tell me more? Did you have to provide the UPN or the e-mail address of the user to make it work?

Have just encountered a similar error Resource<EMAIL_ADDRESS>does not exist or one of its queried reference-property objects are not present." when sending emails using PnP. In our scenario the user principal name in AD was different to the email address. Not sure if this helps investigation!

It sure does, because I've been fiddling with that. Can you tell me more? Did you have to provide the UPN or the e-mail address of the user to make it work?

Yes, I discovered it wasn't actually emails but group owner/membership changes throwing the error. It appears that using UPN rather than email addresses will work in this scenario when a user's UPN doesn't match their email in AD. Specifically this was with the 'Add-PnPMicrosoft365GroupOwner' cmdlet but I would guess it'll be the same with anything touching Team/Group owners and members! It's generally not recommended as far as I understand it to have different UPN vs email, so I hadn't seen this before!

@KoenZomers and @thunderstorm654 I can confirm that in my case the users I have issues with have a different UPN then their email address. Do we think that is something that can/would be fixed in PnP or do I need to come up with a workaround?

@KoenZomers and @thunderstorm654 I can confirm that in my case the users I have issues with have a different UPN then their email address. Do we think that is something that can/would be fixed in PnP or do I need to come up with a workaround?

Hi there's a relatively simple workaround for now. Assuming you've already got an email, you can do this to get their UPN:
$pnpUser = Get-PnPUser | Where-Object Email -eq $teamOwnerEmail
$teamOwnerUPN = Get-PnPProperty -ClientObject $pnpUser -Property UserPrincipalName

Then just pass that in as the owners of the group/Team, and still works with a collection e.g. of multiple UPNs for multiple owners. There may be some shortcut to getting more properties in the first line but the above works :)

@thunderstorm654 I think the logic you gave only works if the user is in the web you are currently connected to? I am going to implement the following which I think works regardless of what connection you have.
$user = Get-PnpAzureADUser -Filter "mail eq '$email'"
That object has a UserPrincipalName property directly on it.

@thunderstorm654 I think the logic you gave only works if the user is in the web you are currently connected to? I am going to implement the following which I think works regardless of what connection you have. $user = Get-PnpAzureADUser -Filter "mail eq '$email'"
That object has a UserPrincipalName property directly on it.

Even better! The PnPUser one will ensure the user in the web, so they don't need direct access (and don't get it), but then by doing that they will end up in the user info list. In my scenario the users are all in that web creating items, so didn't think about it. I like your solution better though as covers all bases, nice :)

Hello, thanks for the wonderful discussion here in the issue.
To use the Owners and Members parameters, you will need to provide the UPN.
The underlying Graph API that we use also has the same requirement.
Both the approaches mentioned above by @mdressel  and @thunderstorm654 will work and fetch UPN values which you can then pass to this cmdlet.
So, if your email is different from your UPN, then this will not work, so please use the UPN values.
We have also updated the New-PnPTeamsTeam docs to better clarify the same.
Will close the issue for now, feel free to let us know if issue still exists  :)
Happy weekend, stay awesome 🙏😊

I wanted to add that this isn't just an issue when a user's UPN and email don't match. I am using release 1.10.0, and am unable to use the -Owners param in teams creation because for some inexplicable reason the search against the UPN is case-sensitive. For example, if I have a user whose UPN in AzureAD is listed as<EMAIL_ADDRESS>this will work:
New-PnPTeamsTeam -DisplayName 'Test Team' -Owners<EMAIL_ADDRESS>
But this will not:
New-PnPTeamsTeam -DisplayName 'Test Team' -Owners<EMAIL_ADDRESS>
This behavior is consistent, even when the users' UPN and email are identical. This is an annoying problem for us, since the script that we are using is pulling the users from SharePoint, where the casing may not be consistent with what AzureAD has.


Component Update: Hyperlink - Decoration and Underline

Purpose: We need to update our Hyperlink component in the following ways:

Make it easy to add an icon to the left or right side
Make the underline an option you can turn off

Figma: https://www.figma.com/file/VoWYiiOKKppcYm5QLgPyDC/Components---Inputs?type=design&node-id=1112%3A537&mode=design&t=kdAzH0OJ184MuCgA-1
1.0 Hyperlink w/ Icon
-- 1.1 Make it easy to add an icon to the right and/or left side of the link

-- 1.2 Commonly needed when links are used on their own, away from surrounding text, to help indicate what the link will do.
-- 1.3 Dev should be able to set the icon size to a value that will pair best with the size of the text
-- 1.4 Dev should be able to set the space between the text and the icon using our spacing system.

-- 1.5 Focus style goes around both the text and icon, not just the text.

-- 1.6 Icon color should match the text.
2.0 Hyperlink underline
-- 2.1 This only impacts the hyperlink component. When a hyperlink is used within other elements like paragraphs, the underline is still mandatory

-- 2.2 Dev should be able to turn off the underline when the component is used.
-- 2.3 On hover, the underline is used and is not optional.
3.0 Hyperlink hover color
-- 3.1 Change hover color of all hyperlink styling to the following:
--- 3.1.1 Default = primary-700

--- 3.1.2 Inverse = neutral-100

Looks great, can't find any issues here!


[ISS-63] add job hopper logo

Add #63 results into the release.

Merging file from @edward-malouf  for him.


 Widget not found: {"model_id":"a05785fb-34a8-4d81-a972-8f5d2974026a","version_major":2,"version_minor":0}

for each cell result where I expect a plot or table to be.
I've made a clean install with Anaconda 5.
UPDATE: If I explicitly save widget state before exit, everything works fine. Is it intended this way?

Yes, "save with widget state" is annoying requirement of jupyter.


Update Post + Delete Post

This issue is assigned to work on update post and delete post for frontend.

Completed during iteration 2


Allow to block everything with *

Right now I can block only by tld like com, net.
What if I want to block all sites except whitelisted?

I was wondering the same thing but also for allow. E.g. block everything with a , but then allow certain whilecard matches. For instance someone learning html might want to allow w3 to whitelist www.w3schools.com and w3resource.com, or say html.com and html.org, along with www.google.com so they can get still search and find references needing during study periods where they are coding.


Docs

update docs
bump version by 0.01 so as to allow reflection of change on crates.io

Consider a squash merge.

didn't even see this


Add toString() to BitSet [HPPC-35]

As I said in HPPC-1, it's not easy to determine the contents of a BitSet looking at its 64-bit words in a debugger ;-)

Issue: HPPC-35 (migrated from JIRA), created by Stanisław Osiński (@stanislawosinski), resolved Dec 15 2010

Comment by Dawid Weiss (@dweiss) (migrated from JIRA)
In trunk.


Crate level docs for forest_filecoin

@aatifsyed Let's describe what needs to be done here and why; otherwise, let's close it.


Add CECF token

官方公告：
5月21日 青焰新能源生态正式启动代理和城市运营中心招募
青焰APP内暂停通证认购，启动市场运营中心
5月28日 青焰物联网新能源云矿平台正式启动
青焰新能源生态APP迎来重大更新
新能源装备的购买可在APP内进行，装备凭证可兑换CECF通证
新增钱包功能
6月01日 调整注册奖励
启动新能源绿色云挖矿
6月15日 新增生态角色
【新能源经纪人】负责生态推广，参与生态建设
【新能源顾问】专业新能源工程顾问，可为生态提供技术支持
【新能源供应链】提供新能源装备供应链，光伏、锂电储能、逆变器...
【新能源代理商】负责全球新能源装备项目建设
7月02日 启动青焰新能源供应链电商平台
接入美的、华帝等品牌供应链产品，上架生态商城
7月08日 正式接入新能源生态电站，启动绿色产电挖矿
7月23日 调整生态规则，专注新能源项目建设
8月15日 新能源数据采集器Power Elf正式发布，全面启动市场化推广
团队背景：
CECF新能源通证是由海南青焰数字科技有限公司开发运营。青焰数科在工业互联网领域深耕多年，在物联网系统、软件开发、大数据应用等方面有着丰富的经验；秉承专业、创新、开拓的精神搭建新能源物联网平台Powerwell、新能源4S销售终端Power Showroom，为全球新能源项目建设提供数字化基础设施，推动清洁能源的数字化与互联网化。
当前市场上各个硬件厂商都提供了数据采集和数据展示的平台，但当一个系统里面应用了多个厂商硬件的时候，就会出现数据分隔，无法关联、无法统一应用的问题，朗禾科技物联网平台Powerwell通过数据采集器Power Elf对数据进行统一接入、联合存储、开放应用，极大地方便了合作伙伴整合各个厂商的硬件与数据，低成本地接入互联网实现数字化应用。
项目基本情况：
CECF&CEC新能源区块链项目由海南青焰数字科技有限公司发起。海南青焰数字科技有限公司2021年成立于海南，由资深区块链技术专家和投资者共同成立，为响应国家区块链产业创新以及新能源建设，发起成立CECF&CEC项目。通过以新能源分布式离网电站的建设场景为牵引，构建区块链产业的生态体系。海南青焰数字科技有限公司同成都朗禾科技有限公司达成战略协议，由海南青焰数字科技有限公司独家应用成都朗禾科技有限公司的新能源物联网平台的大数据开发及区块链应用，为中国及全人类的新能源建设做贡献。
媒体报道：
《新快报携手社会各界 关注各行各业践行绿色低碳发展》
https://www.xkb.com.cn/article_657301
《新快报 - 助力“碳达峰 碳中和”推动能源转型及结构调整》
https://mp.weixin.qq.com/s/nEJ_QPVCgP_O4--AQhfoRw
《站上数字新能源领域的全新赛道 为每一个个体赋能》
https://www.toutiao.com/i6964058867386810916/?tt_from=weixin&utm_campaign=client_share&wxshare_count=1&timestamp=1621474503&app=news_article&utm_source=weixin&utm_medium=toutiao_android&use_new_style=1&req_id=202105200935020101351650932828A0FC&share_token=509655ee-37b6-4a82-ab76-1a58deff041d&group_id=6964058867386810916
《天津第一届碳计量论坛》
https://mp.weixin.qq.com/s/h-LgTRpRUIxrBy7Nv3N0Iw
《青焰新能源供应链电商》
https://mp.weixin.qq.com/s/LwgdCe98c32CFLr4RJrJCQ
《青焰新能源生态与美的电器达成长期的战略合作》
https://mp.weixin.qq.com/s/pF6mXwnFzdC-a4jQvnkNFA
《第十届 储能国际峰会暨展览会》
https://mp.weixin.qq.com/s/OFPr-53SSqTLbrr7GN1uig
《青焰数科携手朗禾科技，打造全球首个新能源+物联网+区块链项目》
https://mp.weixin.qq.com/s/1JmHaCwMYVRgz5qcgA43TQ
收录的交易所：
暂无
Gas Limit: 60000

CECF related literature
Official announcement:
On May 21：
Qingyan new energy ecology officially launched the recruitment of agents and urban operation centers
Suspend subscription of pass in Qingyan app and start the market operation center
On May 28：        the new energy cloud mining platform of Qingyan Internet of things was officially launched
Qingyan new energy ecological app ushered in a major update
The purchase of new energy equipment can be carried out in the app, and the equipment certificate can be exchanged for CECF pass
New wallet function
On June 1：
the registration reward was adjusted
Start green cloud mining of new energy
On June 15：
New ecological role on
[new energy broker] responsible for ecological promotion and participating in ecological construction
[new energy consultant] professional new energy engineering consultant, who can provide technical support for ecology
[new energy supply chain] provide new energy equipment supply chain, photovoltaic, lithium battery energy storage, Inverter
[new energy agent] is responsible for the construction of global new energy equipment projects
On July 2：
the e-commerce platform of Qingyan new energy supply chain was launched
Access to Midea, vantage and other brand supply chain products and put them on the shelves of the ecological mall
On July 8：
it was officially connected to the new energy ecological power station and started green power generation and mining
On July 23：
the ecological rules were adjusted to focus on the construction of new energy projects
On August 15：
the new energy data collector power elf was officially released and the market-oriented promotion was fully launched
Project team background:
CECF new energy pass is developed and operated by Hainan Qingyan Digital Technology Co., Ltd. Qingyan digital has been deeply engaged in the field of industrial Internet for many years, and has rich experience in Internet of things system, software development, big data application, etc; Adhering to the spirit of professionalism, innovation and development, build new energy Internet of things platform powerwell and new energy 4S sales terminal power showroom, provide digital infrastructure for the construction of global new energy projects, and promote the digitization and Internet of clean energy.
At present, various hardware manufacturers in the market have provided platforms for data acquisition and data display. However, when multiple manufacturers' hardware are applied in a system, there will be problems of data separation, association and unified application. Powerwell, the Internet of things platform of Langhe technology, carries out unified access, joint storage and open application of data through the data collector power elf, It is very convenient for partners to integrate the hardware and data of various manufacturers, access the Internet at low cost and realize digital applications.
Project basic information:
CECF renewable energy blockchainproject was initiated by Hainan Qingyan Digital Technology Co., Ltd which wasestablished in Hainan in 2021. It was jointly established by senior experts andinvestors in the field. In response to blockchain industry innovation and renewableenergy construction sponsored by the government, CECF&CEC project wasinitiated. By taking the construction scenarios of renewable energy distributedoff-grid power stations as the traction, it will build the ecosystem of theblockchain industry. Hainan Qingyan Digital Technology Co., Ltd. has reached astrategic agreement with Chengdu Langhe Technology Co., Ltd., and HainanQingyan exclusively applies the big data technology and blockchain applicationof the renewable energy IOT platform by Chengdu Langhe which will contribute tothe construction of renewable energy in China and all over the world.
Media publications:
《New express joins hands with all sectors of society to pay attention to all walks of life and practice green and low-carbon development》
https://www.xkb.com.cn/article_657301
《New Express - helping "carbon peak and carbon neutralization" to promote energy transformation and structural adjustment》
https://mp.weixin.qq.com/s/nEJ_QPVCgP_O4--AQhfoRw
《Standing on a new track in the field of digital new energy to empower every individual》
https://www.toutiao.com/i6964058867386810916/?tt_from=weixin&utm_campaign=client_share&wxshare_count=1&timestamp=1621474503&app=news_article&utm_source=weixin&utm_medium=toutiao_android&use_new_style=1&req_id=202105200935020101351650932828A0FC&share_token=509655ee-37b6-4a82-ab76-1a58deff041d&group_id=6964058867386810916
《The first Tianjin carbon measurement Forum》
https://mp.weixin.qq.com/s/h-LgTRpRUIxrBy7Nv3N0Iw
《Qingyan new energy supply chain E-commerce》
https://mp.weixin.qq.com/s/LwgdCe98c32CFLr4RJrJCQ
《Qingyan new energy ecology and Midea reach long-term strategic cooperation》
https://mp.weixin.qq.com/s/pF6mXwnFzdC-a4jQvnkNFA
《The 10th International Energy Storage summit and Exhibition》
https://mp.weixin.qq.com/s/OFPr-53SSqTLbrr7GN1uig
《Qingyan Digital Technology Co., Ltd. and Langhe technology to create the world's first new energy + Internet of things + blockchain project》
https://mp.weixin.qq.com/s/1JmHaCwMYVRgz5qcgA43TQ
Tradeable exchanges:
Not yet
Gas Limit: 60000

抱歉，无法合并贵项目方的PR信息


bugfix: use scoped resource for puppet 4

Thank you William Oldwin for finding and fixing this issue.

Great!


uv.lock does not stay synced with pyproject.toml requirements

Hi there, I'm using uv for a large open source library https://github.com/fastled/fastled
First of all, GREAT TOOL!!!! You eliminated my own duct-taped solution for boot strapping and venv creation.
I've been tracking down an issue where when updating one of the side dependencies that I control, it was difficult to update. You'll see in the pyproject.toml file I have a dependency called fastled which lives in another repo, because reasons.
I would bump the dep to the next version on pypi, update the dep in the pyproject.toml to match, then get an error from uv saying that the program couldn't run because the results were unsatisfiable because of this fastled dependency version number.
I was pretty lost at what was causing this and it seemed like black magic was needed to get uv to run again.
I finally tracked the issue down to the uv.lock file. It does not re-sync with changes in the pyproject.toml file. But it will detect that the changes exist.
I'm sure you have a lot of bugs and features on your plate so I wanted to offer a humble suggestion to avoid this really bad foot gun: simply issue a warning that the dependencies are out of sync with what's in the uv.lock file. Just this informative warning alone would give developers the hint they need to simply delete the uv.lock file and move on.
Again, thank you very much for this tool. You guys ROCK!!!

We usually update the lockfile automatically, so perhaps this is a bug? Can you share a reproducible example?

(Also great to hear you like the tool <3)

It's tricky to reproduce.
The dependency is here: https://github.com/zackees/fastled-wasm
And the target manifesting the bug is the repo in my original post.
What I have to do is:

Update the dependency, bump it to the next revision. Then I sit back and let the builder push to pypi.
Then I go to the target repo (github.com/fastled/fastled), bump up the dep in pyproject.

I can't progress until I delete uv.lock, or do a rain dance and invoking ./clean and ./install over and over again. It will just start working.
If I get a reproducable case Ill post here.
Thanks again!

It sounds like the 10 minutes PyPI cache, perhaps use --refresh-package fastled-wasm to avoid caching that dependency during this update? When you do (2) you should update the lockfile too, do you?

I do not update the cache. However I do enter into the environment via .venv/Scripts/activate (I'm on git-bash on windows FYI) and force the update, which does indeed install fastled to the new version. I then deactivate the environment the run ./test which will then invoke uv run on the linters, then tests. uv fails immediately as soon as it's called.
I'll try and archive the repo as soon as it happens again and see if you can repro it.

I do not update the cache.

I'm not sure what you mean by this? uv will cache dependency metadata for 10 minutes. If you upload a new version, it can take up to 10 minutes for uv to invalidate the cached local metadata.

Nope.
I was iterating on one project for hours trying to get code signing to work via github actions. There were like 40 revisions. It wouldn't update to any of them. I could absolutely enter into the venv environment though, pip would install it just fine. UV wouldn't.
As soon as a I remove the uv.lock file it succeeded.

Just had it happen again
Attached is a full archive of the FastLED repo when it happens
Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled (master)
$ ./install
Using CPython 3.11.5
Creating virtual environment at: .venv
Activate with: .venv\Scripts\activate
  × No solution found when resolving
  │ dependencies:
  ╰─▶ Because only fastled<=1.2.10 is
      available and ci==0.1.0 depends on
      fastled>=1.2.11, we can conclude that
      ci==0.1.0 cannot be used.
      And because only ci==0.1.0 is
      available and you require ci, we can
      conclude that your requirements are
      unsatisfiable.

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled (master)
$ pip install fastled==1.2.11
WARNING: Ignoring invalid distribution ~dvanced-aicode (C:\Users\niteris\AppData\Local\Programs\Python\Python311\Lib\site-packages)
Requirement already satisfied: fastled==1.2.11 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (1.2.11)
Requirement already satisfied: docker>=7.1.0 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (7.1.0)
Requirement already satisfied: httpx>=0.28.1 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (0.28.1)
Requirement already satisfied: watchdog>=6.0.0 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (6.0.0)
Requirement already satisfied: download>=0.3.5 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (0.3.5)
Requirement already satisfied: filelock>=3.16.1 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (3.16.1)
Requirement already satisfied: disklru>=2.0.1 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (2.0.1)
Requirement already satisfied: appdirs>=1.4.4 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (1.4.4)
Requirement already satisfied: rapidfuzz>=3.10.1 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (3.10.1)
Requirement already satisfied: progress>=1.6 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (1.6)
Requirement already satisfied: nodejs-bin[cmd] in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from fastled==1.2.11) (18.4.0a4)
Requirement already satisfied: pywin32>=304 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from docker>=7.1.0->fastled==1.2.11) (306)
Requirement already satisfied: requests>=2.26.0 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from docker>=7.1.0->fastled==1.2.11) (2.32.3)
Requirement already satisfied: urllib3>=1.26.0 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from docker>=7.1.0->fastled==1.2.11) (2.2.3)
Requirement already satisfied: tqdm in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from download>=0.3.5->fastled==1.2.11) (4.66.5)
Requirement already satisfied: six in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from download>=0.3.5->fastled==1.2.11) (1.16.0)
Requirement already satisfied: anyio in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from httpx>=0.28.1->fastled==1.2.11) (3.7.1)
Requirement already satisfied: certifi in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from httpx>=0.28.1->fastled==1.2.11) (2024.7.4)
Requirement already satisfied: httpcore==1.* in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from httpx>=0.28.1->fastled==1.2.11) (1.0.5)
Requirement already satisfied: idna in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from httpx>=0.28.1->fastled==1.2.11) (2.8)
Requirement already satisfied: h11<0.15,>=0.13 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from httpcore==1.*->httpx>=0.28.1->fastled==1.2.11) (0.14.0)
Requirement already satisfied: nodejs-cmd in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from nodejs-bin[cmd]->fastled==1.2.11) (0.0.1a0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from requests>=2.26.0->docker>=7.1.0->fastled==1.2.11) (3.3.2)
Requirement already satisfied: sniffio>=1.1 in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from anyio->httpx>=0.28.1->fastled==1.2.11) (1.3.1)
Requirement already satisfied: colorama in c:\users\niteris\appdata\local\programs\python\python311\lib\site-packages (from tqdm->download>=0.3.5->fastled==1.2.11) (0.4.6)

It's about a 1gb so far to large for github.
However, I've put this on a private server for you to download. I'll keep it open for 24 hours:
https://5a97-98-97-26-124.ngrok-free.app

It also just resolved itself, so the 5 min limit on a the pypi cache seems like it's a good bet that this is what is happening.

I think you just need to add --refresh-package to your uv pip install . invocation then

I think this is a real bug. I know that pypi does NOT reflect versions immediately available.
I am seeing this same error propagate through our massive test chain, as we test lots and lots of devices for our open source projects.
So what I think is happening is this:

New package get's uploaded to pypi
The first pip install fastled==XX will fail, pretty much guaranteed even minutes after it's uploaded

However, immediate repeated pip install XXX==YYYY will make it come online faster

What this means is then

pypi has a poisoned cache until second or 3rd pip install...
uv's cache get's poisoned when it's the first caller to pip install..

Now uv's cache will remain poisoned for the next 5 mins

This issue only affects developers that have a dependency they are updating immediatly. It's also going to periodically cause build flakyness, or docker images could fail to build.
Work around
Since github actions does the pypi publishing, after the artifact is posted, i'm going to run the folllowing:
for i in range(10):
  os.system(f"pip install fastled=={version}")
  time.sleep(1)

Hopefully this hack will force the dependency to become immediately available and prevent the uv cache from becoming poisoned.

Yeah, just hit this again while doing uv sync
Now I have to wait for some timeout to happen. Output:
Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ uv sync
  × No solution found when resolving dependencies:
  ╰─▶ Because only static-npm<=1.0.4 is available and your project depends on static-npm>=1.0.5, we can conclude that
      your project's requirements are unsatisfiable.

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ pip install static-npm==1.0.5
ERROR: Could not find a version that satisfies the requirement static-npm==1.0.5 (from versions: 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4)
ERROR: No matching distribution found for static-npm==1.0.5

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ pip install static-npm==1.0.5
Collecting static-npm==1.0.5
  Downloading static_npm-1.0.5-py2.py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: download in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from static-npm==1.0.5) (0.3.5)
Requirement already satisfied: appdirs in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from static-npm==1.0.5) (1.4.4)
Requirement already satisfied: filelock in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from static-npm==1.0.5) (3.16.1)
Requirement already satisfied: tqdm in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from download->static-npm==1.0.5) (4.66.6)
Requirement already satisfied: six in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from download->static-npm==1.0.5) (1.16.0)
Requirement already satisfied: requests in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from download->static-npm==1.0.5) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from requests->download->static-npm==1.0.5) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from requests->download->static-npm==1.0.5) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from requests->download->static-npm==1.0.5) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from requests->download->static-npm==1.0.5) (2024.8.30)
Requirement already satisfied: colorama in c:\users\niteris\dev\fastled-wasm\.venv\lib\site-packages (from tqdm->download->static-npm==1.0.5) (0.4.6)
Downloading static_npm-1.0.5-py2.py3-none-any.whl (9.0 kB)
Installing collected packages: static-npm
  Attempting uninstall: static-npm
    Found existing installation: static_npm 1.0.4
    Uninstalling static_npm-1.0.4:
      Successfully uninstalled static_npm-1.0.4
Successfully installed static-npm-1.0.5

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ uv sync
  × No solution found when resolving dependencies:
  ╰─▶ Because only static-npm<=1.0.4 is available and your project depends on static-npm>=1.0.5, we can conclude that
      your project's requirements are unsatisfiable.

I can even uv pip install it, but uv sync then fails.
Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ uv pip install static-npm==1.0.5
Audited 1 package in 98ms

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ uv pip install static-npm==1.0.5 --refresh
Audited 1 package in 93ms

Zach Vorhies@DESKTOP-I3718DO MINGW64 ~/dev/fastled-wasm (main)
$ uv sync
  × No solution found when resolving dependencies:
  ╰─▶ Because only static-npm<=1.0.4 is available and your project depends on static-npm>=1.0.5, we can conclude that
      your project's requirements are unsatisfiable.

Oh I see, other's are reporting the problem too:
https://github.com/astral-sh/uv/issues/2538
Can you just make uv sync break through the cache?


Reworking

Chapter 4:

Find out at least one dataset that had different order in Windows
Calculate RoBERTa scores for Disease datasets


Update README.md

added the steps for how to run the project in the local system

please review the changes @bhattabhi013

LGTM ✨

Hey, just one last thing can you add the gssoc23 label to this?


error: format specifies type 'ssize_t' (aka 'long') but the argument has type 'off_t' (aka 'long long')

I am getting the following errors in master. Changing the formatters to %ll fixes the problem. Any ideas why I am seeing this? OS 10.13.4
☁  hyperkit [master] make -j4
cc src/lib/firmware/bootrom.c
src/lib/firmware/bootrom.c:95:49: error: format specifies type 'ssize_t' (aka 'long') but the argument has type 'off_t' (aka 'long long')
      [-Werror,-Wformat]
                fprintf(stderr, "Invalid bootrom size %zd\n", sbuf.st_size);
                                                      ~~~     ^~~~~~~~~~~~
                                                      %lld
src/lib/firmware/bootrom.c:101:22: error: format specifies type 'ssize_t' (aka 'long') but the argument has type 'off_t' (aka 'long long')
      [-Werror,-Wformat]
                    "page size\n", sbuf.st_size);
                                   ^~~~~~~~~~~~
src/lib/firmware/bootrom.c:112:4: error: format specifies type 'ssize_t' (aka 'long') but the argument has type 'off_t' (aka 'long long')
      [-Werror,-Wformat]
                        sbuf.st_size);
                        ^~~~~~~~~~~~
3 errors generated.
make: *** [build/lib/firmware/bootrom.o] Error 1

diff --git a/src/lib/firmware/bootrom.c b/src/lib/firmware/bootrom.c
index 02cdecd..3ae2b6a 100644
--- a/src/lib/firmware/bootrom.c
+++ b/src/lib/firmware/bootrom.c
@@ -92,12 +92,12 @@ uint64_t bootrom_load(void)
         * MMIO space (e.g. APIC, HPET, MSI).
         */
        if (sbuf.st_size > MAX_BOOTROM_SIZE || sbuf.st_size < XHYVE_PAGE_SIZE) {
-               fprintf(stderr, "Invalid bootrom size %zd\n", sbuf.st_size);
+               fprintf(stderr, "Invalid bootrom size %lld\n", sbuf.st_size);
                goto done;
        }

        if (sbuf.st_size & XHYVE_PAGE_MASK) {
-               fprintf(stderr, "Bootrom size %zd is not a multiple of the "
+               fprintf(stderr, "Bootrom size %lld is not a multiple of the "
                    "page size\n", sbuf.st_size);
                goto done;
        }
@@ -108,7 +108,7 @@ uint64_t bootrom_load(void)
        ptr = vmm_mem_alloc(gpa, (size_t)sbuf.st_size);
        if (!ptr) {
                fprintf(stderr,
-                       "Failed to allocate %zd bytes of memory for bootrom\n",
+                       "Failed to allocate %lld bytes of memory for bootrom\n",
                        sbuf.st_size);
                rv = -1;
                goto done;

Please do raise a PR.
Using %lld is probably the least bad option but please also cast the variables with (long long) to eliminate any possible mismatch.
Unclear why you are the only one seeing this so far, might be a more (or less) up to date clang in your toolchain or some CFLAG thing turning on long offsets. Seems a bit odd for ssize_t and off_t to have such different underlying type magnitudes, but oh well.

The diagnostic is new in clang 6.


exclude package

I want to be build a package to deploy on Google Cloud Functions. My code imports @google-cloud/datastore which has many transitive dependencies (as is typical for Node libraries), some of which have binary files, computed imports etc.
Because I'm compiling Typescript and using other libraries in my build/test/package/deploy process,  node_modules contains 100s-1000s of libraries, most of which aren't needed in production.
Ideally, I'd like to be able to build something that looked like:

package.json - {"main": "index.js", dependencies: { "@google-cloud/datastore": "1.4.1" }}
index.js - compiled from multiple TypeScript files in my project and most of the code I'm importing from  libraries and transitive dependencies
node_modules - all of, but only the code that is not included in index.js but is required to run the app.

I've created a simple demo app to show what I'm trying to do:
https://github.com/nalbion/packaged-google-function/blob/master/lib/demo.js
To exclude @google-cloud/datastore and it's transitive dependencies from my compiled demo.js I've added a filterFile:
filterFile: file => {
        return !['@google-cloud/datastore'].includes(file.collection.name);
    },

I'm confused by the lines in the output:
FuseBox.pkg("@google-cloud/datastore", {}, function(___scope___){
return ___scope___.entry = "src/index.js";
});

Google Cloud Functions is also confused:
TypeError: Cannot read property 'default' of null
    at helloWorld (/user_code/demo.js:10:42)

For reference, the demo was working until I tried to add the datastore code:
https://github.com/nalbion/packaged-google-function/blob/no-dependencies/lib/demo.js
I suspect that filterFile is not intended for this purpose, or maybe I'm using it wrong.
Is there an equivalent to filter packages?

hi @nalbion
Based on the output you are requiring something that doesn't exist. FuseBox wasn't able to resolve the package.
Could you require a specific file from the package? It would help if you pasted some links
Besides, there is ignoreModules of FuseBox.init

@nchanged I didn't see ignoreModules in the documentation, but it sounds like what I'm looking for.
I call import Datastore from '@google-cloud/datastore'; and then new Datastore({projectId: 'demo'}); in src/gcf/demo.ts

@nalbion yep, it's not documented ignoreModules, the other option is to add it to instructions
instructions("- google-cloud")

closing this for now


feat(actiongroup)!: migrate to core tokens

Description
Migrates ActionGroup to core tokens. Also consumes latest version of ActionButton
How and where has this been tested?

How this was tested:
Browser(s) and OS(s) this was tested with:

Screenshots

To-do list

[ ] If my change impacts other components, I have tested to make sure they don't break.
[ ] If my change impacts documentation, I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have tested these changes in Windows High Contrast mode.

[ ] This pull request is ready to merge.

Released beta:
3.0.0-beta.0


fix bug where interleaved kill and create lead to duplicate ids

Noticed this due to a "upgrade" system transforming entities into new ones, so it'd delete X entities and create X entities, but only X - 1 would be remembered.
For the test case, what would happen is that you'd observe the sequence:
Create 0
Create 1
Kill 0
Kill 1
Create 2
Create 2

This is due to return i == max_id in the first post-kill create, and then returning max_id in the second call. It also seems to me like max_id is more clearly next_id -- if you disagree I can drop that commit, but that realization helped diagnose this bug.

Thanks! <3

published in planck_ecs = "1.1.0" and entity_component = "1.1.0" ! :hearts:


Add getChannelTeams to ChannelAPI

Feature Request
I noticed that an API endpoint I need, Get Channel Teams under the Channel entity was missing.
I would like to add this request under ChannelAPI class. I can implement it myself using the existing work done for TeamAPI class for the response object.

Sure, you're very welcome to add it :) if you do, please work on the next branch :)

Closing as #125 is already merged.


Support callbacks with nonlinear models

In MathProgBase and JuMP there's a distinction between solvers for LinearQuadratic vs Nonlinear (expression-based) problems. According to the documentation, the callbacks are supported only for MIP solvers (i.e., LinearQuadratic).
For SCIP, this distinction is artificial. We initially implemented the wrapper with a focus on the LinearQuadratic interface, later adding support for the Nonlinear.
Unfortunately, when building a model using @NLconstraint, adding callback functions does not have an effect with SCIP.jl currently (see example).
@mlubin: Is this a restriction inherent in JuMP or MathProgBase? Can we work around this with little effort?
@fserra: I'm confused by the implementation of Nonlinear in SCIP.jl, in particular this code:
type SCIPMathProgModel <: AbstractLinearQuadraticModel
[...]
end

type SCIPNonlinearModel <: AbstractNonlinearModel
    m::SCIPMathProgModel
end

LinearQuadraticModel(s::SCIPSolver) = SCIPMathProgModel(s.options)
NonlinearModel(s::SCIPSolver) = SCIPMathProgModel(s.options)

The type SCIPNonlinearModel is never again used in the code. Might there be a related issue here?

The MathProgBase documentation gives the signature as setlazycallback!(m::AbstractLinearQuadraticModel, f). So if we actually used SCIPNonlinearModel then I would believe this to be the issue. But anyway in NonlinearModel(s::SCIPSolver) we return a SCIPMathProgModel which is in fact an AbstractNonlinearModel?!

Is this a restriction inherent in JuMP or MathProgBase?

It's not in MathProgBase or JuMP only because we've never put effort into supporting callbacks for nonlinear solvers. I would support tweaking the wording to make it clear that callbacks may be implemented in nonlinear models. JuMP will need a few small changes to set up the callbacks for nonlinear model, shouldn't be too difficult.

But anyway in NonlinearModel(s::SCIPSolver) we return a SCIPMathProgModel which is in fact an AbstractNonlinearModel?!

That's not good. There should be a wrapper type which implements AbstractNonlinearModel a lot of the other solver interfaces do this.

Thanks for the info, @mlubin.

JuMP will need a few small changes to set up the callbacks for nonlinear model, shouldn't be too difficult.

Sounds good, should I open an issue for this?

There should be a wrapper type which implements AbstractNonlinearModel a lot of the other solver interfaces do this.

Can you give me an example? I've looked at the code for some solver interfaces, such as IPOPT, Xpress or KNITRO, but they all only implement one type of AbstractLinearQuadraticModel or AbstractNonlinearModel and then use a Bridge to provide also the other. Is that what you suggest for SCIP.jl also? I don't think that we can keep the same generality that we already have w.r.t. expressions.
The issue I see is that if we use separate subtypes for AbstractNonlinearModel and AbstractLinearQuadraticModel, there would be a lot of duplicate methods, and only loadproblem! would really make this distinction.
What I'd like to do here is something like this:
abstract SCIPModel end
type SCIPNonlinearModel <: AbstractNonlinearModel end
type SCIPLinearQuadraticModel <: AbstractLinearQuadraticModel end

but have the latter concrete types also be subtype of SCIPModel so that we can use SCIPModel to provide the common implemenation of functions such as getsolvetime or numvar.
I believe that multiple inheritance is not possible in Julia, but is this maybe a use case for union types?

Another thing I just noticed: The MPB doc says that setwarmstart! takes an AbstractLinearQuadraticModel. But really, it's also called by JuMP for AbstractNonlinearModel (see _buildInternalModel_nlp).

The MPB doc says that setwarmstart! takes an AbstractLinearQuadraticModel. But really, it's also called by JuMP for AbstractNonlinearModel (see _buildInternalModel_nlp).

Yes, that's worth fixing. PRs are welcome.

I tried running my example above using the change in JuMP/#878 and it kind of worked:
<<< Now try with nonlinear constraints:
### lazycb is called! (1)
[src/scip/scip.c:19502] ERROR: invalid SCIP stage <3>
[src/scip/cons_linear.c:640] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:3513] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:16568] ERROR: Error <-8> in function call
Failing with retcode 1 at 118
Failing with retcode 1 at 1374
### lazycb is called! (2)
presolving (16 rounds: 16 fast, 3 medium, 3 exhaustive):
 16 deleted vars, 20 deleted constraints, 14 added constraints, 44 tightened bounds, 0 added holes, 0 changed sides, 1 changed coefficients
 9 implications, 0 cliques
presolved problem has 13 variables (0 bin, 0 int, 0 impl, 13 cont) and 6 constraints
### lazycb is called! (3)
[...]

So the callback is definitely called, but there is some SCIP stage issue (3 = transformed). I don't know yet whether that issue should be dealt with on the CSIP side. I'm guessing it is triggered by the (all-0) warmstart solution added by JuMP for NLPs.

I tried the same again, using SCIP in debug mode (OPT=dbg):
[src/scip/scip.c:247] ERROR: cannot call method <SCIPlockVarCons> in problem transformed stage
[src/scip/scip.c:19464] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:640] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:3513] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:16568] ERROR: Error <-8> in function call
[...]
julia: src/scip/cons_linear.c:770: consCatchAllEvents: Assertion `consdata->eventdata == ((void *)0)' failed.
[...]
while loading /home/bzfschwa/.julia/v0.5/PipeLayout/examples/minlp_callbacks.jl, in expression starting on line 48
[...]
consCatchAllEvents at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons_linear.c:770
consInitLinear at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons_linear.c:13771
SCIPconshdlrInit at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons.c:2389
SCIPsetInitPlugins at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/set.c:4308
SCIPtransformProb at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:12817
SCIPpresolve at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:14366
SCIPsolve at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:14581
CSIPsolve at /home/bzfschwa/.julia/v0.5/SCIP/deps/usr/lib/libcsip.so (unknown line)
optimize! at /home/bzfschwa/.julia/v0.5/SCIP/src/mpb_interface.jl:52

@fserra: Does that mean anything to you?
I double-checked and I create a "fresh" SCIPSolver object for this second run. Also, the same happens when I only run the second part.

I think that the error message and the failing assertion are actually not directly related.

OK, we have a workaround for the SCIP stage problem in CSIP, then we're good.

CSIP is updated, now we only need the change in JuMP merged.

The relevant PR JuMP.jl/#894 was already merged.


Docs: Extract color previews from unassigned Color objects in match/case

We just added color extraction from unassigned objects for previews:
if True:
    # We will now get a color preview for this
   # where you had to explicitly print prior to this.
    Color('red')

We added this for if/elif/else, for/else, white/else, with, and try/catch/else/finally. Unfortunately, we have not yet added it for match/case.
Match case is a bit more complicated and has multiple different types of patterns, some can generate * variables, and it is just generally more difficult to extract the color results by executing line by line in AST. This is because normally this is run via exec because you can't run the start of the block by itself, you need the whole context. We work around this for other blocks by handling the block (evaluating the conditions and applying the block logic ourselves) and then running the block content line by line. To do this in match/case, we need to figure out a way to execute and match the conditions from the AST so we can iterate the case blocks line by line. This may be easier than I think, or we'll have to implement all the pattern match cases individually, covering all their quirks to then know which case block we need to evaluate.
Now, this doesn't mean match/case is currently broken, we just don't have fine enough resolution to extract color previews when a color is under the scope of the match/case block. You can still print:

You can store the objects and display them later:

We just can't get at those objects under that scope yet:

I actually was able to get it to work. It took a bit, but all the pieces kind of build off each other. Now we can get color previews from inside match/case. That handles all the major blocks. We are skipping Async blocks.
This will make it more natural to play with the lib live.


Updated crate-admin to version 0.11.2 which includes following changes:

- removed usage of sys expressions in wrong context
- fixed a template rendering issue in cluster view in Safari
- de-register watches to decrease DOM updates and improve performance in cluster view
- fixed title of `tables` view if no tables exists or connection is down

lacktm


Nextcloud static channel backups

This pull request adds a nextcloud destination for the static channel backups.
The corresponding pull request for the charts can be found here. However this can be merged without changes to the helm charts. If the nextcloud environment variables are not set it will simply skip it.
Additionally it replaces the dropbox dependency with axios.
Since Nextcloud uses a Webdav interface where the filepath is contained inside the url. The NEXTCLOUD_URL environment variable thus takes the whole url for example https://myserver.com/remote.php/dav/files/myuser/backupfolder.
Please see the official documentation here.
I have added a check in the backup routine which results in an error log entry if none of the environment variables for any destination are set. But it will not crash.
If any of the environment variables for either google cloud, dropbox or nextcloud are not set it will skip them and notify in the logs.

LGTM @zoop-btc
you also should remove dropbox from our dependency now (yarn remove ...)

Did an in person review yesterday and agreed @zoop-btc would add tracing to the backup code paths so that we get notified if one fails as part of this PR. This will also help identify missing configuration.

I am waiting for this tracing issue to be resolved before this can be upgraded to use tracing. I've found it while calling asyncRunInSpan for the backups.

Changed to draft until tracing code is completed.

Changed to draft until tracing code is completed.

Is this still WIP?

Changed to draft until tracing code is completed.

Is this still WIP?

would be good to have some opentelemetry plugged into this file so we could have some alert in case there is backup errors

Changed to draft until tracing code is completed.

Is this still WIP?

would be good to have some opentelemetry plugged into this file so we could have some alert in case there is backup errors

I added the tracing. It first creates a root span for servers.trigger.uploadBackup. Inside that span the attributes uploadBackup.destination.dropbox /googlecloud / nextcloud will be set to true or false depending on which environment variables are set. If none are set it'll record a critical error event.

_Example of the error_

Then each subsequent uploadDestination is wrapped in a span again and an event is recorded if it's successful or not. The errorlevel of those are warning.

_Example of an error if Dropbox Token is invalid_

I could only test Dropbox and Nextcloud as I don't have a google service account.

_Here what a successful upload to those two looks like_

Will make sure that the charts-part of this is also up-to-date and then mark both as ready for review.


Create ConvertStringToConsonantcase.java

Why:

Closes #ISSUE_NUMBER
What's being changed:

@anandfresh Hi, how do I do that? :)

@anandfresh Hi, how do I do that? :)

Hey @NoyHanan
You can add Closes #ISSUE_NUMBER in the description of the PR(s)
For this case Closes #5150


Fix as_completed to not need async with.

Codecov Report
Base: 28.70% // Head: 28.70% // No change to project coverage :thumbsup:

Coverage data is based on head (09c9f51) compared to base (0e06b23).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##           stable    #1078   +/-   ##
=======================================
  Coverage   28.70%   28.70%
=======================================
  Files          72       72
  Lines        5852     5852
  Branches      306      306
=======================================
  Hits         1680     1680
  Misses       4109     4109
  Partials       63       63

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


update test-infra

ci-tools needs an updated test-infra so release-controller gets it too.

@smg247 a significant change just merged.  I'm going to watch it rollout, for a bit, before approving this PR.  Hope that's ok!

Works for me!


Return false from instanceof if the type being checked against doesn't exist

Fixes #2272
Previously, checking if a value was an instance of a nonexistent type would result in an uncaught exception. This changes it to return false instead, avoiding having to do an extra feature-test of whether the type exists before calling dyn_into/dyn_ref/etc.

I just noticed that this code seems to raise the target of the generated code from supposedly ES2015 to ES2019. From what I can tell, it's nowhere documented what level is targeted (well, obviously at least ES2015), so it might be acceptable, but at least it seems unnecessary.

Yeah, unfortunately there is no policy in place for a specific ES target and no checks are made in that regard.
I'm not familiar with the differences of these targets, but we can take a look at a PR that reduces the required target. In any case, as far as I can see this PR mitigated/fixed a certain bug, so this definitely needs to be preserved.


snark work prometheus metrics

Added the following metrics:

snark_work_received_gossip: # of snark work received from peers
snark_work_received_rpc: # of snark work received via rpc
snark_work_sent_rpc: # of snark work sent via rpc
snark_pool_size: # of snark works in the snark pool
snark_work_per_block: # of snark work purchased per block
scan_state_snark_jobs: # of jobs in the scan state after every block

Checklist:

[ ] Tests were added for the new behavior
[ ] All tests pass (CI will check this if you didn't)
[ ] Serialized types are in stable-versioned modules
[ ] Does this close issues? List them:

An attempt to order this a little more sequentially.
scan_state_snark_jobs:            gauge of snark work jobs in the current scan state
snark_work_assigned_rpc:        count of snark work assigned to be done by snarkworkers
snark_work_reassigned_rpc:     count of snark work reassigned to be done by snarkworkers
snark_work_completed_rpc:     count of snark work completed by snarkworkers
snark_work_received_gossip:   count of snark work MESSAGES received from peers (includes repeats)
snark_pool_size:                        gauge of current number of snark works in the snark pool
snark_work_last_block:             gauge of snark work purchased from pool in the last block

snark_work_reassigned_rpc: count of snark work reassigned to be done by snarkworkers

This would be slightly different because what we do is remove such jobs from the "seen" jobs list so that it could be reassigned when the next request comes in and not reassign them immediately. So the metric would be # of snarks not completed within a specific time. Maybe snark_work_timed_out?

...  Maybe snark_work_timed_out?

that works
snark_work_timed_out_rpc ?


extlinux-conf-builder: specialisations entries

Motivation for this change
Add entries to the generated extlinux config for sub-generations (specialisations).
Example output after this commit:
Retrieving file: /extlinux/extlinux.conf
10981 bytes read in 1 ms (10.5 MiB/s)
------------------------------------------------------------
1:      NixOS - Default
2:      NixOS - Configuration 334-default (2022-01-13 15:07 - 22.05.20220113.dirty)
3:      NixOS - Configuration 334-foundation (2022-01-13 13:46 - 22.05pre-git)
4:      NixOS - Configuration 333-default (2022-01-13 15:03 - 22.05.20220113.dirty)
5:      NixOS - Configuration 333-foundation (2022-01-13 13:46 - 22.05pre-git)
6:      NixOS - Configuration 332-default (2022-01-13 14:59 - 22.05.20220113.dirty)
7:      NixOS - Configuration 332-foundation (2022-01-13 13:46 - 22.05pre-git)
8:      NixOS - Configuration 331-default (2022-01-13 14:52 - 22.05.20220113.dirty)
9:      NixOS - Configuration 331-foundation (2022-01-13 13:46 - 22.05pre-git)
10:     NixOS - Configuration 330-default (2022-01-13 13:56 - 22.05.20220113.c9d0820)
11:     NixOS - Configuration 330-foundation (2022-01-13 13:56 - 22.05pre-git)
12:     NixOS - Configuration 329-default (2022-01-13 13:46 - 22.05.20220113.c9d0820)
13:     NixOS - Configuration 329-foundation (2022-01-13 13:46 - 22.05pre-git)
14:     NixOS - Configuration 328-default (2022-01-13 13:12 - 22.05.20220113.c9d0820)
15:     NixOS - Configuration 328-foundation (2022-01-13 13:06 - 22.05pre-git)

Particularly, this shows a "foundation" specialisation that represents a different kernel configuration.
Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2789

I'd like to merge this, but I suppose it could have tiny consequences if someone had:

specialisations enabled, albeit never bootable directly
hardware automation for manually entering key-presses into early boot to blindly boot into an old generation.

This seems unlikely, and thus relatively safe to merge, but I don't want to self-merge it.


feat(tutorial): complete step 3

Closes #
{{short description}}
Changelog
New

{{new thing}}

Changed

{{change thing}}

Removed

{{removed thing}}

@mattrosno seems the same issue as https://github.com/carbon-design-system/carbon-website/issues/1297
Locally the build is working. Main difference is yarn version:

versus the 1.15.2 used by netlify

Adding gensync module didn't fix the issue as now it complains about another missing library:
Cannot find module '@babel/helper-create-regexp-features-plugin'


Suggestion: When execution fails, use error message & traceback to debug

When code execution fails, it could automatically take the error message & traceback and ask GPT to suggest a fix. Could be a setting to toggle this to run automatically or have it manually triggered on demand.
Would be a great alternative to just asking what's wrong with a piece of code.

@salomartin thanks for the suggestion. In order to send a natural language prompt, initially you need to identify where the culprit is in your code if there is a compilation issue. In that case you can use, ChatGPT: Find problems to ask it to analyze and find the issue.
Are you thinking more of an automated way of doing it? If so, could you elaborate a little bit more on this? I like the idea of enriching our feature set but just want to understand more on your scenario

In this case I executed python code that gpt had suggested with the vscode run command and got this output in the python debug console. In this case, i just then manually copy-pasted this over to gpt and asked how to fix this issue. It took a few iterations to get it right. If the question to suggest a fix was sent to gpt automatically after running code, it could reduce the iteration time. Of course we could go on from here and get it to apply the fix too, but that'd be a new suggestion already :-)
Output from the debug-console:
Traceback (most recent call last):
File "code_suggested_by_gpt.py", line 34, in
print(extras.Json(results).dumps(indent=2, schema=True))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Runtime errors are maybe a bit too language/version specific?
Maybe compiler errors have a better internal interface in vscode. While programming flutter/Dart, I often get suggestions from before breaking changes.  When I insert that code, I get errors in the "problems" tab (bottom) after a second (some dart tooling is used there). Maybe it's similar for other languages/frameworks.
It would be cool, if those error descriptions get feed back directly into a prompt "Repair this problem", or something.


Update Mapster.Tool to include .net6.0 as target framework

I'm using Mapster.Tool to generate mappers from interfaces. When I tried updating my project to .NET 6.0 an exception is thrown.
Updating the target framework in Mapster.Tool to include net6.0 fixes the issue.
The issue can be reproduced with a new console application running .NET 6.
Program.cs:
using Mapster;

Console.WriteLine("Hello, World!");

public class A
{
    public string Str { get; set; }
}

public class B
{
    public string Str { get; set; }
}

[Mapper]
public interface IMapper
{
    A Map(B test);
}

public class MyRegister : IRegister
{
    public void Register(TypeAdapterConfig config)
    {
        config.NewConfig<A, B>();
    }
}

Running dotnet mapster mapper -a ".\bin\Debug\net6.0\MapsterNet6Issue.dll" outputs the following error
Cannot find assembly path: System.Runtime (type=package, version=5.0.11-servicing.21473.8)
Cannot find assembly path: System.Runtime (type=package, version=5.0.11-servicing.21473.8)
Cannot find assembly path: System.Runtime (type=package, version=5.0.11-servicing.21473.8)
Cannot find assembly path: System.Runtime (type=package, version=5.0.11-servicing.21473.8)
Unhandled exception. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at Mapster.TypeAdapterConfig.<>c.<Scan>b__87_0(Assembly assembly) in D:\git\Mapster\src\Mapster\TypeAdapterConfig.cs:line 619
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Mapster.TypeAdapterConfig.Scan(Assembly[] assemblies) in D:\git\Mapster\src\Mapster\TypeAdapterConfig.cs:line 619
   at Mapster.Tool.Program.GenerateMappers(MapperOptions opt) in D:\git\Mapster\src\Mapster.Tool\Program.cs:line 65
   at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult`1 result, Action`1 action)
   at Mapster.Tool.Program.Main(String[] args) in D:\git\Mapster\src\Mapster.Tool\Program.cs:line 17
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Runtime, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

To get by, I forked the project and confirmed it works by setting Target to Net60 and installing the locally built Mapster.Tool nupkg.
Created PR #390 accordingly

You can compile Mapster tool by changing net6.0 and not net5.0
netcoreapp3.1;net6.0
Then use the executable to run your command.

Well, It was a little more complicated to have  the tool working.
You can found the code here:
https://github.com/Philippe-Laval/Mapster
https://github.com/Philippe-Laval/TestMapster
All projects are compiled under net6.0 (I created a pull request).

Any news on this issue? I'm experiencing the same behavior and I was wondering if any pull request is in planned to be merged to solve this.

Please try the latest version now available on NuGet and let me know how it goes. I'm on standby to resolve any issues you may encounter.

Please try the latest version now available on NuGet and let me know how it goes. I'm on standby to resolve any issues you may encounter.

Seems to be working so far, had to remember to run dotnet tool update Mapster.Tool. Cheers!

Any chance to add .net7.0 target framework?


About input frames and sampling interval

Thank you for your excellent work! By the way I want to know about clip_len and frame_interval for Kinetics. In Appendix A.1, "We evaluate the model on 8, 16, 32 frames and the sampling interval is 16, 8, 4, respectively."  Does this mean for kinetics400/700, the data pipeline (train, val, test) should be the same? For example, in configs/recognition/vit/vit_imagenet_k400.py, the config of data pipeline keeps the same as the paper mentioned.
i.e., clip_len=8, frame_interval=16 for train/val/test pipeline, which is the same as the paper mentioned.
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vit_imagenet_k400.py#L19-L21
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vit_imagenet_k400.py#L32-L39
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vit_imagenet_k400.py#L49-L56
But, for CLIP pretrained, the configs are confused.

vitclip_base_k400, clip_len=32, frame_interval=16 for train pipeline, while clip_len=32, frame_interval=8 for val/test pipeline. However, if clip_len=32, frame_interval should be 4?

https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_base_k400.py#L19-L21
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_base_k400.py#L32-L39
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_base_k400.py#L49-L56

vitclip_large_k400, clip_len=16, frame_interval=16 for train/val/test pipeline. However, if clip_len=16, frame_interval should be 8?

https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_large_k400.py#L19-L21
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_large_k400.py#L32-L39
https://github.com/taoyang1122/adapt-image-models/blob/392647ea000c9bda8e1123c4efb6d61cd398025c/configs/recognition/vit/vitclip_large_k400.py#L49-L56
Thank you.

Hi @BinhuiXie , thanks for your interest in our work. You can safely follow the settings descripbed in the paper. I will update the codes.


Section 4.5 - Relative Motion in One and Two Dimensions

There does not seem to be a subtopic for relative motion and I am not sure where to place the LOs in section 4.5.

I've created a new subtopic under Kinematics(2D and 3D) for Relative Motion. I don't think a Relative Motion subtopic is necessary under Kinematics (1D) but if it is needed just add one there as well.