Introduce email field in user table

The new fields email, and permission are introduced in the user table.
We introduced the email field because name is not unique. We identify a user by email now when we share the resources, name is also showing up to help the user identify the email address.  'uid' is still the primary key internally.
The username for the local login system will still be stored in the name field.
Current systems can be updated automatically with the script \core\scripts\sql\update\user.sql.
In this script, I just rename the current name field since it's actually storing the user's email in the existing database.
After that, I just add the permission field at the end of the row.
Finally, I add the 'name' field after 'uid'.
The 'email' and 'name' will be updated when the user login with google, so the data will be updated as long as the DDL is correct.

In the PR description, please also specify the reason we introduce email and permission fields (e.g., what features we can enable, why the old name field is not enough, etc). And how do we identify a user from now on? do we use name or email or both?
Also, I have seen you included a sql script to update the existing databases. Please also briefly describe how will these changes backward compatible with the existing data, so I can review the sql script accordingly.

In the PR description, please also specify the reason we introduce email and permission fields (e.g., what features we can enable, why the old name field is not enough, etc). And how do we identify a user from now on? do we use name or email or both?
Also, I have seen you included a sql script to update the existing databases. Please also briefly describe how will these changes backward compatible with the existing data, so I can review the sql script accordingly.

Updated.

One comment before reviewing code,

The 'email' and 'name' will be updated when the user login, so the data will be updated as long as the DDL is correct.
Are we implicitly using user's google username as name field from now on? If so, I think we should document it somewhere. If the ddl field name is decided as name, shall we add a comment somewhere in the code to mention that this will store and will be overwrited by google username?

One comment before reviewing code,

The 'email' and 'name' will be updated when the user login, so the data will be updated as long as the DDL is correct.

Are we implicitly using user's google username as name field from now on? If so, I think we should document it somewhere. If the ddl field name is decided as name, shall we add a comment somewhere in the code to mention that this will store and will be overwrited by google username?

name will be user's real name reported by google API instead of the username. This is the behavior of the current system which is not changed in this PR.

One comment before reviewing code,

The 'email' and 'name' will be updated when the user login, so the data will be updated as long as the DDL is correct.

Are we implicitly using user's google username as name field from now on? If so, I think we should document it somewhere. If the ddl field name is decided as name, shall we add a comment somewhere in the code to mention that this will store and will be overwrited by google username?

name will be user's real name reported by google API instead of the username. This is the behavior of the current system which is not changed in this PR.

Understood. But It is not exactly the same as the current implementation. The old name field can be either the user's name (when registered with password), or the name reported by google (when registered with google login). Now I believe we do not have the first cases any more. If so, I think it's better to be explicit to document it. One possible place is leave this difference update in the PR description, if no good place in code can be found.

Discussion 2022.12.07: We leave it here.


Multiple Videos Remembered/Sent with Video Analyzer

I am experiencing a strange behavior specific to the video clips analyzed with OpenAI (I realize it may be an OpenAI issue, but wanted to share)
I have an automation that creates a short video clip (8 sec) and then uploads the video for analysis with the llmvision.video_analyzer service. What I am experiencing is that the response seems to include a summary of several of the past events, like there is a ghost of the previous videos. I get the same issue whether the service is run from an automation or Dev Tools.
A typical response is:
response_text: >-
The images show various views of a porch, with some frames featuring a
delivery person and others showing the empty porch at different times of day.
This would suggest to me that some frames are current and others are "old." There was delivery person at one point today, but not when the service was last run (response above) and when I open the video files I only see the current 8 second clip. Could OpenAI be holding on to previous videos analyzed in the past 24 hours?
I do not see this same behavior with images, only videos.
##Yaml Automation
action:

metadata: {}
data:
filename: /config/www/Front.png
target:
entity_id: camera.front_high_resolution_channel
action: camera.snapshot
metadata: {}
data:
duration: 8
lookback: 0
filename: /config/www/Front.mp4
target:
entity_id: camera.front_high_resolution_channel
enabled: true
action: camera.record
delay:
hours: 0
minutes: 0
seconds: 10
milliseconds: 0
enabled: true
metadata: {}
data:
provider: OpenAI
model: gpt-4o-mini
interval: 2
target_width: 1280
max_tokens: 100
temperature: 0.5
video_file: /config/www/Front.mp4
message: describe what is happening in 1 simple sentence
response_variable: response
action: llmvision.video_analyzer
enabled: true
metadata: {}
data:
message: "{{response.response_text}}"
title: Person Detected
action: notify.mobile_app
mode: single

To test further, I changed the camera and rerun the action, with both a different camera and a different filename. Below is the response I received, suggesting OpenAI must be remembering or analyzing multiple videos. The action was performed on a single video file from a single camera.
##YAML##
action: llmvision.video_analyzer
data:
provider: OpenAI
model: gpt-4o-mini
interval: 2
include_filename: false
target_width: 1280
detail: high
max_tokens: 100
temperature: 0.5
video_file: /config/www/Driveway.mp4
message: describe what is happening
##RESPONSE##
response_text: >-
The frames from the video show two distinct locations and activities:

Driveway Scene: Several frames depict a driveway with a black pickup
truck parked on the concrete. The surrounding area features a well-maintained
lawn and a pathway leading to a house. The frames indicate different times,
but the truck remains stationary, suggesting it is parked there for a while.

Porch Scene: Other frames show a porch area where a person, likely a
delivery worker, is seen adjusting their

Thank you very much for bringing this up!
There was an error in the cleanup so the tmp folder with the extracted frames would not be deleted. This will be fixed in the next release.

Awesome! Thanks for you initial and continued work on this, its been a great addition to HA.

The fix is live. Thanks again for pointing it out!


Crash in approx_most_frequent

Reproduce by running single aggregation approx_most_frequent(c0, c1, c2) over attached vector.
approx_most_fr.tar.gz

It's crashing because buckets is too large.  We will need #3159 to limit the range of the argument in fuzzer.

@Yuhta Can we add a check and throw an exception instead of crashing? Otherwise, a client may inadvertently send a "bad" query and crash the cluster.

@mbasmanova It's hard to draw the line here, something working fine on one machine might be OOM on another, and we don't want to put the value too small here to make it too limiting for the user.

buckets is too large

How large is this? Would it be reasonable to limit 'buckets' to 100 (or 1000) entries?

@mbasmanova That would be too restricting.  Java Presto has a limit of 2^30, but on some machines this can still be OOM.

@mbasmanova That would be too restricting. Java Presto has a limit of 2^31, but on some machines this can still be OOM.

Having buckets value that large doesn't seem to make sense? What's the use case? Does Java Presto OOM as well? We have seen cases in the past where Presto Java lacks checks for inputs and allows a bad query to bring down the cluster. We should not "match" that behavior.

@mbasmanova I would suggest we hook this into the memory arbitrageur and turn the input with too large values into something like a VeloxException based on the runtime configuration.


Add target libraries/executables to executable's RPATH

Checklist

[X] Used a personal fork of the feedstock to propose changes
[X] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

xref: https://github.com/conda-forge/cuda-feedstock/issues/10
There is only one DSO lib to patch, libnvidia-ml.so.
Also this file will have a symlink as shown below.
From $PREFIX/lib/stub/,
libnvidia-ml.so -> ../../targets/x86_64-linux/lib/stubs/libnvidia-ml.so*

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.


fix issue when polymorphic inline info layouts differ in JIT and runtime data

Exact order of next pointers for polymorphic CodeGen JIT and CodeGen Runtime data may differ. So, while walking CodeGen JIT data during BuildJITTimeData data we should pass along the head CodeGen Runtime pointer, otherwise inner calls will not be able to find inlinees who appear earlier in the list.

@dotnet-bot test Ubuntu ubuntu_linux_debug_static

@dotnet-bot test Ubuntu ubuntu_linux_release_static

:shipit:


Shouldn't this be reversed?

Probably, yes. Electron-builder creates installers and electron-packager makes packages. Could you please make PR with a fix?

Fixed with https://github.com/loopline-systems/electron-builder/pull/98.

Here you are: https://github.com/loopline-systems/electron-builder/pull/98
I also updated packager.json to builder.json to be aligned with the naming convention, but this might be a bit controversial and it seems that the former name might have been used elsewhere too.
If you'd want me to revert the name of this file (in another PR), let me know!
Thanks :-)

All good with this one. ;)


Add metadata method

@martinytodorov committed those changes - Alex @radhack is testing it locally

@martinytodorov tested by @radhack and it works locally


Fixed #27556 -- Added Oracle support for IsValid function and isvalid lookup.

https://code.djangoproject.com/ticket/27556

buildbot, test on oracle.


Handle 'This application could not be started' error dialog

Handle 'This application could not be started' error dialog
//cc @richlander @NKarnam15 @mairaw

No, I was planning on adding it to the TOC in my next commit.


This product has been removed from the marketplace. The article needs an update to note that, or simply to be removed.

Same as the subject line:
This product has been removed from the marketplace. The article needs an update to note that, or simply to be removed.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 0b348a89-811c-2d1f-62f6-029d67213ef2
Version Independent ID: 7fbb4ab1-701d-c88a-a14f-d29bbadd6981
Content: View of features and epics on the Feature Timeline - Azure DevOps
Content Source: docs/boards/extensions/feature-timeline.md
Service: azure-devops-boards
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

Note that the live page seems to be stuck in a redirect loop at this time.

@mmBPCS, @mdell-seradex -- thank you for your feedback.
@KathrynEE -- please look into this issue.

@mmBPCS - Thank you for bringing this to our attention. I recently deprecated the articles but made a few errors. I've just published the latest changes and the articles are now fully archived and links should now work as expected. Here are the articles for reference:

View progress using the Epic Roadmap
View portfolio progress with the Feature Timeline


Error detection improvments

Improve the error detection to detect uninitialized variables T
Try to detect all errors in a template instead of stopping on the first one encountered
Rename Keys and Values to GetKeys and GetValues on IDictionary object to avoid conflict with potentials key name
Fix a bug causing generated files to have the wrong file mode (0755 instead of 0644)
Fix a bug causing some strings to be modified by the razor parser
Fix a bug with the extract function causing results to not be in the same order as the keys
Use json strings and json dict by default instead of the generic go lists and maps
Add support fot the new go template feature allowing overwrite of variables using the syntax {{ $var = "value" }}
Stop evaluation on @(var).x, now it wiil result in "value.x" instead of trying to access field x of var
Add an option to disable errors on  --accept-no-value or shorter --nv
Fix a bug to detect divide by 0
Add functions to detect nil object (isNil, isNull, isZero, isEmpty)
Add a function to validate assertion  within template, assert
Add a function to raise errors within template, raise
Fix a problem with ellipsis evaluation @func(arg1, arg2...)
Overwrite Sprig list functions to always get serializable list objects
Improve the documentation
Update to go 1.11

Pull Request Test Coverage Report for Build 338

304 of 579 (52.5%)  changed or added relevant lines in 23 files are covered.
5 unchanged lines in 4 files lost coverage.
Overall coverage increased (+2.9%) to 56.264%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

template/math_base.go
3
4
75.0%

template/math_bits.go
0
1
0.0%

collections/string.go
45
46
97.83%

template/extra_runtime.go
10
11
90.91%

template/razor.go
44
45
97.78%

utils/exec.go
0
2
0.0%

json/generated_impl.go
18
24
75.0%

template/func_table.go
0
6
0.0%

collections/implementation/generic.go
14
24
58.33%

yaml/generated_impl.go
14
24
58.33%

Files with Coverage Reduction
New Missed Lines
%

template/slice.go
1
29.94%

utils/exec.go
1
59.34%

template/extra_math.go
1
38.64%

template/template_process.go
2
10.91%

Totals

Change from base Build 335:
2.9%

Covered Lines:
2542

Relevant Lines:
4518

💛  - Coveralls


Improve formatting and responsiveness of all mini-maps

Closes #190.
Xref: #201.
Without this PR, the permafrost, flammability, and vegetation change mini-maps have some clunky behavior at different browser widths, especially if you resize your browser after the page loads:

The width & height do not always stay the same, so the maps can end up more rectangular than square.
The mini-maps and titles sometimes overlap each other if you resize your browser window after the page loads.
Because some mini-map titles are shorter than others, lines breaks can cause height differences that affect the alignment of the mini-map grid.

We decidedly earlier this week that we probably only want to show a report's qualitative text summary for very narrow resolutions (e.g., a smartphone in portrait mode). The mini-map grids look pretty decent at any resolution larger than 600px, so I don't think it's worth the effort to try to lay them out in a vertical column format that we probably will choose not to show anyway. So, I've reworked them to use HTML tables that help us keep both the map titles and maps lined up and separate without resorting to CSS trickery.
Each of the three problems listed above have been fixed.
This PR also adds the model to the title above permafrost MAGT mini-maps (one of the checkboxes on PR #201).

In addition to the requested changes, this PR has been modified to use just flexboxes and CSS media queries to pull off the same effect as the HTML tables without using HTML tables.


feature/handle-click-behind-old-safearea

This PR aim to fix a use case where a top floating flushbar didn't allow user to click on appbar elements.
For example the following screen, you can see a rectangle with red line (which have added). It's coming from the old safeArea. Because of it, it's not possible to click on the bottom of my Icons.home.

With the new system, the safeArea didn't interact with user click which let the user to interact with the background.

Code used to test.
import 'package:another_flushbar/flushbar.dart';
import 'package:flutter/material.dart';

void main() {
  runApp(
    const MaterialApp(
      home: Home(),
    ),
  );
}

class Home extends StatelessWidget {
  const Home({super.key});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        leading: IconButton(
          icon: const Icon(Icons.home),
          onPressed: () {
            print("Home button tapped");
          },
        ),
      ),
      body: Center(
        child: OutlinedButton(
          onPressed: () {
            Flushbar<void>? flush;
            flush = Flushbar<void>(
              borderRadius: const BorderRadius.all(Radius.circular(8.0)),
              reverseAnimationCurve: Curves.decelerate,
              margin: const EdgeInsets.only(
                left: 12,
                right: 12,
                top: kToolbarHeight + 9,
              ),
              padding: const EdgeInsets.symmetric(horizontal: 12),
              messageText: Container(
                key: const ValueKey<String>("message_delete_success"),
                width: double.infinity,
                alignment: Alignment.center,
                child: Row(
                  children: <Widget>[
                    const Expanded(
                      child: Text("THE FLUSHBAR"),
                    ),
                    IconButton(
                      onPressed: () => flush?.dismiss(),
                      icon: const Icon(Icons.close, color: Colors.white),
                    ),
                  ],
                ),
              ),
              isDismissible: false,
              backgroundColor: Colors.orange,
              flushbarPosition: FlushbarPosition.TOP,
              flushbarStyle: FlushbarStyle.FLOATING,
            );

            flush.show(context);
          },
          child: const Text("SHOW FLUSHBAR"),
        ),
      ),
    );
  }
}

Thanks for this . will review

@cmdrootaccess any update ? Many issue are opened and many PR :/

my bad, i have been busy with other projects.
i will find time over the weekend hopefully to address the issues including this pr .

@cmdrootaccess  :) up for this


Kaniko build does not work on rancher installation

Algorithm build in HKube does not work when it is installed in Rancher (possibly permission problem)
Might be related to GoogleContainerTools/kaniko#566

After analysis:
There is a problem with kaniko if the base image was created with docker 18 and k8s runs with docker 17
as long as the base is created with docker 17, or the cluster uses docker 18 it works


#816 Docs : Audited Docs for Copy command

This PR closes Issue : #816
Have changed the docs to be consistent and used  SET Command docs for reference
Do lemme know if there are any changes to be done

Hi @onlybond , Thanks for the changes. DiceDB currently does not support databases hence the argument destination db is not being used by the server. A user can pass the parameter, It will be ignored. Therefore, please update any such reference where it is mentioned that the destination key will be updated in the destination DB.
Thanks

Hey @apoorvyadav1111 , Thanks for reviewing the PR, As you said DiceDB doesn't support databases, so in the docs should i remove all the references which are related to db? So in the parameters only 3 parameters will be given as of now?

So you can remove references which mention that the key will be inserted in database, You can mention in the description about this. But please take that as an argument as we do take it as an argument, but just ignore it. It act as sort of placeholder.

Hi @onlybond, Hope you are well. I am eager to know more about your progress on this issue. Please reach out to me if you need any clarifications.

hello @apoorvyadav1111 I have mentioned about the the key to be stored in another database in the notes. until diceDB supports databases its viable to mention it in the notes section.

hi there I have made the changes. please have a review on it.

All changes done. LGTM


[LUNATIX] Lunatix - Crash on startup

Module Information

Identifier [LUNATIX]
Module Name [Lunatix]
Version [5.3f]

Describe the bug
Critical Exception has occured:
System.OverflowException: Value was either too large or too small for an unsigned byte.
   at System.Number.ThrowOverflowException(TypeCode type)
   at MBBSEmu.HostProcess.ExportedModules.ExportedModuleBase.ProcessEscapeCharacters(ReadOnlySpan`1 inputSpan)
   at MBBSEmu.HostProcess.ExportedModules.ExportedModuleBase.FormatPrintf(ReadOnlySpan`1 stringToParse, UInt16 startingParameterOrdinal, Boolean isVsPrintf)
   at MBBSEmu.HostProcess.ExportedModules.Majorbbs.sprintf()
   at MBBSEmu.HostProcess.ExportedModules.Majorbbs.Invoke(UInt16 ordinal, Boolean offsetsOnly)
   at MBBSEmu.HostProcess.ExecutionUnits.ExecutionUnit.ExternalFunctionDelegate(UInt16 ordinal, UInt16 functionOrdinal)
   at MBBSEmu.CPU.CpuCore.Tick()
   at MBBSEmu.HostProcess.ExecutionUnits.ExecutionUnit.Execute(IntPtr16 entryPoint, UInt16 channelNumber, Boolean simulateCallFar, Boolean bypassState, Queue`1 initialStackValues, UInt16 initialStackPointer)
   at MBBSEmu.Module.MbbsModule.Execute(IntPtr16 entryPoint, UInt16 channelNumber, Boolean simulateCallFar, Boolean bypassSetState, Queue`1 initialStackValues, UInt16 initialStackPointer)
   at MBBSEmu.HostProcess.MbbsHost.AddModule(MbbsModule module)
   at MBBSEmu.Program.Run(String[] args)

To Reproduce
Steps to reproduce the behavior:

Launch MBBSEmu to run Lunatix
Crash at launch time, see error

Expected behavior
Best case: Game runs. Worst case: Game doesn't run, but crash doesn't take down MBBSemu.
Software Information:

OS: Ubuntu 20.04 (64 bit)

Startup crash is caused by a list of issues uncovered now that File Loading/Pathing is fixed:

The current implementation of fnd1st() takes in a wildcard. Need to verify FindFile supporst wildcard, or create a FindDirectory method in the FileHelper and append the wildcard at the end to do a Directory.GetFiles()

The fndblk struct that is passed in isn't being set. Based on looking at Lunatix, this struct is ~43 bytes in size, with the filename starting at byte 30

Implement the fndnxt() ordinal which would find the next file from the previous fnd1st()/fndnxt(), returning 0 if there are no more matching files


Idea: Automatic demo deployment

You can create a github workflow which will deploy demo with the newest version of the theme (to Vercel, or Github Pages).

I will take a look, but there may be breaking changes that need manual tweaks before build

I will take a look, but there may be breaking changes that need manual tweaks before build

Forgot to mention that rebuilds should be run at version bumps, so only well-tested locally stuff will be published. I can make a PR if you don't mind.

sure

Seems that something similar is already implemented. I think you should mention that repo somewhere here in docs.

The demo site was deployed on vercel, triggered by a manual git push.
The idea of automatically triggered deployment at version bumps is actually not achievable, because it may involves copy new items from config.example.toml to config.toml (adding new config items is not a breaking change)


Can't install it through brew…

Today I decided to delete everything related to mailtrackerblocker, just because I was having it duplicated whenever I ran brew upgrade.
Now I'm trying to install it again but Brew reports there's no mailtrackerblocker formulae, nor Cask…
Do I need to clone it from Git?
TiA

That's odd. It's listed on the brew directory.
If you used brew for MailTrackerBlocker in the past, you may have used my tap. I'm still maintaining it but it's safe to remove it in favour of the official tap:  brew untap apparition47.
And no need to git clone. Trying brew update should update your local formulae/cask cache. Otherwise you might need to check on Homebrew's repo for answers: https://github.com/Homebrew/homebrew-core/issues/37724

No luck…
Already ran:
sudo rm -rf brew --cache
and then both for cask and formulae with this results:
❯ brew install --cask mailtrackerblocker
Error: Cask 'mailtrackerblocker' is unavailable: No Cask with this name exists.
❯ brew install mailtrackerblocker
Warning: No available formula or cask with the name "mailtrackerblocker". Did you mean mailcatcher?
==> Searching for similarly named formulae...
This similarly named formula was found:
mailcatcher
To install it, run:
brew install mailcatcher
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.
==> Searching taps on GitHub...
Error: No formulae found in taps.

Happy to hear that!


Custom price lines

Type of PR: enhancement
PR checklist:

[x] Addresses an existing issue: fixes #183
[x] Includes tests
[x] Documentation update

Overview of change:
Implemented custom price lines

@kirchet could you please merge master and then update these lines https://github.com/tradingview/lightweight-charts/blob/6f56aa076c4f711ab07f07557489c0d89f1932a1/codechecks.yml#L6-L8 to make "Build Size" check happy if needed?

@kirchet could you please merge master and then update these lines
https://github.com/tradingview/lightweight-charts/blob/6f56aa076c4f711ab07f07557489c0d89f1932a1/codechecks.yml#L6-L8
to make "Build Size" check happy if needed?

Done


How long does it takes for default training?

System: 4xRTX3090.
Training scripts: Default training scripts:
python -u train_lseg.py --dataset ade20k --data_path ./datasets --batch_size 1 --exp_name lseg_ade20k_l16
--base_lr 0.004 --weight_decay 1e-4 --no-scaleinv --max_epochs 200 --widehead --accumulate_grad_batches 2 --backbone clip_vitl16_384
My system has shown that one epoch takes for 45mins, which is a pretty long time for 200 epochs. Is that a normal procedure? Or we may not need max epochs like that?

Hi @synsin0 ,
Thanks for your interest in LSeg!
As has been mentioned, we used 8 V100 and trained for 200 epochs. I guess, it is because of your batchsize=4, we use batchsize = 4 instead.
Hope this helps.
Best,
Boyi


axom::Array constructor crash on CUDA.

Code like the following resulted in Array::Array trying to initialize elements of a device-allocated array using placement new on the host. The code SEGV'd.
using ExecSpace = axom::CUDA_EXEC<256>;
const int allocatorID = axom::execution_space<ExecSpace>::allocatorID();
axom::Array<int> arr(n, n, allocatorID);

This method calls initialize() with 2 arguments, making the 3rd argument the detault of true, which is to default-construct.
https://github.com/LLNL/axom/blob/70b360815ebed6a25e6ae369bc5efeaa58cacdbc/src/axom/core/Array.hpp#L1084
https://github.com/LLNL/axom/blob/70b360815ebed6a25e6ae369bc5efeaa58cacdbc/src/axom/core/Array.hpp#L1591
I think the root of the problem could be that Array::m_executeOnGPU is not initialized anywhere. Valgrind was logging uninitialized memory in this area and m_executeOnGPU is probably the culprit.
Calling axom::Array(n, n, allocatorID) where allocatorID is a CUDA allocator should not cause a SEGV and it should initialize the data as needed on device.

I'm trying a fix to set m_executeOnGPU based on the memory space, inside of Array::initialize, Array::initialize_from_other, and one of the constructors that calls neither of those methods.

As mentioned previously:
Link to documentation on setting/disabling the Address Translation Services (ATS), and checking if it is enabled/disabled (Point 19): https://lc.llnl.gov/confluence/display/SIERRA/Quickstart+Guide

Thanks @BradWhitlock . We can have @publixsubfan look into this to make sure other issues don't occur.

Update. I've had some trouble reproducing the crash on develop. The Array::m_executeOnGPU member is uninitialized but it does not seem to matter much. When it fails in my branch, it seems like some bad optimization might be at work. I was getting the allocatorID to pass from execution_space::allocatorID() and it seemed (in Totalview) that the allocatorID was getting optimized out. If I make it "volatile" to prevent inlining then I can see it returns 3 and it works normally. The code resembles:
void buildShapeMap(axom::ArrayView<axom::IndexType> &values, axom::ArrayView<axom::IndexType> &ids, int allocatorID)
{
  const axom::IndexType n = // get the size
  values = axom::Array<IndexType>(n, n, allocatorID);
  ids = axom::Array<IndexType>(n, n, allocatorID);
  // Fill values, ids here
}
...

/*volatile*/ int allocatorID = axom::execution_space<ExecSpace>::allocatorID();
axom::Array<axom::IndexType> values, ids;
buildShapeMap(values, ids, allocatorID);

Yes, I believe we need to initialize m_executeOnGPU to an appropriate default value. Good catch @BradWhitlock.
But is this happening with CUDA device-only memory? The value of that variable should be immaterial -- we should be passing through to special logic for that case.


Should not download same files everytime

Should not download same files everytime when call criteria.loadModel();

Can you set DJL cache folder to somewhere else? See: https://docs.djl.ai/docs/development/cache_management.html
In some weird windows machine, Java failed to move tmp files. You should see some Permission error when this happen. The work around is to use a different cache folder.

Ok now, it just use different folder for tmp file. It use the same folder after downloaded.


Fix dag visualization with Var wires

Summary
This commit fixes the dag visualization for DAGs with classical variables. The Var type was not handled in the attribute callback functions for nodes and edges. This was causing the visualizer to fail if the dag contained these types. This fixes this by adding explicit handling for the Var types and using the name attribute of the Var object.
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>9
Details

0 of 6 (0.0%)  changed or added relevant lines in 1 file are covered.
20 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.02%) to 89.754%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

qiskit/visualization/dag_visualization.py
0
6
0.0%

Files with Coverage Reduction
New Missed Lines
%

qiskit/transpiler/passes/synthesis/unitary_synthesis.py
2
88.39%

crates/qasm2/src/lex.rs
5
92.37%

crates/qasm2/src/parse.rs
6
97.61%

qiskit/synthesis/two_qubit/xx_decompose/decomposer.py
7
90.84%

Totals

Change from base Build<PHONE_NUMBER>9:
-0.02%

Covered Lines:
66953

Relevant Lines:
74596

💛  - Coveralls

I'm skeptical of a test here, mostly because it's a visualization test and the only thing I can do besides asserting it doesn't fail is compare the output visualization. I'm not a big fan of image diff tests, and in this case will be very tricky for me to write because I have a newer version of graphviz installed locally and none of the existing graphviz based visualization tests pass locally.

For visualisation, I'm basically fine just asserting that it doesn't fail for the reasons you give. It would be good to make sure that the path is at least run in the tests.

@Mergifyio backport stable/1.1 stable/1.2


Edits to 'Modification of host cellular component' branch

Usually viruses/symbionts dirsupt, not only modify host cellular components. Here are all the 'annotations to modification' terms:

[x] GO:0098865 modification by symbiont of host bicellular tight junctions -> 1 paper, disruption: change label

[x] GO:0098864 modification by symbiont of host tight cell-cell junction
** PMID:24287273 disruption
** PMID:25838979 CACAO: "Our findings have uncovered a novel role of the bacterial protein AvrA in suppressing the inflammatory response of the host through JNK-regulated blockage of epithelial cell barrier function. " -> ok leave as 'modification by symbiont of host tight cell-cell junction'

[x] GO:0052111 modification by symbiont of host structure-> used to mean 'induction of host (phagosome) membrane tubulation' annotated by Antonia  PMID:18996344 -> changed to modification by symbiont of host cell membrane
[ ] AI Create new term modulation of endomembrane organization in another organism

[x]  GO:0052025 modification by symbiont of host cell membrane -> ok no more annotations

PMID:16085396 MTB ->  inhibits macrophage phagocytosis -> changed to "GO:0052067 antiphagocytosis"

GO:0052043 modification by symbiont of host cellular component	-> 5 CACAO to 3 papers:

PMID:24304930 -> formation of replication-associated inclusion bodies

[ ] NTR modification of host intracellular structure for assembly of symbiont replication compartment
[ ] NTR viral factory assembly
[ ] Check definition: GO:0046809 replication compartment - too specific (VZ term)

[ ]  PMID:10913092 -> Vibrio fischeri lux genes play an important role in colonization and development of the host light organ. -> probably need new terms
[ ] NTR light-emitting organ development (for host)
is_a animal organ development
[ ] NTR induction of host light-emitting organ development
is_a modification of host intracellular structure for assembly of symbiont replication compartment

[ ]   PMID:24188660
-> missing MF 'toxin activity' (binary toxins consist of binding component and enzymatic component. The binding component recognizes a cell surface receptor and allows the internalization of the enzymatic component into the cytosol, and the enzymatic component catalyzes the reaction and induces the toxicity)
-> BP: killing of cells of another organism ("Binary toxin causes insect midgut CF-203 cells death via necrosis", so, not cytolysis)
-> "Confocal microscopy revealed that Plu1961/Plu1962 caused a notable decrease in cellular tubulin of CF-203 cells. " -> disruption by symbiont of host cytoskeleton (or, do we need 'host actin' and 'host microtubule' cytoskeleton?)

@dsiegele @genegodbold for a future discussion

Template definiton:
The process by which a symbiont effects a change in the structure or function of a xx of the host.

[ ] GO:0052008 disruption by symbiont of host cellular component  > obsolete and replaced by parent GO:0052043 modification by symbiont of host cellular component

[ ] check all terms in that branch for 'disruption' and that all definitions are standardized


util/tracing, sql: expose span's goroutine IDs

PR: https://github.com/cockroachdb/cockroach/pull/59717
From release notes:

Added the goroutine_id column to the crdb_internal.node_inflight_trace_spans virtual table that represents the goroutine ID associated with a particular span. [#59717][#59717] {% comment %}doc{% endcomment %}

Fixed by https://github.com/cockroachdb/docs/pull/8909.


Translate to pt-BR

Add pt-BR translation to the project

https://github.com/bancodobrasil/mtls-best-friend/pull/5


Hosts entries appended incorrectly to hosts file on Mac

Using 1.0.2 on a Mac, I found that if there is no blank line at the end of the hosts file, the plugin simply appends to the last line, rather than adding a new line.  If there is a blank line already there, it appends to the blank line.  MacOS 10.12.6

@Kellman3000 Can you test with version 1.1.0? If it is still an issue I will find out how to fix it.

I'm seeing this issue with <IP_ADDRESS> on Mac High Sierra.

I'm also seeing this issue with <IP_ADDRESS> on Mac High Sierra (10.13.5)

Also seeing this with <IP_ADDRESS> on Mac High Sierra.

I would think it wouldn't be an issue unless there needs to be a \r\n?  See this part of the code for the plugin:
https://github.com/cogitatio/vagrant-hostsupdater/blob/master/lib/vagrant-hostsupdater/HostsUpdater.rb#L156-L161

I think I've figured out why this is happening. That code you linked to is never actually invoked on most systems, because the hosts file is owned by root and not writeable.
FYI, this is easily reproducible on a standard Linux/Mac OS by saving your hosts file without a new line at the end, and then running vagrant up with hostsupdater enabled.
I haven't quite figured out how to solve this. The problem is that if you fix the conditional to actually work as intended, your hosts file quickly blows up with a whole bunch of trailing newlines. I'm not sure the best way to prevent those.

I've opened a PR to fix this.
Note that this may actually be a dangerous bug. When vagrant hostsupdater appends to a hosts file without a newline, its output gets concatenated with the last line of the host file. Then when hostsupdater removes its lines, it would probably remove what used to be the last line of the file. That's a pretty destructive maneuver :fearful:

I'll take a look at #165 within the upcoming week. hostsupdater looks for the GUID to remove the line. So it will not matter where the line is in the hosts file - it does not remove the last line.

@cgsmith actually I just confirmed via a test that Vagrant Hostsupdater will destructively modify your hosts file if it does not contain a trailing newline.
Easy to test, just add a line to the end of your hosts file that you don't care about and save it without a newline:
<IP_ADDRESS> foobar
Now run vagrant up on something, and examine the hosts file with cat -A /etc/hosts:
<IP_ADDRESS> foobar<IP_ADDRESS>  local.baz.com  # VAGRANT: 95a9b2d48c7c4dfd09aeb13cee9c1d63 (bwd) / 797b93b2-98b4-4784-a529-89565f136b37$
Now run vagrant halt, and poof the foobar entry is removed from your hosts file.

I see what you mean. Thanks for the clarification


[fireurld.service] remove old socket

Solves #15

It is a workaround and signals seems the right way to handle different init systems / autostart imho. No idea about how to handle cleanup after a SIGKILL right now but I believe there are some possible solution.
To be honest I opened this pull request because I adopted the arch package which shipped its own service file with this workaround and I'd rather have it maintained here by the community.
Maybe a compromise could be to push this workaround for now and keep the Issue open to work on a better solution.

arch package

wait, there's an Arch package for it?
... https://aur.archlinux.org/packages/fireurl
need to update the readme

Maybe a compromise could be to push this workaround for now and keep the Issue open to work on a better solution.

Yes. Still we can evaluate Pre or Post cleanup (+ Restart if we want).
I'll wait some time to give other chance to comment and merge it then.

wait, there's an Arch package for it?
... https://aur.archlinux.org/packages/fireurl

sorry, aur package, didn't mean to spook :x

Yes. Still we can evaluate Pre or Post cleanup (+ Restart if we want).
I'll wait some time to give other chance to comment and merge it then.

agreed

sorry, aur package, didn't mean to spook :x

No I meant Arch/AUR I.e. packaged somewhere even launchpad or cope would count.

ExecStartPre could take over a path that is actually used by a fireurld that was started by a different mechanism than systemd, say autostart.

This is true for manually deleting it as well, which seems to be the current option.
Actual graceful shutdown on a signal seems like the correct solution long term but this seems like the most reasonable stopgap.


Propagate OIDC Config for Nutanix

Ensure OIDC config is propogated to KubeAdmControlPlane via
API Server extra args for the Nutanix provider.

/ok-to-test

Uncommented the 1.25 e2e test as other 1.25 tests have now been merged too.


option to override remote image expire

This PR:
Gives user the option to override the remote image hosting's expire date.
imageproxy -override-cache-control="public" -override-expires="Sat, 28 Dec 2199 04:09:32 GMT"

closes https://github.com/willnorris/imageproxy/issues/207

I'll take a closer look at thing later, but my initial thought is that we should probably expose this through a "minCacheDuration" flag using flag.Duration.  Then, we only use that value if it is longer than the cache duration from the origin server.  (We can also set cache-control: public if that's necessary, though I can't remember the behavior of the httpcache package.)
Also, I'll probably want to put some kind of warning in the documentation for this, noting that it is overriding the instructions from the origin server and could cause problems if the origin is not expecting the image to be cached.
See also previous related feature requests: #28 #144

@willnorris that sounds reasonable to automatically set it to "public"
and also good idea to print a warning to the user


autoReset and autoReset for individual sensors

Hi there,
Very useful plugin, thank you! Would it be possible to make the autoreset and autoreset delay functions available for individual sensors? I’m using 2 sensors with this plugin, and want to have diffrent autoresetdelay values for each sensor. would it be possible to look at?
thank you

Hi @teymur1988, apologies for taking so long to respond to your issue.
I agree that having individual autoReset and autoResetDelay fields for each sensor would be useful.
Unfortunately, as I do not use this plugin myself anymore, it is a bit trickier for me to test real-world implementation of changes. I also don't have the spare time to experiment with it manually at the moment.
Please feel free to fork this plugin and make the necessary changes, then I'd be happy to check over the changes in a pull request before merging. If not, this is something I may look into sometime in the future.


Add autocomplete to the users name

in the highlight user form

Using jQuery UI autocomplete plugin: https://github.com/Dataninja/edgesense/commit/2062e7108ddc4d92a792793a6b04707de622ca72

Done in last PR.


JWT authenticator

Add a JWT authenticator so that the Authorization headers get sent when accessing a protected API

Done


Wait for tcpdump to start listening

What this PR does / why we need it:
Installation and execution of tcpdump in the mirroring test is executed
async. On a slower setup, the installation of tcpdump may take more time
than the execution of ping inside the test - causing missed ping reply
and failing the test.
With this patch, we wait for tcpdump to start recording traffic before
the test continues its execution.
This pathc also removes waiting for the tcpdump file to get created,
assuming it is not needed with the wait now happening in preparation
phase.
Special notes for your reviewer:
Release note:

NONE

/lgtm


time_locked needs to be implemented for accounts

Include time_locked informations in our account ledger implementation.

Once we have this, I think we'll be able to produce staking ledgers from the indexer

Possibly related to #496

Blocks #496


Add three new methods to GLTFDocumentExtension and document all of its methods

Implements and closes https://github.com/godotengine/godot-proposals/issues/3305 (at least, most of what matters, and all for now).
This PR does three things:

Re-order the methods in GLTFDocumentExtension to be in the order they are executed in (for easy readability).
Add documentation for all methods in GLTFDocumentExtension (and the below new ones).
Add three new methods to GLTFDocumentExtension:

parse_node_extensions runs when parsing the node extensions of a GLTFNode. This method can be used to process the extension JSON data into a format that can be used by generate_scene_node.
generate_scene_node runs when generating a Godot scene node from a GLTFNode. The returned node will be added to the scene tree. Multiple nodes can be generated in this step if they are added as a child of the returned node; GLTFDocument will handle setting ownership of those children.
convert_scene_node runs when converting the data from a Godot scene node. This method can be used to process the Godot scene node data into a format that can be used by export_node.
The creation of these methods were driven by practical use cases, including physics and audio nodes. With these methods in place, we could also in theory move the light code to a GLTFDocumentExtension class instead of having it be directly in GLTFDocument, but I discussed with @fire and there is no strong reason to do this.

The signatures of the new methods look like this:
virtual Error parse_node_extensions(Ref<GLTFState> p_state, Ref<GLTFNode> p_gltf_node, Dictionary &p_extensions);
virtual Node3D *generate_scene_node(Ref<GLTFState> p_state, Ref<GLTFNode> p_gltf_node, Node *p_scene_parent);
virtual void convert_scene_node(Ref<GLTFState> p_state, Ref<GLTFNode> p_gltf_node, Node *p_scene_node);

Thanks!


Find-GcsObject throws errors

The example at http://googlecloudplatform.github.io/google-cloud-powershell/#/google-cloud-storage has an error in it.  specifically, the line below
Find-GcsObject -Bucket $bucket
should instead read
Get-GcsObject -Bucket $bucket
There is no Find-GcsObject Cmdlet listed at the link below
http://googlecloudplatform.github.io/google-cloud-powershell/#/google-cloud-storage/GcsObject

Thank you for pointing this out.
The documentation has now been fixed.


same deps in 2 different package.json files

Why "hexo": "^3.2.2" and "hexo-admin": "^2.1.0" dependencies are added in 2 different package.json files? (main's project package.json and hexo's package.json)

I tried extracting all dependencies to the main project's package.json, but then Hexo failed to process files. Hexo seems to require that the /node_modules folder to be at the hexo root for it to work properly.


Update package.json

Add "use-reducer-async": "^2.0.1"

Oops, missed this one. No longer needed due to #6


Bi-weekly auto PR from refs/heads/dev into main branch

Summary

Bi weekly automatic PR from dev to main branch

All test passed successfully


dist: redhat: reduce log spam from unpacking sources when building rpm

rpmbuild defaults to logging the name of every file it unpacks from
the archive. This is quite a lot for Java applications.
Make it quiet with the %setup -q flag.

Queued as 113c7d993b038e13fd34a4a40b2aea7e324507f8


x/tools/go/packages: does not obey $GOROOT

I noticed today the go/packages was executing the version of Go that is on my path, not the version that I was setting in $GOROOT. go/packages should probably obey $GOROOT.

This has been decided against in the past: https://github.com/golang/go/issues/26845

This issue can probably be closed


New functions

New commands and HSR content

I will be working on add profile cards by enka

Added a /profile command. Will extend some commands to HSR when I have free time


A question about obstacle avoidance

Hi, Alex
I have 2 questions:

In the track constraints of the MPCC problem quadratic approximation:

for any k, the track boundary (corridor) is computed according to the theta of the corresponding initial guess point. But obviously, the solution might be different from the initial guess, so at the kth point of the solution, the theta changes, thus the track boundary is actually not the value used in our constraints.
So it seems that even when the track constraints are satisfied, the predicted trajectory may have collision with obstacles? How do you deal with this situation?
In obstacle avoidance, the vehicle is treated as a point without considering its heading. Does this matter when the corridor becomes narrow? Sometimes even if the point stays within the corridor, part of the vehicle might be out of the corridor because of its length and heading.
Thanks!

Since the track constraints are approximated at each step in the SQP scheme, if the SQP converges, the track constraints are fulfilled. But if you stop early (as we do here) you could be right, it very much depends on how good a linear approximation works for your actual constraints.

To deal with this at the moment I use a safety margin. The issue is that the safety margin makes the problem conservative. So there is a certain trade-off. It would be interesting to implement orientation dependent constraints, but I did not yet found the time to do it.

Thanks for your answer :smile:


New theme + New plugin

I added a new theme called "ys-pep", based on the "ys" theme which I loved -- but found lacking in several aspects.
I also add a plugin for better experience in using yubikey dongles.

We can't accept more themes. Please make a separate repo with your theme and link to it from the external themes wiki page.
You can also add your theme as a custom theme. See customization for instructions.


Outdated Firefox versions on nightly, desktop and mobile

Only Firefox 117.0 is covered by tests for release or nightly versions.

Hi! Good to here from you. Sorry for the delay -- dealing with a bug but I hope to have the latest tests up soon.

Good to see you here :) No worries at all and thanks for running this great framework

Latest tests have now been posted with Firefox 118.0 or 118.1 on desktop, android, iOS.

Thanks! Nightly is still 117 though? https://privacytests.org/nightly-private.html

Oh! It seems I have a bigger update problem than I thought. Also I need some way to automatically detect these problems. Thanks for raising this issue.

I updated the Firefox Nightly version so it's up to date in the latest issue of PrivacyTests. And I was very happy to see that the Blob API is now partitioned! https://twitter.com/privacytests/status/1712280973529043174
I will keep this issue open until I have better detection of out-of-date browsers.


[cartservice] Bump OTel .NET SDK to 1.9.0

Changes
[cartservice] Bump OTel .NET SDK to 1.9.0 together with other dependencies
Merge Requirements
For new features contributions please make sure you have completed the following
essential items:

[x] CHANGELOG.md updated to document new feature additions
[x] Appropriate documentation updates in the docs - https://github.com/open-telemetry/opentelemetry.io/pull/4704
[ ] Appropriate Helm chart updates in the helm-charts N/A

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

hmmm... we need to clean up the changelog to align with the 1.10.0 release that was just done. I'll need you to update this PR  to get the changelog entry done properly.

Fixed. I think that release process should be extended by updating CHANGELOG. Based on my experience, it is much easier to follow such process by particular repo mainteiners that the unreguleral contributors.


PHP support missing

Current version of extension doesn't work when mixing code in PHP files.

Adding php to documentSelector option doesn't make much sense, since extension doesn't know how to ignore embedded PHP (<?php, <?, <?=, ?>, heredoc opening <<<HTML and closing \nHTML;) tags inside php files when mixing with html code, which is usual for PHP coding. That's why errors, that doesn't related to html, appears for embedded tags every time linter is running with message Special characters must be escaped.
Regex pattern for excluding php tags and its content should run before pass document to linter.
It is very important for php coding.
I hope you will fix this as soon as possible to make every programmer much happier.
Thank you very much!

Linter doesn't work for unsaved php files (and html too), when you just started typing or copy pasting code.

Linter doesn't start to check php file until any other html document is open. That's very annoying, especially when you aren't using html documents in projects at all.

OS: 10.11.6 (El Capitan).
VS: 1.29.1

+1 for this


Source file '\Binding\SkiaSharp.Portable\SkiaPortable.cs' could not be found.

Hello,
Il tried to test the 'SkiaSharpSource.Windows' solution but I can't build: I get an error saying that the class 'SkiaPortable.cs' is missing. Is it normal?

Hi @Goldstrike. this is expected because that file is generated from the binding projects.
In order to generate that file, you need to run this build target:
.\bootstrapper.ps1 -Target externals-genapi

That will create the file. In order to build so that you can actually run the tests or platform libraries, you will need to first build the native library and managed wrappers:
.\bootstrapper.ps1 -Target libs


Use volume mounts for docker/podman

Use volume mounts to avoid needing to rebuild images when making configuration changes. This can be simplified from what David Edell prototyped into just documentation (or commented lines in the compose file) to aide developers to preserve the self-contained built images approach Brian Sipos mentioned while still providing a reference for easy usage when needed. The ion configuration file is probably the single file most likely to need overriding for specific tests.

This represents an explanation within the product guide about how to do this on a development host. The deployment host won't not have the full source tree available.


MNT: rerender

Hi! This is the friendly automated conda-forge-webservice.
I've rerendered the recipe as instructed in #1.
Here's a checklist to do before merging.

[ ] Bump the build number if needed.

Fixes #1

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Let me fix the CI. Looks like we still need a custom CBC file to kick it off...

Thanks @leofang, because I'm clueless :man_facepalming:


Add functions getCanvas and getImage

Added new methods getCanvas and getImage to handle the situation where Jeff is used inside an electron application.

Please check latest changes as per your comments.

lgtm


fix: memory leakage

Summary
FIX memory leakage
timer.After: https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/time/sleep.go;l=154-159
Checklist

[ ] reference any related issues
[ ] updated docs
[ ] updated unit tests
[ ] updated UPGRADING.md
[x] add appropriate tag (improvement / bug / etc)
[ ] ready for review

I don't think this will work. Reading the docs:

It is not safe to manipulate the
provided backoff policy (notably calling NextBackOff or Reset)
while the ticker is running.

We call .Reset.
time.After can leak, but only if it doesn't fire:

The underlying Timer is not recovered by the garbage collector until the timer fires.

In the case of this code I believe the timer will almost always fire since ctx.Done happens when the program terminates.
Were you seeing a memory leak in practice? Maybe there's some other cause?


USERGUIDE.md is 404ing

Small thing, but from the pypi page, when I click the link for the user guide, I get 404. The userguide here in gh seems to be fine.
https://pypi.org/project/hdx-cli-toolkit/USERGUIDE.md

Yes, this is actually a duplicate of #40! I think I decided fixing the link wasn't worth a whole release. Maybe once you finished finding issues I'll do a little documentation/bugfix release.

Resolved in #46


feat: support controller username and password

On the websocket channel previously we only supported sending tokens with this PR we can send a username and password from the eda-server. The username and password can be created in the RH AAP Credential Type.
https://issues.redhat.com/browse/AAP-22060

e2e test in https://github.com/ansible/eda-qa/pull/332


Do not initialize pair HMM when we have zero reads

@davidbenjamin please review

Great job tracking down the source of this error @takutosato!

thanks!


Update README.md

Added myself to list of members and hyperlinks for example projects within SIG-Runtime scope.

/lgtm
:)


It is possible to call masonry methods ?

Hello I would like to know if there is a way to call the http://masonry.desandro.com/methods.html from the directive ? for example i would like to have layoutComplete method.
thanks!

Great idea! When I use angular-masonry with infinite scroll, the whole grid updates which causes flickering.
Does anyone have solution until the directive supports methods?

any news on this matter?

@rbosneag Still taking PRs for this. :)

+1 - I often need to trigger a re-arrange after changing brick classes that control width.

@skattabrain
Unfortunately the only way I found to trigger reload is by broadcasting on the $rootScope, but it does work:
$rootScope.$broadcast('masonry.reload');
you can add other methods by modifying the masonry directive and add a listener, for example: scope.$on('masonry.destroy', ctrl.destroy);
Not ideal, but works.

Is there an update regarding calling methods? Would be very useful! This would for example solve problems/workarounds with calling the masonry rearange method.

+1 we all got the same problem

found the need for this, since I am doing drag drops. I don't see any pull requests addressing this.


Integrate a different hand

Hi @tylerlum
Thanks for your excellent work!
I try to integrate a different hand into the dataset generation code, and like yours, I added 6 joints in the virtual_joints.urdf.
However, when I generate pre-grasp pose, I encountered the following problem.
Could you please help me with it? Thank you!

Glad you are using the dataset generation code. I'd be happy to help.
Could you please send me the following?

the .urdf file you are using
the command you are running

Also, could you try modifying this https://github.com/tylerlum/get_a_grip/blob/main/get_a_grip/dataset_generation/scripts/generate_hand_config_dicts.py#L493 by adding raise so that it doesn't continue but raises the exception.
Replace
    except Exception as e:
        print(f"Exception: {e}")
        print(f"Skipping {object_codes} and continuing")

with
    except Exception as e:
        print(f"Exception: {e}")
        raise

I will add some more documentation about adding a new hand based on this.
Do you have one urdf with virtual joints and one without?
https://github.com/tylerlum/get_a_grip/blob/main/get_a_grip/dataset_generation/utils/leap_hand_info.py
^ this has both regular urdf and virutal joints urdf. Make sure you have both!

Thanks for your reply!
The .urdf file and command I use are listed below.
Also, I create the urdf with virtual joints just like what you did to the Allegro hand.
Thank you!
python get_a_grip/dataset_generation/scripts/generate_hand_config_dicts.py \ --meshdata_root_path ${MESHDATA_ROOT_PATH} \ --input_object_code_and_scales_txt_path data/dataset/${DATASET_NAME}/nerfdata_settled_successes.txt \ --output_hand_config_dicts_path data/dataset/${DATASET_NAME}/lxqs_hand_config_dicts \ --hand_model_type LXQS
lefthand_10dof.zip

I have added a rough guide to adding your own hand here: https://github.com/tylerlum/get_a_grip/blob/main/docs/custom_hand.md
Hope this helps!
If you are still stuck, could you adding the raise as I described above and show me the output?


Fix all initializer list ordering

There are a ton of warnings generated by gcc's -Wreorder. The warning's meaning is explained here. Might not be any actual bugs, but it'd be nice to be on the safe side.
Working branch

I have seen that there are a lot of -Wreorder warnings generated inside /lib/ files.
I think that is not a good idea to modify this files directly, because they will be overwritten if we update these libraries in future.
What do you think?

Yeah, third party libraries shouldn't be modified unless absolutely necessary to integrate with the engine that otherwise wouldn't be useable.

What @dottools said. I've mentioned before the possibility of using git submodules for our lib dependencies, at least those of them for which that's feasible - in that case maybe we could contribute back to those dependencies, maybe fix some warnings, etc. But that might not be a realistic ideal :p.


refactor: use YAML plan with delimiter in KCLVM Rust ver

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[x] N
[ ] Y

2. What is the scope of this PR (e.g. component or file name):

kclvm/runtime/src/value/val_plan.rs
kclvm/src/main.rs
3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[x] Other

use YAML plan with delimiter in KCLVM Rust ver
4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

[x] N
[ ] Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

[ ] Unit test
[ ] Integration test
[ ] Benchmark (add benchmark stats below)
[ ] Manual test (add detailed scripts or steps below)
[x] Other

6. Release note

Please refer to Release Notes Language Style Guide to write a quality release note.
None

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 38 (0.0%)  changed or added relevant lines in 1 file are covered.
3 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.6%) to 59.363%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

kclvm/runtime/src/value/val_plan.rs
0
38
0.0%

Files with Coverage Reduction
New Missed Lines
%

kclvm/runtime/src/value/val_plan.rs
3
0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.6%

Covered Lines:
21947

Relevant Lines:
36971

💛  - Coveralls


Package Sync

Pushing #222 a bit further to see if anything breaks. This

Removes a couple of controlling environment variables, which should simplify local development as well as CI workflows. Specifically TEST_UPSTREAM has been folded into COILED_RUNTIME_VERSION, which can be a version number, "upstream", or "latest", and COILED_SOFTWARE_VERSION is no longer necessary in a package_sync universe.
Removes the setup/cleanup jobs. This also means there are fewer artifacts passed around between jobs, which is nice.
"upstream" is now run on pull requests

It seems that something did break: package_sync currently doesn't support direct URLs to conda packages, which we currently use to point to nightly dask/distributed builds.

I'm happy with the state of package sync and can give a solid thumbs up to merging this!

Just pushed an empty commit to rerun CI (we were getting AWS limit errors in the most recent CI run)

We're getting
 E               coiled.errors.ClusterCreationError: Cluster status is error (reason: Scheduler Stopped -> Errors installing conda packages:
E               package pyarrow-8.0.0-py310hbc2c91e_3_cuda requires arrow-cpp 8.0.0 py310h1f609fe_3_cuda, but none of the providers can be installed) (cluster_id: 50389)

in this CI build cc @shughes-uk

Having to boot my up my x86 macbook for the first time in months to solve this one, not quite sure what's going on!

pyarrow and arrow-cpp was having some difficulty, i've solved it by allowing arrow-cpp to be a more unconstrained. This seems ok as I'm pretty sure only pyarrow depends on arrow-cpp and nobody is really installing arrow-cpp directly

@ian-r-rose I think this might require sorting in the coiled runtime, at first I thought it was a package sync issue but it's actually a symptom of it being hard to sync cross platform
On macos the coiled runtime will use pyarrow 8.0, which fails to install on the cluster
On linux the coiled runtime uses pyarrow 7.0
So macos users I think are going to have a very bad time using pyarrow + the coiled runtime unless it's either pinned to ~7 or ~9.
9.0 seems to work.

The source seems to be a pin on pyarrow 8. With that pin the macos environment generated can't solve properly for linux64, not sure which dep, looking now

Got it working! 🥂

Got it working! clinking_glasses

Woo!

I'm suspicious that's some kind of transient network error. I'll have to ensure they get printed out properly as It looks like it got eaten

This seems to be all green, though I would note that there was a solid amount of mamba/conda/mysterious flakiness that makes me a bit nervous.

I'll add some changes so I can see the flakyness serverside and hopefully push fixes fast. Yell at me in the mean time

Thanks for the empty commit @ian-r-rose -- I think sanity checks like that are a good thing to do at this point. @shughes-uk
ERROR    coiled:states.py:134    | Worker Process         | test_coiled-5b0e15b6-worker-3ee9bf36b8         | error      at 16:48:59 (UTC) | Errors installing conda packages:
nothing provides requested tzdata 2022rc0

is currently popping up

Interesting. That package was brand new yesterday. I think you might have had the bad luck to have tried things exactly after release of the package, when it was propagated to the CDN edge node serving CI but not the one used by the cluster. If i'm right re-running now should work

I was not correct, looks like an issue dealing with the way the version gets parsed!

Just dropping a note that I've spent a fair amount of time the last few days looking at performance testing and looking at some specific performance regressions, and this PR has been invaluable. I keep on cherry-picking commits on top of it.

this PR has been invaluable. I keep on cherry-picking commits on top of it

+1. I wrote the entirety of https://github.com/coiled/coiled-runtime/pull/243 on top of this. I never used the a non-package-sync cluster. I still don't know how you would; I never learned what incantations of coiled-runtime / packages / environment variables needed to be there to get the test suite to figure out what software environment you were trying to use (and how to make your local environment actually match that).
I couldn't imagine trying to develop in this repo without package_sync. Especially as a non-primary developer in this repo who just wants to come in, add some tests, and move on without too much fuss, it's invaluable. I expect this would reflect the experience of package_sync and software environments in general for most Coiled users working in a team, btw.

@gjoseph92 see https://github.com/coiled/coiled-runtime/issues/227#issuecomment-1220830678

Still seeing the same issue with tzdata. Any updates on package_sync @shughes-uk?

I have a fix for this issue, release should be today. Was going to be yesterday but our deploy process blew up a bit

Looks like just that "unknown mamba error" flakyness. I'll try flush out any reliability issues with some 1000 worker clusters monday morning. Seems like it's about 2-3/1000 currently

Squashed some flakyness with mamba with a big hammer of retrying things. Current test failure is something in distributed unrelated to package sync I think?

Current test failure is something in distributed unrelated to package sync I think?

Yep, I'm going to try to re-roll a bunch of times tomorrow to see if anything comes up, but if package sync seems solid, then I think I'd advocate for merging

Another unknown mamba error: https://github.com/coiled/coiled-runtime/runs/7964668431?check_suite_focus=true

Another unknown mamba error: https://github.com/coiled/coiled-runtime/runs/7964668431?check_suite_focus=true

Code isn't live yet, i can get it live tomorrow

This has actually been fairly stable for the last day or so. If you drill into all of the red X's, you'll find the failures are mostly due to a (likely real) regression and flakiness in the new spill tests (cf. #229 )

Opened an issue for that mamba install error.  https://github.com/mamba-org/mamba/issues/1890, might have to be more aggressive with retrying in the meantime (we only really retry once)

I know that @shughes-uk was curious about startup times. Here are a couple of examples of the slowdown generated by including package_sync, it is approximately 3 times slower. Where the 1st element in the latest_duration tuple is the with package_sync. The other two were not, and the threshold is in this case (avg until this PR run + 3 std)
runtime = 'coiled-0.0.4-py3.9', name = 'test_default_cluster_spinup_time', category = 'benchmarks', last_three_duration [s] =<PHONE_NUMBER>467041,<PHONE_NUMBER>258606,<PHONE_NUMBER>1061096), duration_threshold [s] =<PHONE_NUMBER>72597
runtime = 'coiled-upstream-py3.9', name = 'test_default_cluster_spinup_time', category = 'benchmarks', last_three_duration [s] =<PHONE_NUMBER>2541504,<PHONE_NUMBER>7651978, 98.0114254951477), duration_threshold [s] =<PHONE_NUMBER>0766282
runtime = 'coiled-0.1.0-py3.9', name = 'test_default_cluster_spinup_time', category = 'benchmarks', last_three_duration [s] =<PHONE_NUMBER>949951,<PHONE_NUMBER>1072083,<PHONE_NUMBER>0151367), duration_threshold [s] =<PHONE_NUMBER>839115

This were extracted from here https://github.com/coiled/coiled-runtime/runs/8104127629?check_suite_focus=true

Thanks @ncclementi , package sync start time is what i'm actively working on and have some promising stuff for it to hopefully cut it down dramatically

@ncclementi is the exact same set of packages installed on the cluster via package_sync as were in the senv? I'd be really curious if package_sync is pulling in any extra stuff (jupyterlab, pytest, etc.) that isn't actually needed on the cluster, but happens to be installed in the environment that's driving the cluster.

Non Package sync environment
- _libgcc_mutex=0.1
- _openmp_mutex=4.5
- _py-xgboost-mutex=2.0
- abseil-cpp=20210324.2
- aiobotocore=2.3.4
- aiohttp=3.8.1
- aioitertools=0.10.0
- aiosignal=1.2.0
- alembic=1.8.1
- anyio=3.6.1
- argon2-cffi-bindings=21.2.0
- argon2-cffi=21.3.0
- arrow-cpp=8.0.0
- asciitree=0.3.3
- asttokens=2.0.8
- async-timeout=4.0.2
- attrs=22.1.0
- aws-c-cal=0.5.11
- aws-c-common=0.6.2
- aws-c-event-stream=0.2.7
- aws-c-io=0.10.5
- aws-checksums=0.1.11
- aws-sdk-cpp=1.8.186
- babel=2.10.3
- backcall=0.2.0
- backoff=1.11.1
- backports.functools_lru_cache=1.6.4
- backports=1.0
- beautifulsoup4=4.11.1
- bleach=5.0.1
- blinker=1.4
- blosc=1.21.1
- bokeh=2.4.3
- boto3=1.21.21
- botocore=1.24.21
- brotlipy=0.7.0
- bzip2=1.0.8
- c-ares=1.18.1
- ca-certificates=2022.6.15
- cachetools=5.2.0
- certifi=2022.6.15
- cffi=1.15.1
- charset-normalizer=2.1.1
- click=8.1.3
- cloudpickle=2.1.0
- coiled=0.2.27
- commonmark=0.9.1
- cryptography=37.0.4
- cytoolz=0.12.0
- dask-core=2022.8.2a220830
- dask-glm=0.2.0
- dask-labextension=5.2.0
- dask-ml=2022.5.27
- dask=2022.8.2a220830
- dataclasses=0.8
- debugpy=1.6.3
- decorator=5.1.1
- defusedxml=0.7.1
- distributed=2022.8.2a220830
- entrypoints=0.4
- execnet=1.9.0
- executing=1.0.0
- fasteners=0.17.3
- filelock=3.8.0
- flit-core=3.7.1
- freetype=2.12.1
- frozenlist=1.3.1
- fsspec=2022.5.0
- future=0.18.2
- gcsfs=2022.5.0
- gflags=2.2.2
- glog=0.6.0
- google-api-core=2.8.2
- google-auth-oauthlib=0.5.2
- google-auth=2.11.0
- google-cloud-core=2.3.2
- google-cloud-storage=2.5.0
- google-crc32c=1.1.2
- google-resumable-media=2.3.3
- googleapis-common-protos=1.56.4
- greenlet=1.1.3
- grpc-cpp=1.45.2
- grpcio=1.46.3
- heapdict=1.0.1
- icu=70.1
- idna=3.3
- importlib_metadata=4.11.4
- importlib_resources=5.9.0
- importlib-metadata=4.11.4
- iniconfig=1.1.1
- ipycytoscape=1.3.3
- ipykernel=6.15.2
- ipython_genutils=0.2.0
- ipython=8.4.0
- ipywidgets=7.7.0
- jedi=0.18.1
- jinja2=3.1.2
- jmespath=1.0.1
- joblib=1.1.0
- jpeg=9e
- json5=0.9.5
- jsondiff=2.0.0
- jsonschema=4.14.0
- jupyter_client=7.3.4
- jupyter_core=4.11.1
- jupyter_server=1.18.1
- jupyter-server-proxy=3.2.1
- jupyterlab_pygments=0.2.2
- jupyterlab_server=2.15.1
- jupyterlab_widgets=1.1.1
- jupyterlab=3.4.2
- keyutils=1.6.1
- krb5=1.19.3
- lcms2=2.12
- ld_impl_linux-64=2.36.1
- lerc=4.0.0
- libblas=3.9.0
- libbrotlicommon=1.0.9
- libbrotlidec=1.0.9
- libbrotlienc=1.0.9
- libcblas=3.9.0
- libcrc32c=1.1.2
- libcurl=7.83.1
- libdeflate=1.13
- libedit=3.1.20191231
- libev=4.33
- libevent=2.1.10
- libffi=3.4.2
- libgcc-ng=12.1.0
- libgfortran-ng=12.1.0
- libgfortran5=12.1.0
- libgomp=12.1.0
- libgoogle-cloud=1.40.2
- libiconv=1.16
- liblapack=3.9.0
- libllvm11=11.1.0
- libnghttp2=1.47.0
- libnsl=2.0.0
- libopenblas=0.3.21
- libpng=1.6.37
- libprotobuf=3.20.1
- libsodium=1.0.18
- libsqlite=3.39.2
- libssh2=1.10.0
- libstdcxx-ng=12.1.0
- libthrift=0.16.0
- libtiff=4.4.0
- libutf8proc=2.7.0
- libuuid=2.32.1
- libwebp-base=1.2.4
- libxcb=1.13
- libxgboost=1.6.1
- libxml2=2.9.14
- libxslt=1.1.35
- libzlib=1.2.12
- llvmlite=0.38.1
- locket=1.0.0
- lxml=4.9.1
- lz4-c=1.9.3
- lz4=4.0.0
- mako=1.2.2
- markupsafe=2.1.1
- matplotlib-inline=0.1.6
- mistune=2.0.4
- msgpack-python=1.0.3
- multidict=6.0.2
- multipledispatch=0.6.0
- nbclassic=0.4.3
- nbclient=0.6.7
- nbconvert-core=7.0.0
- nbconvert-pandoc=7.0.0
- nbconvert=7.0.0
- nbformat=5.4.0
- ncurses=6.3
- nest-asyncio=1.5.5
- networkx=2.8.6
- notebook-shim=0.1.0
- notebook=6.4.12
- numba=0.55.1
- numcodecs=0.10.2
- numpy=1.21.6
- oauthlib=3.2.0
- openjpeg=2.5.0
- openssl=1.1.1q
- orc=1.7.5
- packaging=21.3
- pandas=1.4.2
- pandoc=2.19.2
- pandocfilters=1.5.0
- parquet-cpp=1.5.1
- parso=0.8.3
- partd=1.3.0
- pexpect=4.8.0
- pickleshare=0.7.5
- pillow=9.2.0
- pip=22.2.2
- pkgutil-resolve-name=1.3.10
- pluggy=1.0.0
- prometheus_client=0.14.1
- prompt-toolkit=3.0.30
- protobuf=3.20.1
- psutil=5.9.1
- pthread-stubs=0.4
- ptyprocess=0.7.0
- pure_eval=0.2.2
- py-xgboost=1.6.1
- py=1.11.0
- pyarrow=8.0.0
- pyasn1-modules=0.2.7
- pyasn1=0.4.8
- pycparser=2.21
- pygments=2.13.0
- pyjwt=2.4.0
- pynvml=11.4.1
- pyopenssl=22.0.0
- pyparsing=3.0.9
- pyrsistent=0.18.1
- pysocks=1.7.1
- pytest-forked=1.4.0
- pytest-xdist=2.5.0
- pytest=7.1.2
- python_abi=3.9
- python-blosc=1.10.2
- python-dateutil=2.8.2
- python-fastjsonschema=2.16.1
- python=3.9.13
- pytz=2022.2.1
- pyu2f=0.1.5
- pyyaml=6.0
- pyzmq=23.2.1
- re2=2022.06.01
- readline=8.1.2
- requests-oauthlib=1.3.1
- requests=2.28.1
- rich=12.5.1
- rsa=4.9
- s2n=1.0.10
- s3fs=2022.5.0
- s3transfer=0.5.2
- scikit-learn=1.1.1
- scipy=1.9.1
- send2trash=1.8.0
- setuptools=65.3.0
- simpervisor=0.4
- six=1.16.0
- snappy=1.1.9
- sniffio=1.2.0
- sortedcontainers=2.4.0
- soupsieve=2.3.2.post1
- spectate=1.0.1
- sqlalchemy=1.4.40
- sqlite=3.39.2
- stack_data=0.5.0
- tblib=1.7.0
- terminado=0.15.0
- textual=0.1.18
- threadpoolctl=3.1.0
- tinycss2=1.1.1
- tk=8.6.12
- tomli=2.0.1
- toolz=0.11.2
- tornado=6.1
- traitlets=5.3.0
- traittypes=0.2.1
- typing_extensions=4.3.0
- typing-extensions=4.3.0
- tzdata=2022c
- urllib3=1.26.11
- wcwidth=0.2.5
- webencodings=0.5.1
- websocket-client=1.4.0
- wheel=0.37.1
- widgetsnbextension=3.6.1
- wrapt=1.14.1
- xarray=2022.3.0
- xgboost=1.6.1
- xorg-libxau=1.0.9
- xorg-libxdmcp=1.1.3
- xz=5.2.6
- yaml=0.2.5
- yarl=1.7.2
- zarr=2.11.3
- zeromq=4.3.4
- zict=2.2.0
- zipp=3.8.1
- zlib=1.2.12
- zstd=1.5.2

Package sync
- _libgcc_mutex=0.1
- _openmp_mutex=4.5
- _py-xgboost-mutex=2.0
- abseil-cpp=20211102.0
- aiobotocore=2.3.4
- aiohttp=3.8.1
- aioitertools=0.10.0
- aiosignal=1.2.0
- alembic=1.8.1
- anyio=3.6.1
- argon2-cffi-bindings=21.2.0
- argon2-cffi=21.3.0
- arrow-cpp=8.0.1
- asciitree=0.3.3
- asttokens=2.0.8
- async-timeout=4.0.2
- attrs=22.1.0
- aws-c-cal=0.5.11
- aws-c-common=0.6.2
- aws-c-event-stream=0.2.7
- aws-c-io=0.10.5
- aws-checksums=0.1.11
- aws-sdk-cpp=1.8.186
- Babel=2.10.3
- backcall=0.2.0
- backoff=1.11.1
- backports.functools-lru-cache=1.6.4
- backports=1.0
- beautifulsoup4=4.11.1
- bleach=5.0.1
- blinker=1.4
- blosc=1.21.1
- bokeh=2.4.3
- boto3=1.21.21
- botocore=1.24.21
- brotlipy=0.7.0
- bzip2=1.0.8
- c-ares=1.18.1
- ca-certificates=2022.6.15
- cachetools=5.2.0
- certifi=2022.6.15
- cffi=1.15.1
- charset-normalizer=2.1.1
- click=8.1.3
- cloudpickle=2.1.0
- coiled=0.2.27
- commonmark=0.9.1
- cryptography=37.0.4
- cytoolz=0.12.0
- dask-glm=0.2.0
- dask-labextension=5.2.0
- dask-ml=2022.5.27
- dask=2022.8.1
- dask=2022.8.2a220830
- dataclasses=0.8
- debugpy=1.6.3
- decorator=5.1.1
- defusedxml=0.7.1
- distributed=2022.8.2a220831
- entrypoints=0.4
- execnet=1.9.0
- executing=1.0.0
- fasteners=0.17.3
- fastjsonschema=2.16.1
- filelock=3.8.0
- flit-core=3.7.1
- freetype=2.12.1
- frozenlist=1.3.1
- fsspec=2022.5.0
- future=0.18.2
- gcsfs=2022.5.0
- gflags=2.2.2
- glog=0.6.0
- google-api-core=2.8.2
- google-auth-oauthlib=0.5.2
- google-auth=2.11.0
- google-cloud-core=2.3.2
- google-cloud-storage=2.5.0
- google-crc32c=1.1.2
- google-resumable-media=2.3.3
- googleapis-common-protos=1.56.4
- greenlet=1.1.3
- grpc-cpp=1.46.4
- grpcio=1.46.4
- HeapDict=1.0.1
- icu=70.1
- idna=3.3
- importlib_metadata=4.11.4
- importlib-metadata=4.11.4
- importlib-resources=5.9.0
- iniconfig=1.1.1
- ipycytoscape=1.3.3
- ipykernel=6.15.2
- ipython_genutils=0.2.0
- ipython=8.4.0
- ipywidgets=7.7.2
- jedi=0.18.1
- Jinja2=3.1.2
- jmespath=1.0.1
- joblib=1.1.0
- jpeg=9e
- json5=0.9.5
- jsondiff=2.0.0
- jsonschema=4.15.0
- jupyter-client=7.3.4
- jupyter-core=4.11.1
- jupyter-server-proxy=3.2.1
- jupyter-server=1.18.1
- jupyterlab-pygments=0.2.2
- jupyterlab-server=2.15.1
- jupyterlab-widgets=1.1.1
- jupyterlab=3.4.5
- keyutils=1.6.1
- krb5=1.19.3
- lcms2=2.12
- ld_impl_linux-64=2.36.1
- lerc=4.0.0
- libabseil=20211102.0
- libblas=3.9.0
- libbrotlicommon=1.0.9
- libbrotlidec=1.0.9
- libbrotlienc=1.0.9
- libcblas=3.9.0
- libcrc32c=1.1.2
- libcurl=7.83.1
- libdeflate=1.13
- libedit=3.1.20191231
- libev=4.33
- libevent=2.1.10
- libffi=3.4.2
- libgcc-ng=12.1.0
- libgfortran-ng=12.1.0
- libgfortran5=12.1.0
- libgomp=12.1.0
- libgoogle-cloud=1.40.2
- libiconv=1.16
- liblapack=3.9.0
- libllvm11=11.1.0
- libnghttp2=1.47.0
- libnsl=2.0.0
- libopenblas=0.3.21
- libpng=1.6.37
- libprotobuf=3.20.1
- libsodium=1.0.18
- libsqlite=3.39.2
- libssh2=1.10.0
- libstdcxx-ng=12.1.0
- libthrift=0.16.0
- libtiff=4.4.0
- libutf8proc=2.7.0
- libuuid=2.32.1
- libwebp-base=1.2.4
- libxcb=1.13
- libxgboost=1.6.1
- libxml2=2.9.14
- libxslt=1.1.35
- libzlib=1.2.12
- llvmlite=0.38.1
- locket=1.0.0
- lxml=4.9.1
- lz4-c=1.9.3
- lz4=4.0.0
- Mako=1.2.2
- MarkupSafe=2.1.1
- matplotlib-inline=0.1.6
- mistune=2.0.4
- msgpack=1.0.3
- multidict=6.0.2
- multipledispatch=0.6.0
- nbclassic=0.4.3
- nbclient=0.6.7
- nbconvert-pandoc=7.0.0
- nbconvert=7.0.0
- nbconvert=7.0.0
- nbformat=5.4.0
- ncurses=6.3
- nest-asyncio=1.5.5
- networkx=2.8.6
- notebook-shim=0.1.0
- notebook=6.4.12
- numba=0.55.2
- numcodecs=0.10.2
- numpy=1.21.6
- oauthlib=3.2.0
- openjpeg=2.5.0
- openssl=1.1.1q
- orc=1.7.5
- packaging=21.3
- pandas=1.4.2
- pandoc=2.19.2
- pandocfilters=1.5.0
- parquet-cpp=1.5.1
- parso=0.8.3
- partd=1.3.0
- pexpect=4.8.0
- pickleshare=0.7.5
- Pillow=9.2.0
- pip=22.2.2
- pkgutil-resolve-name=1.3.10
- pluggy=1.0.0
- prometheus-client=0.14.1
- prompt-toolkit=3.0.30
- protobuf=3.20.1
- psutil=5.9.1
- pthread-stubs=0.4
- ptyprocess=0.7.0
- pure-eval=0.2.2
- py-xgboost=1.6.1
- py=1.11.0
- pyarrow=8.0.1
- pyasn1-modules=0.2.7
- pyasn1=0.4.8
- pycparser=2.21
- Pygments=2.13.0
- PyJWT=2.4.0
- pynvml=11.4.1
- pyOpenSSL=22.0.0
- pyparsing=3.0.9
- pyrsistent=0.18.1
- PySocks=1.7.1
- pytest-forked=1.4.0
- pytest-xdist=2.5.0
- pytest=7.1.2
- python_abi=3.9
- python-dateutil=2.8.2
- python=3.9.13
- pytz=2022.2.1
- pyu2f=0.1.5
- PyYAML=6.0
- pyzmq=23.2.1
- re2=2022.06.01
- readline=8.1.2
- requests-oauthlib=1.3.1
- requests=2.28.1
- rich=12.5.1
- rsa=4.9
- s2n=1.0.10
- s3fs=2022.5.0
- s3transfer=0.5.2
- scikit-learn=1.1.2
- scipy=1.9.1
- Send2Trash=1.8.0
- setuptools=65.3.0
- simpervisor=0.4
- six=1.16.0
- snappy=1.1.9
- sniffio=1.2.0
- sortedcontainers=2.4.0
- soupsieve=2.3.2.post1
- spectate=1.0.1
- SQLAlchemy=1.4.40
- sqlite=3.39.2
- stack-data=0.5.0
- tblib=1.7.0
- terminado=0.15.0
- textual=0.1.18
- threadpoolctl=3.1.0
- tinycss2=1.1.1
- tk=8.6.12
- tomli=2.0.1
- toolz=0.11.2
- tornado=6.1
- traitlets=5.3.0
- traittypes=0.2.1
- typing-extensions=4.3.0
- typing-extensions=4.3.0
- tzdata=2022c
- urllib3=1.26.11
- wcwidth=0.2.5
- webencodings=0.5.1
- websocket-client=1.4.0
- wheel=0.37.1
- widgetsnbextension=3.6.1
- wrapt=1.14.1
- xarray=2022.3.0
- xgboost=1.6.1
- xorg-libxau=1.0.9
- xorg-libxdmcp=1.1.3
- xz=5.2.6
- yaml=0.2.5
- yarl=1.7.2
- zarr=2.11.3
- zeromq=4.3.4
- zict=2.2.0
- zipp=3.8.1
- zlib=1.2.12
- zstd=1.5.2

There's a bug I just. squashed to do with the private dev packages (hence why the dask versions are off) , but @ian-r-rose and me don't think it would have affected the benchmarks
There are extra packages for sure, but no idea if any of them should really affect performance.

Column 1 is only in the senv, column 2 is only from package-sync. They're basically the same; not enough different to attribute the 3x slowdown to I don't think.
        Babel
        HeapDict
        Jinja2
        Mako
        MarkupSafe
        Pillow
        PyJWT
        PySocks
        PyYAML
        Pygments
        SQLAlchemy
        Send2Trash
babel
        dask
dask_core
        fastjsonschema
heapdict
jinja2
        libabseil
mako
markupsafe
        msgpack
msgpack_python
        nbconvert
nbconvert_core
pillow
        pyOpenSSL
pygments
pyjwt
pyopenssl
pysocks
python_blosc
python_fastjsonschema
pyyaml
send2trash
sqlalchemy

@shughes-uk is package_sync downloading and installing in parallel right now? I don't remember.

So, it seems that the differences we saw from test_vorticity and test_dataframe_align were both real upstream changes, and due to  https://github.com/dask/distributed/issues/6597 .
Since those were the biggest outliers, I'm getting more optimistic that we can move forward with this.

@shughes-uk is package_sync downloading and installing in parallel right now? I don't remember.

uses micromamba so yes

Not sure if this matters but I noticed that in the package sync list, we have two dask versions installed. Not sure if that matters/affects but mentioning it for visibility.
- dask=2022.8.1
- dask=2022.8.2a220830

Not sure if this matters but I noticed that in the package sync list, we have two dask versions installed. Not sure if that matters/affects but mentioning it for visibility.

Good eye -- I think @shughes-uk pushed a fix for that yesterday afternoon, are you still seeing it?

I'm doing a census of the failures and reported regressions for this branch during the last week. I'm deliberately removing regressions related to cluster startup time, as those are expected.
Regressions, main

Link

'coiled-0.0.4-py3.9', name = 'test_q8[5 GB (parquet)]

Link

'coiled-0.0.4-py3.9', name = 'test_double_diff',

Link

coiled-0.0.4-py3.9', name = 'test_double_diff',

Link

'coiled-0.0.4-py3.9', name = 'test_double_diff'

Regressions, package_sync

Link

coiled-0.0.4-py3.9 - test_quickstart_parquet
coiled-0.0.4-py3.10 - test_shuffle_simple
coiled-latest-py3.10 - test_shuffle_parquet
coiled-upstream-py3.9 - test_read_spark_generated_data
coiled-0.1.0-py3.10 - test_shuffle_parquet
coiled-0.1.0-py3.10 - test_shuffle_simple
coiled-0.1.0-py3.9 - test_q7[5 GB (parquet)]
coiled-0.1.0-py3.9 - test_read_spark_generated_data

Link

'coiled-0.0.4-py3.9', name = 'test_default_cluster_spinup_time'
'coiled-0.0.4-py3.9', name = 'test_quickstart_parquet'
'coiled-0.0.4-py3.9', name = 'test_read_spark_generated_data'
'coiled-0.0.4-py3.10', name = 'test_shuffle_simple'
'coiled-latest-py3.9', name = 'test_read_spark_generated_data'
 'coiled-latest-py3.8', name = 'test_shuffle_parquet'
'coiled-latest-py3.10', name = 'test_shuffle_parquet'
'coiled-upstream-py3.9', name = 'test_double_diff'
'coiled-upstream-py3.10', name = 'test_rechunk_in_memory'
'coiled-0.1.0-py3.10', name = 'test_shuffle_parquet'
'coiled-0.1.0-py3.9', name = 'test_q7[5 GB (parquet)]'

Link

'coiled-0.0.4-py3.9', name = 'test_jobqueue'
'coiled-0.0.4-py3.9', name = 'test_q8[5 GB (parquet)]
 'coiled-0.0.4-py3.9', name = 'test_q9[5 GB (parquet)]'
'coiled-0.0.4-py3.9', name = 'test_read_spark_generated_data'
'coiled-0.0.4-py3.10', name = 'test_shuffle_simple'
'coiled-latest-py3.9', name = 'test_shuffle_simple'
'coiled-latest-py3.8', name = 'test_shuffle_simple'
'coiled-upstream-py3.9', name = 'test_dataframe_align' (We understand this one! Related to co-assignment)
 coiled-upstream-py3.9', name = 'test_download_throughput[pandas]',
'coiled-upstream-py3.9', name = 'test_jobqueue'
'coiled-upstream-py3.9', name = 'test_q3[5 GB (parquet)]'
'coiled-upstream-py3.9', name = 'test_q9[5 GB (parquet)]'
'coiled-upstream-py3.9', name = 'test_read_spark_generated_data'
'coiled-upstream-py3.9', name = 'test_shuffle_simple'
'coiled-upstream-py3.8', name = 'test_shuffle_simple'
'coiled-0.1.0-py3.10', name = 'test_rechunk_in_memory'
'coiled-0.1.0-py3.9', name = 'test_download_throughput[pandas]'
'coiled-0.1.0-py3.9', name = 'test_q1[0.5 GB (csv)]'
'coiled-0.1.0-py3.9', name = 'test_shuffle_simple',
'coiled-0.1.0-py3.9', name = 'test_xgboost_distributed_training'

Link

coiled-0.0.4-py3.9', name = 'test_q9[0.5 GB (parquet)]
'coiled-latest-py3.9', name = 'test_read_spark_generated_data'
'coiled-latest-py3.8', name = 'test_shuffle_simple'
'coiled-upstream-py3.9', name = 'test_anom_mean'
 'coiled-upstream-py3.9', name = 'test_basic_sum'
'coiled-upstream-py3.9', name = 'test_dataframe_align',
coiled-upstream-py3.10', name = 'test_shuffle_simple',
'coiled-0.1.0-py3.10', name = 'test_shuffle_parquet'
'coiled-0.1.0-py3.10', name = 'test_shuffle_simple'
'coiled-0.1.0-py3.9', name = 'test_anom_mean',
'coiled-0.1.0-py3.9', name = 'test_q7[0.5 GB (csv)]
 'coiled-0.1.0-py3.9', name = 'test_q8[5 GB (parquet)]'
'coiled-0.1.0-py3.9', name = 'test_q9[5 GB (parquet)]'
'coiled-0.1.0-py3.9', name = 'test_read_spark_generated_data'

Link

'coiled-0.0.4-py3.9', name = 'test_q8[5 GB (parquet)]'
'coiled-0.0.4-py3.9', name = 'test_read_spark_generated_data'
'coiled-latest-py3.9', name = 'test_q4[5 GB (parquet)]'
'coiled-latest-py3.10', name = 'test_rechunk_in_memory'
'coiled-upstream-py3.9', name = 'test_dataframe_align' (Again, we understand this one)
'coiled-upstream-py3.9', name = 'test_jobqueue'
'coiled-upstream-py3.8', name = 'test_shuffle_parquet'
'coiled-0.1.0-py3.9', name = 'test_read_spark_generated_data'

Link  -- Many test failures, possibly related to some uptime issues with s3?
Link

'coiled-0.0.4-py3.9', name = 'test_download_throughput[pandas]'
'coiled-0.0.4-py3.9', name = 'test_download_throughput[s3fs]'
'coiled-0.0.4-py3.9', name = 'test_write_wide_data'
'coiled-latest-py3.9', name = 'test_read_spark_generated_data'
'coiled-latest-py3.9', name = 'test_write_wide_data'
'coiled-latest-py3.10', name = 'test_shuffle_parquet'
'coiled-upstream-py3.9', name = 'test_dataframe_align' (again understood)
'coiled-upstream-py3.9', name = 'test_shuffle_parquet
'coiled-upstream-py3.8', name = 'test_shuffle_parquet'
'coiled-0.1.0-py3.9', name = 'test_download_throughput[pandas]'
'coiled-0.1.0-py3.9', name = 'test_read_hive_partitioned_data'
'coiled-0.1.0-py3.9', name = 'test_read_spark_generated_data'
'coiled-0.1.0-py3.9', name = 'test_shuffle'
'coiled-0.1.0-py3.9', name = 'test_write_wide_data'
coiled-0.1.0-py3.9', name = 'test_write_wide_data'

Link

'coiled-0.0.4-py3.9', name = 'test_q3[5 GB (parquet)]'
'coiled-0.0.4-py3.9', name = 'test_quickstart_parquet'
'coiled-latest-py3.9', name = 'test_q3[5 GB (parquet)]'
'coiled-latest-py3.9', name = 'test_read_spark_generated_data'
'coiled-latest-py3.8', name = 'test_shuffle_simple'
coiled-latest-py3.10', name = 'test_shuffle_simple'
'coiled-0.0.4-py3.8', name = 'test_rechunk_in_memory',
'coiled-0.0.4-py3.8', name = 'test_shuffle_simple'
'coiled-upstream-py3.9', name = 'test_dataframe_align',
'coiled-0.1.0-py3.9', name = 'test_read_spark_generated_data'

Link

'coiled-0.0.4-py3.9', name = 'test_vorticity'  (This is weird! Why is this one showing up in 0.0.4?)
'coiled-latest-py3.8', name = 'test_shuffle_simple',
'coiled-latest-py3.10', name = 'test_shuffle_simple'
'coiled-upstream-py3.9', name = 'test_q5[5 GB (parquet)]'
'coiled-upstream-py3.10', name = 'test_rechunk_in_memory'
'coiled-0.1.0-py3.8', name = 'test_shuffle_simple'
'coiled-0.1.0-py3.10', name = 'test_shuffle_parquet'
'coiled-0.1.0-py3.9', name = 'test_dataframe_align' (This is very weird! Why is this one showing up in 0.1.0?)
'coiled-0.1.0-py3.9', name = 'test_q1[5 GB (parquet)]

Link

'coiled-0.0.4-py3.10', name = 'test_shuffle_simple'
'coiled-latest-py3.9', name = 'test_shuffle_simple'
'coiled-latest-py3.8', name = 'test_shuffle_parquet'
coiled-latest-py3.8', name = 'test_shuffle_simple'
'coiled-upstream-py3.9', name = 'test_anom_mean'
'coiled-upstream-py3.9', name = 'test_q7[5 GB (parquet)]',
'coiled-upstream-py3.9', name = 'test_read_hive_partitioned_data',
 'coiled-upstream-py3.9', name = 'test_shuffle'
'coiled-upstream-py3.9', name = 'test_shuffle_simple',
'coiled-upstream-py3.8', name = 'test_rechunk_in_memory'
'coiled-upstream-py3.8', name = 'test_shuffle_simple'
'coiled-upstream-py3.10', name = 'test_shuffle_simple'
 'coiled-0.1.0-py3.8', name = 'test_rechunk_in_memory'
 'coiled-0.1.0-py3.8', name = 'test_shuffle_simple',
'coiled-0.1.0-py3.10', name = 'test_shuffle_simple'
 'coiled-0.1.0-py3.9', name = 'test_q7[5 GB (parquet)]'
'coiled-0.1.0-py3.9', name = 'test_q8[5 GB (parquet)]'
'coiled-0.1.0-py3.9', name = 'test_shuffle_parquet',
'coiled-0.1.0-py3.9', name = 'test_shuffle_simple'

Link

'coiled-0.0.4-py3.10', name = 'test_shuffle_simple'
'coiled-latest-py3.9', name = 'test_q3[5 GB (parquet)]'
'coiled-latest-py3.9', name = 'test_q5[5 GB (parquet)]'
'coiled-latest-py3.8', name = 'test_shuffle_simple'
'coiled-latest-py3.10', name = 'test_shuffle_parquet'
 'coiled-upstream-py3.9', name = 'test_anom_mean'
'coiled-upstream-py3.9', name = 'test_shuffle',
 'coiled-upstream-py3.9', name = 'test_shuffle_simple'
'coiled-0.1.0-py3.9', name = 'test_anom_mean'

Extra observations

This failed because it failed to find pyarrow. Did that not get sync'd successfully?
Why do I see activity in test_vorticity and test_dataframe_align in the non-upstream builds?

My top observation of the above is that the package sync PR has many more detected regressions than the ones on main. And they are not just in the upstream builds, but relatively evenly distributed. But there are some repeat offenders that jump out when they are all arrayed together:

test_quickstart_parquet
test_shuffle_simple
test_shuffle_parquet
test_read_spark_generated_data
test_double_diff
test_rechunk_in_memory
test_q*[5 GB (parquet)]
test_shuffle

Many (but not all!) of these are parquet-related. So one thing we should look at pretty closely is whether there is something different going on in the solves for boto, pyarrow, or s3fs