[suggestion] Error checking on Sack of Holding

Just today, I had a soul shattering experience. after taking several hours to mcguiver up an enhanced storage network from the mod of the same name on my server, along with several generators and other bits and transferring close to 4k items of special loot, resources and mundanes from my super dangerzoned chests (as far as it's cell was concerned) I was moving stuff around my base using the sack of holding to make it faster to not drop inventories and energy storage.
It works fine for machines for the most part, but when I came to the storage drive which held my cells it apparently null pointered and didn't pick it up and picked up a phantom empty copy that was placed when I used the bag again. what this essentially did was destroy my original drive and when placed down place an empty one which as a result destroyed all my items. it also shot some errors into the console. Because when I noticed there was still a drive where I had picked it up from that drive just turned out to be a phantom...
what I'd like to suggest is that you somehow have a form of error checking on the bag that if it was to pick up something but hits a snag with stuff that it doesn't understand  (like projectE alchemical chests from sometime back before they updated, and now these storage drives) it doesn't complete the picking up process or some kind of safeguard.
if not, does this mean my safeguard is only "knowledge" and blacklisting things that are known to have errors?
https://pastebin.com/hC2050fH
Heres a paste of multicraft's console stuff in the section when it happened, to when I left after confirming it with creative and looking for my drive to see if it really went boom and stuff... read from bottom to top, cause multicraft console is upside down so new stuff is at top. :V
there wasn't an actual crash however. thankfully i had to remove a mod earlier in the day cause of errors also which procced an automatic backup due to it's items going poof which I can roll back to and for the bother I can just cheat anything in particular back in...

Note to self: Bug is very easy to reproduce, dont even need that many items
1 disc drive
1 64k disk
add a few stocks of cobblestone
pick it up with the bag and place in different location
Refined Storage version 1.5.34

well, I took the bag and used it on most everything and it worked fine.
but with the storage drive, I picked it up but it was still there but my bag was glowing. I didn't notice at first, i thought it was lag.
I placed down the drive that was in my bag, but it was empty. no cells in it. but it looked like there was cells at first.
I turn to the side, notice the drive with my original cells is still in it's original spot, I at first think it's the duplication bug I used to have with projectE several months ago. but upon interacting with it, it's just a phantom copy due to the nullpointer and the bag meaning to take a copy of it then delete it. and so it disappears when I interact with it to my confusion.
I then proceeded to be completely dumbfounded for a few minutes when the drive i had placed down was empty and no longer had any cells after I checked it and the rendering updated as such resulting in all my items being lost to the hungry bag of holding gods. Then proceeded to test it out a bit more generating more console spam and more phantoms and re-emptied drives, then reset my world to a backup.
that was the only thing I know of so far that "dupes". but it's not really a dupe. it goes to pick up the original and fails so just gets a factory default version. but still deletes the original in the world returning no items as if it did pick it up, only realizing it did after you interact with it and make the phantom object remove itself due to time paradox. you then place it down, and you've got a factory default drive with nothing in it. :V

With error checking it looks like this.  It puts up a "sack.error.pickup" text which eventually will be a nice message like "error trying to pickup block, action cancelled to prevent data loss"
[GIF]
https://i.imgur.com/h6RoQY2.gifv

absolutely brilliant. thats perfect. :)

Release is out with these changes https://minecraft.curseforge.com/projects/cyclic/files/2588760


Container storage fills with puppeteer temp data

Description
While downloading chapters, Teemii does not clean up old puppeteer data. /tmp fills with puppeteer_dev_profile-XXXXXX. I left it overnight and it ballooned to 26GB.
The drive ran out of space and the config file got corrupted. It resulted in a mostly downloaded library that cannot be rescanned.
Version
0.8.0
Steps to Reproduce

Download any chapter
docker exec -it teemii-backend ls -lha /tmp
Witness leftover data

Screenshots
No response
Logs
No response
Platform
desktop
Device
Web
Operating System
Linux
Browser
Edge
Additional Context
No response
Code of Conduct

[X] I agree to follow Teemii's Code of Conduct

Many thanks for noticing this. I'm delivering a patch asap to disable this annoying Puppeteer behavior.

Thanks for the quick response.

Fix available in develop branch


Add helpers for converting between xmap subclasses

This is temporary compost heap while I figure out exactly how this should work
To go from xmap_df to xmap_tbl is basically:
# links <- data.frame(<from,to,weights>)
links |>
  as_xmap.data.frame() |>
  as_tibble() |>
  as_xmap.tibble()

Wrapper for new_xmap()
could be useful internally
new_xmap <- function(x, col_from, col_to, col_weights, subclass = c("xmap_df", "xmap_tbl"), from_set = NULL){
  subclass <- rlang::arg_match(subclass)

  x <- switch(subclass,
              xmap_df = new_xmap_df(x, col_from, col_to, col_weights, from_set),
              xmap_tbl = new_xmap_tbl(x, col_from, col_to, col_weights, from_set)
         )

  return(x)
}

Close for reasons discussed in #93 and https://github.com/cynthiahqy/conformr-project/commit/c218b4e83cecba86b146dd8eab2c2aa787328068


Regressions when compared to the previous build system

I have packaged few pdf files which compared the test file outputs (created using fontdiff) there were few regressions related to diacritic positioning (see darker green [that's the newest build from this directory], reddish is the previous font -- not every darker green [darker color indicates more serious difference] is an error but you'll see some)
regressions.zip
I'd like to find solution to these while maintaining the new structure that builds a single variable file with three axis: weight, width, VLHT

With #5 the only differences that show in fontdiff PDFs are:

the modified uni0773, uni0774 that are now tall in Noto Naskh Arabic but remain short in Noto Naskh Arabic UI
the modified alif and final alif in all composite glyphs, this is one node removed or added for interpolation compatibility
the Urdu numbers in the relevant languages in Noto Naskh Arabic UI
the three dots in uni06B8, uni06B8.fina, uni06D1, uni06D1.fina are now a component of uniFBB9.restricted instead of a scaled component of uniFBB9 which may produce a 1 unit rounding difference


Add a new CLI to help user finish the kubebb quick start

We add kctl start --context ~/.kube/config  as the 1st command which help user finish the quick start:

create a  one-node kind cluster(optional)
deploy kubebb core
deploy building base (optional)

[ ] #263

kctl start
Install

install cluster-component

cert-manager (kubebb/core)  YES
nginx-ingress (random selected)  YES/NO

2.install core YES

PreCheck
Continue
Component Store  NO

installl u4a-component(building-base) YES/NO

PreCheck
Continue

Component Store YES/NO

Uninstall

Uninstall store
U4A
core
cluster-component


Update README.md

Hey Leonard!
Merging into README.md doesn't update the website, just the github repo. I can review this at Hack Night and work through any issues (unless just updating what showed in the README) was the goal

Hi Hunter, thanks for noticing. That was intentional. I wanted to update the README, so potential project contributors could have a super quick description of what we're working on. What I wrote is aimed at them, not necessarily the broader public.

Oh Awesome. In which case, 👍


Allow users to opt-out creating root dir when building Fs backend

https://github.com/apache/incubator-opendal/blob/a04a724059d7631318cb1543382b7462190f43eb/core/src/services/fs/backend.rs#L141-L163
Currently when creating Fs backend, OpenDAL will always create the root dir even if users intend to read data from the Fs backend, which might suprise users.
I propose we could create the necessary dirs in the implemetation of Accessor::write and Accessor::create rather than when we are building Fs backend. Or at least provide users with a option to opt-out this behavior.

I don't understand the purpose of creating a file system backend on a non-existent root. For instance, if a user creates a backend with /not_exist and tries to read hello.txt, they will receive a NotFound error. This can cause confusion for the user who then needs to figure out what went wrong.
I would prefer to return an error instead, but this change will break the fs backend. Let's keep this issue open and see if others share the same concern.

I don't understand the purpose of creating a file system backend on a non-existent root.

Users might not create a fs backend with a non-existent root on purpose.
For instance I once tested oli cat by reading a local file and I mistyped the path, like oli cat /tmp/not-exist/foo, and then a dir named non-exist was created under /tmp.
Another case is that if I run the test test_parse_fs_location locally, a dir named foo/bar will be created under PWD, which supprises me as no read/write operation is actually performed.

Users might not create a fs backend with a non-existent root on purpose.

That's true.

For instance I once tested oli cat by reading a local file and I mistyped the path, like oli cat /tmp/not-exist/foo, and then a dir named non-exist was created under /tmp.

Actually, oli must avoid using /tmp/not-exist as the root directory. Instead, all serivces should stick to / as root to make the internal logic cleaner.

Another case is that if I run the test test_parse_fs_location locally, a dir named foo/bar will be created under PWD, which supprises me as no read/write operation is actually performed.

This is the same issue as I statemented in last comment.

Thank you for the proposal, but I don't think it's a significant enough issue to warrant a change.


Use voluptuous for Honeywell

Description:
Migration of the configuration check to voluptuous. Also this is includes an upgrade of somecomfort to 0.3.1 as discussed on gitter.
Related issue (if applicable): fixes 127528299
Example entry for configuration.yaml (if applicable):
climate:
  platform: honeywell
  username: YOUR_USERNAME
  password:  YOUR_PASSWORD
  region: REGION

Would be nice if somebody could take a look at the changes and run a quick test. Thanks.

🐬


Add new CesiumLegacyTerrain library/namespace

This is a PR into #828 so merge that first.
The term "legacy terrain" here refers to quantized-mesh and layer.json. The main motivation for moving it to its own library is that this makes it easier to use the same code generator we use for glTF and 3D Tiles to generate classes for reading/writing layer.json.
So, this new library / namespace has:

QuantizedMeshLoader (previously in Cesium3DTilesContent)
Generated code for representing, reading, and writing layer.json.
New LegacyTerrainUtilities class with some helpers for working with this format.

Other related bonus changes:

Added support for arrays of arrays in generated classes. This is needed for layer.json availability rectangles.
Fixed some name conflicts between the generated Reader and Writer classes, so that both can be generated into a single library / namespace.
Fixed a bug in resolving schemas during code generation when the schema files are stored on disk rather than on the web.
Changed the generator to avoid generating properties named extensions, extras, or unknownProperties, because these will conflict with built-in fields of the generated classes.

The term "legacy terrain" here refers to quantized-mesh and layer.json.

Kind of a pre-review question, and maybe naïve, but any reason we need to call this format "legacy"?
For example:

The CesiumLegacyTerrain library could just be CesiumTerrain
The class LegacyTerrainUtilities could be QuantizedMeshUtilities

Previous experiences have made me a little biased against using the word "legacy" in code bases, but am open to it if it has use.

Well, I think CesiumTerrain is way too general. Terrain is increasingly represented using 3D Tiles. IMO it's very much fair to call quantized-mesh and layer.json a legacy format at this point, though that doesn't mean it will go away tomorrow. Anyone who has a strong opinion on this, please chime in. If we don't like the term Legacy, I guess maybe I'd call it CesiumQuantizedMeshTerrain? Though that's not quite right either because this library would also include the even more legacy heightmap-1.0 format if we ever feel the need to support it.

Anyone who has a strong opinion on this, please chime in.

I don't have a strong enough opinion to push too hard. More like, if I can convince you with the reasons, great. If not, let's move on.
(take these musings with a grain of salt, just a personal take)

Naming things with "legacy" is naming them based on the plan for them, instead of what they actually do
The plan for a legacy thing can change. It may not go away. Worst case, it actually becomes more prominent
You still have to get past the name, and figure out what the thing actually is

@kring @csciguy8 I would suggest to avoid "legacy" in the name as Cesium is still actively supporting and producing data in quantized-mesh.

I'm getting an error generating the gltf classes.

This was caused by a change in the glTF schemas. It was fixed in this PR:
https://github.com/CesiumGS/cesium-native/pull/851
So we just need to merge in main.

I would suggest to avoid "legacy" in the name as Cesium is still actively supporting and producing data in quantized-mesh.

Thanks @pjcozzi.
Maybe CesiumOGTerrain then? 😆 In all seriousness, I think CesiumQuantizedMeshTerain is the best alternative, so I'll rename it to that.

Maybe CesiumOGTerrain then? 😆 In all seriousness, I think CesiumQuantizedMeshTerain is the best alternative, so I'll rename it to that.

@kring lol. 👍

@csciguy8 I've merged in main and renamed the library / namespace. This is ready for another look.

Your changes look good to me. Thanks @csciguy8!


Wrong AppConfig property value (Set Property: AppConfig) - prepends .csproj name

Trying to run a build using msbuild 15. It fails on trying to copy the .config file from the obj\debug\net452 folder but for some reason it's adding the full .csproj name
so code.exe.config is resolved as Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
08:13:09.957     1>Target "GenerateBindingRedirectsUpdateAppConfig: (TargetId:127)" in file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\amd64\Microsoft.Common.CurrentVersion.targets" from project "C:\Projects\Sitecore.Demo.Retail\src\Foundation\Commerce\Engine\code\Sitecore.Foundation.Commerce.Engine.csproj" (target "ResolveReferences" depends on it):
                   Set Property: AppConfig=obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
                   Added Item(s):
                       AppConfigWithTargetPath=
                           obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
                                   TargetPath=code.exe.config

Here is the Sitecore.Foundation.Commerce.Engine.csproj content:
<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk.Web" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <TargetFramework>net452</TargetFramework>
    <PreserveCompilationContext>false</PreserveCompilationContext>
    <AssemblyName>code</AssemblyName>
    <OutputType>Exe</OutputType>
    <PackageId>code</PackageId>
  </PropertyGroup>

  <!--<ItemGroup>
    <None Update="wwwroot\**\*">
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
    </None>
  </ItemGroup>-->

  <ItemGroup>
    <ProjectReference Include="..\..\legacyCommerce\Plugin.Sample.Habitat\Plugin.Sample.Habitat.csproj" />
    <ProjectReference Include="..\..\legacyCommerce\Plugin.Sample.Payments.Braintree\Plugin.Sample.Payments.Braintree.csproj" />
    <ProjectReference Include="..\..\legacyCommerce\Sitecore.Commerce.Plugin.AdventureWorks\Sitecore.Commerce.Plugin.AdventureWorks.csproj" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Diagnostics" Version="1.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="1.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="1.0.3" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="1.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.OData" Version="6.0.0-alpha1-rtm-121216" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.1.0" />
    <PackageReference Include="Serilog" Version="2.4.0" />
    <PackageReference Include="Sitecore.Commerce.Core" Version="1.0.2301" />
    <PackageReference Include="Serilog.Sinks.Literate" Version="2.1.0" />
    <PackageReference Include="Serilog.Sinks.RollingFile" Version="3.3.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Filter" Version="1.1.1" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="1.4.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="1.0.2" />
    <PackageReference Include="Sitecore.Commerce.Provider.FileSystem" Version="1.0.2301" />
    <PackageReference Include="Sitecore.Framework.Rules" Version="1.1.12" />
    <PackageReference Include="Sitecore.Framework.Rules.Serialization" Version="1.1.12" />
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="1.1.1" />
    <PackageReference Include="Sitecore.Framework.Diagnostics" Version="1.1.4" />
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetFramework)' == 'net452' ">
    <Reference Include="System" />
    <Reference Include="Microsoft.CSharp" />
  </ItemGroup>

</Project>

What is the actual error that you're seeing? With your project (removing project and package references I didn't have), I see successful builds with
_CopyAppConfigFile:
  Copying file from "obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config" to "bin\Debug\net452\code.exe.config".

Are you just running into a MAX_PATH problem like https://github.com/Microsoft/msbuild/issues/1786?

I apologize for leaving the most important part out.
File not found is what I'm getting because it's resolving the config file as Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config as opposed to code.exe.config as it seemingly should be as that's the assemblyName.
08:13:11.290     1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Publish.targets(128,5): error MSB3030: Could not copy the file "obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config" because it was not found.
code.exe.config exists in obj\debug\net452 but Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config does not

That file should be created in the GenerateBindingRedirects target (and is on my machine). Can you build with a diagnostic log (msbuild /flp:v=diag) and share the resultant msbuild.log?
From my repro:
Target "GenerateBindingRedirects: (TargetId:36)" in file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\bin\Microsoft.Common.CurrentVersion.targets" from project "S:\work\msb1873\Sitecore.Foundation.Commerce.Engine.csproj" (target "ResolveReferences" depends on it):
Building target "GenerateBindingRedirects" completely.
Output file "obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config" does not exist.
Using "GenerateBindingRedirects" task from assembly "Microsoft.Build.Tasks.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "GenerateBindingRedirects" (TaskId:16)
...
  Output Item(s):
      FileWrites=
          obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
                  OriginalItemSpec=app.config
                  TargetPath=code.exe.config (TaskId:16)
Done executing task "GenerateBindingRedirects". (TaskId:16)

Here is the log
build-log.zip

@rainersigwald any chance you were able to glance at my log? I've been sitting here pressing F5 for over two hours. Kidding of course but this one is a blocker for me (aren't they always?).
Thanks a bunch!

Continuing to work on this, I changed the AssemblyName in the project properties from code to Sitecore.Foundation.Commerce.Engine and now the "file not found" is looking for:
Sitecore.Foundation.Commerce.Engine.csproj.Sitecore.Foundation.Commerce.Engine.exe.config

@jflsitecore To set expectations, issues filed in this repo aren't under an SLA and we don't guarantee that we'll help you investigate in any kind of timely fashion.

I certainly wasn't "expecting" a quick response. I was gently asking. At this point I imagine we'll probably look at opening an incident for this.
Appreciate you looking at this in the first place. If nobody had a clue by the morning I'll create a request.
Cheers!
Sent from my iPhone
On Mar 16, 2017, at 15:02, Rainer Sigwald <notifications<EMAIL_ADDRESS>wrote:
@jflsitecorehttps://github.com/jflsitecore To set expectations, issues filed in this repo aren't under an SLA and we don't guarantee that we'll help you investigate in any kind of timely fashion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/msbuild/issues/1873#issuecomment-287159107, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AY24vgXYzA1gXe-1gzWUh3MAwBFZJzPVks5rmYcsgaJpZM4MfQIV.
This message and its attachments are for the designated recipient only and may contain proprietary or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other sharing, copying or use of this message by you is strictly prohibited.

This is happening because GenerateBindingRedirectsUpdateAppConfig is running, which specifies that the app config points to _GenerateBindingRedirectsIntermediateAppConfig, which is the filename that is later complained about being missing.
08:13:09.957     1>Target "GenerateBindingRedirectsUpdateAppConfig: (TargetId:127)" in file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\amd64\Microsoft.Common.CurrentVersion.targets" from project "C:\Projects\Sitecore.Demo.Retail\src\Foundation\Commerce\Engine\code\Sitecore.Foundation.Commerce.Engine.csproj" (target "ResolveReferences" depends on it):
                   Set Property: AppConfig=obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
                   Added Item(s):
                       AppConfigWithTargetPath=
                           obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
                                   TargetPath=code.exe.config

That target is condiditoned on Exists('$(_GenerateBindingRedirectsIntermediateAppConfig)'). On my machine, that file exists and is generated by the GenerateBindingRedirects target, which produces it because it gets a list of SuggestedBindingRedirects from ResolveAssemblyReferences. In your log it apparently exists (since that Exists() condition fires) but then doesn't get produced later. That's confusing.
One of the redirected assemblies on my machine is Microsoft.Extensions.Configuration. RAR logs this on my machine:
  Unified primary reference "Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60". (TaskId:15)
      Using this version instead of original version "<IP_ADDRESS>" in "C:\Users\raines\.nuget\packages\microsoft.applicationinsights.aspnetcore\1.0.2\lib\net451\Microsoft.ApplicationInsights.AspNetCore.dll" because AutoUnify is 'true'. (TaskId:15)
      Resolved file path is "C:\Users\raines\.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.dll". (TaskId:15)
      Reference found at search path location "{RawFileName}". (TaskId:15)
      Found related file "C:\Users\raines\.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.xml". (TaskId:15)
      This reference is not "CopyLocal" because at least one source item had "Private" set to "false" and no source items had "Private" set to "true". (TaskId:15)
      The ImageRuntimeVersion for this reference is "v4.0.30319". (TaskId:15)

In your log the equivalent is
Primary reference "Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60". (TaskId:71)
    Resolved file path is "C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.dll". (TaskId:71)
    Reference found at search path location "{RawFileName}". (TaskId:71)
    Found related file "C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.xml". (TaskId:71)
    This reference is not "CopyLocal" because at least one source item had "Private" set to "false" and no source items had "Private" set to "true". (TaskId:71)
    The ImageRuntimeVersion for this reference is "v4.0.30319". (TaskId:71)

That is odd. Can you ildasm or similar and confirm that C:\Users\jfl\.nuget\packages\microsoft.applicationinsights.aspnetcore\1.0.2\lib\net451\Microsoft.ApplicationInsights.AspNetCore.dll has a reference to Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>?

I'll take a peek first thing in the morning ~6am eastern!
Sent from my iPhone
On Mar 16, 2017, at 15:27, Rainer Sigwald <notifications<EMAIL_ADDRESS>wrote:
This is happening because GenerateBindingRedirectsUpdateAppConfig is running, which specifies that the app config points to _GenerateBindingRedirectsIntermediateAppConfig, which is the filename that is later complained about being missing.
08:13:09.957     1>Target "GenerateBindingRedirectsUpdateAppConfig: (TargetId:127)" in file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\amd64\Microsoft.Common.CurrentVersion.targets" from project "C:\Projects\Sitecore.Demo.Retail\src\Foundation\Commerce\Engine\code\Sitecore.Foundation.Commerce.Engine.csproj" (target "ResolveReferences" depends on it):
Set Property: AppConfig=obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
Added Item(s):
AppConfigWithTargetPath=
obj\Debug\net452\Sitecore.Foundation.Commerce.Engine.csproj.code.exe.config
TargetPath=code.exe.config
That target is condiditonedhttps://github.com/Microsoft/msbuild/blob/8ec190f8bb2219d150e5c19bf72e1df89c6e638c/src/Tasks/Microsoft.Common.CurrentVersion.targets#L2065 on Exists('$(_GenerateBindingRedirectsIntermediateAppConfig)'). On my machine, that file exists and is generated by the GenerateBindingRedirects target, which produces it because it gets a list of SuggestedBindingRedirects from ResolveAssemblyReferences. In your log it apparently exists (since that Exists() condition fires) but then doesn't get produced later. That's confusing.
One of the redirected assemblies on my machine is Microsoft.Extensions.Configuration. RAR logs this on my machine:
Unified primary reference "Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60". (TaskId:15)
Using this version instead of original version "<IP_ADDRESS>" in "C:\Users\raines.nuget\packages\microsoft.applicationinsights.aspnetcore\1.0.2\lib\net451\Microsoft.ApplicationInsights.AspNetCore.dll" because AutoUnify is 'true'. (TaskId:15)
Resolved file path is "C:\Users\raines.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.dll". (TaskId:15)
Reference found at search path location "{RawFileName}". (TaskId:15)
Found related file "C:\Users\raines.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.xml". (TaskId:15)
This reference is not "CopyLocal" because at least one source item had "Private" set to "false" and no source items had "Private" set to "true". (TaskId:15)
The ImageRuntimeVersion for this reference is "v4.0.30319". (TaskId:15)
In your log the equivalent is
Primary reference "Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60". (TaskId:71)
Resolved file path is "C:\Users\jfl.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.dll". (TaskId:71)
Reference found at search path location "{RawFileName}". (TaskId:71)
Found related file "C:\Users\jfl.nuget\packages\microsoft.extensions.configuration\1.0.2\lib\netstandard1.1\Microsoft.Extensions.Configuration.xml". (TaskId:71)
This reference is not "CopyLocal" because at least one source item had "Private" set to "false" and no source items had "Private" set to "true". (TaskId:71)
The ImageRuntimeVersion for this reference is "v4.0.30319". (TaskId:71)
That is odd. Can you ildasm or similar and confirm that C:\Users\jfl.nuget\packages\microsoft.applicationinsights.aspnetcore\1.0.2\lib\net451\Microsoft.ApplicationInsights.AspNetCore.dll has a reference to Microsoft.Extensions.Configuration, Version=<IP_ADDRESS>?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/msbuild/issues/1873#issuecomment-287165698, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AY24vo0m7rWISpbFMXO4qyu-CW7B3VAdks5rmY0tgaJpZM4MfQIV.
This message and its attachments are for the designated recipient only and may contain proprietary or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other sharing, copying or use of this message by you is strictly prohibited.

@rainersigwald
I can confirm that
C:\Users\jfl.nuget\packages\microsoft.applicationinsights.aspnetcore\1.0.2\lib\net451\Microsoft.ApplicationInsights.AspNetCore.dll has a reference to Microsoft.Extensions.Configuration v <IP_ADDRESS>
Forgive my lack of experience with this area.
I see that Microsoft.Extensions.Configuration has a version 1.0.0 and 1.0.2 in my packages cache (I initially cleared it and restored packages). Inside both of those folders they only have netstandard1.1 folder in the lib folder. Not sure if that means anything since the applicationinsights is net452?

C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration>dir 1.0.0\lib
 Volume in drive C is Windows
 Volume Serial Number is 3A27-4900

 Directory of C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration\1.0.0\lib

03/17/2017  05:56 AM    <DIR>          .
03/17/2017  05:56 AM    <DIR>          ..
03/17/2017  05:56 AM    <DIR>          netstandard1.1
               0 File(s)              0 bytes
               3 Dir(s)  230,475,849,728 bytes free

C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration>dir 1.0.2\lib
 Volume in drive C is Windows
 Volume Serial Number is 3A27-4900

 Directory of C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration\1.0.2\lib

03/17/2017  05:56 AM    <DIR>          .
03/17/2017  05:56 AM    <DIR>          ..
03/17/2017  05:56 AM    <DIR>          netstandard1.1
               0 File(s)              0 bytes
               3 Dir(s)  230,475,849,728 bytes free

C:\Users\jfl\.nuget\packages\microsoft.extensions.configuration>

One possible mitigation would be to drop an empty app.config file next to the project and let it get piped through the system that way.

My colleague is having an unrelated VS 2015 / .NET Core issue. He’s trying to re-create the projects/solution from scratch. I’ll try your suggestion and at the same time we’ll see if his approach provides us with a working build!
Thanks for the follow-up.
From: Rainer Sigwald<EMAIL_ADDRESS>Sent: Friday, March 17, 2017 2:30 PM
To: Microsoft/msbuild<EMAIL_ADDRESS>Cc: Jean-François Larente<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Microsoft/msbuild] Wrong AppConfig property value (Set Property: AppConfig) - prepends .csproj name (#1873)
One possible mitigation would be to drop an empty app.config file next to the project and let it get piped through the system that way.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/msbuild/issues/1873#issuecomment-287435790, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AY24vh9qZp6c6mj_UL9CTO0m_d5HNWFUks5rmtEzgaJpZM4MfQIV.
This message and its attachments are for the designated recipient only and may contain proprietary or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other sharing, copying or use of this message by you is strictly prohibited.

Team Triage:
Closing this for now. Please followup if you reproduce the problem.


Forcasting usign DSGE model

Hi everyone!
I'm trying to forecast the observable variable using an estimated and calibrated standard DSGE using the Dsgepy library. Using some real data from FRED I calibrated the model but now i have some difficulty to forecast GDP, interest rates, exc. There are some examples or functions already in Python?

The DSGE object in this library currently does not have a forecasting, but I intend to add this functionality in the future. But the matrices that make up the model are available. So you can build your own forecast using those matrices and the Kalman filter. And if you do, please consider contributing on a Pull Request.

Thank you Gustavo, I think that given my level in coding with python I will wait. However, you have scheduled the implications in the next month? Or it's something that probably it will take years?
Best.

This library is a pet project of mine. Is not under active development. But when I find myself bored during a rainy Sunday, I like to work on it. So, I cannot give any deadlines for implementation on any new feature. But I take notes of people's requests, and adding a native Kalman Filter is one the developments that I have at the top of my list.

An interesting way to pass the time 😅
However also since on DSGE models in python there is a lack of libraries, yours has some nice features.
For your knowledge I am attaching a link to a notebook where the forecast feature is implemented: https://github.com/nflp/dsge_forecasting/blob/main/DSGE forecasting.ipynb
Thank you for your time Gustavo! 😊


[Feature Request]: Validate that embeddings on add are numeric

What happened?
We should Validate that embeddings on add are numeric.
Versions
All
Relevant log output
No response

Closed by #481


Restore the NextMessageSize for streamed calls

#9758 is filed to improve tests.

What goes wrong here? Is it possible to just not pass the constructor argument to Call?
The max_receive_message_size property here is a misfeature that I'd really like to kill (since service configs handle this functionality differently and elsewhere)

#9668 #9762

Sorry, I don't get the connection of this issue to the other issues that you just posted.

Comment updated.


How to get Pose Θ(3D joint locations), Shape β from the pre-trained model?

Thank you for your work.
Currently, I run demo on images with 4DHumans, but the output is mesh (with 6890 vertices).
How can I get Pose Θ(3D joint locations of 24 joints), Shape β (betas) with format below on 4DHumans pre-trained model?
{
  "pose": [
    -2.0428097248077393,
    0.11024858802556992,
    -2.333742141723633,
    -0.3365379273891449,
    0.011719911359250546,
    0.15597772598266602,
    -0.18535791337490082,
    -0.05565245449542999,
    -0.0701599195599556,
    0.5772491693496704,
    0.032217949628829956,
    -0.024495713412761688,
    0.15936070680618286,
    0.024038009345531464,
    0.015323049388825893,
    0.219781294465065,
    0.010715791955590248,
    0.03143862262368202,
    0.06791213899850845,
    0.015769684687256813,
    -0.026596715673804283,
    0.09402766823768616,
    0.061577245593070984,
    -0.07461484521627426,
    0.03821762278676033,
    -0.07338492572307587,
    0.09183917939662933,
    0.03599092736840248,
    0.027726667001843452,
    -0.03167799487709999,
    -0.08593683689832687,
    0.13839715719223022,
    0.06819991767406464,
    -0.04061942175030708,
    -0.05313720554113388,
    -0.06050587072968483,
    0.33442655205726624,
    0.0762828141450882,
    -0.033081308007240295,
    -0.02586432732641697,
    -0.26066192984580994,
    -0.03644227609038353,
    -0.016138266772031784,
    0.33069372177124023,
    0.03311172500252724,
    0.17657236754894257,
    0.04906424507498741,
    -0.07966049760580063,
    0.022382577881217003,
    -0.633172333240509,
    -0.9721075892448425,
    -0.05635237693786621,
    0.6847463846206665,
    0.6975662112236023,
    0.23132619261741638,
    -1.2001522779464722,
    0.2783845365047455,
    0.20647276937961578,
    1.0782970190048218,
    -0.22080913186073303,
    -0.08577798306941986,
    -0.1102685034275055,
    0.21007774770259857,
    0.030228132382035255,
    0.13235852122306824,
    -0.22589533030986786,
    -0.07216395437717438,
    -0.07743184268474579,
    -0.12528756260871887,
    -0.10984508693218231,
    0.04706919193267822,
    0.13594649732112885
  ],
  "betas": [
    0.13400566577911377,
    -0.12246442586183548,
    0.7548058032989502,
    0.6801249384880066,
    0.16374310851097107,
    0.17321491241455078,
    0.0016920268535614014,
    -0.013438664376735687,
    0.7659109830856323,
    -0.003035418689250946
  ]
}

You can check out['pred_smpl_params'] in demo.py. This includes the global_orient, body_pose and betas fields that correspond to the SMPL parameters.

@kaenkogashi :

There are 23 body joints and 1 root joint. You can find the root joint pose in d['pred_smpl_params']['global_orient'].
The provided rotations are in (3x3) rotation matrix format. You'll need to convert that to a (3x1) euler angle representation.
Then you can append the root joint and body joint poses to a single array. (I am not sure if the root joint goes at the start or at the end. But as far as I know, it usually goes at the start).

@geopavlakos , @AmoghTiwari
Thank you very much for your help!
I have one more question.
After I get  (3x1) euler angle representation, how do I convert them into 3D joint positions (x,y,z) that can be plotted?
Does the 4D-Humans pre-trained model also output 3D joint positions (x,y,z)?

4D joint positions are already predicted. Just that they are not saved anywhere.
In the demo.py script, after the line out = model(batch) (roughly around line-80). You can dump the variable out using joblib or pkl, and then inspect that variable. I had done this on a local fork of mine here. You can use it for reference.
Once you have the variable out saved in a pkl file, just load that pkl file and search for a key named 'kp3d' or 'j3d' (or something similar).

@AmoghTiwari
Thank you very much for your reply.
I get keys named "pred_keypoints_3d" and "pred_keypoints_2d" below. But the format looks like there are 44 joints? (not 24)
Also the number are decimals. How can I convert these decimals into integer like 3D joint positions (x,y,z)?
'pred_keypoints_3d': tensor([[[ 0.3246, -0.6181,  0.0810],
         [ 0.1618, -0.6862,  0.0181],
         [ 0.2146, -0.5923, -0.1336],
         [ 0.2971, -0.3519, -0.1694],
         [ 0.4247, -0.2721,  0.0391],
         [ 0.0774, -0.6367,  0.1792],
         [ 0.0061, -0.4214,  0.3127],
         [ 0.2192, -0.2901,  0.3288],
         [-0.0017, -0.2209,  0.0271],
         [ 0.0045, -0.1198, -0.0223],
         [ 0.0334,  0.2675, -0.0160],
         [-0.0213,  0.6650,  0.0361],
         [-0.0314, -0.1442,  0.1070],
         [ 0.0443,  0.1899,  0.2736],
         [ 0.0693,  0.5707,  0.4071],
         [ 0.3519, -0.6624,  0.0482],
         [ 0.3340, -0.6696,  0.1120],
         [ 0.3081, -0.7314, -0.0170],
         [ 0.2684, -0.7477,  0.1247],
         [ 0.2586,  0.5859,  0.4818],
         [ 0.1890,  0.5828,  0.5283],
         [ 0.0170,  0.6154,  0.4075],
         [ 0.1676,  0.7250,  0.0811],
         [ 0.1277,  0.7427,  0.0047],
         [-0.0813,  0.7069,  0.0303],
         [-0.0418,  0.6170,  0.0041],
         [ 0.0313,  0.1997, -0.0197],
         [ 0.0257, -0.2265, -0.1207],
         [-0.0643, -0.2641,  0.1538],
         [ 0.0333,  0.1468,  0.2665],
         [ 0.0384,  0.5413,  0.3958],
         [ 0.4115, -0.3083,  0.0389],
         [ 0.3249, -0.3799, -0.1664],
         [ 0.2080, -0.6213, -0.1072],
         [ 0.0890, -0.6579,  0.1545],
         [ 0.0407, -0.4556,  0.3291],
         [ 0.2188, -0.3143,  0.3285],
         [ 0.1698, -0.6807,  0.0264],
         [ 0.3398, -0.7529,  0.0712],
         [-0.0197, -0.2456,  0.0168],
         [ 0.1465, -0.6357,  0.0224],
         [ 0.0368, -0.4860, -0.0071],
         [ 0.2655, -0.6670,  0.0630],
         [ 0.3523, -0.7691,  0.0686]]], device='cuda:0'), 'pred_vertices': tensor([[[ 0.3709, -0.6959,  0.1323],
         [ 0.3634, -0.6816,  0.1258],
         [ 0.3532, -0.6887,  0.1333],
         ...,
         [ 0.3177, -0.7356, -0.0179],
         [ 0.3202, -0.7351, -0.0172],
         [ 0.3202, -0.7312, -0.0196]]], device='cuda:0'), 'pred_keypoints_2d': tensor([[[ 0.0959, -0.3680],
         [-0.0024, -0.4099],
         [ 0.0296, -0.3547],
         [ 0.0799, -0.2090],
         [ 0.1565, -0.1594],
         [-0.0532, -0.3781],
         [-0.0957, -0.2475],
         [ 0.0320, -0.1688],
         [-0.1013, -0.1285],
         [-0.0976, -0.0674],
         [-0.0802,  0.1671],
         [-0.1131,  0.4070],
         [-0.1189, -0.0819],
         [-0.0729,  0.1190],
         [-0.0576,  0.3461],
         [ 0.1124, -0.3952],
         [ 0.1014, -0.3987],
         [ 0.0862, -0.4377],
         [ 0.0618, -0.4456],
         [ 0.0554,  0.3543],
         [ 0.0138,  0.3520],
         [-0.0889,  0.3728],
         [ 0.0011,  0.4427],
         [-0.0230,  0.4544],
         [-0.1494,  0.4324],
         [-0.1256,  0.3784],
         [-0.0814,  0.1260],
         [-0.0851, -0.1324],
         [-0.1386, -0.1540],
         [-0.0795,  0.0932],
         [-0.0761,  0.3286],
         [ 0.1485, -0.1812],
         [ 0.0968, -0.2259],
         [ 0.0256, -0.3721],
         [-0.0462, -0.3912],
         [-0.0750, -0.2679],
         [ 0.0318, -0.1832],
         [ 0.0025, -0.4064],
         [ 0.1051, -0.4495],
         [-0.1122, -0.1435],
         [-0.0117, -0.3793],
         [-0.0781, -0.2890],
         [ 0.0602, -0.3977],
         [ 0.1126, -0.4593]]], device='cuda:0')}

@kaenkogashi: The provided decimal values itself are the joint positions. They are supposed to be in floating point values (decimals) right? Why do you want them to be integers?
I am not sure about the 44 joint thing. I have observed this thing for a few other methods too, but not sure why this is the case. @geopavlakos / @shubham-goel : Could you please clarify this.

@AmoghTiwari
Thank you very much for your comments.

Why do you want them to be integers?
For example, in OpenPose key points detection (as same as 'pred_keypoints_2d' value above) of below picture.
As the pixel size of the image is 683*1024, so the detected blue points (Neck) shall be (x,y)=(357,254).
However, the output of 4D-Humans's  'pred_keypoints_2d'  is not integer, but decimals.
I just wondering if  'pred_keypoints_2d' is not in pixel scale, then how can I visualize the output of 'pred_keypoints_2d' or 'pred_keypoints_3d' ?

@geopavlakos / @shubham-goel
Thank you for your great work!
Would you tell me the relationship between the mesh (6890 vertices ),  Pose Θ(3D joint locations of 24 joints), Shape β (betas), and 'pred_keypoints_2d' or 'pred_keypoints_3d' ?
In 4d-humans, the model predict  Pose Θ and Shape β in the first, then calculate mesh from  Pose Θ and Shape β?
The model predict 'pred_keypoints_2d' or 'pred_keypoints_3d' in which stages?

Regarding the joints, we augment the joints that SMPL outputs with some extra joint/landmark locations that are helpful for training/supervising HMR2.0 (e.g., eyes, ears, etc). For a list of the joints we use, you can check the list here. We use the first 44 of the 49 joints of that list.
The pred_keypoints_2d correspond to the crop that HMR2.0 sees (not to the full image) and they are normalized to [-0.5, 0.5]. If you want to convert them such that they correspond to the pixel space of the full image, you need to multiply with the box_size and add the box_center. In pseudocode:
pred_keypoints_2d_full = pred_keypoints_2d*box_size+box_center
Finally, HMR2.0 predicts pose \theta and shape \beta. If you pass these parameters through SMPL, you will get the mesh vertices and 3D joints, pred_keypoints_3d. If you project the 3D joints to the image plane through the predicted camera parameters, you get the 2D joints, pred_keypoints_2d.

Thank you very much for the reply!

@geopavlakos , @AmoghTiwari
Thank you very much for your help!
I have one more question. After I get (3x1) euler angle representation, how do I convert them into 3D joint positions (x,y,z) that can be plotted? Does the 4D-Humans pre-trained model also output 3D joint positions (x,y,z)?

Hello everyone, I have one more question， how to convert (3x3) rotation matrix format to (3x1) euler angle representation

Check this out: https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.transform.Rotation.html
Within this, look at the ".as_rot_euler()" function.
You can initialize an object using the "from_matrix()" function. And then  use the "as_euler()" function.
A rough pseudo code would be something like this:
r = R.from_matrix(<rotn_matrix>)
euler_angles = r.as_euler('xyz')
Pls read the documentation carefully for more details.

Regarding the joints, we augment the joints that SMPL outputs with some extra joint/landmark locations that are helpful for training/supervising HMR2.0 (e.g., eyes, ears, etc). For a list of the joints we use, you can check the list here. We use the first 44 of the 49 joints of that list.
The pred_keypoints_2d correspond to the crop that HMR2.0 sees (not to the full image) and they are normalized to [-0.5, 0.5]. If you want to convert them such that they correspond to the pixel space of the full image, you need to multiply with the box_size and add the box_center. In pseudocode: pred_keypoints_2d_full = pred_keypoints_2d*box_size+box_center
Finally, HMR2.0 predicts pose \theta and shape \beta. If you pass these parameters through SMPL, you will get the mesh vertices and 3D joints, pred_keypoints_3d. If you project the 3D joints to the image plane through the predicted camera parameters, you get the 2D joints, pred_keypoints_2d.

Thanks for the guide, when I use the formula you provided pred_keypoints_2d_full = out['pred_keypoints_2d'] * box_size + box_center
Buf I can't get the 3d joints to project correctly to the 2d image.
(Where the white circle is the 2d joint)

When I use other images, this has the potential to work, but something is wrong with this one

@AmoghTiwari @kaenkogashi
I got a .pkl file by running track.py, and in that file I extracted the 3d_joints, and intercepted some of the information as follows:
3d_joints: [array([[-2.8718486e-01, -7.5166744e-01,  2.2998667e-01],
[-1.6642219e-01, -6.6351068e-01,  1.0292265e-01],
[-9.6075758e-03, -6.2545604e-01,  2.0373139e-01],
[ 1.0279357e-01, -4.2896354e-01,  3.1757167e-01],
[-6.7978762e-02, -3.5120371e-01,  4.8845506e-01],
[-2.6760888e-01, -5.3899467e-01, -8.4407069e-03],
[-2.5292975e-01, -2.8882870e-01, -7.7291429e-02],
[-4.0486607e-01, -1.5837581e-01,  6.7398876e-02],
[-1.5314636e-03, -2.1615791e-01,  2.6278008e-02],
[ 7.4146807e-02, -1.3799396e-01,  5.2112691e-02],
[ 2.6961237e-01,  1.7003253e-01,  1.7404395e-01],
[ 4.5271403e-01,  5.0126529e-01,  4.6340257e-02],
[-4.7355521e-02, -1.1685089e-01, -5.3188279e-03],
[-2.9106852e-01,  1.1983880e-01,  1.5947574e-01],
[-2.3855132e-01,  5.0827360e-01,  8.3282381e-02],
[-2.4569771e-01, -7.9643202e-01,  2.2368461e-01],
[-2.9768065e-01, -7.8838718e-01,  1.8275031e-01],
[-1.5319166e-01, -7.9065788e-01,  1.7652753e-01],
[-2.6787433e-01, -7.7296245e-01,  8.6717866e-02],
[-3.6278105e-01,  5.8073884e-01,  2.2207400e-01],
[-3.7149715e-01,  5.8517128e-01,  1.4058727e-01],
[-1.9863102e-01,  5.4008555e-01,  3.5266701e-02],
[ 4.3201709e-01,  5.7590663e-01,  2.2899407e-01],
[ 5.0781614e-01,  5.5226767e-01,  1.9430286e-01],
[ 4.8288533e-01,  5.2856648e-01, -1.2670055e-02],
[ 4.5059079e-01,  4.4229305e-01,  3.6990423e-02],
[ 2.3787317e-01,  1.1562512e-01,  1.6140664e-01],
[ 1.2874840e-01, -2.7276033e-01,  7.2813749e-02],
[-1.1671343e-01, -2.1542217e-01, -7.0440099e-02],
[-2.8482258e-01,  8.8945180e-02,  1.4328843e-01],
[-2.2954750e-01,  4.7395110e-01,  6.3516118e-02],
[-7.7342778e-02, -3.7666583e-01,  4.6462077e-01],
[ 9.1598317e-02, -4.6208042e-01,  3.3960426e-01],
[-3.8918350e-02, -6.4481968e-01,  1.8522283e-01],
[-2.5571617e-01, -5.7589465e-01,  4.8243771e-03],
[-2.9661471e-01, -3.1598577e-01, -7.2813287e-02],
[-4.0797624e-01, -1.7939991e-01,  6.2734440e-02],
[-1.7192465e-01, -6.5598309e-01,  1.0553554e-01],
[-2.2858347e-01, -8.3181155e-01,  1.3771361e-01],
[ 6.1788335e-03, -2.4410549e-01,  5.6433678e-04],
[-1.4638546e-01, -6.0630494e-01,  9.5950007e-02],
[-5.9450842e-02, -4.6149611e-01,  4.3473933e-02],
[-2.4040745e-01, -7.2623658e-01,  1.6699974e-01],
[-2.2167268e-01, -8.5226554e-01,  1.3193452e-01],
[ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00]], dtype=float32),
There are 45 lines in total.My understanding is that this is information about the joints of a person detected in that frame, and how he corresponds to each of the 44 joints, for example, which joints correspond to the first row, do you understand that part of the information?


split function

split(i0, [i1, i2, ...]) will return a list of indices [j1, j2, ...] such that a[i0] = concat(a[i1][j1], a[i2][j2], ...).
This is needed primarily so that we can split an index into chunks, but this seems like a good generalization.

split_slices here is a simple version of this.

I believe what's needed here is a way to project one slice onto another. k projecting onto i would give the index j such that i[j] = k, which works like an inverse of a composition https://github.com/Quansight/ndindex/issues/27 (although it is only a true inverse if k is a subset of i).
I'm not really sure what to call this operation. Let's just call it (i^-1)[k] for now for lack of a better notation. Given it,split(i0, [i1, i2, ...]) becomes [(i1^-1)[i0], (i2^-1)[i0], ...].
Note that my "concat" notation above is very ambiguous. It doesn't specify how concatenation works in multiple dimensions. It is also only true that a[i0] would equal the concatenation if a[i1], a[i2], ... completely covered a[i0]. But I don't think that is a necessary requirement for this projection idea.
An idea for the name might be lift, since we are "lifting" k into a slice on i. My main worry is that the operation is noncommutative, so the operation name and syntax should make it clear which direction things are happening in.

I also considered using / for this operation, but I'm not sure about it. A "right" inverse of i[j] = k is ambiguous. Given j and k, there are potentially any number of is such that i[j] = k, since one can just expand i to include more elements than are indexed by j. Also, there may be no such i, if they have incompatible lengths for instance. But given i and k, if k is a subset of i (all elements indexed by k are also indexed by i) one can always find an index j such that i[j] = k. This is basically true by assumption, and the fact that an index j can index anything (some cases may require explicitly listing all indices via an integer array index). If k is not a subset of i, you can still find an index j such that i[j] = k | i, i.e., j selects those elements of a[i] that are also included in a[k].
Here is a picture
Here k is a subset of i, so a[i][j] == a[k]
         +- k -+
         |     |
------- a ---------
       |         |
       +--- i ---+
         |     |
         +- j -+

Here k is not a subset, so a[i][j] = a[k | i]:
     +-- k --+
     |       |
------- a ---------
       |         |
       +--- i ---+
       |     |
       +- j -+

And one can also find a l so that a[k][l] = a[k | i]
       +- l -+
       |     |
     +-- k --+
     |       |
------- a ---------
       |         |
       +--- i ---+

So I'm not sure about the notation k/i for instance (or any other notation or function name, for that matter). It needs to be clear that you are getting j and not l. One is a recursive index of i and another is a recursive index of k. Since both make sense, there is no sort of type system to help you here.

Work started at https://github.com/Quansight/ndindex/pull/36 with the provisional name as_subindex.

This can be closed with the work on #36. There are still some cases that need to be implemented, but the basic idea is there now.


Lucene.Net.Search.ControlledRealTimeReopenThread: Refactor to integrate changes from #513

See #513 and #519.
This re-applies the changes that were introduced in #513 and patches them so they don't deadlock on macOS.

Lock was removed from Dispose(bool)
AtomicBoolean added to ensure Dispose(bool) is only entered by a single thread.
AtomicInt64 added for refreshStartGen, which is updated and read outside of locks.
Use constants for second/millisecond/nanosecond conversions.
Ignored TestStraightForwardDemonstration() and TestMultithreadedWaitForGeneration(), since they often fail in CI because of timing check failures (tolerance is too low)
Renamed intrinsic field to m_signal and marked it protected so subclasses can receive signaling events.
Use ConcurrentSet<T> rather than ConcurrentHashSet<T> in ThreadedIndexingAndSearchingTestCase.
Use using block on LineFileDocs in ThreadedIndexingAndSearchingTestCase.

I strongly prefer my variable naming of intrinsic for the additional EventWaitHandle over the new name of m_signal.  The former helps throw light on the fact that it’s being used to mimic java's ability to signal on the intrinsic class which c# can’t do and hence the use of the extraEventWaitHandle.
You changed refreshStartGen from a long to a AtomicInt64.  I don’t believe this changes is necessary.  This variable is a long in the original java code.
You changed isDispose from bool to AtomicBoolean.  I don’t believe this change is necessary.  Reading and writing a Boolean is inherently atomic  see Is a bool read/write atomic in C#
You removed the holding of a lock on the Dispose method.  This is a major drift from the Java implementation.  Please note the java close method for this class is a synchronized method.
By removing the searchingGen = long.MaxValue;  line in the Dispose method and using a conditional assignment of searchingGen  in the RefreshDone method this new version of the code does not track as closely with the java code as my original ported version.  This change is also why in the Dispose method you needed to replace the call to Intrensic.Set() and it’s very helpful comment with a call to RefrsehDone()   which adds more drift from the java implementation.    It's clear these changes improve the code but they do make it harder to compare to the java code.

Okay, let's try to establish first what we are trying to achieve:

A direct line-by-line port from Java to C#
A (mostly) lockless implementation that functions similarly to Lucene's original design

We tried a direct line-by-line port, and it doesn't function correctly in .NET because of low level differences in the way concurrency is implemented. It was changed to its current (mostly) lockless form in the following 2 commits:
https://github.com/apache/lucenenet/commit/d3621f0c3a02bf3c82603c1d94916e70e06b3035
https://github.com/apache/lucenenet/commit/dfb2e92839dd29cbdc47d8c37af9325ea351709b
IMO, as long as it functions correctly (i.e. passes all of the tests and doesn't deadlock), the form is less important than the function. The fact that it is lockless is more in line with how modern .NET code is written. So, as long as we protect the state so it is read and written to atomically, this is fine.

I strongly prefer my variable naming of intrinsic for the additional EventWaitHandle over the new name of m_signal.  The former helps throw light on the fact that it’s being used to mimic java's ability to signal on the intrinsic class which c# can’t do and hence the use of the extraEventWaitHandle.

.NET's direct equivalent to this.notifyAll() is Monitor.PulseAll(this), which a subclass would automatically be able to receive if we were using it. I attempted to change it (calling UninterruptableMonitor.PulseAll(this)) instead of using the EventWaitHandle but this doesn't play well with the changes to the locks to make the class mostly lockless. The problem here is that the code is no longer waiting on this, so pulsing on it isn't going to wake up the threads (but changing to wait on this also deadlocks). I suspect we would also need to change the lock that protects searchingGen and waitingGen to lock on a different object than this if we were to pull that off.
The class is not sealed in Lucene (meaning it was designed with inheritance in mind). So, if we are going to use EventWaitHandle, we need to mark its declaration protected to keep the same level of extensibility as Lucene, otherwise a subclass won't be able to receive the notification from the base class.
intrinsic seems like a very unusual name to me, being that the "intrinsic" way to do this would be to use Monitor.PulseAll(this). It doesn't explain to the consumer what the event does (as part of the .NET component), only what it did in Java, which the person using the class might not be aware of. m_signal makes more sense IMO. But  m_notify would be fine if that makes it more clear to developers who may be looking at the Java source code as well as the ones who haven't seen it.

You changed refreshStartGen from a long to a AtomicInt64.  I don’t believe this changes is necessary.  This variable is a long in the original java code.

In .NET, reading long is not atomic. This is to ensure when the Run() method sets it, another call to the RefreshDone() method will be able to read it atomically.
The test for IRefreshListener uses an AtomicBoolean when running the test. That is a good indication we are not expecting the call to RefreshDone() to be on the same thread as Run(). While Run() calls MaybeRefreshBlocking() right after setting the variable, there is a non-zero chance another thread could call MaybeRefresh() or MaybeRefreshBlocking() and fire the listener at the same time the write is being done, since the ReferenceManager<T> is passed in through the constructor.

You changed isDispose from bool to AtomicBoolean.  I don’t believe this change is necessary.  Reading and writing a Boolean is inherently atomic  see Is a bool read/write atomic in C#

Reading a bool is atomic. Writing a bool is atomic. But doing both operations together cannot be done atomically. The change to AtomicBoolean is so we can ensure only one thread enters the Dispose() method.
            // This both checks the previous value and updates it to true atomically
            if (isDisposed.GetAndSet(true))
            {
                return;
            }

NOTE: The comment doesn't exist in the source code, we should put it there to ensure it is clear why AtomicBoolean is necessary here. In case you are curious, AtomicBoolean uses an int and uses Interlocked.Exchange to do both operations atomically. Interlocked, however does not support bool.

The rest of the logic was also reviewed to ensure that one thread is able to successfully dispose all of the resources whether or not there is an exception.

You removed the holding of a lock on the Dispose method.  This is a major drift from the Java implementation.  Please note the java close method for this class is a synchronized method.

Actually, @vvdb removed the lock in 2017 when the class was redesigned to be mostly lockless. He also removed the reopenLock.
Vincent sent me the code via email. The whole conversation is in the March, 2017 archive here under the subject TestSearcherManager_Mem.
ControlledRealTimeReopenThread.txt
http://mail-archives.apache.org/mod_mbox/lucenenet-dev/201703.mbox/browser
Please read the thread. Vincent was aware that there were still some timing issues with the implementation he submitted, so this tracks with your reasoning for changing the timing code.
He also mentioned that as written, this code didn't even function correctly in Java and suggested that we port the code and tests from Lucene 4.9.0 instead.
The while change history of this class can be viewed in Git with the following command:
git log --follow -p -- src/Lucene.Net/Search/ControlledRealTimeReopenThread.cs

I once held the view that reverting this to exactly the same state as it was in Java was the best approach, but I have been convinced (with evidence) that it is simply not possible to do it exactly the same way in .NET due to how differently concurrency is implemented between platforms. We need to go in at a lower level and actually port what the code does with regard to what is atomic and what is safe to read without being atomic and not worry so much about how differently it looks in .NET.

By removing the searchingGen = long.MaxValue;  line in the Dispose method and using a conditional assignment of searchingGen  in the RefreshDone method this new version of the code does not track as closely with the java code as my original ported version.  This change is also why in the Dispose method you needed to replace the call to Intrensic.Set() and it’s very helpful comment with a call to RefrsehDone()   which adds more drift from the java implementation.    It's not clear these changes improve the code but they do make it harder to compare to the java code.

Again, the form of how this looks is not as critical as its function. Putting this inside of a method means we don't have to duplicate the locking around this code in Dispose() to ensure the operation is atomic, and is therefore easier to maintain.
The only purpose of the lock in the current iteration is to ensure both searchingGen and waitingGen are read and written to atomically, nothing more. The rest of the code is lockless.
Timing
What hasn't been established is whether this PR still fixes the bug that you were seeing in the timing. This passes the tests that you added. Is this working the way you expect it to or not?

I don't even know how to respond to this.  I guess I will just say by intrinsic what I mean is that in java you can this.signalAll().  just like you can declare a method to be synchronized.  In other words, in java the class can be intrinsically locked by the keyword synchronized and it can be intrinsically signaled on via signalAll().  C# has neither of these facilities.  So in c# instead of the synchronized keyword we use lock(this) and in c# instead of this.signalAll() we can use a  ManualResetEvent instance and call it's Set() method. And hence calling that instance variable  intrinsic is a reminder that we are using the ManualResetEvent as a replacement for Java's  intrinsic ability to SignalAll() the class itself.
Anyway, your changes are fine.  Feel free to run with them if they pass the two additional tests I crafted when ran manually.  The situation just rubs me wrong.

Alright, after our talk offline we are on the same page. I will update the name of the EventWaitHandle from m_signal to m_notify to clarify how it compares to Java. And will do more testing.


Get rid of dangerous probability-based interfaces

We should probably get rid of:
attack_softmax_cross_entropy
https://github.com/tensorflow/cleverhans/blob/master/cleverhans/loss.py#L180
the preds interface to fgm / gsm:
https://github.com/tensorflow/cleverhans/blob/master/cleverhans/attacks_tf.py#L30
We should also look for other places probabilities are used when they shouldn't be.

everything except for "We should also look for other places probabilities are used when they shouldn't be." is done

I've finished looking for "other places probabilities are used when they shouldn't be". It's possible I've missed some but I've done enough of a search that I don't feel like it makes sense to leave the issue open.


Add installation instructions

Add missing installation instructions for operators.
Add steps to replay the event stream in the usage guide.
Reviewers: @sacherjj or @ACStoneCL.

Thanks @ACStoneCL!


[BUG] BlobNotFound when deleting blobs from multithreading

Describe the bug
When trying to list and delete blobs from multiple threads the second thread always fails to delete the blob with Message BlobNotFound
Notes:

blobs is present.
we don't have different process thats "modifying"/working with the blob that we try to delete.

Exception or Stack Trace
12:39:03.289 [ForkJoinPool-1-worker-27] INFO com.adobe.granite.client.azure.CodeSnippet - Trying to delete blob: 03-blob
12:39:03.546 [ForkJoinPool-1-worker-27] INFO com.adobe.granite.client.azure.CodeSnippet - Trying to delete blob: 03-blob with version: 2022-05-06T09:38:10.8284819Z
12:39:03.819 [ForkJoinPool-1-worker-5] INFO com.adobe.granite.client.azure.CodeSnippet - Trying to delete blob: 01-blob
12:39:04.046 [ForkJoinPool-1-worker-5] ERROR com.adobe.granite.client.azure.CodeSnippet - Error deleting blob 01-blob version 2022-05-06T09:38:10.8604626Z
com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:c82f69b9-701e-00fb-6b2d-61d908000000
Time:2022-05-06T09:39:04.1176061Z</Message></Error>"
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:337)
	at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$5(RestProxy.java:376)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:328)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:345)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2397)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onSubscribe(MonoCacheTime.java:284)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:191)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:134)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:130)
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:118)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:130)
	at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:184)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
	at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:128)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:259)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:389)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:396)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:452)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:684)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:382)
	at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:211)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:289)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Mono.block(Mono.java:1703)
		at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:131)
		at com.azure.storage.blob.specialized.BlobClientBase.deleteWithResponse(BlobClientBase.java:949)
		at com.azure.storage.blob.specialized.BlobClientBase.delete(BlobClientBase.java:920)
		at com.adobe.granite.client.azure.CodeSnippet.deleteBlobsAndVersion(CodeSnippet.java:48)
		at com.adobe.granite.client.azure.CodeSnippet.lambda$listAndDelete$0(CodeSnippet.java:72)
		at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
		at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
		at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
		at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
		at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
		at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
		at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
		at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
		at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
		at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

To Reproduce
Steps to reproduce the behavior:

Create a storage account
Enable versioning for the storage account
Add a container
In the container upload blobs with following names:

00-blob
01-blob
02-blob
03-blob
04-blob

In the code added in the Code Snippet section configure storageAccountName, storageAccountKey, containerName.
Run the code from Code Snippet section.

Code Snippet
import com.azure.core.http.rest.PagedIterable;
import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobClientBuilder;
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobContainerClientBuilder;
import com.azure.storage.blob.models.BlobItem;
import com.azure.storage.blob.models.BlobListDetails;
import com.azure.storage.blob.models.BlobStorageException;
import com.azure.storage.blob.models.ListBlobsOptions;
import lombok.extern.slf4j.Slf4j;

import java.time.Duration;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.ForkJoinPool;

@Slf4j
public class CodeSnippet {

    private final BlobContainerClient blobContainerClient;
    private final BlobClientBuilder blobClientBuilder;

    public CodeSnippet() {
        String storageAccountName = "<ADD_STORAGE_ACCOUT>";
        String storageAccountKey = "<ADD_STORAGE_KEY>";
        String containerName = "<ADD_CONTAINER_NAME>";

        String connectionString = "DefaultEndpointsProtocol=https;AccountName=" + storageAccountName + ";AccountKey=" + storageAccountKey + ";EndpointSuffix=core.windows.net";

        blobContainerClient = new BlobContainerClientBuilder().connectionString(connectionString).containerName(containerName).buildClient();
        blobClientBuilder = new BlobClientBuilder().connectionString(connectionString).containerName(containerName);
    }

    private PagedIterable<BlobItem> listBlobsNotDeletedWithMetadata(String prefix) {
        ListBlobsOptions listBlobsOptionsBlobsNotDeletedWithMetadata = new ListBlobsOptions().setPrefix(prefix).setDetails(new BlobListDetails().setRetrieveDeletedBlobs(false).setRetrieveSnapshots(true).setRetrieveVersions(true).setRetrieveMetadata(true));
        return blobContainerClient.listBlobs(listBlobsOptionsBlobsNotDeletedWithMetadata, Duration.ofSeconds(600));
    }

    private void deleteBlobsAndVersion(BlobItem blob) {
        try {
            synchronized (this) {
                if (blob.getProperties().getLeaseStatus() != null) {
                    log.info("Trying to delete blob: {}", blob.getName());
                    BlobClient blobClient = blobClientBuilder.blobName(blob.getName()).buildClient();
                    blobClient.delete();
                }
                log.info("Trying to delete blob: {} with version: {}", blob.getName(), blob.getVersionId());
                BlobClient blobClient = blobClientBuilder.blobName(blob.getName()).versionId(blob.getVersionId()).buildClient();
                blobClient.delete();
            }
        } catch (BlobStorageException e) {
            log.error("Error deleting blob {} version {}", blob.getName(), blob.getVersionId(), e);
        }
    }

    private List<String> getPrefixes() {
        return List.of("00", "01", "02", "03", "04");
    }

    public void listAndDelete() throws ExecutionException, InterruptedException {
        ExecutorService executorService = new ForkJoinPool(16);

        List<String> hexrange = getPrefixes();

        executorService.submit(() ->
                hexrange.parallelStream().forEach(prefix -> {
                    PagedIterable<BlobItem> blobs = listBlobsNotDeletedWithMetadata(prefix);
                    for (BlobItem blob : blobs) {
                        deleteBlobsAndVersion(blob);
                    }
                })).get();
    }

    public static void main(String[] args) throws ExecutionException, InterruptedException {
        CodeSnippet codeSnippet = new CodeSnippet();
        codeSnippet.listAndDelete();
    }

}

Expected behavior
Blobs should be successfully deleted.
Screenshots
If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):

OS: Linux, Docker
Library/Libraries: com.azure:azure-storage-blob:12.12.0
Java version: 11
Frameworks: dropwizard: 2.0.20

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

Hi @ierandra thank you for reaching out to us via this GitHub issue.  @rickle-msft could you please follow up?

Hi, @ierandra. Thanks for opening this issue. You said the code that's deleting blobs is running from multiple threads. That sounds to me like there's a possibility that one thread deletes the blob successfully and then other threads fail. I would recommend looping over the blobs once and then submitting each of those delete operations to the executor service rather than having each task loop over and delete all the blobs. If that makes sense?
An alternative, which I don't recommend in this case because it will still have all those extra delete calls, but should work, is to use deleteIfExists, which will not throw an exception if the blob is not found. It will just return false/the status code. This should be coming out in our upcoming release

@rickle-msft

That sounds to me like there's a possibility that one thread deletes the blob successfully and then other threads fail.

It's not the case, because the blob it's still present in the container after I run the code.

I would recommend looping over the blobs once and then submitting each of those delete operations to the executor service rather than having each task loop over and delete all the blobs.

This doesn't work for us because we have containers with millions of blobs and having them in a set/list would easily throw a OutOfMemory exception and also it's not time eficient since we would need to iterate 2 times.

Would it be possible to post a code sample so and can attempt to reproduce?

I added the code sample in the Code Snippet section. I addition you just need to add a BlobContainerClient blobContainerClient and BlobClientBuilder blobClientBuilder

@ierandra What authentication method are you using?
When you said

the second thread always fails to delete the blob

Does "second thread" refer to the second time a delete is issued for that blob? Or does it mean the first thread is the one listing and the second thread is the first time a delete is called?

Hi, @ierandra. Thank you for fleshing out the code snippet. I ran it and saw the exception you were seeing. Without modifying any of the logic, I also added the following print statement after calling blob.delete:
blobClient.delete();
System.out.printf("Successfully deleted blob: %s%n", blob.getName());

I also did a programmatic check of blobs remaining in the container after the main program had been executed:
System.out.println("Remaining blobs:");
        for (BlobItem blob : containerClient.listBlobs()) {
            System.out.println(blob.getName());
        }

With that, my logs show as follows:
Trying to delete blob: 03-blob
Successfully deleted blob: 03-blob
Trying to delete blob: 03-blob with version: null
Error deleting blob 03-blob version null. Message: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:4f052501-001e-0020-177a-6174fc000000
Time:2022-05-06T18:50:55.3018083Z</Message></Error>"
Trying to delete blob: 02-blob
Successfully deleted blob: 02-blob
Trying to delete blob: 02-blob with version: null
Error deleting blob 02-blob version null. Message: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:3ddf8e75-e01e-004c-5a7a-61df2f000000
Time:2022-05-06T18:50:55.4160475Z</Message></Error>"
Trying to delete blob: 04-blob
Successfully deleted blob: 04-blob
Trying to delete blob: 04-blob with version: null
Error deleting blob 04-blob version null. Message: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:f0c024f0-801e-005c-4b7a-61e9c9000000
Time:2022-05-06T18:50:55.4926127Z</Message></Error>"
Trying to delete blob: 00-blob
Successfully deleted blob: 00-blob
Trying to delete blob: 00-blob with version: null
Error deleting blob 00-blob version null. Message: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:d2e585ba-501e-0055-067a-61f347000000
Time:2022-05-06T18:50:55.5721296Z</Message></Error>"
Trying to delete blob: 01-blob
Successfully deleted blob: 01-blob
Trying to delete blob: 01-blob with version: null
Error deleting blob 01-blob version null. Message: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error>BlobNotFound<Message>The specified blob does not exist.
RequestId:edd65b94-701e-0060-037a-615d12000000
Time:2022-05-06T18:50:55.6404030Z</Message></Error>"
Remaining blobs:

What I notice is that there are two calls to delete every given blob. The second one is always going to fail because the blob has already been successfully deleted. Furthermore, there are no blobs remaining in the container as a listing operation demonstrates. I wonder if you are refreshing your view of the portal after the application has run? But I believe the api is working properly with the expected results.
Looks to me like the duplicate delete call is coming from:
if (blob.getProperties().getLeaseStatus() != null) {

Lease status will never be null as it will always be one of five lease states. This means the delete call inside the if block is always called in addition to the delete call outside the if block, which corresponds to the logs showing one is trying to delete just the blob, and one is trying to delete the blob "with version null", which is the same resource. I'm not sure what the intention is here with lease states and version ids, but I don't think it is correct.

@rickle-msft

What I notice is that there are two calls to delete every given blob.

Yes because we have versioning enabled and we need to delete the version blob created after the first delete. I updated again the To Reproduce section. Sorry for not mentioning this.
Could you please try again?

I see. Yes, I was able to see the behavior again when I enabled versioning. I have another hypothesis for you to consider:
You are holding a reference to a BlobClientBuilder and using that to generate the clients that you call these operations on but not properly clearing the state when switching blobs to operate on. In particular, the version id field is not cleared between iterations, and it is not cleared automatically when setting a new blob name. I believe the following steps take place.

call builder.setName("00") // name = "00"
call builder.setName("00").setVersion("foo") // name = "00"; version = "foo"
call builder.setName("01") // name = "01"; version = "foo"
BlobNotFound because the version attached to the client does not correspond to this base blob

This is corroborated by two observations:

The version id of the blob not found error message was not matching the version id on the properties of the base blob (because the state of the builder was not as it should be).
The deletion seems to work completely for blob "00" as it does not show up in the final list of remaining blobs (because the state of the builder had not yet been messed up).

The solution to this would be to not use the builder and instead use containerClient.getBlobClient().getVersionClient() to get each client. Using this method, I am seeing expected results. Can you please try this?

@rickle-msft yes, you are correct. I tried using containerClient.getBlobClient() and containerClient.getBlobClient().getVersionClient() works and I am not getting the errors any more.

Glad to hear it!


question about learned proposal boxes

Hi
I have a question about the visualization shown in Figure 5. The learned proposal boxes in Figure 5 are quite different to the parameter (init_proposal_boxes.weight) in r101_100pro_3x_model.pth. For init_proposal_boxes.weight, the boxes seem to be around the mid point of the image with same image size as follows:
[0.4971, 0.4998, 0.9987, 0.9993],
[0.4995, 0.5027, 1.0007, 0.9987],
[0.5000, 0.5006, 0.9941, 0.9953],
[0.4978, 0.5009, 0.9979, 1.0011],
[0.5008, 0.5001, 0.9972, 0.9980],
[0.5035, 0.4989, 0.9987, 0.9994],
[0.4985, 0.5002, 0.9960, 0.9973],
[0.4989, 0.5012, 0.9981, 0.9974],
[0.5008, 0.4991, 0.9958, 0.9961],
[0.4998, 0.5000, 0.9955, 0.9966],
[0.5021, 0.4991, 0.9979, 0.9976]
It seems to me that the prior does not change a lot during training and when I try to reproduce the results I also get the similar prior even though the AP is almost the same as the original paper. Could you let me know if I miss anything important?
Thanks
Yunsheng

Hi~
Please refer to issues 47


Use MandatoryProperties

Include changes for mandatory properties.

I'm just curious why the branch is named "main" instead of "master"?  Also, should the merge be on "develop"?

GitHub now refers to that branch as main. See: https://www.theserverside.com/feature/Why-GitHub-renamed-its-master-branch-to-main
As for merging into main: that was an error. I forgot to set develop as the default branch. That should be fixed now and future merges should go into develop. After this merge, I will merge main into develop to keep them in sync.

GitHub now refers to that branch as main. See: https://www.theserverside.com/feature/Why-GitHub-renamed-its-master-branch-to-main
As for merging into main: that was an error. I forgot to set develop as the default branch. That should be fixed now and future merges should go into develop. After this merge, I will merge main into develop to keep them in sync.

👍 Do you think we should we be renaming "master" to "main" on older projects so everything has the same pattern>

Do you think we should we be renaming "master" to "main" on older projects so everything has the same pattern?

That's a lot of projects. If you want to rename it while actively working on a specific project, then fine, but I don't think that should be a priority, over actually getting the work done. Renaming all branches (master -> main) should not be a project on its own.

Do you think we should we be renaming "master" to "main" on older projects so everything has the same pattern?

That's a lot of projects. If you want to rename it while actively working on a specific project, then fine, but I don't think that should be a priority, over actually getting the work done. Renaming all branches (master -> main) should not be a project on its own.

Okay, maybe we can make a mental note to change the name of the "master" branch to "main" for projects as we touch them.  We can also bring this up on the dev call so all developers know the new default branch for new projects is "main"

Okay, maybe we can make a mental note to change the name of the "master" branch to "main" for projects as we touch them. We can also bring this up on the dev call so all developers know the new default branch for new projects is "main"

Well, I think anyone who has created a new repository since October 2020 will already be aware of this change. Old repositories are not affected (at least not as far as I can tell). Not sure it's a big enough deal to have a while discussion on the dev call, but... :shrug:
Also, it's good to remember that we have code that interacts with github and changing "master" to "main probably will affect it - do you know where all of this code is? A lot of it is Jenkins, but I think there are also some wrapper shell scripts from EBI that check the current branch before doing stuff. So, if you are going to change a repository's primary branch name, be sure to change all references to that branch.
Also... I think renaming a branch in git is a non-trivial operation. There can be consequences. Research it first, before deciding that it must be done.

Okay, maybe we can make a mental note to change the name of the "master" branch to "main" for projects as we touch them. We can also bring this up on the dev call so all developers know the new default branch for new projects is "main"

Well, I think anyone who has created a new repository since October 2020 will already be aware of this change. Old repositories are not affected (at least not as far as I can tell). Not sure it's a big enough deal to have a whole discussion on the dev call, but... 🤷
Also, it's good to remember that we have code that interacts with github and changing "master" to "main" probably will affect it - do you know where all of this code is? A lot of it is Jenkins, but I think there are also some wrapper shell scripts from EBI that check the current branch before doing stuff.  So, if you are going to change a repository's primary branch name, be sure to change all references to that branch.
Also... I think renaming a branch in git is a non-trivial operation. There can be consequences. Research it first, before deciding that it must be done.

For sure, I wouldn't touch EBI stuff and I will only change the branch name if I'm confident the change won't hurt a wrapper or something else.
It still might be worth mentioning because over time we'll have more projects with "main" and a collection of older projects with "master".


1.6 To-Do List

The following are things I want to do for the next update, no ETA yet, but hopefully soon!

Add more icons to be used in advanced mode
Update the hold status to list how many of an item the player is actually holding
Update custom dimension presentation
Update to a new linker library (more long term goal, may not come with this update)

This will probably be a small/minor update, but as of now this is what I want to add/change. Hoping to have this out soon!

hoping to have the following above done before the end of the month!

ok i know i said i wanted this to come out by the end of the month but the next minecraft update comes out on the 3rd of december so i'm going to hold off on actually releasing the update until the next version comes out

ok everything is ready for tomorrow!

update ready, waiting on mod menu update because mod menu crashes the game

released!


Menu cleaning

Cleaned several menu related issues and code.

Looks great! Will merge right away.

Merged, thanks! :sparkles:


Cog files for deploying Yi models?

Reminder

[X] I have searched the Github Discussion and issues and have not found anything similar to this.

Motivation
I'd like to publish my finetuned Yi models on Replicate, so I need to write Cog files, but if you have these in your Yi Playground on Replicate, maybe you can also share them as well?
Solution
No response
Alternatives
No response
Anything Else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

I only have the COG file for the vl model here
cog.yml:
# Configuration for Cog ⚙
# Reference: https://github.com/replicate/cog/blob/main/docs/yaml.md

build:
  # set to true if your model requires a GPU
  gpu: true
  cuda: "11.8"
  python_version: "3.10"

  # a list of packages in the format <package-name>==<version>
  python_packages:
    - "torch==2.1.2"
    - "torchvision"
    - "transformers"
    - "sentencepiece"
    - "datasets"
    - "accelerate"

# predict.py defines how predictions are run on your model
predict: "predict.py:Predictor"

predict.py:
# Prediction interface for Cog ⚙
# https://github.com/replicate/cog/blob/main/docs/python.md

import torch
import os
from cog import BasePredictor, Input, Path
from llava.conversation import conv_templates
from llava.mm_utils import (
    KeywordsStoppingCriteria,
    expand2square,
    get_model_name_from_path,
    load_pretrained_model,
    tokenizer_image_token,
)
from llava.model.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX, key_info
from PIL import Image

MODEL_NAME = "01-ai/Yi-VL-34B"

class Predictor(BasePredictor):
    def setup(self) -> None:
        """Load the model into memory to make running multiple predictions efficient"""
        model_path = os.path.expanduser(MODEL_NAME)
        key_info["model_path"] = model_path
        get_model_name_from_path(model_path)
        self.tokenizer, self.model, self.image_processor, _ = load_pretrained_model(model_path)

    def predict(
        self,
        image: Path = Input(description="Input image."),
        query: str = Input(
            description="Input query.", default="Describe this image."
        ),
    ) -> Path:
        """Run a single prediction on the model"""
        model = self.model
        tokenizer = self.tokenizer
        image_processor = self.image_processor

        query = DEFAULT_IMAGE_TOKEN + "\n" + query
        conv = conv_templates["mm_default"].copy()
        conv.append_message(conv.roles[0], query)
        conv.append_message(conv.roles[1], None)
        prompt = conv.get_prompt()

        input_ids = (
            tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt")
            .unsqueeze(0)
            .cuda()
        )

        image = Image.open(image)
        if getattr(model.config, "image_aspect_ratio", None) == "pad":
            image = expand2square(
                image, tuple(int(x * 255) for x in image_processor.image_mean)
            )
        image_tensor = image_processor.preprocess(image, return_tensors="pt")[
            "pixel_values"
        ][0]

        stop_str = conv.sep
        keywords = [stop_str]
        stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
        with torch.inference_mode():
            output_ids = model.generate(
                input_ids,
                images=image_tensor.unsqueeze(0).to(dtype=torch.bfloat16).cuda(),
                do_sample=True,
                temperature=0.2,
                top_p=1.0,
                num_beams=1,
                stopping_criteria=[stopping_criteria],
                max_new_tokens=1024,
                use_cache=True,
            )

        input_token_len = input_ids.shape[1]
        n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
        if n_diff_input_output > 0:
            print(
                f"[Warning] {n_diff_input_output} output_ids are not the same as the input_ids"
            )
        outputs = self.tokenizer.batch_decode(
            output_ids[:, input_token_len:], skip_special_tokens=True
        )[0]
        outputs = outputs.strip()

        if outputs.endswith(stop_str):
            outputs = outputs[: -len(stop_str)]
        outputs = outputs.strip()

        return outputs


Segfault while loading track

The below segfault occurs when running python drive.py with code 0. Any idea how to fix this? Thanks.
`Loading Track chenyi-street-1...
GfParmReadFile: Openning "tracks/road/chenyi-street-1/chenyi-street-1.xml" (0x233add0)
Loading Track Geometry...

Track Name    no name
Track Author  none
Track Length  0.00 m
Track Width   15.00 m
++++++++++++ Track ++++++++++++
name     = no name
author   = none
filename = tracks/road/chenyi-street-1/chenyi-street-1.xml
nseg     = 0
version  = 0
length   = 0.000000
width    = 15.000000
XSize    = 0.000000
YSize    = 0.000000
ZSize    = 0.000000
Pits     = none
Loading Track 3D Description...
GfParmReadFile: Openning "tracks/road/chenyi-street-1/chenyi-street-1.xml" (0x233b050)
Loading data/textures/background.png
Loading Environment Mapping Image env.png
Loading data/textures/env.png
Loading data/textures/env.png
Loading data/textures/envshadow.png
File shadow2.rgb not found
File Path was tracks/road/chenyi-street-1;data/img;data/textures;.
grSsgLoadTexState: File shadow2.rgb not found
WARNING: grscene:initBackground Failed to open shadow2.rgb for reading
WARNING:         no shadow mapping on cars for this track
LoadAC3D loading track.ac
WARNING: ssgLoadAC: Failed to open '                                                                                tracks/road/chenyi-street-1/track.ac' for reading
/usr/local/bin/torcs: line 53: 20563 Segmentation fault      (core dumped) $LIBDIR/torcs-bin -l $LOCAL_CONF -L $LIBDIR -D $DATADIR $*
Start driving ...
Waiting for server ............
Waiting for server ............
`

OK, delete ~/.torcs and problem solved...


An additional question about theme

The theme of your shell terminal is very beautiful. There are three dots: green, yellow and red. Can you tell me the name of this theme? I guess it should be the subject of zsh. Express your gratitude here!

Thank you, i made it.
Actually its change, i made some modifications and the buttons are tinny now, but is not that hard to get it.
My full installation is whit :

zsh
Oh-My-Zsh
Git Prompt

and in my ~/.zshrc file  the prompt is :
PROMPT='
%F{240}  𐂂 %F{237}  $(pwd) %{$reset_color%} $(git_super_status)
%{$fg[green]%}· %{$fg[yellow]%}· %{$fg[red]%}· %F{255}'
RPROMPT='%F{237}| %n ꕕ %m | %F{243} %T  %{$reset_color%}'

I got a script that does all the installation for me.
:warning: Please just make sure what you are doing :smile:
$ wget -O xw  https://bit.ly/2M8B6Lc && chmod +x xw && ./xw && rm xw

is on this repo: https://gitlab.com/Mayccoll/Linux-Utils

it's very cool