Use explicit "vulkan-style" shaders to generate all shader outputs

In order to create a WebGPU backend, we need shaders with separate sampler and texture2D types. Unfortunately there is no existing automated way (that I know of) to convert a sampler2D shader into a split sampler and texture2D shader.
I can think of two solutions to this problem:

create duplicate "vulkan style" shaders with separate samplers and textures
Make the source shaders "vulkan style" with separate samplers and textures, generate spirv from those, then generate compatible shaders for each backend using spirv-cross

(1) seems harder to maintain so I opted to implement (2).
The changes in this pr:

Adapt source shaders to a vulkan style format with split samplers and textures.
Modify shader makefile to generate gl3, gl4, and metal shaders from the source shaders by using spirv-cross
Modify GL backend to account for format changes:

Fall back to a SPIRV_Cross_Combined sampler2D uniform name if a "normal" uniform name cannot be found
Use vec4 types instead of scalar, matrix, vec2, and vec3 types when binding uniforms. This change is the result of the spirv-cross transformation. I would really prefer to not make this change, but I couldn't find a way to force spirv-cross to maintain the original types.

I tested the GL backend and it works as expected.
This PR is not ready to be merged for the following reasons:

We should discuss the above vec4 change in the GL backend and generated shaders
Need to restore #ifdef GL_ES precision highp sampler2D in GL outputs. The ifdef is still in the source shaders, but the initial spirv compilation strips that out. This could be fixed by creating a separate "gles" shader resource folder generated with -DGL_ES. Alternatively we could do a text transformation on the existing GL outputs.
#extension GL_GOOGLE_include_directive : enable is also stripped out. This should also be added to the GL outputs
Haven't tested on android
Haven't tested WebGL backend
Metal backend isn't functional yet. The metal shaders generated appear valid, but I think the backend can't bind uniforms because the argument names have changed:

Original clear.vs.metal
// Automatically generated from files in pathfinder/shaders/. Do not edit!
#include <metal_stdlib>
#include <simd/simd.h>

using namespace metal;

struct main0_out
{
    float4 oFragColor [[color(0)]];
};

fragment main0_out main0(constant float4& uColor [[buffer(0)]])
{
    main0_out out = {};
    out.oFragColor = float4(uColor.xyz, 1.0) * uColor.w;
    return out;
}

New clear.vs.metal
Note the new uColor struct and the argument name changing from uColor to `_12
// Automatically generated from files in pathfinder/shaders/. Do not edit!
#include <metal_stdlib>
#include <simd/simd.h>

using namespace metal;

struct uColor
{
    float4 color;
};

struct main0_out
{
    float4 oFragColor [[color(0)]];
};

fragment main0_out main0(constant uColor& _12 [[buffer(0)]])
{
    main0_out out = {};
    out.oFragColor = float4(_12.color.xyz, 1.0) * _12.color.w;
    return out;
}

This issue seems solvable by adjusting the metal backend to query uniforms using argument type names instead of argument names.
Once again I'm pulling in @kvark  and @tangmi because this relates to WebGPU / wgpu.

Just like #332, there are tradeoffs here ... so i'd love to hear people's thoughts.

This issue seems solvable by adjusting the metal backend to query uniforms using argument type names instead of argument names.

I don't understand this part. In gfx/wgpu world, there is no querying from a backend. You describe the pipeline layout, and the backend uses it for shaders. In case of wgpu, it also validates the shader needs to be compatible with the layout. Names don't mean anything for these.

I can think of two solutions to this problem:

Any reason to ignore the suggestion I voiced on Matrix?

I don't understand this part. In gfx/wgpu world, there is no querying from a backend. You describe the pipeline layout, and the backend uses it for shaders. In case of wgpu, it also validates the shader needs to be compatible with the layout. Names don't mean anything for these.

This is specifically the "pathfinder metal backend", which needs to map a "uniform name string" to a MetalUniformIndex in order to bind a value. It currently uses reflection on the main function's argument names to find the index. Check out MetalDevice::get_uniform_index for context.

Any reason to ignore the suggestion I voiced on Matrix?

This is embarrassing for me, but I convinced myself it didn't work. The example you gave me didn't compile with glslValidator and i tried to adapt it without success. But with a few changes I was able to make it work:
// ifdef instead of if
#ifdef COMBINED_IMAGE_SAMPLERS
uniform sampler2D uSrc;
#else
// "sampler" by itself didnt work for me. i needed to give it a name
layout(set=0, binding=0) uniform sampler uSampler;
layout(set=0, binding=1) uniform texture2D uSrc_;
#define uSrc sampler2D(uSrc_, uSampler)
#endif

I gave up too early the first time I tried and I just started diving into other solutions. I should have circled back with you when I couldn't get it to work.
Given that this works ... its clearly the right solution. I'll put together a new branch and I'll reach out if/when everything works.
Sorry for wasting your time!

I just remembered why I moved on so quickly (and it wasn't just the compilation issues). The #ifdef above works in isolation. You can compile a gl 330 shader using glslangValidator -G330 -DCOMBINED_IMAGE_SAMPLERS and a "vulkan" shader using glslangValidator -V, but how do we make a more complex example work?
#ifdef COMBINED_IMAGE_SAMPLERS
uniform sampler2D uSrc;
#else
layout(set=0, binding=0) uniform sampler uSampler;
layout(set=0, binding=1) uniform texture2D uSrc_;
#define uSrc sampler2D(uSrc_, uSampler)
#endif

uniform vec4 uColor;

The uColor uniform is valid GL330, but it isn't valid Vulkan, which needs to define set/binding and use "block style" uniforms.
Of course we could just do what we did above:
#ifdef COMBINED_IMAGE_SAMPLERS
uniform sampler2D uSrc;
uniform vec4 uColor;
#else
layout(set=0, binding=0) uniform sampler uSampler;
layout(set=0, binding=1) uniform texture2D uSrc_;
#define uSrc sampler2D(uSrc_, uSampler)
layout (set=0, binding=2) uniform uColor {
  vec4 color;
};
#endif

Of course now we should rename COMBINED_IMAGE_SAMPLERS to something like VULKAN_SEMANTICS or NON_VULKAN_SEMANTICS.
But now we're replicating every uniform. Maybe its still the best solution given the constraints discovered in this PR's code. But using a single "vulkan style" shader instead of doubling up on everything is starting to look more attractive.
Can you think of any other solutions to this problem?

I gave up too early the first time I tried and I just started diving into other solutions. I should have circled back with you when I couldn't get it to work.

Sorry, I just wrote this approximately from the way I understand it should work, not copied from actual code that works. Glad you got that working though!

The uColor uniform is valid GL330, but it isn't valid Vulkan, which needs to define set/binding and use "block style" uniforms.

WebGPU doesn't have free-standing uniforms (or push constants), so there needs to be a path were all the free uniforms are put into a uniform buffer. Whether this is the only path (which is compatible to both OpenGL and WebGPU), or an alternative path, is up to debate. Performance-wise, a free-standing uniform could theoretically be put into a scalar register by the driver (especially, AMD). I don't know for sure if they do this.

We should probably rename COMBINED_IMAGE_SAMPLERS to something like VULKAN_SEMANTICS or NON_VULKAN_SEMANTICS.

Vulkan supports both combined and non-combined image samplers, so the part about texture/sampler separation shouldn't be called "VULKAN_SEMANTICS".

But now we're replicating every uniform.

You shouldn't be replicating every uniform. As I mentioned, you'd just be putting all of them into a single uniform buffer. Note that in D3D11/12 there isn't even such a thing as a free-standing uniform, and all those would have to go into a "global" uniform buffer anyway.

Sorry, I just wrote this approximately from the way I understand it should work, not copied from actual code that works. Glad you got that working though!

You don' need to apologize. This was a @cart failure :)

Vulkan supports both combined and non-combined image samplers, so the part about texture/sampler separation shouldn't be called "VULKAN_SEMANTICS".

Fair point. So either we split them out into two separate defs or call it something like WEBGPU_SEMANTICS?

Everything else

I think I'm being dense. I understand that WebGPU needs all uniforms in uniform buffers and that it doesn't support push constants, but I don't see how this solves the "replication" problem in shaders.
uniform vec4 uColor; can't be compiled by glslangValidator in "vulkan semantics" mode. uniform uColor { vec4 color; }; can't be compiled in "GL330 mode". Therefore to support both don't we need to declare the uniform in the shader twice like I did in my second example above?

And we need to compile in "vulkan mode" for webgpu because "GL mode" doesn't support split samplers.

So either we split them out into two separate defs or call it something like WEBGPU_SEMANTICS?

Yep

uniform uColor { vec4 color; }; can't be compiled in "GL330 mode"

Why? OpenGL uniform buffer objects are supported in GL 3.1 as well as GLES 3.0. Therefore, it looks like using them can be the one and only code path supported everywhere.

Once again you're right. I was treating the output of this command as the source of truth:
glslangValidator -G330 shader.fs.glsl -DCOMBINED_IMAGE_SAMPLERS -S frag -E

Because thats what the master branch's shader makefile uses when it produces the gl3/gl4 glsl from glsl source files. I wasn't aware that this actually applies additional constraints (because the -G330 instructs it to produce a spirv binary?)
This command succeeds when I run it on a uniform buffer object with #version 330:
glslangValidator shader.fs.glsl -S frag -DCOMBINED_IMAGE_SAMPLERS -l

Is that the command you would use to validate? If so the makefile will need to be updated because it fails when a ubo is added with the current setup.
The webgpu spirv could then be generated by substituting a higher version number like 450 and passing in -V  --auto-map-bindings
I think the only missing piece is metal support. Sadly I think the use of UBOs in this pr is what broke the "name based argument reflection" in MetalDevice. So that problem will likely still need solving.

sigh *
I made one more mistake. The makefile only uses -G when generating the spirv used to produce MTL shaders. The makefile is fine as-is

You can write GLSL in the way that exposes a uniform buffer via a single global variable, similar to how Metal sees that. I just tested it on shader playground. The source GLSL has:
layout(set = 0, binding = 0) uniform Globals {
    vec4 color;
} u_Globals;

The output Metal shader has:
fragment main0_out main0(constant Globals& u_Globals [[buffer(0)]])

So the name "u_Globals" would be preserved this way.


Launch improvements

Renamed URL /json_schema_launch to just /launch
Added functionality to pick a pipeline and release, if loaded with no inputs
Added functionality to load a schema and launcher through GitHub API

Ties in with updated https://github.com/nf-core/tools/pull/638

Happy to merge this again, like #400 so that it can be tested on the live site.
Only changes to the new launch page, so shouldn't break anything. I think.


Event loop redesign

Closes https://github.com/PistonDevelopers/piston/issues/1146

Since making changes to the game loop requires careful testing, I'll publish a new version with this preparation so it unblocks other issues and makes it easier to compare alternatives.


Gear is not changed to PARK on goal anymore.

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I'm convinced that this is not my fault but a bug.

Description
During a recent test drive, I observed Autoware does not change the vehicle gear to PARK anymore upon reaching goal. Normally, this piece of code is supposed to do so when ARRIVED_GOAL state is received:
https://github.com/autowarefoundation/autoware.universe/blob/001a865bdae040784d3392677cb6d18c82309aac/control/shift_decider/src/shift_decider.cpp#L73-L74
Upon arrival, Autoware state changes quickly from DRIVING to ARRIVED_GOAL, to FINALIZING, then back to WAITING_FOR_ROUTE. Autoware is always in Autonomous mode and the vehicle is always engaged.
Expected behavior
Gear is changed to park upon arrival if park_on_goal_ is true
Actual behavior
Gear unchanged.
Steps to reproduce
Drive autonomously to any goal.
Versions
No response
Possible causes
No response
Additional context
No response

I couldn't observe the problem at commit sha in the universe: f627fc30a6730af52348e0605437d7ed6806170e
Could you recheck whether the issue still occurs or not?

Sorry for my mistake, it still occurs. I am going to work on this.


[New game]: 2048 game

🎮 Game Request
2048 is a popular puzzle game played on a 4x4 grid. The objective of the game is to slide numbered tiles on the grid and combine them to reach the tile with the number 2048.
At the beginning of the game, there are two randomly placed tiles, either with the number 2 or 4. You can use the arrow keys (up, down, left, and right) to move all the tiles on the grid in that direction. When you make a move, all the tiles slide as far as possible in the chosen direction, either merging with another tile of the same number or stopping at the edge of the grid.
Whenever two tiles with the same number collide during a move, they merge into one tile with the sum of their values. For example, if you slide two tiles with the number 2 together, they combine to form a single tile with the number 4. The merged tiles also free up a space on the grid for new tiles to appear.
Point down the features
The game continues until you either reach the tile with the number 2048 or are unable to make any more moves. Your score is based on the value of the tiles you combine during the game. The higher the value of the merged tiles, the higher your score will be.
2048 is known for its simple rules but challenging gameplay, requiring strategic thinking and planning to reach the elusive 2048 tile. It has gained immense popularity on various platforms and has inspired numerous versions and adaptations across different devices and operating systems.
Select program in which you are contributing
GSSoC23
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey  @AbhishekPSingh07  ! Thank you for raising an issue 💗 You have been assigned this issue you can start working on the issue 🎮

Hey @AbhishekPSingh07 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


error: Missing pending deopt entry

From last night:
https://build.chromium.org/p/client.dart/builders/vm-linux-debug-x64-reload-be/builds/391/steps/vm tests/logs/stdio
https://build.chromium.org/p/client.dart/builders/vm-linux-debug-x64-reload-be/builds/391/steps/checked vm tests/logs/stdio
https://build.chromium.org/p/client.dart/builders/vm-linux-debug-x64-reload-be/builds/393/steps/vm tests/logs/stdio
https://build.chromium.org/p/client.dart/builders/vm-linux-debug-x64-reload-be/builds/393/steps/checked vm tests/logs/stdio

@fsc8000 @rmacnak-google

https://build.chromium.org/p/client.dart/builders/vm-linux-release-x64-reload-be/builds/699/steps/vm tests/logs/stdio

The links above have expired, but this is fixed, right? @rmacnak-google

Re-labeling this as a language issue. There is nothing VM-specific we can do,  except reducing the overhead of try-catch to as close to zero as possible. The case of empty try-finally is already fixed.

Undo: Edited wrong issue...

Is this fixed and can be closed?

Yes, I think this was failing after enabling the new lazy deopt mechanism, but is fixed now.


Istio ingress gateway support

We use istio as ingress gateway. The openappsec support for istio would be appreciated.

Envoy is indeed on the roadmap and we plan to support it.

@bilbogh I wandered on to this thread looking for envoy support, Is it still in development ?
Could you point me to some example/documents/PRs on writing a attachment ? I am thinking ext-authz filter is how i should be implementing this ? Does this make sense ?


Alpha v2

Updated a few things and made column names variable

@Trippl7777  just a small consistency request ^^, otherwise looks good. ready to merge?

All fixed up. You should be good to merge


create the method that returns all files

create a method that return the image in all types at once

getAllTypesOfImage

Merge in the wrong branch, sory


Chatterbot response in django is in json format

@buildithub how you are trying? Did you see this error while running example_app ? Could you please explain a bit your problem, so that is easier to help?

yes I ran example_app and the above problem occurs. This is happening since 0.5.3 version. Before that I used 0.5.1 version of the chatterbot and there was no problem.

I can able reproduce this issue

I used 0.5.1 version's django_app with 0.5.3's chatterbot. Still not working. Only working with 0.5.1.

I used 0.5.3 version's django_app with 0.5.1's chatterbot. It's working.

Same answer

I used 0.5.3 version's django_app with 0.5.1's chatterbot. It's working.

Hi everyone, #608 corrects this problem. I apologize about the delay in tracking down the issue.


F#523 add window command

Description

New command window is added
Related Issue :

#523
How Has This Been Tested?

Check if adding window command works with no error
Check if editing window command works with no error
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code follows the code style of this project. it will be added soon
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document. it will be added soon
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Additional Context

run build
deploy to 2

window value: [row_number(), rolling_sum(col1, 2, 2)] group: col2 order: col3
window value: [avg(col1), lead(col1, 3)] group: order: col3


Enable rabbit heart beat and timeout in openstack services

With Kilo, oslo.messaging grew the ability to heartbeat to rabbit (enabled by default) and to have a timeout client side to reset connection (not enabled by default). We should enable the timeout to a reasonable time, like say 30 seconds.
This is typically done in a service config file under the header
[oslo_messaging_rabbit]
with the setting
heartbeat_timeout_threshold = 30

however check the config reference in each service to be certain. Setting this should provide a bit more stability in our services around rabbit disconnects or HA failovers.
Would really like to get this in for the 2.0 release.
This also requires kombu>=3.0.7 and amqp>=1.4.0 so be sure those are expressed in the places we install those packages.

Looks promising, I can get started taking a look at this.
This should help with migrations.

Another one bites the dust!


Reset password interface

Users should be able to reset their own password. If they have 2FA setup, they should be required to re-enter that code.

Fixed in #71


Code workflow : Configure Azure Pipelines

This pipeline configuration builds the application and produces a build for the Release configuration.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: bhallavarun sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

approved


Group Membership Deletion bug

Seems like when deleting group member, the backend does not deal with dependency, which it only delete the group member entity while higher dependency exist (group post created by this member). There should be a solution that if we want to keep the group post until we delete that member (that will be harder). Or: we delete the related group post directly (that will be easier).

Fix the bug by connect Group Post with UserProfile table.


[QUESTION] OOM when load XCOMET-XXL in A100 with 40G memory for prediction

❓ Questions and Help
Before asking:

Search for similar issues.
Search the docs.

What is your question?
I can predict scores with only cpu successfully. But when loaded model to gpu, there is oom error.
Code
from comet import download_model, load_from_checkpoint
#model_path = download_model("Unbabel/XCOMET-XXL")
model_path = "./XCOMET-XXL/checkpoints/model.ckpt"
model = load_from_checkpoint(model_path,reload_hparams=True)
data = [
{
"src": "Boris Johnson teeters on edge of favour with Tory MPs",
"mt": "Boris Johnson ist bei Tory-Abgeordneten völlig in der Gunst",
"ref": "Boris Johnsons Beliebtheit bei Tory-MPs steht auf der Kippe"
}
]
model_output = model.predict(data, batch_size=1, gpus=1)
Segment-level scores
print (model_output.scores)
System-level score
print (model_output.system_score)
Score explanation (error spans)
print (model_output.metadata.error_spans)
hparams.yaml

What have you tried?
What's your environment?

I have a suggestion to run XCOMET-XXL in a 40GB but its still not integrated. In the file: comet/encoders/xlmr_xl.py
Replace the model init to load in 16bits:
def __init__(
        self, pretrained_model: str, load_pretrained_weights: bool = True
    ) -> None:
        super(Encoder, self).__init__()
        self.tokenizer = XLMRobertaTokenizerFast.from_pretrained(pretrained_model)
        if load_pretrained_weights:
            self.model = XLMRobertaXLModel.from_pretrained(
                pretrained_model, add_pooling_layer=False
            )
        else:
            print ("Loading model in f16")
            self.model = XLMRobertaXLModel(
                XLMRobertaXLConfig.from_pretrained(pretrained_model, torch_dtype=torch.float16, device_map="auto"),
                add_pooling_layer=False
            )
        self.model.encoder.output_hidden_states = True

this will load the model with half its memory and should solve your problem. I'll integrate this soon

@ricardorei I did something very similar for the XL. I actually converted it in fp16 then I just changed one line in the feedforward.py
But after I wanted to go even further and use bitsandbytes/HF load_in_8_bit / load_in_4_bit = True but the integration is a mess between lightning and HF.
Last, FYI I did this as a WIP: https://huggingface.co/vince62s/wmt23-cometkiwi-da-roberta-xl
adapting your code in the existing HF XLM-roberta-XL code.
We are trying to implement it in CTranslate2 for much faster inference.

this will load the model with half its memory and should solve your problem. I'll integrate this soon

sorry to tell you and it's still oom

Same here. I cannot run it with A100 40G. However, I can run it with V100 32G, then, I found that the cost of memory is just 13G. The difference is that the lib of comet is installed by pip in V100, but installed by local edit in A100.


Table name not quoted in query for table data tab

I have a table named "Option" on my SQL server database. If I double-click the table in DBeaver to view the data, I get an error about incorrect syntax (OPTION is a keyword). However, the query works when the table is quoted either in double quotes as "Option" or with brackets as [Option]. Seems like a good idea to quote table (and schema, and database) names by default.

Reserved keywords must be quoted.
Quoting all identifiers is simpler but isn't convenient.
I'll add SQL Server keywords in the list of reserved keywords.


new(ci): added arm64 ci build and tests.

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area cmd

/area pkg

/area docs

/area tests

What this PR does / why we need it:
This PR adds arm64 CI jobs.
Fixed tests on arm64 (same buildmode=pie that we already had to do for go build in #151).
Moreover, improved CI: x86 and arm64 build-test jobs will now store artifacts in the workspace,
and they are later retrieved by "images" job.
This way we are able to avoid the qemu build of driverkit for arm64 when running buildx.
Old CI took ~1h, new CI takes ~20m.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

New workflow was tested: https://app.circleci.com/pipelines/github/FedeDP/driverkit/31/workflows/88a43039-088c-4698-ab92-906836208fc3.
Of course i had to disable image push (and docker login).

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: dwindsor, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [dwindsor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/lgtm

LGTM label has been added.  Git tree hash: dd70a963e733aafea8a447028ed8818af2c2e29e

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: dwindsor, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [dwindsor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Error: python-serial cant find candidate for install & python-cryptography not found

I get the folllowing Error messages when using this script on Rpi3B with Raspbian GNU/Linux 11 \n \l
Any workaround to this?

There is:
sudo apt install python3-pip
sudo pip install cryptography pyserial

I think it has something to do with other updated softwares. I am currently working through these issues myself. Facing other changes as well.
I think the general solution is to switch to pipenv and defining versions in the lockfile


Update sourcetree-beta to 2.6b1

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Additionally, if updating a cask:

[ ] If the sha256 changed but the version didn’t,
provide public confirmation by the developer: {{link}}

Appcast and download URLs for beta versions have stopped working at this exact moment. Most likely it's a temporary issue.

https://github.com/caskroom/homebrew-versions/pull/4100


ngx-datatable not showing data after closing and reopening ngx-bootstrap modal popup

I'm submitting a ...  (check one with "x")
[X ] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[ ] support request => Please do not submit support request here, post on Stackoverflow or Gitter

Current behavior

use the ngx datatable in a component that you wish to open in bs modalpopup of ngx-bootstrap , then close the poup and reopen, the table occupies space but does not show any data
Expected behavior

the table should still show data after reopening the modal popup
Reproduction of the problem

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Windows 7

Table version: 0.8.x

9.3.0

Angular version: 2.0.x

2.0

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Chrome

Language: [all | TypeScript X.X | ES6/7 | ES5]
ES5

I too am facing the same issue. any solution?

Also facing the same issue. Needs to resize the browser in order show the data.

Same here... This is quite an old ticket. Is anyone looking at this?

this is still not fixed in latest version, this librarry is nightmare so many bugs... so many open tickets.

Same here, data displayed after resizing the browser.
Anyone got a temporary solution for this?

Hi, had same issue and I resolved just changed the css of ngx-datatable:
datatable-header, datatable-body, .datatable-footer{
width: auto !important;
}

A workaround that worked for me with my own rolled out modal.
After you show the modal, manually trigger a resize event. You may need to wrap it in a setTimeout to make sure that the modal is actually shown in the DOM, before triggering resize event.
setTimeout(() => {
    window.dispatchEvent(new Event('resize'))
}, 0)

The workarounds mentioned didn't work for me, I forced rerender with *ngIf. I have dataArrived set to false and I'm setting it to true with setTimeout of 500


Update go-encrypt import path for custom url

ping @ryanskidmore

Hi again Peter,
Sorry for leaving this so long, completely forgot about it! I think it might be best (in my opinion) if all of the import URLs remain as github.com purely so if your domain goes down for whatever reason, the package is not affected - exception is for gopkg.in, since that is fairly established.
Hope this is okay.

It isn't an option - right now, I don't believe your code will compile from
scratch. You have to do the custom url
On Friday, October 9, 2015, Ryan Skidmore<EMAIL_ADDRESS>wrote:

Hi again Peter,
Sorry for leaving this so long, completely forgot about it! I think it
might be best (in my opinion) if all of the import URLs remain as
github.com purely so if your domain goes down for whatever reason, the
package is not affected - exception is for gopkg.in, since that is fairly
established.
Hope this is okay.
—
Reply to this email directly or view it on GitHub
https://github.com/ryanskidmore/GoWork/pull/9#issuecomment-146949890.

Eh, whatever. I don't quite agree with that but w/e

No as in its part of the golang spec, lol :) custom urls aren't optional.
On Friday, October 9, 2015, Ryan Skidmore<EMAIL_ADDRESS>wrote:

Merged #9 https://github.com/ryanskidmore/GoWork/pull/9.
—
Reply to this email directly or view it on GitHub
https://github.com/ryanskidmore/GoWork/pull/9#event-431698671.

I know, but I mean putting it on a custom URL as opposed to just a github url


fix  The table structure in testdata sql requires fewer fields than that in smaple

What this PR does:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

Codecov Report

Merging #385 (9d7e7d0) into master (78b8f64) will increase coverage by 1.17%.
The diff coverage is 0.00%.

:exclamation: Current head 9d7e7d0 differs from pull request most recent head 0bff304.  Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


CSV upload & Gitlab management

The API should provide operations to manage the upload of new CSV files to the HUB, making them accessible to e.g. QBer. This should act as a tunnel to Gitlab, where all persistent storage of datasets must be centralised.

Related to CLARIAH/qber#61

Closed with CLARIAH/wp4-requirements#17 and 982a651


Create Arrays.md

Description 📜
Added documentation on arrays, added types of arrays, its code, and output programs.
Fixes #3461

Type of change 📝

[x] Documentation (Content Creation in the form of codes or tutorials)

Domain of Contribution 📊

[x] C/CPP

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

Plagiarised from : https://www.thecrazyprogrammer.com/2021/05/advantages-and-disadvantages-of-array.html


Don't Escape Wiki domain update

The Wiki URL for the Don't Escape Wiki was changed/renamed. I think this edit is sufficient, otherwise let me know if not, thanks.

Sorry for the delay, been busy with IRL things and checking if things are ready for the 1.41 upgrade. Getting this pushed today in the next 5 hours.


Release '0.0.4'

Codecov Report

Merging #142 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #142   +/-   ##
======================================
  Coverage    75.1%   75.1%
======================================
  Files          25      25
  Lines        1458    1458
  Branches      230     230
======================================
  Hits         1095    1095
  Misses        363     363

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48a4e0b...c8609ef. Read the comment docs.

Codecov Report

Merging #142 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #142   +/-   ##
======================================
  Coverage    75.1%   75.1%
======================================
  Files          25      25
  Lines        1458    1458
  Branches      230     230
======================================
  Hits         1095    1095
  Misses        363     363

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48a4e0b...c8609ef. Read the comment docs.


[WIP] Bump libgit2

This updates libgit2.
Unfortunately it looks like all the remote APIs changed again :cry:
I hate to burn them down again, but it’s also bothersome that the APIs keep changing.
Anyone have thoughts about how we should address this?
/cc @phatblat

If we're going to #ifdef out the code, we should include the declaration in that so it's obvious to anyone who's using them.

Yup. :+1: I was just doing that so I could get the rest of it building to make sure I didn't miss any API changes.

Who knew push would be so :fire:? I'll take a look at the API changes and see about updating these disabled calls.

Made things happy with the API changes, but not 100% certain everything is correct yet as there are a handful of failing tests.

Down to one broken test which is failing because a newly created remote is having the expected git:// scheme changed to https://. Does libgit2 no longer support the git protocol?

What's the usual update process for libgit2 I see we don't use releases, right?

What's the usual update process for libgit2 I see we don't use releases, right?

No, it's been really ad hoc :disappointed: I'd love to have a more principled process but I doubt that'll happen until libgit2 itself does.

Does libgit2 no longer support the git protocol?

That would be really surprising but I honestly don't know.

@joshaber there was a 0.22.3 release today: https://github.com/libgit2/libgit2/releases/tag/v0.22.3 I guess this does not break APIs. @carlosmn also wrote a comment that 0.23.0 will take a little while still. https://github.com/libgit2/libgit2/issues/3135

Right, sorry, I meant more that since libgit2's still in a high state of flux objective-git will be too.

Sorry about changing the remotes API so heavily, but the current state was rather unfortunate. There will hopefully also be a similar change to submodules. Back in the day I introduced a pattern very similar to an ORM and that just doesn't really work with Git. These are the changes so we don't have to change anymore (hopefully...).
@phatblat if the url is getting changed, you might be hitting an insteadof rule, we support those now.

Well, @carlosmn was spot on. That last failing test was indeed an insteadOf rule I had forgotten about in my .gitconfig to get around corporate firewall issues:

With that, I believe this PR is ready for review.

That @carlosmn knows his stuff.
Thanks a ton @phatblat!

:tophat:


Move build&publish of Token Dashboard dApp to GH Actions

As described in RFC-18, there is a need for a refactorization of Keep
and tBTC release processes in order to reduce human involvment and make
the processes faster and less error prone. A part of this task is
migration from CircleCI jobs to GitHub Actions. This commit creates a
GitHub Action workfow for building of Token Dashboard dApp and
publishing it in keep-test Google Container Registry.

After discussion with @Shadowfiend and @nkuba I modified condition for Dashboard publishing (see 1abf1f4e7fc5212323aabdd283571a7160fc89bc). The publishing will occur only for workflows triggered due to merge to master.


add support for glue.id

The upstream Iceberg documentation details glue.id as a parameter which can be used to set the AWS Glue Catalog ID (AWS Account ID).
This is critical for scenarios where the Glue catalog exists in a separate environment from the workload running pyiceberg (e.g. an account separation between workloads and data lake).
This PR reads glue.id and, if set, injects CatalogId= as a parameter on all boto3 Glue service calls.

@HonahX since you're more familiar with Glue, would you be able to do this review? :)

@HonahX appreciate the review! Pushed a new commit with your suggestions.


treewide: cargoSha256 -> cargoHash

limited to pkgs/applications to limit merge-conflicts, done with
rg 'cargoSha256 = "sha256-' pkgs/applications/ -l | grep \\\.nix$ | xargs grep buildRustPackage -l | xe sd 'cargoSha256 = "sha256-' 'cargoHash = "sha256-'

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

https://github.com/NixOS/nixpkgs/pull/329763#issuecomment-2249126607


cholesky update/downdate?

Is there a Scipy function to perform rank 1 cholesky updates and downdates (as in MATLAB's cholupdate())? I was unable to find anything in the docs and also here in the issues, so I would really appreciate any information. Thanks!
PS: If this has not been implemented here, I would have to implement it for my use-case and if contributions on such a functionality would be welcome, I would be happy to make a contribution here (although I would be limited to python/cython).

LAPACK did not include the code that probably(!) matlab uses from LINPACK since nobody actually polished that code. There is Daniel Kressner's QRUPDATE package but I think the license is "free for noncommercial" and Octave is linking to that.
https://sourceforge.net/projects/qrupdate/
I am not sure if this is actually Kressner's qrupdate.
https://anchp.epfl.ch/QRupdate
There are other people who wrote similar functions but I couldn't find any benchmarks or stability analysis of these code. So I can't assess which way is to go. Hence it is indeed a bit cumbersome how to go about this. Maybe it is a better strategy to first consult this to LAPACK people. If they mention that there is no interest for them to include then we can think of alternatives.

Can this be computed from the qr_update?
Like qr_update and similar qr functions, I think including more linalg decomposition update functions would be useful to have in scipy.

@ewmoore Care to chime in?

xref Reference-LAPACK/lapack#209.

@xoviat matlab uses lapack (linpack actually) already for that.

Yes, it uses linpack but not lapack. So we should open an issue so that LAPACK will add the function, and then can backport it for older LAPACK APIs.

Thank you everyone for your very helpful comments (and please forgive my delayed response). For what it's worth, I completely agree that the ideal solution moving forward would be to have it integrated into LAPACK itself. Hopefully there will be positive news coming from that direction.

Hi everyone, so it has been almost 2 weeks since @xoviat posted the issue on the LAPACK discussion board but nobody over there has responded. I am not too familiar with the development process at LAPACK, so I am wondering if anyone knows if this is "normal"?
PS: My apologies for forgetting to note this earlier, but thank you @xoviat for taking the time to post on their issues board.

some Lapack issues/question about adding QR or Cholesky rank-one update are 7 to 10 years old.
I wouldn't plan on that they get magically added soon.

Not only that but also even if it is added tomorrow to LAPACK, we won't be able to use it since we are barely trying to upgrade to LAPACK v3.4.1. So that would be at least quite a few years until we make it to v3.9.0.
The reason that I've mentioned LAPACK however is because I wanted to get their opinion about the reference implementation so that we can mimick it. There are many versions lying around and I don't know which one is the reference implementation.

The function won't be added to LAPACK unless someone submits a PR to them.
If no one takes the initiative, then it won't happen. Regardless, when is
the next SciPy release? We have plenty of time for this to be added.
Although I'm skeptical that it will make it in (even noting the length of
time) because no one with the capability to tackle this issue seems to have
much interest.

@ilayn That's true, but it would be trivial to backport that particular
function (e.g. only include it if the symbol is undefined).

Of course. Although I was not expecting that it would be added to lapack anytime soon, my question was more in the vein of "are they not interested at all?". Having said that, I definitely was not aware that it would take "at least quite a few years" to get to the latest LAPACK version -- that is good to know.

My impression as an outsider is that there is no reference implementation at LAPACK standards. At least for the QR update discussion I got the impression that there are unresolved numerical/technical details.
(In QR one problem is numerical stability in downdating when matrix is badly conditioned and what's the best way around it.)

Let me be clear: if the function is added to LAPACK, then you will see it
in SciPy by the next feature release (1.10 as of now), assuming that it's
added with a reasonable lead-time to have it in SciPy. I can make sure of
that (assuming there are no objections, which I don't think there would
be). But if the function is not added to LAPACK, then you will probably not
see it added to SciPy because we need to be sure that it is numerically
stable as well.

@ashimb9 LAPACK is like firmware, you change a single comma and people will start complaining about checksums not matching :) It's one of "the" most central libraries so people often tend to follow from a few versions behind.
What I meant is if we know the reference algorithm, we can Cythonize it ourselves. But as I said, so far it is not clear what the update procedure is (at least to me). That's why I've pinged @ewmoore (let's try again one more time) because qr_update is authored by him in Cython and not in LAPACK routines.

On a more optimistic note:
the previously linked http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=2&t=2646
contains the code. dchdr1.txt  and dchur1.txt
AFAIK, the numerical standard in scipy is lower than in LAPACK. So, I don't think there should be a strong objection to having something like what's there, even if eventually it might be possible to replace it with a better LAPACK standard version.
That is, scipy could gain something soon that works for most cases even if it's not the best possible solution for ill conditioned cases.
"The downdating problem needs care." (quote from LAPACK issue) From what I read during the QR updating issue, QR updating is numerically stable, only the down dating might need a warning for numerical stability in ill-conditioned cases.

@josef-pkt Thanks for sharing your experience with the QR update discussion. It helps explain the apprehension on their side re the updates (or specifically the downdates). It would make sense for them hold back on implementation if the instability issues have not been appropriately resolved.
@xoviat I hear you. And thank you for your enthusiasm to bring this functionality to SciPy!
@ilayn Got it. And I agree, it does make a lot of sense to stick to the tried and tested versions, especially when a lot of critical functionality depends on it. Also, I hope my previous comment did not come off as a negative reaction -- FWIW that was certainly not the case.

@ashimb9 No problem at all. Just tried to summarize the intertwined story. Took me a while to figure it out too.

Adding these routines is certainly in scope.  Like anything else, it requires someone to step up and write the code.  You can find plenty of papers that document algorithms.  I don't know which paper we should follow, an interested person needs to find these papers and review them, etc.  AFAIK this only become complex when downdating would cause the matrix to no longer be positive definite.
We shouldn't wait for these to appear in lapack.  It would be nice certainly, but that might also mean waiting forever. One data point for not waiting would be that Julia appears to have merged rank-one updates / downdates in this ticket: Julialang/Julia#14243

Note that I am NOT suggesting that we wait for these to appear in LAPACK. What I am suggesting is that IF someone wants to step up to write the code (I currently don't have the time), it would be more time-efficient for them to submit a PR to LAPACK and then simply copy the FORTRAN file into SciPy. Then, we would get LAPACK vendors to optimize the code for us.

Thanks @ewmoore I'll have a look at them to see what I can do. In the meantime, I've found DOI:10.1007/BF01933218 as the starting point. Julia implementation looks a bit naive with manual Givens rotations. I'll try to make an initial PR soon.

@ilayn Also please feel free to let me know if I can be of any help at any point. And even though I am not familiar with Fortran, if there emerges anything python/cython related I can help with, I would be happy to do so.

@ashimb9 Thanks. The paper I've linked to is from the second author of Alexander et al. There is also a comparison within the paper about that. That's why I've picked it up.

OK here is a basic prototype with very little attention to optimization
from numpy import zeros_like, empty_like, sqrt, spacing
from scipy.linalg import LinAlgError

def algoB(R, z, lower=False):
    """
    Cholesky rank-1 downdate algorithm taken from [1]_ such that
    ``D'D = R'R - zz'``

    Parameters
    ----------
    R : (N, N) ndarray
        The 2D square array from which the triangular part will be used to
        read the Cholesky factor.The remaining parts are ignored.
    z : (N,) ndarray
        Downdate array
    lower: bool, optional
        Whether Cholesky factor is upper or lower triangular Cholesky
        factorization.  Default is upper-triangular.
    overwrite_R: bool, optional
        If set to True ....
    overwrite_z : bool, optional
        If set to True the entries of the input array will be modified during
        the computations instead of creating a new array.

    Returns
    -------
    D : (N, N) ndarray
        The resulting downdated Cholesky factor.

    References
    ----------
    .. [1] DOI:10.1007/BF01933218
    """
    # Just assuming arrayness, shape checks and overwrites etc. are handled.
    # Then... (with overwriting z)

    n = R.shape[0]
    eps = n * spacing(1.)  # For complex this needs modification
    alpha, beta = empty_like(z), empty_like(z)
    alpha[-1], beta[-1] = 1., 1.
    D = R.copy()

    for r in range(n):
        a = z[r] / R[r, r]
        alpha[r] = alpha[r-1] - a ** 2
        # Numerically zero or negative
        if alpha[r] < eps:
            # Made up err msg.
            raise LinAlgError('The Cholesky factor becomes nonpositive'
                              'with this downdate at the step {}'.format(r))
        beta[r] = sqrt(alpha[r])
        z[r+1:] -= a*R[r, r+1:]
        D[r, r:] *= beta[r] / beta[r-1]
        D[r, r+1:] -= a/(beta[r] * beta[r-1])*z[r+1:]

    return D

With this so far any array size worked (tested up to 5K) and only 4x slower than matlab.
n = 1000
A = np.random.rand(n, n)
A = A + A.T + n*np.eye(n)
R = sp.linalg.cholesky(A)
z = np.random.rand(n)
# Uncomment the following to make it occasionally trip up
#z *= 0.95 * np.sqrt(n)
zz = z.copy() # z is overwritten
D = algoB(R, zz)
np.testing.assert_allclose(D.T@ D - (R.T @ R - np.outer(z, z)),
                           np.zeros([n, n]), atol=n*np.spacing(100.))

The parts that need attention

the walking backwards from bottom right corner up if lower is True.
Complex arrays and bookkeeping the datatypes to avoid accidental upcasting.
matlab seems to report the r value where it trips as an optional output. Should we? We try to avoid varying number of outputs.

@ashimb9 Would this be sufficient input for you stress test it and expand? If you need it, I can do the backwards walking too without too much hassle (famous last words).

Hey, license compatible code here: https://github.com/mp4096/cholesky-update?files=1

@xoviat Still using home-baked norms and givens rotations. We need to convert them into LAPACK calls. So again needs work. I don't mind but seems unnecessarily bloated.

@ilayn Thank you! I will give it a shot and get back to you. Also, since you mentioned comparison to Matlab performance, I am wondering if there are any unofficial thresholds on relative performance?

@ashimb9 What kind of thresholds?

Threshold as in the max performance gap relative to Matlab. So you mentioned that it is currently 4x slower than Matlab right now, so I am wondering if "at the most" it can be, say, no more than 2X slower or something like that? Of course ideally we want it faster, but I am just trying to get a sense of the ceiling.

I don't know any such thresholds but of course it should be reasonably fast. 4x is OKish for pure python if there is no other way. But matching matlab speed is a moving post with JIT and etc.
It's a nice goal but the actual target is as fast as possible while keeping an eye on memory cost. Stability and maintainability are, in certain cases, more preferable to ultra-optimized but undebuggable code. Here it is simply a triangular walk over a square array so I wouldn't expect too much discrepancy.

@ilayn I cannot seem to access the paper you have referenced (DOI:10.1007/BF01933218), even through a university-based journal access portal. Would you happen to know of any non-paywall URL for the paper? Alternatively, do you know if the proposed algorithm is the same as in the Alexander et al paper referenced above?

@epsilonmach Here is the excerpt for the algo I've coded above by removing the internal loop by NumPy ufuncs.

As I mentioned above this is also written by Pan who is also the 2nd author of the Alexander et al paper. The author claims this simplifies the hyperbolic rotation method (aka LINPACK code).
Even proposes another algo C but that didn't seem to make too much of a difference though being more complicated. This one however is, obviously, pretty easy to code.

@ilayn That was very helpful -- thank you!

@ilayn gh-16499 says that it follows Pan 1990 for rank-1 updates and downdates and Van De Geijn 2011 for rank-p updates and downdates.
However, in gh-8286, you stated that

downdate and update uses different strategies namely, update is Krause Igel 2005, and downdate is Pan's algorithm that was discussed in [gh-8188]

This suggested that there is some way that these things are always done / must be done. When asked why, you pointed to Julia's code as a reference. You also seemed to indicate that the code you wrote above should not be used.
Do I understand that a requirement for addressing gh-8188 is that we reproduce Julia's strategy for updates/downdates in SciPy, or are other algorithms allowable provided that they meet certain goals? If so, what are those goals that were not met by gh-8286?
Please note that gh-8286 already satisfied these three criteria:

Produce a valid updated Cholesky factor when it exists.
Detect when the updated matrix would not be positive definite.
Exceed the performance of factorizing the updated matrix from scratch.


Support route rules per HTTP verb

Describe the feature
Hello,
Actually, a route rule apply to a pattern but having it per http verb could be useful, especially for rate limiting.
Imagine the route GET /me, I do not need to rate limit it but the route PATCH /me will send an email to the user so I need to rate limit it.
Additional information

[ ] Would you be willing to help implement this feature?

coming in nitro v3 (since rout3 library used for route rule matching natively supports verbs)


Implement emoji (Notion-Version 2021-08-16) and Filter group

Hi,
Is the implementation of the latest version of the Notion api planned?
I would like to access the emoji of the page.
Also is it possible to group filter ? I haven't found the solution yet...
DatabaseQuery().all()/any() can't contain another implementation of them selve from what i see

Thanks
Denis

Hi!  Thank you for this issue!

emoji (Reference): yes this is definitely planned! I have no timeline however, as I am currently in vacation. If you are familiar with Kotlin and have some free time and would like to contribute, this will be welcome!
group filter (Reference): I wasn't aware this was possible with the API until now. I am not sure what happened, maybe this has been added by the Notion team after I implemented the first simple version... In any case, yes I would like to implement this but unfortunately this won't be that simple and will probably introduce a breaking change in this lib's API :)

I will leave this issue open for now.

Hi @DenisD3D, I've just release v1.8.0 which adds support for icons and covers in pages and databases. Group filter is not yet implemented though - so keeping this issue open for now.


test(code-block): add test for html text escaping

Follow up to #169
I forgot to add a test, so...
Just a simple test to ensure, that escaping is performed properly for code blocks.

Just a question: Shouldn't fenced code blocks be structured like this?

  // My code

Because currently, it just consists of a
-tag. And I also only know it the way from my example.

Thanks!


[WIP] Auto wrap YAML BOMs as catalog bundles

Wraps pure-YAML BOMs uploaded to catalog in a bundle, creating a name and version if needed.
This allows us to see all items added as a unit.  This will allow us to prefer loading co-defined items (though that's not fully done, still relies on preferring search path when doing item lookup), and to edit or remove that item as a set (crucially we now keep the BOM, previously we discarded it!).
Docs changed to reflect this and explain differences in another PR.
The basic wrapping was pretty simple but a surprising number of subtleties emerged which this fixes:

java scanning doesn't work in OSGi; docs and errors make this clear
impacts replacement semantics (now it does a hash on the bundle to make most things the same, though a re-post of a non-SNAPSHOT BOM that doesn't declare a bundle will now fail because the random bundle name is stored on the item as containingBundle and is different on a re-post; and needed extra work to remove leaked anonymous bundles)
had to change the way we set up the libraries / search path, to exclude this "bundle" (mainly so tests are unchanged, but also improves efficiency for simple BOMs; also the code is cleaner now)

And there is one thing that emerged that I haven't fixed:  on rebind it became vital to handle the situation where dependent bundles might not be loaded before the depender bundle; see failing testtestLongReferenceSequence.  Not entirely sure how to do this /cc @neykov @geomacy @aledsage.
Depends on #743.

have done a big job to make things use the TypeRegistry instead of the old catalog, and to add to type registry in an UNRESOLVED state first.  this fixes the big problem by adding types from all bundles into an unresolved state, then resolving them.
it has ended up much bigger than i meant BUT it has several main advantages:

we no longer persist catalog item XML for things where we persist the OSGi bundle
we're very close to being able to remove nearly all the clunky "catalog" code; apart from items that were persisted as XML everything uses the new type registry now

the logic for guessing the supertypes is messy, hooking in to the old Guessing interpreter in a brittle way, but if we plan to remove old catalog code (and annotations scanning) we can rewrite it all to be much simpler and nicer.
note this PR is still WIP, not working 100%; we don't support templates in the registry yet, locations need attention, and there are quite a few TODO items to look at.  but wanted to give people sight of it.

This now addresses templates in TypeRegistry -- simply using a "catalog_template" tag.  There was no immediate need for "deployable" or "final" semantics so those have not been added.
This also addresses a newly-found issue to do with locations:  the location registry had a hokey relationship with the old catalog; it's now a cleaner facade.  See comments in BasicLocationRegistry for this.
Finally I found a way to preserve the capability where the same BOM YAML is uploaded multiple times:  if the bundle of the currently installed type was a wrapper, we now allow to replace it if the plan itself is unchanged.  This removes the backwards compatibility issue introduced around uploading a BOM multiple times and it disallowing because the containing bundle was changed.
The REST API however still points at the old catalog routines, so it won't yet show items added through OSGi.  This is a showstopper of course because all BOMs will be added through OSGi.  Fix for that coming soon.

REST API is now compatible.  So this is ready for review @geomacy @neykov @aledsage .
Apologies for the size -- hopefully between this and the ML it is clear why there was so much.  Perhaps I should have disabled it so it could have been merged incrementally but I was not expecting so much more to be needed.
Still it is all good stuff moving forwards.  We are now very close to being able to remove all the catalog code, in favour of the very simple TypeRegistry.

@geomacy This should solve load order problems and any problems around BOM scanner.  (I think we want to disable the auto-scan osgi bundles BTW.)  Let me know if not.  And of course please don't go fixing other things -- ask me if in any doubt.

hi @ahgittin you've accidentally committed an unresolved conflict in CatalogUtils in 47b54d4

oops - thanks @geomacy; fixed that

Attached file contains logs for the above test results, and the persisted bundle data for the second.
test-notes.zip

Thanks @geomacy.  Glad you tried Karaf mode -- didn't think to test that.  Will repair.
BTW wanted to note, this fixes https://issues.apache.org/jira/browse/BROOKLYN-343 .

right, see note on last commit which should solve the karaf mode problem you spotted @geomacy.  also does a little something to solve the situation you encountered @richardcloudsoft .

@geomacy comments thus far addressed, including CatalogBomScanner removed

The build break on 98d9b0a is genuine, I get the same thing:
Failed tests:
  ApplicationsYamlOsgiTest>ApplicationsYamlTest.testDoesNotWrapCatalogItemAndUsesCatalogName:216->ApplicationsYamlTest.assertDoesNotWrap:436 expected [catalogLevel] but found [Application (wiifyf67tn)]
  ApplicationsYamlOsgiTest>ApplicationsYamlTest.testNameInCatalogMetadata:364->ApplicationsYamlTest.assertDoesNotWrap:436 expected [My name in top-level] but found [Application (co8ot9tzt2)]
  ApplicationsYamlOsgiTest>ApplicationsYamlTest.testNameInItemMetadata:382->ApplicationsYamlTest.assertDoesNotWrap:436 expected [My name in item metadata] but found [Application (u1xjn8fsai)]
  ApplicationsYamlTest.testDoesNotWrapCatalogItemAndUsesCatalogName:216->assertDoesNotWrap:436 expected [catalogLevel] but found [Application (s5iotaeji9)]
  ApplicationsYamlTest.testNameInCatalogMetadata:364->assertDoesNotWrap:436 expected [My name in top-level] but found [Application (ojj1x9bq1v)]
  ApplicationsYamlTest.testNameInItemMetadata:382->assertDoesNotWrap:436 expected [My name in item metadata] but found [Application (hdis9rqcee)]
  CatalogOsgiYamlEntityTest.testOsgiNotLeakingToParent:539 Unexpected error message: java.lang.IllegalStateException: Bundle BasicManagedBundle{symbolicName=org.apache.brooklyn.test.osgi.entities.SimpleEntity, version=0.1.2.update, url=null} failed installation: Error installing catalog items: Failed to install org.apache.brooklyn.test.osgi.entities.SimpleEntity:0.1.2.update, types [BasicRegisteredType[org.apache.brooklyn.test.osgi.entities.SimpleEntity:0.1.2-update;null]] gave errors; 2 errors including: UnsupportedTypePlanException: Invalid plan; format could not be recognized, none of the available transformers [brooklyn-camp:CampTypePlanTransformer, java-type-name:JavaClassNameTypePlanTransformer] support BasicRegisteredType[org.apache.brooklyn.test.osgi.entities.SimpleEntity:0.1.2-update;null] expected [true] but found [false]
  CatalogYamlEntityNameTest.testUsesNameInCatalogItemMetadataIfNonSupplied:50 expected [nameInItemMetadata] but found [BasicEntity:faw9]
  CatalogYamlEntityOsgiTypeRegistryTest>CatalogYamlEntityTest.testAddCatalogItemVerySimple:60 expected 'services:' block: BasicRegisteredType[my.catalog.app.id.load:0.1.2;null]
type: org.apache.brooklyn.entity.stock.BasicEntity expected [true] but found [false]
  CatalogYamlEntityOsgiTypeRegistryTest>CatalogYamlEntityTest.testCreateSpecFromCatalogItem:371 » IllegalState
  CatalogYamlEntityTest.testAddCatalogItemVerySimple:60 expected 'services:' block: BasicRegisteredType[my.catalog.app.id.load:0.1.2;null]
type: org.apache.brooklyn.entity.stock.BasicEntity expected [true] but found [false]

thanks @geomacy - let's watch this run, hopefully tests now all good!

tests good now @geomacy

thx @geomacy -- great comments.  all addressed.

@geomacy yes, there's an open question how to treat unresolved items, whether to uninstall the bundle (so it could be fixed) or leave them in the unresolved state; i think the latter (current beahviour) is pretty good, but we'll see what we find
as for deletion you're right there's no easy way to delete a bundle, that's a known issue, to be fixed soon.  (you have to stop brooklyn and go into persisted state, or run some hairy groovy.)
so i think merging.

👍  merging


Process Hangs with activated opcache

Hi,
tbh, I don't know if I am correct here,  but my research showed me earlier issues with interactions with opcache and revolt/event-loop so I hope you might be able to help us figure out this issue.
We have developed an AV SDK and at one point we decided to go async because of connection issues (basically using two connection one websocket and an https connection for an upload). So the first step we did is replacing our old Guzzle http-client with the amphp one replacing our upload feature.
With this SDK we build a nextcloud-app and ran into an issue that (for now, because its not released with that feature) only exists in our pipeline.
The setup locally and in the CI are almost the same, the tests that fail are run in a container but the behaviour differs. Locally everything works fine, but in the CI we always get the "Empty reply from server" error, what basically means the server did not answer. My assumption was that something (like a repeat or a delay) might sill be active in the eventloop but that would also have caused the issue on our local setup.
We could mitigate the issue by disabling the opcache via opcache.blacklist_filename and adding the whole amphp path to the blacklist, but that is almost not an option for the nextcloud app, because the blacklist has to have full-pathes and the customer would have to write it by semself.
I am basically helpless right now. Do you have any Idea, maybe at the SDK code, where use the http-client or some runtime-trick to prevent opcache from caching the amphp directory completely?
I also opened this issue on the revolt repository.

The image we are using has php version 8.2.21 installed.

Ok. To further explain. This CI runs a nextcloud container with an App that we developed.
The test does a put against the nextcloud instance to upload a file. While the upload our App checks against our backend if the file contains a virus, this is done via http PUT. Previously we used Guzzle for the http PUT but we exchanged it for amphp which has revolt as a dependency.
The code in question is this
                $cancellation = new DeferredCancellation();
		$connection = $this->_vaasConnection->GetAuthenticatedWebsocket();
		$pingTimer = EventLoop::repeat(5, function () use ($connection) {
			if ($connection->isConnected()) {
				$connection->ping();
			}
		});

		try {
			$httpClient = (new HttpClientBuilder())
				->skipAutomaticCompression()
				->skipDefaultAcceptHeader()
				->skipDefaultUserAgent()
				->build();

			$request = new Request($url, 'PUT');
			$request->setProtocolVersions(["1.1"]);
			$request->setTransferTimeout($this->_uploadTimeoutInSeconds);
			$request->setInactivityTimeout($this->_uploadTimeoutInSeconds);
			$request->setBody(StreamedContent::fromStream($fileStream, $fileSize));
			$request->addHeader("Authorization", $uploadToken);

			$response = $httpClient->request(
				$request, new TimeoutCancellation($this->_uploadTimeoutInSeconds), $cancellation->getCancellation());
			if ($response->getStatus() > 399) {
				$reason = $response->getBody()->buffer($cancellation->getCancellation());
				throw new UploadFailedException($reason, $response->getStatus());
			}
		} catch (\Exception $e) {
			if ($e instanceof HttpException) {
				throw new UploadFailedException($e->getMessage(), $e->getCode());
			}
			throw new VaasClientException($e->getMessage());
		} finally {
			if (EventLoop::isEnabled($pingTimer)) {
				EventLoop::cancel($pingTimer);
			}
			if (!$cancellation->isCancelled()) {
				$cancellation->cancel();
			}
		}

Looking at the CI we can see, that the request runs into a segmentation fault:

This only happens, when opcache is activated and does not happen, when using Guzzle as http-client. But I also did not manage to reproduce this bug locally.
I did another test, removing the EventLoop::repeat. The segmentation fault keeps coming .

Here you can find the coredumps from the apache child processes:
https://github.com/GDATASoftwareAG/nextcloud-gdata-antivirus/pull/98#issuecomment-2291117317
And in our Dockerfile we have an automated way to get the apache2 debug-symbols for our version:
https://github.com/GDATASoftwareAG/nextcloud-gdata-antivirus/blob/renovate/major-all-major-patch/Dockerfile.Nextcloud#L8-L15

With dump_bt we get this
(gdb) dump_bt executor_globals.current_execute_data
[0x7efca1750180] version_compare(false, "0.3.0", "<") [internal function]
[0x7ffd3f49eeb0] Amp\File\Driver\UvFilesystemDriver::__construct()
/var/www/html/apps/gdatavaas/vendor/amphp/file/src/Driver/UvFilesystemDriver.php:0

Hi @unglaublicherdude!
Thanks for the additional information. I saw this issue when you originally posted it, but was left scratching my head, as there isn't really enough in your initial post for me to have any idea what was happening. I believe "Empty reply from server" is from curl, not amphp/http-client, so I'm not sure what's going on there.
[0x7efca1750180] version_compare(false, "0.3.0", "<") [internal function]

I find this line unusual, since this shouldn't be happening. This could be the result of a bug in opcache optimizing out something which it should not have. false should never be passed to version_compare. This is in amphp/file. I pushed branch, https://github.com/amphp/file/tree/opcache-bug, which changes the code a bit to potentially avoid a bug, if there is one. Could you change your dependency in composer.json for amphp/file to dev-opcache-bug and see if the fix works?

Thank you for the reply. And sorry for my initial post. It was a bit of braindumping.
Thank you for the offered solution. It still does not work, but at least now it seems to be an issue with our app or nextcloud so I can go on debugging that.
(gdb) dump_bt executor_globals.current_execute_data
[0x7f445741c2d0] OC\AppFramework\Utility\SimpleContainer->query("OCA\Files_Trashbin\Trash\ITrashManager", false)
/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php:153
[0x7f445741c200] OC\ServerContainer->query("OCA\Files_Trashbin\Trash\ITrashManager")
/var/www/html/lib/private/ServerContainer.php:145
[0x7f445741c0d0] OCA\GDataVaas\AvirWrapper->OCA\GDataVaas\{closure}() /var/www/html/apps/gdatavaas/lib/AvirWrapper.php:177
[0x7f445741c060] call_user_func(object[0x7f445741c0b0]) [internal function]
[0x7f445741bfe0] Icewind\Streams\CallbackWrapper->stream_close() /var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php:119
[0x7f445741bf70] fclose(resource(#1357862240)) [internal function]
[0x7f445741beb0] OC\Files\Storage\Local->writeStream("files/functionality-parallel.eicar.com.txt.ocTransferId1605683835.part",
resource(#1357862240), NULL) /var/www/html/lib/private/Files/Storage/Local.php:652
[0x7f445741bdd0] OC\Files\Storage\Wrapper\Wrapper->writeStream("files/functionality-parallel.eicar.com.txt.ocTransferId1605683835.part
", resource(#1357862240), NULL) /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php:653
[0x7f445741bcf0] OC\Files\Storage\Wrapper\Wrapper->writeStream("files/functionality-parallel.eicar.com.txt.ocTransferId1605683835.part
", resource(#1357862240), NULL) /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php:653
[0x7f445741bc50] OCA\GDataVaas\AvirWrapper->writeStream("files/functionality-parallel.eicar.com.txt.ocTransferId1605683835.part",
resource(#1357862240), NULL) /var/www/html/apps/gdatavaas/lib/AvirWrapper.php:108
[0x7f445741bb70] OC\Files\Storage\Wrapper\Wrapper->writeStream("files/functionality-parallel.eicar.com.txt.ocTransferId1605683835.part
", resource(#1465903520)) /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php:653
--Type <RET> for more, q to quit, c to continue without paging--
[0x7f445741b900] OCA\DAV\Connector\Sabre\File->put(resource(#1465559264)) /var/www/html/apps/dav/lib/Connector/Sabre/File.php:250
[0x7f445741b7e0] OCA\DAV\Connector\Sabre\Directory->createFile("functionality-parallel.eicar.com.txt", resource(#1465559264))
/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php:148
[0x7f445741b6e0] Sabre\DAV\Server->createFile("files/admin/functionality-parallel.eicar.com.txt", reference, reference)
/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php:1098
[0x7f445741b5b0] Sabre\DAV\CorePlugin->httpPut(object[0x7f445741b600], object[0x7f445741b610])
/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php:504
[0x7f445741b4c0] Sabre\DAV\Server->emit("method:PUT", array(2)[0x7f445741b520])
/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php:89
[0x7f445741b3f0] Sabre\DAV\Server->invokeMethod(object[0x7f445741b440], object[0x7f445741b450])
/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php:472
[0x7f445741b2e0] Sabre\DAV\Server->start() /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php:253
[0x7f445741b280] Sabre\DAV\Server->exec() /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php:321
[0x7f445741b1f0] OCA\DAV\Server->exec() /var/www/html/apps/dav/lib/Server.php:383
[0x7f445741b150] (main) /var/www/html/apps/dav/appinfo/v2/remote.php:35
[0x7f445741b020] (main) /var/www/html/remote.php:172

Look like you managed to find another bug in PHP! As you said, at least this one seems to be happening in your code so hopefully you have an easier time diagnosing it.
I've tagged 3.1.1 of amphp/file, so you should be able to revert your change to composer.json and get the new version with the fix.

I believe this was fixed by changes in amphp/file. If not, please reopen this issue or create a new one. Thanks! 👍


How to undo DDL statements

See this forum post.
Summary:

If you're within the GC window (24 hours by default), you can use AS OF SYSTEM TIME queries to get the previous values and insert/upsert them.
Full-history backup.

We haven't received questions around this, so closing for now.


 Is it working and is it still needed or does it add an additional layer of possibly unneeded complexity?

When include_missing_data=True it allows creation of empty states in the World_QSR_Trace.trace in order to maintain the "same" number of timestamps as in World_Trace.trace.
a) include_missing_data is a bad name; allow_empty_states is probably a better one.
b) don't see the point of it as such assumption is violated in a number of currently implemented QSRs, e.g. MOS, QTC*S
Might want to deprecate as such; as it makes things confusing. It is easier to see what states would have been empty by getting the remainder of the intersection of World_Trace.get_sorted_timestamps() and World_QSR_Trace.get_sorted_timestamps() for most of the QSRs.

I never knew what it was good for so I didn't implement it right I guess. I assumed it triggers some black magic in the abstract class ;)
I don't need anything like this.

Don't think anyone does. Will deprecate, actually remove. (realisedI was talking with myself in the first two posts, hmmm).

Those are the most fruitful discussions some times ;)


Fix Wrong count of products with price search

Questions
Answers

Description?
The price range slider does not update correctly the counter of products with specific prices. But Ask in issue before : https://github.com/PrestaShop/PrestaShop/issues/25819#issuecomment-950121389

Type?
bug fix

BC breaks?
no

Deprecations?
no

Fixed ticket?
Fixes PrestaShop/Prestashop#25819.

How to test?
See Steps to Reproduce section of PrestaShop/Prestashop#25819

This change is

@alexandrebrubeyonds Bro this completely broke pagination. I reverted it in my last PR. We need to find some other way. ;-)

@alexandrebrubeyonds Bro this completely broke pagination. I reverted it in my last PR. We need to find some other way. ;-)

Oh really ? 😨


ORC-508 remove hadoop dependency

This change makes Hadoop an optional dependence for ORC reader/writers.
It creates a new package org.apache.orc.core that contains a new version of OrcFile, OrcConf, etc. that don't use Hadoop classes.
The old classes stay in their old place and should be compatible. The tests required relatively little change.
I also included a new version of the tools that does not include Hadoop. It is naturally much smaller (3.6mb vs 40mb).
I compiled Hive master branch with the patch to find any places where I could minimize the API changes. Most of the changes are things that changed in ORC 1.6 to master, but they aren't much. https://github.com/omalley/hive/commit/12c1e6416990b7387b7bb823a62468e5a49f24d6
Remaining question:

We still need Writable and WritableComparable from hadoop. Those interfaces are tiny and haven't changed in more than a decade. I currently put them into the nonhadoop tool directory, but they should probably be somewhere else. Thoughts?

Other notes/questions:

There is a new implementation of HadoopShims that avoids using Hadoop.
For classes where I needed a second implementation that doesn't use Hadoop, I used *Core.
Do we actually want to commit the nonhadoop module? It is a complete duplicate of tools. It is however, a nice demonstration that you can read and write ORC files without hadoop on the classpath at all.
If we do keep nonhadoop, it currently duplicates the sample data from tools, which should probably be fixed.

Clearly we also need to update a fair amount of documentation also.


Add cross-platform packages with NuGet

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

It is not encouraged to repackage binaries in general. However, I found that NuGet installations can be much smaller than direct compilation outputs, which probably bundle some dotnet runtimes. Using a NuGet package on Windows has an additional benefit in that hiding *.dll files from %PREFIX% prevents conflicts with other packages.

@conda-forge-admin please rerender

Closing because the dotnet tool does not support cross-platform installation.


Enhancement: Changing language Options

Some language Options are not set correct (e.g. if the language is set to English, the text in "generate training data" is still german. This should be changed.

see #138

Tested the executable for that pr. Still works properly.


Check pre-commit ruff formatting for notebooks is set up correctly

I'm not 100% sure the pre-commit checks are fully/properly working for ruff formatting of notebooks. Currently we have:
https://github.com/pymc-labs/CausalPy/blob/9a4daf7cb4b86c56c3f569a18f0212482c3a1986/.pre-commit-config.yaml#L26-L35
Maybe we don't need nbQA at all and solely use ruff-pre-commit? The README at https://github.com/astral-sh/ruff-pre-commit shows that it's pretty simple to make it apply to Jupyter notebooks.

It is not necessary to use nbqa, the latest ruff pre-commit hooks support jupyter and pyi files directly. The third code block shows how: https://github.com/astral-sh/ruff-pre-commit


Initial Jenkins integration of TestGrid

Description:
Integrate TestGrid with Jenkins pipeline.

This work is completed hence closing this issue.


Modify yosys optimizer to operate on secret.generic

See https://github.com/google/heir/issues/268 for context

Hmmm, I was trying to make this nicer by refactoring the part that invokes Yosys into a RewritePattern, but it seems like the Yosys API is global and doesn't support being run in that context. I end up getting a bunch of errors like malloc_consolidate(): invalid chunk size

OK so this is a preliminary idea and I wanted to have the chance to talk to you about it, but that will have to wait for a week due to holidays. Let me just write up my idea here in the mean time.
tl;dr: you can try this:
bazel run //tools:heir-opt -- --pass-pipeline='builtin.module(func.func(secret.generic(yosys-optimizer)))' $PWD/tests/yosys_optimizer/add_one_generic.mlir

result in a gist
And chain it with secret-distribute-generic
bazel run //tools:heir-opt -- --pass-pipeline='builtin.module(func.func(secret.generic(yosys-optimizer)), secret-distribute-generic)' $PWD/tests/yosys_optimizer/add_one_generic.mlir

result in a gist
Longer explanation:
My general idea is that we should maintain the secret.secret as long as we don't have any lower-level FHE dialect involved in the computation. The way we can achieve this in the tosa-to-tfhe-rs flow is as follows:

Convert a model to arith (affine loops + arith loop bodies)
Wrap the computational parts of each loop body in a single secret.generic, as well as the computational parts between each loop.  (This requires somehow ensuring there are no intermediate stores/loads, or pushing them to the start/end of the loop body; I don't remember what we did for this before, but I imagine we just relied on the loads/stores just so happening to be at the start/end of the loop body)
Run (a cleaned up version of) this yosys optimizer pass in this PR on each secret.generic op, possibly with loop-unrolling optimizations as discussed in https://github.com/google/heir/issues/257. The output of that are secret.generic ops, one for each block form the previous step, and each containing many comb ops.
Run secret-distribute-generic on the result, which splits the generics down to single comb ops.
Convert each single op to a cggi op (at which point the i1s and tensor<?xi1>s get converted to lwe.ciphertext).
Run the lwe to cggi pass and onward as usual.

This ensures that comb ops never have overloaded semantics: they are only ever allowed to operate on cleartext types, and it's the conversion of a secret.generic op containing a lut op that is allowed to become an lwe lut op.

should there be asserts that there's only one op per generic body...?

As a precondition for the passes that require this, yes.

whether the change to tensor<?xi1> from #264 is still compatible.

I suspect we can make the outline in this work regardless of which type we choose, but I agree tensor is the better choice.

@asraa Updated this PR and it should be ready at least for us to discuss:

Added "manual" type conversions before and after the secret.generic being converted. This is essentially the arith version of comb.extract_bit and the inverse.
Updated the tests to use secret.generic instead of func.

Still todo:

Replace the manual type conversions with a custom op (secret.cast) which will be cleaned up during the dialect conversion.

@asraa ready when you are

Fixed all but that last remaining comment.


Savefile detected does not match the current game version

Title says it all. I'm currently with the newest public build of the game (0.7.10), and the v0.18 save editor build. Any idea what happened?

For details: I updated the game a few days ago, as well as this save editor, and the save editor keeps loading the savefile from the previous game build, and not the current updated one.

Did the savefile come from an Android version?
Can you attach the savefile?
I use the editor with the latest backer version and it works just fine.

no, it's the pc version. any ideas?

Can you link the save file?

which one?
the old version or the new version from the editor?

The savefile that the editor cannot load.

Faedir - 1791 days.zip
There.

brokenthing.zip
There.

You won't be able to load that file directly in the save editor.
But once I "re-saved" as a slot file (.sol) I was able to load it in the save editor ok.

But I can't even open the file :(

Right, you have to save from the game to a save slot.
Are you saying you can't save to a slot?
That would be a bug in the game or a problem with the browser.
Have you tried the stand alone flash projector?

So I'm supposed to save the file to a slot file. Yes, it appears I can only save as .tits directly from the game. I'll try the flash projector first.

And it worked. Thanks!


Documentation: SwiftUI -> User Interface Element -> Text input and output: Text Editor

As a PO I'd like to have documentation for the TextEditor

@shadowfacts reopened since PR was not merged. Could you please merge the PR?


No Database connection from Ubuntu 14.04 LTS to SQL Server 2008 R2

Steps to reproduce
Unfortunatly I don't know if this occurs only at my setup
Please also see my post: http://stackoverflow.com/questions/40150357/asp-net-core-web-api-on-ubuntu-14-04-lts-doesnt-get-sql-connection-to-mssql-200
The issue
Unable to get an connection from an ASP.NET Core Web API on Ubuntu 14.04 LTS to an MSSQL Server 2008 R2 running on a Microsoft Windows Server 2008 R2
System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.AggregateException: One or more errors occurred. (No such device or address) ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
   at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Data.SqlClient.SNI.SNITCPHandle.d__22.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
   at System.Data.SqlClient.SNI.SNITCPHandle..ctor(String serverName, Int32 port, Int64 timerExpire, Object callbackObject, Boolean parallel)
   --- End of inner exception stack trace ---
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open()
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.QueryMethodProvider.d__3`1.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.d__15`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteObject(TextWriter writer, Object value)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ApplicationInsights.AspNetCore.ExceptionTrackingMiddleware.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ApplicationInsights.AspNetCore.RequestTrackingMiddleware.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.d__2.MoveNext()

Further technical details
EF Core version: (found in project.json or packages.config)
Operating system:  Ubuntu 14.04 LTS
Visual Studio version: Visual Studio 2015
Other details about my project setup:
{
  "dependencies": {
    "Microsoft.AspNetCore.Server.IISIntegration": "1.0.0",
    "Microsoft.Extensions.Configuration.EnvironmentVariables": "1.0.0",
    "Microsoft.Extensions.Configuration.FileExtensions": "1.0.0",
    "Microsoft.Extensions.Configuration.Json": "1.0.0",
    "Microsoft.Extensions.Logging": "1.0.0",
    "Microsoft.Extensions.Logging.Console": "1.0.0",
    "Microsoft.Extensions.Logging.Debug": "1.0.0",
    "Microsoft.Extensions.Options.ConfigurationExtensions": "1.0.0",
    "Microsoft.EntityFrameworkCore.Tools": "1.0.0-preview2-final",
    "Swashbuckle": "6.0.0-beta902",
    "NLog.Extensions.Logging": "1.0.0-*",
    "Microsoft.AspNetCore.StaticFiles": "1.0.0",
    "Microsoft.ApplicationInsights.AspNetCore": "1.0.2",
    "Microsoft.AspNetCore.Mvc": "1.0.1",
    "Microsoft.AspNetCore.Server.Kestrel": "1.0.1",
    "Microsoft.EntityFrameworkCore.SqlServer": "1.0.1",
    "Microsoft.EntityFrameworkCore.SqlServer.Design": "1.0.1",
    "Microsoft.NETCore.App": "1.0.1"
  },

  "runtimes": {
    "win7-x64": {},
    "win10-x64": {},
    "ubuntu.14.04-x64": {}
  },

  "tools": {
    "Microsoft.EntityFrameworkCore.Tools": "1.0.0-preview2-final",
    "Microsoft.AspNetCore.Razor.Tools": "1.0.0-preview2-final",
    "Microsoft.AspNetCore.Server.IISIntegration.Tools": "1.0.0-preview2-final"
  },

  "frameworks": {
    "netcoreapp1.0": {
      "imports": [
        "dotnet5.6",
        "portable-net45+win8"
      ]
    }
  },

  "buildOptions": {
    "emitEntryPoint": true,
    "preserveCompilationContext": true,
    "xmlDoc": true
  },

  "runtimeOptions": {
    "configProperties": {
      "System.GC.Server": true
    }
  },

  "publishOptions": {
    "include": [
      "wwwroot",
      "Views",
      "Areas/**/Views",
      "appsettings.json",
      "web.config",
      "nlog.config"
    ]
  },

  "scripts": {
    "postpublish": [ "dotnet publish-iis --publish-folder %publish:OutputPath% --framework %publish:FullTargetFramework%" ]
  }
}

Updated information to name the correct serer version:
running a Windows Server 2008 R2 (unpatched) with a SQL Server 2008 R2 (Service Pack 3)

Are you using the latest SQL 2008 Servicepack?

Likely this is due to an issue with solving named instances in SqlClient: https://github.com/dotnet/corefx/issues/10920.

No, i am not using servicepack 3. Actually the server is completly unpatched.
Since this issue was linked to another issue stating the problem are named instances, could you please add some information on how to solve this issue  or what actually the problem is.
The linked issue is the first found on google but does not provide any information on the problem nor the solution.
If there are any patches required for the MSSQL server, would you mind to list it? The server is in an offline (not connected to the internet) environment and I will have to download and copy the patches by hand in order to install them.

The current supported version of SQL Server 2008 is the latest Service Pack, as stated here: https://github.com/dotnet/corefx/issues/9439 - so you should update to SQL Server 2008 SP4

I was wrong in the server versions (missing the R2). I am using Windows Server 2008 R2 and SQL Server 2008 R2.
just to leave it here: I installed SQL Server 2008 R2 Service Pack 3 now but the problem remains.

@MontyGvMC Here are some other duplicates of https://github.com/dotnet/corefx/issues/10920 that contain more details and workarounds for the issue with named instance connection strings not working on Linux:
https://github.com/dotnet/corefx/issues/4346
https://github.com/dotnet/corefx/issues/8086#issuecomment-214819752
In general the workaround is to use Data Source=machine,port number instead of Data Source=machine\instance.
According to a comment in https://github.com/dotnet/corefx/issues/4346 is fixed. The bug is assigned to 1.2 but I think the fix went in 1.1. @saurabh500 or @geleems should be able to confirm.

Closing for now with the assumption that this is a duplicate of one of the SqlClient bugs mentioned above. If it isn't please create a new issue in https://github.com/dotnet/corefx/ with repro details.

I am not using instance names

I am not using instance names

Then as I suggested, please open a new issue in https://github.com/dotnet/corefx/ with your repro steps. There isn't much we can do here to help as you are not hitting a problem with EF Core.


tweak(ymap): dont limit ymap entities to drawables

theres no reason to limit ymap entities to only drawable objects, fragments exist, and also its much easier to work with unparented meshes (just a drawable model) if  you have many instances of this model, rather than have it parented to a drawable for it to export properly.

There is also the entity properties panel that is only shown for drawables. But I guess I can merge this for now.


[TwigComponent][Icons] Inconsistency between allowed attributes

Twig components allows to provide an integer as attibute values and casts it, while UX:Icon doesn't allow it.
Valid example:
<twig:Test a="1" b="{{ 2 }}">
  test
</twig:Test>

Currently invalid:
<twig:UX:Icon a="1" b="{{ 2 }}" />

I'm curious to know the real problem behind this (factual) observation ?
What did you want to do here you could not  ?

I was switching from custom icon component. In one twig component I had a loop where icon's are rendered and then binding {{ loop.index0 }} via data attribute and this is where the problem appear. Had to do {{ '' ~ loop.index0 }}.
So this was an expectation error, where you can write numbers to attributes via twig component and so <twig:UX:Icon /> looks like a twig component with some exceptions :)

Let's add this :)

Does this fix from @Kocal solve all your use cases ?
#2149

Yup