Null and empty array handling in parquet reader

Bug description
parquet-tools output is different than velox parquet reader output:
lniu@devrestricted-lniu:~/velox_parquet_test_triage/fail_parquet_files/testArray$ parquet-tools show native_parquet_reader_test2148048251466121645parquet
+-------------------+
| test              |
|-------------------|
| [ 1. nan  3.]     |
| [5.]              |
| [nan nan nan]     |
| []                |
| [ 7. 11.]         |      ---> difference starting here
| [nan]             |
| [13.]             |
| [17.]             |
| [ 1. nan  3.]     |
| [5.]              |
| [nan nan nan  7.] |
| [11. nan 13.]     |
+-------------------+

lniu@devrestricted-lniu:~/velox_parquet_test_triage/fail_parquet_files/testArray$ ~/velox_scan_parquet native_parquet_reader_test2148048251466121645parquet
number of rows: 12
velox type: ROW<test:ARRAY<INTEGER>>
{3 elements starting at 0 {1, null, 3}}
{1 elements starting at 3 {5}}
{3 elements starting at 4 {null, null, null}}
{<empty>}
{2 elements starting at 7 {null, 7}} ---> difference starting here
{1 elements starting at 9 {11}}
{1 elements starting at 10 {null}}
{1 elements starting at 11 {13}}
{3 elements starting at 12 {17, 1, null}}
{1 elements starting at 15 {3}}
{4 elements starting at 16 {5, null, null, null}}
{3 elements starting at 20 {7, 11, null}}

System information
Velox System Info v0.0.2
Commit: 1e186e548833750cdee4b95d829711ddad78aba1
CMake Version: 3.16.3
System: Linux-5.4.0-1063-aws
Arch: x86_64
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 9.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 9.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
Relevant logs
file to reproduce: https://www.dropbox.com/scl/fi/ln4d2tnoz497w309dr2qr/native_parquet_reader_test2148048251466121645parquet?rlkey=3ke9bkml3w35ij3g6gdymmz26&dl=0

file to reproduce: https://www.dropbox.com/scl/fi/ln4d2tnoz497w309dr2qr/native_parquet_reader_test2148048251466121645parquet?rlkey=3ke9bkml3w35ij3g6gdymmz26&dl=0

@yingsu00 @nmahadevuni

I am taking a look thanks!

@jaystarshot I'm working on the refactor of null handling. It should resolve this issue.

We encounter same issue when reading map, will keep tracking this issue.

Hi, I realized this has already been caught in existing test. I did an analysis and found the problem is empty collection related. whenever there is a empty array or map, the final result will be incorrect.
There is already a unit test capture the issue. I use the data https://github.com/facebookincubator/velox/blob/main/velox/dwio/parquet/tests/examples/array_2.parquet . the expect output is:
A array with type ARRAY<VARCHAR> containing 3 elements:

0: 2 elements starting at 0 {a, null}
1: <empty>
2: 2 elements starting at 2 {null, b}

array vector:

offset size  value
0      2     a
2      0     null
2      2     null
             b

the actual output:
size = 0
0: 2 elements starting at 0 {a, null}
1: <empty>
2: 2 elements starting at 2 {null, null}

array vector:

offset size  value
0      2     a
2      0     null
2      2     null
             null

parquet meta data:
lniu@lniu-FXGFKFV Downloads % parquet meta array_2.parquet

File path:  array_2.parquet
Created by: parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)
Properties:
  writer.model.name: example
Schema:
message spark_schema {
  required group _1 (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
}

Row group 0:  count: 3  20.00 B records  start: 4  total(compressed): 60 B total(uncompressed):46 B
--------------------------------------------------------------------------------
                 type      encodings count     avg size   nulls   min / max
_1.list.element  BINARY    G   _     5         12.00 B    3       "a" / "b"

the data in parquet format is:
def rep data
2   0   a
1   1   b
0   0
1   0
2   1

Current findings:

offset and size are right. RepeatedColumnReader::setLengthsFromRepDefs is correct.
on leaf level. pageReader.readPageDefLevels() which calls arrow::DefLevelsToBitmap() that makes leafNulls_ is  “0b10001” . this is incorrect because 3rd def is 0, which is less than max def - 1, means the value doesn’t exist at all. (value has 3 status: exist_value, exist_null, not_exist)
Given size info, ListColumnReader decided to read 4 elements from leafReader that becomes [’a’, null, null, null]

Possible Solutions:

Given we are using arrow library to get nulls and value_to_read information, is it possible we are not using it correct? if so, fix the usage can fix the problem.
don’t reply on arrow::DefLevelsToBitmap() when deciding leaf level data.
Depending on whether 1 or 2 alone can cover all cases (nested array containing empty, array over map containing empty), might need to rebuild whole readPageDefLevels and not use arrow library here.

@qqibrow Thanks for the findings. Yes NestedStructureDecoder::readOffsetsAndNulls() will replace Arrow's rep/def decoder and should correctly process this case. However, it has a known bug, and it requires more changes than just replacing the rep/def decoder. I have the fix to the known bug, but haven't gotten time to send PRs. Once the first Iceberg equality delete PR is out I'll start working on this, hopefully next week.

@yingsu00 is this known issue for map only? or also impact array/struct? We noted parquet parse issue on array

@yingsu00 is this known issue for map only? or also impact array/struct? We noted parquet parse issue on array

@FelixYBW This issue occurs with Array too

does it impact stringview? We hit a bug that a string should be null but it return as ""

I tried various parquet files with different iterations of empty and null.
Through various iterations, confirmed that the result mismatch happened in these cases.

and

and

Seems to be just an issue with the empty array as @qqibrow  mentioned and not related to null atm.
Maybe we are wrongly reading the define and repeatation level of empty ?

@hitarth @qqibrow and me debugged this issue. Seems like the current root cause is us not properly using arrows' DefLevelsToBitmap function properly as we were not updating arrow::LevelInfo's repeated_ancestor_def_level properly.
We found out how arrow populated   repeated_ancestor_def_level here which it apparently uses for empty checks.
So now we are trying to give arrow the correct repeated_ancestor_def_level. I have a wip here but this still needs some work

does it impact stringview? We hit a bug that a string should be null but it return as ""

forgot about it. It's fixed by https://github.com/facebookincubator/velox/pull/9129

Pr https://github.com/facebookincubator/velox/pull/9129 is ready for review


Unify scaled INT8 mm

Fixes #862

The triton kernel in torchao.kernel.intmm_triton is highly coupled to torchao.kernel.autotuner, which makes it unnecessary complex to separate them out. intmm_triton also has inference in mind, with exhaustive autotuning and cached config, which is not suitable for training. Closing this for now.


Private methods in derived class "hide" private methods in base class [moved]

This is Issue 227 moved from a Google Code project.
Added on 2013-07-26T00:25:53.000Z by<EMAIL_ADDRESS>Please review that bug for more context and additional comments, but update this bug.
Original labels: Type-Defect, Priority-Medium

Keith, you fixed this, right?

Just verified that this is fixed


Can we hide the 'powered by Google' image?

Probably didn't look hard enough. But I found a way of hiding it. Just add this prop to the component enablePoweredByContainer={false}


feat: Add Router-Authorization check

Aligns with the subgraph security docs I've drafted

Yeah, the goal is for it to be completely deployment-agnostic—every secrets management technique I've seen supports injecting via env vars. I could see a world where folks want to deploy their subgraphs behind a load balancer / API gateway and do the enforcement at that level, but we can't help them much in the template in those cases 😅

@patrick91 or @michael-watson, is there some alternative method we should use here (like a plugin), or can we merge this and copy over to the JavaScript template?


Add single frame encoder for type-gif

What's Changing and Why
There wasn't a encoder for image/gif so any attempt to create a buffer with mimetype image/gif would fail with the error Unsupported MIME type: image/gif
What else might be affected
Not sure how things hook up with the getBuffer method so could use some guidance here.  All the other encoder functions in the other types seemed synchronous.
Tasks

[X] Add tests
[ ] Update Documentation
[N/A] Update jimp.d.ts
[X] Add SemVer Label

This is looking good. get the lint passing and I'll merge 🚀

Fixed linting @hipstersmoothie

The version for this project is automated through GitHub labels using https://github.com/intuit/auto. Thanks for thinking about it though!

:rocket: PR was released in v0.13.0 :rocket:


Publish engine as an npm package

So that the upcoming editor can declare it as a dependency.
Also allows us to separate Lätkäzombit into its own project in a way that LZ can declare the engine as a dependency.

Done: https://www.npmjs.com/package/@kiigame/adventure_engine

Was not successful, build files did not appear in the package.

Fixed in 0.4.1


Samples for data services like Rabbit MQ and Kafka.

The sample service broker is as below.
https://docs.cloudfoundry.org/services/managing-service-brokers.html
In this case, if a user creates a service instance, it should create or install the service and provide access to it.

Hi , pl help me to understand the issue. Do you need samples for Rabbit MQ and Kafka working with Spring Open Service broker ?

Can you assign this issue to me?


Fixes transposed params in intergrating_vms guide

The istoctl register command had the port and IP fields
transposed.   This fixes that and closes:
https://github.com/istio/istio.github.io/issues/691

Jenkins job istio.github.io/presubmit passed


Move to Circle 2.0

What problem does this feature solve?
At the moment we're using Circle 1.0, which isn't supported from August 2018
What does the proposed API look like?
We should move to circle 2.0, and parallelize the compatability tests—https://github.com/vuejs/vue-test-utils/blob/dev/scripts/test-compat.sh

Resources and guides to help you make the transition https://circleci.com/sunset1-0/

We're using circle 2 now, but the tests aren't paralellized—https://github.com/vuejs/vue-test-utils/blob/dev/.circleci/config.yml


feat(onboarding): require explicit external connection configuration

Currently when the app starts up we detect wether the user is running lnd locally or not, and if we detect that they are we connect zap to that automatically.
This is confusing from a user perspective as they have no idea what happened, or what zap has just connected itself to or why the behaviour of zap is different from another time when they ran it when they didn't have lnd already running on their localhost already.
This feature predates the custom connection screen, and I think conflicts with it. The user should just have two very easy to understand options

default mode runs lnd from resources dir with our own configs (neutrino mode, connected to the zap fullnode backend)
custom mode allows you to choose your connection params explicitly. It should provide the ability top use sensible default values for a localhost connection, and default to the last used settings.

In this PR I remove the lnd process lookup, and add some links to the custom connection settings onboarding screen that allow a user to populate the field with suggested values (those for a localhost based connection). See attached screenshots:
initial form as a new user sees it:

hover over the 'use suggested values' link:

first field populated with suggested value:

all fields populated wiuth suggested values:

I worry this would represent a regression in user experience by requiring knowledge / interaction with values which can be determined automatically. This is more of an advanced user interface imo, than one that should be presented to everyone.

The custom connection screen is for advanced users only - it specifically says that on the screen where users are given the option to use default or custom mode.
My thought process is that at least this way the user has a chance to understand what zap is going to do. It's unpredictable at the moment, and I think the number of people asking for help recently in slack is testament to that fact.
Just because a user might have lnd running on their local machine doesn't mean that they want to connect zap to it.

@JimmyMow there are 2 aspects to this PR.
I think that the first aspect is a must - removing thing thing where we detect if the user is already running lnd on their host and try to connect to it automatically if so.
The second part is to add the suggested value feature into the custom connection onboarding screen, which I'm in two minds about. One the one hand I sort of question why we are suggesting values for localhost over anything else. On the other hand it is possible that some people may want to connect to lnd on their localhost and this potentially makes it easier for them.

Pull Request Test Coverage Report for Build 1140

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 34.675%

Totals

Change from base Build 1138:
0.0%

Covered Lines:
357

Relevant Lines:
897

💛  - Coveralls

I have pulled the second part of this out into a separate PR: https://github.com/LN-Zap/zap-desktop/pull/640
This PR now only deals with removing the automajical behaviour of connecting to an existing lnd instance if we happen to find one.

Tested ACK b54fd26

Lol automajical 😆


E-Commerce Module

#9 mentions e-commerce. This is a ticket to discuss and implement e-commerce support in Beubo along with a way to support payment gateway modules. The addition of sub modules such as payment gateways may need to be considered in #4.

I think you should separate the e-commerce by modules, like a catalogue, documents (quotes, orders), payments, shipping, etc.
I would like to help.

I think you should separate the e-commerce by modules, like a catalogue, documents (quotes, orders), payments, shipping, etc.
I would like to help.

I think you should separate the e-commerce by modules, like a catalogue, documents (quotes, orders), payments, shipping, etc.

I like that idea. Currently I'm trying to figure out the best way to handle plugins. I don't think the current implementation for plugins which supports a few methods using the go plugin package is a good option for an e-commerce module. I think RPC or gRPC may be the best alternative here using something like https://github.com/hashicorp/go-plugin
So there are a few steps left before I think we can start on an e-commerce module.

I would like to help.

I appreciate any help I can get. I need to add a contributing file but basically I'm happy to add any code that improves Beubo and uses the same license (MIT). An these modules you proposed here may be better placed in their own repositories but I'm open to suggestions here. You can message me directly on discord if you would like to discuss something directly, uberswe#2533 is my username
Thank you!

I think you should separate the e-commerce by modules, like a catalogue, documents (quotes, orders), payments, shipping, etc.

I like that idea. Currently I'm trying to figure out the best way to handle plugins. I don't think the current implementation for plugins which supports a few methods using the go plugin package is a good option for an e-commerce module. I think RPC or gRPC may be the best alternative here using something like https://github.com/hashicorp/go-plugin
So there are a few steps left before I think we can start on an e-commerce module.

I would like to help.

I appreciate any help I can get. I need to add a contributing file but basically I'm happy to add any code that improves Beubo and uses the same license (MIT). An these modules you proposed here may be better placed in their own repositories but I'm open to suggestions here. You can message me directly on discord if you would like to discuss something directly, uberswe#2533 is my username
Thank you!

I have now added a contributing.md file in #43

I have now added a contributing.md file in #43


Replace src with raw to fix image paths

Fixes a problem occurred in #4376 .

Codecov Report

Merging #4377 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master    #4377   +/-   ##
=======================================
  Coverage   20.06%   20.06%
=======================================
  Files         153      153
  Lines       30766    30766
=======================================
  Hits         6174     6174
  Misses      23649    23649
  Partials      943      943

Impacted Files
Coverage Δ

modules/markup/markdown/markdown.go
64.48% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bc8782...01aaa2a. Read the comment docs.

LGTM
@JonasFranzDEV When this gets merged can you backport to 1.5?

@techknowlogick sure

@JonasFranzDEV please send a backport PR.


Force update support for mqtt binary sensor

Description:
Adds 'force_update' option support to the mqttbinary sensor. All the changes made are based on force_update support for mqtt sensor https://github.com/home-assistant/home-assistant/pull/6492
Pull request in home-assistant.github.io with documentation (if applicable): https://github.com/home-assistant/home-assistant.github.io/pull/4558
Example entry for configuration.yaml (if applicable):
binary_sensor:
  - platform: mqtt
    state_topic: "home-assistant/window/contact"
    force_update: true

Checklist:

[x] The code change is tested and works locally.

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.github.io

If the code does not interact with devices:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[x] Tests have been added to verify that the new code works.

@dgomes Not sure what you mean about sensor. It already has force update. As for adding to other mqtt components, this was my first contribution so I'm still getting my feet wet.

My point is why not making it part of mqtt platform (thus all components benefit form it, instead of replicating the behaviour)
Similar to what has been done for MqttAvailability


[FEAT] Edit option in answers

add here edit option with format for tables and coding
Something like this:

Why do you need to edit the responses?

For copy and paste format, coding, tables, markdown, etc...

You can copy, paste and edit them elsewhere. I can get the meaning to edit directly in responses. The responses should be exact as the LLMs outputted.


Sort comments by votes, age and reputation

There is no option to sort comments on the new version. It just shows the trending comments.

I could not reproduce this issue I checked code block when filter is change all data goes empty therefor its not able to do that @feruzm @umair1209 Could you please re-test again thanx

@ue presently users can only filter by Trending but other options need to be provided: Filter by

trending (vote weight)
age (time of posting)
reputation (users reputation)

This is possible, other UI has it

@ue Surfer has this, please check it's source code

I confused that was another Issue that was a misunderstanding. I didn't say its not possible or something. Thanx Feruz

Could you please test and close if it okay.
Android Apk Download link: https://install.appcenter.ms/orgs/esteem.app/apps/esteem/distribution_groups/beta_testers
IOS Ipa Download link:
https://install.appcenter.ms/orgs/esteem.app/apps/esteem-1/distribution_groups/beta_testers
Thank you. @umair1209


Send + Sync bounds on the Error type seem to be unnecessary

I started building a small project using this crate and quickly found that the bounds on the return type from TunBuilder::try_build were slowly spreading throughout my project so I dove in to see why it was necessary.
I took Send + Sync from Error here: https://github.com/yaa110/tokio-tun/blob/master/src/result.rs#L2 and the crate builds fine. Is there a particular reason to require these traits?
My main issue is that it cannot be trivially converted to and from std::result::Result<T, Box<dyn std::error::Error>> so it ends up spreading throughout.
If I try to return it in a function that returns Result<(), Box<dyn std::error::Error>>, I get this error:
error[E0277]: the size for values of type `dyn std::error::Error + Sync + std::marker::Send` cannot be known at compilation time
  --> src/bin/main.rs:11:45
   |
11 |     let mut netty = NettyStack::new(if_name)?;
   |                                             ^ doesn't have a size known at compile-time
   |
   = help: the trait `Sized` is not implemented for `dyn std::error::Error + Sync + std::marker::Send`
   = note: required because of the requirements on the impl of `std::error::Error` for `Box<dyn std::error::Error + Sync + std::marker::Send>`
   = note: required because of the requirements on the impl of `From<Box<dyn std::error::Error + Sync + std::marker::Send>>` for `Box<dyn std::error::Error>`
   = note: required because of the requirements on the impl of `FromResidual<Result<Infallible, Box<dyn std::error::Error + Sync + std::marker::Send>>>` for `Result<(), Box<dyn std::error::Error>>`

If I try to return Result<T, Box<dyn std::error::Error>> in a function that returns Result<T, Box<dyn std::error::Error + Send + Sync>>:
error[E0277]: the size for values of type `dyn std::error::Error` cannot be known at compilation time
  --> src/bin/main.rs:15:74
   |
15 |     add_address(if_name, Ipv4Addr::new(10, 0, 0, 1).into(), handle).await?;
   |                                                                          ^ doesn't have a size known at compile-time
   |
   = help: the trait `Sized` is not implemented for `dyn std::error::Error`
   = note: required because of the requirements on the impl of `std::error::Error` for `Box<dyn std::error::Error>`
   = note: required because of the requirements on the impl of `From<Box<dyn std::error::Error>>` for `Box<dyn std::error::Error + Sync + std::marker::Send>`
   = note: required because of the requirements on the impl of `FromResidual<Result<Infallible, Box<dyn std::error::Error>>>` for `Result<(), Box<dyn std::error::Error + Sync + std::marker::Send>>`

Hi, fixed in 0.7.0


boot.cleanTmpDir=true removes files in /tmp too late

See the following:
# systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/nix/store/6j1c354bzz70ylbh4cd0c3x165ic6hh2-systemd-212/example/systemd/system/systemd-tmpfiles-setup.service)
   Active: active (exited) since Tue 2014-11-04 04:04:08 CET; 19min ago

# systemctl status postgresql.service
● postgresql.service - PostgreSQL Server
   Loaded: loaded (/nix/store/q5rk80hmxy57ndl4zh0lq50bw3bcwza8-unit-postgresql.service/postgresql.service)
   Active: active (running) since Tue 2014-11-04 04:04:06 CET; 20min ago

As you can see, PostgreSQL was started before systemd-tmpfiles-setup.service was finished.
The result was that PostgreSQL's Unix socket created in /tmp/.s.PGSQL.5432 got deleted and clients are not able to connect to it anymore.

How should this be fixed?

I suppose the patch is now at https://github.com/NixOS/systemd/commit/5f47c0ee21e46975115a885e666d2cb68f3e32e7
From what I've read, systemd doesn't really allow services to run at shutdown only. One thing we could do, however, is call it in ExecStop instead, which has the same effect. Would that be an acceptable workaround?

This also likely causes https://github.com/NixOS/nixpkgs/issues/12132#issuecomment-171284532, see also https://github.com/NixOS/nixpkgs/commit/b292e19fbddaddd4ada813965e3be3880f15fa7e (I've had no problems since I've disabled boot.cleanTmpDir). I want to look at this issue later, implementing a simple oneshot service that cleans up /tmp on ExecStop, if noone does that before me. @edolstra, would that be an acceptable fix for the problem?

Probably the best fix is to drop that patch (https://github.com/NixOS/systemd/commit/5f47c0ee21e46975115a885e666d2cb68f3e32e7) and find some other way to fix the nixops send-keys issue (assuming it even exists anymore).

Whom can I ping for nixops knowledge? I noticed /tmp had grown over the top today and also got a report from another victim of cleanTmpDir.

@edolstra: you self-assigned – are you (still) working on this?

Not right now.

If we don't make it to 16.09, maybe we want to remove this option or add a warning somewhere?

@domenkozar is it better to make a change in documentation to spell a big warning, or just remove the option?

@abbradar I would prefer this option not to be removed, since I am using it and I have no plans to use nixops send-keys.
To prevent this bug, I am adding the following config options to configuration.nix:
{
    systemd.services = {
      systemd-tmpfiles-setup.before = [ "sysinit.target" ];
      systemd-update-utmp.after = [ "systemd-tmpfiles-setup.service" ];
    };
}

@wizeman Awesome; maybe we want to add this conditionally when boot.cleanTmpDir is enabled to workaround the situation?

On the second thought, we can just move from systemd-tmpfiles; this should work always.

I have a draft of a possible fix here https://github.com/rvl/nixpkgs/commit/70383fc3d18dd3247e431ae15744dbca6021ce53 but I think we all agree it would be preferable to fix NixOps so that the NixOS systemd patch can be removed.
However I can't reproduce the problem. Here is my deployment:
{
  network.description = "Test #4825";

  testserver =
    { config, pkgs, ... }:
    {
      deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 1024; # megabytes
      deployment.virtualbox.headless = true;
      virtualisation.virtualbox.guest.enable = true;

      # deploy the test key
      deployment.keys.secret.text = "secret";

      # uses systemd-tmpfiles
      boot.cleanTmpDir = true;

      # reverses nixos patch to systemd
      systemd.services = {
        systemd-tmpfiles-setup.before = [ "sysinit.target" ];
        systemd-update-utmp.after = [ "systemd-tmpfiles-setup.service" ];
      };
    };
}

After deploying this and ssh'ing into testserver, I see that the test key is correctly installed. What would I need to do to trigger the nixops send-keys problem? @edolstra

I think the problem occurs when you use the send-keys feature to send a LUKS key for an encrypted filesystem. Then you get a cycle in the unit dependency graph (since sshd depends on local-fs.target via sysinit.target, but the filesystems depend on sshd via keys.target).
However, we can probably just drop the systemd patch and if necessary relax the sshd dependencies in NixOps if any keys are specified.

Is this issue still relevant?

AFAICS the buggy systemd patch is still present in NixOS 19.03 but it doesn't seem to be present in the master branch, so it should be fixed in master but not in the stable branch.
That said, I don't know how the nixops send-keys feature

The patch seems to have been dropped in the 239 -> 242 systemd upgrade on purpose and it seems the nixops autoLuks option doesn't currently work on master. See issues #47550 and #62211.
Unless we are considering backporting the fix, it's probably safe to close this issue.

Awesome, I can finally drop my out of tree patches for this :3 I don't think we should backport this, people may rely on autoLuks on stable.


Editing a custom ArrayMesh from the inspector steals mouse control and interrupts the edit

Tested versions
Reproducible in 4.3.stable and 4.4.dev5
System information
Godot v4.4.dev5 - Windows 10.0.22631 - Multi-window, 1 monitor - OpenGL 3 (Compatibility) - NVIDIA GeForce GTX 1660 SUPER (NVIDIA; <IP_ADDRESS>36) - Intel(R) Core(TM) i7-10700F CPU @ 2.90GHz (16 threads)
Issue description
When using the mouse to drag a value for a custom ArrayMesh, and the custom mesh's script uses  ArrayMesh.add_surface_from_arrays to regenerate the mesh, it causes the mouse to lose grip of whatever control it was interacting with, and snap to the middle of the screen.
The use case here is implementing a custom, parametric mesh type. This bug makes it impossible to tweak the mesh with the mouse. You have to manually type in values.
Notes:

This bug also happens if you use something like MeshDataTools or SurfaceTool to modify the mesh.
This bug does not happen if you use something like ArrayMesh.surface_update_vertex_region, but that does not perform all the necessary operations of generating a mesh (e.g. adding new vertices, triggering an AABB recalculation, etc.).

Steps to reproduce

Open the attached project/scene.
Select the triangle node, and inspect its mesh, which has a custom_mesh.gd script attached.
Use the mouse to edit its Size property.
Observe that the value only changes by a small amount before the cursor is stolen away.

Minimal reproduction project (MRP)
bug-report.zip

The reason drag does not work here is because you are replacing the entire mesh and redrawing the full resource inspector. So basically what you try to drag you remove. You just don't see it because it gets replaced so quickly.
When you use the ArrayMesh.surface_update_vertex_region you are actually only updating part of the existing Mesh which is why it works.

The reason drag does not work here is because you are replacing the entire mesh and redrawing the full resource inspector. So basically what you try to drag you remove. You just don't see it because it gets replaced so quickly.
When you use the ArrayMesh.surface_update_vertex_region you are actually only updating data parts of the existing Mesh which is why it works.
You would need to use a custom inspector that can keep your properties and does not target the resource directly. You can't use the generic resource inspector when the resource gets replaced on changes. E.g. Skeleton3D and others all use custom inspectors for this very reason. This is an older issue that I have seen before around here so there are some duplicates to find.

I'm not understanding why recreating a surface in an ArrayMesh would replace the entire resource. Why is that necessary?
Also not understanding the Skeleton3D comparison because that's a node, not a resource.

UPDATE: Replacing the calls with the RenderingServer equivalents does not cause this bug! 😲
RenderingServer.mesh_clear(get_rid())
RenderingServer.mesh_add_surface_from_arrays(get_rid(), RenderingServer.PRIMITIVE_TRIANGLES, arrays)

UPDATE 2:
It seems like the culprit is notify_property_list_changed() being called in ArrayMesh's native methods.
So this is a general issue of mouse control being taken away whenever the inspector has to be rebuilt as a consequence of notify_property_list_changed(). As such, I think this bug can be closed. I'm not sure if this is fully avoidable, but at least there's the workaround of using the RenderingServer.

To add to this yes notify_property_list_changed() causes a redraw of the resource inspector and since a mesh has a lot of "dynamic" properties for the surfaces a new drawn resource inspector has no track of the old slider that was e.g. dragged so the focus and input gets lost.
Skeleton3D was just one example for a custom inspector and why a simple resource inspector often does not work for more complex things. E.g. something touches a bone transform and the entire skeleton inspector would need to redraw. That would be even worse than this issue here with the ArrayMesh.


Catch non existent directories before starting generation

Feature Request
Description of Problem:
Currently if a file path point to a file in a directory which does not exist the generationConfigValidator returns valid and generation is allowed to begin whereupon a NoSuchFileException is thrown.
Potential Solutions:

Add a check into the generationConfigValidator to check whether or not the directory in which the desired output file is located exists.
Potentially tell the user which folder could not be found (the directory path may not exist from an earlier point:
e.g. C:\foo\bar\foobar\outputFile.csv --> Invalid Output - directory  C:\foo\bar does not exist)

Acceptance Criteria:

[ ] Invalid file paths are discovered before generateDataSet is called

I think a message to tell the user so such folder can be found is a good idea


Allow user to specify how many days in the future to generate payoff diagram.

Added exp_adjust parameter to support graphing future date payoff diagram.

Added exp_adjust_date as input parameter and conversion to days to adjust.


[0.8] Listening to events on an element produces different results under ShadowDOM v. ShadyDOM

This is due to event retargeting. We'll need an api (perhaps part of Polymer.dom) to normalize the behavior. It should probably include both the target as seen in ShadyDOM (event.path[0]) and a normalized event.path.

Fixed via https://github.com/Polymer/polymer/commit/739de807d553d6d7885edd093d1b2b8176196322.


Adding new IdentityRole without NormalizedName fails subsequent lookups

I recently hit an issue whereby adding a new role to the system using the single parameter constructor:
_context.Roles.Add(new IdentityRole(“SYSADMIN”));
Would insert the record into the DB, but subsequent calls to look up this role, either by:
_roleManager.FindByNameAsync(“SYSADMIN”)
or indeed
_userManager.AddToRoleAsync(adminUser, “SYSADMIN”);
would both fail.
It appears that adding a role with a single string parameter populates the name field but leaves the NormalizedName field as null. However, it would seem NormalizedName is what is used to perform the lookup so the inserted record is never found thereafter.
I worked around it by using:
_context.Roles.Add(new IdentityRole("SYSADMIN")
{
  NormalizedName = "SYSADMIN"
});

Shouldn't
_context.Roles.Add(new IdentityRole(“SYSADMIN”));
Populate both Name and NormalizedName, or is there a use-case where you'd want Name to be populated but NormalizedName to be blank?

You should use the identity APIs to create the role, roleManager.CreateAsync()


Stylus and CSS customization

Now it's not possible to override stylus variables for customization.
The Calendar stylus vars should be declared with a ?. E.g.: $day-label ?= 24px
This is useful for reducing the css footprint.
Would be also nice if instead of $primary, etc, use var(--q-color-primary) and if the color for borders, backgrounds, etc, are in their own vars. E.g.: $calendarBorderColor ?= $grey-4

Thanks for the info! I'll be looking into it. :heart:

@HenriqueSPin
beta.1 is released. I changed to the ?= for the variables. I'll be updating the rest in a future release, but going to close this for now.


Add ability to inline edit translations

If the backend API allows it, you can provide an updateTranslation endpoint to the translateWidget init parameters. If that is the case, there's an advanced feature that gets enabled, that will show an edit pencil over translated chunks. When clicking on it, an edit <textarea> will show up where the translation can be edited and saved. This is how it works:
When the translation feature kicks in, you get the content translated (As normal):

While holding Ctrl and hovering over it, you will get the edit pencil with a border showing what is going to be edited:

After clicking the edit pencil, you get a <textarea> with the translation to edit:

Finally, clicking Save will send the updated translation to the backend endpoint, and update the element:

This is fantastic! I've also used this project in a cached scenario and this feature would be phenomenal!

Would you be willing to update openapi.yaml and README.md with how to use this new feature?


[Question] Is this an expected behavior for soundex lk82? [ไทย, ไท, ทัย, ทาย]

Question
I'm using the pythainlp.soundex.soundex() with lk82 and found some confusion that the ภาษาไทย is not equal to ภาษาทัย, while ไทย is equal to ทัย.
The ไทย, ไท and ทัย give us ท7000, while the ภาษาไทย give us พ9397, but ภาษาทัย give us พ9393. Is this an expected behavior?
I understand that the ภาษาไทย and ภาษาทัย should give us the same result as  พ9397. Could you please help to advise?
To Reproduce
Steps to reproduce the behavior:

from pythainlp.soundex import soundex
soundex("ไทย","lk82"), soundex("ไท","lk82"),  soundex("ทัย","lk82"), soundex("ทาย","lk82")

result = ('ท7000', 'ท7000', 'ท7000', 'ท9700')

soundex("ภาษาไทย","lk82"), soundex("ภาษาไท","lk82"),  soundex("ภาษาทัย","lk82"), soundex("ภาษาทาย","lk82")

result = ('พ9397', 'พ9397', 'พ9393', 'พ9393')

Desktop :

OS: Windows 10 Home 1903
Python Version

Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]

Version pythainlp==2.0.7

It works as expected in the paper.
It detects if the first char is consonant, and switch with the second character if not.
So, ไทย -> ทไย, which is similar to ทัย. But if it is not the first char. There is no switching.(though you could implement it on your own if you like).


Error deerma.humidifier.jsq5

I have been getting this error for some time, why ?
`Cette erreur provient d'une intégration personnalisée
Logger: custom_components.xiaomi_miot.humidifier
Source: custom_components/xiaomi_miot/init.py:1242
Integration: Xiaomi Miot Auto (documentation, issues)
First occurred: 17:47:00 (1 occurrences)
Last logged: 17:47:00
Antibacterial Humidifier(deerma.humidifier.jsq5): Got MiioException while fetching the state: {'code': -9999, 'message': 'user ack timeout'}, mapping: {'humidifier.on': {'siid': 2, 'piid': 1}, 'humidifier.fault': {'siid': 2, 'piid': 2}, 'humidifier.fan_level': {'siid': 2, 'piid': 5}, 'humidifier.target_humidity': {'siid': 2, 'piid': 6}, 'indicator_light.on': {'siid': 6, 'piid': 1}, 'alarm': {'siid': 5, 'piid': 1}, 'environment.relative_humidity': {'siid': 3, 'piid': 1}, 'environment.temperature': {'siid': 3, 'piid': 7}}, max_properties: 4/8`

Try to add this custom attribute and observe again:
# customize.yaml
humidifier.deerma_jsq5_entity_id:
  chunk_properties: 1
  interval_seconds: 90
  exclude_miot_services: alarm,custom,indicator_light

Try to add this custom attribute and observe again:
# customize.yaml
humidifier.deerma_jsq5_entity_id:
  chunk_properties: 1
  interval_seconds: 90
  exclude_miot_services: alarm,custom,indicator_light

Do you need to create this file and put it in the config directory ?

# configuration.yaml
homeassistant:
  customize: !include customize.yaml

# configuration.yaml
homeassistant:
  customize: !include customize.yaml

I made the changes as you suggested but the error is still there.  Do you have another idea ?

No other idea.

No other idea.

I will delete your integration, and warn that bugs remain unchanged and will propose to delete this integration, there will still be 2 functional and updated without bugs.


ci: update to actions/checkout@v3

github will be deprecating node12-based actions by summer of 2023. checkout@v3 is developed for node16.
https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
Signed-off-by: Tim Whisonant<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
7 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.03%) to 66.025%

Files with Coverage Reduction
New Missed Lines
%

libraries/plugins/xfpga/open.c
1
74.68%

libraries/plugins/xfpga/mmio.c
6
82.87%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
13065

Relevant Lines:
19788

💛  - Coveralls


Make sure open existential ref only takes objects.

We already have this assert in the verifier, but by placing the assert here we
catch the error right where the error occurs instead of later. This just lowers
triage time for such errors.

@swift-ci smoke test and merge


start development of nutils v5

Coverage remained the same at 87.301% when pulling e059f1416440180c1831b93d9a372c50b3bf236f on joostvanzwieten:bump-v5-dev into 5ec1d84317c27d1e2a0cb7c1dcb97aa5df8027cb on nutils:master.


Minor documentaion added for pre-2.0.0 mpi_caf.c

Current coverage is 52.21% (diff: 100%)

Merging #315 into master will increase coverage by 6.87%

@@             master       #315   diff @@
==========================================
  Files             3          3
  Lines          1019       1578   +559
  Methods          64        101    +37
  Messages          0          0
  Branches        193        369   +176
==========================================
+ Hits            462        824   +362
- Misses          479        636   +157
- Partials         78        118    +40

Powered by Codecov. Last update 4575ca1...1f02c3c

LGTM

Please don't close yet, I intent to add more documentation.

@vehre great! Just push commits to the comment-caf-token token branch. I'll keep this open until you're ready.

Oops.  @vehre please ignore my "Closing now." comment.  I'll leave this to you to close when you're ready.

LGTM


Reversed "up" and "down" event direction

Hit an issue when trying to detect a panning element's collision with the sides of the window/an outer element.
Given:

a set of booleans for whether the element is colliding? and with which boundaries t? b? l? r?
a symbol d for e.detail.local.direction

(if (or (not colliding?)
        (and colliding?
            (or (and t? (= d "down"))
                (and b? (= d "up"))
                (and l? (= d "right"))
                (and r? (= d "left")))))
  (translate-according-to-global-deltas...) ; then
  (log-to-console) ; else

The checks for right and left would work, but up and down are reversed. Code is cljs but behavior is the same in js. After checking my code I realized it was in the actual event object.

Devtools output for e.detail
{contact: Contact, recognizer: Pan, global: {…}, live: {…}}
  contact: Contact {options: {…}, DEBUG: false, id:<PHONE_NUMBER>370, pointerInputs: {…}, activePointerInputs: {…}, …}
  global:
    deltaX: 0
    deltaY: -37.78631591796875
    direction: "down"
    distance: 37.78631591796875
    rotation: 0
    scale: 1
    speed: 0.3235129787992604
    speedX: 0
    speedY: -0.3235129787992604
    srcEvent: PointerEvent {isTrusted: true, pointerId: 1, width: 1, height: 1, pressure: 0.5, …}
  live:
    center: {x:<PHONE_NUMBER>671875, y:<PHONE_NUMBER>2421875}
    deltaX: 0
    deltaY: -30.96844482421875
    direction: "down"
    distance: 30.96844482421875
    rotation: 0
    scale: 1
    speed:<PHONE_NUMBER>421875
    speedX: NaN
    speedY: NaN
    srcEvent: PointerEvent {isTrusted: true, pointerId: 1, width: 1, height: 1, pressure: 0.5, …}

If deltaY is negative, the element has moved up.

See the source where direction is calculated..
https://github.com/biodiv/contactjs/blob/dbbb27b5cc5791857010532112ffbcea996a5501/src/contact.js#L721-L728

Either the directions or the start/endpoint should be reversed. 😎

Thanks for reporting, and thanks for including logs and code. The vectors used in contact.js are calculated using the "tip of the arrow minus foot of the arrow" rule. An upward vector would look like this:
^  (Tip)
|
|
|
x  (Foot)
So tip minus foot would be a positive number for upward arrows and a negative number for downward arrows (y-coordinates, deltaY). Can you confirm, that actually panning upwards results in negative deltaY values in your case? You could also try the test/test.htmlof this repository if you get correct values there when panning.
Have you somehow made sure, that the outer element (according to the DOM tree) fully contains the inner element? Given this layout:

it still is possible, that one or more boundaries of inner are outside of outer, if it is positioned/sized using css. If the inner element is not fully contained, it is possible to e.g. pan up and get a collision at the bottom.

Oh thank you for replying. Don't put it past me to be confused! Sorry if I'm not getting it.
I'm under the impression tip minus foot in the case you illustrated would be negative, because the (positionally) lower pixel values on an HTML page are greater, the opposite of a regular coord plane.
I added some quick output to the test canvas with values straight from the event:

in Test.js > showOutput
document.getElementById("deltaX-global").textContent = event.detail.global.deltaX
document.getElementById("deltaY-global").textContent = event.detail.global.deltaY
document.getElementById("direction-global").textContent = event.detail.global.direction
document.getElementById("deltaX-live").textContent = event.detail.live.deltaX
document.getElementById("deltaY-live").textContent = event.detail.live.deltaY
document.getElementById("direction-live").textContent = event.detail.live.direction

Here's a recording of what I got. I pushed the page to the fork I'd made if you want to see.

If my mindset is completely wrong thanks for letting me take your time regardless, ha, feel free close the pr.
It's a fantastic library I'm having fun with it as a replacement to hammer.js.

You got it right :) The pixels increase downwards on the y-axis, so "up" and "down" direction have to be reversed as you described.


Group Triples by Common Subject

This work addresses the bug in #98.

@tmciver Thanks for this bug report and PR, and your efforts in improving the rdf4h library.
Looks good, I'll merge this.
It'd be good if, in addition to the "triples with the same subject should be grouped" test case (and other unit tests), we can also use property based tests using QuickCheck for far greater coverage of possible inputs.
E.g. for the bug that you report, the specification might be:

In the Turtle serialisation of a graph, every subject node appears in a subject position exactly once.

or

In the Turtle serialisation of a graph, a node can appear in a subject position at most once.

This can test for many more possible interleavings of triples with common subjects, in addition to:
<:something> <dc:title> \"Some title\" .
<:another> <rdf:type> <schema:Document> .
<:something> <rdf:type> <schema:Document> .

We probably can't use the Turtle parser to test this, since it desugars < subject , [ (predicate , object) ] > patterns into < subject , predicate , object > triples.
Hhmm..

@tmciver we should probably also have QuickCheck tests to validate properties that we believe should hold when parsers and serialisers are composed. E.g.
forall (g::RDF a) . triplesOf g
                    ==
                    triplesOf (parseString (TurtleParser Nothing Nothing)
                                   (writeRdfString (TurtleSerializer Nothing Nothing) g))

There's probably a few such properties we assume hold that might not. E.g. on the number of triples, the order of triples, ensuring namespaces are/aren't lost, etc.

@robstewart57 I love the idea of using Quickcheck!  That's something I can work on but I'd like to tackle #97 first since the produced Turtle is not valid until that one is resolved.


Weighted prompts feature gives unexpected results

Not sure if I'm missing something but when trying to use weighted prompts it seems whole concepts get removed. For testing I'm using this prompt from the internets as it makes it clear to see due to the green cucumber coloring.
First try vanilla (w:448, h:640, euler_a, 25 steps, cfg:8):
full body shot, benedict cumberbatch as a cucumber knight, intricate, highly detailed, digital painting, artstation, concept art, art by artgerm and greg rutkowski and alphonse mucha, full body shot
This should produce very green knights of house cucumber
Now try:
full body shot, benedict cumberbatch as a:1.0 cucumber:1.1 knight, intricate, highly detailed, digital painting, artstation, concept art, art by artgerm and greg rutkowski and alphonse mucha, full body shot
or even:
full body shot, benedict cumberbatch as a:1.0 cucumber:1.0 knight, intricate, highly detailed, digital painting, artstation, concept art, art by artgerm and greg rutkowski and alphonse mucha, full body shot
The green is gone and cumbersnitches face is missing.
Can someone reproduce this? User error? I'm basing the syntax on the Weighted Prompts section from the features other markdown
Expected behavior
I would expect a weighting of :1.0 not to change anything at all
Environment

GPU: cuda
VRAM: 24GiB
CPU arch: x86
OS: Linux
Python: miniconda, Python 3.9
Branch: main
Commit: 698d30

Hi Spezialspezial, its unclear to me what you are trying to achieve, but I don't think you are using prompt weighting correctly.    If you want to weight how much Benedict is a cucumber vs a knight, I would try something like:
benedict cumberbatch as a cucumber: 0.5 benedict cumberbatch as a knight: 0.5

I'm trying to put emphasis on the cucumber part, making it more cucumbery, but something seems off.
Like in the docs: tell the sampler to attach different levels of priority to them, by adding : to the end of the section you wish to up- or downweight.

@spezialspezial , the syntax is confusing and we have a lot of work ongoing to improve it add a number of other features. Our current prompt weighting (more accurately, prompt blending) is intended to blend whole prompts, not individual tokens.

Oh, okay, got it


'Save as default ' button is enabled even if no FOIPPA section is selected

Describe the bug in current situation
When there is no FOIPPA section selected and 'save as default' button is enabled, then redaction is saved without having any section numbers
Link bug to the User Story
Impact of this bug
Describe the impact, i.e. what the impact is, and number of users impacted.
Chance of Occurring (high/medium/low/very low)
Pre Conditions: which Env, any pre-requesites or assumptions to execute steps?
Test-marshal
Steps to Reproduce
Steps to reproduce the behavior:

Go to redaction app
Select some region for redaction ,then FOIPPA modal comes
When we click on 'save as default' button without checking any sections then the redaction is saved on the right redaction panel without section number
then we are unable to edit that redaction as it is not saved with a section before

Actual/ observed behaviour/ results
Expected behaviour
A clear and concise description of what you expected to happen. Use the gherking language.
Screenshots/ Visual Reference/ Source
If applicable, add screenshots to help explain your problem. You an use screengrab.
[Private Zenhub Image]
Private Zenhub Image
(https://api.zenhub.com/attachedFiles/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBakV3IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9a64928d504c9a76c675c4972e13dd5a1d61a1d7/image.png)

Needs to confirm whether this will be applicable to 'multiselect' as well?  Since if we multiselect two of the redactions and click "save as default' it wont change any section update on there as well @m-prodan
cc @divyav-aot
Private Zenhub Image

working as expected, so closing it


upgrade erlang in emqttd directory

Environment

OS: Ubuntu14.04
Erlang/OTP:
EMQ: 2.0.6

I have problem with beam.smp memory overload which mentioned in issue 308.
I could install erlang by apt-get but how can I upgrade the erlang located in local emqttd directory ?

Upgrade Erlang/ OTP to R18.3+


quick fix try for mus

Overview

Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

Codecov Report

Merging #2083 (5995699) into main (94cd2a6) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #2083      +/-   ##
==========================================
- Coverage   55.70%   55.67%   -0.03%
==========================================
  Files         208      208
  Lines       13161    13162       +1
==========================================
- Hits         7331     7328       -3
- Misses       5085     5089       +4
  Partials      745      745

Impacted Files
Coverage Δ

nodebuilder/header/config.go
53.12% <100.00%> (+0.74%)
:arrow_up:

... and 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Remove HTTP::RequestProcessor

The HTTP::RequestProcessor class simply delegates the request handling for HTTP::Server. I don't think this needs to be a distinct type. There is a 1:1 coupling between each server and processor, and it basically consists of one method and two ivars which can easily be integrated into HTTP::Server directly.
I suppose the class was originally introduced in ecdcea9b945559fdd44b4fb6f993ce20e287b207 as an easy to use interface for testing request processing, since HTTP::Server required to set up a TCP server for that and RequestProcessor operates on IOs which can be easily be tested using IO::Memory.
After #5776 this is no longer the case. Setting up a HTTP::Server instance doesn't anymore instantly open a TCP server. It could easily be used for testing handler chains without going through any network socket.
HTTP::RequestProcessor is part of the public API, but has no documentation. I doubt it has any practical use case where it couldn't be substituted by HTTP::Server directly.

I don't mind the split, purely on a code separation point of view. HTTP::Server dealing with the incmping connections and HTTP::RequestProcessor dealing with an individual connection's lifecycle makes sense to me.
I don't think it needs to be a documented type though.

@straight-shoota yes, it's a single method, but it's a long one and I wouldn't want to make http/server.cr longer. Any complaints of mine are purely stylistic.

Oh, yes. I'd actually too prefer to leave the code in request_processor.cr and just reopen HTTP::Server instead of creating a new class.


How many can join at once on the public server and on a private self hosted server?

How many can join at once on the public server and on a private self hosted server?

We don’t have an exact number but you should be able to handle as many as you’d like.


WIP: Introspection support

Also adds support for interleaved keys and improvements for identity columns.

I'd like to add test for this, but introspection tests are going to require an actual Redshift cluster to connect to. I agree with the sentiment in #9 that it's unreasonable to maintain a cluster just for tests.
I'd propose a separate test suite that doesn't get run on travis, but can be run locally. It would be up to the individual developer to point the optional tests at a Redshift cluster.
That said, I have limited experience with pytest, so I'm open to suggestions about the best way to go about defining optional tests.

I'm calling this WIP for the moment until we define a testing strategy and determine if there are missing pieces of functionality.
I'll comment that the general approach here is to limit the number of queries sent to the cluster. The _get_*_info functions pull info about all tables in all schemas, and other functions pull data as needed out of those large cached blobs of schema info. It's been my experience that the bulk of the latency in Redshift queries is simply waiting for a slot for the cluster, so minimizing the number of overall queries is significant.

There are several fixes needed to address test failures in Travis.

You can create a pytest fixture that creates a redshift-sqlalchemy database engine. Then run all the tests that don't require this fixture
Do you think @SimpleFinance could sponsor or part sponsor a test instance? I could write a script that only boots it when the tests are run.

Do you think @SimpleFinance could sponsor or part sponsor a test instance? I could write a script that only boots it when the tests are run.

That's a possibility. Is there a way to provide AWS credentials to Travis such that they're not publicly exposed?

@jklukas yes, you can encrypt them http://docs.travis-ci.com/user/encryption-keys/

I brought up the idea of Simple sponsoring a test cluster, and it sounds like we can make it happen. I'll set up a completely separate AWS account and we'll link it to Simple for billing. We can probably have that all in place next week sometime.

+1, some minor nitpicks, all of which are safe to ignore if you disagree.

This definitely needs tests of some sort, mocking responses from Redshift would be nicer than nothing. It would be super great if we can test this against a real Redshift.

Although the test I added is passing in Travis, I can't get them to run locally. I load up the relevant PGPASSWORD for the travis user locally and the connection is fine, but I get the following error:
E       sqlalchemy.exc.InternalError: (psycopg2.InternalError) opening multiple cursors from within the same client connection is not allowed.

I'd like to understand that, but regardless I'll plan to pick up next week with filling out more introspection tests.

Maybe something like this: https://github.com/jklukas/redshift_sqlalchemy/compare/master...graingert:introspection-support

I suspect the multiple cursor issue is because there's code like:
def get_something(conn):
    result conn.execute(query)
    for table_name in result:
        _do_something_with_table_name(tablename=tablename, conn=conn)

def _do_something_with_table_name(conn, tablename):
    result conn.execute(query, tablename=tablename)
    for foo in result:
        💥

Maybe something like this: https://github.com/jklukas/redshift_sqlalchemy/compare/master...graingert:introspection-support

I want these tests to be as simple and readable as possible. The approach proposed in the above requires defining the same table in both text and SQLAlchemy constructs, plus keeping test names in sync between two files.
What I'm interested in with these tests is that introspection can regurgitate a table definition. Using the same DDL test block for creating and comparison seems like the simplest approach.

@jklukas I'd rather not run create_table queries as part of unit tests.
In general you don't worry about keeping things in sync in tests, a test failure will verify that they are not in sync.

I'd rather not run create_table queries as part of unit tests.

Is your objection to creating tables at all, or to creating tables via SQL text? If the former, I'm not understanding how it's possible to test introspection without actually creating a table. If the latter, I'm not understanding why it's bad form.

The test suite fixtures automatically creates the tables in models.py just
after the database is created
On 24 Aug 2015 4:50 pm, "Jeff Klukas"<EMAIL_ADDRESS>wrote:

I'd rather not run create_table queries as part of unit tests.
Is your objection to creating tables at all, or to creating tables via SQL
text? If the former, I'm not understanding how it's possible to test
introspection without actually creating a table. If the latter, I'm not
understanding why it's bad form.
—
Reply to this email directly or view it on GitHub
https://github.com/graingert/redshift_sqlalchemy/pull/17#issuecomment-134257716
.

I've been learning a lot about SQLAlchemy internals in putting this together, and it looks like my approach here is fundamentally flawed. I wanted to make reflecting an entire schema fast by sharing the results of one large query to create many Table objects, but it looks like the reflection machinery that exists doesn't support that.
I will likely make make a new branch from master and start over again, overriding just the methods needed to catch Redshift-specific table attributes.

That's fine. I'd rather you reset this branch because it keeps all the
comments in context
On 25 Aug 2015 02:55, "Jeff Klukas"<EMAIL_ADDRESS>wrote:

I've been learning a lot about SQLAlchemy internals in putting this
together, and it looks like my approach here is fundamentally flawed. I
wanted to make reflecting an entire schema fast by sharing the results of
one large query to create many Table objects, but it looks like the
reflection machinery that exists doesn't support that.
I will likely make make a new branch from master and start over again,
overriding just the methods needed to catch Redshift-specific table
attributes.
—
Reply to this email directly or view it on GitHub
https://github.com/graingert/redshift_sqlalchemy/pull/17#issuecomment-134444548
.

I decided to continue with this PR, leaving the queries as-is, but taking the performance hit that comes with not being able to cache between reflection of multiple tables.
I expect this PR is actually complete at this point, with the full suite of reflection tests I've been intending. The tests are failing due to some setup issue ImportMismatchError, which I'm tracking down now.

Tests are now passing by specifying tests as the default argument to tox.

The docstrings aren't being discovered now. Remove that default py.test
argument and ensure that conftest doesn't import redshift_sqlalchemy.
See
https://github.com/graingert/tox_doctest_fail/blob/master/tests/conftest.py
for what triggers the import errors in py.test
On 25 Aug 2015 22:09, "Jeff Klukas"<EMAIL_ADDRESS>wrote:

Tests are now passing by specifying tests as the default argument to tox.
—
Reply to this email directly or view it on GitHub
https://github.com/graingert/redshift_sqlalchemy/pull/17#issuecomment-134742776
.

ensure that conftest doesn't import redshift_sqlalchemy

In this case, conftest.py doesn't import redshift_sqlalchemy. It's https://github.com/graingert/redshift_sqlalchemy/blob/master/tests/test_unload_from_select.py that has from redshift_sqlalchemy.dialect import UnloadFromSelect.
Do you know of a workaround to make UnloadFromSelect available in that test without importing redshift_sqlalchemy?

It should be fine to import in the tests, just not as an import side effect
of conftest.py. For now you can hack it by importing it in the test
function.
If that doesn't fix it I'll mess around in pdb and see if I can find
another pytest doctest bug. Another option is just to use boring old
python -m doctest
On 26 Aug 2015 12:29 am, "Jeff Klukas"<EMAIL_ADDRESS>wrote:

ensure that conftest doesn't import redshift_sqlalchemy
In this case, conftest.py doesn't import redshift_sqlalchemy. It's
https://github.com/graingert/redshift_sqlalchemy/blob/master/tests/test_unload_from_select.py
that has from redshift_sqlalchemy.dialect import UnloadFromSelect.
Do you know of a workaround to make UnloadFromSelect available in that
test without importing redshift_sqlalchemy?
—
Reply to this email directly or view it on GitHub
https://github.com/graingert/redshift_sqlalchemy/pull/17#issuecomment-134769816
.

It looks like the problem is indeed with the doctest setup rather than with test_unload_from_select.  Within the RedshiftDDLCompiler docstring is this line which seems to be the problem:
    >>> engine = sa.create_engine('redshift+psycopg2://example')

If I remove the doctest configuration from tox.ini, tests run smoothly.

I've reverted the default argument, but had to remove doctest from tox.ini for the time being. I also filled out documentation.
I'm looking for a +1 at this point. I'm hoping we can defer debugging of the doctest import conflict to another PR.

The doctests definitely work before your change. Can you run a git bisect?

The doctests definitely work before your change. Can you run a git bisect?

Yes, the doctests worked, but that was before adding conftest.py. I just tried adding doctests back into the tox.ini and ran a few scenarios:

Remove test_simple_query.py and test_reflection.py -> ImportMismatchError
Remove conftest.py -> Tests run successfully
Restore conftest.py, but remove all Reflectionclasses frommodels.py` -> Tests run successfully

It looks like this ImportMismatchError comes up when we have doctests on and we have a model which includes a dialect-specific argument in __table_args__. ReflectionUniqueConstraint was able to be present if removed {'redshift_diststyle': 'EVEN'} from its definition.
So, I'm assuming that the Redshift dialect only gets imported when dailect-specific kwargs need to be parsed?
I have no idea where to go from here.

Ah, move from rs_sqla_test_utils models to line 78 def _redshift_database_tool(): of conftests

Ah, move from rs_sqla_test_utils models to line 78 (def _redshift_database_tool():) of conftests

Damned if that didn't work. I just pushed the change, but I still don't understand why it matters :confounded:
I'm happy to rebase and squash this PR down to a single commit, since history is now a bit messy.

yes, please rebase squash etc.

@jklukas see here for discussion: https://github.com/pytest-dev/pytest/issues/727#issuecomment-133217356

Squashed down to a single commit, and AWS_ACCESS_KEY_ID made explicit.

Removed set_isolation_level (which had exactly the same code the the psycopg2 version) and _isolation_lookup, as the most recent functional change according to git history happened in SQLAlchemy 0.8.0.  This is already the minimum required version, so no change needed to docs.

Rebased on master to make this mergeable again.

Fixed an unused import.

Removed the # noqa and rebased on master. This is mergeable again (for the moment :wink: ).


Fix nil device error for stats push service and use live WireGuard adapter in dev.

I noticed this error when running with the sandbox adapter:
[error] GenServer #PID<0.947.0> terminating
** (UndefinedFunctionError) function nil.peers/0 is undefined
    nil.peers()
    (fz_vpn 0.0.0+git.0.deadbeef) lib/fz_vpn/interface.ex:141: FzVpn.Interface.dump/1
    (fz_vpn 0.0.0+git.0.deadbeef) lib/fz_vpn/stats_push_service.ex:32: FzVpn.StatsPushService.push_stats/0
    (fz_vpn 0.0.0+git.0.deadbeef) lib/fz_vpn/stats_push_service.ex:28: FzVpn.StatsPushService.handle_info/2
    (stdlib 4.0.1) gen_server.erl:1120: :gen_server.try_dispatch/4
    (stdlib 4.0.1) gen_server.erl:1197: :gen_server.handle_msg/6
    (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :perform
State: %{}

The cause of the error was that the server & stats push service modules had their own process dictionaries, so two sandbox adapters were started, one with the wg-firezone device set and the other without causing the nil error.
~So I moved the sandbox to being started conditionally with fz_vpn's application supervisor instead.~

Pull Request Test Coverage Report for Build a5291f0e78c8e5cee14d3567afeb83681d96ecf6-PR-825

5 of 9 (55.56%)  changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 69.242%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

apps/fz_vpn/lib/fz_vpn/interface/wg_adapter/sandbox.ex
5
6
83.33%

apps/fz_vpn/lib/fz_vpn/stats_push_service.ex
0
1
0.0%

apps/fz_vpn/lib/fz_vpn/application.ex
0
2
0.0%

Files with Coverage Reduction
New Missed Lines
%

apps/fz_vpn/lib/fz_vpn/interface/wg_adapter/sandbox.ex
1
87.88%

Totals

Change from base Build 4aadabd5258ef1ac900f0387a2bfab54ba4e97f0:
-0.1%

Covered Lines:
977

Relevant Lines:
1411

💛  - Coveralls


Static Mod Hierarchy

Describe the bug
Perhaps "bug" isn't quite the right term for this, but I've noticed that the codes section and mods section behave differently and was wondering if parity between their behavior would be possible. Codes maintain a static placement in their list regardless of whether the code has been enabled or not. Mods, meanwhile, return all inactive mods to alphabetical order whenever they are saved as deselected. This makes mod organization needlessly difficult.
To Reproduce

Toggle a mod on
Save, note that the position is moved to the top of the hierarchy (or bottom of the mods hierarchy in the event other mods are already enabled)
Toggle a mod off
Save, note that the mod shifts back to its alphabetical position

Expected behavior
The mod would remain in the hierarchical position that it was left in after saving, regardless of whether the mod is on or off (just as a code would). An option to toggle between static and auto-alphabetical behavior for disabled mods would also be an effective resolution
Screenshots
N/A
Desktop (please complete the following information):

OS: Windows 10 Pro - 64-bit
Version (1.1.6 - 8afdbf4)

P.S. Thank you for all of your hard work on this tool. This issue aside, it's been a joy to work with.

After a lot of thought, this is way too complicated and definitely way more complicated than the existing system is.


Rollup of many PRs and other fixes from my fork

@scrooloose If you ever want to bring the original plugin up to speed I'd love to talk about what it would take to merge all these things back from my fork and possible add a maintainer or two. Great plugin, but it needs better out of the box file type support and with so many contributions coming in it was a shame they were all being left to stand on their own. For now I'll continue to pull in any contributions that are simple file type additions and any other improvements that look sane to me.

I've been added as a maintainer and will merge these separately directly to the main repo to avoid at least one of the mixups I made while setting them up in my fork.


Discovering project ID from the local metadata service

Support for auto discovering the project ID when running on Compute Engine, Cloud Run etc.
Verified by running a build on a Compute Engine instance.

@bojeil-google I've made the suggested changes.


Support Dict type field

I have the following pydantic model:
class SourceModel(BaseModel):
    id: str
    data: Dict[str, Any]

And use it to define Strawberry type:
@strawberry.experimental.pydantic.type(model=SourceModel, fields=['id', 'data'])
class Source:
    pass

But I get the following error:
Traceback (most recent call last):
  File "schema.py", line 87, in <module>
    class Source:
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 61, in wrap
    all_fields = [
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 64, in <listcomp>
    get_type_for_field(field),
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 36, in get_type_for_field
    type_ = replace_pydantic_types(type_)
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 22, in replace_pydantic_types
    return type_.copy_with(tuple(replace_pydantic_types(t) for t in type_.__args__))
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 22, in <genexpr>
    return type_.copy_with(tuple(replace_pydantic_types(t) for t in type_.__args__))
  File "/lib/python3.8/site-packages/strawberry/experimental/pydantic/type.py", line 24, in replace_pydantic_types
    if issubclass(type_, BaseModel):
  File "/usr/local/lib/python3.8/abc.py", line 102, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

Field data in my model is embedded document from MongoDB and is structure may varies (I get it from external source), so I can use only Dict for it.
In Graphene I use GenericScalar type for it and it perfect serve my needs. But I want switch to Strawberry and can't without this.

@Dotrox I think we can implement that 😊 I might have some time over the weekend to do so.
I need to think a bit about the implication of doing this for the Dict type, as it might be confusing to some people.
Does Graphene convert to GenericScalar directly?

Does Graphene convert to GenericScalar directly?

I can`t use pydantic models with Graphene, so define all fields directly by myself:
class Source(graphene.ObjectType):
    id = graphene.String()
    data = GenericScalar()

With Strawberry I already use additional decorator for types because I use not pure pydantic BaseModel, but wrapped model class from ODMantic ODM.
This how it really looks now:
def init_model(model):
    def wrap(cls):
        cls_annotations = model.__pydantic_model__.__dict__.get(
            "__annotations__", {})

        for field, field_type in cls_annotations.items():
            if field_type == odmantic.bson._datetime:
                cls_annotations[field] = datetime
            elif field_type == odmantic.bson.ObjectId:
                cls_annotations[field] = str

        cls.__annotations__ = cls_annotations

        return cls

    return wrap

@strawberry.type
@init_model(SourceModel)
class Source:
    pass

Only this way Strawberry can perfectly works with ODMantic models: I need support for MongoDB ObjectId and also ODMantic implicitly change datetime with own version. And as sugar my decorator load all fields from my model, but main reason for it is type conversion.
So Strawberry may support Dict type in similar way (I tried implement it by myself, but my knowledges not enough).

ok! thanks for the additional information, will take a look at it soon :)

Any news?

@Dotrox haven't had time to look into this yet, is it blocking you?

I can't use Strawberry without this.
Maybe you can explain how I can implement this by myself (how add custom type to Strawberry)?

What should be the result of data? A JSON string?

Just added some documentations about an example JSONScalar here

@Dotrox let me know if the docs written by @Speedy1991 are helpful; so we can close this issue 😊

@Dotrox let me know if the docs written by @Speedy1991 are helpful; so we can close this issue blush

Json doesn't fit my needs, but this example show me how create a custom scalar. So I slightly modified it and get what I need:
GenericScalar = scalar(
    typing.NewType("GenericScalar", typing.Any),
    description="The GenericScalar scalar type represents a generic GraphQL scalar value that could be: List or Object."
)

Also I made a custom scalar for MongoDB ObjectId:
ObjectIdScalar = scalar(
    bson.ObjectId,
    serialize=lambda v: str(v),
    parse_value=lambda v: bson.ObjectId(v),
    description="MongoDB ObjectId"
)

It may be helpful for somebody if it will be in docs.

Here's what I needed to do to get JSONScalar working with an inline input value
JSONScalar = strawberry.scalar(
    NewType("JSONScalar", Any),
    serialize=lambda v: v,
    parse_value=lambda v: json.loads(v),
    parse_literal=graphql.utilities.value_from_ast_untyped,
)


Added routes for creating, triggering and deleting builds

Motivation
https://issues.jboss.org/browse/AEROGEAR-7911
https://issues.jboss.org/browse/AEROGEAR-7915
https://issues.jboss.org/browse/AEROGEAR-7914

Pull Request Test Coverage Report for Build 483

145 of 244 (59.43%)  changed or added relevant lines in 6 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-6.2%) to 68.497%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/mobile/secrets.go
0
8
0.0%

pkg/mobile/buildConfig.go
5
27
18.52%

pkg/web/buildConfigsHandler.go
129
198
65.15%

Files with Coverage Reduction
New Missed Lines
%

pkg/mobile/buildConfig.go
1
28.13%

Totals

Change from base Build 477:
-6.2%

Covered Lines:
474

Relevant Lines:
692

💛  - Coveralls


Modified add program/course form

Closes #45
Added required field wherever necessary
Added dynamic addition of 'add course' fields

LGTM


Looking forward to official support for no eject extension react-scripts/webpack

Extended configuration is required
When we create react app with create-react-app, we almost always need to extend more configurations ( react-scripts or webpack ), and we want to not eject to continuously upgrade react-scripts.
The current available methods are as follows, but they all have their own problems.

npm run eject
Cannot upgrade the react-scripts.

Modify the react-scripts source under the node_modules, and then use the patch-packages
Cannot be maintained when upgrading react-scripts.

Using the --scripts parameter, extend the custom react-scripts
It is too expensive to customize a react-scripts for a single project.

Other programs in the community

craco
react-app-rewired
rescripts
......

These programs are great, but there is no way for community support to continue. When there is a destructive update, such as CRA5. Almost can not follow up to support, or even be abandoned.
For example, I used to like using craco. But, when CRA5 is released, because the update is too big, the original author did not have much time to maintain it, and this plan was almost abandoned (LOOKING FOR A NEW TEAM TO LEAD CRACO).

Describe the solution you'd like
Expect that the official can support a method. Without eject, we can extend the configuration ( react-scripts/webpack ), support parameters can be merged into the webpackConfig.
If it can be supported, we can preset some configurations when we make a custom cra-template.

a good proposal

I feel that there simply isn't enough value proposition from react-scripts since webpack 5 to bother, if you're going to be customizing?
Out of the box, webpack 5 supports hot reloading, error overlays, development and production defaults, caching, sensible defaults for input and output..., at that point react-scripts is only really enabling the above, adding babel transforms, CSS bundling, and integrated lint and type-checks. The former two are trivial to add to a webpack.config.js, and the latter are of dubious value and are generally being moved away from by the larger web dev community, in my experience (in favor of CSS-in-JS and separate CI passes + editor highlighting)
It's a little rough (needing to install @webpack-cli/generators, some defaults I don't like, etc.), but there's even npx webpack-cli init to replace npx create-react-app.


When going to singleplayer

When i click the single player button i get this
http://pastebin.com/hkhVF8rT

Seems that the error comes from ExtraPlanets, you're not the first one to report this and you should go on their side.


Populating dates in CDS Connect metadata

@preston @linophth Some of the VHA artifacts have dates populated in the Repository Information metadata section (e.g.,  Publication Dates and Last Review Dates in Primary Care: Tobacco Assessment and Cessation Counseling Documentation Template and many/if not all of the Cardiology artifacts such as Cardiology: Cardiac Catheterization Documentation Template). However, the populated dates do not align with the definition of the metadata fields, which will cause confusion. See the metadata field definitions far below.
For example, the Primary Care: Tobacco Assessment and Cessation Counseling Documentation Template
artifact has the following values populated:
Publication Date
2018-06-14
Last Review Date
2018-05-23
I would like to propose that I remove existing dates and:

Manually enter the date that the artifact is published on CDS Connect in the Publication Date field

Manually enter the date that the artifact is published on CDS Connect in the Last Review Date field, given the recent re-review of the artifact by the VHA clinical domain SMEs and the MITRE review of each artifact prior to publishing.

What are your thoughts?
Thanks,
Sharon

Metadata Field
Date Type
Description

Creation Date
Date
The date that the artifact was created/developed (this is different than the date that the evidence-based reference source was published)

Approval Date | Date | The date the artifact was approved for the Repository by the CDS Connect administrators. This is usually null.
Expiration Date | Date | The date at which artifact no longer effective
Last Review Date | Date | The date of last review (this is usually on or after the Publication Date). For instance, an artifact may be published on 7/12/16 and reviewed annually, therefore the "Last Review Date" is 7/12/18.
Publication Date | Date | The date that the artifact was originally published on the Repository

Your approach makes sense to me.

Okay, thank you. I’ll take care of adjusting the dates on each artifact as I go.
I can also populate Last Reviewed and Publication Dates on the artifacts that don’t have existing dates.
From: linophth<EMAIL_ADDRESS>Sent: Tuesday, November 27, 2018 6:01 PM
To: OSEHRA/vha-kbs-knarts<EMAIL_ADDRESS>Cc: Sebastian, Sharon M<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [OSEHRA/vha-kbs-knarts] Populating dates in CDS Connect metadata (#460)
Your approach makes sense to me.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/OSEHRA/vha-kbs-knarts/issues/460#issuecomment-442251397, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AobGn1vQSn1GW1RvrEZhu2xY-ih0JVPBks5uzcQWgaJpZM4Y2az_.


lodash dependency to README.md

Please add the requirement of lodash for version 2.0.0

npm install lodash --save

fixed in 2.0.1 - thanks :)


Body doesnt glow with the glowing effect

And shouldn't it? Doesn't it even use the third person model?

It uses the third person model. I'm not sure at which point the glow is applied to be missing in this case, and if it where to work, it might just clutter the screen really badly. Will have to check, but might be a case of "it's better turned off" like with the F3+B hitbox in firstperson.

Even if it is, can you make it optional via config?

becuase i like realism, but in case it looks bad and other people dont, then it should definitely be toggleable

Just tested it, player renders in the pure team color(white by default), and I think fixing that would require MixinExtras which I won't look into rn due to old mc versions and Forge/NeoForge.


Tracking don't work

Hi,
i installed your plugin and can't get it work.
No matter production or dev-mode.

More Info:

Matomo V4.5.0 - system check is OK
I tested in production and local development
Plugin options:
siteId: 1,
matomoUrl: https://xxxxxxx,
siteUrl: https://www.xxxxxx.xx,
// localScript: /piwik.js, <<< also tested
exclude: ['/abc/'],
requireConsent: false,
requireCookieConsent: false,
disableCookies: true,
respectDnt: false,
dev: true

I hope you can find time to help.
Best regards

@rsubasoglu I had a similar issue. I compared the script provided by matomo and the script on my website and noticed that its not piwik.php and piwik.js that must be used but matomo.php and matomo.js so:
    matomoPhpScript: "matomo.php",
    matomoJsScript: "matomo.js",

Hope it helps.


Problems in Data Conversion with Excel of Otoole Examples

$ otoole convert excel datafile simplicity.xlsx simplicity.txt config.yaml
OtooleIndexError: TechnologyFromStorage -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY', 'STORAGE', 'MODE_OF_OPERATION']. Data indices are ['REGION', 'TECHNOLOGY', 'STORAGE', 'YEAR'].

After I change 'MODE_OF_OPERATION' to 'YEAR' in config.yaml, the following steps of this section seems work, but there comes another problem in Model Visualization:

$otoole viz res excel simplicity.xlsx res.png config.yaml
TypeError: argument of type 'float' is not iterable

So the res.png was not created in this folder.
I'm new to OSeMOSYS, so I don't know how to solve this problem.

Hi @Hansen-hanhui, thanks so much for flagging this issue!
Quick question, are you able to check your otoole version, please? You can do that through running the command otoole -V from the command line.
If you are running otoole version 1.0.2, you may be experiencing the bug described in this issue ticket. If this is the case, can you please try updating your otoole version to 1.0.3 and try again? To update you otoole version, you should be able to just run the command pip install otoole==1.0.3.
After doing this, can you also report back on this issue ticket please. That way we can either close the issue of figure out what is going on if you are still getting errors. Thanks again! :)

Hi @trevorb1 , thanks for your quick response. the former problem is fixed after I upgrade to 1.0.3 from 1.0.2. However the latter problem still exist, maybe caused by another small problem in config.yaml.
Thanks again for your detailed help! To save your valuable time, it's ok to reply briefly, and I will keep my attention on this project.

Great, happy to hear the first issue is solved @Hansen-hanhui!
Unfortunately, I wasn't able to recreate your second issue. Below are the commands I ran and everything worked for me; I used otoole v1.0.3. Are you using the config file you updated, or the reference one found in the repository? If the issue persists, could you please run otoole in verbose mode (ie. otoole -v viz res ...) and copy over the traceback error?
git clone https://github.com/OSeMOSYS/simplicity.git
cd simplicity
otoole convert csv excel data data.xlsx config.yaml
otoole convert csv datafile data data.txt config.yaml
otoole viz res csv data res-csv.png config.yaml
otoole viz res excel data.xlsx res-xlsx.png config.yaml
otoole viz res datafile data.txt res-txt.png config.yaml

Thanks!

C:\Users\Administrator>git clone https://github.com/OSeMOSYS/simplicity.git
Cloning into 'simplicity'...
remote: Enumerating objects: 189, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 189 (delta 5), reused 18 (delta 3), pack-reused 166
Receiving objects: 100% (189/189), 59.11 KiB | 517.00 KiB/s, done.
Resolving deltas: 100% (73/73), done.
C:\Users\Administrator>cd simplicity
C:\Users\Administrator\simplicity>otoole convert csv excel data data.xlsx config.yaml
C:\Users\Administrator\simplicity>otoole convert csv datafile data data.txt config.yaml
C:\Users\Administrator\simplicity>otoole viz res csv data res-csv.png config.yaml
TypeError: argument of type 'float' is not iterable
C:\Users\Administrator\simplicity>otoole viz res excel data.xlsx res-xlsx.png config.yaml
TypeError: argument of type 'float' is not iterable
C:\Users\Administrator\simplicity>otoole viz res datafile data.txt res-txt.png config.yaml
TypeError: argument of type 'float' is not iterable
C:\Users\Administrator\simplicity>otoole -V
1.0.3

I tried it in cmd of windows11, maybe the problem was caused by not using linux system?
I am just famillar with python language, and I will further explore the python code of otoole so that I can use pycharm to debug.
Thanks!

Thank you @trevorb1 , I have tried WSL on Windows machine, and ALL CODE WORKS! That's so great!
Maybe I should try to learn more about LINUX as it provide more friendly environment than windows cmd.
Thanks again for your help! :)

There are another problem in Model Validation 6. otoole validate. I just copied the code in this section to validate.yaml and there came an error:

root@A3K0GQFTQ2V5QX7:/mnt/c/Users/Administrator/simplicity# otoole validate datafile data.txt config.yaml --validate_config validate.yaml
ScannerError: mapping values are not allowed here
in "validate.yaml", line 18, column 12

Then I checked the difference between otoole/src/otoole/validate.yaml, and found that there are several indent error in Schema. After I corrected the indent errors in this file, the code worked with the result:

Beginning validation
Validating FUEL with fuel_name
17 invalid names:
BAGASSE, DSL, ETH, FEL1, FEL2, FER, GAS, LAND, LANDFOREST, LNDSUGCAN, MOLASSES, RAWSUG, RIVERWATER, SEC_EL, SUGCAN, WATAGR, WATIN
Validating TECHNOLOGY with technology_name
26 invalid names:
BACKSTOP1, BACKSTOP2, ETHPLANT, GAS_EXTRACTION, GAS_IMPORT, GRID_EXP, HYD1, HYD2, IMPDSL, IMPFERT, IMPRAWSUG, LNDFORCOV, LNDRES, LNDSUGPL, LNDSUGPLIR, LNDSUGPLRF, NGCC, RIVER, RIVER_2, RIVWATAGR, SOLPV1, SOLPV2, SUGFACTORY, TD, TD2, WINDPOWER
Checking graph structure
1 'technology' nodes are isolated:
TD2

I believe that I have fixed this problem, and there are no other problems in Examples.  :)


[Docs]: Add Null return value type in code example

Summary
TypeScript threw this error when I attempted to copy and paste in my code:
Type 'null' is not assignable to type 'string'.ts(2322)

The getDefaultKeyBinding() method may return null. So I added | null to the return type declaration. Just thought it would help others who may also be copying/pasting the code!
Test Plan
No testing required.

Yup, can confirm the type is an optional DraftEditorCommand 👍
https://github.com/facebook/draft-js/blob/bc716b279299748d955bbdc398454bff45ea0191/src/component/utils/getDefaultKeyBinding.js#L63


Fix file name error in download example

@ideal Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@ideal Thank you for signing the Contributor License Agreement!


Fix typo in DeviceOrientation documentation

Rename prop from betaa to beta in DeviceOrientation documentation

Hey @jaredpalmer ! I just watched you present at React NYC, and wanted to check out react-fns. Looks super neat, I'll be keeping an eye on it!


Remove gnsst, add loct.

The configuration setting is changed to a more general location timeout setting called loct. It configures the time the device can take to acquire any location information, using GNSS, and fall back to querying neighboring cells and WiFi cells.
The default value should be 300 seconds and generally be fairly large, otherwise fallback to cellular will never occur if GNSS fix is not acquired.

When is this expected to be implemented into the cloud?

Once it lands in nRF Connect SDK main for the following release, probably 2.2.0.

PR https://github.com/nrfconnect/sdk-nrf/pull/8631 has been merged so I guess this change could be done now.


When using type "scale" you need argument: 'alignment'

try official demo and get :
When using type "scale" you need argument: 'alignment'
'package:page_transition/src/page_transition.dart':
Failed assertion: line 174 pos 16: 'alignment != null'

It has transitions which doesnt need alignment, but the doesnt work too, without error, just no effect

Pull request merged, pls use this package by git version:
animated_splash_screen:
    git: 'https://github.com/clean-code-dev/animated_splash_screen'


doc-json-config-example

Added JSON config files and updated related links.

This was a test that is completed.


Add a paraphrase pipeline

Gen -> Train (with low train iterations) -> Paraphrase -> Train (high train iterations) -> Eval
Currently testing this. I'll undraft when I'm convinced it works.

By the way, this does not work yet. I don't know what causes that or how to fix it.


Add API functionality to Sticker Module

Please check if the PR fulfills these requirements

[ ] Tests for the changes have been added (for bug fixes/features)
[ ] Docs have been added / updated (for bug fixes / features)
[x] I have included in this PR's description the statement closes #xx or fixes #xx where #xx is the number of the issue being addressed.

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
This PR Adds API functionality to Sticker Module closes #154

What is the current behavior? (You can also link to an open issue here)

What is the new behavior (if this is a feature change)?

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

Other information:

@kallyas so far I don't see anything out of place. So you can go ahead and merge this, maybe I'll pick up something during testing


fixing url-names at ModelControllerBuilder

The url_name attribute on _register_action_endpoint, except the create endpoint, points to primary key attribute name ( self._model_pk_name, instance to the model name. Another change: use lower case to url_name.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.91%. Comparing base (b51094e) to head (d2ca69c).
Report is 14 commits behind head on master.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##            master     #145      +/-   ##
===========================================
- Coverage   100.00%   99.91%   -0.09%
===========================================
  Files           56       56
  Lines         2452     2450       -2
===========================================
- Hits          2452     2448       -4
- Misses           0        2       +2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@eadwinCode I fix the pyproject. Again, Sorry. Now I will correct on fork ....


Put all images on one Markdown file

For easy comparison - make sure to make slides anyways

LucidChart is useful to learn as well


Ability to only allow view-based shares?

Is there a way we can disable the functionality of the time based views?
Ie only allow notes to be created that are destroyed after the first view?
Thanks.

That is already the case. It is time OR view based, not both. The text in the interface should make it clear in theory.
Btw the default behavior is exatcly that: no time contraint, destroyed after 1 view.
Feel free to reopen if I did not unterstand correctly :)


Support proxy-less endpoints

This issue outlines how networking works in aspire https://github.com/dotnet/docs-aspire/issues/232. Today we spin up a proxy for each endpoint which means the application being proxied needs to be amenable to accepting a port to listen on. The proxy is important for 2 reasons:

We can avoid forwarding the connection until the underling endpoint is ready
We can route to multiple replicas of the underlying service

The scenario that does not work is the one described here https://github.com/dotnet/docs-aspire/issues/232#issuecomment-1884575748

We have also a legacy java http service running on specific port on Windows.
We add the service as a console project managing its execution.
How can I specify its port in WithServiceBinding if it is not running in a container?
If i set the hostPort to the service port, the proxy blocks the port and the service cannot be started.

In this scenario, there can be no proxy because the underlying process can't listen to anything but the port it already has. The only work around for this today is to avoid using WithEndpoint (formerly WithServiceBinding) but that breaks the ability to add references to other resources (WithReference) and it also means the dashboard doesn't show the url for the service.
Open questions:

What happens when a resource has endpoints that are proxy-less and proxied? This probably shouldn't be allowed.

Does DCP already support a way to disable proxying or will this require work there too?

I think that perhaps proxy support is a per endpoint option rather than a per resource option. So perhaps we could do something like this:
WithEndpoint(....other args..., bool? proxy = true)
@karolz-ms can comment on whether DCP still supports proxyless executables/containers.

Yes, DCP supports proxyless mode. It currently designed for singletons, but could be enhanced to support replicated services too.

@JamesNK

Giving this to you @JamesNK

I think the design here needs to be based on the resource itself being proxyless. It solves many of the ambiguities.


Print bootstrap in pilot-agent

As part of logging "refactor", we've lost logs about the start-up bootstrap config. This makes is impossible to diagnose user issues: https://github.com/istio/issues/issues/169,  https://github.com/istio/issues/issues/173, as well as the recent cert straggler issues. Please bring back the log that printed the bootstrap config.

The bootstrap config logging was restored a while back.  Closing.


Fedora COPR Package

I think having a fedora COPR package along side an arch AUR package would be nice and help easier installation. I am interested in spending some time on this and maintaining a COPR package for yin-yang.
I wanted to check there is interest and if there is anything I should be aware of.

I have started a COPR where I can start experimenting with building it.
Please let me know if the maintainers do not wish me to package it on COPR so I can take it down.

That's great! If you have any questions or suggestions, please let me know!
For example, I am thinking about improving the installation process. The installation script is a bit hard to maintain, I would prefer to create an app manifest and let pip handle the rest. Or maybe I could compile everything into a single binary.


move toolbar one button left

It's not an issue but a question.
is it possible to move toolbar one button left so the second one appear first.
if user swipe to right the first one appear.
Thank you.

I found the solution.
[_scrollView setContentOffset:CGPointMake(somewith+8, 0) animated:YES];
thank you


Graph node subscriptions should be incremental

Subscriptions with graph is working fine.
The only issue is every time an entity matches it returns all the matching entity which as time passes becomes huge.
We only need the incremental matching entity data not all matching entity.
The ticket is about exploring and handling this in code
Find below the gist link for the code: https://gist.github.com/abhayks1/68969d2e29a77585894369f99f037b73

Below graph query can be used for the incremental susbcriptions:
query {
  stakeRequesteds(first:100, orderDirection: desc, where: {blockNumber_gt: $blockNumber}) {
    id
    amount
    gasPrice
    gasLimit
    staker
    gateway
    stakeRequestHash
    nonce
    beneficiary
    blockNumber
  }
}

$blockNumber: (last processed block number) -1

I guess as we are working on #86, we wouldn't need this ticket any longer.
@abhayks1 @deepesh-kn views please.

Closing this ticket as its already handled in https://github.com/OpenST/facilitator/pull/94