Incorrect values parsed for Complex Packing due to missing 2nd Scale Factor

Simple Description on the Bug
When parsing data at a constant Isobaric level that is encoded with Complex Packing, there are some messages that have a "Missing" 2nd Scale Factor that cannot be parsed resulting in incorrect values being produced.
Steps to Reproduce
Attempt to parse a GRIB file with a supposedly missing 2nd Scale Factor. I have posted an example GRIB file that displays this here: https://drive.google.com/drive/folders/1Zxs21uamCSF9LVUWHYO-dE-A9rzb91S_?usp=sharing. The error_grib_file contains the troublesome data and the correct_test_file contains GRIB data that can be properly parsed.
Note that I am only parsing messages that are at Isobaric layers (Code Table 4-5 100), not those that are altitude layers. The .idx index file shows which messages are of which type. The specific message that I am testing on is number 295 in the index file.
The following code explains how I was comparing the data from the test file to a different file that should have fairly similar data. Note that this was copy-pasted then adapted, so it will likely require some changes to function properly.
    #[test]
    fn test_grib_format() {
        // read through every message in a grib file and parse the first few values from each
        // message
        let p = "error_grib_file";
        let f = std::fs::File::open(&p).unwrap();
        let rd = std::io::BufReader::new(f);
        let f1 = grib::from_reader(rd).unwrap();

        let p = "correct_test_file";
        let f = std::fs::File::open(&p).unwrap();
        let rd = std::io::BufReader::new(f);
        let f2 = grib::from_reader(rd).unwrap();

        // set up decoder
        let f1 = f1
            .iter()
            .map(|lyr| {
                let ((i, j), fm) = lyr;
                let strn = format!(
                    "\n\nIncorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
                    i,
                    j,
                    fm.describe(),
                    fm.indicator(),
                    fm.grid_def(),
                    fm.prod_def(),
                    fm.prod_def().forecast_time().unwrap().to_string(),
                    fm.repr_def(),
                );
                let dc = Grib2SubmessageDecoder::from(fm).unwrap();

                let dv = dc.dispatch().unwrap();

                let val = dv.take(10).collect::<Vec<_>>();

                (strn, val)
            })
            .collect::<Vec<_>>();

        // set up decoder
        let f2 = f2
            .iter()
            .map(|lyr| {
                let ((i, j), fm) = lyr;
                let strn = format!(
                    "\n\nCorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
                    i,
                    j,
                    fm.describe(),
                    fm.indicator(),
                    fm.grid_def(),
                    fm.prod_def(),
                    fm.prod_def().forecast_time().unwrap().to_string(),
                    fm.repr_def(),
                );
                let dc = Grib2SubmessageDecoder::from(fm).unwrap();

                let dv = dc.dispatch().unwrap();

                let val = dv.take(10).collect::<Vec<_>>();

                (strn, val)
            })
            .collect::<Vec<_>>();

/*
Code to pair up messages together for printing purposes
*/

        f2.into_iter().zip(f1.into_iter()).for_each(|(f, l)| {
            println!("{}", f.0);
            println!("{:?}", f.1);
            println!("{}", l.0);
            println!("{:?}", l.1);
        });

        // ensure the test fails
        panic!()
    }

Expected Behavior
Parse the values properly.
Actual Behavior
Produce values that are wildly inaccurate and nowhere near the values produced by the file that is parsed correctly. Here is example output from the aforementioned message that I tested with:
Correct

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Momentum
  Parameter:                            u-component of wind
  Generating Proceess:                  Forecast
  Forecast Time:                        0
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     20000
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 581327 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 2, 2, 2, 0, 81, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 78, 32, 255, 0, 0, 0, 0, 0] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 195, 167, 184, 23, 0, 0, 0, 1, 9, 0, 1, 0, 98, 88, 209, 154, 255, 255, 255, 255, 0, 0, 105, 56, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 71, 7, 2, 2] }
[-24.74382, -24.64382, -24.64382, -24.543821, -24.44382, -24.44382, -24.34382, -24.24382, -24.24382, -24.14382]

Incorrect

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Temperature
  Parameter:                            Temperature
  Generating Proceess:                  Forecast
  Forecast Time:                        0
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     30
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     Missing
  2nd Scaled Value:                     255
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 675339 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 133, 93, 74, 128, 0, 0, 0, 0, 48, 5, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 64] }
ProdDefinition { payload: [0, 0, 0, 0, 0, 0, 2, 255, 16, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 0, 30, 255, 255, 0, 0, 0, 255] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 70, 174, 52, 0, 0, 0, 0, 2, 8, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 89, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 64, 6, 2, 2] }
[252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28]

Additional Context
Additionally, the Parameter Category and Parameter are seemingly incorrectly parsed as they should both be Momentum, u-component of wind and the incorrect file produces Temperature, Temperature.

@LafeWessel Thank you for reporting.
Since I am a little confused, let me ask a question: what does "incorrect values" mean?
Does "Missing" seem to be incorrect? Or, decoded values are incorrect?
Note
I am sorry if you are familiar with fixed surfaces in GRIB2, but I would like to explain a little about it here.
Fixed surfaces
Each "layer" (or submessage) in GRIB2 has one or two fixed surfaces. Since most layers express values at just one altitude level, they have only one fixed surface (e.g. humidity at 700hPa, temperature at 30cm depth, etc.). For such layers, the second fixed surface is missing.
However, there are cases that it would be nice for layers to have the second fixed surface. Fox example, some values are defined for layers with a certain thickness. Actually, I have not seen so much data that has a second fixed surface, so I asked ChatGPT to explain its usage to me; according to ChatGPT, the second fixed surface has the following uses (I am not sure how much of a lie this explanation contains).
Me:

Each message in GRIB2 files contains 1 or 2 fixed surfaces. In what cases is the second fixed surface used?

ChatGPT:

In GRIB2 files, each message can indeed contain one or two fixed surfaces. The second fixed surface is typically used to represent a secondary level or layer of the atmosphere or Earth's surface that is relevant to the meteorological data being stored.
The second fixed surface is often employed in the following scenarios:

Vertical interpolation: In some cases, meteorological variables may need to be interpolated or derived between two fixed surfaces to provide a more complete representation of the atmosphere. For example, temperature data may be provided at both 1000 hPa (first fixed surface) and 850 hPa (second fixed surface) to allow for vertical interpolation and calculation of temperature values at different pressure levels.

Layer variables: Certain meteorological variables are defined for specific layers or thicknesses in the atmosphere. In such cases, the first fixed surface represents the lower boundary of the layer, while the second fixed surface represents the upper boundary. This allows for the representation of variables like the thickness of a layer, cloud cover within a specific layer, or precipitation accumulation over a particular layer.

Earth's surface and pressure levels: The second fixed surface can also be used in combination with the Earth's surface or pressure levels. For example, a message might include data at the Earth's surface (e.g., surface temperature) as the first fixed surface, and then provide data at a specific pressure level (e.g., 500 hPa) as the second fixed surface.

The inclusion of a second fixed surface in a GRIB2 message enhances the flexibility and completeness of the data by accommodating various vertical representations and calculations. It allows for more comprehensive analysis and modeling of the atmosphere or Earth's surface based on different levels, layers, or transitions between fixed surfaces.

Expressions of fixed surfaces
When a layer does not have the second fixed surface, that fixed surface is expressed as "missing". That status is expressed by setting code 255 to the type of that fixed surface. So, the following means 2nd fixed surface is missing. (I think it would be nice to express the type as "Missing" instead of "code '255' is not implemented"...)
  2nd Fixed Surface Type:               code '255' is not implemented

When a fixed surface is missing, the scale factor and scaled value are usually set to 0xff and 0xffffffff, but there are cases where the scale factor and scaled value are both set to 0. Anyway, following two layers both have a missing second surface:
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0

  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     Missing
  2nd Scaled Value:                     255

Scale factor and scaled value of fixed surface
With a fixed surface type, scale factor and scaled value are used like this:
975 hPa
(975 * 10 ** (-(-2)) Pa (The unit of isobaric surface is defined to be Pa))
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     -2
  1st Scaled Value:                     975

Mean sea level:
  1st Fixed Surface Type:               Mean sea level
  1st Scale Factor:                     Missing
  1st Scaled Value:                     Missing

1.5 m high above the ground
(15 * 10 ** -1 m)
  1st Fixed Surface Type:               Specified height level above ground
  1st Scale Factor:                     1
  1st Scaled Value:                     15

Thanks for the clarification! I confess I do not know that much about the Fixed Surfaces - though I do now.
My report is specifically about incorrect values being parsed from the messages, not a missing Fixed Surface. Though I don't know how the parsing works internally, my thought was that the 255 from the 2nd Scaled Value was "polluting" the values that were parsed from the message. Here's my reasoning:
From the correct message, we see:
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     20000
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0

which produces the following (first ten) values:
[-24.74382, -24.64382, -24.64382, -24.543821, -24.44382, -24.44382, -24.34382, -24.24382, -24.24382, -24.14382]

For the incorrect message, we see:
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     30
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     Missing
  2nd Scaled Value:                     255

which produces the following (first ten) values:
[252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28]

Both of these messages should produce roughly the same data, but they are clearly very far off from one another. My thought was that the 255 from the 2nd Scaled Value from the incorrect message caused the values parsed to be incorrect as they are rather close to 255. Whereas the correct message has 0 for a 2nd Scaled Value and produces the correct values.
I would guess that the 2nd Scale Factor is missing, but my hunch was/is that the placeholder of 255 for the missing 2nd Scaled Factor causes different values to be parsed.
Does that make any more sense, or have I misunderstood what your comment says/how it works internally?

Thank you. I understand the intent of your report. However, so far it doesn't sound like there is a problem with the library. Here are two reasons why I think so.
Section 4 is not used in decoding
First, the information contained in Section 4 (Product Definition Section), such as fixed surfaces, should not affect decoding. A GRIB2 message consists of following sections:

Section 0 (Indicator Section)
b"GRIB", GRIB version, message size, etc.
Section 1 (Identification Section)
Information about the data source, such as issuing authority, reference time, etc.
Section 2 (Local Use Section)
Information defined by each local authorities.
Section 3 (Grid Definition Section)
Definition of grid system. The type of grid system (e.g., equirectangular, Lambert, etc.) and the parameters that specifically define each grid system (e.g., longitude and latitude at the four corners, number of grid points in the latitudinal and longitudinal directions, etc.).
Section 4 (Product Definition Section)
Definition of the meaning of the included values. For example, meteorological elements, fixed surface (altitude level), forecast time, etc.
Section 5 (Data Representation Section)
Definition of what representation is used for including grid point values. Simply put, it defines the compression method and the parameter values used to compress the data with that method.
Section 6 (Bit-map Section)
A bitmap representing a series of missing values in the data (array). The information about whether each grid point value is a missing value or not is separated from the array of grid point values and stored in this section, and only the grid point values other than missing values are compressed using the method defined in Section 5 and stored in Section 7.
Section 7 (Data Section)
The main body of data compressed by the method defined in Section 5. Some parameters are not included in Section 5, but are included here.
Section 8 (End Section)
b "7777".

Of these 9 sections, only Sections 3, 5, 6, and 7 are required to decode grid point values. Information on meteorological elements, fixed surfaces, and forecast times are included in Section 4 function as data attributes and are necessary to understand the meaning of the data, but are not used in the decoding process.
Values
Second, please note the following points in your data.
  Parameter:                            u-component of wind
(snip)
[-24.74382, -24.64382, -24.64382, -24.543821, -24.44382, -24.44382, -24.34382, -24.24382, -24.24382, -24.14382]

  Parameter:                            Temperature
(snip)
[252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28]

The u component of the wind is positive for the east and negative if the wind blows to the west. Also, it is not surprising that wind speeds of -24 m/s can be observed regularly at high altitudes.
The temperature cannot be lower than absolute zero, and it is reasonable to assume that the meteorological data is within 273±100 K. Therefore, the value of 252 K is not particularly strange.

This is very helpful, thank you. After further poking around in the code that I have, I realized a couple things:

The messages were not matching up how I anticipated, so for the example messages that I expected to be "u-component", only one of them actually was. None were being read incorrectly.
The lat-lon grid I was using is flipped on the y-axis for the second set of data I have, so the logic I used to parse it out was showing correct data when I tested at 0 latitude but was incorrect anywhere else.

Thus, the bug is on my end in how I consume the GRIB data, there is nothing wrong with your library! Thanks for all the help, sorry for wasting your time.


Fix: make chatbot functional

please host your api, I'm using my own personal api

Hey @Sheriff-Oladimeji, thank you for contributing! I noticed some issues with the bot

It holds a persona of John Smith, Software Engineer
The bot sometimes doesn't answer, and we have to input the question again.

Maybe you could look into it. The prompt for the bot being a sex education can be found in server.js file

Hey @Sheriff-Oladimeji, thank you for contributing! I noticed some issues with the bot

It holds a persona of John Smith, Software Engineer
The bot sometimes doesn't answer, and we have to input the question again.

Maybe you could look into it. The prompt for the bot being a sex education can be found in server.js file
Other than that, awesome work!

Sure, I'll look into it. Thanks for your feedback.


Add higher quantization level for kNN vector search

Description
There have been multiple discussions around supporting higher vector quantization levels (e.g. more than scalar quantization can provide).
We (elastic) have been doing some research around what the best next step will be for higher levels of quantization in Lucene.
A couple of months ago we came across an interesting paper: https://arxiv.org/abs/2405.12497 and have been actively experimenting with RaBitQ.
Its approach to quantization fits well with Lucene:

Higher levels of compression (32x for floating point)
Very cheap quantization costs
Fast vector similarities
Lower levels of reranking required

In our investigations we have found:

RaBitQ achieves the same or better recall at the same compression ratios as Product Quantization.
RaBitQ is 20-30x faster at quantizing
RaBitQ (in Java) is more than 2x faster to query than PQ (can likely be made faster, where PQ cannot really...)

Let me share some PoC numbers. All our testing with PQ assumed a 32x reduction. Higher levels of reduction then require more reranking, which then becomes the main bottleneck. Our goal was to achieve good recall at only 5x reranking and without Optimized PQ (which is insanely expensive, the PQ already is very expensive).
30M cohereV3 (on a GCP intel machine avx512):

RaBitQ:

Time to quantize vectors: 998363ms
Avg Brute-force query time for Recall@10:50 1776ms with 98% recall
Time to build HNSW graph: 40043229ms

HNSW recall@10:50: 86.2% at 1641qps

PQ:

Time to calculate codebooks: 472260ms
Time to quantize vectors: 12644293ms
Avg brute-force query time for Recall@10:50 5790ms with recall 98% recall.
WE didn't bother building the large graph here as, frankly, its PQ is just too slow.

522k Quora encoded with e5Small (noted for being horrible with binary quantization & higher compression ratios in general) (On my macbook, Arm M1-MAx)

RaBitQ:

1051ms to quantize vectors
Avg Brute-force query time for Recall@10:50 11ms with 99% recall

PQ:

28360ms to build the code books
31364ms to encode the docs
Avg Brute-force query time for Recall@10:50 19ms with 99% recall

Consequently, given all the various trade-offs and benefits, we are working on bringing a Rabitq format into Lucene.
I will open a draft PR soon to make the work more visible.

This sounds super promising!  There is some discussion here about PQ and PCA as well.

Ben, is this what you referred to in your Elastic article?  I read that, got interested, but had trouble ascertaining which Lucene version has this feature.  I suppose it's unreleased?  It would have b been helpful if you clarified the state in that article.

@dsmiley I am sorry for dragging my feet here :(
It is not released yet, the PR has been open for some time, but there arose some weird recall behaviors we wanted to fix & consequently discovered an even better way to quantize at 32x since the original work mentioned. I expect to update the PR for Lucene over the holidays and aim to get it in for 10.2.0: https://github.com/apache/lucene/pull/13651


dispatch_protocol_error: type 90 seq

Hi guys,
I'm having trouble with connecting to any ssh server with your framework. Please find my code below:
NMSSHSession *session = [NMSSHSession connectToHost:@"server"
                                           withUsername:@"username"];
NSLog(@"Auth methods: %@", session.supportedAuthenticationMethods);

if (session.isConnected) {
    [session authenticateByPassword:password]
    NSLog(@"lastError: %@", [session lastError]);
    if ([session isAuthorized]) {
        NSLog(@"Authentication succeeded");
    }
}

However, in my server logs, only the following line appears when trying to execute the above:
dispatch_protocol_error: type 90 seq 7 [preauth]

The applications log says:
2014-01-31 12:22:23.422 Dropshare[31359:1003] NMSSH: libssh2 (v1.4.2) initialized
2014-01-31 12:22:23.422 Dropshare[31359:1003] NMSSH: Start server resolution
2014-01-31 12:22:23.856 Dropshare[31359:1003] NMSSH: Socket connection to <ip> on port 22 succesful
2014-01-31 12:22:24.079 Dropshare[31359:1003] NMSSH: Remote host banner is SSH-2.0-OpenSSH_5.9
2014-01-31 12:22:24.080 Dropshare[31359:1003] NMSSH: The host's fingerprint is <fingerprint>
2014-01-31 12:22:24.080 Dropshare[31359:1003] NMSSH: SSH session started
2014-01-31 12:22:24.117 Dropshare[31359:1003] NMSSH: User auth list: publickey,keyboard-interactive
2014-01-31 12:22:24.117 Dropshare[31359:303] Auth methods: (
    publickey,
    "keyboard-interactive"
)
2014-01-31 12:22:24.146 Dropshare[31359:1003] NMSSH: User auth list: publickey,keyboard-interactive
2014-01-31 12:22:24.179 Dropshare[31359:1003] NMSSH: Public key authentication failed with reason -19
2014-01-31 12:22:24.179 Dropshare[31359:303] lastError: Error Domain=libssh2 Code=-19 "Callback returned error" UserInfo=0x6000000709c0 {NSLocalizedDescription=Callback returned error}

Any hints on this?
 So
`NSString *privateKey = [[NSBundle mainBundle]<EMAIL_ADDRESS>ofType:nil];
             [self.session authenticateByPublicKey:nil
                                        privateKey:privateKey
                                       andPassword:@"passwordForThePrivKey"];`

works.


Experiment in lock-free module invalidation

Pull Request Description

Experimenting with invalidating modules' indexes without requiring full write-context locks. That should significantly improve the execution.
Improving performance by making background job executor run in a larger threadpool than 1.

Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[ ] The documentation has been updated, if necessary.
[ ] Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
[ ] All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:

[ ] Unit tests have been written where possible.
[ ] If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Please find some smaller issues spread as inline comments. The bigger concern is expressed here.

This was a useful feedback although I don't believe I have made it worse than it used to be. Previously we would use a rather broad writeCompilationLock with little consideration on how that affects scheduling of tasks.
Either way, I extracted the logic out of the runtime Module, as I agree that this is not an ideal place for indexing flag. Instead, similarly to PendingEdits, it is now carried around in RuntimeContext. Let me know what you think.

I extracted the logic out of the runtime Module, as I agree that this is not an ideal place for indexing flag. Instead, similarly to PendingEdits, it is now carried around in RuntimeContext.

From a code perspective, such a change doesn't have real impact on the behavior. Whatever flaws were present so far, are still present. The only real impact is organizational one: This PR moves the code away from engine/runtime project - the changes in engine/runtime project are non-controversial now.


Define "MicroProfile Compatibility"

From the "additional goals" category of the program plan, but ties into other goals as well.

decided with
[microprofile-wg] [BALLOT] Replace MicroProfile Compatibility Logo with.pdf


Additional Tests

This PR adds a few additional tests to round out the test suite.

Rebased with master, and deleted a test that's no longer needed.


iOS devices can hit a VRAM limit with 8k textures

as tested with the iPad gen9.
Should somehow write a warning about this and auto handle the case, via downsizing.

The issue was a bit more involved actually, since decoding may or may not also perform a texture allocation. Seemingly fixed by deleting the texture before trying to decode a new image.


Onboard ops for testing to test flight deployments

onboard operations team with iOS and android devices to be able to poke around the app on those devices
ensure rendering is correct across app and all screens (responsive, etc)
no other functionality needs to be tested at the moment

operations is able to access app and is available for testing.


az apim list fails with ModuleNotFoundError

Describe the bug
az apim list fails with ModuleNotFoundError using version 2.0.72 of the CLI, which I installed yesterday.
Command Name
az apim list
Errors:
No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
Traceback (most recent call last):
pip-install-qxmmnr17\knack\knack\cli.py, ln 206, in invoke
cli\core\commands\__init__.py, ln 523, in execute
azure\cli\core\__init__.py, ln 291, in load_arguments
cli\core\commands\__init__.py, ln 290, in load_arguments
pip-install-qxmmnr17\knack\knack\commands.py, ln 97, in load_arguments
azure\cli\core\__init__.py, ln 488, in default_arguments_loader
azure\cli\core\__init__.py, ln 528, in get_op_handler
...
<frozen importlib._bootstrap>, ln 955, in _find_and_load_unlocked
<frozen importlib._bootstrap>, ln 665, in _load_unlocked
<frozen importlib._bootstrap_external>, ln 678, in exec_module
<frozen importlib._bootstrap>, ln 219, in _call_with_frames_removed
cli\command_modules\apim\custom.py, ln 9, in <module>
ModuleNotFoundError: No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.
-No prerequisites_

az apim list

Expected Behavior
The CLI would list the API Mangement resources in my subscription.
Environment Summary
Windows-10-10.0.17134-SP0
Python 3.6.6
Shell: cmd.exe

azure-cli 2.0.72

Additional Context

Environment Summary
Windows7 Enterprise 6.1.7601 64-bit
Python 3.6.6
Shell: powershell.exe
azure-cli 2.0.72
Installer was run with a privilege limited user, then UAC prompt was authorized with a different user with admin rights.

Just ran into this now on windows server 2019. I'm running inside a VM on a private subnet (same vnet as APIM, different subnet). Also 2.0.072.

This is a blocker for me. Any updates?

Any update? I also ran into this issue on Windows 10.
Windows-10-10.0.18362
Python 3.6.6
Shell: cmd.exe
azure-cli 2.0.73

Same problem as #10496? I made a pull request that should fix this: #10623

I'm also having this issue. az apim list fails with:

No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-mjd5h3hf\knack\knack\cli.py", line 206, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-mjd5h3hf\azure-cli-core\azure\cli\core\commands\__init__.py", line 523, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-mjd5h3hf\azure-cli-core\azure\cli\core\__init__.py", line 294, in load_arguments
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-mjd5h3hf\azure-cli\azure\cli\command_modules\apim\__init__.py", line 28, in load_arguments
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-mjd5h3hf\azure-cli\azure\cli\command_modules\apim\_params.py", line 11, in <module>
ModuleNotFoundError: No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'

Version of CLI is:

command-modules-nspkg              2.0.3
core                              2.0.77
nspkg                              3.0.4
telemetry                          1.0.4

Extensions:
aks-preview                       0.4.17
azure-cli-iot-ext                  0.8.6
azure-devops                      0.16.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\joamart\.azure\cliextensions'

Python (Windows) 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.```

I'm running into this same issue.  Same versions as the last comment.

Just run into this issue as well. Using:
`azure-cli                         2.0.78
command-modules-nspkg              2.0.3
core                              2.0.78
nspkg                              3.0.4
telemetry                          1.0.4`

Same here:
azure-cli                         2.0.81
command-modules-nspkg              2.0.3
core                              2.0.81
nspkg                              3.0.4
telemetry                          1.0.4

same here with:
PS C:\Users\Niek> az apim delete
No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
Traceback (most recent call last):
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\knack\knack\cli.py", line 206, in invoke
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli-core\azure\cli\core\commands_init_.py", line 528, in execute
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli-core\azure\cli\core_init_.py", line 300, in load_arguments
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli\azure\cli\command_modules\apim_init_.py", line 28, in load_arguments
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli\azure\cli\command_modules\apim_params.py", line 11, in
ModuleNotFoundError: No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
To open an issue, please run: 'az feedback'

same here with az apim delete:
No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
Traceback (most recent call last):
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\knack\knack\cli.py", line 206, in invoke
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli-core\azure\cli\core\commands_init_.py", line 528, in execute
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli-core\azure\cli\core_init_.py", line 300, in load_arguments
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli\azure\cli\command_modules\apim_init_.py", line 28, in load_arguments
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-0u_uiwkq\azure-cli\azure\cli\command_modules\apim_params.py", line 11, in
ModuleNotFoundError: No module named 'azure.mgmt.apimanagement.models.api_management_client_enums'
To open an issue, please run: 'az feedback'
version info: az --version
C:\WINDOWS\system32>az --version
azure-cli                          2.1.0
command-modules-nspkg              2.0.3
core                               2.1.0
nspkg                              3.0.4
telemetry                          1.0.4
Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\Niek.azure\cliextensions'
Python (Windows) 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.

Would really like an update on this. Same issue as everyone else.

@tehyatt Looks like this has since been fixed and I can confirm that there is no error now. Apologies for no updates on this issue sooner. Unless you are still seeing this error on your end, I believe this could be closed.


Pin nightly version to a latest one known to be working

Lightweight version of #131 - it'd be good to start with a good working base and incrementally work from there.
r? @Manishearth

It's worth noting that I used the same cargo bisect-rustc tool from #131 but with test script rather than check - there were some changes wrt Eq and StructuralEq that needed addressing in nightly-2020-06-10 suggested there; atm this version (nightly-2020-06-01) is what builds and passes current tests.

Hmm, CI still fails, but r=me if you can get it to pass

That's because it tests using the newest nightly with rustup-toolchain-install-master: https://github.com/rust-dev-tools/rust-semverver/blob/0bbdabfc883a4e5c4d6dce21800b01649f0a70fa/ci/run.sh#L10-L21
If you don't mind, I'd prefer to change the CI script to just cargo test (that accounts for rust-toolchain) and then re-introduce the daily CI check using rustup-toolchain-install-master later at some point

CI is green now, merging as per above


Branch not shown for trains

I'm monitoring southbound trains from West Finchley station.
There is no information if trains are via Bank or Charring Cross.
Such information is more relevant than the destination itself.

You are correct! That would be very useful. I had a couple of minutes just now to confirm if the API has that info, and it does.
And, to my luck, past me left the code in a nice state that changing what is used is trivial. I'll push a patch imminently.

https://github.com/morosanmihail/HA-LondonTfL/releases/tag/v0.4.3 should fix it. If it doesn't, I blame the nasty cold I'm currently nursing.

There we go, v0.4.4 should properly fix it. If you see anything wrong, do tell me!

Amazing!
This was probably the fastest fix ever!
Thanks a lot.

No, thank you for raising the issue! Take care!


make it easier to "prune" the scene graph without leaking memory

In https://github.com/phetsims/joist/issues/332, HomeScreenView was leaking memory.  It keeps and reuses references to the screen icons, and puts them in a different HBox, depending on which screen is currently selected.  And unless removeAllChildren is called on the HBox, the icons will retain a reference their parent - in this case, a lot of HBox instances.
I suspect that I could find this type of bug elsewhere in PhET code. The fact that children keep a reference to their parent is (rightfully) transparent.  But it makes to "pruning" some portion of the scene graph awkward and error prone.
@jonathanolson What do you think about adding something to Node like pruneChild and pruneAllChildren, which removes parent/child references for all ancestors?

I suspect that I could find this type of bug elsewhere in PhET code.

It's been the source of multiple memory leaks that I've fixed.

@jonathanolson What do you think about adding something to Node like pruneChild and pruneAllChildren, which removes parent/child references for all ancestors?

So instead of disconnecting a node from all of its parents (e.g. node.detach()), it would also recursively detach all ancestors from their respective parents? Sounds very easy to accidentally nuke screenview connections in a hard-to-debug way.
I've considered some form of dispose() that would work for an entire subtree of nodes, such that they all disconnect, but that can't really be used with the DAG well.

I was thinking that it would recursively call removeAllChildren for all ancestors.  Potential problems with that?

I'd like to see an example of usage, as you would need to know it is disconnected from the ScreenView before calling anyways?

Why would it need to be disconnected from the ScreenView?...
Here's a concrete example, from HomeScreenView.  This replaces lines 253-278. this is the HomeScreenView, a subtype of ScreenView.
var largeIcons = [ … ]; // array of scenery.Image, large icons for the screens
var smallIcons = [ … ]; // array of scenery.Image, small icons for the screens

var iconsParentNode = new Node();

this.addChild( iconsParentNode );

sim.screenIndexProperty.link( function( screenIndex ) {

  // prune children, so that elements of largeIcons and smallIcons aren't left with references HBox
  iconsParentNode.pruneAllChildren();

  // create an array with the appropriate large and small icons
  var icons = _.map( screenChildren, function( screenChild ) {
    return ( screenChild.index === screenIndex ) ? largeIcons[ screenIndex ] : smallIcons[ screenIndex ];}
  );

  // us an HBox to handle the layout of icons
  iconsParentNode.addChild( new HBox( {
    spacing: spacing,
    children: icons,
    align: 'top',
    resize: false }
  ) );

} );

So iconsParentNode.pruneAllChildren() as initial described (handling ancestors), would:

iconsParentNode.removeAllChildren()
this.removeAllChildren(), since it was an ancestor.
If 'this' was being displayed by the screenview, then you get:
a. screenView.removeAllChildren() (yikes)
b. rootNode.removeAllChildren() (there go our screens)

If instead it did removeAllChildren on descendants, we would often run into this case:
var firstImage = require( 'image!SOMETHING/first.png' );
var secondImage = require( 'image!SOMETHING/second.png' );

// Shared image nodes, for better memory usage
// (or maybe they were passed in from somewhere)
var firstImageNode = new Image( firstImage );
var secondImageNode = new Image( secondImage );

var largeIcons = [
  new Node( { children: [ firstImageNode ], scale: 2 } ),
  new Node( { children: [ firstImageNode ], scale: 2 } )
];
var smallIcons = [
  new Node( { children: [ firstImageNode ], scale: 0.5 } ),
  new Node( { children: [ firstImageNode ], scale: 0.5 } )
];

Then we run into the case that iconsParentNode.pruneAllChildren() mucks up the scaled nodes by removing their children (breaking things). This also happens in the general case where each reused icon is not a single node (say we used AlignBox/AlignGroup as described in that issue).
It seems better to just keep a reference to the hbox and mutate its children. Need to dispose or wipe it? Just set hbox.children = [].

Then if we can get rid of resize: false (which is usually the case):
var largeIcons = [ … ]; // array of scenery.Image, large icons for the screens
var smallIcons = [ … ]; // array of scenery.Image, small icons for the screens

var iconsParentNode = new HBox( {
  spacing: spacing,
  align: 'top'
  // Why was resize: false? Usually not needed?
} );
this.addChild( iconsParentNode );

sim.screenIndexProperty.link( function( screenIndex ) {
  iconsParentNode.children _.map( screenChildren, function( screenChild ) {
    return ( screenChild.index === screenIndex ) ? largeIcons[ screenIndex ] : smallIcons[ screenIndex ];}
  );
} );

I had previously tried resize: false and hbox.children = [].  This causes the small icons to jump around, instead of remaining stationary.  There might've been a way to resolve that, but it wasn't a problem I felt like tackling.
I'll withdraw the request for this feature. Closing.


Remove nightly feature

[X] Remove nightly feature that is not really needed
[x] Fix Clippy lints
[X] Make Generator and relevant code generic over AsRef<Path> (this should improve egonomics)

Thank you soo much for the contribution
I'll review it later in the day

About the new crate you added, it'd would more efficient if you could incorporate it into the current gen_attribute crate. This is because crate.io doesn't accept path used to indicate the path of the crate

So I'll merge your branch but I'll drop the drain vectors commit since using drain filter is more efficient


Classes should follow PSR-4

Classes in src/ folder do not follow PSR-4 for namespace and filenames. It causes issues with auto wiring.
E.g. src/Entity/AutoCompleteResult.php should be class AutoCompleteResult in namespace Mathematicator\Search\Entity (instead of current: Mathematicator\Search).

Nette also does not use the PSR-4 for autoloading classes.
I think this is a big BC break that has to be done carefully, due to the compatibility of other packages.

So, I am not able to autowire services in Symfony.
Any thoughts, how to solve this?

My services.yaml is:
# ... other config ...
    Mathematicator\Search\:
        resource: '../vendor/mathematicator-core/search/src/*'

E.g. namespace: Mathematicator\SearchController for src/Controllers does not make sense. I think it should be refactored as a new version ASAP. This package is used only by your packages at this time, so it is relatively easy to refactor with PHPStorm refactoring tool.


How to ensure the correctness

Hi, I'm very curious that how do you ensure the correctness of all the algorithms in MMDetection when update the framework? I've seen the test fold in the root python, but can they cover all the border situations and all the modules?

@liliangqi Before each version release, we will benchmark the algorithm to ensure the correctness of performance.

@hhaAndroid Did you train all the algorithms or just perform inference using the trained models?

Yes


Fixed emitLegacyCommonJSImports in getImports adding JS when truthy r…

Related #8815
Description
Whenever generating hooks with @graphql-codegen/typescript-react-apollo and the following config options:
emitLegacyCommonJSImports: true
documentMode: "external"
importDocumentNodeExternallyFrom: "near-operation-file"

It's appending .js to the Operations import like so (but it shouldn't):
import type * as Types from '../../../../shared/types/graphqlGenerated.d';

import * as Operations from './Foobar.query.js';

The issue originates in @graphql-codegen/visitor-plugin-common specifically this line where the condition checking if emitLegacyCommonJSImports should be inverted according to other examples within the codebase like here.
The solution here is to just append a ! to the condition such that it adds the .js only when emitLegacyCommonJSImports is false.
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
I've included specs with this path, these can be used to confirm that there's a bug in visitor-plugin-common currently where a condition checking emitLegacyCommonJSImports just needs to be inverted.
Test Environment:
See linked issue if you want more information.
Checklist:

[x] I have followed the CONTRIBUTING doc and the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas (N/A)
[] I have made corresponding changes to the documentation (N/A)
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Further comments
Super simple change, I didn't create a reproduction here because in the linked issue, I reference locations in the codebase where the code is correct and how it differs from this specific location.

can you add a changeset (learn more here https://github.com/dotansimha/graphql-code-generator/pull/8816#issuecomment-1382872852) aslo seems like prettier is failing?

can you add a changeset (learn more here #8816 (comment)) aslo seems like prettier is failing?

Yup sorry about that, wanted to do it right away but got distracted. Hopefully I did it properly :).
Also the prettier issue seems to be something unrelated to my changes but I've committed the formatting fix. Let me know if you want me to undo that or if it's ok to include.
Thanks for looking at this, appreciate it!


Add initial Draw.IO flowchart

This adds the Draw.IO diagram to this repository. Accessible via https://draw.io
Authorise GH -> Select repo -> Select branch -> Select file
(27-add-flowchart -> airseeker_v2_pipeline.drawio)
Closes #27

This should be documented in README how to access it.
Also can we make:
Authorise GH -> Select repo -> Select branch -> Select file
(27-add-flowchart -> airseeker_v2_pipeline.drawio)

somehow encoded in the URL ? Such that when you click a link you will get the file displayed immediatelly?

Actually now that I think about it... If I edit this diagram using the diagrams URL the content in the repo will not be updated.
It makes it pretty annoying to always make a PR whenever we change the diagram. I'd just put the URL in README and be done with it.

An alternative solution is to use Mermaid and embed the diagram URL in the README.md

Mermaid

Mermaid looks very cool 🚀

Diagrams/Draw.IO has a URL export function and various other tools that could probably be used to automate things, and can be used for embedding too... but ultimately imo we can't spend too much time on this 🤷

LGTM 🤷 I can't reveiw because I created the PR :D

Authorise GH

Yikes, I'd recommend having it read from https://raw.githubusercontent.com/api3dao/airseeker-v2/b57506ffeafb45dcc79bbce083798de0f759323d/airseeker_v2_pipeline.drawio instead

This looks out of date. What do we do about it?

I'd just put the link to README, the link always shows the up to date content.


error "OpenCV 4.x+ requires enabled C++11 support"

/usr/include/opencv4/opencv2/core/cvdef.h:690:4: error: #error "OpenCV 4.x+ requires enabled C++11 support"
error "OpenCV 4.x+ requires enabled C++11 support"
thannk you !!

Enable C++11 support. Please provide more background info if that does not solve your issue.


Authorization via Environment Variables?

Please add support for authorizing Gmail Object via environmental variables for services like Heroku that recommend using Twelve-Factor App principles, e.g., storing config data in the environment instead of files.

While trying to figure out a good way to Dockerize a simple cronjob that checks email to import some stats (its a work around for their API giving only root access to the WAF.) I was going to go down a path of trying to store the creds in a DB. oauth2client does have a SQLacademy storage module but implementing that and pushing it through seems harder than just fixing this 3 years after it was requested and using Docker secrets to store the refresh token and credentials.
TL;DR; might submit a pr for this soon.


Updated host-utils to automatically include SPA base

Codecov Report

Merging #74 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #74   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          23     23
  Lines         505    512    +7
  Branches       56     58    +2
=====================================
+ Hits          505    512    +7

Impacted Files
Coverage Δ

utils/host-utils.js
100% <100%> (ø)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8833678...5135c47. Read the comment docs.


Exception on flutter v1.5

Running an exception on flutter v1.5, it seems that we need to use Main Thread to callback the result of MethodChannel/EventChannel.
W/System.err( 5858): java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: signaling_threa - 5899
W/System.err( 5858):    at io.flutter.embedding.engine.FlutterJNI.ensureRunningOnMainThread(FlutterJNI.java:605)
W/System.err( 5858):    at io.flutter.embedding.engine.FlutterJNI.dispatchPlatformMessage(FlutterJNI.java:515)
W/System.err( 5858):    at io.flutter.embedding.engine.dart.DartMessenger.send(DartMessenger.java:74)
W/System.err( 5858):    at io.flutter.embedding.engine.dart.DartExecutor.send(DartExecutor.java:149)
W/System.err( 5858):    at io.flutter.view.FlutterNativeView.send(FlutterNativeView.java:142)
W/System.err( 5858):    at io.flutter.plugin.common.EventChannel$IncomingStreamRequestHandler$EventSinkImplementation.success(EventChannel.java:223)
W/System.err( 5858):    at com.cloudwebrtc.webrtc.PeerConnectionObserver.sendEvent(PeerConnectionObserver.java:332)
W/System.err( 5858):    at com.cloudwebrtc.webrtc.PeerConnectionObserver.onRenegotiationNeeded(PeerConnectionObserver.java:425)

It's not the only place where it crashes, it's a lot of calls everywhere
If it was Kotlin, I would create extension method like .successOnUiThread(), something like that
But in Java wrapping every single single in static method...

Suggestion, wrap every Result object in this:
public final class AnyThreadResult implements MethodChannel.Result {
    final private MethodChannel.Result result;

    AnyThreadResult(MethodChannel.Result result) {
        this.result = result;
    }

    @Override
    public void success(Object o) {
        final Handler handler = new Handler(Looper.getMainLooper());
        handler.post(()->result.success(o));
    }

    @Override
    public void error(String s, String s1, Object o) {
        final Handler handler = new Handler(Looper.getMainLooper());
        handler.post(()->result.error(s, s1, o));
    }

    @Override
    public void notImplemented() {
        final Handler handler = new Handler(Looper.getMainLooper());
        handler.post(result::notImplemented);
    }
}

Yes, and eventSink also needs the same processing. All webrtc events are triggered by the source_thread from the native. Basically all result and eventSink need to be processed.

Well, unless someone wants to do that, I will start working on it in few days

Fixed https://github.com/cloudwebrtc/flutter-webrtc/commit/a15119730e3f4652fdf4451333ab2a11a9ebaade


chore(package.json): update package version to v0.3.1

Codecov Report

Merging #52 (fd732f5) into dev (b1ed8c8) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               dev       #52   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            5         5
  Lines          310       310
  Branches        23        23
=========================================
  Hits           310       310


: redux thunk exercise

Drift:
https://video.drift.com/v/ab3YAaogikNmPDHMo8uQsJDQTn9EXbHSa04m88Rcog/

The commit message's format must be proper
Work on the directory structure
Use separate files for the interfaces, constants, etc.
Modularize the code


KYLIN-4098: Add a cube auto merge api

See: https://issues.apache.org/jira/browse/KYLIN-4098

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@d659542). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master     #761   +/-   ##
=========================================
  Coverage          ?   25.71%
  Complexity        ?     6021
=========================================
  Files             ?     1387
  Lines             ?    82796
  Branches          ?    11594
=========================================
  Hits              ?    21288
  Misses            ?    59462
  Partials          ?     2046

Impacted Files
Coverage Δ
Complexity Δ

...java/org/apache/kylin/metadata/model/Segments.java
0% <0%> (ø)
0 <0> (?)

...ava/org/apache/kylin/rest/service/CubeService.java
0% <0%> (ø)
0 <0> (?)

...g/apache/kylin/rest/controller/CubeController.java
0% <0%> (ø)
0 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d659542...ad075cc. Read the comment docs.


Add: Option to select and batch approve file items

Not needed as a dynamic approval approach was taken instead


[bug] Set-commands from MQTT not working

Version
Build/Run method

[x] Docker
[ ] PKG
[ ] Manually built (git clone - npm install - npm run build )

zwavejs2mqtt version: 1.0.0-rc.1
zwavejs version: 6.0.0
Describe the bug
MQTT "set"-messages don't work. Triggering the same from the web-ui works.
To Reproduce
Set Gateway to payload-type JSON time-value
Example 1 (Zipato | RGBW Bulb V2)
Current Value:

Topic: raw/z-wave/LB__01/38/0/Up
Payload: {"time":1611248166243, "value": false}
Setting Value:
Topic: raw/z-wave/LB__01/38/0/Up/set
Payload: { "value": true }
Log:
2021-01-21 21:00:41.436 INFO MQTT: Message received on raw/z-wave/LB__01/38/0/Up/set, '{ "value": true }'

Example 2 (Fibargroup | Single Switch 2)
Current Value:

Topic: raw/z-wave/IWS_01/37/1/CurrentValue
Payload: {"time":1611251607553,"value":false}
Current Value:
Topic: raw/z-wave/IWS_01/37/1/TargetValue
Payload: {"time":1611248166006,"value":false}
Setting Value:
Topic: raw/z-wave/IWS_01/37/1/targetValue/set
Payload: { "value": true }
Log:
2021-01-21 21:06:55.624 INFO MQTT: Message received on raw/z-wave/IWS_01/37/1/targetValue/set, '{ "value": true }'

Expected behavior
The MQTT set messages should work.
Additional context

I recently switched from zwave2mqtt to zwavejs2mqtt
The MQTT-broker in use is mosquitto:2.0.
As zwave(js)2mqtt doesn't support multipart mqtt-topic prefixes I added an additional endpoint for zwave(js)2mqtt which has raw/ as mount-point. In my opinion zwavejs2mqtt shouldn't actually see the topics as raw/z-wave/# but only z-wave/#, but the logs show differently. This setup worked in zwave2mqtt

The idea I had while writing the "Additional context" seems to be part of the issue. I reconfigured to remove the raw/ mount-point. and when trying to set my first example from above actually leads to further log-messages:
2021-01-22 10:09:03.988 INFO MQTT: Message received on z-wave/LB__01/38/0/Up/set, '{\n "value": true\n}'
2021-01-22 10:09:03.991 INFO ZWAVE: Writing true to 38-0-Up
2021-01-22 10:09:04.168 INFO ZWAVE: Node 13: value updated: 38-0-currentValue 0 => 99

(the payload can actually be just true too)
I cannot see the visible results right now, but I will check later.
I'm not sure if that's an issue with zwavejs2mqtt or mosquitto (I think mosquitto is at fault here).
In my case this circumvents the issue, because the raw/ prefix is obsolete in my setup (the raw layer has its own broker now), but others may still encounter this issue.

The idea I had while writing the "Additional context" seems to be part of the issue. I reconfigured to remove the raw/ mount-point. and when trying to set my first example from above actually leads to further log-messages:
2021-01-22 10:09:03.988 INFO MQTT: Message received on z-wave/LB__01/38/0/Up/set, '{\n "value": true\n}'
2021-01-22 10:09:03.991 INFO ZWAVE: Writing true to 38-0-Up
2021-01-22 10:09:04.168 INFO ZWAVE: Node 13: value updated: 38-0-currentValue 0 => 99

(the payload can actually be just true too)
I cannot see the visible results right now, but I will check later.
I'm not sure if that's an issue with zwavejs2mqtt or mosquitto (I think mosquitto is at fault here).
In my case this circumvents the issue, because the raw/ prefix is obsolete in my setup (the raw layer has its own broker now), but others may still encounter this issue.

@sneakpaw When you say mount point, you mean the prefix set on mqtt settings?

@sneakpaw When you say mount point, you mean the prefix set on mqtt settings?

No. Mosquitto has the ability to restrict a tcp-listener to a specific topic (and its subtopics). In my case that would be:
listener 1873
mount_point raw/

zwave2mqtt only has z-wave configured.
So anyone who connects to 1873 should only see raw/#, but as # (without the raw/).
This was the behaviour in mosquitto 1.6, but it seems, that the upgrade to 2.0 broke that.
The main reason, why I had the end point was because zwave2mqtt didn't support raw/z-wave/ as prefix (at least not in the ui)

No. Mosquitto has the ability to restrict a tcp-listener to a specific topic (and its subtopics). In my case that would be:
listener 1873
mount_point raw/

zwave2mqtt only has z-wave configured.
So anyone who connects to 1873 should only see raw/#, but as # (without the raw/).
This was the behaviour in mosquitto 1.6, but it seems, that the upgrade to 2.0 broke that.
The main reason, why I had the end point was because zwave2mqtt didn't support raw/z-wave/ as prefix (at least not in the ui)

This was the behaviour in mosquitto 1.6, but it seems, that the upgrade to 2.0 broke that.

So It's a bug with mosquitto not zwavejs2mqtt?

This was the behaviour in mosquitto 1.6, but it seems, that the upgrade to 2.0 broke that.

So It's a bug with mosquitto not zwavejs2mqtt?

Yes, I think so too.
But the bug just broke a workaround I had for a missing feature in zwave2mqtt.
The issue I had is "fixed", or rather no longer relevant here.
If you want to add support for multi-part mqtt prefixes, that would be nice, but not strictly necessary any more for me.

Yes, I think so too.
But the bug just broke a workaround I had for a missing feature in zwave2mqtt.
The issue I had is "fixed", or rather no longer relevant here.
If you want to add support for multi-part mqtt prefixes, that would be nice, but not strictly necessary any more for me.


Sign loss while parsing rust_decimal::Decimal to sqlx::postgres::types::PgMoney

    let foo = Decimal::new(-314, 2);
    let bar = PgMoney::from_decimal(foo, 2);
    let baz = bar.to_decimal(2);
    println!("foo={} bar={:?} baz={}", foo, bar, baz); // foo=-3.14 bar=PgMoney(314) baz=3.14

This is likely due to PgMoney::from_decimal() taking bytes 4..12 of Decimal::serialize() and just treating that as a little-endian twos-complement integer: https://github.com/launchbadge/sqlx/blob/be189bd11e6bdd14c45c70bdad477e780a82b050/sqlx-core/src/postgres/types/money.rs#L59-L62
However the sign is the MSB of byte 3, so the value isn't actually twos-complement; we also shouldn't be ignoring scale which an integer in bytes 1..2.
It actually doesn't make much sense that scale being passed in here, I think that's supposed to be the precision that the Postgres database is expecting.
The docs make it sound like SQLx is just going to figure that out for you, but that's honestly way out of scope here because that would mean:

Executing SHOW lc_monetary; on the database server
Figure out what frac_digits is in that locale

Use libc::set_locale() and localconv()->frac_digits?

not thread-safe
changes the output of a bunch of C functions across the entire process
sometimes is set to nonsense anyway

Shell out to locale <locale name> frac_digits and read the response (non-portable)

It would be easier to execute select 0::money::text and then count the number of zeroes before a non-digit character (which would cover locales that use a different decimal separator than .) but what about locales that don't use Arabic numerals or use right-to-left writing? It'd be a nightmare and this is all still assuming we're talking to the database and patching the value up on-the-fly.
So yeah, instead we should make it clear that scale should be the precision that the Postgres server is expecting or else it'll interpret the value incorrectly.
I defended money in a previous issue but now that I'm digging more into it, it's honestly super gross. To be fair, a lot of that is also the POSIX locale system which honestly no one should be using anymore.


sorting by custom attributes

For example we have such code
class Person < ApplicationRecord
  has_one :parent_person, class_name: 'Person'
  has_one :detail, class_name: 'PersonDetail'

  alias_attribute :active, :is_active

  def independent
    self.parent_person_id.nil?
  end

  def ensure_detail
    self.detail || self.build_detail
  end
end

class PersonDetail < ApplicationRecord
  belongs_to :person
end

class PersonResource < ApplicationResource
  attributes :email, :address, :is_active, :independent
  filter :address, apply: ->(value) { where(person_details: {address: value}) }
  filter :independent, apply: ->(value) do
    flag = !(value == 'false')
    where(flag ? 'parent_person_id IS NULL' : 'parent_person_id IS NOT NULL')
  end

  def address
    @model.detail.address
  end

  def address=(value)
    @model.ensure_detail.address = value
  end

  def self.creatable_fields(context = nil)
    super - [:active, :independent]
  end

  def self.updatable_fields(context = nil)
    super - [:active, :independent]
  end

  def self.records(options = {})
    super.eager_load(:detail)
  end
end

and we want to sort by address, is_active and independent
in past version of JR I use code like this:
def custom_sorting
  {
    'address' => 'person_details.address',
    'is_active' => 'active',
    'independent' => '(parent_person_id IS NULL)'
  }
end

def self.apply_sort(records, order_options, context = {})
  changed_order_options = []
  order_options.each do |key, value|
    changed_order_options << "#{custom_sorting.fetch(key,key)} #{value}"
  end
  super(records, changed_order_options, context)
end

but after PR #494 it doesn't work
Do you have any idea how can we now sort by custom attributes?

@senid231 Your custom sorting work looks awesome. I'd love the ability to define sorts in the same way as filters. How do we get this into core?

@adambedford there are many bugs in current master so we need to fix them at first
I think this PR will be merged before next stable release comes


Patcher.sh

Make patcher.sh that will auto patch VestaCP files.

http://www.cyberciti.biz/faq/unix-linux-replace-string-words-in-many-files/ - use this as main patcher of files ...

In new version of patcher instead of calling URL just do the return 0;
Example:
https://github.com/serghey-rodin/vesta/blob/master/bin/v-activate-vesta-license#L38
just set answer= to 0


Mast: Watch elaraby-decor.com

Mast requests the watch of the watch_keyword elaraby-decor\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

elaraby-decor\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Daniil in Charcoal HQ


HTML Markup in Question Title

Hi,
I am trying to have a question title and subtitle with further description of the question in my survey. Therefore I wanted to just use basic HTML Markup in the question title but I only get the Markup output as text. Is there any way to change this?

@Chry007 What markdown engine to you use? We are using showdown in this example: http://surveyjs.org/examples/jquery/survey-markdown-matrix/
By the way, we have "add question.description" property in out TODO list. It should be really easy to implement and I may do it tomorrow, for example. In this case, it will be in the next minor update.
Thank you,
Andrew

Hi @andrewtelnov,
thanks for the quick reply. I am not using a markdown engine. All I'm trying to do is set the title to something like this:
question.question_text + "  Subtitle ... "
But as output I get:
Question ...  Subtitle ...

@Chry007  Unfortunately it will not work. The engine consider question.title as text. The correct approach is to use markdown. I gave the link on the demo in the previous reply.
The question.description property functionality is almost done. It will be in the next minor update on this week: https://github.com/surveyjs/surveyjs/issues/579
Thank you,
Andrew

@andrewtelnov All right! :) Thanks for the clarification!

@andrewtelnov Hi Andrew, Just following up on @Chry007 comment above
I too wanted to use HTML or markdown in a question title or description. I was using the showdown markup as per the surveyjs documentation and I am using the latest release.
I can see the description appear under the title but it is not rendering HTML or markdown as I would have hoped it would.
For example,
 { type: "radiogroup",isRequired: true,  name: "section1question1", title: "[Test Link](https://github.com/showdownjs/showdown)Test", choices: ["Yes", "No"], description:"[Test Link](https://github.com/showdownjs/showdown)Test"},
The HTML is not rendered as HTML but has been converted into:
1. * [Test Link](https://github.com/showdownjs/showdown)<span>Test</span>

[Test Link](https://github.com/showdownjs/showdown)<span>Test</span>
Is it possible to print a link out using markdown or print HTML in the title or description of a radiogroup question?
Thanks in advance,
Trevor

@trevcl The description has been already implemented and it supports markdown. Here is the plunker example: https://plnkr.co/edit/Jvou2vokWwvHG7p8TdVa?p=preview
Thank you,
Andrew

Hi Andrew,
Stupid mistake on my part. I didn't realise I would have to include the additional showdown.js file.
All is working as expected.
Thanks,
Trevor

Any plan to include HTML markup or functions in question title? This will make it robust! Some usecase can be

highlight/bold/underline certain phrases in the title
conditional display texts based on other questions (using functions)

@yidan You may use our markdown support. Here is the example
Thank you,
Andrew

@andrewtelnov Thanks Andrew! Got it! So we can include a markdown library to implement the HTML functionality in title text. That's awesome! I was a bit confused by the markdown concept earlier. Appreciate your follow-up!

Markdown is typically the way to go.  But if you really want unprocessed/unsanitized HTML to pass through, it's possible by using the markdown support and omitting any conversion.
Use the raw HTML in place of the 'converted' markdown output, like this:
model.onTextMarkdown.add((survey, options) => {
    options.html = options.text;
});


Replay works, but getevent not showing X, Y lines.

this is what getevent sees:
[<PHONE_NUMBER>59] 0003 0039 000009c0
[<PHONE_NUMBER>59] 0001 014a 00000001
[<PHONE_NUMBER>59] 0000 0000 00000000
[<PHONE_NUMBER>14] 0003 0039 7fffffff
[<PHONE_NUMBER>14] 0001 014a 00000000
[<PHONE_NUMBER>14] 0000 0000 00000000

while the file is:
[<PHONE_NUMBER>29] 0003 0039 000009c0
[<PHONE_NUMBER>29] 0003 0035 00000039
[<PHONE_NUMBER>29] 0003 0036 0000007c
[<PHONE_NUMBER>29] 0001 014a 00000001
[<PHONE_NUMBER>57] 0000 0000 00000000
[<PHONE_NUMBER>57] 0003 0039 ffffffff
[<PHONE_NUMBER>57] 0001 014a 00000000
[<PHONE_NUMBER>57] 0000 0000 00000000

The bizzard is the click is in the correct place :()
The issue is that when I click the screen bymyself is do like the click from the screen was never "UP" so I see multitouch. but when sending another click to the screen with the software it is not a multitouch click but singe one.

Can you try using Peter's technique: https://github.com/Cartucho/android-touch-record-replay/issues/12

@instasck, this should be fixed. Could you please confirm?

@Cartucho  this is not my isssue, I have seen last touch not recorded so I just remove it manually.
The issue is different as discribed ....

Can you please update the code and give it a try? We made many changes in the last commits.

@Cartucho I don't use the *.sh files ... I only use:
getevent -t  /dev/input/event1 | tee record_new
And split the file with my own code.

oh ok, then the issue is not related to my code


scenario cli_help isn't available anymore

since we removed scenarios list the help messages that describe scenarios are not available to user anywhere

good point, I think we need to move all of those scenario descriptions down into the parse_args section of the scenario, vs having it in its own function. This is also nice because the help is self updating and always in line with the current arguments for the scenario, whereas before we were hardcoding the help text.


Support JSON stream to enable import of large size scan files

Description
There is a 256 MB size cap in V8 when loading JSON files. We can get around this issue by loading the JSON with a stream.
Steps to reproduce
Import a ScanCode JSON file larger than 256 MB
Expected behavior
Be able to import a JSON file larger than 256 MB
Actual behavior
When a scan is imported, the following error is thrown: SyntaxError: Unexpected end of JSON input
Reproduces how often:
Anytime you load a file larger than 256 MB
Errors in the Developer Tools
None
System configuration
Operating System: MacOS
AboutCode Manager Version: v2.0.0

a way to get a streamable JSON is to use the new ScanCode json-lines output contributed by @JonoYang