Pre requisites

Fix #100 #37
Adding PreRequisite checks.  Check if user has modified/deleted starter code or tried to import external libraries, and give appropriate feedback.

Hi Sean,
I can confirm that the linter is not catching the style errors you pointed
out; I fetched from origin and merged in the master branch, but was still
able to push while receiving a "Linter tests passed" message.
I'll work on fixing the issues above.
--Aaron
On Tue, Apr 25, 2017 at 4:25 PM, Sean Lip<EMAIL_ADDRESS>wrote:

@seanlip requested changes on this pull request.
Thanks! I've taken a full pass -- a lot of the parts are there, but
there's some improvements that can be made to make things conceptually
clear. PTAL and let me know if you have questions on the comments I've
written.
In client/app.js
https://github.com/google/tie/pull/237#discussion_r113329092:

@@ -33,6 +33,9 @@ tie.constant('CODE_EXECUTION_TIMEOUT_SECONDS', 3);
// Supported languages.
tie.constant('LANGUAGE_PYTHON', 'python');
tie.constant('ALL_SUPPORTED_LANGUAGES', ['python']);
+tie.constant('PYTHON_STANDARD_LIBRARIES', ['collections', 'document', 'image',

I'm not sure we need document, processing, or turtle.
In client/domain/SnapshotObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113329510:

   this._codeEvalResult = codeEvalResult;

 this._prereqEvalResult = prereqEvalResult;

For ease of understanding, let's put codePrereqCheckResult (see other
comment above re naming) above codeEvalResult, since it comes first
chronologically.
In client/domain/PrerequisiteEvalResultObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113330157:

+/**

@fileoverview Factory for creating new frontend instances of

PreRequisiteCheckResult domain objects.

*/

+tie.factory('PrerequisiteEvalResultObjectFactory', [

function() {
var PrerequisiteEvalResult = function(

 code, prereqErrorMessage, starterCode) {

 this._code = code;

 this._prereqErrorMessage = prereqErrorMessage;

 this._starterCode = starterCode;

};

// Instance methods
PrerequisiteEvalResult.prototype.getCode = function() {

Where is this used? Why do we need to store the code at all?
In client/domain/PrerequisiteEvalResultObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113331076:

+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS-IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.

+/**

@fileoverview Factory for creating new frontend instances of

PreRequisiteCheckResult domain objects.

*/

+tie.factory('PrerequisiteEvalResultObjectFactory', [
How about calling this CodePrereqCheckResult, for parity with
CodeEvalResult? The difficulty with PrerequisiteEvalResult is that, in
CodeEvalResult, it's the code that's being evaluated, which implies that
here the prereqs are being evaluated.
In client/domain/PrerequisiteEvalResultObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113331382:

+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS-IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.

+/**

@fileoverview Factory for creating new frontend instances of

PreRequisiteCheckResult domain objects.

The capitalization that's being used here does not match the actual name
of the domain object.
In client/domain/PrerequisiteEvalResultObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113332110:

+// distributed under the License is distributed on an "AS-IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.

+/**

@fileoverview Factory for creating new frontend instances of

PreRequisiteCheckResult domain objects.

*/

+tie.factory('PrerequisiteEvalResultObjectFactory', [

function() {
var PrerequisiteEvalResult = function(

 code, prereqErrorMessage, starterCode) {

 this._code = code;

 this._prereqErrorMessage = prereqErrorMessage;

This is a bit confusing. The purpose of this domain object would
presumably be to store enough information to reconstruct the result of the
prerequisite check.
So, presumably the only field here should be a list called something like
this._prerequisiteCheckFailures. Each entry is a dict with a 'type' key
(whose value is PREREQ_CHECK_TYPE_BAD_IMPORT = 'badImport' or
PREREQ_CHECK_TYPE_MISSING_STARTER_CODE = 'missingStarterCode' -- these
named constants can be declared in app.js). If the key is
PREREQ_CHECK_TYPE_MISSING_STARTER_CODE then the dict also has an extra
'expectedStarterCode' key; you may need something similar for badImport as
well.
The error messages themselves should presumably go in the service which
generates the feedback objects based on the essential info in this domain
object.
In client/services/SolutionHandlerService.js
https://github.com/google/tie/pull/237#discussion_r113332260:

@@ -19,53 +19,74 @@

tie.factory('SolutionHandlerService', [
'$q', 'CodePreprocessorDispatcherService', 'CodeRunnerDispatcherService',

'FeedbackGeneratorService', 'SnapshotObjectFactory', 'TranscriptService',
'CodeSubmissionObjectFactory',

'FeedbackGeneratorService', 'PrerequisiteCheckDispatcherService',
'SnapshotObjectFactory', 'TranscriptService', 'CodeSubmissionObjectFactory',

Nit: let's move 'CodeSubmissionObjectFactory' to the line below for parity
with line 27. Easier to check that the dependencies line up, that way.
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333270:

+// limitations under the License.

+/**

@fileoverview Service for performing pre-requisite checks on Python code snippets.

*/

+tie.factory('PythonPrerequisiteCheckService', [

'PYTHON_STANDARD_LIBRARIES',
'PrerequisiteEvalResultObjectFactory',
function(

 PYTHON_STANDARD_LIBRARIES,

 PrerequisiteEvalResultObjectFactory) {

const LEFT_TRIM_PATTERN = /^\s+/;

var checkStarterCodePresent = function(starterCode, code) {

This is far too complicated / ad hoc. It'll be hard to maintain.
So, just to verify: the point of this prereq check is to ensure that the
functions in the starter code still exist within the learner's code, right?
If so, could we instead restructure the logic in this function roughly as
follows?
expectedTopLevelFunctions = extractTopLevelFunctions(starterCode);
verifyFunctionsExist(code, expectedTopLevelFunctions);

In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333430:

   else {

     while (codeLineIndex < codeLines.length) {

       if (codeLines[codeLineIndex] === starterCodeLine) {

         break;

       }

       codeLineIndex++;

     }

   }

   if (codeLineIndex >= codeLines.length && i < starterCodeLines.length) {

     return false;

   }

 }

 return true;

};

var getCodeLibs = function(code) {

getImportedLibraries
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333478:

       if (codeLines[codeLineIndex] === starterCodeLine) {

         break;

       }

       codeLineIndex++;

     }

   }

   if (codeLineIndex >= codeLines.length && i < starterCodeLines.length) {

     return false;

   }

 }

 return true;

};

var getCodeLibs = function(code) {

 var codeLines = code.split('\n');

 var coddLibs = [];

Spelling is wrong. Might want to use importedLibraries, anyway.
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333538:

       }

       codeLineIndex++;

     }

   }

   if (codeLineIndex >= codeLines.length && i < starterCodeLines.length) {

     return false;

   }

 }

 return true;

};

var getCodeLibs = function(code) {

 var codeLines = code.split('\n');

 var coddLibs = [];

 var pattern = new RegExp('^import\\ \\w+$');

 for(var i=0; i<codeLines.length; i++) {

I'm surprised that the linter isn't catching this, but there should be
single spaces around = and '<', as well as before '('. Ditto elsewhere.
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333702:

   }

   if (codeLineIndex >= codeLines.length && i < starterCodeLines.length) {

     return false;

   }

 }

 return true;

};

var getCodeLibs = function(code) {

 var codeLines = code.split('\n');

 var coddLibs = [];

 var pattern = new RegExp('^import\\ \\w+$');

 for(var i=0; i<codeLines.length; i++) {

   if(pattern.test(codeLines[i])) {

     var words = codeLines[i].split(' ');

     coddLibs.push(words[1]);

You should be able to get the pattern directly from the regexp check by
using capture groups. I recommend doing that instead, it's much cleaner.
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113333867:

 var coddLibs = [];

 var pattern = new RegExp('^import\\ \\w+$');

 for(var i=0; i<codeLines.length; i++) {

   if(pattern.test(codeLines[i])) {

     var words = codeLines[i].split(' ');

     coddLibs.push(words[1]);

   }

 }

 return coddLibs;

};

return {

 // Returns a promise.

 checkCode: function(starterCode, code) {

   if (!(checkStarterCodePresent(starterCode, code))) {

     var errorMessage = ['It looks like you deleted or modified ',

This stuff should go in the service that creates the feedback messages
that get displayed to the user. The object storing the prereq check result
should only contain the error type and the minimal amount of info needed to
reproduce what check actually failed.
In client/services/code_evaluators/PythonPrerequisiteCheckService.js
https://github.com/google/tie/pull/237#discussion_r113334833:

     var errorMessage = ['It looks like you deleted or modified ',

       'the starter code!  Our evaluation program requires the ',

       'function names given in the starter code.  You can press the ',

       '\'Reset Code\' button below to start over.  Or, you can copy ',

       'the starter code below:'].join('');

     return Promise.resolve(

       PrerequisiteEvalResultObjectFactory.create(

         code, errorMessage, starterCode));

   }

   var libErrorMessage = ['It looks like you are importing an external ',

                   'library.  Only standard libraries are supported.  ',

                   'The following libraries are not ',

                   'supported:\n'].join('');

   codeLibs = getCodeLibs(code);

   var extLibs = [];

Again, all this stuff is way too complicated. You could instead do
something like this:
var unsupportedImports = importedLibraries.filter(function(importedLibrary) {
return PYTHON_STANDARD_LIBRARIES.indexOf(importedLibrary) === -1;
});
Make sure that the importedLibraries list is already trimmed, etc. Then
pass the unsupportedImports as a customization to the prereq check result
factory.
In client/services/code_evaluators/PythonPrerequisiteCheckServiceSpec.js
https://github.com/google/tie/pull/237#discussion_r113334996:

+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS-IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.

+/**

@fileoverview Unit tests for the PythonPreRequisiteCheckService.

*/

+describe('PythonPrerequisiteCheckService', function() {

var PythonPrerequisiteCheckService;

Use spaces (and 2-space indentation), not tabs. Again I'm surprised the
linter isn't catching this, I'll take a look. Is this synced with the
latest version of master?
In client/services/code_evaluators/PythonPrerequisiteCheckServiceSpec.js
https://github.com/google/tie/pull/237#discussion_r113335091:

			PythonPrerequisiteCheckService.checkStarterCodePresent(

				starterCode, code);

		expect(starterCodePresent).toEqual(false);

});

});
describe('getCodeLibs', function() {

it('correctly parses import statements', function() {

	var code = ['import numpy\n',

		'import pandas\n', '\n',

		'def myFunction(arg): \n',

		'\treturn arg'].join('');

	var codeLibs =

		PythonPrerequisiteCheckService.getCodeLibs(code);

	expect(codeLibs).toEqual(['numpy','pandas']);

});

it('does not capture strings with \'import\' in text', function() {

Nice test!
In client/services/code_evaluators/PythonPrerequisiteCheckServiceSpec.js
https://github.com/google/tie/pull/237#discussion_r113335132:

		var starterCodePresent =

			PythonPrerequisiteCheckService.checkStarterCodePresent(

				starterCode, code);

		expect(starterCodePresent).toEqual(false);

});

it('returns true when only argument to \'return\' is modified)',

	function() {

		var code = ['def myFunction(arg):\n',

			'\tresult = arg.rstrip()\n',

			'\treturn myResult\n'].join('');

		var starterCodePresent =

			PythonPrerequisiteCheckService.checkStarterCodePresent(

				starterCode, code);

		expect(starterCodePresent).toEqual(true);

});

it('returns false when starter code lines are present, but out of order',

Eh, I'm not sure the order matters...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/tie/pull/237#pullrequestreview-34707776, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AaKvlFRta_Zqkg60EGUrkEDKrIFSJzyUks5rzoDQgaJpZM4NCfoR
.

Thanks, @a-zakem! OK, good to know -- I'll investigate the linter issues.

Thank you, @a-zakem! On the whole, this is looking way better. I have another round of comments.
Also, one request: please could you reply to all comments? In particular, for comments that you've addressed, please reply "Done." -- this helps keep track of which comments have been addressed and which haven't. (I think you missed one.)

@eyurko -- can you PTAL and approve if you have no further comments? Then I'll handle the merge.
Thanks!

Hi Sean,
I think that depends on when the CodeEvalResult object is created.  If the
CodeEvalResult object is created as a result of a syntax error during the
the initial run of the raw code, then the CodeEvalResult object will
contain the raw code.  If the student submission passes the syntax error
check and proceeds to pre-processing, then the CodeEvalResult object will
contain the pre-processed code including the testing code.
In the PreRequisites branch as merged, PreRequisiteEvalResultObjectFactory
was renamed CodePrereqCheckResultObjectFactory, and the
CodePrereqCheckResult does not store the submitted code (raw or
otherwise).  At present, I believe the stored code in the CodeEvalResult is
only used in the FeedbackGeneratorService's _getBuggyOutputTestFeedback
function, to check that the student has submitted different code before
incrementing the hintIndex for the same error.
On Thu, Apr 27, 2017 at 4:47 PM, Sean Lip<EMAIL_ADDRESS>wrote:

@seanlip commented on this pull request.
In client/domain/PreRequisiteEvalResultObjectFactory.js
https://github.com/google/tie/pull/237#discussion_r113826884:

+/**

@fileoverview Factory for creating new frontend instances of

PreRequisiteCheckResult domain objects.

*/

+tie.factory('PreRequisiteEvalResultObjectFactory', [

function() {
var PreRequisiteEvalResult = function(

 code, preReqErrorMessage, starterCode) {

 this._code = code;

 this._preReqErrorMessage = preReqErrorMessage;

 this._starterCode = starterCode;

};

// Instance methods
PreRequisiteEvalResult.prototype.getCode = function() {

Hi @a-zakem https://github.com/a-zakem,
I just realized something -- the code that's used for
PrerequisiteEvalResult is not actually the same as the code in the
CodeEvalResult, right? If I'm not mistaken, the first one is the raw code
and the second is the code after a bunch of extra testing stuff is added to
it...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/tie/pull/237#discussion_r113826884, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AaKvlJyQzqG0E8W0kRndnROoQ57FbkLhks5r0Sj1gaJpZM4NCfoR
.


help

now i have a host.  i want to deploy many nodes with one wallet? is there have solusions? thankx

You cannot share Ethereum accounts nor checkbooks with multiple nodes.
One node, one Ethereum account, one checkbook, one configuration, one data-dir, one set of 3 unique ports.

You cannot share Ethereum accounts nor checkbooks with multiple nodes. It will not work. Each node must be unique and that uniqueness is based on the Ethereum address.
One node, one Ethereum account, one checkbook, one configuration, one data-dir, one set of 3 unique ports.

yeah i get. thx.  and another question: many node on one host., and how to use one clef, thx

You have to manually create the accounts in bee-clef and then tell each of your nodes which account to use with the following command-line argument:
--clef-signer-ethereum-address string     ethereum address to use from clef signer
I do not know how the password is managed in this usage as I don't use clef (yet).  You may find some hints on this in https://github.com/mfw78/apiarist

Hi there!
Thank you for trying out bee :bee: !
We can better help you with your question in our Discord forum. Hope to see you there, and... Welcome to Swarm.
Link to Discord


Missing definition RtlAdjustPrivilege

Missing definition RtlAdjustPrivilege
hope to add

Hi. Thanks for the report.
I added a definition to the headers, but I don't think it's necessary to create a wrapper for it. When using the library, I suggest calling NtxAdjustPrivilege or NtxAdjustPrivileges because they provide more flexibility. Here are some tips on how to use them:

Besides ordinary handles, you can specify NtCurrentProcessToken, NtCurrentThreadToken, or NtCurrentEffectiveToken. The first two options correspond to setting the Client parameter of RtlAdjustPrivilege to False and True, respectively.
You can supply values from TSeWellKnownPrivilege enumeration, so you don't need to lookup a privilege LUID from its name.
Use SE_PRIVILEGE_DISABLED, SE_PRIVILEGE_ENABLED, or SE_PRIVILEGE_REMOVED to indicate the desired state.
To try without failing the function when the privilege is not available, set IgnoreMissing to True and then compare the result to STATUS_NOT_ALL_ASSIGNED. That provides functionality similar to WasEnabled of RtlAdjustPrivilege.

OK. Thank!


Fix integration test for struct_search

Compensate stochasticity

Integration tests run

Codecov Report

Merging #135 (c48bcc9) into main (5c9e419) will decrease coverage by 0.29%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #135      +/-   ##
==========================================
- Coverage   72.31%   72.02%   -0.29%
==========================================
  Files         128      128
  Lines        7336     7336
==========================================
- Hits         5305     5284      -21
- Misses       2031     2052      +21

see 6 files with indirect coverage changes


Can't see shared mode for a volume?

I am using the following command to create/start a container. note that "/net" is a shared location and folders under it are nfs mounts.
docker run -d -h sickrage --restart=always --name=sickrage -v /opt/docker_configs/sickrage/:/config -v /net:/net:shared -e TZ=America/Toronto -ePUID=1001 -e PGID=1001 -p 8081:10093 linuxserver/sickrage:latest
however I cannot see the :shared tag in portainer or when I do a "re-create" i.e. to refresh a container due to an image change it doesn't s setup the volume as ":shared". This creates issues with the container accessing files on that share.
the ":shared" tag isn't seen anywhere on poratiner when viewing the container information. When I inspect the container it does show up as mode shared

Turns out this can be done using the stacks feature (docker-compose).
However I still think it is unfortunate it is (still) not included in the container UI.

Good lord. still an issue with 2.16.2


Stopped working

Updated the plugin to recent version.
Note-taking shortcuts stopped working.

You can provide your Obsidian version, plugin version, and a list of other installed plugins.
If you are more familiar with Obsidian, you can use Obsidian's console to check the error messages and feedback to us.

The Video Player seems to get stuck on buffering with loading bar spinning.
Obsidian v1.7.7
Media Extended v3.2.6

I also have this bug. The video gets stuck loading.
(Obsidian v1.7.7, and Media Extended v3.2.6)
I tried changing "open this url as" to iframe. While this fixed the video loading, video controls are not visible, and screenshots are unsupported, so it's not a great solution.

I seemed to be ok after signing into YouTube via Media Extended settings


This PR has a few extra changes from my code that I think are worth adding.

@j-ro @amy-chinlai  We're using these for a project, so we'll probably have more changes in our fork. Happy to ship them upstream or not, if it's a distraction.

I think we're happy to take changes, but probably more on the substantive end (this form is new!) vs these tweaks around javascript


Try to debug why Travis Linux builds are failing

(don't merge yet)

r? @wycats
(rust_highfive has picked a reviewer for you, use r? to override)


Closing the gap between strict and classic branches

Summary

Added strict checks, combined classic/strict code conditionally and added usergroup determination based on strict/classic.
Changes

Added is_strict to python utils.
Combined classic and strict code based on is_strict checks
Added dynamic usergroup determination based on is_strict checks
Testing

Current tests should cover the changes.
Possible Regressions

Checklist

[X] Read the contributions page.
[X] Submitted the CLA form, if you are a first time contributor.
[X] The introduced changes are covered by unit and/or integration tests.

Notes

Most of this work is merged in separate PRs.


insert records form array

Hello I have the following question
I'm using version 1.4.3 version
I have two grids and I'm using remote load
but I do not return the json with the structure of the documentation
the json returns a custom json
my question as I load the data in the grid
without using the for (....)
and do something similar to:
a1 = [1,2,3,4,5];
a2 = [21,22];
a1.splice.apply (a1, a2);

je I think the answer is simple
w2ui['gridcargos_parciales_alumno'].add(array_name)


enable pnpm mode

Description

Enable pnpm mode to catch issues earlier
Resolves: https://github.com/microsoft/rnx-kit/issues/2783
Test plan

Hi @tarunrajput, thanks for trying to help out. We are not quite ready to enable this yet. As you can see in #2783, there are a number of things that needs to be fixed before we can do that. I will be closing this for now.


[feat] Support for custom recipes

Describe the problem
I love how tauri supports so many recipes out-of-the-box. However, over time, I've built up a bunch of my own create-* packages that I would love to initialize my Tauri app with. Most of these extend existing templates (e.g. packages and configs on top of create-vite).
Describe the solution you'd like
Looking at the svelte merge, it looks like there's some custom, tauri-specific config that's required on top of the create-* package. Is it possible to extend existing recipes in some way?
Alternatives considered
Currently, I initialize a new tauri app and then go about installing packages and configs from my custom templates.
Additional context
No response

That's a great idea. Maybe @jbolda would like to respond.

Honestly at first, I was opposed to the idea, because different front-end setups requires different integration with Tauri but the idea has grown on me and I think it can help reduce the amount of recipes we need to maintain at the core package.
I still don't know what is the optimal way to do this but I think it should be like this:

the custom template is hosted on github

it has a cta-manifest.json that contains some metadata, like beforeBuildCommand, devPath...etc.

_or _

The github repo simply has a js file that exports a default object that follows the recipe interface. This approach will make the custom template have a finer control over how to bootstrap the template and if different package managers need different steps. I am skeptic about the security implications of this but it might be the best way imo because of the fore mentioned advantages and it would require less work on our end.

I know you're trying to target a much broader audience and use-case, but for my uses, I don't even change Vite's ports or configs that much. I add some Vite plugins that I've come to love which can maybe be merged with Tauri's default config? It gets difficult because these are arrays rather than objects 😕
Beyond that, I'm just installing my own set of packages, setting up stylesheets, etc.

I know you're trying to target a much broader audience and use-case, but for my uses, I don't even change Vite's ports or configs that much. I add some Vite plugins that I've come to love which can maybe be merged with Tauri's default config?

Not likely. We only bootstrap base projects. Plugins should be added by the user afterwards. Opinionated/Custom recipes shouldn't be baked in create-tauri-app. That's what you are asking for and I think the above proposal is the solution.

It gets difficult because these are arrays rather than objects 😕

Not sure what you mean by that.

Beyond that, I'm just installing my own set of packages, setting up stylesheets, having my own eslint rules, etc.

All should be done by you afterwards or a custom recipe using the above proposal.


Deprecate Index protocol

As discussed in #497. This deprecates the Index protocol.

Okay, so do we have a general consensus to merge on this one, or is there any other opinion?


JavaScript: Using Object Property Value Shorthand loses semantic highlighting

Issue Type: Bug
Using the es6 syntax for object property value shorthand makes it so the used properties always have the light blue color, instead of the dark blue color of readonly properties. Here's a reproducible code snippet:
const someVariable = "value"
const someObject = {
    someProperty: "other value",
    someVariable
}

In VSCode:

In the example above, someVariable shows up with a light blue color in the object declaration, whereas the expected behaviour would be a darker color that you could get from doing someVariable: someVariable:

VS Code version: Code 1.59.0 (379476f0e13988d90fab105c5c19e7abc8b1dea8, 2021-08-04T23:13:12.822Z)
OS version: Windows_NT x64 10.0.19043
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (12 x 3192)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
19.89GB (5.37GB free)

Process Argv
D:\Users\rodri\Documents\GitHub\hypixel-translators-bot --crash-reporter-id 1aa51578-c79e-468f-b062-c24334e21918

Screen Reader
no

VM
0%

Extensions (17)

Extension
Author (truncated)
Version

better-comments
aar
2.1.0

jsdoc-markdown-highlighting
bie
0.0.1

vscode-eslint
dba
2.1.23

gitlens
eam
11.6.0

EditorConfig
Edi
0.16.4

prettier-vscode
esb
8.1.0

copilot
Git
1.3.2371

vscode-graphql
Gra
0.3.16

discord-vscode
icr
5.7.0

vscode-env
Iro
0.1.0

python
ms-
2021.8.1105858891

jupyter
ms-
2021.8.1236758218

remote-wsl
ms-
0.58.2

vsliveshare
ms-
1.0.4673

vetur
oct
0.34.1

vscodeintellicode
Vis
1.2.14

minecraft-lang-colorizer
zz5
1.0.0

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspyt653:30270858
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
pythondataviewer:30285071
pythonvsuse255:30340121
vscod805cf:30301675
pythonvspyt200:30340761
vscextlangct:30333562
binariesv615:30325510
pythonvssor306:30344512
bridge0708:30335490
pygetstartedc2:30351386
vstre464cf:30350173
bridge0723:30353136

You can modify someObject.someVariable and it is not a readonly property.

You can modify someObject.someVariable and it is not a readonly property.

Yeah I know but the point here is to make it easier to see where someVariable comes from and even though the property itself isn’t readonly, the property that was assigned to it is so it should be dark blue like in the second example


TEGRA_SIGNING_ARGS add sed removal

Hi,
How can I pass extra sed command to TEGRA_SIGNING_ARGS?
I need the extra
-e/persistent.img/d -e/overlay.img/d -e/ota.img/d

otherwise the BUP signing will fail due to  tag not being stripped in the partition layout.
I've tried many variant of this
TEGRA_SIGNING_ARGS += "\\\"-e/persistent.img/d\ -e/overlay.img/d -e/ota.img/d\\\"

But -e always get re-interpreted as command line option to tegra194-flash-helper.sh

That's right, TEGRA_SIGNING_ARGS is only used to pass extra arguments to the flash helper script for adding the -u/-v options to specify the signing keys.
Modifying the partition layout XML file on the fly is done by appending commands to one of the scriptlets.   One example is here, in tegraflash_generate_bupgen_script_append() - it uses sed to append an additional sed command to the BUP generation script for modifying the XML file.

Understood. I won't push my patch then :D
Thanks!


compactor - "write postings: exceeding max size of 64GiB"

Thanos, Prometheus and Golang version used
Prometheus version: 2.11.1
thanos, version 0.6.0 (branch: master, revision:<PHONE_NUMBER>11d561fcf991182b3f60e91ea04843)
  build user:       root@packages-build-1127f72e-908b-40dc-b9e1-f95dcacaaa62-sck6t
  build date:       20190806-16:13:48
  go version:       go1.12.7

What happened
When running the compactor on we get the following output (sanitized to remove internal hostnames):
2019-08-13T15:13:54-07:00 observability-compactor-123456 user:notice thanos-compact-observability-036[20057]: level=error ts=2019-08-13T22:13:54.054751118Z caller=main.go:199 msg="running command failed" err="compaction failed: compaction failed for group 0@{host=\"observability-prom-A98765\"}: compact blocks [thanos-compact-tDlEs/compact/0@{host=\"observability-prom-A98765\"}/01DHRV2ADD0DGW2V0WB84D472P thanos-compact-tDlEs/compact/0@{host=\"observability-prom-B98765\"}/01DHTN0B5GK4M9K1YF344C599J thanos-compact-tDlEs/compact/0@{host=\"observability-prom-B98765\"}/01DHTYSE4G3CPN6PT73JD4NFPC thanos-compact-tDlEs/compact/0@{host=\"observability-prom-B98765\"}/01DHV863B7P32AR6CM46ZJCJDR thanos-compact-tDlEs/compact/0@{host=\"observability-prom-B98765\"}/01DHY92FF48P4CPQMGPVMEVJE8 thanos-compact-tDlEs/compact/0@{host=\"observability-prom-B98765\"}/01DHYJM1E1F58STJERMF6PWY0M]: 2 errors: write compaction: write postings: write postings: exceeding max size of 64GiB; exceeding max size o
2019-08-13T15:13:54-07:00 observability-compactor-123456 user:notice thanos-compact-observability-036[20057]: f 64GiB"

What you expected to happen
Compaction would succeed, or skip these block without error, since they cannot be compacted.
How to reproduce it (as minimally and precisely as possible):
Try to compact blocks that result in the postings part of the resulting index to exceed 64G
Full logs to relevant components
See log lines above.  Let me know if anything else would be useful, but I think that's all we have that is relevant
Anything else we need to know
These blocks are from a pair of prometheus hosts we have scraping all targets in one of our data centers.
This appears to be related to https://github.com/prometheus/prometheus/issues/5868.  I'm not sure if the limit is going to change in the TSDB library, but it would be nice in the mean time if this was considered a warning, or if we could somehow specify a threshold to prevent it from trying to compact these blocks, since they're already fairly large.

@claytono Sorry for the delay. You are right here. I wonder how to check that error? Maybe it is hard to work around. I'm sorry that I'm not very familiar with tsdb.

Sorry for the delay!
IMO, I agree with you, but also have some questions:

how thanos might handle this would be to have a maximum block size

How can we set the maximum block size? In your case, the size of each index files are less than 30GB, but they actually exceeded the max size of 64GB during compacting. Before finishing compact, we don't know whether the new size of index is more than 64GB or not, right? I think it is difficult to find the right size for checking. Maybe I'm wrong.

This might also help with some scenarios using time based partitioning

Sorry, I can't catch your point. Could you explain it?

It looks like this is being addressed upstream: prometheus/prometheus#5884

Yes, if it is fixed, that will be great.

How can we set the maximum block size? In your case, the size of each index files are less than 30GB, but they actually exceeded the max size of 64GB during compacting. Before finishing compact, we don't know whether the new size of index is more than 64GB or not, right? I think it is difficult to find the right size for checking. Maybe I'm wrong.

I don't think we really can.  I think the best we could do would be to have a flag that allowed us to specify a maximum size and index needs to be, and past that we don't try to compact it any more.

This might also help with some scenarios using time based partitioning

Sorry, I can't catch your point. Could you explain it?

We're planning to use the new time based partitioning.  As part of this we expect we'll want to rebalance the partitions occasionally as compaction and downsampling occurs to keep the amount of memory usage roughly equivalent between thanos store instances.  The limiting factor for us for thanos-store capacity is memory usage. That's roughly proportional to index size, so we'd like to ensure we don't end up with thanos stores serving blocks so big that it's just a few blocks.  If a thanos-store process is serving 2 blocks and that consumes all the memory on the host, then our options for rebalancing aren't as good as if we limit index size and the thanos-store process is serving a dozen blocks.  Right now I think the best options we have is to limit max compaction, but that's a fairly coarse tool.
That said, this isn't really related to the issue I opened, it would just be a nice side effect if this needed to be fixed or worked around in Thanos.

Awesome! Thank you for your detailed case.

we don't try to compact it any more.

How about downsampling? If we skip compaction, maybe we could not do downsampling, because of lack of enough series in the blocks. Further information is here.

Don't compact index files that are already x GB

What's your idea for the x? I think we are difficult to find the right x.

If a thanos-store process is serving 2 blocks and that consumes all the memory on the host, then our options for rebalancing aren't as good as if we limit index size and the thanos-store process is serving a dozen blocks

I think it is not our goal for compaction, did you compare the time cost for long time period querying between those two ways?
Now @bwplotka is following the issue #1471 about reducing store memory usage, I think you can also follow that if you would like to.

We just hit this issue as well. This limit is due to the use of 32 bit integers within Prometheus' TSDB code. See here for a good explanation. There is an open bug report as well as PR that will hopefully resolve the issue.

Could this be re-opened until this is resolved?

Yea, let's solve this.
I have another suggestion: I would suggest actually not waiting for a fix here and rather set upper limit for block and start splitting them based on size. This is tracked by https://github.com/thanos-io/thanos/issues/2340
Especially on large setups and WITH some deduplication (vertical or offline) enabled and planned we will have a problem with huge TB size blocks pretty soon. At some point indexing does not scale well with number of series, so we might want to split it....
cc @brancz @pracucci @squat

That sounds good to me and was another fear of mine as well. Sure we could increase the index limit but we will reach that again some day.

This is still an issue.

Yes, not done. How you guys workaround so far this issue?

We have just stopped compacting :(

Still an issue. If anyone knows of a workaround I would be appreciative. Right now we just have the impacted instance not compacting. This will start hurting performance soon.

Still an issue

In the mean time while waiting for https://github.com/prometheus/prometheus/pull/5884

One idea is to mitigate this by detecting this error on the compactor side and excluding those blocks from compacting.
Second is to split into two blocks instead of one.

2 is nicer but requires much more thinking. 1. is not the easiest as well as we need different block planning. But can be done without Prometheus upstream changes.

Help wanted! We need to limit the size of blocks anyway at some point so starting this: https://github.com/thanos-io/thanos/issues/3068

This is still a big issue for us. We now have several compactor instances turned off due to this limit.

We're also now running into this (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11757)

On it this sprint.
Kind Regards,
Bartek Płotka (@bwplotka)
On Thu, 29 Oct 2020 at 08:22, Ben Kochie<EMAIL_ADDRESS>wrote:

We're also now running into this (
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11757)
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1424#issuecomment-718415229,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O2QWPPSKM3URYSV723SNEJ25ANCNFSM4IMAUWDQ
.

As a workaround, would it be possible to add a json tag to the meta.json so that the compactor can be told to skip it?
Something like this:
{
  "thanos": {
    "no_compaction": true
  }
}

The main problem with this approach is that this block might be one portion of a bigger block that "have" to be created in order to get to 2w compaction level. Need to think about it more, how to make this work, but something like this could be introduced ASAP, yes.
I am working on some block size capped splitting, so compaction will result in two blocks, not one etc. Having blocks capped at XXX GB is not only nice for this issue but also caps how much you need to download for manual interactions, potential deletions, downsampling, and other operations.
Just curious: @SuperQ can you tell what size of the block is right now? (mostly sizes of index I am interested in) Probably more than one in order to get 64GB+ postings.

I should have some workaround today.

Here's some metdata on what's failing:
  8.96 GiB  2020-09-20T17:41:23Z  gs://gitlab-gprd-prometheus/01EJP50AZ3P1EN2NWMFPMC5BA7/index
  3.78 GiB  2020-09-21T05:56:53Z  gs://gitlab-gprd-prometheus/01EJQHMW5K9VP7FWW7VD3M9FQB/index
  8.19 GiB  2020-09-23T06:57:05Z  gs://gitlab-gprd-prometheus/01EJWQJQ0M2F33JV47CEFKFK5N/index
 14.08 GiB  2020-09-25T08:00:39Z  gs://gitlab-gprd-prometheus/01EK1XNFBHZ0MB86HBEZ9NFSVV/index
 12.72 GiB  2020-09-27T07:21:45Z  gs://gitlab-gprd-prometheus/01EK711SSHZHPDH5V274M8ESAY/index
 11.35 GiB  2020-09-29T07:32:43Z  gs://gitlab-gprd-prometheus/01EKC6VXQ4A3ZANP0E30KY5AWM/index
 13.31 GiB  2020-10-23T10:03:34Z  gs://gitlab-gprd-prometheus/01ENA7V7YW31BVM1QFRJ57TDVX/index

And an example meta.json

{
"ulid": "01EKC6VXQ4A3ZANP0E30KY5AWM",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000,
"stats": {
"numSamples":<PHONE_NUMBER>9,
"numSeries": 49668912,
"numChunks": 229130151
},
"compaction": {
"level": 3,
"sources": [
"01EK6R1EW9Y76QPSHE2S2GZ1SC",
"01EK6YX646B7RPN4FFFCDR62KP",
"01EK75RXC6HZBKGRB4N8APP638",
"01EK7CMMM611CJ4TJD04BBG6MM",
"01EK7KGBWNC515RFKBDMX4HJSP",
"01EK7TC3463VCS3Z4CXZMEET0X",
"01EK817TC6KVQ59R73D48BRAC9",
"01EK883HM7X7V1SFCH8NZY6FW4",
"01EK8EZ8W5N2JK5ZZY2V4058YG",
"01EK8NV048MXK605DPJ710DMKS",
"01EK8WPQC7ZY51T32WHFMBNPKS",
"01EK93JEN36YBXK58KMTWMPJCK",
"01EK9AE5X3WR35X0FGP6Y1A0PR",
"01EK9H9X586N62EGE972PWPT95",
"01EK9R5MD2B11YRBPE62ETQJX8",
"01EK9Z1BN8DFE5H765E8JTDEJC",
"01EKA5X2XE7959NND514559190",
"01EKACRT59F6KDA9WJ9GP3NZSW",
"01EKAKMHD7VA5WXK10S8EH4274",
"01EKATG8N9Q6EYRR3MYHSE3H10",
"01EKB1BZX6SZRVKMMG83R1SZ2J",
"01EKB87Q5DHDGVK0MGS5BE8PFH",
"01EKBF3ED853GQ0TBWT5YTCTGS",
"01EKBNZ5NNE8GACZDW32RTBWR0"
],
"parents": [
{
"ulid": "01EK89TVSX9ZZAET1FYXC1WHH6",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
},
{
"ulid": "01EK8HESS4GD6YS55ASCCXGA2N",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
},
{
"ulid": "01EK9CXXF25T12TBV0XRGRXD63",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
},
{
"ulid": "01EKA8B4MF78WYY3FM115Q4C6T",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
},
{
"ulid": "01EKB3XJWV3FMJXSMQ493JV6WN",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
},
{
"ulid": "01EKBZBX8MCBM2T05Y739KDRGV",
"minTime":<PHONE_NUMBER>000,
"maxTime":<PHONE_NUMBER>000
}
]
},
"version": 1,
"thanos": {
"labels": {
"cluster": "gprd-gitlab-gke",
"env": "gprd",
"environment": "gprd",
"monitor": "default",
"prometheus": "monitoring/gitlab-monitoring-promethe-prometheus",
"prometheus_replica": "prometheus-gitlab-monitoring-promethe-prometheus-0",
"provider": "gcp",
"region": "us-east1"
},
"downsample": {
"resolution": 0
},
"source": "compactor"
}
}

Due to the interaction of the operator, and helm, we have an excess of overly long labels.

Update. The code is done and in review to fix this issue.

no-compact-mark.json added https://github.com/thanos-io/thanos/pull/3409
Auto marking of blocks that would cause the block to be too big (trigger issue of 64GB index) https://github.com/thanos-io/thanos/pull/3410

You can manually exclude blocks from compaction, but PR for automatic flow for this is still in review: https://github.com/thanos-io/thanos/pull/3410 will close this issue once merged.


A unit converter to easily translate different types of units to each other

This is a great feature as it can be as simple or as complex as the implementer would like. and there are many ways to tackle this. This would be a great first issue for someone to get their feet wet with discord bots

Since nobody's commented on this one, I might try taking it on as my first try with making a cog and service. Though since I'm still fairly new to Python, I can't say whether or not I'd be successful in implementing something anytime soon.

Ohh perfect, i would def suggest looking into a library or api to do this for you, up too you which path you choose. feel free to ping me with any questions

Yeah, I'm sure Python has some sort of library out there that can handle the actual unit conversion formulas. Cause writing out each one by hand as individual functions sounds like a nightmare. I guess my job would just be coming up with a way to take user input, validate it, and feed it to the appropriate conversion function and display the output to the user.


Make the new appimage executable on validation success

Currently, the 'Run' button functionality will make the newly downloaded AppImage executable before executing. What I propose with this pull request is that the newly downloaded AppImage is also made executable when signature validation passes (not on warning/missing signature).

Why would you need that feature? Hardly any AppImage supports the signature validation anyway. It'd probably make more sense to make any updated AppImage executable.

Thank you for reviewing my patch.
I have come across this while working on updates integration for Krita. It is a small usability issue, but I believe it is useful.
As it is now, if a user downloads an AppImage, they express their trust by making it executable and running it. However, if they update the AppImage, they need to make it executable again (unless they use appimaged or AppImageLauncher, most of our users probably do not have any of those). That is quite inconvenient (it adds an extra step on every update) and it might be confusing (the user already expressed their trust earlier). That is why I propose to make the AppImage executable upon update.
From the security point of view, it made sense to me to only make the AppImage executable upon successful signature validation. However, that is just the original idea and I understand your argument that signed AppImages are not widespread at the moment.

The way I had implemented it originally: after a successful update, the permissions of the old version were carried over to the new version. In other words, if you were updating an AppImage that was executable, the updated binary also was executable (unless the signature was tampered with).
Would that satisfy your requirement?

Yes, carrying over the permissions of the old version would work for my use case. I will make the changes to the pull request.
Concerning "unless the signature was tampered with": should I change the permissions only for valid signatures, or also for AppImages, which were never signed (both old and new versions are missing the signature)?

Unsigned AppImages cannot be verified, hence we assume they are good. Only signed AppImages can be verified, and if verification fails, the user should be warned and the file should not get the same permissions as the old version.

I have updated the code: permissions are copied from the old version to the new one when 1) both AppImages are unsigned, 2) signature validation passes. Is this acceptable? Should I make other changes?

Are there any remaining blockers to this? If not, may I ask you to merge it?

Thanks for your contribution!


Autoclose $ not working.

When I type $, it doesn't autoclose, and I cannot find a setting for it in the plugin. Am I missing something obvious?

Hi! @kaevm
This autoclose function should work by default. It can also be toggled in the plugin settings page:

Have you enabled the plugin? After installation, you may need to enable it by clicking here:


Document how to query

This PR adds the user docs on how to query VAST.

@dominiklohmann I requested a review from you regarding the new Formats section.
The PR will also fill in new Ingest and Query sections, but those are yet to come. There's enough meat there to review this part, though.

NB: we need to teach the Markdown linter that this is not an error:
web/docs/understand/formats/zeek.md:94:12 MD010/no-hard-tabs Hard tabs [Column: 12]
web/docs/understand/formats/zeek.md:95:15 MD010/no-hard-tabs Hard tabs [Column: 15]
web/docs/understand/formats/zeek.md:96:13 MD010/no-hard-tabs Hard tabs [Column: 13]
web/docs/understand/formats/zeek.md:97:13 MD010/no-hard-tabs Hard tabs [Column: 13]

Zeek logs have tabs, and it's okay to embed them.


Nnef to nnir

nnef update

This pull request introduces 4 alerts and fixes 1 when merging ec61ba2c8ae5cc2aebe471a9c2b6bb8d19d5f0c0 into 5c12598486e0735c91ea5d0f39def880fc958e78 - view on LGTM.com
new alerts:

4 for Variable defined multiple times

fixed alerts:

1 for Unused import

Comment posted by LGTM.com

This pull request introduces 2 alerts and fixes 1 when merging a8cbbd21ab77ccde5d604a8ce652d7954e90d947 into 5c12598486e0735c91ea5d0f39def880fc958e78 - view on LGTM.com
new alerts:

2 for Variable defined multiple times

fixed alerts:

1 for Unused import

Comment posted by LGTM.com

@hansely is this ready to be merged?

Not yet. Will let you know.

This pull request introduces 2 alerts and fixes 2 when merging ff70f616cfb117ef168502b9e6e6b767a4b9bd7c into 666809f98aa5dbc5334e3edf7c3a16c23fe93a74 - view on LGTM.com
new alerts:

2 for Variable defined multiple times

fixed alerts:

2 for Unused import

Comment posted by LGTM.com


Multiple file modified events fired per file save

Hi,
I'm testing out this nice library in an UE4 setting and noticing that while it is working quite well for add and remove, modified is firing twice whenever I update a file (save it in a text editor). Is this a known thing, or is there some way I can avoid it?
Also, when I add a new file to my watched directory, I see two events: one add and one modified - perhaps this is a hint to the problem. Here is the code I'm testing with in a C++ Actor. Note I'm also printing the current frame, which is sometimes the same for the redundant modified event, sometimes it is a frame later.
// Fill out your copyright notice in the Description page of Project Settings.

#include "FileMonitorTest.h"
#include "./FileWatch.hpp"
#include <string>

#define PRINT_DISPLAY_TIME 3.0f
#define printf(format, ...)                 if (GEngine) GEngine->AddOnScreenDebugMessage(-1, PRINT_DISPLAY_TIME, FColor::White, FString::Printf(TEXT(format), ##__VA_ARGS__), false)

// Sets default values
AFileMonitorTest::AFileMonitorTest()
{
 	// Set this actor to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
	PrimaryActorTick.bCanEverTick = true;

}

namespace {
const char* watchEventTypeToString( const filewatch::Event change_type )
{
	switch (change_type) {
		case filewatch::Event::added:			return "added";
		case filewatch::Event::removed:			return "removed";
		case filewatch::Event::modified:		return "modified";
		case filewatch::Event::renamed_old:		return "renamed_old";
		case filewatch::Event::renamed_new:		return "renamed_new";
		default: break;
	};

	return "(unknown)";
}
}

// Called when the game starts or when spawned
void AFileMonitorTest::BeginPlay()
{
	Super::BeginPlay();

	// we'll add a watcher for file changes in this directory
	std::wstring scanDir( L"E:/epic/Unreal Projects/FileMonitoring/data/scans" );

	IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
	bool dirExists = PlatformFile.DirectoryExists(scanDir.c_str());
	printf("scanDir: %s, exists: %d", scanDir.c_str(), dirExists );

	if( dirExists ) {
		// print out contents of folder using ue4 stuffs
		TArray<FString> fileNames;
		PlatformFile.FindFiles( fileNames, scanDir.c_str(), NULL );
		printf("num files: %d", fileNames.Num() );
		for( int i = 0; i < fileNames.Num(); i++ ) {
			printf( "%s", *fileNames[i] );
		}

		// watch for file changes within scanDir
		static filewatch::FileWatch<std::wstring> watch(
			scanDir,
			[](const std::wstring& path, const filewatch::Event change_type ) {
				FString changeTypeStr = watchEventTypeToString( change_type );using char* directly prints rubbish
				printf("path: %s, event type: %s, frame: %d", path.c_str(), *changeTypeStr, GFrameNumber );
			}
		);
	}
}

// Called every frame
void AFileMonitorTest::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);

}

Another interesting note: if I hit save one an empty file, then I only get one modified event fired. But if it has contents, then I get two modified events fired.

I also have this problem in windows, if I delete and write it is removed\added\modified.
If it is modified directly, it is modified twice.

There seems to be no good way to solve this problem
Most people use the anti-shake function call, or view the last modification time of the file.
You can use procmon to check the occurrence of the event, it should also be twice.

https://stackoverflow.com/questions/1764809/filesystemwatcher-changed-event-is-raised-twice
https://stackoverflow.com/questions/14036449/c-winapi-readdirectorychangesw-receiving-double-notifications


Scan and import from zip files

The tool should (optionally) pick up and display files stored inside zips.

That would be awesome! Especially if it could then extract the one you choose.

FOr me, this is critical. This sets apart the offerings out there. Other apps are only seeing STL and some other 3d formats. To be able to look in and catalog a zip into one "set" would be incredible. What I mean by set is that inside the zip, per model, there is a Supported, Unsupported, STL, LYS files. Obviousy we dont want the catalog to single out each of these, but to have them stacked as a "set". That would be amazing. Sort of like how adobe Lightroom works.


Render schemas on gateway entities

Render the schema on entity pages using the new
spec-renderer and pulling the information from Konnect.
It works similarly to how the api specs work in the docs.
We keep a file app/_data/konnect_oas_data.json with the products'
information we have in Konnect so we can retrieve their API specs.
The file app/_data/konnect_product_ids.yml contains a mapping
of each API spec and its correpsonding product id in Konnect. This
provides a way to access and filter them easily.
The schema is rendered using a vue component that requires 3 arguments:

konnect product id
the version id of the specific product
the path to the schema in the spec

The platform pulls this information from the different configuration
files based on the following metadata in a page's frontmatter:
schema:
  api: gateway/admin-ee
  path: /schemas/Consumer

The key api maps to the following entry in app/_data/konnect_product_ids.yml
gateway:
  admin-ee: 'konnect_product_id'

And the corresponding version information comes from
app/_data/konnect_oas_data.json
NOTES:

This won't work in preview apps/production due to CORS, we need to ask Engineering to add an exception
The vue component I'm using adds an extra title, description and examples. A  new prop will allow us to disable the examples, but we'll need to use a different component if we want to get rid of the title and description. Unfortunately, it's not being exported in the package. We could hide the title and example using css for the demo.
We don't have a schema for Consumer Groups in Gateway EE
You need to set VITE_PORTAL_API_URL (setting it in a .env file would do), its value should be prod's dev portal url


Improve safety of NER build

This PR updates the NER index build to skip entries whose norm terms are none or contain an empty string. This shouldn't happen in practice, but depending on the pipeline used to construct terms, this might happen. A few alternatives to consider:

Have the Term class raise errors when None is given for norm_text
Raise an error in the NER index build

This is actually tricky to solve perfectly. We already throw a ValueError if a Term's text evaluates to False in the constructor (https://github.com/gyorilab/gilda/blob/master/gilda/term.py#L53-L54). So I think checking norm_text the same way could make sense. Still, depending on how the terms are loaded, this could be inconvenient. We could also check for norm_text.strip() I suppose since a norm_text consisting of just spaces doesn't make sense either.


Unable to login on node.js

Hi,  I am having trouble getting the login flow to work on node.js.
I copied the sample code you shared in a comment on issue #19, using a locally generated certificate/private key + API_KEY. Everything compiles and runs.
When the login() routine is invoked, it opens up my default web browser to the TD Ameritrade login page and 1. asks for my account credentials and 2. asks if I want to grant my app permission to access to my account. After a successful auth / grant, the browser is redirected to my REDIRECT_URI (e.g. https://localhost/?code=.....) but the node.js app just sits there.  I tried closing the browser tab and the entire browser, but the node.js app just sits there, apparently not receiving back what it needs from the browser.
What am I missing here?
Please note I am using Windows 10, I am not sure if this matters for how your library handles this login process.  The redirect also appears to be changing my REDIRECT_URI from HTTP to HTTPS so it fails to actually load my local app. Not sure if either of these factors are relevant here but I thought I'd mention both.
Thanks in advance, I'm looking forward to resolving these issues and working with this library.

Hello,
HTTPS is required for OAuth2 authentication. There's instructions in the readme on how to generate a self signed SSL cert. Also, both the local config and the TD App in your TD dev account should point to the same address. Hope this helps!

Hi @knicola ,  thanks for the reply.
Ok I took another crack at this and got the SSL certificate generated, installed into the Windows certificate store and accepted by the Microsoft Edge browser.
Now, the app takes me through the login/grant flow and then gets the code returned to https://localhost:8443/?code=.... but then displays a "Failed to get access token" message, while the node.js app crashes with the following error:
`node:internal/process/promises:246
triggerUncaughtException(err, true /* fromPromise */);
^
Error: write EPROTO B0510000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled:c:\ws\deps\openssl\openssl\ssl\statem\extensions.c:908:
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:98:16)

Node.js v17.1.0`
Is this "SSL unsafe legacy renegotiation disabled" error something I can resolve by generating the certificate in with a different command?
Thanks!

Hey, can you try running it with node v16?

Yes, that worked.   Thanks for your help & creating this project!


triage: Vitessce widget removal - no way to get them to come back if "x'd" out

Feedback on display of viewer for preview (may be generalizable to other previews or actual data views)

 Go to https://portal.hubmapconsortium.org/preview/multimodal-molecular-imaging-data

 Scroll to viewer

 Click x button to right of "Layer Controller" or "Spatial"

 Realize I goofed and now am lost as to how I get it back (blank white screen if close both)

Fill in as much as seems useful:
Describe the issue
To Reproduce
Expected behavior
Screenshots
Environment

Browser:
Browser version:
Portal version: /static/VERSION

Best guesses

Relevant repos:
Developers to involve:

Related to #863

Related to https://github.com/vitessce/vitessce/issues/242


[SPARK-5668] Display region in spark_ec2.py get_existing_cluster()

Show the region for the different messages displayed by get_existing_cluster(): The search, found and error messages.

Can one of the admins verify this patch?

ok to test

Test build #27039 has started for   PR 4457 at commit 4ecd9f9.

This patch merges cleanly.

Test build #27039 has finished for   PR 4457 at commit 4ecd9f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27039/
Test PASSed.

@MiguelPeralvo Do you think we should print the region when launching a cluster, or does that already happen?
Also, I think we should update the help text for the --region option to indicate that this is the region used to find instances, not just launch them.

@nchammas,
Regarding the help text, I'd say that it applies to the destroy, login, reboot-slaves, get-master, stop and start options, not only launch. Something like "EC2 region to launch/destroy/login/reboot-slaves/get-master/stop/start instances in" or "EC2 region where to apply options to: launch/destroy/login/reboot-slaves/get-master/stop/start instances in".
Regarding the region being printed when launching a cluster: As the "get_existing_cluster()"  method gets used by the launch option (also by destroy, login, reboot-slaves, get-master, stop and start), the region will get printed for all these options if the proposed change is applied. I'm in favor of the region being printed out. For example, this would be the beginning of the output of a launch execution:
Miguels-MacBook-Pro:ec2 Miguel$ $SPARK_HOME/ec2/spark-ec2 -k xyzapachespark -i $SSH_HOME/xyzapachespark.pem -r eu-west-1 launch my-spark-cluster

Setting up security groups...
Searching for existing cluster my-spark-cluster in region eu-west-1...
Spark AMI: ami-1ae0166d
Launching instances...
Launched 1 slaves in eu-west-1c, regid = r-679b8425
Launched master in eu-west-1c, regid = r-e69d82a4
...

If we don't apply the proposed change to the "get_existing_cluster()" method, the zone appears, but not the region. For example, this would be the beginning of the output of a launch execution:
Setting up security groups...
Searching for existing cluster my-spark-cluster ...
Spark AMI: ami-1ae0166d
Launching instances...
Launched 1 slaves in eu-west-1c, regid = r-679b8425
Launched master in eu-west-1c, regid = r-e69d82a4
...
Warning: Permanently added 'ec2-54-194-24-152.eu-west-1.compute.amazonaws.com,<IP_ADDRESS>' (RSA) to the list of known hosts.

Test build #27191 has started for   PR 4457 at commit 3923f36.

This patch merges cleanly.

Test build #27191 has finished for   PR 4457 at commit 3923f36.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27191/
Test FAILed.

Test build #27192 has started for   PR 4457 at commit 0a837b0.

This patch merges cleanly.

Test build #27192 has finished for   PR 4457 at commit 0a837b0.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27192/
Test FAILed.

The MiMa fail may be flaky. It passed in my computer.
Jenkins, retest this please.

Test build #27211 has started for   PR 4457 at commit a5514c8.

This patch merges cleanly.

Test build #27211 has finished for   PR 4457 at commit a5514c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27211/
Test PASSed.

Looks good. If there aren't any additional comments, I can merge this soon

LGTM


Wrapper for determining the appropriate profile for a region

Overview
This minor patch will add a wrapper func to determine what profile is necessary for a region.
We're using this logic already in a couple of places, but it can be prone to typos.
Testing/Steps taken to ensure quality
tested on a couple proj; unit tests.

wrong brnach.


[Task] Improved Help

Task Description
Mimic the style of RDanny's help page

Or in this case, completely rip off from RDanny and restyle it to make it for Kumiko


Encodings reported by chardetng-py don't always match up to python's decoding

LookupError: unknown encoding: windows-874

In a previous version, I just passed the entire buffer to encoding_rs and had it handle decoding entirely in rust, but that was a little confusing.
Need more robust aliases

Actually it might be even more complicated than that. https://bugs.python.org/issue25416
I think it's possible that encoding_rs and the python codec will have different output  for legacy single byte encodings

For what it’s worth, I went through and:

Compared the names/aliases. windows-874 is the only one chardetng can produce that does not map correctly in Python, so that’s already fully addressed.

Wrote a quick script to compare the single byte WHATWG encodings definitions (found at https://encoding.spec.whatwg.org/#legacy-single-byte-encodings or more conveniently in git at https://github.com/whatwg/encoding) with the Unicode consortium definitions (found at https://unicode.org/Public/MAPPINGS/). They are all basically identical except for a few exceptions in KOI8-U and windows-1255:
MATCH: IBM866
MATCH: ISO-8859-2
MATCH: ISO-8859-4
MATCH: ISO-8859-5
MATCH: ISO-8859-6
MATCH: ISO-8859-7
MATCH: ISO-8859-8
MATCH: ISO-8859-13
Definitions for KOI8-U do not match!
  Mismatch at 174! WHATWG byte 174 (0xae) = point 1118 (0x045e)) | Unicode byte 174 (0xae) = point 9565 (0x255d))
  Mismatch at 190! WHATWG byte 190 (0xbe) = point 1038 (0x040e)) | Unicode byte 190 (0xbe) = point 9580 (0x256c))
MATCH: windows-874
MATCH: windows-1250
MATCH: windows-1251
MATCH: windows-1252
MATCH: windows-1253
MATCH: windows-1254
Definitions for windows-1255 do not match!
  Mismatch at 202! WHATWG byte 202 (0xca) = point 1466 (0x05ba)) | Unicode byte 202 (0xca) = point <UNDEFINED>)
MATCH: windows-1256
MATCH: windows-1257
MATCH: windows-1258

(The format here is byte <decimal> (<hex>) = point <decimal> (<hex>) where “byte” is the actual byte being decoded and “point” is the unicode code point it should decode as.)
Caveat: I allowed points that are control characters to be treated as undefined for this comparison, which I think is probably reasonable. There are a bunch more encodings that don’t match up otherwise (i.e. a byte is defined as some control character in WHATWG’s definition and is entirely undefined/unmapped in the Unicode Consortium’s definition, or vice versa).

These mismatches are definitely not ideal, but I think are also small enough not to be a problem (at least for the use case here: detecting the encoding of some bytes; not actually decoding/encoding them). None of these bytes show up in chardetng’s frequency data in src/data.rs (assuming I’m reading it correctly), which I think means they aren’t really being considered anyway, so their mapping shouldn’t have a notable impact. (This might all be worth mentioning in some addendum to the the docs, though… idunno.)

Ah, some historical details on the ones that differ between WHATWG and Unicode, if you’re curious:

KOI8-U as defined by WHATWG is actually KOI8-RU. Apparently Internet Explorer historically interpreted KOI8-U this way, and so all other browsers had to follow suit. Discussion: https://github.com/whatwg/encoding/issues/74 and https://www.w3.org/Bugs/Public/show_bug.cgi?id=17053

In windows-1255, the actual implementation in Windows has historically mapped 0xCA to 0x05BA, even though all the specs and code sheets listed it as unmapped. Since WHATWG’s goals are to keep old content working (don’t break the web!), they decided to implement it following how Windows did it in practice, rather than how it was spec'd. Discussion: https://github.com/whatwg/encoding/issues/73

Thanks for looking into that. So it looks like if you use python to decode, it might not be correct. I think probably it's best to just document this and move on.

I'm going to keep this issue open because it's lacking documentation, but it looks like the functional problems are well-addressed.

Yes, good point! I should not have marked that PR as solving this.
I started looking into the multibyte encodings (I think I covered the single-byte ones well), and it seems like the situation is a bit more messy. I’m leaving some notes here so I or someone can document based on them.
Multibyte codecs that Chardetng detects:

GBK
Big5
Shift_JIS
ISO-2022-JP
EUC-JP
EUC-KR

GBK
See #149 for details. Basically, gb18030 is a superset of gbk, which is a superset of gb2312. Chardetng expects decoding with encoding_rs, which treats all of them the same (as gb18030). They are all different decoders in Python, though, so we should (and now do) transform GBK → gb18030, which will work as a decoder in Python for all [correctly detected] cases.
Shift_JIS
Oh dear, this is messy and complicated. Shift_JIS turns out to be a really old and limited spec that has been extended in a multitude of ways that are incompatible with each other.
⚠ TL;DR: If Chardetng returns Shift_JIS, the right way to decode in Python is probably:

Try decoding as ms932.
If that fails, try decoding as shift_jis_2004 (which is a superset of all the others in this family).
If that fails, do whatever your fallback action is for bad detection.

(Caveat: if you want/need to behave like a web browser, skip step 2).
I think it would be lovely if that behavior was bundled up into the detect() function here (but in a way where you could call it separately if you are using EncodingDetector directly), and maybe also use it to inform the confidence of compat.detect(), but I think you could also make the case that it’s out of scope. It also adds in a fair amount of overhead if you aren’t ultimately planning to decode the bytes you’re dealing with (a good reason to put it in detect() and not EncodingDetector).
⚠ End TL;DR
The most popular versions of Shift_JIS are:

Shift_JIS is the original (although in many languages/frameworks, the thing named "shift_jis" is not actually this).

Windows-31J, a.k.a. Windows-932 (in Python: ms932 or cp932) is the most widely used version according to WHATWG and Wikipedia, but I couldn’t find evidence for that cited anywhere (probably just because this was the official/first-part Japanese encoding on Windows). It is a superset of Shift_JIS, with a four exceptions (see below).
The exceptions to being a superset: two notable exceptions (0x5C and 0x7E are treated differently) and two minor ones (0x815F and 0x8160 get visually similar, but not exact same characters). However, in Python, there is a STRICT_BUILD flag that, if false or not set (and it appears to be false on my machine) causes shift_jis to treat 0x5E and 0x7E like Windows-932, and makes Windows-932 handles errors in a slightly different way. I have no idea what environments might set this to true and get “accurate” behavior.

Shift_JISx0213 is a standards-body superset of Shift_JIS that adds a lot of characters. (Note: because of the above strict build stuff, this may or may not be an exact superset of Shift_JIS on any given Python install. 😬)

Shift_JIS_2004 is a standards-body update of of Shift_JISx0213. It adds 10 characters.
On Python, it is not quite an exact superset because one character is different in a weird a subtle way that I’m not sure if is correct or a bug: The bytes 0xFC5A equate to the same glyph, but a different code point! In Shift_JISx0213, it’s U+9B1D and in Shift_JIS_2004, it’s U+9B1C. I think this is mainly a non-issue because the same bytes work with both, and decode to a perfectly equivalent glyph. The only problem is if you do:
some_bytes.decode('shift_jisx0213').encode('shift_jis_2004')

Or vice-versa. That issue is way outside the scope of this package, though.

WHATWG (and Chardetng) treats Windows-932 as Shift_JIS. So technically if Chardetng detects Shift_JIS, it really means windows-932 (or ms932 in Python terms). BUT the detection is loose enough and the overlap in these encodings is big enough that will typically detect Shift_JISx0213 or Shift_JIS_2004 or Shift_JIS content as Windows-932 and give Shift_JIS as a result.
So if Chardetng returns Shift_JIS, the right way to decode in Python is probably:

Try decoding as ms932.
If that fails, try decoding as shift_jis_2004 (which is a superset of all the others in this family).
If that fails, do whatever your fallback action is for bad detection.

(Caveat: if you want/need to behave like a web browser, skip step 2).
In an ideal world, I think that behavior would be bundled up into the detect() function here (but in a way where you could call it separately if you are using EncodingDetector directly), and maybe also use it to inform the confidence of compat.detect(), but I think you could also make the case that it’s out of scope. It also adds in a fair amount of overhead if you aren’t ultimately planning to decode the bytes you’re dealing with (a good reason to put it in detect() and not EncodingDetector).

I didn’t have a chance to dive into the other multibyte ones yet. Quick notes on EUC-KR and Big5, but these need more investigation:
EUC-KR
WHATWG’s and encoding_rs’s definition of EUC-KR is actually UHC/Windows-949, which is an extension on the original EUC-KR. I’m not sure whether that’s a strict superset (in which case this is straightforward and we should swap the name like we did for GBK/gb18030) or not, or any idiosyncracies in how Python treats the two (they are at least separate codecs in Python).
Big5
WHATWG’s and encoding_rs’s definition of Big5 is actually Big5-HKSCS. I’m not sure how much a strict superset this is, or anything else here. Needs more research.
ISO-2022-JP
Haven’t looked into at all yet.
EUC-JP
Haven’t looked into at all yet.

Hm, I think writing up a condensed version of this--something like "these encodings tend to be difficult in general: "--and putting it in the docs might be sufficient.
It's obviously pretty tricky and I appreciate you looking into it.
If someone really wants to do what a browser might do, they should probably take the binary stream and pass it directly to encoding_rs. It's not too bad to add bindings. We could create and encoding_rs_py (lol) binding.

That’s fair. Given that, I think we should probably also (in addition to the docs) rename Shift_JIS to ms932 (like was done with gbk → gb18030) since that is really the closest thing to what chardetng means.
I’m still hoping to write some docs for this, but want to do the research on the other remaining multibyte codecs here first.

More research:
ISO-2022-JP
ℹ TL;DR: None of Python’s built-in encodings is quite equivalent to the WHATWG version of this, but iso2022_jp_ext is the closest (and fairly narrow) superset. For safe usage, it probably makes sense to map ISO-2022-JP to iso2022_jp_ext. ℹ
ISO-2022-JP has a similarly complicated backstory to GBK with incompatible families of extensions. ISO-2022 is a structure for making a codec that is stateful and switches between different sub-encodings when it encounters certain escape sequences (e.g. when it encounters the bytes ESC ( J it switches to interpreting everything after as if it were the roman component of JIS X 0201-1976, and when it encounters ESC $ B it switches to interpreting everything after as if it were JIS X 0208-1983).
Note: JIS X NNNN is an encoding standard, and JIS X NNNN-YYYY is the version of that standard published in year YYYY.

Baseline ISO-2022-JP can switch between:

ASCII
JIS X 0201-1976 (roman characters)
JIS X 0208-1978, and JIS X 0208-1983

Both Python and WHATWG treat both the JIS X 0208 components as JIS X 0208-1983 instead of treating them differently. The -1983 version is a strict superset, so this really just means they are slightly less strict than the specification.

WHATWG ISO-2022-JP is the same as baseline, but it treats the JIS X 0208-1978 version as if it were JIS X 0208-1983 (a superset) and it adds support for:

JIS X 0201-1976 (kana characters) — this and ISO-2022-JP-EXT are the only versions that support the kana characters from JIS X 0201-1976.

ISO-2022-JP-EXT is the same as baseline + support for:

JIS X 0201-1976 (kana characters) (like the WHATWG version above)
JIS X 0212-1990

In Python, this also treats JIS X 0208-1978 as JIS X 0208-1983, like WHATWG does. It’s basically WHATWG + JIS X 0212-1990.

ISO-2022-JP-1 is the same as baseline + adds support for:

JIS X 0212-1990

ISO-2022-JP-2 is the same as ISO-2022-JP-1 + adds support for:

gb2312 (Chinese)
KS x 1001-1992 (Korean)
ISO-8859-1 (the “high” or extended latin part — bytes above 0xA0)
ISO-8859-7 (the “high” or basic greek part — byes above 0xA0)

ISO-2022-JP-3 does not follow from -JP-2. It is the same as baseline + adds support for:

JIS X 0201-1976 (kana characters) (like the -ext and WHATWG versions above) BUT NOT IN PYTHON. Or at least, Wikipedia suggests this is supposed to be supported, but digging through the Python source confirms it is not in Python’s implementation. Trying this in actual Python (v3.12.0) also confirms it is not supported.
JIS X 0213-2000

The basic idea here was to replace all the complicated switches in ISO-2022-JP-2 with JIS X 0213, which pretty much supports it all in a more cohesive way.

ISO-2022-JP-2004 is the same as ISO-2022-JP-3 but replaces the JIS X 0213-2000 bits with JIS X 0213-2004 (effectively a superset).

There are more variations, but these are the most popular and the only ones relevant to Python or WHATWG/chardetng/encoding_rs.

So in Python, nothing is quite equivalent to the WHATWG version (which is what Chardetng is guessing when it says ISO-2022-JP). Straight-up iso2022_jp will fail to decode some things Chardetng thinks are OK, but iso2022_jp_ext is a strict superset of what Chardetng is looking for (the only superset!), so should always succeed. A bit of fiddling around with various characters that are only supported in some of the sub-encodings listed here seems to confirm this in practice (using C Python 3.12.0).
If we are going ahead with remapping names for compatibility, it probably makes sense to map ISO-2022-JP → iso2022_jp_ext.
EUC-JP
ℹ TL;DR: As far as decoding goes WHATWG/chardet/encoding_rs’s concept of EUC-JP is the same as Python’s. We shouldn’t need to treat this one specially. ℹ
EUC encodings are like ISO-2022 and Shift-JIS in that they contain several sub-encodings. Like ISO-2022, they support more/are a bit more flexible than Shift-JIS, but uses a switching mechanism based on what bytes are allowed where, like Shift-JIS, rather than statefully switching modes like ISO-2022. (If you’re familiar with UTF-8, it’s similar.)
For Japanese, there are three popular versions:

EUC-JP combines ASCII, JIS X 0201 kana characters (like ISO-2022-JP-EXT), JIS X 0208, and JIS X 0212. In some versions, the ASCII portion is JIS X 0201 roman characters instead, but both Python and WHATWG/encoding_rs/chardetng use ASCII.
In WHATWG and encoding_rs, the EUC-JP codec does not support encoding JIS X 0212 characters. But they are supported for decoding so we don’t really care for our purposes here.

EUC-JISx0213 combines ASCII and JIS X 0213-2000 (0213 can handle all the stuff from 0201, 0208, and 0212).
Python changes behavior slightly here based on the STRICT_BUILD build-time flag — the same as Shift-JIS. The impact of this is pretty minor and probably not worth worrying about here.

EUC-JIS-2004 is the same as EUC-JISx0213 but upgrades from JIS X 0213-2000 to JIS X 0213-2004 (just a newer version of the standard with a few additional characters).

Neither the WHATWG standard nor encoding_rs/chardetng use anything from JIS X 0213, and pretty much exactly match behavior. We don’t need to do anything special for this family of encodings.

Still remaining:

EUC-KR
Big5

That all makes sense to me. I wonder if documenting it is sufficient, or if we should emit a warning of some kind.
By putting the mapping logic directly into the rust code, we have created a minor problem if the user wants to know what chardetng actually outputs. For instance, if they want to pass the output to a binding of encoding_rs.
Additionally, if we want to emit a warning using the python warnings module, I'm not sure we can really do that because the mapping is already done in rust.

By putting the mapping logic directly into the rust code, we have created a minor problem if the user wants to know what chardetng actually outputs. For instance, if they want to pass the output to a binding of encoding_rs.

So my feel on this is:

Nothing we’ve put in so far really violates this in a problematic way — we’ve only mapped names to their exact equivalents in Python. That said, the name cp874 is not supported in encoding_rs, but there’s no overlap in aliases for that encoding between Python and encoding_rs. I think we made the right choice, given that this is principally a Python library. (Note gb18030 is fine because that name works and has the same meaning in both places).

Mapping names to safe supersets (e.g. ISO-2022-JP → iso2022_jp_ext) is obviously a less clear choice. I definitely hear the argument that this is not quite what chardetng means, and if someone has a WHATWG-compatible codec available, they might want to use that instead, since it’s exactly equivalent. OTOH, the operation this package is fundamentally doing is fuzzily trying to find a workable decoding scheme for some bytes. From that perspective, it might be fair to say that a superset of the detected encoding is almost a better answer. 🤷

Regardless of the above, if this package returned the names chardetng uses, I think it should prefix them, e.g. return whatwg-Shift_JIS instead of Shift_JIS. That prevents confusing collisions, makes it extremely clear what you’re talking about, and lets someone register a WHATWG-compatible codec (e.g. from encoding_rs) with that name in Python so a user can then do some_bytes.decode('whatwg-Shift_JIS') and have it just work. (If there were a hypothetical encoding_rs_py package, I would hope it integrates with Python’s built in encoding machinery this way.)

(Side note: FWIW, after having read up and thought about it more, I think Shift_JIS → ms939 belongs more to the “exact match” category than the “safe superset” category. Also, Python’s canonical name for that is cp939, but both it and encoding_rs recognize ms939, so that one is probably better to use if we did a mapping.)
Some ideas for splitting the difference here…

Add an argument to the EncodingDetector constructor that tells it whether to return Python or WHATWG names. For example:
detector = EncodingDetector(use_python_names=True)
detector.feed(some_ms939_bytes)
detector.guess(tld=None, allow_utf8=False)
# "ms939"

detector = EncodingDetector(use_python_names=False)
detector.feed(some_ms939_bytes)
detector.guess(tld=None, allow_utf8=False)
# "whatwg-shift_jis" or "shift_jis" or "Shift_JIS"

I think .detect() and compat.detect(), which go for simplicity, would set this to use Python names, but the default  could be to use WHATWG/encoding_rs names.

Always return a WHATWG name, but provide a way to register the equivalent Python decoders for it:
chardetng_py.detect(some_ms939_bytes)
# "whatwg-shift_jis"

chardetng_py.register_loose_decoders()
some_ms939_bytes.decode("whatwg-shift_jis")
# Works by using Python’s `ms939` codec.

Interesting, either way. I'll need to think about it for a while.

Oops, forgot a third: have detect() and compat.detect() return Python compatible names, and EncodingDetector return WHATWG names. Provide the transform as a Python function someone call with the result from EncodingDetector.guess() if they want.

Alright, wrapping up with the final two:
EUC-KR
ℹ TL;DR: As far as decoding goes, WHATWG/chardet/encoding_rs’s concept of EUC-KR is equivalent to Python’s uhc/ms949/cp949. Python’s euc-kr is a subset (sort of) and not safe to consider equivalent. ℹ
EUC-KR is structured like EUC-JP, but just uses different sub-encodings. The original version had some major drawbacks, so both Microsoft and Apple developed their own extended versions:

EUC-KR is the original, and combines ASCII (or ISO-646:KR, which is almost the same) with KS X 1001 (for all the Korean characters). In Python, this is called euc_kr. It doesn’t exist in WHATWG-land.

UHC (Unified Hangul Code) or Windows-949 is Microsoft’s extensions on top of EUC-KR. In terms of bytes handled, it is a strict superset of EUC-KR, BUT it handles some byte sequences in a different way. It is by far the most popular version of EUC-KR and is also the most popular Korean-specific encoding. In Python, this is called cp949 or ms949 or uhc. In WHATWG, this is EUC-KR.

What’s with the note about handling some byte sequences differently?
In Korean (like Chinese and Japanese), most characters are made by combining several simpler characters. The big problem in EUC-KR was that only some of these more complex characters were directly supported. It handled others by recognizing certain sequences of simplified characters that make them up.
For example, the character “샾” can be represented as a single 2-byte sequence in UHC, but not in EUC-KR. Instead, EUC-KR represents it with a special sequence start byte pair followed by the 3 characters that make up the character:
Bytes:              A4 D4 A4 B5 A4 C1 A4 BD
                    ^^^^^ ----- ^^^^^ -----
                      │      │    │     └─ ㅍ
                      │      │    └─ ㅑ
                      │      └─ ㅅ
                      └─ <start of sequence>

Decoded as EUC-KR:  샾
    (Code points):  U+C0FE (HANGUL SYLLABLE SYAP)

Since UHC can encode this character as a single 2-byte pair without the special sequence handling (98 DE), it doesn't decode the sequence as a single character, it just treats it literally:
Bytes:              A4 D4 A4 B5 A4 C1 A4 BD

   Decoded as UHC:  ㅅㅑㅍ
    (Code points):  U+3164 ("<invisible>" HANGUL FILLER)
                    U+3145 ("ㅅ" HANGUL LETTER SIOS)
                    U+3151 ("ㅑ" HANGUL LETTER YA)
                    U+314D ("ㅍ" HANGUL LETTER PHIEUPH)

Note that U+3164 isn't a visible or meaningful character; it just indicates that this started a sequence so that encoding can safely do a round-trip. In fact, as far as I understand it, that is the whole reason the sequence decodes this way instead of to a single character in UHC:
some_euc_kr_bytes == some_euc_kr_bytes.decode('uhc').encode('uhc')

Mac OS Korean is Apple’s extensions on top of EUC-KR. It’s a little more complicated. More importantly, it is not supported at all in Python or WHATWG, so I didn’t investigate this one in a lot more depth.

This one is ultimately pretty simple. WHATWG’s EUC-KR is the same as Python’s uhc or ms949. Unfortunately there is no alias that is commonly supported in both Python’s built-ins and encoding_rs (similar to the situation with cp874).
Big5
ℹ TL;DR: As far as decoding goes, WHATWG/chardet/encoding_rs’s concept of Big5 is equivalent to Python’s big5-hkscs (there are some other aliases, but this one works in both Python and encoding_rs, so it’s the one to use). ℹ
The backstory on Big5 is pretty complex and I’m starting to run out of steam here. So here’s a short summary: Big5 was created in a somewhat ad-hoc way by by various computer manufacturers in Taiwan, and was eventually standardized for interoperability. It was still pretty limited, though, so there there is a whole mess of more specialized encodings that extend it. In particular, the characters needed in different regions that used Traditional Chinese characters (e.g. Hong Kong, Singapore, Macau) needed a lot more characters.
Ultimately, one extension (Eten) was extended by the Hong Kong government (GCCS) which was then extended and standardized as HKSCS. It’s generally the most popular of many forks and forks-of-forks. This is the encoding WHATWG means by Big5 (with some encoding idiosyncracies, but we only care about decoding here). In Python, it’s called big5hkscs. Both encoding_rs and Python support the alias big5-hkscs, so if we want to map Big5 to something, it should probably be that.
Note: It’s possible there are some weird edge cases here. I tested several different sets of bytes and characters and encodings a bit, but have run out of steam to read through the decoder source in Python and encoding_rs and the encoding spec, etc., like I have for some of these others.

Overall summary of encodings and their support/similarities/differences between WHATWG/encoding_rs and Python’s built-in codecs:

Single-byte?
WHATWG Name
Python Builtin
Equivalent
Notes

Yes
IBM866
"
✅ Yes

Yes
ISO-8859-2
"
✅ Yes

Yes
ISO-8859-4
"
✅ Yes

Yes
ISO-8859-5
"
✅ Yes

Yes
ISO-8859-6
"
✅ Yes

Yes
ISO-8859-7
"
✅ Yes

Yes
ISO-8859-8
"
✅ Yes

Yes
ISO-8859-13
"
✅ Yes

Yes
KOI8-U
"
❌ No
WHATWG’s version is actually KOI8-RU, which Python does not have built-in support for. 2 mismatched bytes. 0xAE is “ў” in WHATWG and “╝” in Python. 0xBE is “Ў” in WHATWG and “╬” in Python. There is no better matching codec in Python.

Yes
windows-874
cp874
✅ Yes
Different name, but exact same results.

Yes
windows-1250
"
✅ Yes

Yes
windows-1251
"
✅ Yes

Yes
windows-1252
"
✅ Yes

Yes
windows-1253
"
✅ Yes

Yes
windows-1254
"
✅ Yes

Yes
windows-1255
"
❌ No
1 mismatched byte. 0xCA is U+05BA [HEBREW POINT HOLAM HASER FOR VAV] in WHATWG and undefined in Python.

Yes
windows-1256
"
✅ Yes

Yes
windows-1257
"
✅ Yes

Yes
windows-1258
"
✅ Yes

No
GBK
gb18030
✅ Yes
This Python name is an alias that works in both Python and encoding_rs.

No
Big5
big5-hkscs
✅ Yes
Different name, but exact same results.

No
Shift_JIS
ms932
✅ Yes
This Python name is an alias that works in both Python and encoding_rs. These are basically the same, although there is a little complexity on the Python side, where a couple characters decode differently depending on the STRICT_BUILD flag that is set at build-time on Python.

No
ISO-2022-JP
iso2022_jp_ext
❌ No
The Python codec handles a superset of the WHATWG encoding here. Some bytes that would fail in chardetng/encoding_rs will decode OK with it in Python.

No
EUC-JP
"
✅ Yes

No
EUC-KR
uhc
✅ Yes
Different name, but exact same results.

Notes:

This treats any control characters as equivalent even if one of the codecs doesn't support them. You may have to decode and ignore/skip errors to get exactly matching output if you are handling bytes that include control characters.
" denotes names that are the same as the matching built-in codec in Python.

This is overkill in the best possible terms. I definitely think it makes sense to document this behavior. I would almost immediate accept a PR that updates the docs with some information on the more difficult charsets.

Quick update: I ran into some weird behavior with Big5 today and had more energy to dive into the details. It turns out to be kinda weird! I updated my detailed comments above if you want to know more, but the short version is: it’s the only one where there is not a clear equivalent/safe superset in Python’s built-in codecs. big5-hkscs, which I’d recommended as equivalent before, is still close, but less close than I’d thought.


Remove reactions from everyone when paginating and waiting for trashcan reaction.

Closes #1454
This will remove reactions when someone adds it, even if they aren't whitelisted to do so.

When you commit, there should be something like this:

You shouldn't have to run precommit every single time, just once when you setup the project.

When you commit, there should be something like this:

You shouldn't have to run precommit every single time, just once when you setup the project.

Oh so I only have to pipenv run pre-commit once?

I'm thinking we should squash this when we merge due to the messy history at the beginning.

Like a new PR?

Never mind I read it incorrectly

create_task doesn't raise the exception. It has to be suppressed within the coroutine being scheduled.

create_task doesn't raise the exception. It has to be suppressed within the coroutine being scheduled.

So don't suppress it at all?

create_task doesn't raise the exception. It has to be suppressed within the coroutine being scheduled.

So don't suppress it at all?

No. I explicitly wrote it has to be suppressed. You're just suppressing it at the wrong time/place.

create_task isn't where the coroutine is being scheduled?

A correct example:
async def coroutine():
    with suppress(SomeException):
        return await do_thing()

create_task(coroutine)

Oh, so I need to create a function that suppresses HTTPException?

Yes... unless you can think of a nicer way.

Yes... unless you can think of a nicer way.

Would a new utility function work since I'm using it multiple times?

But I don't see why suppress the exception if scheduling.create_task logs the exception and ultimately suppresses it.

Yes... unless you can think of a nicer way.

Would a new utility function work since I'm using it multiple times?

Yeah, sure. See how much you can turn into a utility function. Maybe some logic in the check could be moved to the utility function too.

But I don't see why suppress the exception if scheduling.create_task logs the exception and ultimately suppresses it.

It suppresses the raising of the exception, but it does not suppress the logging of it. It indiscreminantly logs all exceptions. We don't need to see a log for when the message is missing because it's not actually an error that we need to fix.

Right, I forgot to append the _ after reaction, user, but I never saw that before.

Still getting the same error. Can you please test this yourself before pushing the fix?

Still getting the same error. Can you please test this yourself before pushing the fix?

I've tested multiple times and haven't gotten that error, I did get a different one though.

Definitely still getting the same error... I think reaction.message.remove_reaction(reaction_.emoji, user_) needs to be reaction_.message.remove_reaction(reaction_.emoji, user_)

Definitely still getting the same error... I think reaction.message.remove_reaction(reaction_.emoji, user_) needs to be reaction_.message.remove_reaction(reaction_.emoji, user_)

Yeah that makes sense. I'm just wondering why I didn't get any errors.

Ah it's because I reacted to it first so reaction got defined.

Okay, that's fixed, but now there is a different bug. I think the bot is detecting its own reaction as a deletion. It happens fast, so it's hard to tell what's going on. Enter !codeblock and observe that the message immediately gets deleted.

Fixed.


upon resizing the terminal the prompt flashes and is deletable

happens when the edge of the terminal hits the prompt

this was a weird edge case, and I think is actually just a limitation of xterm.js, so I'll close for now


Fix clear introspection signature cache

[x] tests and linter show no problems (npm t)
[x] code is properly formatted (npm run fmt)
[x] description of changes is added in CHANGELOG.md

Landed in https://github.com/metarhia/impress/commit/66c9555cae0acf220e53c0ff47714189511fe3f4


Saving following file deletion does not replace the deleted file

Describe the Bug
When one creates a new diagram, saves it (triggers the save modal), deletes the resulting file via the OS, and saves again, no file is stored on disk. This is due to the fact that no modification was made to the diagram, and hence the save button does nothing. This is inconsistent with most software out there which will still rewrite the file for you. It's an edge case and not a crucial one, but deserves fixing IMO.

Steps to Reproduce

New diagram
Save (modal)
Delete file via filesystem
Save again (no modal)
File is not rewritten

Expected Behavior
File should be rewritten in (5), as per most software out there. (VSCode, notepad, notepad++ kinda)
Environment

OS: Windows 10
Camunda Modeler Version: 5.0.0

This behavior also exists on Linux

I can reproduce it on Mac as well.

I would propose the following change:
Remove the dirty/unsaved check before the save.
https://github.com/camunda/camunda-modeler/blob/e72740c750e9593f890eaeed41332ccc855f8c40/client/src/app/App.js#L1391-L1394
This will work the same as with other Editors (checked with VSCode, overrides file even when nothing was changed). I think it is the most pragmatic option, as we would otherwise have to check if the file was removed/modified externally/... since we last saved it.

closed via https://github.com/camunda/camunda-modeler/pull/2946


Index of mouse event always zero?

I am trying to add tooltip to chart. But It seem index for mouse enter event is always zero?
This is the code I am using:
        svg.append("g")
                .datum(data)
                .attr("class", "candlestick")
                .on('mouseover', function(d, i){
                    console.log('Index: ' + i );
                    var d = d[i];
                    tip.html(function(){
                            var html = "Date: " + d.date + "
";
                            html += "Open:  " + d.open + "
";
                            html += "High:  " + d.high + "
";
                            html += "Low:  " + d.low + "
";
                            html += "Close:  " + d.close + "
";
                            return html;
                    })
                    .style({
                        left: d3.event.pageX + 20 + 'px',
                        top: d3.event.pageY - 20 + 'px',
                        display: 'block'
                    });

                })
                .on('mouseout', function(){
                    tip.style('display', 'none');
                })
                .call(candlestick);

JS fiddle http://jsfiddle.net/sisir/ghox8ewa/4/ (line #79)

Technically there is only 1 element ('g.candlestick') you have registered a mouseover event with which it's __data__ is an array of data points, so it will always issue you the same data. The library also  has a performance trade off in that each individual candle does not equal a single dom element, so tool tips using mouseover events need a little bit more work.
The two ways I generally do this is by:

Add a clear rectangle that covers the chart, add the mousemove/mouseover event then use var domainIndex = financetime.invertToIndex(coordinate[0]). This result represents an index in the scale domain. Use the result to convert to your data: data[domainIndex] // Use this value to render in your tooltip
Use crosshair which will do above for you. It also exposes events that pass the coordinates of the mouse which you can then use to convert to an index to obtain your data also using financetime.invertToIndex(coord[0]) See http://bl.ocks.org/andredumas/045f550b72ad46301130. You could modify the move callback and do this processing there.

I found both of these methods to be greatly more performant than a node per plotted candle with mouse events registered to each. The chart will feel more responsive.

Updated example: http://jsfiddle.net/ghox8ewa/6/

Okey, if I understand correct we are now getting index of the hovered element on the dom and using it to get the data, right?