Implementation of MultiClass SVM

Support Vector Machines (SVM) is a supervised machine learning algorithm for classification and regression tasks. SVM works by finding the optimal hyperplane that separates data points of different classes in a high-dimensional space. The goal is to maximize the margin, which is the distance between the closest data points (called support vectors) from each class and the separating hyperplane. In cases where the data is not linearly separable, SVM uses kernel functions (e.g., polynomial or radial basis function) to map the data into a higher-dimensional space where a linear separation is possible. By solving an optimization problem, SVM ensures that the best hyperplane is chosen, providing robust classification with good generalization to unseen data.
Requirements for implementation:

Nonlinear SVM: Implementing kernel functions to map data to higher dimensions.
Quadratic Programming (QP): The optimization problem SVM needs to solve is a convex QP problem. DAPHNE will need an efficient QP solver for this step.
Linear Algebra Libraries: DAPHNE likely integrates with linear algebra libraries optimized for its backend. It could be leveraged for matrix multiplications, dot products, and more.
Sparse Data Support: DAPHNE is designed for high-performance scenarios, so handling sparse datasets efficiently will be key to ensuring that the SVM scales to large datasets.
DAPHNE handles large-scale matrix and tensor operations. DAPHNE's optimized routines can be used to handle matrix multiplications, transpositions, and dot products.
SVM also relies on vector norms and dot products, which can be efficiently implemented using DAPHNE’s underlying
To search for the best parameters like C and gamma, a strategy like Grid Search, Random Search, or Bayesian Optimization, paired with cross-validation is needed to avoid overfitting.

I am going to work on it.

I am going to work on it.
It is done, but I do not know how to test the code on the DAPHNE platform and add the codes to the DAPHNE repository.

Hi @FAmirjani, thanks for the support! An SVM implementation in DaphneDSL would be very welcome. We collect reusable DaphneDSL scripts that users can import and call as functions in their own DaphneDSL scripts in scripts/algorithms/ (see, e.g., scripts/algorithms/decisionTree_.daph for a recent example).
Yes, it would be great if you could also contribute test cases for the SVM script. The test cases should make sure that your script (1) successfully compiles and executes in DAPHNE (without crashes) and (2) yields meaningful results. General guidelines on testing in DAPHNE can be found in the documentation. For your specific case, I recommend having a look at the test cases for decision trees (test/api/cli/algorithms/DecisionTreeRandomForestTest.cpp) as a recent example. These execute small DaphneDSL scripts (e.g., test/api/cli/algorithms/decisionTreeRealData2.daphne) on a small real-world data set in test/data/wine/ and verify if the resulting accuracy is satisfactory. In case you ported the SVM script from Apache SystemDS, you could also port their test cases to DAPHNE. Don't hesitate to reach out if you need additional hints.
Once your contribution is ready, you can create a pull request (see our contributions guidelines).


Refactor indentation example.

More Comments
Avoid manual token emitting, it has pitfalls
Make sure the Custom Token patterns docs do not refer to manual token emitting.

@christianvoigt You may want to keep track of this.

Thanks, will do.

After reading https://docs.python.org/release/2.5.1/ref/indentation.html
again it seems that manual token emitting matches the algorithm in the spec much better.
This is probably not the only use case where manual Token emitting will be required.
Perhaps it would be possible to mitigate / reduce the pitfalls and provide an API
to support manual emitting.
Need to be investigated farther.

Closing this, will track emitting feature in #415


Update _login_event_handler method to save unsaved sale_line #6889

Coverage remained the same at 68.82% when pulling 323fd722d2477d56174df61c9f24f1691e72f42e on tarunbhardwaj:active-record into 9187f586ab7286c1bfe81608d809e022f05c126a on openlabs:develop.

@PritishC send a PR for tests fix please


Setup Flowbite to work with Phoenix LiveView

I have gone through the guide for setting up Flowbite in Phoenix here, however perhaps the guide might need to be updated.
This issue is similar to #88. Due to the way Phoenix LiveView works, Flowbite components that require javascript stop working when to navigate between LiveViews. This is because LiveView uses a Websocket, does not do a full page reload but just patching the DOM, therefore it does not dispatch  load event which should initialize Flowbite javascript after a page is loaded.
There are some ways to fix this:

Update the assets/js/app.js to dispatch the load event once the LiveView is loaded:

window.addEventListener("phx:page-loading-stop", _info =>
    topbar.hide()
+    window.dispatchEvent(new Event("load"), {
+        bubbles: true,
+        cancelable: true
+    })
})

Similar to the Flowbite js(/node_modules/flowbite/lib/esm/index.turbo.js) for Turbo, we can have js file specifically for Phoenix LiveView. In the file we have something like this:

// setup events for data attributes
-var events = new Events('load', [
+var events = new Events('phx:page-loading-stop', [
    initAccordions,
    initCollapses,
    initCarousels,
    initDismisses,
    initDropdowns,
    initModals,
    initDrawers,
    initTabs,
    initTooltips,
    initPopovers,
    initDials,
]);
events.init();

I have tested both approaches and they work, though for now I think the first approach would suffice, and perhaps included in the setup guide for Phoenix.
I have attached a gif illustrating the issue where the Flowbite components stop working when I navigate between LiveViews and how Bootstrap components worked without making changes in app.js.

Hey @jmnda-dev,
Thanks for the valuable insight, feedback and solutions.
I think that we should go with solution number two and create a flowbite.phoenix.js file just like for Ruby on Rails 7.
If you want to I would be more than happy to overlook a PR for this and would be a great opportunity for you to contribute to open-source, if it's something you'd like of course.
We can also use this starter repo as testing grounds and reference: https://github.com/themesberg/tailwind-phoenix-starter
Ultimately, we would only need to update the docs and we're all good to go :)
Cheers,
Zoltan

Thank you @zoltanszogyenyi, I will create a PR for this issue


Fix segmentInfos replace doesn't set userData

Description
The replace method of SegmentInfos does not overwrite the userData field. This PR fixes that bug. Relevant issue is opened here.

Hi @Shibi-bala and great to see you here. Let's sync up this week and maybe we can help move this PR forward. It's a good catch, so thank you.

Kind of confused why this check is failing. This was never changed and I've tried merging.
1```
. ERROR in /home/runner/work/lucene/lucene/lucene/test-framework/src/java/org/apache/lucene/tests/store/MockDirectoryWrapper.java (at line 49)
import org.apache.lucene.index.SerialMergeScheduler;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Task :lucene:test-framework:ecjLintMain FAILED
The import org.apache.lucene.index.SerialMergeScheduler is never used

Have you merged in the latest main branch, so this PR is uptodate? This could be an issue which already existed when the PR was created.

@uschindler Ah I needed to re-sync my forked repo 😅

Looks good now.

@uschindler hey, thanks for the approval! Read the contributing guidelines, but not entirely sure how to get permissions to merge this PR.

You can't do it.
Please add a Changes entry unter the 9.9 section, commit it to branch and I will merge and Backport your PR.
I am just away from my computer at moment, sorry vor the delay.

@Shibi-bala It's here:
https://github.com/apache/lucene/blob/c228e4bb66ca73c8150d8eaebe2bb999bcc6c9b1/lucene/CHANGES.txt#L147
You need to include your user and the GitHub user(s) that reviewed it/assisted as seen in the other entries.

Made the changes. Thanks @uschindler @MarcusSorealheis  @msfroh  😁


Ability to specify package repository for development

It could be convenient for contributors if there was a way to temporarily change the package repository URL so package changes could be tested before submitting a pull request to the main package repository.
I don't think this would need a UI, just allowing it to be changed via the command line (eg, defaults write) would be sufficient for contributors.

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

:+1:

This is actually on master now thanks to @kattrali!


Ensure all rare OMIMs are classified under rare disease

Related to #170
We want to make sure that any rare OMIMs are classified under rare disease (note we will later obsolete this grouping class, but we will retain a property).
The 4th column here shows there are many OMIMs that are not categorized as rare

From https://github.com/monarch-initiative/mondo-analysis/blob/master/Mondo_Analysis_seaborn.ipynb
I believe the default for any mendelian should probably be rare

@cmungall is there a way to give me a list of all the OMIM diseases that are not classified as rare diseases but should be?
I can then add this superclass assertion in Protege.

@pnrobinson and @ShahimEssaid said they can help with this too

@cmungall, Deepak sent me a list of all the OMIM diseases that are not classified as rare-should all of these diseases should have the super class assertion rare disease? Should I do this manually in Protégé? I'm happy to do so, but just want to make sure there not a way to do it automatically, as doing this manually will take some time.

@nicolevasilevsky can you send me this list?

Sure thing - I uploaded it to google drive here: https://docs.google.com/spreadsheets/d/1x9VCToGoEOY9XIYOc4IDDAA450dvmuADy_G5QI7tZRo/edit#gid=1666029332

Let's first obsolete any new entries in
https://github.com/monarch-initiative/hpo-annotation-data/blob/master/rare-diseases/annotated/omit-list.txt,
we can script this
On Tue, Mar 19, 2019 at 7:59 PM Nicole Vasilevsky<EMAIL_ADDRESS>wrote:

Sure thing - I uploaded it to google drive here:
https://docs.google.com/spreadsheets/d/1x9VCToGoEOY9XIYOc4IDDAA450dvmuADy_G5QI7tZRo/edit#gid=1666029332
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/monarch-initiative/mondo/issues/629#issuecomment-474665155,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGObkffM2U8vrvW4dnkDAt4ikSYn1dks5vYaQegaJpZM4bTZwg
.

Would it be better to just drop the "rare" as an upper level classifier? We could instead add an attribute such as prevalence, which could be set to either Orphanet's prevalence or just "rare" (some ontology term) for OMIM entries for which we have no specific data.

@cmungall @nicolevasilevsky  -- the categorization "is rare" in this file seems to be often wrong (unless it is being used as a flag to denote legacy or entries otherwise deemed to be of low quality)

It seems to me we'd want the upper classifier but only populate it based on inference from an attribute. I don't know that all OMIM diseases should be considered Rare? we need to know prevalence to determine that. Sorry if i missed something, agree with Peter.

Things like sickle cell anemia and Tay–Sachs disease are not really rare (though obviously both demographic dependent).

Yes, remember there is a ticket for obsoleting all rare-X terms (replace
with a property) including the rare root, we are just using this now
because that is what was required for the analysis. Anything currently
classified under rare comes from ORDO, you can see the complete provenance
as axiom annotations in the OWL version.
On Wed, Mar 20, 2019 at 5:18 AM Melissa Haendel<EMAIL_ADDRESS>wrote:

Things like sickle cell anemia and Tay–Sachs disease are not really rare
(though obviously both demographic dependent).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/monarch-initiative/mondo/issues/629#issuecomment-474805149,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGOVBspLpkW0D5tLFPqin2icxUVOYLks5vYicegaJpZM4bTZwg
.

I would say that sickle and Tay Sachs are rare, in the biological sense, and certainly each of these diseases is rare in most populations. It does not make sense to use the prevalence as something that is extremely meaningful!

well, ~2% of Nigerian kids are born with Sickle cell disease. rare worldwide, but not in some places ;-). We need to refine the model and definitions to build  this on the fly. Big project to do well...

is there an open action item here?

can this be closed?


Need to handle error when 'Run Tests' is pressed and there is a fatal TADA error raised

Describe the bug
When loading data from an Excel file using the 'Run Test' button on the 'Flag' tab panel causes a fatal error that locks the screen for the user.  In the console during debugging the message is
Error: (converted from warning) Error in dplyr::mutate: ℹ In argument: `MeasureQualifierCode.Split = strsplit(MeasureQualifierCode, ";")`.
Caused by error in `strsplit()`:
! non-character argument

To Reproduce
Steps to reproduce the behavior:
1. In the 'Load' tab panel, use the "Option C: Upload dataset" option to load the user file tada_output_utah_10rows.xlsx
2. Click on 'Flag' tab.  In the Flag tab panel, click on the 'Run Tests' button
3. See UI error in the browser

If running in developer mode, the following is printed to the console
Error: (converted from warning) Error in dplyr::mutate: ℹ In argument: `MeasureQualifierCode.Split = strsplit(MeasureQualifierCode, ";")`.
Caused by error in `strsplit()`:
! non-character argument

If applicable, include code to reproduce the behavior:
By using a browser() statement, I can see that the MeasureQualifierCode are all the letters 'NA' which in the excel file are blank.
Browse[1]> tadat$raw$MeasureQualifierCode
 [1] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Reminders for TADA contributors addressing this issue
Bug fixes should include the following work:

[ ] Create or edit the code.

[ ] Document all code using line/inline and/or multi-line/block comments
to describe what is does.

[ ] Create or edit tests in tests/testthat folder to help prevent and/or
troubleshoot potential future issues.

[ ] If your code edits impact other functionality in the shiny
app, ensure those are updated as well.

[ ] Run styler::style_pkg(), devtools::document(), and devtools::check()
and address any new notes or issues before creating a pull request.

When run on the excel file I get this result

Browse[1]> .data$MeasureQualifierCode  [1] NA NA NA NA NA NA NA NA NA NA Browse[1]> .data$MeasureQualifierCode.Split Error: (converted from warning) Unknown or uninitialised column: `MeasureQualifierCode.Split`. Browse[1]> flag.datazoom <- .data %>% dplyr::mutate(MeasureQualifierCode.Split = strsplit(MeasureQualifierCode, ";")) Error in dplyr::mutate(., MeasureQualifierCode.Split = strsplit(MeasureQualifierCode,  :    ℹ In argument: `MeasureQualifierCode.Split = strsplit(MeasureQualifierCode, ";")`. Caused by error in `strsplit()`: ! non-character argument
--

Browse[1]> | Browse[1]>
Browse[1]>

Here is some test of the excel, running in TADA_FlagMeasureQualifierCode()
Browse[2]> .data$MeasureQualifierCode
 [1] NA NA NA NA NA NA NA NA NA NA
Browse[2]> is.character(.data$MeasureQualifierCode)
[1] FALSE
Browse[2]> typeof(.data$MeasureQualifierCode)
[1] "logical"
Browse[2]>


tools: add option to use in-tree board-level common code

Summary
This PR fix the failure reported by users after https://github.com/apache/incubator-nuttx/pull/5274
Impact
Custom boards users
Testing
Pass CI

How are the other options going to be used now? I mean those chip specific like STM32_COMMON etc.
Can they be replaced by this new option?

No. Those will coexist. The STM32_COMMON is about to use the code or not even for the boards that are located in NuttX repo. So we can have in-tree stm32 based boards that will not use STM32_COMMON. I will explore if the option is needed and will issue a separate PR to clean it up.
This patch is intended to address issues reported by users while keeping an option that was added by initial PR.

I can't actually test this right now, but I like the idea that users can disable the common directory for out-of-tree boards.

Currently next in-tree board archs have common:
./boards/xtensa/esp32s3/common
./boards/xtensa/esp32/common
./boards/xtensa/esp32s2/common
./boards/arm/rp2040/common
./boards/arm/cxd56xx/common
./boards/arm/stm32/common
./boards/arm/samv7/common
./boards/risc-v/mpfs/common

and there are only two options available BOARD_STM32_COMMON and BOARD_SAMV7_COMMON. So we need either to equip the rest of the cases with BOARD_ARCH_CHIP_COMMON option go to go with CONFIG_BOARD_CUSTOM_ARCH_BOARD_COMMON. Personally I do not know what is better and taking into account next release time frame I'm not sure that I will be able to add  BOARD_ARCH_CHIP_COMMON to all missing places. So I made this simple change that covers error reported by users.

As far as I can tell, these specific options (like BOARD_STM32_COMMON) are used only in the common folder's Kconfig file to select the source files to build.  Wouldn't be better to just replace it with this new config? We won't have to add other chips options and we won't have to deal with this every-time we introduce the common folder for a family.

As far as I can tell, these specific options (like BOARD_STM32_COMMON) are used only in the common folder's Kconfig file to select the source files to build. Wouldn't be better to just replace it with this new config? We won't have to add other chips options and we won't have to deal with this every-time we introduce the common folder for a family.

The specific options (like BOARD_STM32_COMMON) are used by in-tree configs. The in-tree board common is mapped via $(wildcard $(BOARD_DIR)$(DELIM)..$(DELIM)common) and not via $(wildcard $(TOPDIR)$(DELIM)boards$(DELIM)$(CONFIG_ARCH)$(DELIM)$(CONFIG_ARCH_CHIP)$(DELIM)common), so with current code we can't eliminate specific options (like BOARD_STM32_COMMON) because $(wildcard $(BOARD_DIR)$(DELIM)..$(DELIM)common) will always exists for STM32 based in-tree boards.

I have made alternative PR https://github.com/apache/incubator-nuttx/pull/5712 that seems to be move generic. Please take a look.


Latest releases not released as docker image

https://github.com/hyperledger-labs/fabric-operator/pkgs/container/fabric-operator
is behind
https://github.com/hyperledger-labs/fabric-operator/releases

This is fixed now, we can see new images pushed on a release.


Upload fails for .gitignore

Do you read the FAQ?

[X] Yes.
[ ] I am going to read now.

Describe the bug
Uploading a .gitignore file appears to fail in any project on MacOs while uploading any other file works as expected.
To Reproduce
Steps to reproduce the behavior:

Edit .gitignore file
Save file and upload to remote
Get error in console

Expected behavior
Expected .gitignore file to be uploaded to remote server
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: MacOS v10.12.6
VSCode Version: 1.30.2
Extension Version: 1.9.3

Extension Logs from Startup - required
[error] Error: Failure
	at SFTPStream._transform (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2-streams/lib/sftp.js:412:27)
	at SFTPStream.Transform._read (_stream_transform.js:186:10)
	at SFTPStream._read (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2-streams/lib/sftp.js:183:15)
	at SFTPStream.Transform._write (_stream_transform.js:174:12)
	at doWrite (_stream_writable.js:387:12)
	at writeOrBuffer (_stream_writable.js:373:5)
	at SFTPStream.Writable.write (_stream_writable.js:290:11)
	at Channel.ondata (_stream_readable.js:639:20)
	at emitOne (events.js:116:13)
	at Channel.emit (events.js:211:7)
	at addChunk (_stream_readable.js:263:12)
	at readableAddChunk (_stream_readable.js:250:11)
	at Channel.Readable.push (_stream_readable.js:208:10)
	at SSH2Stream.<anonymous> (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2/lib/Channel.js:167:15)
	at emitOne (events.js:116:13)
	at SSH2Stream.emit (events.js:211:7)
	at parsePacket (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2-streams/lib/ssh.js:3291:10)
	at SSH2Stream._transform (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2-streams/lib/ssh.js:689:13)
	at SSH2Stream.Transform._read (_stream_transform.js:186:10)
	at SSH2Stream._read (/Users/my_user/.vscode/extensions/liximomo.sftp-1.9.3/node_modules/ssh2-streams/lib/ssh.js:249:15)
	at SSH2Stream.Transform._write (_stream_transform.js:174:12)
	at doWrite (_stream_writable.js:387:12)
	at writeOrBuffer (_stream_writable.js:373:5)
	at SSH2Stream.Writable.write (_stream_writable.js:290:11)
	at Socket.ondata (_stream_readable.js:639:20)
	at emitOne (events.js:116:13)
	at Socket.emit (events.js:211:7)
	at addChunk (_stream_readable.js:263:12)
	at readableAddChunk (_stream_readable.js:250:11)
	at Socket.Readable.push (_stream_readable.js:208:10)
	at TCP.onread (net.js:594:20)

https://github.com/liximomo/vscode-sftp/blob/master/FAQ.md#error-failure

check directory is exist on remote server
check directory permission

@Shaffron The directory exists, and the permissions are correct as well. I actually just noticed this issue with another set of files.. I've got a contact.php and a sendContact.js file .. both permissions are identical and they are located in the same directory. sendContact.js uploads correctly, contact.php fails every time. It can however be downloaded without issue.

I have the same issue.
To reproduce, create a local .gitignore and try to upload it.


Intrinsify Closue_equals

Many profile samples are taken in this runtime call during Flutter animations on Fuchsia.
/cc @a-siva

Here is the runtime implementation.
https://github.com/dart-lang/sdk/blob/master/runtime/lib/function.cc#L33

The Flutter use case which inspired this issue is actually about storing many closures in the HashSet and frequently calling contains() to check is a particular closure in the set. In such case, the profile showed that majority of time is spent in Closure_equals.
However, I found that Closure_equals is called excessively due hashcode collisions. Hashcode for closures  was calculated as the hascode for the underlying function, so it is the same all implicit instance closures created for a particular method, regardless of different receiver objects. After improving hashcode for closures in CL https://github.com/dart-lang/sdk/commit/203955bc48fdbd03b7b030c4e6dbcd20920201ec, there is a significant speedup of HashSet.contains() for closures and Closure_equals is no longer the bottleneck in this use case.


Does not work with sealed objects.

See ES6 Code below:
class Model {
  constructor(props) {
    this.init();

    Object.seal(this);

    if (props) {
      deepAssign(this, props);
    }
  }

  init() {
    Object.defineProperties(this, {
      one: {
        value: null,
        writable: true,
        enumerable: true
      },
      two: {
        value: {},
        writable: true,
        enumerable: true
      }
    });
    Object.defineProperties(this.two, {
      three: {
        value: null,
        writable: true,
        enumerable: true
      },
      four: {
        value: null,
        writable: true,
        enumerable: true
      }
    });
  }
}

console.log(new Model({one: 'foo', two: {three: 'bar', four: 'baz', five: 'shouldn\'t appear'}, six: 'won\'t appear'}));

What is not working? What did you expect? Please provide a simpler example, preferably a failing test.

@nhardy Your code works exactly as intended:

Object.seal preventing six from being added
Object.seal marking two as non-configurable, but that does not mean you can't add a new property to two (see Object.defineProperty for details about configurable)


Error: Don't know how to parse column type: 1009

Hello team!
This module for Postgres is very promising. To contribute to its quality, I want to report this error to you:
I have this error because my table contains a property of type text array (text []).
I attach the Stack trace:
Error: Don't know how to parse column type: 1009     at decodeText (https://deno.land/x/postgres/decode.ts:233:13)     at decode (https://deno.land/x/postgres/decode.ts:241:12)     at QueryResult._parseDataRow (https://deno.land/x/postgres/query.ts:49:24)     at QueryResult.handleDataRow (https://deno.land/x/postgres/query.ts:61:28)     at Connection._simpleQuery (https://deno.land/x/postgres/connection.ts:322:18)     at async Connection.query (https://deno.land/x/postgres/connection.ts:539:16)     at async Client.query (https://deno.land/x/postgres/client.ts:24:12)

closed by #143 !


Fix constraints not satisfiable message causing compliance flood

ConstraintsNotSatisfiable message types in the subscription status are formatted "constraints not satisfiable: clause1, clause2, clause3 ...". However, the order of the clauses is nondeterministic, which causes compliance event flapping as each update leads to a different ordering of messages, which in turn triggers another update. This was fixed by sorting the message prior to evaluation.
Ref: https://issues.redhat.com/browse/ACM-10204

Some tests might be flaky after this change... I'll check it out

Turns out I had to add the string sorting logic to the check function in the tests as well, should be fixed now!


First-party apps

Some ideas
For scores

Box plot
2D scatter plots (10)

For ensembles

Heat map
ReCom player

Todd

we could set up a website that allows the user to pick an ensemble and two metrics, and we'd render the blob.  The cute thing is that all the computation could be done in the browser, so we don't need a server---it could be a static page.


Supports grpc+json and the curl tool

Implemented support for grpc+Json, and enabled direct communication between curl tool and unary services

Good work. Thanks for your contribution to dubbo-rust.

#145


[* Unit] Fix unit extraction in utterances with compound entity (#2037)

fix to first issue pointed out in #2037 ("12 ft 2 in" returning "12 ft 2", instead of the two separate entities)
added relevant test cases to DimensionModel

@LionbridgeCSII can you resolved the conflict? We should probably still merge this before the compound units PR is ready to merge. What do you think?

@tellarin, conflicts solved


force capture backtraces on VortexErrors

What's the difference vs not forced?

Ah ok so it just overrides the var and always captures. This feels like a decision we might not want to make for library users?

Yeah, I was curious to see if there would be a perf diff if we did this for our errors. Appears to slow down TPC-H by a chunk, so not worth it


Use of cmp builtin function results in "is not defined" warning

Using the cmp builtin function results in a "not defined" warning:
Environment data

Language Server version: Pylance language server 2020.6.1
OS and version: Mac OS Catalina 10.15.5
Python version (& distribution if applicable, e.g. Anaconda): 2.7.18 and 3.8.2

Python is running in a virtualenv, but no extra modules are imported or used
Expected behaviour
No warning is generated
Actual behaviour
The following warning is shown
"cmp" is not defined Pylance (reportUndefinedVariable)
Logs
Info  - 8:42:51 AM] Pylance language server 2020.6.1 starting
[Info  - 8:42:51 AM] Server root directory: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server
[Info  - 8:42:51 AM] Background analysis root directory: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server
[Info  - 8:42:51 AM] Background analysis started
[Info  - 8:42:51 AM] No configuration file found.
[Info  - 8:42:51 AM] Setting pythonPath for service "apiper": "/opt/local/bin/python3.8"
[Info  - 8:42:51 AM] stubPath /Users/andy.piper/awn/users/apiper/typings is not a valid directory.
[Info  - 8:42:51 AM] Assuming Python version 3.8
[Info  - 8:42:51 AM] Assuming Python platform Darwin
[Info  - 8:42:51 AM] Searching for source files
[Info  - 8:42:51 AM] Found 3 source files
[FG] parsing: /Users/andy.piper/Downloads/foo.py (18ms)
[FG] parsing: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server/typeshed-fallback/stdlib/2and3/builtins.pyi (300ms)
[FG] binding: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server/typeshed-fallback/stdlib/2and3/builtins.pyi (119ms)
[FG] binding: /Users/andy.piper/Downloads/foo.py (0ms)
[BG] analyzing: /Users/andy.piper/Downloads/foo.py ...
[BG]   parsing: /Users/andy.piper/Downloads/foo.py (91ms)
[BG]   parsing: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server/typeshed-fallback/stdlib/2and3/builtins.pyi (291ms)
[BG]   binding: /Users/andy.piper/.vscode/extensions/ms-python.vscode-pylance-2020.6.1/server/typeshed-fallback/stdlib/2and3/builtins.pyi (117ms)
[BG]   binding: /Users/andy.piper/Downloads/foo.py (1ms)
[BG]   checking: /Users/andy.piper/Downloads/foo.py (8ms)
[BG] analyzing: /Users/andy.piper/Downloads/foo.py (509ms)

Code Snippet / Additional information
Can be reproduced with a 1-line file:
z = cmp(1, 2)

The cmp function exists only in Python 2. It was deprecated in Python 3.
Pylance works only with Python 3. If you want language support for Python 2, you can use the JEDI language server.

Thanks for the prompt reply!
This makes sense considering that Python 2 has reached EOL, but I didn't come across anything that stated Python 2 wasn't supported by Pylance. Maybe adding this to the docs and/or the description in the VS Code marketplace would prevent others from filing similar bugs in the future?

Yes, that's a good suggestion. Someone else also suggested that, and the README has since been updated to mention that Python 2 is not supported.


Enable enhanced text selection by default

With #7539 merged, we should address the following items before we can enable the feature by default:

[ ] Use a red-black tree or binary search in expandBoundsLTR (see inline TODO item)
[ ] Use reference counting in expandBoundsLTR (see inline TODO item)
[x] Remove dataset usage in text_layer.js as there is unnecessary type conversion
[x] Improve handling cases where each glyph is in a separate div (see https://github.com/mozilla/pdf.js/pull/7539#issuecomment-243205897)
[x] Make enhance text selection a viewer preference

I'm proposing that we close this as WONTFIX for the time being, since:

No real work, to address the remaining points, have happened for a couple years and the enhanceTextSelection-mode is essentially not being worked on or improved at all.

Even with the aforementioned improvements implemented (which only affects the initial parsing), the enhanceTextSelection-mode still won't perform well for pages that contain a lot of textLayer-elements during actual user-interaction.
The explanation here is the need to update the DOM as the user clicks/releases the mouse button, for potentially thousands of elements, which unfortunately will feel sluggish and result in a bad UX in larger and/or complex PDF documents.


On Windows -no-colors cannot be specified from the command line

On Windows with sbt 1.2.6 (NB. bin\sb.bat is not in sync with bin\sbt)
> c:\opt\sbt-1.2.6\bin\sbt.bat -no-colors
[info] Loading settings for project dotty-build-build from build.sbt ...
[info] Loading project definition from C:\dotty\project\project
[info] Loading settings for project dotty-build from build.sbt,plugins.sbt ...
[info] Loading project definition from C:\dotty\project
[info] Loading settings for project dotty from build.sbt ...
[info] Resolving key references (13262 settings) ...
[info] Set current project to dotty (in build file:/C:/dotty/)
[error] Expected symbol
[error] Not a valid command: -
[error] Expected end of input.
[error] Expected '--'
[error] Expected 'debug'
[error] Expected 'info'
[error] Expected 'warn'
[error] Expected 'error'
[error] Expected 'addPluginSbtFile'
[error] -no-colors
[error]  ^

Quick fix: add the following 2 lines in bin\sbt.bat:
74: ) else if /I "%~1" == "-no-colors" (
75:   set _JAVA_OPTS=!_JAVA_OPTS! -Dsbt.log.noformat=true

Originally posted by @michelou in https://github.com/sbt/sbt-launcher-package/issues/260#issuecomment-435340435

https://github.com/sbt/sbt-launcher-package/commit/b5fd78f74bf1dc38bbc4203067dabb0f54dd5ea7#diff-0e467aba72fad1d1010d8efae1c500d6R150


Create Jest snapshot tests of every shared component

Feature
Why is this feature being added?

See #326. We've chosen Jest as our unit testing library, and now we need to utilize it.
What should your feature do?
Integrate Jest
Create snapshot tests for every shared component
Make sure it runs in the build

Hello! Could I take on this one? This would be for all of the components in /src/shared/components, right?

@seeksort yes indeed! Wow, I was so excited to see this issue pop up in my notifications. You'll be integrating our first tests!!! I've assigned myself to this issue as a placeholder for you.

Still working on this, I've just been busy. I had a question though, what are everyone's thoughts on having a central __tests__ directory vs. a subdirectory in each component subfolder? Ryan Florence of React Router suggests having a separate__tests__ subdirectory for each component. I've seen it done both ways, though.

@seeksort no problem, as it seems you're the only person interested in helping with tests 😆
I'm really not experienced enough to have an opinion on the matter. My first inclination is to stick with what people are most familiar with in a central __tests__ directory, since we work with many new developers. I see Ryan Florence's points, but what do you think?


71 Map with points with radius sizes and colors

https://github.com/geovistory/design-system/issues/71

Thanks for your review @ZEDz318 ! You did great!


Vim9: can't use line break after method call operator at script level

Describe the bug
In Vim9 script, we can't use a line break after the method call operator at the script level.
To Reproduce
Run this shell command:
vim -Nu NONE -S <(cat <<'EOF'
    vim9script
    let result = 'abcdefghi'->
        split('\zs')->
        filter('v:val !~# "[aei]"')->
        join('')
    echo result
EOF
)

E260 is raised:
Error detected while processing ...
line    2:
E260: Missing name after ->
line    3:
E260: Missing name after ->
line    4:
E119: Not enough arguments for function: filter
line    5:
E714: List required
line    6:
E121: Undefined variable: result

Expected behavior
No error is raised.
Environment

Vim version: 8.2 Included patches: 1-1253
OS: Ubuntu 16.04.6 LTS
Terminal: XTerm(356)

Additional context
According to :h vim9-declaration, it should be possible to put a line break after a binary operator such as ->:
https://github.com/vim/vim/blob/b146e01a7e41bf5787a2b5ccc6c340ed6001f308/runtime/doc/vim9.txt#L268-L269

The issue is specific to the method call operator.  As an example, .. works fine:
vim9script
let result = 'foo' ..
    'bar' ..
    'baz'
echo result

The code works fine when the line break is before -> instead of after:
vim9script
let result = 'abcdefghi'
    ->split('\zs')
    ->filter('v:val !~# "[aei]"')
    ->join('')
echo result

The issue disappears in a :def function:
vim9script
def Func()
    let result = 'abcdefghi'->
        split('\zs')->
        filter('v:val !~# "[aei]"')->
        join('')
    echo result
enddef
Func()

Non-automatic line continuations fix the error:
vim9script
let result = 'abcdefghi'
    \->split('\zs')
    \->filter('v:val !~# "[aei]"')
    \->join('')
echo result

vim9script
let result = 'abcdefghi'->
    \split('\zs')->
    \filter('v:val !~# "[aei]"')->
    \join('')
echo result

I very much prefer putting the "->" after the line break, it's more readable, it's easier to see that it is a continuation line:
let result = expr
->method()
->method()
I do not consider "->" to be similar to an operator, just like ".member" is not.
So the question is, should we enforce putting the line break before "->", or allow for both?

I very much prefer putting the "->" after the line break, it's more readable, ...

I have the same opinion; I like reading operators/tokens which are aligned; this is only possible when putting the line break before ->.

I do not consider "->" to be similar to an operator, just like ".member" is not.

Ok, I'll use the term "token" instead of "operator" from now on.

So the question is, should we enforce putting the line break before "->", or allow for both?

There are two questions on softwareengineering.stackexchange.com which are relevant:

Line break before/after operator
Should I put newlines before or after binary operators?

It seems that the most upvoted answers either recommend putting a line break before an operator, or to just follow the most commonly used style guidelines.  The only Vimscript style guideline I know of is the one from Google, but it doesn't seem to recommend anything relevant for the current issue.
FWIW, I would enforce putting the line break before ->, but other people may disagree.  No matter what is decided, I think it should be consistent between a :def function and the script level.

Since this line break support is new, it's better to be restrictive at first, and perhaps loosen it up when we decide that is needed.  The other way around, making it restrictive later, doesn't work well.


Multiple scheduler processes?

Hey Evan, beginner here, please bear with me.
I ran the example on the readme and the first time it worked as expected. I did a Control+C exit to exit the program. On subsequent times running the program, nothing is getting enqueued.
Another issue posted here mentioned that he had unexpected behaviour due to multiple processes running. Issue #127 .
Question 2: How do i identify and kill these processes?

everything OK?

@evantahler Apologies for the notifications Evan. Having a tough time implementing this but i'm making progress. I recently started programming.
Just confirming, the way to kill multiple scheduler processes is to call scheduler.end() right? I was looking around with the ps command and scratching my head initially.

Yep! If you "hard kill" it at the process level that's OK too, but it will take some amount of time (default 3 minutes) for a new scheduler to take over

Ah, that explains some strange behaviour.


[@svelteui/core]: add new drawer component

Hi @masagatech, any updates regarding this? Do you need any help?

@mineofcode still want to finish this?


Add Front-end images

What's Changed

Added images to front-end, going to test to see if this works on the server, cause of paperclip.

Coverage remained the same at 99.94% when pulling 37715f98a6d32322942a214647006f8899ef903b on simonv3:master into 7b1215f0cae602e5b1f32adb22d08bece8918167 on openfarmcc:master.


Demo is no longer can be used as a template

@freddy38510  After switch to monorepo is seems that it is no longer possible to use "demo" project (renamed to "playground") as a template. Previously (last working commit for this workflow is of May 4, 2023) I just copied quasar-ssg-vite directory out of tree and it worked (yarn install, quasar ssg dev). Have you considered to provide a template for quasar+ssg+vite project?

It works with current master, however, it need's a bit of hacking (and I don't now, if it is a good way to go...)
git clone git@github.com:freddy38510/quasar-app-extension-ssg.git
cp -r quasar-app-extension-ssg/playground/quasar-cli-with-vite/ ./
cd quasar-cli-with-vite
cp ../quasar-app-extension-ssg/.npmrc ./
# update package.json with "quasar-app-extension-ssg": "^4.7.0"
pnpm i rollup@^2 postcss ouch
quasar ssg dev

I am a bit worried that it only works with<EMAIL_ADDRESS>while ver 3 was available long ago...

Hey @kostyfisik,
With the latest commits, the demo should works. At least it does on Netlify.
In the updated readme you'll find all the steps you need to follow to develop the extension locally.
Let me know if you have any  issues. More changes to come. That's why there's no new release yet.

Dear @freddy38510,
Great news! I am overly interested seeing the upcoming release, thank you a lot!


Change ControlPlane spec.backup feature gate to EnableSharedBackup

Description of your changes

This changes the feature gate for ControlPlane's spec.backup field to EnableSharedBackup, instead of the deprecated EnableControlPlaneBackup.
I have:

[x] Read and followed Upbound's contribution process.
[x] Run make reviewable to ensure this PR is ready for review.
[x] Added backport release-x.y labels to auto-backport this PR, as appropriate.

How has this code been tested

make reviewable

This isn't necessary, I was under the mistaken impression that EnableSharedBackup depends on spec.backup.


Add _taskQueue snippet to document

@fiyazbinhasan  see Add IBackgroundTaskQueue  and this comment
Can you add _taskQueue to your sample?
Take a look at all the comments by @halter73 in #25701
See #25759 for an implementation.
code written in  #25787, just need to update doc.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4fc35621-ae53-cfb5-5eb1-07aefdc5342f
Version Independent ID: 79cb05f8-9ac0-26a8-8434-15de071160c3
Content: Access HttpContext in ASP.NET Core
Content Source: aspnetcore/fundamentals/http-context.md
Product: aspnet-core
Technology: aspnetcore-fundamentals
GitHub Login: @coderandhiker
Microsoft Alias: riande

huan086's https://github.com/dotnet/AspNetCore.Docs/issues/25759 queue implementation is quite different from https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio#queued-background-tasks
Is there anything wrong with the existing implementation that is in the doc? If yes, maybe I'm missing the core details. @Rick-Anderson need your feedback.

Take a look at all the comments by @halter73 in https://github.com/dotnet/AspNetCore.Docs/pull/25701 esp Take a look at all the comments by @halter73 in https://github.com/dotnet/AspNetCore.Docs/pull/25701

If I understood this comment,

Unfortunately, doing background processing the right way is a lot more complicated. Really we should be doing something like what's shown in https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio#queued-background-tasks. Can we just reference the BackgroundTaskQueue from there?

We could just use the queue implementation available in the doc which uses a Channel<Func<CancellationToken, ValueTask>> as a queue whereas in https://github.com/dotnet/AspNetCore.Docs/issues/25759 we have a ConcurrentQueue with a SimaphoreSlim. Is there any issue with one of the implementations?
I would rather use the implementation that is in the doc since it's quite readable and easy to understand.
@huan086 @halter73 any thoughts?

I wasn't aware that we have Channel now. Channel is awesome stuff and we should definitely be using it instead of ConcurrentQueue.
Years ago I looked at the ConcurrentQueue implementation and it contained a spin lock, which is not suitable for web applications. So let's use Channel.
As for Channel<Func<CancellationToken, ValueTask>> specifically, we should test for whether capturing request scoped service would result in ObjectDisposedException.  I strongly suspect it would. The solution would be to only put POCO in the Channel and create a new scope to resolve services in the BackgroundService

Fixed in #25787


[SUPPORT] Request for assistance with an issue regarding ease of use

Tips before filing an issue

Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
A clear and concise description of the problem: The user mentioned 'Easy come . Easy go'.
To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
4.
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version :
Spark version :
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) :
Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.

Hello


bugfix: fix custom rules not executing

Context
Custom rules were not available and I would like to to fix it.
It is caused by copying the custom checks folder after loading the checks.
so, added when files are copied
Description
copying the custom check in the parse_checks_from_folder function here
But check loading is called in load_checks_to_execute before that
There was no process to assign a custom check for that transition, so the custom check was not executed!
Added support for updating pre-loaded sets at the time of copying custom check folders.
Check
Here is an example of gcp.
Prepare the following directory structure rules.
custom_rule
└── cloudsql_test_custom_rule
    ├── __init__.py
    ├── cloudsql_test_custom_rule.metadata.json
    └── cloudsql_test_custom_rule.py

Execute the following command
without custome rule scan
prowler gcp --project-ids {project_id}

with custome rule scan
prowler gcp --project-ids {project_id} -x custom_rule

Confirmed that the number of scans performed and the number of compliant evaluations changes
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Hey! @sejimhp This looks great! Could you fix the tests in order to ensure that all it's correct? Thank you 💯

@pedrooot Thank you for review!
I fixed the test.

I had to add an additional push because it failed in the format.

If you have a look in the tests, now blake it's failing. Could you try to solve it? Using pre-commit should be enough

The coverage drop seems to be caused by main.py, but I couldn't find a test for it...


Getting coordinates of selection to update values

Hi there,
I'd like to be able to update the data of what I have selected on my sheet programatically when I press a button.
A bit more background:
I upload an excel file and I want to select a set of cells and then I press a button on my web that should process the data and then update the cells with the results. I've gotten to the point where I get the data and transform it, but I'm unable to update it back in it's same place, because I'm not able to find how to get their coordinates.
I have been able to do the getJson(true) method and that gives me their column address, but not their row.
I know that with a simple "A9" or "B1", I could split the letter from the number and then know that A is 0 and B is 1, but I wanna know if there's a simpler way, because if my document has over the letter Z, like AA or AB, that's harder to convert to number to place in array.
Thank you!

You have two native methods to deal with names/coords.
jexcel.getColumnNameFromId(x, y);
or
jexcel.getIdFromColumnName('A1', true); // true means return in a array format.

Hi @paulhodel ,
Unfortunately this isn't exactly it.
getColumnNameFromId requires x and y, but I don't have x or y because I have a selection made, so what goes in XY?
The same for getIdFromColumnName, I don't know that my selection is in A1 or B1.
Here is the data I get from the getJson(true) method:

How would I go about updating those values and putting them back in their place?

Sorry, your coordinates you can get by using:
var coords = table.selectedCell;
x1 = coords[0];
y1 = coords[1];
x2 = coords[2];
y2 = coords[3];
One other option is to use:
table.paste(x, y, data);
Where x, y is the initial location, and data should be in CSV format. Rows split by \r\n and Columns \t
To get x, y  you have that:
x = table.selectedCell[0];
y = table.selectedCell[1];


Run kubeadm after static routes

Fixes a race condition where the kubeadm service would fail because it was trying to pull the kube-apiserver image before the static routes were installed.
/run cluster-test-suites

The E2E tests aren't really relevant here as they don't rely on static routes for internet access.
/run cluster-test-suites

/run cluster-test-suites TARGET_SUITES=./providers/capvcd/upgrade

Static routes, the gift that keeps on giving

/run cluster-test-suites TARGET_SUITES=./providers/capvcd/upgrade


Misc errors

Using this schema
 {
      '$schema': 'http://json-schema.org/draft-07/schema#',
      type: 'object',
      additionalProperties: false,
      properties: { email: { type: 'string', format: 'email' } },
      required: [ 'email' ]
    }

And data:
{
   email: "foo"
}

The native message from AJV (v7) is:
 [
      {
        keyword: 'format',
        dataPath: '/email',
        schemaPath: '#/properties/email/format',
        params: { format: 'email' },
        message: 'should match format "email"'
      }
   ]

However this lib returns:
No such property /email in . [query: /email]

Not an improvement :)

This is because of an inconsistency between Ajv 6 and 7. I've now explicitly added support for 6, 7 and 8 with tests for all of them.


Spike using all the new webcomponents/* polyfills

custom-elements (v1 polyfill)
shadydom (v1 polyfill)
shadycss (v1 styling polyfill)

The custom-elements fill isn't working well yet, so as of now we'll continue using the document-register-element polyfill.


ImageNet preprocessing functions

Added

Random sized crop
Scale and center crop
PCA jitter

@sayakpaul
Adding protection to main branch to avoid accidental commits. Requires at least one review approval.

@AdityaKane2001 please also add @MorganR as a reviewer.

@sayakpaul
Thanks for the quick response!

@AdityaKane2001 please also add @MorganR as a reviewer.

Morgan has not yet accepted the invitation. I'll add him as soon as he accepts it.
The script for making tfrecords is in /utils/tfrecords_utils.py
Sure, I'll use Black.
Sure, I'll add a public Colab notebook for this.
I wanted to ask a couple of things:

For testing any of the functions, I do 2 things: Run it through dataset.map() and adding a @tf.function, and checking whether the output images are fine. Most of the functions should not be decorated with @tf.function, but I do so just to test whether they work in graph mode. The read_tfrecs() function in imagenet.py does not play well with @tf.function. Will this be fine in a TPU environment? Does a preprocessing function run in graph mode, even in a TPU system? The dataset gets mapped correctly each time, but as mentioned above, @tf.function gives some issues.
What should be the effect of PCA jitter? I processed a couple of images, but they looked unchanged after PCA jitter was applied.

but as mentioned above, @tf.function gives some issues.

What are those issues exactly?

Does a preprocessing function run in graph mode, even in a TPU system?

It does.

What should be the effect of PCA jitter? I processed a couple of images, but they looked unchanged after PCA jitter was applied.

Did you run it a couple of times? The jittering effect should be pretty visible. I would suggest comparing the output ranges to the FAIR PCA jitter, which would provide us with some more clues. If you want to get rid of PCA Jitter and instead apply something even more generic and more effective like RandAugment, you can do that. Here's a blog post that shows how to apply it. We can keep the RandomResized crop function as it is and then apply the RandAugment.
Also since this is ImageNet-pretraining and using MixUp almost always helps, I would recommend adding a utility for that too. But I totally understand if you'd want to hold off on this.

It gives the following error:
Exception ignored in: <function CapturableResource.__del__ at 0x7f2b77ba2dd0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/tracking/tracking.py", line 269, in __del__
    with self._destruction_context():
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py", line 380, in inner_cm
    old_strategy_stack = self._distribution_strategy_stack
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 5195, in _distribution_strategy_stack
    if not hasattr(self._thread_local, "_distribution_strategy_stack"):
AttributeError: 'FuncGraph' object has no attribute '_thread_local'

It gives this error when I call the function for the second time.
I think that it gives this error as there's a function inside the main function (code here). I'll remove that and try again.
I agree, it'd be good to apply RandAugment from the start itself. As in the blog post, am I allowed to use the vision library?
I think it would be better to keep Mixup as a backup sort of thing, in case we don't get the required accuracy with the rest. Also, we are training the smaller models, which may give us some room to play with. Training on ImageNette in colab itself will give a good idea, I guess.

I'm actually new to GCP and would really like some help on that.
I was thinking to make two instances: One with only CPU for TFRecords making and other with the TPU. I'll delete the first instance after we make and store the TFRecords in GCS. Will it be better to do it this way, or to create only one instance and add the TPU later?

I agree, it'd be good to apply RandAugment from the start itself. As in the blog post, am I allowed to use the vision library?

Why not?
One instance should be enough. You can create an n1-standard-8 instance which is more than enough. You can follow this guide to see how you can create a TPU-attached Jupyter Notebook instance. Just change the script mentioned in the tutorial to install the latest TF version. Let me know how it goes.

Okay.
I have received the credits but  I'm not able to redeem them. I have mailed Jaeyoun regarding the same.
I had seen the script earlier, but frankly, I'm more inclined towards executing a script which does the training so that I don't unintentionally overspend.

Why would you overspend, though? It's the Compute Engine-based concept with a richer GUI. You could use the Terminal that comes with a Jupyter Lab and run the exact same scripts. It's totally up to you. I find this process easier but I understand if you have other preferences.

Oh okay, I'll try this.

@sayakpaul

If you want to get rid of PCA Jitter and instead apply something even more generic and more effective like RandAugment, you can do that

As decided, I have removed the PCA jitter function. Over the weekend, I'll

add RandAugment
make a notebook for illustration.
try the pipeline on ImageNette

I'll request a review again when the pipeline seems good.
/cc @MorganR

@sayakpaul
Should I change the directory structure to:
regnety/
│   ├── config/ ..
│   ├── dataset/ ..
│   ├── utils/ ..
│   ├── main.py
│   └── (any other files)
├── README.md
├── LICENSE
├── .gitignore
└── (any other files, like tutorials)

Yeah that looks good to me.

I would also like to see a couple of image batches after parsing the TFRecords. A notebook or a demo script might be good to test the data loaders out.

Do we want to create a testing script (like a unittest as you mentioned earlier) or a demo notebook on how to use the repo?

A unit test sounds good. Notebook is also good to just interact and debug things.

Ok, then I'll add unit tests for now, will add the illustrative notebook later.

Okay. But for now, I think it would be better if you just create a rough notebook to ensure the dataloader is working as expected. This will save you a lot of time during model training if something goes wrong because of data preprocessing.

Oh, okay sure.

@sayakpaul @MorganR
Today I was planning to fire up the instance and make the TFRecords. Could you please tell if this script is good to go?

@sayakpaul @MorganR
Today I was planning to fire up the instance and make the TFRecords. Could you please tell if this script is good to go?

Looks good to me but here are a couple of things you could do:

Don't shuffle the indices for non-train datasets.
Prefer having a batch of entries inside a single TFRecord. You can create a tf.data.Dataset of the image paths first, batch it, and then map your TFRecord utils to it. Here's an example. This not only will improve your TFRecord creation process but will also speed up the data input pipeline.

Let me know if anything unclear.

Got it, thanks!

@sayakpaul

You can create a tf.data.Dataset of the image paths first, batch it, and then map your TFRecord utils to it

Please take a look at the following code:
def make_image(filepath):

    image_str = tf.io.read_file(filepath)

    if is_png(filepath):
        image_str = png_to_jpeg(image_str)

    if is_cmyk(filepath):
        image_str = cmyk_to_rgb(image_str)

    image_tensor = tf.io.decode_jpeg(image_str)

    height = image_tensor.get_shape()[0] #Here's the issue
    width = image_tensor.get_shape()[1]

    if image_tensor.shape[2] == 1:
        image_tensor = tf.image.grayscale_to_rgb(image_tensor)
        image_str = tf.io.encode_jpeg(image_tensor)

    assert len(image_tensor.shape) == 3

    return image_str, height, width

Since image_tensor is defined during execution itself, it's shape is not predefined. Thus, when I try to map this function to the filepaths dataset, height and width come out to be None. Wrapping it with tf.py_function will not help with speed, as it will execute it eagerly.  Could you please shed some light on this?

What's preventing us from resizing it? When using TPUs, the shapes must be explicitly specified anyway.

We can do that.
I'll use the filter to check whether the image is grayscale and convert it to RGB.
Thanks!

@sayakpaul
Could you please take a look again? I have made changes in tfrecords_utils.py as discussed.

Thanks for making the changes. I think @MorganR will have additional comments. So I will for them.

Looking now
On Mon, Jun 14, 2021 at 12:48 PM Sayak Paul @.***>
wrote:

Thanks for making the changes. I think @MorganR
https://github.com/MorganR will have additional comments. So I will for
them.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AdityaKane2001/regnety/pull/1#issuecomment-860588469,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZIOLDJFALZW23FXSZVXALTSXM75ANCNFSM46MOVPAA
.

@sayakpaul  @MorganR
I wanted to discuss about resizing of the images before saving them to TFRecords. Personally, I think that we should keep the stock images as they are, even at the cost of performance during TFRecords creation, since it is one-time. Resizing the images may result in worse augmentation, as the images will be distorted before RandAugment and random scale and crop.
Also, a small doubt, as far as I understand the objective of random_sized_crop function is to stretch or squeeze the image and then take a random square crop. Is this correct?
Please share your thoughts on the same and please correct me if I am wrong.

I wanted to discuss about resizing of the images before saving them to TFRecords. Personally, I think that we should keep the stock images as they are, even at the cost of performance during TFRecords creation, since it is one-time. Resizing the images may result in worse augmentation, as the images will be distorted before RandAugment and random scale and crop.

I don't you will be able to batch together images of different sizes and serialize those as a single TFRecord shard. In my experience and also from what I have mostly seen this resizing is not that hurtful. Here's a reference. Feel free to also refer to how it's done in models/official. @MorganR feel free to chime in here.

Also, a small doubt, as far as I understand the objective of random_sized_crop function is to take a random square crop and then stretch or squeeze the image. Is this correct?

Yes. It should ideally replicate something like this.

@MorganR
I was following this code.

@sayakpaul @MorganR
Please take a look at the updated code. Following are the important changes:

dataset.imagenet.ImageNet.random_sized_crop uses tf.image.sample_distorted_bounding_box  , which I think was the objective of this. Code is based on tensorflow/models' implementation  of the same.
dataset.imagenet.ImageNet.scale_and_center_crop  is now simpler.
Added a separate python script to be used for making TFRecords. (/regnety/make_tfrecords.py)
Added type annotations.
Made a notebook for testing the dataloader, gist here.

I also have a couple of things to discuss. In previous versions, there was an argument for synset json file path, even though we have that json file in config folder. I have used a small hack to get this path for now without any external input. Going forward, what would be the best way to handle configurations? Also, will it be better if I make a setup.py file and use regnety as a module?

I also have a couple of things to discuss. In previous versions, there was an argument for synset json file path, even though we have that json file in config folder. I have used a small hack to get this path for now without any external input. Going forward, what would be the best way to handle configurations?

For data-related stuff, using separate classes decorated with dataclasses would be a good idea. In general, for handling configuration-related stuff I would recommend either  config.py or config.yaml.

Also, will it be better if I make a setup.py file and use regnety as a module?

Could you provide some pseudocode in Python to help us understand this better?

Also, will it be better if I make a setup.py file and use regnety as a module?

Following is our file structure now. I was thinking of adding a new file setup.py as follows:
regnety
├── regnety/
│   ├── config/
│   │   └── imagenet_synset_to_human.json
│   ├── dataset/
│   │   ├── __init__.py
│   │   └── imagenet.py
│   ├── utils/
│   │   ├── __init__.py
│   │   ├── image_utils.py
│   │   └── tfrecord_utils.py
│   ├── main.py
│   └── make_tfrecords.py
├── setup.py ###
├── README.md
├── LICENSE
└── .gitignore

where setup.py has
from setuptools import setup

setup(
    name = 'regnety',
    package_dir = {'' : 'regnety'},
    packages = ['utils', 'dataset']
)

By doing so, I'll be able to use, say for example
#/regnety/dataset/imagenet.py

from utils import image_utils
...

The other way is to just append the absolute regnety path in PATH in regnety.__init__.py, but I thought this might be simpler.

Try suppressing all the TensorFlow warnings that harmless. Give a descriptive message beside 9469. Append a unit beside the total time taken to prepare the shards.

I'll add these things along with the other changes in the next commit.

@AdityaKane2001 since we are almost about to merge this PR, please consider submitting smaller PRs in the coming days. That way it's easier and more compact for us to review and work on.

Sure, I'll keep one file at a time.

Also, if you agree with a particular suggestion, you can just thumb it up. That way the conversation length is maintained and things look cleaner. Of course, if you think a suggestion needs more clarification or you need to brainstorm more, comment without any hesitation.

Thanks a lot for the reviews! I'll merge the PR now.
@sayakpaul  Here's the demo notebook.


Plugin still spams console on startup

I know the last release says "cleanup logs", but the plugin still spams the console with hundreds of:
"New metadata, drawing graph link types."
when I start Obsidian. See screenshot:

Alright, I put out a new release that should have that fixed. Thank you for letting me know, and please let me know if you find anything else - thanks!

Sorry, I didn't mean to close this until I knew it worked for you! It's interesting that it was spamming it on startup for you, because it should only do that when new metadata is added by dataview...

Sorry, I didn't mean to close this until I knew it worked for you!

That's okay. I tested it now: Updated and restarted, and there was no console log from Graph link types this time. So it seems to have worked.

It's interesting that it was spamming it on startup for you, because it should only do that when new metadata is added by dataview...

Hmm, I see, that is interesting then. It was happening every startup. Dataview always indexes the vault on startup for me. Isn't that normal?

Yes, I believe so. The have 2 different events that I'm attached to, one is the initial indexing, and one is the new metadata. I'm thinking I shouldn't attach to the new metadata event until the indexing is over, because it seems like while indexing the new metadata event gets called a ton of times. Thanks for your help, I will try to account for this soon!


Removes Aditi from the OWNERS files

What type of PR is this?
Not sure.
What does this PR do / why we need it:
$subject
Which issue(s) this PR fixes:
Fixes NA
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

[ ] I have read the test guidelines

How to test changes / Special notes to the reviewer:

/lgtm

/lgtm


Unrecognized option or missing parameter(s): block-outside-dns

Tue Feb  7 14:16:06 2017 Unrecognized option or missing parameter(s) in client.ovpn:14: block-outside-dns (2.3.10)
Tue Feb  7 14:16:06 2017 OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Feb 2 2016
Tue Feb  7 14:16:06 2017 library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.08

That's a warning which is to be expected in non-Windows clients. Nothing wrong with it.
For more details see here: https://github.com/Nyr/openvpn-install/commit/acca10ba1a87b86da3f402ef08e07ad42126913e


Replace sliders with input boxes

Right now, particularly because of the slower performance in Rhomap, the sliders are a source of frustration, and input boxes have been requested :/

We have decided to stay with the sliders, changing the event to change rather than input. Waiting to here back if this is okay.


[Search Block]: Live search triggers several requests to @querystringsearch

Describe the bug
The search block triggers several requests on actions
eg. using the searchbox:
It now triggers three times, one of them being "wrong" (not matching the current query elements), so there's one moment that the result is incorrect. This could affect acceptance tests too.
To Reproduce
Steps to reproduce the behavior:

Add a search block with a custom query
Search for a term in the searchbox
Monitor the network tab, it triggers several requests to @querystringsearch

Expected behavior
We should only trigger one request per action/state change.
Also, when typing a debounce should be added too.

Picking this one up

Working on this one

can you describe it with more details including file or page url if posible

I found to a way to test the amount of the querystring-search queries with cypress. We can work on it in this branch https://github.com/plone/volto/tree/ionlizarazu-search-block-querystring-search @sneridagh


System.NullReferenceException in UnsafeNativeMethods.OnCallback

UnsafeNativeMethods.OnCallback = new UnsafeNativeMethods.WebPMemoryWrite(MyWriter); is causing an exception:
System.NullReferenceException  Object reference not set to an instance of an object.     at UnsafeNativeMethods.WebPEncode_x64(WebPConfig& config, WebPPicture& picture)     at WebP.AdvancedEncode(Bitmap bmp, WebPConfig config, Boolean info)
I can only seem to reproduce this on a server; running in development mode works fine.

I
I have updated the 64-bit library to version 1.2.1
Please try her. If you use the 32-bit version, I recommend you recompile this library. I have put instructions on how to do it in Compile dll.txt

Any updates on this? getting the same error :) and updated library didn't fix it for me

Please, test this change:
[DllImport("kernel32.dll", EntryPoint = "CopyMemory", SetLastError = false)]
You should end up with:
[DllImport("kernel32.dll", EntryPoint = "RtlMoveMemory", SetLastError = false, ExactSpelling = true)]
internal static extern void CopyMemory(IntPtr dest, IntPtr src, uint count);

Hey @JosePineiro , it was about a multi-threaded scenario for me, and the fact that OnCallback was static. The WebP class was being accessed/instantiated by multiple threads therefore OnCallback was being set null, and I was getting those NullReferenceExceptions. I made OnCallback an instance property and made sure to release it properly, it solved it for me. Thanks!


[bug-fix] Filter out mediaType from bodyParams

Related PR
#34, #35
Changes

Filter out mediaType from bodyParams. (If you enter a mediaType as a subproperty of bodyParams, the generated OAS will also contain the mediaType, resulting in an invalid OAS, which should be filtered out. problem detail)
When bodyParams are empty, convert empty object to undefined.

Thanks for the contribution to fixing the bug, I've refactored it a bit so we don't have to check if bodyParams is empty object!

I have published Tspec v0.1.108 including this fix 🎉


Change public API mention of monad to PowerShell

PR Summary
As we are generating docs from comment based documentation in the source code, need to change references from monad to PowerShell to avoid confusing users.  Only public types/methods are changed.
PR Context
Fix https://github.com/PowerShell/PowerShell/issues/12839
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[x] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[x] None
OR
[ ] Experimental feature(s) needed

[ ] Experimental feature name(s):

User-facing changes

[x] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[x] N/A or can only be tested interactively
OR
[ ] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[x] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and opened an issue in the relevant tool repository. This may include:

[ ] Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode
(which runs in a different PS Host).

[ ] Issue filed:

[ ] Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.

[ ] Issue filed:

[ ] Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).

[ ] Issue filed:

[ ] Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

[ ] Issue filed:

@daxian-dbw you should re-review, looks like I forgot to save some of my changes in VSCode so the commit was incomplete, this one has all the changes


Hot reload crashes the mac os desktop embedder

When the first reload attempt has dirty libraries, the macOS desktop embedder silently crashes during reassemble. However, if the first reload has no dirty libraries, then subsequent reloads succeed.
@jonahwilliams @stuartmorgan

I was able to reproduce a crash, not sure if it was the crash. This is happening on the macOS desktop embedder when running in debug mode
flutter run
Building macOS application...
../../third_party/dart/runtime/vm/object.cc: 3716: error: Unable to use class Library:'dart:ui' Class: TextRange which is not loaded yet.
version=2.7.0 (Wed Nov 6 22:44:33 2019 +0000) on "macos_x64"
thread=39179, isolate=main(0x7ff0528b1000)
  pc 0x000000010b818155 fp 0x00007000036f6350 dart::Profiler::DumpStackTrace(void*)
  pc 0x000000010b6b1a92 fp 0x00007000036f6430 dart::Assert::Fail(char const*, ...)
  pc 0x000000010b7a8c21 fp 0x00007000036f6460 dart::Class::EnsureDeclarationLoaded() const
  pc 0x000000010b6faafe fp 0x00007000036f64d0 dart::ClassFinalizer::FinalizeTypesInClass(dart::Class const&)
  pc 0x000000010b6fab6e fp 0x00007000036f6540 dart::ClassFinalizer::FinalizeTypesInClass(dart::Class const&)
  pc 0x000000010b6fa92e fp 0x00007000036f66c0 dart::ClassFinalizer::ProcessPendingClasses()
  pc 0x000000010bab78be fp 0x00007000036f66f0 dart::Api::CheckAndFinalizePendingClasses(dart::Thread*)
  pc 0x000000010bad24ce fp 0x00007000036f6830 Dart_FinalizeLoading
  pc 0x000000010b6a226e fp 0x00007000036f6870 flutter::DartIsolate::LoadKernel(std::__1::shared_ptr<fml::Mapping const>, bool)
  pc 0x000000010b6a2359 fp 0x00007000036f6920 flutter::DartIsolate::PrepareForRunningFromKernel(std::__1::shared_ptr<fml::Mapping const>, bool)
  pc 0x000000010b46b45e fp 0x00007000036f6960 flutter::KernelIsolateConfiguration::DoPrepareIsolate(flutter::DartIsolate&)
  pc 0x000000010b4643c8 fp 0x00007000036f6b40 flutter::Engine::PrepareAndLaunchIsolate(flutter::RunConfiguration)
  pc 0x000000010b463e93 fp 0x00007000036f6d60 flutter::Engine::Run(flutter::RunConfiguration)
  pc 0x000000010b4814cb fp 0x00007000036f6f00 std::__1::__function::__func<fml::internal::CopyableLambda<flutter::Shell::RunEngine(flutter::RunConfiguration, std::__1::function<void (flutter::Engine::RunStatus)>)::$_7>, std::__1::allocator<fml::internal::CopyableLambda<flutter::Shell::RunEngine(flutter::RunConfiguration, std::__1::function<void (flutter::Engine::RunStatus)>)::$_7> >, void ()>::operator()()
  pc 0x000000010b00a783 fp 0x00007000036f6f80 fml::MessageLoopImpl::FlushTasks(fml::FlushType)
  pc 0x000000010b01338c fp 0x00007000036f6fb0 fml::MessageLoopDarwin::OnTimerFire(__CFRunLoopTimer*, fml::MessageLoopDarwin*)
  pc 0x00007fff37d85060 fp 0x00007000036f6fc0 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__
  pc 0x00007fff37d84c0c fp 0x00007000036f8070 __CFRunLoopDoTimer
  pc 0x00007fff37d84752 fp 0x00007000036f80d0 __CFRunLoopDoTimers
  pc 0x00007fff37d65962 fp 0x00007000036f8de0 __CFRunLoopRun
  pc 0x00007fff37d64ebe fp 0x00007000036f8e70 CFRunLoopRunSpecific
  pc 0x000000010b0135bd fp 0x00007000036f8eb0 fml::MessageLoopDarwin::Run()
  pc 0x000000010b00a624 fp 0x00007000036f8ed0 fml::MessageLoopImpl::DoRun()
  pc 0x000000010b0120f8 fp 0x00007000036f8f10 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0> >(void*)
  pc 0x00007fff63ed12eb fp 0x00007000036f8f30 _pthread_body
  pc 0x00007fff63ed4249 fp 0x00007000036f8f50 _pthread_start
  pc 0x00007fff63ed040d fp 0x00007000036f8f78 thread_start

I noticed that TextRange doesn't have an entrypoint annotation (should it?), but I did not think this mattered in debug mode.
cc @a-siva  @alexmarkov

Actually, this isn't an entrypoint issue since the embedder isn't calling into text range.

Current theory from @alexmarkov is that I wasn't loading the core snapshots correctly, but since we were linking the platform in it didn't matter. Verifying that now

Forcing platform linking in debug mode issue casues the crash to disappear. I'm now looking into why we're not correctly supplying the core snapshots.

My read of the situation is:

The desktop targets will need to be updated to load isolate_snapshot_data and vm_snapshot_data in debug mode, using the core snapshots populated in the flutter assets directory.

The section in embedder.cc will need to be updated accept vm_snapshot_data and isolate_snapshot_data in non pre-recompiled mode. @chinmaygarde

In the meantime, we shouldn't keep the desktop targets broken though. I would propose that we force platform linking on desktop platforms for now

It sounds like there was a breaking change to the behavior of the embedding API? That's not supposed to happen, and would affect third-party projects that we don't control.

I don't It wasn't a change to the embedder API, it was a change in how we built the initial kernel file that uncovered it. We never supplied the callbacks for isolate/vm data

I understand that the API didn't change, but it sounds like a change was made such that if someone were using the embedding API in the past, successfully, it will no longer work for them after an update unless they make changes to how they use it. That violates the compatibility intent of that API later as I have understood it.

Just catching up on this. Can you elaborate on:

The desktop targets will need to be updated to load isolate_snapshot_data and vm_snapshot_data in debug mode

Is this something each desktop application will need to change? Or is it something that needs to be changed internally to Flutter?

Also, is this crash reproducible in the desktop example app? Or does it require a more complex app to surface the issue?

We produce two artifacts ( a core snapshot) that contain parts of a precompiled dart-sdk for debug mode only. Android and iOS correctly load this artifact from the flutter assets_directory, but we never did the same for the desktop embedder. It happened to work, because we were previously compiling the entire dart-sdk into the initial kernel dill. To avoid this extra work, I disabled link-platform for debug mode fairly recently, which caused this particular macOS crash: since it doesn't have the core snapshots and doesn't have the linked platform there is no actual dart-sdk loaded into the program.
The workaround is to turn link-platform back on for desktop targets. The actual fix is to updated the embedder to allow loading these snapshots in debug mode, and update the embedder to load them from flutter assets.
The change is internal to the desktop embedders, applications won't need to do anything.

Also, is this crash reproducible in the desktop example app? Or does it require a more complex app to surface the issue?

This crash was trivially reproducible, it might be related to the other macOS crash but it needs more investigation. Matt and I are going to take a look on Friday.

Can you link to the PR where link-platform was disabled? I'd like to correlate that change to when we started seeing problems on our end, as data around whether there is another problem lurking behind that one.

This would have broken in https://github.com/flutter/flutter/pull/44279 and the fix was landed in https://github.com/flutter/flutter/pull/44753

OK. I don't think that change could account for all the issues we have been seeing, because it only landed 9 days ago, and we started seeing issues before that. Still, there was a separate app-specific fix that we landed today that might possibly account for the earlier failures.

We believe this is fixed with #44753. There might have been other issues, but as of last Friday neither me nor Matt could reproduce them.
I believe we'll need to increase the priority of https://github.com/flutter/flutter/issues/42281, this would have caught the issue during postsubmit.


Fix: WebSockets JS API refactoring

Problems Fixed
Impossibility to export types properly
It was not possible to export the type like this:
module.exports.WebSocket = WebSocket

as JS complained about constructor:

Class constructor WebSocket cannot be invoked without 'new'

We had to do it via a function:
module.exports.WebSocket = (url, protocols) => new WebSocket(url, protocols)

It's improper because on the consumer side we lose the original type as it refers exclusively to lambda.
Cause
Validates API wraps every type/function in its own type, thus on the user-end the knowledge about the original type is lost.
Fix
Commented everything related to Validates. In the future we need to have the possibility to activate it for Debug/Test purposes only
Non-functional WebSocket API
Symptoms
All scenes using WebSocket API do not work.
#1236
#935
Cause

The implementation used SocketIO that no scenes have knowledge about, thus there was a protocol mismatch.
Several mistakes in WebSocketApi.js
Wrong marshaling of data between JS and ClearScript in both directions
Impossibility to export WebSocket properly led to absence of callbacks set

Fix

Fresh clean implementation based on ClientWebSocket solely
Various fixes to WebSocketApi.js
Fixes #516 as all allocations are under our direct control and were properly handled

Remaining issues
ColyseusSDK breaks connection sporadically on the host side at 145,60 (Metamall).
SocketException: An established connection was aborted by the software in your host machine.

Possible reason

Scene processes received data for too long. It is clearly visible due to logs and how delayed the instantiation of coins/red rocks is.
WebSocket communication times out on the server side
WebSocket is aborted without a handshake
The scene is still non-functional 😢

How to test

Genesis Plaza dialogs should function properly 🕺 , there should be no errors related to the WebSockets
There is a chance that Metamall (145,60) can function as well under certain circumstances...

MAC testing:
IA dialogues:  they are all triggered now:

However, buttons don't respond and when writing questions to interact with these NPCs, if using keys which perform actions at UI level, those actions are performed, when they shouldn't. Check video

Then the build froze and broke the bar scene only twice: once when interacting with Simone and another time when interacting with Aisha. Here are the logs:
bar scene breaks after interacting with woman NPC with IA in the bar.log
freeze while interacting with simone asking questions.log
Then, dialogues with other NPCs look ok, with the exception of long  dialogues where the press F to skip overlaps with text sometimes:

This issue will be reported separately , too.
Finally, MetaMall. Usually it works ok and coins can be obtained and coin counter works ok. Check video
However, sometimes the scene seems to break and coins cannot be obtained . Check video


Include 'Number of Household Members' in overnight batch processes

thread: https://powerofus.force.com/0D58000002lIn5p

**lurch: add

Tracking W-010669

Pretty sure this would need to be an entirely new job, we don't query for this during the course of our standard rollups

+1

Salesforce has determined that this issue will not be explored further.


[fix] Remove -ansi when building on FreeBSD 10

It is necessary to remove the -ansi flag from SOCI_GCC_CLANG_COMMON_FLAGS in src/cmake/SociConfig.cmake when building on FreeBSD 10. Compiled using the following:
mkdir build
cd build
cmake -G "Unix Makefiles" -DWITH_ORACLE:BOOL=OFF ../../soci-3.2.2
gmake install

The current combination of -ansi and -std=gnu++98 looks schizophrenic at least. It doesn't make sense to ask for one standard and overwrite it with the GNU extensions afterwards. As such, the -ansi can just be dropped.

Ah, I didn't realize both were used. This indeed doesn't make sense, let's drop -ansi.


Mechanics proposed

I am pretty confident now that all examples work as expected.  The new Mechanics API allows a mechanics_io-like interaction on the C++ level (i.e. declaration of abstract shapes implemented by a hidden Bullet-based back-end), and integrates this into mechanics_io.py.  It gives per-shape control over collision margins and a few Bullet-specific options such as gContactBreakingThreshold and support for scaling in the collision engine.
Please review.
Adds a backend selection interface, instead of relying only on overriding classes.  This (and the backend options) is not saved i the HDF5 file. Overriding classes is still supported, although there was an issue with different constructor arguments for the TimeStepping override which has the potential to lead to a bit of confusion.  Tried to make the OCC example work to be sure not to break things completely but it seems still in need of some work in other areas.
In any case, now that examples are working, I feel confident that further work could proceed on the master branch.

Fixed OCC example.


How to link the located particles

Hi,
I am using the multiple particle tracking example notebook, and it works fine. But I realized that it is only the localization, do you have the linking part of this notebook?

Hi! We do have tools for linking as well as detecting. examples/MAGIK contains an example of using our graph-based neural network for tracing. @JesusPinedaC should be able to help more with this!
If the tracing is not too complicated, you can also use a non-deep-learning approach. Let me know if that is of interest.

Hi @XinyueZhang831
In this link, you will find an example of using MAGIK, our graph-based neural network, for detection linking.
MAGIK requires annotated data to be trained—however, a minimal amount of training data has proven to be sufficient to achieve accurate results.
Applying a pre-trained model directly to your data is also possible if a training dataset isn't available. This model has proven to be quite generalizable, capable of adjusting to various movement dynamics. Follow the instructions in the example to download the model.
If you want to know more about MAGIK, here is the paper.

Hi Benjamin and Jesus,
Thank you for providing the example of MAGIK! The work looks fantastic! May I ask if I have to have the parenthood to use the GraphGenerator?

Hello @XinyueZhang831 !
No, parenthood is totally optional. It is helpful in cases where you want to encode specific interactions between particles in the graph, for example, cell divisions or merging events.
However, omit this argument if you are not interested in including this information in the graph representation. DeepTrack will handle it!

Hello @XinyueZhang831 !
No, parenthood is totally optional. It is helpful in cases where you want to encode specific interactions between particles in the graph, for example, cell divisions or merging events.
However, omit this argument if you are not interested in including this information in the graph representation. DeepTrack will handle it!

Hi Jesus,
I was trying to run the code on notebook, but I did not find the pre-trained model. Does this repo includes the pre-trained model?
Also, if I disable the _LOAD_MODEL, an error occurs as following:
Creating graph edges...
100%|██████████| 1/1 [00:52<00:00, 52.76s/it]
Generating 512 / 511 samples before starting training
Epoch 1/10
/usr/local/lib/python3.7/dist-packages/deeptrack/image.py:317: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
return np.array(self.to_numpy()._value)
InvalidArgumentError                      Traceback (most recent call last)
 in ()
16
17     with generator:
---> 18         model.fit(generator, epochs=10)
2 frames
/usr/local/lib/python3.7/dist-packages/deeptrack/models/utils.py in fit(self, x, batch_size, generator_kwargs, *args, **kwargs)
245             return None
246
--> 247         return self.model.fit(x, *args, **kwargs)
248
249     def export(
/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65     except Exception as e:  # pylint: disable=broad-except
66       filtered_tb = _process_traceback_frames(e.traceback)
---> 67       raise e.with_traceback(filtered_tb) from None
68     finally:
69       del filtered_tb
/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
53     ctx.ensure_initialized()
54     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 55                                         inputs, attrs, num_outputs)
56   except core._NotOkStatusException as e:
57     if name is not None:
InvalidArgumentError: Graph execution error:
Detected at node 'model/graph_block_0/GatherV2' defined at (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/usr/local/lib/python3.7/dist-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelapp.py", line 499, in start
self.io_loop.start()
File "/usr/local/lib/python3.7/dist-packages/tornado/platform/asyncio.py", line 132, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.7/dist-packages/tornado/ioloop.py", line 758, in _run_callback
ret = callback()
File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 661, in
self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
self._handle_recv()
File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
callback(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/usr/local/lib/python3.7/dist-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/usr/local/lib/python3.7/dist-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
if self.run_code(code, result):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 17, in
model.fit(generator, epochs=10)
File "/usr/local/lib/python3.7/dist-packages/deeptrack/models/utils.py", line 247, in fit
return self.model.fit(x, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1384, in fit
tmp_logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step
outputs = model.train_step(data)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 859, in train_step
y_pred = self(x, training=True)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1096, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 452, in call
inputs, training=training, mask=mask)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 589, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1096, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/deeptrack/models/gnns/layers.py", line 84, in call
message_inputs = tf.gather(nodes, edges, batch_dims=1)
Node: 'model/graph_block_0/GatherV2'
indices[6,3793,1] = 1020 is not in [0, 1019)
[[{{node model/graph_block_0/GatherV2}}]] [Op:__inference_train_function_12591]
It would be great if you would have any suggestions.
Thank you!

Hi @XinyueZhang831!
Thanks for reporting this issue!
Currently, we are working on it and hope to have it fixed by tomorrow.

Hi @XinyueZhang831!
We have released deeptrack v1.3.1. This new version fixes the bug. Please try it out and let us know if you have any other questions.
The pre-trained model is provided together with the BFC2Cells dataset, which is downloaded in Section 2.1 of the example. Don't forget to run this cell before trying to load the model weights.

Please reopen if the issue persists


Database Handling

need to think on :

ORM? no ORM?

need to have :

upgrade system to upgrade tables with simplicity

After some thinking I think we should use doctrine.
Yml files for the data structure is quite nice, and if memory leaks becomes indeed an issue there are a few tricks that should allow to get around it.

LocalRecrods was started using Doctrine, cross fingers.


dSYMs.zip for last version 6.10

Could you provide the dSYMs for thsi last version 6.10

Hi @JgomesAT,
Sure. Added them to the release: https://github.com/talsec/Free-RASP-Flutter/releases/tag/v6.10.0
Btw. If the iOS SDK version does not change, you can use the same dSYMs as for the previous version. You can find this info in the changelog, currently iOS SDK is version 6.6.3.
Regards,
Tomas

I triad to upload  two days ago a version of my APP , and I had the warning when I finished to upload to apple store


setacq is not reflashing controllers

I have installed CentOS 7 with openvnmrj 2.1 on VNMRS500
The system works fine with the software.
When I install an old controller on the console I am not able to have the new firmware written. Although there is no error message running /vnmr/setacq.
I also tried /vnmr/adm/acq/verifyControllers and got the following error message:
Failed to copy '/vnmr/acq/download/filename' to Controller
I have checked the files and they do exist and owned by vnmr1
Also rsh is enable and hosts.equiv is filled with controller names. Also tftp is enabled

I see that you are on SpinSights. If you search for boot.ini, you will find a document titled "Reflashing script - a tool to manually set firmware on DD consoles".  There is a "Controller_Reflashing.doc" that may help.

Hello Dan,This is also not working. It looks like the tftp service is not working as it works on centos6.I have followed this document. I can delete files but i cannot copy the files. I think itis not an openvnmrj problem but more related to centos 7. Is setacq using tftpboot or rsh?There is no error messages running setsacq. When i have an old controller in the system i can see setacq is not updating the files. When i rlogin the controller and running setacq i see error messages like the files are not uploading.Running from centos6 with vnmrj 4  it is working wellRegards,Ron KerkhofVerzonden vanaf Samsung-tablet.
-------- Oorspronkelijk bericht --------Van: Dan Iverson @.> Datum: 14-09-21  16:12  (GMT+01:00) Aan: OpenVnmrJ/OpenVnmrJ @.> Cc: ronkerkhof @.>, State change @.> Onderwerp: Re: [OpenVnmrJ/OpenVnmrJ] setacq is not reflashing controllers (#619)
I see that you are on SpinSights. If you search for boot.ini, you will find a document titled "Reflashing script - a tool to manually set firmware on DD consoles".  There is a "Controller_Reflashing.doc" that may help.
—You are receiving this because you modified the open/close state.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.
[
{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/OpenVnmrJ/OpenVnmrJ/issues/619#issuecomment-919191677",
"url": "https://github.com/OpenVnmrJ/OpenVnmrJ/issues/619#issuecomment-919191677",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
@.": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

Try running, as root,
systemctl enable tftp.service
systemctl start tftp.service

Assuming this is fixed. Reopen if still a problem


Undefined symbol when using app.py in MedSAM2

I'm running app.py in MedSAM2 in WSL. The environment is Ubuntu 20.04, Python 3.10, CUDA 12.3 and Pytorch 2.3.1. After uploading a video and running process, I  draw box for a prompt and click Segment, then the application raises ImportError: /MedSAM2/sam2/_C.so: undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPKvmm. The symbol _ZN3c1015SmallVectorBaseIjE8grow_podEPKvmm in /sam2/_C.so seems to be not defined.

Have you resolved it?

@ImageBlue
could you please check whether SAM2 is successfully installed by running their notebooks?

@JunMa11 I have run the command pip install -e . and it showed Successfully installed SAM-2-1.0

python setup.py build_ext --inplace worked for me.
Run the above command after building the model.


hot.accept parser stops traversing based on number of args

addresses #6919
What kind of change does this PR introduce?
bugfix
Did you add tests for your changes?
yes
If relevant, link to documentation update:
N/A
Summary
#6919
Does this PR introduce a breaking change?
Not to any external interface

@ooflorent - any thing I can do to help move this along?

Thanks

@sokra / @ooflorent - now that #7083 is merged, is there something I need to do in order to get<EMAIL_ADDRESS>published with these fixes?
Then, I can close #6919
Thanks