Monitoring

hello,
thank you for your work!
building and uploading functions are clear. What about monitoring the serial bus?

haven't gotten around to integrating that in (haven't used serial monitoring enough to warrant doing it inside atom as apposed to in the command line). I know platformio has the capabilities to doing that, it would just be a matter of displaying it in an atom view.

Eli how are you doing that in the terminal?
On Oct 15, 2015 1:40 PM, "Eli Fatsi"<EMAIL_ADDRESS>wrote:

haven't gotten around to integrating that in (haven't used serial
monitoring enough to warrant doing it inside atom as apposed to in the
command line). I know platformio has the capabilities to doing that, it
would just be a matter of displaying it in an atom view.
—
Reply to this email directly or view it on GitHub
https://github.com/vigetlabs/platomformio/issues/9#issuecomment-148468847
.

@az-z see PlatformIO Serial Monitor
@efatsi how is difficult to communicate with platformio serialports monitor process and delegate PIPE.IN and PIPE.OUT to end user?

platformio serialports monitor --echo
It will automatically figure out what port to listen to. More documentation can be found here - http://docs.platformio.org/en/latest/userguide/cmd_serialports.html#platformio-serialports-monitor

@ivankravets wouldn't be difficult at all, I just haven't found the time/motivation to do the work.

@efatsi I understand you. Thanks a lot for your efforts! :beers:

I've just found serial monitor implementation for the Atom https://github.com/spark/spark-dev/blob/master/spec/views/serial-monitor-view-spec.coffee
Maybe, will be helpful for our contributors.

I would love to have this. I don't know if there are other ways but for me this is a hassle, change code, build, make sure my Serial monitor does not keep the port open then try to upload, go to my serial port, open port and look at serial output. It would be so much easier if after upload the atom/platformio would open the port i just uploaded to and next time i upload it closes it automatically so i don't have to do that.

I installed this atom terminal package and ran this inside the atom terminal to see the serial output:
platformio serialports monitor -b 115200
It works really well.
To do this:

make sure you have platformio installed
make sure you know your baud rate or port. For me, it's 115200


React 16 warnings

PropTypes and createClass are being access via the main React library, which will inhibit consumers from using this library with React 16.*
  console.warn node_modules/react/lib/lowPriorityWarning.js:38     Warning: Accessing PropTypes via the main React package is deprecated, and will be removed in  React v16.0. Use the latest available v15.* prop-types package from npm instead. For info on usage, compatibility, migration and more, see https://fb.me/prop-types-docs
console.warn node_modules/react/lib/lowPriorityWarning.js:38     Warning: Accessing createClass via the main React package is deprecated, and will be removed in React v16.0. Use a plain JavaScript class instead. If you're not yet ready to migrate, create-react-class v15.* is available on npm as a temporary, drop-in replacement. For more info see https://fb.me/react-create-class

I use this library with React 16, so it should be fine.
I think those warnings are coming from somewhere else. 🤔
If you look at the code, I don't use createClass at all, and I've installed the prop-types package separately, so I don't import it through the main React package.

Ah, you're right - sorry! These errors popped up for me at the same time as I implemented your library, but were related to a separate change. Thanks for the quick reply!

No worries, hope you like it :)
On Thu, 27 Sep 2018, 15:56 tjbenneche<EMAIL_ADDRESS>wrote:

Closed #97 https://github.com/ZeeCoder/container-query/issues/97.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ZeeCoder/container-query/issues/97#event-1871315595,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADglEgX5V2418iJpKFImDRE0BjW2vhKLks5ufOcSgaJpZM4W7D5P
.


Regeneration of YOLOX model

1. OS you are using e.g. Ubuntu 20.04, WIndows10, etc: Ubuntu20.04
2. OS Architecture e.g. x86_64, armv7l, aarch64, etc: x86_64
3. Version of OpenVINO e.g. 2021.2.185, etc: 2021.4.582
4. Version of TensorFlow e.g. v2.4.1, tf-nightly==2.5.0.dev20210128, etc: v2.6.0-rc1
5. Version of TensorRT e.g. TensorRT6.0 GA, etc: 8.0
9. URL of the repository from which the transformed model was taken
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/132_YOLOX
11. Issue Details
Regenerate and commit the model with the modifications of https://github.com/PINTO0309/openvino2tensorflow/commit/f037817dcaeebe9208293dd598d93f39d7d13602 and https://github.com/PINTO0309/openvino2tensorflow/commit/963fa53a52bd3e7bb6f17784037d395a0f596eee.

Integer quantization model conversion fails (nano, tiny).
This is probably due to the model, not the openvino2tensorflow issue.
The error is:
Integer Quantization started ========================================================
WARNING:absl:Importing a function (__inference__wrapped_model_4389) with ops with unsaved custom gradients. Will likely fail if a gradient is requested.
WARNING:absl:Importing a function (__inference_model_layer_call_and_return_conditional_losses_14552) with ops with unsaved custom gradients. Will likely fail if a gradient is requested.
WARNING:absl:Importing a function (__inference_model_layer_call_and_return_conditional_losses_11454) with ops with unsaved custom gradients. Will likely fail if a gradient is requested.
WARNING:absl:Importing a function (__inference_model_layer_call_and_return_conditional_losses_13233) with ops with unsaved custom gradients. Will likely fail if a gradient is requested.
WARNING:absl:Importing a function (__inference_model_layer_call_and_return_conditional_losses_8610) with ops with unsaved custom gradients. Will likely fail if a gradient is requested.
fully_quantize: 0, inference_type: 6, input_inference_type: 0, output_inference_type: 0
loc("model/tf.concat_14/concat"): error: 'tfl.concatenation' op quantization parameters violate the same scale constraint: !quant.uniform<i8:f32, 0.031797535717487335:-119> vs. !quant.uniform<i8:f32, 0.034242726862430573:-120>
ERROR: Failed to quantize:
Traceback (most recent call last):
  File "/usr/local/bin/openvino2tensorflow", line 2646, in convert
    tflite_model = converter.convert()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 729, in wrapper
    return self._convert_and_export_metrics(convert_func, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 715, in _convert_and_export_metrics
    result = convert_func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 996, in convert
    return self._optimize_tflite_model(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 218, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 208, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 689, in _optimize_tflite_model
    model = self._quantize(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 520, in _quantize
    return _mlir_quantize(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 218, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 208, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert.py", line 236, in mlir_quantize
    return wrap_toco.wrapped_experimental_mlir_quantize(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/wrap_toco.py", line 47, in wrapped_experimental_mlir_quantize
    return _pywrap_toco_api.ExperimentalMlirQuantizeModel(
RuntimeError: Failed to quantize:
All the conversion process is finished! =============================================

It is presumed that this is due to the different input ranges of the concatenated ope.
Therefore, it seems that it cannot be converted to an Integer quantization model due to model constraints.
loc("model/tf.concat_14/concat"): error: 'tfl.concatenation' op quantization parameters violate the same scale constraint: !quant.uniform<i8:f32, 0.031797535717487335:-119> vs. !quant.uniform<i8:f32, 0.034242726862430573:-120>

@NobuoTsukamoto Thank you. I haven't tried it yet, but I can think of two ways.

Downgrade to TensorFlow v2.4.1 and try
Multiply normalization during quantization by 1

However, conviction is not the way to be popular.

It seems that the error can be avoided by setting experimental_new_quantizer = False.
Later, I would like to share the code.

Fixes. v1.16.1

Added --use_experimental_new_quantizer option (Default: False)
Change EdgeTPU compiler option from -sa to -sad
https://github.com/PINTO0309/openvino2tensorflow/releases/tag/v1.16.1

@PINTO0309 -san
Thanks!!

Post-quantization training with experimental_new_quantizer = False reduced accuracy.
There may be model constraints.
YOLOX-Nano 416x416
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.064
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.118
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.063
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.018
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.064
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.120
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.154
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.166
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.031
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.156
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.306

Fixes: 74f3ca68cc785a143380a44f4028d47dfa54c928


Use prettier colors on the CLI reporter?

Summary
https://twitter.com/_nicojs/status/1017154960834945024
Referring to colored output in terminals:

We should use that in #StrykerMutator's clear text console reporter 😎

I was actually referring to the deep linking. It would be awesome if you could click to the exact location where the mutant is located. Pretty colors are nice as well, we already have some colors I think, but as I said in the twitter chat, it might all use a make over 💄. You may work your magic by all means 👍
The code is already prepared to write to file as well (it uses an output stream). Not sure if anyone is waiting for that feature, but if you want, you can add it as well. When writing to file we should remove color codes I guess.
The console report is also pretty verbose, which bothers me a little. It prints several lines per mutant. Not sure if others agree @stryker-mutator/contributors ?

Is there a dedicated emoji for scope creep? 🐍 perhaps?
What is the total feature set discussed here?
💄 Color make over for the CLI if in tty mode?
🔗 Links to original file lines in output?
👽 Emoji decorations in output if in a terminal that supports them?

Is there a dedicated emoji for scope creep?

I think you just coined the emoji: 🐍 it is! 🥇

💄 Color make over for the CLI if in tty mode?

Yes. Well... I'm not sure if tty is the correct check to be making. Non-tty can support colors and emoj.
For example:
In vscode debugger terminal:

In conemu:

In command prompt:

🔗 Links to original file lines in output?
👽 Emoji decorations in output if in a terminal that supports them?

If that is feasible for you :)

There is an issue for emoji support in Windows console: https://github.com/Microsoft/console/issues/190
It seems that there already is some support. So no need to actively check for support I think


Feature - Add Danger to the repository

What
Enhance PRs thanks to danger.systems.

[x] Mentions
[x] Changelog
[x] Prose
[x] Junit
[x] The coding love

Current coverage is 88.64% (diff: 100%)

Merging #12 into master will not change coverage

@@             master        #12   diff @@
==========================================
  Files            23         23
  Lines           599        599
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits            531        531
  Misses           68         68
  Partials          0          0

Powered by Codecov. Last update 775ead0...0408f1a

Coding in VBA

  Generated by :no_entry_sign: danger


Support STSWebIdentity credential provider

Describe the feature
aws-crt-swift 0.14.0 added needed parameters for the STSWebIdentity credential provider.  https://github.com/awslabs/aws-crt-swift/pull/199/files
Create a SDK wrapper for this credential provider and integration tests.
Use Case
To support all standard AWS credential providers
Proposed Solution
No response
Other Information
No response
Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

Add Integration test

Released with:
https://github.com/awslabs/aws-sdk-swift/releases/tag/0.35.0
https://github.com/smithy-lang/smithy-swift/releases/tag/0.40.0


Trim CARO, FAO, and STATO

By 'trim' I mean 'remove'.

Strange, getting this error:
Component is different, updating.
robot --catalog catalog-v001.xml merge --input phenio-edit.owl  -i components/bfo.owl  -i components/fao.owl  -i components/emapa.owl  -i components/fbbt.owl  -i components/go.owl  -i components/hsapdv.owl  -i components/mondo.owl  -i components/mpath.owl  -i components/nbo.owl  -i components/ro.owl  -i components/uberon.owl  -i components/upheno.owl  -i components/wbbt.owl  -i components/zfa.owl  -i imports/merged_import.owl \
        --output tmp/phenio-full-unreasoned.owl
Could not load imported ontology: <http://purl.obolibrary.org/obo/phenio/components/emapa.owl> Cause: http://www.berkeleybop.org/ontologies/phenio/components/emapa.owl
Use the -vvv option to show the stack trace.
Use the --help option to see usage information.
make: *** [phenio.Makefile:22: tmp/phenio-full-unreasoned.owl] Error 1

which only seems strange because this should be a component. I'm guessing this is some phenio-edit.owl -related thing again

The problem was that it (EMAPA) wasn't getting added to the catalog file for some reason

Ah, one underlying error is that source refresh isn't completing because the uberon-bridge-to-nifstd is missing (it 404's on https://raw.githubusercontent.com/obophenotype/uberon/master/src/ontology/bridge/uberon-bridge-to-nifstd.owl).

That bridge got removed, so I'll remove it here, too.
See https://github.com/obophenotype/uberon/commit/ebdc7b157765c359769b9bfda645feb39460ca79

Now I keep encountering this:
Mirror identical, ignoring.
if [ true  = true ] && [ true = true ]; then robot --catalog catalog-v001.xml merge  -i mirror/chebi.owl  -i mirror/eco.owl  -i mirror/ncbitaxon-taxslim.owl  -i mirror/oba.owl  -i mirror/pato.owl  -i mirror/uberon-bridge-to-emapa.owl  -i mirror/uberon-bridge-to-zfa.owl  -i mirror/uberon-bridge-to-ma.owl  -i mirror/uberon-bridge-to-wbbt.owl  -i mirror/uberon-bridge-to-fbbt.owl  -i mirror/uberon-bridge-to-fma.owl  -i mirror/cl-bridge-to-emapa.owl  -i mirror/cl-bridge-to-ma.owl  -i mirror/cl-bridge-to-fma.owl  -i mirror/cl-bridge-to-wbbt.owl  -i mirror/cl-bridge-to-fbbt.owl  -i mirror/cl-bridge-to-zfa.owl -o mirror/merged.owl; fi
make: *** No rule to make target 'imports/chebi_terms_combined.txt', needed by 'imports/chebi_import.owl'.  Stop.
Command exited with non-zero status 2

True, there's no rule to make chebi_terms_combined.txt, but there is one to make $(IMPORTDIR)/%_terms_combined.txt - this just doesn't appear to work for chebi?

(note that goals sometimes fail like this "not exists" because one of their dependencies does not exist:
$(IMPORTDIR)/%_terms_combined.txt: $(IMPORTSEED) $(IMPORTDIR)/%_terms.txt

so check if $(IMPORTDIR)/%_terms.txt and $(IMPORTSEED) exist as files.

True, the IMPORTSEED (tmp/seed.txt) didn't exist.
So I ran
sh run.sh make tmp/seed.txt

and that seemed to work until it raised:
org.semanticweb.owlapi.model.UnloadableImportException: Could not load imported ontology: <http://purl.obolibrary.org/obo/phenio/imports/chebi_import.owl> Cause: /work/src/ontology/imports/chebi_import.owl (No such file or directory)

OK, that's fair, it doesn't exist there, so I did this:
sh run.sh make refresh-chebi

and the mirrors update, but then:
robot --catalog catalog-v001.xml convert --input phenio-edit.owl --format ofn --output tmp/phenio-preprocess.owl
org.semanticweb.owlapi.model.UnloadableImportException: Could not load imported ontology: <http://purl.obolibrary.org/obo/phenio/imports/chebi_import.owl> Cause: /work/src/ontology/imports/chebi_import.owl (No such file or directory)

Even running sh run.sh make imports/chebi_import.owl alone produces that same error.

The solution here was just to create placeholder chebi_terms.txt and chebi_import.owl in the imports directory, as they weren't getting created by other means.
I had to do the same for eco, and pato too.


Fix date bug

fixed issue where load_day_of_data did not read data correctly (hour=0 read whole day not an hour) and related issue where detection_t_series were not named correctly

again - meant to merge into my own fork. Sorry Tom !


Update README.md

Added entry on tmux

Cool, is it like tabs?

It's much more than that. Very useful for remote server management when you want to run multiple things under one login or for terminal emulators like alacritty which do not allow tabs or panes


[Windows] mpv-dev-x86_64-20230807-git-140ec21.7z integrity check failed.

As the title，I am developing a Windows desktop application with flutter_meedu_videoplayer: ^4.2.26，but when run flutter run windows -d，I have encountered the problem:

And，this my flutter doctor:

In addition, I have tried the following command more than once, but the problem still exists:
flutter clean flutter pub cache clean flutter pub get flutter pub upgrade
And, there is a empty project for windwos desktop with flutter_meedu_videoplayer: ^4.2.26 only，It also encounter this issue and cannot run properly:
untitled.zip
So, I need some helps!

media_kit disabled all of my packages, so for now
this is a work in progress, but i think it should work on all platforms only web is not supported for now
  flutter_meedu_videoplayer:
    git:
      url: https://github.com/zezo357/flutter_meedu_videoplayer
      ref: fvp
      path: ./package

let me know if you have a problem with it, for now it will use fvp in all platforms

Thanks, I have tried with it, but I have encountered the problem:

Then, I close the player and opening it again is played.  That is to say, when using it to play videos, there is a gap between the above errors and the video screen appearing, that is: first play is error and then the second play video screen appearing, third is error and fourth is video screen appearing...And, the video screen may have a splash problem （The red line in the picture represents the overall area of the video）:

There is the screen recording for it:
https://github.com/zezo357/flutter_meedu_videoplayer/assets/34667902/12847c56-1d9d-4b02-9f69-fc96003399c5

Thanks, I have tried with it, but I have encountered the problem:  Then, I close the player and opening it again is played. That is to say, when using it to play videos, there is a gap between the above errors and the video screen appearing, that is: first play is error and then the second play video screen appearing, third is error and fourth is video screen appearing...And, the video screen may have a splash problem （The red line in the picture represents the overall area of the video）:

sadly i cant check it right now, but i think it may be related to fvp i will check it before releasing the new version
if you can provide the video you are testing with that will help greatly

Not just one video, all videos are like this:
https://github.com/zezo357/flutter_meedu_videoplayer/assets/34667902/f1f26f67-783c-4e54-9206-7d47d2d51955
There is two videos, and the problem is the same.

okay, will check it, thank you

But, I am waiting for the official version to be launched here. Is there any emergency solution or can the flutter_meedu_videoplayer previous version make the video play normally??

sadly media_kit disables the app if the package is found
but you can add
in pubspec video_player_win: ^2.2.2
and
after initMeeduPlayer
import 'package:video_player_win/video_player_win_plugin.dart';

if (!kIsWeb && Platform.isWindows) WindowsVideoPlayer.registerWith();

this will use another logic to render the video, maybe its better in your case
let me know if it works for you

Very unfortunate, I followed what you said add in pubspec video_player_win: ^2.2.2, and after initMeeduPlayer
import 'package:video_player_win/video_player_win_plugin.dart';
if (!kIsWeb && Platform.isWindows) WindowsVideoPlayer.registerWith();

It has encountered a new issue, causing the video to be unable to play and the screen to go blank:

Additionally, I have also tried use video_player_win: ^ 2.2.2 alone, The error content is consistent with the above

well sadly you will have to wait for the release
and for the problem with the render i opened an issue for it in fvp hopefully it gets fixed
https://github.com/wang-bin/fvp/issues/11
if you cant wait then i will have to say use media_kit directly since they even broke the wrapper i implemented

Enn... I am planning to use media_kit and its related plugins directly to try

same issue

Enn... I am planning to use media_kit and its related plugins directly to try

i hope the best for you, and if you need any help let me know

Okay, Thank you

same issue

media_kit disabled all of my packages, so for now
this is a work in progress, but i think it should work on all platforms only web is not supported for now
  flutter_meedu_videoplayer:
    git:
      url: https://github.com/zezo357/flutter_meedu_videoplayer
      ref: fvp
      path: ./package

let me know if you have a problem with it, for now it will use fvp in all platforms
and if that didnt work for your case sadly you will have to use media_kit directly or wait until i make a new release where i was able to fix the issues in the current logic

I tried and it cannot find the Player from package:flutter_meedu_videoplayer/meedu_player.dart

what do you mean by player?

meedu_player doesnt have a "player" class
do you mean the media_kit one? if thats the case you will need to fully convert your code to use media_kit you cant import this package with media_kit (media_kit will close the app on launch)

oh, thank you, I will find another solution.

but can I have a final confirm.

now flutter_meedu_videoplayer is not related to media_kit (and in the furure)?
these 2 lib cannot be used in 1 project?
is this effect on all platform?
is there anyway that I can use an old version for now, which not have errors (maybe some dependencies override)?

but can I have a final confirm.

now flutter_meedu_videoplayer is not related to media_kit (and in the furure)? yes
these 2 lib cannot be used in 1 project? yes
is this effect on all platform? macos and ios work for now , but most probably wont on the long run
is there anyway that I can use an old version for now, which not have errors (maybe some dependencies override)? No, media_kit deleted all old versions to break any workaround using old versions

Not just one video, all videos are like this:
2023-08-<IP_ADDRESS>.mov
There is two videos, and the problem is the same.

for this can you provide the logs ?
Logger.root.level = Level.ALL;
  Logger.root.onRecord.listen((record) {
    print('${record.loggerName}.${record.level.name}: ${record.time}: ${record.message}');
  });

it will help us greatly

same issue

media_kit disabled all of my packages, so for now
this is a work in progress, but i think it should work on all platforms only web is not supported for now
  flutter_meedu_videoplayer:
    git:
      url: https://github.com/zezo357/flutter_meedu_videoplayer
      ref: fvp
      path: ./package

let me know if you have a problem with it, for now it will use fvp in all platforms
and if that didnt work for your case sadly you will have to use media_kit directly or wait until i make a new release where i was able to fix the issues in the current logic

Can you confirm that the new version using fvp does not support web? If the new version does not support the web, I will have to give up "flutter_meedu_videoplayer" and look for a solution again!

Looking forward to your new version being released, I have been waiting for 12 hours in the discussion area because the old version is also unusable and I have to stop working!

Looking forward to your new version being released, I have been waiting for 12 hours in the discussion area because the old version is also unusable and I have to stop working!

Remind me did you try the fvp version I mentioned? Can you let me know what didnt work for you?

I have tried... Unfortunately, I am not sure where the problem occurred. I cleaned the project and even reinstalled NDK, but the compilation still went wrong! The error prompt did not find the file, but I don't know how to solve it!

What went wrong:
Execution failed for task ':fvp:configureCMakeDebug'.

C/C++: C:\Pub\Cache\hosted\pub.flutter-io.cn\fvp-0.1.0\android\CMakeLists.txt debug|armeabi-v7a : CMake Error at C:\Users\Administrator\AppData\Local\Pub\Cache\hosted\pub.flutter-io.cn\fvp-0.1.0\android\CMakeLists.txt:67 (include):
include could not find load file:

  mdk-sdk/lib/cmake/FindMDK.cmake

I have tried... Unfortunately, I am not sure where the problem occurred. I cleaned the project and even reinstalled NDK, but the compilation still went wrong! The error prompt did not find the file, but I don't know how to solve it!

What went wrong:
Execution failed for task ':fvp:configureCMakeDebug'.

C/C++: C:\Pub\Cache\hosted\pub.flutter-io.cn\fvp-0.1.0\android\CMakeLists.txt debug|armeabi-v7a : CMake Error at C:\Users\Administrator\AppData\Local\Pub\Cache\hosted\pub.flutter-io.cn\fvp-0.1.0\android\CMakeLists.txt:67 (include):
include could not find load file:

  mdk-sdk/lib/cmake/FindMDK.cmake

yes i got it i just opened pr in fvp for a similar error

please check new release

Okay，There is the logs of Exception: No Stream<VideoEvent> for textureId: -1:

And，This is the logs of video can be played, but the screen is splash:
splash_screen.txt

Okay，There is the logs of Exception: No Stream<VideoEvent> for textureId: -1:  And，This is the logs of video can be played, but the screen is splash: splash_screen.txt

In the splash_screen.txt，I have made desensitization modifications to some path data

@coder-xb  this is on the latest version?

@zezo357 Yes, This is flutter_meedu_videoplayer fvp branch

@zezo357 Yes, This is flutter_meedu_videoplayer fvp branch

can you try on the release too?

@zezo357 Yes, This is flutter_meedu_videoplayer fvp branch

can you try on the release too?

Not yet. Currently, for emergency purposes, I am using media_kit directly.

@zezo357 Yes, This is flutter_meedu_videoplayer fvp branch

can you try on the release too?

Not yet. Currently, for emergency purposes, I am using media_kit directly.

its okay, thank you very much.

Have you released the new version?

Have you released the new version?

Yes I did yesterday

4.3.0

Okay, I will try and test it in future versions of my application.

Okay, I will try and test it in future versions of my application.

If you did the migration, then I recommend you stick with it, since not all bugs or issues has been fixed yet,
It's working very good on Android/iOS/web/macos
Windows and Linux are not tested to make sure everything is good yet, it should work correctly but no promises here 😢
If you left the branch of using my package, then all is good try and let me know if you have any issues and I will try my best to solve it

Okay, I will try and test it in future versions of my application.

If you did the migration, then I recommend you stick with it, since not all bugs or issues has been fixed yet,
It's working very good on Android/iOS/web/macos
Windows and Linux are not tested to make sure everything is good yet, it should work correctly but no promises here 😢
If you left the branch of using my package, then all is good try and let me know if you have any issues and I will try my best to solve it

Okay，Thank you

new release working on windows, but can I ask in future release, is there any change that I can use meedu with media_kit? (now Im using media_kit for audio player)

new release working on windows (if I remove the media_kit lib), but can I ask in future release, is there any change that I can use meedu with media_kit? (now Im using media_kit for audio player)

Sadly you would have to request that from media_kit since they are the ones who implemented the logic to terminate any connection between us and them


Will there be support for RustRover and eventually Fleet?

Hi! Just asking to see if there will be support for RustRover and eventually Fleet.

rustrover is now supported

thx

成功激活php goland rustrover非常好用 其他开发的失效了


Create data with relationship works with null id

Hej,
if I send a POST request with something like this
{
  "data": {
    "type": "photos",
    "attributes": {
      "title": "Ember Hamster",
      "src": "http://example.com/images/productivity.png"
    },
    "relationships": {
      "photographer": null
    }
  }
}

I get a 201, but I am expecting an error (400 for example). When I check the database there is no constraints on the table for the foreign key. Is there a way to add constraints on the database or validate the input ?
Thomas

I just tried to create a resource including a relationships object like in your example:
{
  "relationships": {
    "owner": null
  }
}

Which resulted in a 400 error. However, this relationships object works, and is valid:
{
  "relationships": {
    "owner": { "data": null }
  }
}

If you are using the Postgres adapter, there is the useForeignKey option which is not enabled by default. The reason why is explained here. Basically because it doesn't help much if referential integrity is handled at the application level, and also because it's slower to use a join table.

Thanks for the answer. There is something I don't get : your second example breaks the referential integrity. Here is a more concrete example.
const user = {
    email: String,
    password: String,
    phone: String,
    avatar: String,
    role: String,
    customer: ['customer', 'users'],
};

const customer = {
    name: String,
    address: String,
    apikey: String,
    role: String,
    users: [Array('user'), 'customer'],
};

const adapter = [
    postgresAdapter, {
        url: `postgres://${config.get('DATABASE_USER')}:${config.get('DATABASE_PASSWORD')}@${config.get('DATABASE_HOST')}:${5432}/${config.get('DATABASE_NAME')}`,
        useForeignKeys: true,
        generatePrimaryKey: () => uuid.v4(),
    },
];

const store = fortune({
    // Put models here
    customer,
    user,
}, {
    adapter,
});

Then if I POST this request
{
  data: {
    type: 'users',
    attributes: {
      email<EMAIL_ADDRESS>      password: 'invalid',
      phone: 'invalid',
      avatar: 'invalid',
      role: 'admin'
    },
    relationships: {
      customer: {
        data: null
      }
    },
    jsonapi: {
      version: '1.0'
    }
}

I have a 201 and in the database I have something like

Where can I refuse to insert a null reference ?

Well there are two ways to do that: either modify the database schema so that the column is required, or specify an input hook.
The input hook would look like:
const { errors: { BadRequestError } } = fortune

function userInput (context, record, update) {
  switch (context.request.method) {
    case 'create':
      if (!record.customer)
        throw new BadRequestError('A customer is required.')
      break
    case 'update':
      if (update.replace && update.replace.customer === null)
        throw new BadRequestError('Can not unset customer.')
      break
  }
}

To tell fortune to use it, it should be passed into the constructor function:
fortune(recordTypes, {
  hooks: {
    // Order is input, output
    user: [ userInput, null ]
  }
})

A null value in a foreign key column is valid, according to this answer.

Thanks, I will try both solutions and pick one.

I can't successfully register the hook. I have an undefined value.
const store = fortune({
  // Put models here
  customer,
  device,
  user,
}, {
  adapter,
  hooks: {
    customer: [() => console.log('Input hook'), null],
  },
});

And I get

TypeError: Cannot read property 'length' of undefined
at map (node_modules/fortune/lib/common/array/map.js:13:24)
at node_modules/fortune/lib/dispatch/delete.js:71:19
at process._tickCallback (internal/process/next_tick.js:103:7)

That looks correct. Actually you may have discovered an edge case bug: when a delete request is issued without IDs it does not seem to find any records. This should not be allowed to do except directly via the adapter, so it seems to be a messaging problem as well.

It should be fixed in https://github.com/fortunejs/fortune/commit/580587a9a73402eed239859d5779c07f7c555270
Basically the delete without IDs can't work properly via request, so it is just disabled now.

Or it can be fixed by updating the map function to
module.exports = function map (array, fn) {
  var i, j, k = [], l = 0

  if (!array)
    return k

  for (i = 0, j = array.length; i < j; i++)
    k[l++] = fn(array[i], i, array)

  return k
}

I used to use delete without ID to truncate tables in my tests.

It works without ids if I directly call the adapter. So I only have to update
store.delete(/* ... */)

to
store.adapter.delete(/* ... */)

Thanks a lot !

Yep, that was the change. Allowing a delete request without IDs was dangerous because it could potentially drop all records without running the input hook, so it was made more explicit by calling the adapter method directly.

Hej, sometimes (around 50% of the time9 I have an error in my truncate function which is this one
function truncate() {
  return store.adapter.delete('user')
    .then(() => store.adapter.delete('customer'));
}

The error is

TypeError: Cannot read property 'query' of undefined
at Promise (node_modules/fortune-postgres/lib/index.js:485:13)
at PostgreSQLAdapter.delete (node_modules/fortune-postgres/lib/index.js:484:12)
at Object.truncate (test/utils/database.js:32:30)
at Context. (test/api/customer.spec.js:21:21)

The only way that is possible is if you are disconnected already: https://github.com/fortunejs/fortune-postgres/blob/4f827294bf5b100313c588bdebf4651abd11d030/lib/index.js#L152
Are you sure you are not calling disconnect somewhere?

I never called disconnect (I double checked again). And I manually connect like this :
  before(function () {
    return store.connect();
  });

  beforeEach(function () {
    return database.truncate();
  });

Are you sure that the test runner resolves Promises? It is possible that both the connect and delete methods are called simultaneously. If that is the case then you have to chain the Promises yourself:
store.connect().then(() => database.truncate())

I chain myself the Promises and it works ! I will close this issue and try to find another way than calling the connect method into the beforeEach hook.


spec/mempool: First version of a formal specification

A formal specification of the transaction propagation protocol in the mempool would allow to identify problems in the current implementation, propose optimizations, and help design a future implementation.
Tasks

[ ] mempool: First Quint spec of the transaction propagation protocol

When the formal spec is properly documented, there should not be a need of an informal spec.

[ ] mempool: Define safety and liveness properties

Checked with Apalache if/when possible

Will this be done in collaboration with Pierre? Or is it something you would like to do in preparation before he starts?

I plan to do this one with Pierre. Before, I would just like to experiment a bit with Quint to have a better understanding of the language.


Add code of conduct to contributing guidelines

I have discussed this with @atom/core and they intend to adopt and uphold this code of conduct. It will apply to all repositories in the @atom GitHub organization, the discussion forum, and chat rooms.
A blog post on atom.io will be coming soon about it.

:+1:

:+1:
Should this maybe be in its own section in CONTRIBUTING? What about adding it to the Readme too?

What about adding it to the Readme too?

I think it should be in the README

Thanks for this @bkeepers it is a great step forward :feet:  :+1:

I haven't read through the details yet, and I'm sure it's the right move forward.  Does it change anything the core devs or the community currently do in any significant way?

But what's the expected impact of this CoC?

The only immediate expected impact is that when people visit the chat room, repo, and forums, they know the project has a code of conduct in place.

Does it change anything the core devs or the community currently do in any significant way?

No significant changes, but it does present a document to refer to when discussing potential conduct issues with people.

Ok understood. :+1:
I'll read into the details some more as well.


added more information

added a document with research that confirms the illustrations of the data provided, and updated the notebook with the last portion of the graphs made.

@wvelebanks I missed this! Can you review the merge conflicts?

Will do!
On Mon, Dec 23, 2024, 11:15 AM Jacob Harris @.***>
wrote:

@wvelebanks https://github.com/wvelebanks I missed this! Can you review
the merge conflicts?
—
Reply to this email directly, view it on GitHub
https://github.com/datakind/datakit-housing-fall-2024/pull/66#issuecomment-2559969627,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGIKZJ4TSJYMPRI2CJ2WPUL2HAZINAVCNFSM6AAAAABSPNSFLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJZHE3DSNRSG4
.
You are receiving this because you were mentioned.Message ID:
@.***>


MB-2091 Merge storybook test and build and fix failure in cirlce ci

Description
Recent merge to master seems to be causing intermittent failures of storybook_tests on master. Ari and I were able to replicate a similar error locally by reducing the amount of docker memory to 2MB locally and clearing out the image cache by adding --no-cache to line 33 of scripts/run-storybook-tests. But this was not consistent.

Example Failure https://app.circleci.com/pipelines/github/transcom/mymove/16498/workflows/c11c8756-89fb-45a2-8c6e-c2d2a823cd37/jobs/260980 and
Example Success https://github.com/transcom/mymove/pull/3954 https://app.circleci.com/pipelines/github/transcom/mymove/16500/workflows/3ef4bbb3-c14b-408f-9816-dd4d7d8c3c51/jobs/261046

The suggested fix to make it more stable is to increase the executor size from medium to medium_plus.
This PR also merges storybook_tests and build_storybook_app as the latter is not very big or long, and probably a waste of a whole container. This will require a change to the required jobs for a build. I'll touch base with infra for that.
Also noticed we still had --network-concurrency 1 passed to yarn install which shouldn't be required now that we are using a released version of react-uswds. So this PR removes that to allow yarn to install things in parallel once again.
Setup
To run the storybook tests, the exit code should be 0 if everything worked as expected.
make storybook_tests

To test the removal of --network-concorrency 1 you should also run
rm .client_deps.stamp
make client_deps

Code Review Verification Steps

[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

References

MB-2091 for this change
this build failure message in slack
this error

Here is the error I see:
storybook_tests_1  |  FAIL  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |        default
storybook_tests_1  |        Failed with "captureScreenshot timed out after 30000ms" after 4 tries
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app

So my question is, can you configure the screenshot timeout to be longer? I was reading this PR: https://github.com/oblador/loki/pull/140
I think you're saying above that if you restrict the RAM to 2GB (not 2MB) that you see the issue. Maybe its not memory related but CPU related? The resource_class change between medium and medium_plus is only an additional 2GB Ram and an additional CPU. So you'd be going from 2CPU/4GB to 3CPU/6GB. I'm not sure that should have fixed it. You can see the differences here: https://circleci.com/product/features/resource-classes/
I'd love more research on this before just upping the class.

The original failures that we were looking at looked different which seemed to imply a memory issue. Also increasing the time out won't help if chrome has crashed. I'll see if I can increase the timeout or reduce the number of tests in parallel.
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/scenes/Landing
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  PASS  chrome.app/chrome.iphone7/Samples|Form
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|Left Nav
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/scenes/Landing
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app/chrome.iphone7/Components|Left Nav
storybook_tests_1  |        layout
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/scenes/Landing
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app/chrome.iphone7/Components|Tables
storybook_tests_1  |        Table Elements
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |        Standard Tables
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |        Service Item Tables
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/scenes/Landing
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app/chrome.iphone7/scenes/Landing
storybook_tests_1  |        StatusTimeline
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |  RUNS  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app/chrome.iphone7/Components|TabNav
storybook_tests_1  |        default
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |        withTag
storybook_tests_1  |        connect ECONNREFUSED <IP_ADDRESS>:39867
storybook_tests_1  |  RUNS  chrome.app: Running tests
storybook_tests_1  |  FAIL  chrome.app
storybook_tests_1  | Chrome could not be killed kill ESRCH
storybook_tests_1  | Error: Chrome could not be killed kill ESRCH
storybook_tests_1  |     at Promise (/home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:259:28)
storybook_tests_1  |     at new Promise (<anonymous>)
storybook_tests_1  |     at Launcher.kill (/home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:241:16)
storybook_tests_1  |     at Object.<anonymous> (/home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:58:29)
storybook_tests_1  |     at Generator.next (<anonymous>)
storybook_tests_1  |     at /home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:12:71
storybook_tests_1  |     at new Promise (<anonymous>)
storybook_tests_1  |     at __awaiter (/home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:8:12)
storybook_tests_1  |     at Object.kill (/home/circleci/project/node_modules/chrome-launcher/dist/chrome-launcher.js:53:28)
storybook_tests_1  |     at Object.stop (/home/circleci/project/node_modules/@loki/target-chrome-app/src/create-chrome-app-target.js:29:22)
storybook_tests_1  | error Command failed with exit code 1.
storybook_tests_1  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.```

https://app.circleci.com/pipelines/github/transcom/mymove/16496/workflows/d1d4ded5-868e-4f43-ade8-b0decc25f87d/jobs/260914

The original failures that we were looking at looked different which seemed to imply a memory issue. Also increasing the time out won't help if chrome has crashed. I'll see if I can increase the timeout or reduce the number of tests in parallel.

This does feel like its more of a configuration issue for loki than a circleci problem. Let's run that down first and once we've exhausted options we can fall back on the last resort of spending more money.

I don't see how it's a configuration issue, tests were passing and suddenly It can't connect to chrome anymore. Though I agree on running down some loki options for fixing it without increasing the executor size.

I don't see how it's a configuration issue, tests were passing and suddenly It can't connect to chrome anymore. Though I agree on running down some loki options for fixing it without increasing the executor size.

This tells me its not necessarily a CircleCI issue either. It could be that we now have more tests running than we did before, in which case configuring the test runner makes sense. We could also use CircleCI parallelism if this becomes an issue but I'm hoping loki can manage on its own.

The only timeout that looks configurable from the loki docs is loading the story, not taking the screenshot. So I decreased the concurrency from 4 to 2 and upped the retries from 0 to 5, though I think that is just for loading the story not taking the screenshot.
https://loki.js.org/command-line-arguments.html
If this passes then I'll be good with this and keeping an eye on it as it goes

@chrisgilmerproj I made the changes you mentioned, I also removed approve_prod_deploy as part of the update since it's a change Erin and I had talked about. I remembered about it when updating the comment near it since that's caused some confusion. Seems to me that there is no need to block deploy of the storybook site on that job since the storybook static site doesn't have an effect on the deployed app as far as I know. If I have missed something please let me know and I'll change it back.

@chrisgilmerproj I made the changes you mentioned, I also removed approve_prod_deploy as part of the update since it's a change Erin and I had talked about. I remembered about it when updating the comment near it since that's caused some confusion. Seems to me that there is no need to block deploy of the storybook site on that job since the storybook static site doesn't have an effect on the deployed app as far as I know. If I have missed something please let me know and I'll change it back.

The approve_prod_deploy was intended to keep us from constantly pushing out changes whenever someone merged to master. Sounds like the intent has changed and like app-staging you want storybook to be consistent with master. If that's the case then I don't see an issue.

This looks good to me, but I haven't seen any failures of builds in a few days. Did something else resolve that issue?

Looks like it happened again since this, a couple of hours ago so still worth merging to hopefully stabilize it.
https://app.circleci.com/pipelines/github/transcom/mymove/16595/workflows/2cbd9e19-4b33-482b-9c08-6dcbf91f4743/jobs/262770


fix(doc): need to serialize text ranges

[x] I am sure that the code is update-to-date with the dev branch.

close #1476

todo: @zhaolixin7    复现场景：在协同时可以复现报错，dev无法协同待合到staging上验证


Visually indicate outliers in the score table

Like green for lower than mode/median, red for higher.

Nice! I was going to request this feature :)


Doesn't work with the Journals Plugin

Summary
The journals plugin has replaced both the calendar by cain and the periodic notes plugins for me (and a lot others actually). As it is a superior experience.
However I wasn't able to tell this plugin to choose daily or weekly notes, as it doesn't sense that it has them. Even though it does.
Details
No response

It sounds like the Journals plugin is going to have a major revamp, cf. srg-kostyrko/obsidian-journal#54. So I'm happy to add this functionality, but I'd rather wait until srg-kostyrko has revamped his plugin, so I'm redoing a whole bunch of stuff just after implementing it.

Yeah totally! I didn't know he was doing major work to it tbh, I just have used all of the calendar plugins, and found this one to be the best. (Even out of the big ones haha)
You're right to wait until after!
On Saturday, July 13, 2024 at 10:40:59 PM CDT, mirnovov @.***> wrote:
It sounds like the Journals plugin is going to have a major revamp, cf. srg-kostyrko/obsidian-journal#54. So I'm happy to add this functionality, but I'd rather wait until srg-kostyrko has revamped his plugin, so I'm redoing a whole bunch of stuff just after implementing it.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


Access Control

V1 - Users can have admin role or basic role.
V2- Admins can edit permissions of users and archive users
V3 - Admins can set specific permissions for individual users

V1 - Users can have admin role or basic role.
V2- Admins can edit permissions of users and archive users
V3 - Admins can set specific permissions for individual users


Want to Improve the UI of Web3.0-World/Level2/MetaNft_Minting/

Change the threejs platform overall look to look better and also the form UI

@suraj-git07 Great idea .Start working on this .

@suraj-git07 Are us till working on this ?

Hey, is somebody working on this?
If not then I can sure do.

@Vikash-8090-Yadav ser I can on this issue and make it better, Please assign me this issue.


Fix Fix complex type crash when using dict filter facility in the parquet-reader.

Proposed changes
Backport from  #27151

Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case.
Add orc complex types regression test.

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buidall

run buildall


Move block number mapping update out of test_state

Moves block number mapping update to execute_txs out of test_state.

LGTM. One suggestion: make  let mut state = CachedState... not mutable in test_state().

fund_account needs &mut state.


Add option to disable notifications

I would like to be able to disable the notifications that are shown everythime a command was sent to syrinscape. There are multiple ways I could see this change.

Just add an option to allow people to simply deactivate the notifications.
Replace it by better UX/UI solutions. Maybe show a small loading indicator on the button the user clicked until the command was successful. Or display that loading indicator somewhere else unobstrusive but visible. Maybe some small syrinscape icon in the top right corner of the canvas that animates in some way until the command was successful.

Fixed in 0.4.0-beta2


GitHub Actions: Deprecating set-output commands

Description
This repository uses the command set-output, which is deprecated. GitHub Actions recently began adding the following annotation warning all users of slackapi/slack-github-action:

Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

You should replace all command set-output (core.setOutput) with theGITHUB_OUTPUT environment variable (core.exportVariable)
What type of issue is this? (place an x in one of the [ ])

[ ] bug
[x] enhancement (feature request)
[ ] question
[ ] documentation related
[ ] example code related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

I no longer get annotation warnings. Using @actions/core 1.10.0  seems to solve the issue.

Still experiencing this with uses<EMAIL_ADDRESS>


PR: Change reporting of macOS system from Darwin to macOS and include machine type

Description of Changes
In About widget, changed system info reporting for macOS from "Darwin" to "macOS" with appropriately changed release designation.
Also added machine type ('arm64', 'x86_64', 'AMD64', etc) to the system info for all platforms.
In the system info for issue reporting (and "Copy version info" of About widget), operating system value is replaced with the value of platform.platform.
"macOS 13.3" is more meaningful than "Darwin 22.4.0" to users, issue reporters, and developers. Additionally, inclusion of machine type may eliminate the need for developers to ask issue reporters "Is your machine Intel or M1?".
New "About" widget text

New Issue text

Spyder version: 6.0.0.dev0 47a906db (standalone)
Python version: 3.9.16 64-bit
Qt version: 5.15.6
PyQt5 version: 5.15.7
Operating System: macOS-13.3-x86_64-i386-64bit

@spyder-ide/core-developers, what do you think about this PR?

Sounds like a good idea @mrclary ! Maybe is worthy to add this change in 5.x too?

Sounds like a good idea @mrclary ! Maybe is worthy to add this change in 5.x too?

I have no problem with that, depending on what others have to say.

@spyder-ide/core-developers, do you think this should be moved to 5.x?

+1 from me. For that (if others thinks that's ok), please use following commands:
git checkout mac-version
git rebase --onto 5.x master mac-version
git push -f origin mac-version

@dalthviz, did you want to say anything or are you good?

Let's merge this one.


Streaming files

Add stream endpoint to server, so that it can write file chunks directly to a file. This allows the server to receive files that are larger than its host's memory.

LGTM!


Enhancement: groupby performance

The performance of groupby has been degrading over time as we have added different functionality, particularly when we added Series.
groupby needs a better way to parallelize the computation and do the "reduce" portion of the MapReduce.
I will tackle this one.

Resolved by #659


functions in public schema are not exposed via graphql

Describe the bug
A function created in the public schema is not exposed via graphql.
To Reproduce
Steps to reproduce the behavior:

Set up supabase locally using the cli.
Use any of the provided examples here: https://supabase.github.io/pg_graphql/functions/
Observe that no functions are being exposed

Expected behavior
A function created in the public schema should be exposed via graphql.
Versions:

PostgreSQL: [e.g. 15.1]
pg_graphql 1.2.3

Hey @mwahlhuetter,
Function support was only added in pg_graphql version 1.4.
If you're using Supabase, you should be able to upgrade your project to access the latest version here. You'll also need to run
drop extension pg_graphql;
create extension pg_graphql;

after doing so.
I'll close this issue for now, but please feel free to re-open it if upgrading does not solve your problem.

Hi!
Thank you for the fast response. Im not using any project right now, im just using the local containers that are created with the supabase start command. In the GraphQL docs I found this
select * from pg_available_extensions where name = 'pg_graphql'

but the latest available version I get is my current version 1.2.3.
Is there a guide somewhere in the docs how I can upgrade pg_graphql?

For the local CLI you'll need to make sure you're running the latest version, then stop (with the --no-backup option) and start your project. Note that you'll lose any data in your local database by using the --no-backup option.
There's a full guide available here: https://supabase.com/docs/guides/cli/getting-started#updating-the-supabase-cli

That did it! Thank you so much.
I just installed the CLI about two weeks ago and thought i was on the latest version..


add a training script

This PR adds a basic training script that uses Replicate's training API
See https://replicate.com/docs/guides/fine-tune-a-language-model
cc @fofr

@zeke Did you see #3?

Closing as a duplicate of #3

How embarrassing! #3 comes before #4


Database (e.g. postgres)

resolution_fixed   |    by<EMAIL_ADDRESS>
Reviewboard (Trac #66) requires access to a postgresql or mysql database.
At the moment there's some postgres related packages in the repo (e.g. browser:/puppet/modules/postgresql) however these only install packages, they don't handle setting up the DB.
Is there going to be a database puppet module available in the NCI repository that we can use (pref. puppetlabs/postgresql) or should we add this on our end?

Issue migrated from trac:79 at 2024-01-31 18:00:02 +1100

<EMAIL_ADDRESS>changed reviews which not transferred by tractive

<EMAIL_ADDRESS>changed component from accessdev server to NCI Puppet

<EMAIL_ADDRESS>commented

Sorry, just saw this (emails weren't being forwarded).
Yes, we'll be adding the puppetlabs PostgreSQL module to the core framework.

<EMAIL_ADDRESS>changed status from new to accepted

<EMAIL_ADDRESS>commented

Milestone accessdev v1.0 deleted

<EMAIL_ADDRESS>removed milestone (was accessdev v1.0)

<EMAIL_ADDRESS>changed status from accepted to closed

<EMAIL_ADDRESS>changed resolution from `` to fixed

<EMAIL_ADDRESS>commented

Added to core framework (changeset:7a0106b/puppet)


Session > Inactivity: Inactivity is too high

ChargeX reported a high inactivity, could you please check?

Could be a duplicate of https://github.com/sap-labs-france/ev-dashboard/issues/2307. That should be checked.

@WillyLellouch Any news on this one?
Thanks.

@WillyLellouch @jerome-benoit Any news on this one?

@LucasBrazi06 I will need the full copy of the database entries for the consumptions of this transaction. There is not enough info in the logs to determine the source of the issue.

@LucasBrazi06 I will need the full copy of the database entries for the consumptions of this transaction. There is not enough info in the logs to determine the source of the issue.

You don't have the logs but you have the metervalues and consumptions collections, that should be enough.
Logs are too big to be backed up and made the dump failing.

@LucasBrazi06 I will need the full copy of the database entries for the consumptions of this transaction. There is not enough info in the logs to determine the source of the issue.

You can also connect to the customer's tenant to get the logs.

@LucasBrazi06 Yes I already have the logs from the customer's tenant. They are not enough to determine the source of the issue. And I do not have access to the database. I need to be able to see the inactivity evolve over time in the consumptions, which is not logged except in the status updates (which is insufficient)

@LucasBrazi06 Yes I already have the logs from the customer's tenant. They are not enough to determine the source of the issue. And I do not have access to the database. I need to be able to see the inactivity evolve over time in the consumptions, which is not logged except in the status updates (which is insufficient)

I pushed a backup of the AWS DB in our sharepoint, I told Jerome already.
I don't get what is missing?

@LucasBrazi06 I wasn't made aware that the backup was done. Since it is, I have everything I need. Thanks.

@LucasBrazi06 I wasn't made aware that the backup was done. Since it is, I have everything I need. Thanks.

I've forwarded to you the mail with the link to the export previous Thursday.

@LucasBrazi06 https://github.com/LucasBrazi06 Yes I already have the logs from the customer's tenant. They are not enough to determine the source of the issue. And I do not have access to the database. I need to be able to see the inactivity evolve over time in the consumptions, which is not logged except in the status updates (which is insufficient)
I pushed a backup of the AWS DB in our sharepoint, I told Jerome already.
I don't get what is missing?

Nothing is missing. Willy, could you please proceed with the import of the DB to dig the data and identify meter values records correlated to a ongoing transaction that might lead to incorrect inactivity computation?
Thanks.
--
Jérôme Benoit aka fraggle
Piment Noir - https://piment-noir.org https://piment-noir.org/
OpenPGP Key ID : 27B535D3
Key fingerprint : B799 BBF6 8EC8 911B B8D7 CDBC C3B1 92C6 27B5 35D3

The issue is not currently reproducible. We are investigating with the simulator but it would be beneficial if we could have a dedicated station from ChargeX as it seems this particular issue is only encountered on their tenant.

The issue is not currently reproducible. We are investigating with the simulator but it would be beneficial if we could have a dedicated station from ChargeX as it seems this particular issue is only encountered on their tenant.

@WillyLellouch You can request them to run again a transaction on the same charger.
Let me know.

@WillyLellouch You can request them to run again a transaction on the same charger.
Let me know.

This doesn't seem to be tied to any one charging station and the charging stations that have already had the issue do not necessarily have the issue again. Some have had the issue only once, other twice. There are also no commonalities in the transactions affected, be it duration, user, or any other parameter. That's one of the main issue in identifying the problem.


Workarounds for gVisor (and Kata)

Commit 1: ocihook: use prestart (deprecated, though) instead of createRuntime
prestart is deprecated in favor of createRuntime which was added in the OCI Runtime Spec v1.0.2,  however, gVisor and Kata still do not support createRuntime.
Workaround for #787

https://github.com/kata-containers/kata-containers/issues/1935
https://github.com/google/gvisor/issues/7149

Commit 2: ocihook: workaround for gVisor/Kata state annotations bug
gVisor and Kata do not propagate annotations automatically, so we have to copy the annotations from the spec to the state.

https://github.com/kata-containers/kata-containers/issues/3629

Commit 3: CNI: enable "loopback" plugin by default
gVisor requires "loopback" plugin to be enabled explicitly to bring up the loopback interface.

gVisor
# runsc --version
runsc version release-20220131.0
spec: 1.0.2-dev

# nerdctl run -it --rm --runtime=io.containerd.runsc.v1 alpine
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65522
    link/loopback 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/8 scope global dynamic
    inet6 ::1/128 scope global dynamic
2: eth0: <UP,LOWER_UP> mtu 1500
    link/ether 1e:a7:b5:7a:4b:d4 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 scope global dynamic
    inet6 fe80::1ca7:b5ff:fe7a:4bd4/64 scope global dynamic
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
v3.15.0-258-gd5d32e35cf [https://dl-cdn.alpinelinux.org/alpine/v3.15/main]
v3.15.0-259-g8b98485bc8 [https://dl-cdn.alpinelinux.org/alpine/v3.15/community]
OK: 15857 distinct packages available

(still no support for rootless)
Kata
# /opt/kata/bin/kata-runtime --version
kata-runtime  : 2.3.2
   commit   : 1af292c9e693e9bc8e8324a9eb860dad45306fb5
   OCI specs: 1.0.2-dev

# nerdctl run -it --rm --runtime=io.containerd.kata.v2 alpine
FATA[0000] failed to create shim task: Unsupported network interface: bridge: unknown

CNI config

# nerdctl network inspect bridge --mode=native
[
    {
        "CNI": {
            "cniVersion": "0.4.0",
            "name": "bridge",
            "nerdctlID": 0,
            "nerdctlLabels": {},
            "plugins": [
                {
                    "type": "bridge",
                    "bridge": "nerdctl0",
                    "isGateway": true,
                    "ipMasq": true,
                    "hairpinMode": true,
                    "ipam": {
                        "ranges": [
                            [
                                {
                                    "gateway": "<IP_ADDRESS>",
                                    "subnet": "<IP_ADDRESS>/24"
                                }
                            ]
                        ],
                        "routes": [
                            {
                                "dst": "<IP_ADDRESS>/0"
                            }
                        ],
                        "type": "host-local"
                    }
                },
                {
                    "type": "portmap",
                    "capabilities": {
                        "portMappings": true
                    }
                },
                {
                    "type": "firewall"
                },
                {
                    "type": "tuning"
                },
                {
                    "type": "loopback",
                    "name": "lo"
                },
                {
                    "type": "isolation"
                }
            ]
        },
        "NerdctlID": 0,
        "NerdctlLabels": {}
    }
]

Kata v2 no longer supports "bridged" CNI?
https://github.com/kata-containers/kata-containers/blob/3fdc97e11091e0c404c4db49ab1c354075d2c3a2/src/runtime/virtcontainers/network.go#L1117-L1153
cc @liubin

@AkihiroSuda this workaround may not work with Kata Containers. Kata Containers runtime(shim) will create a new network namespace and run hooks in this created namespace. In fact, CNI plugins should run in the host network namespace.
Basically said, in Kubernetes or containerd, kubelet/ctr will create the network namespace and pass it to Kata Containers, nerdctl uses a hook to create the network interfaces, and doesn't take the charge of creating network namespace.
So Kata Containers must do some changes as long as nerdctl uses this pattern to handle CNI plugins.

Linking kata-side fix: https://github.com/kata-containers/kata-containers/pull/3630
Thanks to @liubin

Kata v2 no longer supports "bridge" CNI? https://github.com/kata-containers/kata-containers/blob/3fdc97e11091e0c404c4db49ab1c354075d2c3a2/src/runtime/virtcontainers/network.go#L1117-L1153
cc @liubin

@AkihiroSuda Kata is not bound to any particular CNI plugin, but tries to plumb whatever the plugin created in the pod's netns with an interface (e.g. a TAP interface) passed to the VMM. CNI bridge is supported.
So the bridge plugin will create a veth pair, and leave one end on the pod's netns. Kata scans this netns and finds the veth end. It will then create a TAP interfaces, set TC mirroring between the veth  and TAP and boot the VM with the TAP.
So Kata does not have CNI specific implementations but behaves differently depending on the set of interfaces found on the pod's netns.

@AkihiroSuda this workaround may not work with Kata Containers. Kata Containers runtime(shim) will create a new network namespace and run hooks in this created namespace. In fact, CNI plugins should run in the host network namespace.
Basically said, in Kubernetes or containerd, kubelet/ctr will create the network namespace and pass it to Kata Containers, nerdctl uses a hook to create the network interfaces, and doesn't take the charge of creating network namespace.

@liubin @AkihiroSuda Excuse my ignorance, but what's the reason for nerdctl to not create a netns for the container/pod?

@AkihiroSuda this workaround may not work with Kata Containers. Kata Containers runtime(shim) will create a new network namespace and run hooks in this created namespace. In fact, CNI plugins should run in the host network namespace.
Basically said, in Kubernetes or containerd, kubelet/ctr will create the network namespace and pass it to Kata Containers, nerdctl uses a hook to create the network interfaces, and doesn't take the charge of creating network namespace.

@liubin @AkihiroSuda Excuse my ignorance, but what's the reason for nerdctl to not create a netns for the container/pod?

@sameo I think nerdctl uses a Docker-style network process mechanism, that uses an OCI hook to create the network.
Now Kata Containers run hooks in the network namespace that Kata Containers created:
// Run pre-start OCI hooks.
err = EnterNetNS(sandboxConfig.NetworkConfig.NetworkID, func() error {
	return PreStartHooks(ctx, ociSpec, containerID, bundlePath)
})

For nerdctl, the hook runs in the created namespace, so the CNI plugins also create cni0, eth0 interfaces in that namespace, which leads Kata Containers to report an error.

@AkihiroSuda this workaround may not work with Kata Containers. Kata Containers runtime(shim) will create a new network namespace and run hooks in this created namespace. In fact, CNI plugins should run in the host network namespace.
Basically said, in Kubernetes or containerd, kubelet/ctr will create the network namespace and pass it to Kata Containers, nerdctl uses a hook to create the network interfaces, and doesn't take the charge of creating network namespace.

@liubin @AkihiroSuda Excuse my ignorance, but what's the reason for nerdctl to not create a netns for the container/pod?

@sameo I think nerdctl uses a Docker-style network process mechanism, that uses an OCI hook to create the network.

Right. And those hooks happen to call into CNI plugins iiuc.

Now Kata Containers run hooks in the network namespace(I think this should be reviewed) that Kata Containers created:
// Run pre-start OCI hooks.
err = EnterNetNS(sandboxConfig.NetworkConfig.NetworkID, func() error {
	return PreStartHooks(ctx, ociSpec, containerID, bundlePath)
})

According to the runtime spec, the createRuntime hook must be run in the runtime namespace, i.e. the host namespace. So I think we should run the nerdctl as a createRuntime hook as it's currently implemented, and from Kata we should not run it in the sandbox netns. Our preStart support is also wrong, it should not run in the sandbox netns.

For nerdctl, the hook runs in the created namespace, so the CNI plugins also create cni0, eth0 interfaces in that namespace, which leads Kata Containers to report an error.

If we run the createRuntime in the host ns, and pass the created netns through the linux.namespace entry, wouldn't that work?

I think nerdctl uses a Docker-style network process mechanism, that uses an OCI hook to create the network.

Yes. We rely on OCI hook so that we can support nerdctl run --restart=always.


Does not quit using vue-slideout

Is there an example of using vue-slideout?

Wrong repo sorry, you can ask the question to vue-slideout directly :)
https://github.com/vouill/vue-slideout


Unable to search for "extension type:story"

Thanks  for a great service, only one problem so far: the above search term invariably results in "Server error" message. No other term that I've used causes this behavior. Grateful if you could check it out.

Hi, so there are a couple problems here:
"Server Error" happens when the search query times out. This tends to
happen when the mentions of search terms are rare. It's been like this
forever, and I don't know / remember enough about postgresql full text
search indexing to say exactly why, or how to fix it. I suspect I'm doing
something wrong there.
You can work around this by querying for "extension score>1" instead, which
is (almost) equivalent because only stories (not comments) have scores. I
think this works because postgres uses this higher-cardinality index prior
to filtering the results by the full text index, or something like that.
Which brings us to the second problem – the query I suggested does not time
out, but it returns no results. I don't know why that is. It seems to be
specific to the "extension" search term. Searching for "chrome score>1"
does return results have the "extension" word in them, so the data is
there. "extension" does not appear to be a postgresql reserved word, and
anyway searching for "table score>1" does return results, so even if it
was, it shouldn't be an issue.
Hnapp's postgresql schema is written out in one of the project files, if
anyone knows how to solve either of these issues, PR is welcome. The
existence of this issue bugs me, but sadly I don't have the availability to
figure it out.
--Nikita

Ok thanks! I'm no expert on postgresql fulltext things either so can't help you there atm. One thing I noticed is that the timeout seems to occur for all search terms beginning with "ex", maybe that could be of help to someone with more insight. I'll just avoid those terms in my queries for the time being.

Seems to not be the case, I tried "external score>1" and "experiment score>1" and both worked (one of them took a few seconds, but still). Which ex-word didn't work for you?

OK right, I was not clear; I did not use the 'score>1' query in my tests, I tested using the 'type:story' syntax, and all such that starts with 'ex' fails for me.  Which seems it could  be useful for someone trying to debug the problem.
For all other terms (even those beginning with 'ex') except "extension" I can use the 'score>1' syntax and get the valid results.
Sorry for the confusion.

Ah. It's not specific to "ex" words, but to words that are rarely used in
the set of HN items that you're requesting - that first problem that I
mentioned. For example both "ex" terms that I mentioned are also queryable
with the "type:story" syntax, but for example "extreme type:story" times
out trying to find the first 30 matching items, because there are very very
few stories that match this, so it needs to look at ALL of the data in the
DB, which fails because the query plan or indexing is probably screwed up
in some way. If the index was properly configured I think it wouldn't need
to do what feels like a table scan.

Message ID: @.***>

--Nikita


Add "examples" section

Related to https://github.com/logux/docs/pull/98

I created Recipes with the same purpose as your Example. Do you see a difference between them?

I thought that Recipes are meant to include step-by-step instructions for common cases (auth, deployment, etc.), but in Examples I included a repository with sample application (but if it's better to add an example to Recipes then I'll update the PR)

The difference is too shady. Your example and pretty similar with step-by-step guide.
Let’s avoid creating menu section for the single element. Can you change that PR?

sure, I've updated the PR: https://github.com/logux/docs/pull/98


Parallel.js is slow due to unnecessary per-job overhead

There seems to be a significant overhead per job, not just per worker.
My task is to run the levenshtein algo, comparing a given string with about 200000 target strings and finding the best match. This is ideal for parallelism, because it runs the exact same algo thousands of times, the algo is small, needs little data and no context. Without parallelism, it takes a significant time that's noticeable for the end user.
So, I've used paralleljs to run it on multiple CPU cores at the same time. However, I find that using paralleljs actually makes it slower than before with the synchronous version.
Environment:

node 13.2
Ubuntu Linux 20.04
CPU Ryzen 7 1700X, 8 cores with SMT, i.e. 16 virtual cores

Terminology:

worker: Maps to a OS level thread or process and run on a separate CPU core than other workers, in parallel to the other workers.
job: Running the function once for one of the data values.

I understand the purpose of Parallel.js to create the workers and to distribute the jobs to the workers in an efficient manner and with a comfortable and easy to use API.
Test results
Synchronous
No parallel.js used, all evaluations run in a single JS thread
664 ms with 88959 (longer) values
358 ms with 89548 (shorter) values
93 ms with 31552 values
10 ms with 2994 values
0 ms with 79 values
1 ms with 137 values
1 ms with 823 values
0 ms with 360 values
This is too slow and what we want to improve using parallel.js
Using paralleljs with default options
Same with maxWorkers = 16 (= number of SMT virtual CPU cores)
2067 ms with 88959 values
2036 ms with 89548 values
907 ms with 31552 values
332 ms with 2994 values
272 ms with 79 values
287 ms with 137 values
287 ms with 823 values
280 ms with 360 values
It's 300 times slower for small amount of data. This is might be the overhead of creating worker threads.
It's even 3-6 times as slow for large amount of data. This is somewhat surprising, because it cannot be the overhead of creating the worker threads, because they should be limited to 16 and be re-used for additional data, so I would expect at worst the time of the synchronous execution, plus the 280ms overhead of creating the workers, plus a tiny little overhead for the copy of the data. That would be around 700 ms for the second test, but it's actually 2000 ms, almost 3 times as long.
Interestingly, the first test (longer strings) and second test (shorter test) cost about the same time. This makes a big difference in the synchronous execution, but not with parallel.js, where the larger and shorter strings take about the same time. So, the slowdown isn't due to the pure data copy time. (See also the note to the last test below.) This shows that the amount of data has less of an influence than other factors.
This indicates that there might be something wrong (very inefficient) in parallel.js. There is some significant overhead per job, e.g. that workers are not re-used, by re-created all the time for each job, or some other huge overhead per job.
with maxWorkers = 8
(= number of physical CPU cores without SMT)
2002 ms with 88959 (longer) values
1782 ms with 89548 (shorter) values
778 ms with 31552 values
198 ms with 2994 values
156 ms with 79 values
166 ms with 137 values
173 ms with 823 values
153 ms with 360 values
Split up array
Starting only as many jobs as CPU cores, by manually splitting up the value array before running Parallel
593ms with 88959 values
526ms with 89548 values
357ms with 31552 values
304ms with 2994 values
285ms with 79 values
285ms with 137 values
290ms with 823 values
284ms with 360 values
Here, I first manually split up the data array into one array per CPU core, leading to an nested array with 16 elements, each being an array with thousands of values. This feeds a large array to each worker, one per CPU core. Then, within (!) the worker, I make a synchronous loop over the second level array and the thousands of values.
This is to test whether parallel.js is inefficiently distributing the work, e.g. by creating too many workers, or something with a similar effect. This appears to be the case.
I would expect that Parallel.js creates 1 worker per maxWorker, then makes a synchronous loop over the data and runs the function on each value. If it did, it should get speeds very similar to this here. Given that Parallel.js is about 4 times slower, that indicates that there's something to improve.
Split up array, with 8 workers
578ms with 88959 values
457ms with 89548 values
228ms with 31552 values
167ms with 2994 values
142ms with 79 values
147ms with 137 values
155ms with 823 values
149ms with 360 values
This is coming closest to the synchronous version, because it's also sl
1 Worker
This is the same as the last 2 cases, just with core = 1. This means there is no parallelism and the entire data array is passed into a single worker. This is effectively the same as the synchronous version without any parallel.js, just that it's running in a single worker created by Parallel.js.
1178ms with 88959 values
838ms with 89548 values
316ms with 31552 values
83ms with 2994 values
48ms with 79 values
45ms with 137 values
60ms with 823 values
52ms with 360 values
This should be slightly slower than the synchronous version, due to the the overhead of creating the workers, and a lot slower than the parallel one, due to only 1 CPU core being used. However, it's a lot faster than the non-split parallel version, which should not be the case.
This also shows that the data copy, which still has to happen here, is a lesser factor. The data copy needs to happen here as well. Yet, it is a lot faster than the non-split version, which indicates that the data copy is not the primary reason for the slowness.
Conclusion
Using Parallel.js in this case is dramatically slower than the synchronous version, by 1 or 2 orders of magnitude.
Inherent thread creation overhead
There is a certain overhead of using Parallel.js, which is very significant for small inputs: hundreds of times slower, i.e. 2 orders of magnitude slower. That is due to the fundamental cost of creating threads or processes.
This could be worked around by the caller, by checking the amount of input and avoiding Parallel.js in these cases. The factors to consider are: Amount of input data, execution time per function call, and execution time per thread creation.
Data copy overhead
There is also an overhead for copying the data. However, while that is there, results show that this is not the primary reasons for the slowdown.
Needless overhead
There is a huge overhead per job, not just per worker.
The results suggest that Parallel.js creates more workers than maxWorkers, or has other significant overheads per job with a similar effect.
There should be no overhead per job, only per worker and for the data transfer.
It would be good to look into this.

Test code
This is a reduced test case. It's not the exact code I ran, but similar to what I tested with.
Syncronous
import leven from 'didyoumean2';
let startTime = new Date();
let input = "What is going on";
let matches = ["I don't know", ... ]; // thousands of strings with lengths around 10 to 40 chars.
let results =  matches.map(match => leven(input, match));
results = results.filter(result => result.score <= 0.1).sort((a, b) => a.score - b.score);
console.log((new Date() - startTime) + "ms with", matches.length, "values");

Using paralleljs
import leven from 'didyoumean2';
let startTime = new Date();
let input = "What is going on";
let matches = ["I don't know", ... ]; // thousands of strings with lengths around 10 to 40 chars.
let parallel = new Parallel(matches, { env: { input } })
let results = await parallel
  .require(leven)
  .map(match => leven(global.env.input, match));
results = results.filter(result => result.score <= 0.1).sort((a, b) => a.score - b.score);
console.log((new Date() - startTime) + "ms with", matches.length, "values");

Split-up the array
import leven from 'didyoumean2';
let startTime = new Date();
let input = "What is going on";
let matches = ["I don't know", ... ]; // thousands of strings with lengths around 10 to 40 chars.
const cores = 16;
let lengthPerCore = Math.ceil(matches.length / cores);
let worksplit = [];
for (let i = 0; i < cores; i++) {
  worksplit.push(matches.slice(i * lengthPerCore, i * lengthPerCore + lengthPerCore));
}
let parallel = new Parallel(worksplit, { env: { inputText }, maxWorkers: cores })
let results = await parallel
  .require(leven)
  .map(valuesSplit =>
    .map(match => leven(global.env.input, match)));
results = results.flat(1)
results = results.filter(result => result.score <= 0.1).sort((a, b) => a.score - b.score);
console.log((new Date() - startTime) + "ms with", matches.length, "values");

Looking at the code, the code does seem to try to re-use workers in parallel.map(). It's possible that there are bugs which prevent the reuse, but I'll presume that this is not the case.
The message passing between controlling process and the spawned worker process might be slow. I can see that the data is passed to the worker one by one, and returned one by one. It is quite likely well be that this constant back and forth between the processes is what's so slow. It would match and explain the numbers that I measured.
The fix should be obvious: Pass all values at once into a worker. I.e. do what I did with the split-up code. That leads to a 3x to 4x speedup for workloads with 100000 calls.


Initial upload of VQSR workflow

Hi @jkgoodrich , @konradjk, cc @wlu04. Please have a look at the VQSR workflow and review.
I decided to put resources file paths and parameters required for the workflow to vqsr_resources.json to avoid having too many function/CL arguments in case people want to change the values or files.
There's also a main() function using argparse in case people want to copy the script and run it from the command line. Not sure if this is neccessary though...

Hey all, this is the sort of thing I'd love to pull into Hail mainline. Perhaps best to discuss after v4 is released, but I'd like to ask: Are y'all interested in also/alternatively PR'ing this to Hail?
I want to strike a balance of:

People with relevant knowledge (mostly you all) maintain the code (and perhaps update it as y'all's thinking about VQSR changes).
The interface is stable and easy to use for a wide array of Hail users (which I see as either antagonistic or at best extra work for you all; but exactly the work that Hail is in the business of doing).

Additionally, I want to incorporate this into an example VDS combiner pipeline because folks expect a "jointly-called" dataset to include these annotations.

Oh hello,
I would both 1) be very very interested in PRing this into Hail! 2)
politely would very very much like to hold this until after v4. I
believe KC is actually handling post-v4 priorities at the moment, I can
definitely add this to the list of things to be discussed post-v4!
On Tue, Oct 10, 2023 at 12:31 PM Dan King @.***> wrote:

Hey all, this is the sort of thing I'd love to pull into Hail mainline.
Perhaps best to discuss after v4 is released, but I'd like to ask: Are
y'all interested in also/alternatively PR'ing this to Hail?
I want to strike a balance of:

People with relevant knowledge (mostly you all) maintain the code
(and perhaps update it as y'all's thinking about VQSR changes).
The interface is stable and easy to use for a wide array of Hail
users (which I see as either antagonistic or at best extra work for you
all; but exactly the work that Hail is in the business of doing).

Additionally, I want to incorporate this into an example VDS combiner
pipeline because folks expect a "jointly-called" dataset to include these
annotations.
—
Reply to this email directly, view it on GitHub
https://github.com/broadinstitute/gnomad_methods/pull/470#issuecomment-1755801349,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASXZRYQZ7HVQULKURQSDKZDX6VZ57AVCNFSM54MSKCU2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZVGU4DAMJTGQ4Q
.
You are receiving this because you were assigned.Message ID:
@.***>

I would probably advocate for these two ideas to be independent. We can get this into gnomad_methods, test it out, make sure people can use it, and then when we have more bandwidth, push into Hail where we need to do more robustness testing anyway

@LindoNkambule Is this still the latest and greatest VQSR batch pipeline?
Hail team just generated wave 2 of BGE and I'd like to deliver it to the analysts with VQSR results rather than without.
And is this how you generate the input VCF? If not, could you share the code you use?
vds = hl.vds.read_vds(vds_file)
mt = hl.vds.to_dense_mt(vds)
t = gnomad.utils.sparse_mt.default_compute_info(mt)
t = t.annotate(info = t.info.drop(
    'AS_SB_TABLE', 'AS_QUALapprox', 'AS_VarDP', 'AS_SOR', 'AC_raw', 'AC', 'AS_SB'
))
t = t.drop('AS_lowqual')
hl.methods.export_vcf(dataset = t, output = out, tabix = True)

And what, in your expert opinion, should we use for these parameters? Are the singletons important for a dataset like BGE which, I presume, doesn't have many trios?
    parser.add_argument('--transmitted-singletons', type=str, required=False)
    parser.add_argument('--sibling-singletons', type=str, required=False)
    parser.add_argument('--no-as-annotations', action='store_true')

Hi @danking, Yes, it is the latest pipeline. Not the greatest because the intervals file (UNPADDED_INTERVALS in vqsr_resources.json) does not have chrM  and there were also some regions on chrY that are not covered, so there might be a small drop in the number of variants post-VQSR if the initial input has chrM and/or chrY.
Yes, that is how I usually generate the input VCF.
If there aren't many trios, then I think --transmitted-singletons isn't necessary. @konradjk can you weigh in on this?

@LindoNkambule is the UNPADDED_INTERVALS meant to be the calling intervals used to generate the source GVCFs? I'm pretty sure we should be able to get from the PMs the calling intervals for our GVCFs.

Yes, I just used the current one since Laura said the intervals were optimized for runtime (approximately even runtime). Now that I think about it, I guess I could write a function that takes the input and creates the interval based on the input file...

I just slacked Laura, she mentioned:

The handcurated intervals were optimized for GenotypeGVCFs runtime, so probably not comparable for VQSR runtime

I think either generating intervals directly from the VDS or using the original calling intervals should be fine. AFAICT, the only important part is that the intervals in the interval list contain every variant in the VDS/sites-only-VCF.
Do you know if SplitIntervals merely partitions the intervals or will it actually cut intervals up? If the latter is true, then we could just send it
chr1:1-NNN
chr2:1-MMM
...

and have it split automatically. If the number of splits is sufficiently small and the number of variants in the dataset sufficiently large, I don't think there will be too much imbalance. Maybe near the splits containing the telomeres would be a bit light on actual variant data.

I'm not quite sure what the question is, but yes, if you don't really have many trios or sib pairs, the transmitted and sibling singleton stuff won't do much. It is something that should be encouraged to use but certainly shouldn't be required from a code POV

Don't we usually run VQSR before we've inferred sample relatedness? I don't recall manifest files indicating trios or sib relationships. Is there a usual way for us to deduce that information in preparation for VQSR?


Add panoptes-production-sidekiq-alt pod type

To address high latency in lower priority queues while UserSeenSubjectsWorker jobs in data_high are dominating available threads, this PR adds a single dedicated pod that will service the default (most important), data_medium, and data_low queues. This change adds an additional pod to the minimum total number of pods, but this cost makes sense in the short term (i.e., while panoptes replica DB is out of service during migration).

On hold for now due to more aggressive autoscaling resolving backups w/o need for queue-specialized pods.


Fix formula for regr_slope

Fixes #2940

The code comments say that the denominator is var_pop(x) but apparently it's covar_pop(x, x).
// REGR_SLOPE(y, x)
// Returns the slope of the linear regression line for non-null pairs in a group.
// It is computed for non-null pairs using the following formula:
// COVAR_POP(x,y) / VAR_POP(x)

From the code, it seems to be using VarPop.
It's been a few years and I implemented many of these functions in two days, and I ended up using snowflake/presto/clickhouse as bases, should probably have documented these in the code :-).
I think this one was from snowflake:
https://docs.snowflake.com/en/sql-reference/functions/regr_slope

Thanks @soerenwolfers!

https://github.com/duckdb/duckdb-web/pull/3048


Failed to compile using Solaris Studio 12.6

While trying to compile 4.2.3 using Oracle Developer Studio 12.6 I am getting:
"src/ctx.cpp", line 467: Error: Could not find a match for std::mapstd::string,zmq::endpoint_t::insert(std::pair<const char*, zmq::endpoint_t>) needed in zmq::ctx_t::register_endpoint(const char*, const zmq::endpoint_t&).
and a bunch of other Notes (notices.)
Please, any known workaround for this?
Thanks.

OS: Solaris 11.3
Compiler: cc: Studio 12.6 Sun C 5.15 SunOS_i386 2017/05/30
libc and libc++ are from Solaris 11.3

Closing here, tracking via https://github.com/zeromq/libzmq/issues/3090

@jurajlutter pull libzmq from the master branch, should be all good now


Avout javax.inject dependency

Hi,
I noticed that the javax.inject depndency with jakarta.inject.
Do you intend to maintain (add new features, support, ..) both variants in parallel?

We are already supporting both yes. 6.x for Jakarta.inject and 5.x for
javax.inject
Mentioned in the docs
https://avaje.io/inject/
There is a branch for the javax code and yes we are maintaining both
versions going forward.  This means all changes are made as PRs that are
merged into both branches.
On Sat, 27 Nov 2021, 6:58 am Decebal Suiu, @.***> wrote:

Hi,
I noticed that the javax.inject depndency with jakarta.inject.
Do you intend to maintain (add new features, support, ..) both variants in
parallel?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/avaje/avaje-inject/issues/166, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABTATO6ENL6VMB22H5QCETUN7DF5ANCNFSM5I3EDYXA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I think I can close this now.
TLDR:

for javax.inject use version 5.x (latest now 5.16)
for jakarta.inject use version 6.x (latest now 6.16)

We are maintaining both branches. For example, the fix for #167 is now in version 5.15 and version 6.16.
Re-open or add a further comment if that isn't clear.
Cheers, Rob.

Mentioned in the docs
https://avaje.io/inject/

I read that. I ask you because I see that the versions 5.14 and 5.15 published in Maven Central Repository have jakarta.inject as dependency instead of javax.inject and the last version with javax.inject as dependency is 5.13.
Am I wrong with something?

Oh yeah, I see the confusion. So this dependency below looks like a jakarta.inject one but is actually still javax.inject.
<dependency>
  <groupId>jakarta.inject</groupId>
  <artifactId>jakarta.inject-api</artifactId>
  <version>1.0.5</version>
</dependency>

Now the reason for that is because the old javax.inject dependency didn't include a module-info.  So this dependency is a updated / newer version of javax.inject but with module-info.
The javax.inject artifacts are no longer allowed to be released into the javax namespace - hence it is released as jakarata.inject : jakarta.inject-api : 1.0.5.
So for example, downloading the jar from https://repo1.maven.org/maven2/jakarta/inject/jakarta.inject-api/1.0.5/jakarta.inject-api-1.0.5.jar ... and open up the jar using zip etc we will see it holds javax.inject code + META-INF/versions/9/module-info.class
In summary, the 5.x releases are still javax.inject and that dependency is confusing but correct in the sense it is the latest one for javax.inject.

Thanks for the explanations! I understand now.