Add ReID (historical and on-the-fly).

Abstract
Re-Identification (ReID) is the problem of, given a single image of an object, finding all occurrences of the same subject in a database. The database contains images collected from cameras at different points in time. This is a widely known problem in research that has been solved in many different ways.
The most common approach is to prepare a gallery of images associated with IDs (e.g. person ID, shipment container ID). Then, a sample image of the subject is provided as input to extract its features and compare them against the ReID database. Finally, the output is the top-k similar images with respective IDs sorted based on a similarity score. This is similar a very similar approach to face recognition.
Usecases
In the image below, there are two key usecases where ReID can be applied.

Architecture
In the image below, there is the high-level architecture of our solution. The first architecture useses EdgeMesh for the communication aspects while the second Apache Kafka.

Workflow
In the image below, it's shown the ReID workflow divided into three steps for the edgemesh version. The Kafka version follows the same logic apart from the way data is exchanged between the pods.

PR Details
This PR adds the necessary code to perform re-identification using a multi-layer edge-cloud infrastructure. It provides the following:

New CRDs in the build folder.
The folder examples/multiedgetracking with the full Python code to run the ReID application logic, including Dockerfiles and YAMLs.
The folder lib/sedna/algorithms/optical_flow for optical flow estimation using LukasKanade algorithm.
The folder lib/sedna/algorithms/reid with the algorithm for the ReID similarity functions.
The folder lib/sedna/backend/torch with the code to load Torch models.
The folder lib/sedna/core/multi_edge_tracking with the code for the ReID services.
The folder lib/sedna/datasources/kafka with the wrappers for Kafka (publisher and subscriber) + Fluentd reporter class.
Extra code in lib/sedna/common/ to perform application tracing using Fluentd.
Extra code for the REST endpoints in lib/sedna/services.
Extra code for new GO types under pkg/sedna/v1alpha1.
The folder pkg/globalmanager/controllers/multiedgetracking with the controller code for the ReID service.
Extra code in pkg/localcontroller/.
Two extra dependencies in requirements.txt (fluentd and kafka libs).

I recommend using a tracker because it’s more efficient.Tracker is keep running,the reid model is complete when the traget is searched.


Two small additions for Installation section of README.

Currently the guide assumes that the user has Python 3 in there system, which might not be the case for stable-ver. distros like Debian in which Python 2 is used by default.
I've also added the usage of the -h argument to encourage users to look at the help page.

Thanks!


Update Go versions: 1.13.7, 1.12.16

Update Go versions pending the fix for PR #8.

Side note: I'll work on an API endpoint so the users don't have to get get -u to properly use this tool.


Add ability to pass types in string extension definitions

TODO:

[x] If this PR is a new feature, reference an issue where a consensus about the design was reached (not necessary for small changes)
[ ] Make sure all of the significant new logic is covered by tests
[ ] Rebase your changes on master so that they can be merged easily
[ ] Make sure all tests and linter rules pass
[ ] Update CHANGELOG.md with your change

Resolves #430


Dask bag migration

Brings in stuff from dask.bag.into and its tests. I still need to add costs to the conversions here.

This seems fine to me.

@mrocklin @cowlicks is this ready for merging?

+1

Thanks!


First implementation of pipeline steps

Resolve #283
Description:

For the documentation changes, I recommend looking at cdd8190 alone so you can see the diff.
I added a performance test so that it doesn't take too long to load the HF tables as we add columns to them, but I'm afraid that the tests will be a bit flaky. Not sure what is the best way to proceed and how much buffer to leave.

Checklist:
You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

[x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
[x] PRE-COMMIT. You ran pre-commit on all commits, or else, you
ran pre-commit run --all-files at the end.
[x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact
our users.
[x] DEV CHANGES.

Update the documentation if this PR changes how to develop/launch on the app.
Update the README files and our wiki for any big design decisions, if relevant.
Add unit tests, docstrings, typing and comments for complex sections.

Hello @christyler3030, @nandhinibsn, could you please review my work on the front end? I recommend reviewing those 3 commits separately:

Preliminary work:

Replace makeStyles
Add and test camelToTitleCase

Preliminary work:

Add processing step selector


Switching from libmfx to libvpl

See the benefits here: https://www.intel.com/content/www/us/en/developer/articles/technical/onevpl-in-ffmpeg-for-great-streaming-on-intel-gpus.html#inpage-nav-2
Quote:
"Switching from libmfx to libvpl for the *_qsv codecs in FFMPEG provides an easy way to align with future updates from Intel. Media SDK has already had its final feature updates. Even if security fixes force a new release, features will not change beyond Media SDK’s current 1.35 API implementation. New features will be enabled for new Intel hardware in oneVPL. As an example, one of the first places that oneVPL features start to differentiate from Media SDK is AV1 encode.  Intel released GPUs with the industry’s first hardware accelerated AV1 encoder, which can be accessed via FFMPEG.  This approach is written on top of oneVPL.  In general, Media SDK represents legacy GPU support and oneVPL is for current and future Intel integrated  and discrete GPUs." - Quote from link intel.com article linked above
Fix:
"The switch to oneVPL from Media SDK in FFMPEG, is simple. Just substitute --enable-libvpl instead of --enable-libmfx in the configure stage of FFMPEG compilation.  Both options enable compilation of Intel® Quick Sync Video (*_qsv) codecs, which can be used without change from the FFMPEG command line or in applications.
Users can switch to oneVPL in FFMPEG, even for older Intel hardware." - Quote from link intel.com article linked above

https://github.com/BtbN/FFmpeg-Builds/commit/8fff2b2a8ee851f7b0ff401f6ca64b792a9e9f71

@BtbN Ok, sorry. Is it possible to also enable it for the 5.1 builds?

5.1 does not support libvpl

5.1 does not support libvpl

Ah ok


Refactor test files

Purpose
This is to rename the ballerina files of the test cases in order to identify the use case just by looking at the file name and add new use cases without conflicting.

h1 and h2 names are not clear. File name length s not a problem. Could you please use more meaningful names?

@chrishantha These names were used for following abbrevations (from the spec https://tools.ietf.org/html/rfc7540) in order to add new use cases without conflicting the file names.

h2 - HTTP/2 over TLS
h2c - HTTP/2 over cleartext
h1 - HTTP/1.1 over TLS
h1c - HTTP/1.1 over cleartext

If they are not clear, the longer names would be as follows:

http1_http1_passthorough.bal
http1_cleartext_http1_cleartext_passthorough.bal
http2_http1_passthorough.bal
http2_http2_passthorough.bal

WDYT?

@ldclakmal I prefer to use http1x and http2 rather than using h1 and h2. Also, I think we don't need to repeat the abbreviation in the file if there is no protocol switch. We can just say http2_passthorough.bal instead of http2_http2_passthorough.bal
WDYT?

@VijithaEkanayake Then the following Set 2 will be the updated set according to the suggested format. Is that okay?

Set 1
Set 2

h1_h1
http1s_passthrough

h1c_h1c
http1_passthrough

h2_h1
http2s_http1s_passthrough

h2_h1c
http2s_http1_passthrough

h2_h2
http2s_passthrough

Personally I like the shorter names since it is easy to get the meaning at a glance and h2, h2c these come from the spec itself.  These abbreveations are in use in the industry if you look at [1]
[1] https://line.github.io/armeria/apidocs/com/linecorp/armeria/common/SessionProtocol.html
But since it seems that majority are not familiar with the spec and these abbreveations let's use the set 2 for the names.

Even though I am usually against enigmatic names, I believe in this case we must go with short names as in the spec. I know it sounds foreign to us but for people with HTTP domain literacy it sounds normal.

@ldclakmal Seems these shorter names are widely used in the HTTP domain even in the spec too. Hence +1 to go with the shorter names in this case.


No directory found [.tmp]

I installed the unsilence on both my windows and my Linux machine and in both cases, I am getting the error that a file is not found.
Complete error:
┌──(kali㉿kali)-[~/Desktop]
└─$ unsilence sample1.wav sampout.wav
Calculating Intervals... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Type     ┃ Before          ┃ After           ┃ Difference        ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ Combined │ 0:00:32 (100%)  │ 0:00:22 (68.4%) │ -0:00:10 (-31.6%) │
│ Audible  │ 0:00:20 (62.0%) │ 0:00:20 (62.0%) │ 0:00:00 (0.0%)    │
│ Silent   │ 0:00:12 (38.0%) │ 0:00:02 (6.3%)  │ -0:00:10 (-31.6%) │
└──────────┴─────────────────┴─────────────────┴───────────────────┘

[?] Continue with these options? (Y/n) Y
Rendering Intervals... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Combining Intervals... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
FileNotFoundError: [Errno 2] No such file or directory: '/home/kali/Desktop/.tmp/9381a093-1ede-4d5f-babc-6d97cafdd5dc/out_final.wav' -> '/home/kali/Desktop/sampout.wav'

During handling of the above exception, another exception occurred:

FileNotFoundError: [Errno 2] No such file or directory: '/home/kali/Desktop/.tmp/9381a093-1ede-4d5f-babc-6d97cafdd5dc/out_final.wav'

The command executed on windows and linux are:
unsilence sample.wav sampleout.wav

For installation, I tried with the pip installation and as well as installation from the source.
OS: Windows 10 and Kali Linux

I've encountered this same error myself. It appears to happen only when you give it an audio file, that is, no video. Try passing the -ao / --audio-only flag in, it should work then!

Since this seems to be working when applying the flag, I will close this issue

I got that error now, maybe add autodetection if the input is an audio file will simplify.

I added that to the roadmap


Update Clear-SPSecureStoreCredentialMapping.md

Fix example formatting.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint-server/Clear-SPSecureStoreCredentialMapping.md
:white_check_mark:Succeeded
View (sharepoint-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


States (device_enabled) not updated on Iobroker after made change on Apps

Hi,
is it normal that when an action is done on the android application (Deactivate a camera for example), the information is not updated on the IoBroker side? (on: eufy-security.1.T84XXXXXXXXXX.cameras.T84XXXXXXXXXX.device_enabled)
The action to activate / deactivate the camera works well on IoBroker.
Test on Pan&tilt camera.
Thanks.

Hi, i have the same issue with IOS. When changes are made in App then, status in iobroker will not change.
If changes are been made manually in Iobroker, the cam is only accepting the first order : deactivate, but im not able to start the cam then.
Also 2k Pan&tilt camera.
Please let me know if debug / screens are needed.
Greetings Dennis

Same issue. Not only by App, but the state is not updated when setting within iobroker itself.
e.g. eufy-security.0.TXXX.station.current_mode will not change with setting eufy-security.0.TXXX.station.guard_mode.
And it only happens to certain device. My 2k indoor cam has this problem, but 2k outdoor cam w/ base station works fine.

It looks like on some devices, the info doesn't support push, so status update relies on pulling from server. After reducing the polling interval, the status update seems "timely".

device_enabled is not the case. It never update its status from p2p/server.

@lightgoleft
I am currently working on a new version of the adapter that will use the latest version of the driver. Some of the cited problems are already fixed there.
I will report here as soon as the new version is released and ask you to run the tests again.

Cool, will be excited about the new driver.
From: @.>
Sent: Wednesday, July 28, 2021 8:31 AM
To: @.>
Cc: @.>; @.>
Subject: Re: [bropat/ioBroker.eufy-security] States (device_enabled) not updated on Iobroker after made change on Apps (#146)
@lightgolefthttps://github.com/lightgoleft
I am currently working on a new version of the adapter that will use the latest version of the driver. Some of the cited problems are already fixed there.
I will report here as soon as the new version is released and ask you to run the tests again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/bropat/ioBroker.eufy-security/issues/146#issuecomment-888406570, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF2IBK5OXMBD7PF2UBXGKOLT2APD3ANCNFSM5APJ3SSQ.

@bricevalenza, @lightgoleft, @DM-81
Please test latest version (0.6.0) and let me know:
Note: If you change a setting of a device in the app, it will only be visible in the adapter after the polling interval (default: 10 min.).

@bropat
I tried the new version. But it got some serious problem:

device_enabled never get updated in iobroker when changed by phone app. Tried the indoor cam and outdoor Cam2.
the base station set guard_mode doesn't work at all. No local push (it used to be this mode on eufy basestation 2). No cloud pulling (eufy indoor cam).
It seems not pulling from cloud at all. I only see local push connection established within the log. Missing "Refresh data from cloud and schedule next refresh."

@lightgoleft
Please try to delete all object states of the adapter and restart it (this should fix everything):

Regarding point 3, this message is no longer written in the logs ;)
If not please set the adapter in debug mode and then send me the logs by mail.
You can find the mail address here.

@lightgoleft Version 0.6.1 fixes this problem by restoring all states.

I am kind of busy this week. I will give it a try in the coming weekends.
-------- Original message --------
From: bropat @.>
Date: 8/19/21 8:02 AM (GMT-08:00)
To: "bropat/ioBroker.eufy-security" @.>
Cc: lightgoleft @.>, Mention @.>
Subject: Re: [bropat/ioBroker.eufy-security] States (device_enabled) not updated on Iobroker after made change on Apps (#146)
@lightgolefthttps://github.com/lightgoleft Version 0.6.1 fixes this problem by restoring all states.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/bropat/ioBroker.eufy-security/issues/146#issuecomment-901990034, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF2IBK3FVA4ULKGEMHWJXITT5UMHHANCNFSM5APJ3SSQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

@lightgoleft
Is there any news about this?


Log User-Agent header

At the moment, it's not clear what programs are calling endpoints (and how frequently). It would be nice to log the User-Agent header and log how frequently what User-Agent calls what endpoint.
Some concerns:

just logging every request to a file would lead to more wear on the SD-card of the Pi (maybe logging to tmpfs could be possible)?
having a global object and storing the stats in there might not work well, since there are multitple workers (but we can use https://docs.python.org/3/library/multiprocessing.html#managers)
some scripts might not have a User-Agent header, or the header might be the same for two programs running on another machine, so it probably makes sense to also log the IP address

I think it would not make much sense to try and do this ourselves. Since we already have a Grafana instance running, maybe we can use something like https://grafana.com/oss/loki/ . This way we don't have worry about aggregating stats ourselves and can we keep this application simpler.


Harvester cloud provider enhancement

The epic issue is the collection of the issue related to the Harvester cloud provider enhancement.

[x] https://github.com/harvester/harvester/issues/1676
[x] https://github.com/harvester/harvester/issues/1458
[x] https://github.com/harvester/harvester/issues/1762
[x] https://github.com/harvester/harvester/issues/1580
[x] https://github.com/harvester/harvester/issues/2354
[x] Add migration support
[x] https://github.com/harvester/harvester/issues/2753

Moved to v1.2.1:

[ ] https://github.com/harvester/harvester/issues/1834

Close the epic issue as its sub-tasks are either close or verified.
For the other partially related issues, it will be tracked independently via:

https://github.com/harvester/docs/pull/323
https://github.com/harvester/harvester/issues/4329
https://github.com/harvester/harvester/issues/4232


Light ghost button disabled color is incorrect

Should be transparent white

:tada: This issue has been resolved in version 1.2.0-pre.6 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 2.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Question about "adding 2-hop neighbourhood filtering"

@apoorvumang hi, I am here again. Thanks for your responsible responses to all questions.
I want to make sure if "adding 2-hop neighbourhood filtering" you mentioned in 'Table 5' means that the prediction answer entities will be confined in the 2-hop neighbourhood related to the given topic entity in question at inference?

yes


paddingStart does nothing with react-native-web

Current Behavior
I'm using react-native-web to render tamagui into a website. First off, is this a recommended way of working? It's convenient for sure in that it makes more libraries available for me.
Anyway <YStack paddingStart={\$4`}>does nothing, while<YStack paddingLeft={`$4`}>` gives me a handful of pixels of space as expected.
Expected Behavior
Ideally paddingStart would alias to one side or the other depending on whether the browser language is LTR or RTL. As a fallback it'd either alias directly to paddingLeft or give some kind of error on web.
Tamagui Version
1.120.1

Platform (Web, iOS, Android)
Web

Reproduction
I really apologize I'm really short on time. This is, hopefully, instantly reproducible by anyone. If it's not please do let me know, or close the issue, and I'll loop back in a few days when I have the chance. Again, very sorry.

System Info
No response

Also, this may be something it's better to raise in the necolas/react-native-web repo directly. If so let me know! Either way it's probably good for this issue to exist so future devs can find their way.

We don't use rnw internally, so I guess we may need to fix this. TBH our entire RTL support needs work and tests.


Internal to external sync

@sid-github and @khandpur are the original author for those two commits, we should be good to merge it. please review and let me know!

The author is @sskhandp for the first commit.

I've signed it.

looks like cla/google won't verify it but leave the authors to consent, I think we are good wrt CLA.

fixed the wrong commit author.

looks like the broken ones are fixed after the travis change, waiting for others to complete.


Repeat 1 track not working

The repeat 1 track only feature dosent seem to work, when you click the repeat button so it gets red and a 1 above.
It plays the track then stops after.

Hmm. This may be part of the way Apple handles audio with DRM and direct streams. Ill check it out same with the GPU bug.

This feature should be working now! Please do let me know if it continues. (v1.5.0)


Alignment to left

Custom view is center aligned . Need to bring to left, how can we bring the view to left. My layout is aligned left but still dialog is center aligned. How to bring it to left.

can you please let me know any solution for this

Your view needs to match_parent in terms of width. Nothing in this library messes with your custom view gravity.


fix: Use string keys for return map in getPacketHeights

Description:
Commit Message
type: commit message

see the guidelines for commit messages.
Changelog Entry
version: <log entry>

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have documented my code in accordance with the documentation guidelines
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have run the unit tests
[ ] I only have one commit (if not, squash them into one commit).
[ ] I have a descriptive commit message that adheres to the commit message guidelines

Please review the CONTRIBUTING.md file for detailed contributing guidelines.

Codecov Report

Merging #608 (3171b69) into main (de34373) will decrease coverage by 0.20%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main     #608      +/-   ##
============================================
- Coverage     68.32%   68.12%   -0.20%
  Complexity      408      408
============================================
  Files           129      129
  Lines         12626    12459     -167
  Branches        294      294
============================================
- Hits           8627     8488     -139
+ Misses         3843     3815      -28
  Partials        156      156

Flag
Coverage Δ

java
80.08% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...ore/ibc/src/main/java/ibc/ics24/host/IBCStore.java
80.51% <100.00%> (ø)

... and 9 files with indirect coverage changes

Checked with relay looks good ...


Define carla host and port

By default, the plugin tries to connect to CARLA running on localhost, port 2000 and there is no option to input other values.
I added an additional dialog window to input host and port other than the default values. The dialog is opened after clicking the plugin's "Connect to carla" button.

Host and port are now provided by constructor arguments.


[FLINK-21504][checkpoint] Introduce notification of subsumed checkpoint

What is the purpose of the change
Introduce notification of subsumed checkpoint
Brief change log
Introduce notification of subsumed checkpoint. Add a new InternalCheckpointListener and related implementations.
Verifying this change
This change added tests and can be verified as follows:
SubtaskCheckpointCoordinatorTest
CheckpointCoordinatorTest
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@rkhachatryan @pnowojski , thanks for your review. I have refactored this PR with two commits:
1beb902: Refactor CompletedCheckpointStore#addCheckpoint to CompletedCheckpoint#addCheckpointAndSubsumeOldestOne.
36cfdc4: Introduce the notification of subsumed checkpoint which addressed your comments.

My local CI turns green: https://myasuka.visualstudio.com/flink/_build/results?buildId=362&view=results


frame-mAP or video-mAP?

Hi. It's me again.
In your paper and code, mAP was used for performance comparison.

Is it frame-mAP? or video-mAP?
Do you have any plans to add another one?

Thanks.

In log_coarse_fine_train.txt, there are several mAP values. Is the last mAP value is actual evaluation?

In the paper, mAP was 25.10 with Coarse-Fine model. However, I cannot get that much high value. As your log file, mAP is near 22-23%. How can I get proper result?

The intended here in temporal activity detection, which means making predictions per-frame. So, the given mAP is for frame-wise predictions.

Do you have any plans to add another one?
What do you mean?

Thanks for reply. I means, do you have plans to add code about video-mAP evaluation? or Can you guide me how to edit the code for video-mAP values?

In that case, the task would become recognition (instead of detection). If you still want to do it,
For labels (one-hot): take max across the temporal axis to get video level labels
For logits: correct thing would be to replace the final Linear layers to make video-level predictions (after average pooling over both spatio-temporal axes). Implementation should already be there, but you have to retrain, at least the last layers. Easy thing would be to consider max over temporal axis and use them as video-level predictions


is there a static library?

hello,
i want to use the Easyhook. but i can only find a dynamic library from the Tutorials. Can i use it by static library?
Thanks!

Not currently, I think there is another issue for this.

See #138


Files with singleton categoricals cannot be opened

What I mean by this is that you might have a categorical, which internally in the h5ad is represented by e.g. uns/condition_categories. If there is only one condition (perhaps because it's a subset of data), this will fail the Pandas check for unique categories, because the shape for uns/condition_categories is (1, ).
This can be avoided by when writing the file to h5py, detecting singleton categories and appending a dummy category, to ensure there are at least two unique values in uns/conditions_categories.

Sorry about this. It's a known bug https://github.com/theislab/scanpy/issues/102 and I should already have fixed it. I'll provide a fix and a new anndata release today.

Fixed in https://github.com/theislab/anndata/commit/330cb0d243ae6a81c86efd3cd310982d8066f4a3. Will prepare a new release 0.5.7 just in a bit after fixing another little thing...

0.5.7 is out with a few further minor updates...

Btw: congrats on your SpatialDE paper! And thank you for citing Scanpy. Soon, there will be Scanpy 1.0... a lot better...

Just tested 0.5.7 and it worked! Thanks!
Thanks for the congratulation too :), was looking at choices for highly variable genes detection in Bioconductor, but having a readily available Python defualt was quite a time saver!


Created calculus.py library and created root_compare.py

Adds the calculus.py library, which contains functions for both integration and root finding

Looks good


ProcessorUnit processes tasks without waiting scheduledTime on shutdown

Scenario:

task X (scheduledTime = now + 100sec) is in ProcessorUnit's queue
ProcessorSubscription#close()

ProcessosUnit#close() =>
ProcessPipeline#close() =>
ExecutionScheduler#close() => terminateLatch.countDown()

ExecutionScheduler#schedule() returns immediately for task X without waiting 100sec
task X is passed to DecatonProcessor

Hi ,I am trying to solve the issue ,I want to know that if this happens when the job need retry on the consumer side ?

@yifu-yang Thank you for your interest.
This issue happens when Decaton's scheduled execution feature is utilized.
As you may know, scheduled execution is like below:

Decaton enqueues consumed tasks to ProcessorUnit's executor queue
https://github.com/line/decaton/blob/master/processor/src/main/java/com/linecorp/decaton/processor/runtime/ProcessorUnit.java#L55
If TaskMetadata#scheduledTimeMillis is set, Decaton waits for a while before passing the task to DecatonProcessor.

And since Decaton's retry queueing is based on scheduled execution, the issue can be occurred when a task is retried as you mentioned.
But we found that this issue is already solved by https://github.com/line/decaton/pull/32, which made ProcessPipeline not to pass a task to processor if pipeline is already closed.
Then let me close this issue.
Thank you again.


Fix for MGET crash when upstream Redis Command Stats are enabled

The root cause was not handling all types in a switch statement in RedisCommandStats where I get the command string (ie get, set, etc.). Unfortunately the code path exercised by an mget request from downstream was not exercised so this was not caught by unit or integration tests.
I've explicitly handled all the types; specifically the RespType::CompositeArray which is used in mget. I've also added an integration test that verifies the behavior (and crashes as expected without the fix). The integration test is needed because the mget is downstream-specific; on the upstream side it is converted to gets and the client_impl_test.cc where we tested the RedisCommandStats stats measurement is upstream-specific.
Risk Level: Low
Testing: Local unit/integration on Mac
Docs Changes: N/A
Release Notes: N/A

Do you mind taking a look at CI and filling out the PR description per the https://github.com/envoyproxy/envoy/blob/master/PULL_REQUESTS.md

LGTM.  +1

Oops looks like you need a master merge.
/wait


⚠ Jenkins Plugins Artifactory has degraded performance

In f8b0184, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1035 ms

Resolved: Jenkins Plugins Artifactory performance has improved in cfb064a after 29 minutes.


1.11.3 stable release does not exist

Readme recommends 1.11.3 for 1.11.1
https://github.com/flow-storm/flow-storm-debugger/?tab=readme-ov-file#artifacts
1.11.3 does not exist
https://clojars.org/com.github.flow-storm/clojure/versions

Dang, forgot to upload that one to Clojars! It is there now. Thanks for reporting it.


la-go-live

Docs Build status updates of commit ed8ffaf:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/bots/how-to/update-and-delete-bot-messages.md
:white_check_mark:Succeeded
View

msteams-platform/resources/bot-v3/bot-conversations/bots-conversations.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Remove obsolete methods

[x] ExceptionHelper methods
[ ] StorageFileHelper class

ExceptionHelper part closed by https://github.com/bramborman/DotNetToolkit/commit/d5457c1897a8236517d7dda7afc67b183057083f. Class renamed to ValidationHelper.


Update force_cached.nix

I'm not sure of the root cause yet but we started seeing kill -9 after using this script.
See https://github.com/NixOS/nix/issues/2176#issuecomment-394341125

This seems to have fixed the problem


Error when trying to add Goodwe

Its not possible to connect to Goodwe 15kw with WiFi 2.0

Does the SolarGo/PvMaster mobile app work ?

Hi,
I have the same problem. Yes SolarGo and SEMS is working.
This is the log i get:
Logger: aiohttp.server
Source: /usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py:421
First occurred: April 1, 2024 at 22:27:58 (7 occurrences)
Last logged: 13:42:52
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
resp = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 91, in security_filter_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 25, in request_context_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 79, in ban_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 235, in auth_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 31, in headers_middleware
response = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 68, in handle
result = await handler(request, **request.match_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 72, in with_admin
return await func(self, request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 197, in post
return await super().post(request, flow_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
result = await method(view, request, data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 111, in post
result = await self._flow_mgr.async_configure(flow_id, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 359, in async_configure
result = await self._async_configure(flow_id, user_input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 405, in _async_configure
result = await self._async_handle_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 502, in _async_handle_step
result: FlowResult = await getattr(flow, method)(user_input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/goodwe/config_flow.py", line 97, in async_step_user
inverter = await connect(host=host, family=model_family, retries=10)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/goodwe/init.py", line 51, in connect
return await discover(host, timeout, retries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/goodwe/init.py", line 71, in discover
response = await DISCOVERY_COMMAND.execute(host, timeout, retries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/goodwe/protocol.py", line 154, in execute
await response_future
File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 1228, in _read_ready
data, addr = self._sock.recvfrom(self.max_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionRefusedError: [Errno 111] Connection refused

Also have the same problem, newly installed 15kw hybrid inverter.
Looks like the inverter doesn't respond to UDP 8899 at all(tested with nmap), updated to latest DSP and ARM firmware(according to SolarGo)
DSP: 07.7068
ARM: 08.401

How do you then connect SolarGo ?
Via bluetooth?
If not, can you sniff network traffic ?

I'm using Bluetooth to connect SolarGo.
I've never been able to see the Wifi the inverter is supposed to broadcast..havent found any setting where you can turn on/off this..maybe it's automatically disabled when the LAN module is installed?

Hmm, this seems to be the same issue as https://github.com/home-assistant/core/issues/114173
It looks like the new WIFI/LAN 2.0 adapters switched from UDP to (more standard) modbus/TCP.
And that one is not supported at the moment.
As mentioned in that other ticket, it should be doable, but will be a pain without real device to test with ...

FYI..
I experimented with the modbus connection to my ET15, as far as I can tell it's read only but at least now I can get the statistics/values I need into Homeassistant, PV generation, battery and so on.
Also seem to work fine adding the daily production to the Energy dashboard.
I've added everything to my Github, maybe it could help someone else.(not used to Github but I think it should work :) )
https://github.com/micros01/GoodweHAmodbus

OK, found this about the modbus registers: https://forum.iobroker.net/assets/uploads/files/1666333836822-goodwe_modbus_protocol_hybrid_et_eh_bh_bt__arm205-hv__v1.7-_-read-only_20200226-1.pdf and it states "Ready only"....which I interpreted to read only for the modbus but guess I was wrong.
Then it should be possible to control charging and discharging the battery based on electrical price from the grid(which I get from another sensor in HA)?
If you can hint on how to do that it would be awesome...

Use this instead https://forum.iobroker.net/assets/uploads/files/1703579717824-arm.745.esg2.et30.modbus.protocol.map.20221231..v1.pdf?fbclid=IwZXh0bgNhZW0BMQABHXcRLb0ZpNyMluMOU1YbESC-qq7E6tqPzmVHqIo23sWL3AheL7OYA5a_fg_aem_AeJDg4Y2IXxYAnOrXnNez3tsuNQNVzB0cyxXopC87s3VG7X8rW4mRa4WvMJevb3suLo

Proper support for Modbus/TCP was added in v<IP_ADDRESS>.
When adding the integration, one can select either UDP (8899) or TCP (502).
Enjoy.


feat: :art: improve logo design

Filled the arrow in the logo with white color so that it is better displayed in the marketplace
Also renamed res folder with logos to public

Filled the arrow in the logo with white color so that it is better displayed in the marketplace

Good job, thinks :smile:

Also renamed res folder with logos to public

It is not necessary to rename res folder, the related options should be also updated if so https://github.com/yunsii/vscode-links/blob/dabcbbcc89777208d005900e881429bb3b0efd69/package.json#L27-L31
In the end, just filled the arrow is good enough.

Filled the arrow in the logo with white color so that it is better displayed in the marketplace

Good job, thinks 😄

Also renamed res folder with logos to public

It is not necessary to rename res folder, the related options should be also updated if so
https://github.com/yunsii/vscode-links/blob/dabcbbcc89777208d005900e881429bb3b0efd69/package.json#L27-L31
In the end, just filled the arrow is good enough.

Fixed


Slide: SageWorks Execution Role

Let's make a slide about the SageWorks Execution Role and how it's used

Role is granted 3 main permissions/policies

AWSGlueServiceRole (SageWorks can run Glue Jobs)
AmazonSageMakerFullAccess (SageWorks can read/write to Sagemaker services/artifacts)
SageWorksExecutionRoleDefaultPolicy5C2E224A (limited S3 Access to SageWorks Artifacts and Athena Query results)

Trust Relationship

The client controls what teams (or individuals) have 'AssumeRole` privileges

This is covered by other slide decks


fix: shellcheck's SC2086 issue: Double quote to prevent globbing and word splitting

Fixes SC2086: (info): Double quote to prevent globbing and word splitting.
Here is a summary of the other issues shellcheck reports, per
shellcheck deb-get | grep -o -P 'SC\d\d\d\d' | sort | uniq -c | sort -nr:
    183 SC2292
     66 SC2312
     12 SC2155
     12 SC2076
      8 SC2250
      8 SC2249
      7 SC2207
      4 SC2206
      4 SC2034
      3 SC2178
      3 SC2128
      2 SC2164
      2 SC2046
      2 SC2044
      2 SC1090
      1 SC2154
      1 SC2053
      1 SC2016

Let me know if you are open to my submitting PRs for any or all of these issues. I will submit granular PRs, unless advised otherwise.
Depending on the issue, the PR may be as simple as placing shellcheck disable=SCnnnn at the top of the file.

This is looking a lot better since some recent merges and re-factoring happened. See also #871 and #886 for related efforts.  Currently we are here :
shellcheck deb-get | grep -o -P 'SC\d\d\d\d' | sort | uniq -c | sort -nr
     23 SC2076
     22 SC2086
      8 SC2155
      5 SC2034
      3 SC2064
      2 SC2164
      2 SC1090
      1 SC2206
      1 SC2016

Some pending PRs will also cause/receive conflicts from this one even if re-based now but such is life ...
Your PRs will be very welcome. (Especially since I can possibly then merge your PRs and deal with any conflicts in mine!). I'd suggest SC2155 and SC2076 are candidates for ignoring generally (or granular PRs with discussion)
Some warnings concern "intended" behaviours.
Globally disabling checks is problematic, so I'd suggest generally command/block-level exclusions are always to be preferred.


Missing force_str

Using the plugin BlogFeaturedPostsPluginCached raises an exception:
TypeError: __str__ returned non-string (type __proxy__)
Because this method does not call force_str:
https://github.com/nephila/djangocms-blog/blob/d95802c1475fc7e06ba121a178bb0e755be9bba9/djangocms_blog/models.py#L608

@adrien-delhorme Thanks for reporting this: will try to fix and release a new version today

@adrien-delhorme We've just released 2.0.6 version: could you please check if now it's working as expected?

It's working. Thanks!


Incorrect HMR behavior with webpack 5 and enabled source maps

Version
3.0.5
Reproduction link
https://github.com/yura3d/vue3-webpack5
Steps to reproduce

Clone GitHub repo.
Run yarn and yarn dev.
Browse localhost:8080, type some symbols to input field and/or press counter button to change component's state.
Open src/App.vue and change something in <template> section.

What is expected?
HMR applies, input field and counter still contain their values as state shouldn't be changed according to docs: https://vue-loader.vuejs.org/guide/hot-reload.html#state-preservation-rules
What is actually happening?
HMR applies.
State is lost when bundling with webpack 5 if devtool is set to something with source maps (source-map, eval-source-map, etc).
State is preserved when devtool doesn't set source maps (false, eval), or when bundling with webpack 4 (even with source maps) and almost the same webpack.config.js.

I see this console output on HMR with webpack 5 and source maps:
log.js:24 [HMR] Updated modules:
log.js:16 [HMR]  - ./src/App.vue?vue&type=script&lang=js
log.js:24 [HMR]  - ./src/App.vue?vue&type=script&lang=js
log.js:24 [HMR]  - ./src/App.vue
log.js:16 [HMR]  - ./src/App.vue?vue&type=template&id=7ba5bd90&scoped=true
log.js:24 [HMR]  - ./src/App.vue?vue&type=template&id=7ba5bd90&scoped=true
log.js:16 [HMR]  - ./src/App.vue?vue&type=style&index=0&id=7ba5bd90&lang=scss&scoped=true
log.js:24 [HMR] App is up to date.
hotModuleReplacement.js:215 [HMR] Reload all css

And this on HMR without source maps:
log.js:24 [HMR] Updated modules:
log.js:16 [HMR]  - ./src/App.vue?vue&type=template&id=7ba5bd90&scoped=true
log.js:24 [HMR]  - ./src/App.vue?vue&type=template&id=7ba5bd90&scoped=true
log.js:16 [HMR]  - ./src/App.vue?vue&type=style&index=0&id=7ba5bd90&lang=scss&scoped=true
log.js:24 [HMR] App is up to date.
hotModuleReplacement.js:215 [HMR] Reload all css

It seems component's JS recompiles even when there are no changes in <script> section if source maps are enabled.

The bug is also reproducible in apps created by Vue CLI 5.

A quick google search show Vue CLI is using webpack 5.  I am wondering if any other frameworks are having this problem.  If we can answer that question it will be clear if this is a webpack thing or a vue thing.

I have the same problem.


Discord Support

Today Discord announced their collaboration with Logitech: https://twitter.com/discordapp/status/876937721616375808 I thought it would be cool to have it in Aurora since there is already support for Skype, although it's slowly getting replaced by teamspeak and discord etc.

Duplicate of #297

#297 was closed because it didn't seemed to be possible, but now with https://twitter.com/LogitechG/status/876934012161241089 shouldn't it work with the Logitech wrapper?

@simon-wh The applet path is C:\Program Files\Logitech Gaming Software\ArxApplets


[Question] How to get requested fields inside resolver function?

What did you do?
I want to get only the required fields from the database, based on the requested fields.
What did you expect to happen?
I want an array of the requested fields to transform them into database columns.

@crisu83

I figured it out myself :) All necessary information is available in ResolveInfo $info argument.
$fields = array_reduce(
    $info->getFieldNodes()[0]->getSelectionSet()->getSelections(),
    function (array $carry, SelectionNodeInterface $selectionNode) {
        if ($selectionNode instanceof FieldNode) {
            $carry[] = $selectionNode->getNameValue();
        }
        return $carry;
    },
    []
);

var_export($fields);


virsh.sh with multiple node

virsh.sh doesn't work when there are several nodes with virsh labels.
How to reproduce?

Create several nodes with labels...

kubectl get nodes -l extraRuntime
NAME      STATUS    AGE       VERSION
cmp01     Ready     5h        v1.6.2-2+5270697e4e2c6a
cmp02     Ready     5h        v1.6.2-2+5270697e4e2c6a

Run virsh.sh

Closed by #387


Using WideType Data Container to Increase Memory Bit-Width breaks Multi-CU support

Problem
I am adding complex number support to PyTorch on the FPGA and I rely heavily on the ability to pass multiple samples to the FPGA at the same time using your WideType Single-Instruction, Multiple Data (SIMD) container. For a 512-bit memory bus, I would like to stream the following using my SIMD Vec<T, LOG2BW> Container:
real float:  [R0, R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12, R13, R14, R15]
complex float[2]:  [R0, I0, R1, I1, R2, I2, R3, I3, R4, I4, R5, I5, R6, I6, R7, I7]
quaternion float[4]:  [R0, I0, J0, K0, R1, I1, J1, K1, R2, I2, J2, K2, R3, I3, J3, K3]
Background
The complex/quaternion support is needed for RF/Optical communications. I have seen a wide variety of complex number support in the HLS documentations
|      Type                       |     Synthesis             |          Simulation        |
--------------------------------------------------------------------------------------------
|      std::complex <ap_fixed <>> |             YES           |              YES           |
|      std::complex<float>        |              NO           |              YES           |
|      std::complex<double>       |              NO           |              YES           |
|      complex_wrapper<float>     |              NO           |              YES           |
|      complex_wrapper<double>    |              YES          |              YES           |

The first three support the R, I, R, I memory format, while the last 2 support R, R, I, I memory format.
Proposed Solution
Now that Xilinx Vitis allows us to use Vectorized SIMD container datatypes (see WideType), there should be a built in way to treat std::complex data as a float[2] and std::quaternion as float[4]. If this idea works, the PL code wouldn't need to support std::complex<T> containers (which it currently doesn't).
I have written a blog about this:
pytorch-for-fpga-part-1-heterogeneous-processing
pytorch-for-fpga-part-2-basic-fpga-optimizations
pytorch-for-fpga-part-3-advanced-fpga-optimizations
pytorch-for-fpga-part-4-deploying-pytorch-kernels
pytorch-for-fpga-part-5-complex-and-quaternion-numbers(Coming Soon)
In Part-4 of this blog, SIMD has weird profile performance and it doesn't compile at all when the number of CUs is greater than 1. Here are some questions:

Does the hw_emu of WideType predict the kernel duration time accurately (see Detail)? Can you provide an Example?
Does the WideType data container work with Multiple Compute Units (CU)? Can you provide an Example?
Do you see any long-term problems with supporting complex numbers in this way? Are there potential Bottlenecks? I'm still pretty new to this.

Detail
Here are some logs. Vec<float, 0> (2^0 = 1) achieves the same hw_emu speed as float, which is expected. The Iteration intervals make sense. I'm not sure how to calculate the depth, but the duration of these math kernels seems to go up and down arbitrarily.
float duration 0.063ms
INFO: [v++ 204-61] Pipelining loop 'read'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 3.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 14.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 3.
INFO: [v++ 200-789] **** Estimated Fmax: 411.02 MHz

Vec<float, 0>  duration 0.062ms
INFO: [v++ 204-61] Option 'relax_ii_for_timing' is enabled, will increase II to preserve clock frequency constraints.
INFO: [v++ 204-61] Pipelining loop 'read'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 3.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 14.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 1, Depth = 3.
INFO: [v++ 200-789] **** Estimated Fmax: 411.02 MHz

Vec<float, 1> duration 0.204ms
INFO: [v++ 204-61] Pipelining loop 'read'.
INFO: [v++ 204-61] Pipelining result : Target II = 2, Final II = 2, Depth = 11.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 2, Final II = 2, Depth = 15.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 2, Final II = 2, Depth = 9

Vec<float, 2> duration 0.178ms
INFO: [v++ 204-61] Pipelining result : Target II = 4, Final II = 4, Depth = 13.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 4, Final II = 4, Depth = 17.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 4, Final II = 4, Depth = 11.
INFO: [v++ 200-789] **** Estimated Fmax: 411.02 MHz

Vec<float, 3> duration 0.205ms
INFO: [v++ 204-61] Pipelining result : Target II = 8, Final II = 8, Depth = 17.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 8, Final II = 8, Depth = 21.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 8, Final II = 8, Depth = 15.
INFO: [v++ 200-789] **** Estimated Fmax: 411.02 MHz

Vec<float, 4> duration 0.144ms
INFO: [v++ 204-61] Pipelining loop 'read'.
INFO: [v++ 204-61] Pipelining result : Target II = 16, Final II = 16, Depth = 25.
INFO: [v++ 204-61] Pipelining loop 'vadd'.
INFO: [v++ 204-61] Pipelining result : Target II = 16, Final II = 16, Depth = 29.
INFO: [v++ 204-61] Pipelining loop 'write'.
INFO: [v++ 204-61] Pipelining result : Target II = 16, Final II = 16, Depth = 23.

@dylanbespalko Great articles!
When you increase the bus width, usually you increase the number of arithmetic units accordingly. Seeing the log, Vec<float, 4> achieves II=16. This means the IO bandwidth becomes 16x, but the throughput is the same as the Vec<float, 0> case. II should be always 1 for all the data types.
I suggest modifying your code as follows.
vec_base.h
  Vec<T, LOG2BW> operator+(const Vec<T, LOG2BW>& other) const {
#pragma HLS inline // anjn: 'self' is not necessary
    using VH = VecHost<T, LOG2BW>;
    Vec<T, LOG2BW> ret;
    for (size_t i = 0; i < size(); i++) {
#pragma HLS UNROLL // anjn: Duplicate arthmetic unit
      ret[i] = VH::values[i] + other[i];
    }
    return ret;
  }

vadd.hpp
template <typename DTYPE>
void vadd_dataflow(hls::stream<DTYPE>& result_stream,
               hls::stream<DTYPE>& a_stream,
               hls::stream<DTYPE>& b_stream,
               DTYPE alpha,
               size_t stride) {
  DTYPE a, b;
  vadd: for (size_t j = 0; j < stride*DTYPE::rn()/DTYPE::size(); j++){
#pragma HLS PIPELINE II=1 // anjn: II should be 1
    a_stream >> a;
    b_stream >> b;
    result_stream << a + b;
  }
}

Dataflow.h is also needed to be modified so that the read/write functions also achieve II=1.
For the measured duration,
float duration 0.063ms
Vec<float, 0>  duration 0.062ms
Vec<float, 1> duration 0.204ms
Vec<float, 2> duration 0.178ms
Vec<float, 3> duration 0.205ms
Vec<float, 4> duration 0.144ms

I'm not sure how you measured the duration, I guess this includes various overhead other than kernel execution time, for example, data movement, kernel invocation, interrupt to host. The simplest way to hide these overheads is by using larger data.

@anjn,
Thanks for your quick reply and I think I get what you mean. I am gone until Thursday. I will try to fix it and close the issue then.
Sent from my iPhone

On Feb 25, 2020, at 9:19 PM, anjn<EMAIL_ADDRESS>wrote:

@dylanbespalko Great articles!
When you increase the bus width, usually you increase the number of arithmetic units accordingly. Seeing the log, Vec<float, 4> achieves II=16. This means the IO bandwidth becomes 16x, but the throughput is the same as the Vec<float, 0> case. II should be always 1 for all the data types.
I suggest modifying your code as follows.
vec_base.h
Vec<T, LOG2BW> operator+(const Vec<T, LOG2BW>& other) const {
#pragma HLS inline // anjn: 'self' is not necessary
using VH = VecHost<T, LOG2BW>;
Vec<T, LOG2BW> ret;
for (size_t i = 0; i < size(); i++) {
#pragma HLS UNROLL // anjn: Duplicate arthmetic unit
ret[i] = VH::values[i] + other[i];
}
return ret;
}
vadd.hpp
template
void vadd_dataflow(hls::stream& result_stream,
hls::stream& a_stream,
hls::stream& b_stream,
DTYPE alpha,
size_t stride) {
DTYPE a, b;
vadd: for (size_t j = 0; j < stride*DTYPE::rn()/DTYPE::size(); j++){
#pragma HLS PIPELINE II=1 // anjn: II should be 1
a_stream >> a;
b_stream >> b;
result_stream << a + b;
}
}
Dataflow.h is also needed to be modified so that the read/write functions also achieve II=1.
For the measured duration,
float duration 0.063ms
Vec<float, 0>  duration 0.062ms
Vec<float, 1> duration 0.204ms
Vec<float, 2> duration 0.178ms
Vec<float, 3> duration 0.205ms
Vec<float, 4> duration 0.144ms
I'm not sure how you measured the duration, I guess this includes various overhead other than kernel execution time, for example, data movement, kernel invocation, interrupt to host. The simplest way to hide these overheads is by using larger data.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@anjn,
Thank you so much for your help. Your suggestions reduced the II of vadd_dataflow() to 1.  I have also been trying to fix the read/write functions in Dataflow.h with no success.
I have attached a simplified example WideTypeMemoryRead.txt (.cpp)  where most of the code is copied directly from the Vitis Libraries.
Line6: class BitConv is from blas/L1/include/hw/xf_blas/helpers/utils/types.hpp
Line37: class WideType is from blas/L1/include/hw/xf_blas/helpers/utils/types.hpp
Line103: void readVec2Stream is from blas/L1/include/hw/xf_blas/helpers/dataMover/vecMoverB1.hpp
The only original code in this file is:
extern "C" {

void vfadd(float a[2]) {
#pragma HLS INTERFACE s_axilite port=a  bundle=control
#pragma HLS INTERFACE s_axilite port=return bundle=control

#pragma HLS INTERFACE m_axi port=a  offset=slave bundle=gmem2

hls::stream<WideType<float, 2> > l_strX;
#pragma HLS DATAFLOW
#pragma HLS data_pack variable=a
#pragma HLS data_pack variable = l_strX
//#pragma HLS ARRAY_PARTITION variable=a dim=0 complete
//#pragma HLS ARRAY_PARTITION variable=l_strX dim=0 complete
//#pragma HLS ARRAY_PARTITION variable=a cyclic factor=2 partition
//#pragma HLS ARRAY_PARTITION variable=l_strX cyclic factor=2 partition
  readVec2Stream<float, 2>(a, 2, l_strX);
}

}

I have tried every pragma I can think of to widen the data path, but I keep getting the following compiler response:
===>The following messages were generated while  performing high-level synthesis for kernel: vfadd Log file: /home/jenkins/repos/vitis_kernels/dev/BinaryOpsKernel/v_add/build/kernel/_x/vfadd/vfadd/vivado_hls.log :
INFO: [v++ 204-61] Option 'relax_ii_for_timing' is enabled, will increase II to preserve clock frequency constraints.
INFO: [v++ 204-61] Pipelining function 'readVec2Stream<float, 2u>'.
WARNING: [v++ 204-69] Unable to schedule bus request on port 'p_in_0' (/home/jenkins/repos/vitis_kernels/dev/BinaryOpsKernel/v_add/kernel/vec/vfadd/vfadd.cpp:115) due to limited memory ports. Please consider using a memory core with more ports or partitioning the array.
INFO: [v++ 204-61] Pipelining result : Target II = 1, Final II = 2, Depth = 9.
INFO: [v++ 200-789] **** Estimated Fmax: 291.29 MHz
INFO: [v++ 60-594] Finished kernel compilation
INFO: [v++ 60-244] Generating system estimate report...
INFO: [v++ 60-1092] Generated system estimate report: /home/jenkins/repos/vitis_kernels/dev/BinaryOpsKernel/v_add/build/vfadd/system_estimate_vfadd.xtxt
Add Instance readVec2Stream_float_2u_s readVec2Stream_float_2u_U0 66
INFO: [v++ 60-586] Created /home/jenkins/repos/vitis_kernels/dev/BinaryOpsKernel/v_add/build/kernel/vfadd/vfadd.xo
INFO: [v++ 60-791] Total elapsed time: 0h 0m 35s

The warning is telling me to partition the array, but nothing I have tried seems to achieve II=1. How to I partition the array so that the readVec2Stream achieves an II=1?
WideTypeMemoryRead.txt

@dylanbespalko,
You are trying to fill WideType<float,2> with a 32-bit interface, so it takes 2 cycles. Your code should look like this.
extern "C" {

void vfadd(ap_uint<64>* a, int length) {
#pragma HLS INTERFACE s_axilite port=a  bundle=control
#pragma HLS INTERFACE s_axilite port=length  bundle=control
#pragma HLS INTERFACE s_axilite port=return bundle=control
#pragma HLS INTERFACE m_axi port=a  offset=slave bundle=gmem2

#pragma HLS DATAFLOW

  hls::stream<WideType<float, 2> > l_strX;
#pragma HLS data_pack variable = l_strX

  //readVec2Stream<float, 2>(a, 2, l_strX);
  for (int i=0; i<length; i++) {
#pragma HLS PIPELINE II=1
    WideType<float, 2> f_vec;
    ap_uint<64> a_dat = a[i];
    for (int j=0; j<2; j++) {
      BitConv<float> conv_f;
      f_vec[j] = conv_f.toType(a_dat(32*j+31,32*j));
    }
    l_strX << f_vec;
  }
}

}

@anjn,
Sorry for the delayed response,
This works great!  I'm getting awesome speeds, but I've also tried to make both of the following optimization procedures work using the same code:

SIMD using parallel data path inside a single CU.
Multi-CU

I am finding that the same source code can do both individually:

SIMD works when NUM_CU = 1
Multi-CU works when SIMD ap_int<32> is used for floats.

When I turn both on at the same time, I get the following error in sw_emu
Found Platform
Platform Name: Xilinx
INFO: Reading /home/jenkins/repos/vitis_kernels/aten/src/ATen/fpga/torch.xclbin
Loading: '/home/jenkins/repos/vitis_kernels/aten/src/ATen/fpga/torch.xclbin'
8192
XRT build version: 2.3.1301
Build hash: 192e706aea53163a04c574f9b3fe9ed76b6ca471
Build date: 2019-10-24 20:05:16
Git branch: 2019.2
PID: 24004
UID: 1001
[Sat Mar 14 18:10:23 2020]
HOST: dylan-bespalko-ubuntu
EXE: /home/jenkins/repos/vitis_kernels/dev/BinaryOpsKernel/v_add/build/test
[XRT] WARNING: unaligned host pointer '0x55ba751f5840' detected, this leads to extra memcpy
[XRT] WARNING: unaligned host pointer '0x55ba751fd8c0' detected, this leads to extra memcpy
[XRT] WARNING: unaligned host pointer '0x55ba751e4520' detected, this leads to extra memcpy
corrupted double-linked list
Software emulation of compute unit(s) exited unexpectedly
Terminated

Can you tell me if these features are mutually exclusive? They are somewhat redundant.

Hi @dylanbespalko,
They are not mutually exclusive. Parallel operations in a kernel and multiple instantiations of that kernel make sense.


Would an updated prisma migration be possible?

I'm piddling around with getting a self-hosted / local instance of ShareGPT running - I see there are already a few possibly-similar requests.
I've made a decent amount of progress getting a docker-compose-based environment set up but ran into a semi-blocker in that the initial available migration.sql file doesn't match the current Prisma schema. I could manually parse schema.prisma and guess at some of the structure to get something working, but i was wondering if you'd be willing to add a migration file that gets a new install up to the current schema - on the surface it looks like there are new Save and Comment tables and some field changes..
Further, not sure if you're receptive to such a PR (adding a containerized environment), but happy to contribute it when working if so.

Please to disregard this - it's the result of my unfamiliarity with Prisma. I have an updated database and a "mostly working" docker-compose environment and can proceed from there!


Don't call Stop on daemon if projections are disabled

I have seen the daemon trying to stop when using Marten as document store, using 4.0.0-rc.2
This PR covers a first attempt to try to stop the error message (I know this is not a real issue since is just the log so the code will continue just fine until the end of my app)
I added the debug just for the sake of testing, but open to change this.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: dariogriffo sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@dariogriffo Sorry it took so long to get at this. I'm running it through now.

This error casues my app to crash :/
`fail: Marten.Events.Daemon.AsyncProjectionHostedService[0]
Error when trying to stop the asynchronous projection agent
System.NullReferenceException: Object reference not set to an instance of an object.
at Marten.Events.Daemon.AsyncProjectionHostedService.StopAsync(CancellationToken cancellationToken) in Marten.dll:token 0x600101f+0x1d
Process finished with exit code 0.
`

This error causes my app to crash :/

fail: Marten.Events.Daemon.AsyncProjectionHostedService[0]
Error when trying to stop the asynchronous projection agent
System.NullReferenceException: Object reference not set to an instance of an object.
at Marten.Events.Daemon.AsyncProjectionHostedService.StopAsync(CancellationToken cancellationToken) in Marten.dll:token 0x600101f+0x1d
Process finished with exit code 0.

I don't think the app crashes, there is a try catch and an error message. But th the execution of the flow should be normal.

@dariogriffo Sorry it took so long to get at this. I'm running it through now.

@jeremydmiller any update on this?

This error causes my app to crash :/

fail: Marten.Events.Daemon.AsyncProjectionHostedService[0]
Error when trying to stop the asynchronous projection agent
System.NullReferenceException: Object reference not set to an instance of an object.
at Marten.Events.Daemon.AsyncProjectionHostedService.StopAsync(CancellationToken cancellationToken) in Marten.dll:token 0x600101f+0x1d
Process finished with exit code 0.

I don't think the app crashes, there is a try catch and an error message. But th the execution of the flow should be normal.

OK, I'll look into it more closely, cheers.

This error causes my app to crash :/

fail: Marten.Events.Daemon.AsyncProjectionHostedService[0]
Error when trying to stop the asynchronous projection agent
System.NullReferenceException: Object reference not set to an instance of an object.
at Marten.Events.Daemon.AsyncProjectionHostedService.StopAsync(CancellationToken cancellationToken) in Marten.dll:token 0x600101f+0x1d
Process finished with exit code 0.

I don't think the app crashes, there is a try catch and an error message. But th the execution of the flow should be normal.

OK, I'll look into it more closely, cheers.

It looks like I mistook correalation for causation...i.e. when my app crashs this get's logged....oops.

@dariogriffo So yeah, this isn't causing your app to crash, but it's pretty bad behavior and thank you for the PR. This will be in the next incremental release, but that'll probably be one more RC.


Make index name configurable in config.json

DevOps likes this :)

Thank you! Dev would like if we could have a Reindexierung ;)


Identifier quoting is incorrect

Identifier quoting is incorrect and fails sometimes. Steps to reproduce:

run psql
do the following commands in the psql

postgres=# create table "bad ""table" (id int);
CREATE TABLE
postgres=# insert into "bad ""table" (id) values (1);
INSERT 0 1

run pgweb
try to view data in the "bad ""table"
see the error message instead of a data

This discussion seem to be relevant: https://github.com/golang/go/issues/18478


Add ValueArray bindings

Can't be autogenerated because all the functions return the array itself, and because it's from_glib_none each function would create a copy of the whole array.
Also manually added the Deref impls, very useful

Looks good, thanks

Thanks!


AvgPool2d different from the original

@xternalz
I think your avgpool https://github.com/xternalz/WideResNet-pytorch/blob/master/wideresnet.py#L81
and the original avgpool https://github.com/szagoruyko/wide-residual-networks/blob/master/pytorch/resnet.py#L56 use different strides (yours=kernelsize=8, original=1).

This should be closed as both pooling are the same now.

This should be closed as both pooling are the same now.

@xternalz this issue should be closed now, right ?

This is not an issue. Both implementations have block_width=8, stride=1, padding=0. This is because the defaults for stride and padding are respectively 1 and 0.


Better uninstall dialog for extensions with dependencies. Fix #45197

Self explaining button labels on dialog for uninstalling extensions with dependencies

@camilogiraldo @sandy081 I would use capital casing for button labels


Format With option is missing in VSCode

VSCode used a command called 'Format With', which allowed us to choose the formatting extension. But recently, I noticed that the command is gone. How can I get it back? If removed, please add it back. Thanks.

Found that there should be at least two formatters for a language for this option to appear.


Merge pull request #103 from MicrosoftDocs/main

File upload script PWA

Docs Build status updates of commit 5c5b17b:
:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Fix devfile registry entry name and ingress url

Please specify the area for this PR
What does does this PR do / why we need it:
Fixes the devfile registry entry name for registry viewer component of the helm chart. Fixed name 'Community' changed to value of {{ template "devfileregistry.name" . }} (the deployment name).
Additionally, the ingress url is now define and built under _template.tpl: https://github.com/michael-valdron/devfile-registry-support/blob/1df84bc08021587d954a3f6749699d77b40aaf28/deploy/chart/devfile-registry/templates/_template.tpl#L20-L27
Currently, this is only built from the ingress domain for k8s and devfile/api#1467 needs resolving to either separate this field or not rely on manually setting the domain for OpenShift environments.
Which issue(s) this PR fixes:
Fixes #?
fixes devfile/api#1433
PR acceptance criteria:

[X] Test Coverage

Are your changes sufficiently tested, and are any applicable test cases added or updated to cover your changes?

Documentation (WIP)

[ ] Does the REST API doc need to be updated with your changes?
[ ] Does the registry library doc need to be updated with your changes?

How to test changes / Special notes to the reviewer:

Documentation (WIP)
Does the REST API doc need to be updated with your changes?
Does the registry library doc need to be updated with your changes?

No changes needed.


Create Matrix filter

The filter for matrix type of data.

Unclear how this filter should look like. I think it only makes sense if the user aggregates a matrix to a vector.

Will not be implemented for now.


v1.0 examples Bash

Review Document
Merlin POC
XL/Large Impact:

[ ] Missing preprocessing script for Alibaba dataset
[ ] Feast materlize takes 25min
[ ] Where is the code to pre-process the dataset?
[ ] The pre-processed dataset appears to be 1.6GB
[ ] Without an automated way to get the input data the example has limited utility

[ ] https://github.com/NVIDIA-Merlin/models/pull/309
This PR allows the other issues on top to be resolved

[ ] #181
[ ] Why do we need to install tensorflow in the merlin-tensorflow-inference container?
[ ] 01 notebook: “cd /Merlin/examples/PoC/” there is no directoy

Medium/Small Impact

[ ] Notebook should install any packages that aren’t included in the container (aside from jupyterlab)
[ ] Running jupyterlab from the example notebook path (which the example seems to expect) breaks the images
[ ] README: “Once in the server, navigate to the /Merlin/examples directory and execute the example notebooks.”
[ ] 01 notebook: “In additon to NVIDIA Merlin libraries and Triton library, we are using two external libraries in these series of examples:
[ ] 02 notebook: headings “Feast Apply“ and “Feast Materialize ” are not understandable
[ ] Explain Operator, when they are used and not all upfront
[ ] from merlin.systems.dag.ensemble import Ensemble. gives the following warnings:
[ ] Add the link to the preprocessing script of Ali-CCP dataset in the description of the `
[ ] Feature Engineering with NVTabular` section.
[ ] Add a readme or notebook explaining how to run the preprocessing script of ALI-CPP and prepare train/test files for examples.
[ ] Explain why we are settingthe env variable TF_GPU_ALLOCATOR
[ ] Can we avoid having the same NVT workflow defined in notebooks 04/05 ?
[ ] Avoid TF/cudf/pandas warnings

Merlin Systems
XL/Large Impact:

[ ] Add the link to the preprocessing script of Ali-CCP dataset in the description of the  Feature Engineering with NVTabular section.
[ ] Add a readme or notebook explaining how to run the preprocessing script of ALI-CPP and prepare train/test files for examples.

[ ] https://github.com/NVIDIA-Merlin/models/pull/309
This PR allows the other issues on top to be resolved

Medium/Small impact:

[ ] Explain why we are settingthe env variable TF_GPU_ALLOCATOR

This is an old ticket. Bash was completed.


Add extra info

I've added the ability to pull out extra information from the RAW files that is presented by LibRAW

Thanks for your contribution! I realize this is your first one, so let's walk through it carefully.
One design decision of rawpy was to provide the minimum possible interface to open RAW images, get access to the unprocessed image (the Bayer matrix data), and provide basic functions to postprocess RAW images (although anyone can just take the RAW data and write custom postprocessing algorithms directly in Python). And all that by abstracting as much as possible over the low-level libraw data structures to create a nice Python interface that is easy to understand.
libraw itself provides other functions as well which I think don't belong there, and these are around all the EXIF metadata. Since reading EXIF data from image files is independent of whether the file is a RAW image or not, this should not go into rawpy. There are already many EXIF libraries for Python, so if there is any data not exposed via those libraries, then they should be extended instead. Note that some libraries like ExifRead have a description of "Read Exif metadata from tiff and jpeg files" but in fact they just depend on the EXIF format itself and can also read RAW camera images.
Now, removing all the EXIF data from your PR that leaves us with the .color property (rawpy.ImageColor (cam_mul, pre_mul, cblack, black, cmatrix, cam_xyz, profile)) and the .idata property (rawpy.ImageIData (make, model, raw_count, dng_version, is_foveon, colors, filters, xtrans, cdesc)).
Let's look at each data item separately:
ImageColor (.color)
cam_mul: This is in fact already available via the .camera_whitebalance property.
pre_mul: Same here, via the .daylight_whitebalance property.
cblack and black: This is combined as .black_level_per_channel. black is added to each cblack value and a list is returned. If there is a need to have cblack and black separately, let me know.
cmatrix: Already exposed via .color_matrix.
cam_xyz: Already exposed via .rgb_xyz_matrix
profile: Not available yet.
ImageIData  (.idata)
make and model: Not available yet, but sounds like EXIF metadata.
raw_count: Not available yet. If this is the number of RAW pixels, then this is equivalent to raw.raw_image.size
dng_version: Not available yet, but sounds like EXIF metadata.
is_foveon: Already exposed as .raw_type.
colors: Already exposed as .num_colors.
filters: This describes the Bayer matrix pattern as a single integer. Already exposed in a more useful way as .raw_pattern.
xtrans: Not available yet.
cdesc: Already exposed as .color_desc.
That leaves us with profile and xtrans, assuming my understanding above is correct.
In the libraw API docs it says:

char xtrans[6][6];
char xtrans_abs[6][6];
These matrices contains Fuji X-Trans row/col to color mapping. First one is relative to visible area, while second is positioned relative to sensor edges.

I have to admit I have no idea what this is good for. Do you have a use case for that?

void *profile;
Pointer to the retrieved ICC profile (if it is present in the RAW file).

It turns out that ICC profiles are also part of EXIF, so there is no need to make it available here.
In general (for any open source project) I would advise before doing such undertakings always to open an issue first to see whether things make sense and if there is an actual need.
If you need the xtrans information, feel free to do a bit more research on that and check whether this is part of EXIF or not. If it's not, and if it's helpful in RAW image processing (ideally with a concrete example where it's used) then this would be a good candidate for a PR.

My initial aim for the PR was to include lens data in the library, however I got carried away and added the other structs as well. Having read your comments I've had a look at some Exif readers. It appears that the only ones I can get to work with my files are wrappers around the exiftool binary; but this is ok for my needs.
Thanks for taking the time to go through the PR.

Yes, the current state of EXIF libraries is not the best. In the past I used pyexiftool which also wraps exiftool. I used that to get the maker and model of the camera and lens. Then I used that to correct lens distortion, for which I wrote another library called lensfunpy (which wraps the lensfun C library). The interesting bit is that the lenses database of lensfun follows the naming convention of exiftool in terms of lens name etc. The lens name is not part of the EXIF metadata, only some numeric lens ID, and this is not standardised. All these weird things come from the fact that the lens data is stored in some EXIF extension (called makernotes) which is a proprietary thing of the manufacturers and exiftool is the best tool to decode all of that custom data, I assume by many years of reverse engineering. And so now everyone copied exiftool's naming convention for lens names etc.
Anyway, should we close this PR?

Thanks for the additional information. Yes I am too using that wrapper to call into your lensfunpy library. Thanks for the pointers. Happy to close the PR
Sent from my OnePlus ONEPLUS A3003 using FastHub


Clarification about multi-bin classification loss?

Hi,
Thanks for providing the code. I have a question about the multi-bin classification loss. In the Mono3D paper, it is mentioned that the cross-entropy loss is used for the multi-bin classification of width, height, and length.
However, looking at the code in detection_3d_head.py, I find that it is being used only for angle alpha. Am I understanding the code wrong? Can you please clarify that?
Thank you

When I try to open-source the code, I made it easier to adapt to other datasets with more classes, so the multi-bin classification for rotation is skipped.


remote shell output should distinguish stdout and stderr

Current remote shell run will print both stderr and stdout to stdout.

hard to support, this will need open 2 pipes


It's too slow

In my app, I record the memory usage, but it too slow.
One operation may cost ten minutes.
If run for a while, the app may killed because the memory:Message from debugger: Terminated due to Memory Error
I Test this in iPhone 6 & iphone 5c.  The project is large and many objects are alloced when running.
somebody has the same problem? how should I resolve this. Thanks!

Did you try turning off the backtrace record?
[HINSPDebug recordBacktraces:NO];

Should be better now with 0.2.0 https://github.com/tapwork/HeapInspector-for-iOS/releases/tag/0.2.0

:+1:  Yeah, I have used this version, It's better. thank you

So I can close this issue for now


check is esri-loader script has been loaded

Currently the isLoaded() function is only checking if the require function existed.  This is causing an issue when using the esri-loader module in Electron because this ends up getting evaluated on a process running in NodeJS and therefor the API for require is different.  By adding a check for if the esri-loader script has been loaded if pushes this evaluation into the browser and Electron no longer throws an error.

If this seems like something we are going to go forward with I need to update the unit tests to include the new criteria for the isLoaded() function.

Thanks @kgs916
Ultimately I think  I'm OK w/ this, but I was hoping to avoid the getScript() check. The reason I originally just checked for the existence of window.require() was to support scenarios where the user loads the ArcGIS API for JavaScript by any other method (i.e. hand typing a <script> tag into the document, etc). I we could just document the need to add the data-esri-loader="loaded" attribute to the script tag (need to do that anyway, to support preloading). I can't really think of a scenario where it would not be reasonable to ask that of someone if they don't want to use the bootstrap() method provided by this library.
Would you mind adding the test? I can add the documentation to the README.

Sounds good, @tomwayson.  I should be able to get the unit tests finished up tomorrow morning and push the update.

@tomwayson I've updated the unit tests.  Let me know if you want any more tests to be written to better cover the change.

Thanks @kgs916! I'm working on docs and a release.

This is released as v1.0.0 (breaking change).


Problem downloading data

I'm trying to run the first example on hands on machine learning Book by Geron Aurelien
I'm getting this error: FileNotFoundError: [Errno 2] No such file or directory: 'oecd_bli_2015.csv'
Can somebody help me?

Here is the path of file:- datasets/lifesat/oecd_bli_2015.csv

Hey, you can find data here.


Add signWithMD5 that is able to sign data with RSA private key using md5

Add signWithMD5 that is able to sign data with RSA private key using md5

Thank you very much for your contribution! I'll have a look at your changes and if everything is fine it'll get merged into the repo.

Looks great, but can you please add unit tests to keep test coverage up? It shouldn't be a big deal since you could just copy the test cases from SHA and modify the expected results to the ones which fits the MD5 specification.

Hi, thanks for your project. I added the unit test with "verifySignatureWithMD5" method.