Check for AMQPS protocol, enable ssl if so

This is an Enterprise patch for the issue we've been seeing where if given an amqps:// URI travis-logs can't make a connection . It errors with:
[2016-08-09T10:12:26+0000][travis-logs:logs] com/rabbitmq/utility/BlockingCell.java:77:in `get': java.util.concurrent.TimeoutException
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/utility/BlockingCell.java:111:in `uninterruptibleGet'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/utility/BlockingValueOrException.java:37:in `uninterruptibleGetValue'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/client/impl/AMQChannel.java:367:in `getReply'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/client/impl/AMQConnection.java:293:in `start'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/client/ConnectionFactory.java:647:in `newConnection'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from com/rabbitmq/client/ConnectionFactory.java:694:in `newConnection'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from java/lang/reflect/Method.java:606:in `invoke'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare/session.rb:503:in `new_connection_impl'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from org/jruby/RubyProc.java:271:in `call'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare/session.rb:467:in `converting_rjc_exceptions_to_ruby'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare/session.rb:500:in `new_connection_impl'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare/session.rb:136:in `initialize'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare/session.rb:109:in `connect'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/gems/march_hare-2.16.0-java/lib/march_hare.rb:20:in `connect'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/vendor/bundle/jruby/1.9/bundler/gems/travis-amqp-3966b3651adf/lib/travis/amqp/march_hare.rb:23:in `connection'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/lib/travis/logs/receive.rb:41:in `declare_exchanges'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from /usr/local/travis-logs/lib/travis/logs/receive.rb:25:in `setup'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from ./bin/receive_logs:8:in `(root)'
[2016-08-09T10:12:26+0000][travis-logs:logs]     from $_dot_/bin/./bin/receive_logs:8:in `(root)'

Turns out that march_hare (the Rabbit library we use in travis-logs) also wants either tls: true or ssl: true passed to it when using TLS/SSL. This PR attempts to solve this issue by looking at the protocol passed.
A more thorough fix (for future versions and master) would likely involve a fix in travis-config so that it passes it along, or provides some nicer interfaces for finding out what protocol was used.

(I updated the original PR comment to fix the code block not terminating correctly and the text after it being "code-formatted")
If you're looking for where this is done in travis-config, it's here: https://github.com/travis-ci/travis-config/blob/bc457b6d36119d42edf3a72e2cc13cdc770a2843/lib/travis/config/heroku.rb#L31. For a quick fix, this works, but I agree that this should probably go in travis-config in the end so it gets fixed everywhere where travis-config is used.

@henrikhodne - Thanks for fixing that!
And for sure, I'd prefer a much more thorough fix. Looks like so far this only affects logs. Which is fortunate. Once I get 2.0 out I'll look at implementing a fix at the travis-config level, and getting it into all the apps (and patches for apps). It's a pretty intense process and figured the fix I went with here was in line with some other patterns that existed (such as the postgres one, though the intent there is different). Also need to make sure that passing ssl: true to other Rabbit adapters doesn't have unwanted side effects.
But still, just in case some other app decides to use march_hare getting it in travis-config would be ideal.
Thanks for looking at this! 👍


Use previous selection when focus origins from key event

Selection should be restored to its previous state when using tab to navigate
a page with ProseMirror and other focusable elements with selection (inputs).
This is similar to how textareas work.
Note: While it's possible to create a workaround outside of prosemirror-view by saving the selection and restoring it before/after focus, it does not seem to be a way to avoid scroll jumps when the selection is further down in the document.
The input focus handler restores the previous selection (and DOM selection) already, so ignoring focus when origin is unknown (originated outside of the editor and not via pointer click) makes it restore the previous selection correctly without causing scroll jumps.

This patch seems somewhat scary, because it's hard to know that it won't incorrectly ignore selections in situations where they they are actual user actions that we somehow didn't predict. Does attached patch work for you?


Require ansible 2.10.7 with ansible operator

Type of question

Open question
Question
What did you do?
Hello,
I noticed current operator supports following version of ansible
ansible 2.9.15
config file = /etc/ansible/ansible.cfg
configured module search path = ['/usr/share/ansible/openshift']
ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.8.3 (default, Aug 18 2020, 08:56:04) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
I am in need of certain ansible features which are supported ion 2.10.x. Hence on my mac my playbooks work
ansible 2.10.7
Questions :-

Is there a version of the operator where updated ansible version is present
If not, how do i customize the base image , since its locked down to non sudo at the bottom

Ansible 2.10 contains breaking changes, so will not be supported within the operator-sdk v1.y timeframe. If you want this Ansible version, you can try upgrading your ansible-operator image by following the 2.10 port guide during image build by adding steps to your Dockerfile, although I do not recommend this.
@fabianvf we may want to add an FAQ to answer this, since it is confusing that ansible-operator does not support an Ansible v2.y release.
/triage support
/language ansible

Any details or pointers on what kind of breaking changes were observed when moving to ansible 2.x ?

Take a peek at https://docs.ansible.com/ansible/devel/porting_guides/porting_guide_2.10.html

Thanks @estroz . Couple more questions.

I looked at the above release notes. Was there any one thing that broke the operator-sdk?
When is the v2 version of operator-sdk expected? i helps us make a decision either way...

-Thanks

Ansible 2.10 contains breaking changes, so will not be supported within the operator-sdk v1.y timeframe. If you want this Ansible version, you can try upgrading your ansible-operator image by following the 2.10 port guide during image build by adding steps to your Dockerfile, although I do not recommend this.
@fabianvf we may want to add an FAQ to answer this, since it is confusing that ansible-operator does not support an Ansible v2.y release.
/triage support
/language ansible

I did try to modify the "base.Dockerfile" t include ansible 2.10.X however it errors out at the pipfile section .  It would be great if you share an example.
We cannot directly use the normal docker image and modify as its locked down with user

Best if we add to FAQ


Fix some parts in test/support/protocol_utils.rb to run tests correctly

@ono-max thanks for the update!
@ko1 I think this can be merged 🙂


Loss of video audio

I am mixing mp3 audio into mp4 video, but the resulting old audio of the video is lost

Can you provide a code sample? Are you adding a new track or mixing audio in using AudioOverlayFilter?

I am using lite-demo available and got error field like above, Feature Mux Video And Audio Track

Do you have a sample code for muxing audio into video while keeping the old audiotrack?

Very first demo (Transcode Video/Audio GL) has a capability to apply audio overlay. That does what you need.

I got it, thanks for your help!

Awesome! Let us know if you need more help.


Poor password security

There are a number of serious issues with the way orient currently stores passwords:

User passwords are stored as direct hashes. This means:

I can google for a hash and often retrieve the original password.
I can use rainbow tables to crack common passwords extremely efficiently.
I can use a GPU to bruteforce password hashes with relative ease.
If two or more users share the same password, I recover them all at once.

this is very very very bad.. At the very least we need to introduce per user salts which would defeat the easiest of the attacks, but the correct solution is to use scrypt or bcrypt here. It is possible to fix this without requiring users to change their passwords or having to wait until they next log in, see this post: http://blog.jgc.org/2012/06/one-way-to-fix-your-rubbish-password.html

I can execute a query that retrieves passwords. There are no circumstances where it actually makes sense to be able to retrieve a user's hashed password. It definitely makes no sense for an arbitrary user to be able to retrieve another user's password. Yet at the moment it's as simple as:

SELECT name, password FROM OUser;

and as far as I can tell, it's not actually possible to restrict this using the current permissions system.
Orient currently supports a readonly restriction for properties, I propose we add a writeonly restriction too and apply it to OUser.password. Such properties should never be returned from queries or record load commands.

System passwords are stored in plaintext. While Orient generates a strong random root password on first build, it's stored as plain text in orient's configuration file. If I can find a way to read the configuration file, I can access every record in the system, even if I have no further access. The solution is to generate a strong password on first build, present it to the user for them to remember, and store the strong hash (bcrypt, scrypt) in the config file. So now even if I can read the config file, I can't use the credentials to actually login to the database.

This is security-relevant and has now been open for more than a year… :worried:

Part 3) has been fixed in #2811.

Part 1) has been implemented: #1229

Part 2) has ben fixed by allowing only to "admin" the permission to access to OUser and ORole clusters.


CPAC crashes before completion

Greetings,
I have been running the BIDS-App CPAC docker container image (v1.0.2_disable_log_2) on test participants in two different datasets. CPAC crashes out prior to completion.
The crash error in the pypeline.log file reads:

180330-02:24:47,161 workflow ERROR:
could not run node: resting_preproc_sub-166_ses-1.gen_motion_stats_0.calc_spike_percent.a0.c0
180330-02:24:47,162 workflow INFO:
crashfile: /outputs/crash/crash-20180330-002347-root-calc_spike_percent.a0.c0-8b8c1d4c-4aef-4ba6-a40a-a4b7996d54f7.pklz
180330-02:24:47,163 workflow INFO:

I'm uploading the full pypeline.log file and can send the crash log .pklz file if needed.
Thanks
pypeline.log

Sorry for the late reply.
here is the error in your log file:
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 372, in run
self._run_interface()
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
self._result = self._run_command(execute)
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 613, in _run_command
result = self._interface.run()
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1081, in run
runtime = self._run_wrapper(runtime)
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1029, in _run_wrapper
runtime = self._run_interface(runtime)
File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/interfaces/utility/wrappers.py", line 194, in _run_interface
out = function_handle(**args)
File "", line 17, in calc_percent
Exception: A string was entered for the de-spiking/scrubbing threshold, but there is no percent value.
Interface Function failed to run.
Looks like the scrubbing threshold was set to the wrong value. @sgiavasis would you please give the correct value for this?

Hi @araikes,
Are you running a default pipeline, or a modified pipeline config? If a modified one-Not sure if you have scrubbing enabled, but either way, try entering some typical float value (such as 0.2) into the brackets, without quotes, in the spikeThreshold pipeline option and see if you still get that error.
Also, a much newer version of the C-PAC container (v1.1.0) is available- I would recommend trying that one instead.
Best,
Steve

Hi @sgiavasis,
I'm running the default pipeline. When I pulled the docker image, the older version is what I got. Is there a specific location for v 1.1.0?

@sgiavasis and @ccraddock,
I have successfully updated to version 1.1.0 and finally have a moment to try using this Docker image again. However, I'm running into the following issue:
My data set is BIDS compliant with the structure: bids_dataset/sub-/
Whenever I run the test_config or attempt to use the pipeline, I receive an error that no anatomical images were found at: bids_dataset//sub-/ses-/sub-_ses-*_T1w.nii.
Is there a way to compel CPAC's docker not to look for a site folder? I've tried running the GUI to no avail and do not presently have singularity available to me.

Hi @araikes,
There is a way to bypass the BIDS validator by supplying the --skip_bids_validator flag at the end of your docker run command, but I'm curious as to why this issue is popping up in the first place.
Can you paste the exact error message you get?
Thanks!
Steve


doc/html/boostbook.html redirects to itself

This is related to https://github.com/boostorg/boostbook/issues/29.
The doc/html/boostbook.html file on both develop and master redirects to http://www.boost.org/doc/libs/master/doc/html/boostbook.html, which is the same file on master. In other words, this file performs infinite redirection to itself.
I'm not sure if this file is supposed to be overwritten on the website, but I didn't find anything to that effect in the superproject docs build scripts and release-tools. If it is supposed to somehow work, please let me know how.
If this file is not used then it should probably be removed. Otherwise, if it is supposed to redirect to BoostBook docs, then it should redirect to either tools/boostbook/index.html or tools/boostbook/doc/html/index.html (preferably, the former).
Note that tools/boostbook/index.html was previously redirecting to ../../doc/html/boostbook.html (i.e. the problematic file in the superproject), and this was changed to doc/html/index.html (i.e. the correct file of the generated BoostBook docs in tools/boostbook).
Additionally, there is tools/index.html that has a link to doc/html/boostbook.html. This file needs to be updated if doc/html/boostbook.html is removed.

In terms of boostbook related files, if everything was reverted to boost 1.85.0, it should presumably work again.
After any updates observe the results of the snapshot:  https://www.boost.org/doc/libs/develop/doc/html/boostbook.html

I don't want to revert back to 1.85 state. The issue needs to be fixed at its root.

I don't want to revert back to 1.85 state. The issue needs to be fixed at its root.

Sure.  Keep this in mind though: in previous versions of the boost superproject /doc/html/boostbook.html was not a fully-formatted, correct, static document.
Rather, was just that odd redirect to "http://www.boost.org/doc/libs/master/doc/html/boostbook.html".
Which means in those previous versions, during a build, the documentation was being "generated" and replacing the redirect page.
Going forward in the future, that should also occur. The pages get built or generated.

Thing is, I cannot find where tools/boostbook/doc build is invoked. I tried building doc in the superproject, at current version and with 08330f3f0c6ed9ae4f36ea1ea67ad6bdf082ad9e and https://github.com/boostorg/boostbook/commit/2ba31e6c17e445b30f25692bb310237d421d7fc2#diff-414a16a059e4645ca1574c6f4ba6cff4efbaf75047da905407f1337096af30d8 reverted, and in both cases tools/boostbook/doc is not being built (meaning its html docs are not generated) and doc/html/boostbook.html remains the same circular redirect page. I can see https://www.boost.org/doc/libs/develop/tools/boostbook/doc/html/index.html does not exist, which means BoostBook docs are not being built by release scripts either. I really have no idea how this build is supposed to work, some help would be appreciated.
Now, regarding the dummy doc/html/boostbook.html redirect page, it seems that similar circular redirect pages are also present for other libraries, e.g. doc/html/atomic.html. Specifically Boost.Atomic builds its docs in libs/atomic/doc/html, and this circular redirect page remains unused as long as all links on the website point to the libs/atomic/doc/html location. I suspect, this is what was done when Boost.Atomic switched to the standalone docs build, i.e. links on the website were updated and the dummy redirect page was left behind unused. The same could be done with BoostBook as well, but first we need to figure out how to build its docs during the upload to the website.

@Kojoley Could you have a look, please? In particular, is this line:
https://github.com/boostorg/boost/blob/f400245a6b907e503289281f62ca8176081b8c15/doc/Jamfile.v2#L201
supposed to invoke the build of html documentation in tools/boostbook/doc? And if yes, why is this not happening? I'm invoking b2 -j16 release in doc of superproject.
Other than this line, I cannot see where tools/boostbook/doc build could have been invoked.

Earlier you pointed out a change in the superproject:
https://github.com/boostorg/boost/commit/2403dacaa7c37e8207180a5b5e19ef2b062e21ea
That removed "boostbook.xml".
"boostbook.xml" contains the content of the generated pages.
Reverting the commit ought to fix it, right?
Go back to referencing that file again.
But then... your reply is that solution is not optimal. Or not the root of the problem.   Ok.

Reverting the commit ought to fix it, right?

It doesn't. That is, it doesn't result in tools/boostbook/doc being built when doc is built. And doc/html/boostbook.html is not being replaced either way.

It's been a while since I had touched docs stuff really, I don't even remember how to build release docs. Could you point me to a starting point? b2 doc --release-build?
I suspect the culprit is missing boostdoc or boostrelease target in boostdoc/doc/Jamfile but it's just a guess of mine. I don't know why there is no sanitization for that kind of stuff. I also see that most if not all libraries define one of these target and other as empty, I have no idea why.
Looking at https://github.com/boostorg/boost/blame/master/doc/Jamfile.v2 history the docs building stuff seems to had been implemented by @grafikrobot so he might help untangle this?

I suspect the culprit is missing boostdoc or boostrelease target in boostdoc/doc/Jamfile but it's just a guess of mine. I don't know why there is no sanitization for that kind of stuff. I also see that most if not all libraries define one of these target and other as empty, I have no idea why.

The doc/jamfile deals with it by only building the targets that exist for each library.

So if a library misses documentation it doesn't care? That's not good...

Looking at master/doc/Jamfile.v2 (blame) history the docs building stuff seems to had been implemented by @grafikrobot so he might help untangle this?

I could try.. Although if it used to work and doesn't now not sure why that would be.

There was a change https://github.com/boostorg/boost/commit/2403dacaa7c37e8207180a5b5e19ef2b062e21ea, we fixed one thing and seems to have broken other.

I suspect the culprit is missing boostdoc or boostrelease target in boostdoc/doc/Jamfile but it's just a guess of mine.

Thanks for the idea. Those targets were never there. I've added them and in my local build with --release-build command line option tools/boostbook/doc got built. I've pushed the commit, we'll see if the docs are uploaded to the website.
(BTW, in my local build, tools/quickbook/doc does get built, and tools/quickbook/doc/Jamfile.v2 is also missing these targets. Go figure.)
However, even though tools/boostbook/doc got built, doc/html/boostbook.html was not replaced and remained the self-referring stub.

Try this:
alias boostdoc : boostbook/<format>docbook ;
explicit boostdoc ;
alias boostrelease ;
explicit boostrelease ;

I think this will include BoostBook in the unified Boost documentation twice here - first by including BoostBook in lib-doc-boostdoc-refs and then in tools.
My current boostdoc and boostrelease targets worked on the website, so hopefully after updating links in the superproject and the website this should be resolved.

I think this will include BoostBook in the unified Boost documentation twice here - first by including BoostBook in lib-doc-boostdoc-refs and then in tools.

Then every library is already included twice and it looks like it was intended, in lib-doc-boostdoc-refs and then in libraries https://github.com/boostorg/boost/blob/30c8b4588826e7c8d0b45fd2c47c467296950463/doc/Jamfile.v2#L194-L203 https://github.com/boostorg/boost/blob/30c8b4588826e7c8d0b45fd2c47c467296950463/doc/Jamfile.v2#L210-L214.


Rapid prototyping enhancements

To support the user experience of rapid prototyping on a use case / workflow, there are a few enhancements we can make. This list is TBD and dependent on final design from @opensearch-project/opensearch-ux

show resources that can be deleted when re-running / re-executing a workflow
show recently-ran resources somewhere in an accessible area

Refactoring into clearer meta issues.


New widget design

I changed the toggle widget design to the one in the mockup.
The mockup:

The changed widget:

Thank you very much!


SNOW-1795384:inconsistency with roles in snowpark python

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
<!---
In this section, please add a Snowflake Jira issue number.
<p>Note that if a corresponding GitHub issue exists, you should still include
the Snowflake Jira issue number. For example, for GitHub issue
https://github.com/snowflakedb/snowpark-python/issues/1400, you should
add "SNOW-1335071" here.
--->
Fixes SNOW-1795384

Fill out the following pre-review checklist:

[ ] I am adding a new automated test(s) to verify correctness of my new code

[ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing

[ ] I am adding new logging messages
[ ] I am adding a new telemetry message
[ ] I am adding new credentials
[ ] I am adding a new dependency
[ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes.
[x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

Please describe how your code solves the related issue.
Please write a short description of how your code change solves the related issue.

the test is skipped because it is only reproducable in owner's right stored proc, and this change will not get to sever side in this version


Fix the use of @base

https://github.com/transmute-industries/sidetree.js/pull/43#discussion_r461606155

Possible stale issue, closing.


Program pulls focus from active window when changing wallpaper

I've got a 2 monitor setup on windows 7 and it seems to trigger when in fullscreen applications, like games.
I see the wallpaper changing on one monitor while my game flickers on the other and stops accepting inputs.
I have to then click back on it to resume playing.
I've also witnessed it while the fullscreen game is alt-tabbed

Hay, do you have the 'Info popup' option enabled? If so, can you try disabeling it and see if that helps?
Also worth trying to disable fade mode to see if that makes any difference.

I have a one monitor setup and this happens to me too. I tried disabling the info popup but I felt it was still happening.


CSV output is mangled

Version: 1.5.0-rc16
Docker Container: https://github.com/scr512/docker-diskover-web/blob/master/Dockerfile
When exporting a query to CSV, delimiting breaks... both in Windows land with Excel and with Linux tools (cut)
Example:
$ head -15 diskover_export_file\ \(1\).csv | cut -d, -f 14
filesize
1855808923
1872143765
1619683671
1864039435
1857076267
1850683799
1857484136
8056344320
2018-09-15T09:30:40.856155
2018-09-15T09:30:40.865604
2018-09-15T09:30:40.878265
2018-09-15T09:30:40.893790
2018-09-15T09:30:40.097447
2018-09-15T09:30:40.107179

I clearly expect to see filesize, not a timedate stamp
Here is the first 15 lines without parsing:
dupe_md5,extension,owner,tag,last_change,indexing_date,group,inode,last_modified,last_access,hardlinks,filename,worker_name,filesize,filehash,path_parent,tag_custom
,fsdb,user02,,2018-02-21T14:13:34,2018-09-16T19:07:42.457574,partner,3261587137,2018-02-21T14:13:34,2018-02-21T14:13:34,1,transient0.tran.fsdb,hpc-admin0.11061,1855808923,2d1e78abdd97f1ba0c7423b26f9ddcf7,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/sim/sim.raw,
,fsdb,user02,,2018-02-22T10:38:00,2018-09-16T19:07:42.316971,partner,3300252844,2018-02-22T10:38:00,2018-02-22T10:38:00,1,transient0.tran.fsdb,hpc-admin0.11061,1872143765,29e2a870aced3e73e42dab29d866e2d8,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/VDDPE_OPTIMIZATION/sim/sim.raw,
,fsdb,user02,,2018-02-22T18:07:08,2018-09-16T19:07:42.343690,partner,3317985758,2018-02-22T18:07:07,2018-02-22T18:07:07,1,transient0.tran.fsdb,hpc-admin0.11061,1619683671,280b295ec628826abc241554b2e1fdba,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/WDX4I_AST/CONLY/sim/sim.raw,
,fsdb,user02,,2018-02-22T18:41:35,2018-09-16T19:07:42.379715,partner,3322573322,2018-02-22T18:41:35,2018-02-22T18:41:35,1,transient0.tran.fsdb,hpc-admin0.11061,1864039435,5de9ed413df86ae1dc8e5f106c7aa81d,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/WDX4I_AST/Header_wd/sim/sim.raw,
,fsdb,user02,,2018-02-22T14:50:33,2018-09-16T19:07:42.420287,partner,3312210146,2018-02-22T14:50:33,2018-02-22T14:50:33,1,transient0.tran.fsdb,hpc-admin0.11061,1857076267,3cb2c2cc2298d491d05ce96c7959ffe5,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/WDX4I_AST/Reduce_footer/sim/sim.raw,
,fsdb,user02,,2018-02-22T14:41:14,2018-09-16T19:07:42.397875,partner,3297754556,2018-02-22T14:41:14,2018-02-22T14:41:14,1,transient0.tran.fsdb,hpc-admin0.11061,1850683799,4009b6cac7896686744a039c72cf432a,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/WDX4I_AST/sim/sim.raw,
,fsdb,user02,,2018-02-22T07:39:42,2018-09-16T19:07:42.438273,partner,3292589808,2018-02-22T07:39:42,2018-02-22T07:39:42,1,transient0.tran.fsdb,hpc-admin0.11061,1857484136,47839a89213ccdbb601e24e38e6b5505,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user02/wl_header_update/sim/sim.raw,
,fsdb,user01,,2018-03-06T21:04:22,2018-09-15T09:35:22.213673,partner,3623915697,2018-02-28T23:15:28,2018-02-28T23:15:28,1,c0.fsdb,hpc-admin0.11061,8056344320,d0707d6d0fc8c281c07f78fe653952b9,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/r7/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X84m2fb1fs2w/full/setup_CLK_r_EMA_r/control,
hpc-admin0.11051,fsdb,2269263876,partner,5508efa3bc5e8e57c44db141cb3f3293,,2018-02-14T05:23:14,2018-02-15T21:28:00,user01,,,1,c0.fsdb,2018-09-15T09:30:40.856155,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X16m2fb1fs2w/full/delay_CLK_r_Q_r/control,2805513528,2018-02-14T05:23:14
hpc-admin0.11051,fsdb,2972766162,partner,e8216289179972638df34f3c7e72fbc5,,2018-02-16T06:49:14,2018-02-16T06:49:14,user01,,,1,c0_new4.fsdb,2018-09-15T09:30:40.865604,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X16m2fb1fs2w/full/delay_CLK_r_Q_r/control/ema,2947038418,2018-02-16T06:49:14
hpc-admin0.11051,fsdb,3162196612,partner,769c05ab6c33f016d5275f13dd063783,,2018-02-16T06:51:37,2018-02-16T06:51:37,user01,,,1,c0_new4.fsdb,2018-09-15T09:30:40.878265,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X16m2fb1fs2w/full/delay_CLK_r_Q_r/control/ema/new,2947824310,2018-02-16T06:51:37
hpc-admin0.11051,fsdb,2405317631,partner,612210b3e671b9bf9459641eb291d453,,2018-02-14T05:30:44,2018-02-15T21:28:00,user01,,,1,c0.fsdb,2018-09-15T09:30:40.893790,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X16m2fb1fs2w/full/delay_CLK_r_Q_r/control/new,2809553166,2018-02-14T05:30:44
hpc-admin0.11051,fsdb,6352033331,partner,420d734f5dc1d5fb43948f2db672b319,,2018-02-15T21:25:27,2018-02-15T21:27:59,user01,,,1,c0_new.fsdb,2018-09-15T09:30:40.097447,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X60m2fb1fs2w/full/delay_CLK_r_Q_r/control,2908678752,2018-02-15T21:25:27
hpc-admin0.11051,fsdb,6762929436,partner,50b46f2c2fbf702b1f8a8c10c6f55c07,,2018-02-15T21:34:20,2018-02-15T21:34:20,user01,,,1,c0_new.fsdb,2018-09-15T09:30:40.107179,/my_company/scratch_space/engineering_division/partner/partner_technology/engineering_team/user/user01/partner_partner_technologymy_company_sw_tool/custom_tool/simulation/timing/tt_0p75v_0p75v_85cUHS_FCI/i128X60m2fb1fs2w/full/delay_CLK_r_Q_r/control/new,2918391401,2018-02-15T21:34:20

From just glancing at the data, it appears to be an issue with empty fields. For example, I'm not using custom tags nor dedupe.
Blank values should be
,, or , ,

Works fine with JSON:
$ cat diskover_export_file.json | python -mjson.tool | grep '"filesize":' | head -15
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,
        "filesize":<PHONE_NUMBER>,

The order of the exported fields looks different than on my end, filesize should be field 4 and running cut works on my end on an export csv of files. Can you verify that you have the same results.php and export.php files as the latest on github?
The field order should be:
filename,extension,path_parent,filesize,owner,group,last_modified,last_access,last_change,hardlinks,inode,filehash,tag,tag_custom,dupe_md5,worker_name,indexing_date

That's strange! I'll have a look my Dockerfile as it clones from master and I build the image using --no-cache to avoid Docker being overly aggressive with it's layer caching.
Checking the MD5s from both a git clone on my server into /tmp and inside my Docker container, they match up.
# git clone https://github.com/shirosaidev/diskover-web.git
Cloning into 'diskover-web'...
remote: Enumerating objects: 140, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (93/93), done.
remote: Total 2323 (delta 89), reused 80 (delta 43), pack-reused 2183
Receiving objects: 100% (2323/2323), 47.29 MiB | 18.42 MiB/s, done.
Resolving deltas: 100% (1654/1654), done.
Checking connectivity... done.
# md5sum /tmp/diskover-web/public/export.php
eb90b3f6068b33c00516f5ae9d81b09d  export.php
# md5sum /tmp/diskover-web/public/results.php
2e9e8937e2f0845b51b3e5d214a1b66f  results.php
# docker exec -it diskover-web bash
#  md5sum /opt/diskover-web/public/export.php
eb90b3f6068b33c00516f5ae9d81b09d export.php
#  md5sum /opt/diskover-web/public/results.php
2e9e8937e2f0845b51b3e5d214a1b66f  results.php

I'm looking at a report I've generated through a smart searches.
URL: http://localhost:8081/simple.php?index=diskover&index2=&submitted=true&p=1&resultsize=50&q=%21waveform&doctype=
My search is pretty straight forward:
extension:(fsdb OR raw)
Looking at the header of my CSV export:
worker_name,extension,filesize,group,filehash,dupe_md5,last_modified,last_change,owner,tag,tag_custom,hardlinks,filename,indexing_date,path_parent,inode,last_access

I pushed a small update to v1.5.0-rc16. Can you try to see if you can export now correctly? If not, what version of diskover was the index created with? Are you using diskover latest from github?

Still seeing the same issue though it appears I'm a couple of rc-releases back with diskover (diskover v1.5.0-rc14). Let me kick off a new scan using the latest release and we'll see if this issue then resolves.

I've done some testing. Typically, the data I'm scanning contains 10s-100s of millions of files and directories but when I ran a test scan against my personal scratch area, the CSV export functionality worked. I'm still working to get some more empirical data but it seems that either the path depth (path parent) field is perhaps causing the issue? The mangled-ness occurs irrespective of me exporting the whole dataset or just a single page.

If you go into kibana and look at one of the docs which isn't being printed correctly in the csv, is it's source fields the same and in same order as one of the docs that is printing correctly?

I pushed an update to rc17 just now to try to patch this in exports.php if this is the case. Try it and let's see if this fixes it.

Alright, I'll have a look via Kibana and I'll kick off a new scan using the fixed up exports.php.

Bingo! That seems to have done the trick. I can confirm the patch to exports.php addresses the issue.


Got an error when evaluating and testing the model trained with this code

Thanks for this work,
The training was successful, however I got an error  when I tried to test (demo on video) and evaluating (for performance evaluation like MOTA, IDS etc)

The following is an error when I do demo (on video)

[warning] No nID got!!!
2022-01-24 10:28:34.520 | INFO     | __main__:main:326 - Model Summary: Params: 104.65M, Gflops: 880.83
2022-01-24 10:28:34.524 | INFO     | __main__:main:334 - loading checkpoint
Traceback (most recent call last):
  File "tools/demo_track.py", line 372, in <module>
    main(exp, args)
  File "tools/demo_track.py", line 337, in main
    model.load_state_dict(ckpt["model"])
  File "C:\Users\admin\anaconda3\envs\bytetrack_reid\lib\site-packages\torch\nn\modules\module.py", line 1051, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for YOLOX:
        size mismatch for head.reid_classifier.weight: copying a param with shape torch.Size([40, 128]) from checkpoint, the shape in current model is torch.Size([2, 128]).
        size mismatch for head.reid_classifier.bias: copying a param with shape torch.Size([40]) from checkpoint, the shape in current model is torch.Size([2]).

Is there any modification needed in the code. In [40, 128], 40 in the number of id trained in my model.

The following is an error when I do evaluation

> File "c:\users\admin\desktop\bytetrack_Reid\yolox\core\launch.py", line 90, in launch
    main_func(*args)
    │          └ (╒══════════════════╤══════════════════════════════════════════════════════════════════════════════════════════════╕
    │            │ keys  ...
    └ <function main at 0x000001C75DE9F3A0>

  File "tools\track.py", line 220, in main
    *_, summary = evaluator.evaluate(
                  │         └ <function MOTEvaluator.evaluate at 0x000001C75D08C280>
                  └ <yolox.evaluators.mot_evaluator.MOTEvaluator object at 0x000001C714D7D5B0>

  File "c:\users\admin\desktop\bytetrack_Reid\yolox\evaluators\mot_evaluator.py", line 137, in evaluate
    frame_id = info_imgs[2].item()
               └ [tensor([1080]), tensor([1545]), ['MOT20-04/img1/000001.jpg']]

AttributeError: 'list' object has no attribute 'item'

Thank you.

Any updates @HanGuangXin ,

@HanGuangXin
I managed to solve the first problem (When doing demo on video). I modify the file demo_track.py, by adding code related to re-id.
However, with problem 2 (during evaluation in mot20 or other custom dataset) still got the following error.
Traceback (most recent call last):

  File "tools\track.py", line 290, in <module>
    launch(
    └ <function launch at 0x000001BC9F33A5E0>

> File "c:\users\admin\desktop\byteTrack_reid\yolox\core\launch.py", line 90, in launch
    main_func(*args)
    │          └ (╒══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...
    └ <function main at 0x000001BCAD110550>

  File "tools\track.py", line 167, in main
    val_loader = exp.get_eval_loader(args.batch_size, is_distributed, args.test)
                 │   │               │    │           │               │    └ False
                 │   │               │    │           │               └ Namespace(batch_size=1, ckpt='pretrained/best_ckpt.pth.tar', conf=0.01, devices=1, dist_backend='nccl', dist_url=None, exp_fi...
                 │   │               │    │           └ False
                 │   │               │    └ 1
                 │   │               └ Namespace(batch_size=1, ckpt='pretrained/best_ckpt.pth.tar', conf=0.01, devices=1, dist_backend='nccl', dist_url=None, exp_fi...
                 │   └ <function Exp.get_eval_loader at 0x000001BCAD110DC0>
                 └ ╒══════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════...

  File "C:\Users\admin\Desktop\ByteTrack_reid\exps/example/mot\yolox_x_mot17_half.py", line 100, in get_eval_loader
    valdataset = MOTDataset(
                 └ <class 'yolox.data.datasets.mot.MOTDataset'>

  File "c:\users\admin\desktop\byteTrack_reid\yolox\data\datasets\mot.py", line 48, in __init__
    self.nID = self.get_total_ids()         # TODO: total ids for reid classifier
    │          │    └ <function MOTDataset.get_total_ids at 0x000001BCA03C3820>
    │          └ <yolox.data.datasets.mot.MOTDataset object at 0x000001BCE571F7C0>
    └ <yolox.data.datasets.mot.MOTDataset object at 0x000001BCE571F7C0>

  File "c:\users\admin\desktop\byteTrack_reid\yolox\data\datasets\mot.py", line 140, in get_total_ids
    max_id_each_img.append(int(max(res[:, 5])))
    │               │              └ array([], shape=(0, 6), dtype=float64)
    │               └ <method 'append' of 'list' objects>
    └ []

ValueError: max() arg is an empty sequence

Any suggestion on solving this issue.
Thank you.

Here is the thing. ReID task in MOT is usually treated as a classification task, with track_ids as the class. So get_total_ids() is a function that calculate the total ids in the dataset.
So you maybe should modify the function  get_total_ids() to calculate the total ids in the dataset corresponding datasets.
@NaifahNurya

And the reason of 'size mismatch error' is that in YOLOX, head network and loss calculation is in one Class, so the ReID classifier has to be initialized even if we don't use it when evaluating.
So during evaluation, you should delete parameters of classifier and set strict=False when loading checkpoints. Code can be found in tools/track.py.
@NaifahNurya

@NaifahNurya thanks for sharing your progress i have couple of queries

which dataset did u train it on MOT20 or MOT17 ?
Can you please share your pretrained model as i am facing issues during dataload while training

THanks in adance

@abhigoku10
The model is trained on MOT17, see yolox_s_mot17_half.py for more information.
Trained model will be updated soon, or you can train it yourself.


KAFKA-5886: Implement KIP-91 delivery.timeout.ms

First shot at implementing kip-91 delivery.timeout.ms. Summary

Added delivery.timeout.ms config. Default 120,000
Changed retries to MAX_INT.
batches may expire whether they are inflight or not. So muted is no longer used in RecordAccumulator.expiredBatches.
In some rare situations batch.done may be called twice. Attempted transitions from failed to succeeded are logged. Successful to successful is an error (exception as before). Other transitions (failed-->aborted, aborted-->failed) are ignored.
The old test from RecordAccumulatorTest is removed. It has three additional tests. testExpiredBatchSingle, testExpiredBatchesSize, testExpiredBatchesRetry. All of them test that expiry is independent of muted.

Thanks for the PR @sutambe . Looking over the changes, it seems that there are two cases from the KIP which don't seem to be covered:

Setting the pollTimeout to be the expiryTime of the oldest batch being sent in the produce request. I think we need this to make sure that we expire batches in a timely manner.
Related to the previous point, the current patch doesn't seem to expire inflight requests, which is another feature that the KIP seems to promise.

Have I missed something? Or are you planning on adding the functionality above?

Heads up @sutambe , the following PR just got merged, and may have some conflicts with the current patch : https://github.com/apache/kafka/pull/3743
There shouldn't be any impact on the logic, however.

Friendly reminder that the feature freeze is this Wednesday.

@ijuma Just want to check. Do you think this feature is a "minor" feature?

@becketqin, it is possible to classify this as a minor feature, but the fact that it affects a core part of the Producer puts it in a bit of a grey area. If the PR is almost ready and we miss the feature freeze, my take is that it would be OK to merge it by the end of this week. Later than that and it seems a bit risky.
It's a bit worrying that the merge conflicts haven't been resolved since last week.

I've an new PR but after a rebase I've to fix one more test. Working on that now.

Thanks @sutambe!

@apurvam It's not clear to me why testExpiryOfFirstBatchShouldCauseResetIfFutureBatchesFail and testExpiryOfFirstBatchShouldNotCauseUnresolvedSequencesIfFutureBatchesSucceed are failing. It looks like a batch that should be in IncompleteBatches isn't there. Any thoughts?

I added the following to testExpiryOfFirstBatchShouldCauseResetIfFutureBatchesFail before the first sender.run() call
        Sender sender = new Sender(logContext, client, metadata, this.accumulator, true, MAX_REQUEST_SIZE, ACKS_ALL, 10,
            new Metrics(), new SenderMetricsRegistry(), time, REQUEST_TIMEOUT, DELIVERY_TIMEOUT, 50, transactionManager, apiVersions);

The test still fails.

@sutambe where are those tests failing? The latest PR builder suggests that the clients and core tests all passed.

@apurvam @ijuma @becketqin The Sender and RecordAccumulator are passing now. The failing tests are connect tests that are irrelevant.

@sutambe I don't think the test failures are irrelevant since the same 3 tests failed in all the runs. Further, the cause of the failure is:
java.lang.AssertionError:
  Unexpected method call Listener.onFailure(job-0, org.apache.kafka.common.KafkaException: Failed to construct kafka producer):

I think their mocks may need to be updated to take account of the new configs and attendant ConfigExceptions

@sutambe I had a look at the failing Sender expiry tests. What is happening is that the tests are not modified to account for the fact that the inflight batches can be expired. In the tests, we used to expire a batch sitting in the accumulator but not the inflight batch. When the inflight batch returned, it would be re queued.
But now, the test sends the response for the inflight batch, but when it goes to requeue, it discovers that there shouldn't be an inflight request an raises an exception.
The tests should be updated to account for the new behavior and make sure that the inflight batch is not expired.

Actually, the test reveals a bug in the current patch: the expired inflight batch is not being handled correctly. We should not be trying to requeue it to start with.
So we need two tests: one where the inflight batch is not expired, and the current case. The reenqueue logic in the sender needs to be updated to not reenqueue the expired batches.

@becketqin  Added a couple of tests.
Test a batch is correctly inserted into the in.flight.batches if needed. And not inserted if not needed.
RecordAccumulatorTest.testSoontoExpireBatchesArePickedUp
Test the callback of an expired batch is fired in time when it is in-flight/not in-flight
SenderTest.testInflightBatchesExpireOnDeliveryTimeout

@sutambe I don't see any changes since my comments. Are you sure you have pushed them?

@becketqin  Added more tests
Test when expire an in-flight batch, we still wait for the request to finish before sending the next batch = SenderTest.testWhenFirstBatchExpireNoSendSecondBatchIfGuranteeOrder
Test we are not going to retry an already expired batch = SenderTest.testExpiredBatchDoesNotRetry
Regarding the last test "Test when batch is expired prematurely, the buffer pool is only deallocated after the response is returned. (because we are still holding the batch until the response is returned)". I'm not sure if it's really needed because RecordAccumulator.abortBatches deallocates all incomplete batches (whether they have been sent or not).

@sutambe I am looking at the update patch.

Regarding the last test "Test when batch is expired prematurely, the buffer pool is only deallocated after the response is returned. (because we are still holding the batch until the response is returned)". I'm not sure if it's really needed because RecordAccumulator.abortBatches deallocates all incomplete batches (whether they have been sent or not).

RecordAccumulator.abortBatches() is only used when the producer is force closed. We do not need to worry about the memory consumption at that point anymore because the producer has stopped accepting new messages.

@sutambe It looks there are 3 test failures from connect. Can you check?

@sutambe BTW, I did not see the tests to ensure we are not double deallocating memory. Will you add those tests?

FYI, the connect tests are still failing with:
Caused by: org.apache.kafka.common.config.ConfigException: Must set delivery.timeout.ms higher than linger.ms + request.timeout.ms
	at org.apache.kafka.clients.producer.KafkaProducer.configureDeliveryTimeout(KafkaProducer.java:452)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:380)
	... 35 more

It would be good to get some green builds before this can be considered safe for merge.

@apurvam @becketqin Fixed Connect tests.

I have a high level comment about the inflight batch handling. I was wondering what the advantages/disadvantages of the current approach (where batches are grouped by creation time) would be against adding a Map<TopicPartition, PriorityQueue<ProducerBatch>> to the RecordAccumulator.  This would complement the existing Map<TopicPartition, Deque<ProducerBatch>>.
The priority queue would be ordered by the batch creation time (similar to the current NavigableMap).
We would add to the queue on drain. And we would remove from the queue in Sender.failBatch and Sender.completeBatch, just like we do for the TransactionManager (see: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L589). So we don't have any special logic for
The RecordAccumulator.expiredBatches would now iterate over both maps (TopicPartition -> Deque and TopicPartition -> PriorityQueue), with the priority queue being iterated on first. This doesn't change the time complexity of the algorithm, since we are iterating over all partitions in expiredBatches once anyway.
The advantage of this approach is that managing inflight batches is much simpler (add to the queue on drain, remove on completion). It doesn't complicate the expiry in any way.
The main disadvantage that I see is in computing the earliestDeliveryTimeout. For this you would need to peek at the head of the queue for each partition which has an inflight batch. But this should be fairly cheap, since we would be iterating over 1000's of objects even in the largest installations.
What do you think?

retest this please

I made this comment in a thread, but that may be buried because github collapses comments. There is a bug in the current implementation which means that we will retain references to ProducerBatches in the producer long after they have been successfully completed. The following sequence of assertions in any SenderTest shows the presence of this bug:
    assertFalse(client.hasInFlightRequests());
    // the following should be true, since there are no in flight requests.
    assertFalse(accumulator.soonToExpireInFlightBatches().isEmpty());
    time.sleep(130000);
    // we now 'expire' the completed inflight batches
    sender.run(time.milliseconds());
    // the following shows that we hang on to references for 120 seconds by default.
    assertTrue(accumulator.soonToExpireInFlightBatches().isEmpty());

@apurvam Initially I was under the same impression. One concern of the priority queue is that whenever we need to remove things out of the queue, we will have to iterate over it. Since a producer can poll thousands of times per seconds, and maybe remove tens of thousands of batches. That said, in most cases, the queue would be empty because most of the batches are sent when remaining delivery.timeout.ms is greater than request.timeout.ms. However, it may still slowdown things when retries occur. My take on this is that if the implementation complexity does not increase much, we may want to follow the robust way instead.
Regarding the memory issue. We could not release the memory of an expired batch right away when the batch is still in-flight. This is because the in-flight batch is still referred by the RequestCompletionHandler of the request. We could remove it from the RequestCompletionHandler, but this is a little tricky.

@becketqin I am not sure I follow this:

Regarding the memory issue. We could not release the memory of an expired batch right away when the batch is still in-flight. This is because the in-flight batch is still referred by the RequestCompletionHandler of the request. We could remove it from the RequestCompletionHandler, but this is a little tricky.

Is this a response to my comment above where I shared a snippet from SenderTest to show that we are retaining batches after they are completed? If so, I didn't mean that we need to release an inflight batch right away. What I was pointing out is that the only way the current patch is deleting batches is by expiring them. This may not be a big deal, since the deallocation is done when the batch is completed, but it still seems like an unnecessary anomaly.

Also, it looks like recent commits are causing threads to leak, @sutambe .

One concern of the priority queue is that whenever we need to remove things out of the queue, we will have to iterate over it. Since a producer can poll thousands of times per seconds, and maybe remove tens of thousands of batches. That said, in most cases, the queue would be empty because most of the batches are sent when remaining delivery.timeout.ms is greater than request.timeout.ms. However, it may still slowdown things when retries occur. My take on this is that if the implementation complexity does not increase much, we may want to follow the robust way instead.

Not sure I follow. It would be no worse that what we do today with the Deque except it would be limited only to partitions which had in flight batches. So it is not making things much worse.

@apurvam

Not sure I follow. It would be no worse that what we do today with the Deque except it would be limited only to partitions which had in flight batches. So it is not making things much worse.

I misunderstood your proposal. I though you were suggesting to use a priority queue to store all the in-flight batches. A Map<TopicPartition, PriorityQueue<ProducerBatch>> sounds reasonable.

@becketqin

I misunderstood your proposal. I though you were suggesting to use a priority queue to store all the in-flight batches. A Map<TopicPartition, PriorityQueue> sounds reasonable.

Yes, I meant a separate Map<TopicPartition, PriorityQueue<ProducerBatch>>, to replace the current NavigableMap.

@sutambe Any updates on this? Do you need help addressing the test failures?

@hachikuji  Sorry for disappearing on you guys. I got sucked into an oncall issue. So I had to context switch. I plan to resume work on this PR next week. I've not had a chance to respond to the comments so far. Thank for you the review so far. It's been very useful.

FAILURE
7505 tests run, 2 skipped, 168 failed.
--none--

FAILURE
7910 tests run, 5 skipped, 48 failed.
--none--

FAILURE
7916 tests run, 5 skipped, 46 failed.
--none--

FAILURE
7280 tests run, 1 skipped, 0 failed.
--none--

FAILURE
7964 tests run, 5 skipped, 1 failed.
--none--

FAILURE
7964 tests run, 5 skipped, 1 failed.
--none--

@becketqin @apurvam I've updated the PR. Please take a look

FAILURE
7964 tests run, 5 skipped, 17 failed.
--none--

FAILURE
7964 tests run, 5 skipped, 16 failed.
--none--

Please fix the failing tests, there is a config exception due to the new delivery timeout config.

FAILURE
7280 tests run, 1 skipped, 0 failed.
--none--

FAILURE
4026 tests run, 1 skipped, 1 failed.
--none--

FAILURE
4026 tests run, 1 skipped, 1 failed.
--none--

FAILURE
4026 tests run, 1 skipped, 1 failed.
--none--

SUCCESS
8077 tests run, 5 skipped, 0 failed.
--none--

SUCCESS
8077 tests run, 5 skipped, 0 failed.
--none--

SUCCESS
8077 tests run, 5 skipped, 0 failed.
--none--

@becketqin @apurvam All checks passed

@apurvam @becketqin I updated the implementation to use ConcurrentMap<TopicPartition, Deque<ProducerBatch>>. Please take a look. I don't see the above test failures on my machine.

@sutambe @becketqin It would be nice to unblock this. Can someone else pick up the work?

@hachikuji Yeah, this has been pending for too long. I have spoken to @sutambe and he said he still wants to finish the patch. He will figure out the ETA and see if that works.

@sutambe @becketqin is there any update on the status of this PR?  It would be great if we could get this in the next release.

@guozhangwang @junrao @bbejeck @becketqin We also hit this issue when running Kafka Streams library with some high volume output topics. It would be nice to get this moving and push it to the next release.

becket cant load this page for some reason (some weird issue with his github profile?).
we are ok with you taking over this patch.

@bbejeck @apurvam @becketqin @hachikuji I don't have any update since Dec last year. Sorry, the work has stalled and it has been very hard to find cycles for this effort. I don't mind if Confluent wants to take this effort forward. Better later than never.

cc @abbccdda @yuyang08

@sutambe   i made some change based on your pull request to fix style check and test failure. can i amend the change to this pull request?  cc @becketqin  @apurvam @hachikuji
https://github.com/yuyang08/kafka/commit/69fc79a91d0556408c8037649f1e03aa56206ef2

@yuyang08 I'd suggest you create your own PR against apache kafka trunk and let other reviewers to continue reviewing that one.

@guozhangwang  sure.  will create a separate pull request

@guozhangwang  @apurvam @becketqin  created new pr https://github.com/apache/kafka/pull/5270 for KAFKA-5886

This has been merged via a different PR, closing.


Fix the build (remove import com.getcapacitor.Config)

By removing this line, the plugin works in capacitor 3
It fixes this error:
> Task :capacitor-community-intercom:compileDebugJavaWithJavac FAILED
app/node_modules/@capacitor-community/intercom/android/src/main/java/com/getcapacitor/community/intercom/IntercomPlugin.java:3: error: cannot find symbol
import com.getcapacitor.Config;
                       ^
  symbol:   class Config
  location: package com.getcapacitor
Note<EMAIL_ADDRESS>uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 error

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':capacitor-community-intercom:compileDebugJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

Closes #26

Thanks for the PR. I'll try and merge soon.

@benwinding are you able to update this PR with Capacitor 3 changes for ios as well?

actually if you can, please open a new PR for Capacitor 3. I'm closing this one because it's incomplete.
Thank you

@stewones Seems weird to close a PR which fixes an issue... at least one platform works from this PR

I appreciate your time but no, it doesn't work for cap3. There are more changes to address other than just removing a line of java code
I've closed due to the latest intercom major change for cap2, so it's easier to just send a new fresh one from master rather having to resolve conflict issues, but I'm happy to merge this one if you address all the changes needed, update ts, example app, etc.
For reference
https://capacitorjs.com/docs/updating/plugins/3-0


chore(eslint): Add async/await support in eslint

Right now we have async/await support for node using babel, but Eslint was throwing errors. This commit fix them.

Current coverage is 65.82% (diff: 100%)

Merging #36 into develop will not change coverage

@@            develop        #36   diff @@
==========================================
  Files             4          4
  Lines            79         79
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits             52         52
  Misses           27         27
  Partials          0          0

Powered by Codecov. Last update 6d1adb1...9c76b0d


jax and jaxlib versions

TL;DR is it okay to use the jaxlib version of 0.1.68+cuda110 instead of 0.1.69+cuda110?
I have tried to write a script and construct a conda environment that does not use Docker. When I used the same versions of jax and jaxlib defined in the docker/Dockerfile, I had some issues during the inference time. Scripts were working fine for model_{1,3,4} but raised CUDA_ERROR_ILLEGAL_ADDRESS errors for model_{2,5}. I have no idea why it happened...
So, I tested many variants of the environment and found out that jax=0.2.17 (probably, it is the same version of the original) and jaxlib=0.1.68+cuda110 (it is the version for installing jax with a command
pip3 install jax[cuda110] -f https://storage.googleapis.com/jax-releases/jax_releases.html ) are okay to run smoothly without Docker, but with my custom conda environment.

Hi, we require version of 0.1.69 jaxlib to be able to use CUDA unified memory for running long sequences. If you don't need this you can probably run with 0.1.68, but that might be related to the illegal address error that you see. How long was the sequence you were trying to run?
Some of the other open issues about CUDA versions might also be of help.

I was benchmarking with the CASP14 targets. T1026 (172 residues) raised the issue.
I realized that some of the targets still have issues of the CUDA_ERROR_ILLEGAL_ADDRESS, even though I used jax==0.2.17 and jaxlib==0.1.68+cuda110. Those targets were running okay on CPUs.
For my system information,

NVIDIA driver: 450.36.06
CUDA version: 11.0
jax: 0.1.68
jaxlib: 0.1.68+cuda110
tensorflow: 2.5.0

That's a very small protein, so I'm surprised it's an issue. What GPU are you using? Is it possible to try using the Dockerfile?
You could try disabling unified memory by commenting out these two lines in your script, if you have them:
https://github.com/deepmind/alphafold/blob/main/docker/run_docker.py#L171-L172

I tested with Quadro RTX 6000 and RTX 2080Ti.
I have tested with
(1) jaxlib==0.1.68+cuda110, jax==0.2.17, cudatoolkit=11.0.3 for my custom non-Docker version
(2) jaxlib==0.1.69+cuda110, jax==0.2.17, cudatoolkit=11.0.3 for my custom non-Docker version
(3) (1) or (2) + commenting out the two lines for the unified memory
(4) the same as (2), but with a docker container (the original one)
There was no issue with the (4)... So, there may be some differences between my non-Docker version and the original Docker version... (I thought I implemented my custom non-Docker version with the exact same version of libraries...) I will try it again.

@huhlim Did you solve your CUDA_ERROR_ILLEGAL_ADDRESS problems?  I just ran ~100 proteins from an internal sample, and this cropped up for me in some cases.   As I investigate, it would be helpful if you follow-up here with anything you learned and/or how you resolved your problem.  (I am using Docker at an A100)

@christroat I could not fully resolve the issue. When I turned off the jax.jit compilation of models (initialization of the RunModel class in alphafold/model/model.py), it reduced the chance of the error but did not resolve the issue. I have not had the issue with my Docker system, so I guess my problem is related to our cluster setup... Unfortunately, I gave up to tackle the issue.


About the return value of the createOrUpdateResource method.

After reading the source code, I found that if the createOrUpdateResource method throws a runtime exception, the event will be rescheduled. If it returns UpdateControl, the corresponding crd resource update will be performed according to the returned value.
So the question is, if I need to update the crd resource status and at the same time need to reschedule the event, what do I need to do?

In the golang version of operator sdk, the Reconcile method has both Result and err return values. The Requeue value of Result is true or err != nil, and the event will be rescheduled.

Can it be understood that if the updateStatus or replace method is called to update the CustomResource, the Operator will receive events from the K8s Watch again?
If this is the case, then the user does not need to actively throw runtime exceptions in the Operator.

The update and rechedule can be done at the same time now using UpdateControl, not it extends BaseControl what has the rescheduleAfter method in v2, but similar exists in v1.


Taint tracking tools instead of DFSan.

Angora uses LLVM DFSan for taint analysis. Even it is the best taint tracking tool I can found, it is not friendly if your tested program has  external libraries. Also, it needs source code to compile. I have tries PIN & Libdft. But it is too slow , and Pin 3.x can't use external libraries and libdft only supports 32bits (Vuzzer has migrated it to 64bits).  Any suggestion?

We have implemented the new feature for supporting libdft64 here. https://github.com/AngoraFuzzer/Angora/blob/master/docs/pin_mode.md


In-the-Wild Mode，The result is very different from the true value

crop_size = (544,960),the cam_in [487,487,479.8,266] ,img_size :[540,960],depth_scale =1 ,Autonomous driving scene

crop_size = (544,960),the cam_in [487,487,479.8,266] ,img_size :[540,960],depth_scale =1 ,Autonomous driving scene

Thank you for reaching out but I think your configuration is correct.
Our suggestion is that you could probably use a scale align factor obtained by rough median match. You may apply this factor for your inference. According to our knowledge, the scale drifts from images by the same camera are quite close.
We are still focusing on this to shrink the scale gap between the zero-shot model and real-world application.


[Zone] Storage Affinity

Background

For split rack cluster setup, ECE action plan will create workload volumes, one per node, and affinity of volumes to nodes and zones is set using preferred owners.

Problem

If VM is associated with a zone., currently container is picked in a round robin fashion.
If volume backing the container is active on another node/zone, then IO performance will be degraded

Proposed solution

For container, read the preferred owner from failover cluster and store the info in container spec
During creation of a VM which is associated with zone, try to pick a container in following order

order:

Container with same preferred owner
Container's preferred owner is in same zone
Round-Robin of any available containers

Changes
Moc

Add 'PreferredOwner' field in Container Info message

Moc-Pkg

Add wrapper function for WMI query to fetch preferred owners for the cluster shared volume from the csv's cluster group

Node-Agent

Add new provider type ' Failover cluster' , which is wrapper on top of default local provider.
This additionally fetches the preferred owner for container from failover cluster layer and updates in the container info spec.

Cloud-Agent

During VM creation, if a VM is associated with Zone, then select a container that has same preferred owner or preferred owner is in same zone.

/azp run


(u,v) coordinate error in meta_data_exporting example

Changing the height and width parameters in the meta data exporting example to have a non-square image results in incorrect meta data images (e.g. depth) being extracted. I am using --width 640 --height 480. In this case depth.png is 480 x 640 image instead of 640 x 480.
For reference here is the RGB image

and the depth image

I believe this is due to an error in reshaping the depth_array in this line. It currently reads
depth_array = np.array(depth_array).reshape(opt.width,opt.height,4)

Changing the order of height and width fixes the issue and produces correct depth images
depth_array = np.array(depth_array).reshape(opt.height,opt.width,4)

Given this I am not sure why the original example worked (even with a square image) since it seems like height and width may still have been swapped in the original example.
Of course the same fix needs to be applied to the other metadata fields as well (normals, positions, segmentations, etc.).

Oof, good catch. I’ll make sure to get this fixed before the next merge into master.
We’ll actually likely be updating this example to use the newer visii.render_to_file API, which supports saving raw segmentations as EXR files with signed 32-bit float channels. There are other examples though where we probably make a similar mistake.

These examples have since been updated to use the newer data exporter, so I'm closing this issue. Thanks for the help


ParamSpec trait should be unsafe

See https://github.com/gtk-rs/gtk-rs/pull/400#discussion_r605392775

Maybe also other traits, like the cairo pattern one.

I do not see any pattern trait in cairo...

Indeed, that's solved differently nowadays. gdk::FromEvent is another one though. I think that's all :)


Wallet stuck on specific height

I stack on specific level of Wallet and doesn't go up.
Also in log nothing about to update Wallet
No -> "Updated wallet peak to height"

I reinstalled the application and I also re-sync the wallet. It always stack when arrive to specific height
After reinstall and re-sync I again see following

Please download last version and wait for sync. We had problem with internet On server tonight. Now all restored and we moving to another data center.--Отправлено из мобильной Яндекс.Почты12.08.2021, 16:52, "GregoryGum" @.***>:
I stack on specific level of Wallet and doesn't go up.
Also in log nothing about to update Wallet
No -> "Updated wallet peak to height"
I reinstalled the application and I also re-sync the wallet. It always stack when arrive to specific height
After reinstall and re-sync I again see following
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

I have re-sync db and wallet or only wallet will be enough?

Try to wallet at first. --Отправлено из мобильной Яндекс.Почты12.08.2021, 17:07, "GregoryGum" @.***>:
I have re-sync db and wallet or only wallet will be enough?
—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.


fix: help view translations

fixes: #1100

LGTM!

is this expected? i felt like it should be bold or sth

@eraygundogmus @dopplerDistortion everything is translated to English as expected except the reasons from the filter dropdown are still in Turkish.

@eraygundogmus @dopplerDistortion everything is translated to English as expected except the reasons from the filter dropdown are still in Turkish.

It is because get the data from backend. Please keep in touch with Hande about this subject and open an issue on backend repository.

https://prnt.sc/-bKagqWZYoL_ Also seeing the labels on the map details as not translated. Is this related?

https://prnt.sc/-bKagqWZYoL_ Also seeing the labels on the map details as not translated. Is this related?

yes, we don't have multilanguage support on backend side yet

"Icerik detaylari" for some reason this is not translated. https://prnt.sc/KPKNXTujX2N5

Also SEVICES is written as SERVİCES

"Az Yogun" ......"Cok Yogun" legend from the manual is also in Turkish

@dopplerDistortion please can you fix the problems above?

Above issues resolved @dopplerDistortion Nice Job!


Update Tuya_ZD08.md and Tuya_WL-TH02.md

Both confirmed to work with ZHA, Tuya_WL-TH02.md also supports humidity, temperature and batterypct.

thanks


Mouse support for games

Would love to see mouse support for games, and not just Microsoft Edge.

There is already mouse and keyboard support for Mouse and Keyboard. You can enable this in the settings under 'Inputs'. Make sure to both select the new gamepad driver and mouse and keyboard. It works best when streaming from your home console and be cautious that it may still be very experimental and things are still being worked on so make sure to be prepared for that.


Workflow ui

Created dynamic workflow UI components.
Added multiple examples to demonstrate how they work.
Built the business logic for parsing a JSON object into a workflow UI.

@enisoc

LGTM


Old lockfile when installing

Steps to replicate:

Run npm install @nasa-jpl/stellar --save as instructed in the README
Encounter npm WARN old lockfile:

npm WARN old lockfile
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile

Suggested fix -- commit the updated package-lock.json and package.json to the repository that is generated, post-install.

This should now be fixed

Cheers. 👍


QMAPS-2068 navigate in app menus through history

Description
Allow the app menu to be closed by navigating back (and also to be remembered on page reload).
For that, store the menu state in the url using the same mechanism as Search: a drawer query string param indicating the currently opened menu panel (aka. drawer)
In Maps, can be app or products.
Required a new event sent by the router on page loads so <Menu> can be notified.
It's still a bit hackish, not declarative enough for React, but I think it's the minimum change for the current app structure.
It gave me clearer ideas to clean up our router/history implementation (for now there are still too many URL manipulations in deep components), ideally migrating to React Router, but let's do that separately.

@xem Thanks for the page load bug!
It happened because the routeChange listener is created after the first event occurs, so I added a case for that in the mounting effect of Menu.


Not able to create Master Key

Hi,
Not Able to create master key. need support

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b415b140-12fa-f613-29c5-4aee3904d4bd
Version Independent ID: 73e88688-5e6d-c69c-28d1-5a036789c494
Content: Tutorial - Perform ETL operations using Azure Databricks
Content Source: databricks/scenarios/databricks-extract-load-sql-data-warehouse.md
Service: azure-databricks
GitHub Login: @mamccrea
Microsoft Alias: mamccrea

@Tapasgt Thank you for reaching out.

How are you trying to create a database Master key? Are you following the instructions provided in the Prerequisites section?
Are you seeing any specific error while trying to create master key? If so could you please share it?

Note: It requires CONTROL permission on the database in order to create a database Master key.
If you are having the issue even after having CONTROL permissions, then I would recommend you to please open an issue in Microsoft Q&A forum with above requested information for further assistance as this issue not a feedback on the doc in reference. This repo is to drive improvements towards MS docs.
Please do share the Q&A thread link for reference once you open an issue. Forums are better suited for such queries. We have our engineers and other community members who will be happy to assist.
Please feel free to comment if you have further queries.
Thank you.

Do you know the t-sql command I can use to determine if I have control permissions?  I followed all the instructions for creating the sql server and sql pool. I'm connecting to the server using the admin account I created during that process so I would think I would have control permissions.

@jasonsue53 Is the SSMS session you are attempting to execute the T-SQL connected to master or was it to the user database and now attempting to create database master key in master?
Please refer to this SO thread to check for permissions: https://stackoverflow.com/questions/7048839/sql-server-query-to-find-all-permissions-access-for-all-users-in-a-database.
I have followed the same tutorial and was able to create database master key. Since this issue more of SQL side and not a databricks side I would recommend you to please open up another issue in the SQL document where it says how to create a database master key
Or please open up up a thread in Microsoft Q&A forum for further assistance.  We have our engineers and other community members who will be happy to assist.
As this is not a feedback on the current documentation, we will now proceed to close this thread. If there are further questions regarding this matter, please continue to comment. We will gladly continue the discussion.
Thank you

Hi
I have followed the same document.
The error says that user do not have permission. Then I tried to give the CONTROL permission to the master database user using sa user, but the error showing that the Grantor do not have permission
Thanks & Regards,
Tapas Guhathakurta │ Assistant General Manager - Enterprise Technology & Digital Transformation│ Embee Software Pvt. Ltd. │ Mob: +91<PHONE_NUMBER> │ www.embee.co.inhttps://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.embee.co.in%2F&data=01|01|Tapas.G%40embee.co.in|1194e174b42145cb3f8c08d702597589|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=9t1jqV7Z%2BahtXSQWkwrZZnME2lbo2yt1fVN5ZO9qwz0%3D&reserved=0
Embee Software Private Limited
My Reporting head: Sudhir Kothari, E-mail: Sudhir.kothari<EMAIL_ADDRESS>Local Address: Infinity Benchmark, 6th Floor : Unit No.: Office 9 : Plot : G1, Block : EP & GP : Sector – V, Salt Lake : Kolkata : 700091 : WB, India.
<EMAIL_ADDRESS>From: KranthiPakala-MSFT<EMAIL_ADDRESS>Sent: 20 August 2020 03:41
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Tapas Guhathakurta<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Not able to create Master Key (#61016)
@jasonsue53https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjasonsue53&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=R3iRGrwGanPjhqv26p12wa%2BL3uVJqhnad5dBe7i1jvQ%3D&reserved=0 Is the SSMS session you are attempting to execute the T-SQL connected to master or was it to the user database and now attempting to create database master key in master?
Please refer to this SO thread to check for permissions: https://stackoverflow.com/questions/7048839/sql-server-query-to-find-all-permissions-access-for-all-users-in-a-databasehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F7048839%2Fsql-server-query-to-find-all-permissions-access-for-all-users-in-a-database&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=wp0N8oClyunZctDzPDM4XGQfU49OgaYaBiGLBGLVUXk%3D&reserved=0.
I have followed the same tutorial and was able to create database master key. Since this issue more of SQL side and not a databricks side I would recommend you to please open up another issue in the SQL document where it says how to create a database master keyhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fsql%2Frelational-databases%2Fsecurity%2Fencryption%2Fcreate-a-database-master-key%3Fview%3Dsql-server-ver15&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=rYgp%2BbQf4gBRPR3RfqsEbUFsYOOxWnMC35EKF5BB6QI%3D&reserved=0
Or please open up up a thread in Microsoft Q&Ahttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fanswers%2Fproducts%2Fazure%3Fproduct%3Dall&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=ms%2F0GAQUHpEgji6lK6E7KNf49srLQegdDUda%2Ffg8zGo%3D&reserved=0 forum for further assistance. We have our engineers and other community members who will be happy to assist.
As this is not a feedback on the current documentation, we will now proceed to close this thread. If there are further questions regarding this matter, please continue to comment. We will gladly continue the discussion.
Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F61016%23issuecomment-676752144&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=qdsi4JO9DW4%2FH15PN7BwawjhD7q6LcUgxn7u5UwFtZA%3D&reserved=0, or unsubscribehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHAVGGXJP3FMAK7R3QSW6NTSBREWBANCNFSM4QB4PHTA&data=01|01|tapas.g%40embee.co.in|26ccbba2d9914000912f08d8448cb7d4|718a99124d8d4d9daf9aabe3dca42cb2|0&sdata=ONj4IMpB3HXRd507IvFy06fyclJe%2FLey36xu%2BvOb5cs%3D&reserved=0.
This email, and any attachment, is confidential to the addressee. If you have received this email and are not an authorized recipient please notify the sender and delete this message from your system. If you are not an authorized recipient you must not use, disclose, distribute, copy, print or rely on this email. Email transmission cannot be guaranteed to be secure, error-free or virus-free. Although Embee Software Pvt. Ltd. routinely screens for viruses you are responsible for checking this email and any attachments for viruses and Embee Software Pvt. Ltd. accepts no responsibility for any damage caused to your systems or for loss of data caused by any virus. Embee Software Pvt. Ltd. does not accept liability resulting from errors or omissions in the content of this message following email transmission. If verification is required please request a hard copy version. If this email is of a personal nature, any views expressed are solely those of the author and are not made in the course of the author's employment with Embee Software Pvt. Ltd.

Hi @jasonsue53 Thanks for your response. Since it is specific to SQL issue, I would recommend you to please post your issue in Microsoft Q&A forum : https://docs.microsoft.com/en-us/answers/topics/azure-sql-database.html, which is a better suited forum for such queries. We have our engineers who will be happy to assist.
Thank you


[cloud_firestore] This plugin is not working correctly on iOS, grpcpp related

This is the error I get when trying to run ios app with cloud_firestore as a dependency.
Xcode build done.                                           214.5s
Failed to build iOS app
Error output from Xcode build:
↳
    2020-08-10 22:26:40.416 xcodebuild[33866:363361]  DTDeviceKit: deviceType from 35c01b97dacc7ab0aa241e20fb2593d9ce6c920f was NULL
    2020-08-10 22:26:40.487 xcodebuild[33866:363369]  DTDeviceKit: deviceType from 35c01b97dacc7ab0aa241e20fb2593d9ce6c920f was NULL
    ** BUILD FAILED **

Xcode's output:
↳
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/common/version_cc.cc:22:10: fatal error: 'grpcpp/grpcpp.h' file not found
    #include <grpcpp/grpcpp.h>
             ^~~~~~~~~~~~~~~~~
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/common/version_cc.cc:22:10: note: did not find header 'grpcpp.h' in framework 'grpcpp'
    (loaded from '/Users/yazeed/development/code/myprojects/flutter_projects/beast/build/ios/Debug-iphoneos/gRPC-C++')
    1 error generated.
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/util/string_ref.cc:19:10: fatal error: 'grpcpp/support/string_ref.h' file not found
    #include <grpcpp/support/string_ref.h>
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/util/string_ref.cc:19:10: note: did not find header 'support/string_ref.h' in
    framework 'grpcpp' (loaded from '/Users/yazeed/development/code/myprojects/flutter_projects/beast/build/ios/Debug-iphoneos/gRPC-C++')
    1 error generated.
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/util/status.cc:19:10: fatal error: 'grpcpp/support/status.h' file not found
    #include <grpcpp/support/status.h>
             ^~~~~~~~~~~~~~~~~~~~~~~~~
    /Users/yazeed/development/code/myprojects/flutter_projects/beast/ios/Pods/gRPC-C++/src/cpp/util/status.cc:19:10: note: did not find header 'support/status.h' in framework
    'grpcpp' (loaded from '/Users/yazeed/development/code/myprojects/flutter_projects/beast/build/ios/Debug-iphoneos/gRPC-C++')
    1 error generated.
    note: Using new build system

Hi @YazeedAlKhalaf
Can you please provide your flutter doctor -v, your pubspec.yaml your flutter run --verbose and a complete reproducible minimal code sample
Thank you

I have figured it out, thanks. It was the cocoa pods cache which was having missing files. I deleted it and ran pod install again. Everything worked as usual

Hi @YazeedAlKhalaf
Thanks given your last message I feel safe to close this issue, if you disagree please write in the comments and I will reopen it.
Thank you


Add note about wildcard certs not covering the bare domain.

Release Playbook

[ ] Rebase against master
[ ] Code review
[ ] Merge into master
[ ] Verify changes at http://site-staging.convox.com
[ ] Promote release on site-production in the convox/production Rack

 Document that ACM wildcard certs do not work for the naked domain


[FIX] RCE using execFile()

Bounty URL: https://www.huntr.dev/bounties/1-npm-effect
⚙ Description *
The effect module was vulnerable against RCE because user inputs were concatenated insecurely and executed with exec()
💻 Technical Description *
I simply replaced exec() with execFile() to avoid that arguments could corrupt the command and inject arbitrary code
🐛 Proof of Concept (PoC) *

Create the following PoC:

// poc.js
var Root= require("effect");
var options  = {
image: "& touch Song"
}
Root.edge(options, function(){});

node poc.js
The file Song is created

🔥 Proof of Fix (PoF) *
Same PoC with fixed version: no Song file created :smile:
👍 User Acceptance Testing (UAT)
All ok 👍

@bbeale :smile:,
thanks for the info.
I just updated as you suggested 👍
Cheers,
Mik

Congratulations Mik317 - your fix has been selected! 🎉
Thanks for being part of the community & helping secure the world's open source code.
If you have any questions, please respond in the comments section. Your bounty is on its way - keep hunting!


Dino plop

Hello! Our official mascot is our lord and savior Orpheus, the dino, who is actually the best mascot in existence.

If we can stick Orpheus literally everywhere on the website that still makes it practically useful, we can show a little mascot pride :D.

The repository https://github.com/hackclub/dinosaurs has soooooooooo many images for use, so feel free to use the most topical dino available!
Required (not really lol) reading about our lord and savior: https://workshops.hackclub.com/orpheus/

This is a "first-timers-only" issue. That means that we are requesting that only those new to open source try this one out! Please ask as many questions as you need, I am here to help you create your first contribution to open-source :D

I would like to work on this one

Closed per #49 :D :D :D


Videos don't load properly on ios

🐛 Bug Report
Environment
iPhone
Steps to Reproduce

Open the app, having a good internet connection
Go to a drill with videos
Try to play the video

Expected Behavior
The videos plays correctly after a short loading time

Seen on a

software version 14.1
Iphone SE

Closing since no one complained in the past 2 years


I can't access the attributes

'm doing a search with overview, the theme goes that when I want to access the attributes it never leaves me and I get error messages, so I don't know if it's because I've been doing something wrong.
$cm = new ClientManager([
                'options' => [
                    'fetch' => \Webklex\PHPIMAP\IMAP::FT_PEEK,
                    'fetch_order' => 'desc',
                    'sequence' => \Webklex\PHPIMAP\IMAP::FT_UID,
                ],
            ]);

            $oClient = $cm->make([
                'host' => 'imap.gmail.com',
                'port' => 993,
                'protocol' => 'imap',
                'encryption' => 'ssl',
                'validate_cert' => true,
                'username' =><EMAIL_ADDRESS>                'password' => '*****',
            ]);

            $oClient->connect();
            $folder = $oClient->getFolder('INBOX');
            $messages = $folder->overview($sequence = "190840:190842");

Because if I get the messages.

But when I go through the data you brought me in the foreach it doesn't let me access your data.
            foreach ($messages as $key => $value) {
                dd($value->getAttributes()["subject"]); //Not working ❌
            }

For example, the attributes that I occupied the most in the last library were getFrom(), getSubject(), getDate(). In this new library or version it is occupied so I understood the $message->getAttributes()["attribute_name"];
Am I doing something wrong?

Hi @alexv96 ,
thanks for your question. I guess something got lost in translation :)
The method Folder::overview() does not return an array of messages. Instead it return an array of header values (an overview). Please use Folder::query() if you want to fetch messages.
Best regards,

Hi @alexv96 ,
thanks for your question. I guess something got lost in translation :)
The method Folder::overview() does not return an array of messages. Instead it return an array of header values (an overview). Please use Folder::query() if you want to fetch messages.
Best regards,


Updates to stable modules based on static analysis

Brought this one in to main as it looks good and wanted to inspect the build errors with the latest set of module versions.


Registries vs Database

[ ]  Content
[ ]  Video Script
[ ]  Video
[ ]  Published on microsite

Video script under review. Dependency on Rajeesh

Video script under review. Dependency on Rajeesh

Content added, a new ticket to be created for video


Reduce build output on Unix systems

Description
Reduces amount of output logs produced by configure and make commands during the build process.
Motivation and Context
Please see corresponding issue #182 for more information.
How Has This Been Tested?
npm install -> produces less output
ZK_INSTALL_VERBOSE=1 npm install -> produces full output
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] I have read the code of conduct.
[x] I have read the contributing guide.
[x] I have updated the documentation accordingly (if applicable).

Very nice! I like the possibility to run npm install in dev mode, with verbose output.


EF flex is not loaded into AutoJCM

EF flex is not loaded into AutoJCM. While I checked Auto JCM and export to UE,

The EF morphs cannot be loaded into UE5 and create the corresponding animGraphs in UE5.
The EF flex is below:
https://www.daz3d.com/easy-flex-muscularity-morphs-for-genesis-8-male-s-and-merchant-resource

Sorry for the delay.  We are still reviewing this issue internally.  Thanks for your patience.


[$500]  Distance - Request with empty distance can be created by changing the link to participant link

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.14-1
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Go to workspace chat > + > Request money.
Go to Distance.
Change the link to https://staging.new.expensify.com/create/request/participants/1/<workspacechat_ID>.
Proceed with distance request creation

Expected Result:
In Step 3, user will be redirected to Request money page
In Step 4, distance request cannot be created as the waypoints are empty
Actual Result:
In Step 3, user lands on workspace selection page
In Step 4, distance request with empty waypoints can be created, but an error shows up after creating it
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/133c9d52-4d03-4158-a662-dca54b34fd1f
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01af0bd38709b573a6
Upwork Job ID:<PHONE_NUMBER>030083584
Last Price Increase: 2023-12-19

Proposal
from: @unicorndev-727
Please re-state the problem that we are trying to solve in this issue.
Request with empty distance can be created by changing the link to participant link.
What is the root cause of that problem?
The root cause is that we didn't validate the transaction in IOURequestStepParticipants.js depending on iouRequestType in transaction object. We only check the transaction if iouRequestType is 'scan' here.
useEffect(() => {
        IOUUtils.navigateToStartStepIfScanFileCannotBeRead(receiptFilename, receiptPath, () => {}, iouRequestType, iouType, transactionID, reportID);
    }, [receiptPath, receiptFilename, iouRequestType, iouType, transactionID, reportID]);

What changes do you think we should make in order to solve the problem?
We need to check if the transaction is eligible to create MoneyRequest and if it is not eligible, we need to go back to RequestMoneyPage.
If the iouRequestType in transaction object is distance, we need to check the waypoints.
And if it is manual, we should check if the amount is not zero.
What alternative solutions did you explore? (Optional)
N/A

Proposal
from: @cooldev900
Please re-state the problem that we are trying to solve in this issue.
Request with empty distance can be created by changing the link to participant link
What is the root cause of that problem?
The root cause is that transaction is already created when we click distance tab in MONEY_REQUEST_CREATE page and we didn't consider visiting participant page by deeplink.
What changes do you think we should make in order to solve the problem?
In my opinion, we need to check if the amount in transaction is zero and if it is, we should let the user return back to the MONEY_REQUEST_CREATE because we don't need to create MoneyRequest with zero amount.
What alternative solutions did you explore?
N/A

Proposal
Please re-state the problem that we are trying to solve in this issue.
In Step 3, user lands on workspace selection page
In Step 4, distance request with empty waypoints can be created, but an error shows up after creating it
What is the root cause of that problem?
In here, MoneyRequestParticipantsPage we don't have logic to go back to previous page if the waypoint is invalid
What changes do you think we should make in order to solve the problem?
https://github.com/Expensify/App/blob/52ecad401123dbeced531d63dc722aa92e84f537/src/pages/iou/request/step/IOURequestStepParticipants.js#L50
In here we should add logic to navigate back on the distance request and manual request:

Distance request: navigate back if there is invalid waypoints
Manual request: navigate back if amount is 0

useEffect(() => {
        if (iouRequestType === CONST.IOU.REQUEST_TYPE.DISTANCE) {

            const waypoints = lodashGet(transaction, 'comment.waypoints', {});
            const validatedWaypoints = TransactionUtils.getValidWaypoints(waypoints);
            const isInvalidWaypoint = lodashSize(validatedWaypoints) < 2;

            if (isInvalidWaypoint) {
                navigateBack();
            }
            return
        }

        if (iouRequestType === CONST.IOU.REQUEST_TYPE.MANUAL &&  lodashGet(transaction, 'amount', 0) === 0) {
            navigateBack()
            return;
        }
        // Note that we can upload receipt on both manual request and scan request
        IOUUtils.navigateToStartStepIfScanFileCannotBeRead(receiptFilename, receiptPath, () => {}, iouRequestType, iouType, transactionID, reportID);
    }, [receiptPath, receiptFilename, iouRequestType, iouType, transactionID, reportID, transaction, navigateBack]);

What alternative solutions did you explore? (Optional)
NA

Proposal
Please re-state the problem that we are trying to solve in this issue.

When entering selecting participants on URL just after requesting money by distance directly from a chat (not from FAB) the page didn’t go back to before page. This happen by scan or manual.
When confirming request money by distance without routes, it is confirmed without routes.

What is the root cause of that problem?

There is no going to back page function when isFromGlobalCreate is false at useEffect src/pages/iou/request/step/IOURequestStepParticipants.js. transaction onyx props has isFromGlobalCreate.
{
    ...
    "isFromGlobalCreate": false
}

If just after requesting from chat, isFromGlobalCreate is false at selecting users page for request money. It is true from request money at FAB.
2. There is no error message while confirming by distance when clicking confirming request button without routes at confirm callback src/components/MoneyTemporaryForRefactorRequestConfirmationList.js. It is because confirming by distance without routes is exceptional case.
What changes do you think we should make in order to solve the problem?

Currently selecting users page for split money is only from FAB. So give a function going back if request is came from workplace(isFromGlobalCreate is false) at src/pages/iou/request/step/IOURequestStepParticipants.js. There is navigateToStartMoneyRequestStep function for IOUUtils which can be used for this case.

    useEffect(() => {
///// below
        If(isFromGlobalCreate) {
        IOUUtils.navigateToStartMoneyRequestStep(iouRequestType, iouType, transactionID, reportID));
}
///// above
        IOUUtils.navigateToStartStepIfScanFileCannotBeRead(receiptFilename, receiptPath, () => {}, iouRequestType, iouType, transactionID, reportID);
    }, [receiptPath, receiptFilename, iouRequestType, iouType, transactionID, reportID]);

If new features for selecting uesrs for split except from FAB are added, then this feature should be edited if it is not from global create(FAB).

Give an error if clicking request button on confirming by distance without routes at src/components/MoneyTemporaryForRefactorRequestConfirmationList.js](https://github.com/Expensify/App/blob/5715b544475c670475ddd09878846639d7e52aa8/src/pages/iou/request/step/IOURequestStepParticipants.js#L50).

                if (isDistanceRequest && isDistanceRequestWithoutRoute) {
                    setFormError(‘iou.error.distanceRoutesRequired`);
                    return;
                }

Since it needs localized text, objects needed to be added at each language file at src/languages/.
Alternatives
N/A

@c3024 proposals for you here!

Asked here in an earlier relevant discussion on Slack regarding why these conditions were removed.

I read and get the discussion. If the first error could not be an error because each steps are separated by URL, should the second error be fixed still? Empty waypoint distance request on chat is an issue.

@NicMendonca @c3024
I added the result video.

Go to Distance.
Change the link to https://staging.new.expensify.com/create/request/participants/1/<workspacechat_ID>.
Proceed with distance request creation

Why would a end-user change the URL in a real life example? I am not sure in what use case this would occur in

@dylanexpensify what do you think of this? I don't see this is something an end-user would actually do!

[Asked here] again on Slack about how we should go about this.
cc: @cead22

@c3024 thanks your bringing that up in slack! Looks like the consensus is to do nothing so I am going to close this for now!