Attach files to email from S3

Add a capability to the current Pepipost Java SDK to attach files to an email from an S3 bucket.
Example: A user might be generating all transactional receipts in PDF format in S3 and those need to be sent through emails.

@geniusdibya please let me know if you still need help with above features? i would be more then happy to help out. Thank you.

Yes @chaudharydeepak we still looking for above feature and you are most welcomed for contribution.

Hi @vikram-sahu , sure! i will take a look at this today and respond with any follow-up questions / possible solutions. At bare minimum, I think the application has to allow authentication to S3 on-behalf of user and using S3 exposed api's download / attach file to email - any size restrictions?

yes, we have to restrict size which can be <= 10MB. And you can reach out to us anytime for any questions or query related to feature in this thread or email us to<EMAIL_ADDRESS>

Thank you VIkram - i sent an email to<EMAIL_ADDRESS>from my personal email account with few questions.

Hi, today I finished basic functionality to allow S3 integration -

defined an interface for 3rd party attachement source provider ( S3, dropbox(may be in future) etc. ).
using aws S3 sdk did integration to fetch required files from defined buckets and attach to the email body. Happy path functionality worked - few more basic cases to consider:
a. size limitations.
b. working with encyrpted files on S3.

I used the S3 sdk provided by AWS. Let me know if this is ok, or should we directly invoke the rest apis without using sdk. Per AWS, using sdk is recommended way.

I forked, see changes here -
https://github.com/chaudharydeepak/pepipost-sdk-java
Files:
com.pepipost.api.attachementProvider.AttachSourceProvider
com.pepipost.api.attachementProvider.S3AttachSourceProvider
com.pepipost.utils.PepipostUtil

Thank you @vikram-sahu . I will go ahead with AWS SDK approach.
S3 allows server-side encryption [ i.e. the files stored encrypted at rest ], as well as client side ecnyrption [ i.e. the files are encrypted before transferring over ]. Do we want to consider these functionalities as well? More info here - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingEncryption.html

For now code changed to reflect for client side encrypted files with following schemes -
AWS KMS
AES Symmetric
RSA pub/pvt
Modified code at -
https://github.com/chaudharydeepak/pepipost-sdk-java

Hi @vikram-sahu , your mentioned case should be mostly covered by scenario of client side encryption - wherein the files were encrypted using symmetric or assymetric cryptography before uploading to S3 by the client. Code is already in place for this to allow the users input AES(symmetric) or RSA(asymmetric) keys for decrypting S3 downloaded content, before attaching to the email. I will test some more cases.
Also, code is already in place for base64 conversion before attaching / transferring. I have been to send / verify multiple emails with attachements etc.
Please check modified code at -
https://github.com/chaudharydeepak/pepipost-sdk-java

some sample code here -
		// 2 files in same bucket. They have no encryption applied.
		List< String > filesList_2 = new ArrayList< String >( );
		filesList_2.add( "BW_MEMO2-typeable.pdf" );
		// filesList_2.add( "Microservices_Patterns_v11.pdf" );
		FileObjects fileObjects_2 = new FileObjects( "myotherbucketforpepi", filesList_2 );
		fileObjects_2.setEncyrptionType( "NONE" );

		// A file in a bucket. Before uploading, the file was encyrpted using
		// AWS KMS key.
		// hence we need respective client to download / decrypt the data before
		// document is email attach ready.
		List< String > kms_encryptedFileList = new ArrayList< String >( );
		kms_encryptedFileList.add( "enc_file_kms.txt" );
		FileObjects kms_enc_files = new FileObjects( "myotherbucketforpepi", kms_encryptedFileList );
		kms_enc_files.setEncyrptionType( "AWSKMS" );
		kms_enc_files.setKms_cmk_id( "7c232025-9498-4959-83e5-153345e50ee5" );

		// A file in a bucket. Before uploading, the file was encyrpted using
		// AES Symmetric key.
		// hence we need respective client to download / decrypt the data before
		// document is email attach ready.
		List< String > aes_encryptedFileList = new ArrayList< String >( );
		aes_encryptedFileList.add( "enc_file_aes256.txt" );
		FileObjects aes_enc_files = new FileObjects( "myotherbucketforpepi", aes_encryptedFileList );
		aes_enc_files.setEncyrptionType( "AES" );
		aes_enc_files.setPath_to_keys( System.getProperty( "java.io.tmpdir" ) );
		aes_enc_files.setAes_master_keyName( "secret.key" );

		// A file in a bucket. Before uploading, the file was encyrpted using
		// RSA Asymmetric keys(pub / private).
		// hence we need respective client to download / decrypt the data before
		// document is email attach ready.
		List< String > rsa_encryptedFileList = new ArrayList< String >( );
		rsa_encryptedFileList.add( "enc_file_rsa10241.txt" );
		FileObjects rsa_enc_files = new FileObjects( "myotherbucketforpepi", rsa_encryptedFileList );
		rsa_enc_files.setEncyrptionType( "RSA" );
		rsa_enc_files.setPath_to_keys( System.getProperty( "java.io.tmpdir" ) );
		rsa_enc_files.setRsa_pub_keyName( "public.key" );
		rsa_enc_files.setRsa_pvt_keyName( "private.key" );

Additionally, possibly, the file can be encrypted using interesting ways ( for ex. password protected pdf / word doc etc ), I think we need to approach this as following -

If we allow file decryption directly from S3( first download file, then decrypt, then base64 encode, then attach), we need to define exactly what kind of decryption we support and how the file originally would have been encrypted - algorithms / crypto tech etc. So, for this I will need exact specifics and we can code accordingly.

Else, since we are building sdk, we simply allow users to download file from S3 ( encrypted or not ) and the users handle decryption on their own as they would know how the file was encrypted originally.

my 2 cents - bottom line, we need to define exact expected funtionality. For now, code is already in place to allow the users input AES(symmetric) or RSA(asymmetric) keys for decrypting S3 downloaded content, before attaching to the email. This works for AWS S3 SDK encrypted files.

Sure @vikram-sahu . how should I approach committing the code? currently i have forked from current and making changes there.
Also, I will write some unit testing with few scenarios.

perfect @chaudharydeepak, you can check out a  new branch name "S3 feature addition"  and commit on that branch once you have completed development you can create a PR after that we can have a QA and merge with master.

@chaudharydeepak great.! let me check and have a look.

@vikram-sahu - any feedback here? Please let me know if there are any review comments. Thank you.

@vikram-sahu - any feedback here? Please let me know if there are any review comments. Thank you.

Hey @chaudharydeepak, we are still in the QA phase will let you know in case of any kind of feedback.

@vikram-sahu got it! Thank you.

@vikram-sahu looks like the changes were pulled in to your master branch?

Hi Sir,
Can you please provide code for sending attachment with pepipost

@vikram-sahu looks like the changes were pulled in to your master branch?

hey @chaudharydeepak, we have successfully deployed your contribution!

Hi Sir,
Can you please provide code for sending attachment with pepipost

amazing! Thank you for the update.
I will be more then happy to help out if anything else is needed in the
future - Thank you for the opportunity!
Best Regards,
Deepak
On Mon, Nov 4, 2019 at 12:37 AM Vikram Sahu<EMAIL_ADDRESS>wrote:

@vikram-sahu https://github.com/vikram-sahu looks like the changes were
pulled in to your master branch?
hey @chaudharydeepak https://github.com/chaudharydeepak, we have
successfully deployed your contribution!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pepipost/pepipost-sdk-java/issues/2?email_source=notifications&email_token=AAHDJVRECPULDLX7PJOMEPTQR67KHA5CNFSM4D7P3ZI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6LDSQ#issuecomment-549237194,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHDJVS5CSJZC72JUNVFGZ3QR67KHANCNFSM4D7P3ZIQ
.

--
Deepak Chaudhary


NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedN…

…iFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath.  Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

The first time I ran the test suite I encountered two test failures, not sure if these are consistent yet:

-------------------------------------------------------------------------------
Test set: org.apache.nifi.tests.system.parameters.ClusteredParameterContextIT
-------------------------------------------------------------------------------
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 75.795 s <<< FAILURE! - in org.apache.nifi.tests.system.parameters.ClusteredParameterContextIT
testProcessorRestartedWhenParameterChangedWhenReferencedThroughEL(org.apache.nifi.tests.system.parameters.ClusteredParameterContextIT)  Time elapsed: 1.455 s  <<< ERROR!
org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error deleting Processor: Node localhost:5672 is unable to fulfill this request due to: fa331065-016d-1000-ffff-ffff873ef21d is running
Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict

-------------------------------------------------------------------------------
Test set: org.apache.nifi.tests.system.rpg.ClusteredRemoteProcessGroupIT
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 181.695 s <<< FAILURE! - in org.apache.nifi.tests.system.rpg.ClusteredRemoteProcessGroupIT
testRPGBackToSelfHttp(org.apache.nifi.tests.system.rpg.ClusteredRemoteProcessGroupIT)  Time elapsed: 120.005 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 2 minutes

I attempted to build/run with Java 11 and ran into a few issues...

It looks like the bootstrap.conf in the node directories has the java binary hard-coded to a JDK 8 path:
java=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin/java
This basically launched NiFi with JDK 8, after I built it with JDK 11, so I got one of those errors about incompatible class version of 55 and can read up to 52.
I commented out the above line in each bootstrap and ran again, this time it now used the same JDK as my shell had which was 11, and this time it got a different error:

java.util.ServiceConfigurationError: org.apache.nifi.authorization.UserGroupProvider: org.apache.nifi.authorization.FileUserGroupProvider Unable to get public no-arg constructor
        at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:581)
        at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:672)
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1232)
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
        at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
        at org.apache.nifi.nar.StandardExtensionDiscoveringManager.loadExtensions(StandardExtensionDiscoveringManager.java:155)
        at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:130)
        at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:116)
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:942)
        at org.apache.nifi.NiFi.<init>(NiFi.java:158)
        at org.apache.nifi.NiFi.<init>(NiFi.java:72)
        at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
        at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
        at java.base/java.lang.Class.getConstructor0(Class.java:3342)
        at java.base/java.lang.Class.getConstructor(Class.java:2151)
        at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:659)
        at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:656)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:667)
        ... 11 common frames omitted
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 19 common frames omitted

I think this is because the assembly for the test suite doesn't have the lib/java11 directory setup with the JAX-B dependencies the way the regular assembly does.

Latest changes look good, going to merge, thanks!


How to jump to a question on the specified page?

If you have a button outside, click the button to jump to a certain question on the specified page, how do you implement it?

@fengzhihenxs please check the code:
survey.currentPageNo = 2 // change current page to 3
survey.currentPage.questions[4].focus() // focus specified question

also check: https://surveyjs.io/Documentation/Library/?id=QuestionSelectBase#focus
https://surveyjs.io/Documentation/Library/?id=surveymodel#currentPageNo

OK, thank you. I realized it in another way.

@fengzhihenxs cool! It would be great if you share your realization here for another users

My train of thought is to find the parent element of question and then use scrollTop.


buf: 1.42.0 -> 1.44.0

https://github.com/bufbuild/buf/releases/tag/v1.44.0
https://github.com/bufbuild/buf/releases/tag/v1.43.0
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

breaks the build on aarch64 https://logs.ofborg.org/?key=nixos/nixpkgs.346804&attempt_id=5fb05a64-26a9-425e-8eae-7f8ab5526e47

It seems to be related to the new WASM support added in buf. Disable checks on Aarch64 for now. See #347207.

> created by github.com/tetratelabs/wazero/internal/wasm.(*ModuleInstance).CloseModuleOnCanceledOrTimeout in goroutine 123
>  /build/source/vendor/github.com/tetratelabs/wazero/internal/wasm/module_instance.go:39 +0xa0
> FAIL      github.com/bufbuild/buf/private/bufpkg/bufcheck 8.695s

I tried again, and using Go 1.23, the build was successful.


Await for valid CFRoute status prior domain deletion

Is there a related GitHub Issue?
https://github.com/cloudfoundry/korifi/issues/2109

What is this change about?
This fixes a flake where the domain routes are not deleted upon deleting
the domain. The reason is that the domain may get deleted before the
controllers k8s client manager notices the CFRoutes that reference the
domain.
By awaiting the valid CFRoute status we ensure that controllers
have reconciled the routes at leas once (i.e. are aware of their
existence)

Does this PR introduce a breaking change?
No

Acceptance Steps
N/A

@danail-branekov is this something that could occur in production somehow?

@danail-branekov is this something that could occur in production somehow?

Probably - the domain finalizer is doing a best effort to delete orphan routes. I think that we already agreed that we would accept potential orphan routes when e.g. an app is deleted.
However, IRL it is not really likely that you create a route and immediately (like in a script) delete its domain


react version conflict

react is specified as a dependency which result in multiple versions of react loaded and give me the following error
Uncaught TypeError: Cannot read property 'firstChild' of undefined
Shouldn't react be specified as a peerdependency and not dependency?

Could #194 have fixed this?

I ran into the same thing today, and running npm dedupe in the project directory resolved this. Probably not the long-term answer, but it should do the trick.

I ran into the same thing today as well. I tried running npm dedupe as suggested by bdanderson, and that fixed it for me (for now) as well.

Ditto on same error, with removing offending file/npm dedupe working.

In 0.7.0 we've changed react to a peer dependency. https://github.com/callemall/material-ui/blob/master/package.json#L36
Hopefully, this solves the issue.

i have the same trouble with 0.7.0 version and npm dedupe solved it.


Javascript Error in dropdown-menu

Discussed in https://github.com/orgs/twbs/discussions/40658

Originally posted by Knud13 July 24, 2024
I've found a small javascript error in the dropdown-menu e.g.

  New around here? Sign up
  Forgot password?

Keyboard Tab into the dropdown menu works but if you then use the arrow keys it will generate an error
>  Uncaught TypeError: Cannot read properties of undefined (reading 'parentNode') dropdown.js:98
    at new qi (dropdown.js:98:34)
    at qi.getOrCreateInstance (base-component.js:65:41)
    at HTMLDivElement.dataApiKeydownHandler (dropdown.js:419:31)
    at HTMLDocument.n (event-handler.js:118:19)

you can reproduce the error on the bootstrap docs page
https://getbootstrap.com/docs/5.3/components/dropdowns/#forms
just navigate to the "New around here? Sign up" dropdown-item via keyboard so it has focus and use the arrow keys.
I assume the dropdown-item is looking for a parent  since most dropdown-item are in the format

New around here? Sign up

Forgot password?

Thanks for reporting this issue @Knud13
It's not easy to find, but this is actually a duplicate of https://github.com/twbs/bootstrap/issues/37428 (mentioned in https://github.com/twbs/bootstrap/issues/37428#issuecomment-2025934169 for instance). The error only happens in the documentation that shows the dropdown menus with their dropdown button, just to see easily the rendering; it won't happen in the real life.
https://github.com/twbs/bootstrap/pull/37755 might fix this issue.

Thanks Julien, seems you are correct. In my situation we were using the dropdown-menu as a menu on the page without a toggle button. It appears you need to use a dropdown-menu in a dropdown. I could reproduce the error on the demo page obviously as it's just showing the layout as an example.


Unable to convert the decoded data into readable format

Hi,
My approach to this problem might be wrong so I'll add the code snippet here.
layer_image = layer_count.as_PIL()
item['dat'] = str((base64.b64decode((layer_image.tobytes()).decode("utf-8"))))
print (item['dat'])
I know that the decoded data is in Unit8Array format.
I am trying to convert the layer image data to a readable format.
This is the traceback:
File "C:\Python34\lib\site-packages\django\core\handlers\base.py" in get_response
  132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "E:\myproject\myproject\myapp\views.py" in list
  85. item['asdf'] = str((base64.b64decode((layer_image.tobytes()).decode("utf-8"))))

Exception Type: UnicodeDecodeError at /myapp/list/
Exception Value: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

@kmike Found a way. Can you please close this issue. And thank you for this amazing tool. Still need to explore a lot.

Hey @pythonl1,
Ok, closed!


Parse Error When Trying To Use theme() Inside calc()

I'm using Tailwind CSS with PostCSS via Gulp. Just about everything works great. I'm trying to reference values from the config theme object in my CSS. I'm able to do the following just fine:
height: theme('spacing.12');

However, when I try to use theme() inside of calc() (just like in the documentation):
height: calc(100vh - theme('spacing.12'));

I get a Parse error from gulp-postcss ("Expecting 'SUB', 'LPAREN'....")

Can't seem to reproduce here, works on tailwind.run too:
https://tailwind.run/Wuixda
This must be an issue with another tool in the chain so I'm going to close as likely not anything we can fix in Tailwind, but if you provide a GitHub repo that reproduces the issue I'm happy to pull it down and troubleshoot it.

Can't seem to reproduce here, works on tailwind.run too:
https://tailwind.run/Wuixda
This must be an issue with another tool in the chain so I'm going to close as likely not anything we can fix in Tailwind, but if you provide a GitHub repo that reproduces the issue I'm happy to pull it down and troubleshoot it.

 In my gulp file, I was passing the postcss-cssnext node package into the postcss() call - turns out postcss-cssnext is now deprecated (oops!) - once I removed that, the problem was fixed. Sorry for the false alarm - thanks for pointing me in the right direction!

Take a look at postcss-preset-env if you want/need to write futuristic css.


Data page

Everything on the data page should be classified based on whether or not we wish to continue producing it or not. Everything we plan to continue producing needs to be linked instead of broken.
UNCLASSIFIED

DW-NOMINATE Program With Examples
DW-NOMINATE Probabilities for Legislator Choices: 1st to 112th Congresses (huge file, not sure if we should maintain? People can get this through the main interface if they really want to)

PRODUCE EVERY UPDATE

Percent Voting on the Winning Side by Member -- Houses/Senates 1 - 112 (memberCSV)
Party Unity Scores by Democrat and Republican Members: 1857 - 2012 (memberCSV)
Common Space DW-NOMINATE Scores 1st to 112th Congresses (memberCSV, rollcallCSV)
Cutting Line Angle Files for the House and Senate 1st to 112th Congresses (rollcallCSV)
Party and Chamber Medians, 1 - 112 Congresses (DW-NOMINATE Scores) (partiesCSV)
Party Unity Scores by Congress: 1879 - 2012 (partiesCSV)
Political Polarization Measures: 1879 - 2012 (either as a polarization Rmd or a congress-chamber level csv?)
ORD file for present congress

PRODUCE EVERY SESSION OF CONGRESS

DW-NOMINATE Scores 1st to 112th Congresses (fixed after the end of the last congress, only updated every two years)
One-Congress-at-a-Time DW-NOMINATE (Nokken-Poole) data for the House and Senate 1st to 112th Congresses
ORD files for past congresses

MAINTAIN ONLY AS LEGACY KEITH FILES

Rank Orderings all Houses and Senates -- 1st to 112th Congresses
Clausen, Peltzman, and Issue codes for 1st to 113th Congresses

Static text files that will never change should be put in /static/data/. Legacy files should be put in /static/data/legacy/. Ongoing files should output to /static/data/current/. I will write a cronjob to automagically back up the current folders each week.
The template itself is /var/www/voteview/views/data.tpl. Feel free to edit it and commit changes.
Please use comments to either suggest placement for these files, place them, or note when you've created a script to output them.

This will be handled after the schema change (Issue #22).

I'm going to make my own todo list that I will edit here. I added names so I could tick them off on the list above.
I'm also going to edit your post to reflect our discussions about this page.

[ ] rollcallCSV - CSV files for rollcalls (by congress-chamber), need cutting angles, otherwise done
[ ] memberCSV - CSV files for members (by congress-chamber)
[ ] icpsrCSV - CSV files for icpsr (by congress-chamber)
[ ] partyCSV - CSV file for parties (one file where rows are party-chamber-congress and one file that is over whole db)
[ ] ORD files for past a present congresses
[ ] Polarization .Rmd

Then I will add legacy support that continues to output old .DAT files for legislators and rollcalls, but nothing else. We should encourage everyone to move to .csv

[ ] rollcallDAT
[ ] memberDat

This data page is set up. I will open a new issue with enhancements and ideas going forward, but the basic data is here and available.


Creating a timeout to finish the scan screen

I'm and my friend djalmaafilho, do this feature, we would like if you can merge this to all.
This is a new feature to set a long timeout to finish the scan intent on handler timeout.
Thanks in advance.

Thanks, looks good.
For interest's sake, what is your use case for this (why do you need the timeout)?

Hi @rkistner ,
it's a desire of my costumer, because some ice foods can't be recognized, like icecream...
And after this time, my app change the scan to input manually the EAN barcode. :D

Hi @rkistner ,
it's a desire of my costumer, because some ice foods can't be recognized, like icecream...
And after this time, my app change the scan to input manually the EAN barcode. :D

Sorry, I've clicked on wrong button ;)


cannot get to run on 23.04

So, I downloaded everything and followed all the instructions to install, but it will not install for me on Ubuntu 23.04.  Has anyone been able to install portable on 23.04, and am I missing a step from that is outside of the instructions?

Fixed via https://github.com/wimpysworld/obs-studio-portable/tree/r23172


Remove MBEDTLS_ENABLE_WEAK_CIPHERSUITES configuration option

Rationale
Remove all weak ciphersuites -- These suites are dangerous to use and nobody should be using them (upgrade to a more secure ciphersuite).

Work items for 3.0

Remove all references to MBEDTLS_ENABLE_WEAK_CIPHERSUITES.

Note: weak ciphersuites have already been removed in a previous PR, so all that's left to do is to remove the option.


Fix broken skipWhen on the e2e tests

Something must have changed in the new version of Jasmine.
The skipWhen plugin recurses through the child specs and suites and marks then as disabled.

LGTM


Fixed sort algorithm

See #59 for details.
I had to switch to the latest 0.5.x jest branch in order to make it run correctly.
Anyway, tests are still missing for this functionality...

Thanks! I cherrypicked your change to the library. It's included in the latest beta. You can install it using npm i reactabular@beta. Let me know if you encounter any issues. :+1:

All right! :+1:


Until 2021 data problem

gpt 3.5 has a problem with only having data until 2021. As far as I know, the web has solved this with a plug-in. I wonder if this can't be done in the API. I am Korean, if I ask who the president is gpt will answer the 2021 president, not the current one.

i have to wait until the gpt-4 API with plugin is public so i can check their documentation


Improve binary run safety

WHY are these changes introduced?
Fixes https://github.com/Shopify/develop-app-inner-loop/issues/120
WHAT is this pull request doing?
More details in the issue
Measuring impact
How do we know this change was effective? Please choose one:

[x] n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
[ ] Existing analytics will cater for this addition
[ ] PR includes analytics changes to measure impact

Checklist

[x] I've considered possible cross-platform impacts (Mac, Linux, Windows)
[x] I've considered possible documentation changes

/snapit


"Cannot read properties of undefined (reading '0')" inside modal structure

Hi,
after the issue have been fixed in #74 i've tried to send the modal over, that worked but right after im seeing the modal, discord-modals gives an error in the mapping array (this is only an error on the bot side, nothing wrong happening on discord or anything), while it still shows fine, the error should not happen.
Error:
Generated Modal
Sended Modal
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: Cannot read properties of undefined (reading '0')
    at /home/container/node_modules/discord-modals/src/structures/Modal.js:42:96
    at Array.map (<anonymous>)
    at new Modal (/home/container/node_modules/discord-modals/src/structures/Modal.js:42:40)
    at showModal (/home/container/node_modules/discord-modals/src/structures/ShowModal.js:66:10)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
TypeError: Cannot read properties of undefined (reading '0')
    at /home/container/node_modules/discord-modals/src/structures/Modal.js:42:96
    at Array.map (<anonymous>)
    at new Modal (/home/container/node_modules/discord-modals/src/structures/Modal.js:42:40)
    at showModal (/home/container/node_modules/discord-modals/src/structures/ShowModal.js:66:10)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

At the top the "Generated Modal" and "Sended Modal" should only be a reference at when it happens
Full code of the Modal: https://pastebin.com/aXhG3Kqb
Im using both latest version of discord.js and discord-modals

I hope this is solved, was fixed in Discord-Modals v1.3.8.


[vsts release definition] Fixing helptext

The example for "starting with" search has been removed. However, the instruction still says "Limit results to definitions with this name or starting with this name." The highlighted section has to be removed if not supported.

Starting with this name actually is apt.
PS C:\Users\desattir> vsts release definition list --name N -o table
  ID  Name                         CreatedBy         Created On
----  ---------------------------  ----------------  --------------------------------
   7  aks - Service account tests  Deepak Sattiraju  2018-11-06T06:30:44.773000+00:00
   2  New release pipeline         Deepak Sattiraju  2018-08-28T10:40:51.810000+00:00
   3  New release pipeline (1)     Deepak Sattiraju  2018-10-09T08:24:02.200000+00:00
   4  New release pipeline (2)     Deepak Sattiraju  2018-10-12T08:38:44.370000+00:00
   5  New release pipeline (3)     Deepak Sattiraju  2018-10-22T04:31:57.250000+00:00
   6  New release pipeline (4)     Deepak Sattiraju  2018-11-01T08:41:50.893000+00:00
   8  New release pipeline (5)     Deepak Sattiraju  2018-11-15T06:08:04.463000+00:00

@geverghe This is the function being used: https://github.com/Microsoft/azure-devops-python-api/blob/dev/vsts/vsts/release/v4_0/release_client.py#L330
It's just a different convention, FabCi is equivalent to *FabCi* for this API call.


Remove Procfile and other hosting files

Repo clones and pulls fail because of the 'Procfile' file and there are other files that are only applicable if you are using a specific hosting service.

Repo clones and pulls fail because of the 'Procfile' file

This was because the file somehow had a space on the end that could not be removed through GitHub by editing the file but would register in the file URL (Discord-bot/blob/main/Procfile/Procfile%20) and would prevent the file from being pulled


[BACKLOG-27325] - SWT 4.9 - Window Database Connection lost Focus

@pentaho/kashyyyk please review


StartLimitedTask will duplicate the instances

Using the function above I ran into the issue that a non elevated task would start a duplicate instance.
Im assuming the issue is that withing the method as it first starts a process and then also starts another task:

By removing the code marked as (1) the issue was resolved.

Or adding a "return" after the Process starts does also the job :)


[BUG] Resolve backwards compatibility TypeError

Closes #3414
Hello!
Description
DatasetConfig.from_json is now a classmethod like you would expect. That said, I can't make a nice test for this, as now it fails with:
ValidationError: 4 validation errors for DatasetConfig
questions -> 0
  Discriminator 'type' is missing in value (type=value_error.discriminated_union.missing_discriminator; discriminator_key=type)
questions -> 1
  Discriminator 'type' is missing in value (type=value_error.discriminated_union.missing_discriminator; discriminator_key=type)
questions -> 2
  Discriminator 'type' is missing in value (type=value_error.discriminated_union.missing_discriminator; discriminator_key=type)
questions -> 3
  Discriminator 'type' is missing in value (type=value_error.discriminated_union.missing_discriminator; discriminator_key=type)

Still, at least it can actually parse the file with this PR in place.
Type of change

[x] Bug fix

How Has This Been Tested
Running
from argilla.feedback import FeedbackDataset, TrainingTaskMapping

dataset = FeedbackDataset.from_huggingface(
    repo_id="argilla/stackoverflow_feedback_demo"
)

as well as
pytest .\tests\client\feedback\test_config.py::test_dataset_config_json_deprecated

Checklist

[ ] I added relevant documentation
[x] follows the style guidelines of this project
[x] I did a self-review of my code
[ ] I made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I filled out the contributor form (see text above)
[x] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

Tom Aarsen

#3416 also fixes this, and I prefer that one.


Update tutorial-automate-vm-deployment.md

A few corrections having run through the lab.  (Good lab btw.)
Make sure VM names are consistent between commands.
Ensure output read into variables is in the right format in case user had az configured to default output to table, jsonc, yaml, etc.

@richeney : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@richeney Thank you for the contribution :)


Follow-up on analog info

Description and motivation
#316 introduced an analog info functionality. Going forward, we might want to enhance it further by adding the following aspects:

Add some more summary info like remaining digital-MACs, total number of analog tile mat-vecs
User-definable "columns" to select only some of the columns to display
Improve on layout, e.g. the (- - - ) first entry of the tile section is a little confusing
Count also the peripheral digital ops (e.g noise management, bound management)
Currently only forward pass is counted, but backward / update would be interesting to add for analog training
Add other specifics for analog training with analog optimizers (tiki-taka / mixed precision) which need additional tiles internally and additional digital ops

Hi @IHIaadj will you be able to add these enhacements.

@IHIaadj can you please let us know if you are interested in incorporating some of this feedback to analog info.


fix(Header): subheader and image shorthand

The Header icon/image requires the content div.  The content div should contain all text.
Image
The image shorthand currently does not wrap the content shorthand in HeaderContent.
Subheader
Header subheaders should be rendered as HeaderContent children.  Currently, the subheader shorthand is rendered as a sibling when there is an icon/image.  Example here only shows an icon, but the issue exists for subheaders when there is an icon or an image.

Outputs:

foo

bar

It should output:

    foo

bar

I've marked this help wanted, but I actually need it now.  PR coming.


[Question]:  Groq: client.chat.completions.create function location.

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
Hi, I am trying to get a full understanding on how function a function call message si done, in particular when using Groq with a FunctionCalling agent created in the following way. I want to understand what is the structure of the message passed to the most basic llm agent in the chain.
where the message is passed to
from llama_index.llms.groq import Groq
llm = Groq(
    model="llama3-groq-70b-8192-tool-use-preview",
    api_key=GROQ_API_KEY,
    temperature=0
)

agent_worker = FunctionCallingAgentWorker.from_tools(
    [multiply_tool, add_tool, subtract_tool, query_tool],
    llm=llm,
    verbose=True,
    allow_parallel_tool_calls=False,
    system_prompt=(
       ""
    ),
)

I was able to trace the user input up until the file
https://github.com/run-llama/llama_index/blob/67c7e50e782f9ce12e1fd76b4ac3a131a505f19b/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py
Up to the function call on line  429
response = client.chat.completions.create(
                messages=message_dicts,
                stream=False,
                **self._get_model_kwargs(**kwargs),
            )

however I am unable to find where the definition of  client.chat.completions.create is when I am using a groq model. If I try to print in python the location of the function through
def _get_client(self) -> SyncOpenAI:
        print("_get_client is running")
        if not self.reuse_client:
            print("not reuse client")
            return SyncOpenAI(**self._get_credential_kwargs())

I get sent to an openai module. This one ( on my local compyter it is not a link)
lib/python3.11/site-packages/openai/_utils/_utils.py
which is not part of llamaindex and it is also not Groq.
Can you please point me to the correct source code that handles the user message with function call for through tool use with Groq please?

That is not the information I am looking for. I want the source code of client.chat.completions.create for the particular case of using Groq API through llamaindex

The FunctionCallingAgentWorker uses the lower level llm.chat_with_tools() and llm.get_tool_calls_from_response() functions
For groq, we can see that it its just a light wrapper around the openai llm
https://github.com/run-llama/llama_index/blob/be7f61aade0e1506b718560a713c163184ce6abc/llama-index-integrations/llms/llama-index-llms-groq/llama_index/llms/groq/base.py#L7
Looking at the openai llm class, we can see that the tools are prepared here
https://github.com/run-llama/llama_index/blob/be7f61aade0e1506b718560a713c163184ce6abc/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py#L823
Which that method is called from the base class here
https://github.com/run-llama/llama_index/blob/be7f61aade0e1506b718560a713c163184ce6abc/llama-index-core/llama_index/core/llms/function_calling.py#L40
And then tool calls are parsed from the response here
https://github.com/run-llama/llama_index/blob/be7f61aade0e1506b718560a713c163184ce6abc/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py#L877

Thank you for the quick response. I was actually able to find those files.
I was trying to dig deeper into the function call
response = client.chat.completions.create(
                messages=message_dicts,
                stream=False,
                **self._get_model_kwargs(**kwargs),
            )

found in file (429)
llama_index/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py
I am interested in looking at the prompt that ultimately gets sent to the groq API, when it is no longer formated for llamaindex structure and classes. I am interested in that piece of text.  But I am having trouble tracing back that function `client.chat.completions.create. If I try to print its location with

 import inspect
            print(inspect.getfile(client.chat.completions.create))

I get sent to https://github.com/openai/openai-python/blob/main/src/openai/_utils/_utils.py.  Which clearly is not the place for the groq API call which i am looking for.
Any help with this would be greatly appreciated.
Cheers


Interview of ID Lock fails

So I tried to include my Remotec ZRC-90, "Scene master", (yes, I do have a lot of Z-Wave devices ;)) but the inclusion fails pretty quickly. Based on the log file attached I don't even think it's being interviewed. I attached my cache-files too, there's nothing in them though.
remoteczrc90_log.txt
eabbf325.txt
eabbf325.metadata.txt
eabbf325.values.txt

I also tried to include Everspring ST812, the inclusion seems to work but got "the node is unknown or not initialized yet, discarding..." there too. Do you want me to open a separate issue for that one?

Since both issues seem to affect the node inclusion, you can leave it here.

but the inclusion fails pretty quickly. Based on the log file attached I don't even think it's being interviewed.

It seems that the inclusion went through but the interview was not started afterwards although it should be. You can probably work around it by restarting your application until I've found the source of the problem.

Here's the files for Everspring ST812. When I restart and then manually wake up Remotec, the interview process gets completed.
e81a9a19.metadata.txt
e81a9a19.txt
e81a9a19.values.txt
everspring_st812_inclusion.txt

Issue with inclusion of ID Lock too, don't know if it's related to the above though. Attached log files.
idlock_inclusion.txt

That seems to be something else. The device is included and the interview starts immediately, but then it fails to respond to some messages. First it doesn't respond to SecurityCCCommandsSupportedGet which causes the driver to think it is not secure (which is wrong because it responsed to nonce requests). Then it doesn't respond to Version requests (because the VersionCC is probably only supported securely).
However I'm not seeing the security bootstrapping process in the log. Could it be that the device was previously included securely but you did an insecure inclusion? In that case you might have to factory-reset the device.

Alright! This was actually the first inclusion attempt after I opened the box, but let me try to reset it and try again.

So I did a hard reset of my PC Z-Wave module and then did a hard reset of the Z-Wave module on the ID Lock. Still the same, attaching the inclusion log and all the cache-files.
c3ed057b.metadata.txt
c3ed057b.txt
c3ed057b.values.txt
idlock_inclusion.txt

This time I'm seeing the bootstrapping process succeed. But the device does something weird as soon as the driver tries to communicate securely. The normal flow is
--> Get Nonce
<-- Nonce Report
--> Encrypted Request
<-- Nonce Request
--> Nonce Report
<-- Encrypted Response

What I'm seeing is:
--> Get Nonce
<-- Nonce Report
--> Encrypted Request
... nothing for one second ...
<-- Encrypted Response (unknown nonce)

I'm not sure whats going on and where the nonce is coming from that the device uses to encrypt the response. The encrypted response is discarded because the driver did not create any nonce for that exchange.

Is there anything I can do to debug more? I tried to include it in a friends Fibaro Home Center and it worked fine.

There's no way to see the communication logs in Home Center, is it?

Nope, they have a basic "Inclusion started", "Interviewing device", etc but nothing more... I guess it would be possible to try open-zwave, could that help somehow?

The OZW log would probably help, yeah!

Here's the log file from OpenZWave, hope it helps!
idlock_openzwave.txt

In this log, the node asks the controller for a nonce. Very weird. Wonder why it didn't in the other log.
Maybe try re-interviewing the node in zwave-js (or delete the cache)?

So I ran zwave-js and here's the output from the interview process. I used the same security network key in both zwave-js and OpenZWave.
idlock_interview.txt

Hm that looks better - maybe a temporary hiccup?
The interview mostly goes through, except for the User Code CC which does something it shouldn't.
12:35:12.501 CNTRLR   [Node 002] UserCodeCC: doing a complete interview...
12:35:12.502 CNTRLR » [Node 002] querying capabilities...
12:35:12.504 CNTRLR   [Node 002] Error during node interview: Node #2 does not support the command C
                      apabilitiesGet!

Well, yes, the only difference is that I didn't include it in zwave-js, I did that in openzwave.

Could you try doing the same again in zwave-js now? Wonder if it has to do with the inclusion process.

You mean include it with zwave-js? Reset the lock, reset the z-wave stick and try the inclusion?

yup

Still the same problem, tried multiple
idlock_inclusion_new.txt
times.

Did you do a complete reset? The device does not respond to the NetworkKeySet command, after which the driver must treat it as a non-secure device. The weird thing is that it asks for a nonce to send a reply, but that reply never arrives.
I'll take a look at what OZW does differently.

Yes, did a reset of both the lock and the stick. The only difference I can tell without looking at the code is that in openzwave, you pass true as an argument when doing the inclusion to enable the security inclusion process while in zwave-js it's managed automatically.

I'll do a 1:1 comparison of the data exchange later to see if there is any difference.

So, I've analyzed the first few messages in both inclusion logs (OZW and the last zwave-js):
zwave-js

Inclusion:

NodeFound
AddingSlave
ProtocolDone
Done

Security Bootstrap

 --> SchemeGet (0)
<--  SchemeReport (0)
 --> NonceGet (for NetworkKeySet)
<--  NonceReport
 --> CommandEncapsulation > NetworkKeySet
<--  NonceGet (for the response to NetworkKeySet!)
 --> NonceReport
... nothing ...

OZW

Inclusion:

NodeFound
AddingSlave
ProtocolDone
Done

Security Bootstrap

 --> SchemeGet (0)
<--  SchemeReport (0)
 --> NonceGet (for NetworkKeySet)
<--  NonceReport
 --> CommandEncapsulation > NetworkKeySet
<--  NonceGet (for the response to NetworkKeySet!)
 --> NonceReport
<--  A message that OZW does not understand.
     zwave-js parses it as NetworkKeyVerify with the configured key (the expected response to NetworkKeySet)

The only difference I see is that OZW requires an acknowledgement for the NonceReport and does not use the "Explorer Frame" option. I'm not entirely sure why that would make a difference, but we can try that with a test build I can prepare later tonight.

Version 5.3.2-alpha.0 will be available in a couple of minutes. Please try the inclusion with that version again and create a log. The nonce reports are now transmitted using the same options that OZW uses, so there should be no more difference in communication.

I just tried and still the same issue... I think. Did a full reset of the stick and lock and also had the lock 20-30 cm from the z-wave stick when doing the inclusion just to be sure that there were no problems with range. Attached the log.
idlock_inclusion_alpha0.txt

The security bootstrap process succeeds now:
20:48:47.236 DRIVER « [Node 002] [REQ] [ApplicationCommand]
                      └─[SecurityCCCommandEncapsulation]
                        │ sequenced: false
                        └─[SecurityCCNetworkKeyVerify]

But we're back to this problem https://github.com/zwave-js/node-zwave-js/issues/1037#issuecomment-709296495 now.
Looking at the OZW log it seems that it also receives an invalid response to that command, but chooses to ignore it.
I'm working on a PR to make the interview less strict in many cases, this should be one of them.

This should be fixed in v5.3.2

Alright, tried all three devices in 5.3.2. Remotec, Everspring and ID-Lock. I was not able to include neither of them. Both Remotec and Everspring did same as last time. No interview for Remotec at all, and for Everspring the interview is being made but still says "unknown node" after it's done. I did a reset of everything before I began.
everspring_st812_inclusion.txt
idlock_inclusion.txt
remotec_inclusion.txt

Ohhh, I think I found whats going wrong with the remotec. It gets added as a controller, not as a slave (like all the devices I've seen so far). As a result, a variable does not get set. I'm tracking that part in https://github.com/zwave-js/node-zwave-js/issues/1069 now.
ID-Lock now gets past the inclusion process but does the same it did earlier during the inclusion. It responds without requesting a new nonce, so the driver does not know the nonce it used for encryption. I'll have to check if it is reusing previous nonces or something.
The everspring device has me confused. Directly after the inclusion, the node should be added to controller.nodes with the information you see in the log:
09:26:09.795 CNTRLR   finished adding node 3:
                        basic device class:    Static Controller
                        generic device class:  Alarm Sensor
                        specific device class: Routing Alarm Sensor
                        supported CCs:
                        · Basic (0x20)
                        [...]

But the message "the node is unknown or not initialized yet" implies that the nodeid is not found in controller.nodes. The only location where one can be deleted is after exclusion.
Can you check if the "node removed" is emitted some time during the attempted inclusion?

Alright, worth noting about Remotec is that if I first include it, restart zwave-js and then waking it up, it gets interviewed and works like it should (Central Scene commands gets handled by zwave-js)

Ok, let me know if there's something I can do.

Everspring

"node removed" is not emitted.
The node is present in the cache-files
For debug I put a console.log on driver.controller.nodes during the interview-process and the node is present there too, during and after the interview.
I did notice it always fails on the same interview step:

20:46:10.190 CNTRLR » [Node 003] querying current value for Water Leak... 20:46:10.229 SERIAL » 0x010a001303039c0105250853                                          (12 bytes) 20:46:10.231 DRIVER » [Node 003] [REQ] [SendData]                       │ transmit options: 0x25                       │ callback id:      8                       └─[AlarmSensorCCGet]                           sensor type: Water Leak 20:46:10.236 SERIAL « [ACK]                                                                   (0x06) 20:46:10.241 SERIAL « 0x0104011301e8                                                       (6 bytes) 20:46:10.242 SERIAL » [ACK]                                                                   (0x06) 20:46:10.244 DRIVER « [RES] [SendData]                         was sent: true 20:46:10.258 SERIAL « 0x0107001308000002e1                                                 (9 bytes) 20:46:10.259 SERIAL » [ACK]                                                                   (0x06) 20:46:10.262 DRIVER « [REQ] [SendData]                         callback id:     8                         transmit status: OK 20:46:11.874 CNTRLR   [Node 003] did not respond after 1/3 attempts. Scheduling next try in 500 ms. 20:46:12.389 SERIAL » 0x010a001303039c0105250853                                          (12 bytes) 20:46:12.392 DRIVER » [Node 003] [REQ] [SendData]                       │ transmit options: 0x25                       │ callback id:      8                       └─[AlarmSensorCCGet]                           sensor type: Water Leak 20:46:12.398 SERIAL « [ACK]                                                                   (0x06) 20:46:12.405 SERIAL « 0x0104011301e8                                                       (6 bytes) 20:46:12.407 SERIAL » [ACK]                                                                   (0x06) 20:46:12.410 DRIVER « [RES] [SendData]                         was sent: true 20:46:12.420 SERIAL « 0x0107001308000002e1                                                 (9 bytes) 20:46:12.422 SERIAL » [ACK]                                                                   (0x06) 20:46:12.425 DRIVER « [REQ] [SendData]                         callback id:     8                         transmit status: OK 20:46:12.743 SERIAL « 0x010d00040003079c02000500000069                                    (15 bytes) 20:46:12.749 SERIAL » [ACK]                                                                   (0x06) 20:46:12.754 DRIVER « [REQ] [ApplicationCommand]                       └─[AlarmSensorCCReport]                           sensor type: Water Leak                           alarm state: false 20:46:12.765 DRIVER     the node is unknown or not initialized yet, discarding...

Remotec:
Then it has probably two modes of inclusion: Secondary controller and/or slave - could that be?
Regarding the everspring device:
Could you set a breakpoint here and inspect controller.nodes as well as the detected node ID?
I'm also a bit stumped by the fact that this code is even hit during the interview. The AlarmSensorCCReport should be matched to the previous AlarmSensorCCGet.

Remotec: No, I don't think so. According to the manual this is the way to do it. I did try a few different ways but none worked.
Everspring: attached two print screens from the debugger output

Huh, nodeId: 0...
That seems like a bug in AlarmSensorCCReport (or the node sends corrupted Alarm Sensor commands).

Yup, that's a bug in the CC implementation (or I misunderstood the specs)

Tracking that bug in #1072, so the only thing left here is the ID Lock and its strange nonce usage.

So, I've taken a closer look at the idlock log. The device does reuse nonces, which is clearly forbidden by the Z-Wave specs:

The driver sends a nonce (id 0x7f)

09:44:18.789 DRIVER » [Node 002] [REQ] [SendData]
                      │ transmit options: 0x05
                      │ callback id:      10
                      └─[SecurityCCNonceReport]
                          nonce: 0x7f216b115b3055b3

The device sends a message using that nonce

09:44:18.845 SERIAL « 0x01260004000220988126fbb14a2e48ea38e65073649a6585730fe5a7e7617f451 (40 bytes)
                                                                                ^^
                      445eff88623eb25
09:44:18.845 SERIAL » [ACK]                                                                   (0x06)
09:44:18.846 DRIVER « [Node 002] [REQ] [ApplicationCommand]
                      └─[SecurityCCCommandEncapsulation]
                        │ sequenced: false
                        └─[SecurityCCCommandsSupportedReport]
                            reportsToFollow: 0
                            supportedCCs:
                            · Version
                            · Door Lock
                            · User Code
                            · Association
                            · Association Group Information
                            · Notification
                            · Firmware Update Meta Data
                            · Battery
                            controlledCCs:

a bunch of unencrypted communication

The driver does an encrypted request

09:44:18.954 DRIVER » [Node 002] [REQ] [SendData]
                      │ transmit options: 0x25
                      │ callback id:      13
                      └─[SecurityCCCommandEncapsulation]
                        │ nonce id: 47
                        └─[VersionCCGet]

The device responds, but reuses the same nonce it has used above

09:44:19.835 SERIAL « 0x0125000404021f9881a22080f5d642ed82adc5ac6ad6606740027d0f197f62985 (39 bytes)
                                                                                ^^
                      ea9a71dbc5cac
09:44:19.836 DRIVER   Message with invalid data received. Dropping it

Seems that the ID lock supports firmware updates - unfortunately, I can't see the current version in either OZW or here, since in both cases the VersionReport cannot be decoded. The changelog is pretty vague regarding the Z-Wave fixes.
Could you ask the support if reusing S0 nonces is something that was fixed in an update?

Yeah, the lock supports firmware and the version on my lock is 1.5.6, which is the current version. I can try to contact them tomorrow. I did try to include the lock in a Homey and that worked like it should. I don't know if the log files can help but here's from the inclusion in my Homey.
idlock_homey_inclusion.txt

I'm not too familiar with the Homey log format, but I think that the same is happening there too - Homey just doesn't care like OZW.
If I'm correct, the device seems to reuse nonces after an unencrypted message exchange (lines 433ff):
# Node requests a new nonce
[2020-10-25T20:17:44.752Z] Node[3]: Received application command for COMMAND_CLASS_SECURITY, data: 0x40
[2020-10-25T20:17:44.752Z] Node[3]: [COMMAND_CLASS_SECURITY] {}
[2020-10-25T20:17:44.754Z] Node[3]: [COMMAND_CLASS_SECURITY] [SECURITY_NONCE_REPORT] {"type":"Buffer","data":[17,220,166,43,149,123,179,127]}
[2020-10-25T20:17:44.755Z] Node[3]: sendData to COMMAND_CLASS_SECURITY, params 0x8011dca62b957bb37f
[2020-10-25T20:17:44.756Z] Command[56] start: sendData
[2020-10-25T20:17:44.757Z] ProcessSendData[55]: To node: 3 with data: 0x988011dca62b957bb37f and txOptions: AUTO_ROUTE,EXPLORE
[2020-10-25T20:17:44.783Z] Command[56] end: sendData

# Node responds with that nonce (ID 17)
[2020-10-25T20:17:44.814Z] Node[3]: Received application command for COMMAND_CLASS_SECURITY, data: 0x814adb0c27e5605f8ae809754ccf115d812be98a9bdd7d
[2020-10-25T20:17:44.814Z] Node[3]: [COMMAND_CLASS_SECURITY] {"Initialization Vector byte":{"type":"Buffer","data":[74,219,12,39,229,96,95,138]},"Properties1 (Raw)":{"type":"Buffer","data":[232]},"Properties1":{"Sequence Counter":8,"Sequenced":false,"Second Frame":true},"Command byte":{"type":"Buffer","data":[9,117,76,207]},"Receivers nonce Identifier (Raw)":{"type":"Buffer","data":[17]},"Receivers nonce Identifier":17,"Message Authentication Code byte":{"type":"Buffer","data":[93,129,43,233,138,155,221,125]}}
[2020-10-25T20:17:44.836Z] Node[3]: Decapsulated frame from COMMAND_CLASS_SECURITY to COMMAND_CLASS_VERSION, data 0x148001
[2020-10-25T20:17:44.836Z] Node[3]: [COMMAND_CLASS_VERSION] {"Requested Command Class (Raw)":{"type":"Buffer","data":[128]},"Requested Command Class":128,"Command Class Version (Raw)":{"type":"Buffer","data":[1]},"Command Class Version":1}
[2020-10-25T20:17:44.840Z] Command[4] status: addNode, status: INTERVIEW_CC_15_OF_15

# Controller sends an unencrypted CC
[2020-10-25T20:17:45.357Z] Command[4] status: addNode, status: INTERVIEW_MANUFACTURER_SPECIFIC
[2020-10-25T20:17:45.404Z] Node[3]: sendData to COMMAND_CLASS_MANUFACTURER_SPECIFIC, params 0x04
[2020-10-25T20:17:45.405Z] Command[57] start: sendData
[2020-10-25T20:17:45.406Z] ProcessSendData[56]: To node: 3 with data: 0x7204 and txOptions: AUTO_ROUTE,ACK,EXPLORE
[2020-10-25T20:17:45.460Z] Command[57] end: sendData

# Node replies unencrypted
[2020-10-25T20:17:45.478Z] Node[3]: Received application command for COMMAND_CLASS_MANUFACTURER_SPECIFIC, data: 0x05037300030001
[2020-10-25T20:17:45.498Z] Node[3]: [COMMAND_CLASS_MANUFACTURER_SPECIFIC] {"Manufacturer ID":{"value":883},"Product Type ID":{"value":3},"Product ID":{"value":1}}
[2020-10-25T20:17:45.503Z] Command[4] status: addNode, status: INTERVIEW_FIRMWARE_INFORMATION

# Controller requests a new nonce for the upcoming secure request
[2020-10-25T20:17:45.504Z] Node[3]: sendData to COMMAND_CLASS_VERSION, params 0x11
[2020-10-25T20:17:45.505Z] Node[3]: [COMMAND_CLASS_SECURITY] [SECURITY_NONCE_GET] {"type":"Buffer","data":[]}
[2020-10-25T20:17:45.506Z] Node[3]: sendData to COMMAND_CLASS_SECURITY, params 0x40
[2020-10-25T20:17:45.507Z] Command[58] start: sendData
[2020-10-25T20:17:45.509Z] ProcessSendData[57]: To node: 3 with data: 0x9840 and txOptions: AUTO_ROUTE,ACK,EXPLORE
[2020-10-25T20:17:45.544Z] Command[58] end: sendData
[2020-10-25T20:17:45.562Z] Node[3]: Received application command for COMMAND_CLASS_SECURITY, data: 0x80d7fbddf6da7d4dfd
[2020-10-25T20:17:45.562Z] Node[3]: [COMMAND_CLASS_SECURITY] {"Nonce byte":{"type":"Buffer","data":[215,251,221,246,218,125,77,253]}}

# Controller does an encrypted request
[2020-10-25T20:17:45.585Z] Node[3]: [COMMAND_CLASS_SECURITY] [SECURITY_MESSAGE_ENCAPSULATION] {"type":"Buffer","data":[45,89,69,38,210,34,22,220,195,191,202,215,144,172,220,243,21,218,144,71]}
[2020-10-25T20:17:45.586Z] Node[3]: sendData to COMMAND_CLASS_SECURITY, params 0x812d594526d22216dcc3bfcad790acdcf315da9047
[2020-10-25T20:17:45.586Z] Command[59] start: sendData
[2020-10-25T20:17:45.587Z] ProcessSendData[58]: To node: 3 with data: 0x98812d594526d22216dcc3bfcad790acdcf315da9047 and txOptions: AUTO_ROUTE,ACK,EXPLORE
[2020-10-25T20:17:45.628Z] Command[59] end: sendData

# Node responds encrypted (twice [!], with the already-used nonce 17)
[2020-10-25T20:17:46.471Z] Node[3]: Received application command for COMMAND_CLASS_SECURITY, data: 0x8180e207f5acd7d18eb445cc4453360a527268ec54113df90e5cc7dc5503
[2020-10-25T20:17:46.471Z] Node[3]: [COMMAND_CLASS_SECURITY] {"Initialization Vector byte":{"type":"Buffer","data":[128,226,7,245,172,215,209,142]},"Properties1 (Raw)":{"type":"Buffer","data":[180]},"Properties1":{"Sequence Counter":4,"Sequenced":true,"Second Frame":true},"Command byte":{"type":"Buffer","data":[69,204,68,83,54,10,82,114,104,236,84]},"Receivers nonce Identifier (Raw)":{"type":"Buffer","data":[17]},"Receivers nonce Identifier":17,"Message Authentication Code byte":{"type":"Buffer","data":[61,249,14,92,199,220,85,3]}}
[2020-10-25T20:17:46.714Z] Node[3]: Received application command for COMMAND_CLASS_SECURITY, data: 0x813b50bc404902fee7a071a64417b1ae82a68ba41d11d310d9bb7e119897
[2020-10-25T20:17:46.714Z] Node[3]: [COMMAND_CLASS_SECURITY] {"Initialization Vector byte":{"type":"Buffer","data":[59,80,188,64,73,2,254,231]},"Properties1 (Raw)":{"type":"Buffer","data":[160]},"Properties1":{"Sequence Counter":0,"Sequenced":false,"Second Frame":true},"Command byte":{"type":"Buffer","data":[113,166,68,23,177,174,130,166,139,164,29]},"Receivers nonce Identifier (Raw)":{"type":"Buffer","data":[17]},"Receivers nonce Identifier":17,"Message Authentication Code byte":{"type":"Buffer","data":[211,16,217,187,126,17,152,151]}}

I'm certain that this is a bug in the device. Other drivers just don't care, but reusing nonces is a security risk (putting aside that S0 is less secure than S2 anyways).
Please let me know if the manufacturer is going to do something about this. If not, I can implement a flag to disable nonce expiration for this device only, so that you can at least use it.

Okay, so I tried to contact ID Lock support 3 days ago (monday) and still no reply... could we do a workaround to disable the nonce expiration until they reply? I did read somewhere that even if the lock is updateable, the Z-Wave module is not, so it's possible that this is the only solution either way.

@slangstrom The linked PR should fix your issue. Would you like an alpha release to test?

@slangstrom The zwave module is possible to firmwareupgrade: https://idlock.no/wp-content/uploads/2019/11/OTA-guide-v1.4.pdf
I have upgraded both my idlock 150 to 1.6 firmware.

The zwave module is possible to firmwareupgrade:

Note: you don't need any extra software, you don't have to unpair the devices. Just upgrade from zwave-js

@AlCalzone That is pure awesomeness! I will test it for sure on other devices.

@roflmao Also zwavejs2mqtt allows to do that using the UI :)

@AlCalzone Trying to add my idlock 150 with the latest 1.6.0 firmware (Node 028)
zwave-1.log

Uh that doe

@roflmao can you try the fix-id-lock branch?

@AlCalzone Tried latest master:
zwave-1.log

Hmm this issue is like whack-a-mole:
23:30:45.451 CNTRLR   [Node 028] trying to load device config
23:30:45.452 CNTRLR   [Node 028] no device config loaded

@AlCalzone I dont seem to find any config in the zwave-js repository either..

There is one for specific firmware versions and #1212 adds a fallback one so the version CC interview can be completed:
https://github.com/zwave-js/node-zwave-js/pull/1212/files#diff-30921b89cc20aec16480118e42f425219ba2c86b65e1866bde169bd3965aba58

Right.
How about a log from a complete OZW interview?

Not necessary. Loading the device config file must succeed, so the rest of the interview can continue. I can't reproduce locally though - my test passes 🤔

Can you please check if you have the file config\devices\0x0373\id-150_compat.json in the @zwave-js/config module? And if id-150_compat.json appears in the config/devices/index.json?

Oh, i just broke index.json because of https://github.com/zwave-js/node-zwave-js/pull/1236.

If the files is there, running yarn run config -i should include it in the index - so all good.

It exists on master at least:
https://github.com/zwave-js/node-zwave-js/blob/master/packages/config/config/devices/index.json#L15053-L15059
https://github.com/zwave-js/node-zwave-js/blob/master/packages/config/config/devices/0x0373/id-150_compat.json

zwave-1.log

I think we got further this time. However you're running into the same issue as https://github.com/zwave-js/node-zwave-js/issues/1219. The device does respond, but the driver is too impatient.
You could try increasing the timeout by editing the zwavejs2mqtt sources locally:
https://github.com/zwave-js/zwavejs2mqtt/blob/306b3803d31e13e6dbdd8c0780740552135497b1/lib/ZwaveClient.js#L1295-L1298
--> add an option
      this.driver = new Driver(this.cfg.port, {
        cacheDir: storeDir,
+       timeouts: { report: 10000 }, // or even 15000
        networkKey: networkKey
      })

Looks like the same response as before:
zwave-1.log

@roflmao With latest version now you don't need to edit the code but you can inject settings from settings.json, check readme, there is an hidden setting in zwave that allows to extend ZwaveOptions

@roflmao The device simply does not respond to most requests... Aside from the security nonces, I see a total of 2 requests answered 🤷🏻♂
Could you make me an OZW_Log after all?

Upgraded to latest zwavejs2mqtt:dev tag on docker (1.0.0-alph2 with zwave-js 5.7.0 iirc) and suddenly it succeeds:
zwave-1.log

I still see a shitton of Message with invalid data received. Dropping it. Could you send me your network key, so I can try to investigate whats going on?

Ok, seems like my original fix does not quite do it. These devices reuse the nonces until they request a new one.

@roflmao please give https://github.com/zwave-js/node-zwave-js/pull/1262 a test :)

@roflmao I've gone ahead and merged the PR. Let me know if the issue persists.

@roflmao I've gone ahead and merged the PR. Let me know if the issue persists.

@AlCalzone Thanks, i will try to test it soon and get logs

@AlCalzone Thanks, i will try to test it soon and get logs

@AlCalzone  Interview completes! I havent been able to obtain the logs yet in bin nor store.

@AlCalzone  Interview completes! I havent been able to obtain the logs yet in bin nor store.

:)
alpha.1 was bugged, alpha.2 should write the log again.

:)
alpha.1 was bugged, alpha.2 should write the log again.

I realize I'm a bit late to this issue, hope this is still relevant. While the ID Lock in question does seem to misbehave, it doesn't seem like the problem is it reuses NONCEs. Reusing NONCEs would be a serious security issue that would call into question the reliability of the product. The product's security implementation seems sound. It does, however, seem run into issues that cause it to send the "weird" message that shows up in the logs here as "containing invalid data" and getting discarded by the driver.
As far as I can tell, the issue arises from sending a secured VERSION_GET command to the door lock, before its Lifeline association has been configured. The door lock makes an audible beep when this happens, similar to the sound it makes when the batteries are put it, which leads me to think it's running into some error internally. As a result of receiving the secured VERSION_GET, the door lock transmits a secured VERSION_REPORT (without asking for a new NONCE) using a NONCE it had received previously from the sender and already used to encrypt a previous reply. However, the transmission is sent out as a BROADCAST, not as a singlecast reply to the sender - again hinting at the device being in some error condition. While it's an issue, it seems this is a bug specific to the VERSION_GET command when the Lifeline association isn't configured, and not a problem with the device's Security S0 implementation,
So how do other Z-Wave Controllers get around this and work with this door lock? My guess is they're not "getting around" anything. It's probably a combination of two things:

They're following the recommended steps that the Z-Wave Plus Role Type Specification lays out for interviewing devices after inclusion. After the protocol inclusion is completed, and security is negotiated, the first thing a controller should do with a Z-Wave Plus device is setup its Lifeline association. In this case, that is sufficient to not trigger the bug in the door lock (see traces below). I would guess neither the manufacturer nor the Certification tests caught this bug, since their interview process follows the guidelines. I also think this is reasonable, finding a bug like this would require testing far too many possibilities of interview processes.
Even if they run into the same situation as here, they don't consider missing version information to be catastrophic and halt the interview process. The version information isn't received, they might retry once or twice, but then they move on and accept that the information is missing. While having the information is nice, not having it is fine. Sure, they won't be able to display it, and they won't be able to apply quirk fixes at higher layers that apply only to specific versions, but the device still gets included and operates.

Finally, here's two traces of inclusion for the same door lock. The only difference between them is setting up the Lifeline association. When that happens, early in the process, the device's bug doesn't get triggered and it replies fine to the VERSION_GET command. Note also that, in both situations, there's a lot of secured traffic before the call to VERSION_GET - the door lock follows the S0 specification, always requesting fresh NONCEs from the sender before replying - with the exception of the VERSION_REPORT broadcast described above.
id_lock_150_fail.txt
id_lock_150_pass.txt

I realize I'm a bit late to this issue, hope this is still relevant. While the ID Lock in question does seem to misbehave, it doesn't seem like the problem is it reuses NONCEs. Reusing NONCEs would be a serious security issue that would call into question the reliability of the product. The product's security implementation seems sound. It does, however, seem run into issues that cause it to send the "weird" message that shows up in the logs here as "containing invalid data" and getting discarded by the driver.
As far as I can tell, the issue arises from sending a secured VERSION_GET command to the door lock, before its Lifeline association has been configured. The door lock makes an audible beep when this happens, similar to the sound it makes when the batteries are put it, which leads me to think it's running into some error internally. As a result of receiving the secured VERSION_GET, the door lock transmits a secured VERSION_REPORT (without asking for a new NONCE) using a NONCE it had received previously from the sender and already used to encrypt a previous reply. However, the transmission is sent out as a BROADCAST, not as a singlecast reply to the sender - again hinting at the device being in some error condition. While it's an issue, it seems this is a bug specific to the VERSION_GET command when the Lifeline association isn't configured, and not a problem with the device's Security S0 implementation,
So how do other Z-Wave Controllers get around this and work with this door lock? My guess is they're not "getting around" anything. It's probably a combination of two things:

They're following the recommended steps that the Z-Wave Plus Role Type Specification lays out for interviewing devices after inclusion. After the protocol inclusion is completed, and security is negotiated, the first thing a controller should do with a Z-Wave Plus device is setup its Lifeline association. In this case, that is sufficient to not trigger the bug in the door lock (see traces below). I would guess neither the manufacturer nor the Certification tests caught this bug, since their interview process follows the guidelines. I also think this is reasonable, finding a bug like this would require testing far too many possibilities of interview processes.
Even if they run into the same situation as here, they don't consider missing version information to be catastrophic and halt the interview process. The version information isn't received, they might retry once or twice, but then they move on and accept that the information is missing. While having the information is nice, not having it is fine. Sure, they won't be able to display it, and they won't be able to apply quirk fixes at higher layers that apply only to specific versions, but the device still gets included and operates.

Finally, here's two traces of inclusion for the same door lock. The only difference between them is setting up the Lifeline association. When that happens, early in the process, the device's bug doesn't get triggered and it replies fine to the VERSION_GET command. Note also that, in both situations, there's a lot of secured traffic before the call to VERSION_GET - the door lock follows the S0 specification, always requesting fresh NONCEs from the sender before replying - with the exception of the VERSION_REPORT broadcast described above.
id_lock_150_fail.txt
id_lock_150_pass.txt


Migrate assets to the resources directory

Larval conventions have raw assets being placed under the resources directory. We should stick to conventions wherever possible.

Also, we should try and migrate away from bower and rely solely on npm. Most of the packages we use are on npm and I'd rather rely on one service than two.


ShellyPlus.groovy - No Temp or Humidity Support for Add-On with DHT22 Sensor

Sys.GetConfig returns "addon_type": "sensor," but this doesn't appear to have support for the SensorAddon.GetPeripherals API to identify the device attached to the Add-On module and the ID values of its component sensors, or the Temperature.GetStatus?id={} and Humidity.GetStatus?id={} APIs to identify the current reading of those sensors - temperature and humidity, in this case. Looks like there was support with the Gen 1 APIs, but those appear to have been omitted here for some reason.

Hello!
Gen2 (RPC based) is not backwards compatible with Gen1 though some of the most common calls are still supported.
https://shelly-api-docs.shelly.cloud/gen2/Addons/ShellySensorAddon

The problem I'm trying to point out here is that there's zero support for the Shelly Plus Add-On in the ShellyPlus (Gen2) Hubitat driver. The original (Gen1) Shelly-as-a-Switch driver supports external temperature and humidity attributes from external sensors (see change 3.0.7), but there's no analogous functionality with the ShellyPlus (Gen2) driver.
I have a Shelly Plus 2PM with a Shelly Plus Add-On and DHT22 temp/humidity sensor connected, but the Add-On and DHT22 sensor are invisible to Hubitat with the current Gen2 driver.

That's because I was the only coder for the hubitat drivers and when I retired last year no one wanted to take over. I had 3 strokes and simply cannot do this work anymore. So that's the scoop.... sorry.

Hi Scott - I hope you're feeling better!
Hi Travis, nice to e-meet you - I'm looking for community developers to assist with the drivers, but there have been no takers yet.

Hi Scott - I hope you're feeling better!
I'm doing much better... leaving the US for Germany soon.


Form checkbox label text alignment

Summary
Adding css themeable variables for label-text for mobile
Additional Details
Added vertical-align css variable so that the label text is aligned inline to checkbox
Thanks for contributing to Terra.
@cerner/terra
Deployment URL
https://terra-core-deployed-pr-1245.herokuapp.com/

Looks good. Can we set --terra-form-checkbox-is-mobile-label-text-vertical-align: middle on the mock theme? That way consumers viewing site can see the centered alignment when triggering a mobile resize.

Is the intent of the style to vertical align the label text optically in the middle next to the checkbox. If so, I don't believe this style is achieving that.

seems like the changes work for me.

Here is the deployment link with changes pointed to my forked terra-core:
https://pages.github.cerner.com/terra/cerner-consumer-theme/build/checkbox-update/#/site/components/site/form-checkbox/index

@praneethm381 That deployment link looks good to me. Looks like we won't get perfectly vertical centered with the height of checkbox, but the consumer font size for the label text on mobile makes it look close enough for me.

I think I'll let @neilpfeiffer make the official terra-core approval, but this works for me (oneup to @bjankord about still not getting perfect alignment, but it's better)

This looks like it is no longer needed.


Copy class/property attributes or decorate generated class with custom attributes

My source class looks like this:
[GenerateSerializer]
public class Criteria
{
[Id(0)]
public string Program { get; set; }
}
Is it possible to configure generation of DTO to include those attributes?

Please use Stack Overflow for usage question, thank you :)


feat(Readme): Appended details and general description [#3]

You need to merge @omerasipi's fix to make the pipes go brrrrr


FEAT: Add max line limit

Feat:  Add max line limit to help with loading times

Code complete. WIll push with next udpate.

Pushed to DEVELOP.

added with v 4.5
https://github.com/Monitorr/logarr/releases


Added neDeepLink and noDelete on EntityOptions level. Fix no delete m…

Fixed noDelete options mistake
added possibility to specify noDelete, noDeepLink on entity level for multy entity screens

@sergeysenja1992 @ishkurko plz review


vmap is incompatible with custom pytree

The following repro code
from jax import tree_util, lax, vmap, numpy as np, jit, pmap

class Special(object):
    def __init__(self, x, y):
        shape = lax.broadcast_shapes(np.shape(x), np.shape(y))
        self.x = np.broadcast_to(x, shape)
        self.y = np.broadcast_to(y, shape)

def special_flatten(v):
    return ((v.x, v.y), None)

def special_unflatten(aux_data, children):
    return Special(*children)

tree_util.register_pytree_node(Special, special_flatten, special_unflatten)

def f(x):
    return Special(x, x)

assert jit(f)(np.ones(3)).x.shape == (3,)
assert lax.map(f, np.ones(3)).x.shape == (3,)
assert pmap(f)(np.ones(3)).x.shape == (3,)
vmap(f)(np.ones(3))  # fail!

triggers the error
 Details
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-e7c65535399a> in <module>
     18 assert jit(f)(np.ones(3)).x.shape == (2, 2, 3)
     19 assert lax.map(f, np.ones(3)).x.shape == (2, 3, 2)
---> 20 vmap(f)(np.ones(3))

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/api.py in batched_fun(*args)
    768     _ = _mapped_axis_size(in_tree, args_flat, in_axes_flat, "vmap")
    769     out_flat = batching.batch(flat_fun, args_flat, in_axes_flat,
--> 770                               lambda: flatten_axes(out_tree(), out_axes))
    771     return tree_unflatten(out_tree(), out_flat)
    772

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/interpreters/batching.py in batch(fun, in_vals, in_dims, out_dim_dests)
     32   # executes a batched version of `fun` following out_dim_dests
     33   batched_fun = batch_fun(fun, in_dims, out_dim_dests)
---> 34   return batched_fun.call_wrapped(*in_vals)
     35
     36 @lu.transformation_with_aux

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/linear_util.py in call_wrapped(***failed resolving arguments***)
    152     while stack:
    153       gen, out_store = stack.pop()
--> 154       ans = gen.send(ans)
    155       if out_store is not None:
    156         ans, side = ans

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/interpreters/batching.py in _batch_fun(sum_match, in_dims, out_dims_thunk, out_dim_dests, *in_vals, **params)
     57     out_vals = yield (master, in_dims,) + in_vals, params
     58     del master
---> 59   out_dim_dests = out_dim_dests() if callable(out_dim_dests) else out_dim_dests
     60   out_dims = out_dims_thunk()
     61   for od, od_dest in zip(out_dims, out_dim_dests):

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/api.py in <lambda>()
    768     _ = _mapped_axis_size(in_tree, args_flat, in_axes_flat, "vmap")
    769     out_flat = batching.batch(flat_fun, args_flat, in_axes_flat,
--> 770                               lambda: flatten_axes(out_tree(), out_axes))
    771     return tree_unflatten(out_tree(), out_flat)
    772

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/api_util.py in flatten_axes(treedef, axis_tree)
    106   # TODO(mattjj,phawkins): improve this implementation
    107   proxy = object()
--> 108   dummy = tree_unflatten(treedef, [object()] * treedef.num_leaves)
    109   axes = []
    110   add_leaves = lambda i, x: axes.extend([i] * len(tree_flatten(x)[0]))

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/tree_util.py in tree_unflatten(treedef, leaves)
     68     structure described by `treedef`.
     69   """
---> 70   return treedef.unflatten(leaves)
     71
     72 def tree_leaves(tree):

<ipython-input-36-e7c65535399a> in special_unflatten(aux_data, children)
      9
     10 def special_unflatten(aux_data, children):
---> 11     return Special(*children)
     12
     13 tree_util.register_pytree_node(Special, special_flatten, special_unflatten)

<ipython-input-36-e7c65535399a> in __init__(self, x)
      3 class Special(object):
      4     def __init__(self, x):
----> 5         self.x = np.broadcast_to(x, (2,) + np.shape(x))
      6
      7 def special_flatten(v):

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/numpy/lax_numpy.py in broadcast_to(arr, shape)
   1280 def broadcast_to(arr, shape):
   1281   """Like Numpy's broadcast_to but doesn't necessarily return views."""
-> 1282   arr = arr if isinstance(arr, ndarray) else array(arr)
   1283   shape = canonicalize_shape(shape)  # check that shape is concrete
   1284   arr_shape = _shape(arr)

~/miniconda3/envs/pydata/lib/python3.6/site-packages/jax/numpy/lax_numpy.py in array(object, dtype, copy, order, ndmin)
   2070       return array(onp.asarray(view), dtype, copy)
   2071
-> 2072     raise TypeError("Unexpected input type for array: {}".format(type(object)))
   2073
   2074   if ndmin > ndim(out):

TypeError: Unexpected input type for array: <class 'object'>

The issue here is we are applying np.broadcast_to to an "object" x (instead of a ndarray) when unflatenning the tree under vmap. Is there any workaround available in JAX?

You need a "direct" constructor for unflatten that doesn't apply broadcast_to but rather just assigns the value.
You can use Python tricks to skip __init__ when constructing objects, but the more idiomatic option is to keep class constructors minimal, just doing variable assignment, and to make another dedicated constructor (make_special below) for typical use:
from jax import tree_util, lax, vmap, numpy as np, jit

class Special(object):
    def __init__(self, x):
        self.x = x

def make_special(x):
    return Special(np.broadcast_to(x, (2,) + np.shape(x)))

def special_flatten(v):
    return ((v.x,), None)

def special_unflatten(aux_data, children):
    return Special(*children)

tree_util.register_pytree_node(Special, special_flatten, special_unflatten)

def f(x):
    return make_special(x)

assert jit(f)(np.ones(3)).x.shape == (2, 3)
assert lax.map(f, np.ones(3)).x.shape == (3, 2)
assert vmap(f)(np.ones(3)).x.shape == (3, 2)

(This split between user facing and lower level constructors is pretty common, e.g., it's what NumPy for its ndarray class with array() for construction/coercion.)

By the way, one clue that your original unflatten is doing something really strange is that merely adding jit is changing the output:
>>> f(np.ones(3)).x
DeviceArray([[1., 1., 1.],
             [1., 1., 1.]], dtype=float32)
>>> jit(f)(np.ones(3)).x
DeviceArray([[[1., 1., 1.],
              [1., 1., 1.]],

             [[1., 1., 1.],
              [1., 1., 1.]]], dtype=float32)

You probably don't want that!

one clue that your original unflatten is doing something really strange

Yeah, that's a bad example. I just updated the code to illustrate the issue better. The result is consistent now. jit, map, pmap work as expected but vmap failed.

You need a "direct" constructor for unflatten that doesn't apply broadcast_to but rather just assigns the value.

Thanks! Unfortunately, I need some logic in the constructor to simplify the broadcasting code in other methods of a pytree class. Actually, we want to convert our distributions in NumPyro to pytree to support scan. And we want to allow constructors such as Normal(loc, 1) where loc can have an arbitrary shape. The code Normal(loc, np.ones(np.shape(loc))) is a bit ugly if we incorporate that restriction.

Like I said, you can use some Python magic for an alternative constructor that skips __init__:
from jax import tree_util, lax, vmap, numpy as np, jit, pmap

class Special:
    def __init__(self, x, y):
        shape = lax.broadcast_shapes(np.shape(x), np.shape(y))
        self.x = np.broadcast_to(x, shape)
        self.y = np.broadcast_to(y, shape)

    @classmethod
    def restore(cls, x, y):
        obj = object.__new__(cls)
        obj.x = x
        obj.y = y
        return obj

def special_flatten(v):
    return ((v.x, v.y), None)

def special_unflatten(aux_data, children):
    return Special.restore(*children)

tree_util.register_pytree_node(Special, special_flatten, special_unflatten)

def f(x):
    return Special(x, x)

assert jit(f)(np.ones(3)).x.shape == (3,)
assert lax.map(f, np.ones(3)).x.shape == (3,)
assert vmap(f)(np.ones(3)).x.shape == (3,)

I don't really recommend it as a design pattern (cheap constructors that just do assignment/validation are generally preferred, for this among other reasons), but it works if you need it for backwards compatibility. If doing it from scratch, I would consider having a normal() function that constructs a Normal() object.

Oh, good idea!! I believe we can use your trick to create a temporary instance (with unbroadcasted arguments) and invoke __init__ again after scan. Thanks a lot, @shoyer !

I think this issue is resolved, but if not please let us know @fehiepsi ! By the way, it's great to hear from you again :D


Improved anamnese report

bring the current, collected data up2date. got effect on free text notes as well as TCM properties. dont forget to migrate old data (dont leave invalid sql codes).

[x] incorporate different sources to create new model
[x] update pages sheet (anamnese and adendum sheet)
[x] flyway migration step to update xprop keys/values
[x] update gadsu view (the anamnese sheet should be 1:1 reflected by gadsu's input fields; keep order)

done


fix: make poetry an experimental detector

#167 enabled the poetry detector by default. We don't want to globally enable it yet, but instead move from an opt-in to an experimental detector.

Verification failures are expected as we disabled the poetry detector.


Documentation error

What steps will reproduce the problem?
Navigate to https://www.yiiframework.com/doc/guide/2.0/es/security-auth-clients
What is the expected result?
The related documentation
What do you get instead?
A Not Found (#404) error
Additional info

Q
A

Yii version
2.0

PHP version

Operating system

Hm, weird, this chapter is marked with TBD. @samdark some website issue?

Fixed.


[Bug] Cannot rewind subgraph with multiple entries

Bug report
I have a subgraph with multiple entries that I want to rewind
root@goldskyraw-indexer-0-graph-node-7d99457bd5-tl7f2:/# graphman info QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf
name      | cly2om8v1y9pa01zw0d2eh5vr
status    | current
id        | QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf
namespace | sgd57856
shard     | second
active    | true
chain     | artela-betanet
node_id   | paused
----------+-------------------------------------------------------------------
name      | cly70thk886zb011s2i49cnzy
status    | current
id        | QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf
namespace | sgd57856
shard     | second
active    | true
chain     | artela-betanet
node_id   | paused
----------+-------------------------------------------------------------------
name      | clxv9e0ksx4a801tef6j17rel
status    | current
id        | QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf
namespace | sgd57856
shard     | second
active    | true
chain     | artela-betanet
node_id   | paused
----------+-------------------------------------------------------------------
name      | clxk54zxw3y7a010lh7xtgb1w
status    | current
id        | QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf
namespace | sgd57856
shard     | second
active    | true
chain     | artela-betanet
node_id   | paused

When I try to rewind it, I get the following error
Error: Multiple deployments found for the search : QmQzA62X9YpN7CVL3UaPhCVBWnwGiFsq395fPrkaa9T2Gf. Try using the id of the deployment (eg: sgd143) to uniquely identify the deployment

This error happen regardless of whether I use the IPFS hash or the sgd id.
It seems that this bug was introduced in https://github.com/graphprotocol/graph-node/commit/09db0ce68ffbe768e99a2ce5536c3b148837f6ab. Prior to this commit, I could rewind subgraphs with multiple entries (the old logic would flatten all the nested arrays). Now, whether an sgd id is used or an ipfs hash, this errors out.
This is happening on v0.35.1
Relevant log output
No response
IPFS hash
No response
Subgraph name or link to explorer
No response
Some information to help us out

[X] Tick this box if this bug is caused by a regression found in the latest release.
[ ] Tick this box if this bug is specific to the hosted service.
[X] I have searched the issue tracker to make sure this issue is not a duplicate.

OS information
None

We're also experiencing this issue with 0.35.1.


Empty $store.getters.post if loading

I ended up writing this in $store.getters on my latest site:
        post: state => {
            if( !state.loaded ){
                return {}
            }

            // This is a "post" in the sense of a WordPress post - the first result of the Loop
            return _get(state.loop, '[0]', {})
        },

This way, you don't see any flashes of the previous page's content before the new page loads.
Two questions:

Any problem with adding this in as-is to the boilerplate?
Should we create a boilerplate fallback component to handle loading, or should that be on a per-site basis?

I can't talk to the correct "vue" way to do it, but our boiler plate shouldn't show the wrong content when loading, that is for sure. So if this is the bets way to handle this, then lets do it.

Should we then have a getter for related? So that we can handle the same "flash of wrong content" for things like grids?

@drewbaker I think the related getter can be an extension of the post getter - so for related posts, you'd do:
computed: {
  related(){
    return _get(this.$store, 'getters.post.related', {})
  }
}

That'll default to an empty object while the page is loading and return the correct result when it's available. To do the same thing for children:
computed: {
  children(){
    return _get(this.$store, 'getters.post.related.children', [])
  }
}

We just chatted about this here in Ithaca, I think there are times where you want the to still see the current content while new content is loading in but at least 90% of the time the functionality we want is what @SaFrMo has described above where it returns an empty object during load.
Let's add that code into the boilerplate and if we encounter a scenario where we need it to function the old way we can just revert it manually by deleting those 3 lines.

If I wanted the content o show while losing, couldn’t I just get it out of the $store, rather than using a getter?

yep, that's definitely something to be aware of because this solution won't protect you when you're using the actual loop (showing a grid of directors for instance.)

That’s good I think. Basically if I want to have hidden page loading, i use the getter. Gives us options.

Added in 5c4cd32


consistently running out of memory on eddiehinkle.com accounts

...at least twitter and facebook:

https://brid.gy/twitter/EddieHinkle
https://brid.gy/facebook/10102250513854186

two different silos, so i expect it's from OPD fetching and processing his site, not from polling silo posts.
example log from 3d ago. more recent logs of polls that run out of memory don't show anything interesting. e.g.  :
Params: NestedMultiDict([('last_polled', u'2017-10-10-12-39-30'), ('source_key', u'aglzfmJyaWQtZ3lyIwsSDEZhY2Vib29rUGFnZSIRMTAxMDIyNTA1MTM4NTQxODYM')])
Source: Eddie Hinkle (Facebook)<PHONE_NUMBER>3854186, http://brid.gy/facebook/10102250513854186
Last poll: http://brid.gy/log?start_time=1507906429&key=aglzfmJyaWQtZ3lyIwsSDEZhY2Vib29rUGFnZSIRMTAxMDIyNTA1MTM4NTQxODYM
Updating Eddie Hinkle (Facebook) /facebook/10102250513854186 : {'poll_status': 'polling', 'rate_limited': False, 'last_poll_attempt': datetime.datetime(2017, 10, 13, 15, 54, 11, 839540)}
Using ETag "55bad7a1a2898cf6065927c5a4a561a90d6840c4", last activity id<PHONE_NUMBER>9205976
urlopen GET https://graph.facebook.com/v2.10/me/feed?offset=0&fields=id,application,caption,comments,created_time,description,from,likes,link,message,message_tags,name,object_id,parent_id,picture,place,privacy,reactions,sharedposts,shares,source,status_type,story,to,type,updated_time,with_tags&limit=50&access_token=... {}
Found 60 public activities: [...]
...
Exceeded soft private memory limit of 128 MB with 154 MB after servicing 216 requests total
While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

Params: NestedMultiDict([('last_polled', u'2017-10-12-04-50-07'), ('source_key', u'aglzfmJyaWQtZ3lyGAsSB1R3aXR0ZXIiC0VkZGllSGlua2xlDA')])
Source: EddieHinkle (Twitter) EddieHinkle, http://brid-gy.appspot.com/twitter/EddieHinkle
Last poll: http://brid-gy.appspot.com/log?start_time=1507909111&key=aglzfmJyaWQtZ3lyGAsSB1R3aXR0ZXIiC0VkZGllSGlua2xlDA
Updating EddieHinkle (Twitter) /twitter/EddieHinkle : {'poll_status': 'polling', 'rate_limited': False, 'last_poll_attempt': datetime.datetime(2017, 10, 13, 16, 38, 52, 351860)}
Using ETag None, last activity id<PHONE_NUMBER>83219456_favorited_by_308803734
urlopen GET https://api.twitter.com/1.1/search/tweets.json?q=%22eddiehinkle.com%22&include_entities=true&tweet_mode=extended&result_type=recent&count=50 {}
Found 70 public activities: [...]
Exceeded soft private memory limit of 128 MB with 154 MB after servicing 178 requests total
While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

cc @EdwardHinkle, but not your fault, no need to do anything!

@snarfed it actually might be might fault :) Loking at it, I don't think I paginated my "firehose". I should probably just add a limit max since my firehose isn't used for viewing so you don't get ALL my posts ;) I should just do max posts from last couple of days because I imagine bridgy (and readers) would scan at least every day or so

damn, that didn't fix it. :/

i think we both fixed this. :P :crossed_fingers: