Team sync - 14 November 2024

Please take a moment to update the Post PST A11y Audit project board.
Then please provide your updates using the following template. (You may have work outside the post-audit tasks, please include that work, too.)
- Progress made since last sync, if any
  -
- Currently working on
  -
- Blockers, questions, or anything the team can help you with
  -
- Needs discussion
  -
- Upcoming PTO
  -
- How many hours per week you have allocated to this project/Change in allocation

Previous:  31 October 2024

I won't be able to make it to the call so I'm adding my updates here and I'll follow up on slack

Progress made since last sync, if any

I've corporated Isabella's feedback into the PST Design System doc. It is ready to be published now.

Currently working on

I'm done with the open issues I had, so I'm open to take on more work (or reminders if I missed something!)

Blockers, questions, or anything the team can help you with

Need @trallard's help to publish the PST Design System Documentation

Needs discussion

n/a

Upcoming PTO

I'm off on 18th November. And sick, so I may be a bit slower in responding to things. Please do not hesistate to double ping!

How many hours per week you have allocated to this project/Change in allocation

I have 10 hours / week

Progress made since last sync, if any

Opened: https://github.com/pydata/pydata-sphinx-theme/pull/2039
Opened and merged: https://github.com/pydata/pydata-sphinx-theme/pull/2034

Now I finally have a good understanding of how the new Read the Docs version switcher works. I could apply that knowledge to https://github.com/pydata/pydata-sphinx-theme/issues/1933.

Updated: https://github.com/pydata/pydata-sphinx-theme/pull/1986

Carried over:

https://github.com/pydata/pydata-sphinx-theme/issues/1923
Submit feedback: https://github.com/sphinx-doc/sphinx/pull/12507

Blockers, questions, or anything the team can help you with

Needs review: https://github.com/pydata/pydata-sphinx-theme/pull/2039

No upcoming PTO
8 hours/week


Fixup typos in integration-tests/README.md

Description
This PR fixes some miscellaneous things in the integration-tests README file.

This PR has:

[x] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

Thanks for the reviews! Static check is failing unrelated to this change (I see similar failures on other open PRs):
Error:  processing/src/main/java/org/apache/druid/data/input/FirehoseFactory.java:90 -- Method is never used as a member of this interface, but only as a member of the implementation class(es). The project will stay compilable if the method is removed from the interface.
Error:  processing/src/main/java/org/apache/druid/data/input/InputFormat.java:62 -- Method is never used.
Error:  processing/src/main/java/org/apache/druid/data/input/FirehoseFactoryToInputSourceAdaptor.java:48 -- Method is never used.
Error:  processing/src/main/java/org/apache/druid/java/util/http/client/HttpClientProxyConfig.java:43 -- Constructor is never used.
Error:  processing/src/main/java/org/apache/druid/java/util/http/client/response/FullResponseHolder.java:52 -- Method is never used as a member of this abstract class, but only as a member of the implementation class(es). The project will stay compilable if the method is removed from the abstract class.

Merging because I can't see how changing a README.md file could cause a UT to fail, so assuming the UT failure is flaky.


Support serving images under s3 subdirectories, Fix to make /fit-in/ work; Fix for VipsJpeg: Invalid SOS error plus several other critical fixes.

This PR includes fixes for the following v4 issues:

S3 subdirectories not working.
/fit-in/ url directive not working.
Exif orientation not honoured/followed.
VipsJpeg: Invalid SOS parameters for sequential JPEG errors. This is observable with images taken with Samsung cameras, causing the api/cloudfront endpoint to output a blank json {} instead of the image.
Wrong resizing if the path/filename contains an 'x' with surrounding digits (eg: /image8x8.jpg)

Sample paths in which this was tested on aside from the unit tests:
/100x100/directory1/directory2/image.jpg
/fit-in/100x100/directory1/directory2/image.jpg
/fit-in/100x100/filters:someFilters/directory1/directory2/image.jpg
/filters:someFilters/directory1/directory2/image.jpg
/100x100/filters:someFilters/directory1/directory2/image_8x8x8x12.jpg
You may start using this PR/Fork with this Exported Function zip: https://github.com/innovationlove/serverless-image-handler/releases/tag/4.0.1-ilove
Just replace the code that your Serverless Image Handler Function is using with the package above. USE AT YOUR OWN RISK.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I have manually added this solution to our lambda-function and can confirm that it performs as defined.
@rpong You are a life saver!

@rpong, images without resizing still doesn't work, example:
/directory1/directory2/image.jpg
If you can fix it in that pull request will be super cool, thanks!

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

@rpong, images without resizing still doesn't work, example:
/directory1/directory2/image.jpg
If you can fix it in that pull request will be super helpfull, thanks!

sorry, my bad. this should be fixed now.

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits arounds it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix too. thanks man..

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match
Line 39
           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            }

Please add it to your PR, thank you :)

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match
Line 39
           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            }

Please add it to your PR, thank you :)

that work, thanks !

also, in previous version (thumbor), i have some use case that unfortunately unsupported on current version:

using key path without file extension
Example: https://xxxx.cloudfront.net/64x64/images/svara-radio-logo/download/58e5ecc7cab82dc1700c0043
My workaround is remove path extension checking on parseRequestType method on image-request.js

parseRequestType(event) {
    const path = event["path"];
    // ----
    const matchDefault = new RegExp(
      /^(\/?)([0-9a-zA-Z+\/]{4})*(([0-9a-zA-Z+\/]{2}==)|([0-9a-zA-Z+\/]{3}=))?$/
    );
    const matchThumbor = new RegExp(
      /^(\/?)((fit-in)?|(filters:.+\(.?\))?|(unsafe)?).*$/  // Remove extension on this regex
    );
    const matchCustom = new RegExp(/(\/?)(.*)(jpg|png|webp|tiff|jpeg)/);
    const definedEnvironmentVariables =
      process.env.REWRITE_MATCH_PATTERN !== "" &&
      process.env.REWRITE_SUBSTITUTION !== "" &&
      process.env.REWRITE_MATCH_PATTERN !== undefined &&
      process.env.REWRITE_SUBSTITUTION !== undefined;
    // ----
    if (matchDefault.test(path)) {
      // use sharp
      return "Default";
    } else if (matchCustom.test(path) && definedEnvironmentVariables) {
      // use rewrite function then thumbor mappings
      return "Custom";
    } else if (matchThumbor.test(path)) {
      // use thumbor mappings
      return "Thumbor";
    } else {
      throw {
        status: 400,
        code: "RequestTypeError",
        message:
          "The type of request you are making could not be processed. Please ensure that your original image is of a supported file type (jpg, png, tiff, webp) and that your image request is provided in the correct syntax. Refer to the documentation for additional guidance on forming image requests."
      };
    }
  }

using smart path
For now, my workaround is only omit the smart keyword on parseImageKey method at image-request.js

parseImageKey(event, requestType) {
    if (requestType === "Default") {
      // Decode the image request and return the image key
      const decoded = this.decodeRequest(event);
      return decoded.key;
    } else if (requestType === "Thumbor" || requestType === "Custom") {
      // Parse the key from the end of the path
      const pathParts = event["path"].split("/");

      for (let i = 0; i < pathParts.length; i++) {
        if (
          pathParts[i] !== "" &&
          pathParts[i] !== "fit-in" &&
          pathParts[i].match(/^\d+x\d+$/) == null &&
          pathParts[i].match(/^filters:/) == null &&
          pathParts[i].match(/^smart$/) == null  // add this line
        ) {
          const parts = pathParts.slice(i);
          return parts.join("/");
        }
      }
    }

    // Return an error for all other conditions
    throw {
      status: 400,
      code: "ImageEdits::CannotFindImage",
      message:
        "The image you specified could not be found. Please check your request syntax as well as the bucket you specified to ensure it exists."
    };
  }

using filters fill with same usage as thumbor, ex:filters:fill(white,1)
Currently my workaround on this problem is adding some hardcoded line to convert filter value to color object that accepted by sharpjs, ex: 'white,1' to {r: 255, g:255, b: 255, alpha: 0}

maybe you have more clean workaround for this problem
thank you

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match
Line 39
           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            }

Please add it to your PR, thank you :)

Nice! thank you for this, I added this to the PR but moved it outside the foreach loop so it is processed/looked up only once. Just updated the PR as well so the tests pass and several other critical things that required us to fix before we launched our beta app this morning.

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..
/600x400/products/img/denial-krylon-8inchdeck-1xrun-00.jpg

Still not working for some of my URLS such as the following:
https://xxxxxxxxx.cloudfront.net/400x400/products/img/ricky-powell-beastie-boys-18x13-1xrun-01.jpg

Still not working for some of my URLS such as the following:
https://xxxxxxxxx.cloudfront.net/400x400/products/img/ricky-powell-beastie-boys-18x13-1xrun-01.jpg

thanks for that use case, just pushed a change to the PR which should take care of it, strange that the unit-tests missed this.

Hi @rpong
Thanks for all your hard work. Another issue I just spotted is when you form a URL like so...
https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg
Specifying "400x" with the old Thumbor used to give you the best fit for 400 and figure it out itself. However, doing that now will give you "NoSuchKey". If you use 400x400 for example then it works fine.
https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x400/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x400/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

Hi @adam91holt , noted on this, for now, if this is an actual production use case for you, what we do to circumvent around this is to specify zero and a fit-in.
https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x0/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg
I did not realize they support 400x before, such dimension convention also wasn't in the unit tests so we did not cover that.

Hi guys, I am not a developer, actually I am a curious retailer that managed to deploy the serverless-image-handle script on AWS. I use it a lot for all company reports on PowerBI.
I’ve the template v2 but it stopped working due to the changes on lambda environment to sharpJS I’ve heard.
I’ve wiped my cloudformation stack on AWS and created a new one with the template v4, I knew that it would just work with images on root but I could live with that. I see though that fit-in is not working, and basically that’s almost all I use.
I´ve found rpong work fixing the template v4, but I have no idea how to implement that on my end here. Could anyone make a template with all the commits so I can deploy an cloudformation stack pointing to the updated template?
If that is not a good idea, could someone let me know how do I make the suggested commits (changes) on the code there?
Any help would be useful, I am stuck at the moment.
Regards,
Tiago

Hi guys, I am not a developer, actually I am a curious retailer that managed to deploy the serverless-image-handle script on AWS. I use it a lot for all company reports on PowerBI.
I’ve the template v2 but it stopped working due to the changes on lambda environment to sharpJS I’ve heard.
I’ve wiped my cloudformation stack on AWS and created a new one with the template v4, I knew that it would just work with images on root but I could live with that. I see though that fit-in is not working, and basically that’s almost all I use.
I´ve found rpong work fixing the template v4, but I have no idea how to implement that on my end here. Could anyone make a template with all the commits so I can deploy an cloudformation stack pointing to the updated template?
If that is not a good idea, could someone let me know how do I make the suggested commits (changes) on the code there?
Any help would be useful, I am stuck at the moment.
Regards,
Tiago

You may pack the fork by running the following:
#clone the repo
cd ~
git clone git@github.com:innovationlove/serverless-image-handler.git
cd serverless-image-handler/source/image-handler

#build npm depenedencies
npm run build:init && npm install --arch=x64 --platform=linux --production && npm run build:zip && npm run build:dist

The output of the above will be also be a zip file as follows:
~/serverless-image-handler/source/image-handler/dist/image-handler.zip
Go to your Serverless Image Handler Lambda function -> Scroll down to function code ->  Select Upload a .zip file -> Click Upload ->  Select the generated zip file above -> Click Save -> Click Actions and Select Publish New version.
FOLLOW AT YOUR OWN RISK.
NOTE: If you run into any issues during build, you will need Node Version 11, depending on your platform, a little googling should nudge you to the right path regarding that one.. Goodluck.

Hi guys, I am not a developer, actually I am a curious retailer that managed to deploy the serverless-image-handle script on AWS. I use it a lot for all company reports on PowerBI.
I’ve the template v2 but it stopped working due to the changes on lambda environment to sharpJS I’ve heard.
I’ve wiped my cloudformation stack on AWS and created a new one with the template v4, I knew that it would just work with images on root but I could live with that. I see though that fit-in is not working, and basically that’s almost all I use.
I´ve found rpong work fixing the template v4, but I have no idea how to implement that on my end here. Could anyone make a template with all the commits so I can deploy an cloudformation stack pointing to the updated template?
If that is not a good idea, could someone let me know how do I make the suggested commits (changes) on the code there?
Any help would be useful, I am stuck at the moment.
Regards,
Tiago

You may pack the fork by running the following:
#clone the repo
cd ~
git clone git@github.com:innovationlove/serverless-image-handler.git
cd serverless-image-handler/source/image-handler

#build npm depenedencies
npm run build:init && npm install --arch=x64 --platform=linux --production && npm run build:zip && npm run build:dist

The output of the above will be also be a zip file as follows:
~/serverless-image-handler/source/image-handler/dist/image-handler.zip
In AWS Console: Go to your Serverless Image Handler Lambda function -> Scroll down to function code -> Select Upload a .zip file -> Click Upload -> Select the generated zip file above -> Click Save -> Click Actions and Select Publish New version.
FOLLOW AT YOUR OWN RISK.
NOTE: If you run into any issues during build, you will need Node Version 11, depending on your platform, a little googling should nudge you to the right path regarding that one.. Goodluck.

Rpong,
Thanks for the quick reply and also all the work on fixing the serveless image handler template.
I have one linux server here on work with git installed running as root (so should have all permissions needed) but the git clone did not work.
git clone git@github.com:innovationlove/serverless-image-handler.git
Cloning into 'serverless-image-handler'...
Permission denied (publickey).
fatal: Could not read from remote repository.
Would you happen to know how to overcome this error?
Regards,
Tiago

Rpong,
Thanks for the detailed information could not have done it without you. Had problems to build had to install Node and NPM then update Zlib. I’ve built this script using Debian as system node: v10.15.2 and NPM 6.10.2. When I’ve uploaded the command on AWS Lambda I’ve changed the Runtime from Node.JS 8.1 to Node.js 10.x (was supposed to do this)?
The problem is that my files are on 'sa-east-1' and when I´ve uploaded the .zip file after compiling I’ve got this error: Error occurred while GetObject. S3 Error Code: AuthorizationHeaderMalformed. S3 Error Message: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'
I guess I need to change somewhere on the zip file that this code is for 'sa-east-1' do you know where ? Is there another thing that I need to change in order to build this code on a sa-east-1 s3?
Regards,
Tiago

\

Hi guys, I am not a developer, actually I am a curious retailer that managed to deploy the serverless-image-handle script on AWS. I use it a lot for all company reports on PowerBI.
I’ve the template v2 but it stopped working due to the changes on lambda environment to sharpJS I’ve heard.
I’ve wiped my cloudformation stack on AWS and created a new one with the template v4, I knew that it would just work with images on root but I could live with that. I see though that fit-in is not working, and basically that’s almost all I use.
I´ve found rpong work fixing the template v4, but I have no idea how to implement that on my end here. Could anyone make a template with all the commits so I can deploy an cloudformation stack pointing to the updated template?
If that is not a good idea, could someone let me know how do I make the suggested commits (changes) on the code there?
Any help would be useful, I am stuck at the moment.
Regards,
Tiago

You may pack the fork by running the following:
#clone the repo
cd ~
git clone git@github.com:innovationlove/serverless-image-handler.git
cd serverless-image-handler/source/image-handler

#build npm depenedencies
npm run build:init && npm install --arch=x64 --platform=linux --production && npm run build:zip && npm run build:dist

The output of the above will be also be a zip file as follows:
~/serverless-image-handler/source/image-handler/dist/image-handler.zip
In AWS Console: Go to your Serverless Image Handler Lambda function -> Scroll down to function code -> Select Upload a .zip file -> Click Upload -> Select the generated zip file above -> Click Save -> Click Actions and Select Publish New version.
FOLLOW AT YOUR OWN RISK.
NOTE: If you run into any issues during build, you will need Node Version 11, depending on your platform, a little googling should nudge you to the right path regarding that one.. Goodluck.

Rpong,
I was able to fix the problems mentioned above. Found out that I indeed need to use Runtime from Node.JS 8.1
Added this configs under Lambda Environment Variables
REKOGNITION_REGION: us-east-1
TC_AWS_REGION: sa-east-1
And I was able to run the .zip file that I’ve compiled, but it seems to be the same as before, fit-in isn’t working and subfolders either. Is there a way to know if I compiled the file with all the changes needed?
**I tried to use your git clone command, it didn’t work, so I’ve used like this:
git clone https://github.com/innovationlove/serverless-image-handler**
is this git-clone url correct?
Since subfolders aren’t working even with this new compile I’ve put one test photo on bucket root. As you can see, fit-in didn’t work as it supposed to.
Example of output: https://d17wgxzut1imxc.cloudfront.net/fit-in/600x600/190705003.jpg
Thanks for all the help so far, I am trying to do my best to sort this out.
Regards,
Tiago

hi @tiagopmello , make sure that you did clicked "Save" and Actions -> Publish New Version. You may also need to invalidate your whole cache to make sure the image you are trying to request is not cached by Cloudfront.  You can do this by going to your Cloudfront Distribution -> Invalidations tab, and creating a new invalidation for all your objects  by using this path/* (do at your own risk).

hi @tiagopmello , make sure that you did clicked "Save" and Actions -> Publish New Version after you uploaded the zip file in the Function Code area. You may also need to invalidate your whole cache to make sure the images you are trying to request are not not cached by Cloudfront. You can do this by going to your Cloudfront Distribution -> Invalidations tab, and creating a new invalidation for all your objects by using this path/* (do at your own risk).

Thanks for the reply again. Yes it was published. I`ve clicked publish new version and save. and also I've now invalidated the whole cloudfront distribution cache, it still didn't work.
Can you confirm if the way I've closed the github fork is correct ?
git clone https://github.com/innovationlove/serverless-image-handler
you suggested me to use git clone git@github.com:innovationlove/serverless-image-handler.git
But this command did not work for me it gave permissions error.

hi @tiagopmello , please try to find me "Arpee Ong" on any chat/social media channel, and i'll send you the actual build we are using.

glad that the beautiful lady in the photo is now whole @tiagopmello
https://d17wgxzut1imxc.cloudfront.net/fit-in/605x605/190705003.jpg  :)

glad that the beautiful lady in the photo is now whole @tiagopmello
https://d17wgxzut1imxc.cloudfront.net/fit-in/605x605/190705003.jpg :)

Yes, you are the best. Its working flawlessly.
Thank you!!

Apparently there are reports of some users not able to package and apply the function correctly, so I bundled a release ready to be used.
You may start using this PR/Fork by using this Exported Function zip: https://github.com/innovationlove/serverless-image-handler/releases/tag/4.0.1-ilove
Just replace the code that your Serverless Image Handler Function is using with the package above. USE AT YOUR OWN RISK.

Thank you @rpong, I really appreciate it! Hopefully your fix will make it into the main repo soon.

@rpong Thanks for your massive contribution here. I've added your changes to our review board and hope to have them merged and published shortly in the next minor release. Awesome stuff!

This PR resolves some major BC issues for me. Is there an expected time that we can see this merged back into the main repo?

Any news on this PR making it in?

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

easier to just use this pre-built one: https://github.com/awslabs/serverless-image-handler/pull/130#issuecomment-517566608 goodluck and let us know.

Any updates?

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

easier to just use this pre-built one: #130 (comment) goodluck and let us know.

Worked very well for me in combination with #157

@rpong Thanks for your contribution.
But there is a problem: if the filename contains square brackets, the regular expression does not work.
Example: [test].jpg does not work (throw error 400)

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

easier to just use this pre-built one: #130 (comment) goodluck and let us know.

Worked very well for me in combination with #157

Glad to help, btw just released the same package we use with Node 12.x -> https://github.com/innovationlove/serverless-image-handler/releases/tag/4.0.3
Goodluck.

@rpong Thanks for your contribution.
But there is a problem: if the filename contains square brackets, the regular expression does not work.
Example: [test].jpg does not work (throw error 400)

Will look into this within the week.

@rpong Thank you very much, again! But I don't think it's working properly. How do you crop an image to exactly a specified size?
Before upgrading, if I didn't use the fit-in argument, the image auto-cropped and auto-resized to be EXACTLY the specified size. Now domain.com/fit-in/50x50/image.jpg and domain.com/50x50/image.jpg  produces the same result :-(

Why in this case  https://cloudfront-fake-endpoint.net/fit-in/400x0/image_id.jpg image is not covered, I receive an empty object(
It doesn't work with lambda which runs on node.js 12.x, but I have the older Serverless Image Handler that runs the lambda on node.js 8.10 and this one works properly

Got also some problems with the newest version 4.1. Is it still possible to deploy version 4 and then upgrade the node version with the release from innovationlove?

Thanks for your effort here, @rpong
We have updated our solution, and your pull request has been merged into our master branch. However, as we've merged many pull requests at this time, the source code might not exactly same as what you've done. In addition, due to our internal process, we've merged source code manually, so please understand how your effort has been merged.
You can refer to the recent changes here

@rpong this solution is not working for image size more thane 4MB .
getting error .
{"message": "Internal server error"}

I think this is more a Aws Lambda problem because the request and response size is limited:
Request and response body payload size are maximized to 6 MB. (aws)

Is there a way to use the old fit-in behavior url without having to overwrite the code?

@giovannipds the old fit-it is working on the last release. Just update your stack with the latest template.

@cxrod seriously? I probably have edited the code before putting the stack up the last time I've updated here (29 days ago).

@cxrod I won't re-do this right now since it's already working as I need on my stack 😆 but thanks for letting me know.

@giovannipds sorry for the late reply! I was observing for a long time the PR with the fix, until it was released on February

@cxrod no worries! o/


Make use-smartnoise-sql query-specific

Currently we set it on/off for all queries, but this can be made more granular.

Done


Add some unit tests

Good news everyone!
I added Travis CI integration into master (see the .travis.yml file). If anyone is interested (@Marie89, @mickypaganini), it would be fun to try adding some unit tests. Basically what I have in mind:

Call the tests lwtnn-unittest-XXX, they can live in scripts/ or bin/
They have to return non-zero on failure for Travis CI to realize something is wrong, see the example output that I'm turning on with the likes here
I'd rather not include much (or any) data in the repository (since git won't handle raw hdf5 well) but just checking to make sure the output of some of the basic nets doesn't change with new commits would already be very useful.

Obviously this is just for "fun", so low priority but it may also be very useful down the road.

Nice, I was looking into that too actually! I can expand on it a bit if needed. As you said, this might be low priority now, but before I push anything I run all these checks by hand anyways, so we may as well automate them. Thanks for adding this.

Cool, actually the tests I've created aren't super useful so more extensive ones would be welcome. As I said before I don't really want to include HDF5 files in the repository, but I can think of a few other tests:

I wrote a script that generates fake Keras outputs, which won't work right now (because it was for pre 1.0), but could be used for simple cases.
We could generate some very simple intermediate JSON files and put them in the data/ directory. This would also be useful if we ever end up changing the format, since we might end up with earlier versions of the JSON format uploaded to Athena which we still need to support.
We can also just run wget and download trained nets to the Travis CI nodes from (for example) CERN public pages. We should try to keep the file size minimal obviously, but I just checked with a 40 MB file and Travis didn't give me any problems.

Great, I just closed #22, which in which @mickypaganini added highway layers. I implemented a test which downloads the network from my CERN webspace, which seems like a fine solution to deal with much bigger files. Anyone can also run the test locally by executing ./tests/test-highway.sh.
Once we have some production-ready RNNs and feed-forward networks, it would be great to add some tests for them too.

I think this is "complete"—I added a few unit tests that cover the basic cases, so we now have build tests, and automatic tests for the keras and agilepack (anyone actually using it?) converters. Specific tests should be their own issue. To start with, we should probably add at least one test that starts with each JSON file that we actually end up using, to ensure that we don't update the JSON format (i.e. in #4) and break something.


Docs.MS CI Updates for Azure.Analytics.Synapse.Spark

CI Update for release of Azure.Analytics.Synapse.Spark

Docs Build status updates of commit 932bf29:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

bundlepackages/azure-dotnet-preview.csv
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 0720564:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

bundlepackages/azure-dotnet-preview.csv
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Clean up libpfasst_swe_sphere_expl_sdc and, Fix MemBlockAllocator constructor call

remove function call to solve implicit system (pure explicit SDC here, no ccomp() necessary)
fix MemBlockAllocator constructor call in src/unit_tests/test_sphere_sph_operators_complex.cpp, similarly to what Martin did in similar to what Martin did for burgers.cpp in e9c4f32

Still a draft since I am not able to run the tests at the moment!

All tests are running successfully, this PR is ready to go


Add author details at the end of the post (<3 links)

+1. Could also link author picture back to the team page.


Multiple Cloud Source Repo support

NOT yet ready for review. Still pending cleanup/refactoring and testing; in the meantime you can see the screenshots below.
This PR add support in IJ for the GCP support of multiple cloud repositories. Any reference or reliance on a 'default' repository is also removed.
Upload to Cloud Repo dialog:

Checkout from Cloud Repo dialog:

Selector widget from the upload dialog ("create new repo" button is available):

Selector widget from the checkout dialog ("create new repo" button not available):

Selector widget in loading state:

Selector widget in error state:

Selector widget in empty state (no repos found):

Thanks Rui for reviewing this. You can post feedback directly here on GitHub.
@patflynn - the 'create repo' button currently does not open a dialog. We have no API call for this yet - once the new Source API is release I plan up updating this. Instead, it opens up a browser window. Unfortunately the best I could do is open up the 'developer' page that lists all the repo's and has a "new repo" button at the top. This is because the web console's new repo button does not have a permanently linkable url (it just uses some javascript).
Regarding the spacing - I agree, I'll improve this.

Rui, to install this build to check it out (but keep in mind things may change in the process of review) you can follow this steps:

Clone this repository
Switch to branch 'repository-selector'
Run IDEA with the plugin:
./gradlew clean buildPlugin runIdea
Go to tools->Google Cloud Tools -> Upload to a Google Cloud Source Repository
and
VCS -> Checkout from Version Control -> Google Cloud

@patflynn what are your thoughts on allowing a single GCP project to be uploaded to more than one Cloud Source Repo?
Context:
Previously, when there was only 1 'default' Cloud repo, IJ ran a check to see if there already existed a CSR remote. If the remote was there, then a dialog would come up:

Now that we support multiple CSR repos, we have 2 options:

keep the logic from before and only allow a given project to be associated with a single CSR remote
allow multiple remotes and update the check to occur after selection of the CSR repo - after selection it would check to see if the remote exists. We'd have to update the remote naming scheme - probably to make the remote name equal to the name of the repository ID.

Personally I keep going back and forth on these. On the one hand, since GCP now supports multiple repos, we could just allow a single project to be pushed to multiple repos. Other the other hand, when were talking about a single project, the same thing can (and usually is) achieved with branches.

What does the generic git integration for IJ allow?
On Tue, Jan 10, 2017 at 4:22 PM, Etan Shaul<EMAIL_ADDRESS>wrote:

@patflynn https://github.com/patflynn what are your thoughts on
allowing a single GCP project to be uploaded to more than one Cloud Source
Repo?
Context:
Previously, when there was only 1 'default' Cloud repo, IJ ran a check to
see if there already existed a CSR remote. If the remote was there, then a
dialog would come up:
[image: screen shot 2017-01-10 at 4 17 29 pm]
https://cloud.githubusercontent.com/assets/1735744/21825083/4475dace-d750-11e6-8236-6cf759363da2.png
Now that we support multiple CSR repos, we have 2 options:

keep the logic from before and only allow a given project to be
associated with a single CSR remote
allow multiple remotes and update the check to occur after
selection of the CSR repo - after selection it would check to see if the
remote exists. We'd have to update the remote naming scheme - probably to
make the remote name equal to the name of the repository ID.

Personally I keep going back and forth on these. On the one hand, since
GCP now supports multiple repos, we could just allow a single project to be
pushed to multiple repos. Other the other hand, when were talking about a
single project, the same thing can (and usually is) achieved with branches.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184#issuecomment-271701891,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHf5HRus299i6zMFrfC5I4gZKLsSRlaYks5rQ_aUgaJpZM4Lezod
.

If you have multiple remotes then the generic git integration allows you to select which remote to push to. I guess this answers my question.
Good point.

@patflynn @joaoandremartins @aslo
Other then the enhancement to allow uploading to multiple CSR repos (as discussed above), this is ready for initial review.

I've added support for multiple CSR remotes: https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184/commits/b07b8a1324afbf11ae020d675b55d0407d97e951
Say I have a GCP project with the following repositories:

I can then upload my project to say 'test-repo-one', and locally, a git config with the following remote will be created:

I can then upload the same project to 'test-repo-2' as well, and 2 remotes will be present:

If I then try to upload the project again to 'test-repo-1' I'll get the following dialog:

Finally, if I manually update the remote url for 'test-repo-2' to some non GCP url to simulate a name collision and then try to upload to 'test-repo-2', I'll get the following dialog:

In "selector error state" and "selector empty state", what can user do to in order to move forward?

@hray2003 - in both those cases, the user can either refresh, select a different project, or create a new repository. Perhaps the wording for the 'empty' state can be expanded to indicate that the user can create a new repository.

I don't fully understand the second case.
On Wed, Jan 11, 2017 at 2:38 PM, Etan Shaul<EMAIL_ADDRESS>wrote:

@hray2003 https://github.com/hray2003 - in both those cases, the user
can either refresh, select a different project, or create a new repository.
Perhaps the wording for the 'empty' state can be expanded to indicate that
the user can create a new repository.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184#issuecomment-271971905,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHf5HU6aC1744i4053rRIG3oa6siFMyWks5rRS-6gaJpZM4Lezod
.

Which second case? The 'empty state'? This is just when a selected cloud project has no CSR repos associated with it.

Can we rename "Repository Selector:" to just "Repository:"?

Maybe I'm missing something, but why are we prepending "cloud-platform-" to the remote name? To make sure it doesn't collide with any previously existing remote with the same name? To mark it as a GCR remote? (and then we'd probably run into problems if there were already other remotes named "cloud-platform-*", unlikely as that is)

Finally, I also don't understand the second case (where you manually modify the remote URL so it's not a GCR URL). What are you accomplishing there? Validating/enforcing that a GCR remote has a valid URL? Why does renaming the URL result in a name collision? Why are "upload to a GCR remote" and "add a GCR remote" related?

Also, the title of the window needs to be changed from "Google" to something else like "Google Cloud Source Repositories" and the text on the second window could also use some improvements, e.g., s/GCP remote/Google Cloud Source Repository remote. (and why are we telling we couldn't add a remote when the user is trying to push to an already added remote?)

@joaoandremartins

The "Repository Selector:" wording was a mistake and has since been updated (the screenshot is old). It now said "Cloud Repository".

You are right. I am prepending "cloud-platform-" to avoid naming collisions. If there happen to be collisions, then we show the following dialog:

This is just to similate the naming collision I just mentioned in (2). I specifically set it to a non GCP url because it if was a GCP url, then the user would see this dialog instead:

I'm not sure what you mean about "upload/add a GCR remote". Maybe lets talk more about this part offline.

I agree about your points about improving wording.

and why are we telling we couldn't add a remote when the user is trying to push to an already added remote?

This is because our "Upload to GCP" functionality is not for pushing to a GCP remote. Its for the initial project upload and for automatically creating the local git remote. Subsequently to push to the cloud repository the user is expected to use basic GIT functionality (either in IJ or otherwise).

I was just talking with Etan about this offline and we're now thinking a
workflow might be more flexible with the following strategy:

when 'uploading to GCP' always ask the user what they'd like to call the
remote we create.
For the first remote default it to 'origin'. (this is what the JB github
integration does)
For subsequent remotes possibly concatenate project name and repo name.
don't try to map remote names to CSR repos.  Allow the user to create
multiple remotes pointing to the same uri if they want.

On Wed, Jan 11, 2017 at 3:04 PM, Etan Shaul<EMAIL_ADDRESS>wrote:

@joaoandremartins https://github.com/joaoandremartins

The "Repository Selector:" wording was a mistake and has since been
updated (the screenshot is old). It now said "Cloud Repository".
2.
You are right. I am prepending "cloud-platform-" to avoid naming
collisions. If there happen to be collisions, then we show the following
dialog:
[image: image]
https://cloud.githubusercontent.com/assets/1735744/21864203/b3833bbe-d80e-11e6-89c0-649edf62beef.png
3.
This is just to similate the naming collision I just mentioned in (2).
I specifically set it to a non GCP url because it if was a GCP url, then
the user would see this dialog instead:
[image: image]
https://cloud.githubusercontent.com/assets/1735744/21864231/d020165c-d80e-11e6-9d76-05d5c9acd7b8.png
I'm not sure what you mean about "upload/add a GCR remote". Maybe lets
talk more about this part offline.
I agree about your points about improving wording.
and why are we telling we couldn't add a remote when the user is trying to
push to an already added remote?
This is because our "Upload to GCP" functionality is not for pushing to
a GCP remote. Its for the initial project upload and for automatically
creating the local git remote. Subsequently to push to the cloud repository
the user is expected to use basic GIT functionality (either in IJ or
otherwise).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184#issuecomment-271978616,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHf5HR3Lw4qi7L6t0WyZysQtN0EkERpsks5rRTXOgaJpZM4Lezod
.

@patflynn I've updated the experience to reflect our discussion:
There is now always a "remote name" selector:

The remote is auto-populated with a suggestion:

"origin" if there is no remote named "origin" already
"google-cloud-platform-{repository_name}" if there is already a remote named "origin"

If the user forces a conflict (by manually typing in a remote name that already exists) the following will be shown:

@patflynn @joaoandremartins @aslo
What are your thoughts on the wording of this dialog, specifically the "Remote name" section:

I was going through the process of testing out the flow as a normal user and realized it may not be totally clear that this dialog will generate a new remote for the user. I'm concerned that a user may confuse this with us requiring an existing remote to exist and that its up to the user to type in the matching name.
Here are a few options:

Change the label from "Remote name" to "New remote name"
Have placeholder text that says something like "enter a remote name to be created for this repository"
Have some labeling for the dialog explaining that this creates a new git repo (if one doesn't exist), and a new remote

Does it have to be a new remote? If it doesn't, then I'd be OK with not extra wording -- if the remote doesn't exist yet, we create it for the user.

Yes, it does have to be a new remote. Otherwise its an error.

I think "New remote name" looks good. The only problem is if the user doesn't understand why it needs to be a new repo (e.g., if they haven't checked out the remote repo but want to push there right away. not sure how often/if this would happen at all?). Then we could need some text.

e.g., if they haven't checked out the remote repo but want to push there right away. not sure how often/if this would happen at all?

I'd like to understand this scenario. Not having checked out the remote repo is the majority of use cases. This feature is precisely for "intializing" the remote repo by pushing the project to it and establishing the local remote (sounds like an oxymoron).

I think the case you're referring to is when the user doesn't already have a remote repo with that code, and is starting fresh.
If the code is already in the remote, but the remote somehow isn't synced locally (does this happen at all?), the user might want to use that "Upload to GCSR" dialog to upload it there. Though, thinking about it, I don't think this happens frequently?
Also, what's the story when you check out the remote with git or gcloud? Will we still require the users to create a new remote?

If the code is already in the remote, but the remote somehow isn't synced locally (does this happen at all?), the user might want to use that "Upload to GCSR" dialog to upload it there. Though, thinking about it, I don't think this happens frequently?

This dialog will always create a new remote. Whether or not we allow uploading to a cloud repo that already has code is up for debate: previously it would not allow this. After talking to Patrick we decided that it would (a new remote would still be created pointing to the same cloud url). However, I am going back on this now because there are cases where this could fail: e.g. non-fast-foward changes, which is tough to detect before hand.

Also, what's the story when you check out the remote with git or gcloud? Will we still require the users to create a new remote?

Yes, if they use this dialog in this case (and they shouldn't really) a new remote will be forced.
If I had to summarize what this dialog does it would be as follows:
The dialog creates a new local git repository if one does not yet exist, creates a new remote pointing to the Google Cloud Repository, and finally pushes the code to this remote.

I wonder if it would make sense to add a new issue to iron that bit out, i.e., if the user already has a remote with the code that isn't mirrored in a way our IJ plugin can't use it out of the bat?
In any case, since that dialog will always create a new remote and users using this feature who don't have a correctly set up remote, I think we can do without the extra text? Patrick or Rui might have a different take on that, though.

I'm kind of OK with either.
Could hover text or some extra wording above the fields clarify things?
On Tue, Jan 17, 2017 at 4:54 PM, João Martins<EMAIL_ADDRESS>wrote:

I wonder if it would make sense to add a new issue to iron that bit out,
i.e., if the user already has a remote with the code that isn't mirrored in
a way our IJ plugin can't use it out of the bat?
In any case, since that dialog will always create a new remote and users
using this feature who don't have a correctly set up remote, I think we can
do without the extra text? Patrick or Rui might have a different take on
that, though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184#issuecomment-273312233,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHf5Hfn8q1fxrEetfVyh_DRxOb4rSUcdks5rTTiJgaJpZM4Lezod
.

Here is what I came up with, with help from @joaoandremartins:
Updated action text (to make it clear that this is for "Setup" and not for normal "uploading / pushing"):

Updated remote name label to make it clear that its creating a new one:

I've updated the error messages to be notification popups (instead of an error dialog) and to show the contents of the git error output so the user can debug the problem.
Here is an example of the user attempting to use the dialog to a project to an existing git repo:

And after expanding, you can see the full message in a scrollable pane:

@patflynn @joaoandremartins @aslo
This is ready for review. All functionality has been implemented. Only outstanding task is test coverage.

@joaoandremartins see https://github.com/GoogleCloudPlatform/google-cloud-intellij/pull/1184/commits/d6b25811ee2549176acefefa47f1ff02de68b1a6
I made some non-trivial changes to how we are executing asynchronous code. This is mainly for testability (there was no good way to test the asynchronous code previously without injecting a Thread.sleep into the unit test; now the asynchronous part is easily mocked away), but also for better design (hopefully) and more idiomatic java 8.


KWeb 1.0

[x] Add bootstrap
[x] Redesign canvas to adjust to height and width resizes automatically
[x] Allow editable mode
[ ] Readd hierarchy browser
[ ] Add support for lyrdbs <- Preparation for DRC view and gdatasea

@Ealameda31
@joamatab

it would be great to only check the checkboxes of the layers present in the layout

it would be great to only check the checkboxes of the layers present in the layout

Done ;) and a zoom fit button


Subtype citation header

@crispzips If you move the jira ticket into "Delivered" then Stephen can take a look at it and approve it for deployment to production.


How to select all root nodes when only one of the child node is selected?

My current version:
https://jsfiddle.net/gatsbn/ycawjk2q/1/
For now if I select root node - all child nodes will be automatically selected.
My question:
If I select "Nested1", how to automatically select root nodes "Child1" and "Root node"?

data.instance.select_node(data.node.parents);  is helped


Is there a way to generate refresh token using a CustomGrantValidator?

I'm developing a web app that generate Access tokens using an ExtensionGrantValidator but i cant find a way to generate the refreshtoken. I tried with AllowOfflineAccess = true but its not working.

Does the client send the offline_access scope?

Haven't tried that, now its working, thanks


Inactive owners

HI Doug,
As you have mentioned on your posts I have updated all the inactive owners of the notes to an active owner as a pre step but still am receiving 'This user is inactive' error message.
I was looking at the 'ConvertNotesToContentNotesService' class and noticed below lines(#79,#80) of code where we are setting the owner of the contentNote to be same as whoever has created  the Note.
ownerId = note.createdById, // system requirement, owner and creator must be the same
createdById = note.createdById,
In this case should not we change the code to below lines of code?( if we dont then createdByid is still an inactive user as we just updated the owner to be active)
ownerId = note.OwnerId, // system requirement, owner and creator must be the same
createdById = note.OwnerId,
I have updated the code in the class manually myself but am still receiving the same error message. I have also given the user permission to  'Update Records with Inactive Owners'.
I would appreciate if you could please check those two lines of code and let me know if I am missing something.

Hi @NimaNotes,
Good catch on the createdById vs. ownerId. I'll make the change to use the ownerId instead.
If you've made the code change and still getting error, can you confirm you're in an org that's on Summer '17 (API 40.0) release? It's in the Summer release where Salesforce updated their API to respect the Create Audit Fields on ContentNote object.
Thanks,
Doug

Hi Doug,
Thanks for getting back to me on this. Yes, its summer 17 (API 17.0) release and I am getting the same error after using ownerId instead of createdById.

Thanks for the update @NimaNotes.
Yes, the Attachment's conversion uses the same logic so I'll update it too to use the OwnerId instead of CreatedById.
Thanks!

Fixed per https://github.com/DouglasCAyers/sfdc-convert-notes-to-chatter-notes/pull/5


[core] Restore constraint lambda in case of integration failure.

Currently, the lambda multipliers of the constraints are not reset to their original value if the integration step failed. This is an issue because they are used as initial guess for the constraint solver. This coupling seems to be problematic on gym_jiminy.envs:atlas environment.  Here is a snippet to reproduce the issue:
import numpy as np
import gymnasium as gym
env = gym.make("gym_jiminy.envs:atlas", debug=True)
env.reset()
env.simulator.step(0.73)
for _ in range(10):
    print("-----")
    env.simulator.step(1e-6)

An additional buffers with the values at the previous iteration should be stored next to contactForcesPrev_, fPrev_ and aPrev_.

Two options at may improve convergence:

implement SOR (successive over-relaxation with rfactor going from 2.0 to 0.0 linearly)
implement SSOR (symmetric SOR) with factor fixed to 1.0

Before implementing under-relaxation:

After (https://github.com/duburcqa/jiminy/pull/726):


Pin dependencies

This PR contains the following updates:

Package
Type
Update
Change

@aws-crypto/sha256-js (source)
dependencies
pin
^2.0.2 -> 2.0.2

@aws-sdk/client-sts (source)
dependencies
pin
^3.245.0 -> 3.245.0

@aws-sdk/signature-v4 (source)
dependencies
pin
^3.226.0 -> 3.226.0

@brightcove/kacl
devDependencies
pin
^0.1.11 -> 0.1.11

@getify/eslint-plugin-proper-arrows
devDependencies
pin
^11.0.3 -> 11.0.3

@julian_cataldo/remark-lint-frontmatter-schema
devDependencies
pin
^3.7.6 -> 3.7.6

@release-it/keep-a-changelog
devDependencies
pin
^3.1.0 -> 3.1.0

@types/aws-sdk
devDependencies
pin
^2.7.0 -> 2.7.0

@types/browserify (source)
devDependencies
pin
^12.0.37 -> 12.0.37

@types/chai (source)
devDependencies
pin
^4.3.4 -> 4.3.4

@types/clone (source)
devDependencies
pin
^2.1.1 -> 2.1.1

@types/exorcist
devDependencies
pin
^2.0.0 -> 2.0.0

@types/express (source)
devDependencies
pin
^4.17.15 -> 4.17.15

@types/fs-extra (source)
devDependencies
pin
^11.0.1 -> 11.0.1

@types/glob (source)
devDependencies
pin
^8.0.0 -> 8.0.0

@types/js-yaml (source)
devDependencies
pin
^4.0.5 -> 4.0.5

@types/mime (source)
devDependencies
pin
^3.0.1 -> 3.0.1

@types/mocha (source)
devDependencies
pin
^10.0.1 -> 10.0.1

@types/mock-fs (source)
devDependencies
pin
^4.13.1 -> 4.13.1

@types/node (source)
devDependencies
pin
^18.11.18 -> 18.11.18

@types/node-fetch (source)
devDependencies
pin
^2.6.1 -> 2.6.1

@types/pascalcase (source)
devDependencies
pin
^1.0.1 -> 1.0.1

@types/proxyquire
devDependencies
pin
^1.3.28 -> 1.3.28

@types/qs (source)
devDependencies
pin
^6.9.7 -> 6.9.7

@types/request (source)
devDependencies
pin
^2.48.8 -> 2.48.8

@types/request-promise-native (source)
devDependencies
pin
^1.0.18 -> 1.0.18

@types/semver (source)
devDependencies
pin
^7.3.13 -> 7.3.13

@types/sha.js
devDependencies
pin
^2.4.0 -> 2.4.0

@types/sinon (source)
devDependencies
pin
^10.0.13 -> 10.0.13

@types/uglify-js (source)
devDependencies
pin
^3.17.1 -> 3.17.1

@types/url-parse (source)
devDependencies
pin
^1.4.8 -> 1.4.8

@types/uuid (source)
devDependencies
pin
^9.0.0 -> 9.0.0

@types/xml2js (source)
devDependencies
pin
^0.4.11 -> 0.4.11

@types/yargs (source)
devDependencies
pin
^17.0.19 -> 17.0.19

ansi-regex
resolutions
pin
^5.0.1 -> 5.0.1

aws-sdk
devDependencies
pin
^2.1290.0 -> 2.1290.0

browserify
dependencies
pin
^17.0.0 -> 17.0.0

buffer
devDependencies
pin
^6.0.3 -> 6.0.3

chai (source)
devDependencies
pin
^4.3.7 -> 4.3.7

clone
dependencies
pin
^2.1.2 -> 2.1.2

cross-fetch
dependencies
pin
^3.1.5 -> 3.1.5

cspell (source)
devDependencies
pin
^6.19.2 -> 6.19.2

dts-bundle-generator
devDependencies
pin
^6.13.0 -> 6.13.0

es6-promise
dependencies
pin
^4.2.8 -> 4.2.8

esbuild
dependencies
pin
^0.16.16 -> 0.16.16

eslint-plugin-ban
devDependencies
pin
^1.6.0 -> 1.6.0

eslint-plugin-filenames
devDependencies
pin
^1.3.2 -> 1.3.2

eslint-plugin-local
devDependencies
pin
^1.0.0 -> 1.0.0

eslint-plugin-prefer-let
devDependencies
pin
^3.0.1 -> 3.0.1

exorcist
dependencies
pin
^2.0.0 -> 2.0.0

fs-extra
dependencies
pin
^11.1.0 -> 11.1.0

glob
devDependencies
pin
^8.0.3 -> 8.0.3

got
resolutions
pin
^12.1.0 -> 12.5.3

handlebars (source)
dependencies
pin
^4.7.7 -> 4.7.7

isolated-vm
optionalDependencies
pin
=4.4.1 || >4.4.2 -> 4.4.1

js-yaml
devDependencies
pin
^4.1.0 -> 4.1.0

jsonpath-plus
dependencies
pin
^7.2.0 -> 7.2.0

jsprim
resolutions
pin
^2.0.2 -> 2.0.2

marked (source)
resolutions
pin
^4.0.10 -> 4.2.12

mersenne-twister
dependencies
pin
^1.1.0 -> 1.1.0

mime
devDependencies
pin
^3.0.0 -> 3.0.0

minimist
resolutions
pin
^1.2.6 -> 1.2.7

mocha (source)
devDependencies
pin
^10.2.0 -> 10.2.0

mock-fs
devDependencies
pin
^5.2.0 -> 5.2.0

nanoid
resolutions
pin
^3.1.31 -> 3.3.4

node-fetch
resolutions
pin
^3.2.10 -> 3.3.0

node-fetch
dependencies
pin
^2.6.7 -> 2.6.7

open
dependencies
pin
^8.4.0 -> 8.4.0

parse-path
resolutions
pin
^5.0.0 -> 5.0.0

parse-url
resolutions
pin
^8.1.0 -> 8.1.0

pascalcase
dependencies
pin
^1.0.0 -> 1.0.0

proxyquire
devDependencies
pin
^2.1.3 -> 2.1.3

qs
resolutions
pin
^6.5.3 -> 6.11.0

qs
dependencies
pin
^6.11.0 -> 6.11.0

release-it
devDependencies
pin
^15.6.0 -> 15.6.0

remark-cli (source)
devDependencies
pin
^11.0.0 -> 11.0.0

remark-frontmatter
devDependencies
pin
^4.0.1 -> 4.0.1

remark-lint-no-undefined-references
devDependencies
pin
^4.2.0 -> 4.2.0

request-promise-native
dependencies
pin
^1.0.9 -> 1.0.9

semver
dependencies
pin
^7.3.8 -> 7.3.8

sha.js
dependencies
pin
^2.4.11 -> 2.4.11

sharp-cli
devDependencies
pin
^4.1.0 -> 4.1.0

source-map
dependencies
pin
^0.8.0-beta.0 -> 0.8.0-beta.0

stacktrace-parser
dependencies
pin
^0.1.10 -> 0.1.10

ts-loader
devDependencies
pin
^9.4.2 -> 9.4.2

ts-node (source)
devDependencies
pin
^10.9.1 -> 10.9.1

typedoc (source)
devDependencies
pin
^0.23.10 -> 0.23.10

typedoc-plugin-markdown (source)
devDependencies
pin
^3.14.0 -> 3.14.0

typescript (source)
dependencies
pin
^4.7.4 -> 4.7.4

uglify-js
dependencies
pin
^3.17.4 -> 3.17.4

url-parse
dependencies
pin
^1.5.10 -> 1.5.10

uuid
dependencies
pin
^9.0.0 -> 9.0.0

xml2js
dependencies
pin
^0.4.23 -> 0.4.23

yargs (source)
dependencies
pin
^17.6.2 -> 17.6.2

yn
dependencies
pin
^3.1.1 -> 3.1.1

Add the preset :preserveSemverRanges to your config if you don't want to pin your dependencies.

Configuration
📅 Schedule: Branch creation - "before 3am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Self-Hosted Renovate Bot. Find all logs here.

⚠ Artifact update problem
Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.
♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

any of the package files in this branch needs updating, or
the branch becomes conflicted, or
you click the rebase/retry checkbox if found above, or
you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:
File name: yarn.lock
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

⚠ Artifact update problem
Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.
♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

any of the package files in this branch needs updating, or
the branch becomes conflicted, or
you click the rebase/retry checkbox if found above, or
you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:
File name: yarn.lock
You don't have permission to create /tmp/renovate/cache/containerbase.
curl: (23) Failed writing received data to disk/application
Download failed: https://nodejs.org/dist/v18.14.2/SHASUMS256.txt


chore: move IsLinearMap theorems to their own file

Copyright from https://github.com/leanprover-community/mathlib/pull/834

Seems sensible, thanks!
maintainer merge

bors merge

bors merge


flask-appbuilder: fix flask-babel dep

Motivation for this change
Should fix this https://hydra.nixos.org/build/131278146
Things done
Fixed Flask-Babel dependency version condition

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 106369 run on x86_64-darwin 1

1 package failed to build:

apache-airflow (python37Packages.apache-airflow)

3 packages built:

python37Packages.flask-appbuilder
python38Packages.flask-appbuilder
python39Packages.flask-appbuilder

Result of nixpkgs-review pr 106369 run on x86_64-linux 1

4 packages built:

apache-airflow (python37Packages.apache-airflow)
python37Packages.flask-appbuilder
python38Packages.flask-appbuilder
python39Packages.flask-appbuilder


ByteToMessageDecoder have to watch out for empty buffers

Netty version: 4.0.31
Context:
Unexpected behavior when implementing a ByteToMessageDecoder. Following up in issue #4087.
The Javadoc of ByteToMessageDecoder urges the implementer to let another filter handle the actual frame detection, so that the implementation of ByteToMessageDecoder can focus on decode the content of the frame. Which is a good idea.
However will the decode method always be called when the channel is closed, even if there is no remaining data. This seems illogical (to me) since the prepended LengthFieldBasedFrameDecoder has the task of not pushing through data in your chain which does not have a full frame (other decoders like LengthFieldBasedFrameDecoderdo the same).
So in the end implementations of ByteToMessageDecoder and ByteToMessageDecoder (since issue #4087 is fixed) do need to check for the received buffer size before being able to decode.
Steps to reproduce:

Implement a ByteToMessageDecoder
Add a LengthFieldBasedFrameDecoder and your ByteToMessageDecoder implementation to the chain
Close the connection
decode gets called with an empty buffer

@ctron I wonder why you even implement a ByteToMessageDecoder and not a MessageToMessageDecoder ?
I think this would be the correct thing for you and also result in the desired behavoir.

Well I do have the intention of converting between bytes and messages. So the ByteToMessageDecoder with its Javadoc sounded like a good idea.

@ctron can we send byte array to netty server without using netty lib? I want to write byte array byjava OutputStream, Ios NSInputStream.

@ctron well the ByteToMessageDecoder will "buffer" data until you are ready to consume them. Which is not needed as you already cut out the frames. So MessageToMessageDecoder sounds right to me.

Same confused.

Closing this as explained above.

@normanmaurer,
I also agree with @ctron that this behavior is non-intuitive and hope this issue can be revisited.
Are you saying that for two-way conversion, MessageToMessageCodec<MyMessageType, ByteBuf> is recommended over ByteToMessageCodec? The latter seems more intuitive. Both are mentioned in Netty In Action as valid alternatives.
Also, by sending an empty buffer to decode, I think we are violating the integrity property that any message sent to decode has been previously sent. This is the code path sending empty buffers:
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
    RecyclableArrayList out = RecyclableArrayList.newInstance();
    try {
        if (cumulation != null) {
            callDecode(ctx, cumulation, out);
            decodeLast(ctx, cumulation, out);
        } else {
            decodeLast(ctx, Unpooled.EMPTY_BUFFER, out);
        }
    ...

Why is decodeLast(ctx, Unpooled.EMPTY_BUFFER, out); necessary here?
If we want people to not use ByteToMessageCodec due to this behavior, let's indicate so in the docs.
cc @trustin who is the author of this logic.

decodeLast() is called to give a decoder a chance to perform the last task like creating a new message from a cumulation buffer or releasing resources.
decodeLast() is a sort of finally block, so it must be invoked regardless if the cumulation buffer is empty or not.
The default behavior of decodeLast() is currently to delegate to decode() and thus it will drive decode() into the situation where it has to handle the case @ctron and @alexstaytuned.
Assuming that decode() is not a place to perform a clean-up task but decodeLast() is, I think decodeLast() should not call decode() if the buffer is empty.
What do you think, @normanmaurer ?

I think decodeLast() should not call decode() if the buffer is empty.

sounds reasonable to me. this behavior may extend into codecs too (HttpObjectDecoder).

Let me take care

Am 25.02.2016 um 05:47 schrieb Scott Mitchell<EMAIL_ADDRESS>I think decodeLast() should not call decode() if the buffer is empty.
sounds reasonable to me. this behavior may extend into codecs too (HttpObjectDecoder).
—
Reply to this email directly or view it on GitHub.

was fixed


Cors configuration

This should resolve #226

Coverage increased (+0.02%) to 37.331% when pulling cce38c89bf1e489ffe3ccd8663389dfddfd25140 on cors-configuration into f0457c885b92cb33fd48e42230142e56093b44ac on master.

Just tested it, if I didn't make an error, I don't see the CORS info in the response header:
Current master server response headers:
HTTP/1.1 200 OK
Date: Sat, 28 Dec 2019 15:10:21 GMT
Content-Type: application/xml; charset=UTF-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT
Transfer-Encoding: chunked

PR server response headers:
HTTP/1.1 200 OK
Date: Sat, 28 Dec 2019 15:07:29 GMT
Content-Type: application/xml; charset=UTF-8
Transfer-Encoding: chunked

Just tested it, if I didn't make an error, I don't see the CORS info in the response header:
Current master server response headers:
HTTP/1.1 200 OK
Date: Sat, 28 Dec 2019 15:10:21 GMT
Content-Type: application/xml; charset=UTF-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT
Transfer-Encoding: chunked

PR server response headers (with CORS default config in grobid-service/config/config.yaml):
HTTP/1.1 200 OK
Date: Sat, 28 Dec 2019 15:07:29 GMT
Content-Type: application/xml; charset=UTF-8
Transfer-Encoding: chunked

I tried it with ajax/jquery and it is ok ! , I ll do further tests, something weird..

I need to have a look at it again ...

Basically this implementation does not return headers if the Origin doesn't match the allowed origin or it's absent... it seems the proper way to do so, but it's not as I remember this thing...

ah my bad, I didn't put Origin: http://localhost in the requesting header indeed! I though corsAllowedOrigins: "*" would also match an absent Origin field.
Adding for instance Origin: http://localhost with the default server config works as expected - in the server response header, we have Access-Control-Allow-Origin: http://localhost Access-Control-Allow-Credentials: true.

I just added some minor corrections which were stuck in my local branch with 66de8d57eb72130b5b05002222b38f7358971bcb


Raw UDP over IPv4 traffic with fixed source and destination ports

Config

Two back-to-back connected test ports

One for tx
Another for rx

One Flow with,

Fixed frame size of 128B
Rate of 50 PPS
Packet count 100
Packet headers ethernet, ipv4 and udp

UDP header with,

Fixed source port
Fixed destination port

Validation

Push configuration
Start capture on rx port
Start transmit on tx port
Wait until following is as expected,

Frames tx/rx, bytes tx/rx and transmit state in flow metrics

Fetch capture and ensure following is as expected,

ethernet.ether_type
ipv4.protocol, ipv4.total_length
udp.src_port, udp.dst_port, udp.length
size of each frame
(best effort) checksum of all headers

https://github.com/open-traffic-generator/conformance/blob/main/features/flows/headers/udp/test_udp_header.py
https://github.com/open-traffic-generator/conformance/blob/main/features/flows/headers/udp/udp_header_test.go


Generating certs for services that do not use port 80

If I have a docker-compose file with a bunch of services, for instance mosquitto, is generating certs for such services that do not expose post 80? For instance in this case, mosquitto exposes 2 ports for SSL 8883 and 8083. Without a docker-ized solution here, the general instructions are to use certbot in standalone mode, and the configuration for mosquitto links against the letsencrypt directory directly. (reference)

Hi @SimplicityGuy
I'm not certain that I completely understood your question but you can use nginx-proxy's VIRTUAL_PORT environment variable to proxy to containers that don't listen on the standard 80 and/or 443 ports.

Hi @buchdag
In my case I have a single service which serves content in 80, 3000 and 3001 ports. Will them be mapped to my VIRTUAL_HOST:3000 and VIRTUAL_HOST:3001 ?

I have a similar situation. I'm using a nexus container and it serves 3 ports 8081, 8082, 8083. In my manual nginx config, I map them on different names. Is there a way for the companion to map to multiple virtual hosts in the one container?

I tried VIRTUAL_PORT=8081,8082,8083 with 3 matching VIRTUAL_HOSTS=nexus1.xxxx.xx,nexus2.xxxx.xx,nexus3.xxxx.xx.
I doesn't support it by the looks of it. VIRTUAL_PORT is only used when 1 port is entered.
Would be good to add no?

@moracabanas no they won't. The proxy is meant to serve content over the standard 80/443 ports (or another custom pair of ports but you'll lose the ability to use the companion), not to do something like yourdomain.tld:8000 => one container or container port, yourdomain.tld:8000 => another container or container port.
@jseparovic nginx-proxy's doc does not imply at any point that it provide this feature or that VIRTUAL_PORT accept a comma separated list. I'm not really certain that a project documentation should list every feature it doesn't have ...

I solved my issue swapping my nxinx stack with traefik which enables the use of virtual ports for any service "connected" by docker sock and the use of labels.
I also use docker swarm.
You can consider dockerswarm.rocks to learn this stack by example.
I'm now using it in production and im quite happy with it.
Meanwhile I am learning kubernetes so traefik docs are still usefull and the transition to a large load balanced stack is good.

I would recommend to use nginx to pair less complicated or microservices with a nice and simple stack with proxy load balancing. Otherwise traefik is a no brainer.
Traefik docs are really good.


Webpack support for other types of files (png,woff,woff2,eot,ttf,svg)

See #12

Wow, closed without even a comment...  :confused:

Thanks for your PR, but I ended made a one commit that close similar issues and PRs at once.


Updated script with latest CUDA version.

Reference : https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html
Fixed issue where, test case was not showing the sub-test results in test summary.

Execution is verified.
Test Results Summary
ICA test run on 09/24/2018 23:24:59
ARM Image under test Canonical : UbuntuServer : 16.04-LTS : latest
Total Executed TestCases        1 (       1 Pass,        0 Fail,        0 Abort)
Total Execution Time(dd:hh:mm) 0:0:20
XML file: TestConfiguration

#Sr. Test Name : Test Result : Test Duration (in minutes)
----------------------------------------------------
1. VERIFY-NVIDIA-CUDA-DRIVER-FOR-NC6-VM : PASS : 20.87
	PCI Express pass-through : PASS
	lspci : PASS
	lshw -c video : PASS
	nvidia-smi : PASS

Logs can be found at .\TestResults\2018-24-09-16-24-51-6986

09/24/2018 23:45:54 : [INFO ] Creating 'Azure-43601-buildlogs.zip' from '.\TestResults\2018-24-09-16-24-51-6986'
09/24/2018 23:45:55 : [INFO ] D:\GitHub\LISAv2\Azure-43601-buildlogs.zip created successfully.
09/24/2018 23:45:55 : [INFO ] Copied : .\report\report_TC-43601.xml --> .\report\report_TC-43601-junit.xml
09/24/2018 23:45:55 : [INFO ] Analysing results..
09/24/2018 23:45:55 : [INFO ] PASS  : 1
09/24/2018 23:45:55 : [INFO ] FAIL  : 0
09/24/2018 23:45:55 : [INFO ] ABORT : 0


add browser field to package.json for npmcdn

Planning on featuring this component in one of our starter kit. Would like the browser field so I can do<EMAIL_ADDRESS>without having to specify file path.

thanks, Kevin


scarb init templates

Bring capabilities of https://github.com/cargo-generate/cargo-generate into scarb new and init commands
For example:
$ scarb init --template gh:starkware-libs/starknet-contract
$ scarb init --template gh:software-mansion/protostar-project
$ scarb init --template gh:dojo-engine/dojo-project

We are abandoning working on this, as Foundry is implementing their contract project template, which basically covers our mostly needed use-case: https://github.com/foundry-rs/starknet-foundry/pull/565


Added support for initialising a "dict" from a direct ByteBuffer.

This avoid the critical region otherwise required on the byte array.

Thanks!

Thanks!
Any plans for a binary release that include this?

Yes, there is another PR in progress, I will wait for it to be ready and merged  and will release new version

I have just released 1.5.5-2


Upgrade to Docker 1.9.x

In the meantime what version should I be using?  1.91 doesn't seem to happy at all:
https://gist.github.com/faddat/5eb75e13b6e830be2122

@faddat I recently committed some code that allow versions of docker after 1.7 to be used- see this commit.
Was the error you were seeing using the most recent code from master?

Implemented in #286


Add secrets: inherit to Caller Actions.

See https://github.com/giantmade/help-for-heroes/pull/343 for explanation.

These actions are not applicable to this repo, closing this PR.


SyntaxError: invalid syntax

Hi,
I'm trying to to use the script with this command:
python ExportLists.py playlists
And i've this error:

Traceback
(most recent call last):
File "ExportLists.py", line 4, in ?
from common import *
File "/home/gle/gmusic-playlist-master/gmusic-playlist-master/common.py", line 92
if track.get('storeId') else track.get('id'))
^
SyntaxError: invalid syntax

Any ideas ?
Also, can I export all my music ? and not just all my playlists ?
Thanks in advance !

All tracks will be exported as a playlist called 'all tracks' if configured to do so in preferences.
The line throwing the error is part of a ternary conditional starting from the previous line, try putting it all on one line and see if that helps.  are you using python 2.7?

Ok :)
I installed python 2.7 and the error disappeared,
Thank you


WIP: timer that produces poisson arrivals with given constant throughput

see http://mail-archives.apache.org/mod_mbox/jmeter-dev/201609.mbox/<CAB%3DJe-EBhT-hD-oAorpnyfR%2By%3DrPo6J1W4L96ND_WvmWm2sB1A%40mail.gmail.com>

Hello,
Thanks for the patch.
First review look good to me.
Any reason for not using ThreadLocalRandom.current() instead of Random ?
Regards
Philippe

Any reason for not using ThreadLocalRandom.current() instead of Random ?

TLR does not support setSeed. setSeed is required to get reproducible workloads.

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadLocalRandom.html#setSeed(long) : Throws UnsupportedOperationException. Setting seeds in this generator is not supported.

Hi,
Thanks for the patch too.
I don't understand how to use "Test duration" parameter
Thank
Antonio

I don't understand how to use "Test duration" parameter

The timer is random. Suppose one wants to have "100 samples per hour".
Suppose the test duration is 30 minutes.
The "expected" number of samples is 50. Do you remember that the timer is random. That basically means the number of produced samples for 30 minutes might be 50, it might be 40, and it might easily be 60. At the end of the day, poisson arrivals are not guaranteed to produce exactly 50 samples when running for 30 minutes under 100 samples/hour load.
In practice, everybody wants "round" numbers in the load reports, so "test duartion" parameter allows the timer to account for the expected test duration.
So, the timer ensures that the number of samples would be exactly 50.
Does that make sense?

Thank to the quick answer
"so "test duration" parameter allows the timer to account for the expected test duration."
And if "test duration" parameter is not equal to expected test duration?
E.G. if I define a Loop count and not a Duration in the thread group
I think it's a GUI too complex and I search how to simplify it
I will do more test asap
Antonio

And if "test duration" parameter is not equal to expected test duration?

Nothing breaks. The idea is as follows: each "test duration" the timer produces exact number of samples.
For instance, if configured with "100 samples per hour, test duration 30 min", then

By 30min into the test there will be exactly 50 samples produced
By 60min since the beginning there will be exactly 100 samples produced
By 90min since the beginning there will be exactly 150 samples produced
...

However, the patterns between 0m..30m, 30m...60m, and 60m...90m will all be different. In other words, the timer will create different schedules for each 30m timeframe.

I think it's a GUI too complex and I search how to simplify it

Yeah, the UI looks complicated, however I did try to remove non-required elements.
On top of that, we might need to introduce "variable rate" somehow. For instance, in expression format: like ramp(10/hour..100/hour in 10 min) keep(30min) ramp(..10/hour in 10 min)

Hi,
My opinion is that the GUI is too complex because :
Too much parameter
You try to make too much different things in it
And these things

"Specific number of iterations per hour".
-> I think the GUI need to be like "Constant Throughput Timer", or better in Thread Group (and I think "Constant Throughput Timer" need to be in Thread Group too)

"Bursty load"
I don't know which parameter to use

"Repeatable test profile"
I don't know which parameter to use

"Avoid all thread groups"
In my opinion, need to be in Thread Group

In my opinion if you want to have all this functionality in only one element, a new GUI is necessary.
e.g. A list to choose the goal ("Specific number of iterations per hour", "Bursty load", "Repeatable test profile","Avoid all thread groups" and Custom (like today))
When we select one, only needed parameter are displayed
I have made some basic tests and I am impressed by the precision :-)
Do you plan to make some docs/examples ?
Antonio

Too much parameter

Well, UI is built by JMeter OOB: https://github.com/apache/jmeter/pull/231/files#diff-fc396f633dbdaf0a98c5472613e9190dR25
Do you know if BeanInfoSupport can create "foldable" UI?

Do you plan to make some docs/examples ?

docs -- sure.

Hi Vladimir,
No idea for the UI
Anybody have the answer?
To help me to understand how to use it, can you provide me some JMeter jmx file for each goal ("Specific number of iterations per hour", "Bursty load"....) with a short description (in comment)?
Thank
Antonio

@ra0077 , have you seen the tooltips for the input values of the timer?
I'm sure those should explain all the things.

Yes I have read it but I have some strange behavior in some test
It's why I would like to know how do you use it
Antonio

I have some strange behavior in some test

What is the behavior?
Don't get me wrong, but I think the UI should be obvious. If the UI is obscure and/or the tooltip does not help, then it is a good point to update the UI.

Hi,
In my test plan I have 3 http request and one Exponential Timer at the same level
When I run my script, only first http request are executed whereas with "Constant Throughput Timer", I don't have this problem

Antonio

In my test plan I have 3 http request and one Exponential Timer at the same level

That does not work like this.
Please wrap "expenential timer" into some sampler (e.g. debug action or whatever).
In JMeter all timers/listeners are inherited by sibling elements, thus when you place a timer at the same level as "3 http requests", you basically invoke the timer for each of the page, and that is something odd.
You should have the plan as follows:
DebugSampler
   ExpTimer
http1
http2
http3

Hello,
I will commit this PR unless there is a NO GO within 2 days.
Thanks

I will commit this PR unless there is a NO GO within 2 days.

Philippe, thanks for looking into this. I'm afraid the documentation is missing.

Hi @vlsi ,
Could you confirm your component implements the following features:

"Specific number of iterations per hour".
The very basic requirement is to ensure you end up exactly 50
iterations per hour.
Business customers would not understand if you report load test
results with 47 executions "just because the random was random".
All timers except "Constant" can easily send excessive requests or
send less requests than required. There is no way to control it.

"Bursty load" simulation. There is no easy way to test "50
iterations per hour as 10 bursts of 5 items".

"Repeatable test profile". All the random timers produce different
pattern on each test run. This is not good for low-level analysis
(e.g. compare of CPU% charts, etc).

"Avoid all thread groups" to fire at 00:00:00. By default, all
thread groups would fire at 0, so there would be noticeable spike at
the start of the test.

Thanks

Could you confirm your component implements the following features:

That's true.

Hi @vlsi ,
I tried the component and if I have this:

AS you can see I don't get exact executions.
So am I missing something ?
Thanks

AS you can see I don't get exact executions:

@pmouawad , that's a good question.
The timer uses testStarted (see https://github.com/apache/jmeter/pull/231/files#diff-e5e4a43d21590a051458ab1f0fa8898dR92 ) to get the timestamp of the test start.
As you configure 10000 samples per 180 seconds, it assumes it should generate 10000 samples since the test start.
You are using "startup delay=1sec" in thread scheduler, that causes threads to delay its startup by 1 second, and the threads are run for 180 seconds. So the total test duration becomes something like 181 seconds. The added one second easily gives additional 50-70 samples as you are targeting for 55.5 samples/sec.
The timer itself does not limit the number of samples.
It is not clear what is the use of the non-zero delay and it is not clear how it should play with "test duration" either. Frankly speaking, I would refrain from setting non-zero startup delay.

Hi @vlsi ,
Thanks for your feedback.
I tried many configurations including thread delay to 0, and I never really reach exact sampling on duration.
Is it really supposed to be exact ?

Is it really supposed to be exact ?

Let me ask another question: "is thread scheduler=120sec supposed to make the test to last exactly 120 seconds"?
Note: timers do not limit the number of samples in JMeter. If you want to limit the number of iterations (e.g. to make exactly 10000 samples), you'd use additional counter or something like that.

When I try "startupdelay=0, duration=120sec, threads=10, throughput=10000", I get 10011 samples in the aggregate report. I think it good enough to be considered exact.

Note: target throughput is 10000/120=83.3 samples/sec.  11samples/(10000/120)=0.13 sec. That is those 11 "extra" samples can be due to 130ms spent on initialization/finalization of the test (e.g. in between "test started" and "threads started")

"exact number of samples" makes sense for cases like 10-100-1000 per HOUR, not 5000 per minute.  I don't think there are requirements to aim for "exactly 10000 per minute". In case there are that requirements, I would still discuss that in a separate PR as it would be tightly related with thread startup time, etc.
Have you tried to test rates like 0.5...1 per second?

Hi @vlsi ,
Thanks for your clarifications.
It is just that as per your confirmation, one of the aims of this new component was:
"The very basic requirement is to ensure you end up exactly 50
iterations per hour.
Business customers would not understand if you report load test
results with 47 executions "just because the random was random"."
That's why I tested such things, so I think it needs to be mitigated a bit.
I tested with 1000 hits for 2 minutes and I get 1002 hits so ok.
Now regarding "Repeatable test profile", for 2 runs, I don't see "similar test profile", but it is maybe just a matter of interpretation, what is exactly expected in your aim for "Repeatable test profile"
Finally regarding Batched Departures, I am not sure to understand how it is supposed to influence load and if it is supposed to work correctly with First part in GUI.
Thanks for clarifications.
Regards

Philippe: It is just that as per your confirmation, one of the aims of this new component was:
Vladimir: "The very basic requirement is to ensure you end up exactly 50 iterations per hour.
Philippe: That's why I tested such things, so I think it needs to be mitigated a bit.

Please, be careful: I suggested 50 per hour, and you are testing 50 per second. That is 3600 times higher rate than I suggested. I don't mean the timer works for 50/hour rate only, but 50/second is a bit different thing.

Now regarding "Repeatable test profile", for 2 runs, I don't see "similar test profile"

Let us consider "10 second test with 1 per second on average".
 log record with expected timestamps of the samples.
2017-12-23 21:39:01,449 INFO o.a.j.t.ConstantPoissonProcessGenerator: Generated 11 timings (ExponentialTimer 10 required, rate 1.0, duration 10, exact lim 20000, i11) in 0 ms, restar
t was issued 12 times. First 15 events will be fired at: 0.11401676510964737 (+0.11401676510964737), 0.6130748250774025 (+0.49905805996775515), 3.342639426533685 (+2.7295646014562824
), 3.377172823174598 (+0.03453339664091315), 5.112423358789579 (+1.735250535614981), 5.1755972505795125 (+0.06317389178993338), 5.514713410628992 (+0.3391161600494792), 7.94457420086467 (+2.4298607902356784), 8.509235493336268 (+0.5646612924715981), 8.5888989163364 (+0.07966342300013096), 10.107204382018718 (+1.5183054656823192)

A bit more readable: 0.1, 0.6, 3.3, 3.4, 5.1, 5.2, 5.5, 7.9, 8.5, 8.6, 10.1 seconds. Note: there's exactly 10 samples in between 0..10 sec.
Let me restart the test:
2017-12-23 21:40:19,752 INFO o.a.j.t.ConstantPoissonProcessGenerator: Generated 11 timings (ExponentialTimer 10 required, rate 1.0, duration 10, exact lim 20000, i11) in 0 ms, restart was issued 12 times. First 15 events will be fired at: 0.11401676510964737 (+0.11401676510964737), 0.6130748250774025 (+0.49905805996775515), 3.342639426533685 (+2.7295646014562824
), 3.377172823174598 (+0.03453339664091315), 5.112423358789579 (+1.735250535614981), 5.1755972505795125 (+0.06317389178993338), 5.514713410628992 (+0.3391161600494792), 7.94457420086467 (+2.4298607902356784), 8.509235493336268 (+0.5646612924715981), 8.5888989163364 (+0.07966342300013096), 10.107204382018718 (+1.5183054656823192)

Note: the timestamps are exactly the same (the log timestamp is different, so it is a different execution).
This is super-useful when testing low-rate things (e.g. 100 per hour) as it enables to have the same load profile, thus there's less moving parts when comparing different runs. Otherwise it is hard to tell if the difference in "CPU usage" profile is caused by different sample sequence or different system under test behavior.
Note: in order for "repeatable test profile" to work, you need to set non-zero "randomSeed" (see https://github.com/apache/jmeter/pull/231/files#diff-76a5e8f118fb5e533f52ea80fd10dd0bR58 )
Unfortunately, JMeter is unable to initialize "test element property with a random-generated value". Test elements are created with constant values only, so I had to resort to a 0 being "random". I agree this is more like "easier for developer" rather than "easier for JMeter user", however I'm not sure what is JMeter way of doing that config.

Finally regarding Batched Departures, I am not sure to understand how it is supposed to influence load and if it is supposed to work correctly with First part in GUI.

The idea is "batched departures" keep the load level (total number of samples per hour), however the samples are scheduled to come in pairs, triples, etc.
For instance, with "batch size=2, throughput=10, duration=10" it produces samples as follows: 0.52, 0.52, 0.67, 0.67, 3.4, 3.4, 9.37, 9.37, 9.43, 9.43.
Does the above make sense?
PS. That is why I was saying the documentation is missing

@pmouawad , thanks anyway for giving it a spin.
Do you think it requires changes?
If not, I think I can prepare a doc draft.

Hi @vsi,
It looks ok to me, if you can add doc as separate PR it would be great !
Thanks

@pmouawad , just a quick note: I do not like "exponential" name very much. It is scientific, however it has 0.01% probability to be well-understood by regular engineers. What do you think regarding the name?

@vlsi ,
What about:

Poisson Arrivals timer ?
Precise Throughput Timer ?

"Precise Throughput Timer" sounds goods.
PS. There's "Poisson Random Timer" in JMeter: http://jmeter.apache.org/usermanual/component_reference.html#Poisson_Random_Timer  I guess it is used much less often. Frankly speaking, I do not see a use case for it.
Should we somehow deprecate it?

Hi,
Thank to your (Vlad and Philippe) for your works
I don't think to put Poisson it's a good idea because if user don't know poisson law, he will not understant the goal of this element
+1 for Precise Throughput Timer
Antonio

Is there a better name which makes the difference between the Constant Throughput Timer clearer, or should that be deprecated or just replaced with this?

Hi @vsi,
When do you think you’ll be able to provide a documentation PR ?
Thanks


Do a review and plan improvements to our Flowershow/PortalJS app template

PortalJS / Flowershow is now quite heavily used by us and others. It would be good to do a review of the template to see how we can improve or clean-up in various areas.
NB: Flowershow template is normally the base for PortalJS templates so that is the focus.
Acceptance

[x] Reviewed main areas of e.g. flowershow template https://github.com/datopian/flowershow
[x] Identified any areas for improvement et

Tasks

[x] Do a first pass and flag the main areas to look at @olayway ⏲2h

Ola's notes
Potential areas for improvement:

fix search feature finally... 🔥 not sure what's the issue so hard to estimate, but my guesstimate is 2h
content/config.js file - review customisation options and field names, include all available options in the file for the user (even if with default values or commented out), make sure all the fields are well documented ⬆ ~ 2h
eject Layout component 🔥 ~ 2h

currently imported from @portaljs/core but more often than not it is subject to customisations
only specific UI/logic parts it encapsulates should be componentised and imported in the ejected Layout that lives in the template - this way users (and we) could easily remove/add/swap only specific parts of it - atm if we only want to update e.g. the Navbar, we can't, we have to recreate (or copy) the whole Layout

upgrade to App Router, e.g. for: ⬆ ~ 1/2-1 day

build-in SEO support
easy fetching any layout related frontmatter fields by the Layout itself (no need to pass from page -> to custom app -> to Layout)
...

add back eslint and prettier as dev deps for tech savy users convenience and for ourselves ⬇ ~ 0.5h
we still don't have template versioning/releases set up properly at Flowershow template 🔥 ~ 1/2 day

it would make our lives easier to compare apps built with the template and the original template itself, and thus quickly asses if we need/want to upgrade, just by looking at the CHANGELOG
the same goes for all Flowershow users...

Specific parts of code that could be improved:
https://github.com/datopian/flowershow/pull/582/files

@rufuspollock first pass done

Great, can you update acceptabce and suggest specific next steps if any with estimate

We have a list of improvements and a PR. Only suggestion for @olayway is to perhaps classify by value and cost e.g.

this is a must-have and takes probably ~1d etc.

Once that is done can you close as FIXED.

Prioritised and "guesstimated"