daos: enhance use of lookup hash

Question @adammoody
As the DAOS-related code grows, would you prefer we move DAOS-specific functions to some new daos_util.[h,c]? E.g. hash functions, setup functions.
(But keep the daos_* IO functions in mfu_io.c)
lookup_insert_dir and associated functions have been
refactored and expanded.
mfu_param_path.h

Added dfs_hash to mfu_file.
Removed only_daos from mfu_file, since it is no longer needed

Naming convention

Changed mfu_* to daos_* since they are only used by DAOS.
Changed lookup_insert_dir to daos_lookup_hash.

daos_lookup_hash and associated functions

Gets the hash from mfu_file instead of a global.
Uses a pointer to the name instead of static allocation,
which saves space.
Updated error messages to use MFU_LOG instead of stderr directly.
Added a _new and _delete function for the object handle struct.
Properly deallocate and free each entry.

daos_ IO functions
With the excpetion of daos_opendir and daos_closedir,
all daos_ IO functions now use daos_hash_lookup to
get directory entries.
mfu_util

Added daos_mount as a wrapper for dfs_mount.
Added daos_umount as a wrapper for dfs_umount.
This also destroys the hash.

dcp.c
Now uses the new daos_mount and daos_umount functions.
closes #384
closes #387

Thanks @daltonbohning .  This includes a bunch of nice additions, and thanks for the detailed PR comments.  This all looks good to me other than a few comments.
Yes, I think we as grow the DAOS code, it'll make sense to create a daos_util file to hold the various DAOS-specific functions.

Thanks @adammoody! I'll get these fixed tomorrow, and figure out when/how to best abstract to daos_util

@daltonbohning looks like one of the builds is failing

@dsikich Hmm. It looks like it was only one of the builds, not both. It seems the wget lwgrp failed (network issue?)
I'll rerun this to see if it was just a fluke in the network.
$ lwgrp=lwgrp-1.0.2
0.07s$ wget https://github.com/llnl/lwgrp/releases/download/v1.0.2/${lwgrp}.tar.gz
--2020-09-29 16:10:06--  https://github.com/llnl/lwgrp/releases/download/v1.0.2/lwgrp-1.0.2.tar.gz
Resolving github.com (github.com)... <IP_ADDRESS>
Connecting to github.com (github.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 429 too many requests
2020-09-29 16:10:06 ERROR 429: too many requests.
The command "wget https://github.com/llnl/lwgrp/releases/download/v1.0.2/${lwgrp}.tar.gz" failed and exited with 8 during .

@dsikich Looks like it built this time! Do you see anything in the code that might be an issue either now or in the future?
(I have ran all the current functional tests against this, with no failures)

@daltonbohning no, I don't see anything that stands out to me. Thanks!

@dsikich Thanks!


How to choose only JSON methods when import this repo?

When I tried to use CodeEditTetView on macOS, and only used to parse JSON, I found that it depends on many parsing libraries in other languages, is there a way to remove other libraries according to personal needs?
Because I only need methods related to json parsing.

Thanks!

They get all bundled at build-time. Once you run the app only parsers for languages you actually have opened get instantiated (lazy-loaded).

Thanks for your response!
When I import this repo, I got some errors.

Fix incoming! STTextView implemented some non-backwards compatible features on Swift 5.7.

Got it, I need upgrade to Swift 5.7 version and try to it.
Thanks so much!

Okay, thanks!


CMake error when installing

When I try installing snail on my mac I am receiving CMake errors both using "pip install ." and "python setup.py develop". I also tried installing snail on a Linux cluster, and I also receive CMake errors. I have attached the errors I receive on both as well as the CMake error log.
linux_error.pdf
mac_error.pdf
mac_error_log.pdf
mac_error_pip.pdf

Thanks for the error report. Could you try pulling in the latest changes and running pip install . again? I've had a go at reworking the build configuration and dependencies.
See updated notes on different installation methods in README#installation.

Hi @dianajjaramillo is this all fixed for you now?
Either route should work:

pip install -e . from a checkout of this repository
pip install nismod-snail with no need to clone this repository, downloaded by pip from https://pypi.org/project/nismod-snail/


Add Unsupported error for cases when storcli returns Un-supported.

The SAS3508 returns Un-supported when you ask it:
storcli /c0/dall show all
storcli /c0/vall show all

I'm not sure how we should query this card, but for the time being
just add a new error type of Unsupported.

@raharper adjusted, thank you for careful review.

Closing this in favor of #100.


DST started on March 8th instead of March 13th for 2016

On v2.12.0 I noticed that moment considers 03/08/2016 08:00:00 UTC to be the start of DST.
However, DST starts at 03/13/2016 08:00:00 UTC.
Example:
var orderTimeField = '03/08/2016 08:30:00';
var orderDate = moment.utc(orderTimeField, 'MM/DD/YY hh:mm:ss');
Gives a date object with my local time of 03/08/2016 03:30:00 CDT -05:00
Javascript Date object gives a local time of 03/08/2016 02:30:00 CST -06:00

The problem here is that you're using a two digit year token for a four digit year. It's parsing the first two digits of your year and putting you in the year 2020 instead of the year 2016:
var orderTimeField = '03/08/2016 08:30:00';
var orderDate = moment.utc(orderTimeField, 'MM/DD/YY hh:mm:ss');
orderDate.toDate().toString()
"Sun Mar 08 2020 03:30:00 GMT-0500 (Central Daylight Time)"

If you use a 4 digit year token, everything works fine:
var orderTimeField = '03/08/2016 08:30:00';
var orderDate = moment.utc(orderTimeField, 'MM/DD/YYYY hh:mm:ss');
orderDate.toDate().toString()
"Tue Mar 08 2016 02:30:00 GMT-0600 (Central Standard Time)"

In 2020, DST does in fact begin on March 8th.

Thanks Maggie


Cannot find module @angular/material

I'm trying to add Angular Material to my application but it first required me to upgrade to Angular5 which was fine. But now it's not working as it couldn't find the Material module although I've installed it following the getting started tutorial with:
npm install --save @angular/material @angular/cdk

The error I'm getting is literally:
`ERROR in src/app/app.module.ts(6,52): error TS2307: Cannot find module<EMAIL_ADDRESS>
Which correspond to this import instruction:
import { MatButtonModule, MatCheckboxModule } from '@angular/material';

Is this caused because Angular Material is still an RC version?
Thanks.

Works fine to me. Have you tried to restart angluar cli? Or remove node_modules/ then run npm install again :)

@isra-fel Thanks for your response. I actually deleted the whole project and re-generated it with the newly-installed @angular/cli 1.5 (before that I've removed the whole directory ~/.node_modules where the previous version of @angular/cli was installed, as I set npm config set prefix ~/.node_modules).
Can I know which versions of Material, Animations, CDK you've installed? For me when I installed them I always get a warning that the version required isn't installed (some require @angular/core 5.0.1 other the 5.0.0 version).

@h4k1m0u my material & cdk are 5.0.0-rc0, animations is 5.0.0

Did you actually import for your parent angular module?
@NgModule({
    imports: [
        MatButtonModule,
        MatCheckboxModule
    ]
})

@merlosy Yes I already have those imports in my app.module.ts and in the right place.
@isra-fel I'm not 100% sure it's because of the versions but I've noticed that @angular/material doesn't appear when I do ng --version at the root of my project. Here is its output:
Angular CLI: 1.5.0
Node: 8.7.0
OS: linux x64
Angular: 5.0.1
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

@angular/cli: 1.5.0
@angular-devkit/build-optimizer: 0.0.32
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.35
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.0
@schematics/angular: 0.1.2
typescript: 2.4.2
webpack: 3.8.1

Another thing to mention is that when I install material with npm install -g @angular/material I get the following output (that's why I'm wondering whether materialv5.0 doesn't have some issues with angularv5.0.1):
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

+<EMAIL_ADDRESS>updated 1 package in 2.957s

@h4k1m0u Can you try installing material locally instead of -g?

Thanks very much @isra-fel without the -g parameter it's working. It managed to install @angular/material: 5.0.0-rc0 while angular and the rest of the librairies are v5.0.1:
Angular: 5.0.1
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

Why would you install these as global dependencies?
Your package.json should imho be enough to recreate your project deps, bar node of course.
That also means typescript, webpack etc should be installed locally.

restart vscode

install the npm install --save @angular/material @angular/cdk in  your project directory, as well as the ng add angular/material
it should have to work

I am using ubuntu 16.04 LTS. I faced same problem while getting started with  angular material
I followed all the steps from Angular Material
Then once again suing sudo su I used npm install --save @angular/material @angular/cdk.
It worked..


Migrate to custom checkbox UI

What
This PR migrates our checkboxes to a custom implementation in re_ui. Currently, the only change is the removal of the extension effect. In the future, the customisation might go deeper (e.g. custom tick mark, a variant for popup menu, etc.), possibly all the way to copy/pasting and rewriting the entire checkbox code.
This PR also introduces a lint to forbid the use of ui.checkbox().
Fixes #2727
Checklist

[x] I have read and agree to Contributor Guide and the Code of Conduct
[x] I've included a screenshot or gif (if applicable)
[ ] I have tested demo.rerun.io (if applicable)

PR Build Summary
Docs preview
Examples preview

cc @martenbjork


Error in Clustal W and repAlignLineage

❓ Questions and Help
Hello,
I'm a new user of immunarch and I've question about the Clustal W.
Fisrtly, I have already installed Clustal W according to the guide: https://www.howtoinstall.me/ubuntu/18-04/clustalw/. Then, run the app of ClustalW2.exe.
Finally, I input the following code in Rstudio, but error happened.
What can I do for solve this issue, thank you very much.

Code(example code, without changing)
data(bcrdata)

bcr_data <- bcrdata$data %>% top(500)
bcr_data %>%

seqCluster(seqDist(bcr_data), .fixed_threshold = 3) %>%

repGermline(.threads = 1) %>%

repAlignLineage(.min_lineage_sequences = 2)

Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed!\n",  :
repAlignLineage requires Clustal W app to be installed!
Please download it from here: http://www.clustal.org/download/current/
or install it with your system package manager (such as apt or dnf).
——————————————————————————————————————————
By the way，I have another question，I found that it instead of accumulating counts of the same clonotype, the V and U gene usage analyses seemed to count only those corresponding to clonotype. Is that true? How can I figure it?
Thank you, I am looking forward to your reply.
Dr. Huang

Hello, Dr. Huang!
I'm Aleksandr Popov, a developer of Immunarch package. Thank you for using our software!
About your 1st question: on some systems, Clustal executable is named clustal2 instead of clustal. There is now a fix in Immunarch that adds support for it, but now it's only in development branch; it will be included in the next release. You can install immunarch from a branch with this fix using the following commands:
install.packages(c("devtools", "pkgload"))
devtools::install_github("immunomind/immunarch", ref="bcr-pipeline-fixes")
devtools::reload(pkgload::inst("immunarch"))

Best regards,
Aleksandr

Hello， Professor Aleksandr Popov
Thank you reply to me so quickly.
I have tried to install immunarch from a branch with this fix using the following commands. However, the previous problem remains——"Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed! \ n ". Besides, the software which I use is clustal w2.1. Thank you for helping me to solve this problem!
I am looking forward to your reply!
Sincerely，
Dr Huang
At 2022-10-12 21:34:29, "Aleksandr Popov" @.***> wrote:
Hello, Dr. Huang!
I'm Aleksandr Popov, a developer of Immunarch package. Thank you for using our software!
About your 1st question: on some systems, Clustal executable is named clustal2 instead of clustal. There is now a fix in Immunarch that adds support for it, but now it's only in development branch; it will be included in the next release. You can install immunarch from a branch with this fix using the following commands:
install.packages(c("devtools", "pkgload"))
devtools::install_github("immunomind/immunarch", ref="bcr-pipeline-fixes")
devtools::reload(pkgload::inst("immunarch"))
Best regards,
Aleksandr
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Please check your PATH variable, whether it includes the path to clustalw or clustalw2 executable. You can check it with which clustalw and which clustalw2 commands on Linux or Mac, and with where clustalw2 on Windows. If these commands don't return a correct path to the program, probably you need to add the path manually. You can search online how to edit the PATH variable in your operating system or distro.
Cheers,
Aleksandr

Hello， Professor Aleksandr Popov
I have tried to install immunarch from a branch with this fix using the following commands. However, the previous problem remains——"Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed! \ n ". Besides, the software which I use is clustal w2.1. How can I solve this problem？
I am looking forward to your reply!
Sincerely，
Dr Huang
At 2022-10-12 21:34:29, "Aleksandr Popov" @.***> wrote:
Hello, Dr. Huang!
I'm Aleksandr Popov, a developer of Immunarch package. Thank you for using our software!
About your 1st question: on some systems, Clustal executable is named clustal2 instead of clustal. There is now a fix in Immunarch that adds support for it, but now it's only in development branch; it will be included in the next release. You can install immunarch from a branch with this fix using the following commands:
install.packages(c("devtools", "pkgload"))
devtools::install_github("immunomind/immunarch", ref="bcr-pipeline-fixes")
devtools::reload(pkgload::inst("immunarch"))
Best regards,
Aleksandr
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hello， Professor Aleksandr Popov
I have tried to install immunarch from a branch with this fix using the following commands. However, the previous problem remains——"Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed! \ n ". Besides, the software which I use is clustal w2.1. How can I solve this problem？
I am looking forward to your reply!
Sincerely，
Dr Huang
At 2022-10-12 21:34:29, "Aleksandr Popov" @.***> wrote:
Hello, Dr. Huang!
I'm Aleksandr Popov, a developer of Immunarch package. Thank you for using our software!
About your 1st question: on some systems, Clustal executable is named clustal2 instead of clustal. There is now a fix in Immunarch that adds support for it, but now it's only in development branch; it will be included in the next release. You can install immunarch from a branch with this fix using the following commands:
install.packages(c("devtools", "pkgload"))
devtools::install_github("immunomind/immunarch", ref="bcr-pipeline-fixes")
devtools::reload(pkgload::inst("immunarch"))
Best regards,
Aleksandr
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hello Professor Aleksandr Popov,
Thank you for your reply before.
I have add the new path variable to system, but the R studio remain can not be operated.
Thank you very much,  I am looking forward to your reply!  urgently！！
At 2022-10-17 21:12:03, "黄燕" @.***> wrote:
Hello， Professor Aleksandr Popov
I have tried to install immunarch from a branch with this fix using the following commands. However, the previous problem remains——"Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed! \ n ". Besides, the software which I use is clustal w2.1. How can I solve this problem？
I am looking forward to your reply!
Sincerely，
Dr Huang
At 2022-10-12 21:34:29, "Aleksandr Popov" @.***> wrote:
Hello, Dr. Huang!
I'm Aleksandr Popov, a developer of Immunarch package. Thank you for using our software!
About your 1st question: on some systems, Clustal executable is named clustal2 instead of clustal. There is now a fix in Immunarch that adds support for it, but now it's only in development branch; it will be included in the next release. You can install immunarch from a branch with this fix using the following commands:
install.packages(c("devtools", "pkgload"))
devtools::install_github("immunomind/immunarch", ref="bcr-pipeline-fixes")
devtools::reload(pkgload::inst("immunarch"))
Best regards,
Aleksandr
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hi, Huang!
If you created a new variable, it's probably a wrong place, because every system must have the existing PATH variable. Here are the instructions how to change it on Windows https://www.wikihow.com/Change-the-PATH-Environment-Variable-on-Windows, Mac OS https://stackoverflow.com/questions/7703041/editing-path-variable-on-mac and Linux https://superuser.com/questions/488173/how-can-i-edit-the-path-on-linux . After this, reboot and check if your path to Clustal is now included in your PATH (echo $PATH on Linux/Mac, echo %PATH% on Windows). Also try to launch clustalw / clustalw2 from command line to check that OS can find it.
If it works, run R and execute Sys.getenv("PATH") command to check that the path visible from R, and then Sys.which("clustalw") and Sys.which("clustalw2") to check that at least one of these 2 commands returns not empty path.
Cheers,
Aleksandr

Hello，Aleksandr！
Thank you for your early reply！
I have already changed the PATH on Windows11（Figure1）. Now，I am sure that the path to Clustal is included in my PATH（Figure2）. However, the previous problem remains——"Error in require_system_package("clustalw", error_message = paste0("repAlignLineage requires Clustal W app to be installed! \ n ". I have no idea to figure out it. Thank you for helping me for a long time.
I hope you happy everyday, sincerely.
Dr Huang.
Figure1
Figure2
Figure3
At 2022-10-18 22:26:55, "Aleksandr Popov" @.***> wrote:
Hi, Huang!
If you created a new variable, it's probably a wrong place, because every system must have the existing PATH variable. Here are the instructions how to change it on Windows https://www.wikihow.com/Change-the-PATH-Environment-Variable-on-Windows, Mac OS https://stackoverflow.com/questions/7703041/editing-path-variable-on-mac and Linux https://superuser.com/questions/488173/how-can-i-edit-the-path-on-linux . After this, reboot and check if your path to Clustal is now included in your PATH (echo $PATH on Linux/Mac, echo %PATH% on Windows). Also try to launch clustalw / clustalw2 from command line to check that OS can find it.
If it works, run R and execute Sys.getenv("PATH") command to check that the path visible from R, and then Sys.which("clustalw") and Sys.which("clustalw2") to check that at least one of these 2 commands returns not empty path.
Cheers,
Aleksandr
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hey @huangyan530
I was running into a similar issue on mac. I had clustalw installed and could run it via command line.
I ended up reinstalling clustalw using conda, then pointing RStudio to my conda:
Command Line:
 conda install -c bioconda clustalw

Rstudio:
Sys.setenv(PATH=paste0('/Users/ncborch/opt/anaconda3/bin:', Sys.getenv('PATH')))
#Will need to change the path of this to your conda instance

It was only then I got:
Sys.which("clustalw2")

clustalw2  "/Users/ncborch/opt/anaconda3/bin/clustalw2"

Hi, @huangyan530 and @ncborcherding!
I've made a tutorial how to install Clustal and Phylip on Windows and add paths to executables, so R can find them:
Clustal.and.Phylip.installation.tutorial.zip
On macOS there is /etc/paths file, where you can manually add paths to installed and unzipped executables; here is the instruction: https://www.architectryan.com/2012/10/02/add-to-the-path-on-mac-os-x-mountain-lion/
After editing /etc/paths, I think you also need to restart Rstudio to make Sys.which() command find executables on these paths.
Best regards,
Aleksandr


Only show the file name when processing sarif logs

While processing the SARIF logs for display in the viewer, it is attempting to show the entire URI which never fits in the progress window. Just show the filename.

I believe this is in the context of loading. Since #348 we no longer upgrade logs and haven't needed that progress window. Thus closing.


IOT endpoint Timed out after 30sec. Connection couldn't be established.

Hi,
I keep running into the following issue when trying to call the proxy chrome set up when I try call chromeless with the remote set up.
"Timed out after 30sec. Connection couldn't be established."
I tried from Windows, Ubuntu and from another AWS Lambda always resulting in the above error message.
Have completed step 1 for the proxy set up and deployed The Proxy service to AWS Lambda.
Steps I have done

Changed the url for awsIotHost in the serverless.yml
Changed region in the  serverless.yml
ran npm run deploy successfully and made note of the api key and url.

But the after that I am unable to call this proxy without running into the error "Timed out after 30sec. Connection couldn't be established.".
The script I am running is pretty simple i.e.
  const chromeless = new Chromeless({
    remote: {
      endpointUrl: 'https://xxxxxxxxxx.execute-api.eu-west-1.amazonaws.com/dev',
      apiKey: mykey
    }
  });

const links = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .evaluate(() => {
      // this will be executed in headless chrome
      const links = [].map.call(document.querySelectorAll('.g h3 a'), a => ({
        title: a.innerText,
        href: a.href
      }));
      return JSON.stringify(links);
    })
    // you can still use the method chaining API after evaluating
    // when you're done, at any time you can call `.then` (in our case `await`)
    .scrollTo(0, 1000);

Always results in

node chrome.js
Error: Timed out after 30sec. Connection couldn't be established.
at Timeout._onTimeout (C:\dev\cb-headless-chrome\node_modules\chromeless\dist\src\chrome\remote.js:73:52)
at ontimeout (timers.js:488:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:283:5)

Is there some kind of permission, role I need that maybe is not in the documentation ?
Would love some help since I have no idea how to progress on this, even some idea which url it can't connect to ?
EDIT : After some more debugging I am finding its the IOT Host url that I can't connect to i.e.
wss://xxxxxxx.iot.eu-west-1.amazon.com/.....
Is there a step I am missing to allow connection to this ? I can see when I put my IOT endpoint url I can see the same kind of timeout behaviour.

Hi @stevenfitzpatrick Hm.. To help debug this, let's confirm that the session Lambda handler is working correctly. Could you try the curl command I describe here and share all of the result?

Yes region is eu-west-1
Output here

timeout on name lookup is not supported
Trying <IP_ADDRESS>...
TCP_NODELAY set
Connected to 7hyy4u6p15.execute-api.eu-west-1.amazonaws.com (<IP_ADDRESS>) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
successfully set certificate verify locations:
CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
CApath: none
TLSv1.2 (OUT), TLS header, Certificate Status (22):
TLSv1.2 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS change cipher, Client hello (1):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
ALPN, server accepted to use h2
Server certificate:
subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.execute-api.eu-west-1.amazonaws.com
start date: May 30 00:00:00 2017 GMT
expire date: Feb 28 23:59:59 2018 GMT
subjectAltName: host "7hyy4u6p15.execute-api.eu-west-1.amazonaws.com" matched cert's "*.execute-api.eu-west-1.amazonaws.com"
issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 Secure Server CA - G4
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Using Stream ID: 1 (easy handle 0x68a6f0)

GET /dev HTTP/1.1
Host: 7hyy4u6p15.execute-api.eu-west-1.amazonaws.com
User-Agent: curl/7.51.0
Accept: /
x-api-key: lSEeUDsSP32pEyjsz7biR9FIk3jAsRB62xy79cbq

Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< content-type: application/json
< content-length: 812
< date: Sun, 30 Jul 2017 15:15:11 GMT
< x-amzn-requestid: e1add1b3-7539-11e7-8f36-991948b29d85
< x-amzn-trace-id: sampled=0;root=1-597df7ff-2e7c5cabed1df2f207fe8858
< x-cache: Miss from cloudfront
< via: 1.1 e5740b731d03d61279af4365058ca2f2.cloudfront.net (CloudFront)
< x-amz-cf-id: Q6jCgVvwx62Ex4fA6SyXlTYYYOjdvXh77eXPDapk-7gLoqI51cGNPQ==
<
{"url":"wss://a3i6gl60ogso1d.iot.eu-west-1.amazon.com/mqtt?X-Amz-Date=20170730T151511Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJ7MYCYZUZTZ4LYWA%2F20170730%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f1bf85968b0bd88626412c6acda946bd62a51a9fb5bb5bfe2949735150d923e6&X-Amz-Security-Token=FQoDYXdzEJD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDFUUf%2BsRSvInzKqjLyL6AfemldFtN14xyBp57G5gP3amsxoJUpWnbdebTkIr9evdL4GgeklMaK1vLywOjp1p6c3hwTF7LdTCUqpWLfOinbGZazNfRTexc%2FLmfTk%2F%2FZvAFom6P481B6Xd5RkUBunRwPc784Ah9F4iR3h%2FF5L7zMJpnM50Y7T1vL9yVvJuzKjn7kEKyCLuV%2FOm9lIPfCMiPEcCA%2BpKcppJUO2aWXIs4bthHlz4y%2BQ6DagAysDjyWKNrI1Nnvl7OjdffyWDB8QnAtX9OJ7IBZZ0hIxiljyv5MyHKtQRfvptP7IRUvodxJ8kURIXNz23fMToX2Dapkfy0FVwcvxfDhv8AhUoi%2Bj3ywU%3D","channelId":"cj5qval12000401p49j1ks9rl"}* Curl_http_done: called premature == 0
Connection #0 to host 7hyy4u6p15.execute-api.eu-west-1.amazonaws.com left intact

One thing I didn't understand from the documentation is which you define as the "Session" GET end point.
I.e after npm run deploy, you get the following end points.

endpoints:
GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/version
OPTIONS - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/
GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/

Is that last get end point the one you consider the session one ?

@stevenfitzpatrick yes that's correct—GET /dev endpoint is the new-session endpoint. I'll clarify that in the documentation.

The only other thing I wanted to check was inserting the IOT Endpoint in the serverless.yml. I simply replaced the variable

"${env:AWS_IOT_HOST}"

with string

"xxxxxxxxxxx.iot.eu-west-1.amazonaws.com"

Is that also correct ?

The timeout seems to be happening here in chrome/remote.js
channel_1 = mqtt_1.connect(url, {
                          will: {
                            topic: 'chrome/last-will',
                            payload: JSON.stringify({ channelId: channelId_1 }),
                            qos: 1,
                            retain: false
                          }
                        });

Funny behaviour that I noticed when I print out the url it is connecting to.

wss://xxxxxxxxxx.iot.eu-west-1.amazon.com/mqtt?X-Amz-Date

The url has been changed from amazonaws.com to amazon.com at this point

@stevenfitzpatrick yes that's also correct 😄

ok then I am kinda at a loss why it's not working. I am pretty sure it's due to this line
wss://xxxxxxxxx.eu-west-1.amazon.com
Is there maybe a role I need to add to my credentials, some kind of cert installed on my machine etc ?

@stevenfitzpatrick about amazonaws.com -> amazon.com .. that might be the problem. As far as I know it should be amazonaws.com Something weird must be happening for the domain to get changed.. In aws4 package's code we use to create the presigned URL, there's no mention of amazon.com.

My chromeless version is 1.0.1 and serverless is 1.18.1

@adieuadieu Totally my fault I put down amazon.com instead of amazonaws.com in the serverless config.

my serverless version 1.8.1 ,  when use curl , i got `< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 815
< Connection: keep-alive
< Date: Wed, 02 Aug 2017 05:25:16 GMT
< x-amzn-RequestId: f7e6475d-7742-11e7-9732-d536e1f97d30
< X-Amzn-Trace-Id: sampled=0;root=1-5981623c-8fdf8fd359b663c2a046fa79
< X-Cache: Miss from cloudfront
< Via: 1.1 296daa1256a646119edb8b5abbf1c774.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 6WVkXcnEiqM5h5pu-EYaBESjSrUmfcJfPjy5sAOoAEzPFE9HwZEl4Q==
<

Connection #0 to host km83etka57.execute-api.eu-west-1.amazonaws.com left intact
{"url":"wss://a389xbjfoei0cx.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Date=20170802T052516Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAI4ZGA6WPV6G3Z6AQ%2F20170802%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=a0c1984f283eea68c77b93ccff27125c8d913bd16837f24dba75cfea4ea01cea&X-Amz-Security-Token=FQoDYXdzEM7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDJcSjIPaYhqUe44t3yL6AQZg7AXnEWaRJjBhYiysWjL5955OBlBFrgrz7UZY5s7DZSgj8ehhtaQRdB7Nx52w%2FucP9gQGARX1UCH%2FjUSEHuQtCrDLihM3zBkigXtL0%2F9Nzq9Lu%2FbWAuKJGUqOx8aA9Kzi%2BxxKFrGuIP0EyX6MU513rQ%2FdafH7r8LYQVTUkt1raHJcH3HRjGz6CiXV2Yq507BdVopiaqFjdosjEZDya4AXI%2FSDC2FE0drpFHFNBVru6bjT33Yk1jFW130sOP4HlkGVXBoywc2bbu1Mlmz0NTcNaYwNF1ndDWErjHm9hh1Y42iBIgOLWplfTyhZD8%2B06dI%2FhME3kPc54bsoh7yFzAU%3D","channelId":"cj5ukjhzt000101p6gqqbt09p"}`

but still got 30 seconds timeout ,  could anyone help me figure it out ? thanks

@adieuadieu I'm having the same issue:
If I do the curl everything is fine:
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-2.amazonaws.com (<IP_ADDRESS>) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-2.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev/ HTTP/1.1
> Host: xxxxxxx.execute-api.eu-west-2.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: MY_API_KEY
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 831
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 14:30:37 GMT
< x-amzn-RequestId: 517f1e75-7858-11e7-8b92-1976755a6dfc
< X-Amzn-Trace-Id: sampled=0;root=1-5983338d-3baf6c0a22e7da0a000369e8
< X-Cache: Miss from cloudfront
< Via: 1.1 2297ae65d06c070d63c49ed7259f5fbb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 8ioi901ViVi65L8YOH0pR9COW2YdxAmqeSUpTJoRTeJALmKvF_RmIQ==
<
* Curl_http_done: called premature == 0
* Connection #0 to host xxxxx.execute-api.eu-west-2.amazonaws.com left intact
{"url":"wss://https://xxxxx.iot.eu-west-2.amazonaws.com/mqtt?X-Amz-Date=20170803T143037Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASSAQEQIN6YSWTTMJH2A%2F20170803%2Feu-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=9b9fa351facf65c88364667acd84855a294d7d44cf08e7af4cf22b44621fa9dc&X-Amz-Security-Token=SOME_TOKEN","channelId":"cj5wjgoaf000201qcr7efo48q"}

I decided to redo everything again, this time I used a different time region,  eu-west-1
Still getting timeouts, but a different message when using debug:
Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/fabio/playground/chromeless-test/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:469:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:264:5)

the curl works fine fine:
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-1.amazonaws.com (<IP_ADDRESS>) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev HTTP/1.1
> Host: xxxxx.execute-api.eu-west-1.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: xxxxxx
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 825
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 16:33:32 GMT
< x-amzn-RequestId: 7ce040e4-7869-11e7-a3e9-e326343e7894
< X-Amzn-Trace-Id: sampled=0;root=1-5983505c-2aa9848cd55bdb1bfaf046c8
< X-Cache: Miss from cloudfront
< Via: 1.1 7cddfedd82c4e2198668a64be4934fd0.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: jjwhqOOqqdiNzwvOvDPtn8XqlQSn1qgM4bCGVeDcz60T7A97QJd14A==
<
* Curl_http_done: called premature == 0
* Connection #0 to host 4sqfkimk5h.execute-api.eu-west-1.amazonaws.com left intact
{"url":"wss://xxxxxx.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Date=20170803T163332Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJUVCD2JRUATTLOUA%2F20170803%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=3d1a5501384450b98cfde614ce023b3d8d304dc2fbae414e9d69845549ecffc4&X-Amz-Security-Token=token","channelId":"cj5wnuq9q000301qmplem18li"}

Hi @FabioAntunes at the moment I'm not really sure what might be causing this. I have a number of Proxy improvements and refactors I'm working on which will provide more verbose debug output for tracking these issues down.

We're getting the same error. Looking in the lambda logs I'm seeing
Unable to import module 'src/run': Error

It appears that this line is causing the lighthouse-logger to not be packaged, which is currently required by the chrome-launcher. After commenting it out on our end, we were able to get farther but something is still not right.  Now we're seeing from the test suite:
TypeError: Cannot read property 'port' of undefined

Having an identical issue to @danthegoodman

I downloaded the chromeless zip from S3 and extracted and npm install again with node.js 6.10.3 and uploaded again, and I was able to pass the errors that @danthegoodman and @rfink faced, but now my CloudWatch log shows these messages and errors:
@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

And in my local shell, I'm still getting 30sec the timeout error.

I'm having the same problem with @Kivol, and still found no solution. In fact, I followed the exact the same steps, but no good.

Hi @danthegoodman @rfink @Kivol the Unable to import module 'src/run' and 'port' of undefined errors are now fixed in master branch and will be available in the next release (v1.2) sometime later today. A temporary work-around is to downgrade the chromeless package to<EMAIL_ADDRESS>in serverless/package.json.

Thanks @adieuadieu ! Downgrading to @1.0.1 worked for @danthegoodman and I.

@adieuadieu I am still getting 'port' of undefined and I downgraded to chromeless 1.0.1. Is the next release still on track for release soon?

v1.2.0 has been released. Please update and let us know how it goes. Remember to do a npm install/yarn install after fetching/pulling latest from the repository.

I think I found one relevant bit here... Seems that the regions have to match currently. The iot endpoint has to be in the same region as the functions.

I've updated to the latest version this morning but am still seeing the same Unable to import module 'src/run': Error issue.
I also tried going back to 1.0.1 as suggested as a workaround but am getting the same result.
Is there anything I could be doing wrong? Really not sure what else to try

I've upgraded to v1.2.0 and tried removing the whole stack with ./node_modules/.bin/serverless and deployed again with npm run deploy.
I still see the following error from chromeless-serverless-dev-run lambda function log:
@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

I've resolved the error above. Since I deployed in Windows, the Chrome binary seems to lack the +x permission when uploaded to S3 and lambda. When I tried in Linux it works fine now!
Thanks for the support.

There's a known issue with Serverless packaging executables on Windows upstream in https://github.com/adieuadieu/serverless-chrome/issues/21 and upstream from there in https://github.com/serverless/serverless/issues/3557.
Two work-around solutions are in progress in serverless-chrome:

download chrome on lambda invocation (ick)
check chrome executable permission on invocation. If missing, copy chrome to /tmp (only place we have chmod permission), and chmod +x, then launch (ugh)

Both are lame, but it doesn't seem like the root issue will be addressed in https://github.com/serverless/serverless/issues/3557 anytime soon.

I can confirm all issues resolved from me, using Linux and chromeless 1.2.0. Great job guys! This project is moving quickly.

I have the same issue as @scosbet has...
Unable to import module 'src/run': Error
Did everything that suggest here, nothing helps.

I'm seeing the same issue, only the logs give me the following error:
2017-08-26T15:39:45.237Z	663f6765-8a74-11e7-9bad-9be4bd6fbac9	Error: spawn /var/task/node_modules/@serverless-chrome/lambda/dist/headless_shell ENOENT
at exports._errnoException (util.js:1018:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
at onErrorNT (internal/child_process.js:367:16)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)

I'm using chromeless 1.2.0 and deploying from a Mac. Any pointers on what might be going wrong? I've run the curl command and I'm getting a valid response back.

Still having the same issue, getting the time out and Unable to import module 'src/run': Error in Lambda logs. Running off of master with Node 8.4 and built/deployed from OSX 10.12

hi @FabioAntunes , have you fixed the issue ? i still got this problem after upgrade to 1.3.0

@shawnLiujianwei I haven't used this in a while I'm waiting for Chromeless team make the switch to the Puppeteer.
What I did back then was pulling the master branch, remove all the lambda functions:
./node_modules/.bin/serverless remove

And then deploy them again following the instructions

@FabioAntunes  got it , thanks

Anyone else running into this?
Timing out according to the function's max duration. At a loss.
Best way to debug this?

I've also been having trouble with this, I think others above are right, something's happening with the build/deploy.
I was deploying with node v6.10.3 but the problems went away when building with node v8.10.0 and I can still call it with node v6.10.3
There's a discrepancy on the deployed zip size:

node v6.10.3 -> 53.7M
node v8.10.0 -> 73.8M

I've also noticed that the serverless version:

deploys chromeless 1.3.0 (current version is 1.5.1)
the bucket naming in the serverless.yml doesn't take into account the stage
for now updating the engine required in the package.json to the earliest one that works may be enough

I'm working to a couple of deadlines at the moment otherwise I would drop in a PR

I experience the timeout error as well: "Chromeless Proxy disconnected due to inactivity (no commands sent for 30 seconds)." I've tested with the example provided in the readme.md.
I've done the same as OP: Changed the URL for awsIotHost and region in the serverless.yml and then deployed with npm run deploy which returns the API URL.
My setup is Ubuntu 16.04 LTS. I've deployed using Node JS 8.10 without any luck to AWS Lambda running Node JS 6.10. I've also tried with a fresh image of Ubuntu 16.04 LTS and Node JS 6.10, but no luck here.

I have the same issue with "chromeless": "1.5.1" and OSX.

@adieuadieu Please find below response I got on doing CURL as you mentioned above. I am getting undefined in place of IOT url can you let me know what am I doing wrong?
{"url":"wss://undefined/mqtt?X-Amz-Date=20180511T101714Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAI4LIZPF4NJ2OGH2A%2F20180511%2Fus-east-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=55290f84fcf79ce0d4c0b00b8e11dfc4d83147bed613b6de5fa7199dac5e6196&X-Amz-Security-Token=FQoDYXdzEBIaDExQLk7bn%2B427UBuhiL6AdOtStxl8WkNxaw3tlARJiqodmv9dXOLVQTAS2qupRUd6p93gwsJAC29UDIShhsTTTm%2Ba4qe7AkR0zoRo%2B%2FRkqEc5ujOmrTVC0%2F6AjR8YIioVoLvNEbrjkgAKBbeIY50bUlc9HOivDu9CjvSuAekud93wnIZqRE4nELiYOupU9L7mJqeOGgQdIK54DRF0yhD0lET9kNKpiOc2rLpQ%2Bm1WZqSnfRalvUnUXbWX%2FSW7FZy%2BKavxQKMlE37PeEjsCB%2FFo4tknvNGCnZ5ckyjzYeGxIecoJML%2F%2BE7CyAgzS5d7frC05Km2BODQirdEB4dhN8PvGobXirLbCTY6AonKnV1wU%3D","channelId":"cjh1t56so000401qep2un747z"}

Chiming in with the same issue. It started consistently occurring once we had > 10 image tests.
Unfortunately I haven't been able to track down why the proxy is causing this issue. That said, it's becoming a very serious issue for our team as we're trying to to run 100s of tests but our CI can't consistently pass.
I'm happy to work on a PR if someone can help with some direction.

I do have the same issue. Anybody with more info?

Same issue here. Can curl with no problem and timeout occurs when running code. Shame, this was a perfect solution to something I was working on.

I'm also experiencing this issue - is there any guidance on where we should begin to look to potentially find a fix?

I am also having this issue. Is there any update on this?

I am also having this problem.  Running on Node v8.1.0, Serverless 1.27.3, chromeless 1.5.2.
Curl works without any problems.
I get this output to the console...
Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
at Timeout._onTimeout (/Users/.../node_modules/chromeless/dist/src/chrome/remote.js:73:52)
at ontimeout (timers.js:488:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:283:5)
from line 96 of remote.ts....
channel.on('connect', () => {
if (this.options.debug) {
console.log('Connected to message broker.')
}
So.. from the output.. it looks like it gets here, but then nothing else happens so the timeout never gets cleared.
Any thoughts on what might be the cause of this?  Or how I could debug this further?
Thanks!

@atebyagrue can you tell me more specific?

@cannalee90, after I deployed the solution, the endpoints will be displayed. When I added one of these endpoints to my solution, I did add the trailing slash / at the end of the url. So, instead of
export CHROMELESS_ENDPOINT_URL=https://XYZXYZXYZ.execute-api.us-east-1.amazonaws.com/dev/
I used
export CHROMELESS_ENDPOINT_URL=https://XYZXYZXYZ.execute-api.us-east-1.amazonaws.com/dev
This is what was causing my failure, even though curl worked.

I have the same issue:
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/marekvybiral/projects/chromeless/serverless/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:475:11)
    at tryOnTimeout (timers.js:310:5)
    at Timer.listOnTimeout (timers.js:270:5)

Tried every solution suggested in this thread without any luck. 😥 It looks like it has no problem connecting to WebSocket and subscribes to mqtt thread but then no message is being received so it throws this error.

HI, I am getting the same error. Anyone who was able to solve it is available to help?
error { error: Error: Timed out after 30sec. Connection couldn't be established.
4q9qv
2018-10-29 16:16:28-03:00 at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
4q9qv
2018-10-29 16:16:28-03:00 at ontimeout (timers.js:475:11)
4q9qv
2018-10-29 16:16:28-03:00 at tryOnTimeout (timers.js:310:5)
4q9qv
2018-10-29 16:16:28-03:00 at Timer.listOnTimeout (timers.js:270:5)
4q9qv
2018-10-29 16:16:28-03:00 => awaited here:
4q9qv
2018-10-29 16:16:28-03:00 at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
4q9qv
2018-10-29 16:16:28-03:00 at Promise.asyncApply (imports/api/news/server/news-methods.js:22:7)
4q9qv
2018-10-29 16:16:28-03:00 at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 }
4q9qv
2018-10-29 16:16:30-03:00error { error: Error: Timed out after 30sec. Connection couldn't be established.
4q9qv
2018-10-29 16:16:30-03:00

@stevenfitzpatrick Have you found a solution ?


Write a JavaScript program to print numbers from n to 1

Description
Write a JavaScript program to print numbers from n to 1
Input  : 5
Output : 5 4 3 2 1

How to contribute

Save the solution in program/PrintNumbersFromNTo1.js file
Add PrintNumbersFromNTo1.js file in print-numbers-from-n-to-1 folder

Hello, I have sent a PR for this issue.

!assign


Improve developer experience with bazel

What this PR does / why we need it:
Follow up of #2099. This focuses on making the developer cycle more convenient and faster:

Make efficient use of the bazel server by using the same build arguments for all commands. Prior to that we were  regularly invalidating our own cache when formatting code, generating code, and so on.
Use nogo, which runs vet checks (and more checks) in parallel to the actual go build.
Run gazelle, buildifier, goimports and shfmt on every build action, so that people don't have to run make bazel-generate. gazelle ignores in this case all vendor dependencies, where  people normally never touch things. The calls are made cheap in this case and don't add  much to the execution time.
Make the bazel related make commands the entrypoints and move the pure-go targets to extra targets. This affects: make test, make build, make push and make all.

The first time you do a make cluster-sync, this will still download a lot of bazel dependencies, but follow-up runs are now very fast while coding.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2106
Special notes for your reviewer:
NONE

This one should now also be ready. Let's see.

As a first step though, i'm hesitant to drop this change onto everyone until we gain some more experience with it. Would it be practical for us to maintain our current make targets the way they currently are today, and introduce new make targets for people who want to adopt more bazel functionality?

Hm, I don't think that there are more make-targets left than make and make test which work without bazel. Since we didn't want to maintain bazel and Dockerfiles, you can only push kubevirt via bazel and you can only use the providers with bazel. Also make generate needs bazel and users need to update bazel files (right now explicitly via make bazel-generate) to pass CI, which would be optimized here and users don't even have to be aware of it, like with our code formatters.
The changes do not impact the go-code itself. You can still run arbitrary go commands against our code-base in general. Also you can use make go-all and make go-test and make go-build if you still want them.
But, at the end I can keep it separated a little bit longer, if we want that.

Additional things that we are missing, it's to sync binaries from bazel out directory into _out directory after the build.

Additional things that we are missing, it's to sync binaries from bazel out directory into _out directory after the build.

Added.

@davidvossel would you be ok with having parallel bazel rules (so keep the original ones like you suggested), with one exception, and that is the invocation of all formatters and BUILD updates also when the go commands are invoked?

would you be ok with having parallel bazel rules (so keep the original ones like you suggested), with one exception, and that is the invocation of all formatters and BUILD updates also when the go commands are invoked?

is there a reason why we can't maintain exactly the previous behavior for a little while?

No it is possible, it is just very inconvenient like it is right now. There is no way to pass ci without a properly updated bazel config and there is no way to run make generate without having bazel involved. Also cluster-sync requires it. There is no reason to keep the other targets free of bazel. I want to make this as painless as possible for engineers. I would even go as far as saying that the experience is pretty broken now, after my and artyoms changes and I want to bring it back to a consistent state.
Will switch the targets back, so that the pr can move forward.

No it is possible, it is just very inconvenient like it is right now.

i understand. maybe I'm just being over cautious here.
i don't have a complete understanding of the risks involved with this PR.  If you all are confident the risk of disruption for people is low, then I'm fine with this either way with regards to the make targets. I just know how frustrating it is to have productivity disrupted when something in the environment alters behavior in an unexpected way.

ci test please

Hello @rmohr: I am not very familiar with Bazel. But can we make some changes to this PR to support cross-compilation of binaries needed for Kubevirt to work on non-x86 platforms(like ppc64le) as well?

@nitkon I think it does not in the scope of the current PR, and it will need much more work(get right RPM's), but you are welcome to open an issue(truly I think we already have one for the ARM arch)

@cynepco3hahue : Sure. I will work on that PR. Can you point me to the issue you mentioned above?

@nitkon Will try to find it :smiley:

@nitkon also happy to help you with that, once I find some time. In general, you will have to do the following things:

Create a base image for pcc in https://github.com/kubevirt/libvirt and add it to the WORKSPACE file
Add all RPMs which we reference also for ppc to the WORKSPACE file
Add select statements to all bazel targets where you have to choose the right base image and the right rpms.
Run the bazel run //:push_images target in hack/bazel-push-images.sh with the right --platform flag


feat:add 'force' param in sync methods for copy file forcely

Codecov Report
Patch coverage: 96.33% and project coverage change: -0.32% :warning:

Comparison is base (6d0618d) 98.40% compared to head (1011618) 98.09%.
Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
- Coverage   98.40%   98.09%   -0.32%
==========================================
  Files          37       39       +2
  Lines        5146     5237      +91
==========================================
+ Hits         5064     5137      +73
- Misses         82      100      +18

Files Changed
Coverage Δ

megfile/stdio.py
100.00% <ø> (ø)

megfile/stdio_path.py
100.00% <ø> (ø)

megfile/http_path.py
93.25% <71.42%> (-6.75%)
:arrow_down:

megfile/s3_path.py
98.33% <93.18%> (-0.44%)
:arrow_down:

megfile/lib/http_prefetch_reader.py
96.77% <96.77%> (ø)

megfile/lib/base_prefetch_reader.py
98.40% <98.40%> (ø)

megfile/cli.py
98.67% <100.00%> (+<0.01%)
:arrow_up:

megfile/errors.py
99.47% <100.00%> (ø)

megfile/fs.py
100.00% <100.00%> (ø)

megfile/fs_path.py
99.68% <100.00%> (ø)

... and 8 more

... and 1 file with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Difference Array & Fenwick Tree

Please put these file in their respective folders.
On Fri, 1 Oct, 2021, 7:33 pm Ashutosh Aswal, @.***>
wrote:

You can view, comment on, or merge this pull request online at:
https://github.com/rahulsain/CPP-DSA/pull/10
Commit Summary

Add Difference Array Implementation
https://github.com/rahulsain/CPP-DSA/pull/10/commits/61787cf0996861644fd927fd52d2366531340a9e
Add Fenwick Tree Implementation
https://github.com/rahulsain/CPP-DSA/pull/10/commits/d00efc1b34bb3fd82eb1f7820b632b5b6758c5f0

File Changes

A Phase 1 Placement Specific/31. Difference
Array/difference_array.cpp
https://github.com/rahulsain/CPP-DSA/pull/10/files#diff-7c62a962d9dea068af022c9a77c8011a1c32f480c04aff3d0e67e6349b11d551
(112)
A Phase 1 Placement Specific/32. Fenwick Tree/fenwick_tree.cpp
https://github.com/rahulsain/CPP-DSA/pull/10/files#diff-89d9f2264ccca980bb27563952ba05aaad993d40002113ac92e62a5947580f4f
(70)

Patch Links:

https://github.com/rahulsain/CPP-DSA/pull/10.patch
https://github.com/rahulsain/CPP-DSA/pull/10.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rahulsain/CPP-DSA/pull/10, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMB2LNNS4ZKERTVW2FSXDKTUEW5TDANCNFSM5FEYTAPQ
.

it might go to tree section

I thinking adding label will do the work no need to merge any PRs

I have moved the Difference Array to the Array folder and created a new category for Trees as there wasn't any folder for it.

Merge is required as just adding the label is not working.

I have moved the Difference Array to the Array folder and created a new category for Trees as there wasn't any folder for it.
 Merge is required as just adding the label is not working.

Reload two or three times, it will work


build script fails when folder path has uppercase letters

Sail Version: #.#.#
Laravel Version: 8.x
PHP Version: 7.4
Database Driver & Version:

Description:
Not sure if this is the right repository...this is about the online installer at https://laravel.build to create and run a docker container for a sample Laravel installation.  When the absolute path to the folder in which you want to install has uppercase letters, the installation fails with docker: invalid reference format: repository name must be lowercase.
The fix would be on line 11 of the generated shell script to replace pwd with pwd | tr '[:upper:]' '[:lower:]'
Steps To Reproduce:

create a folder with 1 or more uppercase letters in the name.
from within that folder, run in CLI curl -s "https://laravel.build/example-app" | bash

Reopen this with a the correct version


looking only down

Hi I have a 2D map, it has about 1764 tiles. If I go to reduce the coordinates, then the path is instant, but if I go to the increase, then he is looking for a very long time.

Sorry, it was a mistake in my code.


how to preserve original formatting

I would like to parse a document, remove several ast elements and serialize it to a file. I don't want to change formatting of the whole document. Is it possible?

Yes, if you're okay working with the CST parser API, and you're careful about what you're doing. Stringifying the CST will use the original characters, except for the nodes for which you specify a value. Do note that the CST API for collections in particular is a bit burdensome, as it needs to be able to handle even broken YAML.
Working with the AST level is much nicer, but from there the stringifier no longer takes into account the original source, and instead uses its own rules for formatting, in particular for indentation.
If you're interested in fixing this, I'd welcome a proposal for how to preserve and use indentation information from the source at the AST level.

So i can:

parse a document with keep CST
use document AIP to find AST nodes which should be removed
remove CST nodes associated with found AST nodes
serialize CST tree

like that?
In order to remove CST nodes from a collection would it be enough to remove them from items?

It worked, thx!


How to allow any data- attribute and class by the parser?

http://jsfiddle.net/crl/u7gLdsh1/3
How would can I tell the parser to preserve data-* attributes? (for example when the html is using some UI lib)
Also I don't understand how is the parser actually working with classes: wysihtml5ParserRules.classes is set to 'any' so I thought all classes would be preserved but no, and when I put
wysihtml5ParserRules.type_definitions.text_formatting_object.classes['active']=1; it allows active and others (sounds a bit magic).
But like for data-* I'd need a way to allow any classes without having to put them all in the parser

Would like to know also how telling the parser to preserve data-something attributes?

@pulges sorry to ping you directly, http://jsfiddle.net/crl/u7gLdsh1/10/ the data-thing attribute is lost when going in source mode

Would you give browser and version where the attribute is lost in the example?
Also made an example of undocumented feature for allowing data- attributes for all tags (can be used for some specific tags similarly) http://jsfiddle.net/u7gLdsh1/11/
Also documented the undocumented features.
https://github.com/Voog/wysihtml/wiki/parserRules#attributes

ah thanks much, it was in wysihtml5ParserRules.attributes ok, they are not cleaned now
how do you explain that wysihtml5ParserRules.attributes='any' doesn't work though? http://jsfiddle.net/crl/u7gLdsh1/12/

In the example I think the any works but you also have to look at the tag definition if the tag is allowed
For example in span case it is:
 "span": {
        "one_of_type": {
            "text_formatting_object": 1,
            "text_color_object": 1,
            "text_fontsize_object": 1
        },
        "keep_styles": {
            "color": 1,
            "backgroundColor": 1,
            "fontSize": 1
        },
        "remove_action": "unwrap",
        "check_attributes": {
            "id": "any"
        }
    },

All spans that do not fit the test of being either text_formatting_object, text_color_object or text_fontsize_object are removed by un-wrapping.
The types are defined under type_definitionssection.
Usually it is better to take the parser rules set from advanced.js or simple.js as base and take over features from "advanced and extended" that are needed, to avoid confusion. The extended is mostly for reference of all possible features.
Also allowing all possible attributes on all tags is currently not implemented. wysihtml5ParserRules.attributes='any'

Thanks for explanation @pulges

Thanks starting to see more how it works, it's ok for tags because it makes sense to restrict tags


The input element loses the focus on the search page

🐛 Bug Report
Prerequisites

[X] I'm using the latest version of Docusaurus.
[X] I have tried the npm run clear or yarn clear command.
[X] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
[] I have tried creating a repro with https://new.docusaurus.io
[] I have read the console error message carefully (if applicable)

Description
If we are redirected to the search page and try to search for something, it is almost impossible to type. After typing a symbol, the engine makes a search, but the input element loses the focus!
Have you read the Contributing Guidelines on issues?
Yes.
Steps to reproduce

Open the link:
https://docs.dhtmlx.com/suite/search/
Try typing any text.
The input element loses the focus.

Expected behavior
The input element should have the focus when we type the search query.
Actual behavior
The input element loses the focus when we type the search query.
Your environment

Public source code:
https://github.com/DHTMLX/docs-suite
Public site URL:
https://docs.dhtmlx.com/suite/
Docusaurus version used:
2.0
Environment name and version (e.g. Chrome 78.0.3904.108, Node.js 10.17.0):
91.0.4472.106
Operating system and version (e.g. Ubuntu 20.04.2 LTS):
Xubuntu 18.04.5

Reproducible demo
Demo:
https://docs.dhtmlx.com/suite/search/
Video:
https://files.dhtmlx.com/30d/ce41960b564a0b809964b3fa545f8b42/vokoscreen-2021-06-25_17-16-38.mkv

:o yes this is weird thanks for reporting

Will be fixed in https://github.com/facebook/docusaurus/pull/5104


Transparant Splash Screen

Description
Transparant splash screen does not work on linux (does on win 10).  Command line options are passed as follows:
[
  '/home/benn/datapaq5/target/release-builds/dp5-linux-x64/dp5',
  '--enable-transparent-visuals',
  '--disable-gpu'
]

Environment

electron-splashscreen version: 1.0.0
Electron version: 8.8.1(packaged using electon-packager)
OS: Ubuntu 20.04

Steps to Reproduce
Create a dynamic splash screen with trasparant: true as shown below:
  const splash: Splashscreen.DynamicSplashScreen = Splashscreen.initDynamicSplashScreen({
        windowOpts: windowOptions,
        templateUrl: path.join(__dirname, '/../dynamic-splashscreen.html'),
        delay: 0, // force show immediately to better illustrate example
        splashScreenOpts: {
            height: 900,
            icon: __dirname + path.sep + '..' + path.sep + '..' +
							  path.sep + 'src' + path.sep + 'assets' +
						      path.sep + 'images' + path.sep + 'dp5-256.ico',
            width: 700,
            transparent: true,
            webPreferences: {
                nodeIntegration: true,
                backgroundThrottling: false
            },
        },
    });
  win = splash.main;

  return splash;
}

Expected behaviour
A transparant splashscreen
Actual behaviour
A splashscreen with a white background

Update the same thing happens with the plain command 'npm run electron --enable-transparent-visuals --disable-gpu'

This is not the electron splash screen but instead Elctron.  I fyou want to fix it then put a wait of 500ms on the app ready call.


Catch correct exception when fields are missing during decryption

Currently the exception thrown during decryption (https://github.com/paragonie/ciphersweet/blob/86d951ef458626b9e1671e66be4f833ec75480fd/src/EncryptedRow.php#L535) is not caught. It also does not extend ErrorException but rather Exception.
This causes an issue when fetching only specific properties of a model.

This change breaks the tests. Could you have a look?

@freekmurze It appears the tests are actually breaking on main currently. I'l see if it's a quick fix and open another PR for that if it is.

Could you rebase with current main to fix the tests here as well?

Done 👍

Thanks!


Setting a 'Company Name' in the Configuration UI no longer sets the root tenant name, despite 'Use Configuration Settings'

With versions of ManageIQ prior to Darga, setting the default 'Company Name' in the Configure -> Configuration setting of the UI set the new default name for tags and categories and also the root tenant name.
With Darga RC1 setting a 'Company Name' has no effect, I still see 'My Company Tags', 'My Company Categories', and if I have 'Use Configuration Settings' enabled, my root tenant is still 'My Company'.

@pemcg it appears with the Hammer release being current, this issue can be closed. If not, please remove the "stale" label.


sigproc.ewm - Fixes #47 by expanding dims of scale_arr

fixes #47 - scale_arr now has same ndim as buffered data, which allows for > 2-dim input.

I'd like to have a test written before merging this one; I'll take a crack at that in the coming days.  I suspect this will function just fine, but this unit is quite complicated and used in some important places for us at the moment.


Rewrite SH1106 library into a loadable driver

Rewrite SH1106 library functions to be a loadable Elf/OS driver mapped to the O_VIDEO vector.

Done in separate project Elfos-SPI-OLED-Drivers


Need to Downgrade to gcc 7 to build lib with cuda

FYI
when running 'make' in lib, nvcc complained that it could not find a version of gcc 7 or less. I am running version 8.1
I created at the parent of lib & tools /cuda/bin. In bin 'ln -s  gcc'
In /lib/setup.py I added the argument "--compiler-bindir=../cuda/bin" to extra_compile_args= for nvcc.
This removed the compiler complaint from nvcc.

The symbolic link points to gcc-7. nvcc complains in host_config.h that the
gcc compiler must be 7 or less. Part of #if GCNUC ... #error ...
On Mon, Nov 5, 2018 at 12:27 AM Chen Wu<EMAIL_ADDRESS>wrote:

Thanks. Interesting! Does your symbolic link point to a gcc 7 (or less),
or does it still use gcc 8.1. Just wanted to know if the error is because
nvcc does not work with gcc 8.1 or is it because it cannot locate any gcc.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/chenwuperth/rgz_rcnn/issues/6#issuecomment-435759223,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGCKZ7Co_fx22kxLMf0AFWYsq-9i2BMFks5ur8xUgaJpZM4YNj1J
.

--
"Don't play what's there, play what's not there." - Miles Davis
“Read the directions and directly you will be directed in the right
direction.” — Lewis Carroll
"Science is the belief in the ignorance of experts." - Richard Fenyman

I am closing this ticket as the ClaRAN v0.2 is out (perhaps checkout if you have some time), and it does not require extra CUDA code to do NMS or other bounding box calculations.


Unable to migration autoincrementing BigInt to autoincrementing Int

Bug description
I tried to create a prisma model with connects to a Postgres database. For the ids for the table, I want to use the BigInt datatype, as it says it's supported. However,  when I try to push the new model I get Error querying the database: db error: ERROR: type "bigserial" does not exist and I'm unable to push the model
How to reproduce

Create a new prisma instance as per documentation, connecting to a postgres database
Create the model as described below
Push the model using prisma db push
Very the error returned is
Error: Database error Error querying the database: db error: ERROR: type "bigserial" does not exist 0: sql_migration_connector::sql_database_step_applier::apply_migration           at migration-engine/connectors/sql-migration-connector/src/sql_database_step_applier.rs:11 1: migration_core::api::SchemaPush           at migration-engine/core/src/api.rs:161

Expected behavior
push command should happen as expected as BigInt is supported
Prisma information
enum EntityType {
  club
  individual
}

enum Status {
  active
  dormant
}

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model club_entity {
  id     BigInt @db.BigInt @id @default(autoincrement())
  name   String @unique
  email  String
  type   EntityType @default(club)
  status Status
  points Int    @default(0)
}

model club_transactions {
  id          BigInt @db.BigInt @id @default(autoincrement())
  description String
  pointChange Int
  clubId      Int
}

Environment & setup

Client OS is MacOS
Database: PostsgreSQL
Node: 14.15.4

Prisma Version
prisma               : 2.29.1
@prisma/client       : 2.29.1
Current platform     : darwin
Query Engine         : query-engine 1be4cd60b89afa04b192acb1ef47758a39810f3a (at node_modules/@prisma/engines/query-engine-darwin)
Migration Engine     : migration-engine-cli 1be4cd60b89afa04b192acb1ef47758a39810f3a (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core 1be4cd60b89afa04b192acb1ef47758a39810f3a (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary        : prisma-fmt 1be4cd60b89afa04b192acb1ef47758a39810f3a (at node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash : 1be4cd60b89afa04b192acb1ef47758a39810f3a
Studio               : 0.419.0

Can you please use prisma migrate dev instead of prisma db push and post the content of the SQL migration file that was generated?
What version of PostgreSQL are you using? Where are you hosting it?

I'm using the Postres hosted here: https://supabase.io/docs/guides/database
As for the sql:
/*
Warnings:

The primary key for the club_entity table will be changed. If it partially fails, the table could be left without primary key constraint.
The type column on the club_entity table would be dropped and recreated. This will lead to data loss if there is data in the column.
The primary key for the club_transactions table will be changed. If it partially fails, the table could be left without primary key constraint.
Added the required column status to the club_entity table without a default value. This is not possible if the table is not empty.

*/
-- CreateEnum
CREATE TYPE "EntityType" AS ENUM ('club', 'individual');
-- CreateEnum
CREATE TYPE "Status" AS ENUM ('active', 'dormant');
-- AlterTable
ALTER TABLE "club_entity" DROP CONSTRAINT "club_entity_pkey",
ADD COLUMN     "status" "Status" NOT NULL,
ALTER COLUMN "id" SET DATA TYPE BIGSERIAL,
DROP COLUMN "type",
ADD COLUMN     "type" "EntityType" NOT NULL DEFAULT E'club',
ADD PRIMARY KEY ("id");
-- AlterTable
ALTER TABLE "club_transactions" DROP CONSTRAINT "club_transactions_pkey",
ALTER COLUMN "id" SET DATA TYPE BIGSERIAL,
ADD PRIMARY KEY ("id");

Interesting.

I am unable to reproduce this. I tried it on both supabase and locally:

Maybe try cloning your supabase project if the service allows that and try on new database connection?

Thank you-to clarify:
On an empty table, I'm able to create a schema using BigInt:
-- CreateEnum
CREATE TYPE "EntityType" AS ENUM ('club', 'individual');
-- CreateEnum
CREATE TYPE "Status" AS ENUM ('active', 'dormant');
-- CreateTable
CREATE TABLE "club_entity" (
"id" BIGSERIAL NOT NULL,
"name" TEXT NOT NULL,
"email" TEXT NOT NULL,
"type" "EntityType" NOT NULL DEFAULT E'club',
"status" "Status" NOT NULL,
"points" INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY ("id")

);
-- CreateTable
CREATE TABLE "club_transactions" (
"id" BIGSERIAL NOT NULL,
"description" TEXT NOT NULL,
"pointChange" INTEGER NOT NULL,
"clubId" INTEGER NOT NULL,
PRIMARY KEY ("id")

);
-- CreateIndex
CREATE UNIQUE INDEX "club_entity.name_unique" ON "club_entity"("name");
However; should I try to change the primary back to Int using this migration:
/*
Warnings:

The primary key for the club_entity table will be changed. If it partially fails, the table could be left without primary key constraint.
You are about to alter the column id on the club_entity table. The data in that column could be lost. The data in that column will be cast from BigInt to Integer.
The primary key for the club_transactions table will be changed. If it partially fails, the table could be left without primary key constraint.
You are about to alter the column id on the club_transactions table. The data in that column could be lost. The data in that column will be cast from BigInt to Integer.

*/
-- AlterTable
ALTER TABLE "club_entity" DROP CONSTRAINT "club_entity_pkey",
ALTER COLUMN "id" SET DATA TYPE SERIAL,
ADD PRIMARY KEY ("id");
-- AlterTable
ALTER TABLE "club_transactions" DROP CONSTRAINT "club_transactions_pkey",
ALTER COLUMN "id" SET DATA TYPE SERIAL,
ADD PRIMARY KEY ("id");
I get the error previously raised. Is this expected?

 It does not include BIGSERIAL. so you getting an error message containing that would be super unexpected and weird.

Ok, the error there is that Serial does not exist. That is different but also interesting. For some reason the database does not seem to know these aliases. Maybe @pantharshit00 can reproduce that.

Thanks for the info @dompatmore
I can reproduce this even locally now when you change BigInt to Int on autoincrementing fields.
SERIAL is not a datatype but a shortcut and it does not work on alter statements. So what we generate is ALTER COLUMN "id" SET DATA TYPE SERIAL, doesn't work.
Workaround here would be to remove autoincrement() as well when changing from BigInt to Int and adding it again on separate migration.

This has been actually reported before so closing as duplicate of https://github.com/prisma/prisma/issues/5206

have the same problem. Dropped the table in the database and it migrated successfully


Wrong informations in the anime franchies overview

At the bottom of the anime details page you have the franchise overview.
The informations (airing date, episode count, etc.) of these anime are always the information of the anime which page you are on.

ahahaha nice catch wow how did I not notice that before


Add support for karpenter and spot instances requests

Is your request related to a new offering from AWS?
No
Is your request related to a problem? Please describe.
Karpenter default policy doesn't work with spot instance requests because of missing permissions.
Describe the solution you'd like.

Add the following:
  statement {
    actions = ["ec2:RunInstances"]
    resources = [
      "arn:${local.partition}:ec2:*::image/*",
      "arn:${local.partition}:ec2:*:${local.account_id}:instance/*",
      "arn:${local.partition}:ec2:*:${local.account_id}:volume/*",
      "arn:${local.partition}:ec2:*:${local.account_id}:network-interface/*",
      "arn:${local.partition}:ec2:*:${coalesce(var.karpenter_subnet_account_id, local.account_id)}:subnet/*",
      "arn:${local.partition}:ec2:*:${local.account_id}:spot-instances-request/*", # for spot instances request
    ]
  }

Describe alternatives you've considered.

Additional context

This issue has been resolved in version 5.4.0 :tada:


Add text on accessing ES and Kibana via kubectl proxy

@jlowdermilk

lgtm

Adding ok-to-merge since this is doc-only change.


inconsistency between two versions of linux (Debian and Archlinux) opencv with Aravis support

Hi i have a custom opencv compilation with Aravis support that works fine in Archlinux but fails in Debian 12.
In the Case of Debian the custom opencv works fine in Cpp but fails in the Rust program.
The main difference is that in the case of archlinux i install the aravis lib from the AUR package and in the case of Debian i compile from source. Any ideas of what is wrong or where search???.
PD: i see a difference in where place the  pkgconfig folder in the two instalations could be the source of inconsistency???
Regards

Without the full build log as produced by cargo build -vv it's difficult to say what goes wrong. Potentially take a look at https://github.com/twistedfall/opencv-rust/blob/master/INSTALL.md for some guides to make it work with manual build, probably setting up the correct PKG_CONFIG_PATH might help

yeah the solution was setting PKG_CONFIG_PATH to a the pkgconfig folder generated by aravis.
Regards


Including the imagesloaded.js library causes  images to load two versions of the image, increasing bandwidth by ~4x

I noticed that the images on my site were being loaded twice (once as a jpg, once as a webp), which was killing my Lighthouse score. I looked through the <picture> HTML code and couldn't find anything wrong. I finally chased the double-load down to this function in imagesloaded.

The imagesloaded code creates a second proxy image in the background, loading it as a clone via the original <img src>. However, this forces the browser to load a non-optimized version of the image despite the presence of a <source srcset>, since the proxy image is not managed. In my case, this means it forces the browser to load the largest possible variant of the JPG hero image, increasing the bandwidth used by every page on the site by ~4x.
The bug happens in the following conditions:

Website merely includes imagesloaded.js
Website uses <picture><source srcset><img> convention.

I switched my WordPress theme back to TwentyTwenty to verify that it also happened there, too.

I'm curious @zaneclaes did you ever find a solution here?  I'm noticing the same behavior.

imagesLoaded v5 has been released with proper support for srcset and <picture> This issue should now be resolved.


Q: When using transceiver.Submit(), does the handler get the SubmitSMResp as well?

I am using the transceiver to submit a SM, and receiving the response back successfully, but I also have a handler that wants to process the SubmitSMResp. Right now, it seems that it doesn't get passed to the handler, am I correct?

Submit is a full blocking transaction that handles the response automatically. It won't trigger SubmitSMResp.


GlobalMinGasPrice doesn't support versions outside of v2

GlobalMinGasPrice currently throws an error if if the version is not v2.
https://github.com/celestiaorg/celestia-app/blob/ef3c0d574973fb8aabedb1c1d9626f6a24ccdb31/pkg/appconsts/versioned_consts.go#L27-L36
When running on a chain that is currently on v1, this will cause non-determinism since this value is called in the antehandlers in process and prepare proposal. It will also stop all txs from being able to be created or pass CheckTx.

I was just in the process of fixing it when I realised that the only place this function is getting called is in CheckTxFeeWithGlobalMinGasPrices inside the if clause where the appVersion has to be greater than 1. https://github.com/celestiaorg/celestia-app/blob/ef3c0d574973fb8aabedb1c1d9626f6a24ccdb31/app/ante/fee_checker.go#L32
I could be missing something but I don't understand how this would be causing issues on v1

yeah good point, it shouldn't cause issues on v1, however it will cause issues on versions greater than v2, such as in the testground tests


Example binarized vector representation is syntactically incorrect

Now let's store the genre's for these movies in a list that we will later transform into a binarized vector.
Binarized vector representation is a very common and important way data is stored/represented in ML. Essentially, it's a way to reduce a categorical variable with n possible values to n binary indicator variables. What does that mean? For example, let [(1,3),(4)] be the list saying that sample A has two labels 1 and 3, and sample B has one label 4. For every sample, for every possible label, the representation is simply 1 if it has that label, and 0 if it doesn't have that label. So the binarized version of the above list will be -

> [(1,0,1,0]),
> (0,0,0,1])]

This section has output that contains a brackets mismatch syntax error. Not a huge problem, but probably a bit confusing for a beginner. Otherwise, great tutorial!

Thanks for pointing it out. We will get around to it in the next round of bug fixing hopefully!


Error using PETSc

Hi,
I am trying to use some PETSc functions and I have got an error which seems that something in the installation is wrong. The error:
PETSc/Mat.pyx in petsc4py.PETSc.Mat.multTranspose (src/petsc4py.PETSc.c:126588)()
Error: error code 60
[0] MatMultTranspose() line 2274 in /home/juanlu/miniconda/conda-bld/work/petsc-3.6.2/src/mat/interface/matrix.c
Note that the path which refers to PETSc  file is not in my computer.
Thanks,
Ernesto

This is a duplicate of https://github.com/Juanlu001/fenics-recipes/issues/28. I don't think the path in the traceback is affecting the error, please report it with the corresponding program to FEniCS developers and come back if it's my fault.

Sorry, you are right. It was my fault. The PETSc function is working fine.
Thank you


chore(LocaleModal): Raised test coverage

Related Ticket(s)
2077
Description
I've created tests for LocaleModal so the component's test coverage fulfils our 80% target.

LGTM 👍 - Thanks @RaphaelAmadeu! Given this PR seems to involve change in code-under-test, it'll be good if we can have a deploy preview and image diff so make sure nothing is broken functionally.

Good call, and as discussed in standup we will have QA team do a deeper regression of the component as there is dynamic functionality that percy can't capture. We should try to get localemodal e2e tests in place for this, perhaps set up a priority component list for them to add tests to.


README missing performance data metrics table

While reviewing the README I noticed that performance data metrics are not specified.

That said, I just took a closer look and I see now that only the time metric is currently emitted.
Even so, it is probably worth adding the section/table just to get it into place for when other metrics are added later.


[SYCL] Support SYCL layer for LLaMA2 model

Type of Change

Convert BesTLA int4 weight to SYCL weight
Load SYCL weight to ne_tensor
Activation host->device conversion
Device workspace design
Convert 'output.weight' to SYCL tensor, run LLaMa2  with ARC750

SYCL UT passed and llama2 sycl model_eval with i7-1185G7's Iris iGPU, 12900K(as CPU device) and A770

QKV+FFN on GPU, others on CPU, 9600K+A770:
from 150ms/token to 110ms/token. memcpy between the device and host took the most time.

Disable CPU MHA, run all other layers on GPU A750-8GB, 15 ms/token

Run LLaMa2 all layers on A750: 19.5ms/token
MHA+Rope+RMS=5ms, need further optimization.

n_context = 1024, in=512, out=512, only 5.8GB GPU memory.

what is your name? what is your favorite color?
what is your name? what is your favorite
model_print_timings:        load time =  2672.97 ms
model_print_timings:      sample time =     2.92 ms /    16 runs   (    0.18 ms per token)
model_print_timings: prompt eval time =  2672.43 ms /     6 tokens (  445.40 ms per token)
model_print_timings:        eval time =   258.97 ms /    15 runs   (   17.26 ms per token)
model_print_timings:       total time =  2936.64 ms
========== eval time log of each prediction ==========
prediction   0, time: 2672.43ms
prediction   1, time: 17.88ms
prediction   2, time: 17.26ms
prediction   3, time: 17.33ms
prediction   4, time: 17.85ms

MTL, 54ms/ token, the total latency of int4 gemm is 47ms, optimize rope and layer norm, implement f16 MHA (flash attention) should help reduce this latency.

llama2-7b on A770+9900K:
 Once upon a time, there existed a little girl, who liked to have adventures. She wanted to go to places and meet new people, and have exciting experiences. However, no matter how hard she tried, she always got bored and restless wherever she went.
One day, while wandering
model_print_timings:        load time =   231.00 ms
model_print_timings:      sample time =     6.60 ms /    32 runs   (    0.21 ms per token)
model_print_timings: prompt eval time =   230.26 ms /    32 tokens (    7.20 ms per token)
model_print_timings:        eval time =   523.42 ms /    31 runs   (   16.88 ms per token)
model_print_timings:       total time =   765.82 ms
========== eval time log of each prediction ==========
prediction   0, time: 230.26ms
prediction   1, time: 17.19ms
prediction   2, time: 16.78ms
prediction   3, time: 16.82ms
prediction   4, time: 16.65ms


analytics seems to cause problems with unit testing

The project is here...
https://github.com/lebsral/Ocean-Acidification-Kiosk
Whenever I add the
bower install videogular-angulartics
the tests fail.  (see below)
suggestions?
Last login: Wed Apr  1 13:09:29 on ttys003
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$ gulp test
[13:22:24] Using gulpfile ~/mycode/Ocean-Acidification-Kiosk/gulpfile.js
[13:22:24] Starting 'scripts'...
/Users/lars/mycode/Ocean-Acidification-Kiosk/src/components/voices/voices.controller.js
line 26  col 74  Expected '{' and instead saw 'controller'.
⚠  1 warning
[13:22:25] all files 26.05 kB
[13:22:25] Finished 'scripts' after 575 ms
[13:22:25] Starting 'test'...
INFO [karma]: Karma v0.12.32 server started at http://localhost:9876/
INFO [launcher]: Starting browser PhantomJS
WARN [watcher]: Pattern "/Users/lars/mycode/Ocean-Acidification-Kiosk/src/**/.mock.js" does not match any file.
INFO [PhantomJS 1.9.8 (Mac OS X)]: Connected on socket RpEz47UirnAKORBFAAAA with id 45015482
PhantomJS 1.9.8 (Mac OS X): Executed 2 of 2 SUCCESS (0.003 secs / 0.045 secs)
[13:22:26] Finished 'test' after 1.55 s
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$ bower install videogular-angulartics --save
bower cached        git://github.com/2fdevs/bower-videogular-analytics.git#1.1.1
bower validate      1.1.1 against git://github.com/2fdevs/bower-videogular-analytics.git#
bower cached        git://github.com/luisfarzati/angulartics.git#0.17.2
bower validate      0.17.2 against git://github.com/luisfarzati/angulartics.git#0.17.2
bower cached        git://github.com/imakewebthings/waypoints.git#2.0.5
bower validate      2.0.5 against git://github.com/imakewebthings/waypoints.git#~2.0.3
bower cached        git://github.com/linkgod/SHA-1.git#0.1.1
bower validate      0.1.1 against git://github.com/linkgod/SHA-1.git#*
bower cached        git://github.com/jquery/jquery.git#2.1.3
bower validate      2.1.3 against git://github.com/jquery/jquery.git#>=1.8
bower install       videogular-angulartics#1.1.1
bower install       angulartics#0.17.2
bower install       waypoints#2.0.5
bower install       SHA-1#0.1.1
bower install       jquery#2.1.3
videogular-angulartics#1.1.1 bower_components/videogular-angulartics
├── angulartics#0.17.2
└── videogular#1.1.1
angulartics#0.17.2 bower_components/angulartics
├── SHA-1#0.1.1
├── angular#1.3.15
└── waypoints#2.0.5
waypoints#2.0.5 bower_components/waypoints
└── jquery#2.1.3
SHA-1#0.1.1 bower_components/SHA-1
jquery#2.1.3 bower_components/jquery
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$ gulp test
[13:23:14] Using gulpfile ~/mycode/Ocean-Acidification-Kiosk/gulpfile.js
[13:23:14] Starting 'scripts'...
/Users/lars/mycode/Ocean-Acidification-Kiosk/src/components/voices/voices.controller.js
line 26  col 74  Expected '{' and instead saw 'controller'.
⚠  1 warning
[13:23:15] all files 26.05 kB
[13:23:15] Finished 'scripts' after 578 ms
[13:23:15] Starting 'test'...
INFO [karma]: Karma v0.12.32 server started at http://localhost:9876/
INFO [launcher]: Starting browser PhantomJS
WARN [watcher]: Pattern "/Users/lars/mycode/Ocean-Acidification-Kiosk/src/**/*.mock.js" does not match any file.
INFO [PhantomJS 1.9.8 (Mac OS X)]: Connected on socket ILBjY9ywPh8aQ02hAAAA with id 65337394
ERROR [karma]: TypeError: 'undefined' is not a function (evaluating '$(window)')
at http://localhost:9876/base/bower_components/waypoints/waypoints.js?8f213d0f4dfbb256974d6f1a25b546b7e36f26e1:25
[13:23:16] 'test' errored after 1.38 s
[13:23:16] Error: 1
at formatError (/usr/local/lib/node_modules/gulp/bin/gulp.js:169:10)
at Gulp. (/usr/local/lib/node_modules/gulp/bin/gulp.js:195:15)
at Gulp.emit (events.js:107:17)
at Gulp.Orchestrator._emitTaskDone (/Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/gulp/node_modules/orchestrator/index.js:264:8)
at /Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/gulp/node_modules/orchestrator/index.js:275:23
at finish (/Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:21:8)
at cb (/Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:29:3)
at removeAllListeners (/Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/karma/lib/server.js:220:7)
at Server. (/Users/lars/mycode/Ocean-Acidification-Kiosk/node_modules/karma/lib/server.js:231:9)
at Server.g (events.js:199:16)
Larss-MacBook-Pro:Ocean-Acidification-Kiosk lars$

or to make this more simple this also fails.  But with less extra stuff
https://github.com/lebsral/testvideo2

You need to add the module dependency:
"angulartics",
"com.2fdevs.videogular.analytics",
https://github.com/lebsral/Ocean-Acidification-Kiosk/blob/master/src/app/index.js#L13
See this for more info: http://www.videogular.com/examples/analytics-plugin/

Closing this, please, reopen if you need help.


cmd/compile: -d=checkptr should provide more details when reporting about conversions that straddle multiple allocations

What version of Go are you using (go version)?

$ go version
1.14

Does this issue reproduce with the latest release?
probably?
What operating system and processor architecture are you using (go env)?
linux/amd64
What did you do?
dataOffset += 2 * copy((*[1 << 16]uint16)(unsafe.Pointer(&nextData[0]))[:], c.array())
(Context actually mostly irrelevant; what's important is the lack of explicit length/cap on the slice.)
What did you expect to see?
checkptr thinking i totally know what i'm doing, or telling me what's wrong
What did you see instead?
"unsafe pointer conversion"
I mentioned this in passing because I thought it was complaining that we couldn't prove that the pointer was aligned, but bcmills pointed me at the CL for having checkptr not complain about unaligned access when it's to non-pointer-typed data. Another user (hi kale!) pointed out a more subtle issue: https://go-review.googlesource.com/c/sys/+/225418
It turns out checkptr also complains about slices which lack a length/capacity. That's completely reasonable, but I didn't know it was one of the rules checkptr would enforce, and the diagnostic didn't say what it was doing.
Suggestion: checkptr diagnostics should be distinct, and say things like "potential misalignment", or "slice without length or capacity" or something else that would make it more immediately obvious what it's complaining about.
(It's right, though.)

CC @mdempsky

Go 1.15 will improve the error message somewhat: 1f231d7.
Does that address your concerns, or are you suggesting that checkptr should specifically recognize (*[big]T)(p)[:] and hint the user should add len/cap to the slice?

I think what I'm mostly advocating for is "at the point of deciding to report a thing, flag in some way the basis for the decision". That message would probably have communicated the issue to me (although I might have mistaken it for alignment anyway). So, the specific thing of saying that len/cap is needed might be overkill, but something like "converted pointer points to 8192 bytes, base pointer is 2048 bytes before end of allocation" might be helpful too. But mostly, things like distinguishing between "this looks to be looking outside the object the pointer was derived from" and "this is unaligned" and whatnot seem like they'd be useful. In this case, i  happen to know that c.array() is a []uint16 of length n, and if i use [:n:n] in the slice, it works fine and checkptr doesn't fuss. but since i know that it's a length n slice of uint16, it didn't occur to me to worry about the theoretical bounds of the slice, until someone pointed it out.

I think what I'm mostly advocating for is "at the point of deciding to report a thing, flag in some way the basis for the decision".

Okay. I believe that's what 1f231d7 does. After that CL, checkptr reports unique error messages for each case it checks.

something like "converted pointer points to 8192 bytes, base pointer is 2048 bytes before end of allocation" might be helpful too.

We don't always know the "N bytes before end of allocation" part (e.g., the base pointer might be to a statically-allocated, package-level Go variable, or it might be into C memory), but we could certainly include additional details like that when we do know them.
I think we can simplify the wording though. E.g., maybe something like:
checkptr: converted pointer straddles multiple allocations (points to 2048-byte allocation, but [65536]uint16 is 131072 bytes)

Open to suggestions though.

I think this is a case where more information is almost always better, and any additional insight into the violation will be valuable to the user.
Say I have b [65536]byte, and I obtain a poiter to &b[65530], and cast it to *[16]byte. That's the case where the wording seems hard to me; "points to 16-byte allocation, but [65536]byte is 65536 bytes" would be slightly confusing.

Say I have b [65536]byte, and I obtain a poiter to &b[65530], and cast it to *[16]byte.

In this case, my proposed wording would be: points to 6-byte allocation, but [16]byte is 16 bytes.

Can we do points to byte offset 65530 in a 65536-byte allocation, but [16]byte is 16 bytes.
(Including the offset only if it is nonzero.)

Yeah that seems doable.


Invalid result when making css from Bootstrap4

A few day ago I switched from Live Sass Compiler from Ritwick Dey to your Live Sas Compiler. I have notices changes in result css file, that is produced from Bootstap-4.4.0 scss source.
for example in .form-control selector:
height: calc(1.5em + 0.75rem + 2px);
to:
height: calc(1.5em + 0.75rem2px);
Second plus sign is missing.

Going to investigate this.
Are you using the whole bootstrap or including one of their modules?

I had a compilation issue once but then I wasn't able to reproduce it when I came to testing. Don't know if there was some cache built by SASS that sorted it when I saved it the second time
Can you try saving the file again and see if this resolves your issue?
I'll be updating the SASS module soon, as there is something in their changelog that may be a fix 🤷♂. Sorry hectic time at the minute (with baby, uni, work 😩)

Can you try live-sass-5.5.2-alpha.3.zip

Closing as info required. v6 is also available with a more recent version of SASS included

Sorry I replied to noreply email address. Error is still remaing - tested on your aplha version.

Using bootstrap 4.6.2, the issue is resolved.
Looks like it was an issue with bootstrap that was fixed between 4.4.0 and 4.6.2

Hello Glenn,
I upgraded bootstrap to version 4.6.2 and this error disappeared.
Thank you Petr


Use pm2 to node in production

I like it much the distiction of production of develop, and I wanna sugestion change script of npm start use of pm2 to use in production env

Nice idea, but I haven't used pm2 before. I need to learn how to use this process manager with Docker.


add eslint to client and fix react test

Dev Summary

re-configuring eslint to be helpful

was previously giving us errors in all of the build files which made the output too hard to read

fixes a bunch of missing/extra semis
adding a line to the pull request template to enforce running lint and test commands
fixes our react app.test to check for our logo rather than text that isn't there (was breaking before)

Test Plan

cd client && npm run checks
see that there are no eslint errors or jest errors
remove a ; from any component (i.e. client/src/AboutUs.tsx) and run npm run checks again
see that we get an eslint error and therefore eslint is working

Resources

https://www.youtube.com/watch?v=bfyI9yl3qfE
https://eslint.org/docs/rules/

Might be obvious, but npm run checks failed until I ran npm start - not sure if the client needs to be running or if I just need to have run it at least once, but might be good to include that in the testing details

hmmm, no idea why that's the case. i don't think that should happen
in fact, i didn't run start until after all of my changes as an after thought

Might be obvious, but npm run checks failed until I ran npm start - not sure if the client needs to be running or if I just need to have run it at least once, but might be good to include that in the testing details

hmmm, no idea why that's the case. i don't think that should happen in fact, i didn't run start until after all of my changes as an after thought

Strange - it's working now without the client running, but this is after having run the client at least once. It said something about missing a checks folder I think, which doesn't quite make sense to me


Vulnerability roundup 71: libsndfile-1.0.28: 1 advisory

search, files

[ ] CVE-2018-13419 (nixos-19.03)

Scanned versions: nixos-19.03: e0c7712eac6. May contain false positives.

See also #52974, #57156, and #59164

Still disputed by the upstream. No patch nor reproducer.

19.03 is EOL


Add resume policy instructions for Katib experiments

Fixes: https://github.com/kubeflow/katib/issues/1292.
Blocked by: https://github.com/kubeflow/website/pull/2312.
I've added doc about restarting Katib experiment, please take a look.
/assign @johnugeorge @gaocegege
/cc @RFMVasconcelos @8bitmp3

Please do a rebase.

/hold

@gaocegege It's done.
/hold cancel

LGTM :+1:
/lgtm


Registering multiple DecodeCustomTypeFuncs against a type

Is it possible to register multiple custom types on the same type? I want to accept multiple ways of passing a time.Time type in separate use cases.
I've tried registering them procedurally:
decoder.RegisterCustomTypeFunc(func(vals []string) (interface{}, error) {
		return time.Parse("02/01/06 3:04 PM MST -0700", vals[0])
	}, time.Time{})
	decoder.RegisterCustomTypeFunc(func(vals []string) (interface{}, error) {
		return time.Parse("3:04 PM", vals[0])
	}, time.Time{})

And that didn't work, it tries to use the first one. If it's not currently possible, can this be a feature request? When multiple DecodeCustomTypeFuncs are registered to a type, you could iterate through them on parse error?

Hey @danhardman
I don't think it would be a good idea to register multiple, but rather, why not check both time formats in your custom function?
eg. if first time.Parse format fails, try the second

@joeybloggs Hadn't actually thought of that.
You know that moment when you get in the mindset of doing it the way the library and you forget about the language way? Maybe it's just me! haha should work fine, thank you!

No problem @danhardman we all have those can't see the forest because of the trees moments ;)


No module named “pkg_resources”

Couldn’t install with pip, had to use pipx. First run I got this ModuleNotFoundError. Uninstalling and reinstalling did not help.
Edit: MacOS 14.4.1 on Apple M2 Pro chip (MacBook Pro 2023)

This project is pretty dead, as far as I know. Without paid access to the Twitter API I'm not really sure how far you'll get. Good luck nonetheless.

You'll need the setuptools packages. Try pipx inject rainbowstream setuptools.
But as @johnpiers said, while you may be able to launch the tool, you might still face errors due to Twitter API limitations, preventing its full functionality.