Blank Pdf

When attempting to capture a PDF using AWS lambda it is always coming back blank.
I have tried going to a url (page.goto) and also simply setting content (page.setContent).
I have also tried with and without: await page.emulateMedia('screen');
I can return the content of the page (page.content) but not create a PDF.
I am using nodejs8.10 runtime
Below is the code I am using and I have also attached the PDF returned, right now I am returning the PDF, later I will save this to S3 instead.
Any help would be greatly appreciated, thanks a lot
// index.js
const serverless = require('serverless-http');
const express = require('express');
const chromium = require('chrome-aws-lambda');
const puppeteer = require('puppeteer-core');
const app = express();

const main = async() => {
    const browser = await puppeteer.launch({
        args: chromium.args,
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath,
        headless: true
    });

    const page = await browser.newPage();
    await page.goto('https://example.com', {waitUntil: 'networkidle2'});
    await page.emulateMedia('screen');
    const pdf = await page.pdf({format: 'A4', footerTemplate: '
footer
'});
    // close page?
    return pdf;
};

app.get('/', function (req, res) {
    res.send('Hello World!');
});

app.post('/pdf', async function(req, res) {
    const pdf = await main();
    res.contentType("application/pdf");
    res.send(pdf);
});

module.exports.handler = serverless(app);

response.pdf

+1. PDFs seem to be always blank.

@tonyschollum Which version of chrome-aws-lambda / puppeteer are you using?

@tonyschollum Which version of chrome-aws-lambda / puppeteer are you using?

Hi @alixaxel
I was using:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Cheers

can we have fix and work around of this issue?

@tonyschollum I seem to be able to generate a non-blank PDF (at least with latest version): response.pdf.
Could you please try again with the latest version and tell me if that solves it?

Hi @alixaxel,
I have updated but still seem to be getting back blank PDF's.
Now running:
<EMAIL_ADDRESS><EMAIL_ADDRESS>I have attached the simple example index.js I am testing with if there is anything in there you can see I am doing wrong? I am also deploying this using serverless from a windows machine if that impacts at all.
index.zip
Cheers

+1 still encountering this. Has anyone found a fix?

I had the same issue and felt like I tried everything. As @mitchgillin said there is a good chance the issue is the API Gateway configuration and not this library.
I am using the following code to create the lambda server :
'use strict'
const awsServerlessExpress = require('aws-serverless-express')
const app = require('./index')
const binaryMimeTypes = [
	'application/octet-stream',
	'font/eot',
	'font/opentype',
	'font/otf',
	'image/jpeg',
	'image/png',
	'image/svg+xml',
	'application/pdf' // => This was missing
]
const server = awsServerlessExpress.createServer(app, null, binaryMimeTypes);
exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)

Also you can configure API gateway by adding */* to the binary types

@tonyschollum I am pretty sure that's your issue, the rest of my code was very much like yours.

In my case, I am converting the outputted pdf file to base64 and sending it to the client as text (for various reasons). So I don't see how it is an API gateway issue. But I'll have to retest again. https://github.com/Kikobeats/aws-lambda-chrome was working fine for me in existing setup.

@hisham The binary from Kikobeats/aws-lambda-chrome was exactly the same that is available on this project (I used to provide the compiled binary on the puppeteer Slack channel). @tonyschollum Can you cook up a minimal example that consistently reproduces the problem you're facing? You have a bunch of dependencies that are unrelated to this project, and setting that all up is very time-consuming.

@alixaxel - sorry small clarification, the latest Kiko binary did not work for me. I had to revert to an older version of the binary. This is what I'm running currently:
  "dependencies": {
    "@browserless/aws-lambda-chrome": "~1.10.0",
    "browserless": "~4.1.3",
    "puppeteer": "^1.10.0"
  }

@hisham<EMAIL_ADDRESS>binary also came from this project.
I'm not sure why you have browserless there, but I can only provide support for this project.
Also, I managed to correctly get a PDF and shared it in my previous comment. Please provide a minimal example where I can reproduce this problem, otherwise I'm afraid I cannot offer any kind of support for this.

Hi @alixaxel
Attached is a zip of the simple example I was trying but getting blank PDFs.
chrome-aws-lambda.zip

@tonyschollum The reason you are getting a blank PDF is due to API Gateway misconfiguration.
I suggest you look into Swagger to describe your resources and Content-Types, or upload your file to S3.
Following @batman9494 instructions on https://github.com/alixaxel/chrome-aws-lambda/issues/14#issuecomment-476384301 might do the trick as well.
Here's a the output I got: google.pdf.

Closing this as a invalid question (since it pertains to AWS Gateway).

I hardly disagree. It should work with serverless offline start that has no relation with the API Gateway and still doesn't work.

@charmpitz I'd be happy to reopen this if you share a full repro.

Same issue on my side, except I use page.setContent() instead of page.goTo().
Not related to API Gateway, I upload the PDF to S3.
const chromium = require('chrome-aws-lambda');

module.exports = class PDFGenerator {

  async generate(content, path) {
    const executablePath = await chromium.executablePath;
    const browser = await chromium.puppeteer.launch({
      args: chromium.args,
      executablePath,
      defaultViewport: chromium.defaultViewport,
      headless: chromium.headless,
    });
    const page = await browser.newPage();
    await page.setContent(content);
    await page.pdf({ path, format: 'A4' });
    await browser.close();
  }

}

For anyone struggling, here's a working example (don't forget to enable binary mapping on API Gateway as in https://github.com/alixaxel/chrome-aws-lambda/issues/14#issuecomment-476384301)
const express = require('express')
const app = express()
const chromium = require('chrome-aws-lambda');

app.get('/', async (_, res) => {
  let browser = null;

  try {
    browser = await chromium.puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
      ignoreHTTPSErrors: true,
    });
    const page = await browser.newPage()
    await page.setContent(`

          <title>Test</title>

Testing.

    `);

    const buffer = await page.pdf({
      margin: { top: '30px' },
      scale: 0.98,
      format: 'A4',
      printBackground: true,
      headless: true
    })

    // We can directly serve this buffer to the browser.
    // Set the content type so the browser knows how to handle the response.
    res.type('application/pdf')
    res.set('Content-Length', Buffer.byteLength(buffer, 'utf-8'));

    res.end(buffer)

    await browser.close()
  } catch (error) {
    console.log(error)
    return
  } finally {
    if (browser !== null) {
      await browser.close();
    }
  }
})

const awsServerlessExpress = require('aws-serverless-express')
const binaryMimeTypes = [
	'application/pdf'
]
const server = awsServerlessExpress.createServer(app, null, binaryMimeTypes);
module.exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)

For anyone struggling, here's a working example (don't forget to enable binary mapping on API Gateway as in #14 (comment))
const express = require('express')
const app = express()
const chromium = require('chrome-aws-lambda');

app.get('/', async (_, res) => {
  let browser = null;

  try {
    browser = await chromium.puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
      ignoreHTTPSErrors: true,
    });
    const page = await browser.newPage()
    await page.setContent(`

          <title>Test</title>

Testing.

    `);

    const buffer = await page.pdf({
      margin: { top: '30px' },
      scale: 0.98,
      format: 'A4',
      printBackground: true,
      headless: true
    })

    // We can directly serve this buffer to the browser.
    // Set the content type so the browser knows how to handle the response.
    res.type('application/pdf')
    res.set('Content-Length', Buffer.byteLength(buffer, 'utf-8'));

    res.end(buffer)

    await browser.close()
  } catch (error) {
    console.log(error)
    return
  } finally {
    if (browser !== null) {
      await browser.close();
    }
  }
})

const awsServerlessExpress = require('aws-serverless-express')
const binaryMimeTypes = [
	'application/pdf'
]
const server = awsServerlessExpress.createServer(app, null, binaryMimeTypes);
module.exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)

I think you have a typo:
res.end = res.send(buffer) ?


Add adapt_only_on_bare_turn_on which instantly triggers manual_control when turning on with brightness or color

This PR adds a configuration option adapt_only_on_bare_turn_on, which when enabled doesn't adapt lights when initially turning them when there is color or brightness in the service_data.
To not change the behavior for anyone, the option is false by default.
Closes #12, #89, #271, #314, #416, #435, #528, #581

@basnijholt Is it possible to add a feature that let's you select whether brightness or temperature triggers this? For example, I may want to call the light to 100% brightness but still want the temperature to adapt or vice versa.
This feature is exactly what my wife has been asking for :) Thank you!


/etc/fstab syntax for drvfs and CIFS paths

Your Windows build number:  (Type ver at a Windows Command Prompt)

10.0.17763.615

What you're doing and what's happening:  (Copy&paste the full set of specific command-line steps necessary to reproduce the behavior, and their output. Include screen shots if that helps demonstrate the problem.)

sudo mount -t drvfs '\\HOST\vol' /mnt/vol
I'm trying to replicate the syntax from the command-line in /etc/fstab and not all the googling in the world has seemed to find a solution.

What's wrong / what should be happening instead:

I've tried pretty much every syntax I can think of, and the result is something along the lines of:
The last thing I tried was:
//HOST/vol /mnt/vol drvfs default 0 0
mount: /mnt/vol: wrong fs type, bad option, bad superblock on //HOST/vol, missing codepage or helper program, or other error.
It's not clear to me how to placate mount to get it to do what I want.

It does appear that if you remove default that there is a canned set of mount options already there for for drvfs and it will work.
It would be good to know what mount options are available. Are they documented anywhere?

Is default a mistake for defaults?

Is default a mistake for defaults?

Yes, it was a typo!


EC2 documentation

@jek-a is going to take care of some EC2 documentation for chef-provisioning-fog. Thanks!

I will do it on this week. I'm little busy now. I don't use AWS as a cloud provider, I use our corporate cloud as a cloud provider, becuase it supports EC2 API. Although, I don't use knife.rb. I use AWS approach ~/.aws/config and ~/.aws/credentials to store: aws_access_key_id, aws_secret_access_key. With that approach you can work with multiple accounts. I tried to use knife.rb, but I could not pass custom ec2_endpoint and iam_endpoint variables for our cloud. I hope that this will work.

@jek-a awesome thanks. Any documentation that i can verify then merge is what i'm looking for.
If it's only done via .aws/config and .aws/credentials so be it. Anything right now is better then nothing.
And we do have a "standard" way of the documentation now, something like: https://github.com/chef/chef-provisioning-fog/pull/163

@jek-a can you add https://github.com/chef/chef-provisioning-fog/pull/120 as part of your documentation? Thanks!

@jek-a any progress here?

@jek-a ping :smile_cat:

https://github.com/chef/chef-provisioning-fog/pull/175 Resolves what I was looking for.


BeaconRegionIBeacon at ranging: Unable to parse Identifier.

Hi
I am trying to pass a BeaconRegionIBeacon through the ranging() method but when i pass the major and minor as Int, ranging breaks with the message (Unable to parse Identifier.) and shows me that the minor and major, previously set as int, have been converted to double.
I am actually trying to fix this issue, but if anyone can help me I would appreciate it.

Change it to string. It works fine in his example


Background image is override by column background

Microsoft Reviewers: Open in CodeFlow

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: venniladhanasekaran sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Created new PR -#3678


Icons incorrectly use aria-hidden instead of aria-label, and some lack either

When browsing the internet with a screenreader one depends on aria-label to accurately describe functionality. By contrast, aria-hidden removes the element from the screenreader's DOM tree, as well as its children. Therefore, it is important that interactive elements are labeled, and especially elements that only use an icon.
The following components use aria-hidden for its accompanying material icon, even though the button is an interactive element with no other label, or lack aria-hidden and aria-label:

Action Lists - 🍔 more-vert icon should have aria-label instead of aria-hidden
Alerts - ❌ close icon should have aria-label instead of aria-hidden
Dialog - ❌ close icon should have aria-label instead of aria-hidden
Nav - icons-only are aria-hidden, should have aria-label, or change with breakpoint.
Sheet - ❌ close icon should have aria-label instead of aria-hidden.
Sidebar - icons lack aria-hidden and aria-label, should likely use aria-hidden.
Toast - ❌ close icon is missing aria-hidden and aria-label, should have aria-label.
Tooltips - ❓ help_outline is missing aria-hidden and aria-label, should have aria-label
Topbar - 🍔 menu and ❌ close icons should have aria-label instead of aria-hidden.

To Reproduce
Steps to reproduce the behavior:

Launch Chrome.
Install ChromeVox.
Recommended: Enable focus on cursor in the settings.
Visit the Design Guide with ChromeVox enabled.
Navigate to the components mentioned and interact with the icon(s).
Observe lack of speech, or strange speech. Example: Screen readers will read _, i.e. help underscore outline.

Expected behavior

Purely decorative icons should be aria-hidden.
Interactive elements that combine icons and text should always use aria-hidden if text is sufficiently descriptive.
Interactive elements that exclusively use icons should always use aria-label to describe the function of the element.
All material icons should use either aria-hidden or aria-label.

Actual behavior
As described above, a number of icons use aria-hidden, or lack both aria-hidden and aria-label when one of them should always be present.

Some further discoveries show that rather than aria-label on the icon we should employ it on the parent element if it is interactive, or use a .sr-only span element next to it for static, non-interactive content.


Project needs Code Climate Setup

This should be fairly straightforward. Already signed up for CodeClimate with the NerdDice gem.
Add badge to README along with badge for overall CI build.

Resolved by #11


fix: fix install xhprof error

I completed the 3 steps below:

[x] I've read the Contribution Guide.
[ ] I've updated the documentation. (refer to this for how to do so).
[x] I enjoyed my time contributing and making developer's life easier :)

@bestlong Please help review

So fix it for travis


title: "creating a stylish login and registration page with tailwind …

Checklist

Please note that in case of below checklist not updated accordingly, the maintainers have the right to immediately close your pull request.

[x ] You created a branch from dev branch.
[x ] You PR is raised on dev branch and not onmaster.
[x ] Blog follows Blog Guidelines.
[ x] You have read the Contribution Guidelines before creating this PR.
[ x] Checked for any duplicate blog entry.
[x ] Checked for all grammatical errors in the blog.
[x ] You have not published the blog content anywhere else on the internet on platforms like medium, dev.to etc.
[ x] The content is not copy-pasted from anywhere else on the Internet.

Description of change

This is being rejected as it didn't meet our guidelines or quality standards.
You can refer to the guidelines here:

https://github.com/LoginRadius/engineering-portal/blob/master/GUIDELINES.md
https://github.com/LoginRadius/engineering-portal/blob/master/CONTRIBUTING.md


Celsius Request

Hey, this is the best watch face I’ve seen yet. I was wondering if it’s possible to change the temperature to Celsius?
I can’t seem to figure out how to change it, and am wondering if it’s hard coded for Fahrenheit?
thanks! 🇨🇦

Thanks for the positive feedback! Unfortunately Night Shift does not currently support Celsius. It was originally intended to support Celsius with the standard Fitbit user settings, however I encountered trouble getting it to work properly. This was largely due difficulty with testing and debugging because of lack of proper support of the Fitbit brand by Google. I will take another look at the issue when I have time, and see if I can get toggling between Fahrenheit and Celsius to work, and if I can I’ll push an update.

I agree that this is definitely the best watch face. Celsius weather would make this my default watch face. 🇨🇦

Support for Celsius temperature display is now officially available for Night Shift. It will be necessary to update the watch-face to get access to this new feature. To turn on Celsius:  In Fitbit phone app, on "Today" screen, click on profile picture -> Fitbit settings -> Date, Time & units -> UNITS OF MEASUREMENT. An option to switch between Celsius and Fahrenheit is available across from Temperature. However, it may be necessary to toggle the Length value between "Feet, Miles" and "Centimeters, Kilometers" to actually toggle between Fahrenheit and Celsius in the Night Shift watch face. This seems to be an issue with the Fitbit software. If trouble toggling between Celsius and Fahrenheit persists, try uninstalling the watch-face and reinstalling it.

You’re amazing!! Thank you so much!  😃On Dec 13, 2024, at 10:34 AM, Joshua Horvath @.***> wrote:
Support for Celsius temperature display is now officially available for Night Shift. It will be necessary to update the watch-face to get access to this new feature. To turn on Celsius:  In Fitbit phone app, on "Today" screen, click on profile picture -> Fitbit settings -> Date, Time & units -> UNITS OF MEASUREMENT. An option to switch between Celsius and Fahrenheit is available across from Temperature. However, it may be necessary to toggle the Length value between "Feet, Miles" and "Centimeters, Kilometers" to actually toggle between Fahrenheit and Celsius in the Night Shift watch face. This seems to be an issue with the Fitbit software. If trouble toggling between Celsius and Fahrenheit persists, try uninstalling the watch-face and reinstalling it.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

I agree. Thank you so much for the update, this is a brilliant watch face.
On Sat, Dec 14, 2024, 1:35 a.m. Jsonnos24 @.***> wrote:

You’re amazing!! Thank you so much!  😃On Dec 13, 2024, at 10:34 AM,
Joshua Horvath @.***> wrote:
Support for Celsius temperature display is now officially available for
Night Shift. It will be necessary to update the watch-face to get access to
this new feature. To turn on Celsius: In Fitbit phone app, on "Today"
screen, click on profile picture -> Fitbit settings -> Date, Time & units
-> UNITS OF MEASUREMENT. An option to switch between Celsius and Fahrenheit
is available across from Temperature. However, it may be necessary to
toggle the Length value between "Feet, Miles" and "Centimeters, Kilometers"
to actually toggle between Fahrenheit and Celsius in the Night Shift watch
face. This seems to be an issue with the Fitbit software. If trouble
toggling between Celsius and Fahrenheit persists, try uninstalling the
watch-face and reinstalling it.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are
receiving this because you authored the thread.Message ID: @.***>
—
Reply to this email directly, view it on GitHub
https://github.com/jsh9091/night-shift/issues/1#issuecomment-2542905058,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BNKEVTKQ7S5QFEH4YHH2WD32FPGTLAVCNFSM6AAAAABR33DPPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBSHEYDKMBVHA
.
You are receiving this because you commented.Message ID:
@.***>


Link panellist data to panel set-up

Background
Panellist data is now stored and editable on the Digital Platform, however, it is not currently linked to panel set-up.
User Story
As as Admin, I would like to be able to add panellists from the database to the panel(s) I have set up on my exercise.
Feature(s) Description

[ ] Link the panellists stored in the database with Sift under Reports, e.g. this exercise
[ ] Replace the current Add a member button/modal with a dropdown menu (populated from the panellist database and sorted alphabetically desc by name) with an Add a member button next to it
[ ] When clicked, the button adds onscreen the selected panel member
[ ] Next to the panel member name, there are Edit and Delete buttons - Edit allows the user to choose a different name from the dropdown menu
[ ] There is an Add another member button that allows additional panellists to be added

Acceptance Criteria
It's done when Admins can:

select panellists from a dropdown menu on the Panel > Panellists tab
add selected panellists to a panel
edit or delete panellists added to a panel
add/edit/delete multiple panellists from a panel

Definition of Done

[ ] User stories/acceptance criteria met
[ ] Internal reviews passed (feedback actioned)
[ ] User testing passed
[ ] Relevant technical documentation updated
[ ] User guidance updated
[ ] Deployed and merged without errors

User Testing Steps
Panellist data is now stored and editable on the Digital Platform, however, it is not currently linked to panel set-up. This item allows Admins to add panellists from the database to a panel they have set up in an exercise. They can also edit and delete those panellists from the panel.

Hey team! Please add your planning poker estimate with Zenhub @drieJAC @HalcyonJAC @KoWeiJAC @tomlovesgithub @warrensearle

Have kept this ticket open as, whilst the work was done a long time ago, it hasn't been tested or released yet. So this ticket is useful to keep track of testing & deployment.


navbar

Describe the bug
navbar is  not sticky so people find issue to find or go to top for find it ...
To Reproduce
Steps to reproduce the behavior:

Go to "Go to home page"
Click on '' "
Scroll down to 'footer'
See error "there is no navbar"

Expected behavior
there is no navbar when we scroll down so we find problem to search navbar
Screenshots

I would like to work on this issue, please assign this issue to me under GSSoC'24 @ashutoshgithubs .


File libraries-state.json not found inside '.slicemachine' folder

Versions

@prismicio/next:  "^1.0.2"
next: "13.1.1"
node:  v18.12.1

Reproduction
I followed the prismic documentation to install and configure Prismic in a Next.js project. Both nextjs as well as slicemachine ran without issues up to this point. But when I create a slice in slicemachine and try to simulate the newly created slice, simulator tab (in browser) shows;

And the tab running nextjs shows the following error;

If I add the file libraries-state.json, with an empty object, inside the .slicemachine folder manually there will be no such errors. But again, the file remains empty even while creating and updating slices in slicemachine, which I believe is not the expected behavior.

Hey there, really sorry for the lack of support during the last two weeks!
Actually, importing the libraries-state.json file and feeding it to the Slice Simulator component is no longer needed. The fact that you encountered it was due to a miscommunication on our end on updating our documentation.
The documentation is now updated and shouldn't mislead users like you again into this error: https://prismic.io/docs/setup-nextjs#create-slice-simulator-page
Thanks 🙌


Add code coverage by using goveralls tool

Changes Unknown when pulling 32f84ede0b65275aa20d5cc7e8340b7b8f39c4bc on leonwanghui:code_coverage into ** on opensds:development**.


Identifies 3 examples for documentation along poor-good-best gradient

This should fix #1, and close #20.

Ah yes! I like this. We only need one "evaluate documentation" exercise, and I'm glad the the "answer" has be added to the instructor notes page. Unless @elliewix has an objection, I think this is good to merge.

I'm merging this now - assuming the silence means consent.


Does iwasm support SGX2?

Hello. I changed MiscSelect and DisableDebug option in product-mini/platforms/linux-sgx/enclave-sample/Enclave/Enclave.config.xml file:
<!-- Please refer to User's Guide for the explanation of each field -->
<EnclaveConfiguration>
  <ProdID>0</ProdID>
  <ISVSVN>0</ISVSVN>
  <StackMaxSize>0x100000</StackMaxSize>
  <HeapMaxSize>0x8000000</HeapMaxSize>
  <ReservedMemMaxSize>0x1000000</ReservedMemMaxSize>
  <ReservedMemExecutable>1</ReservedMemExecutable>
  <TCSNum>10</TCSNum>
  <TCSPolicy>1</TCSPolicy>
  <DisableDebug>1</DisableDebug>
  <MiscSelect>1</MiscSelect>
  <MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>

Then I used command:
make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW
./iwasm ../../../app-samples/hello-world/test.wasm

then, I got an error:
./iwasm ../../../app-samples/hello-world/test.wasm
Failed to create enclave from enclave.signed.so, error code: 8196
Fail to initialize enclave.

And when I don't change the Enclave.config.xml, which prevent enclave from running on SGX2 platform. The command will get right output.

Hi, can you provide more details:

What modes combination can you build and run without modifying the sample/Enclave/Enclave.config.xml? For example, Hw & Prerelease, Simulator & Prerelease?
Have you tried running other sgx2 apps? Are they working fine in HW mode and prerelease mode?

First, the original sample/Enclave/Enclave.config.xml is as follows:
<!-- Please refer to User's Guide for the explanation of each field -->
<EnclaveConfiguration>
  <ProdID>0</ProdID>
  <ISVSVN>0</ISVSVN>
  <StackMaxSize>0x100000</StackMaxSize>
  <HeapMaxSize>0x8000000</HeapMaxSize>
  <ReservedMemMaxSize>0x1000000</ReservedMemMaxSize>
  <ReservedMemExecutable>1</ReservedMemExecutable>
  <TCSNum>10</TCSNum>
  <TCSPolicy>1</TCSPolicy>
  <DisableDebug>0</DisableDebug>
  <MiscSelect>0</MiscSelect>
  <MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>

I tried these combination:

I didn't modify the sample/Enclave/Enclave.config.xml, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is no error.
I only modified DisableDebug option to 1, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is as follows:

Failed to create enclave from enclave.signed.so, error code: 8196
Fail to initialize enclave.

I only modified MiscSelect option to 1, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is as follows:

Failed to create enclave from enclave.signed.so, error code: 1
Fail to initialize enclave.

The sgxsdk I used is the version of 2.19. I run the Cxx11SGXDemo in the linux-sgx. I only modified MiscSelect option to 1, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is no error.
About step2, I recently read a doc which told me that to run in Pre-Release mode, I only need to run the makefile with the following options: SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The DisableDebug flag in the enclave config file must remain 0 when running in Pre-release mode.

Would you mind providing the document you read?
For your step 2:

I only modified DisableDebug option to 1, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is as follows:

You cannot launch the enclave in non-debug mode (also called production mode) without Intel signing the enclave, you can refer to this post:
https://community.intel.com/t5/Intel-Software-Guard-Extensions/sgx-enclave-without-debug/td-p/1378801
For your step 3:

I only modified MiscSelect option to 1, and used command make SGX_PRERELEASE=1 SGX_DEBUG=0 SGX_MODE=HW. The result is as follows:

In the document, SGX version 2.19 page 68:
"MiscSelect bit 0, referred to as MiscSelect[0], and MiscMask bit 0, referred to as MiscMask[0] must be set to appropriate values with the rules described below depending on whether UserRegionSize is set. Other bits in MiscSelect and MiscMask are for future functional extensions."
It seems to me that setting MiscSelect option alone is not enough, you may want to try to add more related options following the document.

In the document you share, I find :

If UserRegionSize is configured to a non-zero valid value, that is, the user wants to use the EDMM APIs to manage enclave memory, then:

MiscSelect[0] and MiscMask[0] must be set to 1 and the enclave needsto be loaded on SGX 2.0 platform.

If UserRegionSize is not configured, but the heap, stack and thread related configurations have dynamic part, then:

If MiscSelect[0] = 0, the enclave will be loaded as it has no dynamic components.

if MiscSelect[0] = 1 and MiscMask[0] = 1, the enclave can only be loaded on SGX 2.0 platform.

l if MiscSelect[0] = 1 and MiscMask[0] = 0, the enclave can be loaded on SGX 1.0 and 2.0 platform, and on SGX 2.0 platform, it can utilize the dynamic components.

If UserRegionSize is not configured and the heap, stack and thread configuration only has static components, then:

If MiscSelect[0] = 1, the enclave can only be loaded on SGX 2.0 platform.

If MiscSelect[0] = 0, the enclave can be loaded on SGX 1.0 and 2.0 platform.

My step 3 is MiscSelect[0] = 1 and MiscMask[0] = 1, which means the enclave can only be loaded on SGX 2.0 platform. Do you have tried iwasm in SGX2? Can you share the configuration with me?

I have never tried this feature before, the lower version of SGX 2.0(say 2.2.0 or 2.5.0 for example) didn't support this feature, MiscSelect is always set to 0. This feature is added along the way of release and I am not sure start from which version. So unfortunately I don't have a configuration for your problem.

I feel so bad to hear that.

If I were you, I may try to ask for help at https://community.intel.com/t5/Intel-Software-Guard-Extensions/bd-p/software-guard-extensions or https://github.com/intel/linux-sgx/issues. Those forums should have more SGX experts who have tried this feature and could solve your issue

Hi there,

Does iwasm support SGX2?

The short answer is yes. I'm currently developing on such platforms. Without changing the enclave configuration file, the enclave can be launched on SGX2 platforms (with debugging enabled). Nonetheless, you'll get this warning:

INFO: Enclave configuration 'MiscSelect' and 'MiscSelectMask' will prevent enclave from using dynamic features. To use the dynamic features on SGX2 platform, suggest to set MiscMask[0]=0 and MiscSelect[0]=1.

Assuming you would like to use the dynamic features, you'll need to change the two parameters as mentioned in the message.
Similarly, launching an enclave in prerelease mode requires disabling the debugging mode, as you rightfully pointed out by changing the parameter DisableDebug and compiling with SGX_DEBUG=0 and SGX_PRERELEASE=1. Doing so will lead to a new warning:

INFO: Enclave configuration 'MiscSelect' and 'MiscSelectMask' will prevent enclave from running on SGX1 platform. To make it run on SGX1 platform, suggest to set MiscSelect[0]=0.

Note that launching an enclave with these changes may still lead to the error you face (SGX_ERROR_NDEBUG_ENCLAVE) if your SGX SDK is somehow compiled with debugging enabled. You can test it by setting the second argument of the calls to the function sgx_create_enclave to zero (here: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/product-mini/platforms/linux-sgx/enclave-sample/App/App.cpp#L133-L138).
For prosperity, here is the content of my enclave configuration file (for prerelease mode):
<!-- Please refer to User's Guide for the explanation of each field -->
<EnclaveConfiguration>
  <ProdID>0</ProdID>
  <ISVSVN>0</ISVSVN>
  <StackMaxSize>0x100000</StackMaxSize>
  <HeapMaxSize>0x8000000</HeapMaxSize>
  <ReservedMemMaxSize>0x1000000</ReservedMemMaxSize>
  <ReservedMemExecutable>1</ReservedMemExecutable>
  <TCSNum>10</TCSNum>
  <TCSPolicy>1</TCSPolicy>
  <DisableDebug>1</DisableDebug>
  <MiscSelect>1</MiscSelect>
  <MiscMask>0</MiscMask>
</EnclaveConfiguration>

Hope this helps!
Cheers

Thank you very much!!! It works.


[Feature] Argumented Promotion

Hello,
I am in a situation where an automated levelup/rankup is my go to goal, and the promotion track is kind of grinding against of this.
I have 7 individual groups on my server (called Group1,2,3.. from now on) and every 30 levels (leveling plugin which runs commands) I'd like to have the user be promoted on their current track "Group 1 Track" in this case.
So my Lv. 30 command would look like this;

lp user  promote Group1
lp user  promote Group 2
lp user  promote Group 3

Since not everyone plays Group1, I have to run all the promotions to make sure that every track is promoted with the Lv. 30-60-90-120-150 and so on.
This makes it hard since as an example, Player X has Group1 so on rankup he would be promoted to Rank 2 on Group 1 and Rank 1 on Group 2 and 3 which then makes it so that he has 3 groups instead of 1 intented.
What I am suggesting here is a feature either to somehow have the /lp user  promote check for the current tracks and promote them accordingly or some way of disallowing promotion start to new track if you are already part of one.
Example;
/lp user  promote human if notothertrack
Thank you for your time! If you need any additional information feel free to reply and I try to elaborate on it more.
Regards,
Fawked

A reasonable alternative would be to have an option to enable default assignments every time user permissions change as. That would allow to assign a rankup group and the rules then chose the right upgrade.

How about (optionally) changing the behaviour of the promote command to only run the promotion if they're already on the track?

I think that’s a good idea if it’s an opt-in option. The current behavior is better for the general use case.

That would work as well, then you'd only need to run /lp user <user> promote and it would promote them on their existing track.

No, it would just mean you could run the command multiple times without affecting tracks the user is not yet on.


Caller get_export() implemented for Extern::Func.

To my knowledge, this has not been discussed in an issue. However, comments in the overwritten code make clear that the get_export() function is only currently implemented for Extern::Memory, with potential future support for Extern::Func - which is the update of this PR. This update simply enables user to lookup and return Extern::Func types from Caller structure.

Thanks for the PR! Could you outline a bit the motivation for this change?
Our intention is to sort of keep the Caller type "hamstrung" for now because we don't want it to exist long-term. For now it's the only way to implement WASI APIs, but the intention is that interface types will subsume the need for this API (and we'll remove it once that's implemented and stabilized). With interface types, however, you won't have much access to the calling module's functions, so I'd be wary to enable a pretty critical piece of functionality we're inevitably going to remove.
In any case though I'm curious to hear more about  your use case, since we might still be able to cater to it one way or another!

@alexcrichton @agiachris is working on a test framework for Proxy-Wasm modules (and hopefully WASI in the future), which uses Wasmtime as the runtime. See: https://github.com/proxy-wasm/test-framework/pull/1

Ok thanks for the background. It looks like y'all are primarily using this to lookup the calling module's malloc function, right? If that's the case then that's something which interface types will enable so this seems reasonable to allow.
Could you also update the documentation though to leave the clause that it's only implemented for some types? Additionally can you add some words to the documentation that for functions it should in general follow the guidelines of the interface types proposal for accessing exports and calling them?
Finally, one thing I wanted to point out, y'all will want to be extremely careful when accessing the caller's memory. The main idea is that once you get a raw pointer into wasm memory you can't call wasm again because that pointer could get relocated. In https://github.com/proxy-wasm/test-framework/pull/1 the proxy_get_buffer_bytes export is an example where malloc is called while holding some pointers alive over the call to malloc. Where possible I'd recommend investigating witx and wiggle since they should make implementing all of these host calls with 100% safe code much easier.

@alexcrichton thanks for the suggestion, we will make those changes. I've gone ahead and updated the documentation for the implementation only supporting specific types (Memory and Func), however I'm unfamiliar with the interface types guidelines. Could you please suggest the necessary documentation text for this?

Thanks!


Cannot connect to JMX port

Hello, i'm trying to connect to broker JMX
[centos@slave-03 ~]$ ps aux | grep kafka
centos     512  0.0  0.0 112612   732 pts/0    S+   08:37   0:00 grep --color=auto kafka
root      4173  0.4  2.8 6552620 460940 ?      Ssl  Sep16  10:21 jre/bin/java -cp kafka-mesos-0.9.0-beta.jar -Xmx3500m ly.stealth.mesos.kafka.Executor
[centos@slave-03 ~]$ sudo netstat -tulpan | grep 4173
tcp        0      0 <IP_ADDRESS>:34860     <IP_ADDRESS>:*               LISTEN      4173/jre/bin/java
tcp        0      0 <IP_ADDRESS>:44220     <IP_ADDRESS>:5051      ESTABLISHED 4173/jre/bin/java
tcp        0      0 <IP_ADDRESS>:37293     <IP_ADDRESS>:2181      CLOSE_WAIT  4173/jre/bin/java
tcp6       0      0 <IP_ADDRESS>:1025      :::*                    LISTEN      4173/jre/bin/java
tcp6       0      0 <IP_ADDRESS>:50207     <IP_ADDRESS>:2181      ESTABLISHED 4173/jre/bin/java
[centos@slave-03 ~]$

But the jmc and jconsole connection test keeps saying Unable to connect
Telnet on port works
telnet slave-03 34860
Trying <IP_ADDRESS>...
Connected to slave-03.
Escape character is '^]'.
test
Connection closed by foreign host.

The readme needs to be updated but how to-do this is captured in #69

@joestein @tymofii-polekhin
Hi guys, any updates here?

Resolved


Set APPLICATION_FORM_AWS_REGION for Licensify

This is mandatory following the upgrade to Play 2.7. Currently this is
the same in all regions, so it never needs to be passed as a parameter.

Edit suggestion:

Currently this is the same in all ~regions~ environments

?


Some Core Components display as work in progress and/or v3 in the page editor

See screenshot:

Steps to reproduce

Create site from template
Navigate to and edit the homepage
In Edit Mode, open the components tab in the left side-rail

version
latest version (Standard AEM Site Template 2.1.1)

This should be fixed since https://github.com/adobe/aem-core-wcm-components/releases/tag/core.wcm.components.reactor-2.18.0


[FEATURE]: Allow FQDN for the endpoint in CSI-PowerStore

Describe the solution you'd like
I'm with a customer trying deploy the CSI Driver for PowerStore. We configure the endpoint with a DNS name and I have the error :
Read CSI_LOG_LEVEL from log configuration file, level: debug
{"level":"info","msg":"updating array info","time":"2023-03-16T13:07:00.661953233Z"}
{"level":"fatal","msg":"couldn't initialize arrays in node service: can't get config for arrays: can't get ips from endpoint: https://<domain-name>.com/api/rest","time":"2023-03-16T13:07:00.662665873Z"}

From what I see here : https://github.com/dell/csi-powerstore/blob/64be9636f7a0a525ec656bf43467da95fb922b37/pkg/array/array.go#L230 it looks like we are asking for an IP address.
Why is that ?
It is a bit of a concern for several reasons :

A certificate is almost always issue against a DNS name(s) and rarely on an IP address. And I don't think we can have a trusted certificate by a CA on a private IP
If we have to connect through a load-balancer, or if the IP of the PS changes, it can be an issue to use an IP over an FQDN

The necessary modifications for this feature have been incorporated into the main branch. Starting from version 2.7.0, the PowerStore driver will support it.


msys2-runtime: Allow unprivileged symlink

Tested on Windows 10 Pro (15063.138) x64 and Windows 7 Pro x64.
Added a patch to enable new behavior of symbolic link introduced in Windows 10 Creators Update. (https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/)

Setting this flag unconditionally will cause "Invalid argument" errors on older Windows (source).
Regardless, I don't think we're in any hurry to have this feature, so I'm inclined to close this PR and wait for upstream (which should have it pretty soon). Thanks for your effort, @Einbert-Xeride, and I hope you won't be discouraged to contribute to MSYS2 in the future.

@elieux OK. Thank you. Maybe I've done something wrong while testing on Win7. I may investigate it further.


add wildcard hostname rule to kubernetes gateway

closes #7874
What does this PR do?
Allows a kubernetes gateway api httproute definition to include wildcard hostnames (ie. *.foo.com). As mentioned in #7874 per the spec this is an allowed usage but was originally badly rendered by their documentation renderer (which has now been fixed) so as to properly render the * character, which had previously been interpreted as markdown markup).
Motivation
I've been enjoying using the gateway api via traefik but this became a blocker for rolling it out in what should otherwise be a simple deployment.
More

[x] Added/updated tests
[ ] Added/updated documentation

I did not add documentation but as it is documented in the gateway-api spec and the traefik documentation refers to that, I think it isn't incorrect to omit it.
Additional Notes
I'm not a go developer (I'm a Perl dev with other dynamic languages under my fingers) and a first time contributor (of code) to this project, so if this isn't idiomatic code or if it isn't even the proper generation of rules, please let me know and I'll happily update it (or feel free to correct it for me).

I don't understand why travis failed. How can it be restarted?

I was able to force it to restart with a trivial rebase and force-push but it still fails. I have no idea from that output what the error might be. Rather frustrating.

Looks like it was just some hiccup in travis and/or the docker registry.

Travis is finally passing and now semaphore stops? What is going on?! I'm not changing anything!

And now travis failed again. This is ridiculous. someone else can deal with broken CI systems, I'm done.

Hello @jberger,
Thanks for your contribution!
This PR is now labeled needs-design-review,
as you can see in our contribution guide, this means that we have to take a closer look at the scope of impact for that PR to see, how it would interact with the other parts of Traefik.
We will come back to you soon when we did the first design review iteration.
Don't worry about the CI, we are actively working on CI fixes. :smiley:

Thanks for getting back to me. Honestly one of the biggest reasons I was frustrated with the CI (apologies!) is that in many larger projects, a failing CI report means you don't even get looked at and I was concerned that I wouldn't even hear from you.
For the future reviewer, please note that I took the core of the new business logic from the k8s ingress provider, so I hope that it is at least that correct. My biggest design concern is whether a mixed set of wildcard and non-wildcare hosts should all be grouped together in one HostRegexp rule or should they be two rules ||-ed together. The documentation suggests that you can have non-regexp entries in a HostRegexp rule but I don't know if there's a performance concern. That's up to you.
Thanks again for getting back to me.

Hi @jberger,
We made some modifications, please see our commit.
We separated the Host and HostRegexp rules, manage the error in the listener statuses and add some tests.
Does it seem good for you?

It did take me a couple of reads on the error checking logic before I grokked it but yes it seems fine to me. I made one comment that technically supporting * is specced but not very relevant (for me at least).

Thanks to your comments, the last modifications propose a better implementation following the specification on "" and "*" hostname.
WDYT?

Much clearer! My only question, and this is probably in the weeds and/or pedantic: if someone had hostname foo.com and *, should that just ignore all hostname requirements? I think in this code it would make the hostname effectively require foo.com would it not?
Other than that, LGTM!

@jberger,
I'm glad it is clearer :smiley:
TBH we were wondering how to handle the case where there is a wildcard and, at least, one other hostname.
From a logical perspective, as soon as we use a wildcard, it is like we don't care about the domain name anymore as we will match all of them.
It could be an issue with the way we determine the TLS option to use with the request but as we don't support that feature now, for the sake of simplicity, we have chosen to ignore all other hostnames.
That is why we add the comment:

// TODO Refactor this when building support for TLS options.

If you see another issue with that or something we missed, please let us know :pray:

I see some approvals, is there something I can do to push this over the finish line?

Cheers! Thanks! I look forward to using this!


Placeholder for subscribe

The method doesn't do anything but it should return a correct response

This placeholder seems to have some bad side-effects if HA keeps trying to subscribe again... I will try to replace this placeholder with the real implementation in this PR.

@jdesai61 Actually, can you approve this one so I can merge? The Descriptor branch has been merged into this branch instead of main so this is creating a mess...
In the future, can you just approve and not merge? I will ensure that it merges on the correct branch and delete the branch linked to the PR


Decoding issues on Linux cause Segmentation fault for given sites

Summary.
I have tested the following on my Mac and I get both unicode errors:
For the following:
r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)
I get:
UnicodeError: decoding with 'idna' codec failed (UnicodeError: Invalid character '\x80')
For:
r = requests.get('http://xn--meiner-dta.hamburg', stream=True)
I get:
UnicodeError: ('IDNA does not round-trip', b'xn--meiner-dta', b'meissner')
Which I am good with.
But on linux I get:
Segmentation fault (core dumped)
Crashing my program.
Note the versions of python are slightly different but I guess the following is true for ubuntu.
https://forums.aws.amazon.com/thread.jspa?threadID=255687
Expected Result
Expect it to error out like on MacOS.
System Information
$ python -m requests.help

On Mac
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.3.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

On Linux
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.4.0-1032-aws",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

Hrm. Requests cannot be the cause of this segfault because it's pure-Python code. Can you use GDB or something similar to examine the coredump and check the Python stack to see what code was executing when the segfault occurred?

Lukas, thanks for getting back to me. I tried to run using GDB. I get the following:
test.py
import requests print('Hey') r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)
(gdb) run test.py Starting program: /usr/bin/python3 test.py warning: Error disabling address space randomization: Operation not permitted Hey During startup program terminated with signal SIGSEGV, Segmentation fault.

Sorry, we need a Python backtrace. See this article

Does this work?
(gdb) run test.py
Starting program: /usr/bin/python3 test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Hey

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so

(gdb) bt
#0  0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#1  0x00007ffff6739231 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#2  0x00000000004e9bc7 in PyCFunction_Call ()
#3  0x0000000000524414 in PyEval_EvalFrameEx ()
#4  0x000000000052e12b in PyEval_EvalCodeEx ()
#5  0x00000000004ebdd7 in ?? ()
#6  0x00000000005b7167 in PyObject_Call ()
#7  0x00000000004f413e in ?? ()
#8  0x00000000005b7167 in PyObject_Call ()
#9  0x000000000054d359 in ?? ()
#10 0x000000000055d17c in ?? ()
#11 0x00000000005b7167 in PyObject_Call ()
#12 0x0000000000528d06 in PyEval_EvalFrameEx ()
#13 0x000000000052d2e3 in ?? ()
#14 0x0000000000528eee in PyEval_EvalFrameEx ()
#15 0x000000000052d2e3 in ?? ()
#16 0x0000000000528eee in PyEval_EvalFrameEx ()
#17 0x0000000000528814 in PyEval_EvalFrameEx ()
#18 0x000000000052d82f in ?? ()
#19 0x0000000000529332 in PyEval_EvalFrameEx ()
#20 0x000000000052d2e3 in ?? ()
#21 0x0000000000528eee in PyEval_EvalFrameEx ()
#22 0x000000000052d2e3 in ?? ()
#23 0x0000000000528eee in PyEval_EvalFrameEx ()
#24 0x000000000052e12b in PyEval_EvalCodeEx ()
#25 0x00000000004ebdd7 in ?? ()
#26 0x00000000005b7167 in PyObject_Call ()
#27 0x00000000005262af in PyEval_EvalFrameEx ()
#28 0x000000000052e12b in PyEval_EvalCodeEx ()
#29 0x00000000004ebdd7 in ?? ()
#30 0x00000000005b7167 in PyObject_Call ()
#31 0x00000000005262af in PyEval_EvalFrameEx ()
#32 0x00000000004ed325 in ?? ()

Nope, we need the py-bt

hmm
Undefined command: "py-bt".  Try "help".

Got it.
Needed:
sudo apt-get install gdb python3.5-dbg
(gdb) py-bt
Traceback (most recent call first):
  <built-in method _wrap_socket of SSLContext object at remote 0x7ffff4c43f98>
  File "/usr/lib/python3.5/ssl.py", line 745, in __init__
    server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 305, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 252, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 195, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "test.py", line 3, in <module>

Ok, this segfault is coming from your interpreter. How did you install it?

You can reproduce this if you use the following Dockerfile
FROM ubuntu:latest

RUN apt-get update -y
RUN apt-get install -y python3-pip python3-dev libffi-dev

RUN apt-get install -y vim

COPY requirements.txt /tmp/
RUN pip3 install --requirement /tmp/requirements.txt

requirements.txt
requests==2.18.4

I have the same problems (the unicode ones, not the segmentation fault) in Ubuntu 16.10 with Python 3.6.0b2 and aiohttp==2.2.5
This reproduces the problem:
import asyncio
import aiohttp

async def process():
    url = 'http://xn--x9dj.com/'
    session = aiohttp.ClientSession()
    async with session.get(url) as response:
        print(response)
    session.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(
    process()
)
loop.close()

Traceback:
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 8, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 139, in ToUnicode
    raise UnicodeError("IDNA does not round-trip", label, label2)
UnicodeError: ('IDNA does not round-trip', b'xn--x9dj', b'xn--c09aja')
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe5774205f8>

if I use this domain instead:

ערוץ-1.com

Then I get this:

Traceback (most recent call last):
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 134, in ToUnicode
    label2 = ToASCII(result)
  File "/myenv/lib/python3.6/encodings/idna.py", line 76, in ToASCII
    label = nameprep(label)
  File "/myenv/lib/python3.6/encodings/idna.py", line 58, in nameprep
    raise UnicodeError("Violation of BIDI requirement 3")
UnicodeError: Violation of BIDI requirement 3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 44, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 12, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
UnicodeError: decoding with 'idna' codec failed (UnicodeError: Violation of BIDI requirement 3)
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f31f464c5f8>

I hope this helps clarifying the problem.
Thank you.

Since this is an issue with the Python interpreter and not Requests' code base, I'm going to close this out.
@steven2308 you appear to be using aiohttp and the stacktrace you've provided doesn't contain any Requests code. It appears you're supplying an invalid URI to IDNA which is causing issues. I suggest you follow up with aiohttp if you feel there's an issue.


Ka/generator usage readme

@kalagarraj @monkey-jeff the PR was closed not deleted... all comments and previous code snippets are still visible.


Update Becca and James' Chapter 8 Part 2

@eah13 @jamesma560 Hopefully fixed!

closing this PR in favor of a more recent one.


Fix: reconcile repo server on mountSAToken and mountSAToken Name change setting

What type of PR is this?
/kind bug
What does this PR do / why we need it:
This PR aims to identify all the fields of RepoServer and ensures as CR changes for repo server fields new deployment is triggered with updated values
As a part of PR have seen all fields  of spec.Repo
logLevel, logFormat, image, version are reconciled as a part of init containers
reconcilation on change for env, volumes, volumeMounts, initContainers is already present
sidecarContainers are reconciled as a part of comtainers
left with two fields  mountSAToken, serviceaccount reconiclation logic is in this PR
Have you updated the necessary documentation?

[ ] Documentation update is required by this PR.
[ ] Documentation has been updated.

Which issue(s) this PR fixes:
Fixes #?
How to test changes / Special notes to the reviewer:

Make default deployment of ArgoCD via operator
Create a service Account similar to default service account of a namespace
Add that Service Account into ArgoCD CR .spec.repo.serviceAccount

Check pod using this new ServiceAccount

Thanks @jannfis for merging this


Update-pipeline shouldn't launch a job if nothing changes

Let's say I have a JSON file with two JSON blobs specifying two pipeline specifications.  Then I want to update my code or cmd for only one of the pipelines.  If I do this and then run update-pipeline, Pachyderm will launch a job for both pipelines, even though only one of the pipelines changed.
In other words, update-pipeline will always launch a new job, even if you don't change anything.  This was a little surprising to me.
Maybe we want to run a quick check to see if anything in the spec for each provided pipeline actually changed?  Then, if and only if something changed, launch the job.

Hmm, this gets a little weird I think because what you're describing is also how rerun-pipeline is going to work.

I think @dwhitena is just making a good point that we didn't cover when talking about rerun pipeline. When users have a single JSON spec that defines multiple pipelines, this starts to get weird.
Perhaps only allowing one pipeline per JSON simplifies this? I can see that being annoying in its own right so we might need to tweak our proposal for rerun and/or update pipeline.

stale


Sockets are not closed properly?

I ran into a strange issue, my host reached an open file limit and it looks like the swarm-ingress-cluster container is not closing the docker socket it opens properly.
It looks like the number of open sockets increases by one every ten seconds:
Just do a watch -n 10 "lsof | grep socket | wc -l" inside the container to verify this.
Using version 0.0.3

Are you sure you don't mean v0.3.0? v0.0.3 is long dead and was never even completed. Either way I'll look into it on master see what I can find. I'm just using Docker's API project so the issue may live in there.

Sorry, it was a typo, of course I meant 0.3.0 :)
Thank you.

Yeah you're right I managed to reproduce it easily, but it doesn't happen when I run the project on my Mac. Will look into it further, might be an issue with garbage collection.

What host OS are you running Docker on?

This instance is on an Ubuntu 16.04.1 and I am running Docker version 1.12.1

OK thanks, struggling to see why this connection is being left laying around at the minute.

I've made a few changes, including only instantiating a single Docker client, using a pointer receiver for the client and migrating from the deprecated github.com/docker/engine-api package to github.com/docker/docker/client. Also upgraded to golang 1.7, so hopefully that's everything up to date now. It seems the issue has been fixed as a result.
Can you please confirm that you no longer have the issue with the latest tag? If so I will bump the version and close this issue.
Thanks for your help.

So far so good, looks like the issue is fixed.
Thanks for the quick response!

OK will close this for now, feel free to re-open / add a new issue if there are any more problems.


[bug] ReStructuredText failing for sphinx syntax

The ReStructuredText check fails in those repos which are using sphinx, eg: openwisp-radius.
- <string>:43: (ERROR/3) Unknown directive type "toctree".
-
- .. toctree::
-    :maxdepth: 2
-    :caption: Contents:
-
-    /developer/setup
-    /developer/freeradius
-    /user/settings
-    /user/management_commands
-    /user/importing_users
-    /user/generating_users
-    /user/enforcing_limits
-    /user/registration
-    /user/social_login
-    /user/api
-    /developer/how_to_extend
-    /developer/contributing
-    /general/goals

@KapilBansal320 can we selectively disable some errors?

I will try to find a solution for that.

The ReStructuredText check fails in those repos which are using sphinx, eg: openwisp-radius.
- <string>:43: (ERROR/3) Unknown directive type "toctree".
-
- .. toctree::
-    :maxdepth: 2
-    :caption: Contents:
-
-    /developer/setup
-    /developer/freeradius
-    /user/settings
-    /user/management_commands
-    /user/importing_users
-    /user/generating_users
-    /user/enforcing_limits
-    /user/registration
-    /user/social_login
-    /user/api
-    /developer/how_to_extend
-    /developer/contributing
-    /general/goals

@KapilBansal320 can we selectively disable some errors?

@nemesisdesign Please review pull request #148 for this.


[Bug]: 2.4.0-Snapshot Gradle Build fails do to 403 response from netflix/nebula

Describe the bug
When attempting to run CI/CD tests for Windows installation of the security plugin, I came across a bug where gradle is unable to build core because of the following
Required by:
         project : > org.opensearch.gradle:build-tools:2.4.0-SNAPSHOT:20221105.061416-146 > com.netflix.nebula:nebula-publishing-plugin:4.4.4
      > Could not resolve com.netflix.nebula:nebula-core:3.0.0.
         > Could not get resource 'https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom'.
            > Could not GET 'https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom'. Received status code 403 from server: Forbidden

Looking at the error it seems to be failure to access whatever build dependency netflix/nebula. Notably this issue did not exist as of last Friday 11/4.
To reproduce

Go to this repository: https://github.com/scrawfor99/security/actions/runs/3420665475/jobs/5695777464
Clone or fork the repo
Make a trivial change anywhere in the repo or manually run the workflows
Look at the GitHub Action called "plugin install" under your Actions tab (top middle of your screen on the website)
Wait for the build to fail and scroll down to the failure message
Note: I am not sure this failure occurs every time or not, I have run into it a few times in a row but it may not be every time so if you do not get the failure try rerunning the workflow a couple times

Expected behavior
Gradle build should be able to successfully download all the dependencies it needs.
Screenshots
If applicable, add screenshots to help explain your problem.
Host / Environment
No response
Additional context
No response
Relevant log output
No response

Fixed in https://github.com/opensearch-project/OpenSearch/pull/5132


Docs improvements bounty ideas

read more = broken link: https://ai16z.github.io/eliza/docs/community/stream-notes/
readme is outdated
pnpm run shell not valid
go through and test the docs to be valid
add jsdocs comments to all the code
add links to the code in code snippet examples
check and fix all the issues in github

will add more things to fix for a bounty on https://ai16z.github.io/

When trying to run your custom character in dev mode - pnpm run dev - you need to *pin character to dev shell script

First bullet point is now addressed at https://github.com/ai16z/eliza/commit/951aa3eb8b262b54419b4b3b8ccfd1358ef3ec25 #321 just noting it so others don't attempt it


Extend App\Nova\Resource

I see that "Pktharindu\NovaPermissions\Role" extends "Laravel\Nova\Resource".
Wouldn't it be better to extend "App\Nova\Resource" instead. This way we could create methods to be inherited by all child resources.
For example I'd want to be able to ONLY do this:
namespace App\Nova;

abstract class Resource extends App\Nova\Resource
{
    public function Breadcrumbs() {};
    public function IndexQuery() {};
    public function AbbreviateLabel() {};
}

But at the moment, I'm having to create traits to ensure all resources can inherit; like this:
namespace App\Nova;

abstract class Resource extends NovaResource
{
    use Breadcrumbs;
    use SearchesRelations;
    use AbbreviateLabel;
}

namespace App\Nova;

use Pktharindu\NovaPermissions\Nova\Role as RoleResource;

class Role extends RoleResource
{
    use Breadcrumbs;
    use OrderIndexQueryByName;
    use AbbreviateLabel;
}

Yeah, that makes sense. I'll make a new release fixing this soon!


More tests to cover cross-language class lookups.

Looks OK?

Agreed. Fixed. If OK, will squash.

Squash away!

Please merge.

You're right, try now. Squash?

Looks ready for squashing.

Squashed, ready for merge.


Button should be capitalized by default per MD spec

Bug, feature request, or proposal:
Button text should be in uppercase according to the material design spec.  Should this rule be defaulted in the appropriate contexts in Angular Material?
What is the expected behavior?
https://material.io/guidelines/components/buttons.html#buttons-usage

Button text should be capitalized in languages that have capitalization. For other languages, colored text on flat buttons distinguishes them from normal text.

What is the current behavior?
text is not capitalized
What are the steps to reproduce?
Any button in any sample
What is the use-case or motivation for changing an existing behavior?
Default behavior should follow the spec
Which versions of Angular, Material, OS, browsers are affected?
current version
Is there anything else we should know?
Here's an option to make it work on forms, cards and dialogs and excludes buttons in menus:
form button, md-card button, md-dialog-container button  {
  //:not(md-menu-item)
  text-transform: uppercase;
}

No, it shouldn't be set by default. As @jelbourn mentioned in #2637:

We specifically do not use text-transform as it may have internationalization issues (changing the case may change the meaning of the text in some locales). If you always want to to capitalize the button text, you can add a global rule specific to your application.

What @donroyco said.

That makes sense.  Thanks for the feedback.

@donroyco @jelbourn The button used by md-snackbar does text-transform: uppercase;. It sounds like this may be unintentional.

@ccstevenson yes, that is unintentional and it should be removed.

@jelbourn Can I make a PR for this? Or create a new 'issue' and file a PR in that one...

PR would be great

@ccstevenson @jelbourn PR on the way... #4567


Security Issue

I am using curlrequest in one of my tools and GitHub marks it as security issue:
high severity
Vulnerable versions: <= 1.0.1
Patched version: No fix
curlrequest through 1.0.1 allows execution of arbitrary commands. It is possible to inject arbitrary commands by using a semicolon char in any of the options values.
Code: CVE-2020-7646
Any way to fix this?

I have encountered the same issue.
GitHub has highlighted a specific line that triggered this security flag.
Here's hoping someone can fix it!


ES6 (arrow functions; const>var; let>var); and upgraded all dependencies

Hi @SamuelMarks thanks for taking the time to convert the syntax, but we are happy with the ES5 syntax 👍 especially when it means we can support lots of Node.js versions easily.


Parallelize h5 operations in generate_stats_figs.R

Change  h5 access in generate_stats_figs.R to be parallelizable, replacing the current situation where there is a single file handle repeatedly accessed by rhdf5::H5Dopen inside GetGeneDatamatrix etc.
We want to replace this by functions that take the file path as input (not file handle), so can parallize. Apparently the function rhdf5::h5read will do this.
Related to and required for  #32

@FlicAnderson to invest half a day or so on this, see if it unblocks us (and Ania).

Created new branch h5-open-106 from develop to resolve this issue, rather than from regen-stats-32, to avoid horrors/scope-creep in branch regen-stats-32.
May either cherry pick commits from regen-stats-32 if single-issue, but if that's not easy/possible may just recreate changes in this separate branch for clarity of fixing and testing.

Created new branch h5-open-106 from develop to resolve this issue, rather than from regen-stats-32, to avoid horrors/scope-creep in branch regen-stats-32.
May either cherry pick commits from regen-stats-32 if single-issue, but if that's not easy/possible may just recreate changes in this separate branch for clarity of fixing and testing.

Started with tentative fix process of this issue as applied in regen-stats-32 branch, with 2 changes:
In generate_stats_figs.R, remove hdf5file <- rhdf5::H5Fopen(hd_file) and replace with hdf5file <- hd_file
In read_count_functions.R, change function definition from this:
GetGeneDatamatrix <- function(gene, dataset, hdf5file) {
  hdf5file %>%
    rhdf5::H5Dopen(
      name = paste0("/", gene, "/", dataset, "/reads/data")
    ) %>%
    rhdf5::H5Dread() %>%
    return()
}

to this:
GetGeneDatamatrix <- function(gene, dataset, hdf5file){
  rhdf5::h5read(file = hdf5file, name = paste0("/", gene, "/", dataset, "/reads/data")) %>%
    return()
}

In branch h5-open-106, both changes are made in generate_stats_figs.R because read_counts_functions.R doesn't exist in develop branch, as it's a big addition from #32.
Current vignette run status (commit 32cf983a): fails for each sample as gives error during generate_stats_figs.R stage for each sample.
On interactive investigation:
[1] "Starting: Check for 3nt periodicity globally"
Saving 7 x 7 in image
Warning message:
In write.table(gene_pos_counts_bothends, file = tsv_file_path, append = T,  :
  appending column names to file
[1] "Completed: Check for 3nt periodicity globally"
[1] "Starting: Distribution of lengths of all mapped reads"
Error: Error in h5checktype(). Argument not of class H5IdComponent.
Execution halted

Error seems to lie in:
> gene_sp_read_length <- lapply(gene_names, function(x) {
+   rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", x, "/", dataset, "/reads")), "reads_by_len"))
+ })
Error: Error in h5checktype(). Argument not of class H5IdComponent.

So this section equates to the function GetGeneReadLength() in read_counts_functions.R in branch regen-stats-32, and is because I haven't changed out the rhdf5::H5Aread(), rhdf5::H5Aopen(), rhdf5::H5Gopen()` functions yet.
The same 3-function combo needs checking/updating in the following functions in h5-open-106 branch in generate_stats_figs.R (apart from the bit of code above equivalent to GetGeneReadLength() as just mentioned):

GetGeneLength()
GetGeneReadsTotal()

These should help fix the problem, and is definitely my next step to fix this problem.  Think I stopped the .h5 investigation halfway to swap tasks and missed these other functions when I went back to it.
These functions are quite easily accessible and fixable in regen-stats-32 branch too, but are located in read_counts_functions.R rather than generate_stats_figs.R.

This is the time to start unit testing!
Suggestion, try out: {TestThat} library.
Find a way to save the output and the inputs at key stages, take one or two examples of the input/output relationship, save them, then write test: assert that when we run the function then the output is equal to this. TestThat has tools to build on the assert but key points are, check known input and known output are what are expected despite changes to code.
Steps of suggested process:

Define a test for the function
Change the code to use the different method of accessing .h5 file
Check test still passes
If it does pass, then done.

Run this for all the data-getting/opening() functions, then you're done.
Useful technique for #32 refactoring.

Working on testing the .h5 open function changes in branch h5-open-106.
Created temporary new script test_h5-open-106.R to do this, sourcing generate_stats_figs.R.
4 areas need to be fixed:

GetGeneDatamatrix()
gene_sp_read_length
GetGeneLength()
GetGeneReadsTotal()

I've been working on creating basic (ie non-TestThat) tests for now, checking structures, lengths and ensuring functions work in isolation when swapped out for alternatives which can take the .h5 file location instead of requiring an object of class H5IdComponent as H5Aread() or H5Gopen() and the various others currently in use do.
GetGeneDatamatrix()
As I mentioned previously, GetGeneDatamatrix() seems to be fixable quite easily by swapping out the functions and editing the line which assigns hdf5file a value, giving it the hd_file value, which is the location of the .h5 file for the sample.
# OLD VERSION:
hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file

GetGeneDatamatrix <- function(gene, dataset, hdf5file) {
  hdf5file %>%
    rhdf5::H5Dopen(
      name = paste0("/", gene, "/", dataset, "/reads/data")
    ) %>%
    rhdf5::H5Dread() %>%
    return()
}

# NEW VERSION
hdf5file <- hd_file # filehandle for the h5 file

GetGeneDatamatrixAlt <- function(gene, dataset, hdf5file){
  rhdf5::h5read(file = hdf5file, name = paste0("/", gene, "/", dataset, "/reads/data")) %>%
    return()
}

Running this from the console for the WTnone sample using the following (after running the vignette process to create the .h5 files to have something to access):
R --vanilla --args --num-processes=1 --min-read-length=10 --max-read-length=50 --buffer=250 --primary-id=Name --dataset=vignette --hd-file=vignette/output/WTnone/WTnone.h5 --orf-fasta-file=vignette/input/yeast_YAL_CDS_w_250utrs.fa --rpf=True --output-dir=vignette/output/WTnone --do-pos-sp-nt-freq=True --t-rna-file=data/yeast_tRNAs.tsv --codon-positions-file=data/yeast_codon_pos_i200.RData --features-file=data/yeast_features.tsv --orf-gff-file=vignette/input/yeast_YAL_CDS_w_250utrs.gff3 --asite-disp-length-file=data/yeast_standard_asite_disp_length.txt --count-threshold=64

Then within R, I've run the original and alternate/new version of the function, and am comparing outputs are equal:
# old version outputs:
GeneDatamatrixYAL001C_OLD <- GetGeneDatamatrix("YAL001C", "vignette", hdf5file)

# run new version
GeneDatamatrixYAL001C_NEW <- GetGeneDatamatrixAlt("YAL001C", "vignette", hdf5file)

# check output structure
str(GeneDatamatrixYAL001C_NEW)
# int [1:41, 1:3983] 0 0 0 0 0 0 0 0 0 0 ...

# are outputs equal between old and new?
all.equal(GeneDatamatrixYAL001C_OLD, GeneDatamatrixYAL001C_NEW)
# [1] TRUE

... which they are. So far so good.
The next 3 areas which need fixing don't just involve opening a file connection, they involve pulling out and reading attributes within the .h5 file.  But because we are changing to functions which work from the .h5 file directly rather than via a separate connection object, these involve a bit more than just swapping out one function for another, particularly since these functions involve different nested H5 functions.
I'm in the process of working out what each of those current read/open functions do, and replacing them with the appropriate alternative.
gene_sp_read_length
# gene_sp_read_length:

# OLD VERSION CODE:
# hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file

# > gene_sp_read_length <- lapply(gene_names, function(x) {
#   +   rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", x, "/", dataset, "/reads")), "reads_by_len"))
#   + })

# NEW VERSION CODE:
# TODO

# RUN OLD VERSION:
hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file

gene_sp_read_length <- lapply(gene_names, function(x) {
   rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", x, "/", dataset, "/reads")), "reads_by_len"))
})

str(gene_sp_read_length)
# List of 68
# $ : num [1:41(1d)] 0 0 0 0 0 1 0 0 0 0 ...
# $ : num [1:41(1d)] 0 0 0 0 0 0 0 0 0 0 ...
# $ : num [1:41(1d)] 0 0 0 0 1 0 3 3 9 8 ...
# $ : num [1:41(1d)] 0 0 0 3 5 3 15 16 17 29 ...
# ...

length(gene_sp_read_length)

I still need to work on this, but the work I've started with GetGeneReadsTotal() (see below) should solve this pretty quickly since it's the same 3 functions nested.
GetGeneLength()
# OLD VERSION CODE:
# hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file
# # read length-specific total counts stored as attributes of 'reads_total' in H5 file
# GetGeneLength <- function(gene, hdf5file) {
#   start_codon_pos <- rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "start_codon_pos"))[1]
#   stop_codon_pos <- rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "stop_codon_pos"))[1]
#   return(stop_codon_pos - start_codon_pos)
# }

# NEW VERSION CODE:
# TODO

# RUN OLD VERSION:
hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file
GetGeneLength <- function(gene, hdf5file) {
  start_codon_pos <- rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "start_codon_pos"))[1]
  stop_codon_pos <- rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "stop_codon_pos"))[1]
  return(stop_codon_pos - start_codon_pos)
}
GeneLengthYAL001C_OLD <- GetGeneLength("YAL001C", hdf5file)

# check output structure etc
str(GeneLengthYAL001C_OLD)
# int 3480

# RUN NEW VERSION CODE:
# TODO

# COMPARE OLD AND NEW:
# TODO

Same deal; fixing GetGeneReadsTotal() (see below) should solve this pretty quickly since it's the same 3 functions nested.
GetGeneReadsTotal()

# OLD VERSION CODE:
# hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file
# GetGeneReadsTotal <- function(gene, hdf5file) {
#   rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "reads_total"))
# }

# NEW VERSION CODE:
# hdf5file <- hd_file # filehandle for the h5 file
# GetGeneReadsTotalAlt <- function(gene, hdf5file){
#   rhdf5::h5readAttributes(hd_file, name=paste0("/", "YAL001C", "/", "vignette", "/reads"))[5]
# }

# RUN OLD VERSION:
hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file
GetGeneReadsTotal <- function(gene, hdf5file) {
  rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "reads_total"))
}
GeneReadsTotalYAL001C_OLD <- GetGeneReadsTotal("YAL001C", hdf5file)

# check output structure etc
str(GeneReadsTotalYAL001C_OLD)
# num [1(1d)] 4
head(GeneReadsTotalYAL001C_OLD)
# [1] 4
length(GeneReadsTotalYAL001C_OLD)
#[1] 1

# RUN NEW VERSION CODE:
hdf5file <- hd_file # filehandle for the h5 file
GetGeneReadsTotalAlt <- function(gene, hdf5file){
  rhdf5::h5readAttributes(hd_file, name=paste0("/", "YAL001C", "/", "vignette", "/reads"))[5]
}
GeneReadsTotalYAL001C_NEW <- GetGeneReadsTotalAlt("YAL001C", hdf5file)

# check output structure etc
str(GeneReadsTotalYAL001C_NEW)
#List of 1
#$ reads_total: num [1(1d)] 4

head(GeneReadsTotalYAL001C_NEW)
# $reads_total
# [1] 4

length(GeneReadsTotalYAL001C_NEW)
# [1] 1

# are outputs equal between old and new?
all.equal(GeneReadsTotalYAL001C_OLD, GeneReadsTotalYAL001C_NEW)
# [1] "Modes: numeric, list"
# [2] "names for current but not for target"
# [3] "Attributes: < names for target but not for current >"
# [4] "Attributes: < Length mismatch: comparison on first 0 components >"
# [5] "target is array, current is list"

With this one, I'm pretty close to getting it sorted. The outputs are different because my subsetting isn't quite right. I need to access $reads_total and unlist it probably to match the current output, so this is my next move.
I've looked into the arguments of the current nested functions, and tried to understand them better so I can replicate the functionality
# hdf5file <- rhdf5::H5Fopen(hd_file) # filehandle for the h5 file
# GetGeneReadsTotal <- function(gene, hdf5file) {
#   rhdf5::H5Aread(rhdf5::H5Aopen(rhdf5::H5Gopen(hdf5file, paste0("/", gene, "/", dataset, "/reads")), "reads_total"))
# }

 ## H5Gopen(h5loc, name):
 # h5loc, h5 location identifier - file or group: hdf5file
 # name of group: paste0("/", gene, "/", dataset, "/reads")
 # opens groups of objects /gene/dataset/reads at h5 location hdf5file

 ## H5Aopen(h5obj, name):
 # h5obj: hdf5file,
 # name of the attribute (character): "reads_total"
 # opens attribute "reads_total" from h5obj hdf5file, to be read by H5Aread

 ## H5Aread(h5attribute, buf=NULL):
 # h5attribute: the result of H5Aopen() on "reads_total" attribute
 # buf: a buffer related argument, not currently relevant
 # reads attribute object as created by H5Aopen

Following Edward's comment, will update the gene_sp_read_length bit with info from regen-stats-32 branch to make it into a function (copy updated code across) to make this h5-open-106 code up to date.
Notes about this testing to stay in this issue tracker for now, but will be mined out into the TestThat testing framework later.

In branch h5-open-106, completed test script rscripts/test_h5-open-106.R, to test the 4 areas identified as needing fixes for improved .h5 access.
Following the inital exploration work done here, the original version of the code was added for comparison, and a new version created and tested for the same structure and results on usage.
Each function was tested for 1 gene "YAL001C" from the "vignette" dataset, with hdf5file "vignette/output/WTnone/WTnone.h5", and code covering gene_sp_read_length was tested across  all genes as this code involves an lapply().
The outputs were saved and compared by eye for structure, and for equality using all.equal().
The fixes all pass and seemed to substitute well for the original code, so these fixes were then encorporated into generate_stats_figs.R in commit 3d3c4f6.
This fix was then tested by running generate_stats_figs.R from terminal using the following :
R --vanilla --args --num-processes=1 --min-read-length=10 --max-read-length=50 --buffer=250 --primary-id=Name --dataset=vignette --hd-file=vignette/output/WTnone/WTnone.h5 --orf-fasta-file=vignette/input/yeast_YAL_CDS_w_250utrs.fa --rpf=True --output-dir=vignette/output/WTnone --do-pos-sp-nt-freq=True --t-rna-file=data/yeast_tRNAs.tsv --codon-positions-file=data/yeast_codon_pos_i200.RData --features-file=data/yeast_features.tsv --orf-gff-file=vignette/input/yeast_YAL_CDS_w_250utrs.gff3 --asite-disp-length-file=data/yeast_standard_asite_disp_length.txt --count-threshold=64

which made the appropriate parameters available.
I then sourced  generate_stats_figs.R:
> source("/home/flic/riboviz/rscripts/generate_stats_figs.R")

Attaching package: ‘git2r’

The following objects are masked from ‘package:purrr’:

    is_empty, when

The following object is masked from ‘package:magrittr’:

    add

The following object is masked from ‘package:dplyr’:

    pull

The following object is masked from ‘package:rtracklayer’:

    as.data.frame

The following objects are masked from ‘package:Biostrings’:

    as.data.frame, head

The following object is masked from ‘package:XVector’:

    as.data.frame

The following objects are masked from ‘package:GenomicRanges’:

    as.data.frame, merge

The following objects are masked from ‘package:GenomeInfoDb’:

    as.data.frame, merge

The following objects are masked from ‘package:IRanges’:

    as.data.frame, diff, merge

The following objects are masked from ‘package:S4Vectors’:

    as.data.frame, head, merge

The following object is masked from ‘package:BiocGenerics’:

    as.data.frame

[1] "Created by: RiboViz"
[1] "Date: 2020-04-28 18:44:42"
[1] "File: rscripts/generate_stats_figs.R"
[1] "Version: commit 3d3c4f650c9eb16c9f565eaa791d46ee7a40e141 date 2020-04-28 17:44:11"
[1] "generate_stats_figs.R running with parameters:"
[1] "Starting: Check for 3nt periodicity globally"
Saving 7 x 7 in image
[1] "Completed: Check for 3nt periodicity globally"
[1] "Starting: Distribution of lengths of all mapped reads"
Saving 7 x 7 in image
[1] "Completed: Distribution of lengths of all mapped reads"
[1] "Starting: Biases in nucleotide composition along mapped read lengths"
[1] "Completed nucleotide composition bias table"
[1] "Starting: Check for 3nt periodicity (frame) by Gene"
Parsed with column specification:
cols(
  read_length = col_double(),
  asite_disp = col_double()
)
[1] "Completed: Check for 3nt periodicity (frame) by Gene"
[1] "Starting: Position specific distribution of reads"
Saving 7 x 7 in image
[1] "Completed: Position specific distribution of reads"
[1] "Starting: Calculate TPMs of genes"
[1] "Starting: Correlations between TPMs of genes with their sequence-based features"
Saving 7 x 7 in image
[1] "Completed: Correlations between TPMs of genes with their sequence-based features"
[1] "Starting: Codon-specific ribosome densities for correlations with tRNAs"
Saving 7 x 7 in image
[1] "Completed: Codon-specific ribosome densities for correlations with tRNAs"
There were 16 warnings (use warnings() to see them)

The warnings are fairly standard, and there were no errors on running.
I next tested that the vignette would run:
# vignette run:
# 28 Apr 2020
# h5-open-106 branch, commit: 3d3c4f650c
flic@circe:~/riboviz$ source /home/flic/miniconda3/bin/activate
(base) flic@circe:~/riboviz$ source /home/flic/setenv.sh
(base) flic@circe:~/riboviz$ time python -m riboviz.tools.prep_riboviz -c vignette/vignette_config.yaml
Running under Python: 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]
Created by: RiboViz Date: 2020-04-28 18:51:08.059236 Command-line tool: /home/flic/riboviz/riboviz/tools/prep_riboviz.py File: /home/flic/riboviz/riboviz/tools/prep_riboviz.py Version: commit 3d3c4f650c9eb16c9f565eaa791d46ee7a40e141 date 2020-04-28 18:44:11+01:00
Configuration file: vignette/vignette_config.yaml
Command file: run_riboviz_vignette.sh
Number of processes: 1
# ...
Finished processing 3 samples, 1 failed
Collate TPMs across sample results. Log: vignette/logs/20200428-185108/collate_tpms.log
Count reads. Log: vignette/logs/20200428-185108/count_reads.log
Completed
real	2m19.132s
user	2m58.144s
sys	0m4.047s

This ran successfully (the 1 failed sample being the missing file which we expect to fail).
I then ran the regression test against the regression-test-data-20200221 test data parcel from here:
# regression test against https://github.com/riboviz/regression-test-data-20200221
(base) flic@circe:~/riboviz$ pytest -v riboviz/test/regression/test_vignette.py --expected=/home/flic/regression-test-data-20200221
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/flic/miniconda3/bin/python
cachedir: .pytest_cache
rootdir: /home/flic/riboviz
plugins: cov-2.8.1
collected 72 items
riboviz/test/regression/test_vignette.py::test_index[1-YAL_CDS_w_250] SKIPPED [  1%]
# ...
riboviz/test/regression/test_vignette.py::test_sample_output_pdf[3ntframe_propbygene.pdf-WTnone] PASSED [ 97%]
riboviz/test/regression/test_vignette.py::test_output_tsv[TPMs_collated.tsv] PASSED [ 98%]
riboviz/test/regression/test_vignette.py::test_output_tsv[read_counts.tsv] PASSED [100%]
=================== 42 passed, 30 skipped in 140.26 seconds ====================

Note: 30 are skipped as the test parcel does not include the index and tmp files.
All tested files pass the regression test, and all outputs for the vignette look reasonable and normal.
I'm therefore classing this as a success and submitting pull request for review!

Re-ran regression test with --check-index-tmp, regression test files created using current develop.
(base) flic@circe:~/riboviz$ pytest -v riboviz/test/regression/test_vignette.py     --expected=$HOME/regression-test-data-develop-63241663f2 --check-index-tmp
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/flic/miniconda3/bin/python
cachedir: .pytest_cache
rootdir: /home/flic/riboviz
plugins: cov-2.8.1
collected 72 items
riboviz/test/regression/test_vignette.py::test_index[1-YAL_CDS_w_250] PASSED [  1%]
riboviz/test/regression/test_vignette.py::test_index[1-yeast_rRNA] PASSED [  2%]
# ...
riboviz/test/regression/test_vignette.py::test_output_tsv[TPMs_collated.tsv] PASSED [ 98%]
riboviz/test/regression/test_vignette.py::test_output_tsv[read_counts.tsv] PASSED [100%]
========================= 72 passed in 222.70 seconds ==========================

Ran all tests (excluding regression) and including coverage test:
(base) flic@circe:~/riboviz$ pytest --cov-config=.coveragerc --cov-report term-missing --cov=riboviz --ignore-glob="*regression*"
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/flic/riboviz
plugins: cov-2.8.1
collected 165 items

riboviz/test/test_barcodes_umis.py .................                     [ 10%]
riboviz/test/test_demultiplex_fastq.py ..................                [ 21%]
riboviz/test/test_fastq.py ........................                      [ 35%]
riboviz/test/test_process_utils.py ........................              [ 50%]
riboviz/test/test_sam_bam.py ................                            [ 60%]
riboviz/test/test_trim_5p_mismatch.py ............                       [ 67%]
riboviz/test/test_upgrade_config.py .....                                [ 70%]
riboviz/test/test_utils.py ....................                          [ 82%]
riboviz/test/tools/test_prep_riboviz.py ............                     [ 89%]
riboviz/test/tools/test_prep_riboviz_simdata_multiplex.py .............  [ 97%]
riboviz/test/tools/test_prep_riboviz_simdata_umi.py ....                 [100%]

----------- coverage: platform linux, python 3.7.3-final-0 -----------
Name                                    Stmts   Miss  Cover   Missing
---------------------------------------------------------------------
riboviz/__init__.py                         6      0   100%
riboviz/barcodes_umis.py                   25      0   100%
riboviz/bedgraph.py                        19     19     0%   4-72
riboviz/check_fasta_gff.py                 24     24     0%   4-58
riboviz/compare_files.py                   33     33     0%   4-62
riboviz/count_reads.py                    192     39    80%   133-135, 175-177, 222, 246-248, 251-264, 297-299, 334-336, 384-386, 389-394, 436, 441-443
riboviz/create_fastq_simdata.py           125    125     0%   77-474
riboviz/demultiplex_fastq.py              100     14    86%   233, 238-239, 247-261, 283, 289, 307, 316-319
riboviz/fastq.py                           48      0   100%
riboviz/h5.py                               7      3    57%   26-28
riboviz/hisat2.py                           3      3     0%   5-8
riboviz/logging_utils.py                   24      2    92%   75, 81
riboviz/params.py                          41      0   100%
riboviz/process_utils.py                   50      0   100%
riboviz/provenance.py                      31      2    94%   35-36
riboviz/sam_bam.py                        120     85    29%   101-124, 146-153, 167-179, 196-204, 223-230, 245, 260-346
riboviz/sample_sheets.py                   33      0   100%
riboviz/subsample_bioseqfile.py            23     23     0%   6-52
riboviz/tools/__init__.py                   0      0   100%
riboviz/tools/check_fasta_gff.py           18     18     0%   2-57
riboviz/tools/compare_files.py             18     18     0%   2-73
riboviz/tools/count_reads.py               24      0   100%
riboviz/tools/create_barcode_pairs.py      16     16     0%   2-59
riboviz/tools/create_fastq_simdata.py      14     14     0%   2-51
riboviz/tools/demultiplex_fastq.py         27      0   100%
riboviz/tools/prep_riboviz.py             281     32    89%   354-355, 367, 519-522, 617, 625, 692, 718, 751-759, 773, 819-820, 834-835, 851-864, 876-880, 884
riboviz/tools/subsample_bioseqfile.py      24     24     0%   2-105
riboviz/tools/trim_5p_mismatch.py          27      0   100%
riboviz/tools/upgrade_config_file.py       16     16     0%   2-60
riboviz/trim_5p_mismatch.py                80      7    91%   76-83, 129-132
riboviz/umi_tools.py                        5      0   100%
riboviz/upgrade_config.py                  35      1    97%   191
riboviz/utils.py                           45     13    71%   100-106, 121-123, 198-203
riboviz/workflow.py                       120      1    99%   619
riboviz/workflow_files.py                  20      0   100%
riboviz/workflow_r.py                      20      0   100%
---------------------------------------------------------------------
TOTAL                                    1694    532    69%

========================= 165 passed in 149.97 seconds =========================

This seems good; pull request submitted (#170 )

To close together with pull request.

Merged in pull request #170.
Note to dev team: requires installing package here in R via:
install.packages("here")

Merged in pull request #170.
Note to dev team: requires installing package here in R via:
install.packages("here")

Realised I can notify folks of the merge and newly required R package using @riboviz/developers, but would emailing be preferable for something like this?


[vm/ffi] MSAN failure on handles

/=======================================================================\
| ffi_2/vmspecific_handle_test is new and failed (Crash, expected Pass) |
\=======================================================================/

--- Command "vm" (took 11.000154s):
DART_CONFIGURATION=ReleaseMSANX64 out/ReleaseMSANX64/dart --enable-testing-pragmas --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages /b/s/w/ir/cache/builder/sdk/tests/ffi_2/vmspecific_handle_test.dart

exit code:
-6

stdout:
testHandle
passObjectToC(Instance of 'SomeClass')
result = Instance of 'SomeClass'
HandleReadFieldValue
testClosureCallback Pointer<NativeFunction<(Handle) => Void>>: address=0x7f84c24c2000
ClosureCallbackThroughHandle 0x7f84c24c2000 0x71e000003018
doClosureCallback
() => void
Closure: () => void from Function 'increaseCounter': static.
increaseCounter
ClosureCallbackThroughHandle 0x7f84c24c2000 0x71e00004f818
doClosureCallback
() => void
Closure: () => void from Function 'increaseCounter': static.
increaseCounter
testReturnHandleInCallback
ReturnHandleInCallback 0x7f84c24c200a
returnHandleCallback returning Instance of 'SomeClass'
HandleReadFieldValue
Dart_PropagateError(field_value)
testCallbackReturnException
ReturnHandleInCallback 0x7f84c24c2014
exceptionHandleCallback throwing (Exception: exceptionHandleCallback exception)
callback() returned an error, propagating error
caught (Exception: exceptionHandleCallback exception)
testDeepException
ReturnHandleInCallback 0x7f84c24c201e
callCAgainFromCallback
HandleReadFieldValue
Dart_PropagateError(field_value)
testDeepException2
ReturnHandleInCallback 0x7f84c24c2028
callCAgainFromCallback2
HandleReadFieldValue
Dart_PropagateError(field_value)
callback() returned an error, propagating error
testNull
ReturnHandleInCallback 0x7f84c24c2032
returnHandleCallback returning null
HandleRecursion 1
callback
recurseAbove0(0)
throwing

stderr:
==19168==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x561620b32968 in dart::Api::ClassId(_Dart_Handle*) ../../out/ReleaseMSANX64/../../runtime/vm/dart_api_impl.h:228:9
    #1 0x561620b32968 in dart::Api::IsError(_Dart_Handle*) ../../out/ReleaseMSANX64/../../runtime/vm/dart_api_impl.h:223:27
    #2 0x561620b32968 in Dart_IsError ../../out/ReleaseMSANX64/../../runtime/vm/dart_api_impl.cc:729:10
    #3 0x7f84b6d338ea in HandleRecursion ../../out/ReleaseMSANX64/../../runtime/bin/ffi_test/ffi_test_functions_vmspecific.cc:882:7

SUMMARY: MemorySanitizer: use-of-uninitialized-value ../../out/ReleaseMSANX64/../../runtime/vm/dart_api_impl.h:228:9 in dart::Api::ClassId(_Dart_Handle*)
Exiting

--- Re-run this test:
python tools/test.py -n dartk-msan-linux-release-x64 ffi_2/vmspecific_handle_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8877698707636655312/+/steps/test_results/0/logs/new_test_failures__logs_/0
Cause: https://dart-review.googlesource.com/c/sdk/+/145591

This has to do with poisoning and unpoisoning of the stack when doing FFI calls in which parts of the memory of the stack is sometimes Dart stack and sometimes C stack (see this discussion). The cleanest solution would be to call the MSAN unpoison hooks from the FFI transition/pointer code when the VM is built with MSAN.