READ BEFORE OPEING A ISSUES | About Current Detection | ("Ad blocker is not allowed on Youtube")

This is a Thread to discuss the recent detections answer questions and to clean up the issues page.
What changed why is YouTube detecting this script now?
This script has been UD for many months now what changed?
As of what I'm aware of YouTube has been aware of this script since 6 January 2023 when they took down my YouTube video showing off how to install and use this script.
Recently YouTube has been cracking down on adblocker bypasses for youtube and as this script is very popular at this point and open source we are a big target.
Do I plan on supporting this?
Yes, as u can see I've still been working on a way to get this working again.
Any help is very much appreciated.
Feel free to DM me on Discord (Joelmatic)
Or open a pull request
Why is taking long
Since all the code is open source, both you and I, as well as YouTube employees, can examine how it functions. Currently, I can compile a build successfully on my machine, but YouTube seems to implement patches to counteract the workaround within 2 to 5 hours after each update.
Why did I close all the issues
There was a significant backlog of problems that have since been resolved, along with numerous duplicate issues.
Thank you for all the support it's appreciated as I'm trying to fix this script as I refuse to watch ads or pay to remove them.

If youtube is watching your changes and responding with workarounds, how can you make sure that there is a definite build made?

If youtube is watching your changes and responding with workarounds, how can you make sure that there is a definite build made that works for a longer period of time?

He stops making it open source.

If youtube is watching your changes and responding with workarounds, how can you make sure that there is a definite build made that works for a longer period of time?

He stops making it open source.

he can't really do that, he would probably have to obfuscate the code to prevent youtube from countering it

Or find some magic solution that, even when revealed, still makes detection impossible without seriously affecting non-blocking users. In other words, bombard them with false positives.

I've pushed an update that now allows you to watch YouTube ad-free without any detection (5.4)
Please let me know if it works for u.
Also I'm aware playlist don't work as intended by not auto going to the next video.

Oddly, the previous version now actually sort of works for me. The ads still come up but they all fast forward within about 5 seconds. Maybe whatever yt changed for this update allowed the previous one to work?!

The truth is if youtube wants to put there mind to it they will tear this repo and anything like it apart through engineering solutions against it, even if you found a way to make it closed source (Chrome extension + Obfuscation. + independent extension not on chrome store ) etc , the only way I see this working long term is building many active contributors and also finding a way that is better than a tamperMonkey script as they become hard to maintain after they get big


[sw/vendor, crypto] Update veri-titan to include a hardening patch for modexp.

Best to review commit-by-commit to see the changes in the OTBN modexp file (the file was renamed, so otherwise it shows up as a remove/add).
This PR uses the vendor update process in order to update to the latest version of veri-titan, the project from CMU that generates our formally verified RSA modular exponentiation. Upstream updates that are now included are:

A hardening patch that ensures all loops in modexp execute for the expected number of iterations
Comments from the original handwritten implementation
A RISC-V modexp implementation (currently unused in OT)
Removal of the nops from the end of loops where they are not needed (the previous version always had a nop at the end of every loop)

The code here is still covered by the Dafny verification and fulfills the same formal specification. I've run the proofs and code-generation locally to check for myself.

Also, nice to see us getting rid of some of those nops: any idea if it's made a significant difference to the runtime?

Only a very small difference, something like 1% if I remember correctly -- I calculated it when we first added the code and decided we didn't really mind the extra nops.

Darn! I was hoping we'd suddenly be lightning fast :-)

@jadephilipoom, can you check the CI failure when you get the change? Thanks!

@jadephilipoom, can you check the CI failure when you get the change? Thanks!

It was related to the name change; apparently the OTBN meson rules silently require the target name to match one of the file names. Should be fixed now!


[RSDK-7525] CI deploy to registry

RSDK-7525
Testing
Publishes on release (0.1.9)

Released 0.1.9 on Github and verified it triggers publish to registry workflow
Saw it on the registry as the latest version
Used module at "latest" and saw that the version running was 0.1.9
Used it with mediamtx and ffmpeg x264 with and without passthrough successfully
Used it with mediamtx and ffmpeg x265 successfully
Used it with mediamtx and ffmpeg mjpeg successfully
Ensured it works with our RTSP cam hardware setup in the office

Publishes on pre-release (0.2.0-rc-1)

Released 0.2.0-rc-1 on Github and verified it triggers publish to registry workflow
Verified that you have to manually select the exact version in order to use it and that it wasn't used when "latest" or any major or minor pin was selected. I didn't test functionality because the actual executable is the same as 0.1.9.

Works after merging in Android build stuff

0.2.1 works with RTSP cam hardware

Steps to take before merging, after approval
We need to setup the Viam API key secret for the build action to use. We should not merge this PR until this is set up— either by one of us once one of us gets Github permissions to do so or Eliot.

@nicksanford We can't easily use BIN_OUTPUT_PATH to specify the tarball path in build.sh or meta.json, so I added a new make target module-deploy that copies over the tarball to somewhere platform agnostic


Fix error with sed; run dotnet as root

The 'sed' command was missing an escape character. The dotnet commands failed due to ownership of /tailwind folder and also due to the web app needing to bind to port 80.

lgtm


Support for enumerating hosts from LDAPS source & use ADSI for local account discovery

@joemarriott
I see a few commented code

It is normal?
Did you put those field for future plan?

Hi @mabouchard-dev
Yes I left that commented code in there for future use.  But I'm ok with removing it if you prefer.
Thanks
Joe

@joemarriott No I was just curious, im fine with it.
I just finish testing it and its work fine!
Thanks again for your contribution. It is really appreciated. :)


Code refactoring

Summary
Fixed certain things that SonarLint complained about. They are pretty minor improvements.
Details and comments
Not sure if pulling out certain strings as constants is a good step... Perhaps we could name them better?

Yes, the folder "gauopen" has source code from Gaussian that we don't touch. I even have it not checked by style, pylint, mypy etc.

I added the "backport" status because even though it is not a bug fix, having those changes in stable will reduce the possibility of merge conflicts in future backports.

This should NOT have been labelled backport. Backporting to stable branch is just for bug fixes deemed critical enough to be part of the next point release.


manual backport #251

Description
Manual backport #251
Issues Resolved

Screenshot

Testing the changes

Check List

[ ] All tests pass

[ ] yarn test:jest
[ ] yarn test:jest_integration
[ ] yarn test:ftr

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] Update CHANGELOG.md
[ ] Commits are signed per the DCO using --signoff

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 55.17%. Comparing base (4eb3548) to head (1feace2).

Additional details and impacted files
@@              Coverage Diff              @@
##           workspace     #255      +/-   ##
=============================================
- Coverage      55.60%   55.17%   -0.43%
=============================================
  Files           2260     1193    -1067
  Lines          42470    24235   -18235
  Branches        7539     4086    -3453
=============================================
- Hits           23616    13372   -10244
+ Misses         17394    10191    -7203
+ Partials        1460      672     -788

Flag
Coverage Δ

Linux_2
55.17% <ø> (ø)

Linux_4
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks for fixing the integration test error.


Front end dependencies can move into devDependencies

There are a number of dependencies in package.json that could be moved into a devDependencies section for smaller builds. I don't know the whole list but what jumps out is:
    "@testing-library/jest-dom": "^5.16.2",
    "@testing-library/react": "^12.1.2",
    "@testing-library/user-event": "^13.5.0",

    "@types/jest": "^27.4.0",
    "@types/js-yaml": "^4.0.5",
    "@types/material-ui": "^0.21.12",
    "@types/node": "^16.11.25",

    "js-yaml": "^4.1.0",

Do we really need @types/node?


Not found Base-CenterNet2.yaml.

Hi,i want to use CenterNet2_R2-101-DCN-BiFPN_4x+4x_1560_ST.yaml to demo ,and i have download CenterNet2_R2-101-DCN-BiFPN_4x+4x_1560_ST.yaml and CenterNet2_R2-101-DCN-BiFPN_4x+4x_1560_ST.pth, but i can't find the file of Base-CenterNet2.yaml.so how can i download it.thank you.

Hi, you can found the config here.

Thank you for your rapid response,i have download the model,but i have a new error:
raise KeyError("Non-existent config key: {}".format(full_key))
KeyError: 'Non-existent config key: MODEL.BIFPN'

Hi, can you refer to this and see if the issue can be solved?

Sorry,i have the same file,but i can't solve the problem

Hey @sunset326, not sure if you solved the error but here's what I did, following the solution given in #30
Add this to imports

from projects.CenterNet2.centernet.config import add_centernet_config

Add this line to after get_cfg() command in the setup_cfg() function

add_centernet_config(cfg)

You just need to update the config keys with the ones used by centernet2


Transaction with undelegatebw action signed by cleos sign got unauthorized when pushed

Transaction with undelegatebw action signed by cleos sign got unauthorized when pushed
OS: Ubuntu 16.04
EOS Version: DAWN-2018-04-27-ALPHA

I'm trying to sign and push a transaction like this:
{
  "actions": [
  {
    "account": "eosio",
    "authorization": [{"actor": "snail", "permission": "active"}],
    "data": "0000000000ea30550000000080e8ccc450c300000000000004454f530000000050c300000000000004454f53000000000000000000000000",
    "name": "undelegatebw"
  }
],
"context_free_actions": [],
"delay_sec": 0,
"expiration": "2018-05-08T16:48:24",
"max_kcpu_usage": 0,
"max_net_usage_words": 0,
"ref_block_num": 4526220,
"ref_block_prefix":<PHONE_NUMBER>,
"region": 0
}

It can be signed by cleos sign:
==> ./cleos.sh sign "{\"region\":0,\"ref_block_prefix\":3269722171,\"ref_block_num\":4526220,\"max_net_usage_words\":0,\"max_kcpu_usage\":0,\"expiration\":\"2018-05-08T16:48:24\",\"delay_sec\":0,\"context_free_actions\":[],\"actions\":[{\"name\":\"undelegatebw\",\"data\":\"0000000000ea30550000000080e8ccc450c300000000000004454f530000000050c300000000000004454f53000000000000000000000000\",\"authorization\":[{\"permission\":\"active\",\"actor\":\"snail\"}],\"account\":\"eosio\"}]}"
private key: {
  "expiration": "2018-05-08T16:48:24",
  "region": 0,
  "ref_block_num": 4236,
  "ref_block_prefix":<PHONE_NUMBER>,
  "max_net_usage_words": 0,
  "max_kcpu_usage": 0,
  "delay_sec": 0,
  "context_free_actions": [],
  "actions": [{
      "account": "eosio",
      "name": "undelegatebw",
      "authorization": [{
          "actor": "snail",
          "permission": "active"
        }
      ],
      "data": "0000000000ea30550000000080e8ccc450c300000000000004454f530000000050c300000000000004454f53000000000000000000000000"
    }
  ],
  "signatures": [
    "EOSKicuYAZEuP6LLmZkQyYR9moJajqtbtUQmRK5Nq7FqPy3GTP9Pcvew5Rzr32aw4BMjcXGMyye5KDkE4oz39sW75PYtSJ37V"
  ],
  "context_free_data": []
}

But when I try to push it with -p option, it report missing required authority:
==> ./cleos.sh sign "{\"region\":0,\"ref_block_prefix\":3269722171,\"ref_block_num\":4526220,\"max_net_usage_words\":0,\"max_kcpu_usage\":0,\"expiration\":\"2018-05-08T16:48:24\",\"delay_sec\":0,\"context_free_actions\":[],\"actions\":[{\"name\":\"undelegatebw\",\"data\":\"0000000000ea30550000000080e8ccc450c300000000000004454f530000000050c300000000000004454f53000000000000000000000000\",\"authorization\":[{\"permission\":\"active\",\"actor\":\"snail\"}],\"account\":\"eosio\"}]}" -p
private key: Error 3030001: missing required authority
Ensure that you have the related authority inside your transaction!;
If you are currently using 'cleos push action' command, try to add the relevant authority using -p option.
Error Details:
missing authority of eosio

I have tried some other actions. transfer/newaccount/delegatebw/voteproducer works fine, but undelegatebw responded missing required authority.
The error details said that missing authority of eosio, but I didn't add eosio in authorization and I have the private key of snail only...
P.S. I can push this transaction by cleos push action
./cleos.sh push action eosio delegatebw '{"from":"snail","receiver":"snail","stake_net_quantity":"5.0000 EOS","stake_cpu_quantity":"5.0000 EOS","stake_storage_quantity":"0.0000 EOS"}' --permission snail@active

I wonder that how could you get the hash data??

@guaiguaihw serialized according to contract abi. And you can also get this via abi_json_to_bin rpc interface(I am not tried this way).
I have checked this hex_data by comparing it with the one in succeed transaction pushed by cleos push action command, and it is valid.

Great!! Thanks a lot. @LanfordCai

My fault.


Synchronizing UI graphs and dropdown controls to the data table

A function that inks the record selected on the graph to the data table is required. That way a record selected on the graph will be highlighted on the data table and vise versa.

This is already done


No exe file

I downloaded the only downloadable files in releases, and there is no exe file.

I forgot to upload it :P
I will upload it later.
You can try previous versions.


Support for optional Objects

Currently if the schema has an object property, the form data always contains the object even if it's marked as "required": false.
It would be great if the user could opt out of setting a value for the object.
Possible solution would be for optional objects to have an "Add Item" button similar to the one for arrays (but of course limited to one item).
For example here's a form schema with an optional "widget" object. Desired behavior is that if the user does not specify a widget, the data is {} instead of {"widget":{}}.
{
  "$schema": "http://json-schema.org/draft-03/schema#",
  "type": "object",
  "properties": {
    "widget": {
      "type": "object",
      "id": "urn:jsonschema:com:example:Widget",
      "required": false,
      "properties": {
        "name": {
          "type": "string"
        }
      }
    }
  }
}

Thanks for addressing optional booleans! I don't know if objects are as straightforward, but I hope it doesn't hurt to ask :)

Fixing version v1.5.1

Fixing version v1.5.2


[WIP] feat: add wallet connect transaction popup

What does this PR change?
Fix decentraland/unity-renderer#1690
Our Code Review Standards
https://github.com/decentraland/unity-renderer/blob/master/docs/code-review-standards.md

Superseed by https://github.com/decentraland/unity-renderer/pull/1737


Use Tailwind default screen breakpoints and/or add one for smaller screens

Description

Currently our configuration for screens sizes in Tailwind looks like this:
screens: {
  // Determined by the lower bound of screen width
  tab: '768px',
  desk: '1024px',
  wide: '1216px',
  hd: '1408px',
},

It uses ambiguous custom names, since desk could span all screens that are larger than tab and wide and hd feels unintuitive.
We should consider renaming those to more generic and standard names as Tailwind defaults (md, lg, xl ...), and include a breakpoint for smaller devices; frequent options are 480px and 640px.
Alternatives

Make changes specific to smaller screens manual with media queries on each page. I don't think we really want to go this path.
Additional context

The mockups for mobile designs in Figma have a width of 375px.
@panchovm can also help by sharing what he uses here.
Implementation

[x] 🙋 I would be interested in implementing this feature once we agree on using the default tailwind names or new ones :)

I have been using 375px on mobile mockups to preview them on my phone (iPhone 12 Pro), but I fully agree with following Tailwind definitions. I have always considered Apple iPhone 3GS as the smallest size (320px), but it seems outdated in terms of use.
My understanding is that breakpoints are set for interface changes. In this case, the elements that change significantly are the header, filter area, and audio component. I don't have any preference since those can use the mobile version until reaching the md breakpoint.


Image support

I just realized that docx that does not supported embed data url in . Do you have any ideas for this?
Was looking in a way to add imagens in the zip and reference them.. but seems tricky.
(I've done saving images in the zip, but not sure how to alter the altchunk)
thanks.

To be honest, I've never tried embedding images. I will look into this over the weekend.

We can save the images into the zip, using something like:
   var imgRE = /src="?([^"\s]+)"?\s*/g;
   var canvas = document.createElement('canvas');
    var context = canvas.getContext('2d');

    while ( m = imgRE.exec( html ) ) {
      var url = m[1];
      if(url in images) {
        continue;
      }

      var image = new Image();
      image.src = url;
      context.drawImage(image, 0, 0);
      images[url] = canvas.toDataURL("image/png");
    }

    Object.keys(images).forEach(function(url) {
      html = html.replace(new RegExp('src="'+url+'"','g'), 'src="' + images[url] + '"');
    });

But after it seems a bit tricky.. because I would had to split the html into multiple Altchunks and create Word ML for each image. But it would be really cool to be able to solve it.

Insert img - it is important for me too.
I test insert img to html (data uri):

< img src='data:image/gif;base64,R0lGODlhEAAQA[...skip...]IQA7' />

IE 11 and Chrome 43 show img good, but word 2010 don't show it :(

I've been looking into the problem recently. It appears to be that there is no way of processing images via AltChunk.
AltChunk is a temporary object for a content (html) import to the word ml document. It is converted to xml by Word upon the first save action. AltChunk source is injected as content only, it doesn't tie any local resources (e.g. style sheets, images, etc).
The only workaround is to package the html source into mhtml file and serve it in .doc extension.

OK. Kind of good news here. There is possibility to serve mht content via altChunk.
The release implementing this will be available somewhere next week. Currently it is targets to support the embedded images only (base64 injected via DATA URI), the support for 'regular' images (sourced from static) is also planned.

Fixed by 936f78b

Hi there,
Just tried running the demo with the fixed commit id (1d9fba...), however I am running into the same issue as SkyN (Word informs me this image cannot currently be displayed).
Version of Chrome I am using is 45 and Word 2010. (Image I am using is the cat.jpg as well)
Much appreciated with any help.


Color coordination tweaks

Change the colour of the hr tag to '#B6CEFC80' when in dark theme

assign to me pls

@mshokoya completed


improve pass cancel reason to tablet writer when cancelled (#43388)

backport #43388

run buildall

run buildall


[Package Request]: Add Neovim Nightly

Package Requested

[ ] I would like someone else to build the manifest.
[ ] I would like help so I can submit the manifest.
[X] I have performed a search and couldn't find this package.
[X] I have checked that the installer for this package is not contained in a .zip file
[ ] I think there is a new version available and I have provided the URL.

Please provide the following information
Publisher:  neovim.io
Publisher Url:  https://neovim.io
Package Name:  Neovim Nightly
Description:  Vim-fork focused on extensibility and usability
Package Version:  0.8.0-dev-1092-g89b9eab63
Installer URL:  https://github.com/neovim/neovim/releases/download/nightly/nvim-win64.msi
Current Package Identifier (for new versions):

Thanks for the initial PR. The later updates will be automated.

how do i upgrade this? winget upgrade --id Neovim.Neovim.Nightly doesn't work because the "version" is always 0.11.0 (i.e. the one after the current release) so winget says No newer package versions are available from the configured sources


Weird behaviour on ManyToMany field

I know you don't support Djangae, but there may be some useful information for you to dig :)
https://github.com/potatolondon/djangae/issues/717

Thanks for the input!
The comment on that issue is spot-on: djangae seems to expect you to call .save() after using .add().
If you use factory_boy's post_generation hooks, the DjangoModelFactory base will perform a post-post-generation save as seen in https://github.com/rbarrois/factory_boy/blob/master/factory/django.py#L183.
For a more thorough example, you could find some help here: http://factoryboy.readthedocs.io/en/latest/recipes.html#simple-many-to-many-relationship
Hopefully, this will help future users ;)


[RPC] Break out testing utils from unit tests

Motivation
Breaking out changes from https://github.com/diem/diem/pull/8307 . Refactor of testing helper methods in order to use them in tests in https://github.com/diem/diem/pull/8307 .
Have you read the Contributing Guidelines on pull requests?
yes
Test Plan
No functional changes- all existing tests should pass
Related PRs
https://github.com/diem/diem/pull/8307

/land


Openbanking - DCR Response code should be 201 for POST /register

Currently 200 is returned.

code pushed in jans-auth-server - https://github.com/JanssenProject/jans-auth-server/commit/e855f997ce3028def29d7806bd83ce1f21bcac4a

@maduvena it's wrong. You send 201 for client update. I guess it should return 201 for creation only. For update it should be 200.

Also it's not clear why ticket in in jans-cli if it seems to be about jans-auth-server. Transfering it to jans-auth-server.


Not ARC-compliant

In CHLayoutManager.m, there is an @selector(dealloc), which is illegal. Overall, the custom -dealloc behavior should probably be removed. There is also manual block-copying and block-releasing in CHLayoutConstraint.m

Sorry, but as is mentioned in the readme, CHLayoutManager is no longer maintained. My recommendation would be to add the compiler flags to your project that would make these files be compiled without ARC.

Okay, I thought someone else might be willing to take that on.


If a user leaves the lobby, they are removed from the game party, their data is removed in backend

Tim: 3e5c25638a26c9c1ea3ba1340773ae51422a1997
& thx: Linda: a7c402f348007535a4e17ba79760da14cba333a3


IgnoreYaml doesn't work

My user_settings.json contains "IgnoreYaml": true, however, I still see the front matter being rendered:

Oops, forgot to remove that option. I removed it when I went to a a new scroll algorithm.

Would it be technically possible to add this feature back, in a way that works with the new scrolling algorithm? Would be very useful when authoring documents for Jekyll etc.

Useful how?

The front-matter is currently rendered as if it was a heading, see the screenshot above, which is kind of ugly. No big deal but the IgnoreYaml option was an elegant way to get rid of it.

Does adding a blank line after layout work? It won't be a header then.

It would help the rendering but no one writes front-matter like that. I quite like how GitHub renders it (example), but just hiding it altogether as MDE used to would be best and sounds like a simple thing to do. But maybe there are technical issues with that related to scrolling as you mentioned, I don't know.

It's possible but harder than it sounds. One has to calculate the height of the front matter in preview to subtract it (or is it add?) from the scroll offset. Calculating this generally requires rendering it.

Could it be just removed from Markdown using a regex and then rendered?

That's how it worked before. Still have to figure out how to vertically align the views. It's not a simple offset or percentage of scroll height type calculation (the original algorithm).

I see. It's no big deal if it's too hard to implement properly.


Update summaries to allow support for repeating sections

What is the context of this PR?
Previously summary pages did not support list collectors/repeating sections, this issue was highlighted via a bug when users were attempting to view submitted responses for the berd schema, which contains repeating sections. The view submitted response page was only showing the first two sections prior to the list collector. I have updated the summary context so that summary pages are able to display answers repeating sections.
How to review
Check if anything has been missed. Use the test_view_submitted_response_repeating_sections and test_list_collector_section_summary schema to check the change.
Complete the questionnaire -> Go to the Thank You page -> Click the save or print your answers link -> Check the summary on the page is working.
Check the view submittted response page on the berd schema on this branch and compare with the main branch.
Checklist

[ ] New static content marked up for translation
[ ] Newly defined schema content included in eq-translations repo

 Bug: Issue Printing BERD Submission

Functionally tested and for test_list_collector_section_summary, I'm happy that all my answers now show up on the summary and printed answers, whilst they did not before. Likewise for the new test
Although having made the same change to test_list_collector (adding view_response = True to post_submission) the PDF of answers is empty, is this expected due to additional config that would get this schema to work? They do show when changing the show non item answers option but other surveys have a printed summary working without that so wasn't sure if it was necessary

Re-tested and happy with how the answers are now looking when the sections are created with placeholders, all looking good and rendering as expected
only part that now looks a bit odd to me is for lists inside a section with a static name due to the duplicate titles - this comparison of answering questions vs the summary looks a bit off

I think thats what Mebin was referring to above as well. Is it possible to only have section title for the first of these?

Re-tested and happy with how the answers are now looking when the sections are created with placeholders, all looking good and rendering as expected
only part that now looks a bit odd to me is for lists inside a section with a static name due to the duplicate titles - this comparison of answering questions vs the summary looks a bit off
I think thats what Mebin was referring to above as well. Is it possible to only have section title for the first of these?

Yeah this should be updated now 👍

We need to get this in so they can progress testing. The rendering issue can be fixed in a separate PR.

Happy with the changes, simpler way of managing the group titles 👍 Re-tested and functionally working as expected
Only thing that came up, was I tried the summary with test_list_collector.json and this has a block which adds items to an existing list, so on the summary page it takes both blocks and just shows the same list twice without anything to distinguish between them, I'm not sure when this would get used, but it looks like something that might need addressing in a follow up? I don't necessarily see it as a blocker to this though, so happy to approve if deemed a non-issue

As discussed I will investigate this issue in the follow up PR 👍


chore: bump plugin versions in 1.2

What does this PR do?
chore: bump plugin versions in 1.2 (same as in 1.1.1 - RHIDP-1702 / RHIDP-1694)
Signed-off-by: Nick Boldt<EMAIL_ADDRESS>Screenshot/screencast of this PR
N/A
What issues does this PR fix or reference?
same as https://github.com/redhat-developer/red-hat-developers-documentation-rhdh/pull/133 but for main branch
How to test this PR?
N/A
PR Checklist
As the author of this Pull Request I made sure that:

[x] The Eclipse Contributor Agreement is valid
[x] Code produced is complete
[x] Code builds without errors
[ ] Tests are covering the bugfix
[x] The repository devfile is up to date and works, if one exists
[x] Sections What issues does this PR fix or reference and How to test this PR completed
[x] Relevant user documentation updated
[x] Relevant contributing documentation updated
[x] CI/CD changes implemented, documented and communicated

Reviewers
Reviewers, please comment how you tested the PR when approving it.

@josephca Could you please review the changes in this PR and share your comments?

To assist QE in decision making you can just run the script noted in RHIDP-1702:
container=quay.io/rhdh/rhdh-hub-rhel9:1.1-102
./listPluginVersions.sh -c $container

but of course these versions will change again as we get closer to 1.2RC so you could decide to reject this, or merge it knowing there will be a followup version.
Either way, Doc could validate the change just as easily as QE could, using the script above.


Propagate man pages to user profile

At the moment manpages are not available to the user by default, she has to duplicate the package into her profile to read them with man <program>. This leads to the awkward situation where you have the binary but not the manual.
It would be nice if those got propagated too.

What manpages are you talking about specifically?
I know the Perl manpages are no longer available by default because we moved them into a separate output.

Hm, happened to me quite a few times, but now I’m unsure where. They seem to be propagated by default. I’m going to open another Issue if it happens with a specific program.

They aren't propagated from systemPackages, I believe. On the closure-size branch I was fighting with that by c2fff7283. That could be picked to master, if you think it's a good solution.

@edolstra would have to look at it.


Fix collerek/ormar#588 Bug in queryset count() method

Fixes #588

Thanks for catching that! :)
This is still breaking as the default behavior changes but soon we should release a 0.11 version anyway.

Can you please update the docs too to include the new parameter (check the quick start in index, & readme, update queries/aggregations)?

Done! Let me know how that looks

It seems like it died immediately on Python 3.7, but I'm not sure why. Is there any good way to read the log output?


[WIP] Support GPU bulkstore in vineyardd

What do these changes do?
FEATURE-759, try to enable GPU memory object sharing.

Design Brief

Implement GPUBulkStore with the same functionality interface as BulkStore.
Implement a GPU memory allocator, for now,  naïve "allocate-on-request" is adopted.
Client and Server add gpu-specific session support.
Server sends the cudaIpcMemHandle  to client in reply, client open the handle to get access to the shared GPU memory.

New class:

GPUBulkStore: with the same interface as BulkStore.
GPUBulkAllocator: with the same interface as BulkAllocator.
GPUAllocator: GPU memory allocator, it is left for future work.

Main Related class:

Client: add GPU-specific request method, like createGPUBuffer.
VineyardServer, SocketConnection: add GPU-specific session support.
Payload: add a representation bit is_gpu to distinguish GPU object from CPU object.

Detailed Design

GPU Memory

Implement GPUBulkStore and GPUBulkAllocator for managing objects on gpu.
GPUAlloctor similar to dlmalloc is not implemented.

Client

Add new method, like createGPUBuffer, send GPU-specific request to server.
Client will receive a cudaIpcMemHandle_t handle in reply and open the handle to access to the shared gpu memory .

Server

add gpu_bulk_store_ for managing gpu objects.
add gpu-specific session support and response to client gpu-specific request.
gpu object sharing is accomplished by getting a cudaIpcMemHandle handle and sending the handle to client in reply.

Client/Server interaction

add necessary protocols, like CreateGPUBufferRequest and associated functions.

Related issue number
Feature #759
cc @septicmk

Hi @CSWater, I have left some comments for you, I suggest you consider the following at an early stage:

Make the GPU-feature optional.
Allocate the GPU memory on-demand, building a custom GPU allocator is not in the plan.
Disable the ColdObjectTracker, since this is another orthogonal under-construction feature.
We still need the unit test but don't trigger it in CI since we do not have an online GPU environment.

Hi @CSWater, I have following suggestion:

We should pushdown the ENABLE_GPU marcos in /gpu to make the regular code path more clear. GPU-related methods are always callable but return Status::NotImplement() when GPU feature is not toggled on.
To support heterogeneous objects (blobs are allocated in both GPU and CPU) we should merge GPUBulkStore and BulkStore. You can add something like create_gpu and delete_gpu in the BulkStore.
Rebase the code and solve the conflict after you have finalized your code.


Have you ever thought of building in localStorage and sessionStore?

Hey 👋 , getStorage requires handwriting a set of storage logic which is more cumbersome. Have you ever thought of building in localStorage and sessionStore?

That's true. We haven't, but it sound be great to ship some engines, that way engines can get tested by the community to.

That's true. We haven't, but it sound be great to ship some engines, that way engines can get tested by the community to.

What can I do for you? I'll tell you what I can do for you.


Possible issues with Airbase firmware 1.1.7

we love your work and we have been using your plug for many months and kust loving it, however, today it seems that we have upgraded from airbase version 1.1.3 to 1.1.7 on airbase and also using your plug in latest version 3.1 we can see the status reflected in the apple home app of the unit if controlled from the wall controller, however, we are now unable to control the units from the plugin, any advice would be appreciated, thx Paul

Hi @SASPaul 👋
I am using the latest version of homebridge-daikin-airbase myself and my Daikin Airbase is using firmware version 1.1.7. Changes to the wall controller settings are reflected in the Home app, and changes in the Home app are reflected on the wall controller for me.
As a start for your investigation, could you attempt restarting your Homebridge server in debug mode (adding the -D option to the command line arguments). That should print more logs and you could see if there are any errors being printed.
Let me know.

Hi @SASPaul, I am closing this issue due to inactivity. Feel free to reopen it if you have more information.

Hi Matthieu Di Berardino,
Sorry not to come back to you on this one, all is now working well on the sites using your Daikin plug in.  Very well done on this.  We did need to upgrade the firmware on both ends and then reapply the plugin.
Thanks again for the efforts.
A favour, when you get the chance, can you give me a call on 0429 481 010.  We are planning to write a plug in for homebridge to link to smartthings for lighting control only.  We have all the code and API’s working on the smartthings side, would you be interested in collaborating to on this on the homebridge side?
Regards, Paul
Paul Detering
Director
Smart Automation Systems
A  1/15 Earsdon Street, Yarraville 3013
P   1300 211 644
M +61 429 481 010
W  www.sasys.com.auhttp://www.sasys.com.au/
@.https://www.sasys.com.au/
@.@.***http://www.connecx.com.au/
From: Matthieu Dierardino @.>
Sent: Tuesday, 5 October 2021 1:02 PM
To: yenoiwesa/homebridge-daikin-airbase @.>
Cc: Paul Detering @.>; Mention @.>
Subject: Re: [yenoiwesa/homebridge-daikin-airbase] Possible issues with Airbase firmware 1.1.7 (#33)
Hi @SASPaulhttps://github.com/SASPaul, I am closing this issue due to inactivity. Feel free to reopen it if you have more information.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/yenoiwesa/homebridge-daikin-airbase/issues/33#issuecomment-933996454, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVXFKUBVHL5Y3IVYCA4Q563UFJMAVANCNFSM5ENWMHJQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Hi @SASPaul,
Glad to know that your issues are resolved.
Thank you regarding your offer too, but unfortunately I have a day job that keeps me pretty busy already, so I will pass on it.
Cheers,
Matt


[DOCS] Add 8.13.4 release notes

Preview
https://elasticsearch-hadoop_2226.docs-preview.app.elstc.co/guide/en/elasticsearch/hadoop/master/eshadoop-8.13.4.html

💚 All backports created successfully

Status
Branch
Result

✅
8.13

Questions ?
Please refer to the Backport tool documentation

💚 All backports created successfully

Status
Branch
Result

✅
8.14

Questions ?
Please refer to the Backport tool documentation


TypeError: Cannot read property '0' of null

I love the idea, getting the error of "TypeError: Cannot read property '0' of null" upon basic installation.

Did you include the JS file right before the </body>? Also - can you send me an example of your <span>?


Contributor

Hi, I will like to contribute to Pinecode. Even though I am a beginner in creating a language, I can help

@wmww

Anyway, I wouldn't say I recommend working on Pinecone, as it doesn't appear to have much of a future and I don't have time to help much. If you're curious, however, feel free to wack at it as much as you want. Might learn something, might make something cool. idk.


ftp: add temporary tests to make sure that ftp downloads work

This currently fails as discussed in https://github.com/JuliaLang/Downloads.jl/pull/43. The issue seems to be that switching to FTP passive mode tell the client a :<port> to connect to, which macOS interprets as connecting to a different port on the same host but Linux interprets, incorrectly, as connecting to that port on a host with the name "" which is obviously not going to work.

@giordano, if you or anyone else has time to look into this, I've spent about as much as I can afford at this point. I'll come back to it later, but the core mystery is why switching to passive mode FTP works on macOS but not on Linux, and also why it works in the curl command on Linux but not in libcurl.

If anyone cares to take a look, I've narrowed it down to what's happening in the ftp_state_pasv_resp function.


Include updating offset on scanAgain.

The offset and current character is not updated when jumping to scanAgain after scanning comment.

good catch
thanks for the PR!


Registry unnecessarily redirects blob HEAD requests to storage URL

When the "redirect" option is set to true, it applies to both HEAD and GET requests for blobs. The use case for GETs makes sense - this avoids the registry needing to proxy potentially large blobs from the storage backend to the client. But in the case of HEAD requests, the registry has already done a Stat, and could simply return that information to the client, but instead is issuing a redirect which can add considerable latency.
Am I missing anything that justifies this behavior? Would you be open to a PR to change ServeBlob to avoid redirecting HEAD requests?

@milosgajdos: Any thoughts on this?

From your description, the current behavior seems to comply with the HTTP specs; from RFC9110

9.3.2. HEAD
The HEAD method is identical to GET except that the server MUST NOT send content in the response. HEAD is used to obtain metadata about the selected representation without transferring its representation data, often for the sake of testing hypertext links or finding recent modifications.
The server SHOULD send the same header fields in response to a HEAD request as it would have sent if the request method had been GET. However, a server MAY omit header fields for which a value is determined only while generating the content. For example, some servers buffer a dynamic response to GET until a minimum amount of data is generated so that they can more efficiently delimit small responses or make late decisions with regard to content selection. Such a response to GET might contain Content-Length and Vary fields, for example, that are not generated within a HEAD response. These minor inconsistencies are considered preferable to generating and discarding the content for a HEAD request, since HEAD is usually requested for the sake of efficiency.

I guess this part may be relevant here;

However, a server MAY omit header fields for which a value is determined only while generating the content.

Performing the Stat would fall within that bit, but I guess it still needs to do the Stat in order to get the redirect location?

I don't think it would violate the RFC for the registry to generate its own response to a HEAD using the stat information it already has. In fact, the RFC seems to encourage this kind of optimization:

These minor inconsistencies are considered preferable to generating and discarding the content for a HEAD request, since HEAD is usually requested for the sake of efficiency.

Thinking a bit further; so, in your case, there's HEAD request, but the client sees the redirect, so follows it?
Wondering if that's the part to look into; basically; if the HEAD has the headers it need, it wouldn't have to follow the redirect and do another request. I think that would still comply to the RFC if we do the same for both GET and HEAD
The cache headers should probably be omitted though in that case, as (IIUC) the redirect-to URL would be an ephemeral URL, so result should not be cached? https://github.com/distribution/distribution/blob/8857a1948739db3520e2fb5211979aa62c3dd5da/registry/storage/blobserver.go#L59-L60

I don't think it would violate the RFC for the registry to generate its own response to a HEAD using the stat information

Right, so I think there's two bits here;

Add the extra headers https://github.com/distribution/distribution/blob/8857a1948739db3520e2fb5211979aa62c3dd5da/registry/storage/blobserver.go#L62-L74

Remove the Location header

From the above, it feels like 1. is fine, but 2. may not be.
However, I think clients should be allowed to ignore a Location header (don't follow redirects), so in case the response contains the headers they need, they can ignore the Location header and be done. Clients that don't have that logic would continue to follow the redirect.

However, I think clients should be allowed to ignore a Location header (don't follow redirects), so in case the response contains the headers they need, they can ignore the Location header and be done. Clients that don't have that logic would continue to follow the redirect.

I'm pretty sure existing clients do not have htis logic. Anything written in Go (such as Docker) will follow redirects automatically by default, unless this is explicitly disabled. I do not see the distribution client disabling redirect following: https://github.com/distribution/distribution/blob/8857a1948739db3520e2fb5211979aa62c3dd5da/registry/client/repository.go#L876-L880
Anyway, I think it would make a lot of sense to at least provide an opt-in setting for people who prefer efficiency and cost savings over strict RFC compliance.


Properly handle addition of grandchildren and beyond to hidden nodes.

Objects are added incorrectly with no parent if their grand-parent or higher is hidden.


Forseti Issues, Not able to create module. Inventory is also showing zero count. Forseti Broken

Hi team,
We are facing issues after deploying Forseti server.
##Forseti Version:
"""Google Cloud Forseti."""
version = '2.25.0'
package_name = 'forseti-security'
--We are having an issue with Inventory and Model Module. Also scanner/violation is not working. Inventory is not getting collected. Showing as zero. Also not able to create Model. Even when i list it says no model found. When i try to create a module manually, it shows as broken.
--We also applied a policy however it is not being a being scanned and not showing any violations in security command center
forseti service
--It is running
--We deployed Forseti through terraform code.
--Server is deployed successfully however when we login into server and running inventory command it is not working. CSCC id was provided with correct service credentials.
Logs
--attached is the logs:
a) forseti.log
syslog file it is not allowing to upload as very big. please provide ftp
Please assist asap
[forseti.log](https://github.com/forseti-security/forseti-security/files/48463
forseti.log

Just to let you guys know that we had deployed forseti 3 times however same issue. It successfully created 85 resources however not working. Please call me at +91<PHONE_NUMBER>
Regards,
Amit Gupta

Hi @agupta-1, thanks for creating an issue. I recommend to upgrade to the latest Forset patch v2.25.1. There was an issue fixed with the version you are on; it might not resolve the issue, but don't want you to run into another issue. You can upgrade with Terraform by setting the version in the Forseti configuration to ~> 5.2.0.
I don't see any errors in the log that you sent that indicate why the inventory didn't get created. It doesn't look like you have configured the Service Account to use Domain wide delegation, which is required for scanning G Suite resources. If you do not want to scan those resources, then I recommend you also set the following variables in the Terraform config:
admin_disable_polling    = true
group_enabled                 = false
groups_settings_enabled = false

Please try those changes and then let me know if you are still running into issues. Instead of sharing the log file, if there are issues, you can search Stackdriver Logging for any errors coming from the VM during inventory, and include those error messages; just in case you accidentally share sensitive information.

Currently, reinstalling one more time and will check. Will upload the logs as requested

After re-installation same issue. Inventory is not getting collected even after 14 hours.
@forseti-server-vm-4e3a596a:~# forseti inventory list
{
"id": "1593694846257574",
"startTimestamp": "2020-07-02T13:00:46Z",
"schemaVersion": 1,
"status": "CREATED",
"warnings": "Your inventory contains warning message(s), please run command forseti inventory get<PHONE_NUMBER>257574 for more information.",
"countObjects": 0,
"errors": ""
}
--Forseti model list is not listing any model
--When try to run inventory getting below status however it is always in hung state. not going beyond this screen.

echo 'Running Forseti inventory.'
Running Forseti inventory.
forseti inventory create --import_as 20200702T130639

Forseti.log

2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): retention_days is:
2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): retention_days is not specified.  Will use configuration default.
2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): Purge is disabled.  Nothing will be purged.
2020-07-02 13:06:40,760 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051
2020-07-03 01:47:54,150 WARNING [forseti-security][2.25.0] google.cloud.forseti.services.cli(load): IOError - trying to open configuration file located at /root/.forseti
2020-07-03 01:47:54,151 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051
2020-07-03 01:48:06,586 WARNING [forseti-security][2.25.0] google.cloud.forseti.services.cli(load): IOError - trying to open configuration file located at /root/.forseti
2020-07-03 01:48:06,587 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051

Hi Team,
Can you please check and confirm. The same version which i deployed was somehow working however after re-deployment again it is not collecting inventory and is not able to create module. I am already on latest version (2.25).

I did installed 2.25.1 i.e. the latest version however still facing the same issue. It is still not creating inventory and unable to create model. Please assist.

Hi @agupta-1, I am looking into this issue. It appears that others are having the same issue and I have reproduced it as well. Will provide an update as soon as I can track down what is going on. Similar ticket: https://github.com/forseti-security/forseti-security/issues/3770

Hi @agupta-1, I believe I have found an issue that might be the cause of Forseti not creating a model in your instance. More info here: https://github.com/forseti-security/forseti-security/issues/3774.

Can you give more clarity on "Update model creation to ignore the case of the permission.". How to achieve this task?

I will be providing a fix. I might have something available today from the main branch, if not then next week; and we will provide a patch release as well.

@gkowalski-google Waiting for your update. Thanks

Hi @agupta-1, the fix has been merged into the master branch. You can deploy this version with Terraform by setting forseti_version = "master"'. It is recommended to use the master branch version of the Forseti Terraform module as well, which can be done by using the module sourced from GitHub source = "git::https://github.com/forseti-security/terraform-google-forseti.git"`.
I am working on a patch release that will be out next week.

@gkowalski-google Hi as i can see it was merged in master branch, i did went for clean installation again. And guess what - Issue remains the same. Not sure how to resolve this now. Its almost a month.

@agupta-1 Can you provide your terraform configuration?

@gkowalski-google Do you want me to provide main.tf file?
it is same error which i pasted earlier in case

@agupta-1 Yes, the main.tf would be helpful (please remove any sensitive info). To be sure, can you please filter the Stackdriver VM logs for errors? Some errors can be ignored, I am interested to know if you see anything like that contains sqlalchemy.exc.IntegrityError or Duplicate entry.

@gkowalski-google
Here is main.tf file:
module "forseti" {
source = "terraform-google-modules/forseti/google"
version = "5.2.1"
gsuite_admin_email = var.gsuite_admin_email
domain             = var.domain
project_id         = var.project_id
org_id             = var.org_id
enable_write       = "false"
Configure the Cloud Security Command Centre
cscc_source_id          = var.cscc_source_id
cscc_violations_enabled = true
enabled config validator
config_validator_enabled                  = var.config_validator_enabled
config_validator_violations_should_notify = var.config_validator_violations_should_notify
enable policy library
commented out as tcp/22 is not available via nat gateway
policy_library_sync_enabled   = var.policy_library_sync_enabled
policy_library_repository_url = var.policy_library_repository_url
Enter the type of instance that you want to create. Defaults to n1-standard-2 if left empty
server_type   = var.server_type
client_type   = var.client_type
cloudsql_type = var.cloudsql_type
Enter the locations where resources are to be created
storage_bucket_location = "#####"
bucket_cai_location     = "#########"
server_region           = "######"
client_region           = "######"
cloudsql_region         = "####"
Tell forseti to run every 30 mins
forseti_run_frequency = "0 */1 * * *"
This stops the client and server instance from getting a public ip
client_private = "true"
server_private = "true"
Enter the Network details if you want the Forseti cluster to be created in a specific location. If not Forseti will create its own network environment
network         = var.network
subnetwork      = var.subnetwork
network_project = var.network_project
Enter the tags for the instances created.
server_tags = ["removed"]
client_tags = ["removed"]
Enter an IP range in CIDR notation if you want to allow ssh access to the box. OPTIONAL
server_ssh_allow_ranges = []
client_ssh_allow_ranges = []
manage_rules_enabled = false
}
module "rules" {
source = "./modules/rules"
bucket_id = module.forseti.forseti-cai-storage-bucket
org_id = var.org_id
domain = var.domain
}
--Also as per stack driver logs:
Error 1:
[forseti-security][2.25.1] google.cloud.forseti.services.inventory.crawler(visit): (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) Traceback (most recent call last): File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect return fn() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 425, in connect return _ConnectionFairy._checkout(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 822, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 559, in checkout rec._checkin_failed(err) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 556, in checkout dbapi_connection = rec.get_connection() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 683, in get_connection self.__connect() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 701, in __connect connection = pool._invoke_creator(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 437, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/init.py", line 94, in Connect return Connection(*args, **kwargs) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 325, in init self.connect() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 599, in connect self._request_authentication() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 861, in _request_authentication auth_packet = self._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 657, in _read_packet packet_header = self._read_bytes(4) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 707, in _read_bytes CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/crawler.py", line 126, in visit self.write(resource) File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/crawler.py", line 148, in write self.config.storage.write(resource) File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/storage.py", line 995, in write result = self.engine.execute(Inventory.table.insert(), resource_row) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2143, in execute connection = self.contextual_connect(close_with_result=True) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2192, in contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect e, dialect, self File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 296, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 276, in reraise raise value.with_traceback(tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect return fn() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 425, in connect return _ConnectionFairy._checkout(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 822, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 559, in checkout rec._checkin_failed(err) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 556, in checkout dbapi_connection = rec.get_connection() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 683, in get_connection self.__connect() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 701, in __connect connection = pool._invoke_creator(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 437, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/init.py", line 94, in Connect return Connection(*args, **kwargs) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 325, in init self.connect() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 599, in connect self._request_authentication() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 861, in _request_authentication auth_packet = self._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 657, in _read_packet packet_header = self._read_bytes(4) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 707, in _read_bytes CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
Error2:
[forseti-security][2.25.1] google.cloud.forseti.services.model.importer.importer(_flush_session): Unexpected SQLAlchemyError occurred during model creation. Traceback (most recent call last): File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1236, in _execute_context cursor, statement, parameters, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 328, in _query conn.query(q) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 517, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 732, in _read_query_result result.read() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 1075, in read first_packet = self.connection._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 684, in _read_packet packet.check_error() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/protocol.py", line 220, in check_error err.raise_mysql_exception(self._data) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/err.py", line 109, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (forseti_security.c984caa3da4033d08ef35c3c25afa37d_bindings, CONSTRAINT c984caa3da4033d08ef35c3c25afa37d_bindings_ibfk_2 FOREIGN KEY (role_name) REFERENCES c984caa3da4033d08ef35c3c25af)') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/model/importer/importer.py", line 230, in _flush_session self.session.flush() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2446, in flush self._flush(objects) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2584, in _flush transaction.rollback(_capture_exception=True) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2544, in _flush flush_context.execute() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 416, in execute rec.execute(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 583, in execute uow, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 245, in save_obj insert, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements statement, params File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 980, in execute return meth(self, multiparams, params) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement distilled_params, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1240, in _execute_context e, statement, parameters, cursor, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception util.raise_from_cause(sqlalchemy_exception, exc_info) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 296, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 276, in reraise raise value.with_traceback(tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1236, in _execute_context cursor, statement, parameters, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 328, in _query conn.query(q) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 517, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 732, in _read_query_result result.read() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 1075, in read first_packet = self.connection._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 684, in _read_packet packet.check_error() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/protocol.py", line 220, in check_error err.raise_mysql_exception(self._data) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/err.py", line 109, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.IntegrityError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (forseti_security.c984caa3da4033d08ef35c3c25afa37d_bindings, CONSTRAINT c984caa3da4033d08ef35c3c25afa37d_bindings_ibfk_2 FOREIGN KEY (role_name) REFERENCES c984caa3da4033d08ef35c3c25af)') [SQL: 'INSERT INTO c984caa3da4033d08ef35c3c25afa37d_bindings (resource_type_name, role_name) VALUES (%(resource_type_name)s, %(role_name)s)'] [parameters: {'resource_type_name': 'project/sys-68073348388346939442958192', 'role_name': 'roles/owner'}] (Background on this error at: http://sqlalche.me/e/gkpj)

@agupta-1 The forseti_version var is missing, please set this to master. I am going to be releasing the patch release for this version today, so it would probably make sense to wait for that. Once that is published, then you won't need to set forseti_version and you would just need to update version from "5.2.1" to "5.2.2".

@gkowalski-google I am fine to re-deploy. Hence will wait for your patch.
Also to re-confirm are you saying i need to enter this variable in main.tf file for now as below
module "forseti" {
source = "terraform-google-modules/forseti/google"
version = "5.2.1"
forseti_version = "master"
gsuite_admin_email = var.gsuite_admin_email
domain = var.domain
project_id = var.project_id
org_id = var.org_id
enable_write = "false"
As yoy can see version is already set to 5.2.1 Do i need to remove this one while entering forseti_version = "master"

@agupta-1 Sorry for the confusion. Once patch is released then you can just set version = "5.2.2" and remove the forseti_master var from the config.

@agupta-1  The latest Forseti release is out to fix the model creation issue. Please upgrade at your convenience and let me know if the your issue is resolved. You can upgrade with Terraform by setting the Forseti module version to 5.2.2 or ~> 5.2.0.

@gkowalski-google Did updated however same issue. Apologies however i am sure that something is wrong in the code itself which is not allowing to create the model.
If you want i can re-deploy complete forseti once again. But i have a strong feeling it will produce the same results.

Hi @agupta-1, can you please send any errors that you are seeing in the Stackdriver logs? Can you also please schedule a meeting with me so that we can troubleshoot? My email is gkowalski at google.com.

@gkowalski-google Hi, Sure let me do a fresh install one more time with version 5.2.2 (by changing the version in main.tf) so that it will be easier for you to troubleshoot. Let me drop you a direct email to connect with the schedule.
Thanks

Hi Gregg,
Please find the update as of now:

I checked inventory is showing as created,
~$ forseti inventory list
{
"id": "1596085245764606",
"startTimestamp": "2020-07-30T05:00:46Z",
"completeTimestamp": "2020-07-31T07:43:48Z",
"schemaVersion": 1,
"countObjects": 362184,
"status": "SUCCESS",
"warnings": "Your inventory contains warning message(s), please run command forseti inventory get<PHONE_NUMBER>764606 for more information.",
"errors": ""

I checked model is showing as created as partial success:

$ forseti model list
{
"handle": "3389c80630e665466c8e4ac8abfb2219",
"name": "20200730T050044",
"status": "PARTIAL_SUCCESS",
"description": "{"gsuite_enabled": false, "pristine": true, "source": "inventory", "source_info": {"inventory_index_id":<PHONE_NUMBER>764606}, "source_root": "organization/379193874099"}",
"message": "228588",
"createdAt": "31 July 2020 - 07:43:48"
I am nor sure if the above model is completed or not. Because Still we are not receiving any notification in CSCC even though it is enabled.
Note: I increased forseti security resources to n1-standard-8
Let me know.

Hi @agupta-1, sorry I missed your update. Can you run forseti model get 20200730T050044 and see what the warnings/errors are? You can also filter the server VM logs for any errors during inventory/model creation. Let me know if you can provide any of those details.

@agupta-1 Were you able to see what the model issues were? A partial success for model creation is fairly typical. There are some warnings that get reported when Forseti is trying to process the roles, and these do not break anything. If you are still not seeing any CSCC findings, can you please check the server VM logs for any errors and confirm you followed all steps to set up CSCC integration?


in the zh-CN.json file, and I found 3 wrong Chinese Characters

i made zh-CN.json file, but found 3 wrong Chinese Characters
https://github.com/tim-hub/obsidian-bible-reference/issues/219
I correct this file.
创世纪 is wrong, and 创世记 is correct.             纪 and 记  look very simmilar.
贴后 is wrong, and 帖后 is correct.                     贴 and 帖  look very simmilar.
列王记下 is wrong, and 列王纪下 is correct.      记 and 纪  look very simmilar.
simplified Chinese bible is here.  https://www.bible.com/zh-CN/bible/
new file is here.
zh-CN.json

Please check this release, https://github.com/tim-hub/obsidian-bible-reference/releases/tag/3.2.1


Add <track> element

@petele can you also rename the file to text-tracks.yml in line with audio-video-tracks.yml?


While processing image, it reports RuntimeError: GET was unable to find an engine to execute this computation

When did you clone our code?
I cloned the code base after 5/1/23
Describe the issue
Issue:
I use gradio demo to test conversation. There is no problem when using plain text for conversations.
But it reports "RuntimeError: GET was unable to find an engine to execute this computation" while processing image. And all subsequent conversations also showed this error. The model_worker must be restarted to solve this problem. The image I use is what the webpage provides "What is unusual about this image?".
Command:
python -m llava.serve.controller --host <IP_ADDRESS> --port 10000

python -m llava.serve.model_worker --host <IP_ADDRESS> --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path weights/llava-13b-v0 --multi-modal

python -m llava.serve.gradio_web_server --controller http://localhost:10000

Log:
2023-05-08 19:28:47 | INFO | model_worker | Register to controller
2023-05-08 19:28:47 | ERROR | stderr | INFO:     Started server process [31603]
2023-05-08 19:28:47 | ERROR | stderr | INFO:     Waiting for application startup.
2023-05-08 19:28:47 | ERROR | stderr | INFO:     Application startup complete.
2023-05-08 19:28:47 | ERROR | stderr | INFO:     Uvicorn running on http://<IP_ADDRESS>:40000 (Press CTRL+C to quit)
2023-05-08 19:29:02 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: None. global_counter: 0
2023-05-08 19:29:17 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: None. global_counter: 0
2023-05-08 19:29:20 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2023-05-08 19:29:20 | INFO | stdout | INFO:     <IP_ADDRESS>:48850 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-05-08 19:29:28 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2023-05-08 19:29:32 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2023-05-08 19:29:47 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2023-05-08 19:29:52 | INFO | model_worker | Send heart beat. Models: ['llava-13b-v0']. Semaphore: Semaphore(value=4, locked=False). global_counter: 2
2023-05-08 19:29:52 | INFO | stdout | INFO:     <IP_ADDRESS>:35472 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-05-08 19:29:52 | ERROR | stderr | ERROR:    Exception in ASGI application
2023-05-08 19:29:52 | ERROR | stderr | Traceback (most recent call last):
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
2023-05-08 19:29:52 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     return await self.app(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await super().__call__(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     raise exc
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await self.app(scope, receive, _send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     raise exc
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await self.app(scope, receive, sender)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     raise e
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await self.app(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     await route.handle(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
2023-05-08 19:29:52 | ERROR | stderr |     await self.app(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/routing.py", line 69, in app
2023-05-08 19:29:52 | ERROR | stderr |     await response(scope, receive, send)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
2023-05-08 19:29:52 | ERROR | stderr |     async with anyio.create_task_group() as task_group:
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
2023-05-08 19:29:52 | ERROR | stderr |     raise exceptions[0]
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/responses.py", line 273, in wrap
2023-05-08 19:29:52 | ERROR | stderr |     await func()
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/responses.py", line 262, in stream_response
2023-05-08 19:29:52 | ERROR | stderr |     async for chunk in self.body_iterator:
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/concurrency.py", line 63, in iterate_in_threadpool
2023-05-08 19:29:52 | ERROR | stderr |     yield await anyio.to_thread.run_sync(_next, iterator)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
2023-05-08 19:29:52 | ERROR | stderr |     return await get_asynclib().run_sync_in_worker_thread(
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
2023-05-08 19:29:52 | ERROR | stderr |     return await future
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
2023-05-08 19:29:52 | ERROR | stderr |     result = context.run(func, *args)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/starlette/concurrency.py", line 53, in _next
2023-05-08 19:29:52 | ERROR | stderr |     return next(iterator)
2023-05-08 19:29:52 | ERROR | stderr |   File "/data1/fc/LLaVA/llava/serve/model_worker.py", line 306, in generate_stream_gate
2023-05-08 19:29:52 | ERROR | stderr |     for x in self.generate_stream(params):
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
2023-05-08 19:29:52 | ERROR | stderr |     response = gen.send(None)
2023-05-08 19:29:52 | ERROR | stderr |   File "/data1/fc/LLaVA/llava/serve/model_worker.py", line 251, in generate_stream
2023-05-08 19:29:52 | ERROR | stderr |     out = model(
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
2023-05-08 19:29:52 | ERROR | stderr |     output = old_forward(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/data1/fc/LLaVA/llava/model/llava.py", line 223, in forward
2023-05-08 19:29:52 | ERROR | stderr |     outputs = self.model(
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/data1/fc/LLaVA/llava/model/llava.py", line 126, in forward
2023-05-08 19:29:52 | ERROR | stderr |     image_forward_outs = vision_tower(images, output_hidden_states=True)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 934, in forward
2023-05-08 19:29:52 | ERROR | stderr |     return self.vision_model(
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 859, in forward
2023-05-08 19:29:52 | ERROR | stderr |     hidden_states = self.embeddings(pixel_values)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward
2023-05-08 19:29:52 | ERROR | stderr |     patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-05-08 19:29:52 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
2023-05-08 19:29:52 | ERROR | stderr |     return self._conv_forward(input, self.weight, self.bias)
2023-05-08 19:29:52 | ERROR | stderr |   File "/root/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
2023-05-08 19:29:52 | ERROR | stderr |     return F.conv2d(input, weight, bias, self.stride,
2023-05-08 19:29:52 | ERROR | stderr | RuntimeError: GET was unable to find an engine to execute this computation

Screenshots:

Hi, there is another user who met a similar issue, which seems to be related to CUDA/PyTorch versions.  He has provided detailed solution on how he solved the issue.  Can you try this?
Meanwhile, would you please share your CUDA version, GPU type, GCC version, OS version?  Thanks.

Hi, the detail version info is as below:
CUDA version: 11.0
GPU type: Tesla V100 SXM2
GCC version: 7.5.0
OS version: Ubuntu 18.04

Thanks for your useful info about downgrading torch version.
It works after downgrading the torch version to 1.13.1 and setting the CUDA version to 11.7. I used the following command to reinstall pytorch.
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

Thanks for your useful info about downgrading torch version. It works after downgrading the torch version to 1.13.1 and setting the CUDA version to 11.7. I used the following command to reinstall pytorch. pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

Hi, can I ask you could you run import flash_attn_cuda
I have the same env as you, but the flash_attn_cuda can not be imported.

Thanks for your useful info about downgrading torch version. It works after downgrading the torch version to 1.13.1 and setting the CUDA version to 11.7. I used the following command to reinstall pytorch. pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

Hi, can I ask you could you run import flash_attn_cuda
I have the same env as you, but the flash_attn_cuda can not be imported.

Hi, I only used the demo provided by the author and did not modify any code. Btw, was flash-attn installed successfully? In additional, another user seems to have encountered a problem with flash-attn, and you can also refer to his solution.

Thanks for your useful info about downgrading torch version. It works after downgrading the torch version to 1.13.1 and setting the CUDA version to 11.7. I used the following command to reinstall pytorch. pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

Hi, can I ask you could you run import flash_attn_cuda
I have the same env as you, but the flash_attn_cuda can not be imported.

I'm encountering the same problem. How did you resolve this issue?


Sync model

alignment's get_quantization_config to get quantization config matching to the ones used in fine-tuning
reused the practices from alignment to set torch_dtype and model_kwargs

@sayakpaul more thoughts?

No all good.


Merge latest develop

You just want to update the 7.10 branch, right?

Right

I guess it looks fine, go ahead with merge, or would you like anyone to take a look at something?


[draft] MLFlow parameter and documentation

This PR addresses issue https://github.com/scailfin/madminer-workflow/issues/22, in combination with a similar PR created in the madminer-workflow-ml repository, which lives as a GIT sub-module inside this very own project.
NOTE: This PR should only be merged when:

The madminer-workflow-ml repository PR is merged.
An additional commit is included, changing the GIT sub-modules reference to the madminer-workflow-ml latest state.

The project GIT sub-module references has been updated ✅
This is safe to merge now.


Document that undefined values are not passed to CasperJS-land

I'd been puzzled about a Can't find variable: foo error until I realized JSON.stringify omits undefined values.
It'd be helpful to either document this in the wiki, or use a different serialization method to pass undefined values. (My use case was passing a command line option parsed by commander.js; the option can be undefined when no argument is supplied.)
Example:
Spooky = require 'spooky'

spooky = new Spooky
  child:
    command: './node_modules/casperjs/bin/casperjs'
  casper:
    logLevel: 'info'
    verbose: false
, (err) ->
  if err
    e = new Error 'Failed to initialize SpookyJS'
    e.details = err
    throw e

  spooky.on 'error', (e, stack) ->
    console.error(e)
    if (stack)
      console.log(stack)

  spooky.start 'http://example.com'
  spooky.then [{
    hello: undefined
  }, ->
    @echo hello
  ]
  spooky.run()

Outputs:
$ coffee hello.coffee
ReferenceError: Can't find variable: hello
[ { file: '', line: 3, function: '' },
  { file: '/path/to/node_modules/casperjs/modules/casper.js',
    line: 1553,
    function: 'runStep' },
  { file: '/path/to/node_modules/casperjs/modules/casper.js',
    line: 399,
    function: 'checkStep' } ]
{ [Error: Child terminated with non-zero exit code 1] details: { code: 1, signal: null } }

I've updated the wiki to call this issue out.


Missing NodeJsTools\TestAdapter dir

I'm try to see if we can add our Blazor unit testing tool to VS 2022.  The docs suggest to add a js file under
<VisualStudioFolder>\Common7\IDE\Extensions\Microsoft\NodeJsTools\TestAdapter\TestFrameworks\mocha\mocha.js
While I have found C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\Extensions
I cannot find any NodeJsTools dir.  I'm using VS 2022 Version 17.0.0 Preview 3.1.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 32738a06-835b-5bd5-7a6c-942c681aca97
Version Independent ID: e715d12c-a4aa-eec3-c05f-1ff6243d209b
Content: Unit testing JavaScript and TypeScript - Visual Studio (Windows)
Content Source: docs/javascript/unit-testing-javascript-with-visual-studio.md
Product: visual-studio-windows
Technology: vs-javascript
GitHub Login: @Mikejo5000
Microsoft Alias: mikejo

@ray440 -- Ray, thank you for your feedback.  Please consider these resources:

Visual Studio on Microsoft Q&A
Visual Studio MSDN Forum
Stack Overflow
Reddit

You are also welcome to use the Report a Problem tool to report the issue.
@Mikejo5000 -- Mike, please look into this potential documentation issue.

@ray440 Thanks for pointing this out. Did you install the NuGet package? https://www.nuget.org/packages/Microsoft.JavaScript.UnitTest/ It is possible the folder you mention does not exist until you install the Node.js workload in the Visual Studio installer, which is not mentioned in the doc and might be a gap. I'm not sure whether the Blazor scenario is supported.

@ray440 Did you try installing the NuGet package and Node.js tools?

Yeah, that fixed the issue.  Thanks.

@Mikejo5000 -- Mike, thank you for clarifying this issue.
@ray440 -- Ray, thank you for your followup.


Error handling external file: 'Max errors count reached.'

Hi guys,
I'm currently facing with this issue:
Error handling external file: 'Max errors count reached.'. File/External table name: 'https://xxx.dfs.core.windows.net/dynamics365-financeandoperations/eu-uat.sandbox.operations.dynamics.com/Tables/SupplyChain/SalesAndMarketing/WorksheetLine/SalesLine/SALESLINE_00001.csv'.
Please fix asap!
JP

Disclaimer: I'm not the maintainer of the project.
This is more an issue with synapse's not great error messaging.
But this has happened to be sometimes when:

The data length of the serverless view for the column is too small (defaults to 100, but maybe you need 1000).  Try updating the length of the nvarchar to 1000 or some large number just to test.
The data is not in the expected format in the columns that the data type matches.  Try changing all data columns to NVARCHAR(1000) just for testing.

Try selecting column by column to see which one is having issues.  That's what I've done in the past.

Disclaimer: I'm not the maintainer of the project.
This is more an issue with synapse's not great error messaging. But this has happened to be sometimes when:

The data length of the serverless view for the column is too small (defaults to 100, but maybe you need 1000).  Try updating the length of the nvarchar to 1000 or some large number just to test.
The data is not in the expected format in the columns that the data type matches.  Try changing all data columns to NVARCHAR(1000) just for testing.

Try selecting column by column to see which one is having issues. That's what I've done in the past.

Hi @gregorosaurus,
I already knew that! But columns length are automatically set by the Azure Function...
Do you know how to set default types length?

Disclaimer: I'm not the maintainer of the project.
This is more an issue with synapse's not great error messaging. But this has happened to be sometimes when:

The data length of the serverless view for the column is too small (defaults to 100, but maybe you need 1000).  Try updating the length of the nvarchar to 1000 or some large number just to test.
The data is not in the expected format in the columns that the data type matches.  Try changing all data columns to NVARCHAR(1000) just for testing.

Try selecting column by column to see which one is having issues. That's what I've done in the past.

Hi @gregorosaurus,
I already knew that! But columns length are automatically set by the Azure Function...
Do you know how to set default types length?

Disclaimer: I'm not the maintainer of the project.
I believe you can set it at the "Application settings" of your Azure Function App. Add a new setting:
Name: DefaultStringLength
Value: 1000 or some large number just to test.
Please let us know how it goes for you.
See https://github.com/microsoft/Dynamics-365-FastTrack-Implementation-Assets/blob/master/Analytics/CDMUtilSolution/deploycdmutil.md

Yup, the DefaultStringLength parameter is what you're looking for.
It can be set via the HTTP headers of calling the Azure function, if deployed that way, or using the app.config file to set it.

Just to explain - earlier version of the CDMUtil by default using CSV Parser version = 2.0 for performance reason. parser version 2.0 provides improved performance, however have limited error handling specially in such scenario like string length or data type mismatch. new version of cdmutil we still have parser version 2 as default however you can now switch to parser version 1 to identify such issues easily.

Closing the issue with comment - reopen if required.


Rename Socket Module to TCP

$subject.

Sabthar fixed with PR, https://github.com/ballerina-platform/module-ballerina-tcp/pull/82 .


How to recover a address with the result of verify_eip191?

I used siwe-rs to verify the signature. The [signer] is the result of the verification. How can I recover the signer to the original eth address?
The following method is used, but the result is incorrect:
    use sha3::{Digest, Keccak256};

    let siwe_msg = Message::from_str(string_message).unwrap();
    let sig = <[u8; 65]>::from(signature);

    println!("{}", "verify_eip191 start");
    let signer: Vec<u8> = siwe_msg.verify_eip191(&sig)?;

    let public_key = &signer[1..];
    let process_sha3 = Keccak256::default().chain_update(public_key).finalize();
    let result_slice = process_sha3.as_slice();
    let address_recovery = hex::encode(&result_slice[12..]);

    //address_recovery  is not the expected address

My demo is here with only a few codes:   https://github.com/anylots/siweapp

The public key returned by verify_eip191 is compressed SEC1-encoded (https://docs.rs/k256/latest/k256/ecdsa/struct.VerifyingKey.html#method.to_bytes), but addresses are the hash of the uncompressed SEC1-encoded point.
But, if I may, verify_eip191 already verifies the signer address so you shouldn't really need to rebuild it as you can get it from the Message struct. But in any case, you can copy the code in verify_eip191 and use k256 to write something along these lines:
    use k256::elliptic_curve::sec1::ToEncodedPoint;
    let public_key = &k256::ecdsa::VerifyingKey::from_sec1_bytes(&signer).unwrap();
    let public_key = public_key.to_encoded_point(false);
    let process_sha3 = Keccak256::default()
        .chain_update(&public_key.as_bytes()[1..])
        .finalize();
    let result_slice = process_sha3.as_slice();
    let address_recovery = hex::encode(&result_slice[12..]);