CodeHilite: Support Pygments options

Allows specifying any Pygments option as suggested in https://github.com/Python-Markdown/markdown/issues/334. This also allows using the Pygments inline option for line numbers (#652).
Example:
In [2]: text = """
   ...:     :::php
   ...:     print "Hello world";
   ...: """

In [3]: markdown(text,
   ...:          extensions=['codehilite'],
   ...:          extension_configs={'codehilite': {
   ...:              'lexer_options': {'startinline': True},
   ...:              'formatter_options': {'wrapcode': True, 'linenos': 'inline'}}})
Out[3]: '

1 print "Hello world";\n

'

There's some messiness where the Pygments options overlap with the existing options. I marked the ones that apply only to Pygments as deprecated, but some, like linenums and css_class are used when Pygments is disabled.

I like this. It does not actually close #334 though, as this code does not allow one to set the options in a document (only via API). But I guess that can be the next step.
Also, can you make it issue some DeprecationWarnings when the deprecated options are used?

this code does not allow one to set the options in a document

See #816 which has started the work on that. Interestingly, this makes that more complicated. As this breaks the arguments up into 2 dicts to differentiate between lever and formatter options. Previously, we didn't care about that distinction which made specifying options int he document easier. I'll have to think about that.

I wasn't thinking about setting the options in the document. I'll change it to use a single dict if requested. Then any key=value pair specified in the document that isn't a valid codehilite option could be passed to Pygments.

I'm closing this in favor of #816. However, @pauloue your work was not in vain as it helped me to see how to best approach the matter.


Recaptured Image Forensic

Dear author, congratulate your work !
Have you try applying this method for recaptured image detection problem ? Can I train this model with recapture image dataset ? Thank you for your sharing.

Thanks.
This is the link to the published version of the work. https://www.sciencedirect.com/science/article/abs/pii/S2214212622001247
We have not yet tried this method for recaptured image detection, but we hope this can also be utilized to train on recapture images.


Add Sunstone option to avoid importing VM Template from the marketplace

Description
Sometimes (like in vCenter envs) is preferable not to import the VM Template when importing an app from the marketplace
Interface Changes
Sunstone import flow from marketplace should include an option that works like the "--no" param added here:
44beefa77e3303189273d6b6d961643a554bfa59
Additional Context
Add any other context or screenshots about the feature request here. Or any other alternative you have considered to addressed this new feature.

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

PRs to merge:

one: https://github.com/OpenNebula/one-ee/pull/736
doc: https://github.com/OpenNebula/docs/pull/1436
dev: pending


DRILL-2304: Case sensitivity - system and session options are case sensitive

@jaltekruse please review.

Addressed review comments mentioned on 313c8e3 f33654c

@sudheeshkatkam I reviewed this change while I was looking at 3497 in #98. Other than the few small comments over there this looks good.

Passes all unit and regression tests.

+1


Is it possible to replace image of a layer of psd file with another image?

wanted to replace a particular image from a layer of psd file with another image. I am unable to do the same. let me if i can do the same or not?

I meet the same problem, any idea? Thanks!

it not support now


Modsecurity audit log (JSON) - messages field structure

Describe the bug
Server version: Apache/2.4.57 (Win64)
Server built:   Oct 11 2023 09:54:05
ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/), OWASP_CRS/3.1.0
Problem:
msg.audit_data.messages field is not structured as JSON inside.
Expected format: https://gist.github.com/zimmerle/b22660ab33bd6df444fd
Current format:
    "msg.audit_data.messages": [       "Warning. Pattern match \"^5\\\\d{2}$\" at RESPONSE_STATUS. [file \"F:/WAF/mod_security/owasp-modsecurity-crs-3.2-dev/rules/RESPONSE-950-DATA-LEAKAGES.conf\"] [line \"87\"] [id \"950100\"] [msg \"The Application Returned a 500-Level Status Code\"] [data \"Matched Data: 502 found within RESPONSE_STATUS: 502\"] [severity \"ERROR\"] [ver \"OWASP_CRS/3.1.0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-disclosure\"] [tag \"WASCTC/WASC-13\"] [tag \"OWASP_TOP_10/A6\"] [tag \"PCI/6.5.6\"] [tag \"paranoia-level/2\"]",       "Access denied with code 403 (phase 4). Operator GE matched 4 at TX:outbound_anomaly_score. [file \"F:/WAF/mod_security/owasp-modsecurity-crs-3.2-dev/rules/RESPONSE-959-BLOCKING-EVALUATION.conf\"] [line \"69\"] [id \"959100\"] [msg \"Outbound Anomaly Score Exceeded (Total Score: 4)\"] [tag \"anomaly-evaluation\"]",       "Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file \"F:/WAF/mod_security/owasp-modsecurity-crs-3.2-dev/rules/RESPONSE-980-CORRELATION.conf\"] [line \"96\"] [id \"980140\"] [msg \"Outbound Anomaly Score Exceeded (score 4): The Application Returned a 500-Level Status Code; individual paranoia level scores: 0, 4, 0, 0\"] [tag \"event-correlation\"]"     ],
Is this related to Apache/Modsec versions/OS?

Hello @donatasiv ,
The 'messages' name does have an associated value that is valid JSON. Moreover, that value (the array value) will have multiple json objects within it if there are multiple messages. And each of those individual array items will be a valid json object.
But I guess you're really asking why each of those individual items within the messages array appears as a simple string rather than a more complex json object.
In ModSecurity v3, the format for each of the 'messages' items is subdivided in a manner similar to the suggestion that you linked to from 2015.
ModSecurity v2 is a bit different. To obtain most of the same information in an easily JSON-parsable manner, you could activate part 'K' in your audit log. (My guess is that existence of this alternative source in ModSecurity v2 is why the items in the 'messages' array are each provided as a simple string.)


disable logs

Could we have an option to disable the logs?
They are a bit expensive in AWS lambda
Thank you

@girotomas  now there won't be any logs as default , if you need them , you can set process.env.DEBUG=true, then all logs will appear, include the puppeteer logs


Make blobToText replaceable in Typescript template

I found difficult to replace the functionality implemented in blobToText function.
Because File API (FileReader) is not present in server side code (Angular SSR -> Universal), the calls are always failing with error message "ReferenceError: FileReader is not defined".
Not easy to avoid this issue, because the calls and the function itself hard coded.
It would be great if it can be more flexible somehow or avoid using DOM related functions like File API.

You can replace this particular template (File.Utilities.liquid): https://github.com/RSuter/NSwag/wiki/Templates
What wouldbe a good always working replacement?

Thank you. I'll give it a try.

We solved this by manually editing the generated code to replace responseType: "blob" with responseType: "json" in the options object. Then we could just cast the response body to our dto model and delete the blobToText function.
It would have been great to have a select in the NSwag UI to use json instead of blob in HttpClient like this, would that be possible in the future?

So ideally it would use "json" for JSON responses and "blob" for file/binary responses, correct?

Exactly :-)

@Chreekar thanks for suggesting the responseType change.
For now I did Creekar's change with responseType: json but to prevent changing everywhere in the swagger file I kept blobToText with a minor change. It's not optimal but it works with Angular SSR.
Would be nice with a permanent fix though.
function blobToText(blob: any): Observable<string> {
    return new Observable<string>((observer: any) => {
        observer.next(JSON.stringify(blob));
        observer.complete();
    });
}

@Chreekar
I think that its possible to work with providers for customization instead of manually change the auto generated output.
for instance:

blobToTextProvider which is injectable and the developer can assign the ts class in the UI.

httpRequestOptionsProvider

I haven't understood the concept of providers, is this something I just change in the UI or do I need to create them in code somehow?

Took another go at this. I downloaded AngularClient.liquid and File.Utilities.liquid from https://github.com/RicoSuter/NSwag/tree/master/src/NSwag.CodeGeneration.TypeScript/Templates into a folder I pointed out in the TemplateDirectory setting. Changed the response type and blobToText in them, and now when I generate code I won't have to change anything afterwards.

Took another go at this. I downloaded AngularClient.liquid and File.Utilities.liquid from https://github.com/RicoSuter/NSwag/tree/master/src/NSwag.CodeGeneration.TypeScript/Templates into a folder I pointed out in the TemplateDirectory setting. Changed the response type and blobToText in them, and now when I generate code I won't have to change anything afterwards.

I have got the same problem with the endpoints generated by nswag. can you please let me know where exactly I need to put those file into? The nswag folder that I have got has a service.config,nswag and refresh.bat.

Took another go at this. I downloaded AngularClient.liquid and File.Utilities.liquid from https://github.com/RicoSuter/NSwag/tree/master/src/NSwag.CodeGeneration.TypeScript/Templates into a folder I pointed out in the TemplateDirectory setting. Changed the response type and blobToText in them, and now when I generate code I won't have to change anything afterwards.

Any more info on what you did to fix the issue?

I changed it like this:
AngularClient.liquid
{%-    if operation.IsFile -%}
            responseType: "blob",
{%-    else -%}
            responseType: "json",
{%-    endif -%}

This is important, so you can still download files from the API.
File.Utilities.liquid
{% if Framework.IsAngular -%}
function blobToText(blob: any): Observable<string> {
    return new Observable<string>((observer: any) => {
        observer.next(JSON.stringify(blob));
        observer.complete();
    });
}

I replaced the following to ensure my error parsing worked:
AngularClient.liquid
Before:
const responseBlob =
          response instanceof HttpResponse ? response.body :
          (response as any).error instanceof Blob ? (response as any).error : undefined;

After:
const responseBlob =
          response instanceof HttpResponse ? response.body : (response as any).error;


Generate tags file asynchronously

I have autocmd TextChanged,InsertLeave nested silent! update, so whenever I change anything the editor freezes for a few hundred milliseconds while it updates the tags file. This is getting pretty annoying...

This raises the question: @ludovicchabant do you plan to use job control in vim8, soon?

Or Neovim job control? I'm using Neovim...

@ludovicchabant Doesn't use neovim, but if he does the vim8 job thing it won't take long to adapt it for neovim.

Hi there,
The tags file is already being updated asynchronously by spawning a script that takes care of the heavy lifting. So the only thing you would save by using the job API from Vim8/Neovim would be the ~50 lines of Vimscript that do the preparation for running that script.
Therefore I'm not sure exactly if using background jobs would do a lot of difference -- I don't know if they have overhead or whatever. I do have it on my TODO list to try it out, if only to remove the need for creating a .lock file, but that's low priority. And actually, I imagine most of the freezing you notice must be Gutentags checking for the existing of that .lock file on disk.

the only thing you would save by using the job API from Vim8/Neovim would be the ~50 lines of Vimscript that do the preparation for running that script.

Using job-control would avoid accidentally leaving the temp/marker files (or creating them in the first place).

I don't know if they have overhead or whatever.

Overhead...?

It seems like there's a non-trivial amount of overhead (~400ms) to executing an external command in vim without job control. If you do :!echo hi there's a very noticeable pause before you can use the editor again. @justinmk probably knows more than me about why this is the case.

Depends on how it's invoked, but at least one benefit of job_start() is that if you pass the args as a List then it is run "directly", i.e. without a shell. Whereas :! and system() always use the shell (except in nvim if system() args are passed as a List--analogous to jobstart/job_start).
400ms seems high though.

I made a 4-second video to illustrate what's going on: https://www.youtube.com/watch?v=NSAjNQpCX_U
I typed :call system('echo hi') and immediately hit i and started pressing keys. By the time the command has finished, I've inputted 6 characters. I would expect it to not take a noticeable amount of time at all. This is with nvim -u NONE, the same thing happens in vim -u NONE.

@tbodt Could be your bashrc/profile/etc. Try a different shell.
See also https://github.com/vim/vim/issues/1089 and https://groups.google.com/d/msg/vim_dev/PmNEtGLv6J0/xj80VAteAgAJ .
Even so, it's still an argument in favor of job control IMO.

Hey, that makes sense. My shell is fish.
~Theodore

On Mar 12, 2017, at 5:17 PM, Justin M. Keyes<EMAIL_ADDRESS>wrote:
@tbodt Could be your bashrc/profile/etc. Try a different shell.
See also vim/vim#1089 and https://groups.google.com/d/msg/vim_dev/PmNEtGLv6J0/xj80VAteAgAJ .
Even so, it's still an argument in favor of job control IMO.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I'm using Fish also, but I can absolutely not "miss out" keys like you did in the video... even though I'm on an 8-years old Macbook Pro... My Fish config file is pretty minimal though.
Either way yes, using jobs will certainly be a nice improvement. If anybody has time to do it, please go for it :)  Otherwise I'll do it at some point but I have no ETA as to when that would be.

 I spend a bit of time the other day reducing the startup time from 500 ms to 50 ms. It's really helped a lot.

Okay people, it's a long week-end so I figured I'd hack around with the Vim8 jobs! So the latest code updates include:

Using job_start/jobstart (Vim8 vs Neovim... I wish Neovim would just deprecate its API at this point) for launching the tags generation. No changes here from before since it was already launching a background process before, albeit using the shell.
 The tag generation script still creates/deletes the lock file though, because, I don't know, why not? But the Vim plugin doesn't care anymore at this point.
As another nice side-effect, if you added gutentags#statusline to your statusline, you can now use the GutentagsUpdating and GutentagsUpdated user events to force-refresh it and get proper real-time information about when tags are generating or not. See the changes to the documentation for an example with lightline.
The requirements for Gutentags have been bumped to Vim 8 or Neovim, as I didn't want to support 2 code paths (with/without jobs). If you still need to use Vim 7, you can checkout the vim7 branch, which I'll support for a while.

I have tested the new code with a good combination of use cases, but bugs will probably creep up here and there so as always feel free to open new issues.

The tag generation script still creates/deletes the lock file though, because, I don't know, why not?

Why not: because it's annoying to see it appear in git status.

I wish Neovim would just deprecate its API at this point

A lot of thought was put into the Neovim jobs API to make it powerful yet easy to use.  If you don't want to think about the differences, you can inline https://github.com/prabirshrestha/async.vim which is a thin abstraction around Vim8/Nvim jobs APIs.

Ah thanks for the pointer to async.vim, this might indeed come in handy, although I already wrote my own ad-hoc thin wrapper so we'll see if it can hold up for the limited use we have of it.
About the lock file, you probably already have tags in your global .gitignore, so you could also add tags.lock, but I see you point and I'll remove the lock file in the future.

About the lock file, you probably already have tags in your global .gitignore, so you could also add tags.lock, but I see your point and I'll remove the lock file in the future.

This is what I'm currently doing in my global gitignore file.


Peek code review

Summary
Describe the goal of this PR. Mention any related Issue numbers.
Requirements

[ ] I've read and understood the Contributing Guidelines and have done my best effort to follow them.
[ ] I've read and agree to the Code of Conduct.

I opened this in preparation for a talk tomorrow!


Check rule id error key

Description
With this commits, Smart Proxy will be able to check if the combination of rule ID and error key provided by a customer are present and consistent with the content in the content service.
Now, if some the the enable/disable/voting endpoints are provided with a rule ID + error Key that are not consistent, it will return a 404.
Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
Unit tests (no changes in the code)

Testing steps
Regular UTs
Checklist

[x] make before_commit passes
[ ] updated documentation wherever necessary
[x] added or modified tests if necessary
[x] updated schemas and validators in insights-data-schemas in case of input/output change

TestHTTPServer_GroupsEndpoint is failing

Surprisingly, it works fine locally. I'm trying to re-start the job

Codecov Report
Base: 60.08% // Head: 60.24% // Increases project coverage by +0.16% :tada:

Coverage data is based on head (6e46a5e) compared to base (3ca4748).
Patch coverage: 77.27% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #929      +/-   ##
==========================================
+ Coverage   60.08%   60.24%   +0.16%
==========================================
  Files          22       23       +1
  Lines        3164     3182      +18
==========================================
+ Hits         1901     1917      +16
  Misses       1048     1048
- Partials      215      217       +2

Impacted Files
Coverage Δ

server/request_modifiers.go
66.66% <66.66%> (ø)

server/router_utils.go
70.58% <75.00%> (ø)

server/endpoints_v1.go
100.00% <100.00%> (ø)

content/content.go
77.77% <0.00%> (+0.82%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


bump kotlin

This PR bumbs to Kotlin 1.0.1-2. In the meantime Kotlin 1.0.2 was released.

@hastebrot yeah, sigh... :)

Thanks. I'm going to deploy a new version to Maven Central probably this weekend.

woo-hoo!

<kotlin.version>1.0.2</kotlin.version>
<dokka.version>0.9.7</dokka.version>

Hmm, Kotlin 1.0.2 also needs the new Dokka version 0.9.8.

Together with Kotlin 1.0.2, we’re releasing a new version of Dokka, the Kotlin documentation generation tool. If you’re using Dokka in your project, you need to upgrade Dokka together with Kotlin, because older Dokka versions are incompatible with Kotlin 1.0.2.

via: https://blog.jetbrains.com/kotlin/2016/05/kotlin-1-0-2-is-here/

whoops, we should probably add dokka generation to travis checks

Yep you right. Whole API doc is still a work in progress. We should bump the dokka version and publish the docs somewhere when it's in a solid state.
Unfortunately I had not much time for Kotlin dev lately but hopefully this will change soon. However I'm happily accept pull requests. 👯

New version 0.4.0 is available in Maven Central.


Fix GoogleService

Due to Google changes it's not for sure they will return the exchange in the expected format, but formatted according to a locale.

Trigger Travis

Thanks @driade ! This service is really a pain

Hi, thanks. The code is working though it does intermittently. I'll try to debug it's because of new some new format by Google.


Fix some holidays in méxico

I'm fixing some wrong dates with the Mexican holidays, also I'm fixing the failing tests

Hi there, we've already merged a Mexican PR. Could you verify if everything is fine? If not, please open a new PR with fixes! Thanks!

This a fix for that Mexican holidays as mentioned here
https://github.com/spatie/holidays/pull/23

What exactly did you fix? Also, could you make a PR which is up to date to the latest changes of this branch and fix merge conflicts? Thanks


ID field disappears when using eager

Model.query().select("actions.*")
returns correctly :
[{        id: 1        name: "test" }, {        id: 2        name: "test 2" }]
But when I use eager:
Model.query().select('actions.*').eager('actionMembers')
I get:
[{     name: 'test',     actionMembers: [{         id: 1,         related_data: 'abc'     }] }, {     name: 'test 2',     actionMembers: [{         id: 2,         related_data: 'xyz'     }] }]
In this example, field id_pet is present in both main table and relation table, and when using eagers it disappears from main table's result.
 }

You still get the id. The actions.* selection still gets you the id. I have no idea why action_id is there twice, but without knowing what you are actually doing, there's no way for me to help you. I'll fix any bug you can prove by providing a reproduction.

When I do .select("actions.*") and use eager the id column won't show up in the result.


invalid syntax is not recognized

Hi, just tried recript for the first time and found a bug.
I don't exactly when this happens, but I stripped everything until i got this "minimal example code":
let l = Some(list{1, 2, 3}) |> Obj.magic
module M = {
  switch l {
  | None => list{}
  | Some(l) => l["prop"]
  }
}
}from now on all syntax is "valid",
it compiles and will be ignored in js/ removed on format

link to minimal example in the rescript playground
The Compiler does not have any complains about this.
Of course that code would be an error on runtime and the compiler is tricked here,
but its also syntactically wrong and should give maybe the same message as it does when you remove |> Obj.magic and ["prop"] in the code above. link to example with the error i would expect

@farukg thanks for the bug report! Fixed.


ASSETS-88928 : Update Deliverables Page Fields for Phase 2

https://experience.adobe.com/#/@wfadoberm/so:adoberm-Production/workfront/task/66798f2000163878b4962d8e8221d44e/overview

Test URLs
Before
https://rc--adobe-gmo--hlxsites.hlx.page/qa/program-details?programName=Express_Express for Business Launch&programID=66466a56027b70c2ee6edd2f027cc68a
After
https://assets-88928--adobe-gmo--hlxsites.hlx.page/qa/program-details?programName=Express_Express for Business Launch&programID=66466a56027b70c2ee6edd2f027cc68a

Closing PR, the branch to merge to now is assets-72024
I was unable to update the PR to merge to the branch assets-72024
So I created a new PR https://github.com/hlxsites/adobe-gmo/pull/143


Add storage protections on azure

What this PR does / why we need it
Replaces https://github.com/giantswarm/object-storage-operator/pull/170 towards https://github.com/giantswarm/giantswarm/issues/31006
This PR enables soft delete at the container and blob level so we can restore them up to 7 days after deletion.
The reasons I think this is a good idea is that:

counting the actual number of blobs in the storage account is way harder that I thought it would be and requires us to get some extra permissions.
we cannot use locks on the storage account due to a lack of permissions
and we cannot use immutable policies (that should prevent container deletion if not empty) as this would prevent the loki and mimir compactor to run as the policies would prevent the compactor to edit files (only appends and creations are supported 😮💨 )

Checklist

[x] Update changelog in CHANGELOG.md.

Edit:
Locks do not work because we need a role we do not have

@hervenicol do you think it's a good alternative to the bucket délétion issues?

I understand that now, when a bucket is deleted, we have a delay when we can restore it and its data.
But what happens then if I try to re-create a new bucket with the same name? Does it create a new bucket, or does it restore the previous bucket?

That's an interesting question. I'm also not sure that we're protected if we delete the storage account anyway :(

So if we delete the storage account we're losing everything :(

@marieroque is working on another solution

Closing, @marieroque is working on another fix


game over message at the begining

When i open the html file,the browser says game over.I think there should be a start button or something like that to start the game.There's a broken file symbol at the top left corner of the canvas.

@Redarcher9 Thanks for letting us know about your issue. I haven't experienced this problem and i am unable to replicate it. Does this happen if you use a different browser?

Yes it happens with chromium too. Does it have anything to do with Linux? Do I need to install dependencies to run this?

@Redarcher9 Are you using Ubuntu? Can you try it with Firefox?

This works fine on windows i just tested it.
Doesnt works on ubuntu,the screenshot attached to the first comment was taken from firefox browser.

@Redarcher9 Thanks for testing it on windows. It must be a file association issue of some sort. Let me look in to this.


suggest：develop a script to only install board client environment

In GitLab by @liyanq528 on Jun 20, 2019, 10:31
when board client is changed,we can use it to set env.

In GitLab by @guyingyan on Nov 8, 2019, 08:05
system need installed k8s master and board on the same host, so if reinstall the OS the k8s cluster cannot be used.

In GitLab by @cuiaq123 on Nov 8, 2019, 08:05
请罗天评估是否有必要。
这个问题在需要重新换一个board客户端的场景下会用到。

In GitLab by @sokril on Nov 8, 2019, 08:33
assigned to @luotian-github

In GitLab by @luotian-github on Dec 20, 2019, 06:41
assigned to @cuiaq123 and unassigned @luotian-github

In GitLab by @luotian-github on Dec 20, 2019, 06:42
请再详细描述一下，应用场景，谢谢

In GitLab by @sokril on Dec 26, 2019, 07:11
changed the description

In GitLab by @cuiaq123 on Jan 2, 2020, 05:42
assigned to @luotian-github and unassigned @cuiaq123

In GitLab by @cuiaq123 on Jan 2, 2020, 05:42
比如原来的board的机器坏掉了，想重新用一台机器部署board，并继续使用原来的K8S集群，这种情况。

In GitLab by @luotian-github on Mar 4, 2020, 09:21
assigned to @cuiaq123 and unassigned @luotian-github

In GitLab by @luotian-github on Mar 4, 2020, 09:33
assigned to @guyingyan and unassigned @cuiaq123

In GitLab by @guyingyan on Mar 23, 2020, 07:12
单独安装board可以，但需要在hosts文件中提供master的IP，在安装board环境时，需要证书

In GitLab by @guyingyan on Mar 31, 2020, 06:52
closed

In GitLab by @guyingyan on Mar 31, 2020, 06:52
MR 32 on board-install repo


Add support for default values for properties from objecttypes.xml

Object types contain the default values for properties. When the default property value is used for the tile it is not specified in the tileset file (only the type of the tyle). So in order to determine the default values of the tile properties objecttypes.xml must be read.
It would be nice if TiledReader would automatically merge with the default values from objecttypes for the tile type. This would eliminate needing to manually parse objecttypes.xml by users of the TiledReader library.

I haven't heard of objecttypes.xml before. Where is it located? Does the Tiled documentation mention it anywhere?

I can't seem to find documentation on the storage format actually. But functionally it's documented here:

https://doc.mapeditor.org/en/stable/manual/custom-properties/#typed-tiles.
https://doc.mapeditor.org/en/stable/manual/projects/#what-s-in-a-project

Also just saw this issue on the tiled repo which is someone basically having the same problem as I have. https://github.com/bjorn/tiled/issues/2872.
PS: I updated my original post to indicate better what I would expect TiledReader to do with objecttypes.xml.

This situation looks tricky. I'm not clear on how a parser is even supposed to find the appropriate object types file based on the information in a TMX or TSX file. To support this, would we need to extend TiledReader to read project files too?
As refox1 said, this feature seems badly designed. Besides, isn't implementing "object types" what template files are for?

Object types are more generic types. In the future it will be used for all objects (maps/tilesets also). In my map design using meta tiles I heavily depend on that feature for typed tiles. I don't even use the object layer atm.
A simple solution would be to allow the user of TiledReader to just read object types as separate object. And configure the map reader to use the object types object. At this time moment getMap() is a static method. Maybe make the TiledReader class an instantiable class that can be configured to use an object types file. It is then up to the user to specify the file.
Then when properties are read merge them with the default properties.

Maybe make the TiledReader class an instantiable class that can be configured to use an object types file. It is then up to the user to specify the file.

This sounds sensible - and as it happens, there's already another proposed change to TiledReader that involves making the TiledReader class instantiable. I should have time to add object types support in this form sometime soon.

Cool!

I've added the object types support. Let me know if everything works for your purposes.

Hi, I tested it. But got this error when reading objecttypes.xml.
Exception in thread "main" java.lang.RuntimeException: javax.xml.stream.XMLStreamException: (line 4, column 48): Missing required tag attribute (default) 	at org.tiledreader.TiledReader.readObjectTypes(TiledReader.java:562)
Not all properties in my objecttypes.xml specify a default value. I think that may be the problem.

I see. I've changed the parser so any object type property that doesn't specify a default value gets the standard default value for properties of that type (e.g. "" for a string).

That part works. But I'm still not seeing the properties with default value on the tiles. I think that this maybe only works for the object layer now (which I don't use)?

Sorry about that. I didn't read the Tiled documentation on the subject closely enough to realize that the tiles themselves also need to inherit the custom properties from their types.
I've made another change that hopefully actually got property inheritance working the way Tiled does it, for both objects and tiles. Does it work as expected for you now?

It works now! Thanks!

Excellent. I'll close the issue now.


build(devs-dep): upgrade js-yaml@4

Summary

Upgrade js-yaml and @types/js-yaml to v4
Test plan

Green CI
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information
N.A.

Pull Request Test Coverage Report for Build 485043079

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 93.654%

Totals

Change from base Build 483660943:
0.0%

Covered Lines:
948

Relevant Lines:
978

💛  - Coveralls

Pull Request Test Coverage Report for Build 485043079

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 93.654%

Totals

Change from base Build 483660943:
0.0%

Covered Lines:
948

Relevant Lines:
978

💛  - Coveralls


Issue when parsing an email message

When parsing the attached email message, I received the following messages:
More than one match found for (?:by\s+(?P<by>.+?)(?:\s*[(]?envelope-from|\s*[(]?envelope-sender|\s+from|\s+with(?! cipher)|\s+id|\s+for|\s+via|;)) in<EMAIL_ADDRESS>by vger.kernel.org via listexpand id S230171AbhFJIbY ORCPT<EMAIL_ADDRESS>; Thu, 10 Jun 2021 04:31:24 -0400

More than one match found for (?:by\s+(?P<by>.+?)(?:\s*[(]?envelope-from|\s*[(]?envelope-sender|\s+from|\s+with(?! cipher)|\s+id|\s+for|\s+via|;)) in<EMAIL_ADDRESS>by vger.kernel.org via listexpand id S230171AbhFJIbY ORCPT<EMAIL_ADDRESS>; Thu, 10 Jun 2021 04:31:24 -0400

This email message came from a spam archive ( http://untroubled.org/spam/ ) -- http://untroubled.org/spam/2021-06.7z
1622562157.26369_31.txt

Attached wrong message...here is the correct one.
1623336582.1677509_17.txt

It's not an issue. There isn't a standard to parse the received headers, so I print this message to tuning this regex. Thanks for submission.


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v3.5 to before v4.19.209",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ba4dc61fe8c545a5d6a68b63616776556b771f51",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=829f13d6079cf7a2465522f39acb43033e9b320d"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "ba4dc61fe8c545a5d6a68b63616776556b771f51",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "829f13d6079cf7a2465522f39acb43033e9b320d",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "ipack: ipoctal: fix stack information leak\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.19.209 by commit 829f13d6079cf7a2465522f39acb43033e9b320d, it was introduced in version v3.5 by commit ba4dc61fe8c545a5d6a68b63616776556b771f51. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1001555


Problem about Running Row-Less Universal Schema

Hi,
I am trying to run the row-less version with the "attention-both-lstm" configuration. I used the demo training data by changing the training variables in script from
export LOG_ROOT="${TH_RELEX_ROOT}/models/fb15k/filtered/$NAME" export TRAIN_FILE_ROOT="$TH_RELEX_ROOT/data/FB15K-237/fix_padding_no-min//" export TRAIN_FILE="filtered-train-relations.torch"
to
export LOG_ROOT="${TH_RELEX_ROOT}/models/$NAME" export TRAIN_FILE_ROOT="${TH_RELEX_ROOT}/data/" export TRAIN_FILE="train-mtx.torch"
After running the command ./bin/train/train-model.sh 0 bin/train/configs/rowless/fb15k-237/attention-both-lstm, I got the following error.

I also tried other lstm-related configurations like "max-relation-both-lstm", and they all have the same error while non-lstm configurations like "attention" works fine.
Any idea? Really appreciate it if you can help!
@patverga @davidBelanger @strubell

I think the issue is that the demo data only works with the master branch version of the code and not the rowless version because it has a slightly different format. You can try using the processing script referenced here https://github.com/patverga/torch-relation-extraction/tree/rowless-updates#fb15k-237-and-rowless-models and see if that does the trick.


gtsam: 4.2.0-1 in 'humble/distribution.yaml' [bloom]

Increasing version of package(s) in repository gtsam to 4.2.0-1:

upstream repository: https://github.com/borglab/gtsam.git
release repository: https://github.com/ros2-gbp/gtsam-release.git
distro file: humble/distribution.yaml
bloom version: 0.11.2
previous version for package: null

Holding for Humble sync


Realm async writes really works

I have created a singleton class having following method to write but it crashes at times because of incorrect thread access:
func save<T:Object>(_ realmObject:T) {
    let backgroundQueue = DispatchQueue(label: ".realm", qos: .background)
    backgroundQueue.async {
        let realm = try! Realm()
        try! realm.write {
            realm.add(realmObject)
        }
    }
}

I am getting crash always.

Hi @santhoshs5. Would you be able to provide us with a code snippet of where you call func save<T:Object>(_ realmObject:T) as well as the stack trace for the crash. I'm looking forward to seeing this as we'll be able to better assist you.

@santhoshs5, in your code snippet you're passing an object across threads. Realm objects can only be access on the thread on which they were created. See the documentation's section on passing instances across threads for more information.

@bdash  I considered your input and changed the method to following.
Now I am getting this exception
'RLMException', reason: 'Cannot construct reference to unmanaged object, which can be passed across threads directly'
func save<T:Object>(_ realmObject:T) {
let objectRef = ThreadSafeReference(to: realmObject)
let backgroundQueue = DispatchQueue(label: ".realm", qos: .background)
backgroundQueue.async {
let realm = try! Realm()
guard let realmObject = realm.resolve(objectRef) else {
return // person was deleted
}
try! realm.write {
realm.add(realmObject)
}
}
}

@istx25
I am calling this method as following.
    RealmHelper.shared().save(userInfo!)

Here userInfo object realm object.

As the exception says, unmanaged objects (i.e., objects not associated with a Realm) should be passed directly rather than via a thread safe reference. The fact you're hitting both exceptions from the same code suggests you're passing both unmanaged and managed objects, so you'll need to accommodate that.

@bdash
I do get that to clear the confusion managed object means Realm objects which are derived from Object.
is it good practice to have singleton to manage realm.
Could you explain me more from my code what's wrong. as i am newbie to realm.
One more thing I am using this method for reading from realm.
func fetch<T:Object>(type:T.Type,predicate:NSPredicate?) throws -> Results{
let results:Results!
do{
let realm = try Realm()
if let predicate = predicate
{
results = realm.objects(T.self).filter(predicate)
}
else
{
results = realm.objects(T.self)
}
}catch let error{
throw error
}
return results
}
I am using this method as following
        let results = try RealmHelper.shared().fetch(type: UserInfo.self, predicate: nil)
        setting the results to another variable present in singleton , I am getting crash when i read the object from another thread. What could be the reason

managed object means Realm objects which are derived from Object.

This is not correct.
If you create an object from scratch (for example, let myObject = SomeObject()), that object starts out as unmanaged. You can pass it between threads like any normal object.
Once you add that object to a Realm, it becomes managed. You can't pass it between threads anymore without using the thread safe reference API. If you retrieve an object from a Realm, that object also starts out managed.
Please read our documentation for more information about using Realms in an application with multiple threads or queues.

I have the same problem. How can I check if an RealmObject is managed or unmanaged Object？

The Realm property is nil for unmanaged objects.


Use inline options in typescript

For typescript
I should declare my own module like this.
declare module 'worker-loader!*' {
  class WebpackWorker extends Worker {
    constructor();
  }
  export default WebpackWorker;
}

For inline worker
I should set the inline option, but if module declare start with worker-loader!, it like to override my webpack config with no options after worker-loader
The result
In typescript, inline option in webpack config not work, I mush set inline option on the path after loader
Expected
A solution that, typescript can detect the WebpackWorker and I can config the worker-loader in webpack config not write a query string when importing.

typescript declaration doesn;t change webpack options, you can pass inline options using query string !!webpack-loader?inline=fallback!./my.worker.js

typescript declarations doesn't change webpack options, you can pass the 'inline' option using query string !!webpack-loader?inline=fallback!./my.worker.js

because I declared worker-loader!*, so TS can find WebpackWorker only if I import like import Worker from 'worker-loader!myWorker.ts', and When I import like that, webpack config not work. So every time I want to change the option, I have to modify the declare and the import path.

I don't understand you, please create simple reproducible test repo and describe problems in readme

I don't understand you, please create simple reproducible test repo and describe problems in readme

Meet this problem too. We are using the typescript example from the doc, while if we follow the procedure, the options in webpack seem invalid. For instance, I want inline option but it didn't work as expected.
The webpack config is as follow.
      {
        test: /worker\.ts$/,
        loader: 'worker-loader',
        options: {
          filename: 'JayWorker.js',
          inline: 'no-fallback',
        }
      },
      {
        test: /\.tsx?$/,
        use: [
          { loader: 'ts-loader' }
        ],
        exclude: [
          path.resolve(__dirname, 'node_modules'),
        ],
      }

It will be so nice if you can give the doc more details about Typescript development? Thank a lot!

@mikechabot i followed your suggestions and worked perfectly, if someone have the same issue take a look at this https://github.com/webarkit/ARnft/pull/167#issuecomment-821552693


[skia] Add restrictions to feature harfbuzz

Fixes https://github.com/microsoft/vcpkg/issues/36928
Fix based on comments.

[X] Changes comply with the maintainer guide.
[ ] SHA512s are updated for each updated download.
[ ] The "supports" clause reflects platforms that may be fixed by this new version.
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[X] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[X] Only one version is added to each modified port's versions file.

Obsolete: #36963.

Closing this PR, as this is already fixed in #36963


[5.6] Added test helper to assert that a Job has been queued with a Chain

This PR adds a new test helper when testing Jobs and allows to check if a Job was pushed with a specific Chain. The expectedChain can be an Array of Classes or Array of Objects.
This PR is a continuation of #23381 and addresses @taylorotwell concerns on checking only for the first Job. This PR tries to fix that situation.
This PR is also a continuation of #23508 in regards of pushing the change to branch 5.6 instead of master as @taylorotwell suggested
Cheers.

Extracted a few methods out and did some formatting. Thanks.


Is there a way to speedup the evaluation process ?

Hi,
I'm using :
Func<MyClass, MyClass, bool> evalFunc;   evalFunc = interpreter.ParseAsDelegate<Func<MyClass, MyClass, bool>>(expression1, "X", "Y");   evalFunc.Invoke(itemX, itemY);
On a large number of items and there're multiple expressions I pass depending on a condition.
Is there a way that I compile the expression and cache it ? to not take a lot of time when ParseAsDelegate it.
Or any other advice to speed up the process ?
I may apply this on 20,000 items.\
Regards.

I solved it by parsing the expressions before start iterating over the large amount of items I have.
I speed up the process significantly, what do you think of implementing caching mechanism inside the lib it self ?

Yes, implementing cache internally can be a solution but can have some disadvantages. For now caching it externally it is quite easy because you can do on your app based on your specific needs.
See also this:
https://github.com/davideicardi/DynamicExpresso#performance-and-multithreading
davide


storepb: expose step/range on storepb

Expose step/range on storepb so that Sidecar could use this data to
only retrieve data which is actually needed: instead of sending
everything from min time to max time, let's iterate over using the step
and only send the range time in the future from any given step.
Remote read supports multiple queries in one request so we could
leverage that for this feature
(https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L32).
Prometheus remote read sends XOR encoded chunks so there shouldn't be
more load on Prometheus server, and we should be able to successfully
reduce the necessary bandwidth between Sidecar and Query.
Signed-off-by: Giedrius Statkevičius<EMAIL_ADDRESS>

Remote read supports multiple queries in one request so we could
leverage that for this feature
(https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L32)

For the multiple queries here, how would Prometheus return data in one response? Is it actually doing a union for response serieses of all queries?

@yeya24 I was thinking about https://github.com/GiedriusS/thanos/commit/a186b5be390dfe2425aed71975cab2323533e2d3 - that commit actually works, just need to do a small benchmark whether it actually reduces the amount of network traffic

I am good with the change but let's get more reviews from other maintainers about the proto change.
It would be better to include GiedriusS@a186b5b this commit to this pr and make sure this way has improvement. WDYT?

I am good with the change but let's get more reviews from other maintainers about the proto change. It would be better to include GiedriusS/thanos@a186b5b this commit to this pr and make sure this way has improvement. WDYT?

I want to start small by including these two extra hints in SeriesRequest. I think it will be cleaner to use those hints downstream i.e. in Sidecar to generate those time ranges instead of putting everything in the request. I think we are the most active maintainers :smile: so, of course, if anyone wants to chime in then feel free. Otherwise, I will come back with a simple bench to see whether it helps. And if it does then I'll merge this and work on the 2nd part - using this data in Sidecar


get_dummies doesn't update columns variable

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
Modin installed from (source or binary): pypi
Modin version: 0.6.2
Python version: 3.6.8

Describe the problem
get_dummies doesn't update the dataframe columns variable. EDIT: Actually, it appears that it doesn't update anything in the dataframe, since the .shape of the returned dataframe stays the same.
With pandas:
import pandas as pd

df = pd.DataFrame({'categories': ['one', 'two', 'three', 'four']})
df = pd.get_dummies(df)
df

Gives me:
   categories_four  categories_one  categories_three  categories_two
0                0               1                 0               0
1                0               0                 0               1
2                0               0                 1               0
3                1               0                 0               0

and df.columns outputs:
Index(['categories_four', 'categories_one', 'categories_three',
       'categories_two'],
      dtype='object')

With modin:
import modin.pandas as pd

df = pd.DataFrame({'categories': ['one', 'two', 'three', 'four']})
df = pd.get_dummies(df)
df

Gives
   categories_four  categories_one  categories_three  categories_two
0                0               1                 0               0
1                0               0                 0               1
2                0               0                 1               0
3                1               0                 0               0

But df.columns gives
Index(['categories'], dtype='object')

Hi @Rdlenke, thanks for posting!
This is a bug with the metadata calculation after the computation has finished. We keep the metadata separate for performance reasons, so this will be a quick fix. The computation is still correct, but if you do df.columns that value will not be correct. As you point out shape is also an issue, but shape is using the columns to determine the size.
Thanks again for posting! I will get this fixed today.

@devin-petersohn Well, that explains it. I thought that there was something wrong with _fold, but couldn't point what was it.
Thanks for answering!

Yes, it was a misuse of _fold trying to use an optimization that isn't possible for the get_dummies case. Thanks again for the report!


Change title depending on route

Is your feature request related to a problem? Please describe.
When I try to select which page to go back to, I can't differentiate them:

Describe the solution you'd like
The <title> changing when the route changes, so that I see a descriptive list, maybe:
workflow-type Queries | workflow-id
workflow-type Pending Activities | workflow-id
workflow-type | workflow-id
Workflows | default namespace
other-workflow-type | other-workflow-id
Workflows | default namespace

Yea, this has been on my to-do list for a bit.


Swamy/02jul2022 session6 documentation pass1

Codecov Report

Merging #75 (9396dbf) into main (7bbab87) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #75   +/-   ##
=======================================
  Coverage   36.15%   36.15%
=======================================
  Files          12       12
  Lines         260      260
=======================================
  Hits           94       94
  Misses        166      166

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Add CVE re-scoring configuration to VPA images

How to categorize this PR?

/area auto-scaling
/kind enhancement
What this PR does / why we need it:
Add CVE re-scoring configuration to VPA images, so we can automatically re-compute CVE scores for VPA vulnerabilities.
Release note:

NONE

/approve
/cc @ccwienk @andrerun

/lgtm


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 2439

Questions ?
Please refer to the Backport tool documentation


Pass through option parameters

This allows setting 'disabled' per option, for example.

Thanks!

I released a new version (0.3.0)

Thanks!


Docker

Main additions: docker-compose.yml and Dockerfile
To use, run docker-compose up --build and navigate to localhost:8000 in a browser.
The database is not populated by default, use
docker exec -it datatube-django-1 python ../database/kaggle_import_postgres.py

to populate the database.
Tests may be run with
docker exec -it datatube-django-1 python manage.py test datatubeapp
docker exec -it datatube-django-1 python manage.py test functional_tests

To shutdown the database, use docker-compose down -v.

Alright, we were able to get it to work on Joshua's computer, so I'm merging the code onto main.


G2H does not start

Hello. Foolishly deleted the /tmp/out/ folder and the /etc/normal.xml file. The yellow indicator is now on all the time, it does not appear on the network. What can I do?

It happens. I'd think restoring normal.xml should do trick as the /tmp/out folder is untarred on startup.
normal.xml:
<monitor>
        <process name="/local/bin/property_service">
                <arg desc="loglevel option">-i</arg>
                <arg desc="loglevel 0">/etc/build.prop</arg>
                <arg desc="data path option">-p</arg>
                <arg desc="data path">/mnt/config/prop.dat</arg>
        </process>
        <process name="/tmp/out/mosquitto">
                <arg desc="foreground run"></arg>
        </process>
        <process name="/tmp/out/ha_agent">
                <arg desc="foreground run"></arg>
        </process>
        <process name="/tmp/out/proxy_server">
                <arg desc="foreground run">-D</arg>
        </process>
        <process name="/tmp/out/camera">
                <arg desc="bring up by monitor">-M</arg>
        </process>
        <process name="/tmp/out/ha_master">
                <arg desc="bring up by monitor">-a</arg>
                <arg desc="bring up by monitor">/lib/libha_auto.so </arg>
                <arg desc="bring up by monitor">-G</arg>
                <arg desc="bring up by monitor">/mnt/config</arg>
        </process>
        <process name="/tmp/out/zigbee_agent">
        <arg desc="bring up by monitor">-f</arg>
        <arg desc="bring up by monitor">/etc/zigbeeAgent.conf</arg>
        </process>
        <process name="/tmp/out/ha_driven">
        <arg desc="bring up by monitor">-d</arg>
        <arg desc="bring up by monitor">/mnt/config/ha_driven</arg>
        </process>
</monitor>

I soldered the pins to the correct connectors and connected to the Uart device.
When connected to a device via Putty, I used 38400.8n1.
But Putty won't connect. I do not know what to do. Help me please.

Have you tried other bitrates? It's more likely to be 115200. I usually test by swapping tx and rx as it's easy to get them around the wrong way.

Thx, i connected. But can't etner any command.
On connect error:
'/dev/stdin' -> '/proc/self/fd/0'
[RCS]: /etc/init.d/S90app
Can't find upgrade file
mDNSResponder: mDNSResponder (Engineering Build) (May 27 2019 12:17:34) starting
mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy ( 0)
mDNSResponder: mDNSPlatformSourceAddrForDest: connect <IP_ADDRESS> failed errno 101 ( Network is unreachable)
[RCS]: /etc/init.d/S91telnetd
19700101 03:00:34.124 ERROR monitor 115: get RootElement fail
19700101 03:00:34.125 ERROR monitor 315: parser_processlist[/etc/normal.xml] fail

That is interesting. The getty on the serial port is usually started from inittab - check your tx connection and ground wiring, you might need to hit a few keys during boot to get a response on the console but would be good to confirm if there are any issues there before trying anything else.


PAPP-22082 - Mark the app as fips compliant

Please ensure your pull request (PR) adheres to the following guidelines:

Please refer to our contributing documentation for any questions on submitting a pull request, link: Contribution Guide

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[ ] Testing of all the changes has been performed (for bug fixes / features)
[ ] The readme.html has been reviewed and added / updated if needed (for bug fixes / features)
[ ] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[ ] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[ ] Verify all checks are passing.
[ ] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[ ] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation
[ ] Other (please describe):

Security Considerations (REQUIRED)

[ ] If you are exposing any endpoints using a REST handler,
please document them in the readme.html.
[ ] If this is a new connector or you are adding new actions

[ ] Please document in the readme.html all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[ ] If any actions are unable to run on SOAR Cloud, please document this in the readme.html.

[ ] Are you introducing any new cryptography modules? If yes, please elaborate their purpose:
[ ] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API.
[ ] Are you are marking code to be ignored by Semgrep with nosemgrep?
If yes, please provide justification in an additional comment next to the ignored code.

Release Notes (REQUIRED)

Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.

What is the current behavior? (OPTIONAL)

Describe the current behavior that you are modifying.

What is the new behavior? (OPTIONAL)

Describe the behavior or changes that are being added by this PR.

Other information (OPTIONAL)

Any other information that is important to this PR such as screenshots of how the component looks before and after the change.

Pay close attention to (OPTIONAL)

Any specific code change or test case points which must be addressed/reviewed at the time of GA release.

Screenshots (if relevant)

Thanks for contributing!

Why is the title of this MR to mark the app as fips compliant? It looks like it has been fips compliant since last version: https://github.com/splunk-soar-connectors/carbonblackresponse/pull/5/files#diff-9044acfe063f619eca5ccbe733e77ed7fde788c2f988da44b4c07d12842c232aR16
Jira ticket number is not formatted correctly

@alexa-phantom, The app was marked as fips complaint but was only merged in the next and not released. And all the pipelines were failing in the next. Hence, this PR is to resolve pipeline failures and then release the app as fips_complaint.


Update to MSBuild project model and fix for cross-targeting projects

Lots of changes here, but the key pieces are:

The MSBuild packages are updated to the latest.
We now include the right set of NuGet packages that we'll need to perform restore.
We include the NuGet.Build.Tasks package in the msbuild folder.
MSBuild tests are enabled (hope they pass this time!)
We check to see if there are multiple TFMs and pick one prior to running the "ResolveReferences" task. (Otherwise, we can't process cross-targeting MSBuild projects.)
The MSBuild project model includes the list of TFMs specified in the project file.
The MSBuild project model includes the output path of the project file.

cc @david-driscoll

I wonder if this is ready to merge into dev?

That ran locally grumble


Add example of a full featured doc mapper

There is a simple wikipedia mapper for now, would be good to have one adapted to a logging dataset.
Ideas of dataset for the mapping:

[ ] github dataset https://www.gharchive.org/ https://github-sql.github.io/explorer/
[x] hdfs logs dataset used for demo

We have already a nice example of the logging dataset, I'm closing this issue, we may open new ones on specific interesting datasets.


merge develop

Hi @debsahu,
I used this version quite long at the weekend and had no issues so far. So I‘m completely fine with it. Thank you for your work to implement these improvements.
Best regards
Tobias


Define 'fields' request parameter as optional

The 'fields' parameter was defined as not required in the OpenAPI schema but the API code will error out if it is not included in the request. (I have updated the schema to require the 'fields' parameter for now so it matches the API functionality.)
It looks like we have a default value selected of: feature_id, time_str, wse, geometry  but it is not getting passed to the API Lambda function.
Should the fields parameter be required? Does this subset still make sense or should we default to returning all available data fields?
@cassienickles - "I don't think the fields functionality should be required. To me, it should default to include all fields unless that is overburdensome for performance."
@frankinspace and @torimcd - What do you think?

I'm ok making it optional and defaulting to all fields

I agree, I think that's the most expected behavior


How to use some operators.

Sorry, for my question.
How to filter by fields "must" and "range"?
I used
q = Traps.es.all().filter(time__gte="now-10d") q.filter(severity="ERROR")
but it tell me raise

RequestError: TransportError(400, u'search_phase_execution_exception', u'No query registered for [must]')

How to get range and must fields filter?
P.S.: query search create
{'query': {'filtered': {'filter': {'range': {'time': {'gte': 'now-10d'}}, 'bool': {'must': [{'term': {'severity': 'error'}}]}}}}}

Hello,
the syntax is correct, the must filter of Elasticsearch is mapped to __exact of Django and range to gt, gte, etc.. So it seems that either the query is not generated properly (bug) or the used version of pyelasticsearch is not matching your elasticsearch install ?
I will investigate.

I pushed a fix, let me know if it works for you.

Thank you so much!
It works!
`>> Traps.es.all().filter(time__gte=last.isoformat(), severity="ERROR", server="test")
{'query': {'filtered': {'filter': {'bool': {'must': [{'range': {'time': {'gte': '2016-04-08T11:17:26.100360'}}},
{'term': {'severity': 'ERROR'}}, {'term': {'server': 'test'}}]}}}}, 'size': 1000}
[{u'obj': u'None', u'severity': u'ERROR', u'param': u'test', u'server': u'test', u'time': u'2016-04-08T11:59:43.286331', u'msg': u''}]`
I add "size" to query body in your module, because only 10 results returns basicly.
You can add additional attribute in EsIndexable class for development convenience.
Thank you for help and this project :)


Unable to call overwritten implementation of the RuleListener's onFailure method

Hello,
I'm attempting to use the RuleListener's onFailure method as a means to do exception handling where thrown exceptions may be easier to catch and debug.

I created a new class RulesEngineListenerImpl that implements the RuleListener, and Override the onFailure:

   @Override
    public void onFailure(Rule rule, Facts facts, Exception exception) {
        // todo: convert to proper logging once tested and confirmed
        System.out.println("HELLO WORLD");
        log.error(exception.getMessage());
    }

I then created a unit test RuleListenerImplTest to test that exceptions thrown during rule evaluation by the Rules Engine should move into my custom implementation of onFailure

   @Mock
    private RulesEngineListenerImpl rulesEngineListener;

    @Test
    public void whenTheRuleFails_thenOverriddenOnFailureImplementationShouldBeExecuted() throws Exception {
        // Given
        when(mvelRule.evaluate(facts)).thenReturn(true);
        final Exception exception = new Exception("test exception thrown");
        doThrow(exception).when(mvelRule).execute(facts);
        rules.register(mvelRule);

        // When
        rulesEngine.fire(rules, facts);

        // Then
        InOrder inOrder = Mockito.inOrder(rulesEngineListener);
        inOrder.verify(rulesEngineListener).onFailure(mvelRule, facts, exception);
    }

The test passes yes, but i noted that the terminal output does not include the string that I expected to be printed, which seems to hint that the Overridden method onFailure was never entered. Am i misunderstanding how to use the RuleListener class here?

It looks like you didn't register your listener in the rules engine (I can't confirm since that you didn't share all your test setup). But the listener's onFailure method should be called when a rule throws an exception. Here is a minimal complete example:
import org.jeasy.rules.api.Facts;
import org.jeasy.rules.api.Rule;
import org.jeasy.rules.api.RuleListener;
import org.jeasy.rules.api.Rules;
import org.jeasy.rules.core.BasicRule;
import org.jeasy.rules.core.DefaultRulesEngine;
import org.junit.Test;
import org.mockito.Mockito;

public class RuleListenerTest {

	@Test
	public void whenTheRuleFails_thenOverriddenOnFailureImplementationShouldBeExecuted() {
		// given
		final Exception exception = new Exception("test exception thrown");
		Rule rule = new BasicRule("MyRule") {
			@Override
			public boolean evaluate(Facts facts) {
				return true;
			}

			@Override
			public void execute(Facts facts) throws Exception {
				throw exception;
			}
		};
		Rules rules = new Rules(rule);
		Facts facts = new Facts();
		MyRuleListener myRuleListener = Mockito.spy(new MyRuleListener());

		DefaultRulesEngine rulesEngine = new DefaultRulesEngine();
		rulesEngine.registerRuleListener(myRuleListener);

		//when
		rulesEngine.fire(rules, facts);

		// then
		Mockito.verify(myRuleListener).onFailure(rule, facts, exception);
	}

	static class MyRuleListener implements RuleListener {
		@Override
		public void onFailure(Rule rule, Facts facts, Exception exception) {
			System.err.println("Rule " + rule.getName() + " failed due to '" + exception.getMessage() + "'");
		}
	}
}

This test passes (which means the listener has been called) and you should see the error message in the standard error output:
[main] ERROR org.jeasy.rules.core.DefaultRulesEngine - Rule 'MyRule' performed with error
java.lang.Exception: test exception thrown
   ...
Rule MyRule failed due to 'test exception thrown'

I'm closing this issue for now but feel free to add a comment if you need more help on this.

Thanks! It indeed worked, I did add the listener, but I'm not sure why it didnt work the first time. I'll take a look at my code again.


Extend the NPS popup interval

Signed-off-by: nannanli<EMAIL_ADDRESS>Signed-off-by: AshleyLi<EMAIL_ADDRESS>

This changes seems reasonable, I think it can be merged personally.

There is a broken link that I could not figure out how to fix. Will reopen a new PR based on the latest master branch for this change.


Cookbook 3.3.0 required Internet access to http://www.rubygems/org

After upgrading install from Cookbook 3.2.1 to Cookbook 3.3.0, I encountered the following problem.
STDOUT: Fetching source index from https://www.rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/

I don't have access to Internet on this platform, and I won't have any.
Full log:
[2019-04-03T14:09:02+00:00] WARN: *****************************************
[2019-04-03T14:09:02+00:00] WARN: Did not find config file: C:/chef/client.rb, using command line options.
[2019-04-03T14:09:02+00:00] WARN: *****************************************
Starting Chef Client, version 13.12.3
resolving cookbooks for run list: ["arcgis-enterprise::system", "esri-iis", "arcgis-enterprise::server", "arcgis-enterprise::server_wa", "arcgis-enterprise::datastore", "arcgis-enterprise::portal", "arcgis-enterprise::portal_wa", "arcgis-enterprise::federation"]
W, [2019-04-03T14:09:10.642660 #4732]  WARN -- : You are setting a key that conflicts with a built-in method Hashie::Mash#default defined in Hash. This can cause unexpected behavior when accessing the key as a property. You can still access the key via the #[] method.
W, [2019-04-03T14:09:10.752037 #4732]  WARN -- : You are setting a key that conflicts with a built-in method Hashie::Mash#default defined in Hash. This can cause unexpected behavior when accessing the key as a property. You can still access the key via the #[] method.
W, [2019-04-03T14:09:11.611396 #4732]  WARN -- : You are setting a key that conflicts with a built-in method Hashie::Mash#default defined in Hash. This can cause unexpected behavior when accessing the key as a property. You can still access the key via the #[] method.
Synchronizing Cookbooks:
  - arcgis-enterprise (3.3.0)
  - esri-iis (0.1.1)
  - arcgis-repository (3.3.0)
  - hostsfile (2.4.5)
  - limits (1.0.0)
  - authbind (0.1.10)
  - iptables (1.0.1)
  - windows (4.1.4)
  - windows_firewall (3.0.2)
  - ms_dotnet (3.2.1)
  - nfs (2.3.1)
  - java_properties (0.1.3)
  - openssl (8.1.2)
  - aws (7.3.1)
  - line (0.6.3)
  - sysctl (0.8.1)
  - ohai (5.1.0)
Installing Cookbook Gems:

Running handlers:
[2019-04-03T14:09:18+00:00] ERROR: Running exception handlers
Running handlers complete
[2019-04-03T14:09:18+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 14 seconds
[2019-04-03T14:09:18+00:00] FATAL: Stacktrace dumped to c:/chef/local-mode-cache/cache/chef-stacktrace.out
[2019-04-03T14:09:18+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-04-03T14:09:18+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17'
---- Begin output of bundle install ----
STDOUT: Fetching source index from https://www.rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Could not fetch specs from https://www.rubygems.org/
STDERR:
---- End output of bundle install ----
Ran bundle install returned 17

This problem looks similar to https://github.com/Esri/arcgis-cookbook/issues/53.
The only new third-party cookbook used by 3.3.0 is 'aws' and its dependencies.
aws cookbook requires community_cookbook_releaser gem.
# This gemfile provides additional gems for testing and releasing this cookbook
# It is meant to be installed on top of ChefDK which provides the majority
# of the necessary gems for testing this cookbook
#
# Run 'chef exec bundle install' to install these dependencies

source 'https://rubygems.org'

gem 'community_cookbook_releaser'

Please try to install community_cookbook_releaser gem manually as described in https://github.com/Esri/arcgis-cookbook/issues/53.


Any method could make the result more nature?

As my experiment , the result of glow-tts sounds more like robot than a real person, do you have any method could make it more nature, like the result or autoregressive model, like tacotron eg..thanks.

Could you share your results? My results are a bit robotic too, but I've modified the network to train on multiple speakers via n_speakers and din_channels, which might be pushing the network to its limits.
I'll share some samples from my work soon.

If your concern is the prosody of the synthesized samples such as intonation, some techniques such as prosody embedding, and style tokens could be useful. In my ongoing experiments, such techniques helps to improve sample quality

@jaywalnut310 , do you have any updates with your current experiments?
I'm using your model very successfully in production and am very invested in its development. I would love to fund further improvements. I'm currently sponsoring a few folks on Github, and if you enable sponsorship I'd be extremely happy to contribute. PayPal is also fine.
I'd also love to get in touch via email to discuss some things if you don't mind reaching out. My email address is {my username} @ gmail.com .
Thanks!


Reference "level of git" from centralized collaborative workflow?

Would it be useful to reference the level of git lesson https://coderefinery.github.io/git-intro/14-level/ from "Collaborative distributed version control: Working with remotes and centralized workflow" (https://github.com/coderefinery/git-collaborative/blame/gh-pages/_episodes/01-remotes.md#L250)?

Good idea but this is then in the wrong repo. Moving this issue to git-collaborative.


Feature request: See time that isn't submitted

If a person hasn't "stopped" the clock, it would be great if that was still visible under "Manage time", so it is possible to see who still has the "clock" running.

Enhancement added to https://github.com/danniehansen/activity-timer/pull/42 & merged into 1.1.6


Random error when detonating a nuke?

The mod True Nukes (0.2.10) caused a non-recoverable error. Please report this error to the mod author.
Error while running event True-Nukes::on_script_trigger_effect (ID 153) Flush failed: Invalid argument stack traceback: [C]: in function 'destroy' True-Nukes/control.lua:1520: in function 'optimisedChunkLoadHandler' True-Nukes/control.lua:1876: in function 'atomic_weapon_hit' True-Nukes/control.lua:2093: in function <True-Nukes/control.lua:2039>

Hi, thanks for the bug report.
What were you doing at the time of the crash, and what other mods did you have installed?

the betas of SE+K2 and like 100 other mods.
 Didn't get the error after trimming the size of the planet. I'm quite curious about why it crashed tho 🤔

Yeah, that report looks very odd - I honestly think this might be an internal problem. I'll put a post on the forum. Possibilities that come to mind are things like you running out of memory - you were detonating a 10Mt (I think), on a large loaded surface, the bigger nukes scale with the area loaded at the time, so having scanned the planet might load a big area... idk how space exploration works though...
A few questions on that kind of theory:
What kind of hardware are you running?
What operating system are you using?
How long did the game freeze before failing (and did it play the sounds before dying)?
Were you running anything memory or cpu intensive at the time (like a browser)?
Did the disk activity light go fully on before the crash?
Final thing, just so I can see:
Can I have a screenshot of the traceback (like what comes up on screen)?

Oh, also, can I see the factorio log (https://wiki.factorio.com/Log_file for details)

I've asked on the forum:
https://forums.factorio.com/viewtopic.php?f=25&t=102820

I realize now that this issue might be related to not enough RAM. I have 16GB and just running modded factorio uses around 10GB. Going back to the save I made before detonating the 10Mt nuke, I tried detonating it again and the game froze for a few minutes (like it usually does when doing things like detonating nukes), but then this time it crashed.
According to task manager I was using 90-98% ram from after the nuke detonated until it crashed
I'm using windows
no browser open and disk activity was nothing out of the ordinary

Yeah, that figures. You can get almost any error from memory full, unless the system has a properly organised way of failing on full memory. You should be good for smaller nukes on that, but the bigger ones (100kt+) have performance requirements dependant entirely on the number of chunks loaded. This might have been too many.
(I was asking disk activity because some computers use a bit of disk to supplement RAM, and you can tell when they get really short because their disk light goes full-on, and they get ~1000x slower).

Yeah, factorio log shows it crashed due to Stack overflow.. quite interesting indeed
Thank you for your time :)

No problem at all.


[Blazor] Update debugging instructions for Blazor WASM on VS Code

[EDIT by guardrex to add the metadata]
This is a placeholder issue to update the Blazor WASM debug guidelines with details on how to debug with the new blazorwasm debug adapter.

Create a new Blazor WASM application and open it in VS Code.
Ensure that you have the OmniSharp and JS Debug extensions installed.
Navigate to the "Debug" section on the left. Then, click "create a launch.json file". In the window that pops up, select "Blazor WebAssembly Debug".
Launch the "Launch and Debug Blazor WebAssembly Application" configuration.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ec2f61c7-9f51-6735-e328-3c7b495d70c3
Version Independent ID: 4d06e565-282d-6387-aa90-cb2c2986d35a
Content: Debug ASP.NET Core Blazor
Content Source: aspnetcore/blazor/debug.md
Product: aspnet-core
Technology: aspnetcore-blazor
GitHub Login: @guardrex
Microsoft Alias: riande

We're gearing up to release this as part of the upcoming C# release, likely this week.
I'll submit a PR with the docs that we can collaborate on.

Cool ... we have a big one with ToC updates landing tonight (I hope 🤞) and merging either tonight or tomorrow early AM. Recommend holding off on creating your branch until that PR merges. I'll ping u back here when the coast is clear.

Sounds good. I'll write up the docs in another file for the time being and integrate it when the time is write.
Good luck with the ToC refactor!

@guardrex The O# extension has been released. Are we cleared to update the docs now?

I didn't receive my guidance from DR and Artak yet, so I haven't set up the mega-ToC branch/PR yet. Yes, let's go ahead with this issue, and I'll work the ToC thing after this is done and merged.

Cool! I'll open up a PR with a draft of the text I've produced thus far.

@captainsafia This was fixed by https://github.com/dotnet/AspNetCore.Docs/pull/18570, right?

Correct -- we can close this now.

Cool.

It's a pleasure to work with you, Captain!
— The Doctor (Tom Baker), Dr Who: The Pirate Planet, ©1978 BBC


SSDLite conversion issue [FusedBatchNormV3' (type: 'Add', num of outputs: 1) does not have output 5)]

Hi
First things first, environment config:

tensorflow: 1.15.0
coremltools: 3.1
tfcoreml: 1.1
CUDA 10.1
cuDNN: <IP_ADDRESS>-1

The model that I am trying to convert is based on the TensorFlow ssdlite_mobilenet_v2_coco, retrained on 4 classes on an input of 768x768.
I have exported the graph for inference using export_inference_graph.py.
I have also optimized the graph by removing unused nodes based on the input/output tensors.
Final error (entire stack trace is also available at the end):
tensorflow.python.framework.errors_impl.OutOfRangeError: Node 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/FusedBatchNormV3' (type: 'Add', num of outputs: 1) does not have output 5

Sample script:
import numpy as np

import tensorflow as tf
from tensorflow.python.tools import strip_unused_lib
from tensorflow.python.framework import dtypes
from tensorflow.python.platform import gfile

import tfcoreml
import coremltools

# From where to load the saved_model.pb file.
saved_model_path = "model/saved_model"

# Where to save the final Core ML model file.
coreml_model_path = "/coreml/MobileNetV2_SSDLite.mlmodel"

# The number of predicted classes, excluding background.
num_classes = 4

# The number of predicted bounding boxes.
num_anchors = 11526

# Size of the expected input image.
input_width = 768
input_height = 768

# =============================
# PART 1: MobileNetV2 + SSDLite
# =============================

# Temporary file. You can delete this after the conversion is done.
frozen_model_file = "frozen_model.pb"

# Names of the interesting tensors in the graph. We use "Postprocessor/convert_scores"
# instead of "concat_1" because this already applies the sigmoid to the class scores.
input_node = "Preprocessor/sub"
bbox_output_node = "concat"
class_output_node = "Postprocessor/convert_scores"

input_tensor = input_node + ":0"
bbox_output_tensor = bbox_output_node + ":0"
class_output_tensor = class_output_node + ":0"

def load_saved_model(path):
    """Loads a saved model into a graph."""
    print("Loading saved_model.pb from '%s'" % path)
    the_graph = tf.Graph()
    with tf.Session(graph=the_graph) as sess:
        tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], path)
    return the_graph

def optimize_graph(graph):
    """Strips unused subgraphs and save it as another frozen TF model."""
    gdef = strip_unused_lib.strip_unused(
        input_graph_def=graph.as_graph_def(),
        input_node_names=[input_node],
        output_node_names=[bbox_output_node, class_output_node],
        placeholder_type_enum=dtypes.float32.as_datatype_enum)

    with gfile.GFile(frozen_model_file, "wb") as f:
        f.write(gdef.SerializeToString())

# Load the original graph and remove anything we don't need.
the_graph = load_saved_model(saved_model_path)
optimize_graph(the_graph)

# Convert to Core ML model.
ssd_model = tfcoreml.convert(
    tf_model_path=frozen_model_file,
    mlmodel_path=coreml_model_path,
    input_name_shape_dict={input_tensor: [1, input_height, input_width, 3]},
    image_input_names=input_tensor,
    output_feature_names=[bbox_output_tensor, class_output_tensor],
    is_bgr=False,
    red_bias=-1.0,
    green_bias=-1.0,
    blue_bias=-1.0,
    image_scale=2. / 255)

All stack trace:
C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\python.exe D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py
WARNING:root:TensorFlow version 1.15.0 detected. Last version known to be fully compatible is 1.14.0 .
Loading saved_model.pb from 'model/saved_model'
WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:77: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:77: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-03-05 14:54:53.263674: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:78: The name tf.saved_model.tag_constants.SERVING is deprecated. Please use tf.saved_model.SERVING instead.

WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:78: The name tf.saved_model.tag_constants.SERVING is deprecated. Please use tf.saved_model.SERVING instead.

WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:78: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py:78: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\tools\strip_unused_lib.py:88: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\tools\strip_unused_lib.py:88: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`

Loading the TF graph...
WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py:214: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py:214: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py:215: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

WARNING:tensorflow:From C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py:215: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

Graph Loaded.
Now finding ops in the TF graph that can be dropped for inference
Collecting all the 'Const' ops from the graph, by running it....
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
    return fn(*args)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: Node 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/FusedBatchNormV3' (type: 'Add', num of outputs: 1) does not have output 5

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/WORK/Personal/Innaite/projects/matchpoint_ml_tf2coreml/convert-mobilenet-model-to-coreml-tfcoreml.py", line 109, in <module>
    image_scale=2. / 255)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py", line 718, in convert
    custom_conversion_functions=custom_conversion_functions)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tfcoreml\_tf_coreml_converter.py", line 327, in _convert_pb_to_mlmodel
    tensors_evaluated = sess.run(tensors, feed_dict=input_feed_dict)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
    run_metadata_ptr)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
    run_metadata)
  File "C:\ProgramData\Miniconda3\envs\ConvertTFModelToCoreML\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: Node 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/FusedBatchNormV3' (type: 'Add', num of outputs: 1) does not have output 5

Any thoughts?
I have also seen this is basically the same with: https://github.com/tf-coreml/tf-coreml/issues/336
Thanks!

I'm getting the same error

tensorflow: 1.15.2
coremltools: 3.3
tfcoreml: 1.1

Conversion code:
tfcoreml.convert(
    tf_model_path='frozen_graph.pb',
    mlmodel_path='Model.mlmodel',
    input_name_shape_dict={input_tensor: [1, 300, 300, 3]},
    image_input_names='Preprocessor/sub:0',
    output_feature_names=['concat:0', 'Postprocessor/convert_scores:0'],
    is_bgr=False,
    red_bias=-1.0,
    green_bias=-1.0,
    blue_bias=-1.0,
    image_scale=2. / 255)

Frozen graph:

frozen_graph.pb

@dlawrences Adding the following param to tfcoreml.convert resolved this issue for me
minimum_ios_deployment_target='13'

@dlawrences Adding the following param to tfcoreml.convert resolved this issue for me
minimum_ios_deployment_target='13'

Thanks @bourdakos1 , work for me as well!


Use assertj-core 2.0 or higher

pom indicates 1.7.0

I guess this should be closed: https://github.com/joel-costigliola/assertj-db/commit/02fa0226c3ba3ce7544d7a22e6180eebe754dfc6 ?

OK. I do.


Where is the bitstream?

https://github.com/lowRISC/sonata-system/blob/main/doc/guide/fpga-update.md says
"Download lowrisc_sonata_system_0.bit from THEBESTURL.COM/lowrisc_sonata_system_0.bit"
Needless to say, that URL doesn't work (though apparently I could buy the domain for $995). So, where does one download this?

I just made a bitstream release here: https://github.com/lowRISC/sonata-system/releases/tag/v0.1 I'll update the instructions.

FYI @marnovandermaas - quickly tried the quickstart as well on that release (reprogramming the RP2040 with your example) and it wasn't loading from SPI flash, on serial I just got:
[1B][35mbootloader[1B][0m: Loading software from flash...
[1B][35mbootloader[1B][0m: Failed Magic Check

I opened up a PR with some doc fixes.

FYI @marnovandermaas - quickly tried the quickstart as well on that release (reprogramming the RP2040 with your example) and it wasn't loading from SPI flash, on serial I just got:
[1B][35mbootloader[1B][0m: Loading software from flash...
[1B][35mbootloader[1B][0m: Failed Magic Check

Did you also drag sonata_simple_demo_v0.1.uf2 into the SONATA drive? And otherwise try again after replug?


Add debugging how to guide

We have a long standing customer / user ask to better document how to debug with pulumi; see https://github.com/pulumi/pulumi/issues/1372 for the dialog of how to do this. It's also an opportunity to experiment with micro conetnt, short animated, and re-think how docs/help is consumed.
related to https://github.com/pulumi/docs/issues/11327 - work item for a poc video

Guide launched on our blog: https://www.pulumi.com/blog/next-level-iac-briding-the-declarative-gap/