reqmgr2 TaskChain injection

Hi,
injecting TaskChain dict:
http://pastebin.com/ZxpDLu6F
fails with this error:
400 Bad Request
Invalid spec parameter value: No module named "TaskChain"
seems there is a parameter missing in sanity checks of dictionary received?
tried to inject this to testbed on Monday, February 29, 2016 11:28 AM

Thanks Antanas for the report, could you also point the script you are using and exact command. That will be a big help for debugging. I can try it to test it myself.

the command is this one:
cd /afs/cern.ch/cms/PPD/PdmV/work/McM/dev-submit/
source  /afs/cern.ch/cms/cmsset_default.sh
export SCRAM_ARCH=slc6_amd64_gcc493
if [ -r CMSSW_7_6_1/src ] ; then
echo release CMSSW_7_6_1 already exists
else
scram p CMSSW CMSSW_7_6_1
fi
cd CMSSW_7_6_1/src
eval scram runtime -sh
export X509_USER_PROXY=/afs/cern.ch/user/a/anorkus/.globus/voms_proxy.cert
export PATH=/afs/cern.ch/cms/PPD/PdmV/tools/wmcontrol_testful:${PATH}
wmcontrol.py --url-dict https://cms-pdmv-dev.cern.ch/mcm//public/restapi/chained_requests/get_dict/EXO-RunIIFall15DR7
6-03799 --wmtest --wmtesturl cmsweb-testbed.cern.ch
wmcontrol is a wrapper which makes a POST to reqmgr2. It POSTs the dictionary which i sent with pastebin. Redoing that POST (even with curl) should be enough to reproduce the error ;)

Thank you very much Antanas, I also want to know how McM script is executed as well.
Anyway, you are right it give the 400 error when I just use the post call.
The reason is default config cache url is set to cmsweb and it seems that config cache id (61778aff073bf2507ec558760d0ad333) is only exist in cmsweb-testbed.
Could you add following in your json document (on the top level would be fine not under Task[])?
"ConfigCacheUrl": "https://cmsweb-testbed.cern.ch/couchdb",
Thank you very much for the testing.

but in the dict i've pasted you can see: 'ConfigCacheURL': 'https://cmsweb-testbed.cern.ch/couchdb',

Ah, I was running the test over the link you in the command example below. So that I will run over initial json.
https://cms-pdmv-dev.cern.ch/mcm//public/restapi/chained_requests/get_dict/EXO-RunIIFall15DR76-03799

well as i told wmcontrol is wrapper, it adds those global variables with url etc. The mcm url contructs dictionary from request specific things and knows nothing about rqmgr{1;2}

Hi Antanas, I was able to create the request with http://pastebin.com/ZxpDLu6F (just using post call)
https://cmsweb-testbed.cern.ch/couchdb/_utils/document.html?reqmgr_workload_cache/sryu_McMTaskChain2_wmagent_14_160301_162925_1654
The only thing I had to change is
ConfigCacheURL -> ConfigCache**Url **
I think URL make more sense but as far as I remember, it was Url before so we couldn't change to upper case letter.
looking at wmcontrol code. following might caused the problem but I was wondering there was a no problem before?
https://github.com/ticoann/wmcontrol/blob/master/wmcontrol.py#L808
One other thing is I didn't get the same error you reported initially
Invalid spec parameter value: No module named "TaskChain"

I will try to run exactly the same command to see I can duplicate the error.
wmcontrol.py --url-dict https://cms-pdmv-dev.cern.ch/mcm//public/restapi/chained_requests/get_dict/EXO-RunIIFall15DR7
6-03799 --wmtest --wmtesturl cmsweb-testbed.cern.ch

Thanks for testing.

Hi Antanas
When I ran
wmcontrol.py --url-dict https://cms-pdmv-dev.cern.ch/mcm//public/restapi/chained_requests/get_dict/EXO-RunIIFall15DR76-03799 --wmtest --wmtesturl cmsweb-testbed.cern.ch

I am getting following error which is related to Accept header on request. (I was wondering whether I am using the wrong version of wmcontrol.py. should I use master branch or some other branch for the test?
It seemslike the name of the GT '76X_mcRun2_asymptotic_v12' has a typo in it, missing the final ::All which will crash your job. If insted you're using CondDBv2, you're fine.
Re-encoding for nested dicts
....
Status: 406 Reason: Not Acceptable
Explanation:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>406 Not Acceptable</title>

406 Not Acceptable

Not acceptable

    Powered by CherryPy 3.2.2

you shouldn't care about injection through wmcontrol. As you dont even need to clone it. You should use the install in afs which can be sourced.
Anycase the bug was on wmcontrol side:
https://github.com/cms-PdmV/wmcontrol/blob/master/modules/wma.py#L206
seems reqmgr1 needed the params to be encoded. This i didn't change and it worked fine for non TaskChains request. Which i would suggest to create another issue to parse the Content-Type of data received (if reqmgr2 receives only json)
When injecting TaskChain dict with "Content-type": "application/x-www-form-urlencoded" header it would fail with strange errors like:
Invalid spec parameter value: No module named ******
other notice:
ConfigCacheURL is fine, and seems everything got processed. I will change this to *Url in any case.

Yes, in reqmgr when data is pass as request.body it assumes to be json. We could check the content type and convert correctly. I will create the issue.


Add REST API wrapper

This PR is still in progress. Need to work out the details.
Todos:

[x] update readme with examples and API calls
[x] extent the sample project to use more of the API

Done with the todos. @fraserxu If you're ok with this PR, you can merge it!

:+1:


Models not validated properly

Describe the Bug

At the moment, all model validation is placed in the forms, because this allows for better error handling and more informative error messages in forms. However, when using different methods to create content (e.g. the xliff import), these validation methods don't work anymore. Therefore, we need also validation on the model-level.
Steps to Reproduce

Go to page tree
Import XLIFF-file with e.g. empty page titles
Confirm

Expected Behavior

If the imported data is not valid, it should not be saved to the database.
Actual Behavior

The invalid data is imported.

I'm not sure if validating on the model level is the best solution. We could say that we either use forms OR do some additional validation during IO. The question is, how we make sure that this validation is complete. We could have some sort of helper class that validates a dict (adhering to a set of rules, i.e. keys need to be the same as model properties) against model properties?

I'm not sure if validating on the model level is the best solution. We could say that we either use forms OR do some additional validation during IO. The question is, how we make sure that this validation is complete? We could have some sort of helper class that validates a dict (adhering to a set of rules, i.e. keys need to be the same as model properties) against model properties?

I'm not sure if I understood this correctly. We definitely wouldn't need some kind of custom helper class to validate a dict, because Django already has very powerful validation mechanisms on the model level (similar to form validation, see docu)...

I see your point - I supposed that validation is being done on the form layer. However, I would still insist, that we need different levels of validation during manually editing pages vs XLIFF imports. While I would ask a user to enter a title while filling in the page edit form, I would allow XLIFF imports with empty titles. The problem with XLIFF files is that most users are not able to handel (as in edit) them, and sending them back to translators is not always an option. Therefore it should be possible to import and fix some stuff later in the back end.

Ah ok, interesting. Never thought of this workflow. Maybe they should be imported as status "in-review" (or we create a new status "invalid") to make it clear that they need a manual review before they can be published...


Unit and regression tests

Should be able to tell if code is working without waiting an hour.
Unit tests: small graphs exercising only parts of the code?
Regression tests: save some checkpoints of small canned systems.  Start from these checkpoints and verify the outcome is about the same.

Can you delegate on your minions ?

I added a unit test framework with example unit tests (see 3f51d65, 085754c).
There is also the regression-testing branch but it seems to be outdated.

Moving discussion to issue https://github.com/marian-nmt/marian/issues/76


Ajax as a data source

Can You make that data will be loader through ajax? ui-select has refresh and refresh-delay attrbs

Hi, when You think You'll manage to make new version?

any news??

I haven't found any way to make this.

Do You mean That You won't add it?? Take ui.bootstrap as a source. You have to add this option and Your select component will be the best...

Sorry for bother.
I need Your declaration, because it's very important form me.
Are You making ajax handling or not?
If not I have to look for other solution.

No, Ajax is not planned. Sorry for this.

I am using the component, and this feature is missing for me too. I may develop this internally in the future. Are you open to PRs on this?
Thanks.

Add ajax data source may make this component too wired.


Splat failes on UWP release using Native tool chain

Do you want to request a feature or report a bug?
BUG
What is the current behavior?
When I am using the latest version of Akavache with Splat.
Everything works fine in debug once I try to run in release with native tool chain enabled, splat 6.1.7 produces several errors as below url.
https://gist.githubusercontent.com/EmilAlipiev/9e14587b53173c02f292e0a18a0b3f5f/raw/0576b95823b8aa1ef2809a154ee81c3ee64853d0/gistfile1.txt

This has been slated to be investigated after v7 of splat has been released. This is in a couple weeks.

And thanks for reporting the issue @EmilAlipiev  -- even though you aren't directly using Splat this is probably the better location for the bug regardless.

This seems related to https://github.com/reactiveui/ReactiveUI/issues/1957 -- you  may want to upgrade as a work-around.

6.2.3 is not available directly trough nuget. there was 6.2.2 but it had a bigger problem. Microsoft removed this version from nuget also. currently 6.1.9 is the best package to be used and uploaded on Store. at the moment i am using splat older version with 6.1.9 and it works fine for me

Well next release Splat will be going back to 6.1.9 anyway so this will be solved then.

any news for this issue? latest akavache version is dependent on splat 6.1.7 and this issue blocks me to update akavache.

Hi there. We have other issues at the moment that backers from open collective have requested fixes for https://opencollective.com/reactiveui and those take priority. Hopefully soon though.

The aim is getting a release out soon with Splat time pending within the next week or so.

Fixed as part of the release coming today

@EmilAlipiev Splat v7.0.1 has been released.  Please verify that this issue is resolved in that package?


feat: apple support

@dzhdanNV just an observation you nearly have a C interface any reason for the C++ code to abstract across that. you can also just union a bunch of stuff together for dx12/dx11/vulkan/metal. You also have all this logic to build tables of functions etc ...
struct CmdBuffer {
   union {
      struct {...}dx11;
      struct {...}dx12;
      struct {...}metal;
   };
}

struct NriDevice {
   union {
      struct {...}dx11;
      struct {...}dx12;
      struct {...}metal;
   };
   int (*Method1)(NriDevice*...);
}

been busy with something else, I'll have to catch up to this once i finish up what i want to do with this game project im working on.

Sure, no prob. And tons of thanks for working on native Apple support!

Thanks for the progress. Very appreciated.

so i guess you encode the data to an argument buffer. I guess to reproduce descriptor sets the pool  would allocate an argument buffer for n number of descriptor sets?
fragmentShader(       RasterizerData            in                 [[ stage_in ]],
               device FragmentShaderArguments & fragmentShaderArgs [[ buffer(AAPLFragmentBufferIndexArguments) ]])

To be honest I don't follow. Please, explain the problem and a solution. Is the question about indirect drawing or something else?

I guess to reproduce descriptor sets the pool would allocate an argument buffer for n number of descriptor sets?
Correct. Metal requires to fill argument buffers even for non-indirect draw calls, so you need to store descriptors in it.

any clue how these properties line up with DescriptorRangeDesc?
https://developer.apple.com/documentation/metal/mtlargumentdescriptor?language=objc
for(size_t r = 0; r < descriptorSetDesc.rangeNum; r++) {
            MTLArgumentDescriptor* argDescriptor = [MTLArgumentDescriptor argumentDescriptor];
            const DescriptorRangeDesc* range = &descriptorSetDesc.ranges[r];
            argDescriptor.arrayLength = range->descriptorNum;
            argDescriptor.access = MTLBindingAccessReadWrite;
            argDescriptor.index = range->baseRegisterIndex;
            switch(range->descriptorType) {
                case DescriptorType::TEXTURE:
                    argDescriptor.dataType = MTLDataTypeTexture;
                    argDescriptor.textureType = MTLTextureType2D; // descriptor type does not have this
                    break;
                case DescriptorType::SAMPLER:
                    argDescriptor.dataType = MTLDataTypeSampler;
                    break;
                case DescriptorType::CONSTANT_BUFFER:
                case DescriptorType::STORAGE_TEXTURE:
                case DescriptorType::BUFFER:
                case DescriptorType::STORAGE_BUFFER:
                case DescriptorType::STRUCTURED_BUFFER:
                case DescriptorType::STORAGE_STRUCTURED_BUFFER:
                    argDescriptor.dataType = MTLDataTypeStruct;
                    break;
                case DescriptorType::ACCELERATION_STRUCTURE:
                    argDescriptor.dataType = MTLDataTypePrimitiveAccelerationStructure;
                    break;
            }
        }

An obvious suggestion: you can look at MoltenVK sources and work this way:
NRI => open NRI/VK => open corresponding place in MoltenVK => analyze, rearrange, copy & paste to MTL implementation

(looking at MTL docs...)

An obvious suggestion: you can look at MoltenVK sources and work this way:
NRI => open NRI/VK => open corresponding place in MoltenVK => analyze, rearrange, copy & paste to MTL implementation

(looking at MTL docs...)

I've been looking at the MoltenVK source, thanks for the suggestion. umm, ill have to do a bit of research i don't want to have to extend DescriptorType. maybe there is a more ideal solution?

Why do you need to potentially extend DescriptorType?

Why do you need to potentially extend DescriptorType? Because of this:
                case DescriptorType::TEXTURE:
                    argDescriptor.dataType = MTLDataTypeTexture;
                    argDescriptor.textureType = MTLTextureType2D; // descriptor type does not have this
                    break;

?

https://developer.apple.com/documentation/metal/mtltexturetype
defines 1D/1DArray/2D/2dArray/Cube/CubeArray etc ...

You can do the same trick as here - https://github.com/NVIDIAGameWorks/NRI/blob/205cf0518fbef19b3e6a8506e359d276e68b90b3/Source/D3D11/DescriptorD3D11.hpp#L55. Just identify your texture type by texture parameters, that's all you need.

You can do the same trick as here -
https://github.com/NVIDIAGameWorks/NRI/blob/205cf0518fbef19b3e6a8506e359d276e68b90b3/Source/D3D11/DescriptorD3D11.hpp#L55
Just identify your texture type by texture parameters, that's all you need.

in what way?

AFAIK, MTLTextureDescriptor is a direct analogue of NriDescriptor, so it shouldn't be hard to identify the texture type when creating the descriptor. If I'm right, you can use your TextureViewType to identify the MTLTextureType.

The problem is that there are no descriptors in DescriptorRangeDesc. In other places - yes. In this and hopefully the only one place - no.

https://github.com/KhronosGroup/MoltenVK/blob/1d2ec638532268d076a550262df2e5e5e4fd2c9e/MoltenVK/MoltenVK/GPUObjects/MVKDescriptorSet.mm#L1146C43-L1146C67
This type doesn't seem to be very important, as it's only used for size calculations.

is there any differences with managing memory with objective c. assignment rules and how things are freed?


Home assistant support

I added support for home assistant with a docker container.
I wanted to put this merge request to a dev branch but I did not see one.

This looks like a valuable addition. However, not sure the maintainer is eager on adding Home assistant. See issue #28.
But if this would go ahead, I'd have the following questions:
I noticed that you're using a specific homeassistant docker image for Raspberry Pi 4. Would you need a step of identifying if someone is using a Raspberry Pi 3 and then use homeassistant/raspberrypi3-homeassistant instead?
The linked install guide seems to indicate one should just use: ghcr.io/home-assistant/home-assistant:stable. So maybe that would be ok, but you would loose performance enhancement of the version specific packages?
I believe Homassistant also keeps the Docker Hub updated with recent images. For consistency with the other docker-compose files in the repository, it might be preferred to use the Docker Hub over Git Hub Packages?

I will take a deeper look at how the container is setup. I would like to offer it for more then just the Pi4.
I also thought I canceled this pull request after finding that that issue pointing out this goal of this project is internet monitoring.

Hi @EclipseVision,
I have moved this into a separate Repo and called it Home-Pi.
Basically as a separate solution for Home Automation. Kind of an add-on to Internet-Pi.
So far supporting:

Home Assistant Core (only pulling a generic image, want to change to Raspberry specific)
Home Assistant Supervisor
AirConnect

Check it out and any feedback welcome.
Cheers

this is so useful but looks like Home-Pi is not maintained anymore. adding HA support here would be super cool


feat: add da client can be initialized with the gcs seed

Implement DA client can init with a encrypted seed from google cloud storage. The encrypted seed has been encrypted by a HSM key stored in google KMS.
There are 3 environment variables need to be setup for enabling this feature.
DA_SECRETS_FROM_GCS="true"
DA_SECRETS_GCS_BUCKET_NAME="cronos-zkevm-encrypt-seed--bucket-test"
DA_SECRETS_KMS_DECRYPT_KEY_NAME="projects/zkevm-research/locations/northamerica-northeast2/keyRings/gkms_signer_test/cryptoKeys/avail-signer-seed-op"
Note:
There is a tooling can generate a random seed, encrypted with the assigned kms key, and then upload to the gcs bucket.
https://github.com/JayT106/avail-seed-with-kms/blob/main/src/main.rs
run avail-seed-with-kms  [kms key path] [gcs bucket name]

i.e.
avail-seed-with-kms projects/zkevm-research/locations/northamerica-northeast2/keyRings/gkms_signer_test/cryptoKeys/avail-signer-seed-op  cronos-zkevm-encrypt-seed--bucket-test

Implement DA client can init with a encrypted seed from google cloud storage. The encrypted seed has been encrypted by a HSM key stored in google KMS.
There are 3 environment variables need to be setup for enabling this feature.
DA_SECRETS_FROM_GCS="true" DA_SECRETS_GCS_BUCKET_NAME="cronos-zkevm-encrypt-seed--bucket-test" DA_SECRETS_KMS_DECRYPT_KEY_NAME="projects/zkevm-research/locations/northamerica-northeast2/keyRings/gkms_signer_test/cryptoKeys/avail-signer-seed-op"

looks good however not sure it is relevant in this cronos version.
could you open up a PR upstream and see if they would like to merge it?

Open an PR
https://github.com/matter-labs/zksync-era/pull/3315

Okay to merge because it seems harmless and its a mini improvement.
Some changes is being done on the DA layer with the gateway integration. Will see if it is still useful in later version


Added active hooks to migrate-to-ig-publisher

Still needs:

Formatting changes
Manual link verification
"Summary page" (ask Isaac or myself)
Deprecated hooks

@rizzolol the changes don't build with the IG publisher. Do they work for you locally?


Emit unhandled rejection event from HTML Spec

see:

https://tc39.github.io/ecma262/2016/#sec-host-promise-rejection-tracker
https://html.spec.whatwg.org/#unhandled-promise-rejections

Let event be a new trusted PromiseRejectionEvent
We can't do that, can we?

We can't actually make it "trusted" but we can make an object that implements approximately the same interface (such that it would only be distinguishable by someone intentionally checking to make sure it wan't a polyfill).