DPU build failed because of timing violation by Vitis

I tried to run DPU build flow by simply following the instruction below.
https://github.com/Xilinx/Vitis-AI/tree/master/DPU-TRD/prj/Vitis#52-build-and-run-flow
However, although I didn't touch the configuration, the build finally failed. it seems that hold time violation happened in the last step.
the final executed command is make KERNEL=DPU_SM DEVICE=zcu102. I put the error message below.
Is there anyone who knows about this?
[12:27:45] Starting bitstream generation..
[12:31:39] Run vpl: Step impl: Failed
[12:31:39] Run vpl: FINISHED. Run Status: impl ERROR

===>The following messages were generated while  Compiling (bitstream) accelerator binary: dpu Log file: /home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/runme.log  :
ERROR: [VPL-4] design did not meet timing - hold violation
ERROR: [VPL 60-773] In '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/vivado.log', caught Tcl error:  problem implementing dynamic region, impl_1: route_design ERROR, please look at the run log file '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/runme.log' for more information
WARNING: [VPL 60-732] Link warning: INFO: Platform does not require extraction of debug/profile metadata.
WARNING: [VPL 60-732] Link warning: No monitor points found for BD automation.
ERROR: [VPL 60-704] Integration error, problem implementing dynamic region, impl_1: route_design ERROR, please look at the run log file '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/runme.log' for more information
ERROR: [VPL 60-1328] Vpl run 'vpl' failed
ERROR: [VPL 60-806] Failed to finish platform linker
INFO: [v++ 60-1442] [12:31:39] Run run_link: Step vpl: Failed
Time (s): cpu = 00:01:13 ; elapsed = 03:33:30 . Memory (MB): peak = 1340.699 ; gain = 0.000 ; free physical = 7880 ; free virtual = 50197
ERROR: [v++ 60-661] v++ link run 'run_link' failed
ERROR: [v++ 60-626] Kernel link failed to complete
ERROR: [v++ 60-703] Failed to finish linking
INFO: [v++ 60-1653] Closing dispatch client.
Makefile:70: recipe for target 'binary_container_1/dpu.xclbin' failed
make: *** [binary_container_1/dpu.xclbin] Error 1

Hi @tkclimb
If you have not modified Makefile, you can refer the chapter 5.4
https://github.com/Xilinx/Vitis-AI/tree/master/DPU-TRD/prj/Vitis#54-integrate-the-dpu-in-customer-platform
If you meet some timing issues. you can modify the [vivado] part of prj_config file (prop=run.impl_1.strategy=Performance_Explore) to change another implementation strategy and re-compile the project.
you can simply modify the /config_file/prj_confg. change the line44 prop=run.impl_1.strategy=Performance_Explore to prop=run.impl_1.strategy=Performance_NetDelay_low or prop=run.impl_1.strategy=Congestion_SpreadLogic_low and try again.

@qianglin-xlnx
Thanks, I understand. How should I resume the build without starting from the begging after I changed the configuration? Is there any make target for retry the "impl" (or "synth" if necessary)??

@tkclimb
There is no way. The project must re-build again. It calls the vivado to build the project actually.

@qianglin-xlnx
I tried to make the project with both strategies Performance_NetDelay_low and Congestion_SpreadLogic_low but they didn't work successfully. both case also failed by hold time violation.
I put the error message in binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/runme.log
INFO: [Common 17-83] Releasing license: Implementation
342 Infos, 160 Warnings, 2 Critical Warnings and 0 Errors encountered.
route_design completed successfully
route_design: Time (s): cpu = 08:58:15 ; elapsed = 02:44:19 . Memory (MB): peak = 13644.312 ; gain = 1789.000 ; free physical = 7896 ; free virtual = 49438
source /home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/scripts/_full_route_post.tcl
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7893 ; free virtual = 49435
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7880 ; free virtual = 49421
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7878 ; free virtual = 49420
INFO: [Timing 38-480] Writing timing data to binary archive.
Writing placer database...
Writing XDEF routing.
Writing XDEF routing logical nets.
Writing XDEF routing special nets.
Write XDEF Complete: Time (s): cpu = 00:01:26 ; elapsed = 00:00:29 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 6089 ; free virtual = 49194
INFO: [Common 17-1381] The checkpoint '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/dr_routed_timing.dcp' has been generated.
write_checkpoint: Time (s): cpu = 00:02:30 ; elapsed = 00:01:27 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7511 ; free virtual = 49387
report_timing_summary: Time (s): cpu = 00:00:33 ; elapsed = 00:00:16 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7369 ; free virtual = 49244
report_timing_summary: Time (s): cpu = 00:00:47 ; elapsed = 00:00:18 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7368 ; free virtual = 49244
ERROR: [runtcl-1] design did not meet timing - hold violation
INFO: [Timing 38-480] Writing timing data to binary archive.
Writing placer database...
Writing XDEF routing.
Writing XDEF routing logical nets.
Writing XDEF routing special nets.
Write XDEF Complete: Time (s): cpu = 00:01:26 ; elapsed = 00:00:29 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 5578 ; free virtual = 49017
INFO: [Common 17-1381] The checkpoint '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/zcu102_base_wrapper_routed_error.dcp' has been generated.
write_checkpoint: Time (s): cpu = 00:02:31 ; elapsed = 00:01:28 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 6990 ; free virtual = 49199
ERROR: [Common 17-39] 'send_msg_id' failed due to earlier errors.

INFO: [Common 17-206] Exiting Vivado at Fri Aug  7 14:25:12 2020...

@qianglin-xlnx
I tried to make the project with both strategies Performance_NetDelay_low and Congestion_SpreadLogic_low but they didn't work successfully. both case also failed by hold time violation.
I put the error message in binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/runme.log
INFO: [Common 17-83] Releasing license: Implementation
342 Infos, 160 Warnings, 2 Critical Warnings and 0 Errors encountered.
route_design completed successfully
route_design: Time (s): cpu = 08:58:15 ; elapsed = 02:44:19 . Memory (MB): peak = 13644.312 ; gain = 1789.000 ; free physical = 7896 ; free virtual = 49438
source /home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/scripts/_full_route_post.tcl
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7893 ; free virtual = 49435
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7880 ; free virtual = 49421
get_cells: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7878 ; free virtual = 49420
INFO: [Timing 38-480] Writing timing data to binary archive.
Writing placer database...
Writing XDEF routing.
Writing XDEF routing logical nets.
Writing XDEF routing special nets.
Write XDEF Complete: Time (s): cpu = 00:01:26 ; elapsed = 00:00:29 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 6089 ; free virtual = 49194
INFO: [Common 17-1381] The checkpoint '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/dr_routed_timing.dcp' has been generated.
write_checkpoint: Time (s): cpu = 00:02:30 ; elapsed = 00:01:27 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7511 ; free virtual = 49387
report_timing_summary: Time (s): cpu = 00:00:33 ; elapsed = 00:00:16 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7369 ; free virtual = 49244
report_timing_summary: Time (s): cpu = 00:00:47 ; elapsed = 00:00:18 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 7368 ; free virtual = 49244
ERROR: [runtcl-1] design did not meet timing - hold violation
INFO: [Timing 38-480] Writing timing data to binary archive.
Writing placer database...
Writing XDEF routing.
Writing XDEF routing logical nets.
Writing XDEF routing special nets.
Write XDEF Complete: Time (s): cpu = 00:01:26 ; elapsed = 00:00:29 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 5578 ; free virtual = 49017
INFO: [Common 17-1381] The checkpoint '/home/yamada/tkhome/ghq/github.com/Xilinx/Vitis-AI/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/prj/prj.runs/impl_1/zcu102_base_wrapper_routed_error.dcp' has been generated.
write_checkpoint: Time (s): cpu = 00:02:31 ; elapsed = 00:01:28 . Memory (MB): peak = 13644.312 ; gain = 0.000 ; free physical = 6990 ; free virtual = 49199
ERROR: [Common 17-39] 'send_msg_id' failed due to earlier errors.

INFO: [Common 17-206] Exiting Vivado at Fri Aug  7 14:25:12 2020...

Hi @tkclimb
You can open the comment "#param=compiler.skipTimingCheckAndFrequencyScaling=1" in the prj_config file. it will not check the timing of project.
But I wonder do you change anything of the platform or do you use the vitis 2020.1? There should be not the timing issues integrating 2 B4096 DPU.

Hi, @qianglin-xlnx
I'll try your suggestion, but I didn't change anything from the commit id 524f65224c52314155dafc011d488ed30e458fcb.

Hello, i am getting the same issue. I tried building the Vitis-AI using 2019.2 version. Vitis_Embedded_Platform_Source (checkout 2019.2) and then Vitis-AI (latest version).

Hi @tkclimb @aerovish30
Please modify the Makefile to use the prj_config, as shown in the following. It will integrate 2 DPU core and there will be no time violation issue.

Hi @tkclimb ,
did you manage to solve the problem? I currently have the same issue. I want to compile the project with config_file prj_config_102_3dpu_LPD because I want to re-generate the standard Xilinx DPU-image for research purposes and I run into the same timing error as you wrote in your first post.

I tried different implementation strategies as @qianglin-xlnx proposed. For me with Congestion_SpreadLogic_low the DPU could successfully be built.

Hi @tkclimb
Since we haven't received your reply for a long time, I'm going to close this issue. If you still have any questions, please feel free to reopen it. Thank you very much.


Split stable and develop helm chart repositories

Introduce separate repositories for stable and unstable charts of Airy Core, the prerequisites and the tools.
Instead of having them behind an S3 url (https://airy-core-helm-charts.s3.amazonaws.com/stable), put them behind a more user-friendly hostname (example: https://helm.airy.co and https://helm.airy.co/develop).

is this done @ljupcovangelski can we close it?


[edpm_deploy_baremetal] Allow passing extra nova-compute config

Similar to what done for edpm_deploy[1], required changes on install_yamls side already covered with [2]
Needed by: https://review.rdoproject.org/r/c/testproject/+/54904
[1] https://github.com/openstack-k8s-operators/ci-framework/pull/941
[2] https://github.com/openstack-k8s-operators/install_yamls/pull/729

Tested in https://review.rdoproject.org/r/c/testproject/+/54904

/approve

/approve
/lgtm


Compressed contigs don't work in upload

Annotation service fails with compressed upload files.

any update on this?

I just tested this today and it worked.


Documentation error typo fix: s/Used-defined/User-Defined

Used -> User

Had to disable checks: @travis-ci was stuck. Merging manually, then re-enabling them.

Thanks @vladyslavstartsev


Does not work with Lucene 8.8.1?

I am trying to access an index folder created by Lucene 8.8.1 and getting the error "Not a valid lucene index or corrupted". In the log I found the following:
[2021-02-25T10:57:20,366]  WARN (IndexUtils.java:85) - Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/srv/bfabric/as/payara-5.2020.6/glassfish/domains/domain1/config/b-fabric-index/segments_37"))): 10 (needs to be between 7 and 9)
org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/srv/bfabric/as/payara-5.2020.6/glassfish/domains/domain1/config/b-fabric-index/segments_37"))): 10 (needs to be between 7 and 9)

Does this means I cannot use Luke with Lucene 8.8.x or i am doing something wrong?
If Luke cannot be used with Lucene 8.8.x which version is the latest that can be used?

As of 8.1 Luke has been bundled Lucene itself. Please download Lucene binary package; it includes luke module.

@mocobeta many thanks. this one works.


[topcrash] Fix a typo in a function name

Checklist

[ ] Type annotations added to new functions
[ ] Docs added to functions touched in main classes
[ ] Dry-run produced the expected results
[ ] The to-be-announced tag added if this is worth announcing

Ah-ha! I thought you were using some linter to find spelling mistakes in identifiers


Add tekton operator to local dev cluster

Description
This PR introduces the Tekton operator with a TektonConfig to replace the existing Tekton pipelines, triggers, and dashboard.
Known Issue:
There is currently a bug where we are only waiting for 120 seconds for the webhooks, instead of checking if they have reached a running/ready status, so there is a possibility that the tekton-webhooks are in an unready state when trying to deploy the tasks, pipelines and triggers.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)

Checklist

[x] PR is contained in a single commit, using git amend see some docs

This change is

Is this PR better @MarcusSorealheis ?

@SchahinRohani Could you rebase this? It's likely more reliable than the current setup and will make it easier for me to move it over to flux.


Error

We will fix it

We’ve released a new update, and everything should be working fine now.


Adds application integration for stand-alone usage to the documentation

Related to https://github.com/zendframework/documentation/issues/8

@weierophinney
The idea based on various support questions and was first test during the rework of zend-i18n documentation.
The preview of zend-i18n goes one step further and contains one more example: "Usage in a Slim Application"

An example which illustrates the usage with a layout script should be added.

This repository has been closed and moved to laminas/laminas-view; a new issue has been opened at https://github.com/laminas/laminas-view/issues/4.

This repository has been moved to laminas/laminas-view. If you feel that this patch is still relevant, please re-open against that repository, and reference this issue. To re-open, we suggest the following workflow:

Squash all commits in your branch (git rebase -i origin/{branch})
Make a note of all changed files (`git diff --name-only origin/{branch}...HEAD
Run the laminas/laminas-migration tool on the code.
Clone laminas/laminas-view to another directory.
Copy the files from the second bullet point to the clone of laminas/laminas-view.
In your clone of laminas/laminas-view, commit the files, push to your fork, and open the new PR.
We will be providing tooling via laminas/laminas-migration soon to help  automate the process.


Add element.io

Looking at their pricing page, it's $5 for the Business plan, and that includes SAML. I suspect pricing moved on while awaiting a merge, so let's call this a W, even if that price is higher than the SSO price listed in the PR.


Front-end #2: UI Bug Fixing

Known bugs:

Enter key currently doesn't work. Need fix (bound to Ask button?) <------- Works now
Advanced Ask (without Local RAG) currently doesn't work

The settings and Source button currently doesn't work after the latest update. Need checking and fixing


Gameplay bug

When moving and pushing a block the enemies won't die to it.

We've been able to reproduce the bug, we'll look into it


Add additional worker to test in FakeWorkerModule

Goal
Add an additional worker that we can tweak the execution via a blocking executor. This is needed for testing the delivery service components
This PR was hijacked for this as the actual scheduling service code in previous revisions that were under discussion will come in subsequent PRs.

#1384  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bidetofevil and the rest of your teammates on  Graphite

Merge activity

Sep 25, 10:59 AM EDT: @bidetofevil started a stack merge that includes this pull request via Graphite.


Extract http-bridge integration tests to separate travis job to overcome 50min time limit for a job

Issue Number

Overview

[ ] I have extracted http-bridge integration tests to separate travis job

Comments
It seems that tests added in #608 are causing Test travis jobs go beyond the 50 minutes time limit on merges to master (https://travis-ci.org/input-output-hk/cardano-wallet/builds/574426866). I extracted http-bridge integration to separate job to, hopefully, overcome this.
:warning: I don't know how it will affect coverage. I suppose having all tests in one job was for a reason, so it may be that coverage for http-bridge from integration tests will not be counted in :warning:

It's tempting to merge and see what happens on the basis that we have nothing to lose, but I fear there's actually a possibility we make things worse by having a successful master-build that messes up the cache… 🤔
Maybe @rvl has some guidance on what to do?

:crossed_fingers:


⚠ Bofrost.at has degraded performance

In 20e7e5b, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2085 ms

Resolved: Bofrost.at performance has improved in 2a5d921.


Texture swizzling improvements

@cdavis5e
Thanks very much for contributing the texture swizzling enhancement. That was a huge endeavour and a significant enhancement.  Nice work!
In reviewing the improvements...and making them compatible with caching...I had a couple of design questions I thought I'd raise:

The auxiliary MTLBuffer is held by the MVKPipeline. But that one pipeline might be used in several renderings using different textures. This might set up a race condition where the GPU has not consumed the swizzle data for one texture before the CPU overwrites it with swizzle data for a different texture used in a later draw call.
I'm wondering if we could use MVKCommandEncoder::setVertexBytes(), etc. to dynamically set the swizzle content instead...similar to how MVKPushConstantsCommandEncoderState does. The SPIRV-Cross interaction would stay the same.

Texture swizzle slots between vertex and fragment shaders seem to overlap in the auxiliary buffer. Does this work if both the vertex and fragment shaders use textures with different swizzles?

Currently swizzle_texture_samples is always on for any shader that samples a texture. Shall we add a setting to MVKConfiguration to control this...to allow apps that know they don't need swizzling to turn it off to save on shader complexity and improve shader performance?

If these suggestions make sense to your overall design for texture swizzling...I can implement them.

The auxiliary MTLBuffer is held by the MVKPipeline. But that one pipeline might be used in several renderings using different textures. This might set up a race condition where the GPU has not consumed the swizzle data for one texture before the CPU overwrites it with swizzle data for a different texture used in a later draw call.

I see. Since this is state that depends on the bound descriptors (and later, the current draw parameters), it's not really viable to keep it in an MTLBuffer owned by the pipeline and shared between multiple draws.

I'm wondering if we could use MVKCommandEncoder::setVertexBytes(), etc. to dynamically set the swizzle content instead...similar to how MVKPushConstantsCommandEncoderState does. The SPIRV-Cross interaction would stay the same.

We may be forced to do that. One concern is that the space available with that method is limited: Apple's docs (which are probably accurate for iOS; dunno about Mac) state that this method "is appropriate only for small, one-time-use data buffers. Create a MTLBuffer object instead if your buffer exceeds 4 KB in length..." It's not a concern now, since the 512 bytes needed to represent the component mappings for all 128 textures allowed on Mac should fit easily inside that limit, but it will become one when it comes time to support the VK_EXT_descriptor_indexing extension, which mandates a 500000-image minimum limit. (This whole scheme needs rethinking for that, since we'll need to use indirect argument buffers for that.)

Texture swizzle slots between vertex and fragment shaders seem to overlap in the auxiliary buffer. Does this work if both the vertex and fragment shaders use textures with different swizzles?

This will almost certainly break in that case. It is now obvious to me that each shader stage that needs an auxiliary buffer needs its own auxiliary buffer. (D'oh!)

Currently swizzle_texture_samples is always on for any shader that samples a texture. Shall we add a setting to MVKConfiguration to control this...to allow apps that know they don't need swizzling to turn it off to save on shader complexity and improve shader performance?

It doesn't help my employer's case--I doubt programs designed for Windows are going to be checking for a Mac/iOS-specific extension--but I agree this is probably a worthwhile optimization for programs that we know won't need this.

If these suggestions make sense to your overall design for texture swizzling...I can implement them.

Feel free. Unfortunately, I'm not going to have much time to work on this for the next couple of weeks. My employer has other tasks for me to do. To be honest, I was able to submit the original change because I already had it ready to go. Since it required a SPIRV-Cross update (and therefore, a bump to the MoltenVK version), I wanted to withhold it until I was done updating SPIRV-Cross to support geometry and tessellation shaders. Then I could update SPIRV-Cross once to pull in those changes as well. But, since my free time is limited for now, I probably won't be finishing that for another few weeks.

Sounds good. I'll look into the changes suggested above.

but it will become one when it comes time to support the VK_EXT_descriptor_indexing extension, which mandates a 500000-image minimum limit

When this becomes necessary...MoltenVK can allocate pools of MTLBuffers of various sizes for temporary use. See MVKCommandEncodingPool::acquireMTLBufferAllocation() and it's use in MVKCommandEncoder::copyToTempMTLBufferAllocation().

@cdavis5e @danginsburg (and others)
TL;DR: Should texture swizzling via shaders be on or off by default? And should we set this via an environment variable to allow any game to be easily ported with either requirement?

I'd like to get your opinions.

It doesn't help my employer's case--I doubt programs designed for Windows are going to be checking for a Mac/iOS-specific extension--but I agree this is probably a worthwhile optimization for programs that we know won't need this.

Some games rely on texture swizzling...and to ease porting we should maximize compatibility and turn texture swizzling via shaders on. Other games ignore texture swizzling...and to ease porting we should maximize performance by turning texture swizzling via shaders off.
I'm looking at several couple of options for flipping it on or off:

Dynamically at pipeline compile via a new Vulkan flag in the VkPipelineShaderStageCreateFlags bit mask. This would allow a single game to choose when and where to turn it on at the most granular level possible.

Statically via a new config boolean in MVKConfiguration via vkGet/SetMoltenVKConfigurationMVK.

Statically via a runtime environment variable...to allow different games to turn it on or off via a launch script without having to special-case a MoltenVK-specific API call.

Any preferences? We could conceptually combine all three options.
Also...should texture swizzling via shaders be turned on or off by default? This seems to be a basic compatability-vs-performance question...but we need to pick one.
I've heard both arguments...compatibility so porting is seamless for those games that do rely on swizzling...and performance...since there are many games that never use texture swizzling...and would like to just run as fast as possible after being ported.
Thoughts and suggestions, please?

Thoughts and suggestions, please?

I can see both arguments.  I will say that Artifact paid a 2.5ms penalty on AMD D500 @ 1080p for the texture swizzle behavior which it does not need.  Other games may experience similar cost, which does paint MoltenVK in a negative light for the default behavior.
In a perfect world, in addition to configuration options, MoltenVK could only pay the cost on swizzles once it actually encounters a case where it's needed.  The tough thing here is that it's a cost paid by everyone all the time.  I'm assuming this option is not possible though or you already would have done it that way.
So that being the case, @billhollings feel free to make the call as you see fit.  If you want my vote it would be:

Off by default
Clear MoltenVK error warning messages when an unsupported swizzle is encountered saying "hey I hit a swizzle I can't support by default, set this configuration option to support it"
In the Vulkan Portability Initiative we make this a configuration option so that there is a standardized way to enable it once that spec is done.

I would turn it off by default.
The new texture swizzle enhancement has a performance penalty even if not being used at all. In my eyes it's a waste of CPU / GPU performance that could be used in other places.
Our games are usually pixel shader limited already, so adding this specific feature adds even more overhead.
Since Mac OS devices have on average much slower GPU's, one should try to minimize those unnecessary performance hits. We already have a hard time getting performance right on Mac OS due to the big hardware differences.
So I would opt for 'Statically via a new config boolean in MVKConfiguration...'

@cdavis5e @danginsburg @aerofly
Thanks for the feedback. Okay...when I make the changes above...I'll turn it off by default...and add text to the swizzle error that mentions it can be turned on.
Is there any merit to the idea of allowing it to be set via an environment variable setting...say from within a script that launches the game on macOS?
That way...a single, default-compiled libMoltenVK.dylib could support different settings for different games...and the game does not have to special-case a platform-specific config API call in the code base.
It seems to me that would minimize porting code changes...and make it easier to change from one option to the other based on runtime environment.

I'm in favor of there being an environment variable.

Envvar is fine by me too.

I think for Vulkan Portability it would be preferable to avoid an environment variable and use an extension instead -- otherwise all other portability implementations will have to copy this environment variable setup in order to be compatible. This is already a bit unusual because we've implemented MVK* extensions.

@grovesNL
Thanks...yeah...I've raised this with the Vulkan portability TSG as well...so we can standardize.

@billhollings , I'm trying to use swizzles. I set the components field of my VkImageViewCreateInfo to the swizzle I want. I then got the warning described here which led me to set the environment variable. The warning disappeared but the swizzle isn't happening. I have an R8_UNORM texture, am setting a swizzle of RRR1 but the texture is displayed as red. Is there something else I need to do to enable swizzling?


Docker unable to use applications which use SIOCGIFNAME

Hi,
Docker version 1.9.1, build a34a1d5
Os Rhel 7
Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Trying to run some applications in docker which do
ioctl(3, SIOCGIFNAME, {ifr_index=1, ifr_name="lo"}) = 0
and
ioctl(3, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr=00:00:00:00:00:00})
these commands fail when the command get past the loop back device
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
ioctl(3, SIOCGIFNAME, {ifr_index=0, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=1, ifr_name="lo"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr=00:00:00:00:00:00}) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=2, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=3, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=4, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=5, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=6, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=7, ifr_name=???}) = -1 ENODEV (No such device)
The application wants to get the SIOCGIFNAME of eth0 and its SIOCGIFHWADDR
Expected output is something like
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
ioctl(3, SIOCGIFNAME, {ifr_index=0, ifr_name=???}) = -1 ENODEV (No such device)
ioctl(3, SIOCGIFNAME, {ifr_index=1, ifr_name="lo"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr=00:00:00:00:00:00}) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=2, ifr_name="eno1"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="eno1", ifr_hwaddr=MAC_ADDRESS}) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=3, ifr_name="enp1s0"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="enp1s0", ifr_hwaddr=MAC_ADDRESS}) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=4, ifr_name="local"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="local", ifr_hwaddr=MAC_ADDRESS) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=5, ifr_name="virbr1"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="virbr1", ifr_hwaddr=MAC_ADDRESS) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=6, ifr_name="virbr1-nic"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="virbr1-nic", ifr_hwaddr=MAC_ADDRESS}) = 0
ioctl(3, SIOCGIFNAME, {ifr_index=7, ifr_name="virbr0"}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="virbr0", ifr_hwaddr=MAC_ADDRESS}) = 0
close(3)

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
For more information about reporting issues, see CONTRIBUTING.md.
You don't have to include this information if this is a feature request
(This is an automated, informational response)

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

https://lists.linuxcontainers.org/pipermail/lxc-users/2012-May/003672.html seems relavent.

@cpuguy83 Do you know if this is still issue?

I really don't know anything about this, maybe someone from libnetwork can help?

ping @aboch

@owenhaynes The interface indices of the containers' interfaces are not contiguos, as from the discussion @cpuguy83 linked.
And they seem to increment at each container run:
$ docker run --rm busybox ip link show eth0
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff
$ docker run --rm busybox ip link show eth0
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff
}

Also take a  look at http://stackoverflow.com/questions/14264371/how-to-get-nic-details-from-a-c-program where they suggest to first retrieve a list of interfaces then iterate through their indices.

From what I understand, I do not think this is a docker issue.

Thanks for looking @aboch.
Going to close this one out, but feel free to continue the discussion.
Thanks!


MODOAIPMH-36 Implement verb "ListIdentifiers"

The main logic implementation
Validation of the parameters (MODOAIPMH-37)
Handle default set (MODOAIPMH-39)
Migrating all unit tests to junit5 and removing junit4 dependency

The PR #14 cannot be re-opened so opening new one.
The unit tests resolved on unix system as well (on windows they were passing for some reason without any issues).
PR is ready for review. The only change left is to add static method to initialize helpers which is dept from PR #13.


Publish to a public bucket once the main pipeline goes green

For now we'll use the same bucket, but publish to a different prefix /public/
This prefix allows publich object get via bucket policy
According to the docs, we can also pass in parameters to the stack via URL, which I try to do ... but for some reason this does not work for me :shrug:

@hoegaarden , please resolve the conflict and merge in.


cannot import name '_is_windows'

报错：
ImportError: cannot import name '_is_windows' from 'agentscope.utils.tools' (/home/apienv/lib/python3.10/site-packages/agentscope/utils/tools.py)
version: 0.0.5a1

_is_windows is introduced in 0.0.6. Please install the latest version of Agentscope. You can install AgentScope using the latest source code by pip install -e .


NMODL throws for x' = 1/x.

I tried compiling the following mod file:
NEURON {
    SUFFIX leonhard
}

STATE {
 x
}

INITIAL {
  x = 42
}

BREAKPOINT {
    SOLVE dX METHOD cnexp
}

DERIVATIVE dX {
 x' = 1.0/x
}

It fails with:
terminate called after throwing an instance of 'std::runtime_error'
  what():  PRIME encountered during code generation, ODEs not solved?
make: *** [makemod2c_inc:3: leonhard.cpp] Error 134
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "/home/lucg/git/bbp/nmodl/venv/bin/nrnivmodl", line 115, in <module>
    subprocess.check_call([exe, *sys.argv[1:]])
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lucg/git/bbp/nmodl/venv/lib/python3.11/site-packages/neuron/.data/bin/nrnivmodl', '-nmodl', 'nmodl', 'leonhard.mod']' returned non-zero exit status 2.

indicating that NMODL is unable to solve the ODE, and fails to generate the code required to solve the ODE approximately. Similar for x' = sin(x*x).

The behaviour of nrnivmodl is to generate code that solves the ODE via a numerical solver.


Pixels flip error on GL

Description
Renderdoc capture GL app and generate thumbnail pic. Thumbnail's flip is wrong, so middle two line mismatch.
Steps to reproduce
1 capture an GL app
2 convert to xml+zip
3 open thumbnail pic and you will find middle two line mismatch
Environment

RenderDoc version:  newest
Operating System: win
Graphics API: OGL

Example:

in core.cpp, in ResamplePixels function,  change  "for(uint16_t y = 0; y <= out.height / 2; y++)" into "for(uint16_t y = 0; y <= out.height / 2 - 1; y++)"


[#cleanup] cleanup duplicate code

Codecov Report

Merging #6233 into master will increase coverage by 0.02%.
The diff coverage is 48.68%.

@@            Coverage Diff             @@
##           master    #6233      +/-   ##
==========================================
+ Coverage   39.52%   39.54%   +0.02%
==========================================
  Files        3051     3051
  Lines       91609    91429     -180
  Branches    11932    11884      -48
==========================================
- Hits        36205    36157      -48
+ Misses      52396    52295     -101
+ Partials     3008     2977      -31

Impacted Files
Coverage Δ

...ugin/json_lib/interceptor/ToStringInterceptor.java
0% <0%> (ø)
:arrow_up:

...lugin/json_lib/interceptor/ParsingInterceptor.java
0% <0%> (ø)
:arrow_up:

...netty/interceptor/BootstrapConnectInterceptor.java
0% <0%> (ø)
:arrow_up:

...kson/interceptor/WriteValueAsBytesInterceptor.java
0% <0%> (ø)
:arrow_up:

...erceptor/ChannelPromiseAddListenerInterceptor.java
0% <0%> (ø)
:arrow_up:

.../plugin/fastjson/interceptor/ParseInterceptor.java
83.33% <77.77%> (+20.37%)
:arrow_up:

...n/fastjson/interceptor/ToJsonBytesInterceptor.java
88.88% <83.33%> (+26.38%)
:arrow_up:

.../fastjson/interceptor/ToJsonStringInterceptor.java
88.88% <83.33%> (+26.38%)
:arrow_up:

...plugin/fastjson/interceptor/ToJsonInterceptor.java
88.88% <83.33%> (+26.38%)
:arrow_up:

...in/fastjson/interceptor/ParseArrayInterceptor.java
88.88% <83.33%> (+26.38%)
:arrow_up:

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1aca83d...90ba17a. Read the comment docs.


Lua ACL plugin doesn't work for MQTT 5

VerneMQ Version: Latest (12.06.2021)
Docker image: latest-alpine

I try to connect with hivemq's client and it doesn't fail only when I explicitly set MQTT version to 3 in the client.
Otherwise I see
can't authenticate client {[],<<"someclient">>} from ... due to plugin_chain_exhausted
It's as if my Lua, auth plugin wasn't there for mqtt 5, but I can't find any config options to fix this.
My config:
           - "DOCKER_VERNEMQ_ACCEPT_EULA=yes"
           - "DOCKER_VERNEMQ_ALLOW_ANONYMOUS=off"
           - "DOCKER_VERNEMQ_vmq_diversity.auth_plugin.file=/plugins/auth_plugin.lua"
           - "DOCKER_VERNEMQ_log.console.level=debug"
           - "DOCKER_VERNEMQ_listener.tcp.private=<IP_ADDRESS>:1884"
           - "DOCKER_VERNEMQ_listener.tcp.allowed_protocol_versions=3,4,5"
           - "DOCKER_VERNEMQ_listener.ssl.public=<IP_ADDRESS>:8883"
           - "DOCKER_VERNEMQ_listener.ssl.allowed_protocol_versions=3,4,5"
           - "DOCKER_VERNEMQ_listener.ssl.cafile=/etc/ssl/test_root_ca_cert.pem"
           - "DOCKER_VERNEMQ_listener.ssl.certfile=/etc/ssl/broker_cert.pem"
           - "DOCKER_VERNEMQ_listener.ssl.keyfile=/etc/ssl/broker_key.pem"
           - "DOCKER_VERNEMQ_plugins.vmq_acl=off"
           - "DOCKER_VERNEMQ_plugins.vmq_passwd=off"
           - "DOCKER_VERNEMQ_plugins.vmq_diversity=on"

Thanks!

Sorry, I found it - there are separate hooks for MQTT 5:
    auth_on_register_m5
    auth_on_publish_m5
    auth_on_subscribe_m5


Magic Shortcuts for Markdown

Feature Request 🛍
I like not having to move my cursor when I write text and one way to do that is to use markdown things in your text and have the editor automatically convert them as you write. Am I the only one who likes that kind of editing?
Demoed here on the editor from Slab: https://slab.com

Use Case
As a user who understands markdown, I like to be able to type markdown in the editor and have it converted for me to WYSIWYG.
Possible Solution
Look at sequences of characters during editing and detect markdown things in 'em.
Notes
The example above was written without moving the cursor to go to the toolbar, or having to select parts of the text, but only by typing.

@jeromesimeon I would like to implement this one.

This seems to be lots of small features. Should this be all in one go or should we go one by one?

@d-e-v-esh I agree, this will be incremental. I think we should go one by one and use this Issue to track the changes.

@jeromesimeon @irmerk I created a PR #288 a while ago that does exactly the same thing. Maybe we can build on top of that to implement these features after that is merged.

To reiterate from my comment in #288, that's a good start to build on for this Issue.


Add Noti 0.1.5

Noti is a pushbullet client for macOS. It allows you to see and act upon your Android notifications from macOS.
Homepage: https://noti.center/

[x] brew cask audit --download {{cask_file}} is error-free.

[x] brew cask style --fix {{cask_file}} reports no offenses.

[x] The commit message includes the cask’s name and version.

[x] Named the cask according to the token reference.

[x] brew cask install {{cask_file}} worked successfully.

[x] brew cask uninstall {{cask_file}} worked successfully.

[x] Checked there are no open pull requests for the same cask.

[x] Checked that the cask was not already refused in closed issues.

Regarding the build check, does it really matter that a brew formula already exists? I thought cask & brew used a separate repo?
Eitherway, the one in brew is not related to my project. I can possibly rename it, but would rather not.

Regarding the build check, does it really matter that a brew formula already exists?

Only if the apps related, which is not the case here.


"Unknown" not supported if timestamp expected

So far no lightnings since connecting to HomeAssitant - but it seems this is not expected:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/sensor/init.py", line 388, in state
value = ciso8601.parse_datetime(str(value))
ValueError: Invalid character while parsing year ('u', Index: 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 492, in async_update_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 538, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 511, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/init.py", line 390, in state
raise ValueError(
ValueError: Invalid date/datetime: sensor.last_lightning_strike provide state 'unknown', while it has device class 'timestamp'

I belive I have a variation on this error
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 381, in state
    if value.tzinfo is None:
AttributeError: 'str' object has no attribute 'tzinfo'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 526, in async_update_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 572, in _async_write_ha_state
    state = self._stringify_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 545, in _stringify_state
    if (state := self.state) is None:
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 392, in state
    raise ValueError(
ValueError: Invalid datetime: sensor.last_lightning_strike has a timestamp device class but does not provide a datetime state but

I also have this error
Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 381, in state if value.tzinfo is None: AttributeError: 'str' object has no attribute 'tzinfo' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 540, in async_update_ha_state self._async_write_ha_state() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state state = self._stringify_state() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state if (state := self.state) is None: File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 392, in state raise ValueError( ValueError: Invalid datetime: sensor.last_lightning_strike has a timestamp device class but does not provide a datetime state but <class 'str'>


How to render a observation-dependent policy?

In the current documentation, in order to render an environment with a policy, the policy function must be provided to the launcher:
def random_policy(time_step):
  del time_step  # Unused.
  return np.random.uniform(low=action_spec.minimum,
                           high=action_spec.maximum,
                           size=action_spec.shape)

viewer.launch(env, policy=random_policy)

However, this policy can only takes as argument the timestep and not the observations from the environment.
How can we render an environment policy whose output is determined by the observation of each timestep rather than just a timestep? This is needed for evaluating non-deterministic environments where we can not evaluate first, create a table and render after.

Nevermind, timestep is not the time steo but the dm_control class TimeStep which contains the observation vector.


How to share Video in screen share with participants.

Hi I am using your sdk into iOS to share screen with participants using video call with source ScreenCapturer everything is working fine screen is share with participants but when I play video then participants did not get video they get only blank screen but when I stop video and come back any other page it shows fine.
please help. Thanks.

Hi @rashidabbas,
Are you trying to play video that is copy protected? For security reasons, ReplayKit will not allow you to capture video that is protected by FairPlay DRM (the same video would also require HDCP to be transmitted over an HDMI connection).
We've listed this as a known issue in the example, since Apple and not Twilio controls what kind of content is allowed to be captured by ReplayKit.
Best,
Chris

Hi @charliesantos ,
I am playing youtube videos into browser and I am unable to show video into participants device it shows blank screen.

Hi @rashidabbas,
Are you able to play videos in the YouTube iOS app? Is it just Safari where you're having the problem?
Best,
Chris

I am unable to share screen with any participant. Please guide me as I am broadcasting the screen perfectly using extension but I am stuck on how to view the broadcasting video in that room?????

Hi @hammas1993,
We don't have a dedicated "viewer" role in the ReplayKit example. You can use most of our other example apps to view the extension's broadcast (like the QuickStart app). The only thing to note is that these apps always connect with Tracks. If your Participant is just a passive viewer, you should ensure that you don't provide any Tracks in ConnectOptions.
Best,
Chris

Which code should i use? I have used replaykit to broadcast please specify
the project name for eg like uikit?
On Wed, Jul 24, 2019, 12:01 AM Christopher Eagleston <
<EMAIL_ADDRESS>wrote:

Hi @hammas1993 https://github.com/hammas1993,
We don't have a dedicated "viewer" role in the ReplayKit example. You can
use most of our other example apps to view the extension's broadcast (like
the QuickStart app
https://github.com/twilio/video-quickstart-ios/tree/master/VideoQuickStart).
The only thing to note is that these apps always connect with Tracks. If
your Participant is just a passive viewer, you should ensure that you don't
provide any Tracks in ConnectOptions.
Best,
Chris
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/twilio/video-quickstart-ios/issues/382?email_source=notifications&email_token=ADT2U2LQLVX2DU7H7W6MVLTQA5IRBA5CNFSM4HI2MFXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UDN2Q#issuecomment-514340586,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADT2U2IKQAIWIPXNQN5Y5WLQA5IRBANCNFSM4HI2MFXA
.

I suggest using: https://github.com/twilio/video-quickstart-ios/tree/master/VideoQuickStart

when I start video Recording from sample handler.swift in Room "Test" the broadcasting starts. But when I attempt to join  Room "Test" from VideoQuickStart  broadcasting disconnects??? Why

You need a unique access token with a unique identity for each Participant.

@ceaglest except video in youtube app or any browser participants did not receive video with voice .participant receive blank screen

Hey @rashidabbas,
The YouTube app has a custom video playback pipeline and doesn't use copy protection for their video content. If you have content that is copy protected (e.x. a Netflix video stream) then it will not be possible to capture it using ReplayKit. If you are using AVPlayer, you can check the AVAsset.hasProtectedContent property to see if the content can be copied by ReplayKit.
Closing this issue since there has been no activity since 2019. Let us know if you have any more questions.
Thanks,
Chris


Fix uses of SerializedValues::with_capacity in the driver code

The SerializedValues::with_capacity constructor takes a capacity argument which is supposed to represent the expected total size of the serialized values, but currently we are passing the expected value count, which is guaranteed to be at least 4x too small as each value takes up at least 4 bytes in the serialized form (but usually more, 4 bytes is just for encoding size).
I think we should either:

Get rid of with_capacity, or at least stop using it in the driver because the preallocated space will usually be orders of magnitude too small and it doesn't matter,
Add a method to Value interface which returns the expected serialized size,
Extend the Value interface so that serialize can write to other types than Vec<u8> - this way we could implement a CountingWriter which only counts bytes, then we could estimate the serialized size of Value by serializing the Value using CountingWriter first, which should be a fast operation for most of the types.

If we decide to go with points 2/3, we could implement them together - the method mentioned in 2. would have a default implementation described in 3.

We will most likely go with solution 2, but of course extend new serialization traits instead of Value.


SkyWalking Cross Process Propagation Headers Protocol V2

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ ] Bug
[ ] Requirement
[x] Feature or performance improvement

Requirement or improvement

Please describe about your requirements or improvement suggestions.

SkyWalking Cross Process Propagation Headers Protocol v2 is also named as sw6 protocol. This protocol keeps the same purposes of v1(a.k.a. sw3), which is keep context propagation works.

👍🏻👍🏻


Add theme support by using colors from the current ThemeData

If inBar is true, don't specify any colors -- current theme will be used automatically.
If inBar is false, use theme.iconTheme.color for back and clear icons, theme.textTheme.display1.color for text, and theme.canvasColor for background.
Remove colorBackButton -- color controlled by the theme.
Don't use _clearActive to determine the color of the clear button--onPressed determines the disabled/enabled state and color/disabledColor are used.

Light theme, inBar enabled

Light theme, inBar disabled

Dark theme, inBar enabled

Dark theme, inBar disabled

Thanks!


Attendance is calculated, not store in db?

Looking at https://github.com/carpentries/amy/issues/1177, is it right that attendance is now not being stored in the database but rather being calculated in the app?  I hadn't considered this when we discussed that issue, and am now seeing a potential problem.  We  run SQL queries in redash for a lot of our data needs. Since the database only has a "manual_attendance" column, we aren't getting the calculated attendance result.

Yes, attendance is calculated, not stored in DB. Only manual attendance is stored.

Closing this as there is nothing specifically to do about this - just something we will need to be aware of when we run attendance reports externally.  Also @sheraaronhurt  has said we are no longer requesting attendance data so this is not a priority.


No data shown in dashboard for speedtest or airgradient "Service unavailable"

Hey thx for creating the project again ;)
I seem to have an issue lately getting speedtest data in for the last month or two but i haven't investigated
I just finished creating my first AirGradient sensor today and toggled that feature on in the config and reran the ansible-playbook main.yml and both dashboards (Speedtest and Air sensors) show no data after 4 hours
How can I debug whats wrong?
I've tried deleting all containers in docker from the pi other than pi-hole and recreating the install but that didn't change anything
My config is almost stock other than specifying the Grafana version to fix the Annotation issue
I've changed the frequency of the updates for the speedtest to 180 minutes but other than that everything should be stock
Query inspector in Grafana shows Service Unavailable response being returned by the exporter service?
request:Object
url:"api/datasources/proxy/11/api/v1/query_range"
method:"POST"
data:Object
query:"pm02{job="airgradient"}"
start:1637232720
end:1637405520
step:120
hideFromInspector:false
response:Object
error:"Service Unavailable"
response:"Service Unavailable"
message:"Service Unavailable"

I've tested on the pi through ssh with curl localhost:9925/metrics
that returns
# HELP instance The ID of the AirGradient sensor.
# instance cfa389
# HELP wifi Current WiFi signal strength, in dB
# TYPE wifi gauge
wifi -69
# HELP pm02 Particulat Matter PM2.5 value
# TYPE pm02 gauge
pm02 0
# HELP rc02 CO2 value, in ppm
# TYPE rc02 gauge
rco2 772
# HELP atmp Temperature, in degrees Celsius
# TYPE atmp gauge
atmp 23.9
# HELP rhum Relative humidity, in percent
# TYPE rhum gauge
rhum 54

and those numbers change frequently so it seems the pi is getting the data through
config.yml
---
# Location where configuration files will be stored.
config_dir: '~'

# Pi-hole configuration.
pihole_enable: true
pihole_hostname: pihole
pihole_timezone: Australia/Sydney
pihole_password: "apassword"

# Internet monitoring configuration.
monitoring_enable: true
monitoring_grafana_adansible-playbook main.ymlmin_password: "admin"
monitoring_speedtest_interval: 180m
monitoring_ping_interval: 5s
monitoring_ping_hosts:  # [URL];[HUMAN_READABLE_NAME]
  - http://www.google.com/;google.com
  - https://github.com/;github.com
  - https://www.apple.com/;apple.com

# Shelly Plug configuration. (Also requires `monitoring_enable`)
shelly_plug_enable: false
shelly_plug_hostname: my-shelly-plug-host-or-ip
shelly_plug_http_username: username
shelly_plug_http_password: "password"

# AirGradient configuration. (Also requires `monitoring_enable`)
airgradient_enable: true

# Starlink configuration. (Also requires `monitoring_enable`)
starlink_enable: false

inventory.ini
[internet_pi]
#<IP_ADDRESS> ansible_user=pi

# Comment out the previous line and uncomment this to run inside Rasberry Pi.
<IP_ADDRESS> ansible_connection=local ansible_user=pi

I tried adding the ip for the pi to inventory but the ansible install ended with the following output
PLAY RECAP ***********************************************************************************************************************************************************************************************************************
<IP_ADDRESS>                  : ok=20   changed=2    unreachable=0    failed=0    skipped=15   rescued=0    ignored=0
<IP_ADDRESS>               : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Does anyone know where to look to find the issue please?

Checking the logs for the docker containers using sudo docker logs containeruuid --timestamps
container prom/prometheus:v2.25.2 is restarting often and showing lots of memory errors
2021-11-20T11:36:09.014038866Z panic: runtime error: invalid memory address or nil pointer dereference 2021-11-20T11:36:09.014358293Z [signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x17308a8] 2021-11-20T11:36:09.014428762Z  2021-11-20T11:36:09.014493762Z goroutine 779 [running]: 2021-11-20T11:36:09.033965282Z bufio.(*Writer).Available(...) 2021-11-20T11:36:09.034135491Z 	/usr/local/go/src/bufio/bufio.go:624 2021-11-20T11:36:09.034226116Z github.com/prometheus/prometheus/tsdb/chunks.(*ChunkDiskMapper).WriteChunk(0x4ef0b40, 0x306, 0x0, 0x5e7ca812, 0x17c, 0x5e85bc4a, 0x17c, 0x25f26d8, 0x4b39700, 0x0, ...) 2021-11-20T11:36:09.034295126Z 	/app/tsdb/chunks/head_chunks.go:291 +0x4f0 2021-11-20T11:36:09.034362574Z github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x45bce70, 0x4ef0b40) 2021-11-20T11:36:09.034427939Z 	/app/tsdb/head.go:2102 +0x6c 2021-11-20T11:36:09.034493564Z github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x45bce70, 0x5e85cfd2, 0x17c, 0x4ef0b40, 0x54e7780) 2021-11-20T11:36:09.034559397Z 	/app/tsdb/head.go:2076 +0x24 2021-11-20T11:36:09.034688095Z github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x45bce70, 0x5e85cfd2, 0x17c, 0x536cdbee, 0x3f43f50c, 0x0, 0x0, 0x4ef0b40, 0x56f0001) 2021-11-20T11:36:09.034751533Z 	/app/tsdb/head.go:2232 +0x3a0 2021-11-20T11:36:09.034811689Z github.com/prometheus/prometheus/tsdb.(*Head).processWALSamples(0x3cf8800, 0x5e7ca700, 0x17c, 0x58d0ec0, 0x58d0e80, 0x0, 0x0) 2021-11-20T11:36:09.034872470Z 	/app/tsdb/head.go:397 +0x284 2021-11-20T11:36:09.034931220Z github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func5(0x3cf8800, 0x4bb7b10, 0x4bb7b20, 0x58d0ec0, 0x58d0e80) 2021-11-20T11:36:09.034991793Z 	/app/tsdb/head.go:491 +0x40 2021-11-20T11:36:09.035051168Z created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL 2021-11-20T11:36:09.035109814Z 	/app/tsdb/head.go:490 +0x268
2021-11-20T11:30:53.826030985Z panic: mmap, size 134217728: cannot allocate memory 2021-11-20T11:30:53.826209631Z  2021-11-20T11:30:53.826512652Z goroutine 827 [running]: 2021-11-20T11:30:53.828210153Z github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x4ae6e70, 0x4986120) 2021-11-20T11:30:53.828296924Z 	/app/tsdb/head.go:2105 +0x228 2021-11-20T11:30:53.828362758Z github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x4ae6e70, 0x5e85cfd2, 0x17c, 0x4986120, 0x5d01440) 2021-11-20T11:30:53.828425987Z 	/app/tsdb/head.go:2076 +0x24 2021-11-20T11:30:53.828486977Z github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x4ae6e70, 0x5e85cfd2, 0x17c, 0x536cdbee, 0x3f43f50c, 0x0, 0x0, 0x4986120, 0x5160001) 2021-11-20T11:30:53.828550362Z 	/app/tsdb/head.go:2232 +0x3a0 2021-11-20T11:30:53.828611039Z github.com/prometheus/prometheus/tsdb.(*Head).processWALSamples(0x3d7e500, 0x5e7ca700, 0x17c, 0x5ce6c80, 0x5ce6c40, 0x0, 0x0) 2021-11-20T11:30:53.828673122Z 	/app/tsdb/head.go:397 +0x284 2021-11-20T11:30:53.828733018Z github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func5(0x3d7e500, 0x5104158, 0x5104160, 0x5ce6c80, 0x5ce6c40) 2021-11-20T11:30:53.828794789Z 	/app/tsdb/head.go:491 +0x40 2021-11-20T11:30:53.828853904Z created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL 2021-11-20T11:30:53.828914477Z 	/app/tsdb/head.go:490 +0x268 2021-11-20T11:31:03.497673928Z level=info ts=2021-11-20T11:31:03.496Z caller=main.go:404 msg="Starting Prometheus" version="(version=2.25.2, branch=HEAD, revision=bda05a23ada314a0b9806a362da39b7a1a4e04c3)" 2021-11-20T11:31:03.498200439Z level=warn ts=2021-11-20T11:31:03.497Z caller=main.go:406 msg="This Prometheus binary has not been compiled for a 64-bit architecture. Due to virtual memory constraints of 32-bit systems, it is highly recommended to switch to a 64-bit binary of Prometheus." GOARCH=arm
This issue says it its a problem when running 32 bit prometheus and can be fixed by migrating to 64 bit even when on 32 bit os like raspbian
I've removed the hard coded version from the yaml file for prometheus and changed it to prom/prometheus:latest and the memory errors are gone from the docker logs for now
Nothing coming up in the graphs yet though and it still says service unavailable

I think one of the WAL files is corrupt it seems to fail on loading segment 656 or 657 every time
Does anyone know how to clear these files? i cant seem to connect to the container it doesn't have bash shell

I managed to delete the volume and get this sorted with
docker volume ls
docker-compose down
docker volume rm internet-monitoring_prometheus_data
docker volume rm internet-monitoring_grafana_data
docker-compose up - d
running from ~/internet-monitoring folder
I couldnt find the volume on the sdcard but deleting it through docker worked fine
I can now see data starting to populate in the air sensor dashboard
The internet one will probably work in a few hours after the first test
Hopefully it wont reoccur when i hit 656 wal files again not sure why there were so many
Thanks


Fixed for formatting errors and uncompliant UTF8 documents

Some files had bad UTF-8 compliance or unicode but needed to be UTF-8 :

NBSP  U+00A0 NO-BREAK SPACE
THSP U+202F   NARROW NO-BREAK SPAC
among other nonstandard encoding for PDFkit to process.

I've used prettier to adjust the formatting on most filetypes. There might be some edge case situations that needs to be address

[x] Updated .gitignore
[x] Removed tracked files not needed
[x] Fixed formatting on incompliance in clone.ts
[x] Remove \t tabs and replace with  4 space
[ ] Test edge cases that prettier will not work

Hey @jbirbal-skydom, Thanks for the PR. Will go ahead and look into this later this week and go ahead and push after some testing. Again thanks for contributing.


[BUG] Not training algorithms that use the 'UserPolicyAgendaMultiWoz'

When training policies that use the UserPolicyAgendaMultiWoz such as PG, PPO and GDPL, it throws an Exception.
This Exception is related to the file convlab2/task/multiwoz/goal_generator.py line 181, where the self.corpus_path seems to be None instead of a path.
Indeed, in the UserPolicyAgendaMultiWozclass, we initialize the GoalGenerator with defaults values (corpus_path = None, included), and we do not set this anywhere.
To Reproduce
Steps to reproduce an example of the behavior (PPO policy):

Go to convlab2/policy/ppo
Run python train.py
It will throw an Exception as shown in the image below.

Expected behavior
It should train the policy with the respective algorithm (PPO in this case)
Additional context
It also happens when evaluating an policy using convlab2/policy/evaluate.py script.

I realized that the default goal_model_path in class  GoalGenerator changed to data/multiwoz/goal/new_goal_model_no_police_hospital.pkl which does not exist in directory data/multiwoz/goal.
One possible solution is to run the script convlab2/task/multiwoz/goal_generator.py to create this model path.


VM Migration - warnings in log

Bug Report
Version of OpenNebula

[ ] 5.2.2
[ ] 5.4.0
[ ] 5.4.1
[ ] 5.4.2
[X] 5.4.6
[ ] Development build

Component

[ ] Authorization (LDAP, x509 certs...)
[ ] Command Line Interface (CLI)
[ ] Contextualization
[ ] Documentation
[ ] Federation and HA
[ ] Host, Clusters and Monitorization
[X] KVM
[ ] Networking
[ ] Orchestration (OpenNebula Flow)
[ ] Packages
[ ] Scheduler
[ ] Storage & Images
[ ] Sunstone
[ ] Upgrades
[ ] User, Groups, VDCs and ACL
[ ] vCenter

Description
I see warnings in log when migrating VM
Sat Jan 27 19:14:45 2018 [Z0][VM][I]: New LCM state is RUNNING
Sat Jan 27 19:15:59 2018 [Z0][VM][I]: New LCM state is SAVE_MIGRATE
Sat Jan 27 19:16:03 2018 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/save: line 58: warning: command substitution: ignored null byte in input
Sat Jan 27 19:16:03 2018 [Z0][VMM][I]: ExitCode: 0
Sat Jan 27 19:16:03 2018 [Z0][VMM][I]: Successfully execute virtualization driver operation: save.
Sat Jan 27 19:16:04 2018 [Z0][VMM][I]: ExitCode: 0
Sat Jan 27 19:16:04 2018 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Sat Jan 27 19:16:04 2018 [Z0][VM][I]: New LCM state is PROLOG_MIGRATE
Sat Jan 27 19:16:04 2018 [Z0][VM][I]: New LCM state is BOOT_MIGRATE
Sat Jan 27 19:16:05 2018 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sat Jan 27 19:16:05 2018 [Z0][VMM][I]: ExitCode: 0
Sat Jan 27 19:16:05 2018 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sat Jan 27 19:16:08 2018 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/restore: line 43: warning: command substitution: ignored null byte in input
Sat Jan 27 19:16:08 2018 [Z0][VMM][I]: ExitCode: 0
Sat Jan 27 19:16:08 2018 [Z0][VMM][I]: Successfully execute virtualization driver operation: restore.
Sat Jan 27 19:16:09 2018 [Z0][VMM][I]: ExitCode: 0
Sat Jan 27 19:16:09 2018 [Z0][VMM][I]: Successfully execute network driver operation: post.
Sat Jan 27 19:16:09 2018 [Z0][VM][I]: New LCM state is RUNNING

This also happens on stop operation
Thu Apr 19 13:39:41 2018 [Z0][VM][I]: New LCM state is RUNNING
Thu Apr 19 13:45:27 2018 [Z0][VM][I]: New LCM state is SAVE_STOP
Thu Apr 19 13:45:29 2018 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/save: line 58: warning: command substitution: ignored null byte in input

Hello, this is still there, even on 5.6.1.
Fri Nov 23 15:55:31 2018 [Z0][VM][I]: New LCM state is SAVE_MIGRATE
Fri Nov 23 15:55:58 2018 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/save: line 58: warning: command substitution: ignored null byte in input
Fri Nov 23 15:55:58 2018 [Z0][VMM][I]: ExitCode: 0
Fri Nov 23 15:55:58 2018 [Z0][VMM][I]: Successfully execute virtualization driver operation: save.
Fri Nov 23 15:55:59 2018 [Z0][VMM][I]: ExitCode: 0
Fri Nov 23 15:55:59 2018 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Fri Nov 23 15:55:59 2018 [Z0][VM][I]: New LCM state is PROLOG_MIGRATE
Fri Nov 23 15:55:59 2018 [Z0][VM][I]: New LCM state is BOOT_MIGRATE
Fri Nov 23 15:55:59 2018 [Z0][VM][I]: Virtual Machine has no context
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: pre: Executed "sudo brctl addbr onebr18".
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: pre: Executed "sudo ip link set onebr18 up".
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: pre: Executed "sudo ip link add link team0 name team0.218 mtu 1500 type vlan id 218 gvrp on ".
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: pre: Executed "sudo ip link set team0.218 up".
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: pre: Executed "sudo brctl addif onebr18 team0.218".
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: ExitCode: 0
Fri Nov 23 15:56:00 2018 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Nov 23 15:56:24 2018 [Z0][LCM][I]: VM reported RUNNING by the drivers
Fri Nov 23 15:56:24 2018 [Z0][VM][I]: New LCM state is RUNNING
Fri Nov 23 15:56:27 2018 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/restore: line 43: warning: command substitution: ignored null byte in input
Fri Nov 23 15:56:27 2018 [Z0][VMM][I]: ExitCode: 0
Fri Nov 23 15:56:27 2018 [Z0][VMM][I]: Successfully execute virtualization driver operation: restore.
Fri Nov 23 15:56:28 2018 [Z0][VMM][I]: ExitCode: 0
Fri Nov 23 15:56:28 2018 [Z0][VMM][I]: Successfully execute network driver operation: post.
Fri Nov 23 15:56:28 2018 [Z0][LCM][E]: deploy_success_action, VM in a wrong state

Fixed in master and one-5.10 for 5.10.2


FCOS train loss NAN

hi, I train FCOS and Centernet use same dataset, Centernet sames ok while FCOS loss is NAN at the early beginning. I tried to use gradient clip,but the loss would keep high.

hi, I train FCOS and Centernet use same dataset, Centernet sames ok while FCOS loss is NAN at the early beginning. I tried to use gradient clip,but the loss would keep high.

can you chare how to train?

Hi @letcoN , I am a newbie. I have the same problem with the Centernet. Loss always nan. Can you share pre-trained weight and share how to train. I think maybe my step preparation fault. Thanks in advance!!


Remove unused semantic_version module

The semantic_version library has been added as a dependency but none of its features are used. Why adding a dependency when storing a simple string is just as good?

I'm also a +1 for removing a unused dependency. (NOTE: I thought "none of its features are used" meant that the dependency is unused.)
That said, I did a quick check for how many mentions of semantic_version show up in the codebase and I found six instances. The GitHub UI search functionality agrees and also adds the instance in which the dependency is mentioned in the pyproject.toml file.
$ grep -r -o "semantic_version" --include="*.py" | wc -l
6

This library does in fact appear to be used.
But perhaps your comment is really that the functionality of the semantic_version module is used too lightly. My scan suggests that only the Version class is used.
While I defer to you and others that have a history with this project, it does strike me that one benefit of using the semantic_version 3rd party library is that this library does do a check for whether the version actually follows the semantic version scheme. Whether that benefit outweighs the costs of adding a dependency, I defer to others.


CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Invalid configuration for device '11'

Recently we've been experiencing this issue with both bosh-init & bosh deploy with vsphere cpi where vm creating is failing during clone phase of cpi call.
Tried with multiple cpi versions , below debug logs are from cpi v38
For bosh-init
When deploying for the first time there are no issues, but when deploying again with some change in deployment manifest , bosh-init fails with device error.
If stemcell block is removed from state file and bosh-init deploy is fired again , everything goes smoothly, that means everytime fresh stemcell is uploaded it works.
While debugging , got this stack trace from bosh-init debug logs
I, [2017-03-02T03:46:09.504663 #18705]  INFO -- : Cloning vm: (VSphereCloud::Resources::VM (cid="sc-7b961d98-857b-4590-9ff5-8d2c020471e4")) to vm-78b3a003-3e92-46a5-96be-68579ce27a37
D, [2017-03-02T03:46:09.504803 #18705] DEBUG -- : Running method 'FindByInventoryPath'...
D, [2017-03-02T03:46:09.508589 #18705] DEBUG -- : Running method 'RetrieveProperties'...
D, [2017-03-02T03:46:09.513017 #18705] DEBUG -- : Running method 'CloneVM_Task'...
D, [2017-03-02T03:46:09.534150 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:09.539643 #18705] DEBUG -- : Starting task 'VirtualMachine.clone'...
D, [2017-03-02T03:46:09.540111 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:10.546503 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:11.553379 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:12.599832 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:14.184733 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:16.583993 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-03-02T03:46:20.216892 #18705] DEBUG -- : Running method 'RetrievePropertiesEx'...
W, [2017-03-02T03:46:20.225565 #18705]  WARN -- : Error running task 'VirtualMachine.clone'. Failed with message 'Invalid configuration for device '11'.'.

<RetrievePropertiesExResponse xmlns="urn:vim25"><returnval><objects><obj type="Task">task-322042</obj><propSet><name>info.descriptionId</name><val xsi:type="xsd:string">VirtualMachine.clone</val></propSet><propSet><name>info.entity</name><val type="VirtualMachine" xsi:type="ManagedObjectReference">vm-60846</val></propSet><propSet><name>info.error</name><val xsi:type="LocalizedMethodFault"><fault xsi:type="InvalidDeviceSpec"><property>virtualDeviceSpec.device.backing.parent.fileName</property><deviceIndex>11</deviceIndex></fault><localizedMessage>Invalid configuration for device '11'.</localizedMessage></val></propSet><propSet><name>info.name</name><val xsi:type="xsd:string">CloneVM_Task</val></propSet><propSet><name>info.state</name><val xsi:type="TaskInfoState">error</val></propSet></objects></returnval></RetrievePropertiesExResponse>
</soapenv:Body>
</soapenv:Envelope>
Rescued Unknown: Invalid configuration for device '11'.. backtrace: /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/retryer.rb:13:in `try'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/task_runner.rb:12:in `run'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/vcenter_client.rb:50:in `wait_for_task'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/vm_creator.rb:88:in `create'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/cloud.rb:259:in `block in create_vm'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/vendor/bundle/ruby/2.2.0/gems/bosh_common-1.3262.24.0/lib/common/thread_formatter.rb:49:in `with_thread_name'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/lib/cloud/vsphere/cloud.rb:206:in `create_vm'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/vendor/bundle/ruby/2.2.0/gems/bosh_cpi-2.0.1/lib/bosh/cpi/cli.rb:71:in `public_send'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/vendor/bundle/ruby/2.2.0/gems/bosh_cpi-2.0.1/lib/bosh/cpi/cli.rb:71:in `run'
/home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/packages/vsphere_cpi/bin/vsphere_cpi:42:in `<main>'
[File System] 2017/03/06 09:05:36 DEBUG - Remove all /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/jobs/vsphere_cpi
[File System] 2017/03/06 09:05:36 DEBUG - Remove all /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/blobs/5a85bf97-a29c-4bf6-40cc-ca4be17221a5
[File System] 2017/03/06 09:05:36 DEBUG - Remove all /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/tmp/stemcell-manager347632596
[File System] 2017/03/06 09:05:36 DEBUG - Remove all /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/tmp/bosh-init-release065385170
[File System] 2017/03/06 09:05:36 DEBUG - Remove all /home/.bosh_init/installations/fde44370-8f88-45df-7524-6d792d706eb1/tmp/bosh-init-release627218441
[main] 2017/03/06 09:05:36 ERROR - Command 'deploy' failed: Deploying: Creating instance 'bosh/0': Creating VM: Creating vm with stemcell cid 'sc-7b961d98-857b-4590-9ff5-8d2c020471e4': CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Invalid configuration for device '11'.","ok_to_retry":false}

mentioning <fault xsi:type="InvalidDeviceSpec"><property>virtualDeviceSpec.device.backing.parent.fileName</property><deviceIndex>11</deviceIndex> as invalid device spec
Going through few vmware docs , got the reference about the property being used when Performing Advanced Manipulation of Delta Disks in case of linked clones.
References:
Vmware linked clone notes
Vmware spec - parent property is described here in detail which is used during clone phase
CPI code for backing spec - As of now parent property is not added here in spec , as optional
Observed another thing which is probably cause of above issue

When updating vm with bosh-init , during delete phase , .vmdk file from base stemcell directory is also deleted , which shouldn't be the case because same vmdk should be used next time during vm creation phase.

Contents of stemcell dir on datastore after first Bosh-init deploy

Contents of stemcell dir on datastore after manifest update & Bosh-init deploy
Started deploying
  Waiting for the agent on VM 'vm-7cf258dd-85d8-4a6c-8267-fa75a07b4ddf'... Finished (00:00:00)
  Stopping jobs on instance 'unknown/0'... Finished (00:00:01)
  Unmounting disk 'disk-e8ebd8de-da10-4feb-ad9b-180efa333662'... Finished (00:00:01)
  Deleting VM 'vm-7cf258dd-85d8-4a6c-8267-fa75a07b4ddf'... Finished (00:01:01)
  Creating VM for instance 'bosh/0' from stemcell 'sc-4c722f61-e085-4bea-a377-4ab6518818fd'... Failed (00:02:16)
Failed deploying (00:03:23)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
  Deploying:
    Creating instance 'bosh/0':
      Creating VM:
        Creating vm with stemcell cid 'sc-4c722f61-e085-4bea-a377-4ab6518818fd':
          CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Invalid configuration for device '11'.","ok_to_retry":false}

@cdutra can you take a look at this?

@ronakbanka did you figure out if something is killing disks from iaas side?

@cppforlife didn't find any such thing which is killing disks , but this started happening after our IaaS upgraded from 5.5 to 6.

@ronakbanka how frequently does this happen? almost immediately and consistently? we run lifecycle tests against vsphere 6.0 env which seems to spin up multiple vms per stemcells.

Lifecycle specs run the following tags: https://github.com/cloudfoundry-incubator/bosh-vsphere-cpi-release/blob/e51dcb3d77f28397a7529991228e166acab650e5/ci/pipeline.yml#L64
In the Spec Helper for integration tests we upload a stemcell once: https://github.com/cloudfoundry-incubator/bosh-vsphere-cpi-release/blob/e51dcb3d77f28397a7529991228e166acab650e5/src/vsphere_cpi/spec/integration/spec_helper.rb#L17-L21
All create_vm calls from the CPI will use that stemcell for the rest of the Rspec run. This should exercise creating multiple VMs from a single stemcell image.

@cppforlife this happens very frequently in our setup. Do you guys test against multiple clusters?

@zaksoup Issue is not during create_vm stage but when a vm is being deleted , vsphere is deleting the .vmdk from base stemcell datastore dir , so next time during create_vm stage it fails as no base virtual disk.
We are experiencing this in vsphere 6 env but not in vsphere 5.5 .

@ronakbanka @CAFxX just created a story to try this manually: https://www.pivotaltracker.com/story/show/143189927.

@cppforlife maybe underline in the story that this happens with create-delete-create cycle (see screenshots and repro instructions in https://github.com/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/22#issue-212330458)

Hey @ronakbanka @CAFxX @liuweichu, I am working with @cunnie to reproduce your issue but we're having trouble. Can you please provide us with a copy of your (redacted) manifest and cloud-config?
Here's what we've done so far:

Deploy a bosh director using the latest bosh2 CLI to a vSphere 6.0 with this manifest (generated via bosh-deployment)
set the following cloud-configazs:
- name: z1
  cloud_properties:
    datacenters:
    - clusters: [scarlet-1: {}]

vm_types:
- name: default
  az: z1
  cloud_properties:
    cpu: 2
    ram: 1024
    disk: 3240

disk_types:
- name: default
  disk_size: 3000

networks:
- name: default
  type: manual
  subnets:
  - range: 10.xx.yy.0/24
    gateway: 10.xx.yy.1
    reserved: [10.xx.yy.2-10.xx.yy.249]
    az: z1
    dns: [<IP_ADDRESS>]
    cloud_properties:
      name: scarlet

compilation:
  workers: 5
  reuse_compilation_vms: true
  az: z1
  vm_type: default
  network: default

Upload the latest vSphere Ubuntu Stemcell
Deploy the following manifest---
name: repro-issue

releases: []

stemcells:
- alias: ubuntu
  os: ubuntu-trusty
  version: latest

instance_groups:
- name: dummy
  azs: [z1]
  instances: 2
  vm_type: default
  persistent_disk_type: default
  stemcell: ubuntu
  networks:
  - name: default
    default:
    - dns
    - gateway
# - name: transformers
  jobs: []

update:
  canaries: 1
  max_in_flight: 6
  serial: false
  canary_watch_time: 1000-60000
  update_watch_time: 1000-60000

Delete the deployment
Redeploy.

We were hoping that deleting the deployment and redeploying would exercise the create delete recreate cycle but we see the second deployment complete successfully.
Here is what the stemcell looks like in our vSphere:
Thanks!
Zak + Brian

@zaksoup @cunnie I am using bosh create-env with init configuration , issue is still there ,but i noticed one spec difference when asked out IaaS guys to give hostd logs from 2 vsphere env.
There is a difference in deviceChange.backing.parent , in vsphere 5.5 it is null where as in in vsphere 6 it points to stemcell vmdk , and the vmdk delete is occurring in vsphere 6 where this property is set in config.
one below is from vsphere 5.5
2017-04-05T06:58:39.746Z [3ED40B70 verbose 'Vcsvc.LLPM' opID=276a9a3b-16 user=vpxuser] LLPM reconfig (vim.vm.ConfigSpec) {
-->    dynamicType = <unset>,
-->    changeVersion = <unset>,
-->    name = <unset>,
-->    version = <unset>,
-->    uuid = <unset>,
-->    instanceUuid = <unset>,
-->    npivWorldWideNameType = <unset>,
-->    npivDesiredNodeWwns = <unset>,
-->    npivDesiredPortWwns = <unset>,
-->    npivTemporaryDisabled = <unset>,
-->    npivOnNonRdmDisks = <unset>,
-->    npivWorldWideNameOp = <unset>,
-->    locationId = <unset>,
-->    guestId = <unset>,
-->    alternateGuestName = <unset>,
-->    annotation = <unset>,
-->    files = (vim.vm.FileInfo) {
-->       dynamicType = <unset>,
-->       vmPathName = "[]/vmfs/volumes/53e466e2-887e4d38-b813-6c3be5b6d3f0/vm-2e860675-de64-4607-935a-d92022c1455c/vm-2e860675-de64-4607-935a-d92022c1455c.vmx",
-->       snapshotDirectory = <unset>,
-->       suspendDirectory = <unset>,
-->       logDirectory = <unset>,
-->       ftMetadataDirectory = <unset>,
-->    },
-->    tools = (vim.vm.ToolsConfigInfo) null,
-->    flags = (vim.vm.FlagInfo) null,
-->    consolePreferences = (vim.vm.ConsolePreferences) null,
-->    powerOpInfo = (vim.vm.DefaultPowerOpInfo) null,
-->    numCPUs = <unset>,
-->    numCoresPerSocket = <unset>,
-->    memoryMB = <unset>,
-->    memoryHotAddEnabled = <unset>,
-->    cpuHotAddEnabled = <unset>,
-->    cpuHotRemoveEnabled = <unset>,
-->    virtualICH7MPresent = <unset>,
-->    virtualSMCPresent = <unset>,
-->    deviceChange = (vim.vm.device.VirtualDeviceSpec) [
-->       (vim.vm.device.VirtualDeviceSpec) {
-->          dynamicType = <unset>,
-->          operation = "remove",
-->          fileOperation = "destroy",
-->          device = (vim.vm.device.VirtualDisk) {
-->             dynamicType = <unset>,
-->             key = 2000,
-->             deviceInfo = (vim.Description) {
-->                dynamicType = <unset>,
-->                label = "Hard disk 1",
-->                summary = "3,145,728 KB",
-->             },
-->             backing = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) {
-->                dynamicType = <unset>,
-->                fileName = "[]/vmfs/volumes/53e466e2-887e4d38-b813-6c3be5b6d3f0/vm-2e860675-de64-4607-935a-d92022c1455c/vm-2e860675-de64-4607-935a-d92022c1455c_1.vmdk",
-->                datastore = 'vim.Datastore:datastore-11786',
-->                backingObjectId = "111-2000-0",
-->                diskMode = "persistent",
-->                split = false,
-->                writeThrough = false,
-->                thinProvisioned = false,
-->                eagerlyScrub = <unset>,
-->                uuid = "6000C294-5167-070d-b38d-bfa7cadbb5e2",
-->                contentId = "166b589dceb61a8900586e0dd12f5239",
-->                changeId = <unset>,
-->                parent = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) null,
-->                deltaDiskFormat = "redoLogFormat",
-->                digestEnabled = false,
-->                deltaGrainSize = <unset>,
-->             },
-->             connectable = (vim.vm.device.VirtualDevice.ConnectInfo) null,
-->             slotInfo = (vim.vm.device.VirtualDevice.BusSlotInfo) null,
-->             controllerKey = 1000,
-->             unitNumber = 0,
-->             capacityInKB = 3145728,
-->             capacityInBytes =<PHONE_NUMBER>,
-->             shares = (vim.SharesInfo) {
-->                dynamicType = <unset>,
-->                shares = 1000,
-->                level = "normal",
-->             },
-->             storageIOAllocation = (vim.StorageResourceManager.IOAllocationInfo) {
-->                dynamicType = <unset>,
-->                limit = 500,
-->                shares = (vim.SharesInfo) {
-->                   dynamicType = <unset>,
-->                   shares = 1000,
-->                   level = "normal",
-->                },
-->                reservation = 0,
-->             },
-->             diskObjectId = "111-2000",
-->             vFlashCacheConfigInfo = (vim.vm.device.VirtualDisk.VFlashCacheConfigInfo) null,
-->          },
-->       },

One from vsphere 6 is
2017-03-10T07:16:21.716Z [2BF80B70 verbose 'Vcsvc.LLPM' opID=4155057b-bd user=vpxuser:VSPHERE.LOCAL\rpaas] LLPM reconfig (vim.vm.ConfigSpec) {
-->    dynamicType = <unset>,
-->    changeVersion = <unset>,
-->    name = <unset>,
-->    version = <unset>,
-->    uuid = <unset>,
-->    instanceUuid = <unset>,
-->    npivWorldWideNameType = <unset>,
-->    npivDesiredNodeWwns = <unset>,
-->    npivDesiredPortWwns = <unset>,
-->    npivTemporaryDisabled = <unset>,
-->    npivOnNonRdmDisks = <unset>,
-->    npivWorldWideNameOp = <unset>,
-->    locationId = <unset>,
-->    guestId = <unset>,
-->    alternateGuestName = <unset>,
-->    annotation = <unset>,
-->    files = (vim.vm.FileInfo) {
-->       dynamicType = <unset>,
-->       vmPathName = "[]/vmfs/volumes/5507b83b-7346c089-d2ed-0090fa8aa610/vm-12344265-b3ee-4eb9-923c-8a455d975240/vm-12344265-b3ee-4eb9-923c-8a455d975240.vmx",
-->       snapshotDirectory = <unset>,
-->       suspendDirectory = <unset>,
-->       logDirectory = <unset>,
-->       ftMetadataDirectory = <unset>,
-->    },
-->    tools = (vim.vm.ToolsConfigInfo) null,
-->    flags = (vim.vm.FlagInfo) null,
-->    consolePreferences = (vim.vm.ConsolePreferences) null,
-->    powerOpInfo = (vim.vm.DefaultPowerOpInfo) null,
-->    numCPUs = <unset>,
-->    numCoresPerSocket = <unset>,
-->    memoryMB = <unset>,
-->    memoryHotAddEnabled = <unset>,
-->    cpuHotAddEnabled = <unset>,
-->    cpuHotRemoveEnabled = <unset>,
-->    virtualICH7MPresent = <unset>,
-->    virtualSMCPresent = <unset>,
-->    deviceChange = (vim.vm.device.VirtualDeviceSpec) [
-->       (vim.vm.device.VirtualDeviceSpec) {
-->          dynamicType = <unset>,
-->          operation = "remove",
-->          fileOperation = "destroy",
-->          device = (vim.vm.device.VirtualDisk) {
-->             dynamicType = <unset>,
-->             key = 2000,
-->             deviceInfo = (vim.Description) {
-->                dynamicType = <unset>,
-->                label = "Hard disk 1",
-->                summary = "3,145,728 KB",
-->             },
-->             backing = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) {
-->                dynamicType = <unset>,
-->                fileName = "[]/vmfs/volumes/5507b83b-7346c089-d2ed-0090fa8aa610/vm-12344265-b3ee-4eb9-923c-8a455d975240/vm-12344265-b3ee-4eb9-923c-8a455d975240.vmdk",
-->                datastore = 'vim.Datastore:datastore-1211',
-->                backingObjectId = "725-2000-0",
-->                diskMode = "persistent",
-->                split = false,
-->                writeThrough = false,
-->                thinProvisioned = false,
-->                eagerlyScrub = <unset>,
-->                uuid = "6000C29b-233c-e226-9381-34bf9fc78093",
-->                contentId = "b0e8b00c8d1601f4a87255eba20e04db",
-->                changeId = <unset>,
-->                parent = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) {
-->                   dynamicType = <unset>,
-->                   fileName = "[]/vmfs/volumes/5507b83b-7346c089-d2ed-0090fa8aa610/sc-af1c2748-6463-4cc7-8696-ddcb53b2270c/sc-af1c2748-6463-4cc7-8696-ddcb53b2270c.vmdk",
-->                   datastore = 'vim.Datastore:datastore-1211',
-->                   backingObjectId = "725-2000-1",
-->                   diskMode = "persistent",
-->                   split = <unset>,
-->                   writeThrough = <unset>,
-->                   thinProvisioned = false,
-->                   eagerlyScrub = <unset>,
-->                   uuid = "6000C29b-233c-e226-9381-34bf9fc78093",
-->                   contentId = "b0e8b00c8d1601f4a87255eba20e04db",
-->                   changeId = <unset>,
-->                   parent = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) null,
-->                   deltaDiskFormat = <unset>,
-->                   digestEnabled = <unset>,
-->                   deltaGrainSize = <unset>,
-->                },
-->                deltaDiskFormat = "redoLogFormat",
-->                digestEnabled = false,
-->                deltaGrainSize = <unset>,
-->             },
-->             connectable = (vim.vm.device.VirtualDevice.ConnectInfo) null,
-->             slotInfo = (vim.vm.device.VirtualDevice.BusSlotInfo) null,
-->             controllerKey = 1000,
-->             unitNumber = 0,
-->             capacityInKB = 3145728,
-->             capacityInBytes =<PHONE_NUMBER>,
-->             shares = (vim.SharesInfo) {
-->                dynamicType = <unset>,
-->                shares = 1000,
-->                level = "normal",
-->             },
-->             storageIOAllocation = (vim.StorageResourceManager.IOAllocationInfo) {
-->                dynamicType = <unset>,
-->                limit = -1,
-->                shares = (vim.SharesInfo) {
-->                   dynamicType = <unset>,
-->                   shares = 1000,
-->                   level = "normal",
-->                },
-->                reservation = 0,
-->             },
-->             diskObjectId = "725-2000",
-->             vFlashCacheConfigInfo = (vim.vm.device.VirtualDisk.VFlashCacheConfigInfo) null,
-->          },
-->       },

@ronakbanka Thank you for the hostd output.
Could you please provide the bash steps you ran and a redacted manifest, state.json file, and cloud-config?  Or do you deploy a director with bosh create-env and then do a bosh deploy followed by a bosh delete-deployment and then with a bosh deploy again?
We are unable to reproduce your problem. If you could provide an example set of manifests and an example bash script to demonstrate that would be incredibly useful. Very explicit and specific steps to reproduce will be helpful.
This problem is interesting and we want to make sure we help you out and fix any bugs that might be causing this quickly.
Best @cunnie and @zaksoup

@ronakbanka Hey, we're still very interested in understanding what's causing your problem! Can you please provide the bash steps you ran and a redacted manifest, state.json file, and cloud-config?
Thanks!
@cunnie and @zaksoup

@zaksoup @cunnie We just moved to a new cluster with same version 6 and now we are not experiencing this strange issue . There was some db issues on our previous cluster , was not able to find if it was related to stemcell issue in a way.
can close this now and will see it it happens again.

@ronakbanka sounds good. please re-open this issue if you can reproduce it!
@zaksoup && @cunnie

Hi We're experiencing similar problem after we upgraded to vSphere 6.0 from vSphere 5.5, however we're not creating new environment. We're just scaling down some Diego VMs. If we scale up then there is no issue, however when we scale down it throws error:
Unknown CPI error 'Unknown' with message 'Invalid configuration for device '0'.' in 'delete_vm' CPI method.
Further retries of deploy attempts fail as well.
One workaround we could find is delete the VM via BOSH CCK then deploy again.
Following is the last SOAP Request/Response in the task logs:
= Request\n\nPOST /sdk/vimService HTTP/1.1\r\nSOAPAction: "urn:vim25/5.1"\r\nAccept-Encoding: gzip, deflate\r\nContent-Type: text/xml; charset=UTF-8\r\nCookie: vmware_soap_session=ca6r4ac1a908638470229bc37ca4a4d59fc055ca\r\nUser-Agent: HTTPClient/1.0 (2.7.1, ruby 2.2.4 (2015-12-16))\r\nAccept: /\r\nDate: Fri, 17 Nov 2017 09:16:49 GMT\r\nContent-Length: 726\r\nHost: vSphereHost\r\n\r\n\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/\" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd="http://www.w3.org/2001/XMLSchema\">\nsoapenv:Body<RetrievePropertiesEx xmlns="urn:vim25"><_this type="PropertyCollector">propertyCollector</_this>Taskfalseinfo.progressinfo.stateinfo.resultinfo.error<obj type="Task">task-82346false</soapenv:Body>\n</soapenv:Envelope>
\n\n= Response\n\nHTTP/1.1 200 OK\r\nDate: Fri, 17 Nov 2017 09:16:49 GMT\r\nCache-Control: no-cache\r\nConnection: Keep-Alive\r\nContent-Type: text/xml; charset=utf-8\r\nX-Frame-Options: DENY\r\nContent-Length: 853\r\n\r\n\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/\"\n xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/\"\n xmlns:xsd="http://www.w3.org/2001/XMLSchema\"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance\">\nsoapenv:Body\n<RetrievePropertiesExResponse xmlns="urn:vim25"><obj type="Task">task-82346info.error<val xsi:type="LocalizedMethodFault"><fault xsi:type="InvalidDeviceSpec">virtualDeviceSpec.operation0Invalid configuration for device '0'.info.state<val xsi:type="TaskInfoState">error\n</soapenv:Body>\n</soapenv:Envelope>

This may be related to user permissions (i.e. logging in not as administrator@vsphere.local).

@cunnie : Did you come across this issue recently? Which vCenter and which CPI version?

@EleanorRigby — I haven't run across this issue in 2 years.