gsound: enable introspection/vala when cross compiled

Description of changes
these features were removed as a stopgap in https://github.com/NixOS/nixpkgs/pull/151809 when gobject-introspection and vala didn't support cross compilation, but today they do and we can re-enable them here :)
CC: @Mindavi
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 263107 run on x86_64-linux 1

1 package marked as broken and skipped:

gnomeExtensions.easyScreenCast

24 packages failed to build:

adapta-gtk-theme
cinnamon.cinnamon-gsettings-overrides
gnome-browser-connector
gnome.gnome-flashback
gnome.gnome-panel-with-modules
gnome.gnome-session
gnome.gnome-session.debug
gnome.gnome-session.sessions
gnome.gnome-shell
gnome.gnome-shell.debug
gnome.gnome-shell.devdoc
gnome.gnome-terminal
gnome.gnome-tweaks
gnome.nixos-gsettings-overrides
gnomeExtensions.gsconnect
gnomeExtensions.gsconnect.installedTests
gnomeExtensions.system-monitor
megapixels
mojave-gtk-theme
pantheon.elementary-session-settings
phosh
phosh-mobile-settings
vimix-gtk-themes
whitesur-gtk-theme

29 packages built:

budgie.budgie-control-center
budgie.budgie-control-center.debug
calls
calls.devdoc
chatty
cinnamon.cinnamon-common
cinnamon.cinnamon-screensaver
feedbackd
feedbackd.dev
feedbackd.devdoc
gnome-hexgl
gnome.four-in-a-row
gnome.geary
gnome.gnome-bluetooth
gnome.gnome-bluetooth.dev
gnome.gnome-bluetooth.devdoc
gnome.gnome-bluetooth.man
gnome.gnome-clocks
gnome.gnome-control-center
gnome.gnome-control-center.debug
gnome.gnome-nibbles
gnome.gnome-robots
gnome.gnome-taquin
gnome.iagno
gnome.quadrapassel
gnomeExtensions.pano
gsound
pitivi
squeekboard

many of those i recognize as things which have never cross compiled for me: gnome.gnome-shell, phosh and phosh-mobile-settings, at least. i may as well kick off an overnight job to see if any of those failures build on master though.


Reset global phase of the full state simulator

ResetAll on all qubits and/or deallocating all qubits should reset the global phase of the full state simulator.
The global phase of the wavefunction is arbitrary and has no effect on measurements. However, it is reflected in the DumpMachine and DumpRegister outputs. Currently, the simulator retains a nontrivial phase even when the user goes through a sequence that should reset it to a clean state, such as deallocating all qubits. This leads to the confusing behavior that the same gate sequence applied to supposedly identical initial states (created, e.g., by releasing and allocating all qubits or applying ResetAll to all qubits) leads to different machine states, as observed through the dumps. It also makes the process of checking dumps against reference states for debugging purposes more involved than needed.
An alternative would be to have the dump functions normalize the global phase in a consistent and reproducible way. However, this would not cover some debugging scenarios where one would like the phase between different state dumps to be consistent. (E.g.: Apply the same gate sequence to the computational basis states in order to reconstruct the full unitary matrix implemented by the computation.)

I have a bit of concerned about doing this for performance reasons, but might be ok.
If we're doing this during Reset we should probably expose implemented as an Intrinsic for the simulator. An alternative is to implement this during measurement (M). After all, after measurement the global phase should also be reset 0, correct?
Regarding your last comment, I believe that what you're saying is that if we do this during Dump, we'll be loosing some of the unitary information, correct?
This looks like a good enhancement, I'm going to mark it as "looking for help" as we don't have resources right now to work on it.

Thanks for the comment!
I'm not sure I understand your performance concern: You would only ever reset the phase when the wavefunction is a purely classical computational basis state, so you'd have to set a single amplitude to 1.0. I don't see how this would be significant overhead compared to qubit measurement or allocation.
Regarding phase normalization during Dump: Yes, this would essentially introduce an arbitrary, state-dependent phase shift in every Dump operation, which may again create confusion when users compare different dumps. Imagine a user applying the same operation U to different states and examining the register dumps after each of U|0>, U|1>, U(|0>+|1>)/sqrt(2). From the dumps, they'd conclude that U|0> + U|1> != U(|0>+|1>), i.e. that additivity is violated.

I think the problem here is the assumption that people should be looking at the Dump to understand things. The problem with that is that it trains people that they will always have that resource as a debugging tool, when in reality, you cannot look at the Dump of an actual qubit. IMHO I think its a good thing that the dumps would not match in the example that @jgukelberger gave above because it should be a teaching moment to show that there is information in them that you cannot directly observe.

@crazy4pi314 - I think I understand where you're coming from, and there may be a valid point in saying the Dump operations should not be used in a pedagogic introduction to quantum computing; but applying that argument it would seem like none of the quantum register dumps should be available to the user. However, we do have them because we'd like to provide developers the best debugging tools possible.
My argument is that inconsistent handling of the global phase makes this particular debugging tool less useful than it could be. After all, a big advantage of developing on a simulated/emulated system (in general, independent of the quantum context) is the access to additional control and state information that the real system doesn't expose.

I agree with @jgukelberger, the whole purpose of having a simulator is to allow people to try and debug their algorithms; Dump is a debugging tool and keeping this global phase makes debugging harder.
I also agree with @crazy4pi314 in that people should not rely on it to understand what the algorithm is doing, but as @jgukelberger said, Dump should be used as a debugging and not as a pedagogic tool. I do think we should change it.


feat: Added highlights section and feature cards

Description :
Added the Highlights and Feature Section.
I would like to work under GSSOC'23

@senthil-k8s Please check responsiveness

@kaur-rajdeep Please kindly help him to fix the responsiveness.

okay, I'll do it.


Unpin eslint version in react-scripts

I've run into issues with eslint being the wrong exact version a few times now. Is there a reason fir it being pinned to an exact version?
I have a custom eslint config, so eslint is a peer dependency of my project. Since it's an approximate version, it resolves to 5.8.0. react-scripts then throws a warning when I try to run tests.
The react-scripts package provided by Create React App requires a dependency:

  "eslint": "5.6.0"

The test are failing with a similar error.
The react-scripts package provided by Create React App requires a dependency:
  "eslint": "^5.6.0"
Don't try to install it manually: your package manager does it automatically.
However, a different version of eslint was detected higher up in the tree:
  /home/travis/build/facebook/create-react-app/packages/react-scripts/node_modules/eslint (version: 5.8.0)

Shouldn't 5.8.0 satisfy ^5.6.0? https://docs.npmjs.com/misc/semver#advanced-range-syntax

This package is purposely pinned to prevent unwanted changes coming into the project. Please use 5.6.0 in your project.

Wouldn't semver prevent unwanted changes?

Also is there any documentation on the pinned versions? I couldn't find anything and it would be nice to have an explanation so people don't end up spending time on PRs in the future.

Semver is just a recommendation, there's no guarantees.
We also try to vet our packages before upgrading so users don't get viruses.
We have no formal documentation, but have always pinned package versions to be exact.

Ah okay. My apologize. I took a look back at v1 of the scripts to check what version you had for eslint and saw ^4.4.1. So I thought the pinning was new to v2. However I looked at package.json instead of packages/react-scripts/package.json.
I see the root version is a devDependency, so I assume the approximate version is okay here.
Thanks for clarifying.

versions for published packages should not be pinned.. it creates more complex node_modules install trees - since each pinned version must be satisfied. Instead, it would be far more useful if, for example react-dev-utils had a postinstall script which warned for versions above what is known to work.


[1.3.6-prepare][Feature-5087][SqlTask] Add a switch to send mail and print head logs in SqlTask #5088

pr #5088
issue #5087

@zhuangchong @break60 please take a look UI.thx

@chengshiwen @CalvinKirs
LIMIT limits the number of rows in a SQL query, and displayRows limits the number of rows that a log can print.
Isn't the number of queries the number of items to print?

@chengshiwen @CalvinKirs
LIMIT limits the number of rows in a SQL query, and displayRows limits the number of rows that a log can print.
Isn't the number of queries the number of items to print?

A user can choose not to send mail, so printing a small amount of query results into the log is a good way. The variable displayRows is limited to less than 100, in order to avoid that the number of logs is too large.
For sending mail, the number of rows of query results sent can be larger, for example <= 10000

@chengshiwen @CalvinKirs
LIMIT limits the number of rows in a SQL query, and displayRows limits the number of rows that a log can print.
Isn't the number of queries the number of items to print?

A user can choose not to send mail, so printing a small amount of query results into the log is a good way. The variable displayRows is limited to less than 100, in order to avoid that the number of logs is too large.
For sending mail, the number of rows of query results sent can be larger, for example <= 10000

@chengshiwen @CalvinKirs
I wondered if this would make it more user-friendly,
SQL query result display has the email and log printing, on the front page for users to choose more the result display, set up the article a SQL query number limit parameters, the user can select only the log printing way, also can only choose mail way, also can also choose one of two ways (choice is at the same time may produce a large amount of data, this can agree)
In addition, I choose two display methods of log printing and email sending at the same time. I feel that the data can be viewed in either way many times at a time. Why should I view the data in both ways at the same time?

我想了一下这样调整会不会对用户更友好一些，
sql查询结果展示方式有邮件发送和日志打印，在前端页面可供用户进行多选结果展示方式，设置一个sql查询条数限制参数，用户可以只选择日志打印方式,也可以只选择邮件发送方式,也可以同时选择两种方式（同时选择的时候就是可能会产生很大的数据量，这个可以认同）
并且同时选择日志打印和邮件发送两种展示方法，我感觉多次一举，在任何一种方式都可以查看数据，为什么要两种方式同时查看？

@zhuangchong You can start a discussion in the dev email, and realize this improvement according to the idea you mentioned, not on this 1.3.6-prepare. The current pr is a cherry-pick from dev, and this improvement is the result of the previous discussion on March 17.

@zhuangchong You can start a discussion in the dev email, and realize this improvement according to the idea you mentioned, not on this 1.3.6-prepare. The current pr is a cherry-pick from dev, and this improvement is the result of the previous discussion on March 17.

ok, no problem.


xembed-sni-proxy fails with QT plugin error

Seems related to #249 since adding plasma-desktop to my packages fixes it.

Yeah, that's it. Perhaps these services for Qt based programs should simply add qtbase to home.packages? But that wouldn't work if you for some reason are needing additional plugins.


[bug] prefetch_related stops working in Django 3.0

It used to work in Django 2.x, but now using prefetch_related on a SortedM2M fields make it an empty one.

I added a new test here https://github.com/jazzband/django-sortedm2m/pull/208 and I'm not getting any hits. Maybe because you're using Django 2.x, we start testing with Django 3.0.


Allow user defined "dummy name" (BindingElement) for array destructuring

See: https://esdiscuss.org/topic/non-binding-destructuring-assignment
Example:
/* jshint dummybindingelement: "_" */
var f = ([_,_,_,foo]) => foo;

f([1, 2, 3, 4]); // 4

or...
/* jshint dummybindingelement: "WHATEVER" */
var f = ([WHATEVER,WHATEVER,WHATEVER,foo]) => foo;

f([1, 2, 3, 4]); // 4

"dummybindingelement" is just a placeholder for the sake of illustration.

Note: Currently SpiderMonkey thinks this is a SyntaxError, so I'm researching whether or not that's even correct. So far I can't find any evidence in the spec that supports that.

How about elisions?
/* jshint esnext: true */
var f = ([ , , , foo]) => foo;

f([1, 2, 3, 4]); // 4

Benefits:

no side effects (the runtime doesn't create a dummy binding)
no new code :)

@rwaldron That's not valid ES6. You can't have duplicate bound names in that position (arrow parameter lists are strict formal parameter lists).

@jugglinmike that's discussed in the thread I linked ;P

@michaelficarra I don't see an early error defined for duplicate BindingElement names. Please cite the precise grammar.

@rwaldron
14.2

When the production
ArrowParameters[Yield] : CoverParenthesizedExpressionAndArrowParameterList[?Yield]

is recognized the following grammar is used to refine the interpretation of CoverParenthesizedExpressionAndArrowParameterList :
ArrowFormalParameters[Yield, GeneratorParameter] :
  ( StrictFormalParameters[?Yield, ?GeneratorParameter] )

14.1.2

StrictFormalParameters : FormalParameters

It is a Syntax Error if BoundNames of FormalParameters contains any duplicate elements.

That's not valid ES6. You can't have duplicate bound names in that position (arrow parameter lists are strict formal parameter lists).

The same SyntaxError exception is thrown for:
function f([_,_,_,foo]) { return foo; }

And FunctionDeclaration is never defined with StrictFormalParameters. Following the Runtime semantics and the Grammar rules:
14.2.14 Runtime Semantics: IteratorBindingInitialization
ArrowParameters : CoverParenthesizedExpressionAndArrowParameterList

Let formals be CoveredFormalsList of CoverParenthesizedExpressionAndArrowParameterList.
Return the result of performing IteratorBindingInitialization of formals with arguments iteratorRecord and environment.

14.2.9 Static Semantics: CoveredFormalsList
CoverParenthesizedExpressionAndArrowParameterList[Yield] :
  ( Expression )
  ( )
  ( ... BindingIdentifier )
  ( Expression , ... BindingIdentifier )

1 & 2 Refer to ArrowFormalParameters
ArrowFormalParameters[Yield, GeneratorParameter] :
  ( StrictFormalParameters[?Yield, ?GeneratorParameter] )

StrictFormalParameters : FormalParameters

It is a Syntax Error if BoundNames of FormalParameters contains any duplicate elements.

But when does this get applied to Destructing syntax? https://people.mozilla.org/~jorendorff/es6-draft.html#sec-destructuring-assignment
So far what I've seen described is:
var f = (a, a, a) => a;

But I still don't see where those rules suddenly apply to destructuring, since a single destructuring assignment pattern stands in place of a single parameter, and is applied to a single argument.
var f = ([a, a, a]) => a;

Is closer to:
var f = (a) => {
  var [b, b, b] = a;
  return b;
};

Than something like this:
var f = (a, a, a) => {
  return a;
};

Not that I wish to claim that Traceur is infallible, but it supports my reading of the specification:
// Options: --script
var f = ([_, _, _, foo]) => foo;

console.log(f([1, 2, 3, 4]));

Produces no error. (--script is needed to instruct traceur to execute as a Script, without implied "use strict")
Maybe I've missed something. Can you help clarify?
cc @allenwb

@rwaldron: Right below the last early error I sent you,
14.1.2

FormalParameters : FormalParameterList

It is a Syntax Error if IsSimpleParameterList of FormalParameterList is false and BoundNames of FormalParameterList contains any duplicate elements.

Traceur is wrong.

@rwaldron: If you want to learn more about ES6 early errors, you can see examples of each one at https://github.com/shapesecurity/shift-parser-js/blob/es6/test/early-errors.js

Traceur is wrong.

cc @arv

It is clear that duplicate binding names in a FormalParameterList that is not simple should be a SyntaxError. (Traceur gets this wrong.)
Duplicate binding names in a VariableDeclaration is allowed (AFICT). (Traceur gets this right.)

Correct.

Duplicate binding names in a VariableDeclaration is allowed

but not in a LexicalDeclaration  ( a let or const declaration)

The key piece that was missing from the references above was this:

FormalParameter : BindingElement

Return IsSimpleParameterList of BindingElement.

Then just follow BindingElement and it makes sense. When I was reading last night it just wasnt registering—thanks for patience.


V5 [Typography, Btn's & Forms]

Btn

Removes fixed heights
Changes the padding to use em's
Updates color refs to use the variables.sccs file

Form Inputs

Removes fixed height in favor of min-height so btn txt can wrap

Icons

Removes the top offset on the icon class

Messages

Changes the padding to use em's
Updates styling when used with form elements

@catchamonkey Is this to do with the other pr problem i had the other day..?


Bayesian Coreset Algorithms

What's the new feature?
Add Bayesian Coreset Algorithms (https://www.jmlr.org/papers/v20/17-613.html, https://proceedings.neurips.cc/paper_files/paper/2019/file/7bec7e63a493e2d61891b1e4051ef75a-Paper.pdf)
What value does this add?
Add Bayesian functionality to Coreax.
Is there an alternative you've considered?
No response
Additional context
No response

discussed as a team we would not be proceeding with this ticket for now

We will not be implementing these as a team, but we welcome outside contributions for this ticket


Select API is not very good to use

now is :
 <Select defaultValue={{ value: 'apple', label: 'Apple' }}>
      <SelectTrigger className='w-[250px]'>
        <SelectValue
          className='text-foreground text-sm native:text-lg'
          placeholder='Select a fruit'
        />
      </SelectTrigger>
      <SelectContent insets={contentInsets} className='w-[250px]'>
        <SelectGroup>
          <SelectLabel>Fruits</SelectLabel>
          <SelectItem label='Apple' value='apple'>
            Apple
          </SelectItem>

I think better like this
 <Select defaultValue="apple"> ///here
      <SelectTrigger className='w-[250px]'>
        <SelectValue
          className='text-foreground text-sm native:text-lg'
          placeholder='Select a fruit'
        />
      </SelectTrigger>
      <SelectContent insets={contentInsets} className='w-[250px]'>
        <SelectGroup>
          <SelectLabel>Fruits</SelectLabel>
          <SelectItem  value='apple'> ///here
             <Text>Apple</Text>
          </SelectItem>

I think I had to do that so that it worked the same way for native and web while keeping composability. For example, you can add components inside the SelectItem like in the code snippet you provided where you pass <Text>Apple</Text> as its child.
PRs are welcomed!

Closing due to inactivity.


Micro-components

After using WFP UI on different projects, we realised that proprietary components are not being created fast enough to satisfy the demand, as well as required modularity. Each project has different requirements and may need to build totally custom components that won't ever end-up in WFP UI, or may need to alter a proprietary component to achieve desired result.
This calls for something truly modular, that can be used via SCSS APIs (mixins, extends), as well as via HTML classes. A set of baseline, generic micro-components, which used together can form other bigger components. Inspired by Tachyons.io, I began researching the best way to build such a toolkit right into WFP UI, and do away with big and bulky proprietary components, that use completely custom CSS rules.
The following list is in no way exhaustive nor final, and may look similar to Tachyons' own list (which it is inspired by), in no particular order.
Typography

[x] Font Style
[x] Font Size
[x] Font Weight
[x] Vertical Align
[x] Text Decoration
[x] Text Transform
[x] Leading
[x] Alignment
[x] Whitespace
[x] Tracking

Layout

[x] Box Sizing
[x] Floats
[x] Clearfix
[x] Display
[x] Widths
[x] Max Widths
[x] Heights
[x] Position
[x] Spacing

Elements

[x] Headings

Theming

[x] Colours
[x] Background Colours

I've added a basic, temporary implementation of colours module into the build. This should be expanded to allow many more colour combinations for text, backgrounds, and borders.


Mark tests that use the _Volatile module as REQUIRES: volatile

Follow up to https://github.com/apple/swift/pull/74537.

@swift-ci please test


Preserve the order of properties when creating or updating items #162

Description
Preserves the order of properties defined in the catalogue item when creating/updating an item.
Testing instructions
Add a set up instructions describing how the reviewer should test the code

[ ] Review code
[ ] Check Actions build
[ ] Review changes to test coverage

Agile board tracking
Closes #162

I am assuming that we do not have this ordering issue with catalogue items?

I am assuming that we do not have this ordering issue with catalogue items?

No we don't the properties are just overridden at the moment, so it doesn't have the same issue when editing, and when creating the optional fields just aren't stored.


Does the require("fs") in the VSCode extension code turn into require("graceful-fs") when using SSH remote?

Does this issue occur when all extensions are disabled?: No

VS Code Version: 1.79.0
OS Version: WSL2?

Steps to Reproduce:

WSL2 remote ssh VSCode(1.79.0) uses prettier-vscode extension
TypeError: Cannot read properties of undefined (reading 'uid') happens (please see https://github.com/prettier/prettier-vscode/issues/3020 for more details)

error stack (original: https://github.com/prettier/prettier-vscode/issues/3019)
["ERROR" - 12:05:24 AM] Failed to load module. If you have prettier or plugins referenced in package.json, ensure you have run `npm install`
["ERROR" - 12:05:24 AM] Cannot read properties of undefined (reading 'uid')
TypeError: Cannot read properties of undefined (reading 'uid')
	at Object.statSync (/home/ziimakc/.vscode-server/bin/b380da4ef1ee00e224a15c1d4d9793e27c2b6302/node_modules/graceful-fs/polyfills.js:313:17)
	at p (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:35221)
	at L (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:37052)
	at /home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:36677
	at Function.e.exports [as sync] (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:36723)
	at t.ModuleResolver.findPkg (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:7297)
	at t.ModuleResolver.getPrettierInstance (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:3251)
	at t.default.format (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:13495)
	at async t.PrettierEditProvider.provideEdits (/home/ziimakc/.vscode-server/extensions/esbenp.prettier-vscode-9.13.0/dist/extension.js:1:11367)
	at async U.provideDocumentFormattingEdits (/home/ziimakc/.vscode-server/bin/b380da4ef1ee00e224a15c1d4d9793e27c2b6302/out/vs/workbench/api/node/extensionHostProcess.js:102:45790)
["ERROR" - 12:05:24 AM] Prettier could not be loaded. See previous logs for more information.

As a maintainer of the Prettier code formatter, I've been working on an issue that surfaced after updating to VSCode 1.79.0. Specifically, we've been receiving reports about prettier-vscode ceasing to function when used in VSCode over a remote SSH connection.
The problem was first mentioned in this issue: https://github.com/prettier/prettier-vscode/issues/3020
Upon further examination of the error stack posted in https://github.com/prettier/prettier-vscode/issues/3019, it seems that the original require("fs") function call is being replaced by require("graceful-fs") within the affected environment. This replacement is unexpected and is likely linked to the reported malfunctions.
I would like to know if the VSCode team has any ideas. Thank you.

I don't know why this problem started in 1.79.0, but it seems that this commit in graceful-fs is not included in the version on which vscode remote depends
https://github.com/isaacs/node-graceful-fs/commit/e61a20a052b838f420b98195c232a824a6ac04ee

@alexdima Hi, Is it possible to update the version of graceful-fs that VSCode depends on? Without it, prettier-vscode will remain broken. I also suspect that the VSCode Extension making the call to fs.statSync(foo, { throwIfNoEntry: false }) will be broken as well.


Release 1.2.0

#111

this should be in dev branch

@vayesy @vovacha @FiberMan add your sauce here, please

@vayesy do we want to add standalone cargo etc or not?


SLVUU-127: Include username in local storage layout key

Description
When using the local implementation of layout management, the relevant information is saved to local storage under the keys layouts/layouts, layouts/metadata and api/vui. This can cause problems once a second user is introduced, so we change the keys here to include a username. We also add the username to cookies for the NewTheme showcase example, matching the behaviour of the sample app.
Change List

Add username to local storage keys (application layout, layout and metadata)
Convert private functions in LocalLayoutPersistenceManager to arrow functions
Get actual username from cookies
Add username to cookies for showcase example

Testing
Saving and loading layouts tested using local and remote implementations, from both showcase and sample app.

PR raised to Finos

PR merged


publishing is really really slow

Current Behavior
publishing takes ~60 seconds. In general, even when cache's should be hit the publishing part of the step takes ~60 seconds.
Expected Behavior
publishing should probably take no more than 10 seconds. I
Context (optional)
My build was failing to publish and I couldn't understand it, and I looked and found that a whole minute was just CI publishing to github and so I separated the build/publish step. Now it seems that publish takes 2 minutes by itself, I guess it doesn't hit any cache's of the build workflow, even after all the context is created pushing 6 tiny to githubs internal repository jars still takes 60 seconds.
part (most?) of this bug is arguably just to get gradle to give me enough information to diagnose why. The scans aren't really saying why (IMO) that just the publish step takes so long after the configuration and classes have been built.
https://github.com/xenoterracide/spring-app-commons/actions/runs/10868033227
Self-contained Reproducer Project
Not minimal, but since the main goal here is to provide more information to make the root cause easier to track down, maybe that's a good thing.
https://github.com/xenoterracide/spring-app-commons
at least some of this is almost certainly the lack of lazy configuration regarding versions, and so that version number is changing with every sha for me causing problems.
cache invalidation because having changing credentials invalidates the configuration cache.
Gradle version
8.10.1
Build scan URL (optional)
https://scans.gradle.com/s/dlxt7ooaphzeo
Your Environment (optional)
------------------------------------------------------------
Gradle 8.10.1
------------------------------------------------------------

Build time:    2024-09-09 07:42:56 UTC
Revision:      8716158d3ec8c59e38f87a67f1f311f297b79576

Kotlin:        1.9.24
Groovy:        3.0.22
Ant:           Apache Ant(TM) version 1.10.14 compiled on August 16 2023
Launcher JVM:  21.0.4 (Eclipse Adoptium 21.0.4+7-LTS)
Daemon JVM:    /home/xeno/.asdf/installs/java/temurin-21.0.4+7.0.LTS (no JDK specified, using current Java home)
OS:            Linux 6.6.47-1-MANJARO amd64

We don't know if it's an environmental issue like GitHub being slow at taking uploads or if it is the publishing tasks doing unreasonable work during action execution.
Build scans appear to not show uploads in network activity :/

I'm hoping that at the very least the outcome of this ticket is improvements to providing the information required to diagnose this issue. I have confirmed that the publishing is succeeding. At the very least more timings of the publishing tasks and actually showing the network upload speeds would be good.
I also confirmed that if all task configurations are hit locally publishToMavenLocal takes ~10 seconds. However, I do know that publishToMavenLocal does not do the exact same thing locally as it does when publishing to a remote server. https://gradle.com/s/qfznakct36srq
In grand total the total upload size is 1.1Mb (presuming no signficant additional data is publish to the server)
$ du -h
152K	./security-controller/0.0.1-alpha.0.248+a4389ef2
156K	./security-controller
220K	./commons-jpa/0.0.1-alpha.0.248+a4389ef2
224K	./commons-jpa
148K	./commons-model/0.0.1-alpha.0.248+a4389ef2

152K	./commons-model
212K	./security-model/0.0.1-alpha.0.248+a4389ef2
216K	./security-model
44K	./app-core/0.0.1-alpha.0.248+a4389ef2
48K	./app-core
40K	./test-app-core/0.0.1-alpha.0.248+a4389ef2
44K	./test-app-core
60K	./authn-controller/0.0.1-alpha.0.248+a4389ef2
64K	./authn-controller
144K	./test-authorization-server/0.0.1-alpha.0.248+a4389ef2
148K	./test-authorization-server
1.1M	.

If feasible you could try with 8.9 to rule out a regression or 8.10.2 to rule out it has already been fixed.

8.10.2
https://scans.gradle.com/s/i3bnd7m7ldp7g

8.9
https://scans.gradle.com/s/7inr2z22qilt4

I also confirmed that if all task configurations are hit locally publishToMavenLocal takes ~10 seconds

It's actually less than 0.03 second

However, I do know that publishToMavenLocal does not do the exact same thing locally as it does when publishing to a remote server

Does it matter in that case? If you want a clean test, you can set up a local instance of Maven repository, such as Nexus or Artifactory.
Currently, I think it's safe to assume that this is a networking issue.

I'm hoping that at the very least the outcome of this ticket is improvements to providing the information required to diagnose this issue.

We gave the feedback to the Develocity team.

It still wouldn't be the same case. Publishing to Nexus for example uses straight credentials that don't change. Githubs on the other hand changes every single time and so invalidates part of the cache every single time. It's probably not that part of the code that's doing this though. Gradle has documented performance bugs along this path.
I suppose if I were feeling non lazy I could build the artifacts and then push them with something other than Gradle... Although that's challenging because I have to fetch and parse the metadata...

ok, this is the result of pushing to a local nexus running in a docker container clean build

clean build https://scans.gradle.com/s/lntnpcipcix5g
after build https://gradle.com/s/qwduhslzbam6q (seems to prove network)

this is uploading to github from my laptop

clean build https://gradle.com/s/s56ctnnoqoc4m
after build https://gradle.com/s/74j5xv4g4io6q

results are still curious, but I guess. I'm not certain why there's not a bigger cache it on the publish job since the prior job ran build. Unless that build cache is not restored in the subsequent job by gradle setup.
If feedback to develocity was given should close then?

Thanks for the feedback.
Since it's confirmed that the issue was related to the networking, I'm closing this issue.
Regarding the blocking requests - Gradle uses Apache HTTP client under the hood - you can take a look at the wiring code starting from here. If you test it and find some issues with the networking part - please open a new issue.
Regarding build cache - currently, publishing tasks are not cachable. This can potentially be improved, but it's out of scope for now.


Update the lockRight property on refresh

Reviewers: @fractaltheory
Changes

Update lockRight property on refresh

TODOs:

[ ] +1
[x] Updated README
[x] Updated CHANGELOG

How To Test

Initialize a scooch
Add more items to the scooch
Call $('.scooch').scooch('refresh')
Make sure you can scroll through all the new items.

@MikeKlemarewski LGTM! 👍


Missing requirements in the doc

I am getting the following error:
/Users/.../esp32_reusable_components/custom-components/main/./main.c:2:19: fatal error: unity.h: No such file or directory

What did you execute to come to that?

Since it was not in the doc, I did not clone inside EXTRA_COMPONENT_DIRS.
I will test again, thanks for updating the doc!

Is it now working for you?

hi @mh-dev, almost. Using the correct export, your code compiles find.
There must be however an issue in the/my build process.
Using make TESTS_ALL=1 and flashing, I can see the full list of tests, including your.
Using a make clean + make TEST_COMPONENTS='awesome_component' and flashing, the list is empty.

I performed the same steps and everything is working for me.
I am not sure what the root cause of your problem is. You can try to delete the build directory unfortunately make clean does not remove its content and try it again.

@mh-dev deleting the build folder definitely helps.
Strangely:
make clean && rm -rf && make TEST_COMPONENTS='awesome_component awesome_component_cpp' && make flash && make monitor works
but
make clean && rm -rf && make TEST_COMPONENTS='awesome_component' && make flash && make monitor followed by make clean && rm -rf && make TEST_COMPONENTS='awesome_component_cpp' && make flash && make monitor
shows only the first component.
I am closing anyway here because I think your code works fine. Thanks again!


Update README.md

Contributing to Twilio

All third-party contributors acknowledge that any contributions they provide will be made under the same open-source license that the open-source project is provided under.

[ ] I acknowledge that all my contributions will be made under the project's license.

Hi @rnairtwilio -- can you PTAL when you get a chance? Thanks!


R CMD check detects pragmas that suppress diagnostics

https://travis-ci.org/github/cwida/duckdb/jobs/711921270#L2108

File which contains pragma(s) suppressing diagnostics:
‘src/duckdb.cpp’

R CMD check looks for regular expressions of the form "^\\s*#pragma (GCC|clang) diagnostic ignored", see https://github.com/wch/r-source/blob/trunk/src/library/tools/R/QC.R#L9352.
Can we strip these macros off of the amalgamation, at the expense of noise when building the R package? I have no idea why R CMD check does this, it seems this check has been introduced ~2 years ago. I haven't found a mention of this check in R-exts.

Yeah those should indeed not be there, need to figure out where they come from.

I have created a branch nopragmadiag where those should be zapped.

Merged


Thumbnail doesn't keep proportions

Hi ,
I think It would be preferable if thumbnails are created of maximum width or height and cropped if necessary

check out about "bootstrap grid system",

Thumbnails keeps propotions.

Hi, please check out this picture I've put in products page in serenity demo :
original
thumbnail
Am I missing something ?

There is nothing wrong, it takes center, works as designed


chore(main): release 0.3.0

:robot: I have created a release beep boop
0.3.0 (2024-04-28)
Features

docker image for err-aprs-backend (#59) (a82ca42)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/andrewthetechie/err-aprs-backend/releases/tag/v0.3.0 :sunflower:


Document bug in openssl cms -binary (1.1.1)

Documents a bug in openssl cms -binary for 1.1 whereby it cannot process input using LF line endings correctly. Binary input processing was reworked substantially for 3.0 and backporting these changes doesn't appear reasonable.
Fixes #17797.

24 hours has passed since 'approval: done' was set, but this PR has failing CI tests.  Once the tests pass it will get moved to 'approval: ready to merge' automatically, alternatively please review and set the label manually.

Merged to 1.1.1 branch. Thank you.


Example 5-3 section ending comments

In the tail of this section, it told us that we can invoke script, then type (or paste) the input to get result info.
However, the input is actually the whole line to paste (so script can parse accordingly); meanwhile we need to use the concept ot heredoc to make it work, such as:
getlocal.sh << EOF

Then paste the whole line from cmds.txt, press enter and type EOF to end execution.
In this way I could have correct XML format .info file. Any idea? Should I add comment about this?

If only invoking script name without any argument, after pasting commands (in the format like cmds.txt has), press ctrl-D to stop input, the effect is same as EOF of heredoc.


Local Language instead of english

Hi, I installed R using rig and for some reason, all of the R tool are now speaking to me in Polish...?
Don't get me wrong I appreciate the availability of local languages in software however I would like to be able to choose the language (aka switch to English, as I have everything on my development machine in English). Is there a way to do that?

What is your OS and OS version? What are the R versions you installed? What is the version of rig?

That would indeed be useful (I prefer the English version because of easier-to-google error messages).
I am using rig version 0.6.0 on macOS 14.4 and R versions 4.3-arm64/4.4-arm64.

@mbstadler We can probably add an option for that on macOS, installers default to the system language.
In the meanwhile you can probably work around it by putting this in .Renviron:
LC_ALL=en_US.UTF-8

@gaborcsardi Setting LC_ALL=en_US.UTF-8 in ~/.Renviron does the trick for me - thanks for the tipp!

Also setting the LANGUAGE in /Library/Frameworks/R.framework/Resources/etc/Renviron as 'en' also works for me
LANGUAGE=en

I'll still add an option for this at some point.


refactor(client): remove unused client type aliases

These type aliases cause test build to fail due to strict LINT settings
If these types should be preserved (used outside the test suite), please let me know and I can instead disable the LINT for these types.

Thanks!


The download of GHC

I have installed it, but ./install.sh still choose to download ghc. The problem is, I have to pay a long time to   download it(about  30KB/s).

You should explain more clear about this. Have you take a look on https://github.com/google/codeworld/issues/435?

CodeWorld installs its own versions of dependencies, including GHC, by design.  You can modify install.sh to avoid doing so, but then I couldn't guarantee that the rest of the install process will succeed with the specific versions you have installed.
If you have a better idea how this should work, can you be specific about what you want?  Otherwise, I think we should close this, as things are working as intended.


CodeReifier does not reify recursively

If you call a meta method that returns a MethodInvocationExpression where one of the parameters is a BlockExpression that declares a typed argument, the compiler will choke on the argument's type, saying that the node has not been correctly processed.  (I'm sure there are simpler ways to reproduce this, but that's the real-world case that caused it.)
The cause of the problem appears to be that when the meta method returns its result expression, ProcessMethodBodies grafts it into the appropriate place on the AST, then calls Reify(Expression) on it, which runs reification on that individual node, but not on its subtree.

Fixed on my end

Merged into master


tensorflow crashes when using large image with 3d convolutional network

I'm trying to implement a 3d fully convolutional network on my GPU. But for some reason I get a crash.
Environment info
Operating System: Ubuntu 14.04 LTS
GPU: GeForce Titan X .
Installed version of CUDA and cuDNN: 8.0 and 5
(attach the output of `ls -l /path/to/cuda/lib/libcud*
cud.filelist.txt )
I installed tensorflow version 0.11.0rc2, and it also reproduce in docker installation (gcr.io/tensorflow/tensorflow:latest-gpu)
Example code
The following code reproduce the problem:
import numpy as np
import tensorflow as tf

graph = tf.Graph()

with graph.as_default():
    tf_dataset = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))
    tf_label = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))

    layer1_weights = tf.Variable(tf.truncated_normal((2, 2, 2, 1, 1), stddev=0.1))
    layer1_bias = tf.Variable(tf.zeros(1))

    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
    logits = tf.nn.relu(conv+layer1_bias)

    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_label))
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    batchData = np.random.rand(1, 512, 512, 512, 1).astype(np.float32)
    batchLabels = (np.random.rand(1, 512, 512, 512, 1)>0.5).astype(np.float32)
    feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
    _ = session.run((optimizer, ), feed_dict=feed_dict)

with the following output:

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 11.92GiB
Free memory: 11.68GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
F tensorflow/stream_executor/cuda/cuda_dnn.cc:2440] failed to enqueue convolution on stream: CUDNN_STATUS_NOT_SUPPORTED

Does it work with smaller images?  It vaguely sounds like running out of memory, but it doesn't seem like it should given the numbers and batch size 1.

Can it be a memory issue? My GPU has 12GB. The size of the input is 0.25GB. Anyhow the following size works:  (32, 32, 512, 512, 1) which is larger in size, but smaller in one convolutional direction.

@HggsHntr I'm just a simple computer scientist.  Your questions are reasonable, but it's hard for me to think beyond testing whether a smaller image works or not.

I guess it might be memory issue after all.
I set config.gpu_options.allow_growth = True and traced the memory consumption of the GPU. It raised to the maximum before it crashed.
Also, by changing the stride size to 4 in each direction, or reducing the image size the crash disappear.
So I wonder if this large memory consumption in done intentionally or is it a bug.

@zheng-xq Is there any possible issue with GPU convolution scratch space?  (e.g. can the autotuner selects an algorithm based on available memory and then fail later on)

I've encountered the same issue on tensorflow==0.11.0rc2 with latest keras:
F tensorflow/stream_executor/cuda/cuda_dnn.cc:2674] failed to enqueue convolution on stream: CUDNN_STATUS_NOT_SUPPORTED
...... 104 Aborted                 (core dumped) .......

I was feeding the CNN 10k 200x200x3 images per fit. When I gradually lower the number to 8k, the same issue no longer exist. On the other hand, with the 10k images per fit, the same CNN written in tflearn was just fine.

can confirm a similar issue with large 3D convolutions:
inshape = (2,258,258,34,1)  # tf img-order
filter_shape = (3,3,3,1,32)
x = tf.placeholder('float32', shape=inshape, name='X')
f = tf.placeholder('float32', shape=filter_shape, name='filter')
c = tf.nn.conv3d(x, f, padding='VALID', strides=[1,1,1,1,1])
grads = tf.gradients(c,f)[0]

xx = np.random.rand(*inshape)
ff = np.random.rand(*filter_shape)

with tf.Session().as_default():
    Q = c.eval(feed_dict={x:xx, f:ff})
    gradQ = grads.eval(feed_dict={x:xx, f:ff})

will yield tensorflow/stream_executor/cuda/cuda_dnn.cc:2674] failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED.
It runs fine if I set inshape = (2,257,257,34,1) instead, so there's some issue with the sizes. However I dont think its the usual GPU-out-of-memory issue:

the same convolution runs perfectly fine in theano (there I can even increase to (2,300,300,34,1) without trouble)
it doesnt respond to all dimension the same: decreasing the batchsize from 2->1 will still crash even though we just halfed the input

Turns out that it's somehow related to the gradient computations, i.e. commenting the gradQ =... line, just evaluating the result of the convolutions works!

@redst4r I have a similar issue. I too think there could be some issue when calculating the gradients. Any update on how you resolved it?

@deepak09027 Unfortunately I dont have any solution yet. I was hoping that some googler picks it up here :)

Can confirm this issue as well. I'm having exactly the same problem! Would be great if this could be resolved :)

Similar problem here with deep 2d convolution network and huge batch size (because my input is [seq_length * batch_size, height, width] ).
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:02:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x3a9e500
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 980 Ti
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 4.99GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 0 and 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 1 and 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y N
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   N Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980 Ti, pci bus id: 0000:01:00.0)
F tensorflow/stream_executor/cuda/cuda_dnn.cc:2684] failed to enqueue convolution on stream: CUDNN_STATUS_NOT_SUPPORTED
Aborted (core dumped)

Change batch size to a smaller one solves the issue. It'd great if it can throw a python exception : )

@prb12, @zheng-xq any updates on this issue?

I am having the same issue as well

Same here. I can reproduce this issue with the example code above. Can confirm CUDA 8, CUDNN 5.1 with tf 1.0.
I ran into an interesting aspect when determining the maximum size of the tensor. It run when tensors with shape less than [1, 256, 256, 256,1] are used. The example below will run and sz=256 will break.
Is anyone looking into this?
import numpy as np
import tensorflow as tf

graph = tf.Graph()

sz = 255

with graph.as_default():
    tf_dataset = tf.placeholder(tf.float32, shape=(1, sz, sz, sz, 1))
    tf_label = tf.placeholder(tf.float32, shape=(1, sz, sz, sz, 1))

    layer1_weights = tf.Variable(tf.truncated_normal((2, 2, 2, 1, 1), stddev=0.1))
    layer1_bias = tf.Variable(tf.zeros(1))

    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
    logits = tf.nn.relu(conv+layer1_bias)

    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=tf_label))
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    batchData = np.random.rand(1, sz, sz, sz, 1).astype(np.float32)
    batchLabels = (np.random.rand(1, sz, sz, sz, 1)>0.5).astype(np.float32)
    feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
    _ = session.run((optimizer, ), feed_dict=feed_dict)

I've also run into the same issue, where a P100 with 16GB RAM would get the same error, but I know that my data is not even big enough to take up the entire GPU RAM.
I'm using TF 0.12, CUDA 8.0, CuDNN 5.1.

Mostly its not related to VRAM (I am using latest TF with keras)
In first layer for a VGG like architecture
Convoulution3D(NumChannels, 3,3,3,)(inputs)
if I set NumChannels < 8, it crashes with the above message.
This somehow contradicting with waTeim

Hmm, I am trying to run on tensors of shape (1,125,125,125,128) = 1 gig and getting the standard OOM error. I am wondering how much working memory the 3d conv needs. It seems like doing a conv of this size should work ok.

This issue also goes away for me when I reduce my batch size. Seems like a memory issue that Tensorflow doesn't catch the way it does other OOM issues.

Did anyone find any solution for this?
I am also hitting this issue when trying to train a CNN. When I reduce the batch size to 2 its runs and at this point the GPU VRAM consumption is 217 MB/3072 MB and on increasing the batch size to 3, it gives the "failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED" error. I doubt if this is due to low VRAM.
TensorFlow's response/support has been very weak on the bugs being reported here. Really disappointing.

@tfboyd Any ideas?

same problem .
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate (GHz) 1.3285 pciBusID 0000:05:00.0 Total memory: 15.89GiB Free memory: 15.61GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:05:00.0) F tensorflow/stream_executor/cuda/cuda_dnn.cc:1989] failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED ('Sample data and label: ', 50000, (3072,), array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.])) ('Training data: ', (35000, 3072), (35000, 10), '\nValidation data: ', (15000, 3072), (15000, 10)) Training started !!!!! /cm/local/apps/slurm/var/spool/job109000/slurm_script: line 14: 38773 Aborted                 /home/ksrivastava/tensorflow/bin/python /home/ksrivastava/DL/CIFAR10/scripts/trainGPU.py

I will try to escalate this.  The issue has been open for a long time and reproduced many times based on my skimming of the thread.

thanks @tfboyd !!

Internal email thread started, looking for an owner to "dig in".  Sorry for the delays hopefully we get this moving forward.

b/64718915  @yzhwang is working on it and thank you to @SrivastavaKshitij for following up via email and helping get this prioritized.

Hi @SrivastavaKshitij could you let me know which version of tensorflow you are using? I tested with @HggsHntr 's example code on tf 1.2.1 with CUDA 8 and cudnn 5.1. This seems to be a duplicated issue with https://github.com/tensorflow/tensorflow/issues/11327, which I have already fixed in https://github.com/tensorflow/tensorflow/commit/db596594b5653b43fcb558a4753b39904bb62cbd. Unfortunately the fix is still not included in 1.3rc2. If you believe this is a separate issue, could you provide a reproducer? Thank you!

Hi @yzhwang : I was able to solve the problem. I am working on a cluster and I was using an incompatible version of PGI+OpenMPI with cuda and cudnn version which was causing the problem. But as soon as I corrected it, everything worked fine. If you want I can give u the details.
Thanks

If it is not too complicated, we would like you to provide a more detailed description, and how you solved it. I think it would help other users with the same configurations.
As for this issue, I will close it for now as I have a fix for this (https://github.com/tensorflow/tensorflow/commit/db596594b5653b43fcb558a4753b39904bb62cbd) and it has been verified both internally and on the head of OSS version. @HggsHntr if you believe the fix didn't solve the issue, please reopen it and I can investigate more.
@SrivastavaKshitij Please do put more details if you like even when the issue is closed. Much appreciated!

I have upgraded to tensorflow 1.4 nightly build (windows-gpu) and when I run the following:
import numpy as np
from keras.engine import Input, Model
from keras.layers import Conv3D, Activation
dim = 256
input_shape = [dim,dim,dim,1]
inputs = Input(input_shape)
conv1 = Conv3D(filters=1, kernel_size=(2, 2, 2), strides=(1,1,1), activation='relu', padding='same')(inputs)
logits = Activation('softmax')(conv1)
model = Model(inputs=inputs, outputs=logits)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
batchData = np.random.rand(1, dim, dim, dim, 1).astype(np.float32)
batchLabels = (np.random.rand(1, dim, dim, dim, 1)>0.5).astype(np.float32)
model.fit(x=batchData, y=batchLabels)
it crashes with an error about Conv3DBackpropFilterV2

@yzhwang
I get the same thing as @hadarpo .
Is it the same bug or something else?

Hi @hadarpo and @HggsHntr , I cannot reproduce this on my linux box. I do not have a Windows machine to test this right now. I will ask my colleagues to help reproduce this. Meanwhile, if you could include a full error log here, I will start looking at it and see where we go from there. Thank you!

Hi @yzhwang, thanks for your response. I have tried the latest nightly build (#55) with the original tester of @HggsHntr and got the following error output:
2017-10-02 11:30:17.785801: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_driver.cc:1080] failed to synchronize the stop event: CUDA_ERROR_LAUNCH_FAILED
2017-10-02 11:30:17.785922: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Internal: error destroying CUDA event in context 0000028FBE051ED0: CUDA_ERROR_LAUNCH_FAILED
2017-10-02 11:30:17.787104: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Internal: error destroying CUDA event in context 0000028FBE051ED0: CUDA_ERROR_LAUNCH_FAILED
2017-10-02 11:30:17.787561: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-02 11:30:17.787960: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-02 11:30:17.788460: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-02 11:30:17.788998: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-02 11:30:17.789578: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-02 11:30:17.790140: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-02 11:30:17.790606: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-02 11:30:17.790944: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-02 11:30:17.790987: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-02 11:30:17.791331: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
NotFoundError                             Traceback (most recent call last)
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1322     try:
-> 1323       return fn(*args)
1324     except errors.OpError as e:
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1301                                    feed_dict, fetch_list, target_list,
-> 1302                                    status, run_metadata)
1303
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\contextlib.py in exit(self, type, value, traceback)
65             try:
---> 66                 next(self.gen)
67             except StopIteration:
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
466           compat.as_text(c_api.TF_Message(status.status)),
--> 467           c_api.TF_GetCode(status.status))
468   # Delete the underlying status object from memory otherwise it stays alive
NotFoundError: No algorithm worked!
[[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]
During handling of the above exception, another exception occurred:
NotFoundError                             Traceback (most recent call last)
 in ()
22     batchLabels = (np.random.rand(1, 512, 512, 512, 1)>0.5).astype(np.float32)
23     feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
---> 24     _ = session.run((optimizer, ), feed_dict=feed_dict)
25
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
887     try:
888       result = self._run(None, fetches, feed_dict, options_ptr,
--> 889                          run_metadata_ptr)
890       if run_metadata:
891         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1118     if final_fetches or final_targets or (handle and feed_dict_tensor):
1119       results = self._do_run(handle, final_targets, final_fetches,
-> 1120                              feed_dict_tensor, options, run_metadata)
1121     else:
1122       results = []
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1315     if handle is None:
1316       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1317                            options, run_metadata)
1318     else:
1319       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)
~\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1334         except KeyError:
1335           pass
-> 1336       raise type(e)(node_def, op, message)
1337
1338   def _extend_graph(self):
NotFoundError: No algorithm worked!
[[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]
Caused by op 'gradients/Conv3D_grad/Conv3DBackpropFilterV2', defined at:
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\Scripts\ipython-script.py", line 5, in
sys.exit(IPython.start_ipython())
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython_init_.py", line 125, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\terminal\ipapp.py", line 356, in start
self.shell.mainloop()
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\terminal\interactiveshell.py", line 480, in mainloop
self.interact()
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\terminal\interactiveshell.py", line 471, in interact
self.run_cell(code, store_history=True)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2698, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2802, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 17, in
optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 343, in minimize
grad_loss=grad_loss)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 414, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 353, in _MaybeCompile
return grad_fn()  # Exit early
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\nn_grad.py", line 88, in _Conv3DGrad
data_format=data_format)]
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 964, in conv3d_backprop_filter_v2
data_format=data_format, name=name)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3090, in create_op
op_def=op_def)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1638, in init
self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
...which was originally created as op 'Conv3D', defined at:
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\Scripts\ipython-script.py", line 5, in
sys.exit(IPython.start_ipython())
[elided 7 identical lines from previous traceback]
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 13, in
conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 846, in conv3d
padding=padding, data_format=data_format, name=name)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3090, in create_op
op_def=op_def)
File "C:\Users\hadar\AppData\Local\conda\conda\envs\hadar-tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1638, in init
self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
NotFoundError (see above for traceback): No algorithm worked!
[[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

Hi @hadarpo , I'm trying to reproduce this on a Windows machine. Just a couple of things I want to make sure:

Are you using the head of TensorFlow on our github repo?
Are you using this reproducer (https://github.com/tensorflow/tensorflow/issues/5688#issuecomment-332832739) or the original reproducer (https://github.com/tensorflow/tensorflow/issues/5688#issue-190252047)?
It would be nice if you could also provide us the CUDA version, cudnn version, and display driver version you use.

Thank you!

Hi @yzhwang

I've installed tensorflow from jenkins nightly build using the following wheel:
https://ci.tensorflow.org/view/tf-nightly/job/tf-nightly-windows/M=windows-gpu,PY=35/55/artifact/cmake_build/tf_python/dist/tf_nightly_gpu-1.4.0.dev20170929-cp35-cp35m-win_amd64.whl
I am using the original tensorflow version of the reproducer (even though the keras one gives the same results)
I'm using CUDA 8.0 and cudnn6, and the NVIDIA drivers are as follow:

Thank you!

Hi @hadarpo I updated the original reproducer and ran it on a windows machine with the nightly build TF that you had and CUDA 8/cudnnv6, and I wasn't able to reproduce your error.
Since although it looks like the error was from conv3d_backprop, but the stream has already been polluted by earlier CUDA_ERROR_LAUNCH_FAILEDs in cuda_driver.cc and cuda_timer.cc.

@hadarpo have you tried my suggestions? Since I cannot reproduce it and from the error log you posted I don't think it's related to conv3d_backprop, I will close this issue for now. Feel free to comment or open another issue. Thank you!

Hi @yzhwang ,
Thank you for your work.
I tested the tesnsor-flow 1.4 windows-gpu night build on two different machines:
windows server 2016 + Titan X + cuda 8 + cudnn 6
windows 10 + GTX 1050 + cuda 8 + cudnn 6
On both the script conv_ops_3d_test.py ran without any problem.
On both the modified reproducer code failed: titan X with  CUDA_ERROR_LAUNCH_FAILED and GTX 1050 with CUDA_ERROR_LAUNCH_TIMEOUT .
On the Titan X (12GB of memory) if I change in all places 512 to 202 it works perfectly well, and any number above it produces the same failure.
On the GTX 1050, (3GB of available memory) I can get as high as 255. Any number above it produces the following error:
2017-10-05 22:09:24.921703: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_driver.cc:1110] could not synchronize on CUDA context: CUDA_ERROR_LAUNCH_TIMEOUT :: No stack trace available

I have no idea if these errors relates to the same problem or not.

Hi @HggsHntr , as I can't find any Windows server with Pascal/Maxwell architecture, could you try adding this line to the python reproducer:
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
or type:
set CUDA_LAUNCH_BLOCKING=1
in command line?
This is to force CUDA kernels to synchronize after each call, and hopefully will show us more useful error logs. So that we can tell whether it is related to conv3d kernel or not. Thank you!

Hi @yzhwang , It seems like I get the same error. No new information. please see below.
2017-10-09 20:46:28.203661: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-09 20:46:28.494815: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:01:00.0
totalMemory: 12.00GiB freeMemory: 10.06GiB
2017-10-09 20:46:28.494997: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0, compute capability: 5.2)
2017-10-09 20:47:40.380734: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Internal: error destroying CUDA event in context 0000014DD584E910: CUDA_ERROR_LAUNCH_FAILED
2017-10-09 20:47:40.380902: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Internal: error destroying CUDA event in context 0000014DD584E910: CUDA_ERROR_LAUNCH_FAILED
2017-10-09 20:47:40.382666: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-09 20:47:40.383087: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-09 20:47:40.383572: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-09 20:47:40.384041: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-09 20:47:40.384500: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-09 20:47:40.384520: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-09 20:47:40.384919: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-09 20:47:40.385438: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
2017-10-09 20:47:40.385884: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Invalid argument: input event cannot be null
2017-10-09 20:47:40.386252: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Invalid argument: input event cannot be null
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1322     try:
-> 1323       return fn(*args)
   1324     except errors.OpError as e:

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1301                                    feed_dict, fetch_list, target_list,
-> 1302                                    status, run_metadata)
   1303

~\AppData\Local\conda\conda\envs\tftest\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    466           compat.as_text(c_api.TF_Message(status.status)),
--> 467           c_api.TF_GetCode(status.status))
    468   # Delete the underlying status object from memory otherwise it stays alive

NotFoundError: No algorithm worked!
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

During handling of the above exception, another exception occurred:

NotFoundError                             Traceback (most recent call last)
<ipython-input-1-7824d7b75ec7> in <module>()
     26     batchLabels = (np.random.rand(1, n, n, n, 1)>0.5).astype(np.float32)
     27     feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
---> 28     _ = session.run((optimizer, ), feed_dict=feed_dict)
     29

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    887     try:
    888       result = self._run(None, fetches, feed_dict, options_ptr,
--> 889                          run_metadata_ptr)
    890       if run_metadata:
    891         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1118     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1119       results = self._do_run(handle, final_targets, final_fetches,
-> 1120                              feed_dict_tensor, options, run_metadata)
   1121     else:
   1122       results = []

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1315     if handle is None:
   1316       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1317                            options, run_metadata)
   1318     else:
   1319       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1334         except KeyError:
   1335           pass
-> 1336       raise type(e)(node_def, op, message)
   1337
   1338   def _extend_graph(self):

NotFoundError: No algorithm worked!
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

Caused by op 'gradients/Conv3D_grad/Conv3DBackpropFilterV2', defined at:
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\Scripts\ipython-script.py", line 5, in <module>
    sys.exit(IPython.start_ipython())
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\__init__.py", line 125, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\ipapp.py", line 356, in start
    self.shell.mainloop()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\interactiveshell.py", line 480, in mainloop
    self.interact()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\interactiveshell.py", line 471, in interact
    self.run_cell(code, store_history=True)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2698, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2802, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-7824d7b75ec7>", line 21, in <module>
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\training\optimizer.py", line 343, in minimize
    grad_loss=grad_loss)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\training\optimizer.py", line 414, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 353, in _MaybeCompile
    return grad_fn()  # Exit early
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\nn_grad.py", line 88, in _Conv3DGrad
    data_format=data_format)]
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 964, in conv3d_backprop_filter_v2
    data_format=data_format, name=name)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 2936, in create_op
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 1464, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

...which was originally created as op 'Conv3D', defined at:
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\Scripts\ipython-script.py", line 5, in <module>
    sys.exit(IPython.start_ipython())
[elided 7 identical lines from previous traceback]
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-7824d7b75ec7>", line 17, in <module>
    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 846, in conv3d
    padding=padding, data_format=data_format, name=name)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 2936, in create_op
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 1464, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): No algorithm worked!
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

Hi @HggsHntr , thanks for the info! Since we do not have any Windows machine with Pascal/Maxwell card, I will provide some suggestions on how to further debugging the issue:

os.environ['TF_CUDNN_USE_AUTOTUNE'] = '0'
or type:
set TF_CUDNN_USE_AUTOTUNE=0
This will disable the autotune on convolutions, and force all cudnn convolution calls to use the default algorithm.
If this doesn't solve the problem, that means it is not related to conv3d backprop op and must be other CUDA issues. I can tell you where to go from there. If this solves the problem, then it means that during the profiling of one of the internal cudnn kernels, there was some kernel launch error. I'm working on an extensive test suite for cudnn's convolution algorithms to help us spot which internal algorithm caused the problem. Before I release that tool, there is a hacky way to do it yourself though, you could try the following two things:
Add cuda synchronization code and error checking code after each cudnnConvolutionBackwardFilter call:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/stream_executor/cuda/cuda_dnn.cc#L3398
The code can be something like this:

cudaThreadSynchronize();
cudaError_t err = cudaPeekAtLastError();
if (err != cudaSuccess) {
  LOG(FATAL) << "kernel launch error: " << err;
}

Modify this function to get only one internal kernel for cudnnConvolutionBackwardFilter and test which one caused the failure:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/stream_executor/cuda/cuda_dnn.cc#L2540
Either way, if you manage to reproduce the bug in TensorFlow, I will write a standalone cudnn C++ reproducer according to your shape and internal algorithm used and file it to NVIDIA. Let me know if that sounds OK to you. Another way is to wait for the extensive cudnn test tool I'm working on. I should be able to finish it before November.

Hi @yzhwang ,
I set both env vars as you suggested. still no luck. It is still crashing on network sizes larger than roughly 200.  Let me know if you want to see the errors.
About adding some debugging code messages, I don't mind to give it a try but I don't have a building environment for tensorflow. Is there an instruction page on how to do it?
thanks for all you effort.

@yzhwang
2017-10-11 21:48:30.115149: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-11 21:48:30.402802: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:01:00.0
totalMemory: 12.00GiB freeMemory: 10.06GiB
2017-10-11 21:48:30.402991: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0, compute capability: 5.2)
2017-10-11 21:48:47.601437: E C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:3387] failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1322     try:
-> 1323       return fn(*args)
   1324     except errors.OpError as e:

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1301                                    feed_dict, fetch_list, target_list,
-> 1302                                    status, run_metadata)
   1303

~\AppData\Local\conda\conda\envs\tftest\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    466           compat.as_text(c_api.TF_Message(status.status)),
--> 467           c_api.TF_GetCode(status.status))
    468   # Delete the underlying status object from memory otherwise it stays alive

InternalError: cuDNN Backward Filter function launch failure : input shape([1,512,512,512,1]) filter shape([2,2,2,1,1])
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-1-436eb9c8573a> in <module>()
     28     batchLabels = (np.random.rand(1, n, n, n, 1)>0.5).astype(np.float32)
     29     feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
---> 30     _ = session.run((optimizer, ), feed_dict=feed_dict)
     31

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    887     try:
    888       result = self._run(None, fetches, feed_dict, options_ptr,
--> 889                          run_metadata_ptr)
    890       if run_metadata:
    891         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1118     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1119       results = self._do_run(handle, final_targets, final_fetches,
-> 1120                              feed_dict_tensor, options, run_metadata)
   1121     else:
   1122       results = []

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1315     if handle is None:
   1316       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1317                            options, run_metadata)
   1318     else:
   1319       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

~\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1334         except KeyError:
   1335           pass
-> 1336       raise type(e)(node_def, op, message)
   1337
   1338   def _extend_graph(self):

InternalError: cuDNN Backward Filter function launch failure : input shape([1,512,512,512,1]) filter shape([2,2,2,1,1])
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

Caused by op 'gradients/Conv3D_grad/Conv3DBackpropFilterV2', defined at:
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\Scripts\ipython-script.py", line 5, in <module>
    sys.exit(IPython.start_ipython())
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\__init__.py", line 125, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\ipapp.py", line 356, in start
    self.shell.mainloop()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\interactiveshell.py", line 480, in mainloop
    self.interact()
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\terminal\interactiveshell.py", line 471, in interact
    self.run_cell(code, store_history=True)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2698, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2802, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-436eb9c8573a>", line 23, in <module>
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\training\optimizer.py", line 343, in minimize
    grad_loss=grad_loss)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\training\optimizer.py", line 414, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 353, in _MaybeCompile
    return grad_fn()  # Exit early
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 581, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\nn_grad.py", line 88, in _Conv3DGrad
    data_format=data_format)]
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 964, in conv3d_backprop_filter_v2
    data_format=data_format, name=name)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 2936, in create_op
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 1464, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

...which was originally created as op 'Conv3D', defined at:
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\Scripts\ipython-script.py", line 5, in <module>
    sys.exit(IPython.start_ipython())
[elided 7 identical lines from previous traceback]
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-436eb9c8573a>", line 19, in <module>
    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 846, in conv3d
    padding=padding, data_format=data_format, name=name)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 2936, in create_op
    op_def=op_def)
  File "C:\Users\HggsHntr\AppData\Local\conda\conda\envs\tftest\lib\site-packages\tensorflow\python\framework\ops.py", line 1464, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): cuDNN Backward Filter function launch failure : input shape([1,512,512,512,1]) filter shape([2,2,2,1,1])
         [[Node: gradients/Conv3D_grad/Conv3DBackpropFilterV2 = Conv3DBackpropFilterV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_1, gradients/Conv3D_grad/Shape_1, gradients/add_grad/tuple/control_dependency)]]

Hi,
Might be helpful for some of the people that are having the same problem.
A small workaround for this problem that I am using is changing the driver model from WDDM to TCC.
Doing it by
nvidia-smi -dm 1

@HggsHntr
This solves your problem completely on windows 10?

I am having problem with slim.conv3d_transpose(which is wrapper of convolution3d_transpose?)
What's interesting is that the error message does not occur all the time, almost like a coin flipping...
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1359, in convolution3d_transpose
    outputs = layer.apply(inputs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 575, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\layers\convolutional.py", line 1588, in call
    padding=self.padding.upper())
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 1428, in conv3d_transpose
    name=name)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1083, in conv3d_backprop_input_v2
    data_format=data_format, name=name)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
    op_def=op_def)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): cuDNN Backward Data function launch failure : input shape([1,80,160,160,32]) filter shape([5,5,5,32,32])
	 [[Node: head_block/conv0/conv3d_transpose = Conv3DBackpropInputV2[T=DT_FLOAT, data_format="NDHWC", padding="SAME", strides=[1, 1, 2, 2, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](head_block/conv0/stack, head_block/conv0/weights/read/_4949, decode_block_0/Residual_Block/add)]]
	 [[Node: componentwise_overlap_loss/sub/_4957 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3379_componentwise_overlap_loss/sub", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

@qianyizhang
I am not sure about windows 10. I only tested it on windows server 2016.