Plot entire Efficient Frontier

Hi!
I'm trying to use the package to plot the efficient frontier for a set of assets. Just to make the plot a bit nicer, I would like to plot the entire efficient frontier including the lower part (where the returns are lower then the retrun of the minimum risk portfolio). In the example I only see a plot of the upper part of the frontier. Is there a way to get this entire frontier using the package?
Kind regards,
Joost

Hi  @joostbos123,
great question! Indeed, you should be able to plot the inefficient part of the frontier in most cases.
If you are in the completely unconstrained case (i.e. not even a short-sale constraint), you can use the closed-form solution by Merton (1972) discussed in #300.
If you are in a constrained case, consider the following:
You want to minimize the return for a given volatility. Minimizing the portfolio returns is equivalent to maximizing the negative returns (in fact, the return maximization is even implemented as minimization of negative expected returns).
Hacking a bit into the plot_efficient_frontier() to return the list of sigmas and mus, you can do something like:
# with constraints
ef = setup_efficient_frontier()
ef.add_constraint(lambda x: x <= 0.15)
ef.add_constraint(lambda x: x[0] == 0.05)
_, sigmas1, mus1 = plotting.plot_efficient_frontier(ef)

mean_return, sample_cov_matrix = setup_efficient_frontier(data_only=True)
negative_returns = - mean_return
inefficient_frontier = EfficientFrontier(negative_returns, sample_cov_matrix)
inefficient_frontier.add_constraint(lambda x: x <= 0.15)
inefficient_frontier.add_constraint(lambda x: x[0] == 0.05)
_, sigmas2, negative_mus2 = plotting.plot_efficient_frontier(inefficient_frontier)
mus2 = [-mu for mu in negative_mus2]

plt.figure()
plt.plot(list(reversed(sigmas1))+sigmas2, list(reversed(mus1))+mus2)
plt.show()

This yields the following plot, I'll leave it to you to make the formatting pretty :)

Notice that in the constrained case, the frontier is not  reflection symmetric, which can be seen when "flipping up" the inefficient part of the frontier:
plt.figure()
flipped_up_mus = [(max(mus2) - mu) + max(mus2) for mu in mus2]
plt.plot(list(reversed(sigmas1))+sigmas2, list(reversed(mus1))+flipped_up_mus)
plt.show()

This approach should work for cases where you do not have further concave objectives in the return maximization objective.
Hope this helps!

Thanks a lot for your quick reply and the suggested workaround the  get the entire plot in the constrained case!
However, when I use this method on a set of assets of which some assets have negative expected returns (mu) then I recieve the error "target_return should be a positive float" when trying to plot the lower part. This is caused by negative values of the target_return input argument of efficient_return function in effcient_frontier.py.
Do you have any suggestions to overcome this problem as well?
Kind regards,
Joost

@joostbos123 Yes, you simply need to comment out this assertion.

This is a nice workaround, but unless there is a very simple implementation, I think there are other features that need to be prioritised


Add Discussions Tab

We should add a discussions tab to help with future proofing and better support the social community.

I need to take the time to setup this repo correctly but in the meantime I added the Discussions tab.


⚠ Samsung has degraded performance

In 447344f, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 3014 ms

Resolved: Samsung performance has improved in b4aa595.


apt-get: make sure DEBIAN_FRONTEND is propagated to env

DEBIAN_FRONTEND was set to "interactive" in Dockerfile's using an ARG directive.  For some (undocumented?) reason it seems to let this variable each the environment of RUN commands, when building with Docker.  OTOH Podman does not do that (which AFAICT is consistent with Docker's own doc), and with podman build we experience the problem described in #589.
ARG's documented use anyway is to let the caller of the build to set a variable's value, and we don't need that.  OTOH we also want this env value to persist into the container, so installation of build-deps for a given task do not suffer the same problem, so that undocumented "feature" of ARG in Docker is indeed insufficient, it should really have been ENV from the start.

For completion, we do need the envvar set during the builds of our images. tz which is a dependency of something we use (can't remember what) needs user interaction.
Making the variable persist makes sense to me, it shouldn't do any harm and only help.


Trouble setting up nvim-lspconfig // E5108: Error executing lua vim.lua:265: module 'vim.uri' not found

nvim --version:

❯ nvim --version
NVIM v0.5.0-911-gef3230f35
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/cc -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/megaman/git/neovim/build/config -I/home/megaman/git/neovim/src -I/home/megaman/git/neovim/.deps/usr/include -I/usr/include -I/home/megaman/git/neovim/build/src/nvim/auto -I/home/megaman/git/neovim/build/include
Compiled by megaman@HU-00003276

Features: +acl +iconv +tui
See ":help feature-compile"

  system vimrc file: "$VIM/sysinit.vim"
    fall-back for $VIM: "/usr/local/share/nvim"

Run :checkhealth for more info

nvim-lsp version(commit hash):
76f11625481eddb7bcbf86b3e4f592a1aef35b37

:checkhealth result

health#lspconfig#check
========================================================================
  - ERROR: Failed to run healthcheck for "lspconfig" plugin. Exception:
    function health#check[21]..health#lspconfig#check, line 2
    Vim(lua):E5108: Error executing lua ...ack/minpac/start/nvim-lspconfig/lua/lspconfig/health.lua:3: loop or previous error loading module 'lspconfig/configs'

health#nvim#check
========================================================================
## Configuration
  - OK: no issues found

## Performance
  - OK: Build type: RelWithDebInfo

## Remote Plugins
  - OK: Up to date

## terminal
  - INFO: key_backspace (kbs) terminfo entry: key_backspace=\177
  - INFO: key_dc (kdch1) terminfo entry: key_dc=\E[3~

## tmux
  - OK: escape-time: 0ms
  - INFO: $TERM: screen-256color

health#provider#check
========================================================================
## Clipboard (optional)
  - OK: Clipboard tool found: tmux

## Python 2 provider (optional)
  - INFO: `g:python_host_prog` is not set.  Searching for python2 in the environment.
  - INFO: Executable: /usr/bin/python2
  - INFO: Python version: 2.7.16
  - INFO: pynvim version: 0.3.0 (outdated; from /usr/lib/python2.7/dist-packages/neovim)
  - WARNING: Latest pynvim is NOT installed: 0.4.2

## Python 3 provider (optional)
  - INFO: `g:python3_host_prog` is not set.  Searching for python3 in the environment.
  - INFO: Executable: /usr/bin/python3
  - INFO: Python version: 3.7.3
  - INFO: pynvim version: 0.4.2
  - OK: Latest pynvim is installed.

## Ruby provider (optional)
  - INFO: Ruby: ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
  - WARNING: `neovim-ruby-host` not found.
    - ADVICE:
      - Run `gem install neovim` to ensure the neovim RubyGem is installed.
      - Run `gem environment` to ensure the gem bin directory is in $PATH.
      - If you are using rvm/rbenv/chruby, try "rehashing".
      - See :help |g:ruby_host_prog| for non-standard gem installations.

## Node.js provider (optional)
  - INFO: Node.js: v12.19.0
  - INFO: Neovim node.js host: /usr/local/lib/node_modules/neovim/bin/cli.js
  - OK: Latest "neovim" npm/yarn package is installed: 4.9.0

What language server(If the problem is related to a specific language server):
Would be great if I could set up lsp for python3 development.

Operating system/version:
I use WSL(Debian 10) on Windows(10) 1809 build

❯ cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Terminal name/version:
Alacritty

❯ echo $TERM
screen-256color

How to reproduce the problem from neovim startup
I installed nvim v0.5.0-911-gef3230f35 the following way:
git clone git@github.com:neovim/neovim.git
cd neovim
sudo apt-get install ninja-build gettext libtool libtool-bin autoconf automake cmake g++ pkg-config unzip
make CMAKE_BUILD_TYPE=RelWithDebInfo
which nvim
/usr/bin/nvim
mv /usr/bin/nvim /usr/bin/nvim-stable
cp build/bin/nvim /usr/bin/nvim

Actual behaviour
When opening nvim, the following error welcomes me and neovim/nvim-lspconfig plugin doesn't seem to work at all. I use k-takata/minpac as my package manager so I followed as the installation suggested :packadd nvim-lspconfig but it didn't do anything. Welcoming error:
Error detected while processing /home/megaman/.config/nvim/pack/minpac/start/nvim-lspconfig/plugin/lspconfig.vim:
line   14:
E5108: Error executing lua vim.lua:265: module 'vim.uri' not found:
        no field package.preload['vim.uri']
        no file './vim/uri.lua'
        no file '/home/megaman/git/neovim/.deps/usr/share/luajit-2.1.0-beta3/vim/uri.lua'
        no file '/usr/local/share/lua/5.1/vim/uri.lua'
        no file '/usr/local/share/lua/5.1/vim/uri/init.lua'
        no file '/home/megaman/git/neovim/.deps/usr/share/lua/5.1/vim/uri.lua'
        no file '/home/megaman/git/neovim/.deps/usr/share/lua/5.1/vim/uri/init.lua'
        no file './vim/uri.so'
        no file '/usr/local/lib/lua/5.1/vim/uri.so'
        no file '/home/megaman/git/neovim/.deps/usr/lib/lua/5.1/vim/uri.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file './vim.so'
        no file '/usr/local/lib/lua/5.1/vim.so'
        no file '/home/megaman/git/neovim/.deps/usr/lib/lua/5.1/vim.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'

Expected behaviour
No error and working neovim/nvim-lspconfig plugin

from documentation, I guess you need to sudo make install to install rather than only copying the binary file.
make CMAKE_BUILD_TYPE=RelWithDebInfo
sudo make install

Wow. Works! I don't know why I didn't do it like that for the first time. Gave me a lot of headaches.
Your help is much appreciated! Thank you!

from documentation, I guess you need to sudo make install to install rather than only copying the binary file.
make CMAKE_BUILD_TYPE=RelWithDebInfo
sudo make install

I am having this same error, but needing to complete installation without sudo. Installing from source, I used
rm -r build/
make CMAKE_BUILD_TYPE=RelWithDebInfo CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX=$HOME/neovim_dir"
make install
export PATH="$HOME/neovim_dir/bin:$PATH"

as indicated in this documentation, but I am still getting the very same error indicated above (E5108).
Output of nvim -v:
NVIM v0.6.0-dev+164-g50b30de20
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/cc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET_MATCH_LIMIT -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/user/neovim/build/config -I/home/user/neovim/src -I/home/user/neovim/.deps/usr/include -I/usr/include -I/home/user/neovim/build/src/nvim/auto -I/home/user/neovim/build/include
Compiled by user@host

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/home/user/neovim_dir/share/nvim"


this isn't async

I am confused by this lib - looking at the code - it doesn't seem to run off of the main thread - it just uses process.nextTick or setImmediate to delay processing?

Correct, it avoids blocking the event loop for long periods of time if you know you're going to be stringifying a massive object graph.

yes that makes sense...IMO...and I mean no malice..I think it would be
better to moniker this lib "streaming json" or what not
On Oct 14, 2015 8:34 AM, "Stephen Commisso"<EMAIL_ADDRESS>wrote:

Correct, it avoids blocking the event loop for long periods of time if you
know you're going to be stringifying a massive object graph.
—
Reply to this email directly or view it on GitHub
https://github.com/ckknight/async-json/issues/13#issuecomment-148087919.

async should mean "off the event loop" IMO...but I do understand why this
lib exists and do believe it serves a purpose
On Oct 14, 2015 10:22 AM, "Alexander Mills"<EMAIL_ADDRESS>wrote:

yes that makes sense...IMO...and I mean no malice..I think it would be
better to moniker this lib "streaming json" or what not
On Oct 14, 2015 8:34 AM, "Stephen Commisso"<EMAIL_ADDRESS>wrote:

Correct, it avoids blocking the event loop for long periods of time if
you know you're going to be stringifying a massive object graph.
—
Reply to this email directly or view it on GitHub
https://github.com/ckknight/async-json/issues/13#issuecomment-148087919
.

Asynchrony doesn't necessitate multithreading such as with Web Workers (which would be off the event loop). You're probably looking for the term "Concurrency".

actually, disagree. true parallelism means running on sep processors simultaneously. async and concurrency are the same concept.
there can only be two concepts:
concurrency
parallelism
it's up to you where to put asynchrony
I personally like to put asynchrony in the parallelism bin
but it sounds like you like to put it in the concurrency bin, where concurrency in Node.js can be achieved by process.nextTick and setImmediate.
i think that's fair. but I actually think it's more consistent to reserve the term "async" for operations that involve async I/O and the threadpool in V8 that supports those.
it could be, however, that async operations could never be achieved without setImmediate and process.nextTick in combination with the V8 threadpool, and that would make things more interesting for the purposes of this discussion.

in other words, async is the biggest concept in Node.js and it completely relies on the V8 threadpool and has everything to do with I/O operations. so it's somewhat deceptive to start using the term async to describe the concurrency abilities of the JS event loop. that's why I think async should be thrown into the parallelism bin WRT to the node.js ecosystem.

This conversation can scare adopters of this library if not answered and closed. So, here goes my thoughts.

The fundamental concept of what async means in JavaScript is at debate here. The library works as intended by author

I read the code - it recursively goes through properties in an object and performs stringify in batches

This is in line with what most developers in JS are used to the concept of async (heck even async library works this way!)

If you've got a large object that's blocking event loop for stringification, this module will fix that.

Good work


Give the users one/some examples

Hi
To my death I simply can't understand why there aren't any practical examples in your huge documentation on all kinds of subjects.
In am not an expert on many of the issuses, I am looking up in your documentation. So for me it is often a question on how to use af functionality you offer. I dont need to know a lot of background stuff on a specific issue. I simply want a plug and play solution on how to use the functionality in my own code. You sholuld be able to fulfill this demand with 1 or 2 lines of text/code or a few lines of text/code at the most.
I am working in a large organization so therefore I know,  I am not the only one ending up being frustrated, when we read your documentation. Therefore we often end up reading documentation from other sources like StackOverflow and the like.
The way we see your documentation is, that is wriiten by experts to experts and not from experts to common users. And as mentioned above some practical examples could help a lot here.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ed83069f-39a8-e992-95f7-a8b41154b6f4
Version Independent ID: 144fe168-080f-6557-e4b2-a18adb4b89ed
Content: Logging - EF Core
Content Source: entity-framework/core/miscellaneous/logging.md
Product: entity-framework
Technology: entity-framework-core
GitHub Login: @rowanmiller
Microsoft Alias: avickers

@smaldos64 Could you point to some examples from Stack Overflow where the answer there was much more useful than the EF docs?

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.


Remove realm with no authorization constraint / login module passes

https://github.com/kaazing/roadmap/issues/825

Looks good to me.  You just have extra code that does nothing :-)

@dpwspoon I removed the translator in June2016Validator - not sure if I should remove June2016Validator altogether as there might be stuff to check in the future?

Looks like build failed with compilation error...


Admin Sets should be closed for deposit by default

Admin Sets are currently open by default, despite the fact that we've created a new role for depositing to Admin Sets. Admin Sets should instead be closed by default, with the ability for Admin Set Managers to open them up for deposit by all registered users (via some sort of widget -- design needed here -- in the new/edit forms).
Note: the Default Admin Set should remain OPEN by default, but should be able to be made closed via the above widget.
Cc: @hannahfrost @ggeisler

@mjgiarlo @hannahfrost Proposed design below. There's a lot of explanation but my recommendation comes down to:

Add the Registered Users group to the Depositor role for the default admin set only, when the app is first deployed
Add a button labelled "Allow all registered users to deposit" (btn-info btn-sm) just above the current Depositors table whenever that table does not include the Registered Users group. Clicking the button adds the Registered Users group to the Depositor role.
Hide that button whenever the Depositors table does include the Registered Users group

Details
I'll offer a couple of alternatives for the "widget" that enables the Admin Set Manager to easily open an admin set up for deposit: a button and a checkbox. I recommend the button version, because I unchecking the checkbox duplicates the Remove button functionality that is already there and I think that has the potential for confusion (and hiding a checkbox when it is in one state seems wrong to me).
Button Version
Two scenarios: the default admin set and non-default admin sets.
Default Admin Set
There's no need for the widget in this scenario because we'll assign the Registered Users group to the Depositor role by default. The Admin Set Manager can easily revoke that decision by clicking the Remove button.

If the Admin Set Manager does indeed remove the Registered Users group, the widget should appear as described below.
Non-Default Admin Sets
The Registered Users group is not added by default but because this could be a common goal for many admin sets, we want to make it easy and clear to the Admin Set Manager who wants to open the admin set up. So when the Registered Users group is not currently assigned to the Depositor role, we show a button:

(Note that this mockup assumes that ticket #429 is completed -- that's where the No depositors have been added to this administrative set line comes from -- but I don't think that ticket is a blocker for this one; they can be done in either order.)
We would show the button in the same way for the default admin set if the Admin Set Manager has removed the Registered Users group.
I think we still want to show the button if other users have been added as Depositors:

But when the Admin Set Manager clicks this button to add the Registered Users group to the Depositor role, I think we should then hide the button, since it has no purpose in that state.

Alternative Widget
If we went with a checkbox instead of a button, and didn't hide it, it would look like this:

With that approach, the checkbox is acting as a toggle, which seems clear enough. But it also is potentially confusing because unchecking the checkbox does the same thing as the Remove button in the Registered Users row.

@ggeisler This makes sense to me. I have a slight preference for the Button version.

@julesies reported this as a bug in Sufia as well: https://github.com/projecthydra/sufia/issues/3108 I have asked how high of a priority this is for her use cases, to get a sense of whether we should do this in Sufia or Hyrax.

We believe Admin Sets are now closed for deposit.
TODO

[ ] create a new ticket to implement the "Allow all registered users to deposit" functionality mocked-up above.

Created #994. Closing.


scala.js run

Need to add tests

Looks good to me

@lihaoyi added some tests and runMain command too

sounds good
On Wed, 7 Feb 2018 at 4:06 PM, Nikolay Tatarinov<EMAIL_ADDRESS>wrote:

@lihaoyi https://github.com/lihaoyi added some tests and runMain
command too
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/lihaoyi/mill/pull/127#issuecomment-363956059, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATpwjGa8Ef9yJ62zt_hFpRbBBoqyfy_hks5tSjqdgaJpZM4R6TVV
.


Relationship to another article

Hi,i want to know if this network architecture is improved on the previous network architecture in the artical "A Discriminatively Learned CNN Embedding for Person Re-identification"?I noticed a lot of similarities.

Yes.
Because the best model (in the paper) is with the spirit of "A Discriminatively Learned CNN Embedding for Person Re-identification" and GAN.
The verification loss and identification loss are added.


Index out of bounds while editing instances in Scene05

Hey, I was editing the cars(instances) in Scene05 and I'm not able to edit the objects beyond the obj_id 4. Getting
IndexError: index 5 is out of bounds for dimension 2 with size 5
I checked the latent_codes for Scene05 and it contains objects with object ids >5. How do I edit those instances?
I'm well aware that the last 2 dimensions of pose are obj_id and axis and the last dimension of rotation is obj_id.
Passing an object id > 4 to pose and rotation is raising this error.

Hi! @amoghskanda
It's unrelated to the latent codes of the scene. The track_id of each object is related to both the sequence length and the objects in the scene. For instance, maybe you train a short sequence and there are only 5 cars in the whole sequence.

alright. Thank you for the clarification.


fix: Update install command and fix overflow

Tested, works good. Thanks.

Add
Fixes #86
Closes #87

to the end of the commit message and push again.

Supersedes #87

This PR has moved to #93


doc: tutorial stringraw.cue renders oddly

https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/stringraw.md
Note the markdown uses "```json" causing the red rendering
ie this...

$ cue eval stringraw.cue
{
    msg1: "The sequence \"\\U0001F604\" renders as 😄."
    msg2: """
        A regular expression can conveniently be written as:

            #\"\\d{3}\"#

        This construct works for bytes, strings and their
        multi-line variants.
        """
}

should be ...
$ cue eval stringraw.cue
{
    msg1: "the sequence \"\\U0001F604\" renders as 😄."
    msg2: """
        A regular expression can conveniently be written as:

            #\"\\d{3}\"#

        This construct works for bytes, strings and their
        multi-line variants.
        """
}

Similarly, https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/bytes.md exhibits similar (missing quotes around cue export rendered json due to missing quotes in md)
ref also https://github.com/cuelang/cue/pull/48

Fixed. See Issue #48.

This issue has been migrated to https://github.com/cue-lang/cue/issues/49.
For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.


[Enhancement]: re-add riscv64 support

What changes would you like?
I'm about to remove the riscv64 build as the launchpad builders are having issues. This is a reminder to add it back in later.
Any extra information?
No response

Blocked by https://github.com/canonical/snapcraft/issues/4499


Parameterized Queries using 2 or more parameters

Make it possible to use 2 or more paramters for Parameterized Queries.
For Example
'SELECT * FROM table WHERE id=$$id$$ AND date >= $$date$$'

Hi @francot -- this is totally supported. What's happening when you try it? Explorer should build mutiple text boxes for the different params. I just tested this on my install and all seems well.

Hi @chrisclark you have right. It works.
This error:
syntax error at or near "and" LINE 1: ..T * FROM table WHERE id = AND date ... ^
confused me. I have 2 box for the parameter despite the error.
I note that parametric query always return error if the field is not a char type.
for example using 1 parameter:
 'SELECT * FROM table WHERE date = $$date$$'
return
syntax error at end of input LINE 1: SELECT * FROM table WHERE date  = ^
Anyway I can view the Box.
Why it returns error?
Thanks
f

Yep -- Explorer will try to execute the query no matter what, and of course parameters are invalid SQL (if they aren't quoted) so it returns an error. For a time I had explorer not execute the query at all of parameters were present, but I felt it was more useful to execute it so that, if your parameters were quoted, you could at least figure out if the rest of your SQL was valid.
If you can make a good case for different behavior, I'm happy to consider it. Thanks!


Movieku: Failed to install error

Extension information
Movieku
Steps to reproduce

Open the Extension Repository.
Click the Install button on the Movieku Extension.
The Failed to install extension error appears.

Miru version
1.7.2
Other details

Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.

@appdevelpo


values strings, line #8

In the string #8 .
File:
brevent/src/main/res/values/strings.xml

Brevent recognizes whether you have opened or closed app by this way, preventing apps which are in the Brevent list from running in long time.

It is not correct "in long time", it should be, "At the designated time" in the settings of Brevent.
Regards.

@liyufan

@liudongmiao I hesitated when I translated this sentence, untill I saw a sentence in README.md :
Brevent, black prevent, can app-standby (Since Android 6.0) or force-stop apps without ROOT, prevent apps from running in long time.
So I just used it in translation...

@samforo You probably misunderstood the meaning of the sentence. It means

prevent apps from running for extended periods

rather than

prevent apps from running at a time which was previously specified

The time to which it refers, is not the time that is adjusted in the configuration options?
If it isn't the time to configure in settings, perfect .

Just keep it.

@samforo No, it isn't.
The "time" in the sentence has nothing to do with the "time" in settings. It just means a long period.
Regards.


ConstraintLayout 2.0.1

latest version for ConstraintLayout #966 .
@mattleibow we may have issue with this binding, not sure how to resolve it.
In MotionLayout there are 2 Methods (AddTransitionListener and SetTransitionListener) using interface ITransitionListener
and in region Event implementation for Android.Support.Constraints.Motion.MotionLayout.ITransitionListener there are two implemenatation of ITransitionListener interface so we have error that MotionLayout already contains definition for interface methods....
We can remove one of methods  (AddTransitionListener and SetTransitionListener) and it will work but that is not good solution. Do you have idea how to fix it? or it is maybe Xamarin.Android bug?
I also find simular issue https://forums.xamarin.com/discussion/85194/android-binding-library-question

So the add/remove are the better options to keep, so we can do things to the set.
I believe there is an "eventName" thing you can set to a blank string that might prevent the event generation.
<attr path="xxx" name="eventName"></attr>

That might do it.


migrate owens to slurm

Start to migrate owens to slurm by adding owens-slurm cluster and toggling args based of the adapter.

Moving this to draft, I'm trying wrangle JUPYTER_RUNTIME_DIR along with some other stuff like srun to get this to work correctly.


Static relations

Currently relationship definitions in AR are defined as non static getters. In reality an AR class represents a table and an instance of that class represents a record.
Relations are defined at the table level, but because AR uses non-static getters our query classes need to instantiate a dummy model for greedy queries.
Requirements for AR relationships:

One definition for lazy and greedy loading
Support (cached) retrieval via magic getter
Support getting a query object (currently via the getter that defines it)

Pros of current approach:

Uses "native" magic getters (with a small extension)
Has code completion for the getters since they are normal functions

Cons:

No way to get the relationships without instantiating a dummy model
No way to enumerate relations

In Yii1 we had a single function defining the relationships but we also had the dummy model due to lack of late static binding. This allowed for enumeration but had other down sides.
What if, in Yii3 we try to get the best of both? Suppose we define relationships as static functions:
public static function relatedCustomers()
{
    return Has::many (Customer::class, ['id' => 'customer_id']);
}

Internally we would have a sound definition of the relationship, from this we can:

build a lazy query
build a greedy query
enumerate via reflection

From a consumer point of view nothing changes:

getCustomers via __invoke()
->customers  via __get()

Pros:

Relations are now static
No need for dummy instances
OO definition of relationship unrelated to the query object

Cons:

no auto complete for the getter, so could require additional annotations.
breaks BC

This issue is to collect different point of views so please provide yours!

Breaking BC is OK if there are pros and no other cons.
I don't like that getCustomers() and ->customers are currently different. That isn't explicit.
Ability to enumerate relations is a good addition. It was asked multiple times to have it.

Would become more explicit when we get resolvable properties in core.

Some other thing we lose with my proposal is things like forcing joins in a relation, not sure if that is something people do?
Is there anything in the current query object that you can't or shouldn't be doing in a relation?

Some other thing we lose with my proposal is things like forcing joins in a relation, not sure if that is something people do?

Do you mean joinWith()? Why?

Is there anything in the current query object that you can't or shouldn't be doing in a relation?

Nothing I can think of immediately.

I'll try to work on a simple PoC

@samdark could you move this to active-record? All these repos still taking a bit of getting used to :-/


UG: Increase size/focus of the pictures + draw red boxes to highlight the changes

for add, edit command


RuntimeError: Input type (c10::Half) and bias type (float) should be the same

I am using M1 Pro MacBook and I am trying to develop a stablediffusion using mps.
I changed the part about cuda to mps and changed it from ddim.py to float32 because mps did not support float64.
   def register_buffer(self, name, attr):         if type(attr) == torch.Tensor:             if attr.device != torch.device("mps"):                 attr = attr.to(torch.float32).to(torch.device("mps"))         setattr(self, name, attr)
    def make_schedule(self, ddim_num_steps, ddim_discretize="uniform", ddim_eta=0., verbose=True):         self.ddim_timesteps = make_ddim_timesteps(ddim_discr_method=ddim_discretize, num_ddim_timesteps=ddim_num_steps,                                                   num_ddpm_timesteps=self.ddpm_num_timesteps,verbose=verbose)         alphas_cumprod = self.model.alphas_cumprod         assert alphas_cumprod.shape[0] == self.ddpm_num_timesteps, 'alphas have to be defined for each timestep'         to_torch = lambda x: x.clone().detach().to(torch.float32).to(self.model.device)
Since then, this problem has occurred in conv.py
    def _conv_forward(self, input: Tensor, weight: Tensor, bias: Optional[Tensor]):         if self.padding_mode != 'zeros':             return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),                             weight, bias, self.stride,                             _pair(0), self.dilation, self.groups)         return F.conv2d(input, weight, bias, self.stride,                         self.padding, self.dilation, self.groups)
    def forward(self, input: Tensor) -> Tensor:         return self._conv_forward(input, self.weight, self.bias)
Help me please.

Hey, how is it going? Did you figure it out?

No i didnt 😢

Well, that's too bad. I guess we're stuck then.

Are you having the same problem?

I was having that problem when I altered superresolution.py for my use case. You could try running the pipeline provided by diffusers though.. Since this issue got no response, I switched to running the text2img.py, and now, I am getting new error,
(base) root@stablediffusion$ python scripts/txt2img.py --prompt "a professional photograph ofan astronaut riding a horse" --ckpt v2-1_768-nonema-pruned.ckpt --config configs/stable-diffusion/v2-inference-v.yaml --H 768--W 768"
"RuntimeError: expected scalar type BFloat16 but found Float"

Wait, you're getting the problem from something else, not the superresolution.py script

Ok, I was fooling around, and got "RuntimeError: Input type (c10::Half) and bias type (float) should be the same" error again. It doesn't show up when you use "cuda". Why are you trying mps instead?

You can use this fork supported mps!
https://github.com/Tps-F/stablediffusion

it's not work 🥺
this occurred error
Traceback (most recent call last):
File "/Users/blackcat/study/stablediffusion/scripts/txt2img.py", line 393, in
main(opt)
File "/Users/blackcat/study/stablediffusion/scripts/txt2img.py", line 352, in main
samples, _ = sampler.sample(S=opt.steps,
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/models/diffusion/ddim.py", line 107, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/models/diffusion/ddim.py", line 167, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/models/diffusion/ddim.py", line 215, in p_sample_ddim
model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/models/diffusion/ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/models/diffusion/ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/modules/diffusionmodules/openaimodel.py", line 778, in forward
h = module(h, emb, context)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/Users/blackcat/study/stablediffusion/scripts/ldm/modules/diffusionmodules/openaimodel.py", line 86, in forward
x = layer(x)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/opt/anaconda3/envs/cat/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

After days of troubleshooting, I was able to resolve this by upgrading tensorflow to 2.11.0

Try to use v2-inference-v-mac.yaml

@lakejee-rebel
How long does it take to execute?

Here is!
https://github.com/Stability-AI/stablediffusion/pull/163#issuecomment-1422351441

@Tps-F
It's faster because I reduced the batch size.
Thank you.
Are you interested in object detection like ssd(single shot multibox detector) or YOLO?
I want trying ssd in m1 Mac but that model used to CUDA
how to convert CUDA to MPS?

Shall I do it?

I'd appreciate it if you did that.

@Tps-F
Can i follow you?

Sure!
By the way, There seems to be more than one in ssd and YOLO, which one should I support?
Since we are not going to talk here, would you like to go to the discord or something?

Okay good what is your discord id? I will follow you

Thank you-
Ftps#3389

@Tps-F Hi,
I get a similar error using your fork:
.../venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (float) and bias type (c10::Half) should be the same

I used your v2-inference-v-mac.yaml as well, and updated tensorflow to 2.11.0 as suggested but it doesn't work...
Can I connect with you on Discord? I already sent a request... :)

I would like to see all the logs and what you have run.
Can you show me?

Can I connect with you on Discord? I already sent a request... :)

Sure! But I might as well talk about it here in case anyone encounters a similar error in the future!

return F.conv2d(input, weight, bias, self.stride,

RuntimeError: Input type (c10::Half) and bias type (float) should be the same
I also got the above problem @yyahav . I am using ubuntu not macOS

Likewise, please share your logs with us so I can improve.

Likewise, please share your logs with us so I can improve.

Traceback (most recent call last):
  File "/mnt/workspace/stablediffusion/scripts/txt2img.py", line 388, in <module>
    main(opt)
  File "/mnt/workspace/stablediffusion/scripts/txt2img.py", line 347, in main
    samples, _ = sampler.sample(S=opt.steps,
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/models/diffusion/ddim.py", line 104, in sample
    samples, intermediates = self.ddim_sampling(conditioning, size,
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/models/diffusion/ddim.py", line 164, in ddim_sampling
    outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/models/diffusion/ddim.py", line 212, in p_sample_ddim
    model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
  File "/mnt/workspace/stablediffusion/ldm/models/diffusion/ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/models/diffusion/ddpm.py", line 1335, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 797, in forward
    h = module(h, emb, context)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/workspace/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 86, in forward
    x = layer(x)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/tommy/anaconda3/envs/t2im/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

This is my logs after I run: python scripts/txt2img.py --prompt "a professional photograph of an astronaut riding a horse" --ckpt <path/to/768model.ckpt/> --config configs/stable-diffusion/v2-inference-v.yaml --H 768 --W 768
(Link: https://github.com/Stability-AI/stablediffusion#reference-sampling-script)
Could you give me any suggestion @yyahav and @Tps-F

I know you are using ubuntu, could you try using config for mac?
https://github.com/Tps-F/stablediffusion/blob/mps-cpu-support/configs/stable-diffusion/mac/v2-inference-v-mac.yaml

I think the reason this happens is because you are using fp16

Could you try to remove --xformers flag?

python scripts/txt2img.py --prompt "a professional photograph of an astronaut riding a horse" --ckpt checkpoints/v2-1_768-ema-pruned.ckpt --config configs/stable-diffusion/v2-inference-mac.yaml --H 768 --W 768 --precision full

I dont use --xformers

Could you try to remove --xformers flag?

I think that error is caused by xformers, so I'd like you to try uninstalling it

when use diffuser pipeline , i meet this,
my solution is :
pipe.to("cuda", torch.float16)
try :
variable.to(device,  torch_type)
and then it run no problem

I receive the same error when using txt2img on webui. Running on Docker without venv (see Dockerfile at https://github.com/helv-io/stable-diffusion-webui). It's an untainted build.
Running on a 8GB Tesla P4. Generation works, but in the webui, I lose the progress bar.
Any advice? I tried changing use_fp16 to False, but same thing.

im using ubuntu
transformers==4.25.1
timm==0.5.4
& i change
pretrained_model.encoder.to(torch.bfloat16)
to
pretrained_model.encoder.to(torch.float)
now it works for me

I was getting these, and eventually deduced what was going on.  These errors appeared during the render process when it'd show updates.  Instead of updates, these errors were appearing... I did some research.
I had been moving the default models folder and using my own.  The folder I had didn't have various files it was looking for.
It turns out I needed to merge in the default files to get it working...
Here for example is what I typed in...
cd stable-diffusion-webui
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git SD-hold
cp -r SD-hold/models/* models/
rm -rf SD-hold

That restores the files to the models folder tree, and solved this for me.

I was getting these, and eventually deduced what was going on. These errors appeared during the render process when it'd show updates. Instead of updates, these errors were appearing... I did some research. I had been moving the default models folder and using my own. The folder I had didn't have various files it was looking for. It turns out I needed to merge in the default files to get it working...
Here for example is what I typed in...
cd stable-diffusion-webui
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git SD-hold
cp -r SD-hold/models/* models/
rm -rf SD-hold

That restores the files to the models folder tree, and solved this for me.

This fixed, it, thanks! Perhaps those required files could be moved somewhere else, leaving the models folder empty for user-managed models? This would improve containerization.

i use yolov5 ，when i put ACmix block into my code leading the same mistake,someone can help me?please!
File "c:/Users/xie/Desktop/yolov5-master/train.py", line 541, in main
train(opt.hyp, opt, device, callbacks)
File "c:/Users/xie/Desktop/yolov5-master/train.py", line 374, in train
compute_loss=compute_loss)
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "c:\Users\xie\Desktop\yolov5-master\val.py", line 210, in run
preds, train_out = model(im) if compute_loss else (model(im, augment=augment), None)
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "c:\Users\xie\Desktop\yolov5-master\models\yolo.py", line 209, in forward
return self._forward_once(x, profile, visualize)  # single-scale inference, train
File "c:\Users\xie\Desktop\yolov5-master\models\yolo.py", line 121, in _forward_once
x = m(x)  # run
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "c:\Users\xie\Desktop\yolov5-master\models\common.py", line 947, in forward
pe = self.conv_p(position(h, w, x.is_cuda))
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py", line 467, in forward
return self._conv_forward(input, self.weight, self.bias)
File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py", line 464, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same

I got a similar RuntimeError when using SDXL in the AUTOMATIC1111 interface. The --no-half-vae option fixed it for me.

I believe there is a VRAM memory leak which happens if the model couldn't be upscaled. It causes VRAM to never release the acquired space (unless you restart webui).
If you happened to see your image couldn't be upscaled - it means your VRAM is now leaked. In that case, you should restart the whole webui to be able to utilize it properly.
My specs:
GPU: Nvidia Quadro RTX 5000 GPU (16 GB VRAM).
OS: Fedora 38.
Acquired VRAM before webui is started:
$ nvidia-smi
Sat Aug 12 01:26:05 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:01:00.0  On |                  N/A |
| N/A   55C    P5              10W /  90W |    863MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2998      G   /usr/bin/gnome-shell                        463MiB |
|    0   N/A  N/A      7491      G   /usr/bin/Xwayland                             7MiB |
|    0   N/A  N/A     20394      G   /usr/bin/nautilus                            84MiB |
|    0   N/A  N/A     35817      G   /usr/lib64/firefox/firefox                  301MiB |
+---------------------------------------------------------------------------------------+

Acquired VRAM after webui is fully started, but before the first image generation job is triggered. I see 7 GB of VRAM is now acquired by webui even so there were no any image generation tasks submitted. Not sure if that's expected or not. It could be that webui is loading the checkpoint model into VRAM on the start, but I'm not sure.:
$ nvidia-smi
Sat Aug 12 01:38:02 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:01:00.0  On |                  N/A |
| N/A   55C    P8               4W /  90W |   8326MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2998      G   /usr/bin/gnome-shell                        462MiB |
|    0   N/A  N/A      7491      G   /usr/bin/Xwayland                             9MiB |
|    0   N/A  N/A     20394      G   /usr/bin/nautilus                            80MiB |
|    0   N/A  N/A     35817      G   /usr/lib64/firefox/firefox                  221MiB |
|    0   N/A  N/A     63368      C   python3.10                                 7396MiB |
|    0   N/A  N/A     63662      C   ...diffusion-webui/venv/bin/python3.10      116MiB |
+---------------------------------------------------------------------------------------+

Acquired VRAM after webui is the first image generation job is started, but failed to be upscaled. We see that after the image generation task is stopped and not running anymore, acquired VRAM is constantly staying at 14 GB. Thus, no new image generation jobs can be submitted:
$ nvidia-smi
Sat Aug 12 01:40:05 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:01:00.0  On |                  N/A |
| N/A   62C    P8              10W /  90W |  15318MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2998      G   /usr/bin/gnome-shell                        450MiB |
|    0   N/A  N/A      7491      G   /usr/bin/Xwayland                             9MiB |
|    0   N/A  N/A     20394      G   /usr/bin/nautilus                            86MiB |
|    0   N/A  N/A     35817      G   /usr/lib64/firefox/firefox                  238MiB |
|    0   N/A  N/A     63368      C   python3.10                                14376MiB |
|    0   N/A  N/A     63662      C   ...diffusion-webui/venv/bin/python3.10      116MiB |
+---------------------------------------------------------------------------------------+

Acquired VRAM after webui is stopped. We see if we stop webui - VRAM is fully releasedm and can be reused:
$ nvidia-smi
Sat Aug 12 01:47:15 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:01:00.0  On |                  N/A |
| N/A   53C    P8               3W /  90W |    859MiB / 16384MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2998      G   /usr/bin/gnome-shell                        454MiB |
|    0   N/A  N/A      7491      G   /usr/bin/Xwayland                             9MiB |
|    0   N/A  N/A     20394      G   /usr/bin/nautilus                            86MiB |
|    0   N/A  N/A     35817      G   /usr/lib64/firefox/firefox                  252MiB |
+---------------------------------------------------------------------------------------+

Conclusion: It seems there is some kind of bug which doesn't release VRAM on failures. I'm also not sure why 7 GB of VRAM is acquired during initial application startup. I'm also not sure why another 7 GB of VRAM is acquired after the image generation process (total 14 GB of VRAM).
A simple solution for now: just stop and start webui again. It will release all VRAM and you will be able to generate images again.

Same error here, it's just the opposite : RuntimeError: Input type (float) and bias type ([float](c10::Half)) should be the same].
Only when I use Hires fix, any model, any upscaler.
That's not a VRAM leak because it does it every time I try, even after computer restart.
My command line : COMMANDLINE_ARGS=--upcast-sampling --medvram --xformers --no-half-vae

I have changed torch from 1.13.1 to 2.0.1 and problem dissapeared!
My environment.yml:
name: stable-diffusion-env
channels:
  - defaults
dependencies:
  - python=3.8  # Change to your desired Python version
  - pip
  - pip:
    - ./invisible-watermark
    - torch==2.0.1
    - safetensors==0.3.3
    - accelerate==0.22.0
    - diffusers==0.20.2
    - transformers==4.33.1

This issue still stands with AMD GPUs so I tried elsewhere...
Wrote to ExLama they say the issue is HIP ( or perhaps ROCm ) :
https://github.com/turboderp/exllama/issues/281
So here is my post on the Github HIP project in hopes they'll resolve :
https://github.com/ROCm-Developer-Tools/HIP/issues/3331

I had fp16 error (fixed by using --precision full and disabling fp16)
Then  error mentioned in this thread appeared
inserting the line of - torch==2.0.1 did not help
I checked anaconda and I have 2.0.1 installed
I have intel HD4000 on hp laptop and windows 8.1.
any ideas anybody ? shall I try to force v2-inference.yaml instead of v2-inference-v one next ? if so. how do I do that ?

Fantastic! My problem has solved. It went wrong because I delete the model.pt in VAE-approx, and I restore it and the problem was solved.
When I tried generating an image first time, it says "No such file or directory: '...\\models\\VAE-approx\\model.pt'" but I ignored it and tried generating again. Then it seems it changed to another error, which is like my last comment. But in fact that's because the file hasn't loaded yet.
So if somebody recives the same error, try checking if there's any "different" error informations before. That might be the ture reason for the error.

Same happened for me today
 python scripts/txt2img.py --prompt "capybara" --ckpt .\checkpoints\512-base-ema.ckpt --n_samples 1  --precision=full  --device=cpu --outdir ..\output
RuntimeError: Input type (struct c10::Half) and bias type (float) should be the same

I didn't use this project, however, when I use HuggingFace and THUDM/chatglm2-6b-int4,
model = AutoModel.from_pretrained("THUDM/chatglm2-6b-int4", trust_remote_code=True)
I encounter the similar error
File "/home/username/.cache/huggingface/modules/transformers_modules/chatglm2-6b-int4/quantization.py", line 248, in forward
    output = inp.mm(weight.t())
RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::Half != float

Then I tried and found the solution
model = AutoModel.from_pretrained("THUDM/chatglm2-6b-int4", trust_remote_code=True).cuda()
This may be somewhat related.

Traceback (most recent call last):
  File "train_inpainting_dreambooth.py", line 876, in <module>
    main(args)
  File "train_inpainting_dreambooth.py", line 859, in main
    save_weights(global_step)
  File "train_inpainting_dreambooth.py", line 758, in save_weights
    images = pipeline(
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 818, in __call__
    mask, masked_image_latents = self.prepare_mask_latents(
  File "/usr/local/lib/python3.8/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 597, in prepare_mask_latents
    masked_image_latents = self.vae.encode(masked_image).latent_dist.sample(generator=generator)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/autoencoder_kl.py", line 164, in encode
    h = self.encoder(x)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/vae.py", line 109, in forward
    sample = self.conv_in(sample)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same
Steps:  40%|████████████▊                   | 6000/15000 [1:16:40<1:55:00,  1.30it/s, loss=0.107, lr=2e-6]
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/dblab/.local/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/home/dblab/.local/lib/python3.8/site-packages/accelerate/commands/launch.py", line 923, in launch_command
    simple_launcher(args)
  File "/home/dblab/.local/lib/python3.8/site-packages/accelerate/commands/launch.py", line 579, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_inpainting_dreambooth.py', '--pretrained_model_name_or_path=runwayml/stable-diffusion-inpainting', '--pretrained_vae_name_or_path=stabilityai/sd-vae-ft-mse', '--output_dir=../../../../Realstc_Vision_profile_hyunwoo5_inpaint', '--with_prior_preservation', '--prior_loss_weight=1.0', '--seed=3434554', '--resolution=512', '--train_batch_size=2', '--train_text_encoder', '--mixed_precision=fp16', '--gradient_accumulation_steps=1', '--learning_rate=2e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=51', '--sample_batch_size=4', '--max_train_steps=15000', '--save_interval=1000', '--save_min_steps=6000', '--save_infer_steps=35', '--concepts_list=concepts_list.json', '--not_cache_latents', '--hflip']' returned non-zero exit status 1.

RuntimeError: Input type (c10::Half) and bias type (float) should be the same
This issue occurs in min_steps(6000step) while running well.
It seems to be a problem when saving.
If removed '--mixed_precision=fp16', runtime is tooo long. So I can't test it yet.
I am now testing after changing torch=2.0.2.
Help me plz 🥺

fwiw I encountered this while using a1111, and restarting fixed it

i use yolov5 ，when i put ACmix block into my code leading the same mistake,someone can help me?please! File "c:/Users/xie/Desktop/yolov5-master/train.py", line 541, in main train(opt.hyp, opt, device, callbacks) File "c:/Users/xie/Desktop/yolov5-master/train.py", line 374, in train compute_loss=compute_loss) File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "c:\Users\xie\Desktop\yolov5-master\val.py", line 210, in run preds, train_out = model(im) if compute_loss else (model(im, augment=augment), None) File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "c:\Users\xie\Desktop\yolov5-master\models\yolo.py", line 209, in forward return self._forward_once(x, profile, visualize) # single-scale inference, train File "c:\Users\xie\Desktop\yolov5-master\models\yolo.py", line 121, in _forward_once x = m(x) # run File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "c:\Users\xie\Desktop\yolov5-master\models\common.py", line 947, in forward pe = self.conv_p(position(h, w, x.is_cuda)) File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py", line 467, in forward return self._conv_forward(input, self.weight, self.bias) File "D:\Anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py", line 464, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same

hello，have you solved this problem?

im running forgeui, im getting this error as well, (RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same) but im seeing it on multiple extensions such as forges controlnet openpose and DeepDanboru Object Recognition, what does that error mean in laymen terms, im guessing maybe its a compatibility issue but im not certain. honestly i would have dove into the py files and dug but im pretty new to stable diffusion so i don't want to start changing things and making it worse. any advice would be greatly appreciated.

I was able to resolve this issue using:
  autocast_ctx = torch.autocast(pipe.device.type)

  with autocast_ctx:


Large number of warnings when running run_def_tests.sh

When running run_def_tests.sh (at least on windows) a large number of eslint errors and warnings are reported. Is this intended or an error in my setup?
> eslint npm/**/test*
D:\flow-typed\definitions\npm\backbone_v1.x.x\test_backbone-v1.x.x.js
  15:14  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  17:14  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  19:22  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  49:29  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  52:23  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  54:17  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  55:16  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\blacklist_v1.x.x\test_blacklist_v1.x.x.js
  4:33  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  5:37  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\bluebird_v3.x.x\test_bluebird-v3.x.x.js
   97:37  warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  116:47  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\bunyan_v1.x.x\test_bunyan_v1.x.x.js
  17:20  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\camelcase-keys_v3.x.x\test_camelcase-keys_v3.x.x.js
  3:15  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\commander_v2.x.x\test_commander_v2.x.x.js
  37:30  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\express_v4.x.x\flow_v0.28.x-v0.31.x\test_overrideUseMethodClassExtension.js
  31:21  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  31:31  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  63:62  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\express_v4.x.x\flow_v0.32.x-\test_overrideUseMethodClassExtension.js
  31:21  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  31:31  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  63:62  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\express_v4.x.x\flow_v0.32.x-\test_response.js
  1:1  error  Test has to include at least 1 negative Tests  flow-typed/negative-tests

D:\flow-typed\definitions\npm\express_v4.x.x\test_express_v4.x.x.js
  71:23  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\fuzzaldrin_v2.x.x\test_fuzzaldrin_v2.js
  6:10  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\inquirer_v2.x.x\test_rx_observable_array_v2.x.x.js
  41:17  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\inquirer_v2.x.x\test_rx_observable_create_v2.x.x.js
  9:9  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\isomorphic-fetch_v2.x.x\test_isomorphic-fetch_v2.js
  7:25  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\js-cookie_v2.x.x\flow_v0.25.x-v0.37.x\test_js-cookie-v2.x.x.js
  11:12  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  33:13  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\js-cookie_v2.x.x\flow_v0.38.x-\test_js-cookie-v2.x.x.js
  11:12  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  33:13  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\koa_v2.x.x\flow_v0.37.x-v0.37.x\test_koa_v2.x.x.js
  29:34  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  92:43  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\koa_v2.x.x\flow_v0.38.x-v0.38.x\test_koa_v2.x.x.js
  29:34  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  92:43  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\map-obj_v1.x.x\test_map-obj_v1.x.x.js
  3:18  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  3:79  warning  Unexpected use of weak type "any"     flowtype/no-weak-types
  3:92  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\mocha_v2.4.x\test_mocha_v2.4.x_hooks.js
   8:16  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  19:20  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  31:15  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  42:19  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\mocha_v2.4.x\test_mocha_v2.4.x_it.js
   8:20  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  26:25  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  46:25  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\mocha_v3.1.x\test_mocha_v3.1.x_hooks.js
   8:16  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  19:20  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  31:15  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  42:19  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\mocha_v3.1.x\test_mocha_v3.1.x_it.js
   8:20  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  26:25  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  46:25  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\normalizr_v2.x.x\test_normalizr-v2.js
  4:40  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  5:42  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  8:25  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\object-assign_v4.x.x\test_object-assign_v4.x.x.js
  5:16  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  6:16  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\primus_v5.x.x\test_primus-v5.js
  9:63  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\primus_v6.x.x\test_primus-v6.js
  9:63  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\prismic.io_v3.x.x\test_prismic.io_v3.x.x.js
   44:37  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   49:44  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   53:44  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   55:72  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   56:59  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   71:27  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   74:43  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  103:29  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  140:35  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  148:43  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  157:37  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  164:28  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  165:34  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  166:45  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  300:29  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\ramda_v0.21.x\flow_v0.23.x-v0.27.x\test_ramda_v0.21.x.js
  129:37  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  142:30  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  258:14  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\ramda_v0.21.x\flow_v0.28.x-v0.30.x\test_ramda_v0.21.x_functions.js
  47:16  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  47:27  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\ramda_v0.21.x\flow_v0.28.x-v0.30.x\test_ramda_v0.21.x_object.js
   17:40  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
   33:27  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  106:11  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  112:14  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  114:22  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  116:12  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  120:10  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\react-dnd_v2.x.x\test_react-dnd-v2.js
  72:26  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\react-redux_v4.x.x\flow_v0.30.x-\test_Provider.js
  6:11  error  Parsing error: Unexpected token, expected ,

D:\flow-typed\definitions\npm\react-redux_v5.x.x\flow_v0.30.x-\test_Provider.js
  6:11  error  Parsing error: Unexpected token, expected ,

D:\flow-typed\definitions\npm\redux-form_v5.x.x\test_redux-form_v5.js
  19:25  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  20:20  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  21:21  warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  33:32  warning  Unexpected use of weak type "any"       flowtype/no-weak-types

D:\flow-typed\definitions\npm\redux-saga_v0.11.x\flow_v0.28.x-v0.37.x\test_redux-saga_0.11.x.js
   62:36   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:49   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:47   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:60   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:58   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:71   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:69   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:82   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:80   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:93   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   67:91   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   67:104  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  199:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  532:17   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  541:19   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  564:17   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  568:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  568:31   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  569:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  570:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  571:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  572:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  573:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  574:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  575:27   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  575:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  607:32   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  607:54   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  611:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  611:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  654:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  654:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types

D:\flow-typed\definitions\npm\redux-saga_v0.11.x\flow_v0.38.x-\test_redux-saga_0.11.x.js
   62:36   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:49   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:47   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:60   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:58   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:71   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:69   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:82   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:80   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:93   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   67:91   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   67:104  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  199:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  532:17   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  541:19   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  564:17   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  568:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  568:31   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  569:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  570:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  571:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  572:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  573:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  574:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  575:27   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  575:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  607:32   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  607:54   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  611:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  611:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  654:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  654:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types

D:\flow-typed\definitions\npm\redux-saga_v0.13.x\flow_v0.36.x-v0.37.x\test_redux-saga_0.13.x.js
   60:27   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   60:40   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   61:36   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   61:49   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:47   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:60   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:58   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:71   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:69   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:82   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:80   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:93   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:91   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:104  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   68:55   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   68:68   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  238:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  296:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  354:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  790:17   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  799:19   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  822:17   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  837:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  837:31   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  838:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  839:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  840:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  841:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  842:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  843:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  844:27   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  844:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  876:32   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  876:54   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  880:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  880:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  922:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  922:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  973:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  973:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  974:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types

D:\flow-typed\definitions\npm\redux-saga_v0.13.x\flow_v0.38.x-\test_redux-saga_0.13.x.js
   60:27   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   60:40   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   61:36   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   61:49   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:47   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   62:60   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:58   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   63:71   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:69   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   64:82   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:80   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   65:93   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:91   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   66:104  warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   68:55   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
   68:68   warning  Unexpected use of weak type "any"       flowtype/no-weak-types
  238:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  296:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  354:21   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  790:17   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  799:19   warning  Unexpected use of weak type "Function"  flowtype/no-weak-types
  822:17   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  837:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  837:31   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  838:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  839:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  840:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  841:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  842:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  843:22   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  844:27   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  844:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  876:32   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  876:54   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  880:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  880:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  922:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  922:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  973:28   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  973:37   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types
  974:60   warning  Unexpected use of weak type "Object"    flowtype/no-weak-types

D:\flow-typed\definitions\npm\rxjs_v5.0.x\test_rxjs.js
  145:56  warning  Unexpected use of weak type "any"  flowtype/no-weak-types
  146:52  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\twit_v2.x.x\test_twit_v2.x.x.js
  16:30  warning  Unexpected use of weak type "any"  flowtype/no-weak-types
  17:29  warning  Unexpected use of weak type "any"  flowtype/no-weak-types

D:\flow-typed\definitions\npm\underscore_v1.x.x\test_underscore-v1.js
  172:20  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types
  173:13  warning  Unexpected use of weak type "Object"  flowtype/no-weak-types

✖ 244 problems (3 errors, 241 warnings)

@doberkofler do you wanna send a PR to fix this?

Closing as this behaviour no longer seems to be happening. Please re-open the issue if that's not the case!


[5.1] Call realpath on everything rather than just those in config.php

A similar improvement might be applied for view paths (see config/views.php).


Add test for join dshape with string option types.

Also add xfail test for 'outer' join with mixed option / non option
types.
This is to test a case with join after blaze/datashape#213 bugfix.

Closes #1241.


Request Clearart

Can you add the possibility to download also the clearart?

Artwork Dump will download anything configured and provided by scrapers. A debug log will show an error if AD tried and failed to download an image, otherwise one just wasn't provided by the scraper.


bump version

before we merge this we might want to update the warning mechanism...

before we merge this we might want to update the warning mechanism...

preliminary warnings are collected since #313  We'll refine this in another release.


Difficulty training model with adversarial regularization follow-up

Hi @lwsong,
Thank you for your suggestions. I tried them out but am still struggling to replicate the results. I have some follow-up questions that I'd like to ask you.

What is the adversarial regularization term that you used for your reported results in Table 2?

The loss used for the inference model in the published code is the MSE between predictions in the range [0, 1] and the binary feature (training data/reference data). My loss converges immediately to 0.25 and does not decrease in any further epochs. Did you also find this to be the case, or did the inference model perform better at some point during training?

In Algorithm 1 from the Nasr et al. paper, the classifier learns from a training set (D) and the inference model learns from the same training set (D) and a disjoint reference set from the same population (D’). Later in the paper though, the authors state that D^A (number of known members of the training set) and D’^A are used for training the inference model. Could you please help me understand how D^A and D’^A are used during training?

Are the attack accuracies for a model trained with adversarial regularization reported in Table 2 (Purchase100 – 67.6% undefended, 51.6% defended and Texas100 63.0% undefended, 51.0% defended) computed from your own experiments, or are these numbers taken from the Nasr et al. paper?

Were you able to replicate the results of Nasr et al. for Cifar100?

Thanks again for your time!

For the Purchase100, I use 3.0 as the weight for adversary regularization term; for Texas100, I use 10.0 as the weight. Also, I saved all checkpoints for each training step, and finally selected the checkpoint with accuracy matching the original paper.

I just reran the code. In the first epoch, the MSE loss indeed keeps being 0.25. However, starting from the second epoch, MSE loss gradually decreases to around 0.23. More importantly, I think you should check the top 1 accuracy of the inference model. If it stays around 0.5 during all training epochs, it truly indicates that the inference attack model learns nothing during training.

From my understanding, D^A and D'^A are not used during training the model with adversarial regularization. Instead, they are used in evaluating the privacy risk of the defended model after training: the adversary is assumed to know some training samples (D^A) and some non-training samples (D'^A), it trains the inference model from the defended model's predictions over D^A and D'^A, and finally test the inference attack success over left training and test samples (excluding D^A and D'^A).

The values are taken from Nasr et al. paper. We also ran their attacks on our locally training models, the attack accuracies are pretty close to values reported in their paper.

I did not spend time training CIFAR100 models. If you meet some problems, you can try to use different weights for the adversarial regularization term. For example, if your trained model has pretty low accuracy, you should decrease the weight for the adversarial regularization. Also, as you can see in my response 1, my hyperparameter setting on Texas100 is actually different from the original paper.

Thank you very much @lwsong for your thoughtful answers. I really appreciate you taking the time to reply. I’ve tried incorporating these tips but am still not able to replicate the results. Please find below my response to your comments.

For Purchase100, Nasr et al. report that the classifier has a top 1 training/testing accuracy of 92.2%/76.5% with defense and a top 1 training/testing accuracy of 100%/80.1% without defense. I haven’t been able to come close to reproducing either of these results. In fact, the top 1 training accuracy I get without defense never gets above 1%. I know something is going wrong during training, but I am not sure what the problem is. The model I am using has the same structure as the PurchaseClassifier in your file adversarial_regularization/utils.py. What were the hyperparameters that you used for training? I have tried using the hyperparameters released in Nasr et al.’s code, as well as variations such as altering the optimizer, learning rate, weight initializer, batch size, and activation functions. Additionally, I have tried training using the “random_r_purchase100.npy” saved randomization from your repository. Are these the randomizations that you used to train the models, and did you use 10% for the training data and 30% for the attack (reference) data during the original training?

For Purchase100, the released code for Nasr et al. has an outer loop that conducts adversarial regularization for 20 epochs, but within each epoch the training is conducted for an additional 76 epochs. I see in your posted screenshot that you also train for 20 epochs and are showing data from the 8th epoch. Does this correspond to the 608th epoch, as in Nasr et al.’s code, or the actual 8th pass through all the training data batches? Furthermore, the Nasr et al. code trains the classifier and inference model separately for one and five warmup epochs, respectively. Did you also use the same warmup epoch amounts? I’ve tried varying these numbers, but it doesn’t help either way. Lastly, is the top 1 accuracy of each row from this screenshot the accuracy calculated for a single batch of training data within one epoch? Assuming you use 10% of the Purchase100 data for training and use a batch size of 128, then there will be 153 distinct batches to iterate through for a single epoch. This corresponds to the 153 number that I see, but since there are multiple rows for (101/153), it's unclear to me where the reported accuracies are calculated from.

Thanks for this explanation! It makes more sense now.

See above.

Thanks, I’ll look into this once I can replicate the Purchase100 results!

More generally, have you considered releasing your code for adversarial regularization training? I think that it would go a long way towards supporting reproducible research in this area!
Thanks again for your time.

Hi @ckap100 , I have uploaded the training code for adversarial regularization, please take a look.

The training code is almost the same as Nasr et al. code in their repo, that is also why I did not include it before. The biggest issue you have right now is that your ML model on Purchase/Texas datasets learns nothing as you mention your training accuracy is never above 1%. Clearly, there should be some bugs in your code. You should fix those bugs before evaluating the privacy risks.
In the code, the outer loop with 20 epochs is to train the model on the whole dataset with 20 epochs. The inner loop with a range of 76 is to train the model on every 2 batches. I simply follow Nasr et al.'s code to print the loss and accuracy information, which is average estimation over the current and previous batches.
In summary, you should first figure out why your model's training accuracy is below 1%. After fixing that bug, please go ahead to measure the privacy issues.