[QUESTION] Where to find pubchem_rd-canonical_smiles.smi?

I couldn't find in the source the link you used to build the pubchem_rd-canonical_smiles dataset. Do you have a link I could use?

Although we were unable to open-source our curated dataset of 91 million molecules from PubChem, we can outline the steps taken to achieve it. Using the PubChem dataset from MoLFormer (https://ibm.ent.box.com/v/MoLFormer-data), we canonicalized the molecular strings and removed duplicates to ensure uniqueness. Following this, a molecular transformation process was applied to validate the molecules derived from the unique SMILES strings.

I'm closing this issue because it has been inactive for 34 days.
Please, feel free to reopen this issue if it has been closed in error.


API PSS tests flake

These tests sometimes flake. Also, they take too long to execute, we should shorten the timeouts there and somehow simplify the test to not block for so long.
=== RUN   TestPssWebsocketSingleHandler
--- PASS: TestPssWebsocketSingleHandler (4.73s)
=== RUN   TestPssWebsocketSingleHandlerDeregister
    asm_amd64.s:1374: timed out waiting for message
--- FAIL: TestPssWebsocketSingleHandlerDeregister (20.18s)

more
=== RUN   TestPssWebsocketMultiHandler
    asm_amd64.s:1374: timed out waiting for message
    asm_amd64.s:1374: timed out waiting for message
--- FAIL: TestPssWebsocketMultiHandler (37.28s)


Fix #222 Added a signout URL for Cognito

Just a couple of things to note.

I'm going to make this more generic so it'll work for the other identity providers.
Also, there's currently a bug in how this is implemented with loginWithPopup.

Proposed API
const auth = new SalteAuth({
  redirectUrl: {
    loginUrl: 'https://salte.io/app',
    logoutUrl: 'https://salte.io'
  }
  // ...
});

Closing since you have a new PR


Implement Fast-HotStuff Block Construction Signal

Current dependencies on/for this PR:

feature/proof-of-stake

PR #649

PR #648

PR #650

PR #654

PR #656

PR #657

PR #658

PR #663   👈

This comment was auto-generated by Graphite.


ci: update reference docs

This PR is auto-generated by a nightly GitHub action to update the reference docs from source code in withastro/astro.

Fixing this at the source!


nixos/config/timezone: Disallow spaces

Motivation for this change
#20115 causes a very strange and hard to debug error, and it's easy to accidentally write "New York" instead of "New_York"
I would have preferred a check morally equivalent to tz: lib.pathExists "${pkgs.tzdata}/share/zoneinfo/right/${tz}", but I can't figure out how to do so in a pure manner without hardcoding an enum with all the contents.
This should prevent the most frequent mistake anyway.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

What's the error when user writes timezone with space?

Sorry, I forgot to link issue #20115.
We also just had a user in #nixos who had nixos-install fail in an even wonkier fashion
setting up /etc...
/etc/resolvconf.conf: line 5: hopopt: command not found
/etc/resolvconf.conf: line 6: icmp: command not found
/etc/resolvconf.conf: line 7: igmp: command not found
/etc/resolvconf.conf: line 8: ggp: command not found
/etc/resolvconf.conf: line 9: ipv4: command not found
/etc/resolvconf.conf: line 10: st: command not found
/etc/resolvconf.conf: line 11: tcp: command not found
/etc/resolvconf.conf: line 12: cbt: command not found`

No, I mean, with this PR, what would be the error message?

Oh!

The option value time.timeZone' in /etc/nixos/configuration.nix' is not a null or string without spaces.


Allow backgrounds to dictate their geometry

As discussed in #234, the background program should be able dictate where it would like to be drawn.
If it wants to drawn e.g below the title bar (which it can do so by querying it in D-Bus, hypothetically), then it can do that. It's the one dealing with the background image, it knows the correct thing to do.

This is outdated, and has been fixed.


Will skipping of emoji and tags return?

I updated to the YNAB API version this week and transactions have emoji and hashtags in the memo, where I had opted out of them in v1.

@adambowles i removed them in v2 as i wanted to keep things simple. Feel free to submit a PR if you need them :)

Roger, I'll have a look


Rebuild R3.5.1

Testing rebuild as proposed in PR https://github.com/conda-forge/staged-recipes/pull/10297

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

The old r-base that was getting picked up was from defaults, rather than conda-forge, yes?

@dpryan79 exactly! Do you know a solution?

Despite this rebuild, the recipe for r-rmumps (https://github.com/conda-forge/staged-recipes/pull/10470) is still failing on macos. It continues to pick r-base 3.5 from defaults and hence with the wrong libgfortran3.

Despite this rebuild, the recipe for r-rmumps (conda-forge/staged-recipes#10470) is still failing on macos. It continues to pick r-base 3.5 from defaults and hence with the wrong libgfortran3.

@sgsokol This was not merged yet because windows build for VS2008 failed. New package is not yet available for osx-64.

VS2008 build doesn't matter as this is using mingw

After the merge, r-rmumps works like a charm on osx too. Thank you for fixing this.


Dev

Coverage remained the same at 100.0% when pulling 17999110b241642e15b03aa95113320acfbd4b4e on dev into cc0135f9e5e2c9c19445cf055109d9a8ef721172 on master.


Create OpenMetadata instance with 1M entities to test scalability

Create a faker python script to generate randomized entities for DataSources, Dashboards, Topics
Create a JVM with 8G
Run loader tests to get latencies and capture the performance metrics


One-time version check

Closes #1281
Summary:
Adds a one-time check to the Epidata client, which logs a warning when the current "remote" server version (https://api.delphi.cmu.edu/epidata/version) does not match the client version.
This is limited to debug mode for now but can be easily extended to run by default.
Also adds a test to verify this behavior.
Prerequisites:

[x] Unless it is a documentation hotfix it should be merged against the dev branch
[x] Branch is up-to-date with the branch to be merged with, i.e. dev
[x] Build is successful
[x] Code is cleaned up and formatted

Let's do #1296 first and then you can revisit this

@melange396 since #1296 is merged, we can move forward with this PR; lmk if the current iteration looks good!


bug: Seems to cause Lag over ssh

Neovim version (nvim -v)
0.10.0
Operating system
Linux, Fedora 40
Terminal emulator / GUI
Gnome Terminal
Describe the bug
when using nvim with this plugin over an ssh connection nvim becomes quite slow and laggy, regardless if a markdown file is open or not. Disabling the plugin makes the issue go away.
Expected behavior
no lag
Additional information
No response

Thanks for brining this up, that's definitely interesting.
Outside of markdown files this plugin does listen to a fair number of events: ModeChanged, WinResized, FileChangedShellPost, FileType, & TextChanged.
The main problem here is likely the usage of the TextChanged event, I'll see if I can remove the need for this. Will update shortly!

I've pushed a pretty substantial refactor to how auto commands are setup: https://github.com/MeanderingProgrammer/markdown.nvim/commit/14b3a01fbd7de25b03dafad7398e4ce463a4d323.
This keeps only the FileType / WinResized as auto commands that run globally. Everything else is setup at a buffer level and only on markdown buffers, which should remove most of the background performance penalty.
Would you mind updating the plugin and letting me know if the performance is better? I'm super curious!

I'll give it a shot later today!

As far as I can tell the lag is gone! Thanks so much!

Awesome, glad it's working!


python311Packages.types-redis: <IP_ADDRESS>40311 -> <IP_ADDRESS>40409

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python311Packages.types-redis is: Typing stubs for redis
meta.homepage for python311Packages.types-redis is: https://github.com/python/typeshed
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found <IP_ADDRESS>40409 with grep in /nix/store/qbkwcb2fr82h4vhdhmdjri1bkzvvixp3-python3.11-types-redis-<IP_ADDRESS>40409
found <IP_ADDRESS>40409 in filename of file in /nix/store/qbkwcb2fr82h4vhdhmdjri1bkzvvixp3-python3.11-types-redis-<IP_ADDRESS>40409

Rebuild report (if merged into master) (click to expand)

8 total rebuild path(s)

8 package rebuild(s)

First fifty rebuilds by attrpath
python311Packages.gptcache
python311Packages.guidance
python311Packages.redis-om
python311Packages.types-redis
python312Packages.gptcache
python312Packages.guidance
python312Packages.redis-om
python312Packages.types-redis

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/qbkwcb2fr82h4vhdhmdjri1bkzvvixp3-python3.11-types-redis-<IP_ADDRESS>40409 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python311Packages.types-redis https://github.com/r-ryantm/nixpkgs/archive/ecf60ecf68bcc2f2fc48109e8384f8a09538a2af.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/ecf60ecf68bcc2f2fc48109e8384f8a09538a2af#python311Packages.types-redis

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/qbkwcb2fr82h4vhdhmdjri1bkzvvixp3-python3.11-types-redis-<IP_ADDRESS>40409
ls -la /nix/store/qbkwcb2fr82h4vhdhmdjri1bkzvvixp3-python3.11-types-redis-<IP_ADDRESS>40409/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

16 packages built:

python311Packages.gptcache
python311Packages.gptcache.dist
python311Packages.guidance
python311Packages.guidance.dist
python311Packages.redis-om
python311Packages.redis-om.dist
python311Packages.types-redis
python311Packages.types-redis.dist
python312Packages.gptcache
python312Packages.gptcache.dist
python312Packages.guidance
python312Packages.guidance.dist
python312Packages.redis-om
python312Packages.redis-om.dist
python312Packages.types-redis
python312Packages.types-redis.dist

Maintainer pings
cc @gador for testing.

Result of nixpkgs-review pr 303804 run on x86_64-linux 1

16 packages built:

python311Packages.gptcache
python311Packages.gptcache.dist
python311Packages.guidance
python311Packages.guidance.dist
python311Packages.redis-om
python311Packages.redis-om.dist
python311Packages.types-redis
python311Packages.types-redis.dist
python312Packages.gptcache
python312Packages.gptcache.dist
python312Packages.guidance
python312Packages.guidance.dist
python312Packages.redis-om
python312Packages.redis-om.dist
python312Packages.types-redis
python312Packages.types-redis.dist


Add .lowest_common_ancestor

An alternative to #326 that doesn't use .unscoped

@robotdana sqlite3 tests are failing -- can you look into that?


Fastnlight: Watch Personal[\W_]+Memory[\W_]+Curator

Fastnlight requests the watch of the watch_keyword Personal[\W_]*+Memory[\W_]*+Curator. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Personal[\W_]*+Memory[\W_]*+Curator has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by VLAZ in Charcoal HQ


Build fails with fork/OpenXR-SDK-Source/src/loader/openxr-loader.map: No such file

I am unable to build this code in Linux usig gcc version 9.3.1 20200518 .
Here are the full cmake and make output logs:
openxrSDK_cmake_output.txt
It seems that all required dependencies are in order... proceeding to $make
openxrSDK_make_output_error.txt
x86_64-alt-linux-g++: error: fork/OpenXR-SDK-Source/src/loader/openxr-loader.map: No such file or directory make[2]: *** [src/loader/CMakeFiles/openxr_loader.dir/build.make:270: src/loader/libopenxr_loader.so.1.0.12] Error 1 make[1]: *** [CMakeFiles/Makefile2:460: src/loader/CMakeFiles/openxr_loader.dir/all] Error 2 make: *** [Makefile:149: all] Error 2
CMakeError.log

It appears that the problem was a space in the dir name in the path, where the source was unpacked.
Worth a warning in the Readme, IMHO.


[WIP] Add/update documentation re: plugins

Need to decide where to put the markdown -- it could be a standalone page or a section somewhere else.

squash & merge


Implement "pico" GPT example

Response to request in issue https://github.com/sbrunk/storch/issues/44.
Attempt to rewrite the "pico" example from  Karpathy's "Let's build GPT: from scratch, in code, spelled out" in storch.

@sbrunk or anyone else. I need some assistance in this work . To code the "pico" example, I need the Embedding operator. In my branch I have added this  here. I have also added comments and made sure ScalaDoc is ok (minus the math expressions).
The code I am working on now, is the BiGram class. If I understand the code correctly, I have to pass a Tensor of shape/size (B,T) and get a Float back. According to the native code that seems to be a call to the forward method. So I am using this in the embedding class as per the other modules:
  def apply(t: Tensor[Int64]): Tensor[D] = Tensor(nativeModule.forward(t.native))

And this is a problem because I get the error:
[error] 101 |final class Embedding[D <: DType: Default](
[error]     |            ^
[error]     |class Embedding needs to be abstract, since def apply(v1: T1): R in trait Function1 in package scala is not defined
[error]     |(Note that
[error]     | parameter T1 in def apply(v1: T1): R in trait Function1 in package scala does not match
[error]     | parameter torch.Tensor[torch.Int64] in def apply(t: torch.Tensor[torch.Int64]): torch.Tensor[D] in class Embedding in package torch.nn.modules.embed
[error]     | )

I think this is because we extend from TensorModule :
trait TensorModule[D <: DType] extends Module with (Tensor[D] => Tensor[D]):

In other words, the apply from (Tensor[D] => Tensor[D]) assumes the input and output are of the same type. Do we have other operators were this is not true? If not, how should we handle this?
On a related note, is it possible to constrain the Tensor by its shape?
TIA

In order to keep going I have used the following solution:
  def apply(t: Tensor[D]): Tensor[D] = Tensor(nativeModule.forward(t.native))
  @targetName("apply_T_D")
  def apply[T<:DType](t: Tensor[T]): Tensor[D] = Tensor(nativeModule.forward(t.native))

Is this ok for a final solution?

@hmf You're right Embedding is an example where the input type might be different from the output, so we can't inherit from TensorModule.
Note that @davoclavo has also added Embedding and a few other modules in #36 (haven't been able to finish and merge that yet, unfortunately) and added a more generic TensorModuleBase to tackle this issue:
https://github.com/sbrunk/storch/blob/05f7dbdca35daa0589447ad0d4eadbefe38e1aeb/core/src/main/scala/torch/nn/modules/sparse/Embedding.scala#L58-L68
https://github.com/sbrunk/storch/blob/05f7dbdca35daa0589447ad0d4eadbefe38e1aeb/core/src/main/scala/torch/nn/modules/Module.scala#L125-L127
So eventually we need to merge your solutions but for now you could also just inherit from nn.Module and add then use your apply method:
  def apply[T<:DType](t: Tensor[T]): Tensor[D] = Tensor(nativeModule.forward(t.native))

On a related note, is it possible to constrain the Tensor by its shape?

Right now, we're tracking only the dtype at compile time. We might add that in the future though.

@sbrunk I have looked at the embedding class and my version its pretty close to it. Currently cannot search @davoclavo's branch, but I think I can copy and use that code (minimum set of classes with updated docs). Might be easier on your side.
In the meantime if you do merge into the main branch, I will update accordingly. Ok, with you?

Question about cross entropy functions. IThe orgial code uses something like:
import torch
import torch.nn as nn
from torch.nn import functional as F

...
loss = F.cross_entropy(logits, targets)
...
            probs = F.softmax(logits, dim=-1) # (B, C)

I see that we have 2 options, a function in the Loss package (does not exist yet, only binary version available) and the torch.nn.loss.CrossEntropyLoss version. The storch examples use the latter.
What are the advantages/disadvantages of using one or the other?

I see that we have 2 options, a function in the Loss package (does not exist yet, only binary version available) and the torch.nn.loss.CrossEntropyLoss version. The storch examples use the latter.
What are the advantages/disadvantages of using one or the other?

PyTorch has a functional and a class/module variant for most of it's nn operations. See torch.nn.functional.cross_entropy
and torch.nn.CrossEntropyLoss. The class variant usually inherits from Module to it's easy to put it into containers expecting modules.
The functional variant does not contain any state, you call it directly with the tensor inputs and other arguments. The class/module variant can be initialized first with init parameters, and then later reused for different inputs. If you have modules with learnable weights/parameters, the module variant also helps you manage that state (makes it easier to update all weights of your model etc.).
For stateless ops without weights, like cross_entropy the class variant doesn't have much advantage except for reuse, so you can also just use the functional variant but it doesn't make much of a difference after all.

I will also be available to continue work on the pending PR to get it merged, in case I can help in any way @sbrunk

@davoclavo feel free to take over #36 again if you have capacity. I've merged main into it with some improvements of the native bindings but since Scala Days is only 4 weeks away so I'd like to focus on getting my Storch talk ready first. Happy to help/review etc. but I'm not sure I'll be able to actually work on it before the talk.

@davoclavo Thanks for the assist. Please note that at this time I am working on the very simple "video" version. My aim here is to learn about GPT.
I will look at your code and incorporate all I can to make merging easier.

Questions regarding softmax. I was coding the cross_entropy examples to make sure the typing is correct. In the second example we need the softmax function in the link below. Looking at the code I see we have:
  def softmax[In <: DType, Out <: DType](input: Tensor[In], dim: Long)(
      dtype: Out = input.dtype
  ): Tensor[Out] =
    val nativeDType =
      if dtype == input.dtype then ScalarTypeOptional() else ScalarTypeOptional(dtype.toScalarType)
    Tensor(torchNative.softmax(input.native, dim, nativeDType))

This means that we have explicitly provide the last (usually empty) parameter so:
  val target1 = F.softmax( input=torch.randn(Seq(3, 5)), dim=1L)()

If we don't, we get the error:
[error] 358 |  val loss1 = F.crossEntropy(input1, target1)
[error]     |                                     ^^^^^^^
[error]     |Found:    (gpt.BiGram.target1 : torch.DType => torch.Tensor[torch.DType])
[error]     |Required: torch.Tensor[O]
[error]     |
[error]     |where:    O is a type variable with constraint <: torch.NumericRealNN

I have made that last parameter an implicit. I did the same for logSoftmax. If we do this, we avoid having to provide that last parameter. It seems that only the softmax call was used. Ran the test, had no problem. Ok, with this change or am I missing something?
The original Python example code uses a Tensor.softmax(dim=1) call. This method does not exist in storch. The Python documentation states that it is an "Alias for torch.nn.functional.softmax()." Should we add this? If so, do we add as a standard method or use use Scala 3 extension methods?
TIA

I have made that last parameter an implicit. I did the same for logSoftmax. If we do this, we avoid having to provide that last parameter. It seems that only the softmax call was used. Ran the test, had no problem. Ok, with this change or am I missing something?

That's fine but could you give the following variant a try? It's a solution we already use in other places and avoids both implicits and multiple parameter lists (at the expense of a slightly more verbose type signature).
import Derive.derive

// ...

  def softmax[In <: DType, Out <: FloatNN | Derive](
      input: Tensor[In],
      dim: Long,
      dtype: Out = derive
  ): Tensor[DTypeOrDeriveFromTensor[In, Out]] =
    val derivedDType = dtype match
      case _: Derive => input.dtype
      case d: DType  => d
    val nativeDType =
      if dtype == input.dtype then ScalarTypeOptional()
      else ScalarTypeOptional(derivedDType.toScalarType)
    Tensor(torchNative.softmax(input.native, dim, nativeDType))
}

The original Python example code uses a Tensor.softmax(dim=1) call. This method does not exist in storch. The Python documentation states that it is an "Alias for torch.nn.functional.softmax()." Should we add this? If so, do we add as a standard method or use use Scala 3 extension methods?

Yes, you can add it as a regular method in Tensor delegating to the implementation in nn.functional

That's fine but could you give the following variant a try? It's a solution we already use in other places and avoids both implicits and multiple parameter lists (at the expense of a slightly more verbose type signature).

Done (also for logSoftmax). Compiled and all tests pass.

Yes, you can add it as a regular method in Tensor delegating to the implementation in nn.functional

Done:
  def shape: Seq[Int] = size

  def softmax[Out <: FloatNN | Derive](
      dim: Long,
      dtype: Out = derive
  ): Tensor[DTypeOrDeriveFromTensor[D, Out]] = F.softmax(input = this, dim = dim, dtype = dtype)

  def square = Tensor(native.square())

While trying to replicate the Colaboratory notebook to check the code is working, I tried to do the following:
  // We want x[b,t] = mean_{i<=t} x[b,i]
  val xbow = torch.zeros(Seq(b0, t0, c0))
  for b <- 0 until b0
  do
    for t <- 0 until t0
    do
      val xprev = x(b,º`:`t+1) // (t,C)
      xbow(b,t) = torch.mean(xprev, 0)

The Tensorclass has no assignment operator. I also did not find a method for this in the JavaCPP code. How should one go about assigning a value?
TIA

The Tensorclass has no assignment operator. I also did not find a method for this in the JavaCPP code. How should one go about assigning a value?

The C++ API has a method for assigning values (with indices): See https://pytorch.org/cppdocs/notes/tensor_indexing.html#setter
It's just not that easy to find, because it's named index_put_. It's also mapped via JavaCPP, but was missing in Storch.
https://github.com/sbrunk/storch/pull/53 should add support for it. Could you give it a try?

Found some compiler weirdness with the changes above.These do not comlpile:
      xbow(Seq(b,t)) = torch.mean(input=xprev, dim=0)
      xbow(Seq(b,t)) = torch.mean(xprev, dim=0)

The error is:
method mean in trait ReductionOps: (input: torch.Tensor[?], dtype: torch.Float32): torch.Tensor[torch.Float32] does not have a parameter dim

and (for the last one):
Found:    (0 : Int)
Required: torch.Float32

But these do:
      xbow(b,t) += torch.mean(xprev, dim=0)
      val c = torch.mean(xprev, dim=0)
      xbow(Seq(b,t)) = c
      xbow(Seq(b,t)) = torch.mean(input=xprev, dim=0, true, float32)
      xbow(Seq(b,t)) = torch.mean(input=xprev, dim=0, true)

Maybe some tweaking of the 1st definition may get it working, but seems like a Scala issue.

It looks like the compiler gets confused by the overloaded variants of mean for whatever reason. I've seen this in other places with different generic overloads.
I realized that the default dim argument with an empty seq defaults to the behavior of the overloaded variants, making them redundant so I've removed them now in #53. Could you give it another try with the changes?

@sbrunk Changes work fine. Thanks.

I need the use of Dropout. In Python this seems to return a constructor of sorts (did not check), which can then be applied to a Tensor.
I see that we have a torch.nn.Dropout that is private to the torch package. So the more obvious solution of having a public Dropout class and its companion object will require changes. I have the following questions:

Is the suggested change above ok?
If so, can I go ahead and change this?
If not, what is the storch way?

I would like to use register_buffer. According to the Python API doc, we must pass in a name.
Looking at the org.bytedeco.pytorch.Module we have:
  public Tensor register_buffer(BytePointer name, Tensor tensor) { return asModule()._register_buffer(name, tensor); }
  private native @ByRef @Name("register_buffer") Tensor _register_buffer(@StdString BytePointer name, @ByVal Tensor tensor);
  public Tensor register_buffer(String name, Tensor tensor) { return asModule()._register_buffer(name, tensor); }
  private native @ByRef @Name("register_buffer") Tensor _register_buffer(@StdString String name, @ByVal Tensor tensor);

So in torch.nn.modules.Module something like this  should work:
  def registerB[D <: DType](n: String, t: Tensor[D]): Tensor[D] =
    nativeModule.register_buffer(n, t.native)
    t

However, as an example:
  def register[D <: DType](t: Tensor[D], requiresGrad: Boolean = true)(using
      name: sourcecode.Name
  ): Tensor[D] =
    nativeModule.register_parameter(name.value, t.native, requiresGrad)
    t

the name is implicitly defined. Is there any way I can keep the implicit but still allow manually setting that name?
On a related not, should these functions return a Tensor(t). We are assuming the same tensor is returned, but this is not guaranteed.

I need the use of Dropout. In Python this seems to return a constructor of sorts (did not check), which can then be applied to a Tensor.
I see that we have a torch.nn.Dropout that is private to the torch package. So the more obvious solution of having a public Dropout class and its companion object will require changes. I have the following questions:
1. Is the suggested change above ok?

2. If so, can I go ahead and change this?

3. If not, what is the `storch` way?

I think what you found is the Dropout trait in torch.nn.functional right? The trait is private because it's members are exposed through the package object, so you can call it like this:
torch.nn.functional.dropout(input=torch.rand(Seq(3,3)))
// res2: Tensor[Float32] = tensor dtype=float32, shape=[3, 3], device=CPU
// [[0,4759, 1,4497, 1,7002],
//  [1,2299, 0,0000, 1,1805],
//  [0,0000, 0,0000, 0,0000]]

It corresponds to torch.nn.functional.dropout in Python.
Seems like we're still missing the module variant of Dropout, which corresponds to the Python module you linked to. If you'd like to add that, that would be great! We should put it be under torch.nn.modules somewhere, like the other modules.

So in `torch.nn.modules.Module` something like this should work:

```scala
  def registerB[D <: DType](n: String, t: Tensor[D]): Tensor[D] =
    nativeModule.register_buffer(n, t.native)
    t

However, as an example:
  def register[D <: DType](t: Tensor[D], requiresGrad: Boolean = true)(using
      name: sourcecode.Name
  ): Tensor[D] =
    nativeModule.register_parameter(name.value, t.native, requiresGrad)
    t

the name is implicitly defined. Is there any way I can keep the implicit but still allow manually setting that name?

We could add an explicit optional name parameter, i.e. defaulting to an empty string, or using an Option. If the caller provides a real name, we take that, otherwise, we fall back to the implicit. Ah I see you've just done that below in the buffer impl :)

On a related not, shouldn't these functions return a Tensor(t). We are assuming the same tensor is returned, but this is not guaranteed.

You're right, it's better to use the tensor returned by the native register method.

EDIT 1: we also have the problem of duplicate overload methods due to the use of defaults. What is the way to solve this here? Can I change the names?

Yes please go ahead. Perhaps we can keep register for modules, because it is used quite often, but use registerParameter, registerBuffer for the others.

EDIT 2: In the meantime I will use:
  def buffer[D <: DType](t: Tensor[D], n: String="")(using
      name: sourcecode.Name
  ): Tensor[D] =
    val name_ = if n.trim().isEmpty() then name.value else n.trim()
    Tensor( nativeModule.register_buffer(n, t.native) )

👍

I am at a point right now were the model, with a single head, should be converging. Unfortunately, try as I might, I cannot get this to work. 8-( I have followed the video in an attempt to test it as I go and provide a document with this "journey".
In an effort to debug this, I have the BiGramX with the implementation and use of the model. During execution I observed that using the Shakespeare text, the loss practically did not change.
After some additional exploration, I changed the input code to a string containing only "a". This is a trivial case that should easily converge. Now when I execute this, the loss does not change at all. It seems like the loss is not propagated back. In fact the values of the logits does not change.
I am obviously doing something wrong. Can anyone take a look at the code. The BigramLanguageModel3 class and the training loop below it seems to be were I am messing up. Please note that I tried to keep this as close to the original Python as possible.
TIA.

While trying to implement and debug the multi-head attention mechanism, I have what seems to be unexpected behavior. For a model with the multi-head "only", the code:
    val nuParams = m.parameters.map(_.numel).sum
    println(s"${nuParams} parameters")

Reports:
Multi-head attention
4481 parameters

Now to this model I add the following layer:
    val ffwd = register( FeedFoward(nEmbed) )

where nEmbed = 32. If I count the number of parameters of this layer I get 1056 (nEmbed*nEmbed + nEmbed), which is correct. But the model still reports:
Multi-head attention + FFWD
4481 parameters

Shouldn't that be 4481 + 1056?
TIA

@hmf I have a hunch (not tested). Could you try to wrap your Sequential in your feed forward module inside a register as well like so:
https://github.com/sbrunk/storch/blob/5e1fdf2a7b2d985a58ee7a6f8405cd8d443426b4/examples/src/main/scala/gpt/BiGram.scala#L1316-L1326
- val net = nn.Sequential(
+ val net = register(nn.Sequential(

Right now it's registering the layers inside Sequential as submodules of net, but not net itself as a submodule of FeedForward. In Python this is done implicitly. Perhaps we need a macro at some point to achieve s.th. similar in Storch as well.

@sbrunk I have confirmed that I need to register the inner modules. As for the macro, maybe a single function that traverses the sub-modules and registers them would do. But we also para parameter and buffer registering, so that would also have to dealt with.
Thanks.

I would like to give an update on this endeavor. I have gone through most of the video and am now at the start of the "Block" implementation. I have tried to stick to the video so that I can compare my results. Unfortunately my results show much higher loss (single head and multi head of 3).
 Even though convergence is slow, the good news is that it seems to be stable. See below.
 In regards to the seed I have tried to do as is in the video, but this may not be correct. Also some of the constants may be off. For the final version I will try to replicate this code, so general performance should match.
As for the causes of differences in loss, when I tested the MNIST example in Linux, its behavior was not the same as in Mac. In fact the process did not converge. This was strange. @sbrunk changed the code altering the learning rate so that learning would converge in both OS.

@hmf I'll try to look into this to get a better understanding. Is there anything I need to consider if I want to run it? I.e. I've seen you are using mill, right?

@sbrunk Thanks.

Is there anything I need to consider if I want to run it?

Not really. Simple Scala object. Messy code though. Sorry about that.

I've seen you are using mill, right?

Correct, but it just calls the main. Execution is in the object initialization. Something to correct.
I was hoping to contribute the Mill script (another issue). It is just missing project publishing. When I get time I want to upgrade it to the latest Laika version to avoid the need to override the Helium templates (currently it overrides the header).

I have implemented a clean version of this Python code. It is here. I am able to get a validation error below 2.0 (even less) as shown in the videos. However with an increased number of iterations.
Unfortunately I am unable to use the exact same parameters due to memory issues. I am using a GPU with a whopping 24 Gigabytes. As soon as I start training, CUDA (nvidia-smi) shows over 18 Gigabytes being used. The (smaller) model (13_347_905 parameters) seems to use about 50 Mibs. The training loop uses Pointer.physicalBytes(), which reports a stable 2.2 MiBs after many iterations. So I am at a loss to know why so much memory is used.
I have looked for the APIs but cannot find the calls to get the CUDA memory stats.
Can anyone give me some pointer on how to check were the memory is used and diagnose this issue?
TIA

@hmf yeah right now, we don't really have a good way to do memory profiling. Need to look into that too. Perhaps we can use the JavaCPP Cuda bindings to get better GPU memory usage information.
One idea you could try for now is to run only parts of the model (i.e. just the attention layer etc.) inside a training loop (you can use just random inputs of the right size). That might help to isolate better what part consumes so much memory or where it leaks.

@sbrunk thanks for the suggestions. I have started using a kludge to try and get an idea where the memory is being allocated. What I do is set a Thread.sleep at certain parts of the code and use nvidia-smi to check the memory. Not practical, but it has helped me see that I need to add some additional PointerScopes. Still trying to figure out how the memory accumulates so much.

Perhaps we can use the JavaCPP Cuda bindings to get better GPU memory usage information.

I was hoping the PointerScope and Pointer would help. No luck, but still looking into it. Do you by any chance have any examples of explicitly de-allocating Tensors via these interfaces/classes?
What JavaCPP Cuda bindings are you referring to? I quick look at the API does not reveal too much. I also think that we would need to access this information in a device independent manner.  Maybe via Device?

@sbrunk thanks for the suggestions. I have started using a kludge to try and get an idea where the memory is being allocated. What I do is set a Thread.sleep at certain parts of the code and use nvidia-smi to check the memory. Not practical, but it has helped me see that I need to add some additional PointerScopes. Still trying to figure out how the memory accumulates so much.

Does it allocate too much inside a single iteration already or does it grow over multiple iterations during the training loop?

I was hoping the PointerScope and Pointer would help. No luck, but still looking into it. Do you by any chance have any examples of explicitly de-allocating Tensors via these interfaces/classes?

In Storch itself, I think we only have the image classifier example and https://github.com/sbrunk/storch/issues/5, but you seem to be already doing it this way.
The JavaCPP tests for deallocation and PointerScope might be helpful here. There are also a few issues/discussions in the javacpp/javacpp-presets repo like this: https://github.com/bytedeco/javacpp-presets/discussions/1160

What JavaCPP Cuda bindings are you referring to? I quick look at the API does not reveal too much. I also think that we would need to access this information in a device independent manner. Maybe via Device?

The Java bindings to the CUDA toolkit itself. But that's a long shot, I'm not sure if it provides something usable for us here.

EDIT: Device does not seem to be helpful. Need to look at torch.cuda.memory_stats for possible solution.

It looks like LibTorch provides something like this, see: https://discuss.pytorch.org/t/libtorch-equivalent-of-torch-cuda-memory-reserved/165995
But I haven't found anything related in the JavaCPP PyTorch bindings, so it might need to be mapped in the preset. Perhaps you could open an issue in https://github.com/bytedeco/javacpp-presets to ask about it.

The current V2  implementation  is using the same parameters but will not converge using the original learning rate. The only thing that is missing is the weight and bias initialization. The nn.Module does not seem to have an apply like PyTorch that does this.
So what is the best way forward here? Should we include such a method? What should we name it? Do simply iterate through all layers and apply a function like Python does?
Should I open a new issue to discuss this?
TIA

Good idea. A recursive apply like in Python should be quite useful. And yes, please create  a new issue for this.

Results of v2 on par with tutorial (loss below 2.0), but slower convergence. After a while it diverges.
Buch willift my freat I r'd did

LANDWAGLOUD INA:
Foolvest Hapry, shall he no cacinf it pake thou.
Are withe ret withsks shat usant:
Why somein so, helly how.
G Edwars haven, af of my unt: and,
To douer naty and babt! age she woun wilcy stith fabasten ther na s,
To beethe n as ave thigh bid sty are in in is
To nevirly thed, and our aft shal well, couser's be thringe.

ROMET:
For that ther sthing the meet is than him!
's.

HMOPSAMP'

@hmf amazing work!
I got hands on a larger GPU now and will start playing with it.

@dejvid The examples are in this fork. It is not completed yet - still needs weight initialization (see issue #61). I have had trouble finishing due to an update to the Pytorch. See issue #62

@sbrunk I have created a clean branch with the changes that implement the example. This holds changes for #51 and #61 (apologies for the comment error in the commit).
Some notes in the implementation:

It does not have the performance of the original Python code. I have worked on this for a long time but cannot figure out what is wrong. Needs reviewing.
The use of weight initialization make performance significantly worse. I have left it in. Maybe it should be removed  or corrected (might not be the intended/best initialization).

I now have another issue. After updating to your latest changes (2.1.1), GPU dos not work on my side. This is also true for the original LeNet example (nvidia-smi shows no process using GPU). Note that because of this, current code breaks, but it is not usable without a GPU.
Could check this and give me feedback?
TIA

Thanks @hmf for pushing this forward. Could you create a PR from your branch. That should make it easier to do reviewing.
I'll try to figure out the new GPU issue.

@sbrunk The code assumes a GPU is available. The printMemoryInfo will fail otherwise. Remove it if you need to test with CPU only.
The class is gpt.V2.

I now have another issue. After updating to your latest changes (2.1.1), GPU dos not work on my side. This is also true for the original LeNet example (nvidia-smi shows no process using GPU). Note that because of this, current code breaks, but it is not usable without a GPU.

I can't reproduce the GPU issue at the moment, but I could only try on an RTX 4090 so far, which is ADA architecture, while 3090 is Ampere.
It did work with 2.1.0 for you right?
Could you give at a try with the latest update to PyTorch 2.1.2 by bumping the PyTorch patch version in build.sbt?
- val pytorchVersion = "2.1.1"
+ val pytorchVersion = "2.1.2"

@sbrunk
GPU working with 2.1.2
Note that with:
set ThisBuild / enableGPU := true

In sbt I now get:
[error] stack trace is suppressed; run last core / update for the full output
[error] (core / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch/2.1.1-1.5.10-SNAPSHOT/pytorch-2.1.1-1.5.10-20231204.171720-12-linux-x86_64-gpu.jar: not found: https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch/2.1.1-1.5.10-SNAPSHOT/pytorch-2.1.1-1.5.10-20231204.171720-12-linux-x86_64-gpu.jar

Bur no issues with 2.1.2.
Thanks.

@hmf I'm trying to reproduce your results but it's diverging much faster in my case. I get down to a train loss of 2.3 but then it starts to go up again. At some point, the losses even go NaN.
[info] step 8000: train loss 2.3154821, val loss 2.3707209, mem 942,0 MiB @ 00 00:08:13.783, mean 00 00:00:00.061
...
[info] step 40500: train loss NaN, val loss NaN, mem 944,2 MiB @ 00 00:41:43.711, mean 00 00:00:00.061

Here's how I ran it: Took the branch, disabled weight init, ran with a learning rate of 1e-4.
// modules.foreach(init_weights)
// ...

train(model, 1e-4, 67000)

Any idea what could be different?

@sbrunk  I am trying to rerun the test to confirm all is Ok. Unfortunately I have had to recreate the dev container. I have also merged your latest changes from main. I am now running with initialization off.
As soon as I get some results I will report back.

Any idea what could be different?

At this time, it only occurs to me that the OS libraries (including NVIDIA's stuff) my be different. I am assuming your are using Linux Ubuntu. Below is a list of the setup.
storch_os.txt
We could also try setting up a fixed random number seed for replication.

@sbrunk I have rerun with Pytorch 2.1.2. and get:
[info] step 41500: train loss 1.6008276, val loss 1.7987113, @ 00 01:25:21.304, mean 00 00:00:00.123

Here is the full output:
tmp.txt
I have added:
 torch.manualSeed(1337)

and am running this again. We can then test and compare with this seed.

I can't reproduce the GPU issue at the moment, but I could only try on an RTX 4090 so far, which is ADA architecture, while 3090 is Ampere.

I have noticed that your compute time per iteration is about half of mine (0.123 vs 0.061). Nice 8-)

@hmf I did a run on d8d75b7b773972191dd0a48e374f9531bd2407d2 and it's much closer to your result than before:
train-gpt.v2.txt.
There are still numeric differences but I guess that could be due to slightly different hardware/driver.

@sbrunk I am somewhat skeptical of the results.  I noticed that the first loss is indeed the same value (baring precision errors). For that reason I would expect the next values to be the same (the data is the same and should be loaded in the same order). This also does not bode well for unit tests - something that can be added that can be used to confirm your hypothesis.
Having said this, I am not satisfied with the results. I am considering a ViT implementation that has SOTA performance that can be compared. This case is easier to test than NLP because pre processing should be  simpler. Just a thought.

@sbrunk I am somewhat skeptical of the results. I noticed that the first loss is indeed the same value (baring precision errors). For that reason I would expect the next values to be the same (the data is the same and should be loaded in the same order). This also does not bode well for unit tests - something that can be added that can be used to confirm your hypothesis.

That's true. It seems like I get reproducible results on the same hardware/setup though when I run it twice. I'll have access to yet another GPU type next week and I'll try it there too for comparison.

Having said this, I am not satisfied with the results. I am considering a ViT implementation that has SOTA performance that can be compared. This case is easier to test than NLP because pre processing should be simpler. Just a thought.

That's a great idea. If you'll give it a try, let me know if I can help you in any way.


umail

umail.txt

@Jaloljava lib/domains/stoplist.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@Jaloljava lib/domains/stoplist.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@Jaloljava lib/domains/stoplist.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@Jaloljava lib/domains/stoplist.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@Jaloljava lib/domains/stoplist.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@Jaloljava I have to close the request without merging, because you made a mistake in a request and requested changes in too many files. Most of them do not relate to the organization which you would like to add to this repository. Please ensure that you are familiar with this repository rules located at the bottom of this page and make a new correct request. Thank you for the understanding!


Relative URLs act like absolute URLs

I'm utilizing relative URLs in my app to make my life easier with deeply nested URLs. Unfortunately bb.Intercept seems to be mapping my URLs absolutely.

Bar

Will-fix. Thanks for the report!

The issue is that I'm looking at the anchor tag's href attribute, and not a.pathname. The former returns the actual value of the href attr (unsurprisingly), whereas the latter is the calculated pathname.
In addition to this, I will need to support a root option to put this library in line w. Backbone history's options.

Released as v0.3.2


WIP: Restore for device_tracker

Description:
Add restore (#4614) to device_tracker.
Checklist:
If the code does not interact with devices:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[ ] Tests have been added to verify that the new code works.

Well, I was just about to start looking at this. Glad I happened to look at PRs first!
It looks reasonable to me. I think it goes without saying it would be nice if we could use EntityComponent within device tracker to eliminate some of the duplication, but I'm guessing that wasn't done because the differences are too great.

Migrating this to EntityComponent could open up a whole new can of worms, so lets rather keep for another PR: One of the unnecessary things would be scan_interval that I could think of. Here we also use "see" and only one class of device.
For now the async_added_to_hass is the only duplication introduced here, so should be ok

Another problem with the EntityComponent is that DeviceTracker supports two types of platforms. The router ones that just provide a "get_devices" method and the ones that just call "see"


Linting for ES6 + JSCS rules

Linting support for ES6 and rule set up for Javascript Code Style.

ESLint offers linting for ES5 - ES7 and code style enforcement.  This may be the better option over jshint and jscs.

I need this rule to be added in so that I can define styles below my React class definition.
"no-use-before-define"          : 0

A few suggestions:

no-console & no-alert should be disabled. Some designs call for alert and confirm boxes, and console.logs are necessary for errors.

prefer-template "Unexpected string concatenation" should be disabled. This throws an error when string concatenation happens in an object value, which is sometimes called for and would be superfluous to concat in a variable before hand.
eg:

      width : 118 + 'px',
      height : 118 + 'px',
      lineHeight : 120 + 'px',
    };```

3. `no-underscore-dangle` should be removed. Throws an error when identifiers begin with underscores. I do not name variables or functions myself beginning with an underscore, but it is popular to do so in a lot of npm packages.
eg:
`this.props._onClick()`

4. `vars-on-top` should be removed. Throws an error when vars are not defined at the top of a function. Sometimes the code is easier to read when vars are declared directly above where you need them. For example, if you have a long function with an if statement and a for loop, it would be nice to define a variable used only in the for loop right before it rather than declaring it way out of sight.

The idea behind no-console and no-alert is that you don't use these outside of development, or rather, prevent them from being committed so deployments to production environments aren't littered with these methods.  I agree that no-alert should be disabled, but I think no-console could be good for habit building as console logs in production is a pet-peeve of mine.
I agree with prefer-template and no-underscore-dangle as well.  I use preceding underscores frequently to denote "private" or "hidden" variables.  However, vars-on-top is often considered a good practice and I would like to see that utilized.

I would like to also add this rule because there are many instances while using a switch statement that do not need a default. "default-case"                  : 0


Return type is being incorrectly inferred with finalizeDistillation<T, _distill<T, "out", "base">>

Report a bug
🔎 Search Terms

finalizeDistillation

🧩 Context

ArkType version: ^2.0.0-beta.2
TypeScript version (5.1+): 5.5.4
Other context you think may be relevant (JS flavor, OS, etc.):

🧑💻 Repro
import { type, type Type } from "arktype";
import { ok, err, Result } from "neverthrow";

function validate<T>(
  data: unknown,
  schema: Type<T>,
): Result<Type<T>["infer"], Error> {
  const result = ok({});

  return result.andThen(() => {
    let validated = schema(data);

    if (validated instanceof type.errors) {
      return err(new Error());
    }

    return ok(validated);
  });
}

Diagnostics:
1. Type 'Result<unknown, unknown>' is not assignable to type 'Result<finalizeDistillation<T, _distill<T, "out", "base">>, Error>'.
     Type 'Ok<unknown, unknown>' is not assignable to type 'Result<finalizeDistillation<T, _distill<T, "out", "base">>, Error>'.
       Type 'Ok<unknown, unknown>' is not assignable to type 'Ok<finalizeDistillation<T, _distill<T, "out", "base">>, Error>'.
         Type 'unknown' is not assignable to type 'finalizeDistillation<T, _distill<T, "out", "base">>'. [2322]

repro repo: https://github.com/dmmulroy/arktype-bug

This looks like a TypeScript bug:
import { type, type Type } from "arktype"
import { err, ok, Result } from "neverthrow"
import type { _distill, finalizeDistillation } from "../type/ast.js"

// When unannotated, return type inferred as:
// Result<finalizeDistillation<T, _distill<T, "out", "base">>, Error>
function validateInferred<T>(data: unknown, schema: Type<T>) {
	const result = ok({})

	return result.andThen(() => {
		let validated = schema(data)

		if (validated instanceof type.errors) {
			return err(new Error())
		}

		return ok(validated)
	})
}

// inferred correctly
const out = validateInferred({}, type("string"))
//    ^? Result<string, Error>

// When that exact return type is added, you get an error
function validateAnnotated<T>(
	data: unknown,
	schema: Type<T>
): Result<finalizeDistillation<T, _distill<T, "out", "base">>, Error> {
	const result = ok({})

	return result.andThen(() => {
		let validated = schema(data)

		if (validated instanceof type.errors) {
			return err(new Error())
		}

		return ok(validated)
	})
}

// note this still works externally but would require a cast internally
const outAnnotated = validateAnnotated({}, type("string"))
//    ^? Result<string, Error>

My general experience is that if I'm writing any kind of non-trivial generic in TS, I don't worry too much about casting.
However, this still feels like a bug- I don't think I've ever seen it reject the exact return type it happily infers.
If you can get away with not explicitly annotating the return, that's your best option.
If you need the explicit annotation, you'll likely have to cast internally. Maybe worth opening an issue in the TS repo though.

@dmmulroy Closing, but let me know if you end up testing anything else out and have any issues!


Correction for raw affiliation EDF Hydro, DTG, 38950 Saint-Martin-le-Vinoux, France

Correction needed for raw affiliation EDF Hydro, DTG, 38950 Saint-Martin-le-Vinoux, France
raw_affiliation_name: EDF Hydro, DTG, 38950 Saint-Martin-le-Vinoux, France
new_rors: 03wb8xz10
previous_rors:
works_examples: W4318616179;W4318616177
contact: 41bc3e9234430e4857e29e36ed239b1a:7a893f35af6d30b61106debbefc409 @ recherche.gouv.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-09. The new affiliations should be visible within the next 7 days.


Problem with encoder

I tried to train the model (sketch_rnn_train.py) with pig.npz and to test it I used (https://github.com/tensorflow/magenta-demos/blob/master/jupyter-notebooks/Sketch_RNN.ipynb) when I get the next problem:

Hey @Pauladds Did you manage to fix this? I am still getting the same error only now it's -
ValueError: Cannot feed value of shape (1, 251, 5) for Tensor u'vector_rnn_1/Placeholder_1:0', which has shape '(1, 76, 5)'

Is the input shape hardcoded somewhere? Can't seem to find it.
Thanks,
Niraj

Hello!
Yes, I solve it!
In https://github.com/tensorflow/magenta-demos/blob/master/jupyter-notebooks/Sketch_RNN.ipynb
in "encode" function you have :

So, inside of that, you have: "to_big_strokes"  and one of the parameters of that function is max_len that by default is 251, and you need to change to the number that you need, in your case 76.
For example, you can write:

I hope you understand it! :)
You´re welcome
Paula

That actually worked! Thanks a lot @Pauladds
Had to put in 75 to make it work. Will investigate further as to why this happened. Thanks again!

I know why it is!
It is because the parameter is prefixed to have the longest input length that you enter in the training dataset (In the case of the example it is 250).


Fix errors of bitcoin_fr_FR.ts file for 1.14.7 dev

You have few errors which were corrected. Now, the file is available at:
bitcoin_fr_FR.txt

Hi! master is frozen per https://github.com/dogecoin/dogecoin#branches - it gets updated on release. I am closing this PR. If you want to build ahead of the release, please build with 1.14.7-dev as a source.

Ok I'm going to do this later. Thanks for the precision.

@Loopite We can do one of 2 things, let me know what you prefer:

I can look into your file and see if I can create a pull request in your name, or
I can work on a good guide (pref with some others) re: how to do a pull request

I can work on a good guide (pref with some others) re: how to do a pull request

I think it would be better. Then we'll be able to know what kind of branches are available as source. Meanwhile, I'll work this morning on the fr_FR translations.


Mention about Services

I've recently started learning web-dev using ASP.Net Core and what i understand is that in ASP.Net Core MVC, the model is just an entity object that is mapped to the database. Due to this the Business logic get's concentrated into the Controllers and Code Sharing is hindered. To avoid this ASP.Net Core supports Services, but the documentation doesn't mention much in this context about the services. So, getting a mention of the Services at this page will give the reader introduction about using services.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f9b26c2a-d481-043c-e37b-cbe3ad427d41
Version Independent ID: d7d85488-c055-7aea-0664-2e97bc937a04
Content: Overview of ASP.NET Core MVC
Content Source: aspnetcore/mvc/overview.md
Product: aspnet-core
GitHub Login: @ardalis
Microsoft Alias: riande

Hello @Praveen-Rai ... The topic that pertains to services (dependency injection) at https://docs.microsoft.com/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-2.1 appears early in the TOC. Generally, we hope devs will read the major topics in the Fundamentals node before they look at the MVC node.
Also, the topic does have what you're asking for ... https://docs.microsoft.com/aspnet/core/mvc/overview#dependency-injection ... in the Features area of the topic. It links over to the services (DI) topic.

@guardrex It's fair to assume that the developer reads the documentation sequentially, but on the other hand elsewhere as well the MVC pattern does not mention anything about services. May be other frameworks might support writing business-logic in the Models, i have absolutely no idea on this as i am just walked first few steps towards web-development and MVC. The point i want to make here is that when a person like me starts with ASP.Net Core MVC, the page under discussion would be the first introduction about MVC from aspnet team. So, it would be better if we talk about MVC in context of ASP.Net Core (where services are the business logic containers) and hence we should at least mention Services and their importance.
Secondly, the links that you pointed talks more about Dependency injection and does not explicitly say that one should create services to contain and share the Business Logic. Again, for an expert DI might directly imply Services, but to me DI appears to be an implementation method for Services rather the concept of Service itself ( please correct me if i am mistaken ).
Hope i make sense to you :-)

And how about if we can move this page up in the sequence ?

Thanks for your feedback. I defer to management. @Rick-Anderson @scottaddie

@Rick-Anderson @scottaddie Any comments / updates ??

Details like this wouldn't go in the overview document.  I think it's covered adequately in several places like
https://docs.microsoft.com/en-us/aspnet/core/data/ef-rp/intro?view=aspnetcore-2.1&tabs=visual-studio
and https://docs.microsoft.com/en-us/aspnet/core/tutorials/razor-pages/model?view=aspnetcore-2.1

Let me chime in here as author of some of the content you're referencing.
The Model in MVC refers to not just a given ViewModel or Entity, but to the domain model or application model, which may include services. You won't find any of the typical UI design patterns including "services" in their names (MVC, MVP, MVVM, etc) because services are a part of the Model in these patterns. Services is also an ambiguous term, since there can be services at any/all levels of abstraction. You can have a "service" that exists only in the UI layer for performing some UI work. You can have a "service" that exists only in an infrastructure library for performing some low level detailed work. And as I mentioned you can have "services" in your domain model providing business logic (in addition to logic that may be on entities and value objects, etc. in the domain model).
This link provides some more info: https://deviq.com/kinds-of-models/ Note when referring to the domain model:

The domain model will often include abstractions and services that allow the Controller to operate at a higher level of abstraction, and keep low-level plumbing code from cluttering the Controller and making it harder to test.

I agree with @Praveen-Rai. The role of the services are not emphazised enough. I (as a newbie asp.net core MVC developer) started adding all the business logic into the controller. In my opinion this page should guide the readers when they are trying to learn the recommended or best practices to implement the MVC pattern using this framework, and the services are an integral part of that.


"Given URL is not allowed by the application configuration" when running on Android Emulator

🐛 Bug Report
When I run Facebook login process on the Android Emulator, I am presented with the error "Given URL is not allowed by the application configuration"

Everything is working fine on iOS.
I suspect this is something wrong with the oAuth redirect. I have localhost setup in the Authorised Domains in the firebase console.
To Reproduce
Open up Android emulator and run the app, call await LoginManager.logInWithPermissions, enter test username and password
Expected Behavior
For the login process to complete and the popup to redirect back to the app.
Code Example
I am calling the Facebook login process via this code as per documentation (note the same code is working fine on iOS):
await LoginManager.logInWithPermissions(['email', 'public_profile', 'publish_video'])
I have all the permissions authorised and approved on the Facebook app (and working in another app)
The following lines have been added to strings.xml as per documentation:
<string name="facebook_app_id">**********</string>
<string name="facebook_client_token">********************</string>

The following lines have been added to AndroidManifest.xml as per documentation:
<uses-permission android:name="android.permission.INTERNET" />
<meta-data android:name="com.facebook.sdk.ApplicationId" android:value="@string/facebook_app_id"/>
<meta-data android:name="com.facebook.sdk.ClientToken" android:value="@string/facebook_client_token"/>

      <provider
        android:authorities="com.facebook.app.FacebookContentProvider**********"
        android:name="com.facebook.FacebookContentProvider"
        android:exported="true" />

Environment
System:
OS: macOS 11.6
CPU: (8) arm64 Apple M1
Memory: 142.06 MB / 8.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 16.11.1 - /usr/local/bin/node
Yarn: Not Found
npm: 8.0.0 - /usr/local/bin/npm
Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
Managers:
CocoaPods: 1.11.2 - /opt/homebrew/bin/pod
SDKs:
iOS SDK:
Platforms: iOS 14.5, DriverKit 20.4, macOS 11.3, tvOS 14.5, watchOS 7.4
Android SDK:
API Levels: 29, 30, 31
Build Tools: 29.0.2, 30.0.2
System Images: android-29 | Intel x86 Atom_64, android-29 | Google APIs ARM 64 v8a, android-29 | Google APIs Intel x86 Atom_64, android-31 | Google APIs ARM 64 v8a
Android NDK: Not Found
IDEs:
Android Studio: 2020.3 AI-203.7717.56.2031.7784292
Xcode: 12.5/12E262 - /usr/bin/xcodebuild
Languages:
Java: 1.8.0_292 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.66.1 => 0.66.1
react-native-macos: Not Found
npmGlobalPackages:
react-native: Not Found

This won't be the most helpful comment but it should help you focus your effort where they will count - and it's not with a bug here, this definitely works for me in my app on the emulator, so you have a configuration issue. Since no resolution here will result in a change in the module I'm going to close this, and suggest triple-checking your project config vs the documentation and/or checking/posting stackoverflow for general facebook config help

Any updates @jwainwright87 ? Having same issue in 2023


docker-compose DefaultDataCenter IP adress issue

Hi, i'm trying to use docker-compose to run seaweedfs containers, but I can't access the volume default URL
I used the sample docker-compose.yml
version: '2'

services:
  master:
    image: chrislusf/seaweedfs
    ports:
      - 9333:9333
    command: "master"
    networks:
      default:
        aliases:
          - seaweed_master
  volume:
    image: chrislusf/seaweedfs
    ports:
      - 8080:8080
    command: 'volume -max=5 -mserver="master:9333" -port=8080'
    depends_on:
     - master
    networks:
      default:
        aliases:
          - seaweed_volume

Then I have run the command docker-compose  -f docker/docker-compose.yml up
Got 2 containers running:
dc21c9c68658        chrislusf/seaweedfs   "/entrypoint.sh volum"   3 minutes ago       Up 3 minutes        <IP_ADDRESS>:8080->8080/tcp, 9333/tcp   docker_volume_1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
033d2e1cca42        chrislusf/seaweedfs   "/entrypoint.sh maste"   3 minutes ago       Up 6 seconds        8080/tcp, <IP_ADDRESS>:9333->9333/tcp   docker_master_1

I can use POST request on the PORT 9333, but the URL provided to send the file is <IP_ADDRESS> and it looks like I can't access this IP.
Is there a way to specify a IP for this volume.
I tried this (in docker-compose.yml)
command: 'volume -dir="/tmp/bmpr" -max=5 -mserver="localhost:9333" -port=8080'

Note: I'm using windows 10, and the docker default IP adress is <IP_ADDRESS> (not localhost)

I am not familiar with docker. For the volume server to specify the IP, "weed volume -ip=xxx.xxx.xxx.xxx".

In the docker mode, you would need to access the volume server via localhost:8080.


Make spark3.1 the default profile

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot run azure

@hudi-bot run azure

@hudi-bot run azure

Refer to this green azure ci run: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=10078&view=logs&j=47cf0f2a-901e-5ca1-f652-e53b6abbf660&t=35a68570-76b2-5f68-d601-1bf50f7fbd97
all sections passed

in this pr had to disable some tests that had issues. Now adding back certain tests that were disabled in https://github.com/apache/hudi/pull/6151/files. (if pr 6151 is green then will opt for that as the main one).


add Windows statics

On linux works flawless, I added Windows solution, but on my machine it shows username@hostname correctly, however colors doesn't work anymore. I cross-compiled it so maybe that's a reason. Have to be revieved and  eventually corrected

I did run the code in Powershell and wsl. It works perfectly, as well as the color codes

Also, i have made some changes in the main branch, pls do fork the new code.
And also, there is a variable for your <ROOT_PATH>, its also static

rn, i m having merge conflicts with you code, so for now, I'll be rejecting this pr.

sorry I commited before pull, I'm closing it and send you correct one soon if you like :)

sorry I commited before pull, I'm closing it and send you correct one soon if you like :)
Appretiated :)

and no need for sorrys n stuff ✌


NIOPerformanceTester: more precision and resiliency

Motivation:
Previously, NIOPerformanceTester would use Foundation.Date twice to
calculate the time spend in a given function. This is less precise as it
can be but more importantly, it's not resilient against the user
changing the clock or the computer going to sleep.
Modifications:
Use DispatchTime.now().uptimeNanoseconds
Result:
More precise and resilient benchmark measurements.

@swift-nio-bot test perf please


React: Masthead - On tab, user clicks on profile and then on search icon, the focus remains on profile icon.

Detailed description
On tab - iOS 12, 13, and 14
Issue: User clicks on profile and then on search icon, the focus remains on profile icon.
User has to click on the search field again to enter keyword.
Additional information

@RobertaJHahn @Praveen-sr Just tested this on an iPad and cannot reproduce on any of our environments. It might have been inadvertently fixed since this issue was opened. Suggest closing.

Closing based on Kenny's comments above.


Expose #client via _client() on HTTPRequest

What kind of change does this PR introduce?
A small change that will allow the puppeteer-page-proxy project a path to working again with the latest versions of puppeteer
Did you add tests for your changes?
_client() is not documented on Page yet still exists so I guess it's not required.
Summary

Completely eliminating access to the HTTPRequest._client property will break puppeteer-page-proxy and in turn negatively affect my users. Forcing my project to use older puppeteer version will also negatively affect my users. I am making this PR so I can eventually make a PR/fork on puppeteer-page-proxy
https://github.com/open-wa/wa-automate-nodejs/issues/2755
https://github.com/Cuadrix/puppeteer-page-proxy/issues/78

Does this PR introduce a breaking change?
No

Other information

Fixed in #8556


Sorting enums by their values

Product
Hot Chocolate
Is your feature request related to a problem?
Sorting enums based on their values appears to be unattainable at present. Instead, they are sorted by their names, which may suit certain scenarios but can be cumbersome for others.
The solution you'd like
Given the example below, I aim to have the statuses sorted based on their values. Hence, when queried, I anticipate seeing entries in the order of Open, Pending, and Closed.
public enum Status
{
    Open = 0,
    Pending = 1,
    Closed = 2
}

Currently, enums are sorted alphabetically by their names. Consequently, the resulting sort order for entries would be Closed, Open, Pending, which is not preferred in this context.

@PHILLIPS71
Hot Chocolate introspection returns the values in declaration order, and BCP uses ordering similar to VS Code (by proximity, etc.):

Could you please clarify what you meant by "when queried"?

@glen-84 thanks for looking into this!
I'm referring to if you were to apply a sort order to an enum field it would be nice to have it sort via the value of the enum, instead of the name.

Oh I see.
Could you please provide an example, or at least more details? Is this in-memory, EF Core (which provider), etc.?

Which provider are you using?
Is the enum mapped to a string type or an enum type in the database?

Which provider are you using?
Is the enum mapped to a string type or an enum type in the database?

postgres
a string usually, though there are cases an int type is used

Unless I'm missing something, the ordering is simply going to follow the data type of the column – if it's a string, then it'll sort by name, and if it's an integer, then it'll sort by value.
In other words, if you want status to be sorted by value, then consider storing it as an integer.
The only other options that I can think of:

Custom SQL that does something like:

ORDER BY CASE status WHEN "Open" THEN 0 WHEN "Pending" THEN 1 WHEN "Closed" THEN 2 END

(possibly not great for performance)

Mapping to a PostgreSQL enum, in which case the order is based on the order in which the values are listed when the type is created.

I don't think there's much that we can do here on the Hot Chocolate side.

I realize now that I've misunderstood the sorting process. I initially thought it would sort based on the enum value after conversion. However, since the conversion occurs only after the result set is returned, the sorting is already completed by that point. I'll close this issue since it was a misunderstanding on my part and use one of the options you've provided. Thanks for clearing this up, I appreciate it!


Facebook Token Error

Getting this error when logging into soundbounce using my Facebook. I'm inexperienced with node and was unsure of where the problem lay, so hopefully the attached error logs will be of help.
FacebookTokenError: This IP can't make requests for that application.
   at Strategy.parseErrorResponse (/home/ubuntu/soundbounce/server/node_modules/passport-facebook/lib/strategy.js:198:12)
   at Strategy.OAuth2Strategy._createOAuthError (/home/ubuntu/soundbounce/server/node_modules/passport-facebook/node_modules/passport-oauth2/lib/strategy.js:345:16)
   at /home/ubuntu/soundbounce/server/node_modules/passport-facebook/node_modules/passport-oauth2/lib/strategy.js:171:43
   at /home/ubuntu/soundbounce/server/node_modules/passport-facebook/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:177:18
   at passBackControl (/home/ubuntu/soundbounce/server/node_modules/passport-facebook/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:124:9)
   at IncomingMessage.<anonymous> (/home/ubuntu/soundbounce/server/node_modules/passport-facebook/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:143:7)
   at IncomingMessage.EventEmitter.emit (events.js:117:20)
   at _stream_readable.js:920:16
   at process._tickCallback (node.js:415:13)

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Thanks for this, our server changed IP I'm updating it now.  FYI we're working on v2 over here: https://github.com/pdaddyo/soundbounce-v2/


New Post add-in not working (misses _config.yml?)

Hi Yihui,
I'm having trouble creating new posts with this template using the "New Post" RStudio add-in. Here's the error message I get if I try to execute the add-in on a clone of your repo:
> blogdown:::new_post_addin()
Error in blogdown:::site_root() :
  Could not find config.toml / config.yaml under
   /home/grant/blogdown-jekyll
   /home/grant
   /home
   /

So, the problem appears to be that the add-in triggers blogdown:::new_post_addin(), which in turn searches for one of "config.toml" or "config.yaml". In other words, it's not searching for the actual configuration file — "_config.yml" — attached to the site.
Aside: Should "_config.yml" be added to the list of configuration files that blogdown searches for / recognises? That's the standard naming convention for Jekyll sites at least. I'm happy to raise an issue on the main blogdown repo if you agree.
If I manually copy the "_config.yml" configuration file to "config.yaml", then executing the add-in triggers a new error:
> blogdown:::new_post_addin()
Warning: You are recommended to ignore certain files in config.yaml: set the option ignoreFiles: ["\\.Rmd$", "\\.Rmarkdown$", "_files$", "_cache$"]
Error in setwd(content_file()) : cannot change working directory

I should say that all of the other blogdown integrations with this template appear to work (serve_site(), etc.) I'm also somewhat surprised that no-one else has run into this problem. I don't think there's anything unusual with my system, but I provide my session info below. FWIW, I'm running the latest preview release of RStudio (Version 1.4.869).
Thanks in advance!
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Arch Linux

Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblas_haswellp-r0.3.10.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5          bookdown_0.20       digest_0.6.25       later_<IP_ADDRESS>
 [5] mime_0.9            R6_2.4.1            jsonlite_1.7.1      xtable_1.8-4
 [9] magrittr_<IP_ADDRESS>00 evaluate_0.14       blogdown_0.20.2     rlang_0.4.8
[13] miniUI_<IP_ADDRESS>      promises_1.1.1      rmarkdown_2.4.1     tools_4.0.2
[17] shiny_1.5.0         fastmap_1.0.1       httpuv_1.5.4        xfun_0.18
[21] yaml_2.2.1          rsconnect_0.8.16    compiler_4.0.2      htmltools_0.5.0
[25] knitr_1.30

If you use Jekyll, you need to inform blogdown via a global option blogdown.generator: https://bookdown.org/yihui/blogdown/jekyll.html The it'll look for _config.yml.

That said, I guess I should work harder on this front to just let blogdown automatically recognize Jekyll sites, instead of having users configure this option, but that might a little tricky since hexo also uses _config.yml. I'll see if the automatic detection could be robust enough.

Thanks Yihui.
Just to clarify: I'm using your repo as-is, which bundles the local .Rprofile file containing the blogdown.generator Jekyll spec. Do I need to set this in my global ~/.Rprofile file too?

No. That sounds like a bug. I'll look into it. Thanks for the report!

Should be fixed in the development version of blogdown:
remotes::install_github('rstudio/blogdown')

Note that you should open the RStudio project blogdown-jekyll.Rproj, so that .Rprofile can be loaded correctly (in your original report, I feel .Rprofile was not loaded). Thanks!

Hmmm, I always click on the Rproj file and even checked options('blogdown.generator') to confirm that it was pulling in the local .Rprofile. Regardless, that's something of a moot point since the new dev version appears to have fixed it.
Thanks again, Yihui!


Forward DICOM args in InferencePipeline

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.84%. Comparing base (6e1ffff) to head (e1190bd).

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
+ Coverage   93.83%   93.84%   +0.01%
==========================================
  Files          22       22
  Lines        1783     1787       +4
==========================================
+ Hits         1673     1677       +4
  Misses        110      110

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


In version 4.0.1, hintmanager.setdatabaseshardingvalue cannot be used. When will this problem be fixed

In version 4.0.1, hintmanager.setdatabaseshardingvalue cannot be used. When will this problem be fixed

Can you explain the details?

In version 4.0.1, you should use  hintmanager.addDatabaseShardingValue()  or use spring.shardingsphere.sharding.tables.tablename.table-strategy.hint.algorithm-class-name= configuration.
The current version uses Hint, and you need to configure separate tables.

@joinhappy81 please provide more information
like configuration, SQLs, table structure.

I think the problem is duplicated with #3964 .


Test against php 8.3

Thanks, though this change required a little more work, and some upstream changes. This is now complete, in master, and v5.6.0 will be released shortly.


Proposal types Alliance, Campaign, Hypha

3 Types of Proposals

Campaigns - Seeds for campaigns that will get their own website where people can request rewards for submitting contributions, and verifiers verify that tasks have been done.
Alliances - Seeds that are passed go to their escrow account
Hypha - reciepient can only by Seeds.hypha

Campaigns

Liquid Seeds from gift.seeds
Campaigns are managed by Hypha initially
Campaign funds go into campaign accounts which will have multi sig for our verifiers (1/3) so everyone can verify tasks have been done and release funds
We can create a campaign fund account when the campaign has passed - it could be managed by a special seeds account before handing over control to ambassadors?
Not sure this can be automated. The stewards might have to do this.

Alliances

Escrow Seeds from allies.seeds
we send the Seeds to the escrow account with the org as the beneficiary.
org needs to be the proposer

Hypha

Liquid Seeds from milest.seeds
When we request from milest.seeds account, it can only go to seeds.hypha
I think we can soft enforce this on the app level though?
If random people request from this account... we vote no?!

Tasks

[ ] Enforce campaign type by funding wallet
[ ] Add permissions for all funding wallets
[ ] Put allies funds into escrow rather than directly transferring
[ ] Add permission for vesting contract access so we can escrow funds
[ ] Add unit test for escrow funds
[ ] Add unit test for milest.seeds
[ ] Add unit test for funding with allies.seeds

actcam.seeds (can we create this account as a catchall for campaigns?) It's going to be the standard account that if they forget to put in a campaign receiving address it goes here.

Alliance  change:

deploy accounts, proposals, escrow
updatePermissions


[Suggestion] Configuration File

A configuration file is needed for this mod:

Enable/disable blocks/features
Adjust power usage of entire mod(multiplier) and/or individual blocks

I will never add a way in the config to disable certain blocks.  The whole mod is meant to be played with all the blocks. It doesn't make sense to disable say the importer. Of course, pack makers are free to remove recipes to make the item / blocks unobtainable.
Energy cost config is doable, yes.

I actually only care about the energy costs but in the spirit of completeness I thought I would add what most people end up asking for anyways :)

    public int cableRfUsage;
    public int constructorRfUsage;
    public int crafterRfUsage;
    public int craftingMonitorRfUsage;
    public int destructorRfUsage;
    public int detectorRfUsage;
    public int diskDrivePerDiskRfUsage;
    public int externalStoragePerStorageRfUsage;
    public int exporterRfUsage;
    public int importerRfUsage;
    public int interfaceRfUsage;
    public int processingPatternEncoderRfUsage;
    public int relayRfUsage;
    public int soldererRfUsage;
    public int storageRfUsage;
    public int wirelessTransmitterRfUsage;
    public int gridRfUsage;
    public int craftingGridRfUsage;
    public int patternGridRfUsage;

    public int wirelessTransmitterBaseRange;
    public int wirelessTransmitterRangePerUpgrade;

    public int rangeUpgradeRfUsage;
    public int speedUpgradeRfUsage;
    public int craftingUpgradeRfUsage;
    public int stackUpgradeRfUsage;

Any more suggestions for config options?

Looks good the RF usage was the main thing I could think of.
Only thing I could think of is the speed upgrades effect could be configurable like the range upgrades.

Fixed


output node names (for --export_saved_model)

Hi all,
I was trying to save models with --export_saved_model to be able to re-use them.
It all works, and I am able to reload the models with
tf.saved_model.loader.load(             sess, [tf.saved_model.tag_constants.SERVING], frozen_graph_filename)
The problem though is that I cannot find the output node name (I am using transformer here).
If I look at the output ot the transformer model, I do not see int (i.e., token_id), but float (I assume hidden states?).
I am trying to look into the symbol modality (possibly bottom?), but I still canno find the output node (containing the output token ids).
I also tried to list all the nodes in the saved models, but if I grep for the type int, I do not see the correct one.
(I tried with some softmax ones, but no luck)
Does anyone have some insight on this?
Thanks,
Mirko

from the tensorboard you can get names of all ops

adding some information - I tried to compare the graph I can see on the tensorboard with the nodes I get after freezing the models.
I think they are different, e.g., in the tensorboard I see a loss node, while I cannot see that in the frozen/exported pbtxt.
This seem to make sense, given the tensorboard is looking at the training models (which obviously care about the loss value) while the exported models are kind of the inteference models (not needing the loss - just predictions).
Is that correct?
In any case, I still cannot find some suitable output nodes (for the output token_ids).
I am wondering if some part of the graph does not get serialized?
(e.g., the part that argmax the logits in the decoder to get the token_ids?)
Thanks,
Mirko

I may have found a point to 'intercept' the output node (and possibly rename it so that it can be easily used  after loading pbtxt):
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py#L217
even so, I am still in doubt about the question above - in particular, it would seem that I need to feed the placeholder "targets", which is kind of weird (that should be there only for training models, not to produce predictions)...
I remember I had the same problem with the 'old' seq2seq code (I was freezing the 'training' version of the decoder - that was getting its input from the targets, not from the previous output) - do you know if something similar may be happening here?
(checking now if the eval_run_autoregressive may play a role here..)

hi, how do you set the input as your code, as a placeholder?
now, i want to export the graph out to ensemble in my C++ project, any ideas?
thanks a lot.

you can add an explicit name to the output node by adding the line samples = tf.identity(samples, 'output') before the return statement in https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py#L217
Then, you can train a model/export, and use it, e.g., with:
import tensorflow as tf

def load_graph(exported_models, token_ids):
    with tf.Session(graph=tf.Graph()) as sess:
        tf.saved_model.loader.load(
            sess, [tf.saved_model.tag_constants.SERVING], exported_models)
        graph = tf.get_default_graph()
        # preparing input
        name = 'inputs:0'
        node = graph.get_tensor_by_name(name)
        xs = {}
        xs[node] =  token_ids
        # preparing output
        ys_names = 'output:0'
        ys = graph.get_tensor_by_name(ys_names)
        # run
        return sess.run(ys, feed_dict=xs)

exported_models = 'output/export/Servo/1512144610'
token_ids = [7, 7, 4, 9422, 282, 1]
result = load_graph(exported_models, token_ids)
print('result size: {}\ndump: {}'.format(len(result), result))

unfortunately, this still does not work - if you provide 6 token ids (as in the example above), instead of being parsed as a a sequence with 6 tokens, it will be parsed as 6 sequences of length 1.
Maybe related to how tensor2tensor deals with batches?
(see @lukaszkaiser here: https://github.com/tensorflow/tensor2tensor/issues/17 )

Several sections of this guide may be helpful.

Also take a look at the end of the saved_model.pbtxt file that's part of the export. In the signature_def it has the names of the output tensors.

@mirkobronzi hi, i have read the doc of EstimatorSpec, and it has a item name "export_outputs" which should be the export names
but i don't know how to get the input node as a placeholder to use in predict, do you have a idea about this?

hi, how do you export model? can you show me your command?
i have tried to export my trained model with command below, but it fails
see "https://github.com/tensorflow/tensor2tensor/issues/457"
thanks a lot.
"
t2t-trainer  --data_dir=/home/devops/yuimo/mt/translate_zhen_wmtun_base32k/t2t_data --t2t_usr_dir=/home/devops/yuimo/mt/code/t2t_wmt_zhen/data_generators --tmp_dir=/home/devops/yuimo/mt/translate_zhen_wmtun_base32k/t2t_data/original --problems=translate_zhen_wmtun_base32k --model=transformer --hparams_set=transformer_zhen_wmtun_base32k --output_dir=/home/devops/yuimo/mt/translate_zhen_wmtun_base32k/t2t_data/base --train_steps=5 --eval_steps=50  --keep_checkpoint_max=200 --save_checkpoints_secs=3600 --export_saved_model=True"

thanks a lot @rsepassi - that helps with node names.
So, thanks to that link, I am now using the 'official' commands to run the exported models.
Unfortunately though, I get exactly the same problem, i.e., it is like the code is interpreting N tokens as N separated utterances with a single token.
E.g., (note these medels are trained just a few steps - so, the token ids do not make sense)
saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1]]],[[[2]]],[[[3]]],[[[4]]]]'

Result for output key outputs:
[[151 188  15 319 259 259 259 188  15 319 188 151  15   3  15 319 188 151
  151  15 319 188 151 188 151  15   3  93  15 319 151  15 151 188  15 319
  151 151  15 319 188 151 188 151  15 319 188 151 188 151  15]
 [311 330  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67
   67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67
   67  67  67  67  67  67  67  67  67  67  67  67  67  67  67]
 [374   7  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22
   22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22
   22  22  22  22  22  22  22  22  22  22  22  22  22  22  22]
 [311 330 209 209 209 209 151 209 209 209 209 209 151 209 209 209 209 209
  151 209 209 209 209 209 209 151 209 209 209 209 209 151 209 209 209 209
  209 151 209 209 209 209 209 209 151 209 209 209 209 209 151]]

note how I feed 4 tokens but I get 4 utterances in output.
As a test, I can run this only on a single token only (the first one), and I get exactly the first output above:
s$ saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1]]]]'                    Result for output key outputs:
[[151 188  15 319 259 259 259 188  15 319 188 151  15   3  15 319 188 151
  151  15 319 188 151 188 151  15   3  93  15 319 151  15 151 188  15 319
  151 151  15 319 188 151 188 151  15 319 188 151 188 151  15]]

I think this confirms that, somehow, a N token input is parsed as N utterances (single token).
Any idea why this is happening?

I think the shape of the inputs is wrong the way you're passing them in.
Try [[[[4, 3, 2, 1]]]]
On Mon, Dec 4, 2017 at 10:55 AM Mirko Bronzi<EMAIL_ADDRESS>wrote:

thanks a lot @rsepassi https://github.com/rsepassi - that helps with
node names.
So, thanks to that link, I am now using the 'official' commands to run the
exported models.
Unfortunately though, I get exactly the same problem, i.e., it is like the
code is interpreting N tokens as N separated utterances with a single token.
E.g., (note these medels are trained just a few steps - so, the token ids
do not make sense)
saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1]]],[[[2]]],[[[3]]],[[[4]]]]'
Result for output key outputs:
[[151 188  15 319 259 259 259 188  15 319 188 151  15   3  15 319 188 151
151  15 319 188 151 188 151  15   3  93  15 319 151  15 151 188  15 319
151 151  15 319 188 151 188 151  15 319 188 151 188 151  15]
[311 330  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67
67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67  67
67  67  67  67  67  67  67  67  67  67  67  67  67  67  67]
[374   7  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22
22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22
22  22  22  22  22  22  22  22  22  22  22  22  22  22  22]
[311 330 209 209 209 209 151 209 209 209 209 209 151 209 209 209 209 209
151 209 209 209 209 209 209 151 209 209 209 209 209 151 209 209 209 209
209 151 209 209 209 209 209 209 151 209 209 209 209 209 151]]
note how I feed 4 tokens but I get 4 utterances in output.
As a test, I can run this only on a single token only (the first one), and
I get exactly the first output above:
s$ saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1]]]]'                    Result for output key outputs:
[[151 188  15 319 259 259 259 188  15 319 188 151  15   3  15 319 188 151
151  15 319 188 151 188 151  15   3  93  15 319 151  15 151 188  15 319
151 151  15 319 188 151 188 151  15 319 188 151 188 151  15]]
I think this confirms that, somehow, a N token input is parsed as N
utterances (single token).
Any idea why this is happening?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensor2tensor/issues/448#issuecomment-349066706,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEGWwuutcl6PaOhr_F2DKcezJ5H52zpks5s9EAtgaJpZM4Qtco-
.

thanks @rsepassi , I actually tried that (forgot to mention).
The shape will complain in that case:
$ saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1, 2, 3, 4]]]]'
Traceback (most recent call last):
...
ValueError: Cannot feed value of shape (1, 1, 1, 4) for Tensor u'ExpandDims_2:0', which has shape '(?, 1, 1, 1)'

It seems to me the only accepted shape is [[[[1]]],[[[2]]],[[[3]]],[[[4]]]].
Indeed, it seems the only 'unbound' dimension is the first one (the -1 one):
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
    dtype: DT_INT32
    shape: (-1, 1, 1, 1)
    name: ExpandDims_2:0

Ah interesting. Then this may indeed be a bug. I’ll look into it.
On Mon, Dec 4, 2017 at 12:51 PM Mirko Bronzi<EMAIL_ADDRESS>wrote:

thanks @rsepassi https://github.com/rsepassi , I actually tried that
(forgot to mention).
The shape will complain in that case:
$ saved_model_cli run --dir output/export/Servo/1512078709/ --tag_set serve --signature_def output --input_exprs 'inputs=[[[[1, 2, 3, 4]]]]'
Traceback (most recent call last):
...
ValueError: Cannot feed value of shape (1, 1, 1, 4) for Tensor u'ExpandDims_2:0', which has shape '(?, 1, 1, 1)'
It seems to me the only accepted shape is
[[[[1]]],[[[2]]],[[[3]]],[[[4]]]].
Indeed, it seems the only 'unbound' dimension is the first one (the -1
one):
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_INT32
shape: (-1, 1, 1, 1)
name: ExpandDims_2:0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensor2tensor/issues/448#issuecomment-349101294,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEGW7quU1FqaAC1NZrRAJ4nHvGsl3dpks5s9FtigaJpZM4Qtco-
.

@mirkobronzi i have exported the model, but when loaded it in python ,the predict result is totally wrong, see issue: https://github.com/tensorflow/tensor2tensor/issues/462
have any ideas about this?
or have you succeed with exporting models?
thanks.

thanks again @rsepassi
@yuimo : well, as you see in this thread, I cannot get meaningful results unfortunately..
(but not sure it's your same problem though)

@rsepassi: yes, it is working now - thanks again.
one thing: it would seem that the decoder (using transformer here - not sure it's the same for all models) is running past the EOS.
(note I am sending a single utterance per batch)
Is this expected?
I see a stop_at_eos option in the code:
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py#L661
but it seems to not be used while t2t-decoding or when using export.
(so, also when using the exported models)
Is there a way to enable this?
Thanks,
Mirko

I think I can confirm the above.
Also, I saw (in 1.4.2) that the transformer greedy decoder was running past EOS.
(I was able to fix it locally)
Not sure though this is the case anymore in the new versions.
Anyway, I think I can close the issue - thanks a lot @rsepassi  for all the help.

I find there is no difference when set --export_saved_model=True or False?
$ cd t2t_train/model
$ ls -lh
 81B Aug 10 13:58 checkpoint
 19M Aug 10 13:58 events.out.tfevents.1533880718.localhost
1.5K Aug 10 13:58 flags.txt
952B Aug 10 13:58 flags_t2t.txt
8.6M Aug 10 13:58 graph.pbtxt
3.5K Aug 10 13:58 hparams.json
 24B Aug 10 13:58 model.ckpt-1.data-00000-of-00002
114M Aug 10 13:58 model.ckpt-1.data-00001-of-00002
 15K Aug 10 13:58 model.ckpt-1.index
5.0M Aug 10 13:58 model.ckpt-1.meta


'Unable to resolve type SvgImage from namespace'

When opened MainWindows.axaml, I got this error from output
System.Xaml.XamlException: 'Unable to resolve type SvgImage from namespace https://github.com/avaloniaui Line 124, position 60.' Line number '124' and line position '60'.
IDE extension list:

Problem is I already installed Svg packages for Avalonia and restart VS still got error with resolve type

Mhh, sorry but I've never encountered this.
I'm using Rider for my dotnet solutions and it doesn't have this problem.
Maybe Visual Studio needs more for Avalonia development?

Alright, turns out, there were some bigger changes for Avalonia UI version 11.
I've fixed everything, and it should work with my most recent push to master.