IWW Issue #36, Feb 2nd 🎂 2021

This pull request is for gathering links for the February 2nd issue of the Immersive Web Weekly newsletter for the W3C Immersive Web Community Group.
If you have a link to a news article, event, artwork, or project that is directly related to the immersive web then we welcome your help!
To submit a link comment below using this template:
title: Example title of link
url: https://example.com/
author: Alice Smith
content: A short description of what is at the link and why it's related to the immersive web

Please keep in mind that comments here and all submitted links are public!
We keep each issue pretty short so we won't immediately use every submission but we keep track of them all and they may appear in a later issue.
Deadline
IWW goes out each Tuesday morning around 10am Pacific time.
Trevor will edit each issue on the Monday before publication so submissions after that will roll over for use in future issues.
If you need editorial support then email @TrevorFSmith<EMAIL_ADDRESS>

title: Feb 2 is the last day of the Sundance New Frontiers VR
url: https://fpg.festival.sundance.org/film/ntf/catalog

title: GamesBeat Into The Metaverse (sic)
url: https://venturebeat.com/event/gbfbsummit/on-demand/

title: Hubs Supports Edge and Chrome in WinMR Headsets
url: https://mobile.twitter.com/brianpeiris/status/1354597427752394758

title: Magic Leap partners with Google Cloud
url: https://www.magicleap.com/en-us/news/partner-stories/magic-leap-announces-partnership-with-google-cloud-to-build-the-next-era-of

title: Interaction Toolkit Support in Unity WebXR
url: https://twitter.com/marcel_wiessler/status/1356376306968109068

title: New Series: BabylonJS How It Works
url: https://youtu.be/LBPRmGgU0PE

title: Web Slinger
url: https://spiderman.webvr.link/


I want to contribute to this projects

Hey, I did not how to contact you, so I decided to open a Issue. I'd love to contribute to this project because I was looking for something like, I even thought about building one myself, but I found this projects which is quite advance. Maybe we could talk send me a message...

hey @Santiago-Torres14, thanks for reaching out -- I'd love the help. I've been slowly working away to improve the Admin UI (#30) and email builder (#37). If you'd like to jump on some of the existing issues, I can help you get started or if there's something particular you'd like to work on, feel free to create more issues.


feat: Crossmint type + useCrossmint

Description
as discussed:

changing Crossmint from a class to a type
creating useCrossmint and CrossmintProvider
i added a little utility function createCrossmint, which just validates the api key as well

Test plan
not much to test, just a simple type and hook

Closing as I think another PR with similar changes has landed


wait for thread to start instead of sleep(2)

I think this change will help. Do we want to put an upper limit on how long we are willing to sleep in the loop. I'm being paranoid.

pthread_create succeed so thread should start
Worse comes to worse bats will timeout

One comment though: Can we change the check for the value returned by pthread_cancel() from this: ASSERT(s == 0 || s == ESRCH); to this: ASSERT(s == 0); ?

I thought I did.

One comment though: Can we change the check for the value returned by pthread_cancel() from this: ASSERT(s == 0 || s == ESRCH); to this: ASSERT(s == 0); ?

I thought I did.

I don't see that change there.


Unable to use in project/package where "type": "module"

Great project but I'm having issues with using it in a project/package when "type": "module" is set in the package.json.
I get the following errors:
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/user42/Documents/ITDev/projects/proj123/config.migrate.js from /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js not supported.
config.migrate.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead rename config.migrate.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/user42/Documents/ITDev/projects/proj123/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'

// Using config: config.migrate.js
const { database } = require('@abc-lib/load-config').loadConfig();

module.exports = {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

Also tried a config with -C after renaming the config to config.migrate.cjs but that produced other errors
/Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs:1
const { database } = require('@abc-lib/load-config').loadConfig();
                     ^

Error [ERR_REQUIRE_ESM]: require() of ES Module<EMAIL_ADDRESS>from /Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs not supported.
Instead change the require of index.js in /Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs:1:22)
    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'
}

// Using config: config.migrate.cjs
const { database } = require('@abc-lib/load-config').loadConfig();

module.exports = {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

And also tried using ESM in the config but got these errors:
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/user42/Documents/ITDev/projects/proj123/config.migrate.js from /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js not supported.
Instead change the require of config.migrate.js in /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js to a dynamic import() which is available in all CommonJS modules.
    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'
}

// Using // config.migrate.js
import { loadConfig } from '@awt-lib/load-config';

const { database } = loadConfig();

export default {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

Would be great to use this but I cannot see how to work around this.

I'm also able to reproduce this by adding "type": "module" to the package.json file in the example repository https://github.com/TimMikeladze/arango-migrate-example.
Thanks for opening this issue, I will investigate and get back to you.

Awesome, thank you!

@clintwood Hi, a new version of the package has been published which fixes this issue. You can check  out a functioning example in which the package.json file contains the "type": "module" here:
https://github.com/TimMikeladze/arango-migrate-example.
Let me know if there are any more problems.
P.S. Sorry it took so long to get this done. I got severely side tracked and am just coming back to this project now.


Insights operator API

This is the first part (with operator status API) and follow-up of https://github.com/openshift/api/pull/1193 and the original enhancement https://github.com/openshift/enhancements/pull/1037

/label docs-approved

/test verify
I think that's a flake
Otherwise LGTM
/lgtm

/test verify

/label qe-approved

/label px-approved

/approve


Installing SAP HANA 2.0, express edition (Virtual Machine Method)

Need your help please! ^_^
Issue: hsexsa cannot run in VM method, more details on attached screen capture.
Environment: macbook, VMware Fusion pro 8.5.7

In my experience, it is impossible for initial VM setup to work successfully on a MacBook because it has only 16GB of RAM. I had to perform the initial setup in a host with more RAM (I set the VM to 24GB if I recall) and then did initial setup. Once that was completed, I shut down th VM and copied the .VMX files over to my MacBook and it then has been running fine with 12GB.

Hi,
A new version was released yesterday for the VM addressing this. Could you please let us know how it works? Also, on my machines, I change the RAM setting to 16GB (which is what the host has). It may get slow at times but it finished if I leave the host alone.
Cheers,
Lucia.

Hi Lucia, Yes I was about to test it and it works on my MAC. Please use the latest image from SPS 02.

Hey i831123/Lucia,
So as we know, MacBook laptops top out at 16GB of RAM.  I wasn't aware that you could set RAM of a guest VM to 16GB (otherwise the host has no RAM to host) - I assume it's doing some virtual swapping to pull of this feat?  Very cool if so :)
Do you have a recommended bare-minimum amount of RAM that you'd suggest setting an HXE 2.0 machine at which could run XSA?  I've managed to run mine with 12GB, but only after turning off that bloated XSA HANA Admin Cockpit...

Hi - There is a n SAP HANA express edition sizing guide on the SAP Help Portal that lists the memory requirements for each optional component:
https://help.sap.com/viewer/9e4243e92f244537b2164a57a405a9fd/2.0.02/en-US
Please take a look and let us know if this helps with your installation.

That is very cool.  Is this new?  I was reading through the documentation about a month back and must have missed this otherwise.  This would be an extremely handy reference for the tutorial documentation as well!  Thanks for pointing that out.


Add Swarm.Supervisor - a dynamic supervisor for distributed processes

Implement a new supervisor module which can ensure registered processes are properly restarted according to normal supervisor semantics, but are also restarted/shifted based on cluster topology changes.
This supervisor will be for dynamic children only.

Does this address the problem where currently in the example provided, if the process dies it is then restarted by the supervisor yet it doesn't register with Swarm?

@dax0 yes, though in the current example it shouldn't be restarted since the restart type is :temporary (meaning that if it fails, it is not restarted)

So currently what is the best way to create a process that is managed by Swarm that should be restarted?

The following seems to fail to restart when Worker crashes
defmodule Worker do
	use GenServer
	def start_link(key) do
		GenServer.start_link(__MODULE__, [key], name: tuple(key))
	end

	def tuple(key) do
		{:via, :swarm, key}
	end

end

defmodule Worker.Supervisor do
	use Supervisor

	def start_link do
		Supervisor.start_link(__MODULE__, [], name: __MODULE__)
	end

	def init(_) do
		import Supervisor.Spec
		children = [
			worker(Worker.Instance, [], restart: :transient)
		]
		supervise(children, strategy: :simple_one_for_one)
	end

	def start_child(key) do
		{:ok, pid} = Supervisor.start_child(__MODULE__, [key])
	end

end

With simple_one_for_one, if the process exits with :normal, :shutdown, or {:shutdown, term}, it won't be restarted. How are you simulating the crash?

I'm sending an abnormal exit message with Process.exit
On Mon, Oct 31, 2016, 10:40 PM Paul Schoenfelder<EMAIL_ADDRESS>wrote:

With simple_one_for_one, if the process exits with :normal, :shutdown, or {:shutdown,
term}, it won't be restarted. How are you simulating the crash?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bitwalker/swarm/issues/6#issuecomment-257476902, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAydINA_5H0UvwmFoxU06q6mmYQKphubks5q5qaNgaJpZM4KIBhd
.

I have the same problem when using transient restart strategy and spawning children with Swarm.register_name. The child gets restarted but not registered. If I can help to test anything let me know.

So, we're just starting a project and were looking to use swarm with this exact use case in mind. Is this currently still an issue?

@bschaeffer Yes, if you have a need for supervising the processes being dynamically registered, you have to link/monitor those children and handle restarting them yourself currently - I would like to add a supervisor module to do this out of the box, but as I haven't had the need for it myself yet, and unfortunately lack the time to develop it at the moment, it may be awhile before I can get to that myself. That said, handling the restart yourself as mentioned above is fairly easy to do, so it's not a huge obstacle, just something to be aware of.

That said, handling the restart yourself as mentioned above is fairly easy to do, so it's not a huge obstacle, just something to be aware of.

I'm not sure what you're referring to? Currently transient children aren't restarted automatically and there's no solution provided above. Unless I'm missing something?

@kwrooijen I was referring to the first part of my comment, which is supervising children directly by linking/monitoring them and reacting to down/exit signals appropriately - this is what Supervisor does under the covers (it also does things like handling too many restarts within a given interval, etc., but it's core functionality is simply linking and monitoring the process and reacting to those signals).

I made a makeshift dynamic supervisor that will re-register the name when the child process terminates with reasons other than :normal and :shutdown (Important for hand-off). Tested this with 2 nodes, and now Swarm.registered/1 always returns the actual process. When I call Process.exit(Swarm.whereis_name("process_name"), :a) it will be restarted and re-registered.
Here's the code in a gist: https://gist.github.com/narrowtux/3b21967599e1f42787b24667633f68f0
It's not a full Supervisor in the sense that it does not give you a behaviour, it doesn't accept Supervisor.Spec maps, you don't start it with supervisor(Swarm.DynamicSupervisor), but with worker(), and it won't crash if the child process has restarted too often.
These are all things that I imagine @bitwalker wants to be fixed before merging it into this library, but it's a start for all those who need this feature right now.

beardedeagle, I believe that the flaw with that setup is that if the process crashes and is restarted by the supervisor it will no longer be tracked by swarm. After a restart Swarm.whereis_name/1 returns :undefined. At least that's what my testing here indicates.
I took the route of creating a :simple_one_for_one supervisor (back before DynamicSupervisor existed) that uses register_name/4 inside its start_child.  This method works well in the following scenarios:

it will restart the process if it crashes, re-registering with swarm
if the topology changes and the process is started on a new node it will be supervised again

The thing that I'm not pleased with is that a supervisor could end up supervising a process on a remote node since register_name/4 is at liberty to start the mfa on any node. I have a feeling that this setup may not end well if the network is unstable. This could likely be solved by having the supervisor call the other supervisor and hand over control of the process.
It seems like the best of both worlds might be something like register_supervised_name/5 where it would take name, supervisor, m, f, a (or name, supervisor, spec) and supervisor would be expected to exist as a local name on all nodes. Then swarm could let the process to be supervised by a local supervisor as usual and it could still migrate the process off if the topology changed.

hey guys, I'm working on a supervisor which will be designed exactly to do that what @jerel mentions. It is based on DynamicSupervisor - it's still work in progress (I have to fix API for naming processes) but generally it works, but for now you have to pass name of the process in first parameter of args passed to worker. My repo: https://github.com/kelostrada/swarm_dynamic_supervisor

Just an update here:
To properly handle supervision, support needs to exist within Swarm itself, at least for things which use register_name/5 to handle auto-distribution and the like. This is because those registrations are handled via a primitive form of supervision within Swarm, and in order to mesh nicely with a real supervisor, Swarm needs to delegate those supervision tasks to that supervisor. If you try to supervise things registered with register_name/5, you'll find yourself fighting Swarm's own supervision.
To that end, I've been building out support for using :one_for_one supervisors with Swarm. It expects that the supervisor is a locally registered process (i.e. not via Swarm), and that each node in the cluster is running an instance of that supervisor. This setup allows Swarm to move processes around the cluster properly, while leaning on the supervisor for restarts. Unfortunately, DynamicSupervisor is out, because it eliminated the one facility we had with supervisors to track specific children (which is necessary in order to identify which pid belongs to a given registration), likewise :simple_one_for_one supervisors are out. These restrictions aren't really a problem, but they are still restrictions. I'm hoping to have this support available soon, but I need to do considerable testing, since it touches a lot of internals in the tracker.

@bitwalker I'm not sure why it should touch all the internals, however I created a simple DynamicSupervisor which reregisters names in Swarm on start and after restart (by calling Swarm.register_name which later calls GenServers call to start the process in real supervisor).
You can check my project I linked before, after some testing I think it works nicely and to be honest in real logic it doesn't differ much from the example for :simple_one_for_one that is provided here in README.
My knowledge on the subject is probably limited though so I might be doing something wrong. However for now it seems to be working as expected (process groups don't work this way though).

@kelostrada How are you handling the case where the cluster topology changes? Swarm assigns processes started via register_name/5 based on how the name is sharded across a hash ring containing all of the nodes in the cluster, if nodes come or go, then some percentage of processes will be reassigned to other nodes, and Swarm will perform a handoff process to make that happen, the way this works today is that the new process is started, the old process is asked how to perform the handoff, the old process is then killed, and if handing the state off, we do so to the new process, and then the registry is updated. In your case, at the point the old process is killed, you would attempt to restart the process, via Swarm, and would always return {:error, {:already_registered, pid}} since the new process is already running and registered. If you've specifically handled this problem to take that pid and then supervise it under the correct supervisor, then yeah, it probably works more or less as intended - but if you are raising an error, or not supervising the pid in that case, then you've just orphaned the process from supervision. It's also the case that Swarm is already effectively supervising for you in this case (albeit a very simple form of it), and due to the extra time required to do things via the registry, a lot of supervisor configuration (e.g. restarts/shutdowns) is either not useful anymore, or just plain ignored by Swarm (i.e. Swarm will just keep trying to restart a process, even if the shutdown strategy indicates it should not do so). The restart strategy will be inaccurate, since the time it takes to go through Swarm and contact the tracker on another node is not predictable, so saying "only restart 3 times in 5 seconds" may never actually trigger the overload case if it takes much more than a second to perform startup/registration.
However, if Swarm understands external supervison and delegates it to a true supervisor, then restarts/shutdowns are honored, they don't require going through the tracker, so they behave like any other supervised process; Swarm just needs to consult the supervisor to get the new pid of the restarted process (or in the case where the supervisor dies or the child is removed due to restart strategy, we don't bother re-registering it). We can also coordinate handoff in such a way that the supervisor doesn't think it needs to restart the old process by using terminate_child and delete_child at appropriate points in the handoff process.
I guess all I'm trying to say is that yes, you can get something that works pretty well without explicit support from Swarm, but there are hidden edge cases that may bite you if you aren't aware of them, and it's my opinion that Swarm should support external supervision. I've got most of this done locally, but there are still a few bits left, plus testing to perform. If there are specific supervision requirements you think need to be addressed, now is definitely the time to bring those up.

Thanks for your indepth analysis. Yes you are right I didn't handle this case. However with :ignore handoffs this will still work (although obviously I'm missing a lot of functionalities of swarm this way).
I never said this is not needed to be supported internally by Swarm out of the box. I'm very happy to hear you are working on the solution already. Do you have your work in progress hosted somewhere on github perhaps? I would be happy to help test it.

I think there are a few more issues to address. I wasn't sure where to post them, I guess this post is a good start.
In my case, there is a dynamic supervisor (Regent) that is used to spawn things using Swarm.register_name/5. What gets started is also a supervisor (BallotBox), which in turn starts a BallotBox.Logic process and its associated BallotBox.State process.
The first issue is that even tho the top-level process supervisor is started and distributed properly, starting the child processes becomes quite hard with Swarm:

Since adding name: {:via, :swarm, ...} in the child spec, or simply using Swarm.register_name/2 both result in a deadlock, we're using the later in a Task.async. This means that there is a time (even tho it's short, it exists) during which the process exists but isn't named / registered. That's however a minor problem.

The second problem is probably more important: it is impossible to have anything distributed this way that would accept state handoff: Swarm has no idea on how to move the small processes to another node and therefore cannot possibly handle the transitioning.
I asked around on the Elixir Slack, and pretty much no one around attempted to answer - except for one person who recommended to stay away from Swarm if we also wanted a more complex supervision tree.
So right now, we're trying to mitigate and see if we could maybe conform to Swarm's expectations, for example by having a single supervisor per type of process, so we can use Swarm.register_name/5 for all the processes we want started - but this shows a third issue we don't know how to handle:

Imagine that we have a StateSupervisor and a LogicSupervisor, both in charge of spawning State and Logic processes using a function that Swarm.register_name/5 understands.
If we do so, both processes will have a different name, and therefore they might not end-up distributed in the same node. Not very ideal for "pairs" of classic state + logic processes, as it would add inter-node network delay between the process and its state.
I guess we could solve the problem by implementing our own distribution hashing algorithm, but then again that would go a bit on the "off-road" side of things.

I hope the feedback will help to shape that supervisor @bitwalker mentioned, or maybe point to where some documentation is lacking (I could very easily admit that some of these problems would be related to my usage of Swarm in a wrong way).

@hickscorp Maybe an idea is to let the state handoff/restoration logic live in the BallotBox since that is the level of abstraction that is registered and moved around by Swarm. So you make process-subtree relocation a concern of the subtree's root.

@Qqwy Yeah but that's the thing - a BallotBox is a supervisor, so it cannot handle the handoff message (Where would I implement it? :D )

Has there been any update on this?

What about this issue?

I believe this is still planned, but time has been focused other places. There are ways you can handle this yourself in the interim though.

Are there any examples of how this can be handled in the interim? From reading this thread, it seems that @narrowtux's pseudo Supervisor might be closest to working with restarts and topology changes. But @beardedeagle if you have any examples of what you had in mind, that would be great.

So not sure what @beardedeagle has in mind either ref. the ways you can handle this yourself, @cadamis , but I figured that instead of a worker, i.e. GenServer, I start a Supervisor (with one_for_one) strategy, which starts the worker and also acts as a proxy.
The trick I am using here is that my Supervisor is registered in Swarm, and whenever I send a message, say, GenServer.call, it's being sent to the supervisor.
Supervisor, however, immediately forwards the message to it's started worker, and returns {:noreply ... }, leaving the responsibility of returning value to the caller by the child process.
Unfortunately this also means that things like hand-off when topology changes have to be done on the Supervisor level. Basically it wraps around the GenServer worker and provides all of the infrastructure needed for Swarm + normal supervision functionality, i.e. restarting , and also is a proxy whenever it needs to be called. Not ideal but works.

I ended up working around this by using a standard DynamicSupervisor/Genserver worker setup, and then doing a check in handle_continue to see if the process starting up was registered to Swarm. If not, spawn a task to start under Swarm, and kill the starting up process. It's ugly, and could potentially lead to dropped messages, but will work for my particular use case. Basically this:
  def handle_continue(:load, name) do
    if Swarm.Tracker.whereis(name) == :undefined do
      spawn(fn ->
        Process.sleep(500); Swarm.register_name(name, MyApp.MySupervisor, :start_child, [name])
      end)
      {:stop, :normal, nil}
    else
      state = do_normal_start_stuff(name)  # insert normal logic here
      {:noreply, state}
    end
  end

Assuming do_normal_start_stuff handles state restoration (which it should, under a normal DynamicSupervisor/GenServer worker setup), this works well enough to ensure Swarm workers are restarted by the DynamicSupervisor and still tracked by Swarm. I'll see if I run into any major pitfalls with this.


Versioning or Roadmap?

First of all - love this. A principled, light-weight markup unsaddled by baggage, and designed by someone with the chops to appreciate the nuances of every decision.
I was wondering how stable you considered the current implementation. How far is this from a 1.0 release? What would it take for this to become fully integrated into Pandoc? Do you hope for this to eventually replace Pandoc Markdown?

The current version is definitely "alpha."  I want to get more feedback and think about things some more before saying it's stable in any way.
If djot starts getting used by people, we could integrate it into pandoc.  It would take Haskell implementations of the parser and writer (or we could cheat and just load the Lua). Full integration can wait, though, because we already have bidirectional conversion with pandoc, via djot-reader.lua and djot-writer.lua.
In any case pandoc would continue to support pandoc markdown.


Can I add code snippets for text processing?

Hi, I primarily work on NLP. So, I do a lot of text processing using built-in methods and re module. Can I contribute code snippets of text processing?

Thanks for the idea. However, text processing might require a lot more explaining than we can do in 30 seconds, as the user might not be familiar with many different concepts that they should understand before diving into code and implementation details showcased in a snippet.

@Trinityyi I am not talking about full-fledged text processing. I am interested in simple stuff that we usually google while programming.


Federated service DNS should be created for all zones

 Issue by zihaoyu
Friday Aug 11, 2017 at 17:06 GMT
Originally opened as https://github.com/kubernetes/kubernetes/issues/50529

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
If a federated cluster is a multi-zone cluster, then when a federated LoadBalancer type service is created, there is only one zone DNS record created. For example:
# This cluster spans three zones
➜  kubernetes git:(master) kubectl --context=hurley-beta-federation get cluster beta-us-west-2 -o json | jq '.status'
{
  "conditions": [
    {
      "lastProbeTime": "2017-08-11T17:36:25Z",
      "lastTransitionTime": "2017-08-08T15:15:40Z",
      "message": "/healthz responded with ok",
      "reason": "ClusterReady",
      "status": "True",
      "type": "Ready"
    }
  ],
  "region": "us-west-2",
  "zones": [
    "us-west-2a",
    "us-west-2b",
    "us-west-2c"
  ]
}

In Route53 console, I only see DNS records for one of the zones (us-west-2a). Same thing for us-east-1 federated cluster too.

I also see the following in kube-dns logs when I tried to resolve nginx.default.k8s-beta-federation locally:
I0810 20:31:17.072874       1 dns.go:636] Federation: skipping record since service has no endpoint: {<IP_ADDRESS> 0 10 10  false 30 0  /skydns/local/cluster/svc/default/nginx/6466336132663632}
I0810 20:31:17.147368       1 logs.go:41] skydns: incomplete CNAME chain from "nginx.default.my-federation.svc.us-west-2b.us-west-2.mycompany.com.": rcode 3 is not equal to success

What you expected to happen:
DNS records should be created for each zone of the cluster.
How to reproduce it (as minimally and precisely as possible):

Launch a multi-zone cluster.
Join this cluster to a federation.
Create a federated service with LoadBalancer type.

Anything else we need to know?:
It looks like, as of v1.7.3, this behavior is by design. I think we should address that TODO and add support for multi-zone clusters.
Environment:

Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration**: AWS
OS (e.g. from /etc/os-release): Container Linux by CoreOS 1353.7.0 (Ladybug)
Kernel (e.g. uname -a): Linux ip-10-72-230-146.ec2.internal 4.9.24-coreos #1 SMP Wed Apr 26 21:44:23 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz GenuineIntel GNU/Linux
Install tools: Custom tool
Others:

 Comment by zihaoyu
Friday Aug 11, 2017 at 17:08 GMT

/sig federation

 Comment by zihaoyu
Tuesday Oct 10, 2017 at 16:56 GMT

/sig federation

cc @zihaoyu


[Devportal API] Implementing Application related API flows

https://github.com/wso2/apk/issues/244


Question: how feasible is it to share Freezed objects between goja.Runtimes

The question is more or less in the title. The specific use case is an array with objects that mostly will contain strings, numbers, or other array/objects containing the same.
Obviously having a function and calling that function will be bad and having a regex can trigger #214, but are there such situations for the other types, and are there other things that need to be taken into account ... apart from me thinking that this is a bad idea and that w/e answer you give will change over time.
For more context, you can look at this discussion https://github.com/loadimpact/k6/pull/1739#discussion_r540119684

To give you some of the context, without you having to read a whole PR, the situation is like this:

a function in a single JS runtime returns an array of things (without any restriction what these things are)
we use runtime.ExportTo() to get that array as an []interface{} in Go
we give a properly locked mechanism for other goja runtimes to requests pieces of that array or iterate over it, via a Proxy

The question is, to avoid data races, do we need to serialize and unserialize (via JSON or some other mechanism) the data pieces we give to each goja runtime, so they are completely separate copies. Or can we directly return pieces of the original []interface{} slice, if we recursively call Object.freeze() in the proxy, before we return them to the users?

First of all, freezing a go value wrapped into goja.Object won't work, because it's not possible to store ECMAScript property attributes in the wrapper (as the wrapper is for the entire value, not its properties).
If you have a goroutine-safe go value (e.g. sync.Map), it's possible to share it between runtimes as long as you create an individual wrapper for each runtime, i.e.:
var m sync.Map
runtime1.Set("m", runtime1.ToValue(&m))
suntime2.Set("m", runtime2.ToValue(&m))

If you use a Proxy, then the Proxy itself and the handler also must be created per runtime. If the proxy handler is native it can be shared as long as it's goroutine-safe.
It's important to understand that the wrapped value must be fully recursively goroutine-safe, for example if the sync.Map contains unsafe values (such as regular maps) there will be data races. The recursive safety could in theory be enforced by the proxy handler: when returning property values it could wrap them into another Proxy which will enforce immutability or synchronisation.
Hope this helps.

I think @na-- confused you somewhat. Or at least I think so ...
Rewording the question somewhat: If we have an object that is only:

string, integers, floats, booleans - "simple" types (I am perfectly sure there is an ECMAScript term for it, that I don't know about)
arrays or "maps"/objects which only have keys and properties of the above variants.

Would calling Object.freeze recursively on those and then giving them to other goja runtimes be ... racy.

Yes. Non-primitive goja.Value in general is not goroutine-safe, even for concurrent reads.

I guess as a bonus - would that work if we wrap it in Proxy object for each other runtime, without the original frozen object actually be given to the other runtimes.

No, because the proxy handler will still be accessing the original value.

Each runtime would have its own Value, i.e. we'd call the equivalent of runtime.ToValue(data) for every runtime, but with the same immutable data, and one of the purposes of the freezing would have been to keep data immutable from JS code as well.
But if that's still not a good idea, do you have any suggestions how to handle the underlying problem we're trying to solve? We have some huge static data array (e.g. from parsing a huge 10 MB JSON/CSV/etc. file), and we have hundreds of concurrent goja runtimes. We want to give random access to that data to every runtime, without having a full copy of the huge array in every runtime. Streaming JSON/CSV parsers are not an option for this, each runtime should be able to request any element of the array at any time efficiently. Any suggestions?

What do you mean by 'equivalent of runtime.ToValue(data)'?

I tried to shorten the explanation a bit, to the detriment of quality, sorry...
In reality, say that goData is the huge multi-megabyte []interface{} slice in Go we want to safely share across all runtimes. The idea is that each of the goja runtimes will have a Proxy as a reference to that huge slice, called e.g. jsProxyToTheSharedData. And whenever the JS script contains jsProxyToTheSharedData[i], the Go proxy code will carefully lock the mutex for goData, and return runtime.ToValue(<something>).
The current value for <something> is essentially json.Unmarshal(json.Marshal(goData[i])), i.e. a completely independent but inefficient copy of the original slice.
And while we can probably optimize the copying (via gob, reflect, etc.), I was wondering if the <something> we return couldn't just be goData[i] (i.e. a direct reference to an element in the huge slice), provided that in the Proxy we recursively call Object.freeze() on it and all of its sub-elements before we return it to the actual script. That is, the result of jsProxyToTheSharedData[i] will be completely frozen before the user can do anything with it.
As a nice UX bonus, it will be very obvious that the result is read-only. Whereas with the current copying solution, the following script will be very strange:
var a = jsProxyToTheSharedData[someIndex]; // e.g. this is {foo: 1}
console.log(a.foo); // prints 1
a.foo++;
console.log(a.foo); // prints 2
console.log(jsProxyToTheSharedData[someIndex].foo); // prints 1

Then my initial comment stands. You cannot call Object.freeze() on a native object, i.e. the following:
func TestFreezeNative(t *testing.T) {
	vm := New()
	vm.Set("o", map[string]interface{}{})
	_, err := vm.RunString(`
	o.prop = true;
	Object.freeze(o);
	`)
	if err != nil {
		t.Fatal(err)
	}
}

will fail with TypeError: Host object field prop cannot be made read-only.
What you can do is wrap the value into a Proxy that prevents modifications and also wraps any property into a similar Proxy before returning, thus making it recursive. This is currently the only way to make a native object immutable.
But then you have to consider non-ASCII strings. ToValue() will convert them into UTF-16 which means you won't save any memory this way. Even for ASCII strings it will scan them to make sure they are, which means you'll waste some CPU.
Strings currently are immutable so you can share them, but at the moment they can only be created in a Runtime. I could make a global function NewString(s) Value, similar to NewSymbol() if that helps.

Sorry for the slow reply and the long post, as the saying goes "If I had more time ..."
I actually rewrote the feature to object.Freeze recursively the data before returning, which basically required that I JSON.parse the jsonified data which also lead to some ... other problems/complications.
To be perfectly honest I like the latest iteration more even though it is slightly less performant. The whole idea of this feature, again, is that users want to parameterize their scripts with data that they have dumped in some format (JSON, csv are common) and for that, up until now they needed to load the whole data for each VU or do some hacks as that would've meant that they have a copy of the data multiple times (a few hundred, but maybe thousands).
Example:
function generateArray() {
    return JSON.parse(open("./arr.json"));
}

var s = {};
if (__VU > 0 && __VU < 2) { // We will only load the json for VU 1
    s = generateArray();
}

exports.default = function () { // this is just what gets called by k6 on each iteration as long as there are iterations to be done
    console.log(s.length, __VU);
}

where "arr.json" has 100k lines of {"user": "myuser", "password": "password12345"} and a total of 4.7M takes around 330-360M compared to if we don't load it taking 230-250M. This is with 1000 VUs(js.Runtimes + other stuff) and no babel+corejs, again only 1 of the VUs actually loads the data, the rest just skip the if ;).
./k6 run --compatibility-mode=base -u 1000 -i 1000  arr_noshare.js

This shows me that the parsed JSON expands from around 4.7M to around 90-100M once parsed and saved in memory. If I load it in 10VUs I get 1.2+GB of usage which seems to confirm the number.
If I rewrite the script to use the feature in the PR in question the memory usage is 310-330MB when all VUs have access to it.
So at this point:

we already have made it possible for this data to be shared between VUs in a read-only way. I would argue nobody will have a problem with memory usage which was previously a big problem in these cases. In the above example a 100 VUs would take 10GB+ instead of the current 330M.( both will actually be more once they start executing and the fact that the GC will have less things to traverse is something that I haven't benchmarked).
The memory usage seems to be higher when we keep it as an object in goja even without freezing it and Proxying it. Not by much and I don't think goja should be optimized for this, but probably by enough especially for more complex objects and/or bigger JSONs.
The performance accessing suffers between 20-30% for just accessing a single element s[1], which IMO is fine given that this happens only once per iteration in most cases. And 100-200x (with the current code, the in golang iterator was faster, but not by much) if we for-of over the whole data. This was based on more realistic data, and I don't have time to rerun it with this one. The iterator performance IMO is not all that important as the use case again is to parameterize some (HTTP) request which will definitely take way more time if all of the data is used.
In my experience the majority of the cases the elements of the data are small (maybe 10-20 keys tops) and as such will have a small memory footprint, that + 2. means that we will likely use less memory for data with thousands of elements. The downside is that we keep creating new objects, but given that again the use case (in general) is that a user will get 1 element per iteration, k6 has way too many other places where optimizing the generation of an object will be way more useful.

So for me, this is a yak shaving at this point for k6.
Having said that if you or someone else tomorrow implements "ReadOnlyWrapper" that you can give either a golang interface{} or a goja.Value and it will act exactly like the original object but be read only and save for concurrent reads between VMs - I am all for using it. This was more or less the original idea, but I quickly realized that with the current goja API I will need a lot of reflection and getting of string properties to implement even something close to that. Which is why I scaled it back ... a lot, but more or less to the thing that IMO is what k6 needs.
Given that I will add some things that would've helped (but unfortunately probably all of them will need to be true in order for this to work better):

#227
JSON.stringify/JSON.parse from golang code without the need to call JS code. The golang json.Encode/Decode has slightly different behavior, which may lead to problems.
some ... better way to iterate over an array. I moved to this one as the previous one created 1 more array which likely will be even worse for performance with big arrays, but here I need to iterate in JS code
deepFreeze needs JS code wrapper which means that even with #227 we will still need to call some JS code to wrap the value on each call to the Proxy.

In a lot of cases what I find developing with goja is that I either need to call some amount of methods to get some goja.Value and then make it into a goja.Object and then get a Property as then repeat ... or I can just call a JS code that will do most of that, probably faster because it will skip a lot of the required transformations. I don't know how to fix that without exploding the API which might not be a great idea and will definitely be a lot harder to maintain. Maybe I should go look at v8 or some other implementation for inspiration :shrug: .
The problem I mostly have with the JS code is that ... well it is JS code in the middle of the go code making it harder to write/read/debug. This might be better solvable on the developer's side by just having the code in a separate file or something, but I digress.
I don't think though that neither the ReadOnlyWrapper nor the above list are as interesting or needed as much as just more ES6 support, as that lets us drop stuff from our babel+corejs combo and get both better performance and less memory usage.

I can assure you that using API is way faster than running JS code:
func BenchmarkGetPropertyAPI(b *testing.B) {
	vm := New()
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		vm.Get("Object").(*Object).Get("freeze")
	}
}

func BenchmarkGetPropertyJS(b *testing.B) {
	vm := New()
	b.ResetTimer()
	prg := MustCompile("test.js", "Object.freeze", false)
	for i := 0; i < b.N; i++ {
		vm.RunProgram(prg)
	}
}

BenchmarkGetPropertyAPI-4   	13079428	        94.9 ns/op
BenchmarkGetPropertyJS-4    	 4625163	       257 ns/op

Other than that I take it that you have found a suitable solution so this can be closed?

yeah @dop251, the solution we had previously was already ... suitable IMO :).
I might try to call things through the API more, but my benchmarking of the code doesn't show it is slow enough for this to matter ... so maybe on a second iteration in the future, or if there is more time. Also, #227 will still be needed to remove most of the code we currently need.
Thanks a lot for the discussion and I am going to close the issue now.


[SP-5742] Backport of PDI-18969 - PVFS Connection to Metastore broken in AEL (9.1 Suite)

:white_check_mark: Build finished in 33m 55s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,engine

:ok_hand: All tests passed!
Tests run: 4905, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message


Registrar: user management in the portal

Goal is to enable registrars to manage their own users and their accesses:

add/remove users
set user access rights (billing, epp, both)
set expiration date for access
manage IP whiltelist to enable access to the portal
manage PKI certificates (view (alarm when expiration date is closing and it has arrived), create, sign, revoke, remove)

The regsitrar user that has privileges to manage registrars' users must be added only using admin interface (is super user) this shuld be preferably the accredited user. Users added using the portal for registrars do not have access to user management functionality.

As a first step we should just list the data in the registrar portal - users of the registrar (username, status (active: true/false), permissions(super/epp/billing) and certificate info (incl validity)
Also it would be good to list the whitelisted IPs of the registrar

For listing and managing registrar users, access certificates and allowed IP addresses I propose using the profile view accessible by clicking on the logged in user name in the top right corner of the registrar portal view. In the account view lets us the second column for this functionality: 1st block for users and certificates and 2nd block for IP addresses.
There is the "linked users" block in this column - that has almost no use in production environment and is mostly for testing. Lest move this block to the first column below Balance Auto-Reload block and if possible display it only if the logged in user has associations with multiple registrars - so there are multiple accounts to switch between.
As a results there are three blocks in first column: Details, Balance Auto-Reload, Linked users
And two blocks in second column: API users (username, status (active: true/false), permissions(super/epp/billing) and certificate info (incl validity)) and Allowed IPs (IPv4, IPv6, interfaces)

IP address management:

list allowed ip addresses (ip address, interfaces (API, Billing, Registrar full). Billing is the same as registrar today and Registrar full is api+billing. Same approach as with user permissions
add new ip: single text box and interface selection (drop box perhaps?) - I will create separate ticket for IP address counting
add info text / bullet for ip address field with text

en: "You can enter single ip or a network and IPv4 or IPv6 address address. Keep in mind that the number of allowed IP aadresses is limited as set in the registrar agreement"
et: "Sisestada saab nii üksikut IP kui võrgu aadressi ning IPv4 või IPv6 aadressi. Pea meeles, et lubatud IP aadresside arve on piiratud vastavalt registrilepingus sätestatule"

About certificates .. I think that we agreed, that all certificates will be approved/signed manually by us, ad we will have a possibility to prepare certificate beforehand and registrar can just download it without doing anything else. This way the process will be more secure.


[8.x] Add method to dump the SQL query replacing all bindings

About this PR
This PR adds a new toSqlWithBindings method to the Query Builder class.
It allows developers to dump the query with all binding replaced with it actual value, for example:
Post::where('title', 'like', 'Example%');

With the actual ->toSql() method, it returns this:
"select * from `posts` where `title` like ?"

Of course you can use the ->dd() method, and it returns your bindings as an array, which is useful but it kinda take some time to figure out the place for each binding within your query if you have a query with too many bindings:
"select * from `posts` where `title` like ?"

array:1 [
  0 => "Example%"
]

Using the new proposed ->toSqlWithBindings() method, you can easily see  your query with all bindings:
select * from `posts` where `title` like 'Example%'

Questions
I don't know if my tests were added to the correct class. If not, tell me what class they should be in and I make the correction.

I would look into previous attempts to see why they got rejected

https://github.com/laravel/framework/pull/35910
https://github.com/laravel/framework/pull/35900
https://github.com/laravel/framework/pull/25516
https://github.com/laravel/framework/pull/21796

Also you need to account for:

booleans

true becomes 1 and false becomes 0
I think true already is converted to 1 when concatenating, but false I am pretty sure gets coerced to an empty string

nulls

gets coerced to an empty string
should be an unquoted NULL

Another suggestion is to look how these packages handle this:

https://github.com/barryvdh/laravel-debugbar
https://github.com/laravel/telescope
https://github.com/facade/ignition

They all have some sort of full query display.
Good luck

This PR won't be reviewed until it's out of draft.

I suggest checking out tools like Telescope and DebugBar.


portable version

hello, i am requesting a portable version for windows. i am not searching for something like the command --user-data-dir=
example: https://portapps.io/app/ungoogled-chromium-portable/
but official.

Please file this issue with the corresponding platform repo: ungoogled-software/ungoogled-chromium-windows


Formatting of the comment of the first member of a struct is badly indented

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
forge 0.2.0 (01325bc 2022-08-03T00:04:34.894101923Z)
What command(s) is the bug in?
forge fmt
Operating System
Linux
Describe the bug
After running forge fmt the comment of the first members of a struct is always one indentation too shallow:
    struct MyStruct {
    // first 1
    // first 2
        uint256 field1;
        // second
        uint256 field2;
    }

It doesn't matter if the comment is single- or multi-line, the entire comment has invalid indentation. All the following fields are correctly indented.

@rkrasiuk / @jpopesculian was this fixed recently?

nope, it wasn't. just tested on latest release, issue's still there


Meta data useful for LogUp-GKR

As we approach a stable version of the LogUp-GKR implementation, we should start thinking about refining the interface between the STARK and GKR. For example, in the current implementation, we open the MLEs of all of the main trace columns when only a subset of these is currently needed in LogUp relation of the virtual bus. Moreover, there is currently no mechanism to open the MLEs of shifted main trace columns (e.g., addr') or periodic columns (more generally transparent/non-committed columns).
One first attempt to address the above, is to introduce a meta data structure that can be used in order to build:

The query which is the argument to the final GKR composition polynomial.
The s-column constraints, both boundary and transition.

The following is a skeleton of how the meta data object could look like:

pub struct GkrOracle<E: FieldElement> {
    pub oracles: Vec<Oracle<E>>,
}

#[derive(Debug)]
pub enum Oracle<E: FieldElement> {
    // represents a committed oracle i.e., a column with index `index` of the main trace
    Committed {
        index: usize,
    },
    // represents a shifted (up) committed oracle i.e., a column with index `index` of the main trace
    // that is shifted (up)
    CommittedNext {
        index: usize,
    },
    // represents a transparent oracle i.e., one that can be evaluated by the verifier un-assisted
    TransparentOracle {
        transparent_oracle: TransparentOracle<E>,
    },
}

#[derive(Debug)]
pub enum TransparentOracle<E: FieldElement> {
    EQ { x: Vec<E> },
    Periodic { period: usize, values: Vec<E> },
}

pub fn generate_query_final_check<E: FieldElement>(
    gkr_oracle: GkrOracle<E>,
    eval_point: Vec<E>,
    claimed_evaluations: Vec<E>,
) -> (Vec<E>, /* Object needed for verifying the claimed evaluations */) {
    let mut query = vec![];
    assert_eq!(claimed_evaluations.len(), gkr_oracle.oracles.len());
    for (oracle, claimed_eval) in gkr_oracle.oracles.iter().zip(claimed_evaluations.iter()) {
        match oracle {
            Oracle::Committed { index } => {
                // Use `index` and `claimed_eval` to update the AIR constraints
                // ...

                // Add `claimed_eval` to the query
                query.push(*claimed_eval)
            }
            Oracle::CommittedNext { index } => {
                // Use `index` and `claimed_eval` to update the AIR constraints taking into
                // account that it is the shift of column `index`
                // ...

                // Add `claimed_eval` to the query
                query.push(*claimed_eval)
            }
            Oracle::TransparentOracle { transparent_oracle } => {
                let res = match transparent_oracle {
                    TransparentOracle::EQ { x: at } => {
                        EqFunction::new(at.to_vec()).evaluate(&eval_point)
                    }
                    TransparentOracle::Periodic { period, values } => {
                        // TODO: the following needs to be double checked

                        assert_eq!(*period, values.len());
                        // Get the evaluation point for the MLE of the periodic values
                        let ev_pt = (&eval_point[..*period]).to_vec();
                        // Build the MLE
                        let periodic_mle = MultiLinearPoly::from_evaluations(values.to_vec())
                            .expect("failed to construct periodic MLE");

                        // Evaluate the MLE
                        periodic_mle.evaluate(&ev_pt)
                    }
                };

                // Check that the claimed evaluation matches the computed one
                assert_eq!(res, *claimed_eval);

                // Add `claimed_eval` to the query
                query.push(*claimed_eval)
            }
        }
    }
    (query,)
}

The above doesn't include the logic related to the s-column but should be straightforward to add it.
Given the above, and a way to describe the core relations of the virtual bus (i.e., the fractions), we should be able to detach most of the logic that is currently part of the VM and transfer it to Winterfell.

Thank you! My understanding is that what's currently missing are two things:

Ability to pass shifted and periodic elements of a query into evaluate_fractions_at_main_trace_query() function.
Ability to specify shifted and periodic columns for the final circuit layer (e.g., here).

If I'm understanding the proposal correctly, it is mostly aimed at addressing the first issue above (i.e., computing the query values) - right? And the way it would work is roughly as follows:
First, we define a method on the Air trait that would return a GkrOracle struct - it could look something like this:
trait Air {
    ...
    fn get_gkr_oracle(&self) -> GkrOracle {
        unimplemented!()
    }
    ...
}

By default, this method would be unimplemented, but when a specific prover would use GKR-LogUp, the user would implement this method and would statically specify a set of oracles to be returned (in this context, it is not clear to me how TransparentOracle::EQ oracles are to be built).
Then, based the returned GkrOracle, Winterfell would build queries for each trace row via the generate_query() function described above. However, it is not clear to me what are the eval_point and claimed_evaluations parameters (or rather how they are computed). My understanding is that the only things that we need to build a query are the main trace and the definitions of periodic values. So, shouldn't generate_query() function look more like this?:
pub fn generate_query<E: FieldElement>(
    gkr_oracle: GkrOracle<E>,
    main_trace: &ColMatrix<Felt>,
    periodic_column_mls: Vec<E>,
    row_index: usize,
) -> Vec<E>

Another potential concern is the efficiency of the generate_query() function. My understanding is that it needs to be called for every row in the trace, and so, having generic logic there (i.e., a match statement inside a loop) may slow things down somewhat.

A potential alternative approach could be to define something like a GkrLogUpEvaluator trait. This trait could look something like this:
pub trait GkrLogUpEvaluator {

    type BaseField: StarkField;

    /// Evaluates a query defined by the provided main trace frame and writes the results into the
    /// numerators and denominators.
    ///
    /// Query values corresponding to periodic columns are built internally from the periodic column
    /// definitions specified in this evaluator.
    fn evaluate_query<F, E>(
        &self,
        frame: &EvaluationFrame<E>,
        logup_rands: &[E],
        numerator_wires: &mut [E],
        denominator_wires: &mut [E],
    ) where
        F: FieldElement<BaseField = Self::BaseField>,
        E: FieldElement<BaseField = Self::BaseField> + ExtensionOf<F>;

    /// Gets a list of all oracles involved in LogUp-GKR; this is intended to be used in construction of
    /// MLEs.
    fn get_oracles(&self) -> Vec<LogUpGkrOracle<Self::BaseField>>;
}

/// A slightly simplified version of `Oracle` with `TransparentOracle::QE` removed.
pub enum LogUpGkrOracle<E: StarkField> {
    CurrentRow(usize),
    NextRow(usize),
    PeriodicValue(Vec<E>),
}

Then, we'd add a method to build GkrLogUpEvaluator in the Air trait like so:
trait Air {
    ...
    type GkrLogUpEvaluator: GkrLogUpEvaluator<BaseField = Self::BaseField>;

    fn new_gkr_logup_evauator(&self) -> Self::GkrLogUpEvaluator {
        unimplemented!()
    }
    ...
}

Assuming I didn't miss anything major above, the main drawback of this approach that the implementor is responsible for keeping GkrLogUpEvaluator::evaluate_query() and GkrLogUpEvaluator::get_oracles() methods consistent. The benefit is that evaluate_query() should be pretty efficient since it is "hand-written".

Thinking a bit more about it, maybe an even simpler approach is to do something similar to what we do with auxiliary trace. Specifically:
First, we modify the AirContext struct to return the list of columns involved in GKR-LogUp. This could look like so:
pub struct AirContext<B: StarkField> {
    ...
    /// This would replace the current lagrange_kernel_aux_column_idx field
    pub(super) logup_gkr_oracles: Vec< LogUpGkrOracle<B>>,
    ...
}

impl<B: StarkField> AirContext<B> {
    ...

    pub fn with_logup_gkr(
        trace_info: TraceInfo,
        transition_constraint_degrees: Vec<TransitionConstraintDegree>,
        num_assertions: usize,
        logup_gkr_oracles: Vec<LogUpGkrOracle<B>>,
        options: ProofOptions,
    ) -> Self {
        todo!("implement")
    }

    pub fn logup_gkr_oracles(&self) -> &[LogUpGkrOracle<B>] {
        &self.logup_gkr_oracles
    }

    pub fn has_logup_gkr(&self) -> bool {
        !logup_gkr_oracles.is_empty()
    }

    ...
}

Then, in the Air trait, we get rid of GkrProof and GkrVerifier associated types, and instead add a single method for evaluating
pub trait Air {
    ...
    /// Evaluates a query defined by the provided main trace frame and writes the results into the
    /// numerators and denominators.
    fn evaluate_logup_gkr_query<F, E>(
        &self,
        frame: &EvaluationFrame<F>,
        periodic_values: &[F],
        logup_rand_elements: &[E],
        numerators: &mut [E],
        denominators: &mut [E],
    ) where
        F: FieldElement<BaseField = Self::BaseField>,
        E: FieldElement<BaseField = Self::BaseField> + ExtensionOf<F>
    {
        unimplemented!("evaluation of LogUp-GKR queries has not been implemented");
    }
    ...
}

The benefits/drawbacks are the same as in the approach described in the previous post but this more closely follows the existing patterns in Winterfell.
Two outstanding questions in my mind:

Can we provide periodic values into this function as we do for Air::evaluate_transition() and Air::evaluate_aux_transition()? I think so, but would be good to confirm.
Assuming that the s column will be "hidden" from the user, what is the best way to specify "boundary conditions" for LogUp-GKR?

The issue was intended to address the last part of the GKR protocol, namely the part related to the opening proof which is handled using the two auxiliary trace columns.
As there is a significant overlap between solving the aforementioned issue and the bigger issue of integrating LogUp-GKR into Winterfell, I think we can try to address this later one in this issue as well.
First, we can for the time being forget about the EQ entry. Once we finish with the current roadmap, we can start refactoring the GKR protocol so that we can not only clean up the way EQ is handled but also the way I believe we should treat periodic columns for the time being.
I am leaning towards the second proposal with a slight modification. Instead of specifying the evaluate_logup_gkr_query method we would need to specify evaluate_fractions_at_main_trace_query.
Now using logup_gkr_oracles, we can specify a few methods which, given an EvaluationFrame and periodic values, can build the appropriate input to evaluate_fractions_at_main_trace_query. Thus, evaluate_logup_gkr_query is well defined from logup_gkr_oracles and evaluate_fractions_at_main_trace_query.
To make this work, we would need:

To assume that evaluate_fractions_at_main_trace_query and logup_gkr_oracles are in sync.
That the logup_gkr_oracles are ordered, first by kind as in
 and second by id i.e., index within the main trace.

The main downside of the above is that periodic columns will have to be part of Vec<MultiLinearPoly<E>> for the input here. This is unoptimal,  as the MLE and its folds will have a repeating structure, but not by much as the main extra cost comes from storing the fully materialized MLE and folding by randomness both of which are almost negligible when compared to the rest.

Another potential concern is the efficiency of the generate_query() function. My understanding is that it needs to be called for every row in the trace, and so, having generic logic there (i.e., a match statement inside a loop) may slow things down somewhat.

generate_query is only relevant to the checking the final evaluation claims and thus is called only once by the verifier. I changed the name of that method to make it clearer.

Assuming that the s column will be "hidden" from the user, what is the best way to specify "boundary conditions" for LogUp-GKR?

The boundary conditions for LogUP-GKR should be computed from the public inputs using a method which we should probably add to AirContext.
Assuming the s columns is later removed, we would need only boundary constraints against the Lagrange kernel. The constraints against the s columns will be, from the perspective of the verifier, just evaluating an expression during constraint evaluation at the OOD point.


Added function to toggle phones

Turn on forwarding to phone 2:
client.togglePhone(2, true);
Turn off forwarding to phone 3:
client.togglePhone(3, false);
Turn on forwarding to phone 3 and use callback:
client.togglePhone(3, true, function(error, response, body) {
    var data = JSON.parse(body);
    if(error || !data.ok)
        console.log('Error: ', error, ', response: ', body);
    else
        console.log('Forwarding toggled successfully.');
});

chad, thanks for this. I will hold off on merging it, because I am working on a common method for setting different GV settings, which will include the ability to toggle forwarding.


Hello world in C

This Pull Request resolves Issue #2
Description:
In this Pull Request we have done the following additions/edits for the code:

Are there any updates here or should we close the PR and label it as spam?


What should we use for fee payment status when no fee is owed?

We need to be consistent in how we represent workshops that don't have an admin fee, and how we distinguish them from workshops whose admin fee status we don't yet know.What we want is a single field that can be NULL (we don't know the status), False (no fee owed), $0 (we know that no fee is owed), or >$0 (the amount owed).  The two fields in event are admin_fee (currency) and fee_paid (boolean, possibly null).

On Thu, May 07, 2015 at 07:29:42AM -0700, Greg Wilson wrote:

False (no fee owed), $0 (we know that no fee is owed), or >$0 (the
amount owed).

That sounds like “we're adjusting admin_fee to be the outstanding
balance for this event”, which I'm still against 1.  Are my earlier
reasons unclear?  How much accounting is amy supposed to handle?

admin_fee is supposed to always be the total amount that was agreed -
we're not going to adjust in real time, and we're certainly not going to
try to use this as an invoicing system.  But we do need to distinguish
these cases reliably so that we know who to talk to about what:

don't know status
know status - fee is zero
know status - there is/was a fee, but it hasn't been paid yet
know status - there is/was a fee, and it has been paid

On Thu, May 07, 2015 at 09:43:49AM -0700, Greg Wilson wrote:

don't know status

admin_fee is NULL.  fee_paid is irrelevant.  I'd probably set it
false, but you could use NULL or true if you like.  I don't think this
really matters.

know status - fee is zero

admin_fee is 0.  fee_paid is still irrelevant.

know status - there is/was a fee, but it hasn't been paid yet

admin_fee is > 0.  fee_paid is false.

know status - there is/was a fee, and it has been paid

admin_fee is > 0.  fee_paid is true.
That's how the currently proposed 1:
self.filter(admin_fee__gt=0).exclude(fee_paid=True)
finds unpaid fees anyway.

OK, where/how can we implement checks to make sure that these
constraints are always obeyed?  And is there an easy way in Django to
combine two values into a single displayed field?  I think non-technical
users will find it easier to work with one widget (since fee is
conceptually one thing) than keeping two widgets in sync.

On Thu, May 07, 2015 at 09:55:10AM -0700, Greg Wilson wrote:

OK, where/how can we implement checks to make sure that these
constraints are always obeyed?  And is there an easy way in Django
to combine two values into a single displayed field?  I think
non-technical users will find it easier to work with one widget
(since fee is conceptually one thing) than keeping two widgets in
sync.

What we're charging and whether you've paid seem like two conceptual
things to me.  Maybe just some JavaScript to hide the fee_paid field
if admin_fee is 0?

What about something like this:

On Fri, May 08, 2015 at 05:31:32AM -0700, Piotr Banaszkiewicz wrote:

https://cloud.githubusercontent.com/assets/72821/7536285/e31601f4-f58e-11e4-8d31-c53c05f39590.png

Looks good to me.  Not sure how easy it will be to wire up on the
Django side, but since it's your proposal I expect your comfortable
figuring that out ;).

It's really easy in pure HTML, but yeah - not sure how to do this in Django (yet). Btw, as soon as one highlights the text input on the right ("Value in USD"), the corresponding radio box selects - no JavaScript required.
If you know a better description than "Other" (maybe "Value"?), let me know.

On Fri, May 08, 2015 at 10:42:13AM -0700, Piotr Banaszkiewicz wrote:

If you know a better description than "Other" (maybe "Value"?), let me know.

I like “Value” more than “Other”.  I'm split between “Value” and “Yes”…

As @wking predicted, I have some troubles implementing this in Django.
Let's postpone it to v0.4.

Now that we've renamed the field to indicate that it's "have we invoiced", I think we can close this.

Okay, good for me, because I had hard time making this widget in Django…


RangeError: Maximum call stack size exceeded

TypeScript Version:  3.4.5

Search Terms:
I was trying to convert ION to JSON using recursive function. My recursive function only have at most 10 layers depth. I don't know why it returns stack over flow error. If My ION file has less 1000 lines, this code works. However if the file contains more than 1200 lines, it return stack errors.
Code
const getJsonFromIonReader = function(reader) {
console.log('count: ', count++);
const ionType = reader.type();
if (ionType.name === 'struct') {
const result = {};
reader.stepIn();
let current = reader.next();
while (current != undefined) {
  result[reader.fieldName()] = getJsonFromIonReader(reader);
  current = reader.next();
}
reader.stepOut();
return result;

} else if (ionType.name === 'list') {
const result = [];
reader.stepIn();
let current = reader.next();
while (current != undefined) {
result.push(getJsonFromIonReader(reader));
current = reader.next();
}
reader.stepOut();
return result;
} else if (ionType.name === 'null') {
return null;
} else if (ionType.name === 'int') {
return reader.numberValue();
} else if (ionType.name === 'bool') {
return reader.booleanValue();
} else if (ionType.name === 'timestamp') {
return reader.timestampValue().toString();
} else if (ionType.name === 'decimal') {
return reader.decimalValue().toString();
} else if (ionType.isScalar === true) {
return reader.stringValue();
}
throw new Error('encountered an ionType that is not handled: ' + ionType);
};
// A *self-contained* demonstration of the problem follows...
// Test this by running `tsc` on the command-line, rather than through another build tool such as Gulp, Webpack, etc.

Expected behavior:
It should be able to convert ION to JSON for at least 4000 lines file
Actual behavior:
It fails when the files size it more than 1000 lines
RangeError: Maximum call stack size exceeded
  at checkExpressionWorker (node_modules/typescript/lib/typescript.js:52222:28)
  at checkExpression (node_modules/typescript/lib/typescript.js:52185:42)
  at checkBinaryLikeExpression (node_modules/typescript/lib/typescript.js:51650:28)
  at checkBinaryExpression (node_modules/typescript/lib/typescript.js:51638:20)
  at checkExpressionWorker (node_modules/typescript/lib/typescript.js:52281:28)
  at checkExpression (node_modules/typescript/lib/typescript.js:52185:42)
  at checkBinaryLikeExpression (node_modules/typescript/lib/typescript.js:51650:28)
  at checkBinaryExpression (node_modules/typescript/lib/typescript.js:51638:20)
  at checkExpressionWorker (node_modules/typescript/lib/typescript.js:52281:28)
  at checkExpression (node_modules/typescript/lib/typescript.js:52185:42)

Playground Link:
Related Issues:

I found that for some reason, TypeScript can't load my 'structured' ION string. I need to remove all the space and /n

Should be fixed by #36248. Regardless 3.4.5 is rather old at this point; a reproducible sample on a newer version would be nice


Manually adding attributes to send to the next agent

New here, just heard of Huginn on a recent Changelog podcast.  Thanks for all the effort to make this awesome!
I have been playing with this and have the following scenario for watching a craigslist listing page.
https://gist.github.com/ryanwhetstone/30f72a6bdc0555e04c7965cb821943bd
Quick idea of what I am doing.

Watch https://cincinnati.craigslist.org/search/cto?query=truck&min_price=7000&auto_cylinders=5 for changes.  When there is a new listing, get the url to the listing detail page and pass this to another website agent.
Go to the detail page and extract the details of the listing (title, price, description and an image)
Pass the details to a formatting agent.
email the new listings.

Sweet, very cool, this works awesome!
The issue I came across is that in the 2nd WebsiteAgent ("Single Craigslist Listing"), I couldn't figure out how to add the url attribute to the event that was then sent to the formatting agent.  I had to add
"url": {
"css": "#qrcode",
"value": "@data-location"
}
to "re-extract" the url.  But this seemed strange to me considering it was available in the incoming agent with the "url" attribute.
So, my question is... Is there a way to manually set an attribute on the event being created?
Thanks!

Hi @ryanwhetstone,
glad you like Huginn 😄 You can set mode to merge which will include the received event data in the emitted event, the downside is that it is not possible to use merge and on_change at the same time.
I think there currently isn't a better approach then yours. It would be nice to have a general option to merge the received event into the generated data of the Agent.

Thanks  @dsander, I wasn't sure how to use the merge mode.  This makes sense.  I think I could use it for the second agent, because I don't care about the on_change part in that case, just the initial one.  I will see if I can get it working with the merge mode.
I agree, though, it would be nice to either merge or just set attributes manually.
Thanks again!

I'm going to close this issue. Please reopen if you have further questions.

@ryanwhetstone
Are you still running your CL truck search agent?
If yes, what query frequency are you using with your agent?

I'm not currently running it, but I had it going once every 12 hours.

Did you try using this scenario?  I feel like I had to change it some to get it working recently.  Like maybe the html on the page changed.  Let me know if you need me to re export the scenario.  I do have similar searches working right now and would be happy to share this with you.

@ryanwhetstone I'm new to Huggin and would like to see your current scenarios to help learn from, could you update if you have a moment please?


Make file is not complete

The following are not included within the make file
make docker-build
docker-push
kirson@nim-airgapped:/tmp/k8s-nim-operator$ make docker-build docker-push IMG=harbor.runailabs-ps.com/guest-repo/kirson-nim:1.0
make: *** No rule to make target 'docker-build'.  Stop.

Fixing the Readme. Please run make build -f deployments/container/Makefile

Closing this, please re-open if you see other issues.


Optionally use percentage for in/out points

The problem
Using explicit pixel values for in/out points can be very accurate but also very finicky and difficult to determine. Especially with complex scenes.
Proposed solution
There could be an alternative option where I could provide percentages out of the total scroll length from start to finish. For example: I want layer 1 to be active from 10% to 15% of the total scroll length.

On second thought, using percentages could be very confusing. With every new layer, the overall scroll length grows, so the timing of every layer would also grow accordingly. Using pixel values seems like the best approach.
Closing this without a fix.


Corrects a 404 link to an old freighter URL

In the past 30 days, the old URL has received nearly 100 clicks, and it's the only one that did get a 404 page on this docs site. This commit fixes the link on the index page. It's possible there are external links, that we can't control. If that's the case, we may want an nginx redirect. But, that's something we'll have to monitor and fix down the road.

Preview is available here:http://soroban-docs-pr556.previews.kube001.services.stellar-ops.com/


add emacs

This adds the plugin developed by akirak/prism-emacs-lisp for the emacs lisp language. The extension is emacs.
I would like to support also lisp and elisp since both of those names are used, but I'm not sure what would be the best way to do that.
Let me know if there's anything missing to be able to add this 😀

I'll try to post an example soon of this working

You can see some of the highlighting at https://prism-emacs-ogfsnjvwlg.now.sh

@Golmote Thanks for the feedback! I'm having some trouble understanding the tests. I'll upload here with some comments on what I'm having trouble with.
I think the renamings I made will cause the previous comments to appear outdated. I'm adding this here as a reminder that they still need to be addressed

Also note, you may have to add some classes to your theme (CSS) to fully take advantage of it. I have tried it with Dracula theme included in the repo, which is a dedicated effort for the language, but I haven't tested it with any other themes for Prism.

@Golmote I'm the original author of the plugin, and I'm sorry for this tedious work. That code was not meant to be merged into the code base of PrismJS. It should be transpiled using babel, but apparently its resulting code will not be suitable for maintenance by human. Therefore I didn't send a merge request to you.
I am currently working on another project, and I'm not sure if I will adapt this stuff to the proper plugin format. Thank you for your comments.

The name emacs relates to a limitation of a blog engine I am using. My blog source contains code blocks in emacs-lisp language, but my blog engine somehow splits the name by hyphen, and the resulting class name becomes emacs. The plugin was named emacs so that it can highlight Emacs Lisp code in my blog, and I didn't consider supporting other Lisp dialects. Perhaps it should have been first named emacs-lisp, but I would still have to add an alias emacs for use in my blog.

@akirak No worries. I noticed that some patterns are not handled by the official themes. It's not the first component doing so though, and it's ok if some parts are tagged but not styled. I'll just check the appearance once the PR is ready to be merged and maybe we'll make some adjustments.  We might also want to add a comment with a link to your original repo, to credit your original work and in case some users are interested in using the Dracula theme you made.
The transpilation to ES5 should not be that tedious in this case: you're really just using tiny bits of syntactic sugar. Transforming const to var, fat arrows to regular functions, and template strings to good old string concatenation should be pretty straightforward.
I understand the naming limitation you had. Maybe we can still add emacs as an alias, since it seems to be the main scripting language used by this editor.
Thanks for showing up on the PR anyway! I hope @JuanCaicedo can take it to a mergeable state. :)

@Golmote I've made some updates in this PR to address your comments, let me know what you think!

@JuanCaicedo Note that I've just updated the components.js file to fix the code style. You'll have to merge gh-pages and fix the conflicts.

bump @JuanCaicedo :)

@paladox Would love to do some work to get this wrapped up, but I've been busy with some other coding project over the past few weeks. I plan to get to it over the next few days 😃

@Golmote sorry for the delay, I'm working through these changes now 😀

@Golmote I've updated based on your last comments 😀

@JuanCaicedo Yay! :tada:


⚠ Bofrost.es has degraded performance

In f9e8090, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1535 ms

Resolved: Bofrost.es performance has improved in d2995eb.


Italian translation

Preview:

Thank you for your contribution! Unfortunately, at this moment in time, we can not accept this translation as it would be hard to maintain it without internal translation/audit process.


iD crashes when an email field is rendered

URL
https://ideditor.netlify.app/#background=LINZ_NZ_Aerial_Imagery&disable_features=boundaries&id=w267161693&map=19.00/-36.97841/174.78714
How to reproduce the issue?
(this only affects the beta version of iD, not production)

select the hotel building (the "hotel" preset shows the email field by default)
note how the editor gets stuck, and the sidebar no longer responds to changes

Screenshot(s) or anything else?

Which deployed environments do you see the issue in?
Development version at ideditor.netlify.app
What version numbers does this issue effect?
master branch
Which browsers are you seeing this problem on?
Microsoft Edge

taking a look right now

a hotfix is prepared and waiting to be applied on the openstreetmap-website repo


installation problem on Windows 10

I have used node red managed palette menu to install node-red-contrib-xiaomi-ble but I have an error message that seems to indicate that this file is missing: node-pre-gyp
2020-02-05T22:39:53.564Z [err]  WARN Tried to download(404): https://github.com/abandonware/node-bluetooth-hci-socket/releases/download/0.5.3-4/binding-0.5.3-4-node-v72-win32-x64.tar.gz
The complete log file is attached
2020-02-05T22_39_59_037Z-debug.log
any clue on how to fix that?

Could you please redirect this issue to the issue tracker of that library ( https://github.com/abandonware/node-bluetooth-hci-socket ) ?

I have done that and followed their instruction to run npm install --verbose @abandonware/bluetooth-hci-socket which seems to install ok.
bluetooth-hci-socket.log
I could then run npm install node-red-contrib-xiaomi-ble which seems to install without critical error as you could see in the log below.
node-red-contrib-xiaomi-ble.log
But then when I launch node red I could not see the Xiaomi BLE node.
Am I missing something?

Are you sure that xiaomi-ble was installed to the same folder where node-red is?

Ah, sorry, I see that it is in correct place (C:\Users\regis.node-red)
Maybe there are some errors during startup of node-red?


Zoolandia3 cs

Added version 3, removed unused old version2, and clarified instructions in version1

:+1:


eigenvalue of a generalized eigenvalue problem

Hi,
Does anyone know if there is a member function under DenseMatrix that can solve generalized eigenvalue problem, i.e. Av=\lambda Sv where A and S are of class DenseMatrix and they are symmetric positive definite and invertible.
Thanks a lot!
Tom

No, I don't think there is.  Although it wouldn't be difficult to write such a method.  One question would be should this be a member function of DenseMatrix or a "friend" function.

Hello @mlstowell @tzanio ,
Thanks for your reply! Do you mean I can derive the eigenvalue of the generalized system by using seem existing member functions of the class. I tried $S^{-1}Av=\lambda v$ and then using Eigenvalues of the class DenseMatrix.
My matrix A and S are of size 16 by 16, the problem is the eigenvalues I derived using MFEM are different from what I tested in Matlab using the same matrices A and S(do the same thing---just find eigenvalues of S^{-1}A). Do you have any suggestion on the issue? Thank you so much!

Hi, @tomzzh,
No, I did not mean that this could be done with existing members of the DenseMatrix class.  The trouble with S^{-1}A is that this matrix is not symmetric but MFEM's DenseMatrix class assumes the matrix will be symmetric when computing eigenvalues.  Consequently, we should not be surprised that Matlab gives a different result.
I do have some code somewhere that computes generalized eigenvalues using MFEM DenseMatrix objects.  I'll try to find it and paste it into a comment.
Mark

Hello @mlstowell,
Thanks for your code. You mean this code only works for the case A(a in your code) and S(b in your code) are symmetric and there is no other constrain on A and S? I am doing Generalized Multiscale Finite Element and my matrices are systemic for most of cases. I will read and try to test your code. Thanks again.

Hi, @tomzzh,
LAPACK routines are written in Fortran so they typically don't come with C or C++ compatible header files.  To load Fortran functions from C or C++ you need to declare the functions with their C equivalent argument types.  For the DenseMatrix class this is done near the top of densematrix.cpp where you can paste the following.  Note that the implementation that I posted earlier, everything after the first two lines, should be placed in densematrix.cpp someplace after the declaration of dsygv_.
extern "C" {
int   dsygv_  (int *itype, char *jobz, char *uplo, int * n,
double *a, int *lda,
double *b, int *ldb, double *w,
double *work, int *lwork,
int *info);
}

problem solved. Thank you!

Hi, @tomzzh ,
Glad to hear it worked for you.  I also wanted to let you know that we will be incorporating this code into the next release of MFEM (although I'm not sure when that will take place).
Best wishes,
Mark

hello @mlstowell ,
Thanks for letting me know and it will be better if the non symmetric versions of eigenvalue problems will also be included.
Thanks again.


feat: add detailed json output for task list

Summary

added method to extract environment -> feature -> task mapping
build serializable struct of the mapping
add cli entry --json for pixi task ls

Context
In building out the task provider for pixi-vscode to run tasks, I felt that the current output from the task ls was insufficient to provide the user with a good overview of the task through the task runner.
initial implementation with the json output:

Todo
Wanted to get some feedback and add this feature before proceeding with the vscode extension.
Looking for some guidance on if this was the right way to implement this feature, tried to take inspiration from the info --json.
Also wondering if/how the cli commands are tested and some guidance on writing tests for this feature

This looks good to me!
Perhaps it would make sense if we start documenting these JSON outputs into some form of schema as well.

I still think a schema would be very welcome. This would have helped a lot for developers e.g. developing against a JSON API.
Conda was always terrible with their undocumented JSON api..

But in any case, awesome work!


what is the  .ply file ?  What dose it do ? for PnP ?  or ......

.ply file contains the 3D object CAD model. We use it to compute the 3D bounding box of the object. This reference 3D bounding box is then used during PnP.


Segment with ONNX or PT model based on CPU/GPU availability

Checklist
GitHub

[x] I've given this PR a concise, self-descriptive, and meaningful title
[x] I've linked relevant issues in the PR body
[x] I've applied the relevant labels to this PR
[x] I've assigned a reviewer

PR contents

[x] I've consulted ivadomed's internal developer documentation to ensure my contribution is in line with any relevant design decisions
[ ] I've added relevant tests for my contribution
[x] I've updated the relevant documentation for my changes, including argparse descriptions, docstrings, and ReadTheDocs tutorial pages

Description
This PR fixes #961 and adds 2 features:

After training, both the ONNX and PT versions of the model are saved under the model folder. Up until now, when conversion to ONNX was possible, only the ONNX version was saved, otherwise, only the PT version.
When running segment_volume (through the --segment command or API), the version of the model used (ONNX or PT) is based on its availability and on GPU availability. Hence, when GPU is available, PT is used (if available) otherwise ONNX is used. And vice-versa for CPU using ONNX in priority.

Details of the motivation are explained in #961.
In addition for ADS, we chose to package our model repo with both the ONNX and PT version, ex: SEM model.
This PR will allow the segmentation to be done more efficiently on large microscopy images by choosing the ONNX version (optimize for CPU) or the PT version (optimized for GPU) depending on the device.
Tagging @joshuacwnewton because the segment_volume API is also used by SCT.
Linked issues
Fixes #961

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

8 of 14 (57.14%)  changed or added relevant lines in 3 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.03%) to 79.044%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

ivadomed/training.py
4
5
80.0%

ivadomed/models.py
2
4
50.0%

ivadomed/inference.py
2
5
40.0%

Files with Coverage Reduction
New Missed Lines
%

ivadomed/training.py
1
69.14%

ivadomed/main.py
2
55.67%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
4647

Relevant Lines:
5879

💛  - Coveralls


1809

This does not seem to work with Windows 1809 - 17763.194. Error message: "An error occurred while sending data to the Remote Desktop Gateway Server. The server is temporarily unavailable or a network connection is down..."
No issues when connecting from a Windows 1709 build with the same user.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 18fcb539-bdbd-068d-46c0-44058146031e
Version Independent ID: 15e99611-e40e-310c-1625-9e76b27cfe99
Content: Publish Remote Desktop with Azure AD App Proxy
Content Source: articles/active-directory/manage-apps/application-proxy-integrate-with-remote-desktop-services.md
Service: active-directory
GitHub Login: @barbkess
Microsoft Alias: barbkess

@bally-singh
Thanks for your feedback! We will investigate and update as appropriate.

Can you run the 'fix my network' and 'connect to the internet' wizard and try again?


mutate props anti pattern in field arrays example?

In this example, https://redux-form.com/7.4.2/examples/fieldarrays/ why there's a mutate prop like so
<button type="button" onClick={() => fields.push({})}>
Isn't in react we discourage mutation?

There's no mutation going on.
fields prop is a pseudo array. Whenever you call push (or move, or remove, etc), all you are doing is dispatching an action to Redux, which then does its magic.

@gustavohenke thanks for the clarification. One more question, using the arrow function on the onClick won't create a new function on every rerender? Should I move it outside of the render? like
`const fieldRemoveHandler = index => e => this.props.removeField(index)
// ...
`


Duration compare/equality method?

Would this be hard to add? I am thinking something like the isSame() method that standard moments have, but for durations.
Currently I'm just comparing durations in my code by doing
moment.duration().toISOString() === moment.duration().toISOString()
Additionally, if they could be compared like moment.duration() === moment.duration() that would be great

I think isEqual is a reasonable feature request. There are some caveats, though: I think for example, that moment.duration({ hours: 24 }) and moment.duration({ days: 1 }) should not be equal, and if the caller wants that they should compare valueOf() values. In other words, I think it should compare the durations partwise, just like your toISOString thing does.

I don't know if this is possible, but perhaps we could use the == operator for non-strict comparisons (eg moment.duration({ hours: 24 }) == moment.duration({ days: 1 }) would be true) and === for the same condition would be false. However this might not logically work when you go to compare something like moment.duration({ days: 30 }) == moment.duration({ months: 1 }) because the number of days in a month varies...

Just to further complicate things, it would also be great if other comparison's could be supported (say, moment.duration() > moment.duration()) - My assumption would be that it would take a stepwise approach and compare each unit of time, starting at the largest.
For example, moment.duration({ hours: 24 }) > moment.duration({ days: 1 }) would evaluate to false because { days : 1 } is larger than { hours: 24 } (which is essentially days: 0, hours: 24 )
But perhaps this might be too confusing and result in unexpected behavior for people who don't read documentation before using it..

Perhaps the only sane, sensible approach with expected results is to just compare the raw millisecond value for both equality and gt lt comparisons?
moment.duration({ days: 1}).asMilliseconds() === moment.duration({ hours: 24 }).asMilliseconds() true
moment.duration({ days: 1}).asMilliseconds() > moment.duration({ hours: 23 }).asMilliseconds() true
moment.duration({ days: 1}).asMilliseconds() > moment.duration({ hours: 24 }).asMilliseconds() false
moment.duration({ days: 1}).asMilliseconds() > moment.duration({ hours: 25 }).asMilliseconds() false
moment.duration({ months: 1}).asMilliseconds() > moment.duration({ days: 30 }).asMilliseconds() false (according to the duration docs, length of a duration in months is defined as 30 days, so this would be equal)
Thoughts?

I'm hesitant to add this to the codebase, since (as you mentioned) one can compare the millisecond values. I can budge if you have a strong argument for it.
We do have isSame/isBefore/isAfter/isSameOrBefore/isSameOrAfter for moments, but I don't exactly love those either.

RIght, if all we're doing is comparing milliseconds, then you can just do
+someDuration === +someOtherDuration

I think the only real case for isEqual is checking that both durations have the same units with the same values.

Even so - I'm not sure that comparing durations with the same units is useful, since duration, i.e. "difference between two timestamps", feels the same in any unit.

Well, whether it's genuinely useful or not is a fair question, and I'm OK on closing. But I firmly disagree with this:

feels the same in any unit

Moment's imprecise higher order units make this untrue (and should!). An illustration:
var dur1 = moment.duration({months: 1});
var dur2 = moment.duration({milliseconds: dur1.asMilliseconds()});
+dur1 === +dur2  //=> true
var now = moment();
now.toString();   //=> "Fri Dec 29 2017 16:38:49 GMT-0500"
now.add(dur1).toString(); //=> "Mon Jan 29 2018 16:38:49 GMT-0500"
now.add(dur2).toString();  //=> "Wed Feb 28 2018 16:38:49 GMT-0500"

In other words, those durations have totally different behaviors, despite corresponding to the same number of milliseconds. If you want to ask "are these objects the same?", you need to include everything that affects their behavior (i.e. equivalent objects can be used interchangeably). Since that's unit-dependent, the check would have to be too.


Support to render a GitHub users to be a link

close #18

Codecov Report

Merging #23 (1573855) into master (31ffcfe) will increase coverage by 14.52%.
The diff coverage is 98.46%.

@@             Coverage Diff             @@
##           master      #23       +/-   ##
===========================================
+ Coverage   31.38%   45.90%   +14.52%
===========================================
  Files           1        2        +1
  Lines         188      220       +32
===========================================
+ Hits           59      101       +42
+ Misses        121      116        -5
+ Partials        8        3        -5

Flag
Coverage Δ

unittests
45.90% <98.46%> (+14.52%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

main.go
23.71% <0.00%> (-7.67%)
:arrow_down:

function/github.go
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31ffcfe...1573855. Read the comment docs.


Meissa Runner Aborts Runs if Tests Are Retried

2018-03-12T10:54:28.4677043Z Test agent with tag Agent1 finished tests execution on machine Machine1.
2018-03-12T10:54:28.4677043Z Test agent with tag Agent1 starts waiting for new jobs on machine Machine1.
2018-03-12T10:55:18.7664325Z UPDATE Test Run on machine Machine1 3/12/2018 10:55:18 AM
2018-03-12T10:55:25.9193674Z Abort Test Agent Run because test runner lost connection with test agent on machine: Machine1 3/12/2018 10:53:23 AM 3/12/2018 10:53:25 AM current status InProgress
2018-03-12T10:56:18.8316025Z ### All Test Agent Runs Finished! ###

Fixed in the latest version


Add a waffle flag to guard the unlisted addons submissions (bug 1161536)

Fixes bug 1161536

This looks good to me. r+


Incorrect comments in setup.py

Description
Documented URLs are wrong for TPU in setup.py, please see: https://github.com/google/jax/blob/1bcf8d646dfa9f5c87a42280b37bb125c46d447f/setup.py#L86 as an example.
This should reference the URL documented in: https://github.com/google/jax#pip-installation-google-cloud-tpu
What jax/jaxlib version are you using?
JAX 0.3.17
Which accelerator(s) are you using?
TPU
Additional system info
Python 3.8
NVIDIA GPU info
No response

https://github.com/google/jax/commit/8d8643664c9fa9a0ab3ac2eaff0d9d003cbf8852 fixed this.

Thanks mate, have a good day!