BUG: CUDA_VERSION is set to 0 in sdist

I was investigating the bug reported in https://github.com/conda-forge/cupy-feedstock/issues/98, and it turns out the reason is on Conda-Forge we built CuPy from sdist, but the source has CUDA_VERSION inlined to 0:
// cupy_backends/cuda/api/driver.cpp

/* ... */

static PyObject *__pyx_pf_13cupy_backends_4cuda_3api_6driver_2get_build_version(CYTHON_UNUSED PyObject *__pyx_self) {
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("get_build_version", 0);

  /* "cupy_backends/cuda/api/driver.pyx":141
 *
 * def get_build_version():
 *     return CUDA_VERSION             # <<<<<<<<<<<<<<
 *
 *
 */
  __Pyx_XDECREF(__pyx_r);
  __Pyx_INCREF(__pyx_int_0);
  __pyx_r = __pyx_int_0;
  goto __pyx_L0;

  /* "cupy_backends/cuda/api/driver.pyx":140
 * ###############################################################################
 *
 * def get_build_version():             # <<<<<<<<<<<<<<
 *     return CUDA_VERSION
 *
 */

  /* function exit code */
  __pyx_L0:;
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

/* ... */

static CYTHON_SMALL_CODE int __Pyx_InitGlobals(void) {
  if (__Pyx_InitStrings(__pyx_string_tab) < 0) __PYX_ERR(0, 1, __pyx_L1_error);
  __pyx_int_0 = PyInt_FromLong(0); if (unlikely(!__pyx_int_0)) __PYX_ERR(0, 1, __pyx_L1_error)
  return 0;
  __pyx_L1_error:;
  return -1;
}

I don't have a good solution or patch for now, as it seems to imply sdist should be generated per CUDA version. Perhaps I am overlooking some obvious fix?
cc: @jakirkham @kmaehashi

Hmm... I don't come up with the good solution either other than installing the Cython during the installation of CuPy so that CPP code is generated in the user envirnoment.
I think the same can be said for use_hip compile time env. (sdist should be buildable on HIP but I guess currently not)

Yeah, in #4556 I am removing use_hip and making HIP_VERSION propagated together with CUDA_VERSION, but it wouldn't solve the problem I think. On Conda-Forge we have no problem switching to the git source, so this issue is more for awareness.

Discussed a bit in the team, adding Cython to setup_requires in setup.py and exclude cpp files from sdist sounds good to us. This way we can also resolve https://github.com/cupy/cupy/pull/4494 without dirty hacking.

It's great to solve two bugs at once 👍


[Docs generator] Extended external classes/structs/modules do not get documented

In my shard, I extend a struct from another library by adding some convenience methods. I defined an alias in my namespace to the struct in the external namespace.
Now, crystal docs just generates a page with the alias definition (which contains an invalid link to the external struct) without bothering to create a page with the new methods.
This, however, only happens when I extend a class/struct/module that a library already defined (see notes)
Steps to reproduce:

File structure:

├── lib
│   └── a
│       └── src
│           └── a.cr
└── src
    └── main.cr

File contents

# lib/a/src/a.cr
module A
  # A very useful struct
  struct Test
    def initialize(@prop : Int32); end
  end
end

# src/main.cr
require "a"

module B
  alias Test = A::Test
  struct A::Test
    # Multiplies @prop with the given factor
    def multiply(factor)
      @prop * factor
    end
  end
end

test = B::Test.new(3)
puts test.multiply(5)

Generate docs: crystal docs --project-name Test --project-version 1.0.0
Result:

Expectation
My expectation was that this would generate another page under the A module in which the added methods were described.
Notes

When the alias is removed, no page is generated at all
When the external struct from module A is extended at the toplevel (not inside module B), the result is the same
When no library defines module A, the result is as expected

Your code shouldn't even compile.
For some reason, module B; struct A::Test; reopens ::A::Test. That's wrong. It should actually create a new type B::A::Test because the struct definition is in the B namespace.
I suppose the docs generator might be expecting this and that's the reason why it doesn't even show A::Test. It would be documented if you remove the B namespace.

@straight-shoota You are right, that is, however, another issue I was not even going for ^^ (but the code above compiles, btw i don't know why though).
I did a little research on my end:
It seems that my issue has a different root cause. For some reason, if I extend the external struct like this:
module A
  struct Test
    def method(); end
  end
end

it is documented as expected.
If I write the short form like this:
struct A::Test
  def method(); end
end

it is not documented.
If I declare a new struct B::Test, everything works just fine.
Somehow, the short form is treated differently than the verbose one.

Another observation:
When I include a module A; end somewhere in the project, everything works as intended, even when using the short form.

This issue does not seem to fit the discription above anymore, should I edit it or open a new one?


Prius steer rate factor is wrong

The rate factor in the DBC was commented "TBD", so I went ahead and calculated it for you with my good sensor.
Stock OP:

This PR:

@pd0wm Not sure if this would screw up your INDI Kalman Filter, but the plus side is - I'm now uploading accurate rate data from this point on. Not sure if it'd help you refine the actuator model:
BO_ 35 SECONDARY_STEER_ANGLE: 8 XXX
  SG_ ZORRO_STEER : 7|32@0- (0.004901594652,0) [-500|500] "" XXX
  SG_ ZORRO_RATE : 39|24@0- (0.004901594652,0) [-500|500] "" XXX

https://github.com/zorrobyte/openpilot/blob/devel_ZSS/opendbc/toyota_prius_2017_pt_generated.dbc

Hey! INDI works just fine!

Yeah, seriously, my INDI ZSS branches have been a dream for the last couple weeks, but I will definitely check out your updates.

The existing steer rate value seemed correct when I've dashboarded it, albeit "noisy".  I'm also not aware of any steer rate sensors that that have such a high resolution, since 1 degree per second is really slow.

@Gernby not sure how carstate handles it, but at least in Cabana, there's no way I'm steering -350 degrees a second. Unit deg/s

@zorrobyte , not sure what you were doing, but 3.5 deg/s seems indeed very small. An alternative way to get a proper factor is to take the raw derivative of the angle and compare it with STEER_RATE

After thinking about it, I agree that my rate is off. Thank you for the review, I should calculate an actual rate from Cabana using timestamps, instead of just scaling it by the same factor as my angle reading.


[win-x64] Support for 'Process.GetCurrentProcess().Handle'

Both
System.Diagnostics.Process.GetCurrentProcess().Handle
and
System.Diagnostics.Process.GetCurrentProcess().SafeHandle.DangerousGetHandle();
throw
System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property. ---> System.Exception: Method '[System.Diagnostics.Process]Interop+Advapi32.AdjustTokenPrivileges(SafeTokenHandle,bool,TokenPrivileges,int32,native int,native int)' requires non-trivial marshalling that is not yet supported by this compiler.
   at Interop.Advapi32.AdjustTokenPrivileges(SafeTokenHandle, Boolean, Interop.Advapi32.TokenPrivileges, Int32, IntPtr, IntPtr) + 0x23
   at ht.win32!<BaseAddress>+0x45cfd
   at ht.win32!<BaseAddress>+0x128b54

   --- End of inner exception stack trace ---
   at ht.win32!<BaseAddress>+0x128c08
   at ht.win32!<BaseAddress>+0x128a4c
   at System.Diagnostics.ProcessManager.OpenProcess(Int32, Int32, Boolean) + 0x28
   at System.Diagnostics.Process.GetProcessHandle(Int32, Boolean) + 0x18c
   at System.Diagnostics.Process.OpenProcessHandle() + 0x2a

Is there any alternative for getting a handle to the running process?

This was fixed by https://github.com/dotnet/corefx/pull/30539. The next CoreRT NuGet package should have a fix.
Thank you for reporting this!

Thats very fast, thanks 👍


Please add an option to enable TCP_NODELAY (+TCP_QUICKACK)

TCP_NODELAY is a socket setting that disables Nagle's algorithm and allows packets to be sent across the network individually (without queuing ACKs one by one). Together with applying TCP_QUICKACK, each of packets can be ack-ed immediately. This is especially useful for links with high packet loss rate, because of the smaller transmission granularity.

Does this achieve similar effect to ServerSpeeder and BBR?
I'd very much like to see such improvement which lets me ditch ServerSpeeder on Windows since there is no other BBR equivalent for Windows, upload speed without such implementation is hindered too much by high latency connections.

Does this achieve similar effect to ServerSpeeder and BBR?
I'd very much like to see such improvement which lets me ditch ServerSpeeder on Windows since there is no other BBR equivalent for Windows, upload speed without such implementation is hindered too much by high latency connections.

they are totally different and has no direct connection.


Language extension mapping doesn't seem to work

We have all our React files as .js. However, this doesn't seem to work:
jscpd -p src/ --languages jsx --languages-exts jsx:js

If we write this, it runs jscpd, but using javascript vs jsx:
jscpd -p src/ --languages javascript

thank you for the report, will check

language mapping implemented in the alpha version of jscpd.
You can try to install npm i<EMAIL_ADDRESS>-g and use right mapping.


Create 0658-find-k-closest-elements.java

File(s) Modified: _0658-find-k-closest-elements.java
Language(s) Used:  java
Submission URL: https://leetcode.com/problems/find-k-closest-elements/submissions/1186375336/
https://leetcode.com/problems/find-k-closest-elements/submissions/1186418707/

Hi, it seems there was a PR that already merged the solution, that resembles the solution provided in the official video. Thus, let me close this PR for now. However, if you have any solutions to other questions, I'm happy to review. Sorry again for the late catch up.


Detect violations of owl:complementOf in OWLClassTypeRule

It should be detected as a modeling violations that we assign ex:idv1 to both ex:clsA and owl:complementOf(ex:clsA)

Feature added with checkin 7e44bbd


Firebase storage stops working after "Unable to create StorageReference" error

Please fill in the following fields:
Unity editor version: 2019.3.6f1
Firebase Unity SDK version: 6.13.0
Source you installed the SDK (.unitypackage or Unity Package Manager):unitypackage
Firebase plugins in use (Auth, Database, etc.):Auth, Database, Storage
Additional SDKs you are using (Facebook, AdMob, etc.): none
Platform you are using the Unity editor on (Mac, Windows, or Linux): Windows
Platform you are targeting (iOS, Android, and/or desktop): Android (eventually)
Scripting Runtime (Mono, and/or IL2CPP):/
Please describe the issue here:
I noticed that if I apply a wrong URL to StorageReference, that gives me "Unable to create StorageReference" error, that's fine, but then I can't connect to the storage anymore, even if I type a correct StorageReference afterward (that gives no error), seems like I need to reset/clear the storage reference somehow.
Here is my code, if I remove the line with the arrow it works perfectly if I leave it as is the code is stuck at
GetBytesAsync and never continues, no catch, no fail.
What I tried-  remove the await, using 2 different storageRefernces, resetting storage to DefaultInstance
My code is very basic-
    string wrongurl= "Not a good url"
    string goodurl ="gs://...."
    storage = FirebaseStorage.DefaultInstance;
    const long maxAllowedSize = 1 * 1024 * 1024;

     try{
    ===> storageRef =storage.GetReferenceFromUrl(wrongurl);}
    catch{}  //always caught because wrong url

    try{storageRef =storage.GetReferenceFromUrl(goodurl);}
    catch{}  //no catch because good

    try{
    var task2 = await storageRef.GetBytesAsync(maxAllowedSize).ContinueWith(t => t);

    if (task2.IsFaulted || task2.IsCanceled)
        {
              Debug.Log(task2.Exception.ToString());
            // Uh-oh, an error occurred!
        }
        else
        {}
    }
    catch{} // no catch

Please answer the following, if applicable:
Have you been able to reproduce this issue with just the Firebase Unity quickstarts (this GitHub project)? Yes
What's the issue repro rate? (eg 100%, 1/5 etc)
5
Fir

Hi shlighter!
I'm having a hard time reproducing your bug in the Unity Quickstart by doing the following:
I modified the GetStorageReference function a little bit to ensure that I'm always getting the default instance rather than creating a new FirebaseStorage for every call:
protected StorageReference GetStorageReference()
{
  // If this is an absolute path including a bucket create a storage instance.
  if (storageLocation.StartsWith("gs://") ||
      storageLocation.StartsWith("http://") ||
      storageLocation.StartsWith("https://")) {
    var storageUri = new Uri(storageLocation);
    var firebaseStorage = FirebaseStorage.DefaultInstance;
    // var firebaseStorage = FirebaseStorage.GetInstance(
      // String.Format("{0}://{1}", storageUri.Scheme, storageUri.Host));
    return firebaseStorage.GetReferenceFromUrl(storageLocation);
  }
  // When using relative paths use the default storage instance which uses the bucket supplied
  // on creation of FirebaseApp.
  return FirebaseStorage.DefaultInstance.GetReference(storageLocation);
}

I'd first add some random garbage to the "Storage Location" field in the UI to make sure I was uploading to a bad place then clicked "Upload".
Without relaunching, I'd fix my url and click "Upload" again.
The second time always works both in the Editor (MacOS) and on Android.
Can you confirm whether or not repeating these steps causes the bug to occur? What if you update to the latest (6.14) SDK?
Thanks,
--Patrick


Extend Power without statsmodels

The current implementation of power does not allow extending power beyond the available counts without using statsmodels.
I've found a more reliable, light way way to extend power beyond the available values without relying on statsmodels. It removes some of the issues I saw in Statsmodels, and worked well for a set of known distributions.
I know we're in beta, but is this a feature worth adding?

Is it possible to make these improvements in statsmodels itself? That way the statsmodels and scikit-bio communities can both benefit from these changes.

AFAIK, Statsmodels is not a dependency of scikit-bio, and so leaving them out means not including the ability to extrapolate power beyond the effect size being presented. Id like to add that functionality, if not through this, then by wrapping statsmodels.
I'm not sure Statsmodels would want this implementation, because it goes a different direction that what they've used. The approach there has been to use an equation solver within the class. I've chose to solve the equation explicitly for exactly what I need, and generate a result. While it eventually falls prey to the same limitations as equation solving, I get fewer spurrious results. The difference in implementation makes sense, given that theirs is a general tool and mine is about 60 lines of code (tests included) to target a very specific function.
I'd also prefer to go this route because I have clear models of this behavior

@JWDebelius, sorry for the slow reply on this. I'm a bit confused about how the power code is supposed to be applied, and understanding that would help figure out whether this makes sense to add to scikit-bio. Would you be willing to put a quick example together for the scikit-bio cookbook?

I would be happy to do that. I have a couple of pretty rough first-pass
notebooks that I've been working with for my paper/dissertation, and are
about 95% code right now, or I can put together a new notebook using the 88
soils dataset. The former is probably quicker, there later is probably
better long term.
On Tue, Sep 22, 2015 at 5:00 PM Greg Caporaso<EMAIL_ADDRESS>wrote:

@JWDebelius https://github.com/JWDebelius, sorry for the slow reply on
this. I'm a bit confused about how the power code is supposed to be
applied, and understanding that would help figure out whether this makes
sense to add to scikit-bio. Would you be willing to put a quick example
together for the scikit-bio cookbook?
—
Reply to this email directly or view it on GitHub
https://github.com/biocore/scikit-bio/issues/1079#issuecomment-142456780
.

Whichever is quicker should be fine, thank you!
Sent from my phone, please excuse typos and brevity.
On Sep 22, 2015 7:44 PM, "J W Debelius"<EMAIL_ADDRESS>wrote:

I would be happy to do that. I have a couple of pretty rough first-pass
notebooks that I've been working with for my paper/dissertation, and are
about 95% code right now, or I can put together a new notebook using the 88
soils dataset. The former is probably quicker, there later is probably
better long term.
On Tue, Sep 22, 2015 at 5:00 PM Greg Caporaso<EMAIL_ADDRESS>wrote:

@JWDebelius https://github.com/JWDebelius, sorry for the slow reply on
this. I'm a bit confused about how the power code is supposed to be
applied, and understanding that would help figure out whether this makes
sense to add to scikit-bio. Would you be willing to put a quick example
together for the scikit-bio cookbook?
—
Reply to this email directly or view it on GitHub
<
https://github.com/biocore/scikit-bio/issues/1079#issuecomment-142456780>
.

—
Reply to this email directly or view it on GitHub
https://github.com/biocore/scikit-bio/issues/1079#issuecomment-142474962
.


Adjust heigt to parent box

Description
I would like to have the datagrid to adjust its heigt based on the parent box is it in and then presents a acrollbar in the table body like here in my mui example:

Currently, if the box size is smaller than the datagrid size, it just cuts off:

implemented in #82
use height property to customize table height, with 100% it will fit parent container.


Capturing images from Web Cam

Fixes #371

@petele Whilst I would love to get it into for the launch, there is no rush on this one.


updater: missed students

Students 015005184, 015005155, 015008424 not saved in Oodikone db

Students are not among all_student_numbers.txt, so the script needs to be rerun


Update Member Access code sample

Add more accurate and specified code

@baruchiro : Thanks for your contribution! The author, @mikeblome, has been notified to review your proposed change.


Pipeline applyyaml CRD

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):

Install kube-prometheus operator in cluster
Create servicemonitor.yaml in git repository:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: app-metrics
spec:
selector:
matchLabels:
app: myapp
endpoints:

port: metrics-port

Create .rancher.pipeline.yml in git repository:
stages:

name: Deploy
steps:

applyYamlConfig:
path: ./servicemonitor.yaml
timeout: 60
notification: {}

Setup and execute pipeline

Result:
09:53:51 from server for: “./servicemonitor.yaml”: servicemonitors.monitoring.coreos.com “app-metrics” is forbidden: User “system:serviceaccount:p-9h7jb-pipeline:jenkins” cannot get servicemonitors.monitoring.coreos.com in the namespace “p-9h7jb-pipeline”
Other details that may be helpful:
Running from command line works fine:
$ kubectl --as=jenkins -n p-9h7jb-pipeline apply -f servicemonitor.yaml
servicemonitor.monitoring.coreos.com/app-metrics created
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher 2.1.5, rancher 2.2.2
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): vSphere, RancherOS
Kubernetes version (use kubectl version): 1.11.5
Docker version (use docker version): 17.3.2

The build pods are using jenkins serviceaccount in the pipeline dedicated namespace. It has edit permission inside the project. That's a clusterrole for most native resources generated by kubernetes but it doesn't include CRD resources.
If you want to deploy stuff beyond the default scope, you can grant additional permissions to the jenkins service account, for example:
kubectl create clusterrolebinding --clusterrole=cluster-owner --serviceaccount=p-sxvhk-pipeline:jenkins custom-crb

NB: The above is giving * permission for * resources to the SA. You probably want to use fine-grained roles/rolebindings according to your needs.

We probably don't want to grant * permissions to the SA by default. Privilege escalation happens where project-members can access roles/rolebindings in the project although they should not.

Thanks for suggesting. Granting permissions (less that cluster-owner) works.


Record when a subgraph was created

The deployment_schemas table should have a column created_at that records when a deployment was created. The information could be gleaned from subgraph_version, but that's a bit too cumbersome, and longer term, subgraph_version should go away.
The created_at timestamp should also be copied to unused_deployments when a deployment is recorded as unused.

@lutter could we pick this up alongside tracking graph-node version for deployments? cc @tilacog

Yeah, I think so; it should be a fairly straightforward database migration, and might not require any code changes since we can just set the default for the new column to now()

@aodhgan this is referencing storage on the graph-node in the database, but your request is interesting and something that we are investigating (better support for deployment & management of multiple subgraphs from the cli) - I do not think what you describe is currently surfaced.

@aodhgan Until we have better tooling ready, you can query the management table in your postgresql by yourself:
SELECT * FROM subgraphs.subgraph
If you then want to remove a subgraph, simply run:
graph remove --node http://localhost:8020/ [username/subgraphname]


[FLINK-23025][upsert-kafka] Fix upsert-kafka produce duplicates when enable object reuse

What is the purpose of the change
Fix upsert-kafka produce duplicates when enable object reuse.
Verifying this change

Add tests in BufferedUpsertSinkFunctionTest to verify the results  if enable object reuse.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

The failed test is about elaseticserach...

@flinkbot run azure


Option 4 broken

Option 4 to list payment history does not return anything anymore since last release.

is this still open?

No. It is fine now. Thats why i closed it.


add default delay in requestPairingCode to avoid errors

sometimes its necessary to set a delay to avoid error requesting a pairing code when connection was not estabilished yet, the delay can be changed or removed

Here I already do this in the implementation of the function, I don't believe that this delay is the responsibility of the lib

Here I already do this in the implementation of the function, I don't believe that this delay is the responsibility of the lib

There is no information in the documentation that it is necessary to use delay to request the code, new users will face problems and will not find the solution, opening unnecessary issues, at least it could include this in the documentation, you implemented this because you had problems that could have been avoided


Option to disable line numbers in search editor

When I get my search results to show in a search editor, most of the time I copy it to reuse in other part of my code base. If the line number can be hidden (which don't offer much use anyway), this scenario can be improved.
The latest version of VS Code I've confirmed to not have this feature
Version: 1.44.0-insider
Commit: efd7548df65de8e775015b67863190e6bd4934fc
Date: 2020-03-13T06:01:51.669Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 18.7.0

Just a partial workaround in the meantime: the column selection mode might work for you in many instances.

which don't offer much use anyway

In fact the line number is the means by which we link lines to their locations in the source text, so it must be present.
An extension however could implement a command to filter out the line numbers on copy/paste. Or use column selection as noted above.

@JacksonKearl I don't see any practical use for including the line:column prefix when someone does a right click copy on search results.


Error when building on Solaris 10 sparc

Hello, when building the source code on Solaris 10 (sparc) I get some errors related to inline functions. Any hint to fix this?
bash-3.2# uname -a
SunOS martin 5.10 Generic_147147-26 sun4v sparc sun4v
bash-3.2# CC -V
CC: Sun C++ 5.13 SunOS_sparc 2014/10/20
bash-3.2# python setup.py build
sh: git: not found
WARNING: git describe failed with: 256
WARNING: Unable to determine version. Using <IP_ADDRESS>
running build
running build_ext
building 'pyodbc' extension
/usr/lib/python2.6/pycc -DNDEBUG -KPIC -DPYODBC_VERSION=3.0.0-unsupported -DPYODBC_UNICODE_WIDTH=2 -I/opt/DataAccess64/ODBC/dm/include -I/usr/include/python2.6 -c /tmp/pyodbc-master/src/buffer.cpp -o build/temp.solaris-2.10-sun4v-2.6/tmp/pyodbc-master/src/buffer.o -g0
"/tmp/pyodbc-master/src/pyodbc.h", line 69: Error: "inline" is not allowed here.
"/tmp/pyodbc-master/src/pyodbc.h", line 69: Error: DWORD is not defined.
"/tmp/pyodbc-master/src/pyodbc.h", line 69: Error: grf is not defined.
"/tmp/pyodbc-master/src/pyodbc.h", line 69: Error: Badly formed expression.
4 Error(s) detected.
error: command '/usr/lib/python2.6/pycc' failed with exit status 2

I'm not sure about inline - it is C++ code and inline has been around a long time.
I can see how DWORD is going to be a problem.  It looks like it can be swapped out for 'unsigned int', though.  I'm only using it for some internal flags and not the ODBC interface.  Do you want to try that?  I don't have any Solaris boxes available at this time.

Closing due to inactivity.


Memory issue when training 1024 resolution

I'm trying to train a 1024x1024 database on a V100 GPU.
I tried both the tensorflow version and the pytorch version.
Despite setting batch-gpu to 1, the tensorflow version always run out of system RAM(after the first tick, system ram total 51gb), and
the pytorch version alway run out of cuda memory(before the first tick).
Here are my training settings:
python run_network.py --train --metrics 'none' --gpus 0 --batch-gpu 1 --resolution 1024 \
 --ganformer-default --expname art1 --dataset 1024art

Also, I always encounter the warning:
tcmalloc: large alloc

Did dome investigation on the tensorflow model, turns out the problem occurs when saving snapshot images. There is probably some kind of memory leak when saving large images in visualize. It is fine when not saving images.
Will investigate further later

Hi, thanks for reaching out! I noticed indeed that the visualization takes a lot of RAM but haven't yet tracked the issue since it's not a stateful module and so I'm not sure where specifically it could lead to a memory leak. However, I think the issue is that when it tries to make a visualization it holds in a memory 28 model outputs at the same time (including stacks of attention maps), so reducing the grid size of saves images here:
https://github.com/dorarad/gansformer/blob/main/training/misc.py#L306
could mitigate the issue.
I'll be making couple changes so that memory consumption is reduced by default, and looking forward to hearing if you find by any chance anything further!

Btw, I'm currently running fine with only saving output images and not saving attention maps

I'll update the default options in accordance with that so the people won't get memory issues. Thank you for the openning this issue!


Performance and visual issues with deferred shading on Safari

After the move to deferred shading, I'm seeing some visual issues with the lisp on Safari, as well as really slow performance.
lips_2.hom.zip

This doesn't seem to be a problem on Chrome.

To be clear this is a static 3d image which does not use deferred shading, so it must be an indirect consequence of the change.

All rendering is done through the deferred pipeline now

Sure, but I mean those cylinders in the 3d image are genuine geometrical
cylinders, not produced by the kernel method (right?).
On Wed, Jan 19, 2022 at 5:52 PM Nathan Corbyn @.***>
wrote:

All rendering is done through the deferred pipeline now
—
Reply to this email directly, view it on GitHub
https://github.com/homotopy-io/homotopy-rs/issues/341#issuecomment-1016717750,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACQ4OHXOO5LP633RBYIJFEDUW327LANCNFSM5MKLQYPA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID:
@.***>

Ah, yes, I see what you mean - that is the case


Create easier access to selecting saved filters

Creates easier access to selecting web filters by moving saved filters
out of the filter form to a seperate popup. Now clicking the saved
filter applies the filter (and perfomes the page reload).
Signed-off-by: Abishek V Ashok<EMAIL_ADDRESS>Fixes: https://github.com/MLH-Fellowship/pyre-check/issues/30

Screenshots:

I am not totally convinced about this UI. It's a bit confusing the we now have 2 different forms for the filter and they can overlap like in this picture:

Moreover selecting a saved filter would still require 3 clicks instead of 4 which is not a great improvement.
I can imagine 2 approaches:

Keep the Filters like it is now and once clicking on a Saved Filter automatically apply it. This would reduce the interaction from 4 -> 3
Have a sidebar which shows the filter and allows to quickly select them. Interaction would be just one click (internally we have something similar)

cc @dkgi if he has opinions on the UI design here

+1 to @r0rshark's sidebar idea

Personally think what we have internally is pretty terrible and it be great if we could figure out how to do better here that'd be great. Maybe this could be a side-panel with three sections:

"Create": what we currently have hidden in the button and would be collapsed by default
"Search": the search for existing ones as shown in this diff
"Recent": a list of filters the user has looked up recently (maybe stored in the browser storage?)

@r0rshark @gbleaney @0xedward @dkgi
Here you go, sidebar as everyone suggested :)
https://user-images.githubusercontent.com/8947010/126631543-37aaa2f5-32ed-4283-b1e0-a165cda15383.mov

The filter wasn't removed from recent filters on delete, sorry I messed that up, but fixed now :)

@dkgi what do you think about this approach? Is it close enough to your ideas? :)

@abishekvashok this looks pretty awesome! Couple of things I notice:

I think the current version might be a bit confusing to users since the users attention has to move across the full screen when selecting a filter to apply it. I'm wondering if we could move the whole content of the filter dialog into the sidebar (or would that be to small?).
I think it's visually confusing that the Runs / Run 8 and the issues don't align anymore. Maybe if we move the sidebar to the right?

@dkgi thank you for your time and valuable pointers.
yep, suggestion 2 definitely makes sense. But I am not sure if we have enough room for the entire filter form in the side panel, especially since it can introduce multiple levels of collapses. But I am not that good at UI, so I'll let you pick :)
suggestion 1 and 2:

suggestion 2 only:

Let me know which of the above looks good to you - or even something other than this  - and I'll push the commits as needed :)

Don't have strong opinions here. Option 2 does look a bit squeezed. I could also see the list of recent filters becoming too long and hiding the filter configuration at the bottom. Maybe we could have the filter options first, in a separate card, that is collapsible and collapsed by default? If the size is an issue we could also just move the filter button down and make it the same width as the card with the recent filters to make this visually more coherent.
Curious to hear what others think though.

If the size is an issue we could also just move the filter button down and make it the same width as the card with the recent filters to make this visually more coherent.

Nice suggestion @dkgi ! I tried this because making filters as a collapsible card would look like the same collapsed.
How does this look?

CC: @r0rshark @0xedward @gbleaney

We might want to tweak this a bit in follow-ups but generally this looks reasonable to me. Maybe change "Filter" to "Filter..." to make the pop-out a bit less surprising. Curious to hear what others think.

Done

The UI looks good to me but I am having trouble testing this locally
git fetch origin pull/27/head:easier_filters
git checkout easier_filters
npm install
npm run-script build

shows an Error:
  Line 894:48:  React Hook "useMutation" is called conditionally. React Hooks must be called in the exact same order in every component render. Did you accidentally call a React Hook after an early return?  react-hooks/rules-of-hooks

Which I think is caused by moving the deleteFilterMutation from the FilterModal to the FilterControls. @abishekvashok  can you have a look at this?

@r0rshark it was due to the position where we check if we have a filter fetching error, it should be after the mutation.
We moved the mutation and associated objects out of the model to FilterControls because it's part of the sidebar and not the model anymore :)
I actually used modal.error there and after testing I reviewed the diff to double check and saw a model and alert, so changed to alert to make everything uniform but didn't test - my mistake! Fixed it now.

Maybe I am missing something but trying this
git fetch origin pull/27/head:easier_filters2
git checkout easier_filters2
npm install
npm run-script build

the page doesn't load and the console shows:
react-dom.production.min.js:209 TypeError: Cannot read property 'indexOf' of null
    at SubMenu.js:591
    at new i (connect.js:52)

@abishekvashok Can you reproduce?

Hey, I couldn't reproduce the issue at first- but when I tried deleting the recent filters I got it, and it took me a bit of time to figure out why and it was due to setting defaultOpenKeys (conditionally) to [] when we don't have a recent filter, but yeah, figured another way todo it and it works now - sorry for back and forth but it should be good for testing.

Fixed merge conflicts :)

Nice looks good to me. Only minor nit can we keep the "Recent Filter" always in the same order (maybe alphabetical) it's a bit confusing they keep changing position after having clicked on them

Hey @r0rshark done! Now it doesn't move to top. I didn't do the alphabetical one, because then the filter that is first in the list in the db when used will always remain on top! Nice suggestion btw, now UX looks cooler :)


Weight File for PASCALVOC

I can find the Cityscapes and COCO weights, but couldn't find any PASCALVOC weight file. As I reproduce the same code without modifications of yours, also checked the config file (as below), there was a slight gap of 2-3%. I got 76.91 on 732 split, which yours were 79.9. Can you upload any weight file for PASCALVOC? Thank you. Really appreciate your works by the way!
[2023-04-10 19:41:31,486][    INFO] {'backbone': 'resnet101',
'batch_size': 2,
'conf_thresh': 0.95,
'config': 'configs/pascal.yaml',
'criterion': {'kwargs': {'ignore_index': 255}, 'name': 'CELoss'},
'crop_size': 321,
'data_root': '/data2/ksy/PASCALVOC2012/',
'dataset': 'pascal',
'dilations': [6, 12, 18],
'epochs': 80,
'labeled_id_path': 'splits/pascal/732/labeled.txt',
'local_rank': 0,
'lr': 0.001,
'lr_multi': 10.0,
'model': 'deeplabv3plus',
'nclass': 21,
'ngpus': 2,
'port': 20024,
'replace_stride_with_dilation': [False, False, True],
'save_path': 'exp/pascal/unimatch/r101/732',
'unlabeled_id_path': 'splits/pascal/732/unlabeled.txt'}

You should maintain a total batch size of 8. However, your current total batch size is only 4 (2 imgs / GPU x 2 GPUs).


Fix google collab link

Fixes broken link on precipitation model jbook to run in collab.

Just a quick fix in the collab link


Fix #117 - Add longer health check timeout on register-buddy container

Australia is a loooong way away from hub.docker.com, which causes the register-buddy sidecar to constantly crash as seen in #117. Every request to /healthz calls the external API service before responding, and 1 second timeout is simply not enough time.
Adding a 3 second check timeout here fixes the issue, vs the default liveness check timeout of 1 second.
➜  ~ /usr/bin/time curl -ivs http://<IP_ADDRESS>:8889/healthz
*   Trying <IP_ADDRESS>:8889...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8889 (#0)
> GET /healthz HTTP/1.1
> Host: <IP_ADDRESS>:8889
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 27 Jan 2022 04:37:56 GMT
Date: Thu, 27 Jan 2022 04:37:56 GMT
< Content-Length: 0
Content-Length: 0

<
* Connection #0 to host <IP_ADDRESS> left intact
        1.98 real         0.00 user         0.00 sys

Bump.


17.1.5 sudo not working in 'ask password' mode

I'm using 17.1.5, when setting sudo to 'ask password' under System -> Settings -> Administrator sudo does not work.
Dissallow -> does not work (to be expected)
No password -> works fine
Ask password -> asks for password but the password of the user logged in via ssh is not working.
I tried with both Disable integrated authentication on and off. I also tried with Permit password login on and off, as expected this only toggled the sshd setting.
The user I am using is member of the admin group.

You have to be more specific about your setup. Generally, if "no password" works, "ask password" rejecting the password means its wrong :)

Ha you just beet me too it.
I managed to fat finger the password (its a very long one) on both account creation, webgui login (4x) and account testing (2x).
But had the correct one for sudo 9x in a row. I figured it out by temporarily setting it to a simple password for further testing.
Good news I guess, I am not a donkey ;)

We spent a lot of time getting this just right with PAM integration, you can even use 2FA from SSH or login or sudo. Whatever works with GUI login now works reliably everywhere else :)

Yeah I noticed the 2FA, I will probably play with that once I got all things sorted :)
Got pretty far this time, aside from losing a day on HA clustering due to SmartOS issues I had to abandon that.
Now poking at DDNS with double-nat (I hate my ISP)

Cool, glad to hear that. And thanks for all the pointers through issues, really appreciated. Makes a better OPNsense!


Integrating Flex, Webhooks as a parameter, Segment Connections

Added features to call webhooks via GPT functions
added a dummy webhook on app.js to handle GET requests and response with dummy JSON
Added function to send calls to Flex
Added universal segment connector to send function name, and all properties to a Segment Track Event

Merged conflicts


Add Online Mod Project Template Generator

Since it is too tedious to clone this quilt-template-mod and then manually modify the Mod_ID and Gradle configuration of the project. I hope QuiltMC official can release an online mod project generator with pre-set ArtifactID and Maven Group, as well as the name, description, Minecraft version, QSL version, etc. of the mod. It's similar to the FabricMC community's template mod generator.

Yes! This is a great idea! I don't think we have the current infrastructure for it, but it is something I want to eventually explore.

Using GitHub repository templates doesn't seem to work, as there is no support for modifying package names online based on developer needs. Generating project templates online and making them available for zip download would probably have to rely on a backend server, and of course renting a server would certainly be a huge overhead, which doesn't make sense. I'm not sure about the fabric community if they rent servers for this feature. It might be possible to utilize GitHub Action to run the backend server functionality instead of renting a server, I think you could try that.

I tried to use a pure front-end (HTML5+CSS3+JavaScript) to create this online Quilt module project generator tool, which theoretically customizes the package name according to the user's needs and supports zip downloads. The tool is packaged zip using JSZip, a third party library.
Tool
Source code

@Jaffe2718 are you aware that the version dropdowns in your tool don't work? nice project though!

@ix0rai
This is because the tool's version information for both Gradle and dependent libraries is crawled in real time from the official website or Maven repository. The theory is that your poor internet connection is causing a timeout on accessing the Gradle website or Maven repository.
And not all Minecraft versions have corresponding QFAPI and QSL, so some versions don't show QFAPI and QSL versions.

I tried to use a pure front-end (HTML5+CSS3+JavaScript) to create this online Quilt module project generator tool, which theoretically customizes the package name according to the user's needs and supports zip downloads. The tool is packaged zip using JSZip, a third party library. Tool Source code

As of now, it seems that there is no way for me to add open source license generation functionality to this tool, as I didn't find any API for it.

@Jaffe2718 I think this output in the firefox console has more to do with it than my connection:

the only extension I have which could be blocking the request is UBlock Origin

@ix0rai
You can try to visit https://services.gradle.org/versions/all to test if your network can connect to gradle official version list.

@ix0rai
I figured out why Firefox doesn't work properly with this tool:

Higher versions of Firefox block cross-origin requests by default, unless Access-Control-Allow-Origin is set in the response header of the target source;
https://services.gradle.org does not set Access-Control-Allow-Origin, so it will be blocked by Firefox by default.

To solve this problem, there are two options below:

Configure the browser:

In the address bar of the Firefox browser type: about:config;
Set privacy.file_unique_origin and security.fileuri.strict_origin_policy to false;
Restart your browser.

Use a browser based on the Chromium kernel, such as Microsoft Edge, Chrome, Arc.

That's a bad workaround at best (and likely doesn't even work - note how these configs say file URIs), and hardly firefox' fault for implementing it better. If anything, a solution would be either a) getting gradle to change that or b) proxying the request.

That's a bad workaround at best (and likely doesn't even work - note how these configs say file URIs), and hardly firefox' fault for implementing it better. If anything, a solution would be either a) getting gradle to change that or b) proxying the request.

Disabling cross-origin requests is the default security policy in higher versions of Firefox, although it is sometimes inconvenient for web developers. This tool I made mainly for chromium kernel browsers because of the large chromium user base. Of course, if you need to support Firefox, you just need to find a suitable API to get the version list of gradle, but I didn't find such an API at the moment, until then, maybe this tool is not suitable for users whose browsers disable cross-origin requests by default.

Of course, my tool is only an example of a tool for testing, and there are a lot of restrictions on its use, and I won't necessarily continue to maintain it afterwards. However, as far as I know, there are two ways to create GitHub Pages, branch-based and GitHub Actions-based (no need to rent a server to realize the back-end), the latter can be used as a back-end server with certain functions, theoretically, Java can be used as the back-end for version information crawling and zip production, and then the front-end is responsible for interacting with the user. In this way the acquisition of version information is done in the backend and will not be limited to the user's browser type. But this project is a lot of work, plus my personal understanding of the structure of the quilt mod project is limited, so I still hope that in the end quilt mc official can produce such a tool, using GitHub Actions to create GitHub Pages should be the optimal solution.

Yeah that's a bit silly. CORS policy stuff isn't something to just ignore or work around; it exists for a reason, and you should expect more browsers to pick up on it over time. The proper solution here is for quilt to provide an API or add something to meta for the information it needs from an external source.

Isnt the only issue with CORS the gradle API? What if we just hardcode that version?

Honestly a "version of gradle supported by given quilt-loom version" isn't a bad idea for something to add to meta

It breals at a lot more places, but I think there is a not so hacky workaround for most stuff. (Remove the QSL version handling and use mr API for QFAPI, make gradle version selection hardcoded (It would be nice if wrapper was included anyways) or use an action to update it, and use quilt meta for the rest)

Anything needed from the quilt maven can be handled by having the quilt maven provide the relevant CORS headers. Quilt meta should already be fine. The gradle one is the only URL I can see there that would pose an actual issue

@Jaffe2718 asked my web friend about this, and it turns out that gradle's CORS configuration doesn't allow explicitly setting the user agent like you have in your code, and chromium browsers mishandle the rules a bit. that should be a simple fix.

So I think the best solution is to create GitHub Pages based on GitHub Actions, where the crawling of version information and generation of zip archives is done on the backend, and the frontend is only responsible for interacting with users. This way, you don't need to consider the user's browser type.
Even Quilt MC's officially adopted tool https://lambdaurora.dev/tools/import_quilt.html can't be used in Firefox because of CORS issues.

?
https://lambdaurora.dev/tools/import_quilt.html works fine for me on my desktop firefox?

My Firefox browser is developer edition, this tool https://lambdaurora.dev/tools/import_quilt.html cannot work.

So in order to adapt to all users' browsers, I don't think we can start by solving the problem of browser CORS policies. The best solution is obviously to implement and deploy the backend using Java Spring and GitHub Actions.

I already know how fabric mc online mod project generation tool works by analyzing fabric mc's web repository https://github.com/FabricMC/fabricmc.net . They specialize in deploying no backend, also GitHub Pages. the packaging zip is the JSZip library, as for the gradle it has gradle-wrapper.properties and gradle-wrapper.jar as static resource files, and you can't choose the gradle version when packaging, but it's mandatory to use the one provided by his gradle.

Couldn't we just add a proxy route to the gradle API through cloudflare ?

Or you could try kindly asking Gradle first :)
I mean, that API doesn't exactly have any uses within Gradle, only by third-party software like this, so they shouldn't be opposed to that

Perhaps Firefox users should try using a plugin first to avoid the CORS issue, after all, there isn't any currently available API to get Gradle information either.

The behavior on Firefox is the correct behavior. Bypassing it or working around intended behavior, when actually using CORS is the proper solution, is absurd

When I switch to a stable version of Firefox, even this tool https://lambdaurora.dev/tools/import_quilt.html may not be able to use.😓
My Firefox version is <IP_ADDRESS>70

At present, I do not have a way to compatible with Firefox browser, can only temporarily do not use a high version of Firefox, can temporarily use chromium kernel browser instead.

Once again, the solution is either to proxy the stuff properly to obey CORS stuff, or to ask gradle to return the proper CORS header from their API.

It seems like the CORS issue has now been solved by removing User-Agent.
pull 1
Tool

When I switch to a stable version of Firefox, even this tool https://lambdaurora.dev/tools/import_quilt.html may not be able to use.😓 My Firefox version is <IP_ADDRESS>70

At present, I do not have a way to compatible with Firefox browser, can only temporarily do not use a high version of Firefox, can temporarily use chromium kernel browser instead.

This is weird, I have never seen such issues while being a Firefox user.

When I switch to a stable version of Firefox, even this tool https://lambdaurora.dev/tools/import_quilt.html may not be able to use.😓 My Firefox version is <IP_ADDRESS>70

At present, I do not have a way to compatible with Firefox browser, can only temporarily do not use a high version of Firefox, can temporarily use chromium kernel browser instead.

I'm on 118.0.2, and I am not getting this. Your Firefox instance is bugged.


Add iceberg_arrow library

Add a new library iceberg_arrow to leverage arrow-backed implementations (e.g. filesystem and parquet).
Add Apache Arrow 18.1.0 as a third-party dependency to libiceberg_arrow.

I have added a iceberg-arrow library to opt in an Arrow-backed implementation in the future. I follow a similar pattern of arrow-cpp to manage third-party libraries. Let me know what you think. @lidavidm @pitrou @raulcd @Fokko @Xuanwo @gaborkaszab

e.g. it seems we could integrate find_package and FetchContent more naturally and not rely on the pile of code Arrow uses: https://stackoverflow.com/questions/73621403/cmake-integrating-fetchcontent-with-find-package

That makes sense! Let me spend some time to simplify it.

Well, it seems that I cannot make the Windows CI happy. It complains unresolved Arrow symbols during linking. I think I need a Windows PC to debug it.

This seems real complicated, ISTM that find_package and FetchContent mentioned by @lidavidm should be an easy way, I tried that in #13, maybe you can check that.

@zhjwpku Yes, I am already working on simplifying it by integrating FetchContent_declare with FIND_PACKAGE_ARGS. They are still in my local copy and I'm polishing them. I'm too busy these days so it may take a while to complete.

@zhjwpku Yes, I am already working on simplifying it by integrating FetchContent_declare with FIND_PACKAGE_ARGS. They are still in my local copy and I'm polishing them. It is pretty easy to import third-party libraries. The dirty part is to export targets and config files. I'm too busy these days so it may take a while to complete.

Sure, thanks. Take your time :)

This is ready for review now.
Thanks @lidavidm and @zhjwpku for approval!

Thanks @kou for your suggestion! Now the CMake implementation is greatly simplified. FYI, the installed directory looks like below:
├── include/
│   ├── iceberg/
│   │   ├── puffin.h
│   │   ├── table.h
│   │   └── demo_arrow.h
├── lib/
│   ├── libiceberg_vendored_arrow.a
│   ├── libiceberg_arrow.a
│   ├── libiceberg_core.a
│   ├── libiceberg_puffin.a
│   ├── cmake/
│   │   ├── IcebergConfig.cmake
│   │   ├── IcebergConfigVersion.cmake
│   │   ├── IcebergTargets.cmake
│   │   └── IcebergTargets-debug.cmake
└── share/
    └── doc/
        └── Iceberg/
            ├── LICENSE
            └── NOTICE


Connnector error: NullConstraintViolation

Versions

"prisma2": "^2.0.0-alpha.942"
"@prisma/client": "^2.0.0-alpha.942",
OSX
Node v12.1.0

Schema
// This is your Prisma schema file,
// learn more about it in the docs: https://pris.ly/d/prisma-schema

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id           Int     @id @default(autoincrement())
  // Global unique identifier!
  phoneNumber  String  @unique
  // Pre-Computed values for fast access.
  currentRisk  Float?
  groupRisk    Float?
  events       Event[] @relation("UserEvents")
  interactions Event[] @relation("UserInteractions")
}

model Event {
  id          Int       @id @default(autoincrement())
  user        User      @relation("UserEvents")
  timestamp   DateTime  @default(now())
  eventType   EventType
  interaction User?     @relation("UserInteractions")
  // Attachments, handled by event type implementation
  // might change
  data        String?
}

enum EventType {
  // Having travelled in a high risk area before
  HighRiskArea
  // Moving to another part of Germany.
  Location
  // Social interaction with another user
  Interaction
  // Developed Sympthoms
  Sympthoms
  // Diagnosed Sick
  DiagnosedSick
  // Marker inserted in past timeline when covid is diagnosed, to ease model calculation.
  MarkerSick
}

When trying to run the following query the I received the error message below:
  async locationEvent(
    state: string,
    region: string,
    timestamp: Moment = moment(),
  ) {
    await this.client.event.create({
      data: {
        eventType: EventType.Location,
        timestamp: timestamp.toDate(),
        user: {
          connect: { id: this.user.id },
        },
        // Best encoding!
        data: JSON.stringify({ state, region }),
        interaction: null
      },
    })
  }

Current workaround
It seems like the generated migrations were broken which resulted in the DB + Prisma schema being out of sync. I had to wipe my DB completely, delete the migrations folder and re-create & re-run migrations from scratch.
Further notes
Having prisma status (see https://github.com/prisma/specs/issues/400) would have helped to debug this further.

I wasn't able to reproduce this. Also, I am also unsure what part of the stack is broken here. The issue on a first look seems to belong to Prisma Client but in the workaround, you have mentioned that you needed to reset your database in order to generate new migration.
Can you please elaborate this a bit?

@schickling I am going to close this as its been 3 months now and there is no activity here.
Please reply if you want us to take another look.


UI Apps

The UI apps doesn't show when I add them

Hey, did you try the Github version or the repository version? The Github version should be fixed.

Hey, did you try the Github version or the repository version? The Github version should be fixed. The repository one is waiting for approval

I only tried the repository version, I'll try the GitHub one


[RFE] Use crc setup to start the daemon if not running

Since our future plan to decouple tray and installer from the crc and tray should only talk to crc daemon as it currently doing but right now it also starts it. In the future if a developer doesn't need to use tray or in a CI use case where CI is a mac/windows host then crc setup and crc start should provide user expected environment.
This will also kind of align with some developer mind share where they really don't want to have any tray app but consume application as it is.
I will create an issue on tray side which should first check if daemon is running and if it is then don't start and consume the existing daemon.

This is related to https://github.com/code-ready/crc/issues/2557 and https://github.com/code-ready/crc/issues/2459

Yep

Closing this as this should be mostly solved in git.


Building TensorFlow 1.12 with CUDA 10

I get the error below when I try to build TensorFlow on a Windows 2016 server.
I followed this guide step-by-step: https://mc.ai/install-tensorflow-gpu-with-cuda-10-0-and-cudnn-7-4-for-python-on-windows-10/
It is my first time trying to something like this - so please elaborate your answers :-)
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
WARNING: Option 'experimental_shortened_obj_file_path' is deprecated
Loading:
Loading: 0 packages loaded
INFO: Build options have changed, discarding analysis cache.
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded)
WARNING: C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/BUILD:1992:1: in srcs attribute of cc_library rule @grpc//:grpc_nanopb: please do not import '@grpc//third_party/nanopb:pb_common.c' directly. You should either move the file to this package or depend on an appropriate rule there. Since this rule was created by the macro 'grpc_generate_one_off_targets', the error might have been caused by the macro implementation in C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/bazel/grpc_build_system.bzl:172:12
WARNING: C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/BUILD:1992:1: in srcs attribute of cc_library rule @grpc//:grpc_nanopb: please do not import '@grpc//third_party/nanopb:pb_decode.c' directly. You should either move the file to this package or depend on an appropriate rule there. Since this rule was created by the macro 'grpc_generate_one_off_targets', the error might have been caused by the macro implementation in C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/bazel/grpc_build_system.bzl:172:12
WARNING: C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/BUILD:1992:1: in srcs attribute of cc_library rule @grpc//:grpc_nanopb: please do not import '@grpc//third_party/nanopb:pb_encode.c' directly. You should either move the file to this package or depend on an appropriate rule there. Since this rule was created by the macro 'grpc_generate_one_off_targets', the error might have been caused by the macro implementation in C:/users/administrator/_bazel_administrator/xv6zejqw/external/grpc/bazel/grpc_build_system.bzl:172:12
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded)
WARNING: C:/tensorflow/tensorflow/contrib/learn/BUILD:17:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: C:/tensorflow/tensorflow/contrib/learn/BUILD:17:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
WARNING: C:/tensorflow/tensorflow/contrib/timeseries/python/timeseries/state_space_models/BUILD:230:1: in py_library rule //tensorflow/contrib/timeseries/python/timeseries/state_space_models:filtering_postprocessor: target '//tensorflow/contrib/timeseries/python/timeseries/state_space_models:filtering_postprocessor' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
WARNING: C:/tensorflow/tensorflow/contrib/seq2seq/BUILD:23:1: in py_library rule //tensorflow/contrib/seq2seq:seq2seq_py: target '//tensorflow/contrib/seq2seq:seq2seq_py' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
WARNING: C:/tensorflow/tensorflow/contrib/bayesflow/BUILD:17:1: in py_library rule //tensorflow/contrib/bayesflow:bayesflow_py: target '//tensorflow/contrib/bayesflow:bayesflow_py' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
WARNING: C:/tensorflow/tensorflow/contrib/timeseries/python/timeseries/BUILD:354:1: in py_library rule //tensorflow/contrib/timeseries/python/timeseries:ar_model: target '//tensorflow/contrib/timeseries/python/timeseries:ar_model' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
WARNING: C:/tensorflow/tensorflow/contrib/timeseries/python/timeseries/state_space_models/BUILD:73:1: in py_library rule //tensorflow/contrib/timeseries/python/timeseries/state_space_models:kalman_filter: target '//tensorflow/contrib/timeseries/python/timeseries/state_space_models:kalman_filter' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
WARNING: C:/tensorflow/tensorflow/contrib/BUILD:13:1: in py_library rule //tensorflow/contrib:contrib_py: target '//tensorflow/contrib:contrib_py' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of tf.contrib.distributions to tfp.distributions.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded).
INFO: Found 1 target...

I just tried to run "py configure.py" via the Windows CMS instead of via the Bash/msys2 cmd... and no tensorflow ran for something like 10 min - but then failed again with the error-message below:
What does this mean?
host_defines.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 860.118s, Critical Path: 96.34s
INFO: 2217 processes: 2217 local.
FAILED: Build did NOT complete successfully

RESOLVED!
GPU support
To make the TensorFlow package builder with GPU support:
bazel build --config=opt --config=cuda --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package
Bazel build options
**Use this option when building to avoid issue with package creation: https://github.com/tensorflow/tensorflow/issues/22390
--define=no_tensorflow_py_deps=true**
Building TensorFlow from source can use a lot of RAM. If your system is memory-constrained, limit Bazel's RAM usage with: --local_resources 2048,.5,1.0.
If building with GPU support, add --copt=-nvcc_options=disable-warnings to suppress nvcc warning messages.


How to implement this into Unity Android Build?

Hey there! I need to make an Android App with Unity, and that App can open a browser that scan AR target, thing is, A-Frame or three.js seems not working because it can't open Android device's camera.
I was desperately searching ways to solve this problem so if you feel disrupted plz ignore me...

Sorry for late answer.
unity-scene-web-exporter export scene data, not scripting or logic.
unity-scene-web-exporter is based on A-Frame or three.js.
If A-Frame or three.js can't solve your problem, then unity-scene-web-exporter can't solve problem.
You should search other solution. some search keywords.

webcam html5
webar
artoolkit web


Init-testing-for-GitAPITest

JENKINS-60940 - Converting GitAPITestCase to junit 4
Created a new class for testing GitAPITestCase class which is a non abstract class. Updating the junit version from v3 to v4.
Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING doc

Types of changes
What types of changes does your code introduce? Put an x in the boxes that apply

[x] Infrastructure change (non-breaking change which updates dependencies or improves infrastructure)

@MarkEWaite, I created a new class named GitAPITestUpdate which is a non abstract class, that will contain all the tests related to GitAPITestCase. I just initialized the class with the basic functionalities. I'll add the test methods in each commit in an incremental manner.

@MarkEWaite, I created a new class named GitAPITestUpdate which is a non abstract class, that will contain all the tests related to GitAPITestCase. I just initialized the class with the basic functionalities. I'll add the test methods in each commit in an incremental manner.

I don't object strenuously to this approach, but I suspect you would have a better outcome if you migrated one test at a time and inserted the additional items needed (WorkingArea type items) to support the test only as needed.  I know that one of my concerns was that the number of tests per test class is quite large  (142 tests in JUnit 4 GitClientTest, 60 tests in JUnit 4 GitAPITest, 60 tests in JUnit 4 GitAPITest, and 48 tests left to convert in GitAPITestCase) and that makes it more challenging to use multiple test processes to run tests faster on multi-core machines.

I just added the Working Area type items initially because most of the tests in GitAPITestCase use working Area class. If I add any of the test, the working area is required to run the test. I only added Working Area class and it's dependencies. Slowly I can start adding the testing in each step.

Hi @MarkEWaite, Quick question on upgrading code to junit4. Should I upgrade the test cases that are deprecated too?

I forced push due to a wrong commit message.

Thanks @MarkEWaite, I'll upgrade them as well.

Thanks for the pull request.
There are some things that are lost in the draft pull request that you're proposing.

GitAPITestCase is extended by CliGitAPIImplTest, JGitAPIImplTest, and JGitApacheAPIImplTest so that the same test source code is executed with three different git implementations.  I don't see how those three implementations are being tested in the tests in
test_lsTree_non_recursive seems to have been deleted and not replaced with a new implementation.  I don't see any new calls to lsTree() in the new code

There are some tests that migrated but have not been removed from their original JUnit 3 implementation class.  Tests like testPushDeprecatedSignature and testTrackingSubmodule still exist in the JUnit 3 implementation.
Some of the tests were renamed while others retain their original names.  It is a little easier for me to review if they retain the same name or if they are consistently all renamed in the transition.
The formatting in the new file uses leading tabs.  The contributing document advises that

New code should follow the link:https://github.com/jenkinsci/scm-api-plugin/blob/master/CONTRIBUTING.md#code-style-guidelines[SCM API code style guidelines].

in the SCM API code style guidelines it says:

Use spaces. Tabs are banned.

It also states that Java blocks are 4 spaces.
The // moved comments on private methods seem like they won't help you much.  Until the private method is unused, it needs to remain in the JUnit 3 file.  When the private method is no longer used, the spotbugs checks should tell us that the method is unused and can be deleted.

Thanks for the review @MarkEWaite,

I thought I'll extend the classes CliGitAPIImplTest, JGitAPIImplTest & JGitApacheAPIImplTest in the end to GitAPITestUpdate, but I'll update the extended class in the upcoming PR.
I also thought of removing the junit3 tests together at once, but understood it's better to remove them after upgrading it to junit v4.
After upgrading a few tests I realized that the names used are all camelCase. Hence I abruptly changed it to camelCase. The names for all the tests will be renamed to camelCase.

I converted tabs to spaces. Just asking out of curiosity :bulb:, tabs makes the file size smaller than spaces, Ascii value of Tab used once will be smaller when compared to Ascii value of Space used four times. Then why use space :thinking: :thinking: :thinking:

Moved all the tests from GitAPITestCase to GitAPITestUpdate. Need to remove the unnecessary methods in GitAPITestCase and then delete the file.

I converted tabs to spaces. Just asking out of curiosity 💡, tabs makes the file size smaller than spaces, Ascii value of Tab used once will be smaller when compared to Ascii value of Space used four times. Then why use space 🤔 🤔 🤔

Good question.  I like code formatting standards because they avoid back and forth changes that involve only style changes.  The choice of tabs vs. spaces is somewhat arbitrary (see the holy war on stackexchange for discussion of both sides), but the SCM API coding standard chose spaces and the Netbeans IDE that I sometimes use prefers spaces and the emacs configuration that I frequently use prefers spaces.
Eventually, I hope to implement mandatory automated formatting of new files as they are added to the repository.  Existing files cannot be automatically formatted because they will cause conflicts and break the many pending pull requests in the repository.

Thanks @MarkEWaite. Didn't know there was a war between Tabs and spaces :smiley:. The automated formatting would make things easier. Can we create a config file for different kinds of IDE to support the required code style?

Hey @MarkEWaite, can you review the PR?

@Hrushi20 I'm out of the office this week on holiday.  I won't review it until next week.

Oki @MarkEWaite. Enjoy your holidays.

@Hrushi20 the tests need to pass in CI.  There isn't much motivation for me to review code that has failing tests.  If you need help understanding why the tests are failing, I may be able to assist with that, but generally I'd prefer that tests are passing before I review the code.

@MarkEWaite All of the test which are failing throw this error ```hudson.plugins.git.GitException: org.eclipse.jgit.api.errors.ServiceUnavailableException: Signing service is not available
The tests in the CliGitImpl are running successfully. Only the tests in the JGitImpli are failing either because the methods doesn't support JGit or the method is deprecated.

@MarkEWaite All of the test which are failing throw this error
hudson.plugins.git.GitException: org.eclipse.jgit.api.errors.ServiceUnavailableException: Signing service is not available

The tests in the CliGitImpl are running successfully. Only the tests in the JGitImpli are failing either because the methods doesn't support JGit or the method is deprecated.

Check the CI jobs.  I don't see that error on any of the failing tests.

I am moving the methods to another abstract class. This abstract class is inheriting the GitAPITestUpdate. The fields present in the GitAPITestUpdate are private and static. Can I convert them to package protected so that the sub class can access those fields?
Eg:
 private static String defaultBranchName = "mast" + "er";

I am not able to use default Branch Name in the subclass.

I am moving the NotImplementedInJGit methods to another abstract class. This abstract class is inheriting the GitAPITestUpdate. The fields present in the GitAPITestUpdate are private and static. Can I convert the required fields to package protected so that the sub class can access those fields?
Eg:
 private static String defaultBranchName = "mast" + "er";

You're welcome to make them package protected so that they can be accessed elsewhere in the test package.

Hey @MarkEWaite, all tests have passed successfully :smile: :smile: :smile:

Thanks @Hrushi20 !  Since you've requested a review, could you change the pull request so that it is no longer "Draft"?

FYI: Seems this major relocation did not account (carry over) at least my recent fix from PR #886, will re-post shortly.
I did not trawl git history further to see if other tests were changes while this PR was in the works, and whether those changes got lost too or got accounted for.


Add StreetComplete page

In the "Mapping with a Smartphone, GPS or Paper" chapter, a page about mapping with StreetComplete should be added.
I know it's "just" completing quests about existing map features and not "real" manual mapping, but it's an easy and fun way to contribute to OSM with a smartphone.

Thank you for pointing us at this. We know that the chapter on mobile mapping is pretty much out of date and refreshing this is on our todo list. But as we are just a few volunteers this might take a while.

I'd would love to see this section updated, and help translate it in French

@michael63-osm So adding StreetComplete page would be welcome? I would be happy to add it but I have a conflict of interest here (I am one of people making it).

We have no problem with that as long as it fits with the style of other pages, i.e. presents the features in a rather technical way. We do not want to add simple marketing flyers containing sentences like 'this is world's best software to accomplish anything you might ever dream of" - I think you get the point.
We already received the occasional input particularly for the mobile mapping chapter and after a quick review we added it. There was one app which we did not cover because it advertised itself as a simple GPS track visualizer but required all kinds of permissions from reading your list of contacts to sending SMS. Typically when covering software I would like to install it myself to have a look at it but in that case I stopped when being asked to grant all permissions imaginable.
If a project offers good documentation on its homepage then a page on LearnOSM might simply contain a general description what it is about and where it differs from other products and then provide a link to the documentation. E.g. when I revised the documentation about QGIS I mentioned that there are tons of tutorials and books out there and that there is no way we can adequately cover the complete product. But on our website people will be particularly interested in how to load OSM data and how to handle them. So that is what I covered in our small guide.
If you want to add something in Markdown try to keep formatting aligned with the other guides. Usually our pages do not contain fancy formatting so if you feel more comfortable writing plain text or some open document format this will not be a big problem. For Markdown you may want to have a look at our translators' guide (https://learnosm.org/en/contribute/translator/#translating-a-module---using-0200-12-31-beginner-as-an-example) which informs translators what are special commands for site navigation.
Just ask if anything is unclear - general questions may best go to<EMAIL_ADDRESS>and keep in mind that we are all volunteers who take care in our spare time. Please be patient if an answer takes a while.

Some things take way longer than we would like but in the meantime a contributor found the time and had the expertise to write an in-depth guide  on StreetComplete: https://learnosm.org/en/mobile-mapping/streetcomplete/
LearnOSM now covers MapSwipe as well and a revised version of the guide on Vespucci is nearly finished.


(npm module) Doesn't work on Electron 1.8.2 (culprit: graceful-fs^3.0.11)

This goes away if I manually upgrade graceful-fs.
This seems to also break in Node 6 and is actually already corrected in this repo but no new version was published to npm.

Same error in electron 2.0

Electron 6 requires Node 12 which has an incompatibility with graceful-fs 3. @electron-forge/publisher-s3 depends on "s3" so it makes that publisher with Electron 6 fail.

Looks like  the graceful-fs version is updated. As stated by @dbuezas, the change is in master but no NPM published version is available with this change. Please publish to NPM.


feature: generate ressources for specific paltform

What version of the CLI are you using? Output of ionic --version
3.0.0-beta8
Short description of the problem:
In CLI 2.0 it was possible to generate the icons and splash screen for a specific platform with the platform tag like ionic resources android
In V3 it should also be possible because I use different Icons for Android and iOS. Otherwise, the command always overrides the icons and splash screens of both platforms.
ionic cordova resources android

This will be in the next release of the CLI. Thank you!


Verilog wire should not be converted to chisel Mem()

I tryied sv2chisel with the verilog source named fake_differential.v.
In this code, there is a «wire array» :
    wire [1:0] tmds[3:0];

sv2chisel convert it to a chisel memory (class Mem()).  :
  val tmds = Mem(4,UInt(2.W))
  tmds(3) := in_clock
  tmds(2) := in_red
  tmds(1) := in_green
  tmds(0) := in_blue

I'm not sure that is a good idea to convert it to Mem(). Shouldn't it be a Vec() ?

Hi @Martoni,
Thank you for raising this issue and sorry for my very late answer.
I just fixed this in https://github.com/ovh/sv2chisel/commit/d58059a76f90f812930be2a90bb944f999efe24e


Copying a permalink fails for a repo with an SSH remote

@connor4312 is there anything you can think of about your setup that's out of the ordinary? I can't reproduce this with just an ssh remote:

In that case, should be fixed by d344871a337098047e9d7f357b18de18c3761ae7


Why not use LiteLLM?

Hi @haseeb-heaven,
Curious what litellm was missing to be useful here? Any feedback would be great
https://github.com/haseeb-heaven/open-code-interpreter/blob/4c142036c430b1184c8eb420c8af804638ee73d1/interpreter.py#L7

Thanks for suggestion, If you have reference for using LiteLLM with HugginFace models that would be great i would add definitely add that.

Found some Resources and will add that in next update.

Added Support with LiteLLM now :)


[Bug]: Incorrect Page load execution

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
When Query1 depends on Query2, Query2 is not run before Query1 on page load.
Steps To Reproduce
A user reported this issue on intercom
https://app.appsmith.com/applications/619c86ded3208212f819ce08/pages/619c86ded3208212f819ce0f/edit/api/619f3b88f6f14050cbd7d103
Environment
Production
Version
Cloud

I wonder if this should not be reopened to include JS functions e.g. I have a function that process data from 2 APIs, and that function (filtered, merged data) is the source of a Table.
I like to specify that Api1 and Api2 are run first before the function, all at page load.
Currently, these settings are in different places: settings of each Query + setting of JS Objects where one can select the function to run a page load.
Ideally, you have all queries and JS functions (from all JS objects) in one page, and you can set the execution order.

@yan-hic the order of execution is automatically handled by the platform. So if your function is reading data from API1 and API2, they will automatically be run on page load before the function.
If you need control over them, you can create a new function and manually run the APIs on page load and simply mark the function to run on Page Load
fetchData: async () => {
   API1.run();
   API2.run();
}

@Nikhil-Nandagopal fair enough, I agree this can be done programmatically but it can become complex if one needs to consider all interactions and change dependencies in JS code. I maintain that one place to simply control the order of execution of all Page Load objects is useful, and in line with no-code spirit.
Just my 2 cents, no need to re-open anything.


Add Client Specific Subjects in Claims

Currently every Client get's the same Subject for the User. And this is in case if ASP.NET Identity the UserId. So this is a little Information leaked.
Please add an option (Global and/or per Client) so that it is possible to send a client specific Subject to the client.
From my investigation this can be achieved by DefaultClaimsService which make a   (subjectId+clientId).Sha256()
And also https://github.com/IdentityServer/IdentityServer4/blob/74827bc4df39c2234d9490ba5ce032ce806ab475/src/IdentityServer4/Validation/EndSessionRequestValidator.cs#L86
must be changed to recreate the hash, and check for this.
Other Places?
Now seems the time to add this, because 2.0 is not yet released. I willing to implement this.

If you are going to that trouble, why not enable pairwise sub. See the oidc spec if the term is unfamiliar to you. I can help if useful.
..Tom's phone
On Sep 24, 2017, at 11:29 AM, Albert Weinert <notifications<EMAIL_ADDRESS>wrote:
Currently every Client get's the same Subject for the User. And this is in case if ASP.NEThttp://ASP.NET Identity the UserId. So this is a little Information leaked.
Please add an option (Global and/or per Client) so that it is possible to send a client specific Subject to the client.
From my investigation this can be achieved by DefaultClaimsService which make a (subjectId+clientId).Sha256()
And also https://github.com/IdentityServer/IdentityServer4/blob/74827bc4df39c2234d9490ba5ce032ce806ab475/src/IdentityServer4/Validation/EndSessionRequestValidator.cs#L86
must be changed to recreate the hash, and check for this.
Other Places?
Now seems the time to add this, because 2.0 is not yet released. I willing to implement this.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/IdentityServer/IdentityServer4/issues/1557, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKxq1ia8cTpoFk-H7ve-wSJMma01v-V7ks5slp-UgaJpZM4Ph-sx.

Yes - we are interested in implementing pairwise subject identifier. It's been on our todo list for a long time.
@DerAlbertCom  If you are interested in this - we can start a discussion. This won't happen for 2.0 anymore though.

I thought a little more about your proposal. Assuming that the goal is for the (subjectId+clientId).Sha256() to be unguessable there must be some secret value known only to the Idp (OP). That could be either the subjectId itself or a salted hash were the salt was highly protected.
It may not reasonable for the subectId to be protected in  a database that is used for lots of other purposes; perhaps it needs to be held in a separate secured db?

From: Dominick Baier<EMAIL_ADDRESS>Sent: Sunday, September 24, 2017 9:38 PM
To: IdentityServer/IdentityServer4
Cc: tom jones; Comment
Subject: Re: [IdentityServer/IdentityServer4] Add Client Specific Subjects in Claims (#1557)
Yes - we are interested in implementing pairwise subject identifier. It's been on our todo list for a long time.
@DerAlbertComhttps://github.com/deralbertcom If you are interested in this - we can start a discussion. This won't happen for 2.0 anymore though.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/IdentityServer/IdentityServer4/issues/1557#issuecomment-331775320, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKxq1hJfRg8Spk0sO8yN00JflCbDUChuks5sly5UgaJpZM4Ph-sx.

I've read through the specs (http://openid.net/specs/openid-connect-core-1_0.html#SubjectIDTypes).
So, basicly  (sector_identifier+account_id+salt) then using SHA or AES. I'm for SHA, don't see really additional security in AES here. Performancewise SHA should be faster, but it think in general the generation and validation of the pairwise subject id is not a performance problem. And on the IdentityServer side we have all needed Information to recreate the hash.
I have have some problems with the specs, because the sector_identifier should be either a sector_identifier_uri, or a redirect_uri (if only one is present). But with these kind identifiers a Client cannot be moved to another domain (renaming company, mergers, product changes etc.).
So, the questions i have. Should we go to the "trouble" with a sector_identitier_url or just simply use a simpler sector_identifier (like clientid) instead of a volatile redirect_uri? The sector_identitifer_uri seems not commonly in use. And if we do this, this may be optional. If the client want's to use redirect_uri or sector_identifier then it will be used, otherwise the clientid.
I think this is also in the specs, because the example 3 for creating the subject is about generated GUIDs as sector_identifier.
Also, as I understand the spec ist that the IdServer should announce the availability of the paire_wise subject. And then the Client MAY opt in for pair wise subject, this should an opt out. But the client registration is not a part of IdServer4, so no problem on that side ;)

We should add the extra property needed to Client now for 2.0 if we're going to do this.

First suggestion

SectorIdentitierUri
SectorId (if we don't go with ClientId)
PairWiseSubjectSalt
EnablePairWiseSubject

I reread the pairwise section.
It is unclear if sector ID is required because of the odd mixture and must and should.
Sector ID is not needed in my application and it will not support sectors in my OP database.
It seems unclear what would happen if sectors were presented by (one of) the client in one case and not in another.
I guess it is a judgement call on sector ID.

We will discuss this...

I think we'll start with adding PairWiseSubjectSalt whose semantics are that if it's null then we do what we already do today, and if it's non-null then we use it to calculate a pair-wise subject id. We'll add the property for 2.0, and in 2.1 we'll actually implement it. And if you want to do the work on it @DerAlbertCom , then let's start a issue/PR to discuss.

Ok, I've added the property. The next thing is for 2.1 to add the implementation. For now, I'll mark this as a 21. enhancement. If you want to start on a PR, you can @DerAlbertCom .

For clarification.
Following things must be done for this.

Adapt ClaimsFactory to calculate the pair wise subject?
Change EndSessionRequest Validator, to validator the pair wise subject.
Announce the availability of pair wise subject in the metadata

The Pairwise Subject could be calculate with
a. (subject+salt).Sha256() as base64
or
b. pbkdf2(subject, salt, 10000) as base64

Marking as 3.0 since we anticipate breaking changes. I think we all knew this, but just doing it for formality.


Migrate to django-ninja

related https://github.com/ImageMarkup/tracker/issues/143

@danlamanna the tests are now passing; this is probably a good time to do a mid-stream code review to make sure the changes I made are reasonable.

This is now ready for final review. All references to DRF have been removed from this codebase. Here's a comparison of the old and new swagger pages:

These are very sweeping changes, so there's a high likelihood that some behavioral changes have slipped through the cracks. If you have a good way to manually verify zip downloading end-to-end, that would be good, as I have not done so locally.

I believe all feedback has been addressed, PTAL.


build is getting failed with spark 2.3 version .

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building spark-iforest 2.4.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-iforest ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/bizviz/git_dataprep_anomaly_library/dataprep/spark-iforest/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.3.2:compile (default) @ spark-iforest ---
[INFO] artifact commons-codec:commons-codec: checking for updates from repo1
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  com.twitter:chill_2.11:0.8.4 requires scala version: 2.11.8
[WARNING]  org.apache.spark:spark-core_2.11:2.3.0 requires scala version: 2.11.8
[WARNING]  org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ spark-iforest ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.3.2:compile (compile) @ spark-iforest ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  com.twitter:chill_2.11:0.8.4 requires scala version: 2.11.8
[WARNING]  org.apache.spark:spark-core_2.11:2.3.0 requires scala version: 2.11.8
[WARNING]  org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ spark-iforest ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ spark-iforest ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.3.2:testCompile (test-compile) @ spark-iforest ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  com.twitter:chill_2.11:0.8.4 requires scala version: 2.11.8
[WARNING]  org.apache.spark:spark-core_2.11:2.3.0 requires scala version: 2.11.8
[WARNING]  org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.7:test (default-test) @ spark-iforest ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-iforest ---
Discovery starting.
Discovery completed in 1 second, 386 milliseconds.
Run starting. Expected test count is: 7
IForestSuite:
19/02/26 15:25:46 WARN Utils: Your hostname, my-engine resolves to a loopback address: <IP_ADDRESS>; using <IP_ADDRESS> instead (on interface enp0s25)
19/02/26 15:25:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/02/26 15:25:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

default parameters
set parameters
split data
sample features
*** RUN ABORTED ***
java.lang.NoSuchMethodError: org.apache.spark.ml.util.Instrumentation$.instrumented(Lscala/Function1;)Ljava/lang/Object;
at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:475)
at org.apache.spark.ml.iforest.IForestSuite$$anonfun$1.apply$mcV$sp(IForestSuite.scala:107)
at org.apache.spark.ml.iforest.IForestSuite$$anonfun$1.apply(IForestSuite.scala:97)
at org.apache.spark.ml.iforest.IForestSuite$$anonfun$1.apply(IForestSuite.scala:97)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:67)
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:11 min
[INFO] Finished at: 2019-02-26T15:26:07+05:30
[INFO] Final Memory: 29M/401M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project spark-iforest: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Process finished with exit code 1

please help @titicaca

ML api has changed on spark v2.4.x,
Try the version v2.3.0 https://github.com/titicaca/spark-iforest/releases/tag/v2.3.0

@titicaca thanks .. i will try with this .

@titicaca It is having some transitive dependency with spark 2.4. automatically downloading 2.4 spark libraries when i used this in a sbt project .

my build.sbt is as follows
scalaVersion := "2.11.8"
resolvers += Resolver.mavenLocal
libraryDependencies += "com.typesafe.play" % "play-json_2.11" % "2.6.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.3.0"
libraryDependencies += "org.specs" % "specs" % "1.4.0" % Test
libraryDependencies += "org.specs2" %% "specs2-core" % "2.4.16" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % Test
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.9"
libraryDependencies += "com.twitter" %% "util-eval" % "6.43.0"
libraryDependencies += "org.apache.spark.ml" % "spark-iforest" % "2.3.0"

Is there any building error?

No build error . IForestExample running fine . But when i used this library in a sbt project , I am getting following error

19/02/27 10:16:45 INFO DAGScheduler: Job 7 finished: zipWithIndex at IForest.scala:452, took 0.567872 s
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ml.util.Instrumentation$.create(Lorg/apache/spark/ml/Estimator;Lorg/apache/spark/rdd/RDD;)Lorg/apache/spark/ml/util/Instrumentation;
at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:480)
at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:317)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:153)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:149)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableViewLike$Transformed$class.foreach(IterableViewLike.scala:44)
at scala.collection.SeqViewLike$AbstractTransformed.foreach(SeqViewLike.scala:37)
at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:149)

Emm, it looks like a spark version error. Could you double check your spark version and spark-iforest pkg in your sbt project?

I removed my .ivy2 cache , then build again . it start working .
@titicaca do you have any perfomance report (Time taken to process 1 billion records ) of the algorithm ?

I am closing the issue . thanks

Hi, I'm getting the same error in version 2.4.0 with Spark 2.4 at runtime, although the tests all pass.
iforest = IForest(contamination=0.3, maxDepth=2)
model = iforest.fit(nz_df)

Results:
Py4JJavaError: An error occurred while calling o161.fit.
: java.lang.NoSuchMethodError: org.apache.spark.ml.util.Instrumentation$.instrumented(Lscala/Function1;)Ljava/lang/Object;
	at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:495)
	at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:334)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:280)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:748)

Any ideas what could the problem be?