Global metrics of bounding box

First of all I'm excited about this project especially it is wasm friendly! I'm wondering if we have API that can return the global bbox like FreeType2 does: https://www.freetype.org/freetype2/docs/tutorial/step2.html, section 3 Global Metrics of bbox.

This is not implemented yet, but can be easily done.

I'm new to this library. can you please show me which part of the code? I can probably try to implement it.

It's pretty non-trivial, actually. I will do this myself.

Turns out it was very easy.

Thanks! It works great.


Euclidean Distance Transform

Given the importance and wide use of the distance transform of binary images in different computer vision applications, I wonder if it would be possible to add a Euclidean distance transform with gradient support.
There exists very nice O(n) solutions already implemented in C/C++. For example. Opencv has implemented this paper here.
Here's another fast implementation in PyDIP.
This would be a tremendously useful feature. Thanks!

@martinwicke @ebrevdo @rmlarsen do you know if there's a TensorFlower who'd be interested in this?

I am  a beginner contributer?Is this suitable  for  first timer?Which  lebel  for  first time or beginner in  tensorflow?

Unfortunately I do not think this is a good project for a beginner. This is probably best for someone with experience writing TensorFlow ops.

Are there any updates on this? This will be an amazing feature when available!

Following :)

Hey, guys. Should be done within this week. I'll update this thread once I open a PR.

Thank you so much for your great work.

Is there an update on this feature by any chance? Thanks!

@skye Sorry about the delay, I've created a PR with a barebones implementation. I'll add faster algorithms and test cases over the next few hours.

Great job !  How would we be able to access this custom_op ?

@ahatamiz
The changes haven't been merged into the main repository yet, so clone my repository, and switch to this branch: euclidean-distance-transform.
Here's the build guide for tensorflow_addons.
Then you can use the transform as such:
from tensorflow_addons.custom_ops.image import euclidean_dist_transform

x = ... # binary image
y = euclidean_dist_transform(x)

Note:

x must be a binary image (0 and 1 only)
x must be of type tf.uint8. Use tf.cast if it isn't.
x must be of shape [N, H, W, 1] or [H, W, 1]
The dtype of y can be set like this: y = euclidean_dist_transform(x, dtype=tf.float32). tf.float32 is the default.
The current algorithm is of quadratic complexity O(N^2) where N is the number of pixels. I'm working on the faster O(N) algorithm right now, and I'll push it soon.
The Op is currently CPU only.
The Op is not differentiable, and running tf.gradients(y) will return [None].

Let me know if you find any problems or issues with the Op.

@Squadrick
That's amazing ! this op will be tremendously helpful for many researchers and I am sure people will come to appreciate its importance soon !
The only concern though is the fact that Op is non-differentiable. It would be great if we could have gradients defined for this op.
In many applications, this op will be needed for connecting two layers in a sense that it initializes the second layer with the output of the first layer. In essense, both of the layers have learn-able parameters thus not being able to back-propagate the of error through this op would limit its use.

@Squadrick awesome! Once it's ready with full test cases etc., maybe ping @sofienbouaziz to take a look.

@skye Sure thing.
@ahatamiz Couldn’t really figure out what the gradient of the operation would be, considering the fact that the function is defined on binary images. If you could help out with that part, I can implement fairly easily.

@Squadrick Thanks for the contribution! Just quickly parsing through the code I have the same comment than @ahatamiz that this op is non-differentiable and it would be great if we could back-propagate through it. The algorithm currently used is O(n^2)  is the goal to then implement a O(n) algorithm?

@sofienbouaziz I'm working on the O(n) algorithm right now, I'll push it within a day.
Like I mentioned before, I'm not sure how this op can be differentiable considering the fact that the op is defined on binary images.

@Squadrick Awesome for the O(n) version! Let me think about how to differentiate this op during the week-end. Just out of curiosity do you think it would be possible to develop this op purely in python?

@sofienbouaziz I think @Squadrick is right that gradients make no sense for a binary input. However, there is an generalization for arbitrary functions (that you and @ataiya implemented?) that if I understand correctly should be differentiable. If there was an op for this you would also get the binary version for free by composing it with an indicator function.
I'm not sure if the generalized version would do exactly what @ahatamiz wants, but it would be very nice to have an op for.

As an experiment I tried implementing a simple version of the generalized transform in python, and it is indeed differentiable. With a careful choice of indicator function it seems you can even "learn" a binary image by supervising its distance transform, so it might be suitable for use as a layer in a network as @ahatamiz suggested.
Below is an example showing the gradients. Here, the mask is learned from an L2 loss between the DT of the mask and the DT of the ground truth mask.

Thanks @Squadrick for you great work on this and thanks @sofienbouaziz for taking a look into this. Also thanks for your thoughtful contribution to the discussion @drebain
@Squadrick, you are right in the sense that the op may not be differentiable if we were to utilize on binary images. However,  what I had in mind is very close the implementation by @drebain. One can pass the output of sigmoid, before thresholding, to a (generalized) distance transform layer and still allow for gradient flow. This is useful since in most vision applications, the output of sigmoid (or softmax) can be fed into this layer for more interesting operation down-stream.
The implementation by @drebain is definitely correct, but unfortunately it runs out of memory on bigger images ( like 256 by 256 ) on a GPU like Titan XP and also is not as fast, due to its O(n^2) complexity.

According to its documentation, the default O(n) L2 implementation in OpenCV uses the multi-pass algorithm based on the generalized transform, so if @Squadrick has implemented that, then I think all that would need to be done is to expose a version of the op that doesn't apply the thresholding indicator function.

@ahatamiz
The changes haven't been merged into the main repository yet, so clone my repository, and switch to this branch: euclidean-distance-transform.
Here's the build guide for tensorflow_addons.
Then you can use the transform as such:
from tensorflow_addons.custom_ops.image import euclidean_dist_transform

x = ... # binary image
y = euclidean_dist_transform(x)

Note:

x must be a binary image (0 and 1 only)
x must be of type tf.uint8. Use tf.cast if it isn't.
x must be of shape [N, H, W, 1] or [H, W, 1]
The dtype of y can be set like this: y = euclidean_dist_transform(x, dtype=tf.float32). tf.float32 is the default.
The current algorithm is of quadratic complexity O(N^2) where N is the number of pixels. I'm working on the faster O(N) algorithm right now, and I'll push it soon.
The Op is currently CPU only.
The Op is not differentiable, and running tf.gradients(y) will return [None].

Let me know if you find any problems or issues with the Op.

@Squadrick . I cloned the branch you mentioned and built from the source, but it seems like we are missing a file:
/addons-euclidean-distance-transform/tensorflow_addons/custom_ops/image/python/_image_ops.so  cannot open shared object file: No such file or directory
Also, as @drebain  mentioned, I think it would be helpful to also allow for feeding logits before sigmoid or softmax and not just binary images ( for a back-propagating error ) . Currently, this layer seems to only accept binary images.

@ahatamiz I haven't gotten to fully review the PR, but I recognize that error and you likely just need to move out of the src directory after pip install. The .so files aren't built until bazel compile and your current working directory does not contain them

@seanpmorgan Yes. You were right and it worked.
Also, it seems like this layer requires TensorFlow version >= 2. Would it be possible to allow older versions like 1.9 for now ?  @Squadrick

@ahatamiz I haven't gotten to fully review the PR, but I recognize that error and you likely just need to move out of the src directory after pip install. The .so files aren't built until bazel compile and your current working directory does not contain them

Thanks @seanpmorgan . However I also have an issue with version incompatblity as only Tensorflow 2.0.0 is allowed. I have not yet transferred everything to Tensorflow 2.0.0 and was hoping to get it working on older versions.

@ahatamiz Ah, thats a bit of a problem. Addons is only scoped to be TF 2.x+ and it'll take a lot of convincing to get us to reconsider that. You can however make it work on your own fork of the repo by altering this line:
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/__init__.py#L54

@drebain Very cool colab! Indeed the code you are referring to is an old implementation I did of this particular paper and as you pointed out it is differentiable. Yes, you are right that the gradient doesn't make sense in the case of binary image, however I was thinking we could do something similar than what has been done for quantization where the forward pass would be computed on the binary image and the backward pass on some sort of relaxation.

I am closing this issue as the Euclidean Distance Transform op was merged. If updates are needed let's reopen one with the necessary details.
@Squadrick Thanks for this very useful op.

Thanks you @Squadrick  and @sofienbouaziz .
Is it also possible to provide GPU support for this op ?

@ahatamiz GPU support for TensorFlow Addons is currently WIP. Ref #294. Once this is merged, I'll add GPU support.

Hi,
I just want to confirm.  Can the current implementation of the TF addons Euclidean Distance Transform be used in a loss function?  I noticed the discussion above on the issue of gradients when the distance transform is applied on binary images.  Was this resolved?

@emckenzi123 No. The current implementation of euclidean distance transform does not have a gradient defined.

@emckenzi123 If you do not need to run on large images, the code in my notebook from above ( https://colab.research.google.com/drive/1-Fykc6FVi3pUtFx7spP3OAdn267TKwbD ) might work as a loss function. You do need to allow arbitrary real values for pixels (not just binary) in order for the gradients of a distance transform to be meaningful though.

@drebain I tried your posted method (thank you for making that available) however I ran into an OOM error since my images are large.  I looked into the paper you linked to, and found a python version here.  It uses Cython, and I naively tried to just use this code in a training loop's loss, without success (no training of weights).  I went through and tried to replace ops with tensorflow ops to see if I could get something, but that just ended up being really slow.  I should note that if I use the Cython code outside of tensorflow, it's super fast.  I could try to keep working on this, or do you guys think it's actually non-trivial to turn the linked python code into something TF can use in a loss function?

@emckenzi123 I haven't used Cython and TF together before, but I think you could write a wrapper around the code you linked that manually defines the gradient using tf.py_function and tf.custom_gradient. If I'm reading correctly that the code gives you the index of the pixel corresponding to the minimum of d(p,q)+f(q) you should be able to compute the gradient. That would let you keep the Cython implementation I think, while still being differentiable.

Thank you so much for your help.  I defined the gradient to be 2 * sqrt(distance transform), since the derivative of (p-q)**2 is 2 * (p-q).
In case someone in the future runs into something similar, I have used @drebain 's previous colab notebook as a template to demonstrate the code for this.  The link is here


Remote SSH Interpreter

I'm running PyCharm on a windows machine, setup with a remote SSH interpreter on a Linux server.  Autocomplete runs for local windows folder structure, starting with ''; however, all files will be run on the remote Linux machine and should use that folder structure and have autocomplete start with '/'

Hi Andy, this might be very hard to implement as essentially the file path details need to be fetched from the interpreter that's been configured. This interpreter can be Python or PHP or anything else. Also, the intellij plugin SDK needs to provide a means to achieve this which I'm not sure it does. I think this is a very rare use-case to warrant this effort.


Cesium Tiling 1.1 data

I have generated tiling data from cesium ion.
https://github.com/CesiumGS/3d-tiles/tree/draft-1.1/specification/ImplicitTiling
but the viewer is not rendering the data. the issue is unable to read file from subtree structure.
How to bind these placeholder values levels and x y z ?
{
  "asset": {
    "version": "1.1",
    "extras": {
      "ion": {
        "georeferenced": false,
        "movable": true
      }
    }
  },
  "schema": {
    "id": "cesium-tiling-pipeline",
    "classes": {
      "tile": {
        "properties": {
          "tightBoundingBox": {
            "name": "Tight Bounding Box",
            "type": "SCALAR",
            "componentType": "FLOAT64",
            "array": true,
            "count": 12,
            "semantic": "TILE_BOUNDING_BOX"
          }
        }
      }
    }
  },
  "geometricError": 21.495847061053393,
  "root": {
    "boundingVolume": {
      "box": [2.15731143951416, 10.192809104919434, 0.8775157928466797, 12.410633087158203, 0.0, 0.0, 0.0, 12.410633087158203, 0.0, 0.0, 0.0, 12.410633087158203]
    },
    "geometricError": 0.7218289051132697,
    "refine": "REPLACE",
    "content": {
      "uri": "tiles/{level}/{x}/{y}/{z}.glb"
    },
    "metadata": {
      "class": "tile",
      "properties": {
        "tightBoundingBox": [2.15731143951416, 8.006588459014893, -0.9987449645996094, 12.410633087158203, 0.0, 0.0, 0.0, 10.224412441253662, 0.0, 0.0, 0.0, 10.534372329711914]
      }
    },
    "implicitTiling": {
      "subdivisionScheme": "OCTREE",
      "subtreeLevels": 6,
      "availableLevels": 6,
      "subtrees": {
        "uri": "subtrees/{level}/{x}/{y}/{z}.subtree"
      }
    }
  }
}

Yes I didn't add support for implicit tiling yet, you're the first to ask.
I will add it. Any chance you can share a small dataset for testing?

@ebeaufay
implicit tiling datasets.
https://firebasestorage.googleapis.com/v0/b/dive-testing-268508.appspot.com/o/grid.zip?alt=media&token=7520776b-cd5e-474f-a0cc-071a902ac371
https://firebasestorage.googleapis.com/v0/b/dive-testing-268508.appspot.com/o/Sample (1.1).zip?alt=media&token=ca577f13-5d1b-4648-82b5-2de4e5ca6589

Alright I'm working on this but the Implicit tiling is definitely a big complication on the format.
Rant :
They talk about performance improvements but browsers are so performant at decoding json the gains seem minimal compared to the increased complexity. This kind of spec is why the standard is going to stay tied to very specialized and expensive software and OGC keeps wondering why their standards aren't more popular.
Anyhow, thanks for the sample datasets. I'll update this thread when I have something working

Hi @ebeaufay  any updates?

Well I got a second request so it's definitely happening. It's planned for February.

support for OGC3DTiles added in v9.2.15.
Support includes gltf tiles, multiple contents and implicit tiling but not tile metadata. metadata can still be stored in the glb meshes.
On a side note, the stated advantages of 1.1 implicit tiling are:

performance
1 on 1 mapping with other formats that have implicit tiling (cbd,... )
easy to replace tiles.


Minimize and speed up the Collatz conjecture bench

Now that I've had a bit more time to tinker with crepe, I was able to minimize the Collatz conjecture benchmark which, in my opinion, makes it a lot more clear. This also made it a lot faster:
collatz/9               time:   [2.2401 us 2.2420 us 2.2440 us]
                        change: [-50.991% -50.896% -50.813%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) low mild
  5 (5.00%) high mild
  3 (3.00%) high severe
collatz/77031           time:   [37.722 us 37.743 us 37.765 us]
                        change: [-47.089% -46.975% -46.839%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
collatz/9780657630      time:   [125.33 us 125.67 us 125.96 us]
                        change: [-50.623% -50.390% -50.161%] (p = 0.00 < 0.05)
                        Performance has improved.

Nice!! Cool to see that in both cases, the asymptotic time complexity appears to be the same (all are around -50%). I think this means the semi-naive evaluation is probably working.


Support for DocuSign API 2.1

We would like to take advantage of some of the new filtering capabilities available in version 2.1 of the API. When are you guys planning on getting the API updates into the gem?

The API v2.1 Ruby SDK release is currently scheduled for this August.

From: Vilmos Csizmadia<EMAIL_ADDRESS>Sent: Wednesday, July 10, 2019 11:38:32 AM
To: docusign/docusign-ruby-client
Cc: Subscribed
Subject: [docusign/docusign-ruby-client] Support for DocuSign API 2.1 (#23)
We would like to take advantage of some of the new filtering capabilities available in version 2.1 of the API. When are you guys planning on getting the API updates into the gem?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/docusign/docusign-ruby-client/issues/23?email_source=notifications&email_token=ADDJAWTXQMGOVSYNBCKNVIDP6YUCRA5CNFSM4H7SDN6KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6NTASA, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADDJAWRL443AGM2GOOL7CY3P6YUCRANCNFSM4H7SDN6A.

Unfortunately we didn't make the August plan. I'm checking on the current schedule. Internal ticket DCM-3546

Good news, this issue has been fixed with the 3.0.0.rc.1 release of the SDK and gem. The source is available as branch ngopala/v2.1-19.2.02.
It will be released to master and the gem will be updated in the next week.
Please comment on this issue if you spot any issues with the RC (release candidate) or the forthcoming release.
Thanks,
Larry


The file "accept" attribute should handle with "type/*"

Now it only accepts ".ext" way to extensions

It was particulary implemented in #72. It need to add validation of MIME types like in HTML input.
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file
https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types
I think it can be new rule accept with paramenters like in input (audio/* and other) with new validation method for this case.
Add functionality here
Add test here
Add example here and in README
Contribute to development branch


Pretty printing: Quoting in table names

PostgreSQL (and, I think, many other RDBMs) requires table identifiers to be quoted in many cases. But HsSqlPpp does not quote them.
For example, printStatement generates:
select ... from Record r, Submmission s ...
while we need:
select ... from "Record" r, "Submission" s ...
Seems such a quoting might be done with simple patch. In Pretty.lhs replace convDqi definition with following:

convDqi :: SQIdentifier -> Doc
convDqi (SQIdentifier _ is) = hcat $ punctuate (text ".") $ map (text . quot) is
where quot x = """ ++ x ++ """

Thanks for the report!
No thought has been given to the issue of quoted identifiers, even though the parser sort of handles them, so thanks for the patch. I will try to see if there is anything else preventing the proper handling of identifiers that need quoting.
Let me know if you have any other issues.

I've looked into this a little more:
the abstract syntax doesn't record whether an identifier appears quoted. It preserves the case of the identifier (I think). The type check ignores case (which is wrong for quoted identifiers). The pretty printer is unable to tell whether to quote an identifier if it e.g. uses letters only.
So parsing something like 'select * from Reader' could be parsed then pretty printed as either 'select * from Reader' or 'select * from "Reader"'. If the table name in the catalog is e.g. 'reader', then the original sql is fine, but outputting the identifier with quotes would change the meaning of the sql query.
I think the correct solution is to represent quoted and unquoted identifiers explicitly in the ast, (with fixes to the parser, typechecker and pretty printer). This doesn't seem too difficult and I have put it on my todo list.

fixed in this commit:
https://github.com/JakeWheat/hssqlppp/commit/f2339a9e8bb1ea12d9e689ad5f417f6e3d786451
not comprehensively tested, please reopen bug with example if you find something still broken for quoted identifiers


Bug in passing "TestCaseFilter" parameters to "vstest.console.exe" in "VSTest@2" task

Consider this code sample (fabricated for this post) which is part of tests running in Azure DevOps build/release pipeline:
// .NET Framework 4.8 project with referenc to these 2 packages:
//  <package id="MSTest.TestAdapter" version="3.1.1" targetFramework="net48" />
//  <package id="MSTest.TestFramework" version="3.1.1" targetFramework="net48" />

using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace SampleTest
{
    [TestClass]
    public class Tests
    {
        [TestMethod]
        [DataRow(0)]
        [DataRow(1)]
        [DataRow(-1)]
        public void TestIsPositive(int x)
        {
            Assert.IsTrue(x >= 0);
        }
    }
}

When the VSTest@2 task in the pipeline has rerunFailedTests: true, the rerun fails with this message:
Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
From the logs, the executed command which produces this error is like this:
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)"
When I execute this command locally with a slight variation (adding a pipeline to the /TestCaseFilter parameter's start or end) like this:
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)|"
then the vstest executable will run all the 3 tests in the TestIsPositive method and not just the failing one.
Referring to this page TestCase filter I understand that the test cases must be escaped (when there are some characters as in the case of using DataRow(...)] or [DynamicData(...)]) and especially if the tests have URL data that has query parameters in them.
Here are a few variations to try (some will pass some will fail with different reasons):
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)|"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"|FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)|FullyQualifiedName=SampleTest.Tests.TestIsPositive (1)"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive (-1)|FullyQualifiedName=SampleTest.Tests.TestIsPositive (1)|"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive \(-1\)"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive \(-1\)|"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"|FullyQualifiedName=SampleTest.Tests.TestIsPositive \(-1\)"
vstest.console.exe "SampleTest.dll" /TestCaseFilter:"FullyQualifiedName=SampleTest.Tests.TestIsPositive \(-1\)|FullyQualifiedName=SampleTest.Tests.TestIsPositive \(1\)"

Hence, this is to report a bug in which the VSTest@2 task needs to properly escape the string passed in the rerun because as a developer using this task, I have no direct control over the parameters being generated by it and can't enforce escaping myself.
Regards,
George

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 5a378422-4758-62c7-1793-7ce4724079aa
Version Independent ID: 5a378422-4758-62c7-1793-7ce4724079aa
Content: VSTest@2 - Visual Studio Test v2 task
Content Source: task-reference/vstest-v2.md
Service: azure-devops-pipelines
Sub-service: azure-devops-pipelines-tasks
GitHub Login: @steved0x
Microsoft Alias: sdanie

@v-galatrash -- George, thank you for your detailed feedback.  Please this product issue here:

Developer Community

cc @steved0x for awareness
I am closing this issue now.  You are welcome to @ mention me for any followup.  We hope to hear from you again.

Thanks @WilliamAntonRohm . I will post in the Developer Community shortly.


Make nomad-aws modularizable

I had some issues invoking nomad-aws from another root module because it was redefining the AWS provider, which callable modules should not do.
I've thus refactored nomad-aws to remove the AWS provider, and to make it clear that it is a reusable module, I've moved it into shared/modules while leaving the example invocation intact in its original place.

Is the change of directory necessary here? It breaks lots of our docs and I'm not sure what the benefit is

If there are references to that specific directory from outside of this repository, I understand that the doc updates may be a problem at this time. My reasons for refactoring were to:

Avoid nesting example code inside of a module, which could lead to confusion or potentially unwanted execution
Be consistent in placing reusable modules at the same location, in a way that clearly demarcates them as reusable

I will close this PR and create a new one -- likely easier than trying to rewrite this one.


Add storage and network top consumers

Description

Add storage and network top consumers.

Align meaning of CPU consumption for nodes and pods in the top 5 list.

Ref: https://jira.coreos.com/browse/CNV-1157
Screenshot

@mareklibra hi,
a - Please review the metrics.
b - Can we align metrics used in the utilization boards with the ones in top consumers ?

@mareklibra @rawagner @suomiy please review.
wip because not sure about metrics.

Removed WIP

@mareklibra hi,
a- please review
b - do you know who can can know more about what metrics we should use ?


35.17     inlined from 'Jack::JackNetExtSlave::JackNetExtSlave(const char*, int, const char*, jack_slave_t*)' at ../common/JackNetAPI.cpp:541:17,
35.17     inlined from 'jack_net_slave_t* jack_net_slave_open(const char*, int, const char*, jack_slave_t*, jack_master_t*)' at ../common/JackNetAPI.cpp:1054:73:
35.17 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/bits/stdio2.h:64:35: note: '__builtin___snprintf_chk' output 2 or more bytes (assuming 66) into a destination of size 64
35.17    64 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
35.17       |          ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
35.17    65 |                                    __bos (__s), __fmt, __va_arg_pack ());
35.17       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
35.17
35.17 [141/177] Compiling common/netjack.c
35.45 [142/177] Compiling common/JackProxyDriver.cpp
35.48 [143/177] Compiling linux/alsa/hdsp.c
35.65 [144/177] Compiling linux/alsa/JackAlsaDriver.cpp
35.92 [145/177] Compiling linux/alsa/alsa_seqmidi.c
36.35 [146/177] Compiling linux/alsa/alsa_rawmidi.c
36.51 [147/177] Compiling linux/alsa/alsa_midi_jackmp.cpp
37.71 [148/177] Compiling linux/alsa/generic_hw.c
38.16 [149/177] Compiling common/memops.c
39.26 [150/177] Linking build/common/libjackserver.so
41.05 [151/177] Compiling linux/alsa/hammerfall.c
41.19 [152/177] Compiling linux/alsa/alsa_driver.c
41.26 [153/177] Compiling linux/alsa/ice1712.c
41.41 [154/177] Compiling linux/alsarawmidi/JackALSARawMidiOutputPort.cpp
41.43 [155/177] Compiling linux/alsarawmidi/JackALSARawMidiUtil.cpp
41.43 ../linux/alsa/JackAlsaDriver.cpp: In function 'jack_driver_param_constraint_desc_t* enum_alsa_devices()':
41.43 ../linux/alsa/JackAlsaDriver.cpp:625:68: warning: '%d' directive output may be truncated writing between 1 and 11 bytes into a region of size between 0 and 127 [-Wformat-truncation=]
41.43   625 |                 snprintf(device_id.str, sizeof(device_id.str), "%s,%d", card_id.str, device_no);
41.43       |                                                                    ^~
41.43 ../linux/alsa/JackAlsaDriver.cpp:625:64: note: using the range [-2147483648,<PHONE_NUMBER>] for directive argument
41.43   625 |                 snprintf(device_id.str, sizeof(device_id.str), "%s,%d", card_id.str, device_no);
41.43       |                                                                ^~~~~~~
41.43 In file included from /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/stdio.h:936,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/cstdio:42,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/ext/string_conversions.h:45,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/bits/basic_string.h:4097,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/string:54,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/bits/locale_classes.h:40,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/bits/ios_base.h:41,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/ios:44,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/ostream:40,
41.43                  from /opt/ct-ng/x86_64-ffbuild-linux-gnu/include/c++/13.2.0/iostream:41,
41.43                  from ../linux/alsa/JackAlsaDriver.cpp:23:
41.43 In function 'int snprintf(char*, size_t, const char*, ...)',
41.43     inlined from 'jack_driver_param_constraint_desc_t* enum_alsa_devices()' at ../linux/alsa/JackAlsaDriver.cpp:625:25:
41.43 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/bits/stdio2.h:64:35: note: '__builtin___snprintf_chk' output between 3 and 140 bytes into a destination of size 128
41.43    64 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
41.43       |          ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
41.43    65 |                                    __bos (__s), __fmt, __va_arg_pack ());
41.43       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
41.43
41.43 [156/177] Compiling linux/alsarawmidi/JackALSARawMidiDriver.cpp
42.31 [157/177] Compiling linux/alsarawmidi/JackALSARawMidiSendQueue.cpp
42.36 [158/177] Compiling linux/alsarawmidi/JackALSARawMidiReceiveQueue.cpp
42.38 [159/177] Compiling linux/alsarawmidi/JackALSARawMidiPort.cpp
42.46 [160/177] Compiling linux/alsarawmidi/JackALSARawMidiInputPort.cpp
42.58 [161/177] Compiling jack.pc.in
42.97 ../linux/alsarawmidi/JackALSARawMidiPort.cpp: In member function 'bool Jack::JackALSARawMidiPort::TriggerQueueEvent()':
42.97 ../linux/alsarawmidi/JackALSARawMidiPort.cpp:250:27: warning: 'c' may be used uninitialized [-Wmaybe-uninitialized]
42.97   250 |     ssize_t result = write(fds[1], &c, 1);
42.97       |                      ~~~~~^~~~~~~~~~~~~~~
42.97 In file included from /opt/ffbuild/include/alsa/asoundlib.h:31,
42.97                  from ../linux/alsarawmidi/JackALSARawMidiPort.h:23,
42.97                  from ../linux/alsarawmidi/JackALSARawMidiPort.cpp:24:
42.97 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/unistd.h:369:16: note: by argument 2 of type 'const void*' to 'ssize_t write(int, const void*, size_t)' declared here
42.97   369 | extern ssize_t write (int __fd, const void *__buf, size_t __n) __wur;
42.97       |                ^~~~~
42.97 ../linux/alsarawmidi/JackALSARawMidiPort.cpp:249:10: note: 'c' declared here
42.97   249 |     char c;
42.97       |          ^
42.97
43.70 ../common/JackNetAdapter.cpp: In constructor 'Jack::JackNetAdapter::JackNetAdapter(jack_client_t*, jack_nframes_t, jack_nframes_t, const JSList*)':
43.70 ../common/JackNetAdapter.cpp:92:28: warning: 'char* __builtin_strncpy(char*, const char*, long unsigned int)' output may be truncated copying 64 bytes from a string of length 127 [-Wstringop-truncation]
43.70    92 |                     strncpy(fParams.fName, param->value.str, JACK_CLIENT_NAME_SIZE);
43.70       |                            ^
43.70
43.72 [162/177] Linking build/jackd
43.74 [163/177] Linking build/common/netmanager.so
43.74 [164/177] Linking build/common/profiler.so
43.75 [165/177] Linking build/common/libjacknet.so
43.76 [166/177] Linking build/common/netadapter.so
43.78 [167/177] Linking build/jack_loopback.so
43.80 [168/177] Linking build/jack_netone.so
43.96 [169/177] Linking build/jack_proxy.so
44.38 [170/177] Linking build/jack_dummy.so
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'jack_process':
44.50 ../linux/alsa/alsa_rawmidi.c:862:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   862 |         write(str->wake_pipe[1], &r, 1);
44.50       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'alsa_rawmidi_stop':
44.50 ../linux/alsa/alsa_rawmidi.c:372:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   372 |         write(midi->in.wake_pipe[1], &c, 1);
44.50       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c:373:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   373 |         write(midi->out.wake_pipe[1], &c, 1);
44.50       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c:374:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   374 |         write(midi->scan.wake_pipe[1], &c, 1);
44.50       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'alsa_rawmidi_start':
44.50 ../linux/alsa/alsa_rawmidi.c:349:17: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   349 |                 write(midi->in.wake_pipe[1], &c, 1);
44.50       |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c:355:17: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   355 |                 write(midi->in.wake_pipe[1], &c, 1);
44.50       |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c:356:17: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   356 |                 write(midi->out.wake_pipe[1], &c, 1);
44.50       |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'scan_thread':
44.50 ../linux/alsa/alsa_rawmidi.c:788:25: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   788 |                         read(wakeup.fd, &c, 1);
44.50       |                         ^~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'midi_thread':
44.50 ../linux/alsa/alsa_rawmidi.c:926:25: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result]
44.50   926 |                         read(pfds[0].fd, &c, 1);
44.50       |                         ^~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'midi_port_open_jack.isra':
44.50 ../linux/alsa/alsa_rawmidi.c:459:27: warning: '%s' directive output may be truncated writing up to 63 bytes into a region of size between 54 and 117 [-Wformat-truncation=]
44.50   459 |                     "%s%s %s (%s)",
44.50       |                           ^~
44.50 In file included from /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/stdio.h:936,
44.50                  from /opt/ffbuild/include/alsa/asoundlib.h:32,
44.50                  from ../linux/alsa/alsa_rawmidi.c:31:
44.50 In function 'snprintf',
44.50     inlined from 'midi_port_open_jack.isra' at ../linux/alsa/alsa_rawmidi.c:457:4:
44.50 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/bits/stdio2.h:64:10: note: '__builtin___snprintf_chk' output between 17 and 144 bytes into a destination of size 128
44.50    64 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
44.50       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50    65 |                                    __bos (__s), __fmt, __va_arg_pack ());
44.50       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'scan_device':
44.50 ../linux/alsa/alsa_rawmidi.c:415:9: warning: '__builtin_strncpy' specified bound 64 equals destination size [-Wstringop-truncation]
44.50   415 |         strncpy(port->device_name, snd_rawmidi_info_get_name(info), sizeof(port->device_name));
44.50       |         ^
44.50 ../linux/alsa/alsa_rawmidi.c:420:57: warning: '%s' directive output may be truncated writing up to 63 bytes into a region of size between 44 and 60 [-Wformat-truncation=]
44.50   420 |         snprintf(port->name, sizeof(port->name), "%s %s %s", port->id.id[2] ? "out":"in", port->dev, name);
44.50       |                                                         ^~
44.50 In function 'snprintf',
44.50     inlined from 'midi_port_init' at ../linux/alsa/alsa_rawmidi.c:420:2,
44.50     inlined from 'scan_port_add' at ../linux/alsa/alsa_rawmidi.c:709:2,
44.50     inlined from 'scan_port_update' at ../linux/alsa/alsa_rawmidi.c:693:10,
44.50     inlined from 'scan_device' at ../linux/alsa/alsa_rawmidi.c:676:3:
44.50 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/bits/stdio2.h:64:10: note: '__builtin___snprintf_chk' output between 5 and 84 bytes into a destination of size 64
44.50    64 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
44.50       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50    65 |                                    __bos (__s), __fmt, __va_arg_pack ());
44.50       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50 ../linux/alsa/alsa_rawmidi.c: In function 'scan_cycle':
44.50 ../linux/alsa/alsa_rawmidi.c:511:50: warning: '%d' directive output may be truncated writing between 1 and 11 bytes into a region of size between 0 and 63 [-Wformat-truncation=]
44.50   511 |                 snprintf(name, sizeof(name), "%s %d", port->name, num);
44.50       |                                                  ^~
44.50 In function 'snprintf',
44.50     inlined from 'midi_port_open' at ../linux/alsa/alsa_rawmidi.c:511:3,
44.50     inlined from 'scan_port_open' at ../linux/alsa/alsa_rawmidi.c:730:8,
44.50     inlined from 'scan_cycle' at ../linux/alsa/alsa_rawmidi.c:627:12:
44.50 /opt/ct-ng/x86_64-ffbuild-linux-gnu/sysroot/usr/include/bits/stdio2.h:64:10: note: '__builtin___snprintf_chk' output between 3 and 76 bytes into a destination of size 64
44.50    64 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
44.50       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50    65 |                                    __bos (__s), __fmt, __va_arg_pack ());
44.50       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44.50
44.51 [171/177] Linking build/jack_alsa.so
44.70 /opt/ct-ng/lib/gcc/x86_64-ffbuild-linux-gnu/13.2.0/../../../../x86_64-ffbuild-linux-gnu/bin/ld: /opt/ffdl/scripts.d/50-jack.sh/build/jack_alsa.so: version node not found for symbol<EMAIL_ADDRESS>44.70 /opt/ct-ng/lib/gcc/x86_64-ffbuild-linux-gnu/13.2.0/../../../../x86_64-ffbuild-linux-gnu/bin/ld: failed to set dynamic section sizes: bad value
44.70 collect2: error: ld returned 1 exit status
44.70
44.78 ../common/JackNetDriver.cpp: In function 'Jack::JackDriverClientInterface* Jack::driver_initialize(JackLockedEngine*, JackSynchro*, const JSList*)':
44.78 ../common/JackNetDriver.cpp:787:32: warning: 'char* __builtin_strncpy(char*, const char*, long unsigned int)' output may be truncated copying 64 bytes from a string of length 127 [-Wstringop-truncation]
44.78   787 |                         strncpy(net_name, param->value.str, JACK_CLIENT_NAME_SIZE);
44.78       |                                ^
44.78
45.78 Waf: Leaving directory `/opt/ffdl/scripts.d/50-jack.sh/build'
45.79 Build failed
45.79  -> task in 'alsa' failed with exit status 1 (run with -v to display more information)
------
Dockerfile:167
--------------------
 165 |     FROM layer-45 AS layer-50-jack
 166 |     ENV SELF="scripts.d/50-jack.sh"
 167 | >>> RUN --mount=src=scripts.d/50-jack.sh,dst=/stage.sh --mount=src=/,dst=$FFBUILD_DLDIR,from=<IP_ADDRESS>:64647/local/dl_cache:dlcache_e22660b951173f253bc44896e6191ef9b2c850fa75dd63930e585623eb8f259e,rw run_stage /stage.sh
 168 |     FROM layer-45 AS layer-50-kvazaar
 169 |     ENV SELF="scripts.d/50-kvazaar.sh"
--------------------
ERROR: failed to solve: process "/bin/sh -c run_stage /stage.sh" did not complete successfully: exit code: 1
+ rm -f /tmp/tmp.PzgZRybPBU.toml

maybe I made a mistake somewhere?

You cannot statically link alsalib either, so I guess that's what's broken there.


[Story] Design an organization card for the organization search page

We are going to create a search page for organizations and need to create a component that represent an organization. We have an org "card" that uses the avatar component, but this format likely restrict too much the amount of information we want to put in that item. The goal of this ticket is actually to:

[ ] List the information that we want to include for each org item listed in the org search page
[ ] If the existing org "card"/avatar does not meet the above requirement, design a new component in Figma

First design idea for organization card:

The footer would be the total number of challenges supported, let it be by organization, sponsorship, and/or data contribution.

The card looks great! Here is my feedback:

"+N other users": replace "users" by "members" in case we want to add people who are not yet users. "members" would also work if we decide to list only registered users.
Description: What about removing this description to simplify the data collection and declutter the search page? If the user don't know the org, it's a good opportunity for them to learn more about this org by visiting their page on OC.
In the future we could show the tags directly assigned to the org or indirectly linked to it by looking at the tag that appear the most frequently on the pages of the challenges the org supports.

Thanks for the design. I like you have included # of members and # of connected challenges, which I think they are helpful info to explore organizations. One concern I have is whether we could get the banner info of organizations. I feel using the avatar is enough for the organization.

Thank you for feedback!   I mostly tried to stick with the same layout & design for the challenge card, but I agree with you both regarding the descriptions and banner pictures.  Here's the next iteration with implementations of the feedback:

@vpchung Can you please add a link to the Figma doc before closing the issue?

Link to design on Figma
Closing now!


Add: PKCE flow for android native

Signed-off-by: nishihere19<EMAIL_ADDRESS>Fixes #
Short description of the Issue.
PKCE for android apps
Proposed Changes

Added the PKCE flow for android apps
The query params are optional and authorization works normally for the web clients.

Sure, will implement.

others looks fine, can merge after these changes.


Orbit the camera around an object sample

@tschie here is the sample for the orbit camera controls ready for review!

@tschie this sample has been reworked and is ready for review!

@tschie these latest commits take into account your additions (thanks for that) and some additional tweaks I made to them. Could you take a look please, we should be nearly there now :)

@Rachael-E In the Cockpit view, the heading slider stops at the min/max range. But in the Center view, the slider continues past the min/max range. I think I prefer the former. Either way, they should probably be consistent.


Update Project.toml

New version, @timholy  I wonder if we should start with 1.0.0 because I am serious about maintaining this for ImageComponentAnalysis.jl

@zygmuntszpak  I think after https://github.com/JuliaImages/Polygon.jl/pull/4, it's ready to register again pls check once again though if I missed out anything


Update lower boundary in Python pip pyproject when bumping versions #6631

What are you trying to accomplish?
The goal of this PR is to correct the behavior of the versioning strategy in Dependabot for Python dependencies. Specifically, when using the "increase" strategy.
Why: The current behavior was widening the range (e.g., from >=8,<9 to >=8,<10), which is not the intended behavior. Instead, it should keep lower bound version as the new latest version and just move forward the upper boundary version (e.g., from >=8,<9 to >=9,<10) as mentioned Documentation for Versioning Strategies
Issues this affects or fixes:

Addresses the issue reported in #6631.
First noticed at #6625 (comment).

Anything you want to highlight for special attention from reviewers?
The changes involve modifying the requirements for the pip ecosystem to ensure correct behavior for the "increase" strategy. Although the issue suggests above major versioning (e.g. >=8,<9), our "increase" strategy mandates that the lower bound is always  to be the latest version (e.g. >=8.0.1,<9). This approach aligns with the guidelines provided in the Dependabot documentation. Therefore, the changes have been made accordingly.
How will you know you've accomplished your goal?

Updated smoke tests should run successfully.
https://github.com/dependabot/smoke-tests/pull/212
Testing on https://github.com/dsp-testing/dependabot-updates-issues-6631 should create PR according to the changes

Checklist

[x] I have run the complete test suite to ensure all tests and linters pass.
[x] I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
[x] I have written clear and descriptive commit messages.
[x] I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
[x] I have ensured that the code is well-documented and easy to understand.

Smoke tests are updated: https://github.com/dependabot/smoke-tests/pull/212/files
So the PR needs to be merged to make smoke tests successful.

Tagging @jeffwidman and @deivid-rodriguez for PR reviews too

@kbukum1 The python CI test is failing with your last commit

@kbukum1 The python CI test is failing with your last commit, so I have revoked the approval.

Fixing it

ers w

fixed

Also, I think it's fine to treat this a bug and change how Dependabot updates manifests here.
However, as pointed out by @sanderr here, it'd be nice to implement the widen strategy too to allow keeping the same behavior as before this PR, so that people considering this bug a feature, can still get the behavior they want :)

Thanks @deivid-rodriguez!
I was going to take a look at this--just running late--but @kbukum1 I'll wait until you put up another follow-on PR addressing some of this feedback and then take a deeper look myself.

I am currently preparing a doc for the current increase and widening versioning strategies. After going through reviews here I plan to create a new issue that is going to be used for finalizing both versioning strategies. Is that sound ok @deivid-rodriguez , @jeffwidman

The changes of this PR are reverted in #10194. The versioning strategy needs to be re-planned, and a solution will be implemented according to the new plan.


CustomFunc should allow Ptr types as well

CustomFunc are helpful in creating custom clone functions. However, it doesn't work well with Ptrs. Because when adding *Struct, it gets resolved to Struct and similarly when looking up from the map, only the Struct is used.
There are some cases, when custom pointers would be helpful.
For example, in my case, under certain circumstances, I want the ptrs to reuse the same reference and sometimes not.
I am working on a model checker, where the spec might say, multiple alternate paths a process can take, and we would want to explore both the paths. So, from the parent, I would create separate clone of entire variable spaces for each fork, but within each fork I would want to reuse the variable.
Marking it opaque pointer doesn't help, as I would want to actually clone in some cases. So, the ideal solution would be to make allocator.SetCustomFunc(...) to not dereference the Ptr type, and when checking ptrs, and use the custom func when present, before dereferencing the ptr

SetCustomFunc is designed to set custom clone function for a struct, not other types. To support custom function for any type may take too much impact on performance.
One possible way to implement the feature you mentioned could be that you can consider to define a new struct to wrap the pointer and implement the logic on this new struct.
Something like following.
type MyPointer struct {
    *ActualPointer
}

Here is a workable sample for you. Hope it can help you.
type T struct {
	shouldClone bool
	data        []string
}

type Pointer struct {
	*T
}

values := map[string]Pointer{
	"shouldClone": {
		T: &T{
			shouldClone: true,
			data:        []string{"a", "b", "c"},
		},
	},
	"shouldNotClone": {
		T: &T{
			shouldClone: false,
			data:        []string{"a", "b", "c"},
		},
	},
}
clone.SetCustomFunc(reflect.TypeOf(Pointer{}), func(allocator *clone.Allocator, old, new reflect.Value) {
	p := old.Interface().(Pointer)

	if p.shouldClone {
		np := allocator.Clone(old).Interface().(Pointer)

		// Update the cloned value to make the change very obvious.
		np.data = append(np.data, "cloned")
		new.Set(reflect.ValueOf(np))
	} else {
		new.Set(old)
	}
})

cloned := clone.Clone(values).(map[string]Pointer)
fmt.Println(cloned["shouldClone"].data)
fmt.Println(cloned["shouldNotClone"].data)

// Output:
// [a b c cloned]
// [a b c]

Live demo: https://go.dev/play/p/3V6VoHPRabd

Thanks. I did consider this solution, but that would work if clone is the only thing the application does. But there are lot more stuff that goes on, and having to use a wrapper in place of a pointer (Note type alias doesn't work), at every code path using this is a pain.
I would love to hear more on why you think there will be a significant performance hit for supporting custom function for pointers to struct? Are you worried, supporting custom functions to pointers to struct would impact non-users of the feature as well or would it slow down only when the feature is used?

A quick hack. https://github.com/huandu/go-clone/pull/24
I did not implement this for other cases like cloneSlowly or other variants. I would love to implement it if you are okay with this approach. Do let me know.

why you think there will be a significant performance hit for supporting custom function for pointers to struct?
If the SetCustomFunc supports both struct and pointer, it means the method Allocattor.loadStructType will be called for every pointer visited. As you can see, it uses several reflect methods to do its job and should not be called many times. I'm aware of calling this method too many times.

A quick hack. #24
I'll review it later, maybe next Tue. I'm on vacation now. If it's possible, can you please tell me more about your use case with some key code? It can help me to find out the best soluttion.

I want a way to retain the relationship between the pointer values, when cloning. From my understanding cloneSlowly function does that for a single clone operation. For me, I need a way to be able to use the same visited graph if possible across multiple invocations.

@jayaprabhakar I got your point. I think it's OK to add some features in Allocator to provide more powerful APIs with some performance penalty for complex use cases.
I'll submit a PR soon to address your need.

Thanks a lot.
For additional context on what I am trying to build - I am implementing a new model checker - the first step of it is state space exploration.
For example: https://fizzbee.io/tutorials/probabilistic-modeling/#simulating-a-6-sided-die-with-a-fair-coin
(Simulate a fair 6-sided die with a fair coin, fizzbee will explore every possible path.)
To do that, every fork points where we have to choose a path between multiple options, I will deep clone the entire state-space, and evaluate each path separately one after another until no new state is found.
Unfortunately cloning the state-space is not that simple. For example: a parent struct, has multiple fields. I want to clone a few of them, but not all. Each of these fields might be struct ptrs that themselves have some fields that should be cloned and some don't. In addition to that, I would have to preserve the relationship between the fields. If two references/ptrs point to same object before clone, then after cloning, both the ptrs should refer to a common new cloned object.
To make it even harder, I use starlark to maintain python-like syntax. In starlark, after executing an expression, starlark freezes the value (so it doesn't allow any new modification). This cannot be simply cloned but I have to use custom functions.
With these, if I clone the container/parent/root object with clone.Slowly(), it won't work simply, because I need to add custom function to skip a few fields. But those individual fields must be deep cloned as well. But it doesn't have access to the same visitor graph :( So, clone.Slowly here will not be sufficient)

Hi @huandu, did you get a chance to work on this? Or can you give some guidance on how to do it?

@jayaprabhakar Sorry for the delay. I'm super busy these days and have no time to finish this change. I'll try to spare some time on this weekend.

No problem, let me know if there is something I can do to make it easy for you to finish it.
I am facing an issue with Cloning a specific object type. I use another library starlark extensively, and when I do clone, for one of the types alone, the clone is not working.
m1 := starlark.NewDict(5)
m2 := clone.Clone(m1).(*starlark.Dict)

m2.SetKey(starlark.MakeInt(2), starlark.MakeInt(1))
fmt.Println("added to clone", m1, m2)

https://go.dev/play/p/SRQMnBef26Q
In this case, value added to m2 is ignored. I couldn't figure out exactly what causes this. The code for the library is here,
https://github.com/google/starlark-go/blob/046347dcd1044f5e568fcf64884b0344f27910c0/starlark/value.go#L841
Is there some specific feature that is causing this issue.
I can try again using Custom Fn but the issue would be with deduplication. I want to use clone.Slowly(), and I wanted to let you know so if this is a missing feature or a bug in the clone implementation.


JSON exception on run

I'm seeing a JSON exception which appears to be raised when parsing the list of allowable TLDs. Multiple different machines, clean build etc, can anyone see what's going on here?
[28/12/2023 10:02:12] Found 1 files:
[28/12/2023 10:02:12]   .csv   1 files: 0.4 Gb
[28/12/2023 10:02:12] Output will be saved to "D:\Temp\File.txt".
[28/12/2023 10:02:12] Report will be saved to "report.txt".
Press ANY KEY to continue ['Q' to Quit; 'I' for info]:
[28/12/2023 10:02:13] Extracting...
[28/12/2023 10:02:13] Reading "D:\Temp\file.csv" [0.4 Gb]
[28/12/2023 10:02:13] An error occurred while parsing 'D:\Temp\file.csv'L3: 'N' is invalid after a single JSON value. Expected end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 12623.
[28/12/2023 10:02:13] An error occurred while parsing 'D:\Temp\file.csv'L2: 'N' is invalid after a single JSON value. Expected end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 12623.
[28/12/2023 10:02:13] An error occurred while parsing 'D:\Temp\file.csv'L4: 'N' is invalid after a single JSON value. Expected end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 12623.
[28/12/2023 10:02:13] An error occurred while parsing 'D:\Temp\file.csv'L5: 'N' is invalid after a single JSON value. Expected end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 12623.
Continue? [y/n]:

Using the JsonSerializer to read csv?
Not sure exactly where JSON would be failing here. The only place it's used is to cache the IANA TLD list so that it's not hit every time the program is run.

@alirobe
JsonSerializer isn't used to read the .csv itself, it's an exception bubbled up while using the .csv parser

It may also be that the cached tld.json is simply corrupted somehow, if you want to check yours?
I'll see about adding some more checks to it

I must be doing it wrong as I do not see an error.
[28/12/2023 8:24:02 PM] Read 9,925,000 lines from "TestBreachDataV2.txt"
[28/12/2023 8:24:02 PM] Read 9,950,000 lines from "TestBreachDataV2.txt"
[28/12/2023 8:24:02 PM] Read 9,975,000 lines from "TestBreachDataV2.txt"
[28/12/2023 8:24:03 PM] Read 10,000,000 lines from "TestBreachDataV2.txt"
[28/12/2023 8:24:03 PM] Finished reading files
[28/12/2023 8:24:03 PM] Extraction time: 1.2m
[28/12/2023 8:24:03 PM] Addresses extracted: 10,000,000
[28/12/2023 8:24:03 PM] Read lines total: 10,000,000
[28/12/2023 8:24:03 PM] Read lines rate: 134,473/s
[28/12/2023 8:24:03 PM] Saving to disk..
[28/12/2023 8:24:28 PM] Addresses saved to addresses_output.txt
[28/12/2023 8:24:28 PM] Report saved to report.txt
C:\Users\steph\Downloads\EmailAddressExtractor-main\src\bin\Debug\net8.0\AddressExtractor.exe (process 81092) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

Thanks folks, seems more stable after the latest PR. I'm travelling at the moment so hard to devote time to properly debug, should have added that it was running ok in debug mode from VS but failing when directly running it via the exe. Anyway, let's see how this goes now, thanks all 😊


A question about preprocessing

Hi,
I have some problems in preprocessing.
I download the data cnn_stories_tokenized and dm_stories_tokenized, however, it's all *.story. In your preprocess.py, it requires *.json as input, can you provide the code that transform *.story to *.json? I meet some problem in reading the sample json with you load_json function in data_builder.py. Appreciated!

please see sample json file for reference. You can generate them by using json as outputFormat in corenlp.

Is the generation of JSON file performed by:
"python preprocess.py -mode format_to_lines -raw_path RAW_PATH -save_path JSON_PATH -map_path MAP_PATH -lower " ?
Seems that this command is not generating as well.

Thanks for reply. I will try it.

Hi,
Please check the updated instruction for preprocessing the data.
It should work fine now.

It works now, appreciate!

python preprocess.py -mode format_to_lines -raw_path RAW_PATH -save_path JSON_PATH -map_path MAP_PATH -lower
This command returns me nothing, and no output files are generated as well. What error could it be?


Docs update

Grammar, reformatting & implements note in DNET issue #2029

Horrid commit history, can't resolve in any easy way unfortunately

Waiting on @lhjt


House Price Prediction

Hello! I'm a GSSoC'22 contributor. I would like to contribute to this repository by making a project for Predicting the House Prices using Linear Regression. Please assign me this issue.

Hello! I'm a GSSoC'22 contributor. I would like to contribute to this repository by making a project for Predicting the House Prices using Linear Regression. Please assign me this issue.

hey @Shweta2024 Please choose other open issues.


Implement Local Phone Storage

We need to be able to store things like Kp values, personal settings, etc.

Alex and Dustin have confirmed that local storage is working, but we need to restructure how it is stored.


What is license of package

Question
The license file is missing

according to https://github.com/react-native-community/react-native-masked-view/blob/master/package.json
it is MIT

Hi @FonDorn
I see you are listed as an author of this project inside https://github.com/react-native-community/react-native-masked-view/blob/master/package.json
Is it possible to explicitly add a license file with your name and/or any other names needed in the copyright?


⚠ Bofrost.lu has degraded performance

In 7f3ba75, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 884 ms

Resolved: Bofrost.lu performance has improved in eaecd1e after 7 minutes.


Trigger point of object detection in tensorflow lite

I am doing object detection using tensorflow  lite. My code detects the object & add a label with it & also display the label with it, but after that, I want to display the label of an object which is detected separately.
& I want that point (trigger point) in code from where the label is displayed .
can you please help me to find out code .
I am giving link of  my source code :
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/android

@chiragmak10 Could you mention more details about the issue and its context. Please provide as many details as possible to find a solution faster. Also, I noticed that you forgot to provide link to your source code. The link you provided is from TF Lite examples. Thanks!

@chiragmak10 If your referring to the label on the tracked output, its being set at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/android/app/src/main/java/org/tensorflow/demo/tracking/MultiBoxTracker.java#L188

Closing this support issue. Please post this kind of support questions at Stackoverflow. Thanks!


List: Support dynamic height elements

Dynamic height elements:

[x] Have a provisional height, provided by the virtual delegate
[x] Get an accurate height value when they actually get rendered
[ ] Can get their height value invalidated when the list's width changes

This comes at a performance cost.
This will be useful for the settings editor @roblourens  as well as the debug console @isidorn .

🎉 🎉 🎉


Not run on Windows

There appear to be two separate issues:

The yaml file in secrets.ini.example is incorrect, and
Windows uses cp1252 by default, which breaks when handling non-ascii characters.

For the first issue, remove the quotes around the string.  What was happening for me is that an extra tick was getting added to the string.  That is, if my key was abc123, then python was calling the API with my key as abc123'.  Using the following, unquoted template fixed the issue:
# Rename this file to 'secrets.ini'.

[API_KEYS]
GOOGLE_MAPS = API_Key

For the second, use the -X utf8=1 argument:
python -X utf8=1 fetch.py --directory output/england/london --rating 4.5 --reviews 5000 --operator and --query "attractions in London"

@xobs - I have updated the secrets.ini.example and removed the quotations. For the second issue, I'm not sure where the non-ASCII characters come from. Perhaps it's the text from the Wikipedia article. Can you confirm if the issue still persists if you remove the print statement?

@Suleman-Elahi you might want to double-check you are using Python 3.6 or above.

I'm using Python 3.7.3.  Here is the output when I try to search for something in Tiblisi, Georgia:
❯ python fetch.py --directory output/georgia/tiblisi --rating 4.5 --reviews 5000 --operator and --query "attractions in tiblisi"
Attractions In Tiblisi -> ['National Botanical Garden of Tbilisi city', (41.685574, 44.8029141), 'tourist_attraction, park, point_of_interest, establishment', 4.6, '12 Bambis Rigi Street, Tbilisi, Georgia', "Tbilisi (English:  tə-bih-LEE-see, tə-BIL-ih-see; Georgian: თბილისი [tʰbilisi] (listen)), in some countries also still known by its pre-1936 international designation Tiflis ( TIF-lis), is the capital and the largest city of Georgia, lying on the banks of the Kura River with a population of approximately 1.5 million people. Founded in the 5th century AD by Vakhtang I of Iberia, since then Tbilisi served as the capital of various Georgian kingdoms and republics. Between 1801 and 1917, then part of the Russian Empire, Tbilisi was the seat of the Imperial Viceroy, governing both Southern and Northern Caucasus.Because of its location on the crossroads between Europe and Asia, and its proximity to the lucrative Silk Road, throughout history Tbilisi was a point of contention among various global powers. The city's location to this day ensures its position as an important transit route for various energy and trade projects. Tbilisi's diverse history is reflected in its architecture, which is a mix of medieval, neoclassical, Beaux Arts, Art Nouveau, Stalinist and the Modern structures.Historically, Tbilisi has been home to people of multiple cultural, ethnic, and religious backgrounds, though it is currently overwhelmingly Eastern Orthodox Christian. Its notable tourist destinations include cathedrals Sameba and Sioni, Freedom Square, Rustaveli Avenue and Agmashenebeli Avenue, medieval Narikala Fortress, the pseudo-Moorish Opera Theater, and the Georgian National Museum.The climate in Tbilisi mostly ranges between 30°C to 18°C.", 'https://en.wikipedia.org/wiki/Tbilisi', 6786]
Traceback (most recent call last):
  File "fetch.py", line 112, in <module>
    writer.writerow(data)
  File "D:\Software\Miniconda3\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0259' in position 189: character maps to <undefined>
smcro@CUBOID   yosys-bld  D:\Code\trip-planner   master ≢                                                                                 [09:08]
❯ python --version
Python 3.7.3
smcro@CUBOID   yosys-bld  D:\Code\trip-planner   master ≢                                                                                 [09:09]
❯

The following diff fixes the issue:
diff --git a/fetch.py b/fetch.py
index 778a8af..07cc9f0 100644
--- a/fetch.py
+++ b/fetch.py
@@ -60,7 +60,7 @@ filename = ' '.join([q.capitalize() for q in query])
 wikipedia.set_lang(args.language)

 columns = ['name', 'coordinates', 'types', 'rating', 'formatted address', 'summary', 'url', 'reviews']
-with open(args.directory + f'/{filename}.csv', 'w') as out_file:
+with open(args.directory + f'/{filename}.csv', 'w', encoding='utf-8') as out_file:
        writer = csv.writer(out_file, delimiter='|')
        writer.writerow(columns)
        for place in places:

@Suleman-Elahi you might want to double-check you are using Python 3.6 or above.

Yeah, my Python version is 3.7.1. I had the same issue on Linux though.

@Suleman-Elahi I assume you mean you had the same No results found for query error, in which case it's probably caused by quoting your API key inside settings.yaml.  Remove the quote and see if that fixes it.

Yeah, thanks man. It is working now.

encoding='utf-8'

@xobs Thanks for proposing the fix. Pushed into master in: https://github.com/adl1995/trip-planner/commit/22ba61ca4cab5bbd561e620819cb0be1b9d966e8


fix: Typos in log messages and method names

Summary
Changes

Fixed typos in log messages and updated terms like ASGs & self-managed consistently across methods.
Updated depricatedApi to deprecatedApi in method name, arguments and file name.
Simplified null check on presence of Managed Nodegroup

User experience
No changes
Checklist
If your change doesn't seem to apply, please leave them unchecked.

[X] I have performed a self-review of this change
[X] Changes have been tested
[ ] Changes are documented

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Codecov Report
Base: 10.89% // Head: 10.89% // No change to project coverage :thumbsup:

Coverage data is based on head (85debad) compared to base (9e94a28).
Patch coverage: 4.47% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #28   +/-   ##
=======================================
  Coverage   10.89%   10.89%
=======================================
  Files          10       10
  Lines        1974     1974
=======================================
  Hits          215      215
  Misses       1759     1759

Impacted Files
Coverage Δ

eksupgrade/src/eks_get_image_type.py
13.72% <ø> (ø)

eksupgrade/src/eksctlfinal.py
10.50% <0.00%> (ø)

eksupgrade/src/self_managed.py
14.85% <ø> (ø)

eksupgrade/starter.py
11.05% <0.00%> (ø)

eksupgrade/src/preflight_module.py
3.60% <1.88%> (ø)

eksupgrade/src/boto_aws.py
10.16% <100.00%> (ø)

eksupgrade/src/k8s_client.py
20.19% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Media Type parameter spec lacks example of multiple values

We noticed a minor point of confusion around the Media Type parameter formatting for extensions/profiles:
The examples in section 5.4.1 show a space between the ; and ext, but the examples in the atomic operations extension spec all exclude this space. Are both options valid?
Section 5.4 talks about spaces in the extension, but doesn't not provide an example of the space separated list of URI (and where the quotes mentioned in the Note belong), so it is hard to interpret the correct syntax, especially for multiple values of a parameter.
I suspect that the post ; space is optional but that a correct example of the a parameter with multiple values would be `ext="http://uri.ex/1 http://uri.ex/2".

possible duplicate of #1768 ?

Yes. But as the question came up twice, let's keep this one open and add an example in the spec. Maybe even an example in addition shown an Accept header listing JSON API mime type two times with a different set of profiles. A PR is welcome.

@Roanmh I created #1773 to add an example. Would appreciate your review if you have time.


the broken path for the release package on 'pecl.php.net'

There is a bug in config.m4 of php-ext release package on 'pecl.php.net'. It's some broken include path.
From line 8 to 10:
  PHP_ADD_INCLUDE(../../grpc/include)
  PHP_ADD_INCLUDE(../../grpc/src/php/ext/grpc)
  PHP_ADD_INCLUDE(../../grpc/third_party/boringssl/include)

But, all of three include path are broken. It should be just one line:
  PHP_ADD_INCLUDE(third_party/boringssl/include)

Some logs with error.
while configure
...
checking for awk... awk
checking if awk is broken... no
checking whether to enable grpc support... yes, shared
./configure: line 4107: cd: ../../grpc: No such file or directory
./configure: line 4138: cd: ../../grpc/src/php/ext: No such file or directory
./configure: line 4169: cd: ../../grpc/third_party/boringssl: No such file or directory
checking for ld used by cc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
...

while make
...
In file included from /Volumes/RamDisk/grpc-1.0.0/src/core/lib/security/credentials/google_default/google_default_credentials.c:46:
In file included from ./src/core/lib/security/credentials/jwt/jwt_credentials.h:38:
./src/core/lib/security/credentials/jwt/json_token.h:38:10: fatal error: 'openssl/rsa.h' file not found
#include <openssl/rsa.h>
         ^
1 error generated.
make: *** [src/core/lib/security/credentials/google_default/google_default_credentials.lo] Error 1

Probably dupe of #7922

stanley-cheung commented 6 minutes ago
Probably dupe of #7922

oh yes, it's about the same problem.
should i close this issue?

What platform / OS version did you get this error? And how can I reproduce?

@stanley-cheung
stanley-cheung commented 6 hours ago
What platform / OS version did you get this error? And how can I reproduce?

MacOS and CentOS 7.X x84_64
steps:
wget https://pecl.php.net/get/grpc-1.0.0.tgz
tar zxf grpc-1.0.0.tgz
cd grpc-1.0.0
phpize
./configure    ## get the warning about three broken pathes
make    ##  get the fatal error that cannot find 'openssl/rsa.h'

or
pecl install grpc    ## for new install
pecl upgrade grpc    ## for guys who installed a previous version

Hi, maybe you should read the discuss #7923.

The pecl package is not considered source so we don't support unzipping it and then running phpize, ./configure etc. If you want to compile from source, you need to follow the instructions in the src/php/README.md.
I have just tried on Mac and I am able to do a fresh install of pecl install grpc. If pecl install grpc fails for you, can you show me the log please?

@stanley-cheung
stanley-cheung commented 15 hours ago • edited
The pecl package is not considered source so we don't support unzipping it and then running phpize, ./configure etc. If you want to compile from source, you need to follow the instructions in the src/php/README.md.
I have just tried on Mac and I am able to do a fresh install of pecl install grpc. If pecl install grpc fails for you, can you show me the log please?

I have checked out the installation with pecl on CentOS, it's perfect. Sorry for my negligence.
The solution with pecl to install grpc ext is good.
But still i cannot build grpc extensions from source on pecl.php.net, and this is exactly the way to build ext with homebrew on Mac. So i'm trapped between 'fixing' grpc release package on pecl site for source building and 'fixing' homebrew formula modifying 'config.m4' before phpize. The latter hacking solution is not elegant, i think.
Additional, the php installed with homebrew by default has no pear module, there is no pecl command. homebrew encourages to install php ext with brew install instead of pecl install. May we should consider the situations that homebrew formula and many other automatic scripts which build php ext from the source downloaded on 'pecl.php.net'. If they have to clone the whole grpc repo, there is a huge cost in network transmission.

Yea the current status is that we only support the one line package install thru pecl install, or compile from source thru github. Previously we looked into supporting brew install but run into a bunch of problems. I will keep this in mind to see if we can support brew install in the future.


Support Singularity?

For security and management reasons, we want to use Singularity instead of Docker. The difficulty with Singularity for me right now is that little documentation exists about how to run a container with Desktop environment. It's something our users want to use. Since there are similarities between Docker and Singularity, would it be possible to adapt x11docker for Singularity? I can also hack and write some code, but from the first look of 8k+ lines of code, I think it is better to ask you first @mviereck .

Since there are similarities between Docker and Singularity, would it be possible to adapt x11docker for Singularity?

I'd be interested to support singularity, too. However I am a noob here, and have no experience with singularity at all.
Singularity recently appeared in the debian repository, so I can run some tests.

I can also hack and write some code

Can you point me to a beginner's introduction? Or can you provide me a few lines of code to get a container running? If you can introduce me a bit into singularity and help me above some hurdles, we might be able to adapt x11docker together.

For security and management reasons, we want to use Singularity instead of Docker.

Where do you see a security advantage compared to Docker?

The difficulty with Singularity for me right now is that little documentation exists about how to run a container with Desktop environment.

Basically it would be same as with docker: share a unix socket and provide X access credentials. Compare: https://github.com/mviereck/x11docker/wiki/Short-setups-to-provide-X-display-to-container. If sharing a unix socket is not possible for some reasons, an X over TCP setup should work.
At least a basic setup should not be too hard.

Where do you see a security advantage compared to Docker?

We first tried docker for NVIDIA DGX2 server in a multiuser environment setup. One of the big deal breaker there is that docker doesn't respect SLURM job specifications of number of CPU. That gives a user unconstrained access to all available GPUs. I read somewhere that one may have to enable some kernel feature and recompile the kernel so Docker will respect cgroup settings. That is not something we want to do. Singularity on the other hand respects SLURM job specification. Singularity is developed with HPC+multiuser environment in mind.
Another problem, to run Docker, one must be in Docker group, or worse, to have sudo right. This is another deal breaker. I tried with x11docker but this seems only secure if an admin starts a container and give access to the normal users. User should not be able to build image or start a container, otherwise the virtual root in the container may do privilege escalation to and effect things outside the container.
All in all, I see there are to many tweaks to do, that they both takes time and introduce users to more complex workflow.
For complete picture, NVIDIA is also developing something similar to Singularity but it uses Docker images after conversion. That is it doesn't have an Dockerfile-like:
https://github.com/NVIDIA/enroot

Can you point me to a beginner's introduction?
I recommend singularity's own documentation for all things but the GUI desktop environment :)
https://sylabs.io/guides/3.4/user-guide/quick_start.html

You may also watch this slow pace beginner guide for usage of Singularity
https://www.youtube.com/watch?v=gev_nIw02u4&list=PLO7bpxApY95TlTY7pMyPIk0RaJQD85NnG&index=4

share a unix socket and provide X access credentials. Compare: https://github.com/mviereck/x11docker/wiki/Short-setups-to-provide-X-display-to-container. If sharing a unix socket is not possible for some reasons, an X over TCP setup should work.
I will check this out while waiting for you :)

A quick first success!
singularity shares /tmp from host. This includes /tmp/.X11-unix/ with the X unix sockets from host. This allows X access

Authenticate X access with xhost (run on host):
xhost +SI:localuser:$USER
Get a GUI image:
singularity pull docker://x11docker/lxde
Run singularity image:
singularity shell lxde_latest.sif
Run GUI:
DISPLAY=:0 lxterminal

I'll reply to your post a bit later in detail, just wanted to tell this quickly. :-)
This setup is unpolished and insecure, but already a proof of concept.
btw: These days I have some internet connection issues, so I might be slow to respond until I'm online again.

I am not familiar with xhost, how do I add a Kerberos user instead of localuser? I checked man xhost and tried: xhost +SI:krb:$USER
And got this:
<EMAIL_ADDRESS>being added to access control list
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  109 (X_ChangeHosts)
  Value in failed request:  0x12
  Serial number of failed request:  7
  Current serial number in output stream:  9

I am not familiar with xhost, how do I add a Kerberos user instead of localuser?

ok, I see, it is some sort of network setup. I am not familar with network setups in general.
There are two ways to allow access to X, either with xhost or with a cookie in $XAUTHORITY. Let's find out how your current X access is done.

Please check the output of plain xhost if there is already an useable entry.
Check echo $XAUTHORITY if there is a cookie file.
If XAUTHORITY is set and the file is already available in singularity, try:

DISPLAY=:0 XAUTHORITY=/path/to/cookie lxterminal

(Check also if DISPLAY on host is really :0 or :0.0.)

Does singularity not run on your local system at all but on a remote server without a display?
In that case you need to allow access to your local display. If you access the server with ssh, try with ssh -X and check out DISPLAY in the ssh session. DISPLAY should look like <IP-adress>:10.  In that case no unix socket is involved.

We first tried docker for NVIDIA DGX2 server in a multiuser environment setup. One of the big deal breaker there is that docker doesn't respect SLURM job specifications of number of CPU. That gives a user unconstrained access to all available GPUs.

docker has a new option --gpus to specify which NVIDIA GPUs and which of its capabilities can be used by the container. That might be partially what you need. Compare https://github.com/docker/cli/issues/1200

Another problem, to run Docker, one must be in Docker group, or worse, to have sudo right. This is another deal breaker.

Yes, that is quite unfortunate.
podman might be an alternative for containers without root, I am waiting for a debian package: https://github.com/containers/libpod

I tried with x11docker but this seems only secure if an admin starts a container and give access to the normal users.

x11docker does its best to isolate untrusted applications, but to run docker the user must be trusted. That is a no-go in a multi-user environment.
However, there is one possibility to run docker or x11docker without giving the user root permissions or membership in group docker:

The admin creates a launcher script, e.g. /usr/local/bin/x11docker-xfce:

#! /bin/bash
x11docker --desktop x11docker/xfce

Make an entry in /etc/sudoers that allows the user to run this command with sudo and without a password.  The user does  NOT  need to be member of group sudo.

someuser ALL=NOPASSWD:/usr/local/bin/x11docker/xfce

Though, this user cannot pull or build images. That would need further setup.

Haven't check the version on Debian, I would recommend to compile and install the latest stable version from github (see link to the documentation in the end). From version 3.3, Singularity supports fakeroot feature, which means users do not need sudo right to build from a Definition file

Debian currently ships with singularity version 3.1.1, but I think that is ok for now. It is enough to check out X setups, and I'll use the GUI images I've developed for docker.

security advantage compared to Docker

Some first thoughts about the security setup of singularity:
So far I think the only (but important) security advantage of singularity is that the user is not in group docker.
I see disadvantages: The user is not protected at all. By default, entire HOME is shared. If a malicious application is installed with singularity, it can spy passwords, steal data, , change ~/.bashrc etc. pp. It only does not have root privileges.
Also entire /dev seems to be shared. I am not sure if that is a good idea.
Only entirely trusted images should be used with singularity. But the concept allows to run arbitrary images found somewhere in the net.

Only entirely trusted images should be used with singularity. But the concept allows to run arbitrary images found somewhere in the net.

Wouldn't this drawback applied for all programs that users have direct SSH access to host, including Docker images? A solution then is a web interface with prebuild images that users can launch. If they want a customized image, they have to send build requests. That then will sounds a lot like Cloud technologies like  Azure, Google, AWS, or the self-hosted Openstack.
Thanks and agree with you about other things.

I tried and could start a clean lxterminal and mate-terminal (with the mate image).
user@clientlaptop:~/ ssh -X user@front-end
user@front-end:~ mkdir runuser
user@front-end:~ srun --pty --gres=gpu:1 --x11 singularity shell --nv -B ./runuser:/run/user/$UID mate_latest.sif
Singularity mate_latest.sif:~/singularity> mate-terminal # terminal came up clean on my laptop

Problems come when I tried a full desktop environment:
mate-session
Failed to run gnome-keyring-daemon: Failed to execute child process “gnome-keyring-daemon” (No such file or directory)
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Unable to find provider '' of required component 'dock'
Window manager warning: Screen 0 on display "localhost:73.0" already has a window manager
[1570186467,000,xklavier.c:xkl_engine_start_listen/]    The backend does not require manual layout management - but it is provided by the application
Window manager warning: Screen 0 on display "localhost:73.0" already has a window manager

(mate-panel:90445): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:28.312: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
Unable to open desktop file /usr/share/applications/firefox.desktop for panel launcher: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

(caja:90457): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:28.804: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
dbus-daemon[90507]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=140196 pid=90489 comm="/usr/lib/mate-applets/trashapplet " label="unconfined")
dbus-daemon[90507]: Successfully activated service 'org.a11y.atspi.Registry'
SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry

(mate-settings-daemon:90420): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:29.106: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported

(polkit-mate-authentication-agent-1:90497): polkit-mate-1-WARNING **: 12:54:29.130: Error getting authority: Error initializing authority: Could not connect: No such file or directory

** (at-spi2-registryd:90514): WARNING **: 12:54:29.199: Failed to register client: GDBus.Error:org.gnome.SessionManager.AlreadyRegistered: Unable to register client

** (at-spi2-registryd:90514): WARNING **: 12:54:29.199: Unable to register client with session manager

** (mate-settings-daemon:90420): WARNING **: 12:54:29.342: Clipboard manager is already running.

** (caja:90457): WARNING **: 12:54:29.494: Can not calculate _NET_NUMBER_OF_DESKTOPS

** (caja:90457): WARNING **: 12:54:29.494: Can not calculate _NET_NUMBER_OF_DESKTOPS

** (caja:90457): WARNING **: 12:54:29.494: Can not get _NET_WORKAREA

** (caja:90457): WARNING **: 12:54:29.494: Can not determine workarea, guessing at layout

^Cmate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client6' failed to reply before timeout
mate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client1' failed to reply before timeout
mate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client7' failed to reply before timeout
srun: error: _half_duplex: wrote -1 of 4096
Gdk-Message: 12:56:25.473: mate-session: Fatal IO error 4 (Interrupted system call) on X server localhost:73.0.

Wouldn't this drawback applied for all programs that users have direct SSH access to host

Of course, a user can always damage everything he has access to. He can download arbitrary binaries or scripts from the net and execute them.
However, not everything is available from secure repositories, and not every dependency can be fulfilled easily.. Containers help here.
It is a question of the defaults:

Docker:

Docker runs containers with fairly secure defaults. A malicious application will have a hard time to break out of docker run -ti evil/image although it has root permissions in the container.
Docker allows to degrade container isolation with several option. The evil image can do any harm to the system with e.g. docker run -v /:/hostroot  --cap-add=ALL evil/image. It can harm the user space with docker run -v $HOME:$HOME  evil/image.
A quite good wrapper could be designed to allow harmless docker options only.

Singularity

Singularity protects the root system, but shares entire HOME by default. (And also at least /tmp and /dev).
This is a comfortable, but quite insecure default. Running a singularity container one time is enough to install a trojan in user space.
The problem is: the user is encouraged to to run anything like on MS Windows, and he'll get the same problems like on MS Windows. And he might thing: "hey, it is a container, it is secure."
I prefer secure defaults that can be reduced optionally.
For singularity, I'd recommend a default that does not allow to share entire HOME but only directories specified by the user. Also /tmp and /dev and maybe others should only be shared on demand. That would already improve user space security a lot.

A solution then is a web interface with prebuild images that users can launch. If they want a customized image, they have to send build requests.

The user should be aware of the risks. There might be a repository "entirely trusted", another one "basically trusted", and the rest is "unknown wild beast".
At least the "unknown wild beast" should have quite restricted defaults.

For singularity, I'd recommend a default that does not allow to share entire HOME but only directories specified by the user. Also /tmp and /dev and maybe others should only be shared on demand. That would already improve user space security a lot.

This can be done by configuring singularity.conf. Mine is at /usr/local/etc/singularity/singularity.conf
# MOUNT PROC: [BOOL]
# DEFAULT: yes
# Should we automatically bind mount /proc within the container?
mount proc = yes

# MOUNT SYS: [BOOL]
# DEFAULT: yes
# Should we automatically bind mount /sys within the container?
mount sys = yes

# MOUNT DEV: [yes/no/minimal]
# DEFAULT: yes
# Should we automatically bind mount /dev within the container? If 'minimal'
# is chosen, then only 'null', 'zero', 'random', 'urandom', and 'shm' will
# be included (the same effect as the --contain options)
mount dev = yes

# MOUNT DEVPTS: [BOOL]
# DEFAULT: yes
# Should we mount a new instance of devpts if there is a 'minimal'
# /dev, or -C is passed?  Note, this requires that your kernel was
# configured with CONFIG_DEVPTS_MULTIPLE_INSTANCES=y, or that you're
# running kernel 4.7 or newer.
mount devpts = yes

# MOUNT HOME: [BOOL]
# DEFAULT: yes
# Should we automatically determine the calling user's home directory and
# attempt to mount it's base path into the container? If the --contain option
# is used, the home directory will be created within the session directory or
# can be overridden with the SINGULARITY_HOME or SINGULARITY_WORKDIR
# environment variables (or their corresponding command line options).
mount home = yes

I tried and could start a clean lxterminal and mate-terminal (with the mate image).

Great!

Problems come when I tried a full desktop environment:

I am not sure what causes the crash. One singularity error is shown:
srun: error: _half_duplex: wrote -1 of 4096

Though, a desktop should normally not be started without an additional X server. Otherwise it interferes with your already running desktop.
You can try nested X server Xephyr and run the mate desktop in Xephyr.
x11docker can help you to set up Xephyr or other X servers. Run e.g.:
x11docker --xephyr --wm=none --showenv

This will open a Xephyr window and show DISPLAY and XAUTHORITY on stdout.

About x11docker and singuarity: Although technically possible, I think it does not make sense to include singularity support in x11docker.
x11docker aims to isolate applications as good as possible. It follows the principle of least privilege.
singularity aims to isolate applications as few as possible to avoid hurdles for users. It gives a lot of privileges by default.
Maybe x11docker could drop those privileges with command line options regardless of system wide settings. But it would work against the intentions of singularity. So this does not make sense.
I'd rather like to support podman when it is available as a debian package. podman allows to set up containers without root privileges, but is otherwise similar to docker in its default setup.

You can still use x11docker to run an X server and provide this X server to singularity applications.
Setting up X on a remote server with GPU support is out of scope of x11docker, but if you want to set this up, I'm happy to help with some hints.
Closing now, but feel free to ask further questions.


kernel: disable ZSWAP, remove zVM hypervisor on s390x

Unless we run kata VM as a hypervisor, we may need
CONFIG_S390_HYPFS_FS and CONFIG_SYS_HYPERVISOR.
CONFIG_S390_VMUR is for z/VM hypvervisor.
Remove CONFIG_ZSWAP and its dependencies to match other arches.
Fixes: #421
Signed-off-by: Tuan Hoang<EMAIL_ADDRESS>

cc @alicefr

/test

Hi Graham @grahamwhaley, Hi James @jodh-intel,
Would you mind having a quick review and/or approval ?
Many thanks !

@devimc : Hi, friendly ping :D