Why is the matrix calcs so slow

I'm using osx and the native extensions
libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze" % breezeVersion,
  "org.scalanlp" %% "breeze-viz" % breezeVersion,
  "org.scalanlp" %% "breeze-natives" % breezeVersion,
)

trait KNN {
  type MDouble = DenseMatrix[Double]
  type VDouble = DenseVector[Double]

  def classify(input: MDouble, dataSet: MDouble, labels: Array[Double], k: Int): Int = {
    var timestamp: Long = System.currentTimeMillis
    val dataSetSize = dataSet.rows
    val tiled: MDouble = tile(input, dataSetSize, 1) // tiled data set
    // calculates the distance => √(x2-x1)^ + (y2-y1)^
    val diffMat: MDouble = tiled - dataSet
    val sqDiffMat: MDouble = diffMat :^ 2.0
    val sqDistances: VDouble = sum(sqDiffMat(*, ::)) // sum per row
    val distances: VDouble = sqDistances :^ 0.5 // sqrt
    val sortedDistances: Seq[Int] = argsort(distances) // sort the arguments
    val mapAcc = (0 to k).map { i => labels(sortedDistances(i)) }.groupBy(identity).mapValues(_.size)
    val res = mapAcc.foldLeft((0.0, 0)) { (t, curr) => if (t._2 > curr._2) t else curr }
    var lastTimestamp: Long = System.currentTimeMillis
    println("elapsed time in answer")
    println((lastTimestamp - timestamp) / 1000d)
    res._1.toInt
  }
}

this algorithm takes about 0.15 seconds, and in numpy the same algorithm takes about 0.035, I'm testing with a bigger matrix about 1934 x 1024

can you give me a self contained main method? what version of breeze are
you using?
On Thu, Sep 17, 2015 at 3:16 PM, Michel Perez<EMAIL_ADDRESS>wrote:

I'm using osx and the native extensions
libraryDependencies ++= Seq(
"org.scalanlp" %% "breeze" % breezeVersion,
"org.scalanlp" %% "breeze-viz" % breezeVersion,
"org.scalanlp" %% "breeze-natives" % breezeVersion,
)
trait KNN {
type MDouble = DenseMatrix[Double]
type VDouble = DenseVector[Double]
def classify(input: MDouble, dataSet: MDouble, labels: Array[Double], k: Int): Int = {
var timestamp: Long = System.currentTimeMillis
val dataSetSize = dataSet.rows
val tiled: MDouble = tile(input, dataSetSize, 1) // tiled data set
// calculates the distance => √(x2-x1)^ + (y2-y1)^
val diffMat: MDouble = tiled - dataSet
val sqDiffMat: MDouble = diffMat :^ 2.0
val sqDistances: VDouble = sum(sqDiffMat(*, ::)) // sum per row
val distances: VDouble = sqDistances :^ 0.5 // sqrt
val sortedDistances: Seq[Int] = argsort(distances) // sort the arguments
val mapAcc = (0 to k).map { i => labels(sortedDistances(i)) }.groupBy(identity).mapValues(_.size)
val res = mapAcc.foldLeft((0.0, 0)) { (t, curr) => if (t._2 > curr._2) t else curr }
var lastTimestamp: Long = System.currentTimeMillis
println("elapsed time in answer")
println((lastTimestamp - timestamp) / 1000d)
res._1.toInt
}
}
this algorithm takes about 0.15 seconds, and in numpy the same algorithm
takes about 0.035, I'm testing with a bigger matrix about 1934 x 1024
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447.

would be nice to have the numpy code for comparison too
On Thu, Sep 17, 2015 at 3:21 PM, David Hall<EMAIL_ADDRESS>wrote:

can you give me a self contained main method? what version of breeze are
you using?
On Thu, Sep 17, 2015 at 3:16 PM, Michel Perez<EMAIL_ADDRESS>wrote:

I'm using osx and the native extensions
libraryDependencies ++= Seq(
"org.scalanlp" %% "breeze" % breezeVersion,
"org.scalanlp" %% "breeze-viz" % breezeVersion,
"org.scalanlp" %% "breeze-natives" % breezeVersion,
)
trait KNN {
type MDouble = DenseMatrix[Double]
type VDouble = DenseVector[Double]
def classify(input: MDouble, dataSet: MDouble, labels: Array[Double], k: Int): Int = {
var timestamp: Long = System.currentTimeMillis
val dataSetSize = dataSet.rows
val tiled: MDouble = tile(input, dataSetSize, 1) // tiled data set
// calculates the distance => √(x2-x1)^ + (y2-y1)^
val diffMat: MDouble = tiled - dataSet
val sqDiffMat: MDouble = diffMat :^ 2.0
val sqDistances: VDouble = sum(sqDiffMat(*, ::)) // sum per row
val distances: VDouble = sqDistances :^ 0.5 // sqrt
val sortedDistances: Seq[Int] = argsort(distances) // sort the arguments
val mapAcc = (0 to k).map { i => labels(sortedDistances(i)) }.groupBy(identity).mapValues(_.size)
val res = mapAcc.foldLeft((0.0, 0)) { (t, curr) => if (t._2 > curr._2) t else curr }
var lastTimestamp: Long = System.currentTimeMillis
println("elapsed time in answer")
println((lastTimestamp - timestamp) / 1000d)
res._1.toInt
}
}
this algorithm takes about 0.15 seconds, and in numpy the same algorithm
takes about 0.035, I'm testing with a bigger matrix about 1934 x 1024
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447.

@dlwh Here is all the scala code https://github.com/mrkaspa/ScalaMLExamples/blob/master/src/main/scala/com/mrkaspa/ml/knn/ImageProc.scala and the python code is this one https://github.com/pbharrin/machinelearninginaction/blob/master/Ch02/kNN.py#L103

Thanks. you're like the 3rd or 4th person who's asked about squared
distance. I guess it's time I figured it out.
On Fri, Sep 18, 2015 at 5:48 AM, Michel Perez<EMAIL_ADDRESS>wrote:

@dlwh https://github.com/dlwh Here is all the scala code
https://github.com/mrkaspa/ScalaMLExamples/blob/master/src/main/scala/com/mrkaspa/ml/knn/ImageProc.scala
and the python code is this one
https://github.com/pbharrin/machinelearninginaction/blob/master/Ch02/kNN.py#L103
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-141442653.

@dlwh does this only happens with squared distance? I thought it wasn't loading the native extensions

broadcasting can be a bit slow. I'll check out what's going on this weekend.
It looks like you aren't running it multiple times. The JVM takes a while to warm up (and it has to load the .so natives on first use). Can you time it running 4-5 times? I imagine we'll still be behind, but it might be more fair.

@dlwh I made some meassures and the instruction that takes much time is the tile method
elapsed time in tile(input, dataSetSize, 1)
0.228
elapsed time in tiled - dataSet
0.008
elapsed time in diffMat :^ 2.0
0.022
elapsed time in sum(sqDiffMat(*, ::))
0.03
elapsed time in sqDistances :^ 0.5
0.0
elapsed time in argsort(distances)
0.0

great thanks. that'll be relatively easy to fix
On Fri, Sep 18, 2015 at 1:50 PM, Michel Perez<EMAIL_ADDRESS>wrote:

@dlwh https://github.com/dlwh I made some meassures and the instruction
that takes much time is the tile method
elapsed time in tile(input, dataSetSize, 1)
0.228
elapsed time in tiled - dataSet
0.008
elapsed time in diffMat :^ 2.0
0.022
elapsed time in sum(sqDiffMat(*, ::))
0.03
elapsed time in sqDistances :^ 0.5
0.0
elapsed time in argsort(distances)
0.0
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-141565451.

thanks, I'll be pending

i made a few improvements. If you're not satisfied, could you specify the actual sizes of the matrices you're using rather than doing file io? Easier to reproduce

also, don't use argsort if you just want the top k elements. use argtopk

Great how can I test the improvements?

The params
val tiled: MDouble = tile(input, dataSetSize, 1) // tiled data set
input = Matrix(1 x 1024)
dataSetSize = 1934

sbt update (assuming 0.12-SNAPSHOT) and try it out
On Sat, Sep 19, 2015 at 11:34 AM, Michel Perez<EMAIL_ADDRESS>wrote:

The params
val tiled: MDouble = tile(input, dataSetSize, 1) // tiled data set
input = Matrix(1 x 1024)
dataSetSize = 1934
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-141696481.

@dlwh It improved a lot but numpy keeps being faster
scala >> 89.50 sec
python >> 44.79 sec

just so i know: that's 89.5 seconds for how many repeats?

It calls the same method 1934 times one per each file

ok, here is a much better implementation from Breeze's point of view. First, we transpose (and copy) the data so that it's (1024 x 1934), which is better for breeze because it's column major.
Then we use broadcasting for the subtraction rather than explicitly tiling.
      val diffMat = dataSet(::, *) - input
      val sqDiffMat = diffMat :^ 2.0
      val sqDistances = sum(sqDiffMat(::, *)) // sum per column
      val distances = sqDistances :^ 0.5 // sqrt
      val k = 5
      val sortedDistances = argtopk(distances.t *= -1.0, k) // sort the arguments
      val mapAcc = (0 until k).map { i => labels(sortedDistances(i)) }.groupBy(identity).mapValues(_.size)
      val res = mapAcc.foldLeft((0.0, 0)) { (t, curr) => if (t._2 > curr._2) t else curr }
      res._1.toInt

All in all this is about 5x faster on my machine.

For this line
val diffMat = dataSet(::, *) - input

I get this compilation error:
Error:(29, 34) could not find implicit value for parameter op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That]
    val diffMat = dataSet(*, ::) - input
                                 ^

Error:(29, 34) not enough arguments for method -: (implicit op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That])That.
Unspecified value parameter op.
    val diffMat = dataSet(*, ::) - input
                                 ^

input should be DenseVector, 1024 long.
On Sat, Sep 19, 2015 at 5:03 PM, Michel Perez<EMAIL_ADDRESS>wrote:

For this line
val diffMat = dataSet(::, *) - input
I get this compilation error:
Error:(29, 34) could not find implicit value for parameter op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That]
val diffMat = dataSet(*, ::) - input
^
Error:(29, 34) not enough arguments for method -: (implicit op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That])That.
Unspecified value parameter op.
val diffMat = dataSet(*, ::) - input
^
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-141721277.

Did it work?
On Sep 19, 2015 5:04 PM, "David Hall"<EMAIL_ADDRESS>wrote:

input should be DenseVector, 1024 long.
On Sat, Sep 19, 2015 at 5:03 PM, Michel Perez<EMAIL_ADDRESS>wrote:

For this line
val diffMat = dataSet(::, *) - input
I get this compilation error:
Error:(29, 34) could not find implicit value for parameter op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That]
val diffMat = dataSet(*, ::) - input
^
Error:(29, 34) not enough arguments for method -: (implicit op: breeze.linalg.operators.OpSub.Impl2[breeze.linalg.BroadcastedRows[breeze.linalg.DenseMatrix[Double],breeze.linalg.DenseVector[Double]],KNN.this.MDouble,That])That.
Unspecified value parameter op.
val diffMat = dataSet(*, ::) - input
^
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-141721277.

The numbers are like this for this line of code
elapsed time in dataSet(*, ::) - vec
0.098 sec
elapsed time in dataSet(*, ::) - vec
0.135 sec
elapsed time in dataSet(*, ::) - vec
0.128 sec

The hole method for doing all the KNN calculation in python takes 0.045 sec I think the problem is in that line it's slower than python may be they do that through the native library

Like I said, Breeze is column-major, so you're much much much better off doing columnwise computations:
  def timeBroadcastSubtract(reps: Int) = {
    val dataSet = DenseMatrix.rand[Double](1024, 1934)
    val input = DenseVector.rand(1024)
    cforRange(0 until reps) { i =>
      // calculates the distance => √(x2-x1)^ + (y2-y1)^
      dataSet(::, *) - input
    }
  }

  def timeBroadcastRowSubtract(reps: Int) = {
    val dataSet = DenseMatrix.rand[Double](1934, 1024)
    val input = DenseVector.rand(1024)
    cforRange(0 until reps) { i =>
      // calculates the distance => √(x2-x1)^ + (y2-y1)^
      dataSet(*, ::) - input
    }
  }

The difference in those two methods:
[info]            benchmark    ms linear runtime
[info]    BroadcastSubtract  6.67 ====
[info] BroadcastRowSubtract 42.64 ==============================

That is, if you rearrange your computation, you can get an order of magnitude speedup.

(this is just how caches work on modern cpus. I could have made Breeze row-major like numpy, but it's too late to change it now.)

But I'll have to rethink the solution to use column-wise operations rather than row-wise operations then

i pasted in the solution earlier?
On Thu, Sep 24, 2015 at 8:41 PM, Michel Perez<EMAIL_ADDRESS>wrote:

But I'll have to rethink the solution to use column-wise operations rather
than row-wise operations then
—
Reply to this email directly or view it on GitHub
https://github.com/scalanlp/breeze/issues/447#issuecomment-143113468.

Uhm I mean I'll have to change how the solution is implemented to take advance of the speed of the column-wise faster operations


Setup loaders

Overview
This PR setup the loaders module, which loads many of the files required for processing by loading them into pandas dataframe. The following loaders are added:

sound speed loader
travel times loader
gps solutions loader

Additionally, astropy is getting added as a dependency to take advantage of its powerful time module to handle leap seconds as mentioned in #74.
TODO
Tests are still needed, but since this PR is already pretty big, I will put this in a separate PR in the future.

This also resolves #16, #17, #18


Use new "macOS" naming where appropriate.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

Not quite a mass replacement as I've used OS X and Mac OS X where describing specific older versions and added compatibility methods for things in the DSL.
Fixes #961.

Current coverage is 82.43% (diff: 100%)

Merging #1019 into master will increase coverage by 7.17%

@@             master      #1019   diff @@
==========================================
  Files           360        340     -20
  Lines         15543      14190   -1353
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits          11697      11697
+ Misses         3846       2493   -1353
  Partials          0          0

Powered by Codecov. Last update f4a8d28...3f9cce0

I'd prefer to avoid renaming purely-internal identifiers like variable names; this'll affect git-blame's usefulness.

@mistydemeo Given past experience people are going to keep submitting PRs to update these. As we've just had a few mass-rubocop and generic code adjustments I think it's worth biting the bullet on this one and at least getting them all done together.


[SR-12200] swift test --filter shouldn't spawn a new process for every test case

Previous ID
SR-12200

Radar
rdar://problem/59496030

Original Reporter
Tainnor (JIRA User)

Type
Bug

Additional Detail from JIRA

Votes
0

Component/s

Labels
Bug

Assignee
None

Priority
Medium

md5: a8bae54f74d0eec10cbf521fd0b1193f

Issue Description:
this makes testing unnecessarily slow compared to not using "–filter".
See https://forums.swift.org/t/swift-test-filter-is-slow-or-how-to-run-tests-fast-on-ci/32315/5

Yes, this has been resolved.


PLT-724: Remove Script type

It can be moved to plutus-apps if needed.
Added serialiseUPLC, deserialiseUPLC and serialiseScript functions for Flat serialisation.

Sorry, I'm off at the moment!

What's the point in not having an alias or newtype for UPLC.Program UPLC.DeBruijn DefaultUni DefaultFun ()? This is heavily inconvenient for downstream users and a huge breaking change.

What's the point in not having an alias or newtype

This package is the ledger API, not a script utils library, and so is not meant to offer anything not needed by cardano-ledger. There's a plutus-script-utils package in plutus-apps, which looks like a good place for such things as the Script newtype to live in.

What about plutus-tx-plugin?

The Script newtype wraps a UPLC program, and is not specific to Plutus Tx. People who write their scripts using Plutarch or other tools may also want to use it. So I wouldn't put it in plutus-tx-plugin either.

@zliu41 I mean, I don't think cardano-ledger uses plutus-tx-plugin.

I mean, I don't think cardano-ledger uses plutus-tx-plugin.

Ah, yes. If you mean there are plutus-tx things in plutus-ledger-api that cardano-ledger doesn't need, you are quite right. Their dependency should be reversed - plutus-tx should depend on plutus-ledger-api. I've been wanting to do that for a while.


Not possible to connect without losing Cellular network

Hi there,
I'm looking for a solution to keep Cellular data even being connected to Device Wifi.
Usual workaround for it is to provide Manual IP/Subnet mask set up for the network like:
https://stackoverflow.com/questions/53106731/how-to-configure-ipv4-address-and-subnet-mask
But unfortunately I'm not able to do it programmatically for now,
have you checked any possible solutions for this?
That's a killer feature for a tool like this

Hi @punov!
Unfortunately I haven't done or researched anything in that direction because it was out of scope for the requirements that I had when I was implementing this.
Best regards,
Marko


High CPU Usage on MacOS

Users report full CPU usage on MacOS even though the node has finished syncing.  We would expect the CPU usage to be near zero.
It is possible that this is not a bug in the node, but just the wallet making repeated queries.
logs-2020-04-07T224954.0732Z.zip

Testing on another Mac does not replicate the issue (i.e. the CPU usage drops to zero once synced).  Here are the corresponding logs.  In total, the process has used 4:11:27.65 CPU since starting (which is high), memory is stable at 773MB.  It has received 6.21GB of data and sent 12.0MB.  It has written 2.81GB to disk and read 5.34GB.
logs-2020-04-08T091633.0206Z.zip

The two issues I have observed on Mac so far are:
TUI (LiveView) is reporting CPU usage of 70% of above when in fact the cardano-node process seems to be using under 1% of the CPU
Similarly, the Network In field of the LiveView TUI is sometimes showing some ridiculous value (but in the heading, i.e. what the maximum value was observed - not the current value) - e.g.
Network In:<PHONE_NUMBER>0000000.0 KB/s

This is resolved in https://github.com/input-output-hk/cardano-node/pull/986

Both @nikola.glumac and @darko-mijic have tested this, and confirmed to be resolved.

If it is resolved, we can close it.


Nebraska Sensor Issue

Nebraska called around 5.  She was unsure if she needed to connect her tablet to the gateway to check if the installation was successful (i think video confusion). I walked her through checking on the dashboard on her phone.  We had to wait a few minutes for the gateway to come up, waited aa few more minutes for the nodes to show on dashboard (they didn't).
I called Esleyther to see what to do next, I usually cheat and just plug into tablet to speed things along, I know we dont want new folks doing this.   Esleyther looked at her stuff on the dashboard and noticed the firmware was older.   He is going to keep an eye on it aand reach out to her if needed.  I told her she could drive home.
@ethansweep @elhenriq  Tomorrow (Ethan has my shift)  she was going to install the other set.  She asked if it would have the same issue, I looked at the firmware on that one.  last time it pinged the dashboard in march and it looked the same.  So I said hold off, but she is going to call the hotline tomorrow to check in.

i think that was a good call. easier to update in the office than field

4:15 5/21/21  Tawana and Esleyther updated the firmware in the lab today, she reinstalled in the field, its been 30 min and gateway hasn't connected to dashboard. Eslyether is giving her a call...


Updates to fix operand ordering problem with pow, div and minus

closes #361
The python overrides for rpow, rsub and rdiv invoke
the scala/java map ops with the arguments reversed so that the
correct operation is performed.

Please test this

retest this please

test this please


Testing: Add skip test for non-existent writable io if nixio is not installed

The current error is very hard to understand
file_or_folder = PosixPath('non_existant_file.nix'), args = (), kwargs = {}
ios = [<class 'neo.io.nixio.NixIO'>, <class 'neo.io.nixio_fr.NixIO'>]
io = <class 'neo.io.nixio_fr.NixIO'>

    def get_io(file_or_folder, *args, **kwargs):
        """
        Return a Neo IO instance, guessing the type based on the filename suffix.
        """
        ios = list_candidate_ios(file_or_folder)
        for io in ios:
            try:
                return io(file_or_folder, *args, **kwargs)
            except:
                continue

>       raise IOError(f"Could not identify IO for {file_or_folder}")
E       OSError: Could not identify IO for non_existant_file.nix

It turns out this error only occurs because nixio is not installed. We should skip this test if nixio is not installed.
The other option would be to instead give a meaningful warning (ie install nixio to run this test), but not everyone would require nixio, so I think it is a better idea to just skip for those that don't have this.

@apdavison, this how been flagged by some people for failing and since the error is not super helpful it might be better to get this into main sooner rather than later. If you get a moment can you look this over?

Just tested this on my windows computer and got the error:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

So I changed to skip this test if we don't have nix or we are on Windows. Windows has a hard time releasing files so it is not worth testing this tiny feature on Windows. I'm also getting other errors with some of the nix tests in general so if we decide to test io/rawio we will need to potentially work on Widows testing side separate. Anyway I added a note in the body of the text on the step which causes the test to fail on Windows.


Stalling after boot

IRController used to work fine for me, but after updating to recent versions (including the current latest), I've run into a consistent issue.
When installing and booting IRController, it installs fine and initially functions. I can setup and load the page in the browser and send codes. But, after a few minutes or so, it seems to totally stop working. I'm assuming it's hung up some how. The page won't load and it doesn't receive commands from Alexa or via the browser.
If I reboot the ESP-12E (v2), it works fine for a few minutes and then the same thing happens.
Could you help with some troubleshooting to figure out what could be happening? It there some logging I could turn on to see where its getting hung if I'm monitoring it from my computer?
This is happening on two separate boards, both of which used to work fine with older versions of IRController. But, regrettably I didn't keep track of which new versions broke it. I would try to keep them up-to-date and didn't pay too close of attention since it all installed fine and would initially work. I would only notice later on that they stopped working and figured the next update may fix things. But, that seems to not be the case after some months. So, maybe I'm the only one with this issue? Seems odd that the same thing is happening on two separate boards though.
It's worth noting that I can't be positive if the trouble is because of the latest IRController or the latest ESP8266 Arduino libraries or some other library. I just know that some older installation used to work fine. If I get the time, I could download older and older versions to try to see which one works reliably to at least have some clue as to the changes that broke things for me.
Thanks for any help you can provide with this, I would love to have these two IR blasters working reliably again.

What build settings are you using?
Specifically what board, what Arduino ESP8266 Core version, and what lwIP settings
You need to be on:
Esp8266 2.4.0
LwIP v1.4 prebuilt
Sent from my iPhone

On Dec 17, 2018, at 11:17 PM, Pico Mitchell<EMAIL_ADDRESS>wrote:
IRController used to work fine for me, but after updating to recent versions (including the current latest), I've run into a consistent issue.
When installing and booting IRController, it installs fine and initially functions. I can setup and load the page in the browser and send codes. But, after a few minutes or so, it seems to totally stop working. I'm assuming it's hung up some how. The page won't load and it doesn't receive commands from Alexa or via the browser.
If I reboot the ESP-12E (v2), it works fine for a few minutes and then the same thing happens.
Could you help with some troubleshooting to figure out what could be happening? It there some logging I could turn on to see where its getting hung if I'm monitoring it from my computer?
This is happening on two separate boards, both of which used to work fine with older versions of IRController. But, regrettably I didn't keep track of which new versions broke it. I would try to keep them up-to-date and didn't pay too close of attention since it all installed fine and would initially work. I would only notice later on that they stopped working and figured the next update may fix things. But, that seems to not be the case after some months. So, maybe I'm the only one with this issue? Seems odd that the same thing is happening on two separate boards though.
It's worth noting that I can't be positive if the trouble is because of the latest IRController or the latest ESP8266 Arduino libraries or some other library. I just know that some older installation used to work fine. If I get the time, I could download older and older versions to try to see which one works reliably to at least have some clue as to the changes that broke things for me.
Thanks for any help you can provide with this, I would love to have these two IR blasters working reliably again.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks for getting back so quickly!
These are the boards I'm using: https://www.aliexpress.com/item/2015-New-product-Wireless-module-NodeMcu-Lua-Wifi-Nodemcu-WIFI-Network-Development-Board-Based-ESP8266-High/32521100830.html?spm=a2g0s.90423<IP_ADDRESS>424c4dYjL5QN
I see now the latest IRController version has a custom PlatformIO configuration. I dropped a slightly modified source (hard coding configuration for each IR Blaster as well as commenting out light flashes) file into my previous PlatformIO project which was using default settings. This may be my problem!
I'll re-upload using the custom configuration shortly and let you know if that works properly.

I had the board correctly set to "nodemcuv2" in my previous PlatformIO settings, but my platform was incorrectly the default latest version of "espressif8266" and I did not have the lwip2 build flag, so that was also incorrectly using the default setting.
I've just re-uploaded using the custom configuration included in the latest project and everything seems so far so good. But, because of the kind of issue it's been, I'll see if things stay working after an hour or so to confirm that there isn't another issue going on.
Sorry if this was just my own oversight. Hopefully if that's all it was this may help someone in the future :-)
Thanks again for getting back to me so quickly!

Its all good. It’s a known bug caused by a memory leak with WiFiManager and newer versions of lwIP
From: Pico Mitchell
Sent: Tuesday, December 18, 2018 12:35 AM
To: mdhiggins/ESP8266-HTTP-IR-Blaster
Cc: Michael Higgins; Comment
Subject: Re: [mdhiggins/ESP8266-HTTP-IR-Blaster] Stalling after boot (#233)
I had the board correctly set to "nodemcuv2" in my previous PlatformIO settings, but my platform was incorrectly the default latest version of "espressif8266" and I did not have the lwip2 build flag, so that was also incorrectly using the default setting.
I've just re-uploaded using the custom configuration included in the latest project and everything seems so far so good. But, because of the kind of issue it's been, I'll see if things stay working after an hour or so to confirm that there isn't another issue going on.
Sorry if this was just my own oversight. Hopefully if that's all it was this may help someone in the future :-)
Thanks again for getting back to me so quickly!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

My two IRControllers continue to function perfectly this morning. Closing this issue 👍

Glad to hear it!


pytest_warning_captured deprecated in pytest 6

The plugin uses pytest_warning_captured, which was deprecated in pytest 6.
https://docs.pytest.org/en/stable/reference.html#pytest.hookspec.pytest_warning_captured
Looks like there's a new hook, pytest_warning_recorded, with similar enough API:
https://docs.pytest.org/en/stable/reference.html#pytest.hookspec.pytest_warning_recorded

I asked the pytest mailing list for options on how to deal with this.
Got this from Bruno Oliveira:
"...
An alternative is to check if the new hook exists, like how we used to do in pytest-xdist:
https://github.com/pytest-dev/pytest-xdist/blob/7bf654775dcd2fef9c0ebafbea982b7d180b740e/src/xdist/remote.py#L142-L164
Where we implement one hook or the other, giving preference to the new hook when available.
...
"
I think the xdist fix would work for pytest-json-report

Thanks for the instructive report! Patched in the latest release.


unpack throw exception

I have tried this pkcs11 lib  on ubuntu 19.04 with python 3.7.3, run with a vendor HSM(EngageBlack), all works good.
However, when I  use it on an Linux from scratch system, using same HSM, I got following error.
bash-4.3# python3
Python 3.7.4 (default, Sep 24 2019, 10:51:40)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/usr/lib/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(pkcs11.KeyType.AES, 256)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

are there any other dependencies I need to take care of?
Thanks

What architecture are you on?

virtualbox linux 64 bits
bash-4.3# uname -m
x86_64

bash-4.3# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                3
On-line CPU(s) list:   0-2
Thread(s) per core:    1
Core(s) per socket:    3
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               3292.396
BogoMIPS:              6584.79
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              6144K
NUMA node0 CPU(s):     0-2

What I'm guessing is that one of the #defines we use to set the sizes of things in pkcs11.h on your LFS system is missing. And so it's ending up as the wrong size. You might have to print out the contents of the #define environment the shared object is being compiled with.

Do you mean these defines?
#define CK_PTR            *
#define CK_DEFINE_FUNCTION(returnType, name) returnType name
#define CK_DECLARE_FUNCTION(returnType, name) returnType name
#define CK_DECLARE_FUNCTION_POINTER(returnType, name) returnType (* name)
#define CK_CALLBACK_FUNCTION(returnType, name) returnType (* name)

#ifndef NULL_PTR
#define NULL_PTR          0
#endif

I find that cryptoki.h has been added to master branch but not in release 0.5.0

The defines in 0.5.0 were in setup.py, in master this has been changed. Does it work better in master?

tried master branch still get same error:
Python 3.7.4 (default, Oct  3 2019, 04:30:14)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> from pkcs11 import KeyType, Attribute
>>>
>>> key = session.generate_key(KeyType.AES, 256, template={
...     Attribute.SENSITIVE: False,
...     Attribute.EXTRACTABLE: True,
... })
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "pkcs11/_pkcs11.pyx", line 433, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 586, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 637, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.1.dev33+g46ed66a-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 132, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.1.dev33+g46ed66a-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 117, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

Something is wrong with the length of that type. I would start by having a look how long the value v is. Also what item v refers to.

It might be related to how Python was built.  I can reproduce the issue by following steps:

on ubuntu14.04, download Python-3.7.4  source code and untar it,

guarantee libssl-dev installed, if not sudo apt install ligssl-dev

enter into python source folder and build python:

    *  ./configure --enable-shared
    *  make
    * sudo make install

sudo pip3 install python-pkcs11

Now run python3, import pkcs11,   and create one aes key, same error again
Python 3.7.4 (default, Oct  4 2019, 09:08:14)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(pkcs11.KeyType.AES, 256)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
    return Object._make(self, key)
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
    object_class = self[Attribute.CLASS]
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
    return _unpack_attributes(key, value)
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
    return unpack(bytes(value))
  File "/home/service/python-pkcs11-0.5.0/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/home/service/python-pkcs11-0.5.0/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

Okay so I'm guessing it's a mismatch between how long Python thinks an L is and how long PKCS#11 thinks an L is. Is one being compiled for a different word length than the other?

more info
import pkcs11
from pkcs11 import KeyType
lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
token = lib.get_token()
session = token.open()>>> >>> >>> >>>
>>> key = session.generate_key(KeyType.AES, 256)
unpack attributes
Attribute.CLASS <MemoryView of 'array' object>
len of value 4
b'\x04\x00\x00\x00'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 39, in pkcs11._pkcs11._unpack_attributes
  File "/usr/local/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/local/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

same os, same pkcs11 lib but python3.4 from ubuntu
>>> import pkcs11
>>> from pkcs11 import KeyType
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(KeyType.AES, 256)
unpack attributes
Attribute.CLASS <MemoryView of 'array' object>
len of value 8
b'\x04\x00\x00\x00\x00\x00\x00\x00'
unpack attributes
Attribute.ENCRYPT <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.DECRYPT <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.SIGN <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.VERIFY <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.WRAP <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.UNWRAP <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.DERIVE <MemoryView of 'array' object>
len of value 1
b'\x00'

Okay so I'm guessing it's a mismatch between how long Python thinks an L is and how long PKCS#11 thinks an L is. Is one being compiled for a different word length than the other?

Does pkcs#11 mean   pkcs11 module or python-pkcs11?
pkcs11 module is a shared library provided by hsm vendor and it is working with ubuntu python3.4 + python-pkcs11,  however  same pkcs11 module does not work with python3.7 built from scratch with python-pkcs11.
I hooked my own trace library into vendor hsm, and get some log:
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] virtual CK_RV PKCS11_Trace::C_GenerateKey(CK_SESSION_HANDLE, CK_MECHANISM_PTR, CK_ATTRIBUTE_PTR, CK_ULONG, CK_OBJECT_HANDLE_PTR)
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] session 100
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM type 4224/CKM_AES_KEY_GEN
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM pParameter 0x0
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM ulParameterLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulcount 14
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 0 /  CKA_CLASS
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x2130ec0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 8
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info]<PHONE_NUMBER>000000
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 258 /  CKA_ID
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x15d0210
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info]
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 3 /  CKA_LABEL
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x15d0210
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info]
.....
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] virtual CK_RV PKCS11_Trace::C_GetAttributeValue(CK_SESSION_HANDLE, CK_OBJECT_HANDLE, CK_ATTRIBUTE_PTR, CK_ULONG)
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] session 100
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] hObject 101
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] ulCount1
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] pTemplate 0x7ffdcda2d970
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] type 0 /  CKA_CLASS
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] pValue 0x0
[2019-10-10 08:50:24.826] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:25.070] [PKCS11_Trace] [info]  CKR_OK
[2019-10-10 08:50:25.070] [PKCS11_Trace] [info] type 0 /  CKA_CLASS
[2019-10-10 08:50:25.073] [PKCS11_Trace] [info] pValue 0x0
[2019-10-10 08:50:25.073] [PKCS11_Trace] [info] ulValueLen 4

** notice that when generate a aes key, the CKA_CLASS's ulValueLen is 8 but when query the key
the ulValueLen returned is 4 **.
if the problem is in pkcs11 module(vendor's share library), we can't explain why the module works with python3.4 + python-pkcs

We'll assume the PKCS#11 module is correct. But that our compilation of the headers might be wrong. Especially if the LFS machine is missing some #define (those headers are full of dynamically sized types).

Especially if the LFS machine is missing some #define (those headers are full of dynamically sized types).

I can reproduce the issue on ubuntu 1404 as well.

Build python-3.7.4 from source
python-pkcs11-0.5.0 (pip3 install or install from source)

@Benjamin-Shengming Thank you
@danni You're right. It's due to the wrong size definition in my vendor's cryptoki. Until they fix it, using "I" instead of "L" in ulong definition is a workaround, for me.
_ulong = (Struct('I').pack, lambda v: Struct('I').unpack(v)[0])     # deaults.py:117

@danni  I believe this issue stems from default and standard size differences in python. When I checked the sizes of all prefixes(in both python2 and python3):
    The optional first format char indicates byte order, size and alignment:
      @: native order, size & alignment (default)
      =: native order, std. size & alignment
      <: little-endian, std. size & alignment
      >: big-endian, std. size & alignment
      !: same as >

>>> calcsize("L")
8
>>> calcsize("@L")
8
>>> calcsize("=L")
4
>>> calcsize("<L")
4
>>> calcsize(">L")
4

It's clear that my system does not use the standard size for unsigned long in default. To make sure that the standard size is used, we could modify _ulong as
_ulong = (Struct('=L').pack, lambda v: Struct('=L').unpack(v)[0])     # deaults.py:117

Would this be the correct fix?

No. You want to be using native sizes and endianness, not standard sizes. The definition from PKCS#11 for CK_ULONG is:
typedef unsigned long int CK_ULONG;

Which is your architecture's native long.
Standard sizes serve to provide a platform independent way for Python implementations to exchange binary data. We could consider measuring the length of v as a workaround for broken vendor implementations, but I'm not wild on that idea.
Are you by any chance using the same implementation as @figbux above?

@fubber1nflux working for me!

I'm not using cross platform things so much, but in this situation, even if native or standard, there will be no righteous length of CK_ULONG (or whatever other types). Even ISO C standard defines type lengths as "the values shall be replaced..." and "...shall be equal or greater in magnitude to those shown".
So everyone better hopes their vendor uses a compiler following the same specs with the ones in their systems. I don't have a true solution suggestion for this, but measuring the length as @danni says is something nice I think. This will be more significant when using network hsm's, where at the other side of the cable you are communicating with an embedded device with possibly smaller address space range.

That's not strictly true here. You have a C calling convention defined by your operating system that defines the lengths of types, endianness, etc. Else you'd never be able to link two libraries together ever. This means that unsigned long int is the same type across your whole system, unless you're doing something nutty like using Cygwin on Windows, in which case that's likely to always break.
Also this has nothing to do with the network, only the library that we're linking to. The library can do whatever it wants to deal with endianness and type length, it could talk in XML for all it wants, we don't know anything about that.
Are all of you using the same HSM vendor? Why type does their documentation tell you to use?

Are you by any chance using the same implementation as @figbux above?

That would be correct. However, as I just discovered, the issue runs deeper that that.
The code that generates this exception is this:
_pkcs11.pyx:617
    def __getitem__(self, key):
        cdef CK_ATTRIBUTE template
        template.type = key
        template.pValue = NULL
        template.ulValueLen = <CK_ULONG> 0

        # Find out the attribute size
        assertRV(_funclist.C_GetAttributeValue(self.session._handle, self._handle,
                                     &template, 1))

        if template.ulValueLen == 0:
            return _unpack_attributes(key, b'')

        # Put a buffer of the right length in place
        cdef CK_CHAR [:] value = CK_BYTE_buffer(template.ulValueLen)
        template.pValue = <CK_CHAR *> &value[0]

        # Request the value
        assertRV(_funclist.C_GetAttributeValue(self.session._handle, self._handle,
                                     &template, 1))

        return _unpack_attributes(key, value)

The problem here is that C_GetAttributeValue fetches object attributes from the pkcs library, which are returned in raw form in template struct. This matters because in our case the pkcs library actually communicates with a piece of software that is running on a remote platform which has a different architecture than our own. That causes the fact that even though both the library and the remote application are compiled with identical headers, actual type sizes will differ (ulong is 64 bits on my machine while it is 32 bits on the remote one). And since C_GetAttributeValue returns unprocessed raw data, python-pkcs11 is trying to unpack the data using an incompatible type.
Now, an argument can be made for "Why is the pkcs library not handling the type differences?". But I don't see how that would be possible since it is a packed raw data that is unpacked by the application that is using the library. Interesting thing here is that in the pkcs11 standard(p26) it is stated that:
Note that pValue is a “void” pointer, facilitating the passing of arbitrary values.  Both the application and Cryptoki library MUST ensure that the pointer can be safely cast to the expected type (i.e., without word-alignment errors).

If I understand correctly, as long as the raw data can be safely cast to the attribute type, it is valid. At this point, since python-pkcs11 unpacks the data, I believe It should also handle the casting.
Don't get me wrong, I'm not trying to suggest integrating anything vendor-specific here. This seems more like an unforeseen but valid case that's well within the standard.

I am not an expert on the PKCS#11 spec by any means, but I think you've misunderstood things.
By word alignment errors, I take it to mean alignment within the struct, that is to say, on many architectures you can only cast pointers that are aligned to 4 byte boundaries (e.g. ARM). So pValue has to be correctly aligned for your architecture. This has nothing to do with the data type you expect to cast pValue to, which is given in section 4.
Again, it's important to note that types in PKCS#11 are architecture dependent. If they weren't there would be a lot more code to normalise word size and endianness between operating systems, architecture and platforms. I think if you're just packaging up binary from one system and transporting it across the network to another machine and expecting those types to have the same meaning, the PKCS#11 implementation is broken.
You would have to expect the transfer to happen over an intermediate, network independent format that used the native typing on both ends. I don't see how it can work any other way. It's not safe to read binary off another system.

These are valid (and obvious but im just not experienced enough i guess) points. Looks like I had a narrow perspective on the bigger picture. Thank you for the clear explanation.


Compatibility with Puppet 7

Compatibility with Puppet 7 would be nice. I get following error with Puppet 7:
Could not evaluate: uninitialized constant Puppet::SSL::Key

This change in one of the forks may be a starting point:
https://github.com/btolab/puppet-module-puppet_certificate/commit/8b30f8eb387e898b47b5090e136f17ab34e2b5dd

Here's a summary of the removed code in Puppet 7:
https://tickets.puppetlabs.com/browse/PUP-10252
I don't know if there are any valid/compatible replacements available in Puppet 7. So maybe just add the required files to this module?

Before I try bumbling through the code, has anyone else made progress on refactoring this already for 7 ?

I'm currently working on this, and might have something working.

@mosesdd @huckabeec Would you like to give https://github.com/reidmv/puppet-module-puppet_certificate/pull/11 a go?


Terminations on newlines

If I have some content in gDocs which is marked up like this (in HTML just to illustrate)
My bolded text
My regular text

This addon will generate this:
**My bolded text
**My regular text

Which makes sense, but it breaks the markdown interpretation based on what is really intended. It would be better if it figured that out and gave this:
**My bolded text**
My regular text

Unfortunately there is no way in gDocs for you to realize that you are committing that formatting mistake, so we can't really lump it on the gDocs editor who wrote it.
Great addon by the way, thanks for sharing!

As with the other bug, we can do this. I'll close these two bugs when I get that change in.

Fixed in v 1.0β10. Please reopen (or open another bug) if still a problem.


Clarify 3.4.2 Choice of structure and parameter estimation technique

https://github.com/egouldo/EcoConsPreReg/blob/b5d4ed469c4ceb09c2f57db28f570960785baea0/analysis/preregistration_template.Rmd#L211-L213

@hannahsfraser : HF- I think that it might be double dipping to include structure here as well as in 3.3

I will instead remove the text from 3.3

f753664e0919c0cb04295fbb02aa0bf0a9a4827f


Fix return values of preset_mode(s) properties

It is incorrect to return None as a result of proprety call because in such a case state_attr call will return None as well, which means "Unknown attribute".
Instead for preset_mode(s) PRESET_NONE constant should be used for consistency.
Breaking Change:

Description:
Related issue (if applicable): fixes #
Pull request with documentation for home-assistant.io (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):

Checklist:

[ ] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.
[ ] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[ ] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Devs, do I need to do anything else or it's ready to be merged?


Link to latest comment

Fixes #173
Todo:

[x] Adapt unit tests
[x] Test with release notification (I don't know if they return a latest_comment_url)

I'm using this daily and didn't find any problems yet.

I think that this PR is so out of date, that even if we wanted to merge it, we couldn't. @lgeiger feel free to reopen if the conflicts are fixed!


Added BSG for Bournemouth School for Girls (bsg.bournemouth.sch.uk)

Added BSG for Bournemouth School for Girls (bsg.bournemouth.sch.uk)
Key Stage 3 - Computer Science
https://wearebsg.uk/teaching-learning/the-bsg-curriculum/
https://wearebsg.uk/school-information/contact-us/

@Martyn-Dev First stage of verification passed, official domain is wearebsg.uk as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://wearebsg.uk/

@Martyn-Dev Thank you for your patience while waiting for a review and processing your request. Pull request merged.


MSVC12 Compatibility

I really like the library design and would like to use it cross platform. Unfortunately there are a few roadblocks for using it with MSVC12 (VS2013):

In json11.cpp the Statics struct can't be initialized because not every member has an initializer
Missing noexcept and snprintf
The templates that should define implicit constructors for map and vector like objects seem to fail because std::declval() can not be properly resolved and therefore begin can't be found

The first one can be easily fixed by removing the initializers and adding them to the initializer-list in statics():
struct Statics {
	const std::shared_ptr<JsonValue> null;
	const std::shared_ptr<JsonValue> t;
	const std::shared_ptr<JsonValue> f;
	const string empty_string;
	const vector<Json> empty_vector;
	const map<string, Json> empty_map;
};

const Statics & statics() {
	static const Statics s {
		make_shared<JsonNull>(),
		make_shared<JsonBoolean>(true),
		make_shared<JsonBoolean>(false),
		"",
		vector<Json>(),
		map<string, Json>()
	};
	return s;
}

The missing noexcept and sprintf are a bit harder. MSVC12 does not (yet) include them. It is possible to create a preprocessor  define for noexcept after the STL header includes (as an error is thrown when trying to define the missing noexcept myself). snprintf can be replaced by _snprintf but unfortunately behaves a bit different (snprintf truncates and always 0-terminates, _snprintf does not 0-terminate if the buffer is too small, also the return values are different which are not used anyway).
I didn't invest enough much time digging in c++11s templating features to understand the third problem yet. A simple solution is to comment out the two constructors but I am sure there is a better solution.
This is an issue for me and I don't know if you are interested in supporting MSVC12 (don't know if this gets better with the next compiler update).
PS: Having this library header only would be really nice too, but that's secondary for now

I was able to get it to compile by adding this right above "namespace json11"
#ifdef _MSC_VER
	#if _MSC_VER <= 1800 // VS 2013
		#define noexcept
		#define snprintf _snprintf_s
	#endif
#endif

_snprintf_s is identical to the unix snprintf, and since noexcept isn't supported in VS2013, we just erase them.

@retrop we can mimic noexcept with throw()
and I don't think snprintf is equal to _snprintf_s
see:

http://stackoverflow.com/questions/12833241/difference-between-c03-throw-specifier-c11-noexcept
https://github.com/llvm-mirror/libcxx/blob/master/include/support/win32/support.h

note:
I know that on paper noexcept != throw() but the way it is used in json11 they are more or less equal.
#ifdef _MSC_VER
    #if _MSC_VER <= 1800 // VS 2013
        #define noexcept throw()
        #define snprintf _snprintf
    #endif
#endif


Incorrect import command example in registry.md/replication.md

Describe the bug
The current documentation as of v.3.8.1 contains a copy-paste error within two import command examples.
Expected behavior
The example within registry.md should actually look like this
terraform import harbor_registry.main /registries/7.
Additonally, the example provided within replication.md should also be corrected to
terraform import harbor_replication.main /replication/policies/1.

hi,
I don't understand the issue with this import command terraform import harbor_registry.main /registries/7
I see it, for the 2nd one

You are correct, it was my bad v.3.8.1 contains a correct import command for registries. However, on the main documentation  still links to v.3.7.1 and there it states an incorrect command.

the link you copy is the BESTSELLER/harbor provider which has been migrated to goharbor/harbor (this project) : https://registry.terraform.io/providers/goharbor/harbor/latest


Test operators

Marked as WIP cos' this branch still has no support for multiple test per operator.
Closes #26, #27 & #24

Closed via squashed commit 8526f86587d2d6f3c6efb9658a674a90204be483


Not working with charset="ISO-8859-1"

Hi,
I want to use the polyfill inside a website with charset="ISO-8859-1" meta tag at the document head, but I can't because of the special char you are using for the variable in row 3.
This is the error I get: Uncaught SyntaxError: Unexpected token ILLEGAL (ofi.common-js.js:3).

Once you minify it should work. Alternatively you can require the already minified file in version 2.5.3 (just published)
require('object-fit-images/dist/ofi.browser.js')


BugFix: Type interpretation has broken

react-markdown do not work with TypeScript using default configuration.
I fix it. Please check and merge this.

I can confirm that master is indeed broken and that this fix works.

Pardon my ignorance, I have not delved into the world of TypeScript; are there tools available that could help test/validate these?
Thanks for your contribution!

Released as 2.4.4

Thanks, I also fixed the test incidentally. It has not worked :)


👷🔒 Fix security vulnerabilities

npm audit fix and more.

:tada: This issue has been resolved in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Payload

I am creating subscription just fine and passing a payload it's a guid string. All good..
When checking the subscription, Payload is blank, need something to check against the server.
InAppBillingPurchase purchase = await CrossInAppBilling.Current.PurchaseAsync(productId, ItemType.Subscription, uguid);
Works fine...
var verify = DependencyService.Get();
var purchases = await billing.GetPurchasesAsync(ItemType.Subscription, verify);
            foreach (var item in purchases)
            {
                if (item.ProductId.ToString() == productId)
                {
                    await DisplayAlert("Purchase", item.State.ToString() + " " + item.TransactionDateUtc.ToLocalTime() + " PL: " + item.Payload + " PID: " + item.ProductId + " ID: " + item.Id, "OK");

                }

            }

The item.Payload is always blank, not the guid string I passed? Any ideas?

Basically trying to check on application load if the user has a subscription that is active, and update our server subscription expires on date. Tried the verify to see if i could save a receipt id or something to the server and then pull that back???

Did you read through: https://jamesmontemagno.github.io/InAppBillingPlugin/SecuringPurchases.html
Also based on the type of subscription you should be tracking it.

Payload is only for Android and is what is used to verify the transaction from google.

Hey thanks for responding your awesome! Yes I did go through that and I can pass the verify variable to the purchase and do a GetPurchaseAsync, but not sure what to do from there, sorry...
Is/does the verify variable have return items in it from the purchase side? I couldn't see anything but a VerifyPurchase routine? Trying to grab something from the purchase to save to our server so I can check the subscription and update a subscription expires on the server side...
thanks man for all your work!

ok so the verify variable does return receipt? thanks

Yup! that is the signed data :)

Hey so this is what I got: Prob not correct, but I get to the else and the var verify is null and there is the VerifyPurchase bool event... Trying to see how to get that "signed data" Sorry kinda new to this...
var verify = DependencyService.Get();
InAppBillingPurchase purchase = await CrossInAppBilling.Current.PurchaseAsync(productId, ItemType.Subscription, "payloadisforandroid", verify);
if (purchase == null)
{
//Something went wrong with the purchase
UserDialogs.Instance.Loading().Hide();
await DisplayAlert("Subscribe", "We are sorry, but the subscription did not go through.", "Ok");
					}
					else
					{
                        await DisplayAlert("debug", verify.ToString(), "OK");
						//Purchased, save this information
						var id = purchase.Id;
                        Helpers.Settings.STOREIDSetting = purchase.Id;
                        Helpers.Settings.ISSTORESetting = "true";

}


⚠ Samsung has degraded performance

In 6e9f744, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2513 ms

Resolved: Samsung performance has improved in e5b0cea.


Add license to setup.cfg.

Codecov Report

Merging #175 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #175   +/-   ##
=======================================
  Coverage   53.38%   53.38%
=======================================
  Files         340      340
  Lines       26196    26196
  Branches     3418     3418
=======================================
  Hits        13984    13984
  Misses      11502    11502
  Partials      710      710

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e43dbd...ebdc3a5. Read the comment docs.


Crime Analysis of India

ML-Crate Repository (Proposing new issue)
:red_circle: Project Title : Crime Analysis of India
:red_circle: Aim : Analyze the crime of India in different aspects and different sectors.
:red_circle: Dataset : https://www.kaggle.com/rajanand/crime-in-india
:red_circle: Approach : Try to use 3-4 algorithms to implement the models and compare all the algorithms to find out the best fitted algorithm for the model by checking the accuracy scores. Also do not forget to do a exploratory data analysis before creating any model.
Hello, ML-Crate contributors, this issue is only for the contribution purposes and allocated only to the participants of SWOC 2.0 Open Source Program and JWOC '22 Open Source Program.

📍 Follow the Guidelines to Contribute in the Project :

You need to create a separate folder named as the Project Title.
Inside that folder, there will be four main components.

Images - To store the required images.
Dataset - To store the dataset or, information/source about the dataset.
Model - To store the machine learning model you've created using the dataset.
requirements.txt - This file will contain the required packages/libraries to run the project in other machines.

Inside the Model folder, the README.md file must be filled up properly, with proper visualizations and conclusions.

:red_circle::yellow_circle: Points to Note :

The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.
"Issue Title" and "PR Title should be the same. Include issue number along with it.
Follow Contributing Guidelines & Code of Conduct before start Contributing.
This issue is only for 'SWOC' and 'JWOC' contributors of 'ML-Crate' project.

:white_check_mark: To be Mentioned while taking the issue :

Full name :
GitHub Profile Link :
Participant ID :
Approach for this Project :
What is your participant role?

[ ] SWOC 2.0 Participant.
[ ] JWOC 2022 Participant.
[ ] Contributor

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

Full Name: Sriniketh J
GitHub Profile Link: srini047
Approach for this Project: Perform EDA on Dataset, build models focussing different types of crimes.
Participant Role: JWOC

Issue assigned @srini047


Editor window

Creates Artwork and Critique Editor Window Within the Content Management Portal.

Adding artworks with critiques is fully functional


MTU Limit Question

My coworker and I have a difference in understanding of the requestMtu method's limit values specifically around the Android 14 MTU changes. the limit set here is 517, which is the new limit in Android 14. My coworker is concerned that this should be 512 instead, noting the discussion in this issue from your other library.
My understanding is that the discussion and associated changes from your other library are discussing limiting payload input size in your API for sending so that it will always fit in the MTU even with the changed behavior near the new limits. However in this library it is just setting the MTU on the Android side of the connection, which can go to 517 safely, and either android will ensure that any packets sent will fit their payload in the MTU or it's the user's responsibility to check the MTU.
My coworker is concerned that this limit should be lowered to the data limit (512), worrying that setting this to 517 would allow data up to 517 bytes per packet.
If you could just clarify for us what the actual function of setting the MTU to 517 is, we'd greatly appreciate it!

Hi,
Since Android 14, I think, whatever param you will give, will request highest possible MTU, that is 517.
This will allow you to send 512 bytes of data.
In general, teh Att(ribute layer) MTU contains 2 bytes for attribute handle and one for operation, so 3 bytes are "eaten".
That's why the lowest MTU is 23 which allows sending 20 bytes of data.
However, with this 517 I believe there's something else added here. Some additional 2 bytes, but I don't know what for.
The max size of a characteristic value is 512, so it's not possible to send longer data packets anyway, even with Long Write/Long Read. You may send multiple packets, one by one, to achieve that.
TL;DR
In general, Android requests 517, which allows sending 512 bytes. The peripheral should reply with max supported value.

so calling requestMTU(517) will not allow sending more than 512 bytes in a packet, correct?

Right. It also depends on the peripheral how much it supports. The MTU negotiations settle on the lowest supported value by both sides.

Thank you!


data:i-am-amrendra.json

@Pradumnasaraf  I think one of the commits had a failed build before you merged it

@Pradumnasaraf  I think one of the commits had a failed action before you merged it

Yep, I did that mistake 🙁. That caused the action to fail on other PRs. No worries, everything is sorted out now.

Thank u very much @eddiejaoude @ChinmayMhatre @Pradumnasaraf  for helping me out.🫡


added unit tests for glm

Hi,
I added unit tests for glm but for some reason. Git cannot recognize any folders I added and any new files I added. So I deleted the .gitignore file then the push succeeded.
Thanks,
Yuan

Actually, can you help me push this .gitignore files back but excluding things that might affect my push or future pushes? I need to push filenames like test_****.R in the future.

Another issue I am having is when I run devtools::load_all(".") in R-package folder, this error pops up:
Error in find.package(package, lib.loc, verbose = verbose) :
  there is no package called ‘xgboost'

Can you look into this issue? I don't think this is caused by my new testing code.
Thanks,
Yuan

Addition: library(xgboost) or require(xgboost) both work. find.package('xgboost', lib.loc="~/R/win-library/3.2", verbose = verbose) also works. Could you try to see if you can re-produce this problem?

Okay. I think you guys have just added basic tests on your own. Would you please let people know next time at least?

@terrytangyuan sorry about the not being responsive. I think @hetong007 was busy in the past few month so he might not being able to check the PRs..

Understood. No worries.
On Sep 5, 2015 3:44 PM, "Tianqi Chen"<EMAIL_ADDRESS>wrote:

@terrytangyuan https://github.com/terrytangyuan sorry about the not
being responsive. I think @hetong007 https://github.com/hetong007 was
busy in the past few month so he might not being able to check the PRs..
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/xgboost/pull/437#issuecomment-137988865.

Hi @terrytangyuan, thank you for the PR! Do you mind to open a new PR and create the test in a new test file? Your test is something not being covered right now.

@hetong007 No problem. Sounds good.

@hetong007 So I am facing the same problem again. How do you usually run the tests for this package? I tried to create a new .Rproj file and ran R CMD test but didn't work.

@terrytangyuan I run the test by make Rcheck at ROOT.
to add the file, you can try git add -f R-package/tests/testthat/xxx.R


Refs#28628--abolished use of '\d' in regex

'\d' has different matching rules in Python2 and Python3.

Does #9185 supersede this PR?

Right. #9185 is a more updated version passing more tests.


Persistent HTTP connections

In AWS, Gateway is calling a service ELB behind a R53. I have updated the R53 entry to point to a new ELB after a new deployment, however Gateway (or netty) is still calling the old service.
My thinking is that somehow, once it establishes the connection to the service, the connection is never renewed or expired, or there is a DNS caching configured somewhere?
If I halt momentarily the traffic, the connection will expire and re-create the connection, but as long as the connection is active is never expired.
Is there any fix for this ? Maybe setting some TTL to the connections ?

Gateway uses reactor-netty to make connections. java is notorious for cached dns connections. @violetagg?

I have tried forcing a 30 second DNS TTL before starting boot, but no luck.
java.security.Security.setProperty("networkaddress.cache.ttl", "30");

According to https://github.com/reactor/reactor-netty/issues/1158 there is an option to set a maxLifeTime to the connection pool. Is there a way to override this configuration from Gateway ?
ConnectionProvider.builder("my-pool")                   .maxLifeTime(Duration.ofMillis(60))                   .build();

you can create a HttpClientCustomizer bean.


Heterogeneous timestamp domains with multiple cameras?

Camera Model: D415
Firmware Version: <IP_ADDRESS>
OS: Ubuntu 18.04 with kernel 4.18.20-041820-generic
Platform: PC
SDK Version: librealsense-2.29.0
Segment: Telecommunications
For this test we have 10x D415 cameras attached to our machine. We're highly confident of bus bandwidth and power.
We've recently added an assert to our frame grabber that I was surprised to see always trigger. In pseudo code:
if (rs2_video_frame.get_frame_timestamp_domain() != RS2_TIMESTAMP_DOMAIN_GLOBAL_TIME) {
    // assert here, unexpected behavior
}

Our software reports that:
Frame from sensor<PHONE_NUMBER>46) contains non-global timestamp domain (System Time)

So far, we've consistently seen this only for the 1st camera in the list that comes back from query_sensors()
Hardware resetting the camera doesn't help.
We are not using a hardware synchronization cable in this instance.
Questions:

Is this the expected behavior?

If not, should we explicitly set the desired time-stamp domain on each device/sensor?

Thank you!

Hi @evercoast-smm
No, this is not the expected behavior, in fact only one camera reporting system time on Linux is very strange. Could you please add rs2::log_to_file(RS2_LOG_SEVERITY_DEBUG, "filename.log"); and share the logs? If you have direct support account with Intel, you can also share this way.

Also, dmesg output... @ev-mp - additional suggestions?

@evercoast-smm, please provide application/kernel logs for review as @dorodnic mentioned.
Additionally, please answer the following :

What do you mean by

So far, we've consistently seen this only for the 1st camera in the list that comes back from query_sensors()

Is this reproduced only for the first device in a list, and for the rest the ts domain is global_ts ?
Is also reproduced for other cameras as well ?
Is it a temporal or persistent issue, i.e  does the  system_ts get replaced by global_ts after some period or not ?
Since kernel 4.16 the frame metadata, upon which the global_ts is based is handled differently at the kernel level, and it is not guaranteed anymore that all frames will receive it.
Though we haven't received user reports that this actually happened to them  so far.

@dorodnic , attached please find the log that you requested as "librealsense2.log":
librealsense2.log
@ev-mp , attached please find the dmesg log that you requested as "dmesg.log":
dmesg.log
^^ The above was a 20 second run with with 8x D415 cameras streaming RGB8 color at 1920x1080, and Z16 depth at 1280x720.
Camera status from our API:
firmware for sensor serial #<PHONE_NUMBER>47) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>56) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>82) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>32) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>94) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>14) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>46) is (<IP_ADDRESS>)
firmware for sensor serial #<PHONE_NUMBER>50) is (<IP_ADDRESS>)
interface for sensor serial #<PHONE_NUMBER>47) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>56) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>82) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>32) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>94) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>14) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>46) is (USB 3.2)
interface for sensor serial #<PHONE_NUMBER>50) is (USB 3.2)
@ev-mp "Is this reproduced only for the first device in a list, and for the rest the ts domain is global_ts ? Is also reproduced for other cameras as well ?"
Yes, what you said above is correct. Furthermore, we've tried removing and replacing the offending camera.  The issue will then transfer to the new 1st camera in the list of devices returned by query_sensors().
@ev-mp Is it a temporal or persistent issue, i.e does the system_ts get replaced by global_ts after some period or not ?
Persistent. Across reboots, etc. We're developing a detailed series of unit tests for our product, and have consistently noted this since we started trapping it a week or so ago. I just did several back to back 60 second runs in response to your question about it being temporal. It does not appear to be, we see it on every frame. Only for one sensor. And that sensor is consistently the 1st sensor in the list from query_sensors().
Thanks.

Please kindly let us know if this issue still persists.
Thank you.

We can confirm that this issue has been resolved in 2.31 with
FW <IP_ADDRESS>. Note that we can also confirm that it will only work when
using an LTS kernel.
Thank you!
Evercoast http://www.evercoast.com  |  Sebastian Marino  |  Co-founder &
CTO
On Fri, Feb 7, 2020 at 6:36 PM RealSense Customer Support <
<EMAIL_ADDRESS>wrote:

Please kindly let us know if this issue still persists.
Thank you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/IntelRealSense/librealsense/issues/5131?email_source=notifications&email_token=AGOGSFSFZYTBQT4E67Y5W3DRBXV73A5CNFSM4JFE74VKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELE7NOA#issuecomment-583661240,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGOGSFUPAUVBZKHCF7IDMELRBXV73ANCNFSM4JFE74VA
.

Hi Sebastian,
Thank you very much for the update.
Closing this ticket now.


Add a .unique() method to Iterator.

I've found it very useful to be able to discard duplicates from iterators. I think this would be a useful addition to itertools, what do you think?

Hi, that sounds useful. What do you think about this:

It should be in the itertools trait
Use a single struct, not two
Add it in the form of .unique_by with the closure giving the key
You're welcome to put it in a new module, but not a public submodule.

This was superseded by PR #37


Milestone tracking for "Prettify"

This is an issue where I collect my thoughts about the next milestone and track issues. I might create an actual GitHub milestone if there are enough related GitHub issues.
Next Milestone
My current high-level plan for zzio has these milestones:

Gameplay working from London to leaving Endeva (without trainer fight)
Polish experience of that gameplay section (<- we are here)
Start adding fights

The reason for this is this in-between milestone is to ensure that the systems are working stable before beginning another massive section of this project.
Tasks to do
in no particular order
Must-have

[x] #269  Enough combiner parts to support fairy glows (only needs ParticleEmitter Particle/Models)Maybe check necessary parts for "Slight Quake" spell and/or "Impact"?
First steps to scene effects

[x] Combiner
[x] Lens Flares

[x] #308
[x] Fog rendering
[x] #298  If we do need to add sorting, I would like to know before the next milestone
[x] #274  Just to clean up RWBS and most likely speed up loading
[x] #309
[x] Mlang pipeline overrides to reduce stored variantsAlthough instead I added variant filters to Mlang reducing both compiled programs as well as stored variants by more than half
[x] Remove legacy PipelineCollection
[x] #313
Cleanup dependencies

[x] Veldrid does not need OpenGL or Direct3D anymore
[x] Veldrid.SPIRV will not be necessary anymore Only present in zzmaps anymore pending #270
[x] Mlang.MSBuild should not add runtime references
[x] Replace Veldrid.SDL2 with Silk.NET.SDL  This should also preemptively fix issues for the cross-platform stuff
[x] Remove Veldrid.StartupUtilities after that ^
[x] Upgrade ImGui.NET
[x] #265
[x] Replace SafeStbImageSharp with SDL  One dependency less

Logging

[x] Serilog looks good, how about that?
[x] Is enrichment by class name possible?
[x] Logging should be configurable by CLI options
[x] Log to Remotery

[x] Readd ImGuizmo after fixing cimgui on Windows
[x] Replace DeepSource #312

Maybe just dotnet-format?
Recheck if .editorconfig works good enough

Remotery  Performance seems not as good as expected, Remotery should easily show the culprit and be helpful for any future performance analysis

[x] Preparing bindings repository
[x] Add to Zanzarah, Game, UI and Sequential*System

Sound  What would fights be without the pew-pew?

[x] Prepare bindings repository
[x] Integrating ~SoLoud~ MojoAL + Silk.NET.OpenAL + NLayer
MojoAL supports multiple contexts per device, so multiple windows should just work
[x] Spawning 2D and 3D sounds of actual gameplay as PoC

Cross-platform testing  If possible both for developer as well as user experience  Leaving the possibility to postpone larger issues if they come up  Of course this should be one of the last topics for this milestone

[x] Windows x64  No good access to Windows ARM right now, maybe ask MM
Linux

[ ] Debian-based x64
[ ] Arch-based x64
[ ] Raspberry PI  It apparently is Vulkan-conformant  Quick test: vkcube is running and it can almost build (no native shaderc). Not bad, but no dev on RPi4 for now :)
[ ] Steam Deck  Why not

Mac OSX  The newer Vulkan SDK and MoltenVK really do not like Veldrid very much, even with Pizzas fork. Maybe fix myself and talk to Pizza about merging. (meaning porting to Pizzas fork might be necessary for this)

[ ] x64
[ ] ARM
[x] Add GitHub CI for osx-arm (thanks to the ARM runners)

Good-to-have
Meaning I am not sure yet whether to tackle these in this milestone
Postponed

#137  Apart from item-scripts this is "easy" work without much benefits for other systems
Add map markers  Same reasoning as ^, complete systems that are available in the supported gameplay section
Remotery: Add Vulkan samples  Veldrid currently does not give us a native pointer to CommandList so no GPU samples for now...
#180  I might want to think a bit longer on this.
Mobile ports  This will be only exploration which larger issues occur.

Android
iOS (remember Silk.NET.MoltenVK.Native for this)

Preliminary results of Cross-platform testing
✔  Windows x64
Still works
❔  Windows ARM64
Untested, but I might have access to a device...
There will be several missing native dependencies.
🚧  Linux Debian/Arch-based x64

❌  libdl is not loaded properly due to an outdated NativeLibraryLoader. Switching to Pizzas Veldrid fork should fix this
✔  MojoAL was linked statically with SDL2, causing conflicts. This was fixed.
❔ Debian is untested
❔ Intel GPUs are untested

❌  Rasperry PI (Linux ARM64)
There are many missing libraries and the libdl problem might appear as well.

SDL2  Ask the Silk.NET folks whether they might include Linux ARM64 builds
cimgui/cimguizmo

I guess I should rework ImGui-Nativebuild properly, put all artifacts into an archive and include more platforms
Maybe send both repo changes as PR, maybe they get accepted...

🚧 OSX x64/ARM64
I am putting these two together as they are very similar.

There is a branch with temporary fixes  With these both platforms seem to work fine.
❔  libvulkan.dylib is not found, copying them from the Vulkan SDK fixes this.  It is questionable whether this is just a developer task or whether this should be fixed somehow, maybe with the Ryujinx MoltenVK dependencies?
🚧 Vulkan portability mode with MVK

The big validation error. It can be fixed rather easily in Veldrid but this is neither fixed with Mellinoe nor Pizza, maybe prepare a proper PR and send to both.

🚧  D24_U8 is unsupported

The easier fix would be to just check that the other platforms support F32_S8 and switch
The better fix would be to introduce pipeline overrides to the ShaderVariantCollection after all and switch per platform

Vulkan validations

🚧 VUID-VkBufferCopy-size-01988  This I believe is a Veldrid bug as all other commands are checked for size==0 except one. (For us we can workaround by checking empty atomics in WorldMesh)
🚧 VUID-VkShaderModuleCreateInfo-pCode-08737  This is a Mlang bug, we have to figure out the right pairs of Vulkan-SPIRV-SPIRVBinary versions and create modules accordingly
❔ VUID-VkImageCreateInfo-imageCreateMaxMipLevels-02251  This and the following could be Veldrid bugs, but also could be fixed in Pizzas fork.
❔ VUID-VkPresentInfoKHR-pImageIndices-01430
❔ VUID-vkAcquireNextImageKHR-fence-01287
❔ VUID-vkAcquireNextImageKHR-surface-07783

I have only encountered this on ARM64 but this might occur on x64 as well: F2 key does not work unless it is enabled in the system settings. Thanks Apple.

❔ Steam Deck (Linux x64)
We should expect similar problems like for Windows/Desktop Linux here.
❔ Proton
untested
❔ Steam Runtime
untested


Move features into tree for performance

The current performance is unacceptable. After profiling the code, it is clear that there are issues with cache locality. Here is the closure taking up 71.85% of all the time:
let lookup_distance = |index| (lookup(index) ^ feature).count_ones();

This performs a single trivial operation, so it looks innocuous, but it seems that the way hwt allows the user to store features outside of the tree is actually working against it, making it much slower than linear search (except maybe on huge data sets > 1B).
The API should be changed to give back features and not indices. The user can use their features to look up their data afterwards, which should have negligible performance impact since they only need to do a few lookups (k lookups for k-NN).
This has two negative impacts:

This could make the tree slightly larger.
This will make it harder for the user to store the data however they want.

Seeing as performance is the most important thing here, that takes precedence over these issues.

What about identical features that refer to distinct items?

@abonander My bad for getting back a bit late, I was visiting the family for Easter. Currently, I plan to just have the user disambiguate that. It isn't clear what one might do if two features overlap. For instance, you might just throw one of them out if you don't expect overlap (like descriptors over a bunch of completely unique features). In other cases the user might map one feature to multiple entries, like in photogrammetry applications where you might want geometric verification filtering via RANSAC to match a repeating pattern. There would be a problem if two features overlapped and you were using the lowes ratio from the two features coming out of hwt, in which case you might miss the duplicate, but it could technically be checked when you get the feature back if it has duplicates (although at that point maybe you would just remove it from the tree).
Before it was possible to do this in the tree by using the u32 identifiers, but it wont be possible anymore since that really wasn't good for cache locality. Alternatively, I could add a helper wrapper over the tree that maps features back into a vector of values associated with them. For the time being I am just trying to do everything I can to boost the performance though, and I have learned that what I was doing did not appease the cache on my Intel processor. It seems obvious in retrospect, but I didn't realize that cache locality would have destroyed the performance like that. I am still trying to make more improvements right now to improve branch prediction and cache locality performance, after which I will make another release.

Actually, the way it is being coded now will allow for the lowes ratio to be computed still because the feature can show up twice in the tree, so it can be returned back twice.

This has been finished (along with other optimizations). Benchmarks are on the README.md now.
Unfortunately, after adding more tests I found that the tree broke down at further depths and so the fixed tree's performance only beats linear search once you reach over 1 million features. Keep in mind that the original paper was working with a linear search that took ~2.5 seconds, while I am finding that linear search on modern CPUs is a tiny ~80 milliseconds for the same dataset size. My profiling doesn't leave much room for optimization, but SIMD (using packed_simd) is definitely an option moving forwards (which would definitely speed up the relevant critical sections).
This issue has been addressed, but the performance is still lacking, so I intend to create a tracking issue for improving performance as a follow-up to this.


Removed conflicting default tags

Author Checklist

[ ] I validated this in QA or have noted below why it cannot

Release Notes

[ ] Includes a migration
[ ] Migration is Zero Downtime
[ ] Includes Infrastructure Changes

Reviewer Checklist

[ ] Follows best practices
[ ] No large transaction boundaries
[ ] Queued messages published outside of transactions (if applicable)
[ ] API calls happen outside of transactions (if applicable)

This file is managed by this template

 Vanta - update tags

 Vanta - update tags


Bad behaviour for named pipe (mkfifo)

Issue Type: Bug
The presence of a named pipe (created via mkfifo on Linux) results in undesirable IDE behaviour; on (at least) Linux (possibly also macos)
mkdir x && cd x
mkfifo this-is-a-named-pipe
code .

You now have

Problems:

the pull "doing something" indicator on editor tab never stops ... doing... - IMHO, it should not even attempt doing anything with pipes.
"Unknown File Type" - yeah, it's a named pipe ;)
Trying to delete the pipe from vscode results in the operation never completing

None of that is really breaking, and you'd be right in wondering why I would actually have named pipes in a directory (hierarchy) which vscode touches, but then ...
I think the overall expectation is

don't try to open named pipes
perhaps show "Named Pipe" if you encounter one (find -type p)
make delete work

VS Code version: Code 1.66.2 (dfd34e8260c270da74b5c2d86d61aee4b6d56977, 2022-04-11T07:49:24.808Z)
OS version: Linux x64 5.17.4-200.fc35.x86_64
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz (8 x 2903)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: enabled_onrasterization: disabled_softwareraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: disabled_softwarevideo_encode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
1, 1, 0

Memory (System)
31.32GB (20.93GB free)

Process Argv
. --crash-reporter-id 80e0ae97-0bc4-4a76-8337-f435f2ec1554

Screen Reader
no

VM
100%

DESKTOP_SESSION
/usr/share/xsessions/plasmax11

XDG_CURRENT_DESKTOP
KDE

XDG_SESSION_DESKTOP
KDE

XDG_SESSION_TYPE
x11

Extensions (20)

Extension
Author (truncated)
Version

project-manager
ale
12.5.0

path-intellisense
chr
2.8.0

xml
Dot
2.5.1

gitlens
eam
12.0.6

vscode-drawio
hed
1.6.4

vscode-peacock
joh
4.0.1

milkdown
mir
0.0.12

vscode-docker
ms-
1.22.0

python
ms-
2022.4.1

vscode-pylance
ms-
2022.4.2

remote-containers
ms-
0.231.6

remote-ssh
ms-
0.78.0

remote-ssh-edit
ms-
0.80.0

vscode-remote-extensionpack
ms-
0.21.0

vscode-commons
red
0.0.6

vscode-xml
red
0.20.0

vscode-yaml
red
1.6.0

code-spell-checker
str
2.1.11

code-spell-checker-german
str
2.0.3

shellcheck
tim
0.19.2

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
pythonvspyl392:30443607
pythontb:30283811
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
pythonvsnew555:30457759
vscscmwlcmt:30465135
cppdebugcf:30475217

FWIW, I just ran into the "Trying to delete the pipe from vscode results in the operation never completing" issue.

Not a common problem, but it's something I did hit so I hope this is fixed


Genesis Block error, fixed.

Genesis Block error due  to  missing profile in configtx.yaml file.
When you run the present program, it creates a genesis block which doesn't  look right, plus I couldn't find where the genesis block was being created in the present network.sh file in test-network folder.

I do not think that this PR is necessary. If you use the install scripts: https://hyperledger-fabric.readthedocs.io/en/latest/install.html these files should be installed on your machine.

I agree with @nikhil550 so I'm closing this. If you believe we're missing something please, reopen and explain.
Thanks.


AUTH_KEY_DUPLICATED

tip:
rpc error 406: AUTH_KEY_DUPLICATED caused by invokeWithLayer
I seem to be why this is happening. I can be sure that my account is not lost. When this problem occurs, I did not operate the telegram account, only one cli is running
I did some research on how to re-use the session file when recreating it. I'm curious as to why this is happening, and then how to fix it.
My operating environment: running in windows10, using socks5 proxy, socks5 proxy sometimes has a high delay problem, for this case, I set up the operation of recreating the client instance, that is, if this error message is returned, it will log in again, If the login is successful, delete the old client instance (drop) and use the new client operation
Telethon #777
tdlib AUTH_KEY_DUPLICATED
Telethon #754
webogram #1674

This error is likely outside our control https://github.com/LonamiWebs/Telethon/blob/v1.27.0/telethon_generator/data/errors.csv#L18. That's all I know.


Snowflake ID - when GTID is unavailable

We need to create a monotonically increasing Snowflake ID - when GTID is unavailable

Implemented.


Where do I download the training images from?[Question]

Question
Hello, where can I download the images from in the dataset?
https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K

Hi, thank you for the interest.
You can follow the instruction here to prepare for the necessary checkpoints needed for training models: https://github.com/haotian-liu/LLaVA/blob/main/README.md#visual-instruction-tuning

Please download the annotation of our instruction tuning data llava_instruct_158k.json, and download the COCO train2017 images here.


[$500] Chat - A single line code block becomes multiline in edit mode

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Navigate to https://staging.new.expensify.com/
Open any 1:1 chat
Send a single line code block in any chat (single line)
Edit the message

Expected Result:
A single line code block should be single line in the edit mode too
Actual Result:
A single line code block becomes multiline in edit mode
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/8aa090b2-a1fe-4296-aad2-8efcad16e1e9
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~013245aeb07eeecaf4
Upwork Job ID:<PHONE_NUMBER>268701184
Last Price Increase: 2023-11-20

This isn't a bug, code fences are block content and putting them back inline for editing purposes doesn't server a purpose when the output will always be block level and never inline.

Thank you @erquhart


Fix bugeous crtime value on macOS

During certain operation, macOS use some helper that send request to the mountpoint with crtime set to 0xffff83da4f80. That value correspond to -2_082_844_800u64 which is the difference between the date 1904-01-01 and 1970-01-01 because macOS epoch start at 1904 and not 1970.
https://github.com/macfuse/macfuse/issues/1042
Fix #217

Thanks for digging in and figuring out the root cause here!

Awesome!


Deprecated random module

Since Elixir 1.3 (I guess) the random module has been deprecated. Hence we get these warning when compiling Faker.
warning: random:seed/1: the 'random' module is deprecated; use the 'rand' module instead
  lib/faker/avatar.ex:43
...

Would it be helpful if I make PR to replace random module with rand module? Or is there any concern?

Hi, thanks, it's been done in #53. Sadly don't have time to fix this and release new version.

Oh sorry I didn't see that. Thanks! Hope you can find some time : )


Callbacks to sync on serviceworker state changes.

When state changes are triggered from one browser tab, there should be a way to get callbacks from service worker so other tabs can sync up. These could be login/logout/timeout events.

Because the SW is centralised, all tabs will see the same view on outbound requests. Are you describing a need similar to push notifications? Specifically to notify other tabs when a user has been logged out / timed out?

yes when a logout event occurs, all open tabs should clear state and show the password prompt.


Move towards using "label" field for type alias's for legends

In some cases the entry in the type field might be too long for the layout legend. We may want to use the label field for users to put in a truncated alias.

nevermind


Fixing the BlockingCallbackTransform for a single list output

The current implementation does not work for the following example:
app.layout = html.Div([
    dcc.Input(id="input", value=""),
    dcc.Dropdown(id="dropdown", options=[]),
])

@app.callback(Output("dropdown", "options"), Input("input", "value"), blocking=True)
def update(value):
    return [value + '_1', value + '_2']

That's because the BlockingCallbackTransform will see that the update function returns a list - and when creating a wrapper around it, it will add the new output value of that wrapper function into that list - instead of adding it as a second return value.
In the example above it results in a value like ["aa_1", "aa_2", end_server_id], instead of [["aa_1", "aa_2"], end_server_id].
In this PR I attempt to fix it by making the wrapper generation code context aware, using the expected number of output values when processing the output into a list. If only one output value is expected, even if it's already a list, it is wrapped into a list so that more values can be added next to it.
It also fixes the wrapper generation code for when the single output value is a None (which previously would be converted into [] instead of [None]).
The presented example is a minimal code that allows for reproducing this behavior, but the most notable example of when a user would encounter this issue is when generating the data for a DataTable on an interval using the blocking=True parameter.

Codecov Report

Merging #153 (a0e16f7) into master (35bc3e6) will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff           @@
##           master    #153   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files           8       7    -1
  Lines         938     861   -77
======================================
+ Misses        938     861   -77

Impacted Files
Coverage Δ

dash_extensions/enrich.py
0.00% <0.00%> (ø)

dash_extensions/snippets.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35bc3e6...a0e16f7. Read the comment docs.

I tried reproducing the issue with the latest version of dash-extensions. My app looks like this,
from enrich import DashProxy, html, dcc, Output, Input

app = DashProxy()
app.layout = html.Div([
    dcc.Input(id="input", value=""),
    dcc.Dropdown(id="dropdown", options=[]),
])

@app.callback(Output("dropdown", "options"), Input("input", "value"), blocking=True)
def update(value):
    return [value + '_1', value + '_2']

app.run_server()

and I don't see any errors. Can you check if the error persists in the latest version? And if it does post an updated MWE? :)

I have made an attempt to implement you fix in the current code base. Does is look OK to you? :)
https://github.com/thedirtyfew/dash-extensions/pull/163

Yes, looks good! 🎉

Moved to #163