Some issues and requests

Hi,
First of all great work on this integration, enjoying it a lot! I was able to test it today on two short trips and have a couple of suggestions and some malfunctioning sensors. Of course I know that you already have a to-do list and may not be able to implement all, but here is my list. Keep up the good work!
Missing sensors/requests:

An essential status that is missing is if the caralarm is active.

In the service sensor, the status of 'brake pads wear indicator' is missing. This is a parameter in my official app so it's probably in the data somewhere.

My the doors sensor, my bonnet status is always 'unknown'.

My car has a glass roof with closeable blinds. The status of these blinds are not detected by the sunroof sensor. As with the bonnet status, I do not know if this status is even reported in the server data. How could we find out (also for other sensors and cartypes)?

Wouldn't it be simpler and more consistent if the user preferences (miles/kms, celcium/fahrenheit) are derived from the preferences stored in Home Assistant itself? Those are stored with the unit_system:metric setting in the configuration.yaml.

It would be cool to be able to pull entire trips from the server so that they can be plotted on a map.

Apart from the Last updated timestamp, a usefull variant of this sensor would be 'time since last contact'.

Malfunctioning sensors:

It seems the position is not updated automatically after a trip. Only after I have opened the official website or app the locations is also updated in home assistant. Other sensors such as range and fuel are updated automatically after a trip without having to open the official website.

Timestamp is reported in GMT, not the local time.

Pressing the lock button has no effect, with an error in the log:

2020-04-18 12:23:21 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.139695046379984] 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 134, in handle_call_service
connection.context(msg),
File "/usr/src/homeassistant/homeassistant/core.py", line 1232, in async_call
await asyncio.shield(self._execute_service(handler, service_call))
File "/usr/src/homeassistant/homeassistant/core.py", line 1255, in _execute_service
await handler.func(service_call)
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 213, in handle_service
self._platforms.values(), func, call, required_features
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 412, in entity_service_call
future.result()  # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 600, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 443, in _handle_entity_call
await result
File "/usr/src/homeassistant/homeassistant/components/lock/init.py", line 117, in async_unlock
await self.hass.async_add_job(ft.partial(self.unlock, **kwargs))
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/config/custom_components/jlrincontrol/lock.py", line 51, in unlock
if self._data.wakeup.get("state") != "SLEEPING":
AttributeError: 'NoneType' object has no attribute 'get'

Hi @stefanv84 , thanks for the feedback and the road test!
On the sensors, I will look to add these.  However, it would be very helpful if you could provide a status and attributes output as I do not see any brake wear info on mine.  See the debug into in the readme for how to do.
On the user prefs, are you seeing that they still do not get picked up?  I put a fix in to hopefully make sure they do.  I don't have these settings in my config.yaml and I am trying to find something consistent.  Other issue is that some of the sensor data is only in Km and the way I have built this needs a bit of amending to be able to convert the right ones.  I have on the list - but guessing you are in Europe using Kms?  The default is to use UK imperimetric metrics (not a real word just a UK farse that derserved a farse word!) if it cannot get from JLR but maybe I should make this metric as prob more popular default.
I will put the trips on the list but the weirder one is the position not updating.  I think if we get that updating on a regular interval then HA will plot the trips itself using the person card.
The last updated attribute on the info sensor is the last contacted date and time.  Maybe we should rename it to this?  Does it appear to show correctly?
Aware the lock does not yet work.  I am currently working on the services part of this integration and this will be fixed as part of that.  Should be this weekend.
Again, thanks for using/testing this, being patient and giving great feedback.
Mark

Ah, I had not seen the debug info, will post that soon.
The user prefs are being picked up correctly. Still I wondered if it would be more consistent to grab these from the HA environment. It's an option in the config.yaml and can be seen on 192.168.X.XXX:8123/config/core. Both solutions would work I guess.
Agree the position not updating by itself but only when the official site/app is opened is the strangest issue here. Will try to test a bit more but don't have a lot of long trips these days unfortunately.
The last updated attribute is correct but is showing in GMT, while I am in CET time. Because of that the displayed time seems 2 hours off.     Error is HTTP Error 503: Backend fetch failed
2020-04-18 17:35:12 INFO (MainThread) [custom_components.jlrincontrol] JLR InControl Update Received
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Doors
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol.lock] Getting state of vehicle lock
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Info
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Windows
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Tyres
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Service Info
2020-04-18 17:35:12 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Range
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Unable to get wakeup info.  Error is HTTP Error 503: Backend fetch failed
2020-04-18 17:36:14 INFO (MainThread) [custom_components.jlrincontrol] JLR InControl Update Received
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Doors
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol.lock] Getting state of vehicle lock
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Info
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Windows
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Tyres
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Service Info
2020-04-18 17:36:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Range
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Unable to get wakeup info.  Error is HTTP Error 503: Backend fetch failed
2020-04-18 17:37:17 INFO (MainThread) [custom_components.jlrincontrol] JLR InControl Update Received
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Doors
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol.lock] Getting state of vehicle lock
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Info
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Windows
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Tyres
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Service Info
2020-04-18 17:37:17 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Range

Stefan,
There is a new version in the dev branch which now has services and an alarm sensor. I have also taken some of your guidnace on using HA metrics which i think is probably better. I have not promoted to the master branch yet as it is a lot of changes and want to make sure it is stable beforehand. Would you be able to do some testimg for me.  The lock now also works from the door sensor but you need to add pin:  to config.  Can you also test the services to see if they work for you. Any failures should show in the error log.

Sorry should also add that there is a breaking change in this that the sensors you have added with show unavailable as there was a bug in the unique id which when fixed causes HA to create new sensors. Best way to upgrade is to remove the integration from config, restart then delete the entities from within the configuration - entities menu option. The add the intefration back with the dev branch.
This is a real pain but if i didnt fix now it will be a problem later on.

Mentioning issues #9 and #10 also as these improvements deal with some of these issues.

I tested the Dev branche, followed the comment-restart-uncomment guide and it seems to work.
I succesfully tested the services; "charging off", "charging on", "honk_blink" and everything seems to work. Need to leave home now so no longer able to test but first impression is great!

The dev version works for me and picks up the preferred Miles/KM measurements. New sensors are added, 'Last Contacted Age' and 'Alarm'.
A lot of services don't work for me but I think this is type/subscription dependent. The log says: "Service X is not available on this vehicle". Calling the update_health_status does work, with sensors such as Doors and Alarm being correctly updated after the update.
The repeating part of the log still states one error:
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Unable to get wakeup info.  Error is HTTP Error 503: Backend fetch failed
2020-04-20 17:07:14 INFO (MainThread) [custom_components.jlrincontrol] JLR InControl Update Received
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Doors
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol.lock] Getting state of vehicle lock
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Info
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Windows
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Alarm
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Tyres
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Service Info
2020-04-20 17:07:14 DEBUG (MainThread) [custom_components.jlrincontrol] Updating Disco Sport Range

Need to leave home now so no longer able to test but first impression is great!

Leaving home.  Oh I remember that! ;-)

I live in the Netherlands where we not have a full lockdown but still able to move... under tight conditions..
Do you have specifics you want me to test?

I live in the Netherlands where we not have a full lockdown but still able to move... under tight conditions..
Do you have specifics you want me to test?

I think as youre the only one here with an iPace at present, it would be great to test all the EV services including max soc and also i think we need to do some work on the EV battery sensor to make sure we are showing the right attributes.
Also any feedback on how you want to use it and whether it supports these objectives.
Thanks

@stefanv84 I will be adding another update today to the dev version as the final updates before making this the current master version.
I think I have fixed the issues you have raised here with the following exceptions.

Brake pad wear indicator - this is not in your status update so it may need another api call to get it.  Will have to refer this to @ardevd as I have no way of finding out the call needed.
Roof blinds - again, not in the status update - presume no sensor on the car for this.
Pulling trips - not really sure what we would do with this and whether there is any real way to show in HA - see 4.
Position updates - now we have the update health status service, can you see if that updates the position info.  The only thing I know the app does is call this when opening.  If this doesn't work then I guess there must be another service call we need to find.  Maybe using the last trip to get the position info.

Can you confirm all other issues are now resolved in the version v0.5alpha with tonights updates.

Hi Mark, this version works well without errors in the log.
Position remains fiddly but I was able to pinpoint the problem a bit more. If I move the car, the health update service does not update the position even though it does update the other sensors. Also if I first open the official app (which does show the new position), the health update service does not update the position in HA. Only when I restart HA, the position is correctly updated, regardless of whether I open the official app. Hope this info helps pinpointing the problem. Seems that the health update service updates all the sensors except for the position data.
Amazing to see this integration coming together so quickly, and looks great!

@stefanv84 I think position should now be working.  I noticed it wasn't showing any updates in the logs after the first update on restart.  Seems to be my lack of understanding of device trackers as they do not work the same way as other platforms.  Updated dev branch if you could let me know if this is now working.

Works perfect now!

Can we close this issue now?

Yes of course, I’ll close it


Provide fixed line spacing support

Include line-height as part of the output if it is configured and prepare a new release with this new feature.

Build failing but not because of our changes. The job failing thinks we didn't include all the changes in the bundle, but this is not the case. As the linter is passing and tests as well, I'm merging this PR.


Update notebooks to match format requirements

Description
This PR updated several tutorials to match the format requirements of contribution guideline.

[x] module tutorials
[x] pathology
[x] microscopy/multichannel_microscopy_classification.ipynb
[x] full_gpu_inference_pipeline/client/non_ensemble/client.ipynb
[x] federated_learning/openfl/openfl_mednist_2d_registration/workspace/Monai_MedNIST.ipynb
[x] deployment

Checks

[x] Avoid including large-size files in the PR.
[x] Clean up long text outputs from code cells in the notebook.
[x] For security purposes, please check the contents and remove any sensitive info such as user names and private key.
[x] Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the ./figure folder
[x] Notebook runs automatically ./runner.sh -t <path to .ipynb file>

I see permerge errors https://github.com/Project-MONAI/tutorials/actions/runs/3910825263/jobs/6683476358
perhaps this command can address them https://github.com/Project-MONAI/MONAI/blob/4b464e7b6bee1b52fa0be4d72a4662f384f095af/runtests.sh#L114-L119

I see permerge errors https://github.com/Project-MONAI/tutorials/actions/runs/3910825263/jobs/6683476358
perhaps this command can address them https://github.com/Project-MONAI/MONAI/blob/4b464e7b6bee1b52fa0be4d72a4662f384f095af/runtests.sh#L114-L119

Thanks, @wyli! It has been fixed according to your suggestions!

I'm merging this for now because there's a fix for runtests.sh, please raise more pull requests if @mingxin-zheng has more comments

@wyli Thanks. I am on travel today and had limited access to internet. I will look at these notebooks and raise issues if there are any.

Regarding the formats, everything looks good to me @KumoLiu . Thank you!


Releasing MSA.BuildingBlocks.ServiceClient version 1.0.0

Hi. I want to use a new version of the ServiceClient package. Could you please release it?

Also, it is not for this case. But I found an issue with an example.  I could fix it. Could I create the issue for that case and commit under that number?

Are you going to prepare the fix before release?

Yes, sry, going to make it today.


Test ODM on ROKS

[x] Improve Pre-Requisites with Cluster version, size, nodes flavours.... or at least Reference the Detailed System Requirements

[x] Add a section to configure external DB

[x]  Explain how to retrieve the ICD DB Credentials

[x] oc create route reencrypt my-sticky-dc-route --service=odm23-on-roks-odm-decisioncenter --cert=tls.crt --key=tls.key --dest-ca-cert=tls.crt

Thanks for the input. Doc has been updated


WEBMKS Failed in Najdorf Version due to certs/server.cer* permissions

Discussed in https://github.com/ManageIQ/manageiq/discussions/22060

Originally posted by ccesario August 16, 2022
Hi folks, currently Im running ManageIQ Lasker version, and VMware WebMKS Console it working as expected.
Today I tried upgrade this appliance to Najdorf version and I got error with WebMKS, so I download the Najdorf  VMware appliance stable version and deploy it, but the same problem happen.
When I try access VMware WebMKS console I got the following screen

Does someone has any tip abou fix it ?
Regards
Carlos

So I don't think that we can chown these as manageiq:manageiq like we do with certs/v2_key because httpd also needs it and that runs as root and we don't want to open up read to other.  So I think we have to do
# ls -l certs/
total 12
-rw-rw----. 1 root     manageiq 1107 Aug 18 15:36 server.cer
-rw-rw----. 1 root     manageiq 1704 Aug 18 15:36 server.cer.key


Q: How to compute a gradient w.r.t. selected parameters?

Hello all,
I have a very simple question. How to compute gradients w.r.t. a subset of parameters. Let's say I have a model like:
struct Model: Layer {
    var a = Tensor<Float>(1.0)
    var b = Tensor<Float>(2.0)
    var c = Tensor<Float>(3.0)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return exp(log(b * a * input) + c)
    }
}

How do I compute a gradient w.r.t. to only a and b?

There are two options.

If you want to totally disable the gradient for c:

struct Model: Layer {
    var a = Tensor<Float>(1.0)
    var b = Tensor<Float>(2.0)
    @noDerivative
    var c = Tensor<Float>(3.0)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return exp(log(b * a * input) + c)
    }
}

Then the Model.TangentVector doesn't have c.

If you want to disable just in callAsFunction:

struct Model: Layer {
    var a = Tensor<Float>(1.0)
    var b = Tensor<Float>(2.0)
    var c = Tensor<Float>(3.0)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return exp(log(b * a * input) + withoutDerivative(at: c))
    }
}

Then the Model.TangentVector have c but its value is 0.

@t-ae, thanks for an example. One more question: is there a way to do the trick with withoutDerivative for a model from a third-party library? In TensorFlow, it is pretty easy, I can change variables to non-trainable.

@t-ae, looking at current solutions I realize that models in swift are not flexible enough for research purpose. It happens all the time when you need to learn only the parameters subset of the model. So, my question transforms into: "How to compute a gradient w.r.t. selected parameters without editing the model?"

looking at current solutions I realize that models in swift are not flexible enough for research purpose. It happens all the time that you need to learn only the parameters subset of the model. So, my question transforms into: "How to compute a gradient w.r.t. selected parameters without editing the model?"

I'm curious why you want to "differentiate with respect to a subset of stored properties of a Differentiable-conforming type"? Is your use case optimization (preventing unnecessary differentiation), or something else?

Every Differentiable-conforming type has a fixed TangentVector associated type.
In your snippet, Model.TangentVector is a struct with a b c stored properties - this never changes.
import TensorFlow
struct Model: Layer {
  var a, b, c: Tensor<Float>

  // Compiler synthesizes:
  // struct TangentVector: Differentiable, AdditiveArithmetic, PointwiseMultiplicative, ElementaryFunctions, VectorProtocol, KeyPathIterable {
  //   var a, b, c: Tensor<Float>
  //   ...
  // }
  ...
}

If you apply a differential operator like func gradient to Model, you'll get back values of type Model.TangentVector. This is fixed. No types exists representing just "derivatives of a and b".

Then the Model.TangentVector have c but its value is 0.

In general, zero derivatives like this are correct.
struct Model: Layer {
    var a = Tensor<Float>(1.0)
    var b = Tensor<Float>(2.0)
    var c = Tensor<Float>(3.0)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return exp(log(b * a * input) + withoutDerivative(at: c))
    }
}

withoutDerivative(at: c) causes c to have zero derivative. So small changes in input cause no change in c, hence a zero derivative.

@t-ae's response https://github.com/tensorflow/swift-apis/issues/691#issuecomment-590707440 is great!

Use @noDerivative on stored properties so they don't appear in TangentVector.

Use @noDerivative var c if you never need derivatives with respect to c.

Using withoutDerivative(at:) to prevent differentiation through specific values.
import TensorFlow
struct Model: Layer {
    var a = Tensor<Float>(1.0)
    var b = Tensor<Float>(2.0)
    var c = Tensor<Float>(3.0)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        let c = withoutDerivative(at: self.c) // here
        return exp(log(b * a * input) + c)
    }
}

let c = withoutDerivative(at: self.c) tells activity analysis not to mark c as active and not to propagate activity through c. This means that the compiler knows that c doesn't need a derivative value and does less work. I think this ~addresses (or is set up to address) optimization concerns.

+1 to @t-ae and @dan-zheng 's responses. One other thing to add:
if you'd like to dynamically control whether particular parameters are updated during the optimization process, you can also "zero-out" the computed derivative before updating the model's state. While this doesn't save on computation of the derivatives themselves (except on top of a lazy tensor runtime... stay tuned!), this would allow you fine-grained control of operations & updates.
There are a couple ways to spell the "zeroing-out". The most imperative might be something like:
var grads = gradient(at: model) { model in loss(model(input)) }
grads.c = .zero
optimizer.update(&model, along: grads)

but other ways of spelling this include property wrappers (as @t-ae alluded to).
In order to provide more effective help, could you perhaps describe what you're trying to do // your larger context?

There are a couple ways to spell the "zeroing-out". The most imperative might be something like:
var grads = gradient(at: model) { model in loss(model(input)) }
grads.c = .zero
optimizer.update(&model, along: grads)

Note: using grads.c = .zero is actually problematic in general and not recommended. When per-instance zero tangent vectors are done, you can write:
grads.c = grads.zeroTangentVector // `grads.zeroTangentVector` has the same shape as `grads`

The Differentiable.zeroTangentVector instance property is more correct than the AdditiveArithmetic.zero static property in some cases (e.g. for types containing arrays and tensors).
https://github.com/tensorflow/swift-apis/issues/656#issuecomment-588080813 has more info on zeroTangentVector, if you're interested. zeroTangentVector is important for automatic differentiation correctness, too.

@awav: I hope your question was answered! Your use case isn't clear, so it's have to provide a targeted suggestion. Feel free to reopen if you'd like to chat more.

@awav: I hope your question was answered! Your use case isn't clear, so it's have to provide a targeted suggestion. Feel free to reopen if you'd like to chat more.

:D I was writing an answer when you closed it. Can you re-open it?

Is your use case optimization (preventing unnecessary differentiation), or something else? Your use case should drive this discussion!

@dan-zheng, well I can give you two examples. First one is the Gaussian process model. This type of models can have variational parameters and hyperparameters (kernel parameters). The training loss is ELBO (evidence lower bound). It is common to train these type of models with natural gradients updating only variational parameters and use SGD-like optimisers for hyperparameters.  So, one iteration step consists of two optimiser updates. The second example is about debugging/prototyping/making new models, sometimes you (researcher/user) know the optimum for a subset of parameters and you want to check the difference between losses with and without true values. Another scenario when you want to hold a part of the model to see how it affects the training procedure. Often this technique is very effective, you may call it burning or warming up before training.

@awav: I'm curious why you want to "differentiate with respect to a subset of stored properties of a Differentiable-conforming type"?

I'm used to that. TensorFlow and Pytorch have Module and normally models are bags of parameters and optimisers accept a list of variables instead of differentiable objects :)

Can you re-open it?

Sure! Feel free to do it yourself, since it's your unresolved issue!

well I can give you two examples.

Could you please share some reference implementations? I'm not an expert, so code will greatly help me understand your examples.

First one is the Gaussian process model. This type of models can have variational parameters and hyperparameters (kernel parameters). The training loss is ELBO (evidence lower bound). It is common to train these type of models with natural gradients updating only variational parameters and use SGD-like optimisers for hyperparameters. So, one iteration step consists of two optimiser updates.

To me, "one iteration step consists of two optimizer updates" sounds expressible to me in Swift. Differential operators are higher-order functions (functional APIs) that produce T.TangentVector values. You can pass T and T.TangentVector values around freely and update T with T.TangentVector (i.e. optimize T) however you like.
Some folks have implemented non-trivial optimization algorithms in Swift using these APIs, which give me confidence that the fundamental design is okay!

I'm used to that. TensorFlow and Pytorch have Module and normally models are bags of parameters and optimisers accept a list of variables instead of differentiable objects :)

Yes. Differentiable Swift's programming model is different from these frameworks, for good reason (we have types and can't be as dynamic!). I still feel it's possible to ~accomplish your use case (sounds like you care about optimization) given the current differentiation APIs in Swift. And if not, that's a great opportunity for us to brainstorm support for unsupported use cases.

The custom differentiation tutorial shows some cool techniques like derivative surgery Differentiable.withDerivative(_:), rematerialization instead of checkpointing, and manually chaining pullbacks. The last technique may be relevant to "one iteration step consists of two optimizer updates".
If you'd like, please share code that you don't know how to write in Swift! We can make progress on specific examples. 🙂

Sure! Feel free to do it yourself, since it's your unresolved issue!

I didn't have an option to reopen an issue.

Could you please share some reference implementations? I'm not an expert, so code will greatly help me understand your examples.

Examples of natural gradients and adam optimisers for training SVGP model.

Sounds like you care about derivative optimization, avoiding inefficient zero derivatives

Yes, this is a part of the concern.

The custom differentiation tutorial shows some cool techniques like derivative surgery Differentiable.withDerivative(_:), rematerialization instead of checkpointing, and manually chaining pullbacks. The last technique may be relevant to "one iteration step consists of two optimizer updates".

Thanks, I will try it and will give feedback soon.


Aspect ratio not support yet

Aspect ratio is a useful feature in yoga，We'd be glad to see it in core layout :)

This was added in 43e2cb5d4df180184fc64af506661123207182c7. Cheers!


TikTok User is broken.

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2<IP_ADDRESS>. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
Everywhere
Description
Downloading a profile on tiktok is not working anymore.
Verbose log
[starshine@fedora Documents]$ yt-dlp --verbose --write-pages https://vm.tiktok.com/ZSeS3XPNF/
[debug] Command-line config: ['--verbose', '--write-pages', 'https://vm.tiktok.com/ZSeS3XPNF/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2<IP_ADDRESS>
[debug] Lazy loading extractors is disabled
[debug] Python version 3.9.2 (CPython 64bit) - Linux-5.11.12-300.fc34.x86_64-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [generic] Extracting URL: https://vm.tiktok.com/ZSeS3XPNF/
[generic] ZSeS3XPNF: Requesting header
[redirect] Following redirect to https://www.tiktok.com/@yoleendadong?_d=secCgwIARCbDRjEFSADKAESPgo86dEAh3Z%2Fn77L39WI3Uw5iXYilTYBvihD6Kwq5t4m7W2nW6sxvq%2BlQUXyoIDSvKz2O0N13sGgRnzQO20VGgA%3D&language=en&sec_uid=MS4wLjABAAAAB5eD4nYSVXUm-EenMG1lv61JF4RiHlM-wP04GZa5rzvgBVo8QRcNSagAGLtW-HCx&sec_user_id=MS4wLjABAAAAOkLgiKnZIBqD1c5ZIUaRfpOfVgv2Rps_-JgoEV197JSpXx_bt3-aUbAmir-jFo0q&share_app_id=1233&share_author_id=6621187864221384710&share_link_id=4c1bf44b-1d24-4640-9428-ea21569080e4&timestamp=1637932859&u_code=dlegb2cfck6b3g&user_id=7026701991499318278&utm_campaign=client_share&utm_medium=android&utm_source=copy&source=h5_m&_r=1
[debug] [tiktok:user] Extracting URL: https://www.tiktok.com/@yoleendadong?_d=secCgwIARCbDRjEFSADKAESPgo86dEAh3Z%2Fn77L39WI3Uw5iXYilTYBvihD6Kwq5t4m7W2nW6sxvq%2BlQUXyoIDSvKz2O0N13sGgRnzQO20VGgA%3D&language=en&sec_uid=MS4wLjABAAAAB5eD4nYSVXUm-EenMG1lv61JF4RiHlM-wP04GZa5rzvgBVo8QRcNSagAGLtW-HCx&sec_user_id=MS4wLjABAAAAOkLgiKnZIBqD1c5ZIUaRfpOfVgv2Rps_-JgoEV197JSpXx_bt3-aUbAmir-jFo0q&share_app_id=1233&share_author_id=6621187864221384710&share_link_id=4c1bf44b-1d24-4640-9428-ea21569080e4&timestamp=1637932859&u_code=dlegb2cfck6b3g&user_id=7026701991499318278&utm_campaign=client_share&utm_medium=android&utm_source=copy&source=h5_m&_r=1
[tiktok:user] yoleendadong: Downloading webpage
[tiktok:user] Saving request to<EMAIL_ADDRESS>[download] Downloading playlist: yoleendadong
[tiktok:user] yoleendadong: Downloading user video list page 1
[tiktok:user] Saving request to yoleendadong_https_-_api-t2.tiktokv.com_aweme_v1_aweme_post_user_id=6621187864221384710_count=21_max_cursor=0_min_cursor=0_retry_type=no_retry_device_id=1641026827630030711_version_name=20.9.3_version_cod_e99ac00921ad2119c6c52b8c3243b05b.dump
WARNING: [tiktok:user] Expecting value: line 1 column 1 (char 0). Retrying...
[tiktok:user] yoleendadong: Downloading user video list page 1 (attempt 1)
[tiktok:user] Saving request to yoleendadong_https_-_api-t2.tiktokv.com_aweme_v1_aweme_post_user_id=6621187864221384710_count=21_max_cursor=0_min_cursor=0_retry_type=no_retry_device_id=1641026827630030711_version_name=20.9.3_version_cod_c39ef3447684b83de68cc169062ba245.dump
WARNING: [tiktok:user] Expecting value: line 1 column 1 (char 0). Retrying...
[tiktok:user] yoleendadong: Downloading user video list page 1 (attempt 2)
[tiktok:user] Saving request to yoleendadong_https_-_api-t2.tiktokv.com_aweme_v1_aweme_post_user_id=6621187864221384710_count=21_max_cursor=0_min_cursor=0_retry_type=no_retry_device_id=1641026827630030711_version_name=20.9.3_version_cod_0aedcb3c912daf71338ef6175c55e1b5.dump
WARNING: [tiktok:user] Expecting value: line 1 column 1 (char 0). Retrying...
[tiktok:user] yoleendadong: Downloading user video list page 1 (attempt 3)
[tiktok:user] Saving request to yoleendadong_https_-_api-t2.tiktokv.com_aweme_v1_aweme_post_user_id=6621187864221384710_count=21_max_cursor=0_min_cursor=0_retry_type=no_retry_device_id=1641026827630030711_version_name=20.9.3_version_cod_9516b2fd46e8122ff7256f5830720ab0.dump
ERROR: yoleendadong: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 988, in _parse_json
    return json.loads(json_string)
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 988, in _parse_json
    return json.loads(json_string)
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1318, in wrapper
    return func(self, *args, **kwargs)
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1556, in <lambda>
    lambda self, i: ie_entries[i - 1]
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/utils.py", line 4141, in __getitem__
    self.__cache.extend(itertools.islice(self.__iterable, n))
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/tiktok.py", line 460, in _entries_api
    post_list = self._call_api('aweme/post', query, username,
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/tiktok.py", line 73, in _call_api
    return self._download_json(
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 977, in _download_json
    res = self._download_json_handle(
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 963, in _download_json_handle
    return self._parse_json(
  File "/home/starshine/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 992, in _parse_json
    raise ExtractorError(errmsg, cause=ve)
yt_dlp.utils.ExtractorError: yoleendadong: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

[tiktok:user] playlist yoleendadong: Downloading 0 videos
[download] Finished downloading playlist: yoleendadong

It just went up again, can you try again?

It just went up again, can you try again?

Sure.

Tiktok API has been going down frequently. If you see this message in the future, your best bet is to increase the reties or try again after some time
[tiktok:user] yoleendadong: Downloading user video list page 1 (attempt 3)
ERROR: yoleendadong: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

Experiencing this now. So this is TikTok's mobile API servers going down? Does that mean that TikTok is actually broken at this time, like using the app wouldn't work? Because the website still works for me.
It would be helpful if the error message for this was improved.


Various constraint handling fixes

Fixed:

InputManager using raw xwayland coords
InputManager using raw position hint
InputManager not scaling the regions if xwayland:force_zero_scale
InputManager not using scaled boundaries of the region

Notable for #3340 #3204
Should fix both, that's at least the goal.
If you are affected, let me know how it goes with this mr

waiting for confirmation of fixing from any of the 2 issue members

#3204 confirmed fixed, waiting for #3340

well guy is away, will merge then anyways

@vaxerski : see https://github.com/hyprwm/Hyprland/issues/3340#issuecomment-1740035823
Things got lot better but the lock area geometry is still incorrect


[TODO_BLOCKER] @Olshank's Iteration16 TODOs

Objective
Tackle all TODO_BLOCKERS created by / assigned to @Olshansk
Origin Document

Review & update all instance of Olshansk
2.Review & update all instances of TODO_BLOCKER:
Review & update instances of @h5law
Use TODO_MAINNET or TODO_BLOCKER or TODO_BETA where appropriate

A first execution of the following command:
grep -r "Olshansk" --include=\*.{go,md}

Provided the following results

Goals

Tackle all TODO_BLOCKER(@Olshansk) in this iteration

Deliverables

[ ] Deliverable #1
[ ] Deliverable #2

Creator: @Olshansk
Co-Owners: [OPTIONAL - GitHub handle of co-owner(s)]

Duplicate of #584


Update test-file-04.md

@v-caxian : Thanks for your contribution to the Azure documentation! The author, @TinaLiuGirl, has been notified to review your proposed change.


AttributeError: 'Callback' object has no attribute 'validation_data'

URL(s) with the issue:
Please provide a link to the documentation entry, for example:
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback
Description of issue (what needs changing):
Currently, the page https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback says validation_data: Deprecated. Do not use., but if I attempt to access the attribute validation_data inside the callback I get the error AttributeError: 'MyCustomCallbackClass' object has no attribute 'validation_data'.. You should change the documentation to remove the attribute validation_data, which apparently was removed.
See also this issue https://github.com/tensorflow/tensorflow/issues/27318.

I would like to help with this issue. Is this just a case of having the documentation instead state that validation_data has been removed/ removing the line entirely?

@joshz123 Have you seen https://github.com/tensorflow/tensorflow/pull/36414?

@nbro no I had not, I guess that answers my question. Thanks!

@nbro
as there is pr to monitor this issue, pleas let us know if we can move this issue to closed status.

@Saduf2019 If the PR is supposed to solve the problem and integrate the solution into the next TF release, yes, you can close this issue, otherwise, you should leave it opened.

@nbro This was resolved already by this fix. I cannot see validation_data attribute in the  TF callback page.
Can you please verify once and close the issue if this was resolved for you. Thanks!

I still have this issue today. Could I know if it has been solved?

@ying-hao As this an old issue, can you please open a new issue with a simple standalone code to reproduce the issue? Thanks!


Adding Type Definition for isTransaction

Attempt to fix #4397 by adding type definition to types/index.d.ts . That's my first PR, if I should do it in another way, let me know.

thank you!


Removed default arguments from process_apps

What does this PR do?
While investigating a support case I added a bunch of unit tests that would be nice to merge on master. Also removed useless and confusing default values for few functions, hence the Fixed label.
Motivation
Support case investigation
Review checklist

[ ] PR has a meaningful title or PR has the no-changelog label attached
[ ] Feature or bugfix has tests
[ ] Git history is clean
[ ] If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Codecov Report

Merging #2337 into master will decrease coverage by 13.33%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #2337       +/-   ##
==========================================
- Coverage   85.93%   72.6%   -13.34%
==========================================
  Files         197       4      -193
  Lines       13490     219    -13271
  Branches     1455      42     -1413
==========================================
- Hits        11593     159    -11434
+ Misses       1484      53     -1431
+ Partials      413       7      -406


Terminate all containers within pod after main container completes

Resolves #1422

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

LGTM
I have occurs some times ,     when main container has finished,  but wait container still running .
@jessesuen Pls approve


replace sample link

This PR:

replaces the link of an archived sample with a newer sample
adds the link for one new sample
adds the link for one new sample

p.s. the new sample (2) is a React project, however I couldn't figure out how to add a React logo. I would appreciate some help with this : )

@derisen : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @rwike77 - Could you review the proposed updates to your article? Thanks.

@derisen  Hi, the updates look good.  I just pushed a react logo.  It's in the media folder with the other logos:  media/sample-v2-code/logo_react.png  So you can add a link to the react logo now.  Thanks.

@derisen  OK, I think we're all set here.  #sign-off

@rwike77 The image doesn't display in GitHub preview (although the others do), but I see it in master and it's referenced correctly, so we'll merge it and check for it on live later today.

It looks good on the live site 👍
https://docs.microsoft.com/en-us/azure/active-directory/develop/sample-v2-code


Bump openssh to 0.8, tokio to 1.0, and rand to 0.8

The current changes are insufficient. I was able to make it work (in a different branch) with a version of openssh that also uses tokio 1.0: https://github.com/vitorenesduarte/tsunami/commit/49e4ba27f0231e3159bd7bbe6f6a6786ca3a5ed2
I'll update this PR once there's a new version of openssh.

There's now a new version of openssh! :tada:

Awesome! This PR should be ready now.

I think we still need to hold off a bit here. Even after the bumps here, cargo tree tells me that we have some outstanding dependencies on old tokio:
$ cargo tree -i tokio:0.2.24
tokio v0.2.24
├── h2 v0.2.7
│   └── hyper v0.13.9
│       ├── hyper-tls v0.4.3
│       │   ├── reqwest v0.10.10
│       │   │   └── ubuntu-ami v0.1.2
│       │   │       └── tsunami v0.11.0-beta.10 (/home/jon/dev/minor/tsunami)
│       │   └── rusoto_core v0.45.0
│       │       ├── rusoto_ec2 v0.45.0
│       │       │   └── tsunami v0.11.0-beta.10 (/home/jon/dev/minor/tsunami)
│       │       ├── rusoto_sts v0.45.0
│       │       │   [dev-dependencies]
│       │       │   └── tsunami v0.11.0-beta.10 (/home/jon/dev/minor/tsunami)
│       │       └── tsunami v0.11.0-beta.10 (/home/jon/dev/minor/tsunami)
│       ├── reqwest v0.10.10 (*)
│       ├── rusoto_core v0.45.0 (*)
│       ├── rusoto_credential v0.45.0
│       │   ├── rusoto_core v0.45.0 (*)
│       │   └── rusoto_signature v0.45.0
│       │       └── rusoto_core v0.45.0 (*)
│       └── rusoto_signature v0.45.0 (*)

Specifically, through ubuntu-ami which uses reqwest 0.10: https://github.com/seanmonstar/reqwest/issues/1123), and through rusoto 0.45: https://github.com/rusoto/rusoto/issues/1845. We should probably wait for those, or the bump won't do much (and probably won't even work!)

Interesting. It was working for me, but probably only because I was using the baremetal provider. I'll keep an eye on those issues.

See also #115, which might be a way for us to get away from a dependency on hyper-tls.

You forgot the link 😛

Oh, ignore me, I got projects mixed up in my head when commenting :sweat_smile: I meant https://github.com/jonhoo/fantoccini/pull/115, but that's not relevant to this!

Once the following PRs are merged and new a version is released, we should be good to go:

[ ] https://github.com/akshayknarayan/ubuntu-ami/pull/1
[ ] https://github.com/rusoto/rusoto/pull/1880

Once the following PRs are merged and new a version is released, we should be good to go:

[ ] https://github.com/akshayknarayan/ubuntu-ami/pull/1
[ ] https://github.com/rusoto/rusoto/pull/1880

I merged https://github.com/akshayknarayan/ubuntu-ami/pull/1 and released it as v0.2.0: https://crates.io/crates/ubuntu-ami

I merged https://github.com/akshayknarayan/ubuntu-ami/pull/1 and released it as v0.2.0: https://crates.io/crates/ubuntu-ami

Awesome. Thanks @akshayknarayan!

Awesome. Thanks @akshayknarayan!

I just shipped rusoto v0.46.0!

I just shipped rusoto v0.46.0!

Great news! Thank you for your work @benesch.

Great news! Thank you for your work @benesch.

The CI failure is because we're still pinning nightly, and a fairly old one at that. You can probably update all instances of nightly-2020-05-15 to just say nightly now, and we should be good! And then we can do a follow-up PR that moves to running against stable and beta as well.

The CI failure is because we're still pinning nightly, and a fairly old one at that. You can probably update all instances of nightly-2020-05-15 to just say nightly now, and we should be good! And then we can do a follow-up PR that moves to running against stable and beta as well.

Thanks, it looks like it will pass this time. I'll also push a commit fixing/ignoring the clippy warnings I noticed in CI. Let me know if I should revert it or address those warnings in some other way.

Thanks, it looks like it will pass this time. I'll also push a commit fixing/ignoring the clippy warnings I noticed in CI. Let me know if I should revert it or address those warnings in some other way.

Oh, I was getting type_complexity (which is still on by default) and cyclomatic_complexity (which is now off by default) mixed up!

Oh, I was getting type_complexity (which is still on by default) and cyclomatic_complexity (which is now off by default) mixed up!

Awesome, thank you for the work! Will release shortly.

Awesome, thank you for the work! Will release shortly.

Released in 0.11.0 :tada:

Released in 0.11.0 :tada:


Possible to implement min/max zoom?

Would it be possible to implement a feature to constrain how far the user can zoom in or out?

Can you share a link to the native SDK docs that describe this feature (if it exists, of course)?

I'm not sure if this is the kind of links you are looking for...
Android: https://www.mapbox.com/android-docs/maps/overview/#mapview-xml-attributes
also: https://www.mapbox.com/android-docs/api/map-sdk/5.1.4/com/mapbox/mapboxsdk/maps/MapboxMapOptions.html#minZoomPreference-double-
iOS: https://www.mapbox.com/ios-sdk/api/4.5.0/Classes/MGLMapView.html#/c:objc(cs)MGLMapView(py)minimumZoomLevel

Looks good, thanks!

Any updates on this?
Thanks!

Any updates on this?
Thanks!

@NeriCodec are you still able to use this code snippet with the latest version of this plugin? (5.0.1)
export function onMapReady(args) {
    let map = args.map;
    let nativeMapView = map.ios ? map.ios : map.android;
}

But even before I am able to access the maximumZoomLevel property on iOS and the setMaxZoomPreference method on Android, it says that map.android / map.ios is undefined.

@felixkrautschuk Do you have a repository I can look at? Have you looked at the updated demos in here in github?

@Yermo thanks for your fast reply.
I checked the updated demo app (NativeScript Core + TypeScript) and there you try to access the native element th same way and it also returns "undefined" (for Android and iOS).
https://github.com/Yermo/nativescript-mapbox/blob/0781ed5a206c64a4e266f34a780d0572aaf1842b/demo/app/main-page.ts#L74

Ok. Thank you. I will check it out.

@Yermo I created a pull request for minZoom/maxZoom here https://github.com/Yermo/nativescript-mapbox/pull/365 and added that to the demo app as well. You can now call setMinZoom and setMaxZoom methods and set the minZoom and maxZoom properties from XML.
There is still the problem with getting the native instance with args.android (even though it is not related to the functionality within the pull-request).
On iOS, there is only one element, the MGLMapView instance which exposes all methods for interacting with the map.
On Android, you have the MapView and the MapBoxMap.
So I was not sure which one to set for this.nativeMapView.
Actually the MapBoxMap is the element for manipulation and interacting with the map.


Authentication error

Hi, I ran a PowerShell script to check bulk IP addresses from a .csv file format, It runs but does not generate an output file in the folder. I created a new key on abuseDB after creating an account. I am seeing an authentication failure on the PS console.
access denied (authentication_failed)
Your credential could not be authenticated: "credentials are missing." you will not be permitted access until your credentials are verified
I'm just wondering, apart from "Key": do I need to specify any username/password inside a script? or is there something else I am missing

You do not need to enter a username/password inside the script. Can you post a screenshot of your input as well as the Powershell console output?


Hide Configuration Button When Sheet is Locked

Considering the dramatic effect this has based on this being enabled or disabled, I believe the config option should be hidden (or at least disabled) when the sheet is locked.

Just pulled the changes so the checkbox is disabled when sheet is locked


C# any cpu

compiling C# .net for multiple cpu gives the ability for the process to be x86 on x86 machines and x64 on x64 machines, so how I can make easyhook work on both?

As long as you have all the EasyHook binaries it will automatically use the correct native binary for the target process architecture.

nice, how easyhook decides on runtime? using IsWow64Process api?

Yes

@spazzarama there is a way to put easyhook.dll(assembly), EasyHook32.dll and EasyHook64.dll inside program.exe?(some kind of static linking)

@chico-depressive short answer, yes it would be possible, but it will also be hard.
EasyHook32/EasyHook64 are native DLLs, so you would have to use some sort of load from resource file solution for those. Then if you need to get around the WOW64 boundary you are missing the Svc32/64.exe.
EasyHook.dll could in theory be merged using ILMerge, but not without code changes (I've done this before). Another item is EasyLoad32/64.dll.


Tests indicate races in cache and receiver

I'm not super familiar with the race detector in go, so I'm uncertain if this is indicative of a race in the test code vs a race in the library code being tested. Either way, I suspect it should be cleaned up and the race detector enabled in the "test" make target so it gets better coverage in travis.
$ go test -race ./cache
....

==================
PASS
Found 9 data race(s)
FAIL	github.com/lomik/go-carbon/cache	1.570s

$ go test -race ./receiver

....

==================
PASS
Found 19 data race(s)
FAIL	github.com/lomik/go-carbon/receiver	1.170s

After poking at this for a little bit I think the race is actually in logging.Test which allows an unsynchronized bytes.Buffer to be written/read in different goroutines. I'm actually a little puzzled by how the logging package is envisioned to be used since I see the logrus package directly imported and used all over the place and even a smattering of the standard library log gets used during the initialization of carbon-agent.
FWIW I /think/ we should be initializing a logrus logger instance in the lomik/go-carbon/logging package and then providing that logger to all the various packages that want to log things.
It looks like logrus also provides some decorated logger instances designed as testing shims. I'm going to take a quick crack at converting the racy tests to use these facilities directly for now and perhaps try and do a logging overhaul in another issue/pr.

Some races fixed here https://github.com/lomik/go-carbon/commit/87068714a87493500227cb50679186ca17942046
Nothing interesting, only tests code affected
$ go test -race ./cache
...
==================
PASS
Found 1 data race(s)
FAIL	go-carbon/cache	1.608s

$ go test -race ./receiver
ok    go-carbon/receiver 1.242s

There was only one problem
go test -race ./cache
==================
WARNING: DATA RACE
Read by goroutine 44:
  github.com/Sirupsen/logrus.(*Entry).Debugf()
...

Previous write by goroutine 39:
  github.com/Sirupsen/logrus.SetLevel()
...

==================
PASS
Found 1 data race(s)
FAIL	go-carbon/cache	1.602s

Race condition in logrus:  https://github.com/Sirupsen/logrus/issues/295
Also only affects tests. In normal operation, the logging level is made once at the start (before first log attempt from concurrent thread) and never changing.
There is a chance that the fix in the near future https://github.com/Sirupsen/logrus/pull/374

Great! I'm eager for this to be fixed so we can add race detection to standard Travis tests.


Allow Usage as a value type?

All the types in bitty seem to be ref objects, it's not clear why as there are no methods on BitArray and the type is only two pointers big anyway.

BitArray2D makes this even worse, it has two pointless allocations because the internal BitArray is a ref

It's very easy to make a mistake with a value type and have it copy the whole array, which can be very large.
But I do understand where you are coming from; the extra indirection can hurt you. You can always fork this library for your use case.


Error running example

Hello,
Thanks for your contribution, I am trying to run it in python 2.7, anaconda distribution, following your example and I get the following error when running the cluster_ensembles command:
Traceback (most recent call last):
File "", line 1, in
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 297, in cluster_ensembles
store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 190, in store_hypergraph_adjacency
FILTERS = get_compression_filter(byte_counts)
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 138, in get_compression_filter
if 2 * byte_counts > 1000 * memory()['free']:
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 63, in memory
with open('/proc/meminfo') as file:
IOError: [Errno 2] No such file or directory: '/proc/meminfo'
Do you have any idea what could be the problem?
Thanks

/proc/meminfo is the path to a file in Linux, holding information about
free memory.
Cluster_Ensembles was never meant to be compatible with Windows or OS X.
A fix would involve using the psutil module to change the getChunkSize from
the Cluster_Ensembles package:

import psutil
mem = psutil.virtual_memory()
free = mem['free']

Gregory
On Oct 16, 2016 11:57 AM, "Omar Guerrero"<EMAIL_ADDRESS>wrote:

Hello,
Thanks for your contribution, I am trying to run it in python 2.7,
anaconda distribution, following your example and I get the following error
when running the cluster_ensembles command:
Traceback (most recent call last):
File "", line 1, in
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 297, in cluster_ensembles
store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 190, in
store_hypergraph_adjacency
FILTERS = get_compression_filter(byte_counts)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 138, in
get_compression_filter
if 2 * byte_counts > 1000 * memory()['free']:
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 63, in memory
with open('/proc/meminfo') as file:
IOError: [Errno 2] No such file or directory: '/proc/meminfo'
Do you have any idea what could be the problem?
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AK3j51SWrg_fjVkmez2dRzLEtgLJX5Mfks5q0kl7gaJpZM4KYAWm
.

Thanks a lot. This got me one step closer. Now I am getting the following error:
Traceback (most recent call last):
File "", line 1, in
File "/Users/me/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile
execfile(filename, namespace)
File "/Users/me/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
builtins.execfile(filename, *where)
File "/Users/me/Dropbox/Projects/dynamicLogrolling/code/04 - qualityFunction.py", line 100, in
consensus_clustering_labels = CE.cluster_ensembles(MC)
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 301, in cluster_ensembles
cluster_ensemble.append(consensus_functions[i](hdf5_file_name, cluster_runs, verbose, N_clusters_max))
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 615, in CSPA
return metis(hdf5_file_name, N_clusters_max)
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 938, in metis
labels = sgraph(N_clusters_max, file_name)
File "/Users/me/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 1202, in sgraph
with open(out_name, 'r') as file:
IOError: [Errno 2] No such file or directory: 'wgraph_CSPA.part.370'
Any ideas?

Check that gpmetis is up and running on your machine.
Gregory
On Oct 17, 2016 6:00 AM, "Omar Guerrero"<EMAIL_ADDRESS>wrote:

Thanks a lot. This got me one step closer. Now I am getting the following
error:
Traceback (most recent call last):
File "", line 1, in
File "/Users/me/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py",
line 866, in runfile
execfile(filename, namespace)
File "/Users/me/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py",
line 94, in execfile
builtins.execfile(filename, *where)
File "/Users/me/Dropbox/Projects/dynamicLogrolling/code/04 -
qualityFunction.py", line 100, in
consensus_clustering_labels = CE.cluster_ensembles(MC)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 301, in cluster_ensembles
cluster_ensemble.append(consensus_functionsi
http://hdf5_file_name, cluster_runs, verbose, N_clusters_max)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 615, in CSPA
return metis(hdf5_file_name, N_clusters_max)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 938, in metis
labels = sgraph(N_clusters_max, file_name)
File "/Users/me/anaconda/lib/python2.7/site-packages/
Cluster_Ensembles/Cluster_Ensembles.py", line 1202, in sgraph
with open(out_name, 'r') as file:
IOError: [Errno 2] No such file or directory: 'wgraph_CSPA.part.370'
Any ideas?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-254165145,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j57X9tgC66yHTvykBBxlRNcdoKwJ3ks5q00cngaJpZM4KYAWm
.

Thanks Gregory.
@oguerrer, here are what I did to make it work in Mac system:

Change 'memory' function in Cluster_Ensembles.py in Cluster_Ensembles packages (e.g., ~/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/Cluster_Ensembles.py)
import psutil
mem = psutil.virtual_memory()
mem_info['free'] = mem.free
mem_info['used'] = mem.used
Download METIS binary file 'gpmetis' from https://github.com/mjdousti/Squash/tree/master/metis/mac
Add the path (e.g., ~/bin) to gpmetis in .bash_profile
export PATH="~/bin:$PATH"

Exactly right.
Note that this issue doesn't arise for shmetis, which Cluster_Ensembles
compiles on your machine.
Gregory
On Oct 21, 2016 5:03 PM, "Chih-Hsu Lin"<EMAIL_ADDRESS>wrote:

Thanks Gregory.
@oguerrer https://github.com/oguerrer, here are what I did to make it
work in Mac system:

Change 'memory' function in Cluster_Ensembles.py in Cluster_Ensembles
packages (e.g., ~/anaconda/lib/python2.7/site-packages/Cluster_Ensembles/
Cluster_Ensembles.py)
import psutil
mem = psutil.virtual_memory()
mem_info['free'] = mem.free
mem_info['used'] = mem.used
Download METIS binary file 'gpmetis' from https://github.com/mjdousti/
Squash/tree/master/metis/mac
Add the path (e.g., ~/bin) to gpmetis in .bash_profile
export PATH="~/bin:$PATH"

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-255464701,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j5ylKw34T1M-z0sP4_gRODenluE79ks5q2SizgaJpZM4KYAWm
.

Thank you guys for the conversation - it helped me to run the library under OSX.
Unfortunately I am having the same issue with the last error: FileNotFoundError: [Errno 2] No such file or directory: 'wgraph_CSPA.part.50'.
I have experimented with different ways to add path to file gpmetis and finally Ive put it into usr/local/bin and now Im getting same error saying "Permission denied" both in terminal and in PyCharm.
Would appreciate any help!

You might need to change the permission of the file gpmetis.
Go to the directory, e.g., cd usr/local/bin
chmod 755 gpmetis

Thank you, ChihHsuLin! It worked.

There is another problem arising:
File "/usr/local/lib/python3.5/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 309, in cluster_ensembles
cluster_ensemble.append(consensus_functions[i](hdf5_file_name, cluster_runs, verbose, N_clusters_max))
File "/usr/local/lib/python3.5/site-packages/Cluster_Ensembles/Cluster_Ensembles.py", line 727, in MCLA
temp = squared_MCLA[i:min(i+chunks_size, N_rows), :].todense()
File "/usr/local/lib/python3.5/site-packages/scipy/sparse/csr.py", line 285, in getitem
return self._get_submatrix(row, col)
File "/usr/local/lib/python3.5/site-packages/scipy/sparse/csr.py", line 428, in _get_submatrix
check_bounds(i0, i1, M)
File "/usr/local/lib/python3.5/site-packages/scipy/sparse/csr.py", line 424, in check_bounds
" %d <= %d" % (i0, num, i1, num, i0, i1))
IndexError: index out of bounds: 0 <= 184973695 <= 2387, 0 <= 2387 <= 2387, 184973695 <= 2387
I have not installed metis library (there is a DLL issue), just used the bin file gpmetis.
Probably Gregory meant it when he wrote that  Cluster_Ensembles has never meant to run under OSX

Hello,.Thank you for helping.I am trying to run it in python 2.7 on windows,how can I make it work.I have no idea to get a solution. I'm a newer.
I'm looking forward to your help.Thanks a lot.

As mentioned in Cluster_Ensemble's PyPI and GitHub documentations, only
POSIX-compatible operating systems are supported.
Please try on Linux or OS X and let me know of any issue you might
encounter then.
Gregory
On May 17, 2017 8:16 PM, "stshi8808"<EMAIL_ADDRESS>wrote:

Hello,.Thank you for helping.I am trying to run it in python 2.7 on
windows,how can I make it work.I have no idea to get a solution. I'm a
newer.
I'm looking forward to your help.Thanks a lot.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-302265800,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j50pbQSFdGNcvbPtsTfW9FCeY0EB2ks5r643kgaJpZM4KYAWm
.

Thanks a lot for helping me. When I use the package,I want to get each labels results of the samples with CSPA,HGPA,MCLA. I try to do like this as an example:
import numpy as np
import Cluster_Ensembles as CE
cluster_runs = np.random.randint(0, 8, (11, 50))
CSPA_clustering_labels = CE.CSPA(**hdf5_file_name**,cluster_runs, verbose = True, N_clusters_max = 8)
print CSPA_clustering_labels

But I don't know how to set the first Parameter(hdf5_file_name),because I don't have a hdf5_file.Could you give me the answer of above example?I am really newer in python.
Thanks a lot.I'm looking forward to your help.Thanks a lot.

Please refer to the documentation of PyTables for installation of this
package, its dependencies (including the HDF5 C library and numexpr), as
well as basic I/O operations with *.h5 files.
Gregory
On May 31, 2017 9:33 AM, "stshi8808"<EMAIL_ADDRESS>wrote:

Thanks a lot for helping me. When I use the package,I want to get each
labels results of the samples with CSPA,HGPA,MCLA. I try to do like this as
an example:
import numpy as np
import Cluster_Ensembles as CE
cluster_runs = np.random.randint(0, 8, (11, 50))
CSPA_clustering_labels = CE.CSPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)
print CSPA_clustering_labels
But I don't know how to set the first Parameter(hdf5_file_name),because I
don't have a hdf5_file.Could you give me the answer of above example?I am
really newer in python.
Thanks a lot.I'm looking forward to your help.Thanks a lot.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-305187774,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j5yD7qZXg2H9mpwl08cfxzx6m9tWfks5r_Ww3gaJpZM4KYAWm
.

Thanks Gregory，I tried the following code with referring to the Cluster_ensembles function to build the .h5 file, like this:
import numpy as np
import Cluster_Ensembles as CE
import tables
cluster_runs = np.random.randint(0, 8, (15, 150))
hdf5_file_name = './Cluster_Ensembles.h5'
fileh = tables.open_file(hdf5_file_name, 'w')
fileh.create_group(fileh.root, 'consensus_group')
fileh.close()
hypergraph_adjacency = CE.build_hypergraph_adjacency(cluster_runs)
CE.store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
consensus_clustering_labels = CE.CSPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)

But the error is
AttributeError: 'module' object has no attribute 'build_hypergraph_adjacency'
so the build_hypergraph_adjacency can not be imported?How can I correct it？
Thanks a lot.

These days I tried to use Cluster_Ensembles package to do my paper. I want to get each labels results of the samples with CSPA,HGPA,MCLA. I try to do like this as an example:
import numpy as np
import Cluster_Ensembles as CE
import tables
cluster_runs = np.random.randint(0, 8, (15, 150))
hdf5_file_name = './Cluster_Ensembles.h5'
fileh = tables.open_file(hdf5_file_name, 'w')
fileh.create_group(fileh.root, 'consensus_group')
fileh.close()
hypergraph_adjacency = CE.build_hypergraph_adjacency(cluster_runs)
CE.store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
consensus_clustering_labels = CE.HGPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)

But there is an error:IOError: [Errno 2] No such file or directory: 'wgraph_HGPA.part.4'
But the MCLA and CSPA are available to get the result,only HGPA can't work.Can you help me?

Possibly with some more detail?
On Jun 1, 2017 2:02 AM, "stshi8808"<EMAIL_ADDRESS>wrote:

These days I tried to use Cluster_Ensembles package to do my paper. I want
to get each labels results of the samples with CSPA,HGPA,MCLA. I try to do
like this as an example:
import numpy as np
import Cluster_Ensembles as CE
import tables
cluster_runs = np.random.randint(0, 8, (15, 150))
hdf5_file_name = './Cluster_Ensembles.h5'
fileh = tables.open_file(hdf5_file_name, 'w')
fileh.create_group(fileh.root, 'consensus_group')
fileh.close()
hypergraph_adjacency = CE.build_hypergraph_adjacency(cluster_runs)
CE.store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
consensus_clustering_labels = CE.HGPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)
But there is an error:IOError: [Errno 2] No such file or directory:
'wgraph_HGPA.part.4'
But the MCLA and CSPA are available to get the result,only HGPA can't
work.Can you help me?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-305397087,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j56hXNU5wU6vtIImKHHEh6kw0md1_ks5r_lPogaJpZM4KYAWm
.

Can you try the following code:
import numpy as np
import Cluster_Ensembles as CE
import tables
cluster_runs = np.random.randint(0, 8, (15, 150))
hdf5_file_name = './Cluster_Ensembles.h5'
fileh = tables.open_file(hdf5_file_name, 'w')
fileh.create_group(fileh.root, 'consensus_group')
fileh.close()
hypergraph_adjacency = CE.build_hypergraph_adjacency(cluster_runs)
CE.store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
consensus_clustering_labels = CE.HGPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)
print consensus_clustering_labels

The output : IOError: [Errno 2] No such file or directory: 'wgraph_HGPA.part.4' ,I don't know how to solve this problerm

Thanks Gregory,I try the these coede:
import numpy as np
import Cluster_Ensembles as CE
import tables
cluster_runs = np.random.randint(0, 8, (15, 150))
hdf5_file_name = './Cluster_Ensembles.h5'
fileh = tables.open_file(hdf5_file_name, 'w')
fileh.create_group(fileh.root, 'consensus_group')
fileh.close()
hypergraph_adjacency = CE.build_hypergraph_adjacency(cluster_runs)
CE.store_hypergraph_adjacency(hypergraph_adjacency, hdf5_file_name)
consensus_clustering_labels = CE.CSPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 8)
print consensus_clustering_labels

It can work successfully. Also MCLA is also successful. Only HGPA has that error:No such file or directory: 'wgraph_HGPA.part.4' .
Hope your reply, sorry to dsiturb.

The output is
*****
INFO: Cluster_Ensembles: HGPA: consensus clustering using HGPA.

#
INFO: Cluster_Ensembles: wgraph: writing wgraph_HGPA.
INFO: Cluster_Ensembles: wgraph: 15 vertices and 23 non-zero hyper-edges.
#

#
INFO: Cluster_Ensembles: sgraph: calling shmetis for hypergraph partitioning.
/bin/sh: 1: /home/shituo/Hypergraph_Partitioning/hmetis-1.5-linux/shmetis: not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shituo/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 922, in runfile
    execfile(filename, namespace)
  File "/home/shituo/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 95, in execfile
    builtins.execfile(filename, *where)
  File "/home/shituo/crazy.py", line 1295, in <module>
    consensus_clustering_labels = HGPA(hdf5_file_name,cluster_runs, verbose = True, N_clusters_max = 4)
  File "/home/shituo/crazy.py", line 622, in HGPA
    return hmetis(hdf5_file_name, N_clusters_max)
  File "/home/shituo/crazy.py", line 947, in hmetis
    labels = sgraph(N_clusters_max, file_name)
  File "/home/shituo/crazy.py", line 1175, in sgraph
    with open(out_name, 'r') as file:
IOError: [Errno 2] No such file or directory: 'wgraph_HGPA.part.4'

Where can I get shmetis?

I find the shmetis, It can work now.Thanks a lot

That was a common problem. Metis and shmetis are supposed to be compiled
when installing Cluster_Ensemble. On some machines (including yours), they
do not. Marco Sardelich posted a little fix some time ago on
Cluster_Ensemble page on GitHub.
Glad it works now and all the best with your research.
Gregory
On Jun 1, 2017 6:56 AM, "stshi8808"<EMAIL_ADDRESS>wrote:

I find the shmetis, It can work now.Thanks a lot
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/GGiecold/Cluster_Ensembles/issues/5#issuecomment-305460017,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3j5wCMoH4j18I3kBc-36cpfwkKzDIvks5r_pjGgaJpZM4KYAWm
.

Please help me with the second error. I have tried so many ways to add path but I am still getting the error:
IOError: [Errno 2] No such file or directory: 'wgraph_CSPA.part.50'
I am working on a Mac. I downloaded gpmetis.dms but I have no clue what to do with it. Should I run it or should I put it in the right directory. Please help.

@FatimaAnwer you have to mv the 'gpmetis' file to dir:  /usr/local/bin
and 👍
chmod 755 gpmetis

@GGiecold @ChihHsuLin @oguerrer @isvanilin @stshi8808
Please help me.
I want to use Cluster_Ensembles library for my research. I am using google colab, but when I use this command  "consensus_clustering_labels=CE.cluster_ensembles(final_ensemble,N_clusters_max=int(np.max(final_ensemble)+1))"
I got this error:

INFO: Cluster_Ensembles: CSPA: consensus clustering using CSPA.

INFO: Cluster_Ensembles: wgraph: writing wgraph_CSPA.

INFO: Cluster_Ensembles: sgraph: calling gpmetis for graph partitioning.
IOErrorTraceback (most recent call last)
 in ()
3 #from functools import reduce
4
----> 5 consensus_clustering_labels=CE.cluster_ensembles(final_ensemble,N_clusters_max=int(np.max(final_ensemble)+1))
3 frames
/usr/local/lib/python2.7/dist-packages/Cluster_Ensembles/Cluster_Ensembles.pyc in sgraph(N_clusters_max, file_name)
1199
1200     labels = np.empty(0, dtype = int)
-> 1201     with open(out_name, 'r') as file:
1202         print("INFO: Cluster_Ensembles: sgraph: (hyper)-graph partitioning completed; "
1203               "loading {}".format(out_name))
IOError: [Errno 2] No such file or directory: 'wgraph_CSPA.part.15'
I read the last comments, but I am still confused about which file "gpmetis or shmetis" I must download and how can I add this file to a path in google colab?


Not working with Nano Every

Hi, this looks like a really good library :) but I can't get anything to compile for the Nano Every.  AVR dude seems to have problems with the get clock (what you have listed as a prototype) and set clock functions and a few other things I think related to setting clocks for the I2C bus. I tried to copy the error messages but I not able to. Is there maybe something I can change in the header file to get it working?  Thanks! :)  Rainer S.

Thanks for reporting the issue.
I am not familiar with the Nano Every. The getClock is indeed experimental as it is not a standard implement  feature on all boards. Tomorrow I will do a first investigation (here it is 3.30 AM now) if time permits. Otherwise it will be Saturday.
Do you have a link to the board?

Found the board, so link is not needed.

OK, Thanks for looking into this! :)
Cheers,
Rainer S.
On Thu, 20 Jan 2022 18:36:51 -0800, Rob Tillaart @.***> wrote:
Found the board, so link is not needed.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: @.***>

installed NANO  every board
compiled the minimum example and got errors

TWBR => this is an Arduino UNO low level I2C register.  I understand this issue,
_wire->requestFrom(address, (uint8_t) 2);  error =>  casting error, also understood

Both seems easy fixable however I do not want to introduce a problem for the existing platforms.
I will create a develop branch in which this I try to solve the issue.
Can you do some testing with the real hardware?

Problem 1 is related to this issue - https://github.com/arduino/Arduino/issues/11457

Please verify if develop branch works for you - https://github.com/RobTillaart/ADS1X15/tree/develop
It compiles and I did not found regression of other platforms.

Sorry I just woke up and I have a dentist appointment this morning, but I'll try this out ASAP! :)
On Fri, 21 Jan 2022 05:43:48 -0800, Rob Tillaart @.***> wrote:
Please verify if develop branch works for you - https://github.com/RobTillaart/ADS1X15/tree/develop
It compiles and I did not found regression of other platforms.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: @.***>

No problem, take your time
Good luck at the dentist.

I tested both minimum and performance sketches with ADS1015 and Nano Every and all is good! :)
Thanks you very much :)
Cheers,
Rainer S.

Good to hear,
I will merge and create a new release

Released 0.3.5 !


PyTorch implementation of TimesFM

I implement a PyTorch version of TimesFM here. It includes the essential components required to operate the model effectively. Hope it helps. :D

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

This is awesome, @tangh18! 🎉
When running python convert_ckpt.py, I'm seeing
jaxlib.xla_extension.XlaRuntimeError: FAILED PRECONDITION: DNN library initialization failed.
Checking if there are any ideas about how to resolve this? JAX version is 0.4.26, CUDA version is 12.4, and cuDNN version is <IP_ADDRESS>, which should be compatible.

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

Thanks for your attention! I conducted tests on an A100 server, running the process 45 times with each run consisting of a batch size of 32 and a context length of 2880. The processing time with PyTorch was 3.89 seconds, while JAX completed the task in 1.61 seconds. However, this comparison may not be entirely fair. The PyTorch version did not handle paddings, and its implementation was also not optimized.

This is awesome, @tangh18! 🎉
When running python convert_ckpt.py, I'm seeing jaxlib.xla_extension.XlaRuntimeError: FAILED PRECONDITION: DNN library initialization failed.
Checking if there are any ideas about how to resolve this? JAX version is 0.4.26, CUDA version is 12.4, and cuDNN version is <IP_ADDRESS>, which should be compatible.

The code in convert_ckpt.py about JAX only involves creating the model and loading the checkpoint. Have you successfully run timesfm in JAX? If so, the issue you're encountering should not occur.

This is awesome, @tangh18! 🎉
When running python convert_ckpt.py, I'm seeing jaxlib.xla_extension.XlaRuntimeError: FAILED PRECONDITION: DNN library initialization failed.
Checking if there are any ideas about how to resolve this? JAX version is 0.4.26, CUDA version is 12.4, and cuDNN version is <IP_ADDRESS>, which should be compatible.

The code in convert_ckpt.py about JAX only involves creating the model and loading the checkpoint. Have you successfully run timesfm in JAX? If so, the issue you're encountering should not occur.

Thanks for your super cool contribution! I think what @sebastianpinedaar means is performance in terms of accuracy of the model. Have you been able to reproduce the accuracy metrics reported in the original paper?

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

Thanks for your attention! I conducted tests on an A100 server, running the process 45 times with each run consisting of a batch size of 32 and a context length of 2880. The processing time with PyTorch was 3.89 seconds, while JAX completed the task in 1.61 seconds. However, this comparison may not be entirely fair. The PyTorch version did not handle paddings, and its implementation was also not optimized.

Thanks for the info @tangh18! As @melopeo pointed out, I was rather curious about the MAE and scaled MAE performance comparison between the original jax TimesFM and the pytorch version, at least in a couple of datasets. I mean this as a sanity check. Although there are some difference in the preprocessing, hopefully the difference is not too big.

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

Thanks for your attention! I conducted tests on an A100 server, running the process 45 times with each run consisting of a batch size of 32 and a context length of 2880. The processing time with PyTorch was 3.89 seconds, while JAX completed the task in 1.61 seconds. However, this comparison may not be entirely fair. The PyTorch version did not handle paddings, and its implementation was also not optimized.

Thanks for the info @tangh18! As @melopeo pointed out, I was rather curious about the MAE and scaled MAE performance comparison between the original jax TimesFM and the pytorch version, at least in a couple of datasets. I mean this as a sanity check. Although there are some difference in the preprocessing, hopefully the difference is not too big.

Although using the official PyTorch implementation, I took the same weight conversion processing and was able to reproduce the results like this. Hopefully this helps!
'mse': np.float32(0.4324413),
'smape': np.float32(0.7251805),
'mae': np.float32(0.40476117),
'wape': np.float32(11.708796),
'nrmse': np.float32(19.022907),
'num_elements': 20160,
'abs_sum': np.float32(696.91077),
'dataset': 'etth1',
'freq': 'h',
'pred_len': 96,
'context_len': 512

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

Thanks for your attention! I conducted tests on an A100 server, running the process 45 times with each run consisting of a batch size of 32 and a context length of 2880. The processing time with PyTorch was 3.89 seconds, while JAX completed the task in 1.61 seconds. However, this comparison may not be entirely fair. The PyTorch version did not handle paddings, and its implementation was also not optimized.

Thanks for the info @tangh18! As @melopeo pointed out, I was rather curious about the MAE and scaled MAE performance comparison between the original jax TimesFM and the pytorch version, at least in a couple of datasets. I mean this as a sanity check. Although there are some difference in the preprocessing, hopefully the difference is not too big.

Although using the official PyTorch implementation, I took the same weight conversion processing and was able to reproduce the results like this. Hopefully this helps!
'mse': np.float32(0.4324413),
'smape': np.float32(0.7251805),
'mae': np.float32(0.40476117),
'wape': np.float32(11.708796),
'nrmse': np.float32(19.022907),
'num_elements': 20160,
'abs_sum': np.float32(696.91077),
'dataset': 'etth1',
'freq': 'h',
'pred_len': 96,
'context_len': 512

I haven't learned how to build models with JAX, and I would like to ask how to convert the weights from the JAX checkpoints into a format that can be loaded into a PyTorch model, if I'm using the PyTorch version of the model provided on the official pytorch models.

nice! thanks for this! Did you have a chance to compare the performance between the original and the new implementation performance?

Thanks for your attention! I conducted tests on an A100 server, running the process 45 times with each run consisting of a batch size of 32 and a context length of 2880. The processing time with PyTorch was 3.89 seconds, while JAX completed the task in 1.61 seconds. However, this comparison may not be entirely fair. The PyTorch version did not handle paddings, and its implementation was also not optimized.

Thanks for the info @tangh18! As @melopeo pointed out, I was rather curious about the MAE and scaled MAE performance comparison between the original jax TimesFM and the pytorch version, at least in a couple of datasets. I mean this as a sanity check. Although there are some difference in the preprocessing, hopefully the difference is not too big.

Although using the official PyTorch implementation, I took the same weight conversion processing and was able to reproduce the results like this. Hopefully this helps!
'mse': np.float32(0.4324413),
'smape': np.float32(0.7251805),
'mae': np.float32(0.40476117),
'wape': np.float32(11.708796),
'nrmse': np.float32(19.022907),
'num_elements': 20160,
'abs_sum': np.float32(696.91077),
'dataset': 'etth1',
'freq': 'h',
'pred_len': 96,
'context_len': 512

I haven't learned how to build models with JAX, and I would like to ask how to convert the weights from the JAX checkpoints into a format that can be loaded into a PyTorch model, if I'm using the PyTorch version of the model provided on the official pytorch models.

https://gist.github.com/TeddyHuang-00/fc2238f6f5956a9906c8c206edef2603
You are welcome :smile:


Connection recovery problem less than minimum Idle

When kill all connection from database, Hikari recover connections that less than has been configed.
For example, if I set setMinimumIdle(10) after kill all ten connection from database Hikari will create 9 connection. No more or less.
Environment
HikariCP version: 3.1.0
JDK version     : 1.8.0_111
Database        : |MySQL|... unchecked others
Driver version  : 5.1.6

@zaurhasanov I have added an additional unit test and am unable to reproduce the issue you have reported.


Chrome Web Store EU Trader/Non-Trader developer identification - republished

Hi @ruthtxh I was checking latest user geographical distribution for TagUI Chrome extension (to update user distribution map), and I saw a prompt asking me if I'm a trader or non-trader. See the definition here. This looks like an EU prompt for EU consumer protection. I chose non-trader because it will ask me for a business address which is obviously invalid for me.
Raising an issue here so that you can try out and see if you want to update the Chrome extension account settings as trader. It will require entering AI Singapore's business address. The pro of updating I believe is it will not prompt EU users of something like below, base on the definition link above.

If you are a non-trader, the consumer will be informed that consumer rights stemming from consumer protection laws may not apply to any contracts between you and the consumer. Please note that it is the developer’s responsibility to accurately self-declare their trader/non-trader status.

The link to the Chrome extension developer console is https://chrome.google.com/webstore/devconsole and it is under AI Singapore group publisher account at<EMAIL_ADDRESS>which I believe you and Najib are already inside.

Want to add that I'm not recommending to choose trader or non-trader, but wants to highlight that there is such a thing for EU market. And you guys can weigh whether to choose either, or just ignore since if there's a problem users would feedback by now.

Hi @kensoh noted on the trader account issue, we are looking into this.
We are also looking into the possibility of changing the group publisher account to an internal group instead of the current external group (tagui-rpa@googlegroups.com), I will let you know if I require any information from you. Thanks!

Sure! Let me know if any info or action required from my side :)

Hi @kensoh , we are republishing the chrome extension with a new publisher account that is tied to AISG instead and as such we will track the downloads from the new source. Will link it up when it is ready, thanks!

Oh I see, the<EMAIL_ADDRESS>is already tied to AISG and is created by Najib. I assume you meant that there is a new one that maybe ends with @aisingapore.org. I got the following email from Google saying that the app will be deleted.
Fyi, to see if you would want to publish the new extension first successfully, before deleting the old one, and finally mapping all links to the new one. Otherwise there may be a gap between deleting and the new one being available. I haven't publish for a long time so there may be new requirements including changing some parts of the code base / config for an app to be able to be published.

Hi there,
We've received your request to permanently delete the group publisher account<EMAIL_ADDRESS>As part of the deletion process, all extensions, themes and apps associated with this group publisher will also be permanently deleted.
You can learn more about group publisher deletion here.
Sincerely,
Chrome Web Store Developer Support

@kensoh I see, I have already feedback to Najib about the group publisher. I've updated the link to the new extension here and yes some codes had to be change to suit the new manifest format as well as the api, will update the backup source code on github soon.

Oh @ruthtxh nice republishing the extension, making it compliant and all the styling touches!
Since I've known you, you have been great at tackling on new challenges and tools. Thumbs up 👍🏻👍🏻

Hey does this mean that if we have a non-trader account and we publish an extension under it, we can't post-hoc swap the account to a trader account?
And will thus lose all of our userbase if we decide to switch? Am I understanding this correctly?

Closing but please ping back if this is still relevant or an issue.


Scrape reddit

TODO: check TODO in code
TODO: more then 100 posts
TODO: Comments
TODO: more coins and more subreddits

@FloBaumgartner should also review this when the PR is ready, since it's your project and i'm only interested on the functionality. But i can advise you to install these three vscode extensions:

Pylint - usual linting for code
Mypy - advanced type checking, might need to pip install the mypy module on your python env
sourcery - more advanced code recommendations for cleaner code

this will improve your code drastically and correct typing will keep you from unknowingly accessing None values which might throw errors

i would also advice you to get some inspiration from current scrapers in this repo, for example return a dict instead of a dataframe, so it's consistent across the repo


From private repo

@dougeby : Thanks for your contribution! The author, @andredm7, has been notified to review your proposed change.


chore(package): update standard to v12

resolves #159
This PR updates standard to v12 (see the v12 changelog here).
After installing v2, I ran npx standard --fix to resolve violations that standard was newly reporting.

Thanks @macklinu!


DMA + HBM with one channel +  Top-n-Bottom 4 CUDA

Large data vectors can be written directly to an EVA in DRAM
This includes a new CUDA DMA read/write interface - 1GB 'o data in COSIM? No problem!
The number of used memory channels is configurable when using DRAMSim3 (within reason... don't go too crazy).
Chip configuration has victim caches on both the north and south side of the manycore array with the IO interface at XY=(0,1)
Fixes a bunch of tests that had hardcoded values for target tile coordinates.
Low-level API for invalidating, validating, and flushing every set in victim cache.
Adds some helper functions for low-level vcache operations (i.e. address of a specific way).
Disables the micron memory model when using ideal memory.
Low-level API function hb_mc_config_get_dram_y is deprecated

Passes F1 regression, except 1 test (linker error, not hardware related)

unless anyone objects I'm going to go ahead and merge this.


Corrected contribution links

Hey,
i noticed that the contributing links in the readme were linking to the wrong file.

Ups... Thank you!


502 on /report/github.com/minon*

I noticed that the badges to my projects stopped working (some time ago actually, just getting around to creating this issue...) and I get a 502 when I navigate to my project's pages on goreportcard.com. Other projects are working perfectly fine, so this prompted me to play around with the URL and I ended up noticing that I get a 502 on any URL that matches /report/github.com/minon (notice how https://goreportcard.com/report/github.com/mino results in a standard application error but https://goreportcard.com/report/github.com/minon results in a 502 from nginx). This is problematic for me since my username is "minond" -- ha! I ran goreportcard locally and I'm able to successfully generate and view reports for all of my repos, so I'm guessing that the request isn't getting past nginx:
Loading report on https://goreportcard.com/report/github.com/minond/captainslog:

Loading report for same repo on instance of goreportcard running locally:

Any way, thanks for maintaining goreportcard, I really like this tool.

Hi @minond
We were having some file corruption issues with boltdb, so we switched to dgraph's badger. But it looks like we're still using boltdb in one place, and the code went down that path and panicked when accessing the boltdb file.
I'll look into removing the last pieces of boltdb, but I'm not sure when I can get to it -- hopefully in the next couple of days.

@minond could you try again? I've tried a couple of your repos and it seems to work now.

It’s working, thanks for looking into this!


handle nil in config options (broken in 1.2.2)

Context
In rake-compiler 1.2.1 and earlier, if an extension did something like this, rake-compiler handled it gracefully:
  Rake::ExtensionTask.new("gem", spec) do |ext|
    ext.config_options << ENV["EXTOPTS"] # this is probably going to be nil
  end

It is OK to do this because in 1.2.1 and earlier, the command array was executed as:
  sh cmd.join(' ')

However, this was changed in 1.2.2 to
  sh *cmd

which of course is preferable. However, this breaks extensions like nokogiri that do something silly like above.
Details
I've fixed Nokogiri (see https://github.com/sparklemotion/nokogiri/pull/2894) but I do think that there may be other gems impacted by this change.
This PR does a few things:

extract a method make_makefile_cmd so that we can unit test it
backfill basic test coverage
backfill test for #215
write a test for this nil case, and fix it by returning the result of cmd.compact

This is a big PR because of the extraction and test coverage. Please let me know if you would prefer this to be done a different way.

Thanks!


Eco mode power is unavailable

In version <IP_ADDRESS> the Eco charge power setting is unavailable. Furthermore, Eco charge mode and Eco discharge mode cannot be enabled.
Version <IP_ADDRESS> is working properly
(On device goodwe gw10k-et)

Error message (Failed to call service select/select _option. "' not supported between instances of 'None Type' and 'int')

The individual items are enabled upon request to inverter (to check if it is supported by the inverter model/fw).
Since the UDP is unreliable, sometimes it happens that even after 3 retries the proper response is not received.
Could you restart the HA to ensure it was not case like this ?

Actually, there seems to be really a bug there ! Going to fix it.

Fixed in v<IP_ADDRESS>


Release 0.5.0

Changes Unknown when pulling fa77ea9a8f0f176e1f4584ad0af66d7d9bceb0c6 on mparke:release-0.5.0 into ** on linode:release**.


Numba error running on large datasets

When running the code on main, tests run fine, but when running on "real" data, I get:
  File "/home/adam/Desktop/cellfinder-core/src/cellfinder_core/main.py", line 69, in main
    points = detect.main(
  File "/home/adam/Desktop/cellfinder-core/src/cellfinder_core/detect/detect.py", line 167, in main
    cells = mp_3d_filter.process(async_results, callback=callback)
  File "/home/adam/Desktop/cellfinder-core/src/cellfinder_core/detect/filters/volume/volume_filter.py", line 94, in process
    self._run_filter()
  File "/home/adam/Desktop/cellfinder-core/src/cellfinder_core/detect/filters/volume/volume_filter.py", line 113, in _run_filter
    self.cell_detector.process(middle_plane)
  File "/home/adam/miniconda3/envs/numbatest/lib/python3.10/site-packages/numba/experimental/jitclass/boxing.py", line 61, in wrapper
    return method(*args, **kwargs)
  File "/home/adam/miniconda3/envs/numbatest/lib/python3.10/site-packages/numba/typed/dictobject.py", line 760, in impl
    raise KeyError()

The code run is:
if __name__ == '__main__':
    from cellfinder_core.main import main as cellfinder_run
    from cellfinder_core.tools.IO import read_with_dask
    signal_array = read_with_dask("/media/adam/DATA_4TB/data/MS_cx_left/stitchedImages_100/2")
    background_array = read_with_dask("/media/adam/DATA_4TB/data/MS_cx_left/stitchedImages_100/4")

    voxel_sizes = [5, 2, 2]
    detected_cells = cellfinder_run(signal_array, background_array, voxel_sizes)

The data is fairly standard serial two-photon whole-brain image used to test cellfinder many times before. This works on the previously released versions (albeit with the memory issues).

I can at least reproduce this locally on the medium sized 100-plane subset of data I have.