Setup of maip

Hi, I am currently in the process of setting up the maip platform on the VM, however I am having a couple of issues with this. I have the UI started but if I go into any section of the system it will say error{} in the top right corner of the screen.
Would you be able to give me a hand with this?
Thanks

Hello Tim,
Did you install it by running the script or using the Dockerfile? On which system did you perform the installation? I have only tested it on Ubuntu 20.04.6 LTS - Focal. If possible, can you share a screenshot of the error?
Best,
Manh

Hi Manh,
I have ran the start-client.sh scrip and I am able to get on to the UI. I am running it on a Ubuntu 20.04.6 LTS-focal. I have attached image below of the error when trying to get into the all arrears of the system like build, models, predict, etc.

Thanks for any help
Tim

Hi Tim,
It seems that you have not started the server yet. Please make sure you run the script start-maip.sh first on one terminal and then the script start-client.sh on another terminal.
Best,
Manh

Hi,
ahhhh perfect, that is sorted now appreciate it
Appreciate the help with this.
Thanks,
Tim

Hi,
Apologies now whenever I build a new model it doesn’t appear within the all model section.
Also when i change from activity classification to anomaly detection and then I change to build a model it will automatically revert back to activity classification.
Best,
Tim

Sorry, changing the app on client side is not handled properly now. I will fix this issue soon to make the state “app” more persistent on page navigation. If you want to change the AI application, please update this line with values ad or ac (by default). Also keep in mind that the repo is under development :D.

Hi,
Just want to check are we able to build model on the client side becuase when I build a model, it say that it is successfully built but in the All models section there is none within the section
Thanks

You are trying to build models for which applications? For activity classification ('ac'), it normally takes several seconds to build a model. For 'ad', it takes a bit longer, several minutes to complete, especially if you upload pcap files. You can also check the log on the terminal where you are running the server to see if there are any problems. Perhaps, you can start with the pcap files first.

trying to bulid models for 'ad', I built a model about 15 minutes ago with the existing pcaps provided and the system indicated that the model was successfully built, however there is no models within the All models section of the system which is a bit strange as seen in the below

I guess you may encounter some issues in parsing the upload pcap files. I just checked and can build models on my local machine. Thus, I recently added the mmt's reports of 2 pcap files, so that you can select them without uploading pcap files (commit 993b4cb).

On the server, you should see the log like this:

Finally, after 1 or 2 minutes, you should see the newly built model in the table of all models.

At least, running the latest version, you should see the model model_2 there (for performance testing purpose). Hope this helps.

Hi Tim,
First, you need to check whether there are any issues in the model building process in the log training_f3044...log. If everything is ok, you will see the new model located at deep-learning/models or you can run the GET request /models.
Hope this helps.

Hi,

get this module not found error, how would this be installed?

Hello,
I see, please follow this script to install necessary python libraries: https://github.com/Montimage/maip/blob/main/install.sh#L13C1-L13C55. Also pay attention to the version of Python 3. If you installed it on Ubuntu 20.04, it should be fine.

Hi,
I am receiving this after all of the modules have been installed and I have tried to build a model, just wondering if you have seen this issue before.

AVX needs to be supported by the CPU. Try this: https://stackoverflow.com/questions/58198180/run-tensorflow-2-0-on-cpu-without-avx

Would it be possible if you could create a model with the latest reports for normal traffic and bot traffic in ad and change the training parameters to be different than the model that is pushed already?
Let me know if this is possible.

Just checking if the above is possible?
Thanks
Tim

Hi Tim,
Yes, we can build new models by fine-tuning the parameters, such as the number of epochs and the size of batches for CNN and SAE. This should be done using the POST request /build or on "Build Model" tab.

Hi,
perfect, currently having issues with the avx instructions as it is saying that it isn’t supported on my devices, so trying to get something sorted for as I am currently a final year uni student looking into this software.
Would you know the params that are used for the model 2 that is already in the repo in anomaly detection section

I think that you can generally ignore this warning concerning avx instructions, and TensorFlow should still run as expected on your system, please check.
The model_2 is built as shown in the video demo with following parameters: number of epochs for CNN 5, number of epochs for SAE 2, size of batches for CNN 16, size of batches for SAE 32. The video showed that using 2 epochs for the SAE resulted in better performance compared to using 3 epochs for the tested training pcaps.

Get this within the logs of the system whenever I build a model and after checking the all model there is no model that are appearing. Just wanting to check what way do you find is best to start the platform, to allow for me to be able to build a model.

I ran the server with the using the installation instruction on the website

Would you have a break down of the exact AI python libraries that are used with in the system? and how they are used?

Sorry, it seems that you need to fix this issue in order to run tensorflow. According to this, what you can try is to uninstall tensorflow and then install tensorflow-cpu version 2.11.0.
All python libraries I've used are here https://github.com/Montimage/maip/blob/main/src/server/deep-learning/requirements.txt: lime and shap for XAI, the rest for extracting features, building and running AI models.

Sure, I uploaded this model here.

Thank you for all of your help appreciate it

Hi,
There currently is an issue with the model 1 that was uploaded as there is no building status within the training folder how would this be fixed or could this be fixed.

Hi Tim,
Sorry, I forgot adding building files of model_1, please check the commit c50315f. Also, please find the pcaps here for testing. You can also use normal pcaps or malicious pcaps by searching for "botnet pcap files" on Internet.
Best.

Hi,
No bother, appreciate your help
Thanks,

Hi,
Sorry for hassling so much, I am having issue with the predict section when a model is picked and I add a pcap file then click predict, the system seem to be unresponsive would there be any reason for this?

Hi,
There is also issues with the SHAP and LIME where I am getting a 304 response error with the model. Have you seen this before?
Thanks

Hi Tim,
As you don't use the GUI, for prediction, you need to run 2 APIs in sequence as follows:

POST /mmt/offline to produce analysis report of existing pcap files in /maip/pcaps/, for example with the body request:

{
  "fileName": "youtube7.pcap"
}

If ok, you will find the reports in csv in the folder /maip/src/server/mmt/outputs/. In case you want to upload a new pcap file, please use the POST /pcaps request.

POST /predict using the following body request for youtube7.pcap:

{
  "predictConfig": {
    "modelId": "model_1",
    "inputTraffic": {
      "type": "report",
      "value": {
        "reportId": "report-youtube7.pcap-624b4fe1-422c-4b81-bc4a-2d3a47919b8a",
        "reportFileName": "1695745314.129769_0_youtube7.pcap.csv"
      }
    }
  }
}

I plan to improve the POST /predict to support "type": "pcap" so that the first POST request is executed automatically.
I added the reports of 3 testing pcap files in the latest commit so that you don't need to run the first step for those pcap files.
Regarding the issues of XAI, can you provide the logs? Thanks.

Hi,
How would you start the GUI to allow for me to use the predict if that would be easier?
This is what I am seeing on the logs for the XAI

Producing SHAP explanations can take a significant amount of time, even hours/days, depending on the size of the dataset or the number of samples being explained. I'd recommend running LIME first, as it is fast. You see several 'GET /api/xai' requests, which are intended to continuously track the status of the XAI process. You can wait for the XAI process to complete and then disregard those GET requests.
To initiate the GUI, start the server first and then the client, following the steps you've already executed. Hope that helps.

Hi,
That is no bother at all, will have to run it and leave it, just want to be sure, is this likely to be the same for the attacks as it is getting the same responses in the logs

Apologies I miscommunication i am on the GUI trying to run the task but they just have the loading wheel

Hi,
Hope you are keeping well,
Just looking to see if there is any information in the pcap files that have been uploaded to the repo in terms of:
What traffic is used within the pcaps?
For the bot pcap what network attack utilised such as DDOS or Denial of Service?
and for the honeypot pcap where is this obtained?
Hopefully you will be able to help me.
Thanks
Tim

Hello Tim,
The pcap files are extracted from this public dataset. The botnet attacks may consist of remote shell, file upload/download, capturing screenshots and key logging, etc. The example honeypot pcap can be obtained from multiple sources, like our honeypots or public datasets. Hope this help.
Best.

Hi,
Hope you are keeping well.
I am currently in the process of trying to create models for analyzing the best model parameters for my dissertation to show the accuracy numbers and F1 score for all of the models , apart from the 2 already made within the repo. I was wondering if it would be possible for some more model to be created and added to the repo.
Please see attached the model that i would need created.
I appreciate all of the help and hope that this is possible to be completed.
Thanks
Tim
models.txt

Hi,
Just want to check if the above is possible?
Thanks

Hi Tim,
I will try to build those models this week.
Best,

Hi,
thank you very much, appreciate you help.
Kind regards

Hello Tim,
I recently built different models you've requested (https://github.com/Montimage/maip/commit/413de42aa85ab4b7fa5657ae7d3ee60b2d19fd3c). Please share with us if you find any interesting results during your testing phase, e.g., which configurations perform best. Thank you.

Best.

Hi,
Thanks for this, I am also currently trying to run a gan drvien attack on model 2 to test the robustness and get the performance score. how long would this usually take as this has been running quite a while for me so far?
Also, at the start of the attack there was a few errors that came up on GUI but then went away would this be a problem?
Thanks

Hello Tim,
Thank you for reporting the bugs. I fixed it in the latest commit, so that you can run all 3 attacks against model_2. Actually I added the dataset generated using CTGAN dataset for model_2. I will later update the API to automatically produce those GAN datasets with different configurations.
Since you're currently playing with our tool, if it's ok for you, I'll soon share our evaluation google form with you via email. Thus, you can provide us with your feedback to help us improve the tool. Thanks.
Best,

Hi,
That is strange this is what I am getting

would you know why?
Thanks Tim

Apologise it is now this,

I see, you failed to produce the poisoning dataset of the GAN-based dataset. First, please make sure you get the latest version of code. Check again if the ctgan_samples.csv located in the folder /maip/src/server/deep-learning/attacks/model_2 on your machine. Then, run again the server. If you perform successfully the GAN attack, you should see two new csv on the same folder: ctgan_poisoned_dataset.csv and ctgan_poisoned_dataset_view.csv (for visualization purpose only).
Hope that helps!

Hi,
Ahh got it sorted thank you for you help.
Just to let you know the node modules folder isn't on the repo anymore which stop me from getting on the the GUI
Thanks,
Tim

Hi,
Sorry to be a hassle, How long does it usually take to get the resilience metrics for before and after the attack?
Thanks,
Tim

The entire process (producing a poisoned GAN dataset and computing resilience metrics) took around 10-15 seconds. The second step involves retraining the model using the poisoned dataset generated from the first step. Both steps depend on the size of the GAN dataset; however, in this case, it is quite fast.

I have been running the resilience for quite a while now would there be any specific reason why it would be taking longer to complete.
Thanks

@Tim-Cunnane: Can you please provide a copy of the server log when you compute the resilience metrics? It is difficult for me to debug without it. If you want compute resilience metrics for other models (not model_2), you may need to copy the GAN dataset ctgan_samples.csv to the corresponding folder of the tested model in /attacks.

There is just like this:

I also receive this error sometimes when start the resilience metric:

@Tim-Cunnane: It's a bit strange as I have not encountered this issue. Please check again if the poisoned GAN dataset is successfully generated as below. Also, calculating the resilience metrics requires retraining the model. I'm not sure if the problem is due to your hardware issues.

Hi,
Yeah all of the poisoned dataset are there within that folder. I think i may have found the issue where it is the AVX instruction but i have install the link that you suggest in the above comment and i am getting this when running the client of MAIP would there be any specific reason for this? The repo is fully up-to-date

Thanks

Stange why the above is happen and i am stuggling to figureout the reason for this.
Thanks

That is not a bother, just wanted to check what type of AVX instruction does the CPU need to be compatible for such as AVX512, AVX1 or 2?

What CPU do you have to allow for this to be completed?
Thanks

please see below issue when running the client. I have clone the repo to ensure that this actually an issue with the repo and this is what appears:

Sorry for the hassle
Thanks,
Tim

Hi,
Sorry i believe that i have found a laptop that will allow me to be able complete the tasks. I came across an issue when installing the nodejs using this command curl -sL https://deb.nodesource.com/setup_19.x | bash
I am receiving this error:

I was wondering if you would know a fix to this issue?
Appreciate any help that you give me.
Thanks

Nice try. You can copy the error message on google, I found this potential solution.
... | sudo bash

Hope this works.

Hi,
Thank you for all of your help, Just want to see if you had any other pcaps from the public dataset that you got the botnet pcap from such as the DDos attack or DoS, etc.
Thanks


[CHORE] Mark releases as not prerelease

If we do this then the latest release shows up in the sidebar which is better for visibility

The consensus from #290 is that we shouldn't do this automatically without noting that this is not "production ready" software so closing for now


Data Objects: Computer & Host

#24 : user attempted to authenticate to computer: We will get back to you about the "computer" data element. We need to review this concept and define which term would align with OSSEM Common Data Model.

For now, we have defined to use Device when referring to Computer or Host #52


PeoplePicker component filtering people with same names

Category
[x ] Enhancement
Version
Please specify what version of the library you are using: [    2.10 but it is in new version also     ]
Expected / Desired Behavior / Question
PeoplePicker should filter selected person not by name
Observed Behavior
PeoplePicker - after selecting person 1, second person is filtered - they have same names
Problem:
/**

Checks if list contains the person

@param persona
@param personas
*/
private listContainsPersona = (persona: IPersonaProps, personas: IPersonaProps[]): boolean => {
if (!personas || !personas.length || personas.length === 0) {
return false;
}
return personas.filter(item => item.text === persona.text).length > 0;
}

Resolution:
/**

Checks if list contains the person

@param persona
@param personas
*/
private listContainsPersona = (persona: IPersonaProps, personas: IPersonaProps[]): boolean => {
if (!personas || !personas.length || personas.length === 0) {
return false;
}
return personas.filter(item => item.id === persona.id).length > 0;
}

Hello @Matej4386 , I have created a new PR for this as #1620
Thanks,
Nishkalank

Closing as fixed in version 3.16.0


How to gets the new value of rename?

jstree version js 3.3.9
rename config is :
'rename':{
'label':'rename',
'action':function(obj){
var inst = jQuery.jstree.reference(obj.reference);
var clickedNode = inst.get_node(obj.reference);
inst.edit(obj.reference, clickedNode.val);
}
}
Ths

It will be in the rename event that fires later. You can also check the edit method signature - there is also a callback you can use.


[Source Request] --

Source name
JetAnime
Source link
https://ssl.jetanimes.com/
Language
French
Other details
please notify me if the site does not work so that I can change the link
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a title with source name.
[X] I have checked that the extension does not already exist on the website extensions list or the app.
[X] I have checked that the extension does not already exist by searching the GitHub repository and verified it does not appear in the code base.
[X] I will fill out all of the requested information in this form.

Thanks. But fail to fetch video for this source when start playing an episode (any anime)


Experimental feature: opt-in fast build

This opt in config setting allows for skipping the memory + time heavy parent namespace preparation logic needed for edge cases involving model rebuilds, forward ref simplification, and typevar substitution.
With future refactors, we can hopefully make this speedup the default by eliminating the need for the __pydantic_parent_namespace__ cache, but that will take significantly more effort.
This is a more practical follow up to https://github.com/pydantic/pydantic/pull/10026.
Closes https://github.com/pydantic/pydantic/issues/8652
Closes https://github.com/pydantic/pydantic/issues/9905
I think for now it's fair to say this closes the following issues, though as mentioned above, there's lots of room for improvement in the default case. I'll open an issue to reflect this.

Seeing a 5-10x speedup on the examples given in the above issues, specifically k8_v2.py models, with this small modification:
from pydantic import BaseModel as PydanticBaseModel, Field, ConfigDict

class BaseModel(PydanticBaseModel):
    model_config = ConfigDict(experimental_fast_build=True)

python pydanticv2.py  7.17s user 0.21s system 95% cpu 7.693 total

Compared to the performance on main:
python pydanticv2.py  34.71s user 30.23s system 79% cpu 1:21.32 total

Happy to get some feedback on the naming of this experimental config flag.

@MarkusSintonen, curious to hear your feedback as well 👍

Not sure why those tests are failing on CI - they're passing for me locally...

Alright, I'm using xfail with strict=False on some of the tests - seems ok, given that we're merely showcasing patterns that might not work with a given setting.

Alright, I'm using xfail with strict=False on some of the tests - seems ok, given that we're merely showcasing patterns that might not work with a given setting.

Seems like test_recursive_generic_typeddict_in_function_1 and 2 were unexpectedly passing, so they could be removed. But yes, let's keep them as we are unsure if they really work.

Any benefit/drawback of using an env variable instead? I may remember we talked about this but unsure.

In general, we want to try avoiding mutating global behavior like this bc it might effect models used by third party libraries, which is unpredictable and hard to debug. @adriangb has some strong opinions on this subject, if I recall...

Following up further - we thought about introducing an environment variable for things like.
Also, we already have enough trouble juggling config settings, field constraints, and runtime settings to validation / serialization functions, so I'm inclined to avoid sourcing user preferences from another location.

See https://github.com/pydantic/pydantic/issues/10074 regarding next steps and a by-default plan for this logic.


Make packages inherit rules when they have a base

Is your feature request related to a problem? Please describe.
At the moment if you have this scenario:
In xrepo:
package("foo")
    set_urls(...)
    ...

...while also containing rules.
And in your local script you define another package that uses it as a base:
package("bar")
    set_base("foo")

The rules from foo aren't available when bar is used.
Describe the solution you'd like
Make packages inherit rules when they have a base.
Describe alternatives you've considered
No response
Additional context
No response

https://github.com/xmake-io/xmake/pull/4212
try
xmake update -s github:xmake-io/xmake#base

It works, thanks.


JAX-based gradient descent plateaus

Description
I'm writing my own implementation of some numerical solution papers that solves differential equations using machine learning. However, my implementation fails to converge for examples with slightly complex terms: for example, it can solve the one with exp(y), but it can't solve 4*(exp(y)+exp(y/2)). And when my implementation converges, it does so slower than reported results; what they achieve with 100 epochs I achieve with 1000.
Now, perhaps there is a lot to explain in terms of this research to get us to the same page, but here I just want to know how JAX might have been causing the plateau here. I'm suspecting the stacking operation which happens in the first line of the function N(w, x). There I'm just trying to represent a scalar by plugging it to first 5 Chebyshev polynomials, placing each result to a vector respective to the order of the polynomial and getting a 5-dimensional representation.
GPU returns the same result but slower.
import jax
from jax import random, grad, vmap, jit
import jax.numpy as jnp

jax.config.update("jax_enable_x64", True)

def chebyshev(n, x):
  if n == 0:
    return jnp.ones_like(x)
  elif n == 1:
    return x
  else:
    return 2*x*chebyshev(n-1, x) - chebyshev(n-2, x)

def N(w, x):
  # numerical transformation
  x = jnp.stack([chebyshev(i, x) for i in range(5)]).T
  x = x.T if x.shape[0]==1 else x
  # learning
  z = jnp.dot(w, x).squeeze()
  return jnp.tanh(z)

def trial(w, x):
  return 1 + (x**2)*N(w, x)

trial_vect = vmap(trial, (None, 0))
grad_trial = grad(trial, 1)
grad_trial_vect = vmap(grad_trial, (None, 0))
grad2_trial = grad(grad_trial, 1)
grad2_trial_vect = vmap(grad2_trial, (None, 0))

inputs = jnp.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.])

@jit
def error(params, inputs):
	term = grad2_trial_vect(params, inputs) + (2/inputs)*grad_trial_vect(params, inputs) + 4*(2*jnp.exp(trial_vect(params, inputs))+jnp.exp(trial_vect(params, inputs)/2))
  	return jnp.sum(0.5*term**2)

grad_error = jit(grad(error, 0))

key = random.PRNGKey(0)
params = random.normal(key, shape=(1, 5))

epochs = 1000001
lr = 0.0000005

for epoch in range(epochs):
    if epoch % 100000  == 0:
      print('epoch: %3d error: %.6f' % (epoch, error(params, inputs)))
    grads = grad_error(params, inputs)
    params = params - lr*grads

Result:
epoch:   0 error: 5474.779454
epoch: 100000 error: 1281.560668
epoch: 200000 error: 1279.912948
epoch: 300000 error: 1279.364945
epoch: 400000 error: 1279.091227
epoch: 500000 error: 1278.927101
epoch: 600000 error: 1278.817733
epoch: 700000 error: 1278.739640
epoch: 800000 error: 1278.681087
epoch: 900000 error: 1278.635557
epoch: 1000000 error: 1278.599140

System info (python version, jaxlib version, accelerator, etc.)
jax:    0.4.23
jaxlib: 0.4.23
numpy:  1.25.2
python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
jax.devices (1 total, 1 local): [CpuDevice(id=0)]
process_count: 1

If the problem is slow convergence, I would check the learning rate. 0.0000005 seems very small.

If the problem is slow convergence, I would check the learning rate. 0.0000005 seems very small.

Oh, I tried many learning rates, indeed I thought lr decay would solve it, but it didn't. Then I forgot to run with the largest appropriate one before sharing. Anyway, with .5 it converges to error: 1278.272054 then it's stuck there.

Without digging further, my guess would be that you've found a local optimum.

Paper omits training details, so this is possible. I'd like to discuss here if this implementation of hard-coded representation is correct; does it hurt the backpropagation or is it fine?
  # numerical transformation
  x = jnp.stack([chebyshev(i, x) for i in range(5)]).T
  x = x.T if x.shape[0]==1 else x

That formulation should be fine when it comes to autodiff. The only issue is that the recursive chebyshev implementation you're using is not as efficient as it could be: I suspect that the compiler optimizes it well enough, but it will generate a lot of duplicate statements and lead to slow compile times

Is the gradient norm approximately zero, i.e. do the iterates stop moving (in addition to the loss not progressing)?
To rule out numerical approximation issues, you could try comparing to Autograd, since it is close to a drop-in replacement (no vmap though) and it'd use NumPy for numerical operations.
Can you link the paper in question?

Also, another thing I noticed: if you change the initial params state by using a different seed for random.key, you end up with a different answer (also with zero gradient). This strongly suggests that you have a multimodal likelihood space, and you're landing in a local rather than a global optimimum. One way around this would be to use more sophisticated optimization methods; see for example the jaxopt package.

Is the gradient norm approximately zero, i.e. do the iterates stop moving (in addition to the loss not progressing)?
To rule out numerical approximation issues, you could try comparing to Autograd, since it is close to a drop-in replacement (no vmap though) and it'd use NumPy for numerical operations.
Can you link the paper in question?

Tuning the learning rate along training, for the above equation (loss function) I got error: 70.535873 grad norm: 0.010273, so yes, they vanish (Additional question here: I knew about learning rate decay, but I had never increased learning rates during training before, yet increasing it got me out of local optimums here. But if I had started with that learning rate, loss would return nan.  Any source on this?)
This particular equation is not solved in the paper proposing this architecture, but I expect it to solve it, at least reach an error < 0. Except for final two examples, I can replicate the results. Here is the paper: Chebyshev Neural Network based model for solving Lane–Emden type equations

Also, another thing I noticed: if you change the initial params state by using a different seed for random.key, you end up with a different answer (also with zero gradient). This strongly suggests that you have a multimodal likelihood space, and you're landing in a local rather than a global optimimum. One way around this would be to use more sophisticated optimization methods; see for example the jaxopt package.

Few experiments with Adam resulted in same kind of behavior. And that one paper I mentioned, where it reported convergence in 100 epochs but my JAX implementation did after 1000, it reported using vanilla GD and so that's how I did it. I'll look into it though, thanks!

Additional question here: I knew about learning rate decay, but I had never increased learning rates during training before, yet increasing it got me out of local optimums here. But if I had started with that learning rate, loss would return nan. Any source on this?

I think it depends on the problem at hand, but in some problems choosing good initializers is difficult and so if we "warm up" (ie increase) the step size at first we can get into better optimization regimes. I think that is common in deep learning: see e.g. this paper and references, though I don't know what the canonical references are. On the theory side, the work of Altschuler and Parrilo like here and here (and Altschuler's 2015 masters thesis under Parrilo) offer some explanation of non-monotonic, not-just-constant-or-decaying step sizes.
Should we keep this issue open, or close it until new questions arise? (Sorry, I'm just not sure if there are outstanding JAX questions at the moment!)


Version not updating in bin

Version: 1.6.0
OS: Windows 10
Try to run atom from command line, get following error:
/c/Users/XXX/AppData/Local/atom/bin/atom: line 2: /c/Users/XXX/AppData/Local/atom/bin/../app-1.5.3/resources/cli/atom.sh: No such file or directory

( temporarly ) fixed by manually updating version to 1.6.0 in line 2 of /c/Users/XXX/AppData/Local/atom/bin/atom
similar issue and fix for apm in the same directory.
Atom installed using Windows installer and updated in-app.

/cc @paulcbetts

If you still have the squirrelsetup.log file available please post it here so we can see what happened.

@damieng Sure, log attached.
SquirrelSetup.txt

Hmm, there's no reference in there whatsoever to the resources\bin folder or its contents...

I'm not 100% sure this is an installer / updater issue, is Cygwin supported in atom.sh?

atom.sh does work in cygwin - the problem here and in a number of other issue seems that the resources\bin folder is not always being installed/upgraded.

Just tried latest 1.6 installer from the site and it created the bin folder
with the contents correctly but also doesn't log that it created them in
the installer. Will need to further investigate.
On Tue, Apr 12, 2016 at 2:12 PM Paul Betts<EMAIL_ADDRESS>wrote:

I'm not 100% sure this is an installer / updater issue, is Cygwin
supported in atom.sh?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/atom/atom/issues/11217#issuecomment-209104370

Tracking this bug in https://github.com/atom/atom/issues/11450to combine multiple issues


add attributes to shapefile

@csmoore @kgonzago please review and merge

@csmoore  really not sure what you want me to combine.

I could be confused/mistaken- are these sections the same/similar and able to be combined:
https://github.com/Esri/distance-direction-addin-dotnet/blob/issue299-rework/source/DistanceAndDirection/ArcMapAddinDistanceAndDirection/ArcMapAddinDistanceAndDirection/Models/FeatureClassUtils.cs#L370
https://github.com/Esri/distance-direction-addin-dotnet/blob/issue299-rework/source/DistanceAndDirection/ArcMapAddinDistanceAndDirection/ArcMapAddinDistanceAndDirection/Models/FeatureClassUtils.cs#L790
https://github.com/Esri/distance-direction-addin-dotnet/blob/issue299-rework/source/DistanceAndDirection/ArcMapAddinDistanceAndDirection/ArcMapAddinDistanceAndDirection/Models/FeatureClassUtils.cs#L165
https://github.com/Esri/distance-direction-addin-dotnet/blob/issue299-rework/source/DistanceAndDirection/ArcMapAddinDistanceAndDirection/ArcMapAddinDistanceAndDirection/Models/FeatureClassUtils.cs#L584

They could probably be refactored into one function but would take a lot of time. I think given time we may consider that at a later date. One sets up fields and attributes for shapefiles the other for featureclass. I was working within the boundaries of existing functions...

OK - just wanted to check - you may want to create a v.after next issue to consider for later date


Consistency Update

Events had numbers inconsistently shown. some had #, some not, some 2 digits, some not.
I started a standard, based on past events of Data Saturday #x - yyyy

Would it make sense to pre-empt future consistency with using three digits and leading zeroes, a la T-SQL Tuesday?


metadata() returns original width and height after resize and rotation

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?
Using<EMAIL_ADDRESS>Latest dist is<EMAIL_ADDRESS>What are the steps to reproduce?
const img = sharp('abc.png')
const {width, height} = await img.metadata()

const { width: w2, height: h2} = await img.resize({width: width *2}).rotate(45).metadata()

What is the expected behaviour?
w2 should be changed to the actual width after resize and rotate. But it remains unchanged.
I had to use
const meta = await sharp( img.toBuffer()).metadata()

to get the new meta after transform.
Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?
Are you able to provide a sample image that helps explain the problem?
What is the output of running npx envinfo --binaries --system?

This is the expected behaviour. I think the example provided here needs to await the toBuffer call also to work correctly.
- const meta = await sharp( img.toBuffer()).metadata()
+ const meta = await sharp(await img.toBuffer()).metadata()

The output is only generated when toBuffer or toFile (or, implicitly for Streams, pipe) are called.

Fixed the typo. It was keyed in on spot.
Thanks for the clarification. Maybe clarify it in documentation? I was confused by it and spent some time to find it out.

Added via commit https://github.com/lovell/sharp/commit/2b8e4d20dea94f77fee2ae2f75b400550ce9abd0, thanks for the suggestion.


coap.net

Hi,
Please let me know how i will provide the security in coap server. Is there any possible way to provide security. How to implement proxy between client and server.
Please share me any link or document for COAP which contains all the details about COAP .
Thanks

This issue is kind of duplicate with #27. Shall we discuss there?


Coup game that got stale

a46b773c-89bd-4243-858f-e0a9943991a1

Gameflow rules are only executed once. They should be continually executed until there are no applicable rules.

Successfully reproduced the problem and added a test case for this.


Fetching related data along with suggestions

I'm fetching the names of cities from a database using typeahead. As the user types, bloodhound fetches the names of cities from the server. The database also contains additional info like city population, neighboring cities etc.. which doesn't need to be displayed to the user immediately but it's needed if the user selects a city from the suggestions. I can configure the server to send that additional data along with suggestions when bloodhound makes a request. But how do i store it & retrieve it from bloodhound when a result is selected? I'm trying to avoid a subsequent request when a suggestion is selected by the user.

Any comments on this?

Apologies for closing without comment – was trying to do some clean up after leaving typeahead.js alone for too long.
If the user selects a suggestion, you can use the typeahead:select event to get the suggestion object. If the user types the entire value without selecting a suggestion, it's a little trickier. Is the latter scenario something you need to deal with?


LDC: Valgrind detect memory leak

Arch: x86-64 (zen3)
OS: ArchLinux (glibc)
Note: on alpine-musl not issue.

build - output
$> cat hello.d
void main() {}
$> ldc2 hello.d
$> valgrind --leak-check=full --show-leak-kinds=all ./hello
==109403== Memcheck, a memory error detector
==109403== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==109403== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==109403== Command: ./hello
==109403==
==109403==
==109403== HEAP SUMMARY:
==109403==     in use at exit: 96 bytes in 2 blocks
==109403==   total heap usage: 229 allocs, 227 frees, 47,488 bytes allocated
==109403==
==109403== 24 bytes in 1 blocks are still reachable in loss record 1 of 2
==109403==    at 0x484ABC0: realloc (vg_replace_malloc.c:1690)
==109403==    by 0x128800: core.internal.container.common.xrealloc(void*, ulong) (in /home/kassane/Downloads/hello)
==109403==    by 0x128475: core.internal.container.array.Array!(core.gc.gcinterface.Range).Array.insertBack!().insertBack(core.gc.gcinterface.Range) (in /home/kassane/Downloads/hello)
==109403==    by 0x147CEC: _DThn16_4core8internal2gc4impl5protoQo7ProtoGC8addRangeMFNbNiPvmxC8TypeInfoZv (in /home/kassane/Downloads/hello)
==109403==    by 0x122C0D: rt.memory.initStaticDataGC().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109403==    by 0x1241D8: rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109403==    by 0x121853: rt_init (in /home/kassane/Downloads/hello)
==109403==    by 0x121CFC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109403==    by 0x121C36: _d_run_main2 (in /home/kassane/Downloads/hello)
==109403==    by 0x121A8C: _d_run_main (in /home/kassane/Downloads/hello)
==109403==    by 0x1217F1: main (in /home/kassane/Downloads/hello)
==109403==
==109403== 72 bytes in 1 blocks are still reachable in loss record 2 of 2
==109403==    at 0x484ABC0: realloc (vg_replace_malloc.c:1690)
==109403==    by 0x13B191: core.gc.registry.registerGCFactory(immutable(char)[], core.gc.gcinterface.GC() function) (in /home/kassane/Downloads/hello)
==109403==    by 0x49B1DFD: call_init (libc-start.c:145)
==109403==    by 0x49B1DFD: __libc_start_main@@GLIBC_2.34 (libc-start.c:347)
==109403==    by 0x1216E4: (below main) (in /home/kassane/Downloads/hello)
==109403==
==109403== LEAK SUMMARY:
==109403==    definitely lost: 0 bytes in 0 blocks
==109403==    indirectly lost: 0 bytes in 0 blocks
==109403==      possibly lost: 0 bytes in 0 blocks
==109403==    still reachable: 96 bytes in 2 blocks
==109403==         suppressed: 0 bytes in 0 blocks
==109403==
==109403== For lists of detected and suppressed errors, rerun with: -s
==109403== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

static-build - output
$> valgrind --leak-check=full --show-leak-kinds=all ./hello
==109547== Memcheck, a memory error detector
==109547== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==109547== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==109547== Command: ./hello
==109547==
==109547== Syscall param set_robust_list(head) points to uninitialised byte(s)
==109547==    at 0x47E34A: __tls_init_tp (in /home/kassane/Downloads/hello)
==109547==    by 0x434C12: __libc_setup_tls (in /home/kassane/Downloads/hello)
==109547==    by 0x434768: (below main) (in /home/kassane/Downloads/hello)
==109547==  Address 0x4000870 is in the brk data segment 0x4000000-0x4000eff
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x45B9F7: malloc (in /home/kassane/Downloads/hello)
==109547==    by 0x47CCCD: _dl_get_origin (in /home/kassane/Downloads/hello)
==109547==    by 0x47FCB6: _dl_non_dynamic_init (in /home/kassane/Downloads/hello)
==109547==    by 0x481768: __libc_init_first (in /home/kassane/Downloads/hello)
==109547==    by 0x43485D: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x45BAB9: malloc (in /home/kassane/Downloads/hello)
==109547==    by 0x47CCCD: _dl_get_origin (in /home/kassane/Downloads/hello)
==109547==    by 0x47FCB6: _dl_non_dynamic_init (in /home/kassane/Downloads/hello)
==109547==    by 0x481768: __libc_init_first (in /home/kassane/Downloads/hello)
==109547==    by 0x43485D: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x45A490: _int_malloc (in /home/kassane/Downloads/hello)
==109547==    by 0x45B282: tcache_init.part.0 (in /home/kassane/Downloads/hello)
==109547==    by 0x45BAC3: malloc (in /home/kassane/Downloads/hello)
==109547==    by 0x47CCCD: _dl_get_origin (in /home/kassane/Downloads/hello)
==109547==    by 0x47FCB6: _dl_non_dynamic_init (in /home/kassane/Downloads/hello)
==109547==    by 0x481768: __libc_init_first (in /home/kassane/Downloads/hello)
==109547==    by 0x43485D: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x404A24: _d_dso_registry (in /home/kassane/Downloads/hello)
==109547==    by 0x4348F4: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x453EAA: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x453EDD: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x453EF8: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451229: __libc_cleanup_push_defer (in /home/kassane/Downloads/hello)
==109547==    by 0x44E555: _IO_link_in (in /home/kassane/Downloads/hello)
==109547==    by 0x44BFED: _IO_new_file_init_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x44944D: __fopen_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x454043: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451275: __libc_cleanup_pop_restore (in /home/kassane/Downloads/hello)
==109547==    by 0x44E609: _IO_link_in (in /home/kassane/Downloads/hello)
==109547==    by 0x44BFED: _IO_new_file_init_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x44944D: __fopen_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x454043: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451229: __libc_cleanup_push_defer (in /home/kassane/Downloads/hello)
==109547==    by 0x43F6BE: __vfscanf_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x49B552: __isoc23_sscanf (in /home/kassane/Downloads/hello)
==109547==    by 0x45412C: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451275: __libc_cleanup_pop_restore (in /home/kassane/Downloads/hello)
==109547==    by 0x43FA37: __vfscanf_internal (in /home/kassane/Downloads/hello)
==109547==    by 0x49B552: __isoc23_sscanf (in /home/kassane/Downloads/hello)
==109547==    by 0x45412C: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451229: __libc_cleanup_push_defer (in /home/kassane/Downloads/hello)
==109547==    by 0x44E199: _IO_un_link.part.0 (in /home/kassane/Downloads/hello)
==109547==    by 0x448E34: fclose (in /home/kassane/Downloads/hello)
==109547==    by 0x454072: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451275: __libc_cleanup_pop_restore (in /home/kassane/Downloads/hello)
==109547==    by 0x44E352: _IO_un_link.part.0 (in /home/kassane/Downloads/hello)
==109547==    by 0x448E34: fclose (in /home/kassane/Downloads/hello)
==109547==    by 0x454072: pthread_getattr_np (in /home/kassane/Downloads/hello)
==109547==    by 0x40D8E1: core.thread.osthread.attachThread(core.thread.threadbase.ThreadBase) (in /home/kassane/Downloads/hello)
==109547==    by 0x40DE48: thread_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401B2E: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x459569: _int_free (in /home/kassane/Downloads/hello)
==109547==    by 0x45C0E0: free (in /home/kassane/Downloads/hello)
==109547==    by 0x403360: rt.minfo.ModuleGroup.sortCtors(immutable(char)[]) (in /home/kassane/Downloads/hello)
==109547==    by 0x403FD8: rt.minfo.rt_moduleCtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x4044B8: rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401B38: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x459569: _int_free (in /home/kassane/Downloads/hello)
==109547==    by 0x45B507: _int_realloc (in /home/kassane/Downloads/hello)
==109547==    by 0x45C4F5: realloc (in /home/kassane/Downloads/hello)
==109547==    by 0x40343A: rt.minfo.ModuleGroup.sortCtors(immutable(char)[]) (in /home/kassane/Downloads/hello)
==109547==    by 0x403FD8: rt.minfo.rt_moduleCtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x4044B8: rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401B38: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x459569: _int_free (in /home/kassane/Downloads/hello)
==109547==    by 0x45B507: _int_realloc (in /home/kassane/Downloads/hello)
==109547==    by 0x45C4F5: realloc (in /home/kassane/Downloads/hello)
==109547==    by 0x403D43: rt.minfo.ModuleGroup.sortCtors(immutable(char)[]).doSort(ulong, ref immutable(object.ModuleInfo)*[]) (in /home/kassane/Downloads/hello)
==109547==    by 0x40348C: rt.minfo.ModuleGroup.sortCtors(immutable(char)[]) (in /home/kassane/Downloads/hello)
==109547==    by 0x403FD8: rt.minfo.rt_moduleCtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x4044B8: rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401B38: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x459569: _int_free (in /home/kassane/Downloads/hello)
==109547==    by 0x45C0E0: free (in /home/kassane/Downloads/hello)
==109547==    by 0x4034D5: rt.minfo.ModuleGroup.sortCtors(immutable(char)[]) (in /home/kassane/Downloads/hello)
==109547==    by 0x403FD8: rt.minfo.rt_moduleCtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x4044B8: rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401B38: rt_init (in /home/kassane/Downloads/hello)
==109547==    by 0x401FDC: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x4287BF: rt.profilegc._staticDtor_L94_C1() (in /home/kassane/Downloads/hello)
==109547==    by 0x404105: rt.minfo.rt_moduleTlsDtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x40450E: rt.sections_elf_shared.DSO.opApplyReverse(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401BF5: rt_term (in /home/kassane/Downloads/hello)
==109547==    by 0x402031: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x45C077: free (in /home/kassane/Downloads/hello)
==109547==    by 0x42882C: rt.profilegc._staticDtor_L94_C1() (in /home/kassane/Downloads/hello)
==109547==    by 0x404105: rt.minfo.rt_moduleTlsDtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x40450E: rt.sections_elf_shared.DSO.opApplyReverse(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401BF5: rt_term (in /home/kassane/Downloads/hello)
==109547==    by 0x402031: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x413D17: rt.lifetime._staticDtor_L534_C1() (in /home/kassane/Downloads/hello)
==109547==    by 0x404105: rt.minfo.rt_moduleTlsDtor().__foreachbody1(ref rt.sections_elf_shared.DSO) (in /home/kassane/Downloads/hello)
==109547==    by 0x40450E: rt.sections_elf_shared.DSO.opApplyReverse(scope int(ref rt.sections_elf_shared.DSO) delegate) (in /home/kassane/Downloads/hello)
==109547==    by 0x401BF5: rt_term (in /home/kassane/Downloads/hello)
==109547==    by 0x402031: rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() (in /home/kassane/Downloads/hello)
==109547==    by 0x401F16: _d_run_main2 (in /home/kassane/Downloads/hello)
==109547==    by 0x401D6C: _d_run_main (in /home/kassane/Downloads/hello)
==109547==    by 0x401AD1: main (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x459569: _int_free (in /home/kassane/Downloads/hello)
==109547==    by 0x45C0E0: free (in /home/kassane/Downloads/hello)
==109547==    by 0x408AEF: core.internal.container.common.xrealloc(void*, ulong) (in /home/kassane/Downloads/hello)
==109547==    by 0x4083B8: core.internal.container.array.Array!(rt.sections_elf_shared.DSO*).Array.popBack() (in /home/kassane/Downloads/hello)
==109547==    by 0x404813: _d_dso_registry (in /home/kassane/Downloads/hello)
==109547==    by 0x43258B: call_fini (in /home/kassane/Downloads/hello)
==109547==    by 0x435790: __run_exit_handlers (in /home/kassane/Downloads/hello)
==109547==    by 0x4358FF: exit (in /home/kassane/Downloads/hello)
==109547==    by 0x432A10: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451229: __libc_cleanup_push_defer (in /home/kassane/Downloads/hello)
==109547==    by 0x44F730: _IO_flush_all (in /home/kassane/Downloads/hello)
==109547==    by 0x44FDD9: _IO_cleanup (in /home/kassane/Downloads/hello)
==109547==    by 0x4358A2: __run_exit_handlers (in /home/kassane/Downloads/hello)
==109547==    by 0x4358FF: exit (in /home/kassane/Downloads/hello)
==109547==    by 0x432A10: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451275: __libc_cleanup_pop_restore (in /home/kassane/Downloads/hello)
==109547==    by 0x44F947: _IO_flush_all (in /home/kassane/Downloads/hello)
==109547==    by 0x44FDD9: _IO_cleanup (in /home/kassane/Downloads/hello)
==109547==    by 0x4358A2: __run_exit_handlers (in /home/kassane/Downloads/hello)
==109547==    by 0x4358FF: exit (in /home/kassane/Downloads/hello)
==109547==    by 0x432A10: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451229: __libc_cleanup_push_defer (in /home/kassane/Downloads/hello)
==109547==    by 0x44FDF7: _IO_cleanup (in /home/kassane/Downloads/hello)
==109547==    by 0x4358A2: __run_exit_handlers (in /home/kassane/Downloads/hello)
==109547==    by 0x4358FF: exit (in /home/kassane/Downloads/hello)
==109547==    by 0x432A10: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547== Conditional jump or move depends on uninitialised value(s)
==109547==    at 0x451275: __libc_cleanup_pop_restore (in /home/kassane/Downloads/hello)
==109547==    by 0x44FF8B: _IO_cleanup (in /home/kassane/Downloads/hello)
==109547==    by 0x4358A2: __run_exit_handlers (in /home/kassane/Downloads/hello)
==109547==    by 0x4358FF: exit (in /home/kassane/Downloads/hello)
==109547==    by 0x432A10: (below main) (in /home/kassane/Downloads/hello)
==109547==
==109547==
==109547== HEAP SUMMARY:
==109547==     in use at exit: 0 bytes in 0 blocks
==109547==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==109547==
==109547== All heap blocks were freed -- no leaks are possible
==109547==
==109547== Use --track-origins=yes to see where uninitialised values come from
==109547== For lists of detected and suppressed errors, rerun with: -s
==109547== ERROR SUMMARY: 84 errors from 26 contexts (suppressed: 0 from 0)

Sanitizer=memory
$>  ./openldc2-0655f30f-linux-x86_64/bin/ldc2 hello.d -fsanitize=memory
kassane:~/Downloads $ ./hello
FATAL: Code 0x6423f3fbdbf0 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.
==109599==Process memory map follows:
        0x6423f3f9e000-0x6423f3fbd000   /home/kassane/Downloads/hello
        0x6423f3fbd000-0x6423f4052000   /home/kassane/Downloads/hello
        0x6423f4052000-0x6423f408e000   /home/kassane/Downloads/hello
        0x6423f408e000-0x6423f4095000   /home/kassane/Downloads/hello
        0x6423f4095000-0x6423f40a0000   /home/kassane/Downloads/hello
        0x6423f40a0000-0x6423f59f4000
        0x7e3f53e00000-0x7e3f53f00000
        0x7e3f54000000-0x7e3f54100000
        0x7e3f54200000-0x7e3f54300000
        0x7e3f54400000-0x7e3f54500000
        0x7e3f54600000-0x7e3f5498e000
        0x7e3f54a00000-0x7e3f54a04000
        0x7e3f54a04000-0x7e3f54a28000   /usr/lib/libc.so.6
        0x7e3f54a28000-0x7e3f54b83000   /usr/lib/libc.so.6
        0x7e3f54b83000-0x7e3f54bd8000   /usr/lib/libc.so.6
        0x7e3f54bd8000-0x7e3f54bdc000   /usr/lib/libc.so.6
        0x7e3f54bdc000-0x7e3f54bde000   /usr/lib/libc.so.6
        0x7e3f54bde000-0x7e3f54be6000
        0x7e3f54be6000-0x7e3f54bea000   /usr/lib/libgcc_s.so.1
        0x7e3f54bea000-0x7e3f54c05000   /usr/lib/libgcc_s.so.1
        0x7e3f54c05000-0x7e3f54c09000   /usr/lib/libgcc_s.so.1
        0x7e3f54c09000-0x7e3f54c0a000   /usr/lib/libgcc_s.so.1
        0x7e3f54c0a000-0x7e3f54c0b000   /usr/lib/libgcc_s.so.1
        0x7e3f54c0b000-0x7e3f54c19000   /usr/lib/libm.so.6
        0x7e3f54c19000-0x7e3f54c99000   /usr/lib/libm.so.6
        0x7e3f54c99000-0x7e3f54cf5000   /usr/lib/libm.so.6
        0x7e3f54cf5000-0x7e3f54cf6000   /usr/lib/libm.so.6
        0x7e3f54cf6000-0x7e3f54cf7000   /usr/lib/libm.so.6
        0x7e3f54cf7000-0x7e3f54cf9000
        0x7e3f54d0c000-0x7e3f54d1d000
        0x7e3f54d1d000-0x7e3f54d1e000   /usr/lib/ld-linux-x86-64.so.2
        0x7e3f54d1e000-0x7e3f54d45000   /usr/lib/ld-linux-x86-64.so.2
        0x7e3f54d45000-0x7e3f54d50000   /usr/lib/ld-linux-x86-64.so.2
        0x7e3f54d50000-0x7e3f54d52000   /usr/lib/ld-linux-x86-64.so.2
        0x7e3f54d52000-0x7e3f54d54000   /usr/lib/ld-linux-x86-64.so.2
        0x7ffefc59d000-0x7ffefc5be000   [stack]
        0x7ffefc5e5000-0x7ffefc5e9000   [vvar]
        0x7ffefc5e9000-0x7ffefc5eb000   [vdso]
        0xffffffffff600000-0xffffffffff601000   [vsyscall]
==109599==End of process memory map.

Reference

https://github.com/ldc-developers/ldc/issues/4617

more tests
$> ./openldc2-0655f30f-linux-x86_64/bin/ldc2 hello.d -fsanitize=memory -static
/usr/bin/ld: /home/kassane/Downloads/openldc2-0655f30f-linux-x86_64/bin/../lib/libldc_rt.msan.a(msan_interceptors.cpp.o): na função "InitializeCommonInterceptors()":
/home/runner/work/llvm-project/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:10448:(.text._ZL28InitializeCommonInterceptorsv+0x61bb): aviso: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /home/kassane/Downloads/openldc2-0655f30f-linux-x86_64/bin/../lib/libldc_rt.msan.a(msan_interceptors.cpp.o): note: the message above does not take linker garbage collection into account

I think upstream had something similar to this a while ago, possible we introduced something new here but also possible there is a commit from them we can cherry pick to solve it.

It's also worth investigating whether this only happens on archlinux.
I'll try the same test on another distro (glibc only).

https://github.com/ldc-developers/ldc/issues/4617#issuecomment-2056760386

that ConservativeGC object itself is malloc'd during runtime/GC initialization and most likely never freed.

https://github.com/opendlang/opend/blob/45eb187c85cf85bd5af4a6e32d0b173d81756882/druntime/src/core/internal/gc/impl/conservative/gc.d#L138-L147

That makes sense, I guess it is kinda a leak because it isn't freed explicitly, but it is meant to live through the lifetime of the process. Do you know if there's an easy way to just tell valgrind "don't worry about this"?

Suppressions?

valgrind-suppression-stdout
$ valgrind --leak-check=full --show-reachable=yes -v --track-origins=yes --show-leak-kinds=all --error-limit=no --gen-suppressions=all --log-fd=9 ./start 9>>memcheck.log
$ ./suppression.sh memcheck.log
{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: reachable
   fun:realloc
   fun:_D4core8internal9container6common8xreallocFNbNiPvmZQe
   fun:_D4core8internal9container5array__T5ArrayTSQBp2gc11gcinterface5RangeZQBi__T10insertBackZQnMFNbNiQCcZv
   fun:_DThn16_4core8internal2gc4impl5protoQo7ProtoGC8addRangeMFNbNiPvmxC8TypeInfoZv
   fun:_D2rt6memory16initStaticDataGCFZ14__foreachbody1MFNbNiKSQCc19sections_elf_shared3DSOZi
   fun:_D2rt19sections_elf_shared3DSO7opApplyFMDFKSQBqQBqQyZiZi
   fun:rt_init
   fun:_D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv
   fun:_d_run_main2
   fun:_d_run_main
   fun:(below main)
}
{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: reachable
   fun:realloc
   fun:_D4core2gc8registry17registerGCFactoryFNbNiAyaPFZCQBwQBu11gcinterface2GCZv
   fun:call_init
   fun:__libc_start_main@@GLIBC_2.34
   fun:(below main)
}
kassane:~ $ ./suppression.sh memcheck.log > druntime.supp
kassane:~ $ valgrind --leak-check=full --show-leak-kinds=all --suppressions=druntime.supp ./start ==112438== Memcheck, a memory error detector
==112438== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==112438== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==112438== Command: ./start
==112438==
==112438==
==112438== HEAP SUMMARY:
==112438==     in use at exit: 96 bytes in 2 blocks
==112438==   total heap usage: 232 allocs, 230 frees, 47,595 bytes allocated
==112438==
==112438== LEAK SUMMARY:
==112438==    definitely lost: 0 bytes in 0 blocks
==112438==    indirectly lost: 0 bytes in 0 blocks
==112438==      possibly lost: 0 bytes in 0 blocks
==112438==    still reachable: 0 bytes in 0 blocks
==112438==         suppressed: 96 bytes in 2 blocks
==112438==
==112438== For lists of detected and suppressed errors, rerun with: -s
==112438== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

References

suppression script
doc - suppression-errors


Llama-70-B causes GPU memory leak without explicit garbage collection

I am working on a project that uses nnsight's trace to implement attribution patching. I found that in certain cases triggering an explicit garbage collection after calling nnsight's trace method is the only way to prevent GPU memory leaks -- this seems related to circular object references that aren't cleared when the torch cache is emptied or when known variables are explicitly deleted.
I think @saprmarks has also encountered this issue when working on feature circuits. Also cc-ing @Butanium who mentioned adding .value fixes it in the scenario they encountered.
nnsight-llama.zip contains the necessary files to reproduce this leak.

Fill out the config file
Run the llama-attr-patching.py code file. It contains an explicitly enabled garbage collection at the moment. But you can disable it by commenting out line 121 to reproduce the bug.
requirements.txt contains my current conda environment.
GPU: Nvidia H100

I've commented out lines 112-114 because they make no difference to the memory leak when uncommented.
Please let me know if more info is needed.
cc @JadenFiotto-Kaufman

Hey @arunasank. I recreated part of your script below (to not have to worry about a config or zip file):
from nnsight import LanguageModel
import torch
model = LanguageModel("meta-llama/Llama-2-7b-hf", device_map='auto', dispatch=True, torch_dtype=torch.float16) # Load the model

mlp_effects_cache = torch.zeros((model.config.num_hidden_layers, model.config.hidden_size)).to("cuda")
attn_effects_cache = torch.zeros((model.config.num_hidden_layers, model.config.hidden_size)).to("cuda")

def test():

  attn_layer_cache_prompt = {}

  mlp_layer_cache_prompt = {}

  attn_layer_cache_patch = {}

  mlp_layer_cache_patch = {}

  with model.trace("hello", scan=False, validate=False) as tracer:
      for layer in range(len(model.model.layers)):
          self_attn = model.model.layers[layer].self_attn.o_proj.output
          mlp = model.model.layers[layer].mlp.down_proj.output
          mlp.retain_grad()
          self_attn.retain_grad()

          attn_layer_cache_prompt[layer] = {"forward": self_attn.save()}
          mlp_layer_cache_prompt[layer] = {"forward": mlp.save()}

      logits = model.lm_head.output.save()

  loss = logits[:, -1, 0] - logits[:, -1, 0]
  loss = loss.sum()
  loss.backward()

  with model.trace("hello", scan=False, validate=False) as tracer:
      for layer in range(len(model.model.layers)):
          self_attn = model.model.layers[layer].self_attn.o_proj.output
          mlp = model.model.layers[layer].mlp.down_proj.output

          attn_layer_cache_patch[layer] = {"forward": self_attn.save()}
          mlp_layer_cache_patch[layer] = {"forward": mlp.save()}

  for layer in range(len(model.model.layers)):
    mlp_effects = (mlp_layer_cache_prompt[layer]["forward"].value.grad * (mlp_layer_cache_patch[layer]["forward"].value - mlp_layer_cache_prompt[layer]["forward"].value)).detach()
    attn_effects = (attn_layer_cache_prompt[layer]["forward"].value.grad * (attn_layer_cache_patch[layer]["forward"].value - attn_layer_cache_prompt[layer]["forward"].value)).detach()

    mlp_effects = mlp_effects[0, -1, :] # batch, token, hidden_states
    attn_effects = attn_effects[0, -1, :] # batch, token, hidden_states

    mlp_effects_cache[layer] += mlp_effects.to(mlp_effects_cache.get_device())
    attn_effects_cache[layer] += attn_effects.to(attn_effects_cache.get_device())

for i in range(100):

  test()
  print(torch.cuda.memory_reserved(0))

I dont see memory requirements increase. Can you make sure youre using the latest version of nnsight? Otherwise I'm not sure this is an nnsight problem. Is there something I am missing?

@JadenFiotto-Kaufman It does seem to happen on the dataset + model combination I shared, so it's definitely not a data independent thing! The memory usage drops to 40G (about half of what it is without garbage collection) when I set explicit garbage collection!
@jkminder Thank you for sharing these. Hopefully it will help us debug the issue.

This was an issue on my side since I wasn't detaching tensors. Closing for now.


Setup Go builds and tests in the CI

This PR adds new steps to build and test Go code and a new workflow to format and lint it. I've added a single GraphQLService test to ensure the CI is running tests. If you feel there's an opportunity to write more, let me know and I'll gladly do it.
Example runs:

Format error
Linter error. Below is an image demonstrating the annotation created when a linter error is found:

Closes #108

I've simplified the assess-go-code-quality job as I found out the linter had a gofmt option, so I removed the extra step. Also added a few more linters:
exhaustive: check exhaustiveness of enum switch statements
goconst: Finds repeated strings that could be replaced by a constant
godox: Tool for detection of FIXME, TODO and other comment keywords
gomnd: An analyzer to detect magic numbers.
lll: Reports long lines
misspell: Finds commonly misspelled English words in comments

I've added new changelog entries for the changes in this PR. I promise this is the last review request 😅


Some bugs in your pytorch code

I found that your code didn't support bachtchsize>1 training ,in the kernel function spx_max_pooling_backward_kernel, you didn't allocate block dimension for batch

This means that you only allow for batchsize=1 for training ? @bermanmaxim

I think this means you need to split your batch first into individual images before sending it to the pooling layer. @Mathijssch do you confirm?

however when training in segmentation task, batch size is a key factor for performance, set batchsize=1 is not a correct choice.

Yes, @bermanmaxim's suggestion is correct, that would be the only option in this implementation, which is indeed a bit limiting. This limitation is not present in the Pytorch version, though.

wontfix as we do not want to invest time in this project right now, PR welcome however! Thanks for your understanding.


Scara robot in UnitTest.py not correct

The scara robot in UnitTest.py is made up of two links connected by a single revolutr joint.
This makes this a simple R configuration.
However a scara robot has a RRP configuration.
Either the ne or the robot model has to be fixed

fixed since the test was removed


DEPRECATION on setup.py

pip reports the following warning on install
DEPRECATION: mdutils is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

I take note about this! Thanks


Add user messages for interactive work

In pins for R, we have pretty comprehensive messages for interactive use, like showing users what server they are connected to, what the full qualified name of the pin turned out to be, etc:
library(pins)
b <- board_rsconnect()
#> Connecting to RSC 2022.09.0 at <https://colorado.rstudio.com/rsc>
b %>% pin_write(1:10, "test-test-test")
#> Guessing `type = 'rds'`
#> Writing to pin 'julia.silge/test-test-test'

Created on 2022-10-27 with reprex v2.0.2
These messages are a "nice to have" and would give Python users a more similar experience to R users.

Ah, thanks! Can we dedicate this issue to enumerating what user messages exist?
It seems like there are two messages in your example:

board init messages. This appears to be exclusive to board_rsconnect(), so maybe we can include as part of https://github.com/rstudio/pins-python/issues/93 ? That way, it can be done along with interactive prompts for logging in with this board type.
pin write messages. This appears to happen across backends. I've opened an issue to add: https://github.com/rstudio/pins-python/issues/166.

The other messaging I am familiar with is (not Connect specific):

pin versioning messages. This happens when you prune versions or make a new version.

library(pins)
b <- board_temp()
b %>% pin_write(1:10, name = "really-nice-numbers")
#> Guessing `type = 'rds'`
#> Creating new version '20221101T224133Z-4d1eb'
#> Writing to pin 'really-nice-numbers'
b %>% pin_write(11:0, name = "really-nice-numbers")
#> Guessing `type = 'rds'`
#> Replacing version '20221101T224133Z-4d1eb' with '20221101T224133Z-b697a'
#> Writing to pin 'really-nice-numbers'

Created on 2022-11-01 with reprex v2.0.2

handling the cache, like this (I believe these are already implemented for Python?)

library(pins)
b <- board_rsconnect()
#> Connecting to RSC 2022.09.0 at <https://colorado.rstudio.com/rsc>
cache_info()
#> Cache info: '~/Library/Caches/pins'
#> * ms365-9b18e4bd90db1ecf03203234e7fe6f8d: 0
#> * ms365-e8085e502e3bd6dc64ccccc78403a110: 0
#> * packages: 0
#> * rsc-e62371cfd77db754024f9c5ed3556a73: 37.1M
#> * s3-pins-test-hadley: 0
#> * s3-pins-testing-julia: 1.37K
#> * s3-vetiver-careful-gaur: 0
#> * s3-vetiver-ornery-urva: 0
#> * s3-vetiver-pubescent-akitainu: 0
#> * s3-vetiver-tangy-pitbull: 0
#> * url: 218M
cache_prune()
#> Delete 16 pin versions, freeing 2.54M ?
#> Error in utils::menu(c("Yes", "No")): menu() cannot be used non-interactively

Created on 2022-11-01 with reprex v2.0.2


As a user I want to be able to use my microphone in order to talk to other participants in both the Game Room and the active Game session.

Acceptance Criteria

[ ] At the start users can choose to enable/disable microphone input
[ ] When first visiting the web-application, users are prompted with a permission prompt to use a user’s microphone.
[ ] A user can find instructions on how to enable microphone permission if they revoked the permission at some point.

Not implemented


Update ES Translation

coverage: 60.672%. remained the same
when pulling e06e0b20744989e70d99c1e6c007f14662e80071 on Roboron3042:master
into 3762bc4f2a2d2aed67b9900e9c8d9c2f97a4475d on newsboat:master.


Add com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-1

Add com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-1 (close #1396)

Moved to #1395


Add CMake build

With this changes it is possible to easily integrate rax into any CMake project as well as build it as standalone project.

Hi @antirez, Can you please review this? Any reason not to such feature?

Hello, I dislike cmake, and I think that such a simple lib should not have a build system. Thanks anyway for providing this.


Support restat when rebuilding manifest

As a fix for #874, we started reloading the entire manifest even if the
manifest was never rebuilt due to a restat rule. But this can be slow,
so call State::Reset instead, which also fixes the original crash.
Fixes #987

And State::Reset should be safe -- it's used for this purpose extensively in the ninja tests.

Thanks, that was easier than expected. If you haven't, can you afl-fuzz with this merged for a bit? I think that found the other crash iirc (see readme)


Refactor ISO 8601 parser for ordinal date

It is much easier to just add the amount of days.

Benchmark for before and after?

This refactoring isn't focused on performance but to make the code easier to understand. I didn't do any benchmarks. But I can try to see how it goes and maybe improve it further (Time.new and immediately adding days can probably be optimized).

Hm, well okay. The previous version was actually pretty fast. It seems going through Time constructor and then #year_month_day_day_year can't beat the performance. I couldn't get it faster than 50% of the original speed.
So never mind then, I guess that's not worth changing it.


3wafeat(point): add HashID method for point

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.zhengbo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Bug: Can't download archives

What happened?
Tested on rar, zip and gz files. Works fine on Chromium, but in Firefox I'm getting an error. The logs don't show anything useful. I'm not even seeing the request to the file in them, while I see other requests.

Version
latest (ghcr.io/diced/zipline or ghcr.io/diced/zipline:latest)
What browser(s) are you seeing the problem on?
Firefox
Zipline Logs
No response
Browser Logs
No response
Additional Info
No response

Not able to reproduce this, I just tried this on a 7z, zip, tar, tar.gz, zip and they all seem to work. The problem might be with the files you are uploading, but I don't have a way of really fixing this since it is most likely a firefox issue.

Turns out i had some malformed nginx config that resulted in the Content-Disposition header being present twice for certain archive files (some weird issues i had with the file upload server i was using before). Chromium apparently just doesn't care about that, but firefox completely blocks the request then, probably some security settings that are on by default. I should have probably checked the nginx config before opening an issue, oops :see_no_evil:

Turns out i had some malformed nginx config that resulted in the Content-Disposition header being present twice for certain archive files (some weird issues i had with the file upload server i was using before). Chromium apparently just doesn't care about that, but firefox completely blocks the request then, probably some security settings that are on by default. I should have probably checked the nginx config before opening an issue, oops 🙈

No problem, glad it's fixed though!


Feature request: mongomock compatibility

I've just encountered minimongo and it looks really great, but I don't think it will work with e.g. mongomock out of the box. mongomock is a drop-in substitute for pymongo which doesn't use a real mongodb backend but performs operations in-memory (not necessarily efficiently) so that you don't need the hassle of setting up a real mongodb instance for unit testing.
As an example:
>>> import mongomock
>>> client = mongomock.MongoClient()
>>> client.db.collection.insert({'foo': 'bar'})
ObjectId('586651608eda9b078e37499e')
>>> list(client.db.collection.find({}))
[{'foo': 'bar', '_id': ObjectId('586651608eda9b078e37499e')}]

It would seem better if minimongo wrapped around an existing MongoDB client (which could then be either a pymongo or a mongomock client) or an existing MongoDB database (ditto).

FWIW Mongo-Thingy now supports Mongomock out of the box, and I found the idea here, so thank you @vegard!


SSR with Partial Prerendering

Feature request
What problem does this feature solve?
When creating an SSR Vue.js app, some of the pages don't change much, like a Contact us page.
Therefore, it'd probably be useful to come up with a more natural way of setting up SSR and prerender-spa-plugin.
What does the proposed API look like?
Some of the routes require SSR, while other are prerendered in advance.

 If you want to prerender some pages instead of SSRing them, you'll have to setup the tooling for that but it's tooling not part of Vue core

I think there was some discussion regarding this in Nuxt.

I do think that in order to better differentiate Vue.js as a project comparing to React.js, for example, there should be even more amazing boilerplates (like vue-hackernews) and combining SSR with Prerendering, IMHO, is one of them.
@yyx990803 Thank you. However, when I was using nuxt.js, I remember looking at every message on Gitter and issue on GitHub, and I don't remember any examples/talks. I've also just tried searching for all logical keywords on Google and their repo, but with no result.


Opening dictionary in new tab shows a 404

Since I'm using a free version of heroku the app is not always online, so opening the app with any path that is not "/" will result to it not loading, I can't fix it on my end. :(

I guess potential solutions are:

Pay for hosting
Change to another free host that supports the feature

For now though, no need to change


Won't compile on macOS with Apple Silicon M1

When building on Mac with Apple Silicon the arm64 version of rust-musl-cross is not available:
% make
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "/Users/k0gen/Dev/hello-world-wrapper"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo +beta build --release
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I'm building it manually from https://github.com/Start9Labs/rust-musl-cross

M1 Mac Build Failed!
docker buildx build --tag start9//main: --build-arg ARCH=aarch64 --platform=linux/arm64 -o type=docker,dest=docker-images/aarch64.tar .
[+] Building 1.4s (1/1) FINISHED
 => [internal] booting buildkit                                            1.3s
 => => starting container buildx_buildkit_epic_einstein0                   1.3s
ERROR: invalid tag "start9//main:": invalid reference format
make: *** [docker-images/aarch64.tar] Error 1
make: *** Deleting file `docker-images/aarch64.tar'

Please follow the instructions in the README. Your docker buildx command is malformed.


Error scanning for devices: Error: Could not start scanning, state is unauthorized (not poweredOn)

Hi running RPi 3B+ 10.8
Already setup on noble
getting this on scan

#3


Query string affects route

When accessing the default route with a query string added we get a 404.
Reproduce:

http://trident3.vlab.cs.hioa.no:8081 => OK
http://trident3.vlab.cs.hioa.no:8081/?asdf=asdf => 404 Page Not Found

Since the Router is moved from this repo into the web appliance repo, this issue can now be closed.


solution to "adb remote secure_mkdirs failed: Permission denied"

[x] I have read the FAQ.
[x] I have searched in existing issues.

Environment

OS: Windows
scrcpy version: 1.24
installation method: Windows release
device model: Redmi K50
Android version: 12

Describe the bug
When you root your phone with Magisk, looks like Magisk changed some settings and causing
adbd cannot run as root in production builds when you type adb root, and causing adb remote secure_mkdirs failed: Permission denied when you executing the bat.
This might not be a fixable bug, but I still want to post my solution due to lots of people facing this problem too, since adbd insecure and adb_root won't work on Android 12.

adb shell
su
magisk resetprop ro.debuggable 1
stop;start;

After phone restarting, you should be able to use scrcpy properly.
Note: You might need to do it again if you reboot your phone.

I wonder where this happens?
I think it can be fixed without root. Do not use adb to push dir and just mkdir in the process of scrcpy in andorid.
Hope it helps.

i had this same error when trying to push a whole directory to an android device. it uploaded the first file in the directory (which took time), but failed immediately when trying to actually create the file on the other end. but it DID create the directory for the file, so the second time the same push command worked without issues.
my workaround is to create the directory first (with adb shell mkdir) before pushing a directory of files to the device. that works flawlessly every time.
this is a very strange and silly bug in adb to be honest.


Feature request: UI improvement - select for detail view

It would be great to have a select dropdown (even better with a search filter) on the dashboard to jump directly to a detail view of a website. Main reason being that with a lot of websites it's either scrolling or using the browser text search. Also once found I cant CMD+Click for a new tab but have to visit detail view and when I go back I jumped right back to the top and the scrolling begins anew.
Here is a mock-up of what I mean:

Would be a great feature to have..

I need to change the View Details from a button into a link. That should allow you to CMD+click it.

I need to change the View Details from a button into a link. That should allow you to CMD+click it.

That would help navigating around. A quick navigation to the detail view of a website would be ideal in the end. Sorry I can't help out here with a PR.. react and next is just not my field of expertise.

Not sure about the drop down. But I think a search box can be useful. Also adding a button or link on the website settings page could help as well.

Fixed in #167


Introduction (or change) of aspectj binding inside agent-java-cucumber6 breaks build of tests

Describe the bug
When I try to update to version from 5.3.1 to 5.3.2 I get multiple compile errors in my gradle build.
Seems something was changed added that is incompatible with existing / explicit usage of AspectJ.
Steps to Reproduce
Steps to reproduce the behavior:
Use aspectj in your test code explicitly.
Expected behavior
The implementation of the agent should not disrupt other, unrelated code.
There should be no errors during build or runtime.
Actual behavior
Compile fails with multiple errors about missing classes although I just changed the version in my build.gradle file:
<...>/CryoAspect.java:5: error: package org.aspectj.lang does not exist
import org.aspectj.lang.ProceedingJoinPoint;
                       ^
<...>/CryoAspect.java:6: error: package org.aspectj.lang.annotation does not exist
import org.aspectj.lang.annotation.Around;
                                  ^
<...>/CryoAspect.java:7: error: package org.aspectj.lang.annotation does not exist
import org.aspectj.lang.annotation.Aspect;
                                  ^
<...>/CryoAspect.java:14: error: cannot find symbol
@Aspect
 ^
  symbol: class Aspect
<...>/CryoAspect.java:31: error: cannot find symbol
    public String processElement(ProceedingJoinPoint joinPoint) {
                                ^
  symbol:   class ProceedingJoinPoint
  location: class CryoAspect
<...>/CryoAspect.java:27: error: cannot find symbol
    @Around(value =<EMAIL_ADDRESS>*) && ("
     ^
  symbol:   class Around
  location: class CryoAspect

Dependency versions
client-java: 5.2.13
agent-java-cucumber6: 5.3.2
Additional context
Add any other context about the problem here.

@JuergenKindler Yes, because this dependency was marked as implementation in the new release. This means it is hidden from user now, and actually I'm thinking of removing it at all, sine this is pure auxiliary dependency and not all of our users use need AspectJ.
It was generally a mistake that we specified it as api and share to users, users should control their code not us. So to address this issue please choose and specify your own version of AspectJ.

Thanks a lot @HardNorth !


Update EventHubs to support WebSockets

Describe the feature
Add support for WebSockets for EventHubs.
Be able to set HubWithWebSocketConnection
Release Note
RELEASE NOTE: UPDATE Support for EventHubs over WebSockets

Possibly something that can be looked into once #2101 is complete. But this will not be worked on until #2101 is done.


Is the a TypeScript type definition available?

Something on the order of "@types/pdf-lib".
If not, as embarrassed as I am to ask, how can I get VS Code to recognize the types in this excellent library?

Hello @reselbob!
The TypeScript types are included in the pdf-lib NPM package. You do not need to install anything additional to get them. VSCode should just pick them up. If it’s not, the only cause I can think of is that you have plugins or configurations preventing it from doing so (e.g. the Deno plugin might cause this - haven’t tested it myself though).

Thank you for your prompt response. Everything seems be OK on this end.
I am using your project in my demonstration of the strategy pattern here: https://github.com/reselbob/designpatterns

Are you sure you've setup/configured TypeScript correctly for your project? It's unlikely to be an issue on pdf-lib's end, as pdf-lib has thousands of users, and this is the first I've heard of the types being unavailable. If you're still having trouble, please reopen this issue with some additional diagnostic info and error messages.

And, to be clear, it's possible there's an issue on pdf-lib's end. It just surprising that if there was one, it hasn't been reported until now. And I don't have any more information to go off of right now.

Popping in here because I am seeing the same thing - running a small proof of concept in Deno with the library downloaded from skypack.dev and I'm not getting any typedefs picked up by VSCode.
For what it's worth, I also dug into the JS files linked to from<EMAIL_ADDRESS>and there don't appear to be any TS typedefs in there? I am a Deno noob, though, so IDK if they're supposed to be somewhere else at a predictable (and so unlinked) location.

OK having seen the conversation in #484 as well, I tried downgrading to the [deprecated] deno VSCode extension and it still did not work for me. I was still seeing the PDFDocument.load function have the type signature PDFDocument.load(pdf: any, options: any): any.
At this point, I was frustrated because I could see from the npm download that it should be working, so I spent too much time digging into this this morning.
Finally, I found a little note at the bottom of this page in Skypack's (formerly Pika) documentation that you have to add ?dts at the end of a skypack/pika import to get the TS definitions for deno sent alongside the code. Adding that to the import solved the issue even with the latest, official plugin.
TDLR;
import { PDFDocument } from 'https://cdn.skypack.dev/pdf-lib@^1.11.1?dts'; works


Weird change of scale when rotating

See this example.  The shark swims a full turn but when it reaches the end of its path something weird happens, for a second it becomes unstuck in space and time and travels through the dimensions in a trippy way.

Swimming-shark.zip
I realize that I'm doing something wrong here, and this is fixable from the code. The issue is that the model is changing while rotating in a strange way, and maybe we can prevent this.
I believe this is happening because the array of points that the shark is following to swim in circles starts and ends with the same set of coordinates, so for a second it's trying to do a Slerp to a rotation of 0 or something like that.
Here's the part of the code that's clearly causing the problem when path.target and path.origin are the same value:
        rotate.fraction = 0
        rotate.originRot = transform.rotation
        let direction = path.path[path.target].subtract(path.path[path.origin]).normalize()
        rotate.targetRot = Quaternion.LookRotation(direction)

      transform.rotation = Quaternion.Slerp(
        rotate.originRot,
        rotate.targetRot,
        rotate.fraction
      )

I'm thinking maybe there's some kind of check we can do when you set a rotation, that if the value isn't valid it returns an error instead of doing something crazy. What do you think?
Here's another example of the same happening in a different scene:

@nearnshaw can we close this one?

yes, let's close it, it never was an issue in Unity, thankfully :)


Remove code that checks if the user is in a TTY

Description
globus-sdk==3.47.0 handles this class of errors for us by informing users on what to do when input receives EOFError during login flows.
[sc-37236]
Type of change

Bug fix (non-breaking change that fixes an issue)
Code maintenance/cleanup

Cross-ref: https://github.com/globus/globus-sdk-python/pull/1093

(fixed merge conflict)


Test for https://github.com/box/augmented_types/issues/20

Test for Compliance with PHP Specification for class names issue.

Verified that @yborovikov has signed the CLA. Thanks for the pull request!


Sort imports according to PEP8 for utility_meter

Description:
I have sorted the imports for the utility_meter component according to PEP8 using isort.
I think it would be good if in the future we can add isort to the .pre-commit-config.yaml.
To do that I will first fix all sorting issues per component.
Black and isort do not play nice by default, however, using the following .isort.cfg config:
[settings]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88

it works.
This PR affects:

homeassistant/components/utility_meter/__init__.py
homeassistant/components/utility_meter/sensor.py
tests/components/utility_meter/test_init.py
tests/components/utility_meter/test_sensor.py

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[x] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[x] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[x] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

Done!


Add SpacesDialect for stripping thymeleaf whitespaces.

Codecov Report

Merging #80 into master will decrease coverage by 0.94%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master      #80      +/-   ##
============================================
- Coverage     49.87%   48.92%   -0.95%
  Complexity      230      230
============================================
  Files            50       53       +3
  Lines          1237     1261      +24
  Branches        139      142       +3
============================================
  Hits            617      617
- Misses          564      588      +24
  Partials         56       56

Impacted Files
Coverage Δ
Complexity Δ

...thymeleaf/AttributesInnerWhitespacesProcessor.java
0% <0%> (ø)
0 <0> (?)

...ommons/springmvc/thymeleaf/EmptyTextProcessor.java
0% <0%> (ø)
0 <0> (?)

...ons/commons/springmvc/thymeleaf/SpacesDialect.java
0% <0%> (ø)
0 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef33677...45fb5e8. Read the comment docs.


Don't include Hamcrest outside of test execution

It can lead to PHPUnit clashes. See https://github.com/hafriedlander/phockito/issues/32 and documented fix at https://github.com/hafriedlander/silverstripe-phockito/pull/3

Fix at https://github.com/silverstripe-labs/silverstripe-spellcheck/pull/2


v0.28.3 RemoteWriteSettings default is missing

I just updated my vm operator from 0.25.1 to 28.3.
vmagents redeploy in CrashLoopBackOff state with this error :
{"ts":"2022-09-07T11:53:41.574Z","level":"error","caller":"VictoriaMetrics/lib/persistentqueue/persistentqueue.go:143","msg":"cannot open persistent queue at \"vmagent-remotewrite-data/persistent-queue/1_3D3BC0D0B83BD05F\": cannot create directory \"vmagent-remotewrite-data/persistent-queue/1_3D3BC0D0B83BD05F\": mkdir vmagent-remotewrite-data: permission denied; cleaning it up and trying again"}
{"ts":"2022-09-07T11:53:41.574Z","level":"panic","caller":"VictoriaMetrics/lib/persistentqueue/persistentqueue.go:147","msg":"FATAL: cannot create directory \"vmagent-remotewrite-data/persistent-queue/1_3D3BC0D0B83BD05F\": mkdir vmagent-remotewrite-data: permission denied"}

The operator doesn't set these 2 parameters :
-remoteWrite.maxDiskUsagePerURL
-remoteWrite.tmpDataPath
the workaroud consists to specify them in the vmagent resources :
  remoteWriteSettings:
    maxDiskUsagePerURL:<PHONE_NUMBER>
    tmpDataPath: /tmp/vmagent-remotewrite-data

This regression is possibly related with
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2946

It's not expected, it breaks default behavior for VMAgent.

Must be fixed at v0.28.5 version. Feel free to re-open if it not help.