Support file-like object in load/info/sox_effects

This PR adds support for file-like object to apply_effect_file, load/info functions of sox_io backend and soundfile backends.
With this, one can load audio data

From stdinimport sys
import torchaudio
data, sr = torchaudio.load(sys.stdin.buffer)

From an archive without extracting it in the local filesystemimport tarfile
import torchaudio

with tarfile.TarFile(archive_path, 'r') as tarobj:
    data, sample_rate = torchaudio.load(tarobj.extractfile(member_name))

Streaming over the network
With requestsimport requests
import torchaudio

with requests.get(url, stream=True) as resp:
    data, sample_rate = torchaudio.load(resp.raw)

With boto3import boto3

s3 = boto3.client('s3')
obj = s3.get_object(Bucket=bucket, Key=key)
data, sample_rate = torchaudio.load(obj['Body'])

For sox_io backend, the new loading function load_audio_fileobj, which works on any Python object with read method that returns bytes.  It is added and bound via PyBind11.
For soundfile backend, pysoundfile natively supports file-like object
~Note The first function argument is changed from filepath to src, this is because~
~1. The data source is no longer necessarily filepath.~
~2. The legacy backends ("sox" backend and "soundfile" backend with legacy interface) use src.~
closes #754 #800

In case you don't like file-like object, file-object is actually defined to be a synonym in the python glosassry. Same with stream. https://docs.python.org/3/glossary.html#term-file-object

I'm a fan. Using pybind11 in eager mode and torchbind for deployment only really is opening a lot of doors. What's the error behavior when someone is using a pybind11 feature that torchbind does not support? For now we've been using this dual registration approach for performance reasons since pybind11 has lower eagermode overhead.

I'm assuming pybind is used since torchbind does not support bytes object, is that correct?
Are you planning to expose this to torchaudio.load? There is no test there.
Is this ready for review? It is currently marked as draft.

I'm a fan. Using pybind11 in eager mode and torchbind for deployment only really is opening a lot of doors. What's the error behavior when someone is using a pybind11 feature that torchbind does not support? For now we've been using this dual registration approach for performance reasons since pybind11 has lower eagermode overhead.

Ah, so it's for performance reason then? I know we observe performance difference in some context, but did we measure a significant difference here? There's a note in the description about being ok with low performance in the description for part of the PR.

I'm a fan. Using pybind11 in eager mode and torchbind for deployment only really is opening a lot of doors. What's the error behavior when someone is using a pybind11 feature that torchbind does not support? For now we've been using this dual registration approach for performance reasons since pybind11 has lower eagermode overhead.

Ah, so pybind is picked for performance reason then? I know we observe performance difference in some context, but did we measure a significant difference here? There's a note in the description about being ok with low performance in the description for part of the PR.

It's been picked for performance reasons in the context of torchtext where the dual pybind11/torchbind approach originated.
Using it for the purposes of getting eagermode-only features is new.

@cpuhrsch

In case you don't like file-like object, file-object is actually defined to be a synonym in the python glosassry. Same with stream. https://docs.python.org/3/glossary.html#term-file-object

well, actually, I prefer the notion of file-"like". Because in my personal experience, more often it's not a file but a something else like data streamed over the network. (still, I guess one can claim it's a file object on raw level Linux point of view)

I'm a fan. Using pybind11 in eager mode and torchbind for deployment only really is opening a lot of doors. What's the error behavior when someone is using a pybind11 feature that torchbind does not support? For now we've been using this dual registration approach for performance reasons since pybind11 has lower eagermode overhead.

So the behavior is,

if the function is being scripted, then it directly tries to call function bound via TorchScript. In this case, passing anything other than string object is a failure, because of TS limitation.
if the function is not being scripted, it inspects the input argument, then based on the type, it uses either TS binding (for str) or PyBind11 version (bytes). And in this case, if the input type is file-like object, then perform read() to fetch the data as bytes then pass it to PyBind11 version.

Using it for the purposes of getting eagermode-only features is new.

And it's a long-waited feature
#800 #754

@vincentqb

I'm assuming pybind is used since torchbind does not support bytes object, is that correct?

yes

Are you planning to expose this to torchaudio.load? There is no test there.

It is tested. Checkout the tests added.

It is tested. Checkout the tests added.

I meant: there are tests for the load functions within both sox_io and soundfile backends, but not one calling directly the torchaudio.load function. Did I miss it? If we're confident this covers what we need and will cover what is needed of a new backend in the future, then what is there already in the pull request would be enough.

Are you planning to expose this to torchaudio.load? There is no test there.

It is tested. Checkout the tests added.

I meant: there are tests for the load functions within both sox_io and soundfile backends, but not one calling directly the torchaudio.load function. Did I miss it? If we're confident this covers what we need and will cover what is needed of a new backend in the future, then what is there already in the pull request would be enough.

Check out how the backend tests are written. torchaudio.load is not the right way to test.

So the behavior is,

if the function is being scripted, then it directly tries to call function bound via TorchScript. In this case, passing anything other than string object is a failure, because of TS limitation.
if the function is not being scripted, it inspects the input argument, then based on the type, it uses either TS binding (for str) or PyBind11 version (bytes). And in this case, if the input type is file-like object, then perform read() to fetch the data as bytes then pass it to PyBind11 version.

Makes sense. We could either as a follow-up or using this as a vehicle look into how we can improve the error message, if necessary, when someone tries to convert a model into TorchScript that calls these functions using bytes.

So the behavior is,

if the function is being scripted, then it directly tries to call function bound via TorchScript. In this case, passing anything other than string object is a failure, because of TS limitation.
if the function is not being scripted, it inspects the input argument, then based on the type, it uses either TS binding (for str) or PyBind11 version (bytes). And in this case, if the input type is file-like object, then perform read() to fetch the data as bytes then pass it to PyBind11 version.

Makes sense. We could either as a follow-up or using this as a vehicle look into how we can improve the error message, if necessary, when someone tries to convert a model into TorchScript that calls these functions using bytes.

Well, once the function is scripted, we cannot do much about it, because TorchScript runtime won't run the function if the input type is invalid. However it seems that the error message is clear if the reason for the failure is wrong type.
 foo.py
import torch
import torchaudio

path = 'test/torchaudio_unittest/assets/sinewave.wav'

# eager execution - works
with open(path, 'rb') as file_:
    torchaudio.load(file_)

load = torch.jit.script(torchaudio.load)

# jit execution with str - works
load(path)

# jit execution with file-like obj - does not work.
with open(path, 'rb') as file_:
    load(file_)

Traceback (most recent call last):
  File "foo.py", line 10, in <module>
    load(file_)
RuntimeError: load() Expected a value of type 'str' for argument 'filepath' but instead found type 'BufferedReader'.
Position: 0
Value: <_io.BufferedReader name='test/torchaudio_unittest/assets/sinewave.wav'>
Declaration: load(str filepath, int frame_offset=0, int num_frames=-1, bool normalize=True, bool channels_first=True, str? format=None) -> ((Tensor, int))
Cast error details: Unable to cast Python instance to C++ type (compile in debug mode for details)

@cpuhrsch Update the docstring to mention the difference between scripted/eager mode.

Hi @mthrok, I wonder if the parameters num_frames and offset could be used with network streams and if it would download only the requested frames?
Thanks

Hi @mthrok, I wonder if the parameters num_frames and offset could be used with network streams and if it would download only the requested frames?
Thanks

Hi @mthrok, I wonder if the parameters num_frames and offset could be used with network streams and if it would download only the requested frames?
Thanks

Hi @igorgad
Thanks for the question, the answer is (unfortunately) no. The goal of this PR is to extend the support to Python's file object, which is the protocol requests happen to implement, thus it works for data transferred over the network. There is no specific optimization or operation carried out for online data transfer.
Then the question is, can we (relatively easily) extend the support for such efficient network data transfer? I think the answer is no, because of the limitation libsox poses. In my understanding, libsox works on FILE object pointer in  C level. (which is abstracted away from user API of libsox) You can use some URL with sox command but that's just calling wget in subprocess and piping the data to stdin of the process sox command is running. There is no network utility libsox utilizes, and plugging such library into libsox is not trivial.
Maybe ffmpeg has that kind of capability, but currently torchaudio does not bind ffmpeg, though it's in my wish list. (but we do not have an action plan for it)
If you want that feature, maybe you can file an issue with feature request and provide your workflow?  (total length of the audio file and the protocol you are using to fetch the data) Though I cannot guarantee that we can full fill your request, but it will certainly help us understand your demand.

Hi @mthrok, I wonder if the parameters num_frames and offset could be used with network streams and if it would download only the requested frames?
Thanks

Hi @igorgad
Thanks for the question, the answer is (unfortunately) no. The goal of this PR is to extend the support to Python's file object, which is the protocol requests happen to implement, thus it works for data transferred over the network. There is no specific optimization or operation carried out for online data transfer.
Then the question is, can we (relatively easily) extend the support for such efficient network data transfer? I think the answer is no, because of the limitation libsox poses. In my understanding, libsox works on FILE object pointer in  C level. (which is abstracted away from user API of libsox) You can use some URL with sox command but that's just calling wget in subprocess and piping the data to stdin of the process sox command is running. There is no network utility libsox utilizes, and plugging such library into libsox is not trivial.
Maybe ffmpeg has that kind of capability, but currently torchaudio does not bind ffmpeg, though it's in my wish list. (but we do not have an action plan for it)
If you want that feature, maybe you can file an issue with feature request and provide your workflow?  (total length of the audio file and the protocol you are using to fetch the data) Though I cannot guarantee that we can full fill your request, but it will certainly help us understand your demand.

Yes. Indeed I think that plugging a network utility into libsox isn't the best option. I investigated the use of FFmpeg to load files from GCS buckets, but it doesn't work unless you set a public URL.
I'm currently investigating options to scale the training of our models on a managed cloud infrastructure, and one of the options is the Ai-Platform running custom containers.  The problem is that we cannot mount NFS shares with our data on the containers due to the lack of the privileged run mode. The dataset is also big to copy from a remote path at the beginning of each experiment.
This way, and inspired by this implementation, would you consider a cloud_io backend (probably with fsspec) using miniaudio a good solution to access files directly and efficiently from buckets? Of course, it requires some tinkering to translate audio_frames into file offsets but I think that's relatively easy for wav files.
Thanks

Yes. Indeed I think that plugging a network utility into libsox isn't the best option. I investigated the use of FFmpeg to load files from GCS buckets, but it doesn't work unless you set a public URL.
I'm currently investigating options to scale the training of our models on a managed cloud infrastructure, and one of the options is the Ai-Platform running custom containers.  The problem is that we cannot mount NFS shares with our data on the containers due to the lack of the privileged run mode. The dataset is also big to copy from a remote path at the beginning of each experiment.
This way, and inspired by this implementation, would you consider a cloud_io backend (probably with fsspec) using miniaudio a good solution to access files directly and efficiently from buckets? Of course, it requires some tinkering to translate audio_frames into file offsets but I think that's relatively easy for wav files.
Thanks

Yes. Indeed I think that plugging a network utility into libsox isn't the best option. I investigated the use of FFmpeg to load files from GCS buckets, but it doesn't work unless you set a public URL.
I'm currently investigating options to scale the training of our models on a managed cloud infrastructure, and one of the options is the Ai-Platform running custom containers. The problem is that we cannot mount NFS shares with our data on the containers due to the lack of the privileged run mode. The dataset is also big to copy from a remote path at the beginning of each experiment.
This way, and inspired by this implementation, would you consider a cloud_io backend (probably with fsspec) using miniaudio a good solution to access files directly and efficiently from buckets? Of course, it requires some tinkering to translate audio_frames into file offsets but I think that's relatively easy for wav files.
Thanks

@igorgad
Your question made me realized that I did not test on those frame slicing. While I was adding test, I looked into the read behavior, and turned out that libsox is smart enough to stop reading once it read enough samples for num_frames and frame_offset options. I think it still needs to read until the beginning of the frames.
Regarding cloud_io backend, I once had a conversation with @cpuhrsch about adding specific I/O functions for different data source like database, S3 but we do not have a concrete plan for adding it yet. We are considering getting rid of the notion of backend and moving to format-based automatic backend selection. From there we can add I/O functions that are specific to format or data source.
Giving some thoughts on this, I think the process can be broken down into the following steps;

Query a header and detect the format
Compute the byte position using the format information from 1.
Fetch the corresponding part of the data
Perform decoding

Then here are the questions that came up to me;

How can we perform efficient data transfer?
Looking at AWS S3 documentation, they accept range header so it is possible to download only the related part, but I guess it has to initiate a new connection. I do not have experience in other cloud storage but if the cloud vendor accepts range header or some equivalent, this should be possible.
How can we compute bytes range?
I think there are two potential questions to be answer

Can you always get a header?
I am not an expert in audio formats, but I think there are formats that do not have a global header, in this case, it's hard to get the understanding of the whole file from the beginning part of the file.
Can we always get the precise byte range?
I think there are formats that support variable bit rate. In this case, it will be difficult to know the byte range from header.

As you mention, if one knows that the format is WAV, then solving 2 will be easy, but solving it for general case will be difficult. Also with cloud provider, one has to think of a way to pass access tokens to the object that is making the network access. So I think, for now, to solve your problem, having your own custom implementation is the easiest solution.

Yes. Indeed I think that plugging a network utility into libsox isn't the best option. I investigated the use of FFmpeg to load files from GCS buckets, but it doesn't work unless you set a public URL.
I'm currently investigating options to scale the training of our models on a managed cloud infrastructure, and one of the options is the Ai-Platform running custom containers. The problem is that we cannot mount NFS shares with our data on the containers due to the lack of the privileged run mode. The dataset is also big to copy from a remote path at the beginning of each experiment.
This way, and inspired by this implementation, would you consider a cloud_io backend (probably with fsspec) using miniaudio a good solution to access files directly and efficiently from buckets? Of course, it requires some tinkering to translate audio_frames into file offsets but I think that's relatively easy for wav files.
Thanks

@igorgad
Your question made me realized that I did not test on those frame slicing. While I was adding test, I looked into the read behavior, and turned out that libsox is smart enough to stop reading once it read enough samples for num_frames and frame_offset options. I think it still needs to read until the beginning of the frames.
Regarding cloud_io backend, I once had a conversation with @cpuhrsch about adding specific I/O functions for different data source like database, S3 but we do not have a concrete plan for adding it yet. We are considering getting rid of the notion of backend and moving to format-based automatic backend selection. From there we can add I/O functions that are specific to format or data source.
Giving some thoughts on this, I think the process can be broken down into the following steps;

Query a header and detect the format
Compute the byte position using the format information from 1.
Fetch the corresponding part of the data
Perform decoding

Then here are the questions that came up to me;

How can we perform efficient data transfer?
Looking at AWS S3 documentation, they accept range header so it is possible to download only the related part, but I guess it has to initiate a new connection. I do not have experience in other cloud storage but if the cloud vendor accepts range header or some equivalent, this should be possible.
How can we compute bytes range?
I think there are two potential questions to be answer

Can you always get a header?
I am not an expert in audio formats, but I think there are formats that do not have a global header, in this case, it's hard to get the understanding of the whole file from the beginning part of the file.
Can we always get the precise byte range?
I think there are formats that support variable bit rate. In this case, it will be difficult to know the byte range from header.

As you mention, if one knows that the format is WAV, then solving 2 will be easy, but solving it for general case will be difficult. Also with cloud provider, one has to think of a way to pass access tokens to the object that is making the network access. So I think, for now, to solve your problem, having your own custom implementation is the easiest solution.


Is this still being maintained?

Hey there! I was checking the project and was wondering if there's still certain kind of maintenance of it. I'd love to collaborate with whatever roadmap you all created for it

Hi, not really any current maintenance, the industry as a whole has moved a lot with commercial players offering very similar with significant advantages since we created this way back when.
Is there anything you saw here, that you found particularly interesting and worth bringing up to scratch?
Thanks
Chris

Hi both, with this in mind, I am moving this to attic. Thanks, Rob

Hi Rob, what does moving to the attic mean?

Hi @techjacker,
At the moment, its a buffer zone for repos that can be either deleted or archived.  As there is very little ownership of repos, I'm concerned to do anything more permanent to repos until absolutely sure it is not needed.  There is a UKHomeOffice-attic organisation which I've moving these to in the short term whilst we find out if they are being used.
Any questions, please get in touch<EMAIL_ADDRESS>

Thanks for the info Rob.
It seems a shame to delete or archive it when the community is still getting value from it. I'd be happy to take ownership of it if HO doesn't want the responsibility of maintaining it any longer. I'm confident I could take on the responsibility as I authored most of the code.

No problem Andrew.
If any sort of value is being had, then happy to leave it in attic or even the main organisation!
There hadn't been any contributions or any actions on it for many years so it was assumed that it wasn't in use.
Happy to leave in attic for now.  Have you still got access to it now?

Ok thanks Rob.
I just have read access but don't worry if it's a pain to grant write.


Type issues with mapVertices

Hello everyone,
after removing the keyType, vertexValueType static fields from the Graph class ,
the SingleSourceShortestPaths example fails with the following exception:
Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class flink.graphs.library.SingleSourceShortestPaths$InitVerticesMapper' could not be determined. This is most likely a type erasure problem. The type extraction currently supports types with generic variables only in cases where all variables in the return type can be deduced from the input type(s).
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:288)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:223)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:193)
	at flink.graphs.Graph$ApplyMapperToVertexWithType.getProducedType(Graph.java:129)
	at org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:149)
	at org.apache.flink.api.java.typeutils.TypeExtractor.getMapReturnTypes(TypeExtractor.java:78)
	at org.apache.flink.api.java.DataSet.map(DataSet.java:160)
	at flink.graphs.Graph.mapVertices(Graph.java:109)
	at flink.graphs.library.SingleSourceShortestPaths.run(SingleSourceShortestPaths.java:27)
	at flink.graphs.Graph.run(Graph.java:1001)
	at flink.graphs.example.SingleSourceShortestPathsExample.main(SingleSourceShortestPathsExample.java:30)

I tried retrieving the key type inside mapVertices using TypeInformation<K> keyType = ((TupleTypeInfo<?>) vertices.getType()).getTypeAt(0) and then passing it to ApplyMapperToVertexWithType as a parameter, but this leads to the following:
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: Task flink.graphs.Graph$ApplyMapperToVertexWithType@32535e27 not serializable:
	at org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:97)
	at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:59)
	at org.apache.flink.api.java.DataSet.clean(DataSet.java:133)
	at org.apache.flink.api.java.DataSet.map(DataSet.java:162)
	at flink.graphs.Graph.mapVertices(Graph.java:110)
	at flink.graphs.library.SingleSourceShortestPaths.run(SingleSourceShortestPaths.java:27)
	at flink.graphs.Graph.run(Graph.java:1008)
	at flink.graphs.example.SingleSourceShortestPathsExample.main(SingleSourceShortestPathsExample.java:30)
Caused by: java.io.NotSerializableException: org.apache.flink.api.common.typeinfo.BasicTypeInfo
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
	at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:286)
	at org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:95)
	... 7 more

Any suggestions?
Thanks!

It seems that the type info becomes a variable in the class and thus part
of the data that needs to be serialized. Since the type info is not
serializable, this must fail.
You can solve this by making the field transient, or by not having such a
field at all, but lazily extract the type in the "getReturnType()" or
"getProducedType()" method.
On Mon, Jan 5, 2015 at 12:32 PM, Vasia Kalavri<EMAIL_ADDRESS>wrote:

Hello everyone,
after removing the keyType, vertexValueType static fields from the Graph
class
https://github.com/project-flink/flink-graph/commit/d26445f0fbf346dad9464e08554d3279f1cdb4cc
,
the SingleSourceShortestPaths example fails with the following exception:
Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class flink.graphs.library.SingleSourceShortestPaths$InitVerticesMapper' could not be determined. This is most likely a type erasure problem. The type extraction currently supports types with generic variables only in cases where all variables in the return type can be deduced from the input type(s).
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:288)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:223)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:193)
at flink.graphs.Graph$ApplyMapperToVertexWithType.getProducedType(Graph.java:129)
at org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:149)
at org.apache.flink.api.java.typeutils.TypeExtractor.getMapReturnTypes(TypeExtractor.java:78)
at org.apache.flink.api.java.DataSet.map(DataSet.java:160)
at flink.graphs.Graph.mapVertices(Graph.java:109)
at flink.graphs.library.SingleSourceShortestPaths.run(SingleSourceShortestPaths.java:27)
at flink.graphs.Graph.run(Graph.java:1001)
at flink.graphs.example.SingleSourceShortestPathsExample.main(SingleSourceShortestPathsExample.java:30)
I tried retrieving the key type inside mapVertices using TypeInformation
keyType = ((TupleTypeInfo<?>) vertices.getType()).getTypeAt(0) and then
passing it to ApplyMapperToVertexWithType as a parameter, but this leads
to the following:
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: Task flink.graphs.Graph$ApplyMapperToVertexWithType@32535e27 not serializable:
at org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:97)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:59)
at org.apache.flink.api.java.DataSet.clean(DataSet.java:133)
at org.apache.flink.api.java.DataSet.map(DataSet.java:162)
at flink.graphs.Graph.mapVertices(Graph.java:110)
at flink.graphs.library.SingleSourceShortestPaths.run(SingleSourceShortestPaths.java:27)
at flink.graphs.Graph.run(Graph.java:1008)
at flink.graphs.example.SingleSourceShortestPathsExample.main(SingleSourceShortestPathsExample.java:30)
Caused by: java.io.NotSerializableException: org.apache.flink.api.common.typeinfo.BasicTypeInfo
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:286)
at org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:95)
... 7 more
Any suggestions?
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/project-flink/flink-graph/issues/46.

Currently, I'm using TypeExtractor.createTypeInfo inside getReturnType to get the key type, but this gives the first exception. Is this what you mean by "lazily extract the type"?
Making the field transient seems to work :)


Implement Image Representation algorithms

The pipeline currently has 2 image representation algorithms:

FRQI (2010)
NEQR (2013)

Important QIR methods that need to be added:

[x] MCRQI (2011)
[ ] QUALPI (2013)
[ ] NAQSS (2014)
[x] INEQR (2015)
[ ] GQIR (2015) (is the superset of INEQR)(should merge INEQR with this)
[ ] FRQCI (2016)
[ ] OCQR (2018)
[ ] BRQI (2018)
[ ] QRMW (2018)
[ ] QRMMI (2018)

Hi I can help with the implementation of Bitplane Representation of Quantum Images

Hi @Dljdd, this is great. Could you open a separate issue, titled something like “Implement Bitplane Representation of Quantum Images”, so that I can assign you to it and also mark it for hacktoberfest?

Created the new issue #131

Hi, shall I implement QRMMI for the lib?

Hi @Dabjazz, that would be amazing. I would recommend opening a new issue, titled something like "Implement QRMMI" so that I can assign it to you and also add the hacktoberfest label to it. Let me know if you have any questions!

@SaashaJoshi  Done


Problem with encoded urls (%3A)

When sending a RESTCONF request using python requests , it is automatically URL encode as percent-encoded but after 96cbf73 I only get:
# requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://[fe80::2a0:85ff:fe00:301%d3a]:443/restconf/ds/ietf-datastores%3Aoperational/ietf-yang-library%3Amodules-state
# 12:44:18 infamy0:test #

If i remove that patch, it works again.
 curl -v -u admin:admin -H "Accept: application/yang-data+json" -kX GET "https://[fe80::2a0:85ff:fe00:301%d3a]:443/restconf/ds/ietf-dat
astores:operational/ietf-yang-library:modules-state"

works though.

I'm afraid that our code is correct, but if you can point out to a section in any relevant RFC which proves us wrong, we'll of course fix our bug(s).
The problem is that the RESTCONF RFC defines rules on what MUST be percent-encoded, and if these rules are not followed, you will have problems working with list and leaf-list string keys. This is also nicely illustrated in a Stackoverflow post. But let's look into the standards:

RESTCONF defers to RFC 3986 for a definition of URL and its processing
the percent-encoding definition in RFC 3986 says that it MUST NOT be used for some reserved characters
it also explicitly says that two URL strings which are only different in percent-encoding of a reserved character, are in fact two different URLs
the colon : is listed as one of the reserved characters.

So, you could argue that our implementation is broken because it does not accept an URL like this:
/restconf/ds/ietf-datastores:operationa%6C

...because that one should be interpreted as /restconf/ds/ietf-datastores:operational, but I think it would be actually a violation of 3986 if we treated this:
/restconf/ds/ietf-datastores%3Aoperational

...as an URL that's equivalent to /restconf/ds/ietf-datastores:operational.
I think that https://github.com/urllib3/urllib3/issues/3135 is the relevant bugreport for the Python implementation.
But again, this is a complex matter, so if you can point out any standard which says otherwise, please say so.

Fixed this by rewrite URLs in nginx.


Remove stage connection from filter criteria in data-connection list

Description
Changed data-connection filter condition to remove stage connection from all registered jdbc implementors.
Because stage connection is a type of connection that user can not create.

Related Issue : integrated test
How Has This Been Tested?

move to Data Storage > Data Connection
open DB Type layer.
You can see that the stage connection has been deleted.

Need additional checks?
NA
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have added tests to cover my changes.

Additional Context

run build
deploy to 3


ensure state chainbase is dirty if exiting due to error that leaves state in inconsistent state

Some exceptions can cause nodeos to shut down but leave the state chainbase in an inconsistent state. Other ones, like a database_guard_exception, will cause nodeos to shutdown in a consistent state.
A new function should be added to chainbase to put it into a mode where it expects to shutdown in a dirty way. When in this mode, certain database operations (like commit or creating a session) are free to abort since the developers should not be doing such operations when chainbase is put into this mode. But the main purpose of this mode is to not reset the dirty flag to false in the destructor of chainbase.
When an exception that will shutdown nodeos leaving the state chainbase in a consistent mode  is encountered, the new function should not be called because it is desirable for the state chainbase to not be dirty after shutdown. However, if other exceptions are encountered which will shutdown nodeos and likely leave it in an inconsistent mode, then the new function should be called.
For example in controller_impl::replay, we should catch all exceptions (other than database_guard_exception) in the code that replays irreversible blocks, then call the function to force the state chainbase to stay dirty, and then rethrow the exception. This will properly handle the situation during replay of irreversible blocks where an unexpected exception can cause nodeos to shutdown leaving the state chainbase in an inconsistent state.

In order to focus our efforts on issues that are currently creating difficulty for the community we are closing tickets that were created prior to the EOSIO 2.0 release.  If you believe this issue is still relevant please feel free to reopen it or create a new one.  Thank you for your continued support of EOSIO!


Pass additional arguments to docker run

What should we add or change to make your life better?
I tried to add Mongo DB as a Docker container, but I couldn't mount the volume correctly on Windows using tye.yml.
The problem is a combination of the following things:

Mongo doesn't work with volumes mapped to Windows filesystem (-v c:/something:/data/db). It needs the named volume (which is mapped to a filesystem in the host VM) - I need to run docker volume create --name=mongodata (I can do it manually, it's a one-time operation), and then docker run ... -v mongodata:/data/db.
When I use mongodata as source in tye.yml, it thinks that it's a relative path and make a full path for it - so c:/something/mongodata is passed)
I haven't found a way how to pass additional arguments to docker run using tye.yml or how to work with named volumes

Why is this important to you?
I can work-around this by running the container on my own, but I think that Mongo is quite a popular database so more people can run into this issue, and I also think that tye.yml could offer more flexibility because many containers need some special configuration.
Anyway, GREAT JOB! I love Tye so far. ❤

Thanks! Recategorizing this as a bug since something doesn't work.

When I use mongodata as source in tye.yml, it thinks that it's a relative path and make a full path for it - so c:/something/mongodata is passed)

I think this might be the think to fix here. We can support a -name on volumes instead of source.

I haven't found a way how to pass additional arguments to docker run using tye.yml or how to work with named volumes

This is also something we should support.

The PR for named volumes is up

I've fixed the named volumes, support, will leave this open for docker run command support. I'm a bit hesitant to give that amount of power for now.

@davidfowl Wow, that was quick! The named volume support is great - this should help everyone who is using Mongo. I agree that giving the option to specify any docker run args can be dangerous.

Closing this because we don't plan to directly support configuring the args of docker run right now. If you have something you need, please ask, we'd love to talk to you.


(vcom 66): Execution of vlib.exe failed

I don't know if this is only from my setup, but this is necessary to type
vlib work
before the command
do simulate-modelsim.tcl
I am using Modelsim 18.1 and now, the tests run perfectly

Hello, could you check the branch https://github.com/fransschreuder/8b10b_VHDL/tree/issue-1_add_vlib_work?
Thanks for reporting

Hello,
Yes, it works properly! Also, it was not possible for me to run the command
do ./compile_all.do -c
but I think it is surely due to my environment. I have to run this command without -c or I get this error: "Error: couldn't open "-c/component_list.txt": no such file or directory"
If you want, I replace the process named "alt7_proc" in the encoder by a conditional statement and from what I understood from the tests, it works properly. Here is the modification I have done:
CURRENT CHANGE
alt7_proc: process(fi, gi, hi, KI, ei, dispin, di, l31, l13)
variable dispval : std_logic;
begin
if dispin = '1' then
dispval := (not ei and di and l31);
else
dispval := (ei and not di and l13);
end if;
alt7 <= fi and gi and hi and (KI or dispval);
end process;
becomes
INCOMING CHANGE
alt7  <= (fi and gi and hi and (ki or (not ei and di and l31))) when dispin = '1' else (fi and gi and hi and (ki or (ei and not di and l13)));
My pleasure, thank you for coding this 8b10b block!
Regards
----- Mail original -----
De: "Frans Schreuder" @.>
À: "fransschreuder/8b10b_VHDL" @.>
Cc: "Loan GUILBAUD" @.>, "Author" @.>
Envoyé: Lundi 12 Avril 2021 14:44:28
Objet: Re: [fransschreuder/8b10b_VHDL] (vcom 66): Execution of vlib.exe failed (#1)
Hello, could you check the branch https://github.com/fransschreuder/8b10b_VHDL/tree/issue-1_add_vlib_work?
Thanks for reporting
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/fransschreuder/8b10b_VHDL/issues/1#issuecomment-817779563

I don't think that changes anything functional, does it? I think the process is slightly more readable, but it's a matter of taste.

I don't know, because the difference would be in the implementation and I haven't tried both revisions. But the risk with the process and the variable could be that more than only logical gates would be instantiated, even if the process is combinatorial. It is only visible with implementation, it could be interesting to see the difference between these 2 codes!
Regards
----- Mail original -----
De: "Frans Schreuder" @.>
À: "fransschreuder/8b10b_VHDL" @.>
Cc: "Loan GUILBAUD" @.>, "Author" @.>
Envoyé: Lundi 12 Avril 2021 16:50:59
Objet: Re: [fransschreuder/8b10b_VHDL] (vcom 66): Execution of vlib.exe failed (#1)
I don't think that changes anything functional, does it? I think the process is slightly more readable, but it's a matter of taste.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/fransschreuder/8b10b_VHDL/issues/1#issuecomment-817877132


pull request for EasyPR-Java

Some changes included in this pull request

add the project files for Eclipse project;
add unit tests;
modify CoreFunc::features function to make tests pass;
adapt the SVM and ANN model file to liuruoze/EasyPR 1.3;
refactor some folder structure and java classes;
format the code and document.

The simple unit tests passed under following condition
Windows7 64bit
Eclipse (Luna)
jdk1.8.0_45
junit 4
Javacv 0.11

fix project and support for eclipse


http/2: correctly handle 100-continue

Previously we did not correctly handle "expect: 100-continue" in the http/2
codec. Because of protocol differences, we handle 100-continue directly in the
codec. This change implements support in the http/2 codec similar to http/1.1
where we immediately send a continue response.
This commit also adds partial support for the client codec for 1xx responses.
The code will raise decodeHeaders(...) twice in the 1xx response case. This
is good enough for testing, but will probably crash higher layers if a client
actually responded this way. Right now we effectively trust clients so this
can be handled in a follow up if necessary.
fixes https://github.com/lyft/envoy/issues/561

@lyft/network-team @jpinner-lyft @andrewwebber

@htuch could you give this a look also please.

Do any Envoy use cases motivate forwarding the 100-continue to the backend? It seems that in general the original intent of the expect header was to have the proxy forward it and then determine from the backend whether or not to reply 100.

This might be legacy by now, and it certainly would hurt performance to round-trip unconditionally.

I don't know of any currently, and we already do direct response for HTTP/1.1. Being able to correctly proxy the 1xx through from server to client would be a giant change. My preference would be to do this and then if we actually find a case that requires 100 continue server support (I really doubt this will happen) we can cover it then.

@lyft/network-team @htuch updated


Can't decompile one method, always the same

Hi.
I tried every known java decompiler and the result is always the same on one method, it just does not decompile., but the method is simple.
In jadx it is always inconsistent code error
/* JADX WARNING: inconsistent code. */
/* Code decompiled incorrectly, please refer to instructions dump. */

it is in utils.Api class, apiRequest method, jar file http://www59.zippyshare.com/v/27225685/file.html
Is it really that hard to fix it?

Thank you so much it's work successfully.

how to run jadx using command line

in the bin directory, ./jadx-gui

Thanks
On Mar 4, 2017 3:22 PM, WY<EMAIL_ADDRESS>wrote:
in the bin directory, ./jadx-gui
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/skylot/jadx/issues/26#issuecomment-284140996, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGRJuy0I-e6PLiY0vlIbt3QjLi1oxkIXks5riTRXgaJpZM4C5Mds.


manifest_parser: remove multi-output depslog restriction

It seems to work as intended.

Cc: @RedBeard0531

A test that it works as intended would be good (not only that it parses without error).

Here's a test file (courtesy of @RedBeard0531)
rule TOUCH
    command = touch $out

rule COPY
    command = echo 'using $in' && for file in $out; do cp $in $$file; done
    deps = msvc
    msvc_deps_prefix = using

rule CHECK
    command = for file in $in; do if ! test -e $$file; then return 99; fi; done

build source: TOUCH
build dest1 dest2: COPY source
build test1: CHECK dest1
build test2: CHECK dest2
build test3: CHECK dest1 dest2
build test: phony test1 test2 test3
default test

ninja -t deps dest1 and ninja -t deps dest2 both output the expected depslog information. What's the best way to add this to the test suite?

Tests should also cover gcc parsing for the following two cases:
a b: in

and
a: in
b: in

It should also work with either of:
a: in

b: in

Especially when one of the outputs is an implicit output. This is important for cases like -gsplit-dwarf -MMD because the compiler seems to only list the "primary" output in its deps files.
That said, if it is harder to do this for make-style deps, getting it just for msvc-style would be an improvement for a few of my use cases, so I'd be happy to see msvc-style support land independently if that would get it in an earlier release.

Also found out the place to implement the tests for this (build_test.cc). I'll work on that this weekend.

@RedBeard0531 Noted, I'll add a test for that as well. Though we should get GCC to mention the .dwo file in the deps log if it doesn't already.

Tests added. @RedBeard0531 he b: in (or even b a: in) is currently treated the same as if you had a depfile not agreeing on the "primary" output name and is basically ignored. It's up to the generator to get the "primary" output correct.

Odd. Those failures aren't happening locally. Debugging.

Ugh. I wish GTest gave more information :/ . Can anyone else reproduce the test failures?

Passing on my local macOS build too. Will try with Windows later.

Ugh. I wish GTest gave more information :/ . Can anyone else reproduce the test failures?

Don't blame GTest for that. The Ninja code base has it's own special set of code that pretends to be GTest and does a horrible job of it.
Quoted from test.h:
// A tiny testing framework inspired by googletest, but much simpler and
// faster to compile. It supports most things commonly used from googltest. The
// most noticeable things missing: EXPECT_* and ASSERT_* don't support
// streaming notes to them with operator<<, and for failing tests the lhs and
// rhs are not printed. That's so that this header does not have to include
// sstream, which slows down building ninja_test almost 20%.

<sarcasm> Obviously 20% is a huge concern when dealing with a codebase that compiles in under a minute </sarcasm>
I encourage you to consider writing a pull request that makes the configure.py script's "--with-gtest" actually do something (currently it does nothing at all, as the letters "gtest" are found exactly once in configure.py).
But probably don't start working on that until you've asked on the Ninja mailing list https://groups.google.com/forum/#!forum/ninja-build , as there's a decent chance (in my opinion) that such a PR would be either ignored or rejected.

I encourage you to consider writing a pull request that makes the configure.py script's "--with-gtest" actually do something (currently it does nothing at all, as the letters "gtest" are found exactly once in configure.py).

I think it would be better to remove that switch and always use googletest.

But probably don't start working on that until you've asked on the Ninja mailing list

Or opened an issue where you suggest the change.

as there's a decent chance (in my opinion) that such a PR would be either ignored or rejected.

PRs should no longer be ignored, I'm trying to change that. If you think there's an open one which should be looked at, just ping me :)

PR for GoogleTest: #1562
You can cherry-pick the commit to get more information about the failed test.

Thanks, will look at this in the next week or so.

Ah ha. Installing re2c gets me to agree with the CI now. Tracking it down.

@jhasse Now that the cc.in file has right code too, CI is passing.

This is a rather severe limitation that prevents listing correct dependencies for a c++20 module scanner that outputs both a dyndeps file and a compiler flags file, when the compile task depends on both but order only for dyndeps and implicit for flags file. Additionally, the overall restriction has been lifted with the dyndeps patch because dyndeps can add implicit outputs to a deps=gcc build.
What is remaining to getting this merged?

There's the conflict to resolve, but more importantly, there is the test case I commented on above. What was labeled a "buggy" depfile is no accepted. Should it continue to be rejected? If so, further work is necessary.

Rebased and now rejects the case again.

We've the limitations around multiple outputs with deps when building Rust targets with Ninja. I've tested this change locally and it seems to be working as expected. Can we get this merged?

I've rebased on top of master.

Any chance of this for 1.10 too? :) @jhasse

Let's see. It probably needs someone more experienced than me to review.

Hmm. @bradking? @nico? Thoughts on this PR?

Updated to use std:: on all lines added/changed in this PR. I left std::strerror alone since <cstring> is not included anyways.


Add support for React Developer Tools replacement

This adds support for a drop-in replacement for the React Developer Tools extension loaded by BetterDiscord. It also allows users without a Chrome installation to load the extension.
The extension has to be placed in an extensions folder in the BetterDiscord folder. Its structure can either be the exact same as in Chrome or simply the extension without version subfolders. The drop-in always takes priority over a potential version installed in Chrome.

@Strencher have you had a chance to review?


Don't emit warnings eagerly

Note that warnings are still emmitted eagerly, I'm leaving that for another day. It shouldn't be too hard to change CompileResult to be a custom enum with a Warning variant.
Originally posted by @jyn514 in https://github.com/jyn514/rcc/issues/32#issuecomment-557458719

Actually now that CompileError is an enum it should be really easy to add a Warning variant

No wait that doesn't work, the backend doesn't emit am object file unless there's been no errors. Hmm ... I'll have to mess around with this

Closed in #173


Mark which books you already have in your inventory in Kourend Library plugin

This was added to the Kourend Library plugin quite some time ago in 931cb7adf26a3c4166cd3d49aa2c713421a0d222


Parameter : template override

HI and tank for ajax !!
i have big problem with css !! i need to override all cookie_consent.html.twig can it be possible to add in "config.yaml" parameter to choise template to render?
Thank

No that's not possible. You should just create a new template file in app/Resources/CHCookieConsentBundle/views/cookie_consent.html.twig. Then you can either

use defined blocks by extending the existing twig file by adding the following in your twig file

{% extends '@!CHCookieConsent/cookie_consent.html.twig' %}

or create your own custom html with the data available in your twig file

you mean Template/CHCookieConsentBundle/views/cookie_consent.html.twig i'm on SF4.2

yes that's right

Sorry i did not all understand : my problem is that i have to change all structures of cookie_consent.html.twig and i can’t extend form '@!CHCookieConsent/cookie_consent.html.twig'. It's only to change "block" i need to change all to render as modale overlay.

So i find where was my problem : (SF4.2)
Template/CHCookieConsentBundle/views/cookie_consent.html.twig
TO
Template/bundles/CHCookieConsentBundle/cookie_consent.html.twig
Thank's

Just a note for people working with Symfony 5
One needs to create template file in:  templates/bundles/CHCookieConsentBundle
In addition, you may encounter a problem with "title" block name which is an example one in the docs.
The name title for a block is quite common. Therefore in my case was already defined. Unfortunately, in my case, I did not get simple error info but an infinite loop looking like this:
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".

Just a note for people working with Symfony 5
One needs to create template file in:  templates/bundles/CHCookieConsentBundle
In addition, you may encounter a problem with "title" block name which is an example one in the docs.
The name title for a block is quite common. Therefore in my case was already defined. Unfortunately, in my case, I did not get simple error info but an infinite loop looking like this:
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".
Jan  3 12:24:32 |INFO | REQUES Matched route "ch_cookie_consent.show".


[Image Tiles Service] Consistency bug: AssetManagerHost vs ServiceHostUrl in appsettings.json

We should have either AssetManagerHost and ServiceHost or AssetManagerHostUrl and ServiceHostUrl.

Fixed in PR #27.

lol not much point assigning me these issues if they are opened, assigned, a fixed committed and merged over a weekend before i get to see them ;)
thanks for the code improvement @senakafdo


how can i use other dictionaries?

love this app

In settings of the app.
It doesn't support v3 / structured content yet, but the old dictionaries would work.

ok、完成しました、完璧です！
ありがとうございます！


Prototype: regenerate all APIs with a new generator

As far as I can see all changes are the resource name properties for the nested types. So we should be good.

Great - will close this.

(This isn't for merging, it's just to get a diff.)

Closing again


Add version command

Description
This PR adds a command to print out project's version.

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          2 Warnings

:warning:
Big PR, consider splitting into smaller

:warning:
Unless you're refactoring existing code or improving documentation, please update CHANGELOG.md.

Here's an example of a CHANGELOG.md entry:
* [#2](https://github.com/sschizas/HTTPsimctl/pull/2): Add version command - [@sschizas](https://github.com/sschizas).

  Generated by :no_entry_sign: Danger


Random logouts

Sometimes the app randomly makes me redo the setup process and I have to relogin to my apple account and stuff, when that's done everything works fine (aside from the storage bug) and chats are still there as well.
if there's anything more I can provide in terms of information let me know.

Fixed pending release, logouts should only happen when Apple triggers them.


ValueError: could not convert string to float when computing shap_values for numpy.array input features type

Hello,
I'm having this error when I try to generate shap_values for a LightGBM model that have categorical input features:

Here you have a simple code to reproduce the warning:
df = pd.DataFrame({'A': ['Foo', 'Bar', 'Foo', 'Baz'], 'B': [1, 1, 0, 1], 'C': [1, 0, 1, 1]})
df['A'] = df['A'].astype('category')
lgb_train = lgb.Dataset(df[['A', 'B']], df['C'])

gbm = lgb.train({'objective': 'binary', 'min_data': 1, 'min_data_in_bin': 1},
                lgb_train,
                num_boost_round=1)

shap.initjs()
explainer = shap.TreeExplainer(gbm)
shap_values = explainer.shap_values(df[['A', 'B']].values)

my shap version: '0.39.0'
Also, it works if I provide directly the dataframe instead of the values but idk why then.

If you change the last line of your example to shap_values = explainer.shap_values(df[['A', 'B']]) (without the .values), it should work with the categorical feature.
However, the __call__ function is still not working with categorical features (i.e. explainer(df[['A', 'B']]) will fail): #2144

@thatlittleboy Candidate for closing?

Fixed by dsgibbons#88, and these changes were merged into the main repo and released in version 0.42.0. Should not be an issue anymore


Tech: API Gateway Setup

Description: Set up an API Gateway for centralized communication between frontend and backend services.
Tasks:

Select and configure an API Gateway solution.
Define API routes and security policies.
Test endpoint performance.

The first task is resolved in #16


Report connection failures and their reasons

When connection to the media center fails, the screen will just show "Connecting..." forever. I would like to see more precisely what is happening: is Kore just waiting for a response from the media center, or was the connection rejected due to bad credentials, or something else? The way it is, troubleshooting is hard, if not an outright matter of luck.

I understand what you mean, but the thing is, there's no magic bullet for it. Consider:

If it shows connecting, it is waiting for a response. If this is happening a lot to you, try going into the server's settings (in Kore), and unchecking the "Use TCP" option. It will use HTTP by default, which tends to either connect or not.
Connection errors are better diagnosed on the server's settings (in Kore). Namely, bad credentials are identified and reported to the user.
Apart from bad credentials, every other error is almost impossible to catch: The remote can't ever know if your firewall isn't correctly configured, if there's no route to Kodi because it's on a different network, etc. All these errors get reported as timeouts...

I understand everything will be impossible to catch. Especially, when it's a network issue (no route, blocked by firewall etc.), I don't expect Kore to figure that out. What I would expect Kore to do is to display a message when an error occurs on connection. In the above cases, a message stating "The connection timed out" would be sufficient for me.
I didn't find any diagnostic options in the server setting for a server that is already configured. I guess I need to explain myself better here: I had my media center set up but the next update reset my configuration so that my user name and password were no longer valid. Here I would assume Kore gets an unambiguous error code back from the server – and in that case, it should just display an appropriate message to the user.
And there are probably a few more connection errors that may happen: for instance, when TCP is used, the server may simply reply with a RST when there is no listener on that port (possible reasons: wrong port, wrong server or Kodi not running).
All of these may happen when connecting to a server that has previously been set up successfully. Therefore it's important to have this functionality not only in the setup activity but on every connection attempt.


how sass is compiled ?

.scss file to www folder ...

Please check this file:
https://github.com/caiobiodere/cordova-template-framework7-vue-webpack/blob/master/template_src/webpack.config.js


display resolved search results

Merged this for release. Please comment if any issues are present after review.


Add endpoints

This is editing FuzzDB?

Yes!!!

That then needs to be done upstream to FuzzDB, rather than in seclist,and we re-import it (else when it gets reimported, your work is lost).
Or update another file...


Display categories and category's childs with relationship

Hello Everybody.
My name is Tuyen.
I have problem with yajra datatables
This is my app's construct
MySQL table
categories
- id
- name
- parent_id (this is root category id)
.....

Category Controller
.....
public function datatableCategories(){
    $categories = Category::with('childrens')->get();
    return Datatables::of($categories)->make(true);
}
.....

Relationship function in Category Model
....
public function childrens(){
    return $this->hasMany('Category','parent_id');
}
.....

I want to display data like this image.

Please help me.
Thank you so much

Hi Tuyen, you can follow this demo for ref. Also, I suggest you remove get() on $categories = Category::with('childrens')->get(); for better response.

Hi Yajra
I want to display like that on table
Category Name

Child Category 1
Child Category 2
.....

Hello!
I think we need more function for your package for show children rows of parent row.
Like
$categories = Category::with('childrens')->where('parent_id',0)->get();
return Datatables::of($categories)->addChild('child_row',function($category){
//something for show child category as a row after parent row
})->make(true);

Thank you

@tuyenlaptrinh thanks for the suggestion but I think it's not workable to add something like addChild since we still have to manage the UI using javascripts.
Anyways, if you have some implementation in mind, please do not hesitate to submit a PR. Thanks!

@tuyenlaptrinh did you managed to achieve this?

Maybe try rowGrouping plugin https://datatables.net/examples/advanced_init/row_grouping but haven't tried if it will also work on nested relations.

@yajra unfortunately row grouping plugin is not the solution in this case


Hide select component button on seperate window

          This comes with another issue.

From separate window
Click select component button (Attachment 1)
Select a component in the page
Floating panel appears, it should not based on my settings (Always hide),
And it doesn't disappear without clicking somewhere else, causing you to lose the selected component scope.

Originally posted by @atilkan in https://github.com/vuejs/devtools-next/issues/459#issuecomment-2243133762

@webfansplz @alexzhang1030 At least let us configure the button setting, seems you resolved this issue by making some assumptions

Well, inspect button in separate window is nonsense😂


chore: upgrade loader-utils dependency for minimist security patch

@NekR any chance you could take a look at this? thanks!

@NekR any chance you could take a look at this? thanks!


set the default nats cr in test to unique strings

Description
Changes proposed in this pull request:

...
...
...

Related issue(s)

/test pull-nats-manager-unit-test

/test pull-nats-manager-unit-test


DFKI poster

all posters of the module projects float top under the respective header except the poster by DFKI (in both the german and english page). The DFKI should be adjusted to fit the others

Duplicate it for both MP1 and MP2

+1


Add custom required privilege definitions for clusters that need them

IM operations have default required privileges (e.g. view for read attribute, operate for write attribute) but can override these defaults with custom required privileges (e.g. administer for anything to do with Access Control Cluster).
This functionality was intended to be provided by ZAP (#10246) and Ember (#10247). In the interim, it is being provided by a simple class (#14283).
Regardless, there is still the task (this issue) of populating the data for the custom required privileges. Each cluster that has custom (non-default) required privileges will have to specify them in src/RequiredPrivilege.cpp (interim approach) and cluster XML (desired ZAP/Ember approach).

Reopening just so I have an issue to track adding the actual XML declarations for any remaining custom privileges. Normally this would be done by cluster authors, but it's easier overall for me to do them in one big swoop.

I think after #17115 and #17134 PRs it's really only the pump control and configuration cluster that needs updating; that one's waiting for a small spec change regarding the privileges (which currently require manage for read as well as write, but probably will be updated to only require manage for read).

Probably should also wait for this other pump configuration-and-control-cluster.xml change to land.
PR #17181

Will be fixed for pump-configuration-and-control-cluster in #17181

I do see @tlykkeberg-grundfos is handling the last access definitions for pump in #17181 so I will close this issue now. Thanks!


Creation of user-focused endpoints

Creation of user-focused endpoints
As a user, I want to have endpoints that allow me to retrieve the relevant data/metadata information in a programmatic way.
Endpoints to create are documented in https://docs.google.com/document/d/1wnZbT5HcA8mQrHXVrTOCUC-qH6xZVONpH7IxdVT64sY/edit#
Acceptance criteria

[x] Endpoints are available and documented

@zperova done. Applied on dev.


The path does not seem to contain a valid Bitnami installation using bitnami/phpbb container.

What steps will reproduce the bug?
I am using the phpBB container provided by Bitnami. Despite the documentation directing me to use the bncert-tool, this tool is unable to recognize the Bitnami installation which seems to be in its default location at /opt/bitnami. Moreover, I have attempted to use several alternative paths with no success.
I am seeking assistance to understand why the tool cannot locate the installation and how I might go about resolving this issue. It is crucial that the documentation aligns with the actual functionality of the tool to prevent such issues in the future.
What is the expected behavior?

Welcome to the Bitnami HTTPS Configuration tool.

Bitnami installation directory
Please type a directory that contains a Bitnami installation. The default
installation directory for Linux installers is a directory inside /opt.
Bitnami installation directory []: /opt/bitnami
Warning: The path does not seem to contain a valid Bitnami installation. Make
sure you have proper access rights, or choose a different directory.
Press [Enter] to continue:
What do you see instead?
It should be specified somewhere which files it is looking for and/or there should be a flexible way to adjust the paths.
Additional information
No response

Hi @EneasVVA,
Thanks for using Bitnami, and sorry for the delay on getting back to you. The bncert tool is not meant to be used in our containers, only in our cloud images and virtual-machines solutions. In your case, the tool is looking for some files not present in containers solutions, which ends up returning that "Bitnami installation not found" message.
As an alternative, you can manually install and run the Let's Encrypt binary (lego tool) as described in the guide linked below
https://docs.bitnami.com/aws/how-to/generate-install-lets-encrypt-ssl/#alternative-approach
Hope it helps!


test

@abonas CI is broken / and tests fail locally :(

should have checked in that Gemfile.lock :(

stole working versions from https://travis-ci.org/abonas/kubeclient/jobs/120659814 and narrowed it down to the webmock upgrade ...

Thanks @grosser :+1: . Why the extra (empty) patch? You could probably squash them.

@grosser thank you for your contribution! as @simon3z suggests, please remove the empty patch and change the PR title to a more clear one. thanks!

updated

good to go @abonas ?


Consider removing split and product methods on decoders

Decoder has a couple of methods that were inherited from Argonaut that I don't think are widely used (at least I don't remember ever using them or seeing them used):
def split[B](decodeB: Decoder[B]): Either[HCursor, HCursor] => Decoder.Result[Either[A, B]]
def product[B](decodeB: Decoder[B]): (HCursor, HCursor) => Decoder.Result[(A, B)]

I'd like to remove them for a couple of reasons.
The first is that the method name product in the Cats world typically suggests the semantics of Decoder#and (which I'd like to deprecate as and and rename product).
The second reason is that these methods are pretty trivial to define yourself: a.product(b) is just a.kleisli *** b.kleisli, and if Cats's Choice had +++ (I'm not sure why it doesn't) a.split(b) would just be a.kleisli +++ b.kleisli.
Alternatively we could rename them to free up product and be a little more idiomatic (probably something like split => splitTagged and product => split, although that's obviously confusing as hell and I'm open to alternative suggestions). I'd rather just get them out of the API, though.

Done in #798.


feat(pipelines)!: support package installation by environment

Please detail a migration path for the breaking change in a doc PR or a changelog entry (this can be done via the body of the commit message, it will automatically be appended in the change log)

Please detail a migration path for the breaking change in a doc PR or a changelog entry (this can be done via the body of the commit message, it will automatically be appended in the change log)

code refactored to avoid migration path


[FEATURE] Make Mod App Buttons accessible in all states

Are there concrete plans for how to make mod app buttons accessible in other states?
The VAB has its own APP. BAR that could be integrated into.
The KSC and the Tracking Station don't their own App Bar, but as far as I can see the "inflight app bar" is still loaded with the parent gameobject disabled, maybe it could be forced on with only mod app icons enabled?
The main menu has no app bar and would need its own implementation.
In theory it might be a good idea to have mods be able to specify on which screens they should be accessible (you wouldn't need a Maneuver Node Editor in the VAB or an Alarm Clock in the main menu), but since mod windows persist you could run into issues where you can't close a mod window that you opened earlier.

You could provide a toggle to choose between quick deployment and deterministic propagation delay, this should be opt-in. think that could be solved by user-specific invites. Yes, all of the app buttons would be able to set by mod creator. We would be able to be user-specific, and the buttons would be able to set by mod creator. This code would return the build number and last deploy timestamp, which would allow adding multiple buttons or lists at once, by user/list/button ID. We would be able to be user-specific, and the buttons would be able to be user-specific. This could be achieved by having a dynamic rate limiting implementation.

You could provide a toggle to choose between quick deployment and deterministic propagation delay, this should be opt-in. think that could be solved by user-specific invites. Yes, all of the app buttons would be able to set by mod creator. We would be able to be user-specific, and the buttons would be able to set by mod creator. This code would return the build number and last deploy timestamp, which would allow adding multiple buttons or lists at once, by user/list/button ID. We would be able to be user-specific, and the buttons would be able to be user-specific. This could be achieved by having a dynamic rate limiting implementation.

This sounds like it was written by a bot

Also added a checklist for what locations should be added to

Currently working on a separate general AppBar library mod that would provide this functionality

KSC tray added in https://github.com/SpaceWarpDev/SpaceWarp/pull/261


Special Character in Values

Special characters in values
Documentation says: URL filters already support all special characters in field values, except the single quote ('). That's the only character you need to escape. To search for a single quote character, use two single quotes ('').
However url filtering with question marks (?) in values is not working and I had to escape them, replacing with "&3f" to make URL filtering work.
Thanks,
Luca

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 70b360f3-21a7-adc2-896a-b1dfd5d9e384
Version Independent ID: 15e8c09f-46ad-68f9-75e6-c3d5c2c214e5
Content: Filter a report using query string parameters in the URL - Power BI
Content Source: powerbi-docs/collaborate-share/service-url-filters.md
Service: powerbi
Sub-service: pbi-collaborate-share
GitHub Login: @maggiesMSFT
Microsoft Alias: maggies

Hi, @LucaFontana -- Good question, sorry for the delay! I'm checking with the product team and will report back.
Thanks!

Hi, @LucaFontana -- right you are! Belatedly, updating the article with your feedback. Changes should be live tomorrow afternoon Jan 6 Pacific time.
Thanks!


Missing type for ignore option

In #485 the src/lib/main.d.ts was not updated, so I get a type error when using:
import { config } from '@dotenvx/dotenvx'

config({ ignore: 'MISSING_ENV_FILE' })

releasing with next patch or minor version


[Documentation Suggestion] Firmware docstrings and automated docgen

Update the europi.py firmware file to document public api methods with proper Python docstrings.
This will then enable us to be able to automated documentation generation tools like Sphinx and readthedocs.

I have this working in my fork, publishing to GitHub Pages: https://mjaskula.github.io/EuroPi/
remaining work:

[ ] a nicer theme
[ ] markdown support
[ ] update code and docs with new EuroPi class


Plan download page won't load

Please complete the following fields as applicable:
Expected behaviour:
When I click on the 'Download' tab for a plan, I should be taken to a page to download a PDF
Actual behaviour:
I see an error page.
Steps to reproduce:
Go to a plan's details page and click the "download" tab

Page loads and plan downloads. Closing out


tweak(client/server): performance optimization

By use of Jenkins method instead of GetHashKey.
Change GetPlayerPed(-1) to PlayerPedId().
Minor clean up for readability.
Remove unused config Config.BlacklistedVehicle.

Thanks!


Add TypeScript definition file

Do you have any plans to provide a TypeScript definition file for your library? which is, btw, awesome.

I'll actually try to generate it from the routes.json file.

Ehh PR welcome :)

Including this possibly helpful link for future reference: https://www.stevefenton.co.uk/2013/01/complex-typescript-definitions-made-easy/

Would love this as well :).

Fix is up on #479.

Current typescript lib is causing compile errors with Typescript 2.1.4. Fix is relatively easy as far as I can tell:
node_modules/github/lib/index.d.ts(1328,18): error TS2314: Generic type 'Array<T>' requires 1 type argument(s).
node_modules/github/lib/index.d.ts(1413,19): error TS2314: Generic type 'Array<T>' requires 1 type argument(s).
node_modules/github/lib/index.d.ts(1420,19): error TS2314: Generic type 'Array<T>' requires 1 type argument(s).

Replacing Array with Array<string> or string[] at these locations fixes the compile issues.

Pull request submitted.


[GUI]Problems in viewing the integer attribute due to invalid data

The attribute is integer but accepts the values "a" and "B": the chart is broken at invalid values

Attribute is integer but accepts float values:

This has already been fixed, right?

This issue will not be prioritized as it does not break the UI.


refactor: export component animations for reuse

Exports all of the component animations so they can be reused by consumers.
Fixes #8904.

If we are exporting them all publically perhaps we should "namespace" them?  We could put all of the definitions inside of respective objects for animations.  I worry about polluting the exported namespace.
For example:
export const ExpansionPanelAnimations = {
  indicatorRotate: AnimationTriggerMetadata = trigger('indicatorRotate', [
    state('collapsed', style({transform: 'rotate(0deg)'})),
    state('expanded', style({transform: 'rotate(180deg)'})),
    transition('expanded <=> collapsed',
               animate(EXPANSION_PANEL_ANIMATION_TIMING)),
  ]),
  expansionHeaderHeight: AnimationTriggerMetadata = trigger('expansionHeight', [
    state('collapsed', style({
      height: '{{collapsedHeight}}',
    }), {
      params: {collapsedHeight: '48px'},
    }),
    state('expanded', style({
      height: '{{expandedHeight}}'
    }), {
      params: {expandedHeight: '64px'}
    }),
    transition('expanded <=> collapsed',
               animate(EXPANSION_PANEL_ANIMATION_TIMING)),
  ]),
  bodyExpansion: AnimationTriggerMetadata = trigger('bodyExpansion', [
    state('collapsed', style({height: '0px', visibility: 'hidden'})),
    state('expanded', style({height: '*', visibility: 'visible'})),
    transition('expanded <=> collapsed',
               animate(EXPANSION_PANEL_ANIMATION_TIMING)),
  ])
}

Good point @josephperrott, I was hoping that this would come up during the review since I'm not a fan of just exporting them as plain variables either. Switch over to namespaces.

Refactored it based on our discussion @jelbourn. Note that I'm not a fan of having to add the typings to all of the constants, but it seems like the simplest way of ensuring that the properties are immutable. Initially I was going for something like {readonly [key: string]: AnimationTriggerMetadata}, but that really only prevents people from adding new keys, it doesn't break if they try to override the existing ones.


Fix grammar

Replace 2 instances of It it with It
Signed-off-by: Nicholas Seemiller<EMAIL_ADDRESS>

LGTM
I'm going to merge this in. The harbor tests has failed however that is due to flakiness from TanzuNet and not related to this change.
The local-registry and GCP tests have passed.


Allow cloning Tokens

Thanks! I included this in 0.1.4. (Sorry for the slow response - I am on vacation this week and was off the grid until just now.)


chore: fix amplify-cli structure

Description of changes:

Discard the 'lib' folder inside 'src'
Move templates files to template dir (amplify-skeleton)
Fix require path for modules previously in lib

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #3329 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master    #3329    +/-   ##
========================================
  Coverage   60.13%   60.13%
========================================
  Files         254      254
  Lines       12439    12439
  Branches     2437     2575   +138
========================================
  Hits         7480     7480
+ Misses       4637     4622    -15
- Partials      322      337    +15

Impacted Files
Coverage Δ

...ges/graphql-transformer-core/src/util/fileUtils.ts
11.76% <0%> (ø)
:arrow_up:

...psync-simulator/src/resolvers/pipeline-resolver.ts
9.37% <0%> (ø)
:arrow_up:

...es/graphql-dynamodb-transformer/src/definitions.ts
84.84% <0%> (ø)
:arrow_up:

packages/amplify-appsync-simulator/src/index.ts
73.68% <0%> (ø)
:arrow_up:

...c-simulator/src/schema/directives/aws-subscribe.ts
26.66% <0%> (ø)
:arrow_up:

...amplify-appsync-simulator/src/server/operations.ts
28.45% <0%> (ø)
:arrow_up:

...mplify-appsync-simulator/src/resolvers/function.ts
9.67% <0%> (ø)
:arrow_up:

...lify-graphql-types-generator/src/compiler/index.ts
90.32% <0%> (ø)
:arrow_up:

packages/graphql-mapping-template/src/print.ts
37.77% <0%> (ø)
:arrow_up:

...es/graphql-transformer-core/src/stripDirectives.ts
35.29% <0%> (ø)
:arrow_up:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a28d62...b46ac19. Read the comment docs.


Specifig Implementation required to cover Data Protection Requirements into AA, EaTL, OamLog

AdministartorAdministration: Named-User need to be deleted after on Month. Information that theNamed-User will leave, is expected as eMail to the product owner

"OaM-Log: Need to be cleared after 1 Month automatically

"Execution and Trace - Log: Need to be cleared after 1 Month automatically

Those above are official reqirements by the Telefonica Data Protection Organization. The requirements above are in the official DGSVO Reporsitory assigned to the product owner Thorsten Heinze.

@alattoch : Above requirements cannot be specified by OAS, CONFIGfile or testcases. Could you please contact implementers to agree how to assure implementation.

Will be fixed in Version 2.0 as part of elastic stack
...
This policy says, that new index is created every 15 days and old index will be kept for another 15 days after rollover and then it is removed. That means we have 30 days history.
2.	Modify Logstash output in logstash pipeline config:
output {
elasticsearch {
hosts => "http://<IP_ADDRESS>:9200"
ilm_enabled => "true"
ilm_rollover_alias => "controller-karaflog-rollover"
ilm_pattern => "{now/d}-000001"
ilm_policy => "keep10m-policy"
}
..


Merge Branches in Computation Graph

We note that we can take this idea of contracting chains further by eliminating branches.
In other words, a binary tree of depth 2, (with update steps at the leaves) requires (assuming all functions on edges are unique) 6 matrices, 2 intermediary vectors, and performs 6 matrix-vector multiplications. This could be re-written as an equivalent 1-deep 4-ary tree which requires just 4 matrices, 0 intermediary vectors, and performs 4 matrix-vector multiplications. This result cannot be improved upon, unless edge pruning of duplicate operations is performed.
This speed-up and memory improvement of an entire decapode in this fashion is unlikely to occur, since Op2s are likely to break this downwards tree-like property.
Originally posted by @lukem12345 in https://github.com/AlgebraicJulia/Decapodes.jl/issues/93#issuecomment-1403819186

This issue would allow for the most speed-ups in downstream Decapodes simulations ATM, since most bottlenecks relating to operator efficiency are resolved.


NullPointerException when loading mapping

Hi,
I'm new to ModelMapper and I'm not sure about how I should use it in this case :
I have an entity where I have a fileName. In my DTO I want to get the file content. So I would like to get the file content and set it in my DTO during mapping.
I try to writte a mapping :
@Configuration
public class FabricMapper {

	public FabricMapper(ModelMapper modelMapper, Environment env) {

		Condition<Fabric, FabricDTO> isFile = ctx -> ctx.getSource().getPictureFileName() != null;

		modelMapper.createTypeMap(Fabric.class, FabricDTO.class).addMappings(mapper -> {
			mapper.map(src -> src.getFabricMaterial().getId(), FabricDTO::setFabricMaterialId);
			mapper.map(src -> src.getFabricType().getId(), FabricDTO::setFabricTypeId);
			mapper.when(isFile).map(src -> {

				String filePath = env.getProperty("images.path")
						.concat("/")
						.concat(String.valueOf(src.getOwner().getId()))
						.concat(env.getProperty("image.fabric.path"))
						.concat("/")
						.concat(src.getPictureFileName())
						.concat(env.getProperty("image.extention"));

				File file = new File(filePath);
				String imgB64 = "";
				InputStream targetStream;
				byte[] imageBytes;
				try {
					targetStream = new FileInputStream(file);
					imageBytes = new byte[(int) file.length()];
					targetStream.read(imageBytes, 0, imageBytes.length);
					targetStream.close();
					imgB64 = Base64.encodeBase64String(imageBytes);
				} catch (FileNotFoundException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}

				return imgB64;

			}, FabricDTO::setPicture);
		});
	}

}

First mappings (FabricMaterial and FabricType) are working, but thrid one gives me a NullPointerException when I try to run the app, saying that src is null (obviously).
I suppose then that I shouldn't writte  my mapper this way, but I cannot find a way to writte one. I've read ModelMapper documentation but it's quite opaque for me.
Any suggestion about how to get file content from file path in a mapper? Is that possible with mapper or should I do it manually, outside of the mapping?
Many thx :)

Hi @Lempkin,
The map(...) only accept getter and setter. For more complicated case, please use converter using(converter).map(...). Thanks.


Add GH_PAT secret

As per koj-co/upptime#20, I see that you don't have a GH_PAT set. If you add this secret, your issue will be fixed.
In the Uptime CI logs, you can see the same error: https://github.com/soehlert/upptime/runs/974862548?check_suite_focus=true.

Wonderful, thanks for the issue. I have added this secret.


Optional logger

Thanks for writing this library! I would really like to be able to pass in an optional logger and have context passed around for handling cancelation. Hope this is something you'd be willing to incorporate. Thanks!

would like to get https://github.com/ammmze/wsman/pull/1 merged first, if possible.

@jacobweinstock Sorry...I'm not sure why I missed all this previous effort from you on this. Unfortunately I sort of gave up on this effort. Originally I had started it because I wanted to get AMT support in Sidero for my home lab. But then sort of lost interest and decided not to pursue Sidero in my home lab as it is way overkill as I don't change my lab often enough to need it. But I see you have pursued it and it looks like your fork was picked up in the bmc-toolbox project. Nice work!


[Feature] Add a changelog

Description
This could be a great way to keep track of the changes LinksHub has.
Screenshots
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[ ] I am willing to work on this issue (optional)
[ ] I am a GSSoC'23 contributor

Hey @ujjawaltyagii! :) I'm looking to assign this issue to someone. I know you're free, so do you want to work on it?

Hey @CBID2 can you please elaborate?

Hey @CBID2 can you please elaborate? Any suggestions or references how do you want it to be?

Hi @Lucif3r-in! :) I noticed LinksHub doesn't provide users with updates & I think a changelog would help with that.

Hey @CBID2 can you please elaborate? Any suggestions or references how do you want it to be?

Hi @Lucif3r-in! :) I noticed LinksHub doesn't provide users with updates & I think a changelog would help with that.

You suggest a markdown file with changes done. Correct me if I am wrong.

Yup. Here's more info about change logs.

@CBID2 I would need some time to figure it out :)
Also I have college exams starting from tomorrow T_T

@CBID2 Do you have any package in mind that will automate this process which you have used earlier?

@CBID2 Do you have any package in mind that will automate this process which you have used earlier?

Package? No, not really.

@CBID2 Do you have any package in mind that will automate this process which you have used earlier?

Package? No, not really.

Should we do it manually? I am looking for some package, if I don't find anything I will raise a PR with a Changelog.md that need to be updated manually🚀.

Automation could work. I just wonder what package would suffice

@rupali-codes, you got any suggestions?

@CBID2 I would need some time to figure it out :)
Also I have college exams starting from tomorrow T_T

Take your time @ujjawaltyagii! :) Good luck on those exams! :)

Release alone can be sufficient. However, we can implement a changelog for better documentation.
Since I am new to this, I am eager to work on it and learn more about the process.

@CBID2 Is anyone working on this issue?

Release alone can be sufficient. However, we can implement a changelog for better documentation.
Since I am new to this, I am eager to work on it and learn more about the process.

@CBID2 Is anyone working on this issue?

Sure go right ahead @Angelo

I will let you know about the process and the approach after raising a PR.

Not needed, as semantic drafting of release using gh is done by tamalCodes.


Feature request: Create a PR on the repo

I gave it a github api token that has access to my playground repo: https://github.com/catid/swe_agent_playground
Token is in the keys.cfg but it doesn't seem to read that so I also manually exported the key envvars.
It completed the task but did not create a PR or branch and does not have any useful errors explaining why:
         Let me know if you would like me to explain or modify anything! Otherwise, I think we can go
         ahead and submit this as the solution.

INFO     🎬 ACTION (primary)
         submit

INFO     Found submission:

INFO     Saved trajectory to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/catid__swe_agent_playground-i1.traj
INFO     Saved predictions to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/all_preds.jsonl
(swe-agent) ➜  SWE-agent git:(main)

Also no useful logs explaining why it did not submit anything in the trajectories folder.
Also noted that this project fails to start if the repo is empty.  I'd prefer to have it work on empty repos too.

Same happens to me: https://github.com/princeton-nlp/SWE-agent/issues/4#issuecomment-2032853966

This repo does not automatically interact with the github repo/issue that you are running it on. It generates a diff which can then be applied to a repo via git, but it is not an automated tool that manages repos. If that was the case every test in the SWE-bench lite would have thousands of PRS from people running this tool and others like it.
In trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-          2.00__install-1/catid__swe_agent_playground-i1.traj you can see this is a JSON file. Scroll to the bottom. You are looking for info and a sub-key called submission.
you can then apply the patch manually that was proposed

Timothy is correct. Although it could be cool to make this as an optional flag for private, non SWE-bench repos.

ok so the repo needs to be Public? I might have similar problem...
I have created an issue on my repo and used this cmd: python run.py --model_name claude-3-opus-20240229 \  --data_path https://github.com/cocobeach/Killer-Wasp/issues/1 --config_file config/default_from_url.yaml
(where is tha config_file?)
///ERROR:
Traceback (most recent call last):
File "/home/maxtensor/Dev/SWE-agent/run.py", line 223, in
main(args)
File "/home/maxtensor/Dev/SWE-agent/run.py", line 66, in main
env = SWEEnv(args.environment)
File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/swe_env.py", line 88, in init
self.data = get_instances(self.data_path, self.args.base_commit, self.args.split)
File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/utils.py", line 302, in get_instances
issue = api.issues.get(owner, repo, issue_number)
File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 62, in call
return self.client(self.path, self.verb, headers=headers, route=route_p, query=query_p, data=data_p)
File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 121, in call
res,self.recv_hdrs = urlsend(path, verb, headers=headers or None, debug=debug, return_headers=True,
File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 218, in urlsend
return urlread(req, return_json=return_json, return_headers=return_headers)
File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 119, in urlread
if 400 <= e.code < 500: raise ExceptionsHTTP[e.code](e.url, e.hdrs, e.fp, msg=e.msg) from None
fastcore.net.HTTP404NotFoundError: HTTP Error 404: Not Found
====Error Body====
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest/issues/issues#get-an-issue"
}
Any ideas?
Thanks a lot Great Repo, we all love it!!

@cocobeach No, it does not matter if the repo is public, private, or anything. The functionality to automatically push the diff to Github simply does not exist. This repo does not do that. If you wanted it to do that though, you can add it as a function the agent can all.

Ok then I don't understand, what happens with the issue number we give it when we hit run.py, what is it supposed to do in theory? Where are we supposed to retrieve the answer?

Ok then I don't understand, what happens with the issue number we give it when we hit run.py, what is it supposed to do in theory? Where are we supposed to retrieve the answer?

The scope of the project so far was to demonstrate and quantify the ability to solve real world gh issues. As @farouqaldori  correctly pointed out, you can see the git diff and semi-manually apply it.
I will look into whether we can add PR functionality soon.

Had the same error where it couldnt access my private repo but then added my github token to my bash profile and then it worked. @cocobeach

ok so the repo needs to be Public? I might have similar problem...
I have created an issue on my repo and used this cmd: python run.py --model_name claude-3-opus-20240229 \ --data_path https://github.com/cocobeach/Killer-Wasp/issues/1 --config_file config/default_from_url.yaml
(where is tha config_file?)
///ERROR:

Traceback (most recent call last):
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 223, in <module>
    main(args)
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 66, in main
    env = SWEEnv(args.environment)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/swe_env.py", line 88, in __init__
    self.data = get_instances(self.data_path, self.args.base_commit, self.args.split)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/utils.py", line 302, in get_instances
    issue = api.issues.get(owner, repo, issue_number)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 62, in __call__
    return self.client(self.path, self.verb, headers=headers, route=route_p, query=query_p, data=data_p)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 121, in __call__
    res,self.recv_hdrs = urlsend(path, verb, headers=headers or None, debug=debug, return_headers=True,
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 218, in urlsend
    return urlread(req, return_json=return_json, return_headers=return_headers)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 119, in urlread
    if 400 <= e.code < 500: raise ExceptionsHTTP[e.code](e.url, e.hdrs, e.fp, msg=e.msg) from None
fastcore.net.HTTP404NotFoundError: HTTP Error 404: Not Found
====Error Body====
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/issues/issues#get-an-issue"
}

Any ideas?
Thanks a lot Great Repo, we all love it!!

Had the same error where it couldnt access my private repo but then added my github token to my bash profile and then it worked. @cocobeach

As reported in #25, the GITHUB_TOKEN was not read from keys.cfg so far, so you had to specify it as an environment variable. This will be fixed with #31 :)


Data handling

Codecov Report

Merging #11 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #11   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           2      2
  Lines          68     62    -6
=====================================
- Hits           68     62    -6

Impacted Files
Coverage Δ

include/lars/event.h
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3fbd8a...df790b4. Read the comment docs.


cp2k_pdos.py on ldos files

Hi,
I try to use cp2k_pdos .py to process a file that comes from
relevant input in the cp2k input file
     &PRINT
         &pdos
            nlumo -1
            components .true.
            &ldos
                components .true.
                list 13, 129, 159, 160, 224, 251, 257, 367, 377, 726
            &end
         &end
     &END PRINT

of course the produced file fails on cp2k_pdos.py
I understand why it fails. and I can patch it to read the file but my question: are there any pitfalls that I shall have in mind on using this on a subgroup of atoms, rather than the full system?
Regards,
Alin

From a script perspective: not that I could think of. But also from the CP2K-side should everything work as expected (especially since it's in the postprocessing part).
cp2k_pdos.py is the only tool I haven't migrated yet to https://github.com/cp2k/cp2k-output-tools

Great! I wanted to be sure I missed nothing.