Docker logging blocks and never recovers

You can permanently block a production container by listening too slowly on a log socket with any client. Docker eventually blocks the container process, and it never recovers. The container will remain blocked on stdout/stderr unless it's handling output asynchronously and dealing with output overruns on its own. This is new behavior introduced in November, which utterly changes the behavior of logging in Docker. It is not even reflected in the changelogs.
Output of docker version:
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 20
 Running: 20
 Paused: 0
 Stopped: 0
Images: 26
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-202:16-4407302-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 9.992 GB
 Data Space Total: 107.4 GB
 Data Space Available: 97.38 GB
 Metadata Space Used: 11.25 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.136 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /mnt/var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /mnt/var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-83-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 29.4 GiB
Name: ip-10-3-13-191

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on AWS, repeatable on VMWare
Steps to reproduce the issue:

Simulate a high throughput production app with some heavy timestamp logging: docker run -d busybox sh -c "while true; do date; done" . Note the container ID for later.
Fetch blockingreader.go from here . This connects up to a container, sets up some pipes, and then never reads from them, letting Docker buffer internally.
go get github.com/fsouza/go-dockerclient
docker logs --tail=2 -f <your_container_id> in another terminal
go run blockingreader.go <your_container_id>
Watch the log output for a while and the app will stop logging once it fills the buffer and blocks. You    can't tell from just busybox but the app is not just not logging, it's actually totally blocked. You could hook GDB up to it if you really want to see. We took stack dumps of our app to prove it. This is bad because you've now blocked a production app with a client connection to the logs.
CTRL-C or kill or your blockingreader process which will release the Docker client socket and you would expect to see Docker unblock the container. In extensive testing, this does not happen. Even when the blocked reader is gone, Docker continues to block the container.

Describe the results you received:
Docker eventually blocks on high output logging if any client is slow and the internal logging buffer fills. This was not expected. It then also does not release the container when the socket is disconnected. This was really not expected.
Describe the results you expected:
Docker would continue to accept logging output from containerized applications without blocking (or leaking GBs of RAM like it used to). I would have expected a ring buffer implementation or some kind of sliding window without blocking at all.
Additional information you deem important (e.g. issue happens only occasionally):
100% repeatable. We tripped into this issue because we use Mozilla's Heka to log all of our container output. Heka connects to the Docker logs endpoint and sometimes becomes slow under heavy logging. In this way you can take out an entire server of Docker containers because of the permanent blocking behavior of the logs.

ping @aaronlehmann @unclejack PTAL

Can you test with v1.11, very similar issue of what you are describing was fixed with #21840, at least this should fix the behavior after ctrl-c.

@tonistiigi I don't have access to 1.11 at the moment, will have to test tomorrow (I'm on UTC+1). You should be able to repeat the steps above in a minute or two if you have a 1.11 to test against.

@relistan I tested with your script and #21840 does seem to solve this. Note that point 6 behavior is somewhat by-design and is not fixed. If you have one reader that is very slow and one that is very fast we first buffer  to memory but when this buffer fills up we are going to rely on the slower reader speed(0 in your case). If we didn't do that we would just run out of memory. Maybe we could provide something to tweak the buffer size. The bug in v1.10 is that once you close the slow reader the fast one remains blocked. In v1.11 docker logs will start showing new output after blockingreader.go is closed.

Thanks for testing @tonistiigi. I don't believe that is the right behavior in item 6 above, even if it's by design. Docker should drop log messages to the slow reader, not block the app itself. Note that in #6 it's not just the slow output that is blocked, it's the whole stdout of the container. That's a big change in functionality vs 1.9 and also not very friendly behavior in a production environment. For example, someone watching the docker logs command over a slow link should not be able to block a production application.

cc @aaronlehmann

I don't think dropping data from the log stream would necessarily be the correct behavior. This could break the formatting of the logs and lead to unexpected gaps in the log data. Perhaps it would be reasonable as an option. Alternatively, the buffer size could be made configurable.

@aaronlehmann so you are saying its acceptable behavior to block after a buffer is reached in a production environment?  Dropping data or corrupting the formatting is definitely much better than blocking(thus making the app unusable) would it be feasible to at least have this as a config option to drop instead of block in such cases?

Isn't there a separate buffer for each reader? That's what it looks like from the code. So if I have the daemon logging to a file, and two clients watching docker logs then I have at least three buffers, no? So why would we block the other readers and the application itself when one of the docker logs calls is slow, rather than simply dropping data on the slow link? Honestly, if Docker is going to block the whole container on stdout/stderr for one slow reader, then it is unsafe to ever use docker logs in production.

correctness > speed. Something consuming the data and getting the bogus input is potentially far more dangerous than the decrease in speed. The buffer there is implemented to handle this problem and the size is something we thought was unreachable for any normal usage. Of course usage may vary, so configuring the size or using disk as a cache are solutions here.
I don't think this has anything to do with logs or the logging speed, the problem appears because there is an attach stream that refuses to read data. (Of course if the logdriver was considerable slower than attach stream then this would be opposite).
There can be 2 buffers per io stream. One if there is an attach stream and other if a log driver is attached. New docker log commands don't create new buffers.

@tonistiigi we're talking about logs here, right? Having worked on production systems for over 20 years I can't remember a time when I would have preferred to have complete logs over having the application running with the exception of audit logging, something generally not entrusted to generalized logging anyway. This is the reason many people don't log to disk, and why traditional stream logging was done over UDP, for example.
I know you said you don't think it's about logging speed, but it is. The tool I provided above is the fastest way to make it happen so it doesn't read anything. In the production environment where this hit us, Heka was consuming the stream, but it had gotten backed up and was slow. I know this because we replaced our log appender with an asynchronous one. We then attached metrics to the remaining buffer size for our asynchronous appender. You can see it go up and down as Heka consumes messages. Of course since we're on 1.10.3 it eventually sits at low water mark forever because the Docker engine stops consuming.
Making the buffer bigger would make this happen less frequently perhaps, but it is not a solution. Also no one has addressed that this was a fundamental change in the Docker engine and that it wasn't in the changelogs.
If there are actually people who prefer this behavior, can we have a CLI flag to switch it? If not, I'm not sure what solution I'll recommend in production, including in "Docker: Up and Running".

correctness > speed indeed, a production application thats blocked and unusuable because a peripheral system is having a bad time is not correct - it's completely dead and unusable in a way that might impact customers.
logging is peripheral to the primary function of the application.
This behaviour leads to a system that as soon as you flag an app in docker to enable debug level logging for a while then in the case where the debugging is volumous the system under debug becomes unusable - and impossible to debug.  Operationally you'd much rather sacrifice logging integrity.
Systems where logging is so important as to be a critical function of the app will/should not be doing that logging to STDOUT, so sacrificing integrity of STDOUT logs seems reasonable to me over sacrificing the entire app.

+1 @ripienaar @relistan
I understand there are applications where it is absolutely critical that no log is lost. I would argue that is by far outweighed by applications that are more than happy to shed a few logs to survive a heavy load event.
If it were configurable you could give people the option of 100% log capture but I would say the default should be to throw logs on the floor for a consumer with a full buffer.

Something consuming the data and getting the bogus input is potentially far more dangerous than the decrease in speed.

Someone able to cause a denial of service by using a slow log reader is even more dangerous, as "blocked" in this case is just another way to say DoS. In the field, one simply never assumes that syslogs are 100% complete because syslog, like ICMP, is one of the first things that is dropped to ensure the responsiveness of the service.
Some systems note "## syslog events were dropped" so that purposeful gaps can be differentiated from gaps caused by communication interruption. That may be helpful here to some extent.

@rnelson0 In this case, a slow reader could have caused unbounded buffer growth, eventually causing the entire daemon to crash.
Crashing the daemon is much worse than blocking a single application.
A slow docker logs consumer would not even be able to block the application.
The reason the application would get blocked is because of https://github.com/docker/docker/blob/master/pkg/broadcaster/unbuffered.go#L27
docker logs does not use this.
Only the logging driver itself (for writing logs to the log service) and docker attach would be in there.

Sorry if it was unclear, I was making analogies between a log reader dropping logs and how syslogs are treated, but not trying to directly relate them.

The speed of a docker logs consumer does not have an impact on the container stdio write speed. What happens to the logs depends on the behavior of the log driver. If you use syslog driver and can configure it to drop logs on syslog buffer filling up it should start to drop them. JSON-file logger that is the default will not drop logs and never did before v1.10.

@cpuguy83 @tonistiigi Are you suggesting that the reproduction steps above do not cause the application to be blocked? It sounds like you're now saying that this bug is not valid. It seems that #21840 also proves that this can happen. I suggest that you try it on 10.1.3 with the reproduction steps above, it definitely blocks. And, we have production evidence that it does. I thought we were all agreed on the premise at least and simply arguing the correctness of the implementation.
@tonistiigi the JSON logger never dropped logs before v1.10 but it did not block the container, either. It had a different issue: it just blew up in memory. That was very bad behavior but at least in production we could manage around it: you had a little time before things crapped out and you could monitor for it. What we see now is the container just blocks. If you believe the write speed should not be affected then it looks likely that there is instead a concurrency issue somewhere.
The problem with all of the other drivers, including the syslog driver, is that you give up the ability to look at the log stream from a single container with docker logs. That has, up to now, been the advantage of running the JSON-file logger with Heka or some other process consuming the output stream from a socket.

@relistan The example is contrived. This would only happen with either a blocked/slow logging driver (note this is a driver not docker logs consumers`) or by a blocked/slow attached client.
(Some) Logging drivers can be configured for async logging, this is your choice to risk dropping logs.
Issues with recovery after unblocking (e.g. by closing down the client connection) are fixed in 1.11.
I was able to reproduce the recovery bug on master (not on 1.11), so I'm investigating this regression.
I could see about implementing a way to make sure that attached clients can't block, but this would also be a significant change for some.

And I take that back, I'm not seeing the recovery issue on master.

@relistan docker logs and docker attach have very different behavior. docker attach directly consumes output to the client(limiting throughput using a 1M buffer if the client link is slow). There is a bug in v1.10 for docker attach, I reproduced it and it is already fixed in v1.11. docker logs is different and reads already logged messages back from the driver. This shouldn't have any other purpose in your reproduce script other than you use this output to verify that the streams are being blocked by another component. So if you don't run your production apps in foreground over a slow link this should not affect you.

@cpuguy83 @tonistiigi the behavior of docker logs is not the concern.  what are you suggesting the behavior of the container and daemon should be with an attached blocked/slow client?

@mjvaldez Attached clients get a 1MB buffer. This is a rather large buffer and clients would probably have to just not be reading from the attach stream at all to hit this buffer limit.
In practice, in any environment, why is something attached to a container in such a way?

@cpuguy83 please see comments above where relistan describes the use case and ramifications of the behavior we saw in production.

@mjvaldez AFAICT this is centered around logging. Can you explain how you have logging setup?
Docker does have support for async logging in some of it's drivers, which seems like you might want to utilize here.

@relistan What version of mozilla-heka do you use? Seems old versions used the attach stream but it was fixed in https://github.com/mozilla-services/heka/commit/bd5c4e853852e6d1c292e389da7ce0cacb318fb6

The fundamental misunderstanding here is that it was not clear to me (and I think also at least the authors of Logspout, and Heka until recently) that /attach and /logs do not hit the same buffers inside the daemon. It doesn't help when, even with code to look at @aaronlehmann continued to refer to "logs" when he can see that we're using /attach:

I don't think dropping data from the log stream would necessarily be the correct behavior. This could
break the formatting of the logs and lead to unexpected gaps in the log data.

I was not aware of that and I guess no one at Docker (at least none of you guys) knew that at least two of the major third party logging systems were until not very long ago using the /attach endpoint to consume logs. I can assure you because at New Relic we ran Heka in production for quite some time, and at Nitro for a few months on v1.9, that this did not block containers until v1.10. It did leak memory like crazy when the reader got slow. At New Relic we ran a custom patched build that resolved the memory leak issue that @didip and I patched here.
I am a major contributor on the Heka Docker plugin and we've been running it at Nitro just fine until we upgraded to Docker v1.10 in production and the change took out our site not long after. We'll pull in Rob's recent changes to the Heka Docker plugin that switches it to the /logs endpoint and expect that it resolves this issue at least.
Nonetheless, this functional change to the behavior of the /attach endpoint really needs to go into the release notes for v1.10. It certainly would have raised a red flag for me. And I suggest adding some clarity to the Remote API documentation for the /attach endpoint which in various places refers to pulling in logs.
I would suggest that a 1MB buffer (not quite, it's 1e6) is also not anywhere near big enough. With average log line average length around 100-150 bytes , this is 6666 to 10k lines of buffer. A high throughput production app can blow that buffer very quickly, especially in debug logging mode.
Finally, an explicit warning that docker attach can block a container is a critical addition. It's not mentioned in the docs. And anyone with experience prior to v1.10 will be reasonably certain that it doesn't.

Inadvertently closed rather than commented.

I'm closing this as the issue was fixed in v1.11(#21840) and documentation was clarified in #22611 . If a variable buffer size is needed this can be discussed in a separate issue. I'm not sure if it is a good idea as reaching this limit should be an indication that the endpoint is being misused.

Just fyi, for 1.14 we've added a new logging mode that you can pass through via log-opt.
--log-opt mode=non-blocking --log-opt max-buffer-size=4m, for example.
See #28762 for details.

still hang on write(1 on docker 18.01.0-ce even with mode=non-blocking

@Laisky mode=non-blocking means the container will not be blocked on write for logs.
IF you have a client attached to the container stdio and the client is not consuming it (or is consuming too slowly), you will get blocks on write.

@cpuguy83 thank you for your answer.
I can reproduce the problem by simply run one container with docker 18.03.0-ce, build 0520e24 (also on 18.01):
docker run -itd --rm --log-driver=fluentd --log-opt="fluentd-address=<IP_ADDRESS>:24225" --log-opt tag="test.sit" --log-opt mode=non-blocking --log-opt max-buffer-size=50m --log-opt fluentd-async-connect=true --name test1 test

the docker image of test is a python script with infinite loop to print string, such like:
import datetime

while 1:
    print(datetime.datetime.now())

and the fluentd-address is just an random ip, the fluentd-async-connect can make sure this container starting properly.
after running a few minutes, the python script in this container will be blocked:
# strace -tt -p 26500

Process 26500 attached
17:00:09.487341 write(1, "\n", 1

@Laisky I cannot reproduce
Logs are full with:
ERRO[2019-02-01T23:37:40.443055483Z] Error writing log message                     container=23d79a64ce8356808efaf9d232280c4dc58ae8d9d573ff5dd37733ed7ee9718f driver=fluentd error="fluent#appendBuffer: Buffer full, limit 1048576"

Until I stop the container.


[CT-2485] [Bug] RunOperationResultsArtifact does not contained expected data with programmatic invocation

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
I am attempting to get results from programmatic invocations of DBT operations.
As a toy, I am trying to get the output from dbt-codegen's generate_model_yaml macro (https://github.com/dbt-labs/dbt-codegen/blob/e24e5dc5daca47ba350415a59790d6b741be246b/macros/generate_model_yaml.sql#L50). I'm imagining a script that writes these schema YML files automatically.
The macro prints to the log in addition to the return command. I would expect that these would be included in the dbtRunnerResult class, specifically in the results item of  RunOperationResultsArtifact.
My script:
from dbt.cli.main import dbtRunner, dbtRunnerResult

# initialize
dbt = dbtRunner()

# create CLI args as a list of strings
cli_args = [
    "run-operation",
    "generate_model_yaml",
    "--args",
    """{"model_names": ["my_model"]}""",
]

res: dbtRunnerResult = dbt.invoke(cli_args)

print(res)

Expected Behavior
Output from return function from codegen macro is returned in the dbtRunnerResult class.
Steps To Reproduce
Run the script I posted in Python for a model with dbt-codegen installed.
Relevant log output
DBT logs expected output, returned object is this (note results:[])

dbtRunnerResult(success=True, exception=None, result=RunOperationResultsArtifact(metadata=RunOperationResultMetadata(dbt_schema_version='https://schemas.getdbt.com/dbt/run-operation-result/v1.json', dbt_version='1.5.0rc2', generated_at=datetime.datetime(2023, 4, 24, 21, 22, 36, 550559), invocation_id='e239cd3c-4fdc-40a9-b39b-7707539ce014', env={}), results=[], elapsed_time=1.699451, success=True))

### Environment

```markdown
- OS: mac OS13.2.1 (22D68)
- Python: 3.9.11
- dbt: 1.5.0-rc2

- dbt-codegen: 0.9.0

Which database adapter are you using with dbt?
snowflake
Additional Context
No response

Opened this ticket in response to Slack thread https://getdbt.slack.com/archives/C2JRRQDTL/p1682370679158779?thread_ts=1682368271.410369&cid=C2JRRQDTL

@ttusing Very cool! Thanks for taking the v1.5 RC for a spin :)
The run-operation command doesn't include the value returned by the called macro to its result. It probably should! We get access to it here. We just don't then include it in the RunOperationResult / RunOperationResultsArtifact.
The codegen.generate_model_yaml macro both logs and returns its contents, so it's doing everything that it needs to.

In the meantime... what if we tried this with "interactive compile" instead?
from dbt.cli.main import dbtRunner, dbtRunnerResult

# initialize
dbt = dbtRunner()

# create CLI args as a list of strings
cli_args = [
    "compile",
    "--inline",
    "{{ codegen.generate_model_yaml(model_names = ['my_model']) }}",
]

res: dbtRunnerResult = dbt.invoke(cli_args)

print(res.result.results[0].node.compiled_code)

>>> print(res.result.results[0].node.compiled_code)
version: 2

models:
  - name: my_model
    description: ""
    columns:
      - name: id
        description: ""

I'll admit, res.result.results feels a little wonky to look at. This is a super-generic runner; here are the types involved:
print(type(res))                        # dbtRunnerResult
print(type(res.result))                 # RunExecutionResult
print(type(res.result.results))         # List[RunResult]
print(type(res.result.results[0].node)) # SqlNode

I'm going to change this from a bug to an enhancement. We can discuss & estimate this piece in particular:

The run-operation command doesn't include the value returned by the called macro to its result. It probably should! We get access to it here. We just don't then include it in the RunOperationResult / RunOperationResultsArtifact.

Since this use case is quite new (with programmatic invocations in v1.5), and has a valid workaround (maybe even better?), I don't think this is something we'd prioritize highly.

per BLG 5/23:
@jtcohen6 We;d like to solve this for the general case-- specifically we'd like to have some sort of consistency in our task RunResults and we'd like to contract and document it for our users.  We can either expand this ticket to do that or start something new but we're going to need your input on that.


Switched ol.source.ImageWMS BBOX coordinates, when using some custom projection?

My code snippet looks like this:
new ol.layer.Image({
	source: new ol.source.ImageWMS({
		url: 'http://<IP_ADDRESS>:8080/geoserver/BP/wms',
		params: {'LAYERS': 'BP_inzinerine_infrastruktura'},
		serverType: 'geoserver',
		projection: ol.proj.get("EPSG:3346")
	})
})

This code generates URL:
http://<IP_ADDRESS>:8080/geoserver/BP/wms?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=BP_inzinerine_infrastruktura&CRS=EPSG%3A3346&STYLES=&WIDTH=1871&HEIGHT=944&BBOX=296090.96544524713%2C5901662.523222233%2C867505.2989656775%2C6189965.639653706
And I get an empty image...
If I manually switch BBOX values like shown below, I get a correct one.
http://<IP_ADDRESS>:8080/geoserver/BP/wms?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=BP_inzinerine_infrastruktura&CRS=EPSG%3A3346&STYLES=&WIDTH=1871&HEIGHT=944&BBOX=5901662.523222233%2C296090.96544524713%2C6189965.639653706%2C867505.2989656775
Is there an easy fix for this? I am new to WMS, so there may be something important I don't understand at the moment. Or is it a GeoServer issue? The problem is that this service is not mine.

I have just found some dirty solution by looking at ol.js code. Projection must have g property with "ne" value in order to work properly... Is there some cleaner solution? At least it works!

How do you define your projection object?  I was using too basic projection definition.
I have just defined my CRS with ol.proj.Projection() using proper axisOrientation value and it works like a charm!
Thank you again!


Change references form to reactive/adaptive design

First off, the reason this is such a mess is because PBDB is not using industry standard terminologies for citation - cf., endnote, RIS format - and then is not even using its own internal fields correctly. For example, it uses pubtitle for book publisher even though they have a separate publisher field that is always blank. I dunno.
ANYWAY, the plan is to make this form reactive - i.e., available/displayed fields will be conditional upon publication type selection - this will (when finished) supersede the info provided by the pbot team associated with #71.
--- START FORM ---
First field: PBDB ID option (great as already implemented)
Second Field: Publication Type (picklist of journal article, standalone book [pbdb.pubtype: book], edited book of contributed articles [pbdb.pubtype: book], Contributed Article in Edited Book [pbdb.pubtype: book chapter], Unpublished [does not go to pbdb?])

journal article is self-explanatory

Standalone Book is what is referred to in publishing as a monograph (not to be confused with a biological/taxonomic monograph!! >.<). This means that the entire book was written by a consistent author or set of authors. War and Peace by Tolstoy is a monograph or Dragons of Autumn Twilight by Margaret Weis and Tracy Hickman is a monograph. Importantly, fields like series/volume/issue/editor have no meaning/use in this context - hence why they are omitted.

edited contributions is the most common type of book we will actually encounter. The difference is that it consists of articles written by different authors. Annoyingly, these volumes can be one-off OR released in a series. Therefore fields like volume/number may or may not be relevant. Furthermore, if the entire volume is being cited then the editors are the authors and no editors are recorded.

journal article (PBDB equivalent is journal article)

PBOT Display Name
PBOT Property Name
PBDB Field Name
Type
Required

DOI
doi
doi
free text
FALSE

Title
title
reftitle
free text
TRUE

Year
year
pubyear
4-digit integer
TRUE

Authors
NA
NA
NA
TRUE

Journal
journal
pubtitle
free text
TRUE

Volume
volume
pubvol
integer
TRUE

Number
number
FALSE
integer
FALSE

Pages
Pages
firstpage, lastpage
integer range
FALSE

Standalone Book (PBDB equivalent is book)

PBOT Display Name
PBOT Property Name
PBDB Field Name
Type
Required

DOI
doi
doi
free text
FALSE

Title
title
reftitle
free text
TRUE

Year
year
pubyear
4-digit integer
TRUE

Authors
NA
NA
NA
TRUE

Publisher
publisher
pubtitle
free text
TRUE

Pages
Pages
firstpage, lastpage
integer range
TRUE

Type
booktype
NA
thesis, monograph, compendium, other
TRUE

Edited Book of Contributed Articles (PBDB equivalent is book)

PBOT Display Name
PBOT Property Name
PBDB Field Name
Type
Required

DOI
doi
doi
free text
FALSE

Title
booktitle
reftitle
free text
TRUE

Year
year
pubyear
4-digit integer
TRUE

Editors
NA
NA
NA
TRUE

Publisher
publisher
pubtitle
free text
TRUE

Pages
Pages
firstpage, lastpage
integer range
TRUE

Contributed Article in Edited Book (PBDB equivalent is book chapter)

PBOT Display Name
PBOT Property Name
PBDB Field Name
Type
Required

DOI
doi
doi
free text
FALSE

Chapter Title
title
reftitle
free text
TRUE

Book Title
booktitle
pubtitle
free text
TRUE

Year
year
pubyear
4-digit integer
TRUE

Authors
NA
NA
NA
TRUE

Editors
editors
editors
free text
TRUE

Publisher
publisher
publisher
free text
TRUE

Pages
Pages
firstpage, lastpage
integer range
TRUE

Unpublished

PBOT Display Name
PBOT Property Name
PBDB Field Name
Type
Required

Title
title
NA
free text
TRUE

Year
year
pubyear
4-digit integer
TRUE

Authors
NA
NA
NA
TRUE

Description
description
NA
long free text
TRUE

This is in
paleobot/pbot-api@51548582b868d22cb25e79fbf52059c36116699a
44e56f0c13b9b3230a2b7b28e4ca45b8c42d6643
and
fa878ec9e20110279da5daaf0d95eac742c92f6d
b75d54bb3ea975f90989ec9e3442e2479507eeac
65f51823425b59bfc980b2fd4fbbf37efac53f68
cd0823e5bfd808b7781ac8e751387e28345b719a
8475b8a97aa9124e2208f3bd4c31c2071056dd7b
ebdcb1637835de0c0ee05da3724f2ede51149744
f1ff1e7cb4232d728a94655bc4c359a3a6ab96f2
6d62ae74951ba136c36e5363a98d6334129d95d5
e794ab220d2003a79370697ead400464904ef6e4
559d463cb3d2c99b7220a612fda11491baadeab3
72373ead5aa7fc6ad2f63222557de0a66fe81534
3c91c2e09c4c70cdccc665737973ffbaf0398336
63e27c0ad6ffbdcdefa84c4d6debe122d5ff5244


Protect form submit from pre-init use (close 4360)

The details of the requirement and the occurrence are contained in Issue #4360. This fix has been tested against the issue noted there and fixes it.
Simple addition to prevent use of contextStorage if not initialised. This is a valid check as it can't be already sent (therefore the flag check isn't required) if the storage isn't even initialised.
Other holes may exist with the same pattern but only the form submit is handled here. The other likely candidate is _onJsError() if the error is triggered prior to the driver start.

@testcafe-build-bot retest

@testcafe-build-bot retest

@AndreyBelym I'll add the test. I prepared example when researched the issue

@testcafe-build-bot retest

@testcafe-build-bot retest


Fix-Vesting(Origin).withdrawTokens

Fix for issue 178
Lock date for Origin vested tokes is<PHONE_NUMBER> -> GMT: Friday, March 26, 2021 10:28:15 AM
VestingLogic passes startDate + cliff as a timestamp
startDate =<PHONE_NUMBER> -> GMT: Friday, February 26, 2021 10:28:15 AM
cliff = 1776077
startDate + cliff =<PHONE_NUMBER> -> GMT: Thursday, March 18, 2021 11:49:32 PM
===
How to fix:

Adjust date using Staking.timestampToLockDate
Add 2 weeks (we need to select end of period, by default it will be start of period) if dates before and after adjustment are different (it's the case only for Origin Vesting contracts).

===
Checked using script check_origin_vestings.py
It's better to check on real data rather than using unit tests, it's an integration issue.
checked-origin-list.csv - script result

That's a dirty hack that should be refactored later. Considering tight schedule it is ok (sort of).
Please note that it's difficult to prove whether there are any other missed edge cases.

Agree, but we don't have another option to unlock Origin Vesting contracts.
This fix shouldn't affect any vesting with correct dates.
Also it can be removed as soon as all tokens of Origin Vestings claimed.


Commit creation

We could redesign PartitionState to store a list or map of changes instead of a
parent sum. clone_child() would then reset the list of changes.
This would optimise for few changes to a large set of elements (which is the
intended usage); the current optimises more for many changes to a small set of
elements.
We could support both of these via separate types, plus possibly a third type
which does not support modification. There would be a clone_... method for
each type and a Partition would need to be able to create a commit from each
type which supports modifications.

State remembers parent: has to store a Sum (currently 32 bytes)
State remembers Vec of changes: 24 bytes (I think)
State remembers HashMap of changes: 40 bytes (I think)

[ ] Code tag #0021

The main part of this was implemented in order to allow metadata of new states to be set in a "correct" way.
The remaining question: should the new MutPartState continue to operate by mutating a set of elements or instead build a set of changes?


decimal/benchmarks/pi_test.go broken by changes to dnum

decimal/benchmarks/pi_test.go no longer builds because of changes to github.com/apmckinlay/gsuneido/util/dnum  package.
cp pi_test.go pi_test-.go
sed -e '1,$s/NewDnum(false, \([0-9]*\), 0/FromInt(\1/' < pi_test-.go > pi_test.go

I'll take a look. Thank you for letting me know!

FYI, s/dnum.Cmp/dnum.Compare/
I think dnum is undergoing some cleanup.

Awesome. If you don't mind me asking, what're you using the library for? It's super interesting to see what people use it for.

You'll be sad(ish).  I work on the Go compiler, we have an inadequate (in my opinion) set of benchmarks, so if anyone with a mildly popular project has also written benchmarks, and I find them, and they are not too much of a duplicate of something we've already got (and not too noisy, not too slow, etc) then I incorporate them into the list of benchmarks to run.  See
https://github.com/golang/go/wiki/Benchmarks
I get + build + run/test then using https://github.com/dr2chase/bent (this is Google code under my account to avoid the annoying problem of "where do we put this"; I need to get the copyrights etc in place).
This may or may not lead to more attention paid to whether your code runs well and/or compiles under the development version of Go.  The benchmark runner also includes a run-tests option.

That's still pretty neat. Glad to see it could be of help!


Add nomis integration flag to prisoner validity check

This is to inform send-money about different behaviour that needs
to be used for prisoners who are in prisons where NOMIS integration
has been rolled out.

This is for deciding what copy to use in emails/confirmation page (i.e. do we tell them that they will receive an email when the prisoner is credited).


Switch to chart.js for the release activity page

This PR switches the charts in the /releases/activity page from Highcharts to chart.js, improving multiple things:

Highcharts is proprietary, and I'm not actually sure docs.rs fits into the cases where we don't need a commercial license for it.
Highcharts doesn't fit well with CSP, requiring style-src 'unsafe-inline' to work.
This removes the last dependency we fetch from cdnjs, fixing #921.

Rendered example of the page with chart.js:

Test failure is probably #1251 making the test flaky, probably needs reworking to inject more test crates to ensure there is a result.

Test failure is probably #1251 making the test flaky, probably needs reworking to inject more test crates to ensure there is a result.

If anyone wants to test it locally I recommend running this query to fill some data in:
UPDATE config SET value = '{"counts":[233,281,431,564,390,188,230,291,320,352,458,332,349,338,358,410,467,579,841,397,565,298,407,532,511,456,484,426,326,295],"dates":["19 Dec","20 Dec","21 Dec","22 Dec","23 Dec","24 Dec","25 Dec","26 Dec","27 Dec","28 Dec","29 Dec","30 Dec","31 Dec","01 Jan","02 Jan","03 Jan","04 Jan","05 Jan","06 Jan","07 Jan","08 Jan","09 Jan","10 Jan","11 Jan","12 Jan","13 Jan","14 Jan","15 Jan","16 Jan","17 Jan"],"failures":[16,9,6,7,29,8,6,8,4,13,18,12,11,4,19,18,28,44,23,17,14,8,10,28,19,14,7,13,8,17]}' WHERE name = 'release_activity';

I assume we have no tests covering this page

Yeah, we don't really have a way to do UI tests as of today.

If anyone wants to test it locally I recommend running this query to fill some data in:
UPDATE config SET value = '{"counts":[233,281,431,564,390,188,230,291,320,352,458,332,349,338,358,410,467,579,841,397,565,298,407,532,511,456,484,426,326,295],"dates":["19 Dec","20 Dec","21 Dec","22 Dec","23 Dec","24 Dec","25 Dec","26 Dec","27 Dec","28 Dec","29 Dec","30 Dec","31 Dec","01 Jan","02 Jan","03 Jan","04 Jan","05 Jan","06 Jan","07 Jan","08 Jan","09 Jan","10 Jan","11 Jan","12 Jan","13 Jan","14 Jan","15 Jan","16 Jan","17 Jan"],"failures":[16,9,6,7,29,8,6,8,4,13,18,12,11,4,19,18,28,44,23,17,14,8,10,28,19,14,7,13,8,17]}' WHERE name = 'release_activity';

I assume we have no tests covering this page

Yeah, we don't really have a way to do UI tests as of today.


Error report: Brewfile already exists

brew config output
HOMEBREW_VERSION: 3.4.4
ORIGIN: https://github.com/Homebrew/brew
HEAD: 5f5af43244eaece1f09d695603e1a261676713a0
Last commit: 6 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 05ee6d711f33a7db2c363b3c3cf03096d23756d3
Core tap last commit: 7 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.rw5fhR8p2N/org.xquartz:0
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.8 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 13.1.6 build 1316
Git: 2.35.1 => /opt/homebrew/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.3.1-arm64
CLT: <IP_ADDRESS>.1.1645755326
Xcode: 13.3
Rosetta 2: false

brew doctor output
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Verification

[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?
I reinstall the brew and use brew bundle install --global with Brewfile. Although I successfully install those old pkgs. But after this, every time I open a new terminal window or tab, brew always reports an error, like:
Last login: Fri Apr  1 18:12:13 on ttys000
Error: /Users/username/Brewfile already exists
(base)  wurining@MacBook-Pro-16 >

What happened (include all command output)?
Last login: Fri Apr  1 18:12:13 on ttys000
Error: /Users/username/Brewfile already exists
(base)  wurining@MacBook-Pro-16 >

What did you expect to happen?
Dont show the error. And I wanna know the reason.
Step-by-step reproduction instructions (by running brew commands)
Just open a terminal.

Assuming you're running Zsh as your shell, man zsh and read the STARTUP/SHUTDOWN FILES section. Your error strongly suggests that one of your shell's startup files contains a brew bundle dump --global command, which does not overwrite the old Brewfile.
You either want to brew bundle dump --global --force to force overwriting, or probably remove that line, as it doesn't make much sense to re-run a bundle dump every time you start a new shell.

@gromgit
Thank you so much! Your advice looks quite useful. There indeed exists a brew bundle dump command in the file .zprofile. I cannot remember when it was added. Now I've cancelled this action.
.zprofile content:
eval "$(/opt/homebrew/bin/brew shellenv)"
eval "$(/opt/homebrew/bin/brew shellenv)" brew bundle dump


Convert theme switcher to tsx

Convert theme switcher component to typescript

I'll review this one after we merge the bonds upgrades to develop to go with https://github.com/OlympusDAO/olympus-frontend/pull/720

Hey @alexleventer, thanks for you contribution. I noticed you've been doing some TS refactors. While these are appreciated, pretty much all of the app is refactored and pending the V2 migration.
We have a PR open in here, feel free to take a look and if you'd like to improve TS components, I recommend checking this branch which has the last batch of components.
Saying this so you won't end up repeating work that has been done.
There's a bunch of issues listed here that need attention if you wanna dip your toes further!


The Oct 1 comment needs a year attached

[Enter feedback here]
Reference comment: SQL Server licenses acquired before October 1
SQL Server Enterprise edition has both host-level and by-VM license options.
There is no year attached to October 1...
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 3d9b71dd-761f-8aa3-bdcf-982f3d2da4e9
Version Independent ID: 6553a011-d3b5-f9a4-2014-c3b9373c82a7
Content: Run SQL Server VM on an Azure Dedicated Host - SQL Server on Azure VMs
Content Source: azure-sql/virtual-machines/windows/dedicated-host.md
Service: virtual-machines-sql
Sub-service: management
GitHub Login: @bluefooted
Microsoft Alias: pamela

Added the year (2019) in the following PR: https://github.com/MicrosoftDocs/sql-docs-pr/pull/29717

#please-close


Isn't working as desired with TextChangedI

I added TextChangedI as an event. vim-auto-save abruptly shifts cursor position to the left while inserting text. Guess it is a escape and entering insert mode with i issue.

@MahbubAlam231 I've just tested TextChangedI and do not see any issues with cursor shifting (neovim v0.2.2). AutoSave doesn't send Esc / i keys. You should try to disable all other plugins and custom configs/mappings to check if there any conflicts.

I'm using Vim 7.4, could that be any issue?

@MahbubAlam231 I don't know 😕

Anyway thanks.


Fix factory girl dependency for Solidus < 2.5

We need to load a factory_bot version that has factory_girl in it
to support Solidus versions < 2.5
This change also includes conditional logic for the database
interface gems.

Thank you! Fixed with PR https://github.com/boomerdigital/solidus_amazon_payments/pull/68


LibusbDeviceDescriptor

I was trying to build the list devices example under windows 64Bit with the latest Nim devel branch and vcc. I also used the latest libusb master branch. Everything compiled successfully but some product ids were printed as 0000. After changing the type cint to uint16 for idProduct and idVendor in LibusbDeviceDescriptor everything worked fine.

Thanks, I'll take a look soon!

Sorry for the delay - I've been too busy at work. Please let me know of any other issues in the future, thanks!


Is it okay to multiple tasks on CDC sink case with ensuring exactly-once?

Hello, everyone..!
Debezium MySQL Source connector should operate with one task at any one time.
Then, is it okay to deploy multiple connector tasks on this iceberg sink connector to consume CDC topic (produced by Debezium source connector)?
I guess, it is okay to deploy multiple sink tasks, thanks to only one coordinator task communicating with control topic, out of multiple tasks.
Is it okay to deploy multiple tasks to sink CDC topic with exactly-one semantics?

Yes it is fine to use multiple tasks.
Keep in mind the general kafka sink connector rule; do not use more tasks than partitions available on the CDC topic as you will just be wasting resources then (because each partition can be handled by at most one task).


All: Re-introduce selection dropdown, but place left of options cog

Could also be placed in navbar but in center?

Can you make this clearer? What is the selection dropdown?

We had it before, it contains the list of selected families
On Feb 13, 2015 6:47 AM, "Ankur Sethi"<EMAIL_ADDRESS>wrote:

Can you make this clearer? What is the selection dropdown?
—
Reply to this email directly or view it on GitHub
https://github.com/metapolator/ddt/issues/66#issuecomment-74242329.


feat(guide): Add new Git article How to Undo Things

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX
PR adds a new article for the Git section that summarizes all the ways to undo changes, from discarding working directory changes to undoing multiple commits. I thought having these in one place would be useful in case folks forget the exact command (git checkout or git reset?) to perform a specific action.

Haha, thanks @Manish-Giri, me too - they say write what you know, right? 🤣


does this work on visionos

esp since i got here while googling SpatialUI - unity's shabby prefabs for VisionOS UI

Spatial does not support VisionOS currently.

why not?


Remove version: SDCC.SDCC version 4.2.0

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

Close with reason: URL works;


Question: Is it possible to have different loggers with different transports?

Before I start, I just wanted to say great library.  I learned a bit more about using nest from looking at your code.
Anyway, from what I can tell, we can create several different loggers using @InjectLogger("someContext") but they all share the same configuration from single global logger.  What I need to be able to do is create different loggers that write different things to different files.
Specifically, I would like a separate request log for the API requests, authentication log, error log, and debug log.  The error log and debug log can share the same config since the only difference is log level but the authentication log and request log are entirely different things and I don't want information from there going into a debug log.
I assumed that I could do something like @InjectLogger(AUTH_LOG) and @InjectLogger(REQ_LOG) to create the logs, but don't quite see how to configure them.

Just to add to my previous comment, I'm looking at how what I'm requesting might be achieved (I'm completely new to winston so forgive me if I'm missing something obvious).
// nestjs-winston-logger.module.ts
export class NestjsWinstonLoggerModule {
  static forRoot(options: LoggerOptions): DynamicModule {

So perhaps what I need is some other kind of parameter in forRoot.
type LoggersConfig = {
  defaultOptions:LoggerOptions,

  // so if I have a separate logger that I need different settings for, the provider can look up the settings based on the context
  loggerSpecificOptions?:{
    [context:string]:LoggerOptions
  }

}

And use factory becomes something like
static forRoot(params:LoggersConfig): DynamicModule {

   // ...

    const loggerProviders: FactoryProvider<NestjsWinstonLoggerService>[] = contexts.map(
      (context) => {
        return {
          provide: getLoggerToken(context),
          useFactory: () => {
            let options = params.loggerSpecificOptions && params.loggerSpecificOptions[context] ? params.loggerSpecificOptions[context] : params.defaultOptions;
            const logger = new NestjsWinstonLoggerService(options);
            logger.setContext(context);
            return logger;
          },
        };
      },

I see you added the good first issue label.  I'm fine with picking this one up if you think the approach is ok.

I see you added the good first issue label. I'm fine with picking this one up if you think the approach is ok.

Feel free to go for it!

I created a pull request which you can check out and give me some CR feedback.
https://github.com/owen1002/nestjs-winston-logger/pull/8/commits/dd0a1d19c6d1f202e2dc99bfdc6b9fbc9e668f12#
With regards to your mongoose comment, I didn't fully understand it at first but after looking at it a bit more, I found this.
// from https://docs.nestjs.com/techniques/mongodb
@Module({
  imports: [
    MongooseModule.forRoot('mongodb://localhost/test', {
      connectionName: 'cats',
    }),
    MongooseModule.forRoot('mongodb://localhost/users', {
      connectionName: 'users',
    }),
  ],
})

I think you may be suggesting something where we allow multiple NestjsWinstonLoggerModule.forRoot(...) function but specify the injection token.
imports:[
  NestjsWinstonLoggerModule.forRoot(config, AUTH_TOKEN),
  NestjsWinstonLoggerModule.forRoot(config, SOME_OTHER_TOKEN),
]

@owen1002 - can you add your thoughts?  I don't mind continuing with the review.  It seems to be blocked at the moment though

Sorry for the late response, I'm trying to understand where we are in.
Just to confirm, this use case is to have multiple logger configs to be used in one module right?
What is blocking us from using the below setup?
@Module({   imports: [     NestjsWinstonLoggerModule.forRoot('LoggerA', loggerAConfig),     NestjsWinstonLoggerModule.forRoot('LoggerB', loggerBConfig),   ], })

Yes, I have to admit, I'm a bit foggy too.  It has been a while since I looked at it and I've worked on other things in the meantime while I was waiting.
Yes, you are right.  The feature I needed was ability to have different log files (and therefore logger instances) with different config. Skimming through the pull request, it looks like I didn't initially understand that I could have multiple forRoot imports for the module, but then once I did that, I ran into a bug relating to the context.
I think an an appropriate next step might be to just re-create the use case I was trying to do and see if I run into the context issue with the newest version of the library.  Not sure when exactly I'll get around to it but if you are available and interested in reviewing, I will try to do it in the next week or two.  If you can give me feedback as I go, I can stick with it until we can close it out.  In the end, it might just be an update to documentation to spell out the specific use case.


Some Questions

Just found this program and have some questions.
I primarily use tvheadend, with an iptv provider.
In tvheadend, the original m3u8 seems to be fine, no issues with display of the playlist.
Your m3udoctor.py, makes some minor changes, as below.
Referencing the original m3u8, how would these manifest as errors or issues?
`original
#EXTINF:-1 tvh-chnum="948" tvg-id="hallmarkchannel.us" tvg-name="HALLMARK CHANNEL" tvg-logo="https://tv-fanatic-res.cloudinary.com/iu/s--gBrQRyxY--/t_full/cs_srgb,    162 f_auto,fl_strip_profile.lossy,q_auto:420/v1434689610/hallmark-channel-logo.jpg" group-title="US-ENTERTAINMENT",HALLMARK CHANNEL
163 http://my.provider.zzz.as:port/me/password/1XXXX5623
m3udoctor.py
#EXTINF:-1 tvh-chnum="948" tvg-id="hallmarkchannel.us" tvg-name="HALLMARK CHANNEL" tvg-logo="https://tv-fanatic-res.cloudinary.com/iu/s--gBrQRyxY--/t_full/cs_srgb%    162 2Cf_auto%2Cfl_strip_profile.lossy%2Cq_auto:420/v1434689610/hallmark-channel-logo.jpg" group-title="US-ENTERTAINMENT",HALLMARK CHANNEL
163 http://my.provider.zzz.as:port/me/password/1XXXX4356
original
#EXTINF:-1 tvg-id="" tvg-name="Music Choice 00: 70s Audio Channel" tvg-logo="http://primestreamstv.com/logos/MC POP .png" group-title="MUSIC",Music Choice 00:       2 70s Audio Channel
3 http://my.provider.zzz.as:port/me/password/2XXXght
m3udoctor.py
#EXTINF:-1 tvg-id="" tvg-name="Music Choice 00: 70s Audio Channel" tvg-logo="http://primestreamstv.com/logos/MC POP .png" group-title="MUSIC",Music Choice 00:       2 70s Audio Channel
3 http://my.provider.zzz.as:port/me/password/2cxzsad
original
#EXTINF:-1 tvg-id="" tvg-name="PC: Golf PC: Open " tvg-logo="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/NBCUniversal_Peacock_Logo.svg/2560px-NBCUniversal_Peacock_Logo.svg.png" group-title="US-PEACOCK",PC: Golf PC: Open
http://my.provider.zzz.as:port/me/password/2thuygfd
m3udoctor.py
#EXTINF:-1 tvg-id="" tvg-name="PC: Golf PC: Open " tvg-logo="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/NBCUniversal_Peacock_Logo.svg/2560px-NBCUniversal_Peacock_Logo.svg.png" group-title="US-PEACOCK",PC: Golf PC: Open
http://my.provider.zzz.as:port/me/password/2opulip
original
#EXTINF:-1 tvg-id="ESPNPlay100.jm" tvg-name="(JM) ESPN PLAY 100:" tvg-logo="https://artwork.api.espn.com/artwork/collections/airings/b2ca1b6e-a870-4177-b782-0a12650c0b53/default?width=640&apikey=1ngjw23osgcis1i1vbj96lmfqs&timestamp" group-title="ESPN PLAY",(JM) ESPN PLAY 100:
m3udoctor.py
#EXTINF:-1 tvg-id="ESPNPlay100.jm" tvg-name="(JM) ESPN PLAY 100:" tvg-logo="https://artwork.api.espn.com/artwork/collections/airings/b2ca1b6e-a870-4177-b782-0a12650c0b53/default?width=640&amp%3Bapikey=1ngjw23osgcis1i1vbj96lmfqs&amp%3Btimestamp" group-title="ESPN PLAY",(JM) ESPN PLAY 100:
`
I generally have more issues with the epg data, the tv_grab_file has problems digesting the xml file, I have to socat to xmltv.sock, that seems to work. The xml never validates, to many errors.
Are you aware of some script, or program similar to m3udoctor.py, that checks and repairs xmltv ?
Thanks

Hi,
Thanks for reaching out and apologies for the late reply: I've been busy.
It looks to me that the only difference between the original entry and the
one produced by the m3udoctor module is that the value of the tvg-logo
fields has been url-encoded. I don't know how familiar you are with URLs
and HTTP requests in general, but - in a nutshell - the standard says that
certain characters, when present in a URL string, have to be encoded with a
%XY form. For example, some commas in your entries have been converted to
the %2C code.
how would these manifest as errors or issues?

The most common http clients take care of encoding these characters before
sending a request, but some of them don't and might crash.
Are you aware of some script, or program similar to m3udoctor.py, that

checks and repairs xmltv ?

I'm not, sorry. I still haven't looked into XTMLTV.
I hope this helps.
Cheers.
Beer4Ever83
Il giorno mar 28 nov 2023 alle ore 23:38 halfagascan <
@.***> ha scritto:

Just found this program and have some questions.
I primarily use tvheadend, with an iptv provider.
In tvheadend, the original m3u8 seems to be fine, no issues with display
of the playlist.
Your m3udoctor.py, makes some minor changes, as below.
Referencing the original m3u8, how would these manifest as errors or
issues?
`original
#EXTINF:-1 tvh-chnum="948" tvg-id="hallmarkchannel.us" tvg-name="HALLMARK
CHANNEL" tvg-logo="
https://tv-fanatic-res.cloudinary.com/iu/s--gBrQRyxY--/t_full/cs_srgb,
162
f_auto,fl_strip_profile.lossy,q_auto:420/v1434689610/hallmark-channel-logo.jpg"
group-title="US-ENTERTAINMENT",HALLMARK CHANNEL
163 http://my.provider.zzz.as:port/me/password/1XXXX5623
m3udoctor.py
#EXTINF:-1 tvh-chnum="948" tvg-id="hallmarkchannel.us" tvg-name="HALLMARK
CHANNEL" tvg-logo="
https://tv-fanatic-res.cloudinary.com/iu/s--gBrQRyxY--/t_full/cs_srgb%
162
2Cf_auto%2Cfl_strip_profile.lossy%2Cq_auto:420/v1434689610/hallmark-channel-logo.jpg"
group-title="US-ENTERTAINMENT",HALLMARK CHANNEL
163 http://my.provider.zzz.as:port/me/password/1XXXX4356
original
#EXTINF:-1 tvg-id="" tvg-name="Music Choice 00: 70s Audio Channel"
tvg-logo="http://primestreamstv.com/logos/MC POP .png"
group-title="MUSIC",Music Choice 00: 2 70s Audio Channel
3 http://my.provider.zzz.as:port/me/password/2XXXght
m3udoctor.py
#EXTINF:-1 tvg-id="" tvg-name="Music Choice 00: 70s Audio Channel"
tvg-logo="http://primestreamstv.com/logos/MC POP .png"
group-title="MUSIC",Music Choice 00: 2 70s Audio Channel
3 http://my.provider.zzz.as:port/me/password/2cxzsad
original
#EXTINF:-1 tvg-id="" tvg-name="PC: Golf PC: Open " tvg-logo="
https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/NBCUniversal_Peacock_Logo.svg/2560px-NBCUniversal_Peacock_Logo.svg.png"
group-title="US-PEACOCK",PC: Golf PC: Open
http://my.provider.zzz.as:port/me/password/2thuygfd
m3udoctor.py
#EXTINF:-1 tvg-id="" tvg-name="PC: Golf PC: Open " tvg-logo="
https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/NBCUniversal_Peacock_Logo.svg/2560px-NBCUniversal_Peacock_Logo.svg.png"
group-title="US-PEACOCK",PC: Golf PC: Open
http://my.provider.zzz.as:port/me/password/2opulip
original
#EXTINF:-1 tvg-id="ESPNPlay100.jm" tvg-name="(JM) ESPN PLAY 100:"
tvg-logo="
https://artwork.api.espn.com/artwork/collections/airings/b2ca1b6e-a870-4177-b782-0a12650c0b53/default?width=640&apikey=1ngjw23osgcis1i1vbj96lmfqs&timestamp"
group-title="ESPN PLAY",(JM) ESPN PLAY 100:
m3udoctor.py
#EXTINF:-1 tvg-id="ESPNPlay100.jm" tvg-name="(JM) ESPN PLAY 100:"
tvg-logo="
https://artwork.api.espn.com/artwork/collections/airings/b2ca1b6e-a870-4177-b782-0a12650c0b53/default?width=640&amp%3Bapikey=1ngjw23osgcis1i1vbj96lmfqs&amp%3Btimestamp"
group-title="ESPN PLAY",(JM) ESPN PLAY 100:
`
I generally have more issues with the epg data, the tv_grab_file has
problems digesting the xml file, I have to socat to xmltv.sock, that seems
to work. The xml never validates, to many errors.
Are you aware of some script, or program similar to m3udoctor.py, that
checks and repairs xmltv ?
Thanks
—
Reply to this email directly, view it on GitHub
https://github.com/Beer4Ever83/ipytv/issues/70, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF2HBBGQIHXOHVLODCO3CITYGZRWPAVCNFSM6AAAAAA76QCOC6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYTKNBUGE4TEMQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


LandingPage: Fix failing Webpack build

See similar issue https://github.com/kiwicom/margarita/pull/624

@RobinCsl looks like you have unhappy eslint

Looks like my ESLint plugin on VSCode was disabled... Bear with me.

Looks like my ESLint plugin on VSCode was disabled... Bear with me.

Never trust VSCode 😱


Invalid URI in ReleaseInfoViewModel

Hi, when I open any excel document, the add-in takes some time to load. Enabling the logging, I saw this (don't know if it is related though):
2018-11-23 16:14:54.3183|INFO|Bovender.Versioning.ReleaseInfo|WebClient_DownloadStringCompleted: Status: FailureToParse
2018-11-23 16:14:54.3406|WARN|Bovender.Versioning.ReleaseInfoViewModel|SendProcessFinishedMessage: Exception occurred!
2018-11-23 16:14:54.3798|WARN|Bovender.Versioning.ReleaseInfoViewModel|System.UriFormatException: Invalid URI: The URI scheme is not valid.
   at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
   at Bovender.Versioning.ReleaseInfo.Parse(String rawInfo)
2018-11-23 16:14:54.3798|INFO|Bovender.Mvvm.Messaging.Message`1|Sending message to 1 subscriber(s)
2018-11-23 16:14:54.3798|DEBUG|Bovender.Mvvm.Messaging.Message`1|Message content is a Bovender.Mvvm.Messaging.ProcessMessageContent
2018-11-23 16:14:54.3952|WARN|XLToolbox.Versioning.ReleaseInfoViewModel|Exception during update check
2018-11-23 16:14:54.3952|WARN|XLToolbox.Versioning.ReleaseInfoViewModel|System.UriFormatException: Invalid URI: The URI scheme is not valid.
   at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
   at Bovender.Versioning.ReleaseInfo.Parse(String rawInfo)
2018-11-23 16:14:54.3952|INFO|Bovender.Mvvm.ViewModels.ProcessViewModelBase|SendProcessFinishedMessage: Sending message
2018-11-23 16:14:54.3952|INFO|Bovender.Mvvm.Messaging.Message`1|Sending message to 1 subscriber(s)
2018-11-23 16:14:54.3952|DEBUG|Bovender.Mvvm.Messaging.Message`1|Message content is a Bovender.Mvvm.Messaging.ProcessMessageContent
2018-11-23 16:14:54.9359|INFO|Bovender.Mvvm.ViewModels.ProcessViewModelBase|UpdateProgress: No longer processing, disposing update timer

The error is easy to explain, but it is definitively not the cause of any delays. When checking for a new version, the add-in downloads a text file that contains the current version and download URL. I had recently published an update, but while it worked well on my development machines, when I went to my work place and installed it, it would make Excel crash. So I stopped distributing the update after a few hours by adding invalid characters to the version information file. The error message was introduced by myself to be able to have a way to detect mistakes in the version information file, except that in this case the mistake was deliberate. Neither the text file download (which happens asynchronously) nor the error message can cause any delays.
Now for the actual question at hand: Why does it delay your start-up? I guess you have a situation where you start Excel and you see the splash screen with the notice "Loading XLToolboxForExcel..." and it seems to take forever, is that correct? If you have not done so yet, I would like to suggest to disable (not uninstall) the add-in (e.g. via File > Options > Add-ins > COM add-ins > Go, then clear the relevant check mark; or by enabling the Developer ribbon tab via File > Options > Customize ribbon, then placing a check mark in front of "Developer" in the right-hand list, which will make a new tab appear on the ribbon that contains a "COM add-ins" button). Disabling and re-enabling the add-in could confirm that it is indeed the Toolbox that delays startup.
Then, have a look at the time stamps in the entire Toolbox log file. They may be informative, but they might also be misleading, e.g. when a longer time span in that log simply indicates that the Toolbox was idle. But it depends, I would have to examine it closely. If you want, you could send me the entire file (for example to daniel at xltoolbox dot net, or via Telegram https://t.me/bovender).


Update block-unmanaged-customizations.md

updating a new known limitation for activitymimeattachments

This has been re-created in the private repo via https://github.com/MicrosoftDocs/power-platform-pr/pull/8774.
Closing without merging.


[TypeSpec Requirement] Add suppression mechanism

Design for new suppression mechanism for all new checks in repo azure-rest-api-specs going forward:

Starting from the file/folder you are analyzing, walk up the directory tree until you find a file named suppressions.yml.

If none found, then nothing is suppressed.
The suppressions file closest to the file being analyzed wins.  There is no concept of merging suppressions from files in nested parent directories.

Simple YAML schema, something like:

- tool: TypeSpecRequirement
  path: data-plane/*.json
  reason: Brownfield service not ready to migrate

- tool: TypeSpecValidation
  code: FolderStructure
  reason: Must use non-standard folder structure to avoid breaking changes

Tools should print a warning message for any rules suppressed (don't silently ignore suppressed rules)

To add few details:

the tool determines the type of the entry. Different tools may have different keys.
we also consider jsonPath key for a JSON path inside the file pointed to by the path.
First tool candidates are TypeSpecRequirement, TypeSpecValidation as well as SameVersionBreakingChanges and CrossVersionBreakingChanges

Note to self -- once this is in, I've volunteered to write the process/policy draft (to save time) and will hand it off to ARM for them to wordsmith/post to their site.

Thanks for the questions, these help clarify the design.

There are exactly-duplicate entries. Do we silently ignore this and just read one entry (which one?)
There are almost-duplicate entries. Like exactly the same, but differing only e.g. by reason?
What if there are multiple entries per (tool, path, jsonPath)? E.g. two entries having the same tool, path and jsonPath, but different code ? I understand this is logical OR as in: both suppressions apply. This is also interesting when there is overlap in the sets of paths targeted by two rules. E.g. one has path of data-plane/**/*.json and the other one has data-plane/myService/**/*.json.
What if one rule makes another one redundant because targets a super-set of file paths, with the same suppressions?

Since we only allow suppressions, I think we can start really simple:

If tool Foo is about to raise error code Bar for path Baz, it searches the list (from the top of the file to the bottom), and uses the first matching suppression it finds.

If a suppression does not have a code, then all errors from the check are suppressed.

If an error is suppressed, the tool may either silently ignore the error, or print a warning with the reason from the suppression.  I think this may be case-by-case depending on the tool, and whether the warnings are helpful or just noise.
As an optimization, a tool may skip running the code for a rule entirely, if it knows any errors will be supressed anyway.  Again I think this is case-by-case depending on the tool.
When in doubt, err on the side of more logging, running rules even if you know they are suppressed, etc.  This might find bugs in our tools or other issues with the spec.

If we allowed re-enabling a code suppressed by an earlier entry, it gets more complicated.

How do we reconcile multiple parsers? One will be in specs repo written in PowerShell, and one in openapi-alps repo in TypeScript. Are we going to make an effort to somehow deduplicate them? My concern here is slight differences about minor corner cases, like how exceptions are handled, error messages emitted, etc.

If the implementation is as simple as I described above, we can start by just duplicating the code in each tool.  A YAML parser library should do most of the heavy lifting.

How do we protect against a case where someone submits a PR that immediately has suppressions.yml entry that silences any errors? We hope the PR reviewers will still catch it? In other tooling we have we explicitly add a label that says suppressions were added and PR reviewer explicitly must add a label like "suppressions approved".

I think this is up to the ARM and data-plane reviewers, and the answer may be different for each.  We could potentially re-use the existing Approved-Suppression label that applies to readme.md.
@konrad-jamrozik: Do you know how the existiing Approved-Suppression label works?  What checks for the presence of this label, and how does it determine if it's needed?  If any new suppressions were added to readme.md?

How do we handle malformed suppressions file? Assume no suppressions are present and produce some error? Output to where?

If the file cannot be parsed as valid YAML, fail fast with a clear error message
Most other "errors" I would ignore.  For example, if the tool, code, or path does not exist

You could argue we should "validate" the suppressions file to detect such errors, but I think this is going too far, at least to start.
If a suppression has say a typo and isn't working, it's should be simple for the spec author to figure out and fix.
If we "validate" the suppressions file, we'd either need to add another check just for this, or add the validation to every tool (which then requires sharing code or a lot of duplicate code).

@mikeharder big thanks for the reply! Awesome analysis and design.
Re labels for suppressions there have been some discussions how to improve this part of the process.
Notably, in #6622, in this comment @weshaggard and I proposed completely scrapping the suppressions label. There was a follow-up discussion within the GH issue what that would entail.
Additional context:

https://github.com/Azure/azure-sdk-tools/issues/7277
https://github.com/Azure/azure-sdk-tools/issues/6509
https://github.com/Azure/azure-sdk-tools/issues/6035
https://aka.ms/azsdk/pr-suppressions

From technical standpoint SuppressionReviewRequired is added when the code detects a change in one of the AutoRest config README suppressions section. Source: getSuppressions().
That label then blockes the PR until Approved-Suppression is added, per requiredLabelRules.ts.


Remove Text filenames with insufficient usage

Description

Removes test.me filename as it only had 228 files on GitHub, of which only 110 are relevant
Also removes a duplicate filename entry.
The other entries should be investigated for usage.
Filenames checked for usage:
search query e g. filename:read.me language:text extension:me

[x] = removed in this PR
[ ] :+1: duh COPYING
[ ] ❗ 66 COPYING.regex
[ ] :+1: 2.4K COPYRIGHT.regex
[ ] ❗ 90 FONTLOG
[ ] :+1: duh INSTALL
[ ] ❗ 22  INSTALL.mysql
[ ] :+1: duh LICENSE
[ ] ❗ 52 LICENSE.mysql
[ ] :+1: duh NEWS
[x] ➖ README.1ST
[ ] :+1: 2.5K README.me
[ ] ❓ 316 README.mysql
[ ] ❗ 22 README.nss
[ ] :+1: 3.6K click.me
[ ] :+1: 6.9K delete.me
[ ] ❓580 keep.me
[ ] :+1: 1.9K package.mask
[ ] :+1: 1.1K package.use.mask
[ ] ❗ 173 package.use.stable.mask
[ ] :+1: 7K read.me
[ ] :+1: 1.6K readme.1st
[x] ❗ 110 test.me
[ ] ❔ 830 use.mask - search throws a fit though
[ ] ❗ 162 use.stable.mask - ditto

Checklist:

[x] I am associating a language with a new file extension.

[x] I am adding new or changing current functionality

[ ] I have added or updated the tests for the new or changed functionality.

I automatically recognised README.nss and COPYRIGHT.regex as my handiwork; note that running
$ git blame -L 6033,6074 lib/linguist/languages.yml

may expedite review of the other filenames.

I recall seeing most of them are tagged as "6 years ago" while removing the go.mod filenames.
Here's the git blame:

Un-PR'd commit is https://github.com/github/linguist/commit/b032886c21d8397ca0be56fa91b5454e2f16d7c1

Went through all of them, ❗= not enough, ❓ = possibly not enough

Alright


Merge master into release

@gregg-miskelly, @chuckries, @caslan, @rajkumar42: I scanned through the changes and it looks right to me. Do any of you want to double-check the debugger changes?

@DustinCampbell Yeah, we were going to do a quick smoke test of the .vsix. Do you have one built already? (If not, we can build it)

We should build the final from the release branch. Once I merge, could you build it from there?

That works with me.

look good?

Yup. Merge it.


JSON Schema support

Writing a schema document by hand is error prone and not a pleasant experience. A macros is a good idea to simplify the process and provide helpful hints. Getting ideas from both JSON Schema and JTD I arrived to this DSL. While the first one is overengineered garbage, the other is overly simplistic and not fit for more dynamic languages.
Consider the following:
  {
    "node": {
      "feature": "petal_length",
      "threshold": 2.45,
      "left": {
        "class": "setosa"
      },
      "right": {
        "feature": "petal_width",
        "threshold": 1.75,
        "left": {
          "feature": "petal_length",
          "threshold": 4.95,
          "left": {
            "feature": "petal_width",
            "threshold": 1.65,
            "left": {
              "class": "versicolor"
            },
            "right": {
              "class": "virginica"
            }
          },
          "right": {
            "class": "virginica"
          }
        },
        "right": {
          "class": "virginica"
        }
      }
    }
  }

The DSL language is closer to the JDT spec, but instead of discriminator-mappings we have "oneOf", "anyOf", "allOf". instead of primitive number types taken from C++, we only kept "int", "float" and "number". The justification for this, is that for stricter verification the fromJson can be extented.
The schema for this specific example is defined as:
defineSchema IrisClassifierSchema:
  properties:
    node(`ref` = node)
  definitions:
    node:
      oneOf:
        properties:
          class(`enum` = ["setosa", "versicolor", "virginica"], nullable = true)
        properties:
          feature(`enum` = ["sepal_length", "sepal_width", "petal_length", "petal_width"])
          threshold(`type` = number)
          left(`ref` = node)
          right(`ref` = node)

First I considered producing intermediate rules JSON, which is parsed at compile-time to produce the checks. However it makes no-sense when nim macros exist. And the benefit of compatibility with a standard is lost.
So it produces this code:
type
  IrisClassifierSchema = object

proc errorMsg(msg: string, path: string): string =
  result = msg & " at path: " & path

template adderr(msg: string, path: string) =
  errors.add(errorMsg(msg, path))
  result = false

proc isValidNode(tree: JsonTree; n: NodePos; path: var string; errors: var seq[string]): bool

proc isValidNode1(tree: JsonTree; n: NodePos; path: var string; errors: var seq[string]): bool =
  type
    NodeKey = enum
      class
  result = true
  if n.kind == opcodeObject:
    var nodeKeys: set[NodeKey] = {}
    let oldLen = path.len
    for x in keys(tree, n):
      path.add '/'
      path.add x.str
      case x.str
      of "class":
        if NodeKey.class in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.class
        let classVal = x.firstSon
        if classVal.kind != opcodeString or
            classVal.str notin ["setosa", "versicolor", "virginica"]:
          adderr("invalid class value", path)
      else:
        adderr("invalid object field", path)
      path.setLen(oldLen)
    for e in low(NodeKey)..high(NodeKey):
      if e notin nodeKeys:
        adderr("key not found", path & '/' & $e)
  else:
    adderr("expected object", path)

proc isValidNode2(tree: JsonTree; n: NodePos; path: var string; errors: var seq[string]): bool =
  type
    NodeKey = enum
      feature, threshold, left, right
  result = true
  if n.kind == opcodeObject:
    var nodeKeys: set[NodeKey] = {}
    let oldLen = path.len
    for x in keys(tree, n):
      path.add '/'
      path.add x.str
      case x.str
      of "feature":
        if NodeKey.feature in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.feature
        let featureVal = x.firstSon
        if featureVal.kind != opcodeString or
            featureVal.str notin ["sepal_length", "sepal_width", "petal_length", "petal_width"]:
          adderr("invalid feature value", path)
      of "threshold":
        if NodeKey.threshold in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.threshold
        let thresholdVal = x.firstSon
        if thresholdVal.kind != opcodeFloat:
          adderr("invalid threshold value", path)
      of "left":
        if NodeKey.left in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.left
        result = result and isValidNode(tree, x.firstSon, path, errors)
      of "right":
        if NodeKey.right in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.right
        result = result and isValidNode(tree, x.firstSon, path, errors)
      else:
        adderr("invalid object field", path)
      path.setLen(oldLen)
    for e in low(NodeKey)..high(NodeKey):
      if e notin nodeKeys:
        adderr("key not found", path & '/' & $e)
  else:
    adderr("expected object", path)

proc isValidNode(tree: JsonTree; n: NodePos; path: var string; errors: var seq[string]): bool =
  let oldErrLen = errors.len
  result = false
  if result xor isValidNode1(tree, n, path, errors):
    result = true
    errors.setLen(oldErrLen)
  if result xor isValidNode2(tree, n, path, errors):
    result = true
    errors.setLen(oldErrLen)
  if not result:
    adderr("oneof failed to match", path)

proc isValidRoot(tree: JsonTree; n: NodePos; path: var string; errors: var seq[string]): bool =
  type
    NodeKey = enum
      node
  result = true
  if n.kind == opcodeObject:
    let oldLen = path.len
    var nodeKeys: set[NodeKey] = {}
    for x in keys(tree, n):
      path.add '/'
      path.add x.str
      case x.str
      of "node":
        if NodeKey.node in nodeKeys:
          adderr("duplicate key", path)
        nodeKeys.incl NodeKey.node
        result = result and isValidNode(tree, x.firstSon, path, errors)
      else:
        adderr("invalid root field", path)
      path.setLen(oldLen)
    for e in low(NodeKey)..high(NodeKey):
      if e notin nodeKeys:
        adderr("key not found", path & '/' & $e)
  else:
    adderr("expected object", path)

proc isValid(tree: JsonTree; t: typedesc[IrisClassifierSchema]; errors: var seq[string]): bool =
  var path = ""
  result = isValidRoot(tree, rootNodeId, path, errors)

var errors: seq[string] = @[]
echo isValid(classifier, IrisClassifierSchema, errors)
if errors.len > 0:
  echo "Validation errors:"
  for err in errors:
    echo err
else:
  echo "Validation successful"

Besides the jsonpak/schema module which would contain the macro, the functions in module builder should be refactored to provide more strict validation akin to JDT. So numbers have range checks, object variants are implemented and opt-in path-based error reporting.

According to https://datatracker.ietf.org/doc/html/rfc8927#name-allowing-additional-propert

Note that "additionalProperties" does not get "inherited" by subschemas.

And:

Note from Figure 1 that only schemas of the "properties" form may have a member named "additionalProperties".

Very easy to enforce:
properties(allowExtra = true):
  a(string)

Formal rules for constructing a schema:
Top-level Schema Structure

Schema Definition:

A schema is defined using the defineSchema macro followed by a schema name.
The schema name should be a valid Nim identifier.

Definitions:

The definitions keyword is used to define reusable subschemas.
definitions are allowed only at the top level of a schema.
Each definition is identified by a unique name.
Definitions can be referenced elsewhere in the schema using the ref attribute.

Properties

Properties:

The properties keyword is used to define the properties of an object.
Properties are required by default.
The allowExtra = true attribute can be used with properties to allow additional properties.
Each property is defined using a key-value pair, where the key is the property name and the value is the property definition.

Property Definition

Property Definition:

A property definition can be a simple type definition, a reference or a nested schema.
Simple type definitions include int, float, number, string, and bool.
A definition may include attributes such as minimum, maxItems, format, and others specific to the property type.
A property can be marked as nullable using the nullable = true attribute.
A property can be marked as optional using the optional = true attribute.

Enum:

The enum keyword is used to specify a list of allowed values.
Currently, enum is specified to work only with strings.

References:

The ref keyword is used to reference a subschema defined in definitions.
References can only point to subschemas declared within the same schema's definitions.

Nested Structures, Arrays and Tuples

Nested Properties:

Properties can be nested, allowing the definition of complex objects.

Arrays:

Arrays are defined using the keyword elements.
The elements keyword is followed by the type, reference, or schema definition for the array elements.

Tuples:

TBD

Schema Composition

Schema Composition:

The DSL includes a few keywords for combining schemas together.
The keywords used to combine schemas are:

allOf: (AND) Must be valid against all of the subschemas
anyOf: (OR) Must be valid against any of the subschemas
oneOf: (XOR) Must be valid against exactly one of the subschemas
not: (NOT) Must not be valid against the given schema

Special Validation Features

Number Validation:

Numbers support the following attributes for validation:

minimum: Specifies the minimum value allowed for the number.
maximum: Specifies the maximum value allowed for the number.
exclusiveMinimum: If set to true, the number must be strictly greater than the specified minimum.
exclusiveMaximum: If set to true, the number must be strictly less than the specified maximum.

String Validation:

Strings support the following attributes for validation:

format: Specifies a format for the string value. Supported formats include "email", "date", "time", "date-time", "uri", "hostname", "ipv4", "ipv6", etc.
minLength: Specifies the minimum length allowed for the string.
maxLength: Specifies the maximum length allowed for the string.
pattern: Specifies a regular expression pattern that the string must match.

**Array Validation:

Arrays support the following attributes for validation:

minItems: Specifies the minimum number of items required in the array.
maxItems: Specifies the maximum number of items allowed in the array.
uniqueItems: If set to true, the array must contain only unique items.

Example Schema Definition
defineSchema ExampleSchema:
  properties:
    id(`type` = int)
    description(`type` = string, nullable = true)
    metadata(`ref` = metaData, optional = true)
  definitions:
    metaData:
      properties:
        created(`type` = string, format = "date-time")
        createdBy(`type` = string)

This is misguided. Generating code directly can be beneficial if you just need to log errors. But to be able to repair malformed JSON you need to have the schema in memory. A helpful macro should assist in writing correct schemas, prevent recursion, etc. Maybe even a switch that allows to support multiple standards from one definition.


Ci workflow

Summary
Summary about this PR

clean unused workflows
use dtolnay/rust-toolchain instead of actions-rs/toolchain, cause it's unmaintained
Using relative path to reference the action in the current branch for rust-setup

Closes #1864

close, completed through other prs.


nixos/pam: ability to calls an external command

I would like to receive an email notification for every successful login in a server by taking advantage of the pam_exec module.
An example setup is given on the webpage https://medium.com/@alessandrocuda/ssh-login-alerts-with-sendmail-and-pam-3ef53aca1381.
I managed to implement the setup on NixOS and I modified nixos/pam in order to do that.
I do not fully understanding the implications of the nixos/pam module. Also I would be glad to have feedback about the pull request that I will reference here in a few minutes.

Here is how I use this:
{ config, lib, pkgs, ... }:
let
  notify = with pkgs; writeScriptBin "login-email-notification" ''
    #! ${runtimeShell} -e

<EMAIL_ADDRESS><EMAIL_ADDRESS>
    SUBJECT="SSH Login ''${PAM_USER}@''$(hostname).''$(domainname)"

    MESSAGE="
    A user signed into your server through SSH.

    User: ''${PAM_USER}
    Host: ''$(hostname).''$(domainname)

    From IP address: ''${PAM_RHOST}

    if [ ''${PAM_TYPE} = "open_session" ]; then
      echo "''${MESSAGE}" | ${pkgs.mailutils}/bin/mail -n -a From:"''${EMAIL_FROM}" -s "''${SUBJECT}" "''${EMAIL_TO}"
    fi

    exit 0

  '';
in
{
  security.pam.services.sshd.externalCommand = {
    type = "session";
    control = "required";
    command = "${notify}/bin/login-email-notification";
  };
}

@Mic92: any feedback?

I hope that https://github.com/NixOS/nixpkgs/pull/105319 gets ready soon. Than this module becomes easier to integrate.

Hello,
I know this issue is stale for quite some time now, but I'm curious if there is any plan to support this use case in the long term? I'd be interested in a similar use case, but I don't like the idea of modifying nixos/pam myself :)

@Lqp1 Now it's possible to add pam_exec in any place you want, something like that:
    security.pam.services.login.rules.session.env-pre-systemd = {
      order = config.security.pam.services.login.rules.session.systemd.order - 1;
      enable = true;
      control = "optional";
      modulePath = "${config.security.pam.package}/lib/security/pam_exec.so";
      args = [
        "debug" "log=/var/run/pe-001.log" "${envDumper}"
      ];
    };

Indeed, it works like a charm! Thanks a lot for the pointer @binarin !


Can I register interceptor to Jongo queries?

Hello
Can I register interceptor to all queries to log slow ones / do other useful job?

Not yet.
BTW this has been already discussed here https://github.com/bguerout/jongo/issues/27 but never implemented

I think it should be some kind of interceptor - (e.g. https://jersey.java.net/documentation/latest/user-guide.html#d0e9745 )


Please add how to re-open a PR after auto-closer closes it

The auto-closer came by and closed my PR. I want to reopen it. I've tried #please-open and a bunch of other stuff and nothing seems to work. Our Azure Monitor docs team is baffled too. Can you help? Here's an example PR.
https://github.com/MicrosoftDocs/azure-docs-pr/pull/178091

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2c68d84b-6a64-722b-bf11-78d6e1009c4b
Version Independent ID: 3b4e7a64-72d4-b6b0-2c23-bb2b0017becf
Content: GitHub contribution workflow for major or long-running changes - Contributor Guide
Content Source: Contribute/how-to-write-workflows-major.md
Product: non-product-specific
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Hi @mattmccleary - You've reached us at our external-facing contributor guide. I'll reach out to you internally to point you to the article that you're looking for. Thanks!


Update session object-level access control logic and add SessionService

This commit makes it so that a user must pass one of three checks in
order to operate on a session. The checks are as follows:

The user is the "session user", meaning the user's id matches the
value of the userId property on the session instance.
The user has a permission of the form "session.."
where operation is "create", "read", "update", or "delete". This
permission allows them to operate on all sessions where the user
in the session has a SPID that matches the SPID in the permission.
The user has a permission of the form "session.."
where operation is the same as above and id is the id of a session,
which is a GUID.

The first rule allows users to modify their own session in so far as the
property-level access control checks allow them to. The second rule
allows "weak" admins - which are administrators of service providers - to
manage the sessions of users under their service. The third check allows
"super" admins to be able to do anything. Note that the second check
won't always pass for super admins because it will fail if the userId of
the session is not set.

I would rather not see sessionDAO made available to users. I would prefer an authToken service with specific methods for desired use-cases, like:
SessionId createAuthToken(userid, ttl);

@jlhughes @kgrgreer Updated. Now there's a SessionService interface with methods to create and destroy sessions. It includes an access control check that only lets one create or destroy sessions for users in a SPID if they have the session.create.<spid> or session.delete.<spid> permissions, respectively.
sessionDAO is still served but the authorization methods on Session were changed to only check for session.create.*, session.read.*, session.update.*, and session.delete.* permissions, which only admin will have.

Example requests:

Create a session

http://localhost:8080/service/sugar?service=sessionService&method=createSession&interfaceName=foam.nanos.session.SessionService&userId=1348

Create a session with a custom TTL

http://localhost:8080/service/sugar?service=sessionService&method=createSessionWithTTL&interfaceName=foam.nanos.session.SessionService&userId=1348&ttl=9223372036854775807

Destroy a session

http://localhost:8080/service/sugar?service=sessionService&method=destroySession&interfaceName=foam.nanos.session.SessionService&accessToken=f8bb5d03-79c4-f594-e234-5dde0e8d1847


Need 3.6.2 support for Jadeite

Probably should make Oscar3.1.1pre-release2

If it wasn't too difficult, would it make sense to replace the version dropdown with a version field that can be filled in by users ... then it might not be required to  make a jadeite release to coincide with every gemstone release  ... i suppose this probably would make sense to go along with clientlibs support, since the user would download a new clientlib (or internal dev create new directory in clientlibs and copy libs there manually) and not need to rely on a jadeite release to provide the new libs ... just thinking out loud :)

https://github.com/GemTalk/Jadeite/releases/tag/Oscar-3.1.1-pre-release2


DNS challenge cannot be validated

Hey there,
thanks a lot for this super useful caddy plugin!
I was trying to use this but realised that caddy gets stuck verifying the DNS challenge, even though a TXT record is created by caddy in my Hetzner DNS console.
Inspecting the DNS record more closely, I see that my domain appears twice in the TXT record (something like _acme.challenge.test.example.com.example.com)
Could this be the source of the problem?
Best,
Leander

Hi le-ander,
your issue is caused by libdns/hetzner#1, which depends on another issue.
The problem is that according to this comment, caddy uses full domain name sans trailing dot. Hetzner on the other side, always adds the domain to TXT record name.
Implementing a workaround in libdns/hetzner sould be easy. I will give it a try next days.

Thanks a lot for the update, I think I understand the problem now.
Very much looking forward to your workaround solution.
Thank you very much for your efforts. Highly appreciated!

Fix/workaround has been implemented. I will close the issue. Feel free to reopen it or create a new one , if you still encounter problems.

Awesome, thanks so much! Will give this a try.


Repartition ephemeral disk when OS reload

Hi,
This PR focused on repartition ephemeral disk and was originated from https://github.com/cloudfoundry/bosh-agent/issues/184

When OS reload, agent will repartition the ephemeral disk and set ephemeral_disk_partitioned to true.

When OS reboot or agent restart, agent will check ephemeral_disk_partitioned field and skip repartitioning if it is true.

/cc @mattcui @dpb587  Please help to review

Reopen after implementing  EphemeralDevicePartitioner: https://github.com/cloudfoundry/bosh-agent/issues/184#issuecomment-416772112

Reopen a PR: https://github.com/cloudfoundry/bosh-agent/pull/190


David's take on Serializer

Remove serialize_fixed_size_array
Pure interface (https://github.com/serde-rs/serde/issues/219)
Remove 'static requirement for string names of things
Turn struct_elt and struct_variant_elt keys back to &str
Distinguish all the different state types
Pass &mut state to *_elt()
Pass only the state to *_end()

I'm gonna take a look at it later.
Fixed size array means bincode doesn't need to serialize the length.
Now we don't have any defaults anymore, this requires much more work on the serializer impl side. I'm fine with it, but are users?
Static makes sense for struct field namesand such, not sure why they shouldn't be static. Types should help us catchbugs, lifetimes can help, too

Bincode does not use fixed-size currently (nobody does). I am fine with keeping the method if they plan to use it.
There are only a handful of Serializers, and the defaults are actively harmful to most of them. See https://github.com/serde-rs/serde/issues/219.

thanks!


Change constructor to roundup

Round up fee-rate so that resulting fee calculations will not fall short.
proposed fix for https://github.com/rust-bitcoin/rust-bitcoin/issues/3806

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

4 of 4 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.002%) to 82.803%

Totals

Change from base Build<PHONE_NUMBER>3:
0.002%

Covered Lines:
20420

Relevant Lines:
24661

💛  - Coveralls


sqlalchemy update in python 3.5

I have model User
# __init__.py
from flask_sqlalchemy import SQLAlchemy

from server import app

db = SQLAlchemy(app)

from server.db.models import User

db.create_all()
# models.py
class User(db.Model):
    __tablename__ = 'users'
    mail = db.Column(db.String, primary_key=True)
    img = db.Column(db.String)

    def __init__(self, mail):
        self.mail = mail
        self.img = hashlib.md5(mail.encode()).hexdigest()

    def __repr__(self):
        return '<User %r>' % self.mail

When i try to update on row, i get error
>>> import server.db as h
>>> s=h.models.User.query.filter_by(mail='i@negash.ru').first()
>>> s.img='965fe6f7e5727271c4c67fc43f325288_NEW_DATA'
>>> h.db.session.commit()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/scoping.py", line 157, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 874, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 461, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2137, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2257, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2221, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 172, in save_obj
    mapper, table, update)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 732, in _emit_update_statements
    execute(statement, multiparams)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1015, in _execute_clauseelement
    fn(self, elem, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/crate/client/sqlalchemy/compiler.py", line 88, in crate_before_execute
    return rewrite_update(clauseelement, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/crate/client/sqlalchemy/compiler.py", line 64, in rewrite_update
    for key, val in _params.items():
AttributeError: 'list' object has no attribute 'items'

use docker image
FROM python:3-alpine
RUN pip install flask
RUN pip install flask-sqlalchemy
RUN pip install crate

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/39272880-sqlalchemy-update-in-python-3-5?utm_campaign=plugin&utm_content=tracker%2F505660&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F505660&utm_medium=issues&utm_source=github).

thanks for reporting, we'll take a look at this

@Negashev issue has been fixed and applied in 0.16.5 which got already released. Have fun and thanks for reporting. 👍


High I/O Wait times since upgrading to 2.1.1

Since upgrading to 2.1.1, began to see unusually large iowait percentages. Not since the initial migration to v2 have we seen such utilization:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.03    0.00    6.49   42.83    0.10   41.54

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda            1525.90    101767.59       257.90<PHONE_NUMBER>9   27114938

In our case the utilization is coming from this process:
postgres  14xx  3.1  2.7<PHONE_NUMBER>56 ?       Ds   Sep01  55:10  \_ postgres: 12/main: algorand pgdb <IP_ADDRESS>(49538) SELECT

And we are seeing high import times, too:
algorand-indexer[14yy]: will start from round >8842225
algorand-indexer[14yy]: accounting updated through round 8842226
algorand-indexer[14yy]: round r=8842226 (32 txn) imported in 609.294487ms
algorand-indexer[14yy]: will start from round >8842226

There is no one running queries on the DB through the API most of the time so we assume all that activity is due to synchronizing with algod.
If there is something not working correctly in our configuration, where would one look in order to root cause it?

Something we cannot yet explain just happened. For the last hour, after close to 50 hours of the unusual behavior, it has gone back to a normal-looking state:
From iostat:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.61    0.00    9.90   14.32    0.26   53.91

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              57.00       272.00       616.00        272        616

And notably from ps, this process is no longer permanently on SELECT/D state but in idle/S state:
postgres  14xx  3.0  2.7<PHONE_NUMBER>96 ?       Ss   Sep01  56:06  \_ postgres: 12/main: algorand pgdb <IP_ADDRESS>(49538) idle

Was there a database schema change or scrub in the update?

This is most likely due to a database migration, we need to figure out a way to make these processes more visible


Revert "Add password field support for default value"

Reverts Laravel-Backpack/CRUD#4783

A new inspection was created.

The inspection completed: No new issues


Could not copy the file BackgroundWorker.js because it was not found.

Hello,
I changed my background page to bg.js in manifest and everything was working fine for many weeks but all of a sudden I am getting errors,
Could not copy the file BackgroundWorker.js because it was not found.
It take me to,
<Copy SourceFiles="@(_BrowserExtension_Project_BuildOutput_BrowserExtension_FilesToCopy)"
      DestinationFolder="$(_BrowserExtension_Project_BuildOutput_BrowserExtension_Directory)\%(_BrowserExtension_Project_BuildOutput_BrowserExtension_FilesToCopy.CopyToRelativeDirectory)"
      SkipUnchangedFiles="true" />

It is searching BackgroundWorker.js but I already changed that name to bg.js in the manifest file.
Can you please let me know how to solve this? All my projects are showing the same error message.
Thanks

This is caused by the new feature for background worker generation from C# to JS in v1.5.0. A workaround for this is to change the PackageReference version in your project to v1.4.2 until a fix is issued.

This has been resolved in v1.5.1.