Netlink support for FreeBSD 13.2

Closes #3194, to the point that netlink-sys builds.  This is not yet sufficient for full rust-netlink support though, as other crates like netlink-packet-route are not yet able to build on FreeBSD.

The freebsd14 warning about NETLINK_GENERIC not being used is likely about its shadowing. Should we add a cfg restriction to the one in freensd/mod.rs? I'm not even sure it has any usefulness, and is not listed in the semver list for freebsd, so maybe we should just nuke it instead?

The freebsd14 warning about NETLINK_GENERIC not being used is likely about its shadowing. Should we add a cfg restriction to the one in freensd/mod.rs? I'm not even sure it has any usefulness, and is not listed in the semver list for freebsd, so maybe we should just nuke it instead?

Because it's already defined:
https://github.com/rust-lang/libc/blob/0b758afcdc34c9a72508a481ba151857d0b469de/src/unix/bsd/freebsdlike/freebsd/mod.rs#L3342

I see it's already defined.  The problem is, FreeBSD has 2 defines with same name and different values, each for a different API and defined in their own header file.  The separate header files allow this to work for C by creating a poor man's namespacing, but how are we to handle that in the Rust case?
Some options I can see include:

change the definition of the other FreeBSD API providing the clashing define (breaking API for them, no clue how widespread its use is in Rust or even if it is used at all - can we know that?)
use a different name for the FreeBSD Netlink API's const: that make the FreeBSD Rust Netlink API incompatible with that of Linux-like OS, which does not look clean and has to be handled specially in crates building on this to provide the rustified API (or really on any crate using the libc Netlink API - that raises some documentation issues to make sure this shape-changing API would be properly used)
change the definition of the Netlink API in libc crate, possibly with a compatibility layer on linux-like for transition (which would make the Netlink API more special than the rest of libc)

What do you think?

Regarding NETLINK_GENERIC, ask @GuillaumeGomez .  He's the one who added the definition from if_mib.h .  Also, grep.app shows that his sysinfo crate is the only one that's using it as he intended.  There are many more that try to use your version.  Including https://github.com/bytecodealliance/rustix/blob/db143034d80dbe2dc62368b732378c89f16e7569/src/net/types.rs#L800 , which seems to be confused by the definition in libc.  One version or the other will have to change.  I suspect that the least overall pain to the community would come by removing or renaming the definition from if_mib.h
Also, you should move your definitions out of the freebsd13 and freebsd14 modules and into the freebsd/mod.rs file.  The former two should only be used for stuff that changes between versions.  Symbols that newly appear in a newer version of FreeBSD can safely be defined by libc in for all OS versions.  For example, ino_t changed between versions, so it must be defined in freebsd11, freebsd12, etc.

FreeBSD and its handling of APIs is a nightmare. They don't care about compatibility, they change types/constants instead of creating new ones. So if NETLINK_GENERIC exists in two different places with different values, to prevent breaking libc's API, here's what I recommend: adding docs to the current NETLINK_GENERIC telling where to look for which values and then create both constants in two submodules (which maybe mention each others?).

What do you mean @GuillaumeGomez ?  This problem is caused by if anything too much backwards-compatibility, not too little.  The NETLINK_GENERIC mib that you're using was added in 1996, and it still works.  The NETLINK_GENERIC that @ydirson wants, by contrast, is based on an RFC written in 2003.  By then it was too late to change the name of the old constant, but changing the name of the new constant would break programs' cross-platform compatibility.  Hence isolating them in separate headers.
Is there any precedent for adding submodules to libc?

I mean that if they wanted to change the value of NETLINK_GENERIC, they should have created a new constant (called NETLINK_GENERIC2 for example) instead of just changing the value. They often break API, the worst being in structs where they change the fields order or even remove/add some.

Is there any precedent for adding submodules to libc?

No clue but if there is a good enough reason, I think it could be ok (up to them though!).

I mean that if they wanted to change the value of NETLINK_GENERIC, they should have created a new constant (called NETLINK_GENERIC2 for example) instead of just changing the value. They often break API, the worst being in structs where they change the fields order or even remove/add some.

Are you talking about stuff like struct stat?  That, I think, is really cool.  FreeBSD uses ELF symbol versioning to change structs like that.  That means zero problems for C code; old code will still compile, and old binaries will still run.  And yet new code can take advantage of new features without introducing new syscalls like Linux's stat64.  The only problems come when using FFI, like Rust does.  Rust basically chose to be easily cross-compilable at the expense of not being able to take advantage of ELF symbol versions.

Is there any precedent for adding submodules to libc?

No clue but if there is a good enough reason, I think it could be ok (up to them though!).

I don't see why it couldn't be done.  I think it's preferable to move the NETLINK_GENERIC mib into a submodule, because unlink the NETLINK_GENERIC protocol family it is platform-specific.  If we do that, what do you think the submodule should be named, and what other symbols should go with it?

Are you talking about stuff like struct stat? That, I think, is really cool. FreeBSD uses ELF symbol versioning to change structs like that. That means zero problems for C code; old code will still compile, and old binaries will still run. And yet new code can take advantage of new features without introducing new syscalls like Linux's stat64. The only problems come when using FFI, like Rust does. Rust basically chose to be easily cross-compilable at the expense of not being able to take advantage of ELF symbol versions.

I don't know enough about this to comment. From my understanding, having multiple types depending on which minimum version you support is much better and simpler. Anyway, there is no point debating over whether or not FreeBSD took the right decision here.

Also, you should move your definitions out of the freebsd13 and freebsd14 modules and into the freebsd/mod.rs file.

Pushed this, but:

it won't compile anywhere before the name clash gets solved, so I stacked a temporary commit to comment the if_mib definition out to keep the branch usable
libc-test now reports problems on freebsd12, which was the reason why I had moved the code in version-specific modules

* libc-test now reports problems on `freebsd12`, which was the reason why I had moved the code in version-specific modules

You will have to add an exception in libc-test.  See copy_file_range for an example.

Starting to introduce namespaces in libc will have an impact on those generated tests, we'll likely need someone mastering this part?

Now rebased on top of #3367 to get rid of the previous hack around NETLINK_GENERIC

:umbrella: The latest upstream changes (presumably #3341) made this pull request unmergeable. Please resolve the merge conflicts.

Rebased onto 0.2.151

Since 0.2 was branched out I cannot really rebase this branch any more to keep it usable (rustix just pushed a new version demanding libc 0.2.152), so I pushed the new rebased version to https://github.com/xcp-ng/rust-libc/tree/freebsd-netlink-0.2

This last push removes the pcap/nflog part, which is indeed not directly related, and not necessary for our use case.
That removed part is at https://github.com/xcp-ng/rust-libc/tree/freebsd-pcap-nflog in case it is useful for a new PR.

... and rebased to avoid failure of obsolete freebsd12 build job

:umbrella: The latest upstream changes (presumably #3587) made this pull request unmergeable. Please resolve the merge conflicts.

@tgross35 What does this S-blocked tag imply?  Anything that could be added to CONTRIBUTING.md?

It just means that something needs to happen before this can move forward. In this case I believe that is https://github.com/rust-lang/libc/pull/3367.

Makes sense, and now I realize that #3367 is itself blocked, and indeed needs a rebase (and maybe it should be added to the 1.0 milestone too) - but I only saw it by chance because of your post in #3248, received no notification of the tag change, and @bors did not report the need to rebase either :shrug:

No need to rebase just yet, I need to read into things better and figure out what series of events needs to happen. I'll post an update when I figure that out.


bump kimchima to v0.4.8

update kimchima to v0.4.8

Close this PR,  new kimchima version will be applied


[FEATURE] Pinecone vector size per upsert

Currently there is no limit for vectors per one upsert.
For Pinecone it's:
Max vector dimensionality is 20,000.
Max size for an upsert request is 2MB. Recommended upsert limit is 100 vectors per request.

@robertmaj90 could you please clarify? What are you proposing? Thanks

Hey @langchain4j!
It seems like when you upserting big pdf file, they're a limitation for request size. I would propose to divide it for more requests - send chunk of embeddings one by one.

https://docs.pinecone.io/legacy/limits


[W4.4b][W14-A2]Ang Yee Chin

@Ang-YC
Some comments added. Please ack & close the PR after reading comments.

Noted with thanks =)


ContractInfo query: return NoSuchContract system error instead of sdk error

Closes #687.

Do we also map sdkerrors.ErrInvalidAddress in https://github.com/CosmWasm/wasmd/blob/9a17505cf8b637942cad78e294aa1377d7f7b2f6/x/wasm/keeper/query_plugins.go#L486
to a system error? For that I would need to create a proper system error in wasmvm.

And, what about errors returned from the other queriers? (Bank, IBC, Staking, and Stargate)

What's the progress here? Is it blocked on something?
I think this is a good solution, and while we can do a larger overhaul, this fixes a clear issue with contracts now.

I plan to improve on this after holidays. I agree this is good enough for now. We can merge if you / @alpe agree.

I was waiting for the other PR but 👍  to have this first.


Add JWT Authentication

Add JWT Authentication to secure API endpoints.
@see https://api-platform.com/docs/core/jwt/

Done.


Add support for -i pipe:0

Hello, I need to transcode live videos,
essentially reading from an InputStream and pipeing the output into an OutputStream is what I need to do.
Unfortunately writing the file to disk is not possible for my use case.
I tried using 2 unix sockets which did work for some formats, but some input formats did not work that way. notably the MP4 demuxer
didnt like demuxing from a unix socket. It does however work with demuxing from stdin.
Sadly your CLI wrapper has no support for demuxing from stdin, so I have to rely on ProcessBuilder directly for this one task.
I wouldnt mind developing this as a feature and making a PR to this repo if you are intrested in this.
My idea would be to add a function to FFmpegBuilder that accepts a supplier to a InputStream.
A bit like this:
FFmpegBuilder builder = ...
Supplier<InputStream> streamSupplier = ...
builder.setStdIn(streamSupplier)

At some appropriate location in the library I would add this code or something similar:
(This may possible require starting of a "background" thread and I will maybe have to make adjustments for java 8 compat.)
Supplier<InputStream> streamSupplier = ...
Process proc = ...
if (streamSupplier != null) {
try(OutputStream outputStream = proc.getOutputStream();
    InputStream stream = streamSupplier.get()) {
    stream.transferTo(outputStream);
}
}

After these changes someone could then call
builder.addInput("pipe:0") to pipe the input from the InputStream.
For my usecase the output would still be a unix socket since my output muxer can write to the unix socket
and the input would be from the InputStream.
Tell me if you have any intrest in this feature and I will begin making it and give you a PR shortly.

Thank you for the recommendation! This seems to be somewhat related to #318. At least the implementation should get way easier.
Would a tcp/udp/unix socket also be acceptable?
As there is only one stdin for ffmpeg, using pipe would limit us to one input of this kind, while using a tcp socket would allow more flexibility.
Example:
new FFmpegBuilder().addTcpInput(myInput1).done().addTcpInput(myInput2).done()...

Could produce something like:
-i tcp://<IP_ADDRESS>:2000?listen -i tcp://<IP_ADDRESS>:2001?listen

Unfortunately the mp4 demuxer of current head/master ffmpeg does not demux some mp4's from a unix stream or tcp stream. It is however happy to decode it from pipe:0. I have tried both from unix and tcp stream. Didnt work only pipe:0 worked.
Also another problem I have is that I cannot really use tcp. Customer will ask why I need this port and whats sent over it. They will complain that its not encrypted... (even if I only bind localhost)
This leaves me with unix which doesnt work and pipe:0 that only works using process builder.

Interesting. Are you aware that you won't be able to use the progress API, as it also opens a temporary TCP port?
If we add this, we must ensure the OS buffers are not overfilled, specifically FFMPEG's stdout.
Currently, we block by reading stdout; with this, we would also be concerned about writing stdin at the same time.
It may be time to move the stdout/stdin handling onto a separate thread.
Position in Code.

I do not use the progress api. Its not needed for my use case.
And yes pumping the data must be done in a seperate thread. I have not implemented additional synchonization in my process builder variant. I have not observed any issues but I would not exclude that my source storage is not capable of delivering data fast enough to overfill ffmpegs input buffer.
What I have observed tho is that ffmpeg is much more sensitive about the output buffer beeing full. Since I primarily output to unix sockets, setting the recv buffer size to a very large size and pumping the data from the unix socket into a cache before it is streamed to the end client for display, was necesarry. I tried streaming directly to the end client but the client consumed data too slowly so the output buffer filled up. This caused broken pipe errors in ffmpeg.

Speed shouldn't be too big of an issue, depending on how you load the data from storage. Not reading fast enough would prevent filling the buffer on the other side. However, that depends on many factors; you should test that in your environment.
Small update:
I got the following code working:
InputStrream in = Files.newInputStream(Paths.get(Samples.big_buck_bunny_720p_1mb));

FFmpeg ffmpeg = new FFmpeg();
ffmpeg.setProcessInputStream(in);
FFmpegBuilder builder = ffmpeg.builder().addInput("pipe:").addOutput(Samples.output_mp4).done();

ffmpeg.run(builder);

I'm not quite ready to merge, but the code is already available on my fork.

This looks good to me.
Do you call close() on the InputStream?
Its very important you make it clear who should call close() on the InputStream.
My person preference would be that caller calls close() on the stream after run(builder) returns.
This would allow for try with resources around ffmpeg.run(builder)
I do not think I have the time to build a branch and import it into my package management tool. Its quite complicated/time consuming. I would be willing to check it out if you provide it on a maven repository (not maven central, but an actual maven repo with a public URL as that is much easier to import, many larger projects have a snapshot repo for such purposes)

The stream is not closed. In most cases, a try-with-resources should be fine, but I didn't want to limit use cases. I'll make sure to include it in the documentation.
I'm currently on vacation (flying back tomorrow evening). I'll see what I can do about a snapshot repository. Currently I'm thinking of using jitpack, but I'll see if there are better/other options.


setuptools outputs "ERROR: command '/usr/bin/ccache' failed with exit status 1" during build_ext with no further details - would more detail output be possible?

I am trying to build a self-written package in a cross compile environment with the Android NDK. I am building other packages in the same environment successfully, so in theory the compiler and such should work.
However, this specific package the cpp extension runs through perfectly, but the .pyx/Cython extensions abort during build_ext with a very weird error:
  running build_ext
  building 'sfxscan._scan' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/sfxscan
  /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -fomit-frame-pointer -D__ANDROID_API__=21 -mandroid -isystem /ndk/sysroot/usr/include/arm-linux-androideabi -I/ndk/sysroot/usr/include/arm-linux-androideabi -isysroot /ndk/sysroot -I/home/userhome/.local/share/python-for-android/build/python-installs/unnamed_dist_1/include/python3.7 -DNDEBUG -g -fwrapv -O2 -Wall -DANDROID -fomit-frame-pointer -D__ANDROID_API__=21 -mandroid -isystem /ndk/sysroot/usr/include/arm-linux-androideabi -I/ndk/sysroot/usr/include/arm-linux-androideabi -isysroot /ndk/sysroot -I/home/userhome/.local/share/python-for-android/build/python-installs/unnamed_dist_1/include/python3.7 -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -I/home/userhome/.local/share/python-for-android/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/userhome/.local/share/python-for-android/build/venv/include/python3.6m -c sfxscan/_scan.cpp -o build/temp.linux-x86_64-3.6/sfxscan/_scan.o  -Ivendor-include -Lvendor-lib  -lz
  arm-linux-androideabi-gcc: error: : No such file or directory
  arm-linux-androideabi-gcc: error: : No such file or directory
  Android detected, will not build taglib!
  RUNNING BUILD_EXT...
  ERROR: command '/usr/bin/ccache' failed with exit status 1
  DONE RUNNING BUILD_EXT.
  error: command '/usr/bin/ccache' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for sfxscan

As you can see here, "RUNNING_BUILD_EXT" ... "DONE RUNNING BUILD_EXT." I've put such that they frame the build_ext command run:  https://github.com/JonasT/sfxscan/blob/753c5bedae81401829ea25ab76b3e60584c45fec/setup.py#L43 (with the ERROR being a printed out exception).
This is most likely my own fault. However, the thing is that build_ext doesn't even tell me here what it tried to build before it just crashes with that error. So why I'm filing this ticket is basically, wouldn't it be better to provide actual useful output here before spilling out that error? Like for example, what extension did it even try to build? It would be neat if that could be improved
The bogus   arm-linux-androideabi-gcc: error: : No such file or directory might also be related in some way, I haven't seen them for any other package and that seems pretty weird. But I really can't tell why they're output or what it even tried to compile

Ok so I managed to collect the full error backtrace with more debug output hackery:
[DEBUG]:   	  Traceback (most recent call last):
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
[DEBUG]:   	      extra_postargs)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
[DEBUG]:   	      spawn(cmd, dry_run=self.dry_run)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/spawn.py", line 36, in spawn
[DEBUG]:   	      _spawn_posix(cmd, search_path, dry_run=dry_run)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
[DEBUG]:   	      % (cmd, exit_status))
[DEBUG]:   	  distutils.errors.DistutilsExecError: command '/usr/bin/ccache' failed with exit status 1
[DEBUG]:
[DEBUG]:   	  During handling of the above exception, another exception occurred:
[DEBUG]:
[DEBUG]:   	  Traceback (most recent call last):
[DEBUG]:   	    File "setup.py", line 47, in run
[DEBUG]:   	      super().run()
[DEBUG]:   	    File "/tmp/pip-build-env-o2ie9anp/overlay/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 78, in run
[DEBUG]:   	      _build_ext.run(self)
[DEBUG]:   	    File "/tmp/pip-build-env-o2ie9anp/overlay/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
[DEBUG]:   	      _build_ext.build_ext.run(self)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
[DEBUG]:   	      self.build_extensions()
[DEBUG]:   	    File "/tmp/pip-build-env-o2ie9anp/overlay/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions
[DEBUG]:   	      self.build_extension(ext)
[DEBUG]:   	    File "/tmp/pip-build-env-o2ie9anp/overlay/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 199, in build_extension
[DEBUG]:   	      _build_ext.build_extension(self, ext)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
[DEBUG]:   	      depends=ext.depends)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/ccompiler.py", line 574, in compile
[DEBUG]:   	      self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
[DEBUG]:   	    File "/usr/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile
[DEBUG]:   	      raise CompileError(msg)
[DEBUG]:   	  distutils.errors.CompileError: command '/usr/bin/ccache' failed with exit status 1
[DEBUG]:

It would be nice if right before doing raise CompileError it would tell me what extension it tried to compile, or even better also print the full compiler output. (I'm not sure if the No such file or directory is related at all, since it seems to be possibly before the build_ext call, unless stdout/stderr buffering is doing super weird  things)

Ok this appears to be just an stdout flushing abnormality, I've dug deeper and it seems the ccache-line is actually the it fails, my added bug print lines just occur afterwards since I don't flush. oopsies


Endpoint plugins configurable via YAML

Fixes #4685.

@richard-cox if you'd like to give it a shot, here's the required setup:

Create a Stratos API key ($STRATOS_TOKEN) and add an endpoint :

curl -k -H "Authentication: Bearer $STRATOS_TOKEN" -X POST "https://localhost:5443/api/v1/endpoints" -H  "accept: application/json" -H  "Content-Type: application/x-www-form-urlencoded" -d "endpoint_type=private_github&cnsi_name=private_github&api_endpoint=https://api.github.com&skip_ssl_validation=false"

Get the generated $ENDPOINT_GUID and add the token with your GitHub private token ($USERNAME and $GITHUB_TOKEN):

curl -k -H "Authentication: Bearer $STRATOS_TOKEN" -X POST "https://localhost:5443/api/v1/tokens" -H  "accept: application/json" -H  "Content-Type: application/x-www-form-urlencoded" -d "cnsi_guid=$ENDPOINT_GUID&system_shared=false&connect_type=creds&username=$USERNAME&password=$GITHUB_TOKEN"

Proxy the API requests:

curl -k -H "Authentication: Bearer $STRATOS_TOKEN" 'https://localhost:5443/api/v1/proxy/$ENDPOINT_GUID/search/repositories?q=secret-repo-name+in:name+user:$USERNAME&page=1&per_page=100'

@richard-cox here's a sample cfapppush message for GitHub:
{"message":"{\"project\":\"ikapelyukhin/secret-potato-repo\",\"branch\":\"master\",\"type\":\"gitscm\",\"commit\":\"72a44937df79ffde9b1c9db72784158620d571d6\",\"url\":\"https://github.com/ikapelyukhin/secret-potato-repo\",\"scm\":\"github\",\"endpoint_guid\":\"Z6KJAVYeHPkiEPuZty0-kArqors\"}","timestamp":1605041032,"type":30001}

endpoint_guid is new here.
For GitLab:
{"message":"{\"project\":\"ikapelyukhin/secret-potato-repo\",\"branch\":\"master\",\"type\":\"gitscm\",\"commit\":\"a5c60281ad4cfa9bd3decb629c4347e759fc050e\",\"url\":\"https://gitlab.suse.de/ikapelyukhin/secret-potato-repo\",\"scm\":\"gitlab\",\"endpoint_guid\":\"RlZecuoWsHPFZE4ImUoWb3_wG9s\",\"username\":\"ikapelyukhin\"}","timestamp":1605041032,"type":30001}

endpoint_guid and username are new here.

Codecov Report

Merging #4751 (d4b9a1d) into master (00dbcf9) will decrease coverage by 26.06%.
The diff coverage is 9.09%.

@@             Coverage Diff             @@
##           master    #4751       +/-   ##
===========================================
- Coverage   53.71%   27.64%   -26.07%
===========================================
  Files        1127       59     -1068
  Lines       39118     4619    -34499
  Branches     4951        0     -4951
===========================================
- Hits        21011     1277    -19734
+ Misses      17930     3164    -14766
- Partials      177      178        +1


Updated commercial_paper.md (hyperlink error)

Brackets hyperlink was wrong. It was pointing to sublimetext.com instead of brackets.io.

Thanks for the contribution! However, this is a read-only mirror. please read the contribution guide. Also, feel free to join us on chat!


Work around lack of Unix.set_nonblock on Windows using threads

On Windows Unix.set_nonblock fails.
This PR implements a workaround method for pulling the output without blocking by using threads.  I can't say I like the workaround, but implementing proper non-blocking IO on Windows would seem to require a lot more work.  This workaround has been tested at least on Mac OS and Windows.

I'm closing this as reverting PR403 would be a much simpler solution.

#414 merged to undo the issues that we had with #403.


NLTK for vietnamese

Have nltk supported vietnamese language?
In case it haven't. How can I contribute to make ntlk support vietnamese language?
It would be like this
>>> import nltk
>>> sentence = "Vào tám giờ thứ sáu, tôi cảm thấy không được khỏe."

>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['Vào', 'tám', "giờ", 'sáng', 'thứ sáu', 'tôi', 'cảm thấy', 'không', 'được', 'khỏe', '.']
>>> tagged = nltk.pos_tag(tokens)
>>> tagged[0:5]
[('Vào', 'IN'), ('tám', 'CD'), ("giờ", 'JJ'), ('sáng', 'NN'), ('thứ sáu', 'NNP'), ]

Hi @stevenbird,
What do you think ? Probably we can port these
http://jvntextpro.sourceforge.net/

@rain1024 would you like to do some porting, or contribute wrappers for external Java libraries?

@stevenbird : yes. I'm glad to do this.
@longdt219: can we do this together?

Yes sure @rain1024

hi @longdt219
can I have your email? I will contact to you for more information :smile:

Hi @rain1024,
I emailed you but probably we can discuss here so that others can join the discussion.

@rain1024 @longdt219,
How about porting this https://github.com/rockkhuya/DongDu as first step? Which is aimed for word segmentation and written in C++ by the way.
I don't know C++ or Java but that tool must have the best performace so far, according to http://xltiengviet.wikia.com/wiki/Kỷ_lục_tách_từ

Hi, me again,
After searching around for a while I found that word segmentation in Vietnamese is a really hard problem, not to mention POS tagging.
I had an idea inspire by https://github.com/mesnilgr/is13 for using deep learning to learn word embeddings, and I'll try to implement it. Some interesting may come, or not :smile_cat:

I've implemented a neural net for Vietnamese word segmenting here https://github.com/manhtai/vietseg. Have a look!
It's not so good for now. But at least I've tried, huh? :smile:

About the performance, it look OK though. However, what's the baseline ?
What is the dependencies ? using network.py from https://github.com/mnielsen/neural-networks-and-deep-learning probably is not a good way w.r.t maintenance and licensing. The idea is we don't want to rely on external code.
Using Theano (python based) for this might be a better (and simpler) solution.

Thanks, I'm looking for a baseline and will add it soon.
Theano may be better but not simpler, network.py is an independent file with less than 300 lines of code.
Anyway, it's only a quick and dirty implementation. I've added future works to README file, and that's for working in the future :smile_cat:

@longdt219 @rain1024 I have been using jvntextpro2 for awhile and it's pretty decent. It's written in Java and also an opensource project. We may choose to port this as well.

Bumping the issue ;P
I've written a JVnTextPro wrapper some time ago but it's not properly documented and the coding style is outdate but I hope it helps.
Would be great to see other Asian languages annotators wrappers/ports too =)

@alvations: are you interested in porting JVnTextPro to NLTK :P ?

@letuananh after much thinking, yes. After the PTB tokenizer is merged, interface to JVN would be something on my todo list. Care to help?

:+1: It would be great to support Vietnamese

wow... this is awesome stuff. Would love to have Vietnamese support!

@manhtai do you plan to continue on your project. it sounds awesome.

Coming back to this issue after the next minor release =)
But meanwhile take a look at https://github.com/alvations/underthesea

@rain1024 How about your original porting plan? I reached here because i have ported a python version vnTokenizer and planning if it's possible to port into nltk. I also saw your continuous good job of underthesea and have a question about your next step.

@u8621011 underthesea isn't my work but they're doing a good job =)
I'm not sure how much mileage we can get if we start porting from Jvntextpro. But I think I won't be able to take another try at porting until late July.
Vietnamese support is surely on the list of things I personally would like to see and work on in NLTK.

@u8621011 Glad you asked. Our next steps in underthesea are integrating more modules such as speech synthesis, machine translation and (simple) chatbot for Vietnamese and improving speed and accuracy in current modules (word segmentation, pos tagging, chunking, named entity recognition, text classification and sentiment analysis).
About porting plan in nltk, I think we can write code in pure python to do word segmentation task (perhaps with cython to speed up performance) at the moment. I and my friend @trungtv have an accepted pull request in spacy 2 months ago.


How to tell what menu item was clicked in generic click function

I'm loading the menu dynamically in a plugin system. I define 1 function for the fun property of each menu item. There doesn't seem to be a way for me to tell when inside this function what menu item was clicked.

The tag itself (and the text) is available in the 2nd argument to your function or the event.  From that you can extract the text and determine the menu item:
arguments[1].target
value:
Some Menu Item
or:  event.target.innerText
If it's a submenu you can determine the parent item and figure out the context, should you have duplicate menu items.
Using JQuery:  $(event.target).parent().parent()[0]
result:


txn version 2->3

it should have always been like this on berkeley

!ci-nightly-me

!ci-build-me

!ci-nightly-me

!ci-build-me

https://buildkite.com/o-1-labs-2/mina-end-to-end-nightlies/builds/1621 nightly is good except for the usual rustc toolchain thing...

!ci-build-me

!approved-for-mainnet


Fixes an issue on cleanup with mapped directories

Resolves: #131

@jhowardmsft - Has verified the fix

LGTM

@jterry75,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


parser fails to find and attach script to root node of a scene when running a specific scene (run current)

Godot version
whatever version came out around the end of September
System information
Windows 11
Issue description
Okay so I have a script on the root node of an area, and the script will not attach it's self if I run the current scene, but only if there's any changing the scene, which is something that I had to point out, also sorry I couldn't provide more details, my desktop is currently dead and I won't be able to access it for a long time so I'm reporting the issue before I forget
Steps to reproduce
Have a node3d scene
Attach a script to the file
Preload another scene within the script, set it to change to said scene at a trigger
(Optional) add some visual elements to differentiate the scenes
Run the current scene
Minimal reproduction project
Unable to provide currently

Running the full game mitigates the error, it's only when you run the current scene

This issue is impossible to act on as is and will be closed.

There's indeed nothing actionable in this issue.
It sounds like you're having a bug in your code, or wrong expectations that you can run a scene standalone while it has dependencies on other scenes not in the scene tree.
You should ask for support on other community platforms like Discord, Facebook or Reddit.
GitHub is only for bug reports, which clears steps to reproduce them.

nevermind, found the error, it was a cyclical error im a dumb


[discovery] Configuration not found should be an error

Previously when we cannot read from the configuration file,
an error message is printed to stderr and we keep going.
This commit changes this behavior so we will just fail out.

LGTM.
Are there any other types of error where we continue when we should be stopping? (It might be worth doing an audit for "cannot"...)

After some audit with "cannot" "can't" "error", etc it looks like this is the only one we have. Of course, we'll be fixing them if we find them later.


Request class hardcoded in GuzzleHttp\Psr7\modify_request

I'm using an extended GuzzleHttp\Psr7\Request with the guzzle client:
class FooRequest extends \GuzzleHttp\Psr7\Request {

}

$client = new Client(...);
$request = new FooRequest(...);
$response = $client->send($request);

The problem is, that GuzzleHttp\Psr7\modify_request creates a new GuzzleHttp\Psr7\Request instead of maintaining the class of the supplied instance.
The fix is very simple:
In function GuzzleHttp\Psr7\modify_request,
replace
return new Request(

with
$requestClass = get_class($request);
return new $requestClass(

that way, a new instance of the correct class is created.

@adick PSR-7 classes are not supposed be extended.
To be more precise: modify_request accepts an object implementing PSR-7 Request and returns one implementing the same interface. That's the guarantee, nothing more.
Your proposed solution wouldn't work for all cases, because the constructor is not part of the interface, so you can easily define your own constructor with different parameters. In that case the proposed solution would just break.
Sorry, but this won't end up in the library for the above reasons.

@sagikazarmark Thanks for the hint, I'll implement Psr\Http\Message\RequestInterface and map the response via mapResponse middleware instead of extending the Request.


I detected as l

Basic Information
5.3.0
Operating System
Ubuntu 22.04 Jammy
Other Operating System
Linux
uname -a
Linux
Compiler
No response
Virtualization / Containers
No response
CPU
No response
Current Behavior
Some letters are not detected well.
Picture demonstrating the issue:

Expected Behavior
No response
Suggested Fix
No response
Other Information
No response

Which data files did you use? There a three repos of data files: tessdata, tessdata_fast, tessdata_best. Also, there are 'eng' and 'Latin' data files. You can try and see if using different models gives better results.
I and l look almost the same in this font (Arial?).
The text in your example is not similar to the text that was used during training. In such cases, the ocr will be less accurate.
Also, using the wordlist in the data model does not help Tessract to better recognize the text in this image.


Alternative fix fonts issue in printing

Unfortunately  #130 prints the whole web page for me rather than just the PDF. This method renders in the original document first and copies to the iframe. Inefficient, but effective.
Fixes #66

Thanks for this PR! It fixes a problem for me with unicode placeholder elements instead of the correct rendered font character in the resulted print (using ubuntus 'print to file' function).
Performance seems ok to me, not much different to the current implementation in master. Don't know if there are any other caveats or disadvantages with this solution.
:+1: from me :)

This also fixes a "missing font" issue in the print() function...

请问一下 插件支持滚动翻页吗

any reason this fix not merged with the main repo ?

I'm also very interested in this PR getting merged since I'm currently relying on a fork with this fix for getting PDF printing to work :-)

I'm also very interested in this PR getting merged since I'm currently relying on a fork with this fix for getting PDF printing to work :-)

+1

+1

+1

+1

+1

good


New UI design part 3: Pipeline Editor

Description
The new design of Pipeline Editor, including UX improvements and performance optimization.

New feature

[x] File Manager
[x] Session panel
[x] Execute interactive run button and the dropdown
[x] New mini header bar
[x] keyboard support
[x] Read-only banner

[x] job run
[x] environment build in progress
[x] jupyterlab build in progress

[x] Step details design

[x] New layout & design
[x] Add ability to remove incoming connections
[x] Fix sorting incoming connections
[x] Add outgoing connections
[x] Add ability to remove outgoing connections
[x] Add "info" tooltips to incoming/outgoing connections
[x] Add ability to add new connections

[x] Pipeline file name responsiveness
[x] Saving status chip

Bug fixes

[x] consecutive right clicks on the canvas would open multiple context menus

Could be implemented in #1060

[ ] new canvas grid

[ ] make infinite
[x] update to new design (dots)

[ ] Step selector dragging performance issue (to be verified)

I've gone over the PR a few times, and done a lot of testing.
All I can find right now is a performance regression with the step details panel resize. I'm looking at that now. The idea is to replace that handler with the standardized <ResizableContainer> , but it currently doesn't work when it's anchored to the right, so I'm fixing that first.

Great work @iannbing - this is huge, so I'll start dropping what I find into this comment throughout the day and structure it as much as possible!
File manager
Icon button distribution

[ ] On file manager resize, evenly distribute icon buttons across width rather than hang right

selection style

[ ] Could we change style to blue when selected?

duplicate selection

[ ] When you make a selection and then keyboard through file manager, you get a strange thing where there appear to be two selections. Could we just make the most recent keyboard or click interaction as a single selection?

inconsistent cursor on pipeline links

[ ] Make both finger pointers

Canvas
Canvas background pattern

[ ] Make the canvas bg pattern repeat forever without a border

Alert padding

[ ] Reduce vertical and horizontal padding for alert to 8px

Toolbar
Primary button text

[ ] Change text size of primary button to ButtonSmall to match text buttons

Button corners

[ ] Fix strange button masking on right side

Button spacing

[ ] Increase from 8px to 16px horizontally

Logs dialog

[ ] Have a separate URL for Logs dialog than from pipeline UI

Keyboarding order

[ ] Remove additional tab item between logout button and new pipeline button

Thanks for all the feedback @ncspost. I'll go over this today. :+1:

Thanks for all the feedback @ncspost. I'll go over this today. 👍

Thanks for all the feedback @ncspost. I'll go over this today. 👍

np - there'll be more!

Project page

[ ] Add "rename" functionality to menu on Projects page

Zoom button

[ ] Text input is really dodgy when deleting and entering text because it immediately responds - maybe we shouldn't change zoom until hit submit
[ ] Change zoom shortcut to cmd + and cmd -
[ ] Change default zoom increments to: 2%, 3%, 6%, 13%, 25%, 50%, 100%, 200%
[ ] Rename "centre view" to "zoom to fit" and move to top section
[ ] Add "Zoom to 100%"

Docs link should link: https://docs.orchest.io/en/stable/fundamentals/pipelines.html#pipelines

When changing pipeline via sessions section the tooltip does a weird jump to the top of the file manager

Something weird happens when I try to stop pipeline sessions

@ncspost

Make the canvas bg pattern repeat forever without a border

We deemed this out of scope for this iteration, it's not as simple as it seems and a definite rabbit hole. It's not worth the time investment now.

Is it possible to display the status of environments built in the alerts?

@ncspost

Is it possible to display the status of environments built in the alerts?

Yes it is, but I don't want to scope creep this PR too much. I think it's more important to focus on getting the whole re-design shipped, and then we can focus on these types of features.

When you make a selection and then keyboard through file manager, you get a strange thing where there appear to be two selections. Could we just make the most recent keyboard or click interaction as a single selection?

This is the default behavior of the MUI tree view. It is that way, because otherwise you wouldn't be able to select multiple files using the keyboard, the same behavior exists in VSCode:
https://user-images.githubusercontent.com/8259221/183058572-710b6d28-7a0d-4307-ac86-13a19e57e617.mp4
So it's pretty standard behavior, and I think we should conform to it. I also don't want to override standard behavior in MUI.

@ncspost

Button text: "Build environment(s)"

This doesn't quite make sense when the environments are being built. :thinking:
I think you can add your copywriting suggestions directly here in GitHub, it should also clear up any confusion about which states there are. :+1:
I moved all the copy into the same switch-block, so you can see exactly. Here's the relevant section for it: https://github.com/orchest/orchest/pull/1147/files#diff-0417fe8b0e3c0b05c707c00c9ba655729990a300c911889013d3fe691042ce3cR62-R92

Question: Does the environment build autostart when opening a pipeline? Or do you need to press the alert button?

The build should start immediately when you open the editor, but there could be an edge-case where that doesn't happen and the user has to trigger it.

@ncspost I have fixed all remarks now except for those noted below.

Add numbers to incoming order so users know sort order

I had this in the first iteration, but we decided against it because it felt confusing
when reordering, see this internal Slack thread.

Can intercom button move and hang to the left of the step detail panel when open?
Intercom button hangs over step details once I have clicked it

I think so, yes … but Intercom isn't displaying for me. 🤔
Will take a look once it does! :+1:

Question: It doesn't look like log unread dots are implemented in the step detail panel. Will those be implemented later?

This is definitely a rabbit hole that I don't think is worth the time investment now.

Add "rename" functionality to menu on Projects page

Out of scope since this PR Is for the pipeline editor. But definitey something we can do in the base branch/PR https://github.com/orchest/orchest/pull/1060

Change zoom shortcut to cmd + and cmd -

The library we use for hotkeys seems to work inconsistently with CTRL/CMD++, I spent a couple of hours trying to fix it, but got nowhere. Let's revisit this one.

Disable an canvas panning on empty state

I disabled the selection rectangle, but I couldn't find a consistent way of disabling panning after spending a good amount of time on it. I don't think it's super important, but if @iannbing knows how to easily solve it, feel free to pick it up.

Something weird happens when I try to stop pipeline sessions

I cannot reproduce this, so I'm guessing it was something on your machine, or something that has been fixed since then.


Can't access TableStorage

Preflight Checklist

[X] I have installed the latest version of Storage Explorer.
[X] I have checked existing resources, including the troubleshooting guide and the release notes.
[X] I have searched for similar issues.

Storage Explorer Version
1.27
Regression From
no
Architecture
x64
Storage Explorer Build Number
No response
Platform
Windows
OS Version
No response
Bug Description
Unable to retrieve child resources.
Details:
{
"name": "RestError",
"message": "getaddrinfo ENOTFOUND myaccount.table.core.windows.net",
"stack": "RestError: getaddrinfo ENOTFOUND myaccount.table.core.windows.net\n    at ClientRequest. (C:\Users\terre\AppData\Local\Programs\Microsoft Azure Storage Explorer\resources\app\node_modules\se-table-extension\dist\src\index.js:2:182499)\n    at Object.onceWrapper (node:events:642:26)\n    at ClientRequest.emit (node:events:527:28)\n    at TLSSocket.socketErrorListener (node:_http_client:454:9)\n    at TLSSocket.emit (node:events:527:28)\n    at emitErrorNT (node:internal/streams/destroy:157:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:122:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)",
"code": "ENOTFOUND",
"request": {
"url": "https://myaccount.table.core.windows.net/Tables",
"headers": {
"accept": "application/json;odata=minimalmetadata",
"x-ms-version": "2019-02-02",
"dataserviceversion": "3.0",
"accept-encoding": "gzip,deflate",
"user-agent": "Microsoft Azure Storage Explorer/1.27.0 (win32) azsdk-js-data-tables/13.1.2 core-rest-pipeline/1.9.0 Node/v16.16.0 OS/(ia32-Windows_NT-10.0.19044)",
"x-ms-client-request-id": "3848fbf4-3c7f-476c-8e2c-317f7b1bdfdd",
"x-ms-date": "Tue, 06 Dec 2022 22:42:50 GMT",
"authorization": "Signed SharedKeyLite Redacted="
},
"method": "GET",
"timeout": 0,
"disableKeepAlive": false,
"streamResponseStatusCodes": {},
"withCredentials": false,
"tracingOptions": {
"tracingContext": {
"_contextMap": {}
}
},
"requestId": "3848fbf4-3c7f-476c-8e2c-317f7b1bdfdd",
"allowInsecureConnection": true,
"enableBrowserStreams": false
}
}
Steps to Reproduce

Launch
expand account tab
attempt to expand Tables
boom, error appears.

Actual Experience

Launch
expand account tab
attempt to expand Tables
boom, error appears.

Expected Experience
Show the 30 tables I have in my account
Additional Context
I can access my blob containers fine.
none

Hi @Terrence77 please consider enabling system proxy: https://learn.microsoft.com/en-us/azure/storage/common/storage-explorer-network#use-system-proxy-preview
If that does not work, we recommend you open an Azure support ticket via the portal to help you diagnose why your machine cannot reach your storage account.

I will take look at system proxy.
Thank you.

Closing due to lack of response. If you are still running into this problem we recommend you open an Azure support ticket via the portal. Alternatively you can open a new issue here.


Convert servicemonitor

Done, generating code is working.

@alejandroEsc could you clean up the commit history? Looks like you've pulled in a few things from main that are muddying the diff.

@alejandroEsc could you clean up the commit history? Looks like you've pulled in a few things from main that are muddying the diff.

maybe i did, but i had not even rebased when those showed up, not sure what happened

closing in favor of https://github.com/redpanda-data/helm-charts/pull/1313


Make master ServiceMonitor respect PROMETHEUS_SCRAPE_KUBELETS

What type of PR is this?

/kind bug
What this PR does / why we need it:
This PR updates the template for the ServiceMonitor for the master service, to make it exclude kubelets when that is configured.
Which issue(s) this PR fixes:

Fixes #1922
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/lgtm
/approve


Use LaunchedEffect for system icons

Codecov Report

Merging #281 (cf3783e) into main (0f12aee) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #281      +/-   ##
==========================================
- Coverage   73.06%   73.04%   -0.03%
==========================================
  Files          74       74
  Lines        3304     3305       +1
  Branches      542      543       +1
==========================================
  Hits         2414     2414
- Misses        574      575       +1
  Partials      316      316

Impacted Files
Coverage Δ

...lin/com/alexvanyo/composelife/ui/ComposeLifeApp.kt
0.00% <0.00%> (ø)

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Add archival notice

If this is meant to be archived is there any fork or option that is currently being maintained?
Cheers!

Unfortunately I don't think there is a well-known/well-supported fork that I could point us at :(


Preflight tasks

[x] Rudder inverted (flip RC channel)
[x] Elevator inverted in Stabilized mode (flip RC channel)
[x] Adjust/Document COM_DISARM before take-off (currently at 180s)
[x] Covert all battery plugs
[ ] Charge all batteries
[ ] Charge RC controller
[ ] Bring charged spare batteries for RC controller
[ ] Charge your laptops, bring power adapters
[ ] Cleanup and secure avionics (tape/etc)
[x] Make sure all important variables are logged
[ ] Figure out how to test gear channel (use flaps channel instead?)
[ ] Pack a toolbox and spare parts
[x] Make a charger adapter for new batteries
[ ] (re)calibrate airspeed and take some measurements outside

Closing as done.


TouchPitchHandler is not active by default?

Hi,
Sorry, I'm new to maplibre-gl-leaflet and I'm testing it on an Android device. It seems that TouchPitchHandler is not active by default or it is not working.
For instance in the example:
https://raw.githack.com/maplibre/maplibre-gl-leaflet/main/examples/events.html

on desktop I can rotate and tilt map with mouse right button;
on android, I can't rotate or tilt map by dragging up and down with two fingers, as described in https://maplibre.org/maplibre-gl-js-docs/api/handlers/#touchpitchhandler
The double click to zoom (https://maplibre.org/maplibre-gl-js-docs/api/handlers/#doubleclickzoomhandler) seems to work.

Is this the expected behaviour with maplibre-gl-leaflet? Using pure maplibre-gl-js I can use the two fingers to tilt and rotate.
Thank you very much!

Hi @PedroVenancio.
I can't speak to the specific handler, but it's not possible to tile/rotate in Leaflet, so this plugin disables that entirely in maplibre (if it were enabled, the Leaflet map and the MapLibre map would be misaligned). Does that answer your question? It looks like it may be possible we need to disable the right mouse-button handler as well to keep things consistent.

Hi @lseelenbinder
Absolutely, it makes sense. I'm closing the issue. Thank you very much Luke.


atlantis: 0.19.8 -> 0.20.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/runatlantis/atlantis/releases.
meta.description for atlantis is: "Terraform Pull Request Automation"
meta.homepage for atlantis is: "https://github.com/runatlantis/atlantis"
meta.changelog for atlantis is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.19.9 with grep in /nix/store/z8rfzfv4d7jkqi93ba9hbxf3shv6wgar-atlantis-0.19.9
found 0.19.9 in filename of file in /nix/store/z8rfzfv4d7jkqi93ba9hbxf3shv6wgar-atlantis-0.19.9

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
atlantis

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/z8rfzfv4d7jkqi93ba9hbxf3shv6wgar-atlantis-0.19.9 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A atlantis https://github.com/r-ryantm/nixpkgs/archive/8ade5320dc8cdc5727c308869757013925714578.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/z8rfzfv4d7jkqi93ba9hbxf3shv6wgar-atlantis-0.19.9
ls -la /nix/store/z8rfzfv4d7jkqi93ba9hbxf3shv6wgar-atlantis-0.19.9/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

atlantis

Maintainer pings
cc @jpotier for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/runatlantis/atlantis/releases.
meta.description for atlantis is: "Terraform Pull Request Automation"
meta.homepage for atlantis is: "https://github.com/runatlantis/atlantis"
meta.changelog for atlantis is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.20.1 with grep in /nix/store/6jncbh1vwx2azdvw020xsgh1p8m6h479-atlantis-0.20.1
found 0.20.1 in filename of file in /nix/store/6jncbh1vwx2azdvw020xsgh1p8m6h479-atlantis-0.20.1

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
atlantis

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6jncbh1vwx2azdvw020xsgh1p8m6h479-atlantis-0.20.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A atlantis https://github.com/r-ryantm/nixpkgs/archive/fb9e7d16e2942414c35ccd4d4222997b56bef098.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6jncbh1vwx2azdvw020xsgh1p8m6h479-atlantis-0.20.1
ls -la /nix/store/6jncbh1vwx2azdvw020xsgh1p8m6h479-atlantis-0.20.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

atlantis

Maintainer pings
cc @jpotier for testing.


Please release 3.1.1-SNAPSHOT as 3.1.1-RELEASE

@kongchen We should release 3.1.1-SNAPSHOT as 3.1.1-RELEASE and bump the snapshot version to 3.1.1-SNAPSHOT.
What do you think ?

Good idea. I wanted to ask when will be the next release. :)

@kongchen I've just committed some massive integration test updates, and also committed support for swagger-core 1.5.4 (the latest one as of this writing).
I think we should put a plan together to get master this released, as it has been stable for awhile, and has many, many improvements.

+1, after my latest pull request ;)

Sorry guys, I'll release it soon
2015-11-05 3:36 GMT+08:00 Mortier Andries<EMAIL_ADDRESS>
+1, after my latest pull request ;)
—
Reply to this email directly or view it on GitHub
https://github.com/kongchen/swagger-maven-plugin/issues/244#issuecomment-153840807
.

--
Regards,
Kong

super, could you do it "asap" so I can take it up in our current sprint planning ;)

@kongchen
I see that you've tagged the release - thanks!
It appears that 3.1.1 is not in the mvn repos yet: http://search.maven.org/#search|ga|1|swagger-maven-plugin'
Which repos has the new version been propagated to?

it should be there now.
2015-11-10 0:52 GMT+08:00 Andrew B<EMAIL_ADDRESS>
@kongchen https://github.com/kongchen
I see that you've tagged the release - thanks!
It appears that 3.1.1 is not in the mvn repos yet:
http://search.maven.org/#search|ga|1|swagger-maven-plugin'
Which repos has the new version been propagated to?
—
Reply to this email directly or view it on GitHub
https://github.com/kongchen/swagger-maven-plugin/issues/244#issuecomment-155121061
.

--
Regards,
Kong

Thanks @kongchen


Whitepaper section

This pull request merges in translated copies of the whitepaper from https://bitcoin.org/en/bitcoin-paper into the bitcoincash.org website for better accessibility to people around the world.
This PR also handles linking from:

Homepage > whitepaper
Footer > whitepaper
Getting Started > whitepaper

Could you also maybe rebase on the latest master? I'm not sure why the Sideshift change shows up in there, it seems unrelated.

Ah you're right, looks like english was missed. Same goes with rebase. I'll try to fight with that to tidy up this PR


Syncing stuck after receiving RDB in v4.3.0

问题描述（Issue Description）
In version v4.3.0, a problem occurs during the syncing process. After receiving the RDB file, it only syncs a portion of the data and then gets stuck.
I tested the same setup with version v4.2.2, and it works perfectly without any issues.

I tried to troubleshoot this issue, you can start by synchronizing with “try_diskless=true”.

I reproduced the problem in version 4.3.0, which stems from this commit https://github.com/tair-opensource/RedisShake/commit/f263dd4b7ff401cd026e0ee25a883d85aab2df91
The temporary solution to this problem is as follows:
Use off_reply = true in redis_writer config and set rdb_restore_command_behavior = “skip” in advanced config.
I'm trying to fix this bug. Thanks for raising this issue!


Source code for esp8266-ota-flash-convert

I am looking for the source code of the esp8266-ota-flash-convert used in files esp8266-ota-flash-convert_upg.bin and esp8266-ota-flash-convert.ino-0x81000.bin.
As this project aims to liberate IoT devices from proprietary firmware, I would rather not flash some other binary blobs to my devices.
Thank you,
Jan

I also would appreciate the source.  The way this project is laid out and documented can be a little tough to understand, and I'd like to either improve the documentation via PR or rewrite this whole thing so it feels a little more intuitive.  Being able to see what's happening on the device side in code would be quite helpful to either scenario.

I think I found it: https://github.com/vtrust-de/esp8266-ota-flash-convert
@jalmeroth

Thanks for pointing out, @cscheib! I did a search but couldn't find it. ¯_(ツ)_/¯


[Migration from Gitlab] Incorrect release name leads to full migration failure

Gitea version 1.15.2

Git version: 2.17.1

Operating system:
Ubuntu 18.04
gitea repo cloned and built with :
using TAGS="bindata sqlite sqlite_unlock_notify" make build

Database (use [x]):

[x] SQLite

Can you reproduce the bug at https://try.gitea.io:

[X] No (sensible data)

Log gist:

Debug log :
2021/09/14 10:45:12 modules/task/task.go:54:handle() [E] Run task failed: GetTagCommit[myrelease/0_4]: object does not exist [id: refs/tags/myrelease/0_4, rel_path: ]

Console log :
2021/09/14 15:52:10 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= log -1
2021/09/14 15:52:10 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/14 15:52:10 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/14 15:52:11 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= remote rm origin
2021/09/14 15:52:11 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/14 15:52:11 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /home/local/gitea/data/gitea-repositories/99rgosse/test.git: /usr/bin/git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check

2021/09/14 15:52:11 ...ules/task/migrate.go:68:func1() [E] DeleteRepository: repository does not exist [id: 29, uid: 1, owner_name: , name: ]

2021/09/14 15:52:11 modules/task/task.go:54:handle() [E] Run task failed: GetTagCommit[myrelease/0_4]: object does not exist [id: refs/tags/myrelease/0_4, rel_path: ]2021/09/14 15:52:13 ...s/context/context.go:740:1() [D] Session ID: f61f7159e7a9accf
2021/09/14 15:52:13 ...s/context/context.go:741:1() [D] CSRF Token:

Description
Unfortunately someone here named his release myrelease/0_4
Even gitlab cannot retrieve this release with curl, while on graphic interface it works well.
In gitea, this means that all the repository import fails due to this misleading name, otherwise I have to import all the repository without release, then remake some of them.
There are absolutely no explanation message, so I was wondering if we should catch this error differently ?
...

Well, to be fair that sounds more like a GitLab than a Gitea problem to me.
It shouldn't be possible to create names that are forbidden, or have your API not ready to handle these cases.
From an algorithmic point of view, I can understand the current approach as it is easy to implement.
While it most likely would be possible to simply ignore invalid issues/ milestones/ releases/ branches/ PRs, it certainly is much more work, and some users might complain about missing things in the end.
Even worse is that this approach then will have to be duplicated for any service where Gitea can migrate from, resulting in even more things to implement.

Well, to be fair that sounds more like a GitLab than a Gitea problem to me.
Even worse is that this approach then will have to be duplicated for any service where Gitea can migrate from, resulting in even more things to implement.

Your answer makes totally sense... I was wondering whether it should be handled or not.
From another point of view, if I was to be "testing" gitea for escaping github/gitlab, this would annoy me, because of the lack of explanation in the failure message. Just a warning "gitlab release myrelease/0_4 not accessible" would then "report the fault on the other side".
I will make a Gitlab ticket though, in order to get their answer for their api failure :D

Hello
I'm still waiting for Gitlab answer, but trying to reproduce the bug on a non critical data repository, everything went great !
I close the issue, as it's clearly not Gitea problem
https://try.gitea.io/99rgosse/release_fail
Thanks everyone and sorry for having bothered you

hmm... actually I think it might be a gitea problem:
 2021/09/14 15:52:11 modules/task/task.go:54:handle() [E] Run task failed: GetTagCommit[myrelease/0_4]: object does not exist [id: refs/tags/myrelease/0_4, rel_path: ]2021/09/14 15:52:13 ...s/context/context.go:740:1() [D] Session ID: f61f7159e7a9accf

I'm not certain that should cause the migration to fail completely.

It's a problem similar as #16919

hmm... actually I think it might be a gitea problem:
 2021/09/14 15:52:11 modules/task/task.go:54:handle() [E] Run task failed: GetTagCommit[myrelease/0_4]: object does not exist [id: refs/tags/myrelease/0_4, rel_path: ]2021/09/14 15:52:13 ...s/context/context.go:740:1() [D] Session ID: f61f7159e7a9accf

I'm not certain that should cause the migration to fail completely.

I got the final clue : the person responsible for the badly named release deleted the tag myrelease/0_4 then could not change the name of the release...
Using gitlab rails console, I can change the tag to the new one. Gitlab & Gitea migration are now happy...
Otherwise Gitea failed when calling
this upload release function from gitea_uploader.go
but indeed, the full migration operation fail...


evaluation_threads does not take effect

I tried evaluation_threads = 56, but the time spent for each epoch is pretty the same:
evaluation_threads = 1
Iteration 0 [112.8 s]: HR = 0.4967, NDCG = 0.2792, loss = 0.3568 [2.6 s]
Iteration 1 [25.0 s]: HR = 0.5631, NDCG = 0.3147, loss = 0.3129 [2.5 s]
Iteration 2 [24.1 s]: HR = 0.5887, NDCG = 0.3314, loss = 0.2943 [2.5 s]
Iteration 3 [23.8 s]: HR = 0.6166, NDCG = 0.3512, loss = 0.2832 [2.5 s]
Iteration 4 [23.6 s]: HR = 0.6215, NDCG = 0.3562, loss = 0.2764 [2.5 s]
vs.
evaluation_threads = 56
Iteration 0 [26.1 s]: HR = 0.4965, NDCG = 0.2790, loss = 0.3659 [1.2 s]
Iteration 1 [23.5 s]: HR = 0.5618, NDCG = 0.3141, loss = 0.3094 [1.2 s]
Iteration 2 [23.1 s]: HR = 0.5952, NDCG = 0.3352, loss = 0.2923 [4.5 s]
Iteration 3 [24.0 s]: HR = 0.6075, NDCG = 0.3474, loss = 0.2807 [3.3 s]
Iteration 4 [24.2 s]: HR = 0.6174, NDCG = 0.3528, loss = 0.2756 [4.3 s]

Hello, why did I adjust the evaluation_threads to other values, and the error will be reported:TypeError: 'NoneType' object is not subscriptable.Is it because global variables are set in the evaluate.py? How do I adjust to use multithreading?


fix: sparse tables

🧰 Changes
A recent update seems to have broken rendering of sparse tables?!
I didn't really investigate why it was working before, but I updated the parser code to better initialize tables with default cells.
🧬 QA & Testing

Working in this PR app.

Looks like it's working! Any chance this happens to fix rendering tables with incorrect align data? (This linear ticket.)

It doesn't 😞

This PR was released!
🚀 Changes included in v6.54.0


Open Liberty <IP_ADDRESS>-beta: MicroProfile JWT RBAC 2.0 RC2 Compatible Certification Request

[x] Organization Name ("Organization") and, if applicable, URL:
IBM
[x] Product Name, Version and download URL (if applicable):
Open Liberty <IP_ADDRESS>-beta
[x] Specification Name, Version and download URL:
MicroProfile Interoperable JWT RBAC 2.0
[x] (Optional) TCK Version, digital SHA-256 fingerprint and download URL:
MicroProfile Interoperable JWT RBAC 2.0 RC2 TCK result
[x] Public URL of TCK Results Summary:
MicroProfile Interoperable JWT RBAC 2.0 RC2 TCK result
[x] Any Additional Specification Certification Requirements:
Not applicable
[x] Java runtime used to run the implementation:
Java 8 and 11
[x] Summary of the information for the certification environment, operating system, cloud, ...:
RHEL 8
[x] By checking this box I acknowledge that the Organization I represent accepts the terms of the EFTL.
[x] By checking this box I attest that all TCK requirements have been met, including any compatibility rules.

@teddyjtorres, @Emily-Jiang   Should this CCR be created against yet to be staged MP JWT 2.0 ?

This CCR is for the latest RC, which is the identical to the staged JWT 2.0 to workaround the chicken egg problem.

Why does the reference to the TCK Version show a link to the TCK Results?
(Optional) TCK Version, digital SHA-256 fingerprint and download URL:
MicroProfile Interoperable JWT RBAC 2.0 RC2 TCK result

This CCR is for the latest RC, which is the identical to the staged JWT 2.0 to workaround the chicken egg problem.

But, I think the CCR should still reference the final 2.0 Specification.  Maybe this update can or should be done after the vote is completed, but it looks odd to reference a preliminary version of the Specification for the final TCK results.

Also, like my other comments on the various CCRs, I think you should specific the specifics of the Java 8 and 11 runtimes that were used to certify.

@kwsutter please see the comments from @radcortez and me regarding the CCR requirements here

Hi @teddyjtorres, @ayoho, @Emily-Jiang  I'm closing it now, I believe it has been resolved :-)
Happy Christmas and New Year


#11 JavaScript Classes for User&Session

Was blocked by 2FA since yesterday night and finally has been cleared up.This includes the Javascript classes for the session and user, they a…re pretty bare because I'm unsure what else is needed, feel free to let me know in the reviews.

I think session is good for what I need it for? I imagine it takes in some JSON (returned from the eventual get session servlet, which I haven't looked into) and just wraps that up in a javascript class?

I would write some documentation to accompany this, just some information about the constructor, its parameters, and the methods for this class (I imagine just methods that return what the variables are?)

Also test too

Make sure the file names are lowercase (style) and that they are in the right place (src/main/webapp/scripts)

try putting these under src/main/webapp/scripts


Initial TWAI Driver Implementation

For some visibility, I wanted to put up this pull request with an initial implementation of a driver for the TWAI peripheral. Here's some of my thoughts on what I have currently:
For the acceptance filters, I'm still thinking through ways of designing this code as there are a couple nuances that aren't initially apparent right now. The acceptance filters have two modes, single filter mode and dual filter mode. Within each of these modes the filter will match both standard ids and extended ids if they have matching bit layouts. This appears to be a limitation of the hardware as the hardware does not seem able to differentiate against id formats in the acceptance filter.
TLDR: I want to avoid situations where a filter is created to only accept a standard id of 0x000, and suddenly later in code extended id packets are received. Perhaps clever naming and documentation such as PartialAcceptanceFilter would be better.
For some background on the TWAI peripheral, most of the setup and communication is done with the TWAI_DATA_x_REG registers. In reset mode, they are used to set the acceptance filter. In operation mode, writing to them prepares data for transmission while reading from them reads received packet data. Because of this shared register layout, the driver does a lot of bitwise operations to encode or decode the packets as the bit layouts are significantly different between modes and frame formats.
This will require a patched pac with read access on the TWAI_DATA_x_REG registers. See: https://github.com/espressif/svd/issues/24 This is patched in main for the esp32c3 pac but not pushed to crates.io yet.
As I only have a ESP32C3 at the moment, I've only tested this driver on that platform.

Awesome progress! Thanks for working on this
I personally don't have experience with TAWI/CAN so I hope that someone else can participate. (@jessebraham  @MabezDev ?)
Maybe one nit-pick alrready would be that identifiers like ESPTWAIFrame would better be changes to UpperCamelCase. Also for the final PR the example would need some more documentation but again, this is WIP and most porbably it's all already on you todo-list 😄

Sorry for the lack of activity here. Unfortunately I have never worked with CAN/TWAI so I'm not really sure how to approach this 😅
I will at least get an updated PAC published for the C3 this week. Beyond that, I either need to spend some time reading or try to hunt down a colleague who is familiar with this peripheral.
I will update again as needed, sorry again for the delays!

I've published esp32c3 version 0.5.1, which includes the TWAI patches.

Thank you! I appreciate the help. I'm still working through the acceptance filter designs to make it a bit easier to understand without reading the reference manual. Any feedback is appreciated.

Looks like the embedded-hal team has released the split out version of embedded-can:
https://crates.io/crates/embedded-can/0.4.1. Going to look into it and figure out what it needs for impls, looks like a copy of the code, just a different crate.
https://github.com/rust-embedded/embedded-hal/pull/411

What is the status of this? I skimmed through the code and besides the missing read access on the data registers this seems workable?
Anything I can do to help?

Our colleague @dnedic also took a look at this PR and said it looked fine to him. Would also be happy to help out however I can in getting this PR ready to merge.

Thanks for having @DNedic take a look at this! I had paused some work on this as I got distracted by embassy and all the interesting work going on between these two repos.
Tonight I'm going to go through and add some #[cfg()] so that this it limited to the esp32c3 and play around with adding embedded-can.
As for how people can help, I think it might be nice to get some feedback on the filter and bitselector designs. Does it complicate a bitmask too much? Are there better libraries that would be acceptable?
Adding support for other models is probably best done in another PR, but I believe expanding to the other models should be pretty straight forward, as it appears they use the same peripheral. This would involve patching the SVD files and testing on those platforms. If someone is interested helping add support for other platforms that would be great.

As for how people can help, I think it might be nice to get some feedback on the filter and bitselector designs. Does it complicate a bitmask too much? Are there better libraries that would be acceptable?

Here's my thoughts on this
Your design it very type driven. This makes it easy to know what's going on without reading much documentation.
However I have to admit my initial though was that it was overengineered and simpler APIs should be possible.
... then I read the reference manual.
My ideas:
I tried to come up with an API myself but I'm not overly happy about it either.
My thought process was that there's two scenarios we'd like to cover.

statically known filters

your API is nice here
alternatively you could provide filters as bytestrings b"1100xxxxxxxx"

dynamically set filters

much easier to work with raw code and mask data here

This is what I came up with:
pub struct SingleStandardFilter {
    data: [u8; 8],
}

impl SingleStandardFilter {
    pub const fn from_id_bits(bits: BitSelector<false, 11>) -> Self;
    pub const fn from_id_pattern(filter: &'static [u8; 11]) -> Self;
    pub const fn from_id_raw(code: embedded_can::StandardId, mask: embedded_can::StandardId) -> Self;
    pub const fn with_rtr(self, rtr: bool) -> Self;
    pub const fn with_first_byte_bits(self, bits: BitSelector<false, 8>) -> Self;
    pub const fn with_first_byte_pattern(self, filter: &'static [u8; 8]) -> Self;
    pub const fn with_first_byte_raw(self, code: u8, mask: u8) -> Self;
    pub const fn with_second_byte_bits(self, bits: BitSelector<false, 8>) -> Self;
    pub const fn with_second_byte_pattern(self, filter: &'static [u8; 8]) -> Self;
    pub const fn with_second_byte_raw(self, code: u8, mask: u8) -> Self;
}

// Example Usage:
const FILTER: SingleStandardFilter = SingleStandardFilter::from_id_pattern(b"1100xxxxxxxx").with_rtr(false).with_first_byte_pattern(b"xxxxxx01");

What other folks do:

ESP IDF uses raw values link
On STMs you can specify masks and codes individually link
In MCP2515 you can specify masks and codes individually link

So not very helpful for us...
PS
It seems the <const EXACT: bool> is always false so maybe remove it?

Thanks for those ideas, I really like the bytestrings approach! Yeah, I agree, coming from a background with the STM32 C HAL implementation of bxcan, there was a lot of head scratching about the register layout for the filter on the ESP32.
To me the bytestrings are more readable than the arrays that I used in bitselector so I'm going to play around with some bytestring implementations tonight and probably drop the bitselector if it goes well.

Here's the code I wrote in testing :)
const fn from_id_pattern(filter: &'static [u8; 11]) -> Self {
    let mut acceptance_code = 0u16;
    let mut acceptance_mask = 0u16;

    let mut idx = 0;
    while idx < 11 {
        let shift = 15 - idx;
        match filter[idx] {
            b'1' => {
                acceptance_code |= 1 << shift;
                acceptance_mask |= 1 << shift;
            }
            b'0' => {
                acceptance_mask |= 1 << shift;
            }
            b'x' => {}
            _ => panic!("filter bits must be eiter '1', '0' or 'x'"),
        }
        idx += 1;
    }
    let [code_h, code_l] = acceptance_code.to_be_bytes();
    let [mask_h, mask_l] = (!acceptance_mask).to_be_bytes();

    SingleStandardFilter {
        data: [code_h, code_l, 0, 0, mask_h, mask_l, u8::MAX, u8::MAX],
    }
}

With the good ideas from @dimpolo, I switched to using bytestrings in the implementation for the SingleStandardFilter.
I ended up combining the parameters into the single constructor as I want to force the user to make the decision for every available parameter. I'd rather just have the person familiar with their specific application set an explicit value instead of trying to pick appropriate default values (ex: what is an appropriate default rtr bit filter value?).
I personally like the bytestrings and if other people like them I'll expand them to the other filter options.

Do you have ideas how to cater to this scenario?

dynamically set filters

much easier to work with raw code and mask data here

I think this is in a good state at the moment and I'm almost ready to merge.
With testing against a Raspberry Pi and a MCP2515, the driver appears to work correctly.

Do you have ideas how to cater to this scenario?

dynamically set filters

much easier to work with raw code and mask data here

I don't have any better ideas currently than just setting with raw code and mask so I added an implementation.

Updated driver and example to use PeripheralRef for #249.

As previously mentioned, I've been testing this driver with a Raspberry Pi, MCP2515 HAT, and can-utils. I thought I'd write down a couple basic commands in case people are unfamiliar.
Setting up the Pi is pretty easy.
Adding this line to /boot/config.txt then rebooting enables the can0 network interface:
dtoverlay=mcp2515-can0,oscillator=16000000,interrupt=25

I can then bring up the network interface with sudo ip link set up can0 type can bitrate 1000000 sjw 3. I had to set the sync jump width (sjw) to 3 on the Pi side otherwise the default sync jump width would cause errors.
Viewing packets can be done with candump.

candump can0

Sending packets can be done using either cansend for individual packets, or cangen for fuzzing.

cansend can0 002#ffaabb00
cangen can0 -I i -D r -g 10

Hello @alexbohm, thanks for the PR (and your patience on reviewing!)
I didn't have access to an MCP2515 HAT so I tested with two esp32c3's and modified the examples, one which just transmits and another that prints what it received. I'll explain my process, mostly for future me :D, but also to allow others to test.
With this method, it's possible to test this without a can transceiver by putting the tx pin on each chip into open-drain, and putting a pull-up on the bus wire. See the diagram below (note that you might get away with the internal pull ups at a low enough baud rate):

The code for the transmit example is the same as the supplied one in the PR, but with the following changes:

Remove the call to set_filter - there is nothing to receive here so no need for it.

can_config.set_filter(FILTER);

Rewrite the tx example loop as follows:

loop {
        let id = StandardId::new(0).unwrap();
        println!("Sending can frame with id {:?}", id);
        let frame = twai::EspTwaiFrame::new(id, &[0xAA]).unwrap();
        let _result = block!(can.transmit(&frame)).unwrap_or_else(|e| { println!("Transmit failed with {:?}", e); None });
        delay.delay_ms(1000u32);
    }

Finally ensure that the TX pins on both examples are in open drain output.

With the changes I observed the following output:


test model with different input image size

Hi, It's possible to test the model with different input sizes? I saw the 512 * 384 size in the code
What is the range of score that model can be predict?
@subpic

Yes, it should be possible, the model allows other input shapes as long as they have 3 channels. The output range is related to the range used for the quality scores during training. I believe that was [1,100].

For one image I got a score above 100. Why was it happen? What's the problem?

Even though the training scores are bounded, the predictions are not constrained to [1,100]. It's possible that for some images the predictions will be outside those bounds.


Fix to inverse scaling for predicted values in own_dataset

Codecov Report

Merging #250 into master will decrease coverage by 6.88%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #250      +/-   ##
==========================================
- Coverage   88.77%   81.89%   -6.89%
==========================================
  Files         108      108
  Lines        5246     5246
==========================================
- Hits         4657     4296     -361
- Misses        589      950     +361


Remove surv_rate object

The survival rates object surv_rate parameter in OG-Core is incorrectly specified in that it is currently just an $S$-element list (see default_parameters.json). It would need to be a nested (2-dimensional) $S$-element list to be consistent with the mortality rates rho parameter, immigration rates, and population parameters. However, surv_rate is only used in parameters.py in OG-Core, and it shows up in testing input objects such as testing_params.json and create_params_inputs.pkl.
We propose that we make the following changes to OG-Core:

Remove the surv_rate object from default_parameters.json
Recode line 578 of parameters.py so that surv_rate is just equal to 1 - rho. This way surv_rate does not have to come from  a country calibration demographics.py.
Fix the tests that depend on surv_rate being present.

This will require similar changes to be made to the country calibrations (e.g., OG-USA, OG-ZAF):

Update the rho parameter in og[cntry]_default_parameters.json
Remove the surv_rate parameter in og[cntry]_default_parameters.json
Remove surv_rate from the output of demographics.py

cc: @jdebacker

Resolved with PR #886


Rerender to relax the zlib constraint from 1.2 to 1

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found some lint.
Here's what I've got...
For recipe/meta.yaml:

This recipe is using a compiler, which now requires adding a build dependence on {{ stdlib("c") }} as well. Note that this rule applies to each output of the recipe using a compiler. For further details, please see https://github.com/conda-forge/conda-forge.github.io/issues/2102.

Hi! This is the friendly automated conda-forge-webservice.
I tried to rerender for you, but it looks like there was nothing to do.
This message was generated by GitHub actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/11690654086.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@djsutherland @nehaljwani, sorry to ping directly. Could you review this PR when you have a chance? Thank you! 🙏


Fix abort

This fixes two problems that were making the certification check benchmark crash.
The first is when the tx coordinator sends the abort command to the clocksi vnode, the vnode sends back an ack ( https://github.com/SyncFree/antidote/blob/master/src/clocksi_vnode.erl#L366 ), but the coordinator wasn't expecting it, causing it to crash.  Now it waits for the ack to know that the abort was successful.
The second was the execution of a static tx made up of updates was expecting a reply of either ok or error ( https://github.com/SyncFree/antidote/blob/master/src/antidote.erl#L169 ), but the tx coordinator could return ok, error, or abort, now it returns either ok or error, but in the information about the error it says it was an abort.
Actually this brings up another issue, some of the functions in the antidote.erl transaction interface return ok, error, or abort, while others just return ok or error (with abort being a possible reason for the error included with the error).  Shouldn't we choose one of these and stick with it, I like the second option personally?  But maybe this would break the protocol buffer as well?

Le 18 janv. 2016 à 13:14, tcrain<EMAIL_ADDRESS>a écrit :
This fixes two problems that were making the certification check benchmark crash.
The first is when the tx coordinator sends the abort command to the clocksi vnode, the vnode sends back an ack ( https://github.com/SyncFree/antidote/blob/master/src/clocksi_vnode.erl#L366 https://github.com/SyncFree/antidote/blob/master/src/clocksi_vnode.erl#L366 ), but the coordinator wasn't expecting it, causing it to crash. Now it waits for the ack to know that the abort was successful

I wonder why: is this logically necessary?  Suppose something goes wrong and the ack is lost, is the abort aborted???
Let me reiterate yet one more time: the fate of a transaction logically depends ONLY on unanimous "prepare".  Once all transaction participants have recorded the prepare message in their log, the transaction IS committed, and all entities in the system must reach this same conclusion no matter what happens.  If there is a state where not all participants have recorded prepare, and one knows for sure that this will not happen in the future (e.g. a participant is known to have crashed) then everyone must conclude "abort".  The abort message is an optimisation, and the termination should take place even if participants crash or messages get lost.
				Marc

I agree with this, but this pull request is only to fix a crash where something was not expecting a message, but a message was being sent. So now it doesn't crash during normal execution.
Again, there is no recovery or fixing lost messages anywhere and I didn't implement it here, this has been discussed over and over and is something that is supposed to be considered when we start restructuring the supervisors and start fixing the fault tolerance.
Instead of changing the logic here I just fixed the crash.  I didn't want to make little hacks to ensure "sort of fault tolerance" that would be replaced later anyway. Instead, when someone does do the work to make Antidote fault tolerant they should examine the whole structure and make a valid system.
It would be great if someone took the time to look into Antidote and design a proper fault tolerance system (a unanimous prepare is just a tiny part of this), ensuring both correctness and progress on failure, crash and recovery and even ring transfers.  This way when we design new components we can follow those rules so we don't have to redesign stuff later.  I would be happy to help with this, but I cannot take the lead on it right now as I would like to finish the partial replication component, but before I do that I am trying to get Antidote to work during normal execution.

OK, point taken.
Marc

Le 18 janv. 2016 à 14:42, tcrain<EMAIL_ADDRESS>a écrit :
I agree with this, but this pull request is only to fix a crash where something was not expecting a message, but a message was being sent. So now it doesn't crash during normal execution.
Again, there is no recovery or fixing lost messages anywhere and I didn't implement it here, this has been discussed over and over and is something that is supposed to be considered when we start restructuring the supervisors and start fixing the fault tolerance.
Instead of changing the logic here I just fixed the crash. I didn't want to make little hacks to ensure "sort of fault tolerance" that would be replaced later anyway. Instead, when someone does do the work to make Antidote fault tolerant they should examine the whole structure and make a valid system.
It would be great if someone took the time to look into Antidote and design a proper fault tolerance system (a unanimous prepare is just a tiny part of this), ensuring both correctness and progress on failure, crash and recovery and even ring transfers. This way when we design new components we can follow those rules so we don't have to redesign stuff later. I would be happy to help with this, but I cannot take the lead on it right now as I would like to finish the partial replication component, but before I do that I am trying to get Antidote to work during normal execution.
—
Reply to this email directly or view it on GitHub https://github.com/SyncFree/antidote/pull/192#issuecomment-172529785.

I created an issue here https://github.com/SyncFree/antidote/issues/193 to keep track of the fault tolerance discussion.


CI for docs' code snippets

What's wrong?
The docs are currently include broken code. This is harmful for the adoption of OpenTelemetry as it degrades the project's reputation and ability of users to onboard. See this earlier discussion for more detail.
How to fix?
Let's get CI for the code snippets in the docs so we can ensure they work - or at the very least improve visibility for when they're broken.
Where to start?
I'll propose starting with the Python "Getting Started" docs as that's where I can probably help, and the issue with those docs is documented.

I wonder if there's not an incremental path to right, as such. If we get  the desirables defined and the problem decomposed, there should be an MVP that stands out as an achievable result within a Sprint's worth of effort, I imagine.
I'm looking forward to understanding the issues here, because broken documentation is a critical issue.

As @svrnm mentioned, this is a known issue, that has been discussed in the past, see:

#1635

This repo briefly used a simple include shortcode, but that falls short of what we really need IMHO.
I have had a solution on the back burner (which is successfully being used in other projects), and would be glad to work on a similar feature in the context of OTel if @open-telemetry/docs-maintainers feel that it's a priority.
In a nutshell, the approach relies on code repos to have CI to ensure that their example code is functional. In this repo we'd use tooling to extract example-code snippets from the CI-tested code of the code repos. The extracted code snippets are caches, so we don't always need to extract from the code repo.
What would help me is if you could identify a few target code-snippets in the docs, and point me to the corresponding code in a code repo. I could work on prototyping the code-snippet sync for a few examples to start off with.
(I'll look into the tooling as soon as I can, but most likely I won't have much time until the new year.)

These sophisticated solutions would do wonders for improving the reliability of the documentation. But I wonder if something much simpler might 80/20 the time-to-value issue: keeping a single, end-to-end, working example CI-tested.

keeping a single, end-to-end, working example CI-tested.

I believe that the demo (or a subset thereof) is meant to serve that purpose. Smaller-scale examples would be good too.
Once you have an end-to-end example, inevitably one ends up wanting to refer to code snippets from the docs, so we're back to the need of keeping the docs and code snippets in sync.
We discussed this during the Comms meeting. I proposed to drop a PR (early next year) that illustrates how code snippets can be kept in sync; I'll probably start with go/example/dice since the snippets we'd need are in code repo already. (An alternative would be do use the demo docs.)

That's what moved us away from CI-checked samples at Microsoft

I agree that we don't want to add CI code checks to this repo. In that sense, could we close this issue in favor of #1635 (@svrnm was looking for #1635 but didn't find it at the time) and continue the code-snippet-sync discussion over there? @svrnm WDYT?

Yes, let's continue this discussion on #1635, copy/link the items you want to continue discussing there


How can i build  HugeCTR without docker

When i build HugeCTR follow this doces.(Build HugeCTR from Source) I found that docker have some problem on my super computer when i run the docker image(nvcr.io/nvidia/merlin/merlin-training:21.09) as follow

groups: cannot find name for group ID 1007
I have no name!@56a762eae3f8:/hugectr

so i want to build the HugeCTR without docker .how can i do

my username in DGX is dgxuser4

$id dgxuser4
uid=1005(dgxuser4) gid=1007(dgxuser4) groups=1007(dgxuser4),999(docker),1010(data),1013(ms161)

As posted in this thread:

This sort of error will happen when the uid/gid does not exist in the /etc/passwd or /etc/group file inside the container.

Many ways are provided in this thread to work around this issue, please check it.
HugeCTR requires complex dependencies as shown here, if you want to build by yourself, please make sure you have all of these dependencies properly installed in your environment. We suggest using the container because it warps all of the complexities for you.

One point to clarify before answering your question: you don't need to build HugeCTR by yourself when using the Merlin train container (nvcr.io/nvidia/merlin/merlin-training:21.09) because all executables and libraries of HugeCTR are pre-installed.
For the message: groups: cannot find name for group ID 1007, as posted in this thread:

This sort of error will happen when the uid/gid does not exist in the /etc/passwd or /etc/group file inside the container.

Many ways are provided in this thread to work around this issue, please check it.
HugeCTR requires complex dependencies as shown here, if you want to build by yourself, please make sure you have all of these dependencies properly installed in your environment. We suggest using the container because it warps all of the complexities for you.

Thank you for your answer, it is very helpful to me