Allow removing the need for Cordova file and file transfer plugins

Motivation
File transfer plugin didn't have a release in a few year and the latest code has some fixes that are needed.
File plugin released version is broken and I'm using a branch.
Bottom line, Cordova maintenance is lacking at best and I can't continue to relay on their plugins.
The files that I download and use are for offline to show maps, these files are around 500Mb in total and around 100-200Mb each.
In order to stop the need to relay on Cordova pluings there are a few things that I need to be able to do:

Download large files - when downloading a large file using fetch it saves the entire file to memory and then pass it through the cordova interface to the file system, this is not great and downloading using native code straight to the file system is a lot faster and uses a lot less resources.
Open large files that are saved in the file system.
Get file name and extension

The following is more or less where I use these plugins in my code:
https://github.com/IsraelHikingMap/Site/blob/f43fd3b1f05b7f6bef45ee182486fcf08d5249cf/IsraelHiking.Web/src/application/services/file.service.ts
Search for the following:
this.fileSystemWrapper - When I read from the cache folder for example the file is large as I download to the cache a large file.
this.fileTransfer - Used to download large files
I think I managed to move all other logic to use the capacitor file, but here is where I couldn't.
I'll be happy to test stuff and help push this forward. I fear the day the Cordova plugins will stop functioning...

Thank you for your information on your use case. I will work on adding a download sample to the demo app that you can use to test and evaluate the performance for your specific scenario.
Regarding the storage of the map files, could you please let me know where they are hosted? Are they stored on Amazon S3, your own servers or some other platform? Handling 200 MB files should not be a problem, especially when reading from the system cache folder. Also, if the file format allows, you can further streamline the process by reading only certain parts of the file. For example, if the file starts with an index, you can directly access the information you need, resulting in a significant speed increase.

One of the files is hosted here: and I need to save it to the cache folder before opening it and populating the indexdb with its content:
http://israelhiking.osm.org.il/PointsOfInterest/pois.zip
Other files can be downloaded from open map tiles I think, See here:
https://data.maptiler.com/downloads/tileset/osm/asia/israel-and-palestine/
Let me know if you need any more info.

Also note, that when downloading some files there's a need to authenticate using the Authorization header, so this needs to be supported as well I guess...

I've made some updates to the demo app to allow for more examples to be added in the future. The first example demonstrates the use of fetch with range headers, which you can find here:
https://github.com/qrclip/capacitor-file-chunk/blob/main/demo/src/app/download01/download01.page.ts
I've set up a small test file on my CDN: https://cdn.qrclip.io/dev/pi.zip. It's not a zip file, but due to CORS, it was easier to rename it. It's actually a text file containing repeated digits of pi, totaling around 26MB in size. This example downloads the file to cache and reads a chunk of it. It works with large files, as long as the server supports range headers. I plan to add more examples in the future, but keep in mind that they will load the entire file into memory using XMLHttpRequest, although writing the file should be straightforward.
This current example is efficient in terms of memory usage because it writes each chunk as soon as it's ready before downloading another. There's a class called DownloadFetchRange in the demo app that you can use for testing in your project. Note that encryption is turned off within the class, but this may not be a concern for your use case.

Thanks for the info. I'll see if I can find the time to see if my server supports range requests and that I can use this plugin and the code you wrote.
Does the plugin code has the download fetch range class in it or do I need to copy it to my project?

Yes, just use this class:
https://github.com/qrclip/capacitor-file-chunk/blob/main/demo/src/app/download01/download-fetch-range.ts
It has two observables, one for progress and one for error. And the downloadFile(tUrl: string, tPath: string, tDirectory: Directory). I use bunny cdn for my cdn, amazon s3 should work too. When I've more examples I'll create a separate library, because I don't want to add to the plugin.

I've started looking into this seriously, but it seems too complicated.
The instruction in the readme on how to add this plugin are somewhat complicated.
Yes, they are written fine, but I would prefer them to be done automatically when I npm install the plugin, also there's a dependency on capacitor file system that is not mentioned.
It looks like I need to manage a server for this, is this only for previous versions or also for this version as well?
There are utilities that are used to make things easier, I shouldn't be coping them to my project, but instead they should be available in the npm package - the range download and the manager.
It's a lot of code to include to be able to simply - read a file using chunks or write one.
What I basically would like to do is update the following file in my project:
https://github.com/IsraelHikingMap/Site/blob/d43a0655b5bc8dc0e6902188acfa29f3b7f928cf/IsraelHiking.Web/src/application/services/file.service.ts
This file doesn't have a lot of references to file and file transfer plugin, but I still need to change every call there.
I don't want to bother about chunks, their size and their offset, I want it to work.
Yes, the low level API is important to have for edge cases, but I think there's a need to have a high level API as part of the plugin.
Thanks for all the hard work around this!

Yes, I understand. The only problem is that I need to do some things in the app itself to fix this big problem with Capactitor. Since I can't read the entire file into memory, it would just crash. So the app has to deal with something that there are many options for and is very backend dependent. Right now I don't have time, but it would be easy to develop something plug-and-play that works with S3. Also the localhost is just for speed, but brings a lot of ballast (encryption libraries). It's possible to use just the capacitor bridge (slower), but for smaller files under 1GB it should do. Once I've some time, I'll try to develop a plug-and-play solution that works with a simple npm installation.
Thanks a lot for your time,


Zip installers not getting nested installer type defaults

Brief description of your issue
Installers within zip files are not getting the default values for their nested installer, such as switches and error codes.
Steps to reproduce
Use zip.
Expected behavior
Works as if not zipped.
Actual behavior
Has no command line arguments passed.
Environment
CI/CD

When fixed, re-enable the E2E test InstallZipWithMsi.


Support Python 3.10

Describe the feature
Support Python 3.10.x
How to implement

Update this section
Update here too
And here

How to test

In your local environment, make sure python3 --version reports 3.10
Run the tests locally using these instructions

Additional context
https://github.com/dbeatty10/dbt-mysql/issues/142 is a similar issue for Python 3.11.

Split support for Python 3.11 into its own issue: https://github.com/dbeatty10/dbt-mysql/issues/142


fix issue #1 and added tests - no longer ignoring H0 and V0

Thanks for the merge @MaxArt2501 - do you plan on publishing a new version to npm?


Merge pull request #462 from MicrosoftDocs/master

Deploy October 2021 second publish

Docs Build status updates of commit eb3ee41:
:warning: Validation status: warnings
For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


stream: Change gather_subscriptions_helper for guest users.

I don't actually understand where to use queries_captured() context manager, i.e. which queries does it counts and do I need to use it here in tests.
I've added tests in a separate last commit; Except test_subs: Fix typos in test_previously_s.. commit all can be squashed into one. (All commits passes test_subs test individually.)

I've few more doubts which I'll ask in a review comment.
Testing:

Backend tests
Manually tested and never subscribed streams never appears in "All streams" tab.
Image of Unsubscribed Public Stream:

(Rebasing to latest master and fixing "guest user"->"guest users" typo have made above comment hidden but it's still valid https://github.com/zulip/zulip/pull/9638#discussion_r192554879)

I don't actually understand where to use queries_captured() context manager, i.e. which queries does it counts and do I need to use it here in tests.

We use it in places where we're worried about bugs resulting in our doing database queries in a loop, or otherwise worried about performance (e.g. the / endpoint).  It counts queries that happen during the scope of the context manager only.

At first, I thought we don't want to guest user to access public streams at all, may be due to misleading function name can_access_public_streams but now it's clear that we do want them in invited public streams. Once guest gets subscribed to a public stream do we want them to access history? Probable no, then we have to check for this as well.

Basically, every stream is treated like a private stream for guest users for the purposes of permissions.  So, basically, they can only be added by other people, don't get the automatic access to public streams that normal members do, etc.  Guest users get the same history_public_to_subscribers access as non-guest users, a field which for public streams is always True.

Merged, with the second commit as 8e2337509d32ba5e8bc5ea6c7f6b7cef8d7414df, thanks @shubhamdhama!


Change default upload dir for dev version

#488

Automated message from Dropbox CLA bot
@goelakash, it looks like you've already signed the Dropbox CLA. Thanks!

Merged, thanks @goelakash!  For future reference, GitHub doesn't send a notification when you updated your branch, so next time you should post a comment after you've responded to my comments.

Okay @timabbott will do that.


Deduplicate data relay VDS data

The 30-second data relay data may sometimes contain duplicate sample data, especially if we are recovering from incidents or backfilling.
We should insert some logic into the staging model to deal with this possibly-duplicated-data.

I'm thinking of using snowsql to do that.
Approach 1: a sql query to drop the samples with duplications.
Approach 2: create a new table, so the deduped data will be in the new table, rather than in-place removal as Approach 1.
What's your thoughts? @ian-r-rose

https://github.com/cagov/caldata-mdsa-caltrans-pems/blob/ed2605f906e30bb5c9c788bdd90dc46e6c09f019/transform/models/intermediate/diagnostics/int_diagnostics__real_detector_status.sql#L7-L16

Closing as completed by #304


Word isn't displayed by Hangman when using release v1.0

Issue
When in the installer, it can't find the dictionary file. The JS Dev tools (Ctrl + Shift + I) reveal that it can't find the dictionary file. The error is Error: ENOENT: no such file or directory, open 'dict.txt' (where dict.txt is the dictionary file).
How to reproduce
This occurs only when downloading v1.0 from the release page. When looking in the zip, I see that the .exe folder does NOT contain dict.txt.
Workaround
The current workaround is to go into the folder with Hangman.exe, and navigate to resources\app, and copy dict.py into where Hangman.exe is.
EDIT: 5/26/2020
This is fixed in release 1.1

I am going to add dict.txt manually into the zip file until a better solution is found. Also added a note in Known Issues.

Manually added dict.txt as a fix in 1.1; however, adding it through electron-packager would be nice.

Update: fixed with this commit for the code, and update directions with this commit. New model will be included in next release.


Add libpcre3-dev

partially to encourage people to stop statically linking against
libpcre3, related to CVE-2015-3210

Thanks @flavorjones for this update. You're right we should encourage people to dynamically link their binaries for security purposes.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_73b68334-fd07-4bf9-bf5f-5eb465f128f6/pull/1 is merged into live branch.
Auto archive content to https://github.com/OPS-E2E-PPE/docs-archive-test-target_73b68334-fd07-4bf9-bf5f-5eb465f128f6.git

Docs Build status updates of commit e19e560:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error]

Cannot sync git repo to specified commit because branch Release_Archive_master_2020-01-10-08-58-24 has been deleted or has been force pushed: fatal: couldn't find remote ref refs/heads/Release_Archive_master_2020-01-10-08-58-24

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Added usms.ac.ma and fpk.ac.ma domains

Sultan Moulay Slimane University (USMS) uses two domains for their emails, @usms.ma is mainly used by professors, while @usms.ac.ma is assigned to students.
You learn more about the university and its several institutions through this page : https://www.usms.ac.ma/en/node/111
The Polydisciplinary College of Khouribga (FPK) is an institution of USMS and mainly uses the same domains for their emails (@usms.ac.ma and @usms.ma), but sometimes professors can have an @fpk.ac.ma email.
You learn more about the college through : http://www.fpk.ac.ma/
And here's a link of a long-term (3 years) IT related course is offered : http://www.fpk.ac.ma/formation/licence-fondamentale/sciences-mathematiques-et-informatique

@itsachrafmansari First stage of verification passed, official domain for Faculté Polydisciplinaire de Khouribga is fpk.ac.ma as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@itsachrafmansari I am sorry, the domain usms.ac.ma is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Pack at https://education.github.com/discount_requests/application. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


Cannot use Extension Marketplace

Type: Performance Issue
The error is always net:ERR_TIMED_OUT.
I am sure there is a network, I already installed python and Visual studio code but I can't install any in the marketplace extension.
I also tried to download the latest version.
VS Code version: Code 1.77.3 (704ed70d4fd1c6bd6342c436f1ede30d1cff4710, 2023-04-12T09:16:02.548Z)
OS version: Windows_NT x64 10.0.19044
Modes:
Sandboxed: No

Thank you for opening the issue. Sorry to hear you are experiencing an issue with searching. It appears to be specific to VS Code editor and tools if I'm understanding correctly. It is best to reach out VS Code team at https://github.com/microsoft/vscode/issues.
-VS Marketplace Team


Add new INFO LOG message for groups which cannot be healed by SelfHeal

Changelog entry
...
Changelog category

Improvement

Additional information
...

2024-08-28T13:34:39.904906Z 1 00h01m32.931512s :BS_SELFHEAL INFO: {BSS07@self_heal.cpp:456} group can't be reassigned right now [{[80000000:3:0:0:0] Ready},{[80000000:3:0:1:0] Ready},{[80000000:3:0:2:0] Ready},{[80000000:3:1:0:0] Ready},{[80000000:3:1:1:0] Ready},{[80000000:3:1:2:0] Ready},{[80000000:3:2:0:0] NotReady},{[80000000:3:2:1:0] Ready},{[80000000:3:2:2:0] Ready Faulty Decommitted}] GroupId#<PHONE_NUMBER>


Update plugin to work with JBake 2.6.4 - fix #2

Thanks for the contribution. I tried it, but I get a lot of warnings like:
[WARNING] unable to load engine: java.lang.NoClassDefFoundError: groovy/text/TemplateEngine
and finally also an error
[ERROR] unable to instantiate ParserEngine org.jbake.parser.AsciidoctorEngine
It doesn't process the adoc files.

The warnings are due to this JBake bug.
I haven't seen that error, do you have a small test project that reproduces the error?
I did find another problem related to the orientdb dependency and I pushed an update.

You can reproduce the issues with this project: https://github.com/Blazebit/blaze-persistence
Switch the JBake version in the website project and build it.

I can't reproduce it.
$ git diff  website/pom.xml
diff --git a/website/pom.xml b/website/pom.xml
index 872b375..d43a6e1 100644
--- a/website/pom.xml
+++ b/website/pom.xml
@@ -204,7 +204,7 @@
                 <plugin>
                     <groupId>com.blazebit</groupId>
                     <artifactId>jbake-maven-plugin</artifactId>
-                    <version>1.0.0</version>
+                    <version>1.0.1-SNAPSHOT</version>
                     <dependencies>
                         <dependency>
                             <groupId>org.jbake</groupId>
$ mvn -Djdk8.home=/opt/jdk1.8 -Dmaven.test.skip=true install
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Blazebit Blaze-Persistence ......................... SUCCESS [09:16 min]
[INFO] Blazebit Persistence Checkstyle Rules .............. SUCCESS [01:43 min]
[INFO] Blazebit Persistence Parent ........................ SUCCESS [01:34 min]
[INFO] Blazebit Persistence Core Module ................... SUCCESS [  4.384 s]
[INFO] Blazebit Persistence Core API ...................... SUCCESS [01:52 min]
[INFO] Blazebit Persistence Core Parser ................... SUCCESS [01:57 min]
[INFO] Blazebit Persistence Core Impl ..................... SUCCESS [03:42 min]
[INFO] Blazebit Persistence Integrations .................. SUCCESS [  3.640 s]
[INFO] Blazebit Persistence Integration JPA Base .......... SUCCESS [ 53.773 s]
[INFO] Blazebit Persistence Integration Hibernate Base .... SUCCESS [ 49.359 s]
[INFO] Blazebit Persistence Integration Hibernate 5.2 ..... SUCCESS [01:04 min]
[INFO] Blazebit Persistence Testsuite Base ................ SUCCESS [  2.025 s]
[INFO] Blazebit Persistence Testsuite Base JPA ............ SUCCESS [ 23.282 s]
[INFO] Blazebit Persistence Testsuite Base Hibernate ...... SUCCESS [  5.574 s]
[INFO] Blazebit Persistence Core Testsuite ................ SUCCESS [10:35 min]
[INFO] Blazebit Persistence Entity-View Module ............ SUCCESS [  2.220 s]
[INFO] Blazebit Persistence Entity-View API ............... SUCCESS [ 51.951 s]
[INFO] Blazebit Persistence Entity-View Impl .............. SUCCESS [05:30 min]
[INFO] Blazebit Persistence Entity-View Testsuite ......... SUCCESS [ 17.569 s]
[INFO] Blazebit Persistence JPA-Criteria Module ........... SUCCESS [  3.736 s]
[INFO] Blazebit Persistence JPA-2 Compatibility ........... SUCCESS [ 12.327 s]
[INFO] Blazebit Persistence JPA-Criteria API .............. SUCCESS [ 15.537 s]
[INFO] Blazebit Persistence JPA-Criteria Impl ............. SUCCESS [ 59.436 s]
[INFO] Blazebit Persistence JPA-Criteria Testsuite ........ SUCCESS [ 46.083 s]
[INFO] Blazebit Persistence Integration Hibernate 4.2 ..... SUCCESS [ 27.308 s]
[INFO] Blazebit Persistence Integration Hibernate 4.3 ..... SUCCESS [ 39.302 s]
[INFO] Blazebit Persistence Integration Hibernate 5 ....... SUCCESS [ 52.747 s]
[INFO] Blazebit Persistence Integration Hibernate 5.3 ..... SUCCESS [01:30 min]
[INFO] Blazebit Persistence Integration Hibernate 5.4 ..... SUCCESS [ 32.383 s]
[INFO] Blazebit Persistence Integration Hibernate 6.0 ..... SUCCESS [ 39.548 s]
[INFO] Blazebit Persistence Integration DataNucleus ....... SUCCESS [ 26.535 s]
[INFO] Blazebit Persistence Integration DataNucleus 5.1 ... SUCCESS [ 31.416 s]
[INFO] Blazebit Persistence Integration EclipseLink ....... SUCCESS [ 35.800 s]
[INFO] Blazebit Persistence Integration OpenJPA ........... SUCCESS [ 12.981 s]
[INFO] Blazebit Persistence Integration Entity-View CDI ... SUCCESS [ 56.275 s]
[INFO] Blazebit Persistence Integration Entity-View Spring  SUCCESS [01:15 min]
[INFO] Blazebit Persistence Integration Spring-Data ....... SUCCESS [  2.534 s]
[INFO] Blazebit Persistence Integration Spring-Data Base .. SUCCESS [ 51.387 s]
[INFO] Blazebit Persistence Integration Spring-Data 1.x ... SUCCESS [ 20.109 s]
[INFO] asm-debug-all ...................................... SUCCESS [ 54.703 s]
[INFO] Blazebit Persistence Integration Spring-Data 2.0 ... SUCCESS [03:33 min]
[INFO] Blazebit Persistence Integration Spring-Data 2.1 ... SUCCESS [01:02 min]
[INFO] Blazebit Persistence Integration Spring-Data-Rest .. SUCCESS [ 45.162 s]
[INFO] Blazebit Persistence Integration Spring-Data Testsuite SUCCESS [ 58.580 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data ... SUCCESS [  2.219 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data API SUCCESS [ 33.460 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data Base SUCCESS [01:06 min]
[INFO] Blazebit Persistence Integration DeltaSpike Data Impl 1.7 SUCCESS [ 53.684 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data Impl 1.8 SUCCESS [01:29 min]
[INFO] Blazebit Persistence Integration DeltaSpike Data Rest SUCCESS [  1.084 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data REST API SUCCESS [  5.890 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data REST Impl SUCCESS [ 19.418 s]
[INFO] Blazebit Persistence Integration DeltaSpike Data Testsuite SUCCESS [01:01 min]
[INFO] Blazebit Persistence Testsuite Base DataNucleus .... SUCCESS [ 19.723 s]
[INFO] Blazebit Persistence Testsuite Base EclipseLink .... SUCCESS [ 15.320 s]
[INFO] Blazebit Persistence Testsuite Base OpenJPA ........ SUCCESS [  7.038 s]
[INFO] Blazebit Persistence Documentation ................. SUCCESS [29:53 min]
[INFO] Blazebit Persistence Website ....................... SUCCESS [  6.723 s]
[INFO] Blazebit Persistence Examples ...................... SUCCESS [  1.920 s]
[INFO] Blazebit Persistence Examples Showcase ............. SUCCESS [  2.255 s]
[INFO] Blazebit Persistence Examples Showcase Base ........ SUCCESS [ 11.936 s]
[INFO] Blazebit Persistence Examples Showcase SPI ......... SUCCESS [  9.892 s]
[INFO] Blazebit Persistence Examples Showcase Fragments ... SUCCESS [  2.649 s]
[INFO] Blazebit Persistence Examples Showcase Fragments Basic SUCCESS [ 10.885 s]
[INFO] Blazebit Persistence Examples Showcase Fragments CTE SUCCESS [  5.753 s]
[INFO] Blazebit Persistence Examples Showcase Fragments Spring Data SUCCESS [ 18.746 s]
[INFO] Blazebit Persistence Examples Showcase Runner ...... SUCCESS [  1.418 s]
[INFO] Blazebit Persistence Examples Showcase Runner CDI .. SUCCESS [ 24.068 s]
[INFO] Blazebit Persistence Examples Showcase Runner Spring SUCCESS [01:04 min]
[INFO] Blazebit Persistence Examples DeltaSpike Data REST . SUCCESS [05:24 min]
[INFO] Blazebit Persistence Examples Spring Data REST ..... SUCCESS [06:35 min]
[INFO] Blazebit Persistence Archetype Parent .............. SUCCESS [  2.371 s]
[INFO] Blazebit Persistence Archetype Core Sample ......... SUCCESS [  9.108 s]
[INFO] Blazebit Persistence Archetype Entity-View Sample .. SUCCESS [  6.046 s]
[INFO] Blazebit Persistence Archetype Spring-Data Sample .. SUCCESS [  7.315 s]
[INFO] Blazebit Persistence Archetype Spring-Boot Sample .. SUCCESS [ 13.609 s]
[INFO] Blazebit Persistence Archetype DeltaSpike Data Sample SUCCESS [ 12.669 s]
[INFO] Blazebit Persistence Archetype Java EE Sample ...... SUCCESS [  7.722 s]
[INFO] Blazebit Persistence Bill of Materials ............. SUCCESS [  0.933 s]
[INFO] Blazebit Persistence Distribution Parent ........... SUCCESS [  3.149 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:53 h
[INFO] Finished at: 2019-05-17T15:20:36-07:00
[INFO] Final Memory: 126M/424M
[INFO] ------------------------------------------------------------------------

Did I do something wrong?

@beikov  Any idea why this new plugin works for me but fails for you?

I'm gonna get back to you when I come back from my vacation in July.

Thanks! Sorry it took me so long, but I kind of forgot about this. Just tried it now and it works fine!

Just so you know what didn't work when I first tried it. I was lacking an asciidoctor dependency in the plugin which was included in jbake-core before it seems.
I had to adapt a few things for the new asciidoctor version, but apart from that, it all works now.


problem with double cygwin install to different directory

when i install double cygwin to different directory, with different root-dir and different local-package-dir for test, then i found a problem, the output of apt-cyg dist-upgrade show that local-package-dir point to the same dir
then i modified line 1485 as :
local setup=( ".\${SETUP_EXE}" -R "$(cygpath -wa /)" -B -q -n -g -l "$(cygpath -wa "$(apt-cyg-pathof cache)")" )
could resolve this problem, but why ?
sorry for my pool english.

Hmm, this problem, multiple installation of cygwin, is a little complexity.
I received this issue and read following source code of setup.exe:
https://sourceware.org/git/gitweb.cgi?p=cygwin-setup.git
It seems that setup.rc and root directory will be determined following steps:

Try to read "file://" + cwd + "/" + "setup.rc" at UserSettings.cc line 70-75.
If step 1 failed, then try to read "cygfile:///etc/setup/" + "setup.rc" at UserSettings.cc line 76-81. In this case, "cygfile:///" will be determined by cygpath(), but at this stage set_root_dir() has been never called. So read_mounts(std::string()) determines rootdir with HKEY_LOCAL_MACHINE\SOFTWARE\cygwin\setup\rootdir or HKEY_CURRENT_USER\Software\cygwin\setup\rootdir at mount.cc line 330-360. Note that HKEY_LOCAL_MACHINE takes priority.
When Cygwin setup - Choose Installation Directory dialog displayed, it will determine initial value for root directory with --root option or HKEY_LOCAL_MACHINE\SOFTWARE\cygwin\setup\rootdir or HKEY_CURRENT_USER\Software\cygwin\setup\rootdir at root.cc line 260-260. But this will be not use for read for setup.rc.

As you can see in the above steps, setup.rc seems to be read only once at step 1 or 2 before root directory is determined at step 3.
This should make some troubles in the case as multiple installation of cygwin.
Even more unfortunately the current version of setup.exe has no command line options for giving the location of setup.rc.
Perhaps, the means for giving setup.rc to setup.exe, which we can take, is following, maybe:

Prepare ./setup.rc in the same directory as setup.exe.
Rewrite HKEY_LOCAL_MACHINE\SOFTWARE\cygwin\setup\rootdir or HKEY_CURRENT_USER\Software\cygwin\setup\rootdir.

Please check above commit c3b4d00 on issue-47 branch.

I updated issue-47 branch with 305947b32ba38160a4463d304dce4965d6622771.
Can you try it?


chore: bump shiki peer dependency version

Hello !
It's time to bump shiki to last stable version 🚀
I'm actually blocked by the peer dependency mismatch between my own shiki dependency using last version and this one.

thanks!


Issue with serving modes section (documentation)

Description
Serving example (CV)
This section asks to download
curl -O https://mlrepo.djl.ai/model/cv/image_classification/ai/djl/pytorch/resnet18_embedding/0.0.1/resnet18_embedding.zip

The file has typos on serving.properties
engine=PyTorch
width=224,
height=224,
resize=256,
centerCrop=true,
normalize=true,
translatorFactory=ai.djl.modality.cv.translator.ImageFeatureExtractorFactory
option.mapLocation=true

The rows should not end with commas.
Another issue in this section, after fixing serving.properties and launching the server, I don't get the result in the same format as displayed there. I tried both in sagemaker and local (windows, ubuntu).
curl -O https://resources.djl.ai/images/kitten.jpg
curl -X POST "http://<IP_ADDRESS>:8080/predictions/resnet18_embedding" -T "kitten.jpg"

I get this output, which might be the expected output but it does not match the one from the documentation.
curl -X POST "http://<IP_ADDRESS>:8080/predictions/resnet18_embedding" -T "kitten.jpg"
[
  123,
  89,
  -41,
  59,
  107,
  3,
  -106,
  62,
  -34,

Expected output
According to documentation
[
  {
    "className": "n02124075 Egyptian cat",
    "probability": 0.5183261632919312
  },
  {
    "className": "n02123045 tabby, tabby cat",
    "probability": 0.1956063210964203
  },
  {
    "className": "n02123159 tiger cat",
    "probability": 0.1955675184726715
  },
  {
    "className": "n02123394 Persian cat",
    "probability": 0.03224767744541168
  },
  {
    "className": "n02127052 lynx, catamount",
    "probability": 0.02553771249949932
  }
]

Custom translator section
For the custom translator section, the  translator linked has been moved to "com.examples", however when using this in the package this does not get loaded an I get this error, as it uses the NoOpServing
➜ curl -X POST "http://<IP_ADDRESS>:8080/predictions/resnet" -T "kitten.jpg"
{
  "code": 400,
  "type": "TranslateException",
  "message": "Invalid NDList size: -2555936"
}

Error Message
(Paste the complete error message, including stack trace.)
After removing "package com.examples" from the java file, the server is able to pick the CustomTranslator. But i get the following error:
INFO  ModelInfo Loading model resnet on gpu(0)
INFO  ServingTranslatorFactory Using translator: CustomTranslator
INFO  WorkerPool scaling up min workers by 1 (from 0 to 1) workers. Total range is min 1 to max 2
INFO  ModelServer Initialize BOTH server with: EpollServerSocketChannel.
INFO  ModelServer BOTH API bind to: http://<IP_ADDRESS>:8080
WARN  WorkerThread Failed to predict
ai.djl.translate.TranslateException: ai.djl.engine.EngineException: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/torch/nn/modules/module/___torch_mangle_67.py", line 19, in forward
    _0 = self.fc
    _1 = self.relu
    _2 = (self.bn1).forward((self.conv1).forward(input, ), )
          ~~~~~~~~~~~~~~~~~ <--- HERE
    _3 = self.layer1
    _4 = (self.maxpool).forward((_1).forward(_2, ), )
  File "code/__torch__/torch/nn/modules/module/___torch_mangle_1.py", line 15, in forward
    _1 = self.running_mean
    _2 = self.bias
    input = torch.batch_norm(argument_1, self.weight, _2, _1, _0, False, 0.10000000000000001, 1.0000000000000001e-05, True)
            ~~~~~~~~~~~~~~~~ <--- HERE
    return input

Traceback of TorchScript, original code (most recent call last):
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/functional.py(1670): batch_norm
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py(107): forward
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(516): _slow_forward
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(530): __call__
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torchvision/models/resnet.py(200): _forward_impl
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torchvision/models/resnet.py(216): forward
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(516): _slow_forward
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(530): __call__
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/jit/__init__.py(1034): trace_module
/Users/qingla/pttest/venv/lib/python3.7/site-packages/torch/jit/__init__.py(882): trace
/Users/qingla/pttest/helloworld.py(39): <module>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument running_mean in method wrapper_CUDA__cudnn_batch_norm)

Expected Behavior
Expected output
[
  {
    "className": "n02124075 Egyptian cat",
    "probability": 0.5183261632919312
  },
  {
    "className": "n02123045 tabby, tabby cat",
    "probability": 0.1956063210964203
  },
  {
    "className": "n02123159 tiger cat",
    "probability": 0.1955675184726715
  },
  {
    "className": "n02123394 Persian cat",
    "probability": 0.03224767744541168
  },
  {
    "className": "n02127052 lynx, catamount",
    "probability": 0.02553771249949932
  }
]

Steps to reproduce
I followed the guide as in the readme.
Using: djl-serving_0.24.0-1_all.deb

@segundovolante
Thanks for reporting the issue. I have fixed the zip file.
This model is an image feature extraction model, the out is byte[], so the output is expected.
Please use a different model if you want to use image classification task.

@segundovolante
I created a PR to fix document issue: https://github.com/deepjavalibrary/djl-serving/pull/1237

Thanks, I was able also to get the custom translator working by running the inference on CPU instead of GPU.

Closing this issue as the fix is done and confirmed by the user. @segundovolante Feel free to open a new one, if you have any more questions.


use yo-yo

Big PR!
First step in a larger refactor in preparation for Friends Season 2: Episode 1 ("The One Where They Release It").
I'm using yo-yo in favor of virtual-dom for reasons, then planning on adopting a lot of electron application design patterns learned while working on WebTorrent.app, and working on updating the UI based on feedback in #151.

Can't wait for this episode to drop. Been waiting on the cliffhanger for a year!

I am pushing the button.


SAK-27778 - Chat - please use new date picker for consistency

Chat - please use new date picker for consistency

Thanks @ottenhoff, PR updated!

@ottenhoff It fails because the format is inside the properties:
msgs.custom_date_entry_format
I'll suggest to use the ISO pattern format in the client and the server ignoring the display format.

PR updated using always the ISO format.

This works well for me in EN and SV!

Yeah, better solution. It seems better if "generic looking" utilitiy classes like this should be bumped up into kernel-util or portal-util rather than duplicated since you seem to be using it across multiple PR's. Just easier to fix a bug in one place than multiple, and we'll forget if it's not done now. (Though if you want to clean all that up on a separate ticket once these are in that's okay too)

@jonespm Yeah i thought the same, but better if we get the PRs into Master first!
After all of them merged, i'll move the DateUtil class to kernel-util.
Miguel


Integration Test - Release - 2018/07/24 06:02:33

Auto create for integration test. AnotherAuthorUpdatedArticleFlow

@PRMergerTest1 : Thanks for your contribution! The author, @v-caxian, has been notified to review your proposed change.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

csidevdocs-public/IntegrationTest/BuildIntegartionTestFiles/ChangeSentToAuthor.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


sticky page tabs, and hide-able website header

addresses the sticky behavior in https://github.com/carbon-design-system/carbon-website/issues/578 . i will create a separate issue to track implementation of the tab-stacking behavior described within
todo:

[x] implement 3px top border on page tabs for a11y reasons
[x] on scroll down, add class .website-header-hidden to <body> (or any other high-level parent)
[x] on scroll back up, remove the class
[x] add some 'throttling' so the header doesn't hide/show until you've scrolled a small amount
[x] website header: test animation/transition performance, and consult https://www.sysleaf.com/js-toggle-header-on-scroll/ et al for improvements
[x] ~for some reason  opening the side nav menu jumps the scoll to the top—verify if this is happening in current site or if it is a bug introduced in this PR~ : known issue tracked in https://github.com/carbon-design-system/carbon-website/issues/1061

states and elements to verify when testing:

element: sticky global header

states:

hidden
visible

element: sticky page header

states:

hidden
visible

element: side nav

states:

hidden
visible

element: site switcher

states:

hidden
visible

element: website alert

states:

present
not present
(website alert pushes everything's sticky state down by 2.5rem)

element: page tabs

states:

page with no tabs
page with tabs

2^6 = 64 separate states the site can be in. i tested this across chrome, safari, and firefox, but would appreciate a couple extra eyes 😇

looked into making the state switching & class name propagation into a declarative implementation, but it was causing full-page re-renders, or bumped up against other architectural issues.
will implement this as-is, with this.stickyWrapper = document.getElementsByTagName('body')[0] &       this.stickyWrapper.classList.remove(this.headerHiddenClass) etc to get the desired behavior on our site state.
in the future i plan to pair with @vpicone on the gatsby theme to port over the functionality into a nice declarative way!
all that's left now is some polishing of the interaction, and transition performance

+1 to what @laurenmrice said about pages without tabs. What do we want to happen here from a design perspective?

@laurenmrice thank you, i totally missed this!!
@alisonjoseph atm this is not intended as i understood it, this is a bug because i only tested locally on pages with tabs and didn't see this behavior. it's happening because the header which contains the tabs is fixed at the same height whether or not it contains tabs. so i'll need to think about a way to make it a different height when it doesn't have a child of PageTabs ig.

For pages with no tabs, We would just want the top header to remain sticky, and for the rest of the content on that page to scroll. Example of what i mean on the IDL site. : https://www.ibm.com/design/language/elements/type-basics

omg audrey this looks really good!!!!
Aside from Lauren's point I'm ready to approve.

@laurenmrice @jeanservaas: re: alison's comment, this PR currently includes all fixes pointed out above, in this thread, aside from the global site header being visible on pages without tabs. as she said, unfortunately that involves a bit of a change to how we are architecturally passing around data about the content of the current page, and it would be best addressed after we merge in the big re-architecting of site packages/themes that vince has been leading.
(i also personally think it's kinda weird/inconsistent behavior to have it the global header to be sticky on some pages and not others. but this can be a separate conversation in a separate issue/PR for implementing the behavior 😇)

🎉


Realtime not working while selecting child device/asset flag from configuration page

Steps to reproduce:

Select device group
select indoor map
provide assetType
enable select child device/asset
Error: Realtime device movement is not happening on map.

sorry issue created in wrong repo


X11 HiDPI scaling algorithm is broken

As a well-known fact, X11 locks the DPI to 96 and the current code gets incorrect screen size, resulting in the scale factor always being 1.
The defacto standard is to use Xft.dpi to get font DPI.
Disclaimer: I'm currently working on this, this is a tracking issue.

Hi. I was just burnt by this problem, and I've had a go at fixing it. PR: https://github.com/tomaka/winit/pull/198

@stevebob it sounds like you're fixing another problem. This issue is about hidpi_factor being virtually constant, with how most desktop environments sets the DPI.

Ah, I think I misinterpreted your comment. I set my dpi using xrandr and I observe it changing (with hidpi_factor), but when querying a window's size in points it instead reported its size in pixels, causing the result when querying it's size in pixels to be incorrect. It sounded close enough to this issue that I thought it might be related.

As a well-known fact, X11 locks the DPI to 96 and the current code gets incorrect screen size, resulting in the scale factor always being 1.

This is not well-known to me. If xrandr can freely set the DPI (and @stevebob's reports and my own experience suggest that it can), then it's manifestly the case that X11 does not lock the DPI to anything.

The defacto standard is to use Xft.dpi to get font DPI.

Can you support this claim?

About 96 dpi: https://bugs.freedesktop.org/show_bug.cgi?id=23705 https://bugs.freedesktop.org/show_bug.cgi?id=41115
The term "lock" is somewhat absurd, but I meant that the default is always 96 instead of the autodetect which is the behavior long ago.
Qt uses font DPI, and GNOME also sets that (although GTK itself uses a separate environment variable).

Upping this issue (as I just hit it: most of my system picks up the config in xft.dpi but winit does not).
Would it make sense to use xft.dpi if the value is set, and default to dpi reported by XRandr otherwise?

Few thoughts -

Xft.dpi is global and cannot support mixed DPI environments
If an application wants to use Xft.dpi, is winit really the right place to read that value?
winit should provide access to per-monitor scale factor. If you've got one hidpi monitor and one regular, you'd like your applications to appear reasonable on both (versus having huge/tiny text on one or the other).
Font rendering results (especially with subpixel aa, hinting) are best when physical DPI is used
xrandr provides physical DPI, and it provides it for each monitor.

Agreed that xft.dpi cannot properly handle mixed dpi environments.
But it looks like (cf the previously linked issues on Xorg's bugtracker), that DPI with X11 is a sad story anyway.
Personally, I'm still trying to figure out if I have a way to have XRandR correctly advertize the DPI as being 192 rather than its default 96 other than manually running xrandr --dpi 192 (and even then, it decides to set it to 192x193 for some reason).

@vberger AFAIK, the typical way to compute DPI using xrandr is to take the screen size in millimeters, divide by 25.4, and then divide the resolution by that value. For instance, my main display looks like this:
DVI-I-1 connected 2560x1440+1080+176 (normal left inverted right x axis y axis) 597mm x 336mm

Horizontal: 2560 / (597 / 25.4) = 109
Vertical: 1440 / (336 / 25.4) = 109

When the X server supports RANDR 1.2 or later, the DPI information reported by the core protocol is unreliable. DPI calculations should be done on a per-output (or per-Monitor with RANDR 1.5 or later) basis. This is e.g. what Qt does when QT_AUTO_SCREEN_SCALE_FACTOR=1 is set. My understanding (following this) is that the approach used in Qt is the following:

compute per-output DPI from the output's resolution and physical size;
compute a reference DPI using Xft.dpi falling back to the core DPI if Xft.dpi is unset;
compute per-output scaling as (monitor DPI)/(primary DPI) * (reference DPI)/96;

The individual window scaling then adapts to the output the window is on. As a frequent user of mixed-DPI environments, I find this approach works quite well, and I would recommend it for adoption everywhere.

@Oblomov thanks for the info! I also read your post.This looks like it will be easy to fix now.


Add support for livestreams

Hi, thanks for this plugin!
Here's a small change that adds support for livestreams.

The build is failing because of wrong node version so not related

@andreasgan can you please edit the pull request so that maintainers can push to the branch? I'm not allowed to fix up your commits because that box isn't checked
Or, please squash the last commit onto the first commit, which is what I was trying to do.

Because the fork is in an organization, I can't allow you to edit so I had to add you as a collaborator to the fork.
https://github.com/isaacs/github/issues/1681

But I did the squash

@andreasgan thanks! I did not realize that problem with organizations and PR's. Learned something new. But since you squashed, that's all I needed. You can remove my access to the org.
@yokuze this looks fine to me. Can you take a quick look and see if you have any concerns?

@yokuze this looks fine to me. Can you take a quick look and see if you have any concerns?

@jthomerson This looks fine to me. It's been a while since I've looked at live stream support in Video.js so I can't definitively say this is all that's needed, but I don't have any concerns with this change.

@andreasgan published in<EMAIL_ADDRESS>


No option for custom template

SwiftKit Environment
It would be great if I could provide and select from multiple templates.

Hey @wildthink
Thanks for opening an issue.
To get you right you want to pass in a different Repository URL to clone the Template Project from?

I think that would do the trick. I was thinking that one might want to offer different configurations and/or pre-populate with some starter code or such.
Thanks.

Hey @wildthink
Please check out the latest Release 1.2.5 which now includes the --repository parameter to specify the location where to clone the SwiftKit repository from.

Long parameter
Short parameter
Description

--repository
-r
The SwiftKit template repository url 🌎

$ swiftkit new MyAwesomeKit --repository https://github.com/Custom/MyCustomSwiftKit.git


[SelectField] Add maxHeight prop to pass down to DropDownMenu

Merge into master instead of 0.15-stable
PR replacing #4643

Looks like this is fixing a regression. The maxHeight option disappeared from the Long Example 😱 .

I have added it back with https://github.com/callemall/material-ui/commit/1801e67af05791f95749bb2b0767281fa7637251.


release v1.4.0 indicates incorrect version v1.2.0-dev

https://github.com/bufbuild/connect-go/blob/v1.4.0/connect.go#L35
Not sure what all this impacts. But I do see that the user agent used from v1.4.0 of this library indicates the incorrect version.
I suspect there is an issue with the release process, where this constant is not getting updated.

Ugh, you're right. Fixing it now :/


Monk's Sound Enhancement Tracks too dark

Monk's sound enhancement adds a way to view tracks in a playlist and dark mode makes the test hard to read.

I don't want to make a huge stink about this, but I was hoping to check in on an update on this? it sorta just needs the font changed to like - white or something.

Hi, no worries, you're always welcome to bump old issues.
While a picture is worth more than a thousand worths, I would love to have reproduction steps for producing the picture though.

fix'd for next release

  fix'd for next release

THANK YOU


Summary
UserDialogs.Init(this);
OR Xamarin.Forms - UserDialogs.Init(() => this);

I am not sure I see difference between the two. Is there a reason the docs show a different Init pattern on XF?
API Changes
None
Intended Use Case
Simplification

It is just different ways of getting at the top activity.  I don't take requests for doc enhancements.  Feel free to send a PR in areas you can improve


Fixup: handle console thread termination

During error cases in runTest console thread
isn't terminating and causes test not to exit and
hang forever, lets handle console termination
as part of runTest itself instead in teardown.
Signed-off-by: Satheesh Rajendran<EMAIL_ADDRESS>

The unittest docs say about tearDown:

Method called immediately after the test method has been called and the result recorded. This is called even if the test method raised an exception...

So I'm not sure why this makes a difference, tearDown should be called regardless. What is happening in runTest that is preventing tearDown from being called?

Even i feel the same but am guessing if console thread started in setUp it works fine but if started in runTest, it gets prevented being called by teadDown, not sure why...

I re-read the patch and see that your putting basically the whole test in a try/except, so that's what is probably making the difference but it seems a bit heavy handed. Do you have an example of a failure where we could see exactly where it's getting stuck?

I re-read the patch and see that your putting basically the whole test in a try/except, so that's what is probably making the difference but it seems a bit heavy handed. Do you have an example of a failure where we could see exactly where it's getting stuck?

Just that we want to make sure certain events before the exception is processed in runTest/framework, not sure if that will be heavy though.
Issue can easily be reproduced by failing any test step E:g:- in below test patch, I just made a command intentionally wrong to fail, this will hang the test for ever.
git diff
diff --git a/testcases/InstallUpstreamKernel.py b/testcases/InstallUpstreamKernel.py
index 2686aae..aad68d4 100644
--- a/testcases/InstallUpstreamKernel.py
+++ b/testcases/InstallUpstreamKernel.py
@@ -84,7 +84,7 @@ class InstallUpstreamKernel(unittest.TestCase):
         linux_path = os.path.join(self.home, "linux")
         con.run_command("[ -d %s ] || mkdir -p %s" % (self.home, self.home))
         con.run_command("if [ -d %s ];then rm -rf %s;fi" % (linux_path, linux_path))
-        con.run_command("cd %s && git clone --depth 1  %s -b %s linux" % (self.home, self.repo, self.branch), timeout=self.host_cmd_timeout)
+        con.run_command("cd %s && git clon --depth 1  %s -b %s linux" % (self.home, self.repo, self.branch), timeout=self.host_cmd_timeout)
         con.run_command("cd %s" % linux_path)
         if self.patch:
             patch_file = self.patch.split("/")[-1]

After reading the docs a bit more this appears to be because the CommandFailed exception raised by, for example, con.run_command("cd %s && git clone...") is treated as an error rather than a failure, and so tearDown() isn't called. I was hoping unittest would have a nicer way of catching certain exceptions but as far as I can see we could only do something like
try:
    con.run_command("foo")
except:
    self.fail()

in order to make sure tearDown is called, which is basically what this patch does anyway.
What we probably do need is a way of cleaning up any stale connections etc in run_tests so every test doesn't hit this, I'll see if there's something we can do.

What we probably do need is a way of cleaning up any stale connections etc in run_tests so every test doesn't hit this, I'll see if there's something we can do.

I haven't got to this yet but this is looking fine, I'll merge.


Disabled users showing with 'unknown' status instead of disabled

interesting, I'll check it out

@jirwin  - fixed this in provisioning PR, user is SUSPENDED instead of INACTIVE for disabled state


Portainer fails to pull images with authenticated registry and breaks deployments

Bug description
When using private images, deployments via webhook fail with "pull access denied for mycontainer, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"
Expected behavior
Redeployment via webhook should succees is properly authenticated
Portainer Logs
This is the response I receive
{
  "message": "Failed to update the stack",
  "details": "failed to deploy a docker compose stack 25: failed to pull images of the stack:  IMAGE_NAME Pulling  IMAGE_NAME Error Error response from daemon: pull access denied for USERNAME/IMAGE_NAME, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"
}

Steps to reproduce the issue:

Setup portainer CE on host A
Setup portainer environment on host B (docker standalone)
Add docker hub as an authenticated registry in portainer (expectation is that this will always be used to pull anything from docker hub)
Login to docker hub on both hosts
Create a git stack on host B environment and setup a webhook
Make changes to the repository and push, then call webhook

Result: webhook will fail with message below:
{
  "message": "Failed to update the stack",
  "details": "failed to deploy a docker compose stack 25: failed to pull images of the stack:  IMAGE_NAME Pulling  IMAGE_NAME Error Error response from daemon: pull access denied for USERNAME/IMAGE_NAME, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"
}

Technical details:

Portainer version: 2.18
Docker version: 20.10.12
Kubernetes version (managed by Portainer):
Platform: Ubuntu 20.04
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):

version: "3.5"
services:
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:latest
    restart: always
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./.portainer:/data

Browser: chrome
Use Case (delete as appropriate): Using Portainer at Home.
Have you reviewed our technical documentation and knowledge base? Yes

Additional context
Looks like portainer server always tries to pull the image locally (which makes no sense if it's trying to deploy remotely), and somehow never uses the authenticated docker hub registry, it uses the anonymous one and fails. Solution would be to just leave the remote agent deal with this, but there is no option to get around this because the local pull is hardcoded. I tested the changes below with a custom image and it just works.

I've just been attempting to replicate this myself on a fresh CE 2.18.3 installation but am not running into the same issue you are. The steps I took:

Deploy a new Portainer CE server installation (using your example compose file)
Add a new Agent (2.18.3) on a remote server from the new CE server
Add my DockerHub credentials on the CE server
Create a Git repository with a docker-compose.yml referencing a private image in my DockerHub account
Connect to the remote server from within Portainer
Create a new Git stack from the above-created Git repository, enabling Automatic Updates in webhook mode and copying the webhook
Confirm the stack deployed successfully
Made a change to the docker-compose.yml in the Git repository (in this case, changing the image tag but not the image to force a pull)
Triggered the webhook to update the stack
Confirm the stack redeployed with the new image tag (which it did)

I also then made a different change to the docker-compose.yml file just to make sure (in this case, I published an additional port) and triggered the webhook again, and once again the redeployment was successful, no errors.
One thing to note about my steps above is that I did not do a docker login on either the server or the agent at any point - this shouldn't be necessary to do when using Portainer, so I left it out.
Are you perhaps able to give us some more information on your setup, both on the server and agent side, as well as any specifics you're able to share about your setup that might relate (networking, firewall, etc). Also, if there's anything special about your private image that you're able to share, that might help.

I ran docker login after the issue started, so this happened without it as well.
The servers are part of a tailscale network and can both reach each other
There's nothing special about the image, it's a basic go image built from the following Dockerfile
FROM golang:alpine as builder
ENV GO111MODULE on

RUN mkdir -p /build

WORKDIR /build
COPY go.mod go.sum ./
RUN apk add git && go mod download
ADD . .

# Build
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o /app .

# FROM alpine
FROM gcr.io/distroless/base
COPY --from=builder /app /app

WORKDIR /

EXPOSE 8080

CMD ["./app"]

I'm wondering though, why do we need to pull an image on the server when we want to deploy somewhere else? It's not like we're shipping the image to the agent.

I'm wondering though, why do we need to pull an image on the server when we want to deploy somewhere else? It's not like we're shipping the image to the agent.

We don't - I didn't during my attempted replication of your issue. I've just checked to confirm, the image I deployed on the remote environment is not on my Portainer Server environment at all.

@figassis Would you be able to provide your code changes via a pull request and Portainer server and agent log files? Thanks.

@kamlad sorry for the late reply, sidetracked at work. I'll submit as soon as I clean it up a bit

We were hit by this problem after not resisting the upgrade temptation.
Here is some information that would have helped us, in case it can be useful to others...
Our workaround was to go back to an earlier version (we used both 2.15.1 and 2.16.0).
Note that just downgrading was not enough and the webhooks had to be created again as the problem seems to persist in the database (BTW, we redeployed the impacted stacks from scratch too, just in case).
This being said, we saw that there were many other similar issues that did not help but we missed this one (#7792) before using the workaround... maybe this is a better solution. Please confirm if you follow that path 😃
We would prefer if we could use a more recent version!

After a fresh install of a 2.18.4 (latest) ce server, same problem here, steps:
1- Setup server (latest version)
2- Setup client (docker standlone/edge agent standard with latest version)
3- Setup private registry

4- create a edge group
5- create a edge stack using a private docker hub, ex:
services:
  smartgrid:
        image: "docker.io/jonasfoyfth/smartgrid:latest"

smartgrid Pulling smartgrid Error Error response from daemon: pull access denied for jonasfoyfth/smartgrid, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Otherside, if i pull direct (connect and go to images), works fine:

And compose of edge stack works because image is deployed:

I can confirm this on a clean server (just installed from scratch). Running 2.19.1 CE.

I can confirm this on a clean server (just installed from scratch). Running 2.19.1 CE.

Same with me on 2.19.4 BE, swarm with 5 nodes. It's been happening for a while. Some times updating the stack with the "pull image" toggle turned on will work, but most of the time I receive a "Cannot find image x." This is pulling from a private ECR repo.

As the problem has been there for a while without much visible progress, can someone from the team tell us if it has been identified and if there a good chance it will be fixed in the next release?
Thanks in advance

I can confirm the same problem after upgrading Portainer to 2.20.3 BE.
We have GitLab server with mix of public and private projects with Docker registries. Credentials (username nad PAT) with access to all projects and registeries are configured in Portainer Registeries UI. I can see and pull both public and private images trough Portainer Images UI. And via docker login the same credentials are saved on all Swarm nodes where I can docker pull both public and private images.
When I try to Update a Service in Portainer UI with option "Re-pull image" checked, then it does not pull. The same issue happens when using Portainer API PUT /endpoints/{id}/forceupdateservice. Both method worked fine in older Portainer versions. When it happens, syslog contains errors like:
swarm1 dockerd: level=info msg="Attempting next endpoint for pull after error: Head \"https://docker-registry.xxx.xx/v2/xx/xx/manifests/master\": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.xxx.xx/help/user/profile/account/two_factor_authentication#troubleshooting"
swarm1 dockerd: level=error msg="pulling image failed" error="Head \"https://docker-registry.xxx.xx/v2/xx/xx/manifests/master\": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.xxx.xx/help/user/profile/account/two_factor_authentication#troubleshooting" module=node/agent/taskmanager node.id=xx service.id=xx task.id=xx

In that case Portainer fails to start Tasks with status Rejected and Error message: "No such image: ...". And they are getting rejected until they start on some Node with image from previous deployment. So instead of redeploy with new images it just restarts service using old images.
Partial workaround is to stop and start a Stack. In that case it pulls images fine. However only to nodes where it just started new Tasks. On other Nodes are left outdated images which may be used if Service for some reason migrates to other Nodes.
Only safe workaround is to manually pull images on all Swarm Nodes before updating a service with Portainer.

any updates on this? we used 2.20.1 CE and we faices the same issue.
for the registry image:

registry:2

I can confirm the same problem after upgrading Portainer to 2.20.3 BE.
We have GitLab server with mix of public and private projects with Docker registries. Credentials (username nad PAT) with access to all projects and registeries are configured in Portainer Registeries UI. I can see and pull both public and private images trough Portainer Images UI. And via docker login the same credentials are saved on all Swarm nodes where I can docker pull both public and private images.
When I try to Update a Service in Portainer UI with option "Re-pull image" checked, then it does not pull private images. The same issue happens when using Portainer API PUT /endpoints/{id}/forceupdateservice. Both methods worked fine in older Portainer versions. When it happens, syslog contains errors like:
swarm1 dockerd: level=info msg="Attempting next endpoint for pull after error: Head \"https://docker-registry.xxx.xx/v2/xx/xx/manifests/master\": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.xxx.xx/help/user/profile/account/two_factor_authentication#troubleshooting"
swarm1 dockerd: level=error msg="pulling image failed" error="Head \"https://docker-registry.xxx.xx/v2/xx/xx/manifests/master\": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.xxx.xx/help/user/profile/account/two_factor_authentication#troubleshooting" module=node/agent/taskmanager node.id=xx service.id=xx task.id=xx

In that case Portainer fails to start Tasks with status Rejected and Error message: "No such image: ...". And they are getting rejected until they start on some Node with image from previous deployment. So instead of redeploy with new images it just restarts service using old images.
Partial workaround is to stop and start a Stack. In that case it pulls images fine. However only to nodes where it just started new Tasks. On other Nodes are left outdated images which may be used if Service for some reason migrates to other Nodes.
Only safe workaround is to manually pull images on all Swarm Nodes before updating a service with Portainer.


Add makeContextAware for CompletableFuture

Motivation:

RequestContext is not propagate when users use CompletableFuture asynchronously.

Modifications:

Add RequestContext#makeContextAware for CompletableFuture.

Result:

Fixes #365

Current coverage is 65.12% (diff: 66.66%)

No coverage report found for master at 970f6c0.

Powered by Codecov. Last update 970f6c0...975e552

@trustin Updated~

fixed and squashed~

Great job. Thanks, @jongyeol !


meybe using bit fields rather than arithmetic instruction?

In opcode or in other places (float etc)
http://www.tutorialspoint.com/cprogramming/c_bit_fields.htm

bit fields format can not be used for opcodes, because we have different formats of arguments packing depends of particular opcode.


Unit test fail in AzureVaultKeyGeneratorTest when build with linux file system

This unit test exceptionThrownIfDisallowedCharactersUsedInVaultId expects java.nio.charset.UnsupportedCharsetException to be thrown however before that, Paths.get(filename) would attempt to use FileSystems.getDefault() to validate the file path, and this might return different outcomes depending on the OS system Tessera is being built in. In OS such as linux this method in Paths library will throw a different exception (java.nio.file.InvalidPathException) - causing the unit test to fail

Fixed by #527

Fixed in 0.7.3 release


Creating multi-paragraph amendments

@emanuelschuetze Editing the amendment should now also be supported.

@tsiegleauq Please review later (after merge)


Correct distinction between imported modules and all modules

This PR fixes an issue that has shown up in https://github.com/runtimeverification/pyk/pull/217 due to duplicate productions for <generatedCounter> cell label. While we probably should not be duplicating that production, at least we can fix this by being more disciplined about how we treat modules in a K Definition for now. I've confirmed locally that this allows us to update to latest K.

KDefinition class is modified in the following ways:

The field modules is renamed to all_modules
Cached property module_names is renamed to all_module_names.
Cached property module_dict is added for fast lookups of modules by names (and used in method module(str) -> KFlatModule).
Cached property module_names is added, which returns only the names of modules imported by the main module of the definition.
Cached property modules is added, which returns only the modules imported by the main module of the definition.
Most lookup methods/properties about the definition are modified to work over KDefinition.modules instead of KDefinition.all_modules.
A test is added which shows the difference between KDefinition.modules and KDefinition.all_modules.

KPrint class is modified in the following ways:

When printing out definitions, we use KDefinition.all_modules. This has 2 effects, (i) that modules which are not imported into the main module are printed as well (desired for code generation purposes), and (ii) that definitions which use a requires "..." statement to import other modules, but don't actually have them in context, print out successfully.
When building the unparsing symbol-table, we use KDefinition.all_modules. This is because there are several symbols defined in kast.md that are needed for unparsing (like #EmptyK), which are not included in the definition usually.

I'd like to note that one change which was tricky for me to find was that in many places in KDefinition, we were relying on the default iterator for KDefinition to return KDefinition.modules. This ended up being hard for me to find the breakage, and I had to look through all occurances of self in the outer.py to determine if it needed to be changed to self.modules or self.all_modules. I'd like to be more explicit in the future, and avoid these types of ambiguities. It seems to me that having a default iterator is useful when you want to pass the object into some external algorithm that relies on that, but in our own codebase, we always should know the method/property/field name we want to call instead.

I want to use all_modules and modules instead of modules and imported_modules (which was my first solution), because I think by default most people will want just the imported modules, not all the modules.


Shaders load but don't work

Windows Terminal version
1.13.10336.0 Preview
Windows build number
10.0.19042.1586
Other Software
No response
Steps to reproduce
I have downloaded https://github.com/microsoft/terminal/blob/main/samples/PixelShaders/Rasterbars.hlsl as test.hlsl file and placed it in *C:\TMP* directory.

Ah, shaders don't work with the atlas engine quite yet. This is tracked as a bullet point in #9999 but I'll leave this open to track it fully. Thanks!

@zadjii-msft I will try without Atlas engine.

@UltraStudioLTD Did you set your antialiasingMode to cleartype? If so: That doesn't seem to work at the moment.
Otherwise, could you update or post your settings.json?

@lhecker It didn't work. I updated setting.json file posted to match currently used.

@UltraStudioLTD You have "antialiasingMode": "cleartype", in your settings. Remove that line and it will work. 🙂

@lhecker It didn't work.

By the way here is C:\TMP\test.hlsl file
// A minimal pixel shader that shows some raster bars

// The terminal graphics as a texture
Texture2D shaderTexture;
SamplerState samplerState;

// Terminal settings such as the resolution of the texture
cbuffer PixelShaderSettings {
  // The number of seconds since the pixel shader was enabled
  float  Time;
  // UI Scale
  float  Scale;
  // Resolution of the shaderTexture
  float2 Resolution;
  // Background color as rgba
  float4 Background;
};

// A pixel shader is a program that given a texture coordinate (tex) produces a color.
// tex is an x,y tuple that ranges from 0,0 (top left) to 1,1 (bottom right).
// Just ignore the pos parameter.
float4 main(float4 pos : SV_POSITION, float2 tex : TEXCOORD) : SV_TARGET
{
    // Read the color value at the current texture coordinate (tex)
    //  float4 is tuple of 4 floats, rgba
    float4 color = shaderTexture.Sample(samplerState, tex);

    // Read the color value at some offset, will be used as shadow
    float4 ocolor = shaderTexture.Sample(samplerState, tex+2.0*Scale*float2(-1.0, -1.0)/Resolution.y);

    // Thickness of raster
    const float thickness = 0.1;

    float ny = floor(tex.y/thickness);
    float my = tex.y%thickness;
    const float pi = 3.141592654;

    // ny is used to compute the rasterbar base color
    float cola = ny*2.0*pi;
    float3 col = 0.75+0.25*float3(sin(cola*0.111), sin(cola*0.222), sin(cola*0.333));

    // my is used to compute the rasterbar brightness
    //  smoothstep is a great little function: https://en.wikipedia.org/wiki/Smoothstep
    float brightness = 1.0-smoothstep(0.0, thickness*0.5, abs(my - 0.5*thickness));

    float3 rasterColor = col*brightness;

    // lerp(x, y, a) is another very useful function: https://en.wikipedia.org/wiki/Linear_interpolation
    float3 final = rasterColor;
    // Create the drop shadow of the terminal graphics
    //  .w is the alpha channel, 0 is fully transparent and 1 is fully opaque
    final = lerp(final, float(0.0), ocolor.w);
    // Draw the terminal graphics
    final = lerp(final, color.xyz, color.w);

    // Return the final color, set alpha to 1 (ie opaque)
    return float4(final, 1.0);
}

Are you 100% sure? 😅

You did reload the tab after disabling AtlasEngine / the experimental text renderer right?

I even reopened it

I solved issue by switching to Software Rendering instead of Hardware Rendering

This is interesting actually. Can you tell us more about the hardware you use? What's your CPU and GPU? Do you use Windows Terminal via a Virtual Machine, Remote Desktop, or some other mechanism?
This shouldn't happen and we should try and fix your issue. 🙂
If you have the time and would like to investigate (really just in case), could you enable hardware rendering again and download DebugView and see if any logs come up when you start Windows Terminal ("WT")? If you go into your start menu and search for dxcpl.exe you can find a tool that can enable DirectX debugging for WT. Here's how to use it:

If you're done debugging, don't forget to "Clear" the "Executable List". You need to click "Apply" (or "OK") in the primary window in order for the change to take effect.
All of that is just in case you have the time and interest in investigating this of course. 🙂

@lhecker I will upload details later when I will have time. For now I will enjoy CRT shader that I always wanted (since cool-retro-term)

@lhecker
OK, here are the details:

My CPU is Intel® Core™2 Duo CPU E6750 @ 2.66GHz
My GPU is NVIDIA GeForce 8600 GT or Alternative Technical Specifications (UnOfficial)
I don't use Windows Terminal via any remote or virtual means.

WTPreviewHardwareRendering.LOG

And as I mentioned HLSL rendering works only with Software Rendering with both Preview and Stable versions

Found it:
00000232	1.32877946	[8336] D3D11 ERROR: ID3D11Device::CreateVertexShader: Shader must be vs_4_0. Shader version provided: vs_5_0 [ STATE_CREATION ERROR #167: CREATEVERTEXSHADER_INVALIDSHADERTYPE]

Your GPU only supports feature level 10.0 for DirectX, which

@UltraStudioLTD I've fixed the issue and will submit a pull request soon. Thank you for your great help! The log file you sent me immediately helped me find and fix the issue. 🙂
Unfortunately we're likely "soon" (next year?) dropping support for hardware-acceleration for hardware below feature level 10_1 and your GPU only has 10_0. I know it sounds presumptuous, but I would suggest seeing if you can get a GPU with DirectX 10_1 or 11_0 support somehow. For NVIDIA GPUs these include the GT 200 (10_1), GT 400 (11_0), or newer cards.
Of course Windows Terminal will continue to work on your hardware, but the performance will be rather poor as it will fall back to a software renderer called WARP. It won't be as poor as with the "software rendering" you enabled, but it'll still be significantly worse than an actual GPU.

@lhecker Well, by looks it would be better to upgrade/replace my entire PC since every card and board, including motherboard is very old and not compatible with new cards 😭. Well, thanks for fixing issue 👍


Water Sensors: ERU

The site has been offline for over a month. Unconfirmed connection with issue #202. Can we clarify what is/was going on with this site?

i was going over the data that was downloaded for this farm. i notice all of the sensors were not reading for Cover 1 when they downloaded on 08/06. should send back to check on it.

Letting them know to check on Cover 1


BCH Liquid Error

I catch this bug:
arbitrader_1  | 2019-04-17 00:00:00.752  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Summary: [Long/Short Exchanges] [Pair] [Current Spread] -> [Entry Spread Target]
arbitrader_1  | 2019-04-17 00:00:00.898  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Quoine/Kraken BTC/USD 0.00001 -> 0.0071
arbitrader_1  | 2019-04-17 00:12:14.742  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : ***** ENTRY *****
arbitrader_1  | 2019-04-17 00:12:16.232  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Updated account balances: Quoine $100.02 + Kraken $43.31 = $143.33
arbitrader_1  | 2019-04-17 00:12:16.380  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Entry spread: 0.007233
arbitrader_1  | 2019-04-17 00:12:16.526  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Exit spread target: -0.004967
arbitrader_1  | 2019-04-17 00:12:16.674  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Long entry: Quoine BCH/USD 0.09119497 @ 318.00000 (0.00000 slip) = $29.0000004600000
arbitrader_1  | 2019-04-17 00:12:16.820  INFO 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Short entry: Kraken BCH/USD 0.09054012 @ 320.300000 (0.000000 slip) = $29.00000043600000
arbitrader_1  | 2019-04-17 00:12:17.496 ERROR 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Exchange returned an error executing trade!
arbitrader_1  |
arbitrader_1  | org.knowm.xchange.exceptions.ExchangeException: {"errors":{"order":["product_disabled"]}}
arbitrader_1  |  at org.knowm.xchange.quoine.service.QuoineBaseService.handleHttpError(QuoineBaseService.java:51)
arbitrader_1  |  at org.knowm.xchange.quoine.service.QuoineTradeServiceRaw.placeLimitOrder(QuoineTradeServiceRaw.java:71)
arbitrader_1  |  at org.knowm.xchange.quoine.service.QuoineTradeService.placeLimitOrder(QuoineTradeService.java:64)
arbitrader_1  |  at com.r307.arbitrader.service.TradingService.executeOrderPair(TradingService.java:642)
arbitrader_1  |  at com.r307.arbitrader.service.TradingService.lambda$null$14(TradingService.java:390)
arbitrader_1  |  at java.util.ArrayList.forEach(ArrayList.java:1257)
arbitrader_1  |  at com.r307.arbitrader.service.TradingService.lambda$null$15(TradingService.java:292)
arbitrader_1  |  at java.util.ArrayList.forEach(ArrayList.java:1257)
arbitrader_1  |  at com.r307.arbitrader.service.TradingService.lambda$tick$16(TradingService.java:284)
arbitrader_1  |  at java.util.ArrayList.forEach(ArrayList.java:1257)
arbitrader_1  |  at com.r307.arbitrader.service.TradingService.tick(TradingService.java:284)
arbitrader_1  |  at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
arbitrader_1  |  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
arbitrader_1  |  at java.lang.reflect.Method.invoke(Method.java:498)
arbitrader_1  |  at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
arbitrader_1  |  at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
arbitrader_1  |  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
arbitrader_1  |  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
arbitrader_1  |  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
arbitrader_1  |  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
arbitrader_1  |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
arbitrader_1  |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
arbitrader_1  |  at java.lang.Thread.run(Thread.java:748)
arbitrader_1  | Caused by: si.mazi.rescu.HttpStatusIOException: HTTP status code was not OK: 422
arbitrader_1  |  at si.mazi.rescu.ResponseReader.read(ResponseReader.java:104)
arbitrader_1  |  at si.mazi.rescu.RestInvocationHandler.mapInvocationResult(RestInvocationHandler.java:175)
arbitrader_1  |  at si.mazi.rescu.RestInvocationHandler.receiveAndMap(RestInvocationHandler.java:163)
arbitrader_1  |  at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java:119)
arbitrader_1  |  at com.sun.proxy.$Proxy47.placeOrder(Unknown Source)
arbitrader_1  |  at org.knowm.xchange.quoine.service.QuoineTradeServiceRaw.placeLimitOrder(QuoineTradeServiceRaw.java:65)
arbitrader_1  |  ... 21 common frames omitted
arbitrader_1  |
arbitrader_1  | 2019-04-17 00:12:17.705  WARN 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Polling exchanges took 7865 ms
arbitrader_1  | 2019-04-17 00:12:17.867 ERROR 1 --- [   scheduling-1] c.r.arbitrader.service.TradingService    : Exiting immediately to avoid erroneous trades.
arbitrader_arbitrader_1 exited with code 1
I think that quoine/liquid returns "product_disabled" when some pair does not work for trading.
I am not 100% sure but if we can test this, we can check somehow if BCH is disabled before trading. On my log actually no trade happened. Both kaken / liquid does not enter trade.
Only file was created in the directory and programme exited.
"$ cat .arbitrader/arbitrader-state.json
{"longTrade":{"exchange":"Quoine","orderId":null,"volume":0.09119497,"entry":318.00000},"shortTrade":{"exchange":"Kraken","orderId":null,"volume":0.09054012,"entry":320.300000},"currencyPair":"BCH/USD","exitTarget":-0.004967}"

This is a duplicate of #51 . I'm trying to work out a good way to solve this problem but it's tricky.

just FYI, for liquid, they have a daily maintenance period where the APIs for trading returns product_disabled.
You might be able to check the time and make sure its not going under maintenance
https://help.liquid.com/en/articles/2531810-liquid-daily-scheduled-maintenance

@yoheioka It would be fairly easy to add a configuration option to just black out trading for an exchange during a period of time, although it would be very important to make sure your clock is accurate! Thanks for the suggestion.


[SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version

The current Spark Streaming kinesis connector references a quite old version 1.9.40 of the AWS Java SDK (1.10.40 is current). Numerous AWS features including Kinesis Firehose are unavailable in 1.9. Those two versions of the AWS SDK in turn require conflicting versions of Jackson (2.4.4 and 2.5.3 respectively) such that one cannot include the current AWS SDK in a project that also uses the Spark Streaming Kinesis ASL.

Can one of the admins verify this patch?

Have you tested the current kinesis and works with the new API? Looking online it seems like we might also need to bump our kinesis client library if we upgrade the general aws sdk version.

Note we also depend depend on com.fasterxml.jackson.core 2.4.4 in Spark

I ran the demo with the following two commands against a stream (test-stream1) on my AWS account.  It performed as expected.
bin/run-example streaming.KinesisWordProducerASL test-stream1 https://kinesis.us-east-1.amazonaws.com 100 10
bin/run-example streaming.KinesisWordCountASL testapp test-stream1 https://kinesis.us-east-1.amazonaws.com

I happy to bump the KCL library as well.  With the bumped com.fasterxml.jackson.core version, the project build and passed all tests except docker-integration, which I can't run.  AWS Java SDK 1.10.40 calls a function that isn't provided by Jackson 2.4.x.  If raising that to 2.5.x isn't an option, then we can find whatever the last version of AWS Java SDK to not use it is. I know it works with 1.10.24, so somewhere between 1.10.24 and 1.10.40 could be a temporary solution.

Cool - if we don't need to upgrade the KCL library then thats probably good. Maybe good to see if @tdas or someone similar can take a look?

/cc @brkyvz RE: the KCL library.

@BrianLondon What happens if you change the version of KCL to 1.6.1? Could you also please add a new line to anywhere in extras/kinesis-asl? Otherwise Kinesis tests don't run and we have no idea whether this is a safe change.
Thanks!

Test build #2203 has started for PR 10256 at commit 47d0a59.

Test build #2203 has finished for PR 10256 at commit 47d0a59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@brkyvz Pushed up the KCL version and added a newline to KinesisReciever.  Tests passed for me locally on a hadoop 2.6 build and the kinesis demos ran against a live AWS stream.

@holdenk bump

I think the next step is getting one of the jenkins admins (maybe @srowen or @brkyvz ) since they are already on this PR to tell jenkins ok to test and maybe see if @marmbrus has any concerns about the Jackson upgrade.

@brkyvz @holdenk @tdas any particular reason we depend on aws-sdk-java explicitly? The KCL brings in those dependencies (in fact from 1.5.0 KCL only depends on the necessary AWS SDK components, i.e. core, kinesis, dynamodb and cloudwatch). Depending only on KCL would make things cleaner and avoid potential issues such as not bumping versions at the same time (e.g. #8957 and related mailing list discussion)

I'm not really sure why - skimming the git logs it seems the aws-sdk-java sdk dependency was added at the same time as the kcl dependency.

It appears kcl pulls in 1.10.20.  I'm all for dropping the explicit dependency.  That probably wouldn't have prevented the regression in Spark 1.5.2 though. At any rate, the jackson bump is necessary if someone wants to use the latest aws sdk (e.g. for Firehose).

I removed the explicit dependence on the AWS Java SDK.  There's a newline that was added to KinesisReceiver.scala since the last test run.  I think if tests are re-run the kinesis part should be included, although someone more knowledgable than me about Travis can weigh in.
Tests and the kinesis example worked locally for me for what it's worth.

NAVER - http://www.naver.com/
<EMAIL_ADDRESS>님께 보내신 메일 <Re: [spark] [SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version (#10256)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

FYI, a bunch of spark-redshift users have run into the Jackson dependency conflict issue that's discussed here; see https://github.com/databricks/spark-redshift/issues/133 and https://github.com/databricks/spark-redshift/issues/121. Not commenting either way w.r.t. the dep. concerns themselves, but just wanted to point out that the pain of the Jackson conflict has impacts for a bunch of other applications too.

Yeah, that looks like the same underlying issue of using AWS Java SDK 1.10.0 or later with Spark.  I believe this version bump will fix it.  You can sometimes get around it by adding the following to your sbt build:
dependencyOverrides ++= Set(
  "com.fasterxml.jackson.core" % "jackson-core" % "2.4.4",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4",
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.4.4"
)

That doesn't work however if the calls to the AWS library use any Jackson 2.5.x functionality and some of them do.

I've also run the Kinesis tests locally (against a live AWS account) against this PR and all tests pass. I'm +1 on this subject to agreement for the jackson dependency bump, for which I defer to SQL folks e.g. @marmbrus

If the SQL/Hive tests are passing I'm good.  @srowen usually has good insight on dependency changes.

Can this wait until 2.0? I think it's going to anyway now. Then the dependency issue is fine.

Test build #2271 has started for PR 10256 at commit 64858df.

Test build #2271 has finished for PR 10256 at commit 64858df.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Rebased against current master, ran tests (except for docker) locally, and tested kinesis example against aws hosted stream.  All worked.
My preference would be to get the version bump in as soon as practicable for two reasons.  First, and mostly selfishly, the inability to link any library that requires Jackson 2.5.x features is causing us problems in our production environment.  Second, we currently can just bump the version number because the Spark codebase appears compatible with both Jackson versions at present, but the longer we wait the more likely someone introduces an incompatible change.
That said, I fully understand the reluctance to replace a heavily used library mid release cycle.

Test build #2354 has started for PR 10256 at commit 6f3bbfd.

Test build #2354 has finished for PR 10256 at commit 6f3bbfd.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Ah right, you'll need to acknowledge the dep changes in the build file:
To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
diff --git a/dev/deps/spark-deps-hadoop-2.2 b/dev/pr-deps/spark-deps-hadoop-2.2
index e4373f7..cd3ff29 100644
--- a/dev/deps/spark-deps-hadoop-2.2
+++ b/dev/pr-deps/spark-deps-hadoop-2.2
@@ -84,13 +84,13 @@ hadoop-yarn-server-web-proxy-2.2.0.jar
 httpclient-4.3.2.jar
 httpcore-4.3.2.jar
 ivy-2.4.0.jar
-jackson-annotations-2.4.4.jar
-jackson-core-2.4.4.jar
+jackson-annotations-2.5.3.jar
+jackson-core-2.5.3.jar
 jackson-core-asl-1.9.13.jar
-jackson-databind-2.4.4.jar
+jackson-databind-2.5.3.jar
 jackson-jaxrs-1.9.13.jar
 jackson-mapper-asl-1.9.13.jar
-jackson-module-scala_2.10-2.4.4.jar
+jackson-module-scala_2.10-2.5.3.jar
 jackson-xc-1.9.13.jar
 janino-2.7.8.jar
 jansi-1.4.jar

This LGTM but I do think we'll have to keep it for 2.x only.

Manifest updated.  Is there documented somewhere what all the tests Travis runs are?  It seems to differ from what's described at http://spark.apache.org/docs/latest/building-spark.html

@BrianLondon it should be running dev/run-tests, so that's something you can inspect or run locally if you need to. The doc there describes how to run one test.

Test build #2358 has started for PR 10256 at commit a1f75c1.

Test build #2358 has finished for PR 10256 at commit a1f75c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

I'm going to go for this one since I think we may want to, in general, consider updating Spark dependencies in 2.x anyway. 2.5 is even old for Jackson, so good to bump up here as a side effect I think.

Merged to master

I know this issue is closed, but one suggestion I have would be to specify the latest Jackson 2.5 version, 2.5.5 instead of 2.5.3. While I understand caution in general, Jackson patch versions try to limit inclusion of fixes to safe ones and specifically there are no API changes. This also means that while it is good to keep even patch versions in sync across components there is very low risk of incompatibilities even if patch versions do differ.
In this specific case, jackson-databind 2.5.4 has 10 bug fixes -- see https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.5.4 -- this is the main reason I would recommend use of latest version.

I think that's OK. I was intending to propose a series of dependency updates for 2.x like this sort of thing.

@srowen Ah. Yes, just wanted to mention it; I don't have enough context so there may be specific reasons for picking up particular versions. Glad to see version upgrades coming along. My interests are selfish, being both the main author of Jackson, and a (relative new) Spark user at work. :)
With that, if there are problems from Jackson side I could help, do not hesitate to cc me and/or file issues at https://github.com/FasterXML/jackson-databind (or other specific components). I know version compatibility is an annoying issue, and would like to help if possible.


parsergenerator.build fails on a read-only filesystem

See hylang/hy#1598.

Is there a stacktrace for exactly what lines are triggering this error?

@alex yes: https://github.com/hylang/hy/issues/1598#issuecomment-386064932

Whoops, missed that it wasn't unfurled by default.
If someone wants to write a patch to catch OSError and check if e.errno == errno.EROFS I'd accept that patch.

Actually, I'll write it myself, since I want to refactor something.

As well, I would be inclined to add an argument to builds to disable the cache in case it isn't desired in the first place.

You can pass cache_id=None to the constructor to disable it.

Whoops, accidentally commited directly to master instead of sending a PR: https://github.com/alex/rply/commit/2a3f626081bfe1369c679a51910b4bbc73340268
Feedback welcome

You can pass cache_id=None to the constructor to disable it.

Oh, I missed that, thanks.

Feedback welcome

Shouldn't the calls to tempfile.NamedTemporaryFile and os.rename be guarded, too?

Is there any way I could bypass this problem, even if it's not the best solution?

@alex Could you make a new release of rply soon so we can get this fix into the next release of Hy?

I'd like to try to get rpython fixes into the next release; @cfbolz said he
was hoping to work on them soonish, so hopefully I'll be able to throw
together a release next weekend, does that work?
On Sun, May 20, 2018 at 4:43 PM Kodi Arfer<EMAIL_ADDRESS>wrote:

@alex https://github.com/alex Could you make a new release of rply soon
so we can get this fix into the next release of Hy?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alex/rply/issues/74#issuecomment-390510984, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAADBOFn1-uzorUrch3FZDYmabMnzRJxks5t0dV7gaJpZM4Tv5Ez
.

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

Sure, I doubt Hy will release before mid-June.

Sounds good, I'll aim for a release next week, and if for any reason I
don't have one out by the time you're ready for a Hy release, let me know
and I'll just do it.
On Sun, May 20, 2018 at 4:48 PM Kodi Arfer<EMAIL_ADDRESS>wrote:

Sure, I doubt Hy will release before mid-June.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alex/rply/issues/74#issuecomment-390511310, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAADBDa9UvmCcmU6_-DoiPHZ2urgmtFTks5t0daZgaJpZM4Tv5Ez
.

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

Well, I'm jealous of you then. I added the cache support because building
topaz's parser took like 30 seconds.
On Mon, May 21, 2018 at 12:24 PM Kodi Arfer<EMAIL_ADDRESS>wrote:

Actually, I've made a Hy PR to disable the cache (hylang/hy#1615
https://github.com/hylang/hy/pull/1615) because it doesn't seem to give
a noticeable performance boost, anyway. So, this should be a non-issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alex/rply/issues/74#issuecomment-390705989, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAADBPAOJTrVJBLSAgDy72SANT8TI8zVks5t0upHgaJpZM4Tv5Ez
.

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

I bet that Lisp is a lot easier to parse than Ruby. :)

Understatement of the century :-)
On Mon, May 21, 2018 at 12:28 PM Kodi Arfer<EMAIL_ADDRESS>wrote:

I bet that Lisp is a lot easier to parse than Ruby. :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alex/rply/issues/74#issuecomment-390707159, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAADBDaO8GBEOyP1_TxF1eyMJRTxbAssks5t0us9gaJpZM4Tv5Ez
.

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

I did a release


Precondition Failed (412) Error When Deleting Record

Steps to reproduce this problem:

Open RecordEdit and submit a new record --> redirects to record page (keep this page open)
Open another RecordEdit and create another record.
Go back to the first (Record) page and delete this record gives 412 Precondition Failed error.

The reason:
A new etag is generated when second create request was submitted. Now the first record has an out of date etag and forces it to fail.
Proposed solutions:

Better display of the error/information for the user at UI level. (This still seems inconvenient for the user)
emrestJS handle a failure case.
Karl's comment : it requires UX intervention in general case. ETag mismatch is a conservative signal that something might be out of sync. if you reload and find that the data you based your decisions on is still the same, you can just repeat with the newer ETag you found.  if you find a difference, you would want to prompt the user as in, " has changed. Please review the latest content and resubmit your request." then present a refreshed page where the user can either delete again or make edits again without silently clobbering changes made in parallel.

A sketch from @karlcz on how to deal with 412 errors:

In Chaise:
On 412 in UPDATE case in recordedit,

Show a modal that explains that some data has changed and user must review the data.
Highlight the inputs where the current data in the DB isn't the same as what the user entered in the input. Show current data in blue text beneath input fields (like validation messages).

PR #1006 has been merged, closing now.


Superfluid slashing code

Makes superfluid slashing work. It now starts from a slash validator hook for a validator.
The superfluid module is then responsible for slashing every intermediate account's underlying delegation,
and the undelegations to that validator.
It does this, by getting all synthetic lockups for the underlying denom the IA represents.
It then slashes each of those synthetic lockups.
THought this was already PR'd earlier 😓

For contributor use:

[ ] Targeted PR against correct branch (see CONTRIBUTING.md)
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/)
[ ] Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
[ ] Re-reviewed Files changed in the Github PR explorer

Codecov Report

Merging #908 (a9cd31f) into main (576c171) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #908      +/-   ##
==========================================
- Coverage   20.34%   20.33%   -0.01%
==========================================
  Files         192      192
  Lines       24934    24915      -19
==========================================
- Hits         5072     5067       -5
+ Misses      18929    18920       -9
+ Partials      933      928       -5

Impacted Files
Coverage Δ

x/claim/keeper/hooks.go
28.00% <ø> (+1.07%)
:arrow_up:

x/lockup/keeper/lock_refs.go
38.46% <ø> (ø)

x/superfluid/keeper/hooks.go
38.88% <100.00%> (-0.25%)
:arrow_down:

x/superfluid/keeper/intermediary_account.go
85.71% <100.00%> (+1.50%)
:arrow_up:

x/superfluid/keeper/slash.go
100.00% <100.00%> (+27.77%)
:arrow_up:

x/superfluid/keeper/stake.go
48.46% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 576c171...a9cd31f. Read the comment docs.