Issues while building Clickhouse on Debian

Hi ClickHouse developers,
I tried to follow the build.md documentation to build a .deb for Debian using pdebuild, and I took me a while to make sure that the control's build dependencies were set correctly. Would it make sense to include the following?
--- a/debian/control.in
+++ b/debian/control.in
@@ -1,6 +1,10 @@
 Source: clickhouse
 Priority: optional
 Maintainer: Alexey Milovidov<EMAIL_ADDRESS>-Build-Depends: debhelper (>= 5)
-Standards-Version: 3.8.0
+Build-Depends: debhelper (>= 5), cmake, gcc-6, g++-6, libboost-dev,
+               libicu-dev, libreadline-dev, libmysqlclient-dev, unixodbc-dev,
+               libboost-thread-dev, libboost-program-options-dev,
+               libboost-system-dev, libboost-regex-dev, libboost-filesystem-dev,
+               libglib2.0-dev, libltdl-dev, libssl-dev
+Standards-Version: 3.9.8
 Section: libs

Other misc requests:

It would be really helpful if the debian/rules file was translated to English.
It would be really helpful if the release script had an option to just generate debian files (like control and rules) leaving the build part to the user (for example, say I want to use pdebuild rather than debuild).

Thanks a lot!
Luca

This is reasonable.
But if user has compiled some dependencies from sources and have installed to /usr/local without using debian packages (example: gcc-6), then build will refuse to work.
Is it worth to specify build dependencies, considering that use case?

Hi Alexey,
thanks for the response! I got your point and I agree that this could impact other use cases different than mine, maybe it could be enough to mention it in build.md for people trying to build the deb package by themselves?
The other thing that I would add (for newbies like me) is a suggestion about what git tag use to build the "latest stable". For example, I tried to build two versions, corresponding to the tags "54010" and "v1.1.54083-stable", but in the end of the latter build I get issues like:

Linking problems (might be related to my settings, but it might also be Cmake related?)

[100%] Building CXX object dbms/src/Server/CMakeFiles/clickhouse.dir/main.cpp.o
[100%] Linking CXX executable clickhouse
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(readline.o): relocation R_X86_64_32 against symbol `rl_get_previous_history' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(vi_mode.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(funmap.o): relocation R_X86_64_32 against symbol `rl_abort' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(keymaps.o): relocation R_X86_64_32S against symbol `rl_insert' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(search.o): relocation R_X86_64_32S against symbol `rl_history_substr_search_backward' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(rltty.o): relocation R_X86_64_32S against symbol `rl_vi_unix_word_rubout' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(complete.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
[...]

Debian rule problems, like dh_install not finding the debian/tmp directory and failing.

The above ones do not affect version 54010, so I am wondering if I am missing something trivial or if the docs are missing some indications.
Thanks a lot for the support!
Luca

I also noticed one thing using Lintian:
E: clickhouse source: license-problem-json-evil contrib/libpoco/doc/Acknowledgements.html
E: clickhouse source: license-problem-json-evil contrib/libpoco/doc/90100-Acknowledgements.page
E: clickhouse source: license-problem-json-evil contrib/libpoco/JSON/include/Poco/JSON/Parser.h

https://wiki.debian.org/qa.debian.org/jsonevil
This prevents Clickhouse to be included in Debian distros due to non free licenses.

In fact we don't use Poco/JSON, so I could just remove it from sources.

About stable tag. Latest stable tag is the right one.
Later we will also add two branches for convenience: stable and testing, so you could just checkout stable branch.

About linking with readline. Maybe your system is trying to link clickhouse with PIC (position independent code), for unknown reason. You could

change libreadline.a to readline or libreadline.so in dbms/src/Client/CMakeLists.txt;
Plese tell me, on what system you encounter this issue?

Why don't add gcc-6 to build depends? Because gcc 6 is still rarely available in packages. Users could install it from PPA or build from sources. Second option is also valid choice.
Probably later we will move to clang 4, when it will be released (we have done benchmarks and in results, clang 4 is generating as efficient code as gcc 6, though clang 3.9 is slower). I doubt it will be easy to find packages for clang 4.

Thanks a lot, sorry for the delay :)

If you could remove it and release a stable version it would be massive!
For some reason I didn't see the stable branch, my bad :)
I followed up with your suggestions and I was able to build changing the following:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 863b7f0..90c43e6 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -182,7 +182,7 @@ find_library (BOOST_THREAD_LIB libboost_thread.a HINTS ${BOOST_HINTS})

 # 4. ltdl
 set (LTDL_HINTS "/usr/local/opt/libtool/lib")
-find_library (LTDL_LIB libltdl.a HINTS ${LTDL_HINTS})
+find_library (LTDL_LIB libltdl.so HINTS ${LTDL_HINTS})

 # Directory for Yandex specific files
 SET(CLICKHOUSE_PRIVATE_DIR ${ClickHouse_SOURCE_DIR}/private/)
diff --git a/contrib/libpoco/Net/CMakeLists.txt b/contrib/libpoco/Net/CMakeLists.txt
index de9e618..f58feff 100644
--- a/contrib/libpoco/Net/CMakeLists.txt
+++ b/contrib/libpoco/Net/CMakeLists.txt
@@ -17,7 +17,7 @@ if(WIN32)
         set(SYSLIBS ${SYSLIBS} "ws2_32.lib" "iphlpapi.lib")
     endif()
 elseif(NOT APPLE)
-    set(SYSLIBS ${SYSLIBS} "libanl.a")
+    set(SYSLIBS ${SYSLIBS} "libanl.so")
 endif(WIN32)

 add_library( "${LIBNAME}" ${LIB_MODE} ${SRCS} )
diff --git a/contrib/libpoco/cmake/FindODBC.cmake b/contrib/libpoco/cmake/FindODBC.cmake
index 67cb5d4..0292c7d 100644
--- a/contrib/libpoco/cmake/FindODBC.cmake
+++ b/contrib/libpoco/cmake/FindODBC.cmake
@@ -29,7 +29,7 @@ find_path(ODBC_INCLUDE_DIRECTORIES
 )

 find_library(ODBC_LIBRARIES
-       NAMES libiodbc.a iodbc libodbc.a odbc odbcinst odbc32
+       NAMES iodbc libiodbc.so odbc odbcinst odbc32
        HINTS
        /usr/lib
        /usr/lib/odbc
diff --git a/dbms/src/Client/CMakeLists.txt b/dbms/src/Client/CMakeLists.txt
index a74c261..24311fe 100644
--- a/dbms/src/Client/CMakeLists.txt
+++ b/dbms/src/Client/CMakeLists.txt
@@ -1,5 +1,5 @@
 find_library(READLINE_LIB
-       NAMES libreadline.a libreadline.so
+       NAMES readline libreadline.so
        HINTS "/usr/local/opt/readline/lib")

 find_library(LIBEDIT_LIB

I am trying to build Clickhouse on Debian Sid (using the chroot provided by pdebuild) and from what I can read, .a libraries are usually compiled without -fPIC, meanwhile .so are compiled with it. I am not a great expert but it makes sense in my opinion, but I haven't dig a lot into why the Cmake tries to find either a .a or a .so version of a lib.

I would still provide a complete Build-Depends and then document how to change it to allow other use cases like people building from sources by themselves, but it is also fine the other way around if documented :)

If you could remove it and release a stable version it would be massive!

Already removed. We have new stable version today, I just published it.

For some reason I didn't see the stable branch, my bad :)

We have added stable and testing branched just few days ago.

I followed up with your suggestions and I was able to build the stable 54083 version changing the following

We have added CMake option USE_STATIC_LIBRARIES that may be turned off.
It is in master, not yet in stable branch.
I still cannot guess, why code is not linking with static libraries on your system. Maybe I will experiment on VM by myself, if I will have time.

I would still provide a complete Build-Depends and then document how to change it to allow other use cases like people building from sources by themselves, but it is also fine the other way around if documented :)

Ok.

I think that that dh_install assumes the presence of a debian/tmp dir, but I can't find it in the sources..

I expect, debian/tmp directory should be created here:
https://github.com/yandex/ClickHouse/blob/master/debian/rules#L54
BTW, I am not expert in debian package build system.

Already removed. We have new stable version today, I just published it.

Tested it, lintian does not complain anymore, nice!

We have added CMake option USE_STATIC_LIBRARIES that may be turned off.

I used it (USE_STATIC_LIBRARIES=0) for v1.1.54112-stable and it removes all the linking issues except the following (that I had to patch manually):
--- a/contrib/libpoco/Net/CMakeLists.txt
+++ b/contrib/libpoco/Net/CMakeLists.txt
@@ -17,7 +17,7 @@ if(WIN32)
         set(SYSLIBS ${SYSLIBS} "ws2_32.lib" "iphlpapi.lib")
     endif()
 elseif(NOT APPLE)
-    set(SYSLIBS ${SYSLIBS} "libanl.a")
+    set(SYSLIBS ${SYSLIBS} "libanl.so")
 endif(WIN32)

I still cannot guess, why code is not linking with static libraries on your system.

As far as I know in debian the static .a libraries are not built using -fPIC..

I expect, debian/tmp directory should be created here:

I am not an expert too, but I'll try to spend some time to debug it :)
Thanks!

I was able to figure out the issue, I was missing DAEMONS=clickhouse-server before the git-buildpackage command. Without it, several files under debian/tmp/etc were not created triggering the build.
So the only outstanding issue is the remaining linking problem reported above!

We should use find_*.cmake scripts for libanl library, that is linked inside Poco.

Still very far from being able to build correctly on Debian:
ldd ./usr/bin/clickhouse-server
./usr/bin/clickhouse-server: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by ./usr/bin/clickhouse-server)
	linux-vdso.so.1 (0x00007ffc69583000)
	libreadline.so.7 => not found
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff32f1ec000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007ff32eedc000)
	libicui18n.so.57 => not found
	libicuuc.so.57 => not found
	libicudata.so.57 => not found
	libmysqlclient.so.20 => not found
	libssl.so.1.1 => not found
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff32ecbf000)
	libcrypto.so.1.1 => not found
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff32eaa2000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff32e899000)
	libanl.so.1 => /lib/x86_64-linux-gnu/libanl.so.1 (0x00007ff32e695000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff32e491000)
	libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007ff32e286000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff32df85000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff32dbda000)
	/lib64/ld-linux-x86-64.so.2 (0x00005588236ff000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007ff32d96b000)

The above is the result after using USE_STATIC_LIBRARIES=1, I didn't think about the runtime's repercussions. I tried to play with the -fPIC flag in the codebase but didn't manage to make a build completing correctly. The error is always the same:
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(readline.o): relocation R_X86_64_32 against symbol rl_get_previous_history' can not be used when making a shared object; recompile with -fPIC`
Plus others of course.. How do you usually build your releases and on which platform? Is there any special procedure that you follow other than the one outlined in the build.md doc?

Some info: I do use the release script but I removed the debuild part since I am using git-pbuilder (Debian sid and latest git stable tag).

I also checked the /etc files produced by the build and I have some suggestions:

etc/cron.d/clickhouse-server

#*/10 * * * * root /etc/init.d/clickhouse-server condstart 1>/dev/null 2>&1

This functionality should not be present since admins usually prefer to manage them in other ways (for example leaving the job to systemd). The cron is disabled but it is not super great the fact that it will be installed by default on the target system.

etc/init.d/clickhouse-server

I would not let this script to manage file systems permissions with hardcoded values:
USER=metrika
GROUP=metrika
[..]
		chown -R $USER:$GROUP $LOGDIR
		chown -R $USER:$GROUP $PIDDIR
		chown -R $USER:$GROUP $SYSCONFDIR
				su -l $USER -s $SHELL -c "flock -u 9; exec -a $(generate_program_name $i) \"$BINDIR/$PROGRAM\" --daemon --pid-file=\"$(generate_pid_name $i)\" --config-file=\"$CNFFILE\" $(specific_log_file_for_each_process $i)
[..]

And I would remove the following too:
enable_cron()
{
	sed -i 's/^#*//' "$CRONFILE"
}

disable_cron()
{
	sed -i 's/^#*/#/' "$CRONFILE"
}

is_cron_disabled()
{
	[[ `grep -E "^#.*" $CRONFILE` == `cat $CRONFILE` ]];
}

And all the other references to the cron scripts mentioned in 1).
Thanks!

About cron and init script.
<[admins usually prefer to manage them in other ways]>
We still must have some reasonable default.
How we could make better packages, but leave this cron job as default or as a fallback solution?

How we could make better packages, but leave this cron job as default or as a fallback solution?

I would personally leave the choice to the end user/admin. For example, systemd might be used to restart the service on failure or maybe an admin might not want to remember about the 10 minutes cron during maintenance (so stop - update - start for example).
I would also avoid changing permissions in the init.d file, but rather use something like postinst (so do it once while installing rather than all the times that init.d is executed).

Gentle ping for the build :)

some fixes merged https://github.com/yandex/ClickHouse/pull/316/
now in debian jessie works:  debuild -e USE_STATIC_LIBRARIES=0

@proller: in my use case USE_STATIC_LIBRARIES=0 was already working, and I was happy to have shared libraries rather than everything on one binary. The main issue with this setting is that to satisfy dependencies like gcc-6 you'd need to build in Sid for example, precluding the possibility to use the deb created in Jessie for obvious reasons (our production fleet is all Debian stable). The USE_STATIC_LIBRARIES=1 seems the only viable option, but there is still the problem with relocable code (-fPIC). Can you confirm that you reproduced the problem on your environment?

you can use gcc5+ for building

@proller the "-no-pie" flag seems to have fixed the fPIC issue, I have just built the latest master commit and everything went fine. One little thing left: if I do use ldd to check runtime dependencies I can see the following error on Debian Jessie:
ldd ./usr/bin/clickhouse-server
./usr/bin/clickhouse-server: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by ./usr/bin/clickhouse-server)

Reading through the glibc_compatibility files it seems that I should be able to use older versions of libc..
I am using the GLIBC_COMPATIBILITY flag while building, is there anything else that I need to specify?

Did you try build without GLIBC_COMPATIBILITY ?
GLIBC_COMPATIBILITY needed if you will run pachage on systems older than jessie

@proller yes I tried but same issue, GLIBC_2.23 required. I might be missing something..

Whenever you have time, there are also previous comments that in my opinion would need some more discussion (cronjobs and permissions chowns hardcoded in init files).

I suspect that this is due to the fact that I am building on Debian Sid (to satisfy all the dependencies requested).
You build on sid and then run on jessie? its bad.
Better find way to install gcc5 or gcc6 on jessie ( https://www.google.ru/webhp?#q=debian jessie gcc 5 )

On which platform do you build the debs that you release? What is the correct procedure?
We use ubuntu ( trusty+ ) with available gcc5+ from apt

Whenever you have time, there are also previous comments that in my opinion would need some more discussion (cronjobs and permissions chowns hardcoded in init files).
cron edit will be removed, rc.d script will be simplified.
clickhouse default  user will be changed from 'metrika' to 'clickhouse'

Also here https://github.com/jbfavre/ClickHouse/tree/debian work in progress on true debian package

You build on sid and then run on jessie? its bad.
Yep you are completely right, I started building on Sid to satisfy all the dependencies, without realizing that I'd have an issue with Glibc (got also fooled by the GLIB_COMPATIBILITY flag). As stated above, I'll follow what you mentioned trying to build and use gcc5 on Jessie. Thanks for the links and the suggestions.

cron edit will be removed, rc.d script will be simplified.
clickhouse default user will be changed from 'metrika' to 'clickhouse'
Also please make sure to remove unnecessary chown/chmod in init files.

Also here https://github.com/jbfavre/ClickHouse/tree/debian work in progress on true debian package.
Very nice, just followed the fork, hope that it will get merged to Clickhouse soon.

Some changes:
https://github.com/yandex/ClickHouse/commit/c51801d7762bda374757cf61a062b6c6542762b5
Now you can adjust  init script by creating /etc/default/clickhouse:
to skip old path checks:
CLICKHOUSE_DATADIR_OLD=""
to not touch cron:
CLICKHOUSE_CRONFILE=""
change use/group:
CLICKHOUSE_USER=superman
CLICKHOUSE_GROUP=pupermans


Decode UTF-8 before invoking type=...

Hi,
it seems when I try to use eg. type=ip_address on Python 2.x the bytes are passed as-is to the ip_address function and it fails to decode it because it's not unicode object. If there is no good reason I think it would be good idea to decode UTF-8 before passing it to the type=... function.

This is impossible to change in a backwards-compatible manner, and while it might have made more sense to go "unicode-everywhere" in Click's APIs, I think it's definetly too late now. Afaict it'd also be out of line with many stdlib functions in Python 2, which only handle bytestrings properly. So even if Click started to decode the data to unicode, passing it to a stdlib function would create an implicit conversion using the ASCII encoding, which makes your program crash anyway. Therefore I doubt it'd be worth the compatibility breakage in Click's API.
My personal recommendation is to just use Python 3 if you don't want to care about unicode.

type is not supposed to be used that way anyways. The correct way is to implement a custom click type if you cannot deal with unicode data.


Add Mass Combat Sheet

Just seconding this. Having Mass Battle Force Cards would be amazing. Thanks for all your hard work, by the way!


Unix-like OS support

Thanks to the latest work on Unix-like OS support, we now have working SolutionReader and ProjectReference reader (verified myself under Ubuntu). I am afraid that's not the end.
warn AssemblyInfo file '/home/andrew/dev/NAudio/NAudio/Properties\AssemblyInfo.cs' not found

Well, you see the issue. File exists but not found. We need to make sure to call slash conversion routine everywhere we work with paths.

@andrew-boyarshin what version did you get that with? As the code in question (assuming you don't have a full path in your include) is this Path.GetFullPath(Path.Combine(projectPath, x)) which as far as I know should throw an ArgumentException on illegal chars. Is \ not considered an illegal char for a path on ubuntu?

@hvanbakel 2.0.0 release as global tool, latest .NET Core SDK (2.1.401)+runtime, Ubuntu 18.04.

So when you do Path.FullPath(Path.Combine(“/home/andrew”, “Properties\\assemblyinfo.cs”)) that does not throw (it does on windows for the forward slash).

@hvanbakel will run some checks tomorrow. I'm in final preparation stage for new PR (related to transformations loading and ordering).

Ok, if you have issues that you’re working on, could you assign those to yourself, I want to make sure we’re not doing the same thing.

@hvanbakel what am I doing wrong?
Microsoft (R) Visual C# Interactive Compiler version <IP_ADDRESS>119
Copyright (C) Microsoft Corporation. All rights reserved.

Type "#help" for more information.
> Path.GetFullPath(Path.Combine("/home/andrew", "Properties\\assemblyinfo.cs"));
> Console.WriteLine(Path.GetFullPath(Path.Combine("/home/andrew", "Properties\\assemblyinfo.cs")));
C:\home\andrew\Properties\assemblyinfo.cs

And yeah, that was csi on Win10:
Visual Studio 2017 Developer Command Prompt v15.9.0-pre.1.0

And I don't see it stated explicitly in documentation why it should throw any exception.

I was refering to:
Exceptions
ArgumentException
path is a zero-length string, contains only white space, or contains one or more of the invalid characters defined in GetInvalidPathChars().

the latter part of that sentence but just trying it that must have been a typo.
Then again, given that the code where the error is thrown uses Path.GetFullPath, for some reason that doesn't change the slashes while it does on windows.

I believe we should just MSBuild routine in Extensions. Much simpler and the most precise way to solve that. The only trouble is finding all such places to call that.

I'll do a pass on this one, hoping you can give it a go to test when that's a PR.

@andrew-boyarshin can we flag this one as closed?

I'll do a check today and close this if I find no issues.

v3.0.2 is broken.
The settings file in the tool's NuGet package is invalid: Settings file 'DotnetToolSettings.xml' was not found in the package.
Tool 'project2015to2017.migrate2017.tool' failed to install. Please contact the tool author for assistance.


[WFLY-12540] Move from javax.activation:activation:1.1.1 to com.sun.a…

…ctivation:jakarta.activation:1.2.1
https://issues.jboss.org/browse/WFLY-12540
@asoldano fyi in case there's a reason not to move to com.sun.activation (which is the eclipse-ee4j artifact.) I mention this because the pom's cxf entries have exclusions for com.sun.activation:javax.activation. OTOH this change means we are getting closer to the CXF dep set, which seems good. OTOH maybe there was a reason we stayed on the older javax.activation:activation and you guys know what it is. :)

Dependency Tree Analyzer Output:
New Dependencies:

com.sun.activation:jakarta.activation:jar:1.2.1:compile

Removed Dependencies:

javax.activation:activation:jar:1.1.1:compile

CC @wildfly/prod


REST responses returns HTTP error 200 even if request failed

From project's example, run:
curl -X GET 'http://localhost:8080/test_db/test_table/999

which will not find any records, and in this case service returns HTTP error 200 while it should return 404, as well instead of null it should return error status in JSON reply, something like:
{"error":"record not found"}

I acknowledge that. Before I fix it, we should do the same thing for UPDATE and DELETE when the id (999) is not found, right?

I acknowledge that. Before I fix it, we should do the same thing for UPDATE and DELETE when the id (999) is not found, right?

Yes, and probably in case of authorization get failed, then there should be HTTP error 403

Fixed. Can you try:
GOPROXY=direct go install github.com/elgs/gosqlapi@latest


Type tag 'tex' is not recognized

Hi @jonasbb
First of thanks a lot for your awesome hooks! Since the last version though, I get the following error message:
Updating https://github.com/jonasbb/pre-commit-latex-hooks ...
==> File /home/fuchur/.cache/pre-commit/repoa2l8gbnx/.pre-commit-hooks.yaml
==> At Hook(id='american-eg-ie')
==> At key: types
==> At index 1
=====> Type tag 'tex' is not recognized.  Try upgrading identify and pre-commit?

According to https://github.com/pre-commit/pre-commit/issues/855 this seems to have to do with the types setting.

Hi @klieret,
sorry for the trouble, but I cannot reproduce the problem. Both my existing installation and a fresh one work fine. Did you try upgrading both identify and pre-commit? If that doesn't help could you post steps to reproduce the issue please.
In the last version, I did not change the types setting. I only re-ordered the entries in  .pre-commit-hooks.yaml.
The linked issue does not seem relevant to me. In the issue the error is produced by a typo.

Hey @jonasbb ! Thanks for your quick reply!
I must have messed up something before. Running pip3 install --upgrade identify indeed solved the issue for me when I just tried again :)
Sorry for the noise!


Why am I the only one who can see an issue I opened six days ago?

Six days ago I opened issue #13558 but today I found out that it is apparently hidden so that I am the only one that sees it.  Did I do something wrong, or is there some reason it's still hidden?

Github issue.  Nevermind.


why automatic records remove when i open my desktop app?

Recently added record remove when open desktop app again. can you help me for this.
here is my code
var Datastore = require('nedb');
var ICDB = {};
ICDB.chats = new Datastore({ filename: 'users.db', autoload: true});

Please show code with expected behaviour vs actual behaviour; code above is insufficient to help.

When i build exe file than after remove all saved records.

i wan't to just save all records

don't delete automatic records.

check image

Here is my step
1 Create Desktop App and Make Exe file
2 Open Desktop App By clicking on exe and save some records.
3 close this Desktop app
4 Again Open Desktop App By clicking on exe.
5 My saved record removed.


[WORKAROUND] Login: "Your session has expired" because of invalid Docker system time

Description
Getting the error "Your session has expired" on login. Problem is that an endpoint cannot be found:

Steps to reproduce the issue:

docker run -d -p 9000:9000 --name portainer portainer/portainer
open docker and setup password
login the first time

Technical details:

Portainer version: 1.11.0
Portainer Docker image tag (latest/arm/windows...): 1.11.0
Target Docker version (the host/cluster you manage): docker for windows, 1.12.5
Platform (windows/linux): windows
Browser: chrome, edge

Had the same issue...
Mounting the socket from the host made it work:
docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock --name portainer portainer/portainer

Where is docker.sock on windows? It seems you're running it on Linux

I ran like
docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer

And it did not work.

Oh, that sounds good. I've read the other issue as well but wasn't able to
reproduce the error due the lacking development environment.
Maybe it would help if I could see the current time of portainer in the
logs to help you.

@SeriousM I've pushed some changes on a specific tag on Dockerhub, could you try it?
$ docker run -d -p 9000:9000 --name portainer -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer:fix432

sorry, but that didn't fixed the problem. I was trying to get the date in portainer but there isn't even a bash or something... very clean image btw :)
could you push a dev version where I could give you more information what's going wrong?

The date in the portainer container should be the same as your host machine. If you use a vm, like with Docker for mac or Windows, its the same as the system time returned by ´docker info´.

well, that's very strange! my system time is 31.12.2016 11:47 but docker info reports System Time: 2016-12-29T07:15:27.8156969Z. wtf?

alright, I found the solution now.. and the reason.
solution: restart docker for windows
reason: I use hibernation and the timedrift is actually the time the machine was sleeping. VM doesn't get this information and therefore the big difference.
the code solution would be to calculate the expiration in a different way.

Sounds like it's the same #398 but for Windows.

yes, it seems like that, except that I found the cause and no js errors are involved.

@deviantony tried it on OS X with
$ docker run -d -p 9000:9000 --name portainer -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer:fix432
and it works now, thanks.
Docker version: 1.12.5, build 7392c3b
OS: El Capitan v10.11.6 (15G1212)

@deviantony tried it on OS X with
$ docker run -d -p 9000:9000 --name portainer -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer:fix432
and it works now, thanks.
Docker version: 1.12.5, build 7392c3b
OS: El Capitan v10.11.6 (15G1212)

No luck for me
docker run -d -p 9000:9000 --name portainer -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer:fix432

Environment Details
OS: El Capitan v10.11.6 (15G1212)
Docker version: Version 1.13.0-rc4-beta34.1 (14853)

No luck for me
docker run -d -p 9000:9000 --name portainer -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer:fix432

Environment Details
OS: El Capitan v10.11.6 (15G1212)
Docker version: Version 1.13.0-rc4-beta34.1 (14853)

@petermetz @merbla The issue might be related to your Docker system time.
Could you check the output of docker run busybox date and see if it's different from your system time? Have a look at this issue as it seems that you're both running on MacOS: https://github.com/docker/for-mac/issues/17

Please check if "docker info" datetime correlates with your real system
datetime. If not, restart docker host.

@deviantony @SeriousM yep a restart fixed the issue for me.

I'll let this issue for now, and will add a section in the FAQ about this issue.

I've added an entry about this issue in the FAQ: https://portainer.readthedocs.io/en/stable/faq.html#i-am-getting-the-error-your-session-has-expired-on-login-and-cannot-login-what-s-wrong
Thanks for the help debugging that one guys !

Hi guys, I got this error when using docker for windows and finally fixed it by going to hyper-v-management --> virtual machines --> right click MobyLinuxVM --> integration services and enabling the time sync checkbox in the services list. I hope this helps some of you.

Thanks @nicopeeters, I'll add it to the FAQ !

Hi @deviantony - trying portainer via Docker for the first time and am also getting this problem.

In the command you listed, which one represents the date of your system?

I don't think your issue is the date shift issue, but rather the multiple replicas of portainer.
The JWT may not validate across the 3 replicas, since the JWT secret is randomly generated. (see https://github.com/portainer/portainer/blob/develop/api/jwt/jwt.go#L27)
Also since Portainer uses a file based database, you may be best off using a single replica instead for that reason.

As @Glowbal said, it's probably related to your multiple Portainer setup. Try again with a single replica.

then system time wrong，you need sync ntp time

@yixiongz worked for me, thanks!

@bvisonl ?

I'am still experience the problem (portainer: "your session has expired"). I have docker installed inside Ubuntu VM (virtualbox), and I try to run portainer.
I check the container date using busybox, it show "Tue Feb 12 14:58:36 UTC 2019".
and when i check the system date of Ubuntu, it show "Tue Feb 12 14:58:36 WIB 2019"

Ubuntu have the same system date with my windows.
I'v already tried several solution (nothing it works):

restart portainer
remove and redeploy portainer
restart ubuntu VM

Environment Details
-Ubuntu 16 VM (virtualbox) running on windows 10
-latest DockerCE (18.09.1)
-lates portainer

So anyone can help me how to fix this problem
how do I sync the time, what is the command?

Hi there,
same issue here:
docker run busybox date
Wed Feb 13 10:20:54 UTC 2019
 date
Mi 13. Feb 11:22:20 CET 2019
docker run -v /etc/localtime:/etc/localtime:ro busybox date
Wed Feb 13 11:22:48 CET 2019

Any hints on that?
Best regards


Implement intptrcast methods

Cc https://github.com/rust-lang/miri/issues/224

This is breaking the run-pass/hashmap.rs  test, I'm not sure why. However, this started happening after the last commit (i.e. when forcing intptrcast for binary operations).

lgtm, @RalfJung 	do you want another look?

:umbrella: The latest upstream changes (presumably #787) made this pull request unmergeable. Please resolve the merge conflicts.

Why is this commented out?
Oh, my bad. I was testing if @oli-obk's code would pass or not and forgot to uncomment it. I'll amend it

@bors r=oli-obk

@christianpoveda: :key: Insufficient privileges: Not in reviewers

@bors r=RalfJung,oli-obk let's get this party started :tada:

:pushpin: Commit 7fbf8e53a22925a5ff865b3546f881f7542c4f9e has been approved by RalfJung,oli-obk

:hourglass: Testing commit 7fbf8e53a22925a5ff865b3546f881f7542c4f9e with merge 945f007c0d305c3ec069b5e5d911ef783f6d70e7...

Thank you, this one was really messy D:

:sunny: Test successful - checks-travis, status-appveyor
Approved by: RalfJung,oli-obk
Pushing 945f007c0d305c3ec069b5e5d911ef783f6d70e7 to master...

Awesome, thanks a lot @christianpoveda !


missing shared dataset icon

Hi,
I notice in "Save a copy of a report in a workspace", no.4 Select Save:
[Power BI automatically creates a copy of the report, and an entry in the list of datasets if the report is based on a dataset outside of the workspace. The icon for this dataset is different from the icon for datasets in the workspace: Shared dataset icon
That way, members of the workspace can tell which reports and dashboards use datasets that are outside the workspace. The entry shows information about the dataset, and a few select actions.]
Recently I made the 'save a copy' and the target workspace end up only have the copied report but there is no shared dataset icon under the dataset category. I am wondering if the power bi team removes this feature?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 83c63999-09de-cccf-ae53-ef7fcfbc1186
Version Independent ID: 39407557-14d4-f78b-bfcb-ea95cf501c47
Content: Copy reports from other apps or workspaces - Power BI
Content Source: powerbi-docs/connect-data/service-datasets-copy-reports.md
Service: powerbi
Sub-service: pbi-data-sources
GitHub Login: @paulinbar
Microsoft Alias: painbar

Hi @ErjunQ, thanks for your question. I am checking with the product team and will get back to you. Thanks!

Hi @ErjunQ,
Indeed it seems that in the new look of workspaces (with list view and lineage view), external datasets are no longer shown in list view. External datasets are shown in lineage view however, and are indicated by a different kind of icon. I can imagine that that might not be the most convenient way to see the external datasets, but that's the way it is now.
The article you mentioned is indeed out of date, and shows the old look. Updating that article is now on my to-do list. Thanks for bringing this up! Let me know if you have any further questions.


Entities un-pushable?

Hi. Is it possible to make entities unpushable?
My player pushes my enemies when he hits them and i don't want that to happen. 😃

FXGL uses box2d as the simulation engine. Please refer to their documentation. My guess is that you might be able to set some collision flags, so that player doesn't physically collide with enemies, yet collision events are reported still. I don't know if this is possible via FXGL API, but maybe via box2d API.
If you find out how to do this with box2d and if this isn't possible with current FXGL API, I'm happy to open an issue for it.


Add quick example of select input

Thank you! This is awesome. This is the best React Laravel Boilerplate ever.

Glad you like it - I'll leave this here and not merge as I'd like something a bit more solid to eventually end up in the boilerplate


Orientation at the centre of QR codes

Thanks for the amazing post and code.
If I want to find the orientation of the QR code at the centre of the detected bounding box, what changes should I make in this?

Hi,
Not sure what you're exactly asking for. You just want to know the center coordinate? In that case, it is just the mean value of the 4 detected corner points. You can find it like this:
center = np.mean(points.reshape((4,2)), axis = 0).astype(np.int32)
cv.circle(img, tuple(center), 3, (0,0,255), -1)

Where points is the returned corner points (line 59). The second line simply draws the center on the image. You can put these two lines after line 62. It should look like this.

If you want to just draw the axes at the center, then just add the proper shift. Change line 41 to:
al = 0.3 #axis length
unitv_points = np.array([[0,0,0], [al,0,0], [0,al,0], [0,0,al]], dtype = 'float32').reshape((4,1,3))
unitv_points = unitv_points + np.array([0.5,0.5,0.0]).reshape((1,1,3))

This is what I needed. Lots of thanks to you.

Can we also get the orientation angles in degrees like the one above from this code?

Line 62 returns rvec and tvec. Like your image, tvec is the position of the QR code pattern. Rvec is the rotation matrix (in Rodrigues vector form). If you want XYZ angles, then you need to decompose the rotation matrix into angles.
You need to do something like this:
R = cv.Rodrigues(rvec)

def Decompose_R_ZYX(R):

    #decomposes as RzRyRx. Note the order: ZYX <- rotation by x first
    thetaz = np.arctan2(R[1,0], R[0,0])
    thetay = np.arctan2(-R[2,0], np.sqrt(R[2,1]**2 + R[2,2]**2))
    thetax = np.arctan2(R[2,1], R[2,2])

    return thetaz, thetay, thetax

Line 62 returns rvec and tvec. Like your image, tvec is the position of the QR code pattern. Rvec is the rotation matrix (in Rodrigues vector form). If you want XYZ angles, then you need to decompose the rotation matrix into angles.
You need to do something like this:
R = cv.Rodrigues(rvec)

def Decompose_R_ZYX(R):

    #decomposes as RzRyRx. Note the order: ZYX <- rotation by x first
    thetaz = np.arctan2(R[1,0], R[0,0])
    thetay = np.arctan2(-R[2,0], np.sqrt(R[2,1]**2 + R[2,2]**2))
    thetax = np.arctan2(R[2,1], R[2,2])

    return thetaz, thetay, thetax

 thetaz, thetay, thetax = Decompose_R_ZYX(R)

Although I don't remember if rvec is Rodrigues form or 3x3 matrix form. Just make sure it is 3x3 matrix before calling the decompose function. Also, notice that this is X rotation first. Any other rotation order requires a different decomposition.

Thank you so much for your timely reply.

Hopefully that helped. I'm closing this issue.


HelpDot - Clicking Expensify Classic brings you to main homepage instead of Expensify Classic category homepage

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number:
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Email or phone of affected tester (no customers): N/A
Logs: N/A
Expensify/Expensify Issue URL: https://github.com/Expensify/Expensify/issues/337748
Issue reported by: @maddylewis
Slack conversation: N/A
Action Performed:

Head to https://help.expensify.com/
Click into Expensify Classic - https://help.expensify.com/expensify-classic/hubs/
Click any homepage category (ex: Bank Accounts & Credit Cards)
Click Expensify Classic in the LHN to return to the main, Expensify Classic homepage

Expected Result:
When clicking Expensify Classic from a category page, I would expect to be returned to the Expensify Classic category homepage - https://help.expensify.com/expensify-classic/hubs/:

Actual Result:
When clicking Expensify Classic from a category page, you're brought back to the main homepage where you're prompted to click Expensify Classic or New Expensify - https://help.expensify.com/:

To get back to the Expensify Classic category homepage, you must click Bank Accounts & Company Cards, which to me, is unintuitive.
Workaround:
N/A
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/38016013/1f19d3c9-7e7d-448f-8336-e9e68df42b2f

View all open jobs on GitHub

oh @MitchExpensify - kinda the perfect person to review this.
while the current design is intentional, i think this is an area that could be improved to help navigate the site more intuitive overall.
lmk your thoughts and then we can cc rushat to have a look if we want to move forward with a fix :)

Checking in on what you think about my take above, @maddylewis

chatting more with our tech writer, Ren, and he made the good point that the LHN that shows on the homepage menu is redundant since the categories are listed as icons on the homepage. this makes me think we should maybe eliminate the LHN altogether and only have a back button.
the LHN should still exist once you're in an article so that you can click through the headings and subheadings easily. but the LHN when you're on the homepage and/or subcategory list seems redundant/not useful.

okok im going to move to weekly for now since we're not totally sure if this is a "bug" worth fixing atm.

im gonna close this for now


"Try it out online" link 404s

This link (http://www.goodafternoon.co/decaf/) is a 404

It's a bit out of date, but the gh-pages branch can be accessed here:
http://rainforestapp.github.io/decaf/


Add user avatar to post list

Show avatars of post autors in post

Propably use glide to achive that


feature: styling for bare URLs & comments

Is your feature request related to a problem? Please describe.
This plugin really has come a long way, congrats for that.
The only two things I noticed that are not handled by the plugin yet are <!-- toc --> and bare URLs.
Describe the solution you'd like
I think in both cases, bare URLs and comments, they could be concealed, with only one icon left to indicate the presence of content maybe?
Describe alternatives you've considered
I am not aware of any.
Additional information
No response

For bare URLs are you talking about actual bare URL, like http://www.example.com/, or the ones surrounded by angle brackets, like <http://www.example.com/>?
I'll add support for the latter, but will probably handle it more like emails where the angle brackets are hidden and an icon is added rather than hiding the whole thing. For the former since these are not handled by the treesitter parser I'm not going to handle them.
I've generally stayed away from touching any HTML entities, but maybe comments are a good exception to this. I'll provide a custom_handler version shortly in case I don't end up adding it.

For bare URLs are you talking about actual bare URL, like http://www.example.com/, or the ones surrounded by angle brackets, like http://www.example.com/?

That makes sense, as markdownlint as well as some parsers expect angel brackets, making it somewhat the idiomatic way of writing urls (https://github.com/updownpress/markdown-lint/blob/master/rules/034-no-bare-urls.md)

I'll add support for the latter, but will probably handle it more like emails where the angle brackets are hidden and an icon is added rather than hiding the whole thing.

yep, that sounds good!

Bare URL support added here: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/401a6c9c4cf39e22b8487503dd7dfe37fc7fb064

Here's what a custom handler for concealing HTML comments would look like:
---@param root TSNode
---@param buf integer
---@return render.md.Mark[]
local function hide_comments(root, buf)
    local marks = {}
    local query = vim.treesitter.query.parse('html', '(comment) @comment')
    for id, node in query:iter_captures(root, buf) do
        local capture = query.captures[id]
        local start_row, start_col, end_row, end_col = node:range()
        if capture == 'comment' then
            table.insert(marks, {
                conceal = true,
                start_row = start_row,
                start_col = start_col,
                opts = { end_row = end_row, end_col = end_col, conceal = '' },
            })
        end
    end
    return marks
end

require('render-markdown').setup({
    custom_handlers = {
        html = { parse = hide_comments },
    },
})

Here's what a custom handler for concealing HTML comments would look like

Thanks! I prefer having them not completely hidden, but I realized that it's just the conceal key. Tbh, I think this should be builtin rather than a custom parser. They may be called "html" comments, but they are in practice also the standard comment syntax for Markdown. Many tools (linters, formatters, LSPs) use this comment syntax for their ignore comments.

Bare URL support added here: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/401a6c9c4cf39e22b8487503dd7dfe37fc7fb064

Thanks, looks like a charm!
There appears to a visual glitch when passing such bare URLs though, which does not happens with other conceals by this plugin:
https://github.com/user-attachments/assets/702b8acc-fe29-4089-9182-e7d4c4a45798

That definitely doesn't look good. Out of curiosity what terminal emulator / GUI are you running neovim in?
To remove the angle brackets and add the icon I used a single extmark that hid the entire line then added it back inlined with icon plus the URL minus brackets. Since that's effectively removing and adding back the entire line it ends up having this pretty bad behavior.
Fixed this here hopefully: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/b6b903cc09df1441602fc04665fb12cef576a914.
Separated the logic into 4 smaller marks to hide brackets, add the icon, and highlight the URL.
Can you update and LMK if it's better on your end, I'm not able to replicate that behavior on my end.

Added support for concealing HTML comments here: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/558310ae07b7bf0dffc478160513bb5c845f263c

Sure added support for this here: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/7674543331701d05dc9f878e1fe62d1107dc2f5e
By default no value is set so it will still just conceal the comment out of the box. You just need to provide a value for html.comment.text:
require('render-markdown').setup({
    html = { comment = { text = 'SOME VALUE' } },
})

I used inline text rather than a conceal character so you're free to use a value as long or short as you want.

Perfect, thank you!

Just send you a small coffee (or two, depending on where you live) for all your awesome work and quick responses. Cheers!

Hi, shouldn't be this opt-in instead of opt-out? IMO concealing content is really a personal preference.

I see it in the same way as concealing links and other content that's concealed out of the box. HTML comments are the same level of information as a link URL. If concealing content is intrusive you can opt out of it via:
opts = {
    win_options = { conceallevel = { rendered = 0 } },
}

In which case the HTML comment concealing would also be disabled. Otherwise it's a case by case thing that is left up to you to decide. In general all new features that aren't new options for an existing feature will be enabled by default with some way to opt out.


Changed the method of disabling respawning

I'm going to test this out now.

Everything works perfectly!


TypeError: 'Box' object is not iterable, WHY?

Hi,
I used those lines in my custom environment:
self.observation_space = gym.spaces.Box(np.array([0,0,0,0,0,0,0,0,0,0]), np.array([9,9,9,9,9,9,9,9,9,9]), shape=(10,), dtype=np.int64)
self.action_space = gym.spaces.Box(low=0, high=9, shape=(1,), dtype=np.int64)
but I received this error:
.....\epymarl-main\src\envs_init_.py", line 56, in init
for sa_obs in env.observation_space:
TypeError: 'Box' object is not iterable
thank you

EPyMARL expects the observation and action spaces of each agent to be within tuples with length equal to the number of agents. So you should have something like this:
N_agents = 10
self.action_space = gym.spaces.Tuple(tuple( [gym.spaces.Box(low=0, high=9, shape=(1,), dtype=np.int64]*N_agents ))

Btw, you should also consider using gym.spaces.Discrete instead of Box.

that's right!
It's works.
:)


[tests-only][full-ci] add test to share project drive's resource to a federated user

Description
This PR adds ocm test to check activities of a resource from personal drive shared to a federated user.
Scenario: local user shares resources from project space to federation user

Related Issue

Part of: https://github.com/owncloud/ocis/issues/10073

Motivation and Context

How Has This Been Tested?

locally
CI

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Technical debt
[x] Tests only (no source changes)

Checklist:

[ ] Code changes
[ ] Unit tests added
[x] Acceptance tests added
[ ] Documentation ticket raised:

@saw-jan why test scenario have multiple when for ocm. Any specific reason. Because it seems out of our current practice

@saw-jan why test scenario have multiple when for ocm. Any specific reason. Because it seems out of our current practice

In some cases, it should be fine unless you misuse it. It is always good to have atomic test scenarios but we also have to consider the  growing number of tests
Multiple When can be refactored into Given, Then but in the coming PRs


Two small change suggestions

Sorry - running short of time.  Next time I'll try to generate a pull request.

Allow generating a Java library on MacOS:

$ git diff version3/java/config64.py
--- a/version3/java/config64.py
+++ b/version3/java/config64.py
@@ -9,7 +9,7 @@ org1text="org"
 org2text="apache"
 org3text="milagro"

-if sys.platform.startswith("linux")  :
+if (sys.platform.startswith("linux") or sys.platform.startswith("darwin"))  :
        copytext="cp "
        deltext="rm "
        slashtext="/"

version3/java/DBIG32 and version3/java/DBIG64 have a syntax error around line 235: cast_to_chunk() is in the BIG class.  This should be caught by your test platform...

 		int i,wd,bt;
 		wd=m/CONFIG_BIG.BASEBITS;
 		bt=m%CONFIG_BIG.BASEBITS;
-		w[wd]&=((cast_to_chunk(1)<<bt)-1);
+		w[wd]&=((BIG.cast_to_chunk(1)<<bt)-1);
 		for (i=wd+1;i<BIG.DNLEN;i++) w[i]=0;

was fixed already.

is a good idea..


Edit files

Hello,
How does one edit HTML/PHP and other installation files? I can't seem to find where they are stored after installing with docker. I am using all default settings.
Any help is much appreciated.
Best, Matic

You would need to expose /www/freescout and not expose /data as a volume if you wanted to edit the html files. However, this will lock you into using that version and upgrades will stop working. Are you trying to make a couple small mods? I do similar for my organization and have a seperate git repository that I use to overrwrite the source code files each start of the container to give me the customizations that I want. I drop all my files into /assets/custom and upon container startup it will copy the files into /www/freescout for you.

Hi Dave,
Thank you for your reply. Yes, that’s exactly what I’m looking to do :)
This “hack” you described is really cool idea. I will implement it that way. Thank you again so much.
All the best, Matic


AugmentationStrategy implementation is missing

Hey, I struggle to find where's the implementation for
from augmentationStrategy import augment
from augmentationStrategy import augmentTest

or what package are you using for that

Hey, I struggle to find where's the implementation for
from augmentationStrategy import augment
from augmentationStrategy import augmentTest

or what package are you using for that

I sincerely apologize for any inconvenience caused. The augmentationStrategy.py has been added, and I truly appreciate your valuable feedback. Please don't hesitate to reach out if you have any further questions or concerns.

@wanggynpuer could you also provide data_preprocess/data_list/Indian_pines_train.txt. .../Indian_pines_test.txt and Indian_pines.txt params files. I created my own, but unfortunately there's parsing issue in functions_for_training.py on line sample_path = self.list_txt[index].split(' ')

@wanggynpuer could you also provide data_preprocess/data_list/Indian_pines_train.txt, .../Indian_pines_test.txt and Indian_pines.txt params files.
I created my own from the provided default args values, but unfortunately there's parsing issue in functions_for_training.py on line sample_path = self.list_txt[index].split(' ')
Here's Indian_pines_train.txt I created
dataset=Indian_pines
model_name=HSIVit
use_cuda=True
restore=False
batch_size=64
test_batch_size=64
epochs=300
lr=0.0005
momentum=0.9
seed=0
model_save_interval=300
sheet_id=2
beta=1.0
cutmix_prob=1.0
num_workers=10
pin_mem=True

and error code
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/wd/3d-rcnet/training/functions_for_training.py", line 38, in __getitem__
    label = sample_path[1][:-1]
IndexError: list index out of range

I have now updated the repository, adding the paper reference as well as the Indian_pines_split.txt file to the data_list directory. I hope this helps!


Post Status Updates to GitLab Commits

Looks like GitLab 8.1 allows GitHub style "statuses" to be posted for commits:
http://doc.gitlab.com/ce/api/commits.html#post-the-status-to-commit
Should be easy to integrate

This seems to be working. Comments are still used but that can be a separate issue if people want to get rid of them.


fceux 2.2.3

version bump
remove :DATA patch (most of them merged upstream)
inreplace to bypass X11 requirement (already carried in the above patch, now reported upstream)
use built-in LUA library (brewed one not detected without patch, reported upstream)
disable video logo insertion feature (gd not detected for the same reason above; this should fix #463)
install built-in LUA scripts and palettes to pkgshare
write exec script to correctly set LUA_PATH (otherwise some LUA scripts may fail)
use RELEASE build (no DEBUG caption anymore)
OpenGL framework is not detected on macOS (runs okay without it, but reported upstream)

Merged. Thank you for your contribution to Homebrew!


Install as devDependency?

Isn't vue-tsc a "devDependency"?
On NPM page the installation is:
npm i vue-tsc
Shoudn't it be:
npm i -D vue-tsc ?
Because I just installed it to "dependencies" and it doesn't look good.

But how is it possible to change this?

Why are very upstream issues appearing in this gadget 😂, please forward to https://github.com/npm/feedback/discussions.


WSL 2 only using half the cores of the host machine

WSL 2 is only using up to 50% of the CPUs on my 8c/16t machine.
I currently use an Hyper-V Ubuntu VM and had the same issue initially, as that was the default setting after creating the VM. With Hyper-V I can use the Hyper-V manager to allocate more cores, but I dont see any way of doing that with WSL2.
If WSL2 are Hyper-V VM it would be convenient to be able to configure them in the Hyper-V manager.
Is there anyway to setup the resource allocation of the WSL2 VM's?

@prenaux - Thanks for posting.  We have some changes in progress for configuring various properties of the WSL2 lightweight VM.  These are not traditional VM's.  If it isn't running, it doesn't exist.  They are created on-the-flyso they do not appear in Hyper-V manager.

I'm testing WSL2 on a Ryzen7 1700 CPU, single socket.

Looks like I'm not properly calculating how many cores to give to the WSL2 lightweight VM.  Working on a fix now.

dual 2690v2 - 20 core 40 threads ... and only 8 core on WSL 2 ... setup nginx, php running too slow ...

Looks like I'm not properly calculating how many cores to give to the WSL2 lightweight VM

In specific setups? I only ask because I've got a (lowly?) i7-6700K and my bog-typical make -j8 workloads peg all 4 cores and 8 threads. /proc/cpuinfo shows all 8.

@therealkenc - Yes, it seems only configs are effected.

Figured it out, the kernel is being built to only allow 8, I'll fix this.
root@BENHILL-DELL:/mnt/c/Users/benhill# zcat /proc/config.gz | grep NR_CPUS=
CONFIG_NR_CPUS=8

This article refers to this zipfile which enables other people than me to prove or disprove my claim: Any attempt to run a GNU/Linux executable inside Windows 10 will exhibit considerably longer runtime values than running it on a bare metal GNU/Linux OS - under these circumstances:

Intensive floating point operation load, outcompeting any memory or file io bottlenecks
Shared memory multiprocessing by means of OpenMPI

@benhillis Please let us know when the update is live, I'll try it out.

@benhillis Not sure what your fix is (increase cap or remove it).  I do have some questions:

Will NUMA be reported correctly by the kernel / VM?
Will logical processors from all Windows processor groups be reported? (i.e. more than 64 processors support?)

WSL 1 "reported" only one processor group, but had no cap on the actual number of concurrent processes.
Since WSL 2 enforces the concurrent process limit, this is a much bigger issue.

Will thread affinitization work correctly? This is mostly useful for HPC work, less so for normal server or compilation workloads.

Trying to use a dual SkyLake Xeon (112 total logical processors) with WSL 2 and am currently severely constrained by the 8 proc limit.

@g0blue You can compile the latest kernel to try it out. Following the steps from #4203 but getting the sources from here instead: https://github.com/microsoft/WSL2-Linux-Kernel.
There you can see the changes they did like increasing the cap from 8 to 256 cores/threads.

Thanks @onomatopellan!  This worked great.
It seems that it still only uses 1 processor group, so even though the max cores is bumped to 256, you'll never actually use more than 64.  My dual SkyLake is now limited to 1 socket (56 logical processors).  Much better than 8, but less than what can be used with WSL 1.

Taskmanager during WSL 2 software compilation:

@g0blue - Do you happen to know how many processors (GetSystemInfo)[https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsysteminfo] returns for your machine?

Do you happen to know how many processors GetSystemInfo returns for your machine? I'm assuming 56.

Yes, 56.
Relevant links:

https://docs.microsoft.com/en-us/windows/desktop/procthread/processor-groups
https://docs.microsoft.com/en-us/windows/desktop/procthread/what-s-new-in-processes-and-threads

gnome-system-monitor running in WSL 2 during sofware compilation. :)

@benhillis any update on this? I just updated the Windows machine and reinstalled WSL2 from scratch and its still only using 8 cores, is the only option really to recompile the kernel somehow? If so how do you go about doing that?

Should already be fixed. What version are you running?

Or specifically does your system have multiple processor groups?

The test on Ryzen7 1700, so that's a single processor group. Do I need to re-download the Ubuntu appx? I've reused the one I downloaded a while back (because the Appstore is broken again and refused to dl it...)
I also have a Threadripper 2950, are the 2 NUMA (unit/core?) 2 processor groups?

@prenaux - No you do not need to redownload any apps.  Can you share the output of the following commands?
uname -r
zcat /proc/config.gz | grep -i NR_CPU
dmesg | grep 'Command line'

I'm having the same issue on a Threadripper 2950X.
uname -r
4.19.43-microsoft-standard
zcat /proc/config.gz | grep -i NR_CPU
CONFIG_NR_CPUS_RANGE_BEGIN=2
CONFIG_NR_CPUS_RANGE_END=512
CONFIG_NR_CPUS_DEFAULT=64
CONFIG_NR_CPUS=8
dmesg | grep 'Command line'
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0

@benhillis I can confirm that the issue is fixed - I spent some time today to unstuck Windows Updates (reset caches... etc) and it updated to 19002. So I probably wasn't running the latest build. Tested on a Ryzen 1700.

@benhillis I'm running Ubuntu/WSL and while compiling 3 differente projects at the same time, my host CPU seems capped at 50%. Event tough, /proc/cpuinfo shows 4 cores. Here's a snap showing top under task manager window.

@benhillis I am having a similar issue. I have a multi socket system using two Intel Xeon E5-2699 v3 CPUs. In windows 10 it reports 2 Sockets, 36 cores and 72 logical processors. Those commands in WSL 2 listed above give,
4.19.84-microsoft-standard
CONFIG_NR_CPUS_RANGE_BEGIN=2
CONFIG_NR_CPUS_RANGE_END=512
CONFIG_NR_CPUS_DEFAULT=64
CONFIG_NR_CPUS=256
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=36
If I type in lscpu | grep -E '^Thread|^Core|^Socket|^CPU(' it gives
CPU(s):                          36
Thread(s) per core:              2
Core(s) per socket:              18
Socket(s):                       1
So it appears that only one socket is being recognized. What can I do to make sure that it recognizes both CPUs? Do I need to build a kernal with NUMA support like mentioned here? (https://github.com/microsoft/WSL/issues/5025#issuecomment-609442968) Let me know if I need to open a different ticket somewhere else. Thank you.

I also have the same issue with AMD threadripper. It detects the CPU as a 2 socket system but only uses one of the sockets. Is there anyway to fix it?

Let me know if I need to open a different ticket somewhere else.

It is worth opening new issue if people still aren't seeing all their cores on two-socket rigs, because the resolution for this issue (CONFIG_NR_CPUS stuck at =8) is on a one-way track administratively to closed/fixedin2004 (because "thing fixed" even if "different thing not fixed"). #5025 can stand on its own separately as a yet different ask. [It is tempting to repurpose circa-2016 #1115 but that is WSL1 which is another kettle of fish.]

It is worth opening new issue if people still aren't seeing all their cores on two-socket rigs...

So, I went to open a new issue, and I found #3314 - which is exactly the issue that needs to be opened and was closed as a duplicate of #1115 by you.
To be clear, the issue has nothing directly to do with number of sockets, it's purely an issue with multiple processor groups in Windows. Should I file a bug (feature?) requesting support for multiple processor groups?  Will this also get closed as a duplicate of #1115?

OK, looking at the date for #3314 I see that this was still WSL 1.  So, I suppose open a new feature explicitly asking for WSL 2 support for multiple processor groups?

To be clear, the issue has nothing directly to do with number of socket

To be clear the issue is "WSL 2 is only using up to 50% of the CPUs on my 8c/16t [Ryzen 1700] machine", which has been fixinbound (really fixedininsiderbuilds) since June 2019 (and confirmed by the OP in October).

Should I file a bug (feature?) requesting support for multiple processor groups?

Can your issue (a single-socket Xeon 8180) and the question I was addressing (Greg's dual Intel Xeon E5-2699) be two new issues instead of one? Sure, whynot.

I would suggest a new issue for multi processor groups. I am already looking into this, AMD brought this to my attention as well.

@sanastasiou , Did you find any issue posted? because even when I gave up on WSL 2 and moved back to hyper V, the issue still persists. :(

No one has submitted a new issue for the multi processor groups scenario AFAIK. Doing one is somewhat academic since the devs are aware of the limitation, but it could act as a "like" target.

@therealkenc anything against re-opening this issue? Or simply referencing it from a new issue?

@sanastasiou @therealkenc , I opened a new issue #5423 for the AMD cpu issue.

hi @benhillis,
In one of your comment u asked whether u have 2 processors
In my workstaion i've 2 processors each with 20 cores, butt typing lscpu on wsl only detects only one processor
pls help

lscpu
lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  56
  On-line CPU(s) list:   0-55
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  28
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            4788.90
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse ss                         e2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid p                         ni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c
                         rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow
                         vnmi ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt                          flush_l1d arch_capabilities
Virtualization features:
  Virtualization:        VT-x
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   896 KiB (28 instances)
  L1i:                   896 KiB (28 instances)
  L2:                    7 MiB (28 instances)
  L3:                    35 MiB (1 instance)
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS
                         Not affected
  Srbds:                 Not affected
  Tsx async abort:       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

I have 2 sockets (2 cpus), each with 14 cores and 28 wire cities, now I can recognize all cores well in wsl2, but numa can't recognize 2. This also caused all numa2 cores to be identified as numa1 hyperthreads. This has a significant performance impact.
htop

The same issue still persists with WSL2 in 2024. See the attached image above.
lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  40
  On-line CPU(s) list:   0-39
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz
    CPU family:          6
    Model:               85
    Thread(s) per core:  2
    Core(s) per socket:  20
    Socket(s):           1
    Stepping:            7
    BogoMIPS:            6185.46
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse ss
                         e2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology tsc_rel
                         iable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe p
                         opcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid
                         _single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust
                         bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx5
                         12bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization features:
  Virtualization:        VT-x
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   640 KiB (20 instances)
  L1i:                   640 KiB (20 instances)
  L2:                    20 MiB (20 instances)
  L3:                    35.8 MiB (1 instance)
Vulnerabilities:
  Gather data sampling:  Unknown: Dependent on hypervisor status
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT Host state unknown
  Retbleed:              Mitigation; Enhanced IBRS
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Mitigation; TSX disabled

Microsoft claims that it should use the same number of processors as native Windows:
Microsoft - main-wsl-settings

Hi
Unfortunately, there is no development on this issue. There are constant requests on this issue, but they are marked as "duplicate requests" and directed to old notifications. Is there any development?
Actually, there are multiple expectations regarding this NUMA issue;
1> More than one NUMA needs to be supported.
2> It needs to be able to work with more than 64 processors.
3> We need to be able to decide which NUMA it will work on. (There may be those who want it to work on only one NUMA)
It would be really great if such a development could be made.
Thank you in advance for your efforts.

Thanks for your patience on this - we have identified a fix for this and will be releasing it as an update soon.


undefined method `+' for nil:NilClass (NoMethodError)

By converting a new adoc document to PDF, I get an error message. Unfortunately the document itself is internal and I cannot share it. Is there a way to get the line of the adoc file, which causes the problem?
Using Asciidoctor-pdf 2.3
> asciidoctor-pdf -a pdf-themesdir=media -a pdf-theme=asciidoctor --trace .\en\XXX.adoc
C:/Users/XXX/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3908:in `block (3 levels) in ink_toc_level': undefined method `+' for nil:NilClass (NoMethodError)

                start_dots = last_fragment_pos.right + hanging_indent
                                                     ^
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document/bounding_box.rb:297:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document.rb:528:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3905:in `block (2 levels) in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:4509:in `block in theme_font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/font.rb:60:in `block in font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/font.rb:214:in `save_font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/font.rb:58:in `font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/ext/prawn/extensions.rb:329:in `font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:4507:in `theme_font'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3874:in `block in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `each'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3937:in `block (2 levels) in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document/bounding_box.rb:297:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document.rb:528:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3936:in `block in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `each'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3937:in `block (2 levels) in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document/bounding_box.rb:297:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/prawn-2.4.0/lib/prawn/document.rb:528:in `indent'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3936:in `block in ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `each'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3871:in `ink_toc_level'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:3857:in `ink_toc'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:309:in `block in convert_document'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:612:in `indent_section'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:200:in `convert_document'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/lib/asciidoctor/pdf/converter.rb:152:in `convert'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/document.rb:955:in `convert'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/convert.rb:118:in `convert'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/convert.rb:190:in `block in convert_file'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/convert.rb:190:in `open'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/convert.rb:190:in `convert_file'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/cli/invoker.rb:129:in `block in invoke!'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/cli/invoker.rb:112:in `each'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-2.0.17/lib/asciidoctor/cli/invoker.rb:112:in `invoke!'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/lib/ruby/gems/3.1.0/gems/asciidoctor-pdf-2.3.0/bin/asciidoctor-pdf:46:in `<top (required)>'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/bin/asciidoctor-pdf:32:in `load'
        from C:/Users/AJHMF7Y/AppData/Local/Programs/rubyinstaller-3.1.2-1-x64/bin/asciidoctor-pdf:32:in `'

I'll need at least a reproducible example. I can't fix a scenario I can't
see.

That is understandable. I'll try to find a way next week.

Also, please test with Asciidoctor PDF 2.3.2. You should always be using the latest patch release when filing an issue to be sure there are no old bugs in the picture.

If it's too hard to reproduce the error, you can give me temporary access to a private repository. I'm willing to do that. What I'm not willing to do is guess.

I discovered a reproducible test case. It may not be the way it is defined in your document, but I believe it emulates the problem.
= Document Title
:toc:

== Section 1

== foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo+++
+++

What's happening is that the last text fragment is not rendered. As a result, the position information is missing from the last fragment. And that causes the converter to crash. The converter needs to either handle this case, it track the position of every fragment so it can get the information it needs from the last fragment that was actually rendered.

I'm going to try to port this to main, but it appears like main doesn't have the same problem...and in fact the behavior is different. So more research is needed ;)

@mojavelinux I've found the position in the document. I missed this position when I cleaned up the pandoc converted results. Looks like invalid Asciidoc. But I think it still shouldn't crash here. This is a very small test document causing exact my problem:
= User Manual
:toc: macro
:doctype: book

toc::[]

= *Calloff* +
abc

Removing bold sings ("*") or the "+" will cause asciidoctor-pdf not to crash. But both crashes.

Thanks. That confirms what I suspected the problem was. I also realized
there were potentially a lot of other cases. So I made the code generally
robust.
By the way, using + at the end of a section title is nonsensical. A section
title must be a single line.

Thanks to this bug report, I discovered a different regression in the main branch. I love when we get that kind of side effect. Just wanted to say thanks!

Thank you for your great work on this project!


font, icons scaling for xz3 screen

xz3 screen resolution, although 18:9 ratio, is 1440x2880 pixels, compared to XZ2 variants which are 1080x2160. Changing text size in "settings/display/text size" is partially fixing the problem.
ps not sure if this is an upstream issue?

Its unfortunate limitation of currently available pixel ratios (2.0 is a max): https://github.com/sailfishos-sony-tama/droid-config-sony-tama-pie/blob/master/rpm/header-h8416.inc#L5
Its the same problem for XZ2c, which would have benefited from higher aspect ratio icons available.
Please open issue at TJC and keep it open here. I believe other devices could be approaching the same limit

I opened an issue at TJC, thanks.

Closing it over here as I don't think we can do something on our side. Please ping the issue at TJC, ask at scheduled meetings of community.


fix: lsp-embedded-language-service cleaninterval args type

Based on the @type/node version in the project，Parameter type of clearInterval is
NodeJS.Timeout | string | number | undefined
When the parameter is Nodejs.Timer , The extension can't run

number

J


Feature 949 pinecone support metadata filter

Issue

Closes #
https://github.com/langchain4j/langchain4j/issues/949
Change

Impeletement filter search in dev.langchain4j.store.embedding.pinecone.PineconeEmbeddingStore
General checklist

[X] There are no breaking changes
[X] I have added unit and integration tests for my change
[X] I have manually run all the unit and integration tests in the module I have added/changed, and they are all green
[X] I have manually run all the unit and integration tests in the [core]

Checklist for changing existing embedding store integration

[X] I have manually verified that the {NameOfIntegration}EmbeddingStore works correctly with the data persisted using the latest released version of LangChain4j

@langchain4j I've resolve issues above. Thanks for all the suggestions, I'll pay attention next time. By the way, I found that there is a dev.langchain4j.store.embedding.EmbeddingStore#removeAll(dev.langchain4j.store.embedding.filter.Filter) function that can remove items by metadata filter. However, I failed to use this with Pinecone because Pinecone doesn't support it in serverless mode, only in pod mode, which requires a paid plan. So, I couldn't help to implement it. You can find relevant information here deleting-vectors-by-metadata-filter


doc: Document disabling cargo phases in Rust builds

Description of changes
Since 042adf08d1c ("cargo/hooks: allow hooks to be disabled"), it is possible to disable the default cargo hooks. This allows to switch back to the default phases from stdenv. This can be useful to package projects that rely on Makefiles. I encountered this by chance when creating #245862, so let's document it for future contributors.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

I agree with documenting this, but the usual way to package cargo packages with make files (or meson) is to use stdenv.mkDerivation and use the hooks directly

I got the impression that rustPlatform.buildRustPackage is doing additional things that are necessary for Rust packages even when they are using Makefiles, like vendoring all the dependencies into a derivation before building the actual derivation.

I got the impression that rustPlatform.buildRustPackage is doing additional things that are necessary for Rust packages even when they are using Makefiles, like vendoring all the dependencies into a derivation before building the actual derivation.

This can be done by setting cargoDeps and using rustPlatform.cargoSetupHook, you can search for these words in the repository to find some examples.

This can be done by setting cargoDeps and using rustPlatform.cargoSetupHook

Then maybe this should be documented instead? I have never used that approach and therefore don't feel comfortable documenting it. But it's fine to close this PR if there is a documented way of building rust applications with different build systems.

Here is the documentation

Ah, I never saw that one. I always stopped at https://nixos.org/manual/nixpkgs/unstable/#custom-buildinstall-procedures because it exactly matched my use case. But it seems to give wrong advice for most cases then.

But it's fine to close this PR if there is a documented way of building rust applications with different build systems.

these should still be documented, as sometimes you may want to revert one of the phases but not the others. (eg. when a package has a Makefile just for the purpose of installing files),


Repo transfer?

Hi @khiemdoan thanks again for transferring the snap pacakge. What would you think about transferring this repo as well? If we do that, rather than the current fork, links across the internet will continue working, folks will be able to find the latest source, and we'll be better able to preserve your git history. Would you be amenable to this? The organization we use for public projects is just https://github.com/ngrok
Happy Friday!

Hi @krwenholz ,
I agree to transfer this repo to the organization.
Happy weekends!

There are some errors when I transfer this repo:
ngrok already has a repository in the khiemdoan/ngrok-snap network and You don’t have the permission to create public repositories on ngrok

Ah I think I may know what's wrong. I'm out for the weekend but will get back to you on Monday! Sorry for the hassle.

Okay so I deleted the old repository in our org, which is the first part of the error. In order to resolve the permissions part, it looks like the best option is to transfer to me, krwenholz, and then I can transfer to ngrok where I have create permissions. Sound reasonable?
Thanks for your patience and help!

It's live! Thank you so much @khiemdoan. Have a most excellent week.


JSON Schema validation

See:
https://github.com/readium/webpub-manifest/tree/master/schema
Note that the JSON schemas as divided:
"$ref": "https://URL_TO_OTHER.schema.json"

https://ajv.js.org
https://github.com/epoberezkin/ajv

https://github.com/tdegrunt/jsonschema/issues/255
https://github.com/tdegrunt/jsonschema#dereferencing-schemas

Implemented in https://github.com/edrlab/r2-streamer-js/commit/76bf4484a93bc96bf7e0628454d455455b8c7db2 using ajv (see comment above), with the JSON schema adapted at https://github.com/danielweck/webpub-manifest/tree/master/schema (see PR https://github.com/readium/webpub-manifest/pull/9 ).
Validation is performed for demonstration purposes at the /manifest.json/show/all URL, for instance:
https://readium2.herokuapp.com/pub/L2FwcC9taXNjL2VwdWJzL2NoaWxkcmVucy1saXRlcmF0dXJlLmVwdWI%3D/manifest.json/show/all
(scroll all the way down to the bottom of the page, where the validation report is generated)


Update to december RC

Codecov Report

Merging #9077 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9077   +/-   ##
=======================================
  Coverage   57.12%   57.12%
=======================================
  Files         551      551
  Lines       29100    29100
  Branches     4395     4395
=======================================
  Hits        16623    16623
  Misses      11570    11570
  Partials      907      907

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14e427c...54756bf. Read the comment docs.


Create custom footer

The wiki does not have a footer, create a custom one.
Deadline: 16 October

Custom footer is ready for review 👀
https://github.com/SWE574-Fall2023-Group1/SWE574-Fall2023-G1/wiki

Thank you, looks great! 😀


Simplify the document cloud example

Issue #, if available:
Description of changes:
This PR simplifies the original model as follows.

Remove is_authenticated attribute from context. Now we assume each user is authenticated before requesting authorization.
Remove Public entity type. I think it should amount to unconstrained principal.
Remove DocumentShare entity type. It appears to me that Group is all we need, provided that we remove the Public type.
Remove isPrivate attribute. My rationale is that it can conflict with the public_access attribute.
Merge certain policies into a single policy.
Remove two forbid policies using stale attributes.
Revise the remaining policy such that only resource owner's blocked list takes effects.

@khieta @aaronjeline I'll update the entity JSON file and request JSON files if the policies/schema look good to you.


Task 8: Auto-select optimal Geohash precision

Epic: https://github.com/mzagorskirs/geowave/issues/3

add in code to auto-select the Geohash precision based on the Geohash envelope, pixel width and height.
for example, something similar to: https://github.com/geotools/geotools/blob/main/modules/library/coverage/src/main/j[…]org/geotools/coverage/grid/io/AbstractGridCoverage2DReader.java

I developed an algorithm to automatically select an appropriate Geohash precision as a user zoom into their data/heatmap.
These following demo videos show a combination of how the SLD is used to change the parameters fed into the custom heatmap process, and also how the Geohash precision used increases as the user zooms into the data/heatmap.
Notice that the performance of the Geohash auto-selector algorithm is optimized and there is very little lag while the algorithm is selecting an appropriate precision level.  Also, notice that when a stastistic (e.g. CNT_STATS = count statistic or SUM_STATS = field sum statistic) do not yet exist in the datastore, the equivalent aggregation query is run by default while the statistic is calculated and added to the datastore in the background for each Geohash precision level as the user zooms in.  Only after the statistics are calculated and added to the datastore can they be utilized upon a UI refresh.
The output geojson files show the increasing Geohash precision level, the query type, and the query value in their title for each Geohash bin.  The query values have been verified to be accurate.
https://user-images.githubusercontent.com/43011672/163426533-dc826348-a6bd-4bdd-84e8-080016c59cb7.mp4
https://user-images.githubusercontent.com/43011672/163426510-b00f5bbf-3362-416f-84cb-c7888d857a32.mp4
https://user-images.githubusercontent.com/43011672/163426521-fb8b58aa-d0d4-4db3-9d98-7fabda59a782.mp4

approved


Added .factorypath to gitignore

Quick 1-line change. The .factorypath file is just something that's used by one of my Vim plugins.

Fixed the image display on the user page and the feed. It will no longer show an invalid image icon when no image is uploaded, but it will still display the image correctly if the user does upload an image.

As it turns out, Datastore stores all numbers as longs. What this meant was that when I tried to store status as an int, Datastore freaked out and refused to send the proper data back. Thus, even though it feels very unnecessary, I had to make status a long, just like timestamp.


upgrade jpmml evaluator

Example evaluator is dated, we should move to the new codebase
see
https://github.com/jpmml/jpmml-evaluator

Still relevant @operdeck ? I think this was for a very very old java implementation of pdstools so i assume we don't need this, but it bring up a different question: Should we add pmml to the python tools?

No but will keep the ADM to PMML tool alive for a bit, not planning any improvements however.


fix(optimizer): handle existing select_related in querysets

Note: this is a draft PR! I am testing this out on our code, but I figured I'd open this draft so you're aware :)
Description
The strawberry-django optimizer can automatically add select_related and only clauses to a queryset. However, it currently fails if you enable the only optimization while using querysets that already have select_relateds on them.
We use strawberry at work, and have a model like:
class MyManager(models.Manager):
  def get_queryset(self):
    return super().get_queryset().select_related("foo")

class MyModel(models.Model):
  objects = MyManager()
  foo = models.ForeignKey(Foo, on_delete=models.CASCADE)

This causes issues with the optimizer if you try to execute a graphQL query that doesn't include foo: you get errors like Field cannot be both deferred and traversed using select_related at the same time. We've actually been running without the only optimization for a long time for this reason.
This PR adds some code to the optimizer that inspects the queryset for any existing select_relateds, and includes them in the .only() call.
Types of Changes

[ ] Core
[x] Bugfix
[ ] New feature
[ ] Enhancement/optimization
[ ] Documentation

Issues Fixed or Closed by This PR
N/A, there's no issue for this, though it related to this old message in the Strawberry Discord.
Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[ ] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Note: this is a draft PR! I am testing this out on our code, but I figured I'd open this draft so you're aware :)

Let me know once it is ready to be reviewed :)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.63%. Comparing base (ff50d0a) to head (973c687).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #515      +/-   ##
==========================================
+ Coverage   88.59%   88.63%   +0.03%
==========================================
  Files          40       40
  Lines        3402     3413      +11
==========================================
+ Hits         3014     3025      +11
  Misses        388      388

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.