Timestamp miss milliseconds

Which version of ShardingSphere did you use?
shardingsphere-4.0.0-RC2-1
Which project did you use? Sharding-JDBC or Sharding-Proxy?
Sharding-Proxy
Expected behavior
insert a timestamp value into table with milliseconds
Actual behavior
insert a timestamp value into table without milliseconds
Reason analyze (If you can)
I think maybe related to the Timestamp long construtor in sharding-proxy ? or btm jdbc (but when insert to my db without proxy, it works perfect,so this should not be the cause)???
Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
I am using bitronix tm in my application, just a normal simple insert sql with timestamp value with milliseconds in PreparedStatement. and my client os is windows 10 with Zulu openjdk 1.8.0_232. And the sharding-proxy is in Ubuntu 18.04 with Oracle jdk 1.8.0_211 and MySQL5.7
Example codes for reproduce this issue (such as a github link).
internal application,not available in github

Can you provide your INSERT and CREATE TABLE SQL?

Well, the root cause is that in the Mysql jdbc PreparedStatement setTimestampInternal method,
there is a checking
this.serverSupportsFracSecs
which is to detect  the current connection agains the mysql's version
but when connect to the sharding-proxy, the response of the checking detectFractionalSecondsSupport is false . and BTW, my MySQL server is 5.7.25 version and the jdbc driver is 5.1.43
------below is the code in the MySQL driver PreparedStatement for your reference-----
`
protected void detectFractionalSecondsSupport() throws SQLException {
this.serverSupportsFracSecs = this.connection != null && this.connection.versionMeetsMinimum(5, 6, 4);
}
private void setTimestampInternal(int parameterIndex, Timestamp x, Calendar targetCalendar, TimeZone tz, boolean rollForward) throws SQLException {
if (x == null) {
setNull(parameterIndex, java.sql.Types.TIMESTAMP);
} else {
checkClosed();
        if (!this.sendFractionalSeconds) {
            x = TimeUtil.truncateFractionalSeconds(x);
        }

        if (!this.useLegacyDatetimeCode) {
            newSetTimestampInternal(parameterIndex, x, targetCalendar);
        } else {
            Calendar sessionCalendar = this.connection.getUseJDBCCompliantTimezoneShift() ? this.connection.getUtcCalendar()
                    : getCalendarInstanceForSessionOrNew();

            x = TimeUtil.changeTimezone(this.connection, sessionCalendar, targetCalendar, x, tz, this.connection.getServerTimezoneTZ(), rollForward);

            if (this.connection.getUseSSPSCompatibleTimezoneShift()) {
                doSSPSCompatibleTimezoneShift(parameterIndex, x);
            } else {
                synchronized (this) {
                    if (this.tsdf == null) {
                        this.tsdf = new SimpleDateFormat("''yyyy-MM-dd HH:mm:ss", Locale.US);
                    }

                    StringBuffer buf = new StringBuffer();
                    buf.append(this.tsdf.format(x));

                    if (this.serverSupportsFracSecs) {
                        int nanos = x.getNanos();

                        if (nanos != 0) {
                            buf.append('.');
                            buf.append(TimeUtil.formatNanos(nanos, this.serverSupportsFracSecs, true));
                        }
                    }

                    buf.append('\'');

                    setInternal(parameterIndex, buf.toString()); // SimpleDateFormat is not
                                                                // thread-safe
                }
            }
        }

        this.parameterTypes[parameterIndex - 1 + getParameterIndexOffset()] = Types.TIMESTAMP;
    }
}

`

What I mean is can you provide your INSERT and CREATE TABLE SQL, and we may talk about details.
ShardingSphere do not care concrete database driver.

Well, my test table as follows
CREATE TABLE `TEST_SHARDING_ORDER` (
  `ORDER_ID` varchar(40) NOT NULL,
  `CUSTOMER` varchar(63) DEFAULT NULL,
  `SUPPLIER` varchar(63) DEFAULT NULL,
  `NOTE` varchar(63) DEFAULT NULL,
  `ORDER_DATE` datetime(3) DEFAULT NULL,
  `LAST_UPDATED_STAMP` datetime(3) DEFAULT NULL,
  PRIMARY KEY (`ORDER_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.Timestamp;

public class Sharding {
    public static void main(String[] args) throws IOException {
        String idKeyValue = "100006";
        int sharding = Math.abs(String.valueOf(idKeyValue).hashCode() % 17);

        System.out.println("sharding is " + sharding);

        String jdbcUrl = "jdbc:mysql://<IP_ADDRESS>:3307/sharding_db?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false";
        String username = "root";
        String password = "root";
        try{
            Connection connection = DriverManager.getConnection(jdbcUrl,username,password);
            String sqlString = "INSERT INTO TEST_SHARDING_ORDER (ORDER_ID, CUSTOMER, SUPPLIER, NOTE, ORDER_DATE, LAST_UPDATED_STAMP) VALUES (?, ?, ?, ?, ?, ?)";
            PreparedStatement preparedStatement = connection.prepareStatement(sqlString);
            preparedStatement.setString(1,idKeyValue);
            preparedStatement.setString(2,"customer");
            preparedStatement.setString(3,"supplier");
            preparedStatement.setString(4,"timestamp miss the million seconds");
            Timestamp currentTimestamp = new Timestamp(System.currentTimeMillis());
            System.out.println("currentTimestamp=="+currentTimestamp);
            preparedStatement.setTimestamp(5,currentTimestamp);//这里会因为MySQL jdbc driver this.connection.versionMeetsMinimum(5, 6, 4)数据库版本校验问题导致毫秒丢失
            preparedStatement.setTimestamp(6,currentTimestamp);//这里会因为MySQL jdbc driver this.connection.versionMeetsMinimum(5, 6, 4)数据库版本校验问题导致毫秒丢失
            preparedStatement.executeUpdate();

            preparedStatement.close();
            connection.close();
        }catch (Exception e){
            e.printStackTrace();
        }
    }
}

when run this test,out put would be
sharding is 3
currentTimestamp==2019-11-23 14:29:01.588
and the timestamp in database is 2019-11-23 14:29:01.0
the insert sql statement itself works fine, but the insert value of the timestamp will miss the milliseconds because of the db version checking in the mysql jdbc driver.
I do think ShardingProxy should let the jdbc driver know the real database's version.
e.g.
this.connection.versionMeetsMinimum(5, 6, 4) in the mysql jdbc driver should be true because my MySQL version is 5.7.25

Hello ,any updates on this? if any other information is needed ,please let me know.

Hi @lightning-pro,
I just go through all the messages on this issue. You already have a lot of findings.
I need the clear and brief answers to the following questions:

Have you tested by connecting to MySQL directly? How about the result?
Have you tested by connecting to Sharding-Proxy with raw Connector/J? How about the result?
Is there a bug in Sharding-Proxy? Is the bug related with certain connection pool like bitronix ?
I think you have already found out the root cause, but I can't get the point. Can you figure it out?

1.I have tested by connnecting to MySQL directly, it work perfect
2.Yes, my last response is the pure java main test demo,it purely use the DriverManager to get an connection from Connector/J.you can just copy it and run it from any ide you want
3,yes,there is a bug in sharding-proxy, the bug is not related with any connection pool,it is related to the version checking of mysql server version.
4.Ｉ think i haved descript the root case is very clear.but let me describe it again in Chinese:|
原因就是mysql-connector-j里的com.mysql.jdbc.PreparedStatement的变量serverSupportsFracSecs在方法
protected void detectFractionalSecondsSupport() throws SQLException {         this.serverSupportsFracSecs = this.connection != null && this.connection.versionMeetsMinimum(5, 6, 4);     }
里设置的值为false。直接连数据库设置的值为true。之所以链接sharding-proxy的值为false，是因为this.connection.versionMeetsMinimum(5, 6, 4);这个条件比较满足不了。应该是sharding-proxy没返回版本号？
如果要进一步追踪，就是在MysqlIO.java里的doHandshake(String user, String password, String database)方法，第1043行 this.serverVersion = buf.readString("ASCII", getExceptionInterceptor());
读取到的值为 '5.6.0-Sharding-Proxy 4.0.0-RC2'。我想是sharding-proxy里hard code了版本号？？？。这里导致读取到的版本号为5.6.0，然而我真实的mysql数据库版本号应该是5.7.25.
sharding-proxy返回5.6.0的版本号满足不了mysql jdbc driver的5.6.4版本号的最低要求检查，导致时间戳毫秒数丢失。我想这里描述应该非常的清楚了吧？能否安排解决一下？

@lightning-pro I got it. Thanks for your patience.
I prefer to update the version from 5.6.0-Sharding-Proxy to 5.6.4-Sharding-Proxy.

Hi,@tuohai666

@lightning-pro I got it. Thanks for your patience.
I prefer to update the version from 5.6.0-Sharding-Proxy to 5.6.4-Sharding-Proxy.

I am wondering why you like to hardcode the version number... cause when I debug in the mysql driver, I found that there are a lot of different version checkings for different mysql functions.I am not sure if the 5.6.4 is the highest version number in all the driver source code, for timestamp format itself,it is 5.6.4,but for others, some lower, some maybe higher, I think the most safty way is to check the version agains the mysql server in the proxy startup and combine with your proxy version then set to your constant parameter?
maybe you have your own concern, this is just my opinion...

I understand your concern.
Actually, I checked all the versions as a condition in Connector/J, only 5.6.4 will cause critical troubles. That's why I chose this version.
To explain why hardcode the version number, here are the reasons:

Databases mananged by one Sharding-Proxy instance may have different versions, there's no restricted conditions of the version consistency. In the inconsistent condition, which version should we choose?

Sharding-Proxy does not have different behaviors under different versions. Although we support alterable versions this time, we can't support alterable behaviors. For example, Sharding-Proxy didn't support MySQL protocol before 4.1.

It is not worthy to announce 'Sharding-Proxy can't work if the database versions aren't consistent' just for this issue.

Anyway, I welcome you to submit a pull request, maybe your solution is better, we can discuss more there.

ok, incase you have checked all the versions in Connector/J, it would be enough for this momenet.
Well, for the moment, I don't have a deep knowlege of this sharding product. I am only a new user to this and considering to use this proxy in production if all the test passed ^_^

Hi,
After recompile the source code and test, the insert of the sql with timestamp works ok, I have check the value in the real table, it contains the milliseconds. but when read from the proxy, it miss the milliseconds again.
When I debug the mysql jdbc driver, I found that the value of the length of timestamp field in internalRowData of com.mysql.jdbc.ByteArrayRow are different when connnecting to proxy and database directly.
the length of the value of the timestamp field is 19 when connecting to sharding-proxy
the length of the value of the timestamp field is 23 when connecting to the database directly
I do think there is still a bug in the sharding-proxy for the timestamp value


[Feature Fix] Fix typo and center image of job page in small screen

Purpose
The ID of my github account is wrong on front page and therefore I replace it with the correct one. And also I found the image position on job page is not centered on small screen.
Change

Add correct github ID
Set margin as auto and center the image.

Before Change:

After Change:

Side Effect
None

Awesome! Thanks @haoyuchen1992 !


Prevent log output on Windows

When Daedalus is running on Windows, console GUI shows up while it is running. We should disable it by using silently
Expected: When Daedalus is running, console GUI should not appear
Actual: Console is being displayed while Daedalus is running.

Update:
Using silently does not solve the issue. (TODO): Lookup cabal and stack file of the old launcher code!

I think I've found the solution
https://stackoverflow.com/questions/51015178/how-to-get-rid-of-the-black-console-window-for-a-compiled-haskell-gui-applicatio


Better Folder support

Replace OrganizationUnit with Folder for +19.10 servers.
Support fully qualified folder names.
Add Set-UiPathCurrentFolder to change current folder for a token/session

Fixes #99


UI updates to bottom sheet

This makes the bottom sheet a regular sheet, not a modal. It moves the bottom sheets into ObservationViews where the actual FlatList lives. It snaps the bottom sheet away when a user scrolls down and snaps it back up when a user scrolls to the top of the screen.
Need to get one more test passing.
Closes #177.

Fixed the shadow on Android and changed the prop name to hide.


Rename resetZoom to zoomReset for consistency

I feel like it reads less naturally this way. I’m not a fan

It's a variable name, zoomIn, zoomOut, zoomReset.. category Action.. the copy remains :)

@lipis in/out are not actions though. Could you write zoomIn in the “action category” format? 😛


Limit description text height in panels

Limit panel description text height to a few lines, and enable scrolling

Looks bad with scrolling... maybe something like capping height and letting folks click to see longer descriptions in a dedicated panel? Come back to this


Embed shortcodes should be rendered with templates instead of in PHP

Currently we render shortcodes out with a bunch of HTML::createTag() calls in EmbedShortcodeHandler, which makes it unnecessarily painful to customise the HTML produced. If we switch to using templates, these would be much more easily themeable.
I’d suggest the naming convention:

templates/SilverStripe/View/Shortcodes/EmbedShortcodeProvider_video.ss
templates/SilverStripe/View/Shortcodes/EmbedShortcodeProvider_photo.ss
templates/SilverStripe/View/Shortcodes/EmbedShortcodeProvider_link.ss

The only obvious concern I can see is performance. Templates are parsed + cached of course, so it’s just additional ViewableData lookups we have to worry about. I don’t anticipate a significant slowdown, but happy to profile if anyone’s sufficiently concerned

Are you thinking of just rewriting our existing shortcode implementations to use templates? Or do you want to introduce a new API that would allow you to register shortcodes and directly associate them to a template?

Just converting our existing shortcodes to use templates :)


Added tbColumnDateTimeFilter e2e tests

Coverage remained the same at 97.753% when pulling aa22db08bd1fd7b4d578cdef1e7a6446cf79eac5 on roberttapia:master into db1906fa140e23f8f52482367f28c7eb8cd3e569 on unosquare:master.


1.17 AuxPoW support

Add AuxPoW support to 1.17.
This DOES NOT pass lint tests included in 1.17 Travis, and I'm not entirely sure what to do with that as I'd missed it early on, and it's going to take major changes to fix. Leaning on fixing 1.18, but wanted to flag in case you want to tell me it needs correcting before you'll accept.

@langerhans Can you take a look at this please?


description of issue

Stratux version: Stratux Europe Edition 1.5b2-eu007 BETA
Stratux config: dual  SDR, GPS yes (type?), AHRS yes, battery 20.000 mah Powerbank, battery cable: High quality.
EFB app and version: SkyDaemon latest version. EFB platform: Android 9.0. EFB hardware: Xiaomi Mi Max 3.
Description of your issue.
Tried two different phones, Xiaomi Mi Max 3 and Samsung Galaxy 10, on both platforms the user config data is not stored permanently. After each Power Off and Restart of Startux the configuration of Stratux has to be re-entered. Thanks!

If possible, enable "Replay Logs", reproduce the problem, and provide a copy of the logs in http://<IP_ADDRESS>/logs/stratux/ and http://<IP_ADDRESS>/logs/stratux.log.

Which user config data?


Start-BitsTransfer documentation should have a clear warning that it is not suitable for use in automation

The Start-BitsTransfer commandlet is seemingly a useful tool for initiating bulk file transfers from PowerShell, however one crucial fact is that it depends on the Background Intelligent Transfer Service (BITS) to be running in order to queue up transfer jobs.
In environments where a user is not signed in such as CI, the service is not running and this commandlet is unusable.
There should be a clear warning at the top of this documentation page stating this fact and telling users to avoid attempting to use this commandlet in automation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2ee14777-c3cb-eab0-66af-f75bae048a73
Version Independent ID: 63016f51-2ccf-fdb0-5e29-16418508ce64
Content: Start-BitsTransfer (BitsTransfer)
Content Source: docset/winserver2022-ps/bitstransfer/Start-BitsTransfer.md
Product: w10
Technology: windows
GitHub Login: @JasonGerend
Microsoft Alias: jgerend

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


[Future Request] Create BACNet entities

I was just wondering if it would be possible to add a feature to create entities that'll be available through BACNet or assign the existing ones to the Export Table?
Unless, this is already available? 🤔

Hello,
I just came back from holidays, so apologies for replying late. I've been thinking about this feature as well.
Currently we don't have much of an use case for this, but perhaps when I have some spare time I could implement it.
It could be that you can write down what entities you want in the settings of the add-on. This would make the BACnet device of the add-on create objects that represent these entities in the BACnet network.
Let me know how you think it should be implemented!

It could be that you can write down what entities you want in the settings of the add-on. This would make the BACnet device of the add-on create objects that represent these entities in the BACnet network.

This is exactly what I've meant.
It can be used with various types of devices (especially PLC) and would save hass from processing these in automations.
I was also wondering if there's any way to determinate when the device has lost comms (like disconnected cable of loss of power), unless it's already in.
Thanks!

I was also wondering if there's any way to determinate when the device has lost comms (like disconnected cable of loss of power), unless it's already in.

Currently I don't really keep track of it. I assume the device is there, and poll every minute. Perhaps in the future I can make it a feature to make some kind of alert for it.

Currently I don't really keep track of it. I assume the device is there, and poll every minute. Perhaps in the future I can make it a feature to make some kind of alert for it.

Could you use COV-subscriptions instead of polling? Then poll every 15-30 minutes for a periodic update.
And to check for a device just send a who-is periodically. If no response then declare all points from that device invalid.

Could you use COV-subscriptions instead of polling? Then poll every 15-30 minutes for a periodic update.

This is what's currently happening, a CoV subscription as the main way to get data and a periodic polling as a backup if the CoV fails. The polling rate is configurable, but by default 60 seconds.

And to check for a device just send a who-is periodically. If no response then declare all points from that device invalid.

This is a pretty good idea, might implement this.

If no response then declare all points from that device invalid.

I've been thinking about sending a periodic Who Is request. Once one device fails to respond, what course of action would be appropriate?
It could be that the API just removes that device from the list, and adds it back once it gets a new I Am request.
Another way is to just not update the API points, but keep them available for polling.

I was just wondering if it would be possible to add a feature to create entities that'll be available through BACNet

Just made a start with this, it's out in the dev version if you'd like to play around with it. Not possible to write yet though.

I was just wondering if it would be possible to add a feature to create entities that'll be available through BACNet

@mikihacia I just made a start with this, it's out in the dev version if you'd like to play around with it. Not possible to write yet though.

Just had a chance to play with it and it's been working great so far. Looking forward to test the write function when available.
Doing brilliant job, guys! This stuff should be a part of the core of HASS as the modbus is now.

Writing to present value and out of service should work now, as well as immediate updates when values change. Configuration changed a little bit, so once you reinstall or rebuild after checking updates please check.

I briefly checked and it works like a charm!
This is amazing 🤩.  Thanks!

I'll close this issue, for feature requests Discussions is a great place!


Updating swagger-ui code to the latest version and removing the 'Try it out' buttons

Updating to the latest version to include a recent bug fix which enables using supportedSubmitMethods to decide when should the 'Try it out' buttons be used.

LGTM


Add timeout to requests.post

Maybe do the same for the slack posting code in notifications.py?

other than that :+1:

:+1:


Feature request: Exit status of brew services info xxx indicates the service status

Verification

[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature
The process exit status of brew services info xxx indicates the service status.
What is the motivation for the feature?
For shell scripting:
brew services info xxx >/dev/null 2>&1 && echo "Service Running" || echo "Service Stopped"

This works
 launchctl list homebrew.mxcl.xxxservice>/dev/null 2>&1 && echo "Service Running" || echo "Service stopped"

What alternatives to the feature have been considered?
It is better to add status subcommand for brew service.
brew services status xxx

I don't think we can change the exit code here without breaking existing scripts. Either use --json or grep the brew services list output. Sorry, passing on this.


Replace DetailDisplayer with just using function

Use function to implement something like to_detail_groups is more simple with implement abstract class DetailDisplayer.

Solved by 00c569f5832810e569b7acab37b84c2b6332c24b


Revert "StockLevel fix with NestedLoopJoin and IndexPredicate"

Reverts cmu-db/peloton#445

Coverage decreased (-0.2%) to 64.808% when pulling 2dce70b32d99f18783a212966a6def8c5133c8e4 on revert-445-master into 7287a10f334825d0c18f970a73748d49c1ae3a58 on master.

Coverage decreased (-0.1%) to 64.937% when pulling 2dce70b32d99f18783a212966a6def8c5133c8e4 on revert-445-master into 7287a10f334825d0c18f970a73748d49c1ae3a58 on master.


Fixes to the NoState Prefetch article

What's changed, or what was fixed?
In the Introducing NoState Prefetch article,

Replaced an "a" preceding "HTML document" with "an".
Added backticks around sessionStorage and localStorage.
Added a missing comma.

CC: @petele

:tada: This has been pushed live to https://developers.google.com/web/


generate exercises from goals + limit to 4

instead of hard coding them

post pone, this is quite a challenge to do well, actually, because I need to put in some more logic


Fails to build due to problems with boolean types and constants

As reported in Debian Bug #1012231:

tinyows recently started to FTBFS after some build-dependency got updated:
   dh_auto_build
        make -j3
make[1]: Entering directory '/build/tinyows-1.2.0'
gcc -g -O2 -ffile-prefix-map=/build/tinyows-1.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wl,-z,now -Wdate-time -D_FORTIFY_SOURCE=2 -std=c99 -pedantic -Wall -I/usr/include/postgre
sql -I/usr/include/libxml2   src/fe/fe_comparison_ops.c src/fe/fe_error.c src/fe/fe_filter.c src/fe/fe_filter_capabilities.c src/fe/fe_function.c src/fe/fe_logical_ops.c src/fe/fe_spatial_ops.c src/mapfile/mapfile
.c src/ows/ows_bbox.c src/ows/ows.c src/ows/ows_config.c src/ows/ows_error.c src/ows/ows_geobbox.c src/ows/ows_get_capabilities.c src/ows/ows_layer.c src/ows/ows_metadata.c src/ows/ows_psql.c src/ows/ows_request.c
 src/ows/ows_srs.c src/ows/ows_storage.c src/ows/ows_version.c src/struct/alist.c src/struct/array.c src/struct/buffer.c src/struct/cgi_request.c src/struct/list.c src/struct/mlist.c src/struct/regexp.c src/wfs/wf
s_describe.c src/wfs/wfs_error.c src/wfs/wfs_get_capabilities.c src/wfs/wfs_get_feature.c src/wfs/wfs_request.c src/wfs/wfs_transaction.c src/ows/ows_libxml.c -o tinyows -lfl -L/usr/lib/x86_64-linux-gnu -lpq -lxml
2 -lfcgi
In file included from /usr/include/unicode/umachine.h:52,
                 from /usr/include/unicode/utypes.h:38,
                 from /usr/include/unicode/ucnv_err.h:88,
                 from /usr/include/unicode/ucnv.h:51,
                 from /usr/include/libxml2/libxml/encoding.h:31,
                 from /usr/include/libxml2/libxml/parser.h:812,
                 from /usr/include/libxml2/libxml/globals.h:18,
                 from /usr/include/libxml2/libxml/threads.h:35,
                 from /usr/include/libxml2/libxml/xmlmemory.h:218,
                 from /usr/include/libxml2/libxml/tree.h:1307,
                 from /usr/include/libxml2/libxml/xmlreader.h:14,
                 from src/fe/../ows/ows.h:30,
                 from src/fe/fe_comparison_ops.c:29:
src/fe/../ows/../ows_struct.h:33:3: error: expected identifier before numeric constant
   33 |   false,
      |   ^~~~~
src/fe/../ows/../ows_struct.h:37:19: error: two or more data types in declaration specifiers
   37 | typedef enum Bool bool;
      |                   ^~~~
In file included from src/fe/../ows/ows.h:38,
                 from src/fe/fe_comparison_ops.c:29:
src/fe/../ows/../ows_struct.h:37:14: warning: empty declaration with storage class specifier does not redeclare tag
   37 | typedef enum Bool bool;
      |              ^~~~
[...]

Full buildlog

There seems to be a conflict between stdbool.h as included by ICU in umachine.h and the definitions in ows_struct.h.
This patch works around the issue:
--- tinyows-1.2.0.orig/src/ows_struct.h
+++ tinyows-1.2.0/src/ows_struct.h
@@ -29,12 +29,12 @@

 /* ========= Structures ========= */

-enum Bool {
-  false,
-  true
+enum Bool_ {
+  FALSE = 0,
+  TRUE
 };

-typedef enum Bool bool;
+typedef enum Bool_ bool_;

 #define BUFFER_SIZE_INIT   256

@@ -107,7 +107,7 @@ typedef struct Ows_layer_storage {
   buffer * pkey;
   buffer * pkey_sequence;
   buffer * pkey_default;
-  bool is_geographic;    /* true for a geographic CRS (or a compound CRS
+  bool_ is_geographic;   /* true for a geographic CRS (or a compound CRS
                             whose base is geographic), false for a projected
                             CRS (or a compound CRS whose base is projected) */
   array * attributes;
@@ -117,11 +117,11 @@ typedef struct Ows_srs {
   int srid;
   buffer * auth_name;
   int auth_srid;
-  bool is_geographic;              /* true for a geographic CRS (or a compound
+  bool_ is_geographic;             /* true for a geographic CRS (or a compound
                                       CRS whose base is geographic), false for
                                       a projected CRS (or a compound CRS whose
                                       base is projected) */
-  bool is_axis_order_gis_friendly;   /* true for a CRS whose axis order is
+  bool_ is_axis_order_gis_friendly;  /* true for a CRS whose axis order is
                                         typically easting, northing (e.g most
                                         projected CRS, such as EPSG:32631)
                                         false for example for EPSG:4326 (WGS 84),
@@ -129,10 +129,10 @@ typedef struct Ows_srs {

   /* The two below fields are not properties of the SRS, but of its context
    * of use. */
-  bool honours_authority_axis_order; /* true for a context where the axis order
-                                        as defined by the authority should be
-                                        respected */
-  bool is_long;                      /* true for a context where the srsname must
+  bool_ honours_authority_axis_order; /* true for a context where the axis order
+                                         as defined by the authority should be
+                                         respected */
+  bool_ is_long;                    /* true for a context where the srsname must
                                         be exported as a long URN */
 } ows_srs;

@@ -184,8 +184,8 @@ typedef struct Ows_layer {
   buffer * name_prefix;     /* Nominally concatenation of ns_prefix:name_no_uri, e.g. "tows:world" , or name_no_uri if no ns_prefix */
   buffer * name_no_uri;     /* the name as in the "name" attribute in the config, e.g "world" */
   buffer * title;
-  bool retrievable;
-  bool writable;
+  bool_ retrievable;
+  bool_ writable;
   list * srid;
   ows_geobbox * geobbox;
   buffer * abstract;
@@ -346,8 +346,8 @@ enum fe_error_code {
 };

 typedef struct Filter_encoding {
-  bool in_not;
-  bool is_numeric;
+  bool_ in_not;
+  bool_ is_numeric;
   buffer * sql;
   enum fe_error_code error_code;
 } filter_encoding;
@@ -370,10 +370,10 @@ typedef struct Ows_request {
 #define OWS_MAX_DOUBLE 1e15  /* %f vs %g */

 typedef struct Ows {
-  bool init;
-  bool exit;
+  bool_ init;
+  bool_ exit;
   PGconn * pg;
-  bool mapfile;
+  bool_ mapfile;
   buffer * config_file;
   buffer * schema_dir;
   buffer * online_resource;
@@ -396,12 +396,12 @@ typedef struct Ows {
   int max_features;
   ows_geobbox * max_geobbox;

-  bool display_bbox;
-  bool expose_pk;
-  bool estimated_extent;
+  bool_ display_bbox;
+  bool_ expose_pk;
+  bool_ estimated_extent;

-  bool check_schema;
-  bool check_valid_geom;
+  bool_ check_schema;
+  bool_ check_valid_geom;

   array * cgi;
   list * psql_requests;

Using stdbool.h instead of the custom type is also an option:
--- tinyows-1.2.0.orig/src/ows_struct.h
+++ tinyows-1.2.0/src/ows_struct.h
@@ -24,18 +24,12 @@
 #ifndef OWS_STRUCT_H
 #define OWS_STRUCT_H

+#include <stdbool.h>
 #include <stdio.h>    /* FILE prototype */

 /* ========= Structures ========= */

-enum Bool {
-  false,
-  true
-};
-
-typedef enum Bool bool;
-
 #define BUFFER_SIZE_INIT   256

 typedef struct Buffer {

But this results in warnings:
src/fe/fe_logical_ops.c: In function 'fe_unary_logical_op':
src/fe/fe_logical_ops.c:104:13: warning: increment of a boolean expression [-Wbool-operation]
  104 |   fe->in_not++;
      |             ^~
src/fe/fe_logical_ops.c:115:13: warning: decrement of a boolean expression [-Wbool-operation]
  115 |   fe->in_not--;
      |             ^~

@sebastic using stdbool.h gives no such warnings here on Windows, with Visual Studio (and tested against PostgreSQL 14.3/PostGIS 3.2.1).   This seems like the proper way (?).

@sebastic I'm now able to reproduce the warnings, and your last patch works here.  Can you turn this into a PR ?  (I refer to ows_struct.h/fe_logical_ops.c ).  thanks


Aggregate or down-sample events

In order to relieve pressure downstream and to avoid collation complexity, would be nice if I could aggregate logs within logagent-js.  I'll try to explain more with an example.  I would like logs such as:
{message: 'foo happened', field1: 'type1', metric1: 42, metric2: 1}
{message: 'foo happened', field1: 'type2', metric1: 24, metric2: 2}
{message: 'foo happened', field1: 'type1', metric1: 40, metric2: 3}

to be user-definably aggregated with something like the following before shipping them:
function aggregateFoo(pattern, buffer) {
  if (buffer.foo === undefined) {
    buffer.foo = {};
  }
  if (buffer.foo[pattern.field1] == undefined) {
    buffer.foo[pattern.field1] = {metric1: [], metric2: [], first: now()};
  }
  var agg = buffer.foo[pattern.key];
  agg.metric1.push(pattern.metric1);
  agg.metric2.push(pattern.metric2);
  agg.last = now();
  // Could aggregate in N-minute minimum buckets
  if (agg.last - agg.first > '5 minutes') {
    shipFoo(agg);
  }
  // Could aggregate once N samples collected
  if (agg.metric1.length > 100) {
    shipFoo(agg);
  }
  delete buffer.foo;
}

function shipFoo(agg) {
  // Demonstrating how users may want to define the aggregated event.
  var event = {};
  event.message = 'foo happened ' + agg.metric1.length + ' times';
  event.ts = agg.last;
  event.window = agg.last - first;
  event.metric1 = avg(agg.metric1);
  event.metric2 = max(agg.metric1);
  ship(event);
}

Is this already on the road map, and if not, is it in the scope of what you would like for logagent-js?  I may have the green light to hack my way through this.  If could be generally useful to others, let's discuss further here on how best to approach?

+1 nice!

Looks useful at first glance.  One may want to know the exact time window from which these aggregations were derived (so 'from' and 'to' timestamps may need to be tracked and simply shipped as fields).
Logagent now has support for pluggable inputs and outputs.  Would this be an output, or would it be better to be able to allow pluggable "processors" (the stuff between inputs and outputs)?

My questions:

Is this a parser feature to aggregate messages? Then it would work for all tools using the parser.
No need to change current workflow.
Is this a logs shipper feature, then it needs to be a pluggable "processor". Currently we have only input and output plugins. Inputs emit "data.raw" events, LA does paring and emits then "data.parsed" events. Output plugins listen to "data.parsed".  A "filter/processor plugin" needs to run before "data.parsed" is emitted -> little change in the framework.

@otisg, I think the stuff between.  Plus, there's state to manage, which probably makes this a bit more complicated than an input or output.  Regarding which fields to ship, my hope is that could be completely defined by the user since I expect many different use cases.
@megastef, not sure I know enough to answer.  Now that #22 is complete, I think I can define my filter() to drop the event if my transform() marked it as aggregated.  I think all I may need is some temporary buffer space to store the relevant bits of the event.  Does it make sense to use the pattern object for that or would that be too kludgy?  My transform() would need access to it in order to return the aggregated event as desired (eg time, count).  That would be my naive way of going about it.

Well using #22 is something which would work in the "user land". Setting pattern.aggBuffer or pattern.aggTimer. The pattern object gets only destroyed when the patterns.yml changes (hot-reload).
I think we need filter plugins in any case and it would be the best place for aggregation, before data is passed to output plugins. A simple default could just load a JS function (inline in config) or from a JS file/module.

@erik-stephens I think this is what are looking for. https://www.npmjs.com/package/logagent-filter-output-sql. This outputFilter plugin provides SQL based queries to filter or aggregate log events (already parsed by patterns applied before ...).

Very cool!  Quickly scanned the source of that plugin.  The buffer is flushed based on configured interval?  So if I set interval to 300, it will buffer all events for past 5 minutes and flush the buffer?  That should work nicely for our needs.
If haven't thought about it yet, might want to consider what is expected behavior if user feeds it a bad query.  If it's possible for the query to raise an exception, then buffer might blow up.  Also, looks like buffer will continue to grow if query returns no results.  I would expect buffer to be flushed in that case as well.

Yes, the interval is configurable (value in seconds). Please note that the buffer keeps all records for this time in memory (e.g. 5 minutes in your example). All SQL operations are currently in-memory and the required memory for logagent grows with the number of buffered events until the SQL queries are executed.
Fixed error handling https://github.com/megastef/logagent-filter-output-sql/commit/ce9b45f8ee0d20848547c9083a610d60c47b943b
Thanks for the great feedback.

Apologies if I'm mis-reading the code, but I think makes sense to still emit data.parsed with an empty list when query returns no results - looking at the return if (result && result.length === 0) bit.  As a user of this functionality, I may want to emit that aggregation as a zero event.  Plus, looks like buffer could still grow unbounded if my query never matches any events.  Over all I think this is great, so feel free to close this issue if happy with it - don't want you to feel like have to wait for my "blessing".

Tested the sql output plugin.  One bit I'm missing is how to process the results of the plugin - still needing some user-defined javascript to fill the gaps that alasql leave for our use cases.  I'd like it emitted back up the stream somewhere besides where emit(data.parsed) takes it.  I don't think data.raw makes sense all the time either which has me wondering if there are plans for letting users configure the processing pipeline.  I'm wondering if a distinction needs to be made between input filters, processors, and output filters as mentioned in #43 and instead let processors subscribe to msgs and publish them on specific channels.  I'm using the term "channel" but I think as simple as the event name to subscribe (on()) & publish (emit()) to.  Spit-balling my ideal setup:

I define an input and specify the channel to emit to, let's say my.foo.  The default could be data.raw to preserve current behavior.
I define a sql filter to process msgs on a channel my.foo and emit the processed msg to channel my.bar.
I define a user-defined function filter for msgs on channel my.bar.  Maybe I drop the msg, maybe I perform some side-effect and emit the msg unmodified, maybe I modify the msg.  For this specific example, let's say I emit it to channel my.elasticsearch.
My elasticsearch output filter is configured to index msgs on my.elasticsearch.  The default channel could be data.parsed to preserve current behavior.

If it's as simple as I think it is, I could come up with a PR.  Wanted to gauge where y'all are on this before diving in.  Maybe have other plans or see problems with this approach?

@erik-stephens This is a very good suggestion. Please tell us about:
One bit I'm missing is how to process the results of the plugin - still needing some user-defined javascript to fill the gaps that alasql leave for our use cases.
We have not documented latest Elasticsearch output features like routing logs to different servers/indicies.
What you describe above is exactly the initial idea we had for "Event based architecture".
Down the road it was was not so easy or is not completed:

We used initially https://github.com/asyncly/EventEmitter2 to have wildcard events, but for performance reasons we moved back to default EventEmitter
What you describe above requires to make "listen" and "emit" event names configurable for all existing plugins - this was simply not done (one could say incomplete move to new architecture) and I think it is not much work to add it to all existing input/output plugins (not that many ...). The logagent parser is currently a central functions, where all data flows through and this might needs to be changed to hook it into custom event processors.
Using the "event channels" will create more events than we need - 100k logs might generate 600k or more events at least this was one of my considerations for resource foot print
Input/Output Filters are a bit different than input/output plugins. Filters (including transforms) must  be "serialized" and we decided for generator functions called in "serial" way.  Connecting multiple filters will need configuration for N listen and emit event names - at this point my opinion was it would be too complicated for regular users to understand this.

We are open for good suggestion or PR and we like to improve the usability and reduce JS programming parts for the end users.
Just to complete the current status, we can have workflows like this:
Input File
  - watch files by GLOB patterns
  - emit 'data.raw' new lines from each changed file with a context having source name
  -> Apply N Input Filter (e.g. Grep filter)
    - match source name from 'context'
    - drop | emit depending on 'data' or source
      -> Parser
         - match patterns by sourceName from context
         - match regex / parse structure
         -> Apply N  Output Filter (e.g SQL filter)
            - transform, aggregate,
            - drop/emit 'data.parsed'
            -> N Output Plugins (e.g. Elasticsearch)
               - listen to 'data.parsed'
               - route to Elasticsearch URL and map source name to URL/indices
                 - match source name by regex
                 - map to index name (config)

```

An example config might look like this (sorry it might make much sense, but it outlines the options):
```yaml
input:
  files:
      - '/var/log/**/*.log'
      - '/opt/mylogs/*.log'

inputFilter:
  - module: grep
    config:
      matchSource: !!js/regexp /nginx/
      include: !!js/regexp /.*/i
      exclude: !!js/regexp /OPTIONS/i

outputFilter:
  - module: sql
    config:
      source: !!js/regexp /access.log|httpd|nginx/
      interval: 1
      queries:
        - # calculate average page size for different HTTP methods
          SELECT 'apache_stats' AS _type,
                  AVG(size) AS size_avg,
                  COUNT(method) AS method_count,
                  method as http_method
          FROM ?
          WHERE method is not null
          GROUP BY method

output:
  es-secure-local:
    module: elasticsearch
    url: http://localhost:9200
    httpOptions:
      key: ./client.key
      cert: ./server.crt
      rejectUnauthorized: false
    indices:
      my_secure_logs_index:
        - system\.log
        - access\.log
        - auth\.log
        - wifi.log
  logsene-saas:
    module: elasticsearch
    url: https://logsene-receiver.sematext.com
    indices:
      269d7b2f-0e7d-4e92-ae7f-xxxxxxxxxx:
        - access.log
        - nginx.log
```

One bit I'm missing is how to process the results of the plugin - still needing some user-defined javascript to fill the gaps that alasql leave for our use cases.

I'm trying to consume the systemd journal.  I would like to index all msgs for specific scheduled jobs as one document in elasticsearch.  That's where the sql output filter helps quite a bit but there are some things that are awkward or impossible to do in sql alone:
SELECT
  'foo' as `process`,
  ARRAY(MESSAGE) as `message`,
  ARRAY(DISTINCT _HOSTNAME) as `hostname`,
  `end` - `start` as duration,
  MIN(_SOURCE_REALTIME_TIMESTAMP) as `start`,
  MAX(_SOURCE_REALTIME_TIMESTAMP) as `end`,
  (MAX(_SOURCE_REALTIME_TIMESTAMP) - MIN(_SOURCE_REALTIME_TIMESTAMP)) / 1000000 as `duration`
FROM ? WHERE
  _SYSTEMD_UNIT = "foo.service"

On a related note, I would also like to index the other journal msgs that I'm not aggregating.  Looks like I have to run logagent multiple times: once to feed it msgs that I want aggregated and once to feed the other msgs.  What I'm hoping for is logagent to provide primitives for working with the msg bus/pipeline and let me tap into it various ways.

Using the "event channels" will create more events than we need - 100k logs might generate 600k or more events at least this was one of my considerations for resource foot print

I'm seeing that as user/configuration dependent.  If I want a more complicated processing pipeline, then I would expect a performance tax.

Input/Output Filters are a bit different than input/output plugins. Filters (including transforms) must be "serialized" and we decided for generator functions called in "serial" way. Connecting multiple filters will need configuration for N listen and emit event names - at this point my opinion was it would be too complicated for regular users to understand this.

I was actually having a difficult time understanding the pipeline as it is now.  Your current workflow description helps, though.  I would think having the workflow more user-defined would make it easier for users to understand, but just my opinion.
I'll take a crack at a PR. If goes no where, then no problem.  Will be a good learning exercise.

Documentation is one of the tasks we are working on - including explanation of the pipeline.
Input/Output plugins are event driven, while input filters are hooked before parsing, and output filters after parsing. Then the output plugins listen to the "data.parsed" events. I made a picture, but too complicated to share :)

On a related note, I would also like to index the other journal msgs that I'm not aggregating.
You could use multiple SQL statements in the "queries" section, and use multipe "outputFilter: module: sql".

BTW, for journalctl/json input you could use JSON processing in patterns.yml (before output filter), maybe a helpful example.
json:
  removeFields:
    - __CURSOR
    - __MONOTONIC_TIMESTAMP
    - _TRANSPORT
    - JOURNAL_NAME
    - JOURNAL_PATH
    - CURRENT_USE
    - CURRENT_USE_PRETTY
    - MAX_USE
    - MAX_USE_PRETTY
    - DISK_KEEP_FREE
    - DISK_KEEP_FREE_PRETTY
    - DISK_AVAILABLE_PRETTY
    - DISK_AVAILABLE
    - LIMIT
    - LIMIT_PRETTY
    - AVAILABLE
    - AVAILABLE_PRETTY
    - _CAP_EFFECTIVE
    - _SYSTEMD_SLICE
  unitFilter: !!js/regexp /sematext.*/i
  transform: !!js/function >
    function (sourceName, parsedObject, config) {
       // filter for unit names
       if (! config.uniFilter.test(parsedObject['_SYSTEMD_UNIT'])) {
        parsedObject.logagentDropMessage = true
        return
      }
      parsedObject.logSource = parsedObject['_SYSTEMD_UNIT'].replace('.service','')
      parsedObject['@timestamp'] = new Date(Number(parsedObject['_SOURCE_REALTIME_TIMESTAMP']))
      for (var i=0; i<config.removeFields.length; i++) {
        // console.log('delete ' + config.removeFields[i])
        delete parsedObject[config.removeFields[i]]
      }
     }

And we have an open task for better journald support, while we have that in Sematext Docker Agent:
https://github.com/sematext/sematext-agent-docker/blob/master/lib/tcpLogsene.js
Anyhow, you are welcome to make a PR to get your workflow done.

Apologies for length of this thread and this post in particular.  It's a lot to digest.  I hacked away and have a version that I'm pretty happy with.  It might be too radical a departure from your vision, so would like to gauge that before spending time getting it into a more PR-friendly form.
Everything is a stage in the pipeline - there is no distinction twix inputs, filters, processors, outputs other than how the user wires them up in the pipeline.  Things I like about this approach:

Plugins have a consistent interface: consume events, do stuff, and optionally emit events.  The core can be much simpler - doesn't have to worry about details like input vs output filters.

Allows for msgs to be split into multiple streams/paths and merged.  Allows plugins to applied multiple times in different contexts.

It can be broken up into single-purpose stages or more monolithic ones - whatever makes sense for the user.

Easy enough to debug by injecting debug stages or adding a debugger & stepping through it with a node debugger.

Things I don't like or am not sure about:

The pipeline definitions can be difficult to understand and easy for users to mess up.  However, I think that could be handled with tooling: something to detect loops, dead paths, visualization of the flows, etc.

Performance.  It's fast enough for our needs.  Using the pipeline definition below sans the elasticsearch indexing stage, I get almost 15k msgs/sec on a late 2013 macbook pro.

To give you a sense of the implementation, logagent.js simply provides a name/id, config, and EventEmitter instance (called pipeline below) to the plugins.  It doesn't do much else but listen for termination events.  Most of the implementation gets pushed out to the plugins, even the stats and error handling could be defined as pipeline stages:
$ cat lib/plugins/js.js
'use strict'

function Plugin (name, config, pipeline) {
  var self = this
  self.name = name
  self.pipeline = pipeline
  self.config = config
  self.sandbox = {
    util: require('util'),
    moment: require('moment')
  }
  self.pipeline.on(self.config.inChannel, function (context, data) {
    try {
      context.start = new Date()
      context.stage = self.name
      context.dropped = true
      var copy = Object.assign({}, data)
      if (self.config.function.call(self.sandbox, copy)) {
        self.pipeline.emit(self.config.outChannel, {stage: self.name}, copy)
        context.dropped = false
      }
      context.end = new Date()
      self.pipeline.emit(self.config.statsChannel, context)
    }
    catch (err) {
      self.pipeline.emit(self.config.errChannel, err, context)
    }
  })
}
module.exports = Plugin

Example config defining a pipeline that I'm using to process journalctl -o json | logagent.js.:
pipeline:

  stdin:
    module: stdin

  lines:
    module: lines
    inChannel: stdin

  json-in:
    module: json-in
    inChannel: lines

  # Drop all fields except for the ones I'm interested in.
  clean-journal:
    module: keep
    inChannel: json-in
    # outChannel: clean-journal # not needed since defaults to stage name
    fields:
      hostname: ['_HOSTNAME', '_MACHINE_ID']
      severity: PRIORITY
      process: ['UNIT', '_SYSTEMD_UNIT', 'SYSLOG_IDENTIFIER']
      message: MESSAGE
      container: ['CONTAINER_NAME', 'CONTAINER_ID']
      ts: ['_SOURCE_REALTIME_TIMESTAMP', '__REALTIME_TIMESTAMP']

  # Split the "clean-journal" stream into two: aggregated by specific processes and not.
  unaggregated:
    module: js
    inChannel: clean-journal
    outChannel: cast
    function: !!js/function >-
      function(data) {
        return !data.process.match(/(foo|bar)/)
      }

  # I want output of these scheduled jobs to be indexed as one msg.
  aggregated:
    module: sql
    inChannel: clean-journal
    interval: 60 # seconds
    query: >-
      SELECT
        `process`,
        ARRAY(`message`) AS `message`,
        ARRAY(DISTINCT `hostname`) AS `hostname`,
        MIN(`ts`) AS `ts`,
        MAX(`ts`) AS `end`,
        (MAX(`ts`) - MIN(`ts`)) / 1000000 AS `duration`
      FROM ?
      WHERE
        `process` IN ("foo.service", "bar.service")
      GROUP BY `process`

  # Handle some transormations couldn't figure out how to do in sql.
  post-aggregation:
    module: js
    inChannel: aggregated
    outChannel: cast
    function: !!js/function >-
      function(data) {
        data.message = data.message.join('\n')
        data.hostname = data.hostname[0]
        return true
      }

  # A simple plugin to handle common task of casting to desired types.
  cast:
    module: cast
    inChannel: cast
    outChannel: output
    fields:
      severity: int
      ts: ts-usec
      end: ts-usec

  # Split the msg to multiple outputs: stdout & elasticsearch.
  json-out:
    module: json-out
    inChannel: output
    outChannel: stdout
  stdout:
    module: stdout
    inChannel: stdout

  elasticify:
    module: js
    inChannel: output
    outChannel: es
    function: !!js/function >-
      function(data) {
        if (data._type === undefined) {
          data._type = 'syslog'
          data.type = data._type
        }
        if (data._id === undefined) {
          // Specify a deterministic _id so can re-process logs without duplicates.
          data._id = this.util.format('%s::%s', data.hostname, data.ts.toISOString())
        }
        return true
      }
  elasticsearch:
    module: elasticsearch
    inChannel: es
    # Log to monthly index.
    index: !!js/function
      function (data) {
        return this.util.format('logs-%s', data.ts.format('YYYY.MM.DD')
      }
    api: '2.3'

  # Examples showing how stats & errors could be handled in user-defined way. By
  # default, a stage will emit('stats', {...}) and emit('errors', err, {...})
  # but those channels could be overridden on a per-stage basis.  Feeding
  # to an aggregator stage would be a nice marriage.
  stats:
    module: js
    inChannel: stats
    function: !!js/function >-
      function(context) {
        console.log('stats for stage %s:', context.stage, context)
      }

  errors:
    module: js
    inChannel: errors
    function: !!js/function >-
      function(err, context) {
        console.error('error in stage %s:', context.stage, err)
      }

Seems juicy, will digest.  Have you compared before/after performance with ~equivalent configs to see if these modifications change CPU or memory footprint, and roughly how much?

@erik-stephens could please share your changes? This could be on github or a mail with JS files, so I could play around with your version and see the impact to existing modules (re-use/change/re-write) or the command line interface and @otisg  ideas.

@erik-stephens @megastef:

how would these stages be connected together? with queues? (sorry if this is an ignorant Node.js question)
what happens when one stage is much slower than the others? it creates backpressure?
if one has a stage that is very slow, could one run multiple instances of that stage to improve the overall throughput?

I won't be able to respond fully until early next week, but wanted to respond quickly to your comments:

Since seem more receptive than I thought, I'll go ahead and share what I got.  I clobbered a lot of code more to help me get it working quicker.  That's why I was worried about PR'ing first.  I'm also working on a more apples-to-apples comparison of EPS & memory consumption.

I really like your idea on decoupling stage definitions from how they're wired in the pipeline.

Regarding duplicate named stages, I lean towards raising an exception since allowing for stages to be overridden could make it difficult to reason about a pipeline definition.

A slow stage would increase latency anywhere it's in the stream, but not backpressure since things are non-blocking by default in node.js land.  I'm taking advantage of built-in node.js event machinery, so no need for queues.  I'm new to node.js myself, but I assume node.js will handle the buffering of unhandled events when a stage becomes a bottleneck.  Not sure about the details such as how much memory consumption and how far it can be pushed.

I think node.js suffers from similar single thread issues that the likes of python & ruby do, so even if we could spawn multiple stages it wouldn't buy us much.  I come from a python background and spawning multiple processes is my goto for dealing with its global interpreter lock (GIL) when cpu bound.

A pipeline could have multiple streams/paths of event flow.  I think makes sense to limit one logagent process to one pipeline.  If stage definitions were decoupled from pipeline definitions like you suggest, then breaking out to multiple logagent instances should be a low-fuss way to work around that: one logagent instance for processing pipeline A, one for pipeline B, etc.

Running multiple Logagent instances for multiple pipelines doesn't sound great to me from the operational perspective:

each JS VM has some overhead
starting Logagent may require sudo, etc. so it may not be possible for some people to start
imagine a SaaS like Logsene that can have a GUI for configuring a pipeline, which Logagent could then get.  If 1 Logagent can handle N pipelines defined in N configs, then all that would need to be done would be to drop another pipeline config in the appropriate dir

One more question:

what happens with data that is in-memory/in-flight when Logagent is shut down in this new model? is it lost? does each stage persist it and can pick it up on restart?

gentle nudge @erik-stephens.  We are hoping to nail down the direction ASAP and get a proper release before Santa comes to town.

Apologies for the delay.  You can find a working copy here.  I developed against a journalctl dump and am just now testing against a live system with journalctl being fed to it.  I stomped on a lot of code (more to simplify for me) - I'm considering my version as something to cherry-pick from and nothing close to a PR.
I spent some time trying to get it to work with the stages de-coupled from the pipeline but it was turning out to be just as messy.  Might still be worth considering, but for the sake of time, I abandoned that and instead added support for multiple input & output channels (see my ./config/example.yml).
As it is now, in-memory data will be lost on shutdown.  I'm sure there are a handful of operational details like that would need some more attention before general use.
I haven't gotten to an apples-to-apples comparison about memory consumption.  I don't even have file support in my version yet.  I was seeing some concerning memory usage when feeding it my test file.  It was a 1GB journal dump and it seemed to exhibit leaky behavior.  I went down a bit of node.js memory mgmt rabbit hole and ended up adding a gc stage to allow for scheduled garbage collections but not sure how much it helps - I guess could be used as a blunt instrument / band-aid.  I've had it reading a journal in either 10k-record or 1-minute chunks (which first) and memory is holding steady at around 60MB with the normal gc scheduling.
Next steps for me are:

Deploy this to more servers and see how it performs.
Add file support.
Make sure error handling is solid.

If you guys end up trying this out and get stuck, let me know.  I think it only supports running it with the -c CONFIG  The relatively simple pipeline definition I'm working on had me stumped a few times.  Funny how I rail against DSL's and kinda created one.

Thx @erik-stephens, I will have a look - and might need a few days to understand, review and test a few things.
I see massive changes, a few quick thoughts:

Elasticsearch client is used, I assume we have then no disk buffer when ES is offline/rejecting requests. Disk buffers are part of 'logsene-js'. In addition we do there some checks e.g. for invalid field names etc. - on the other hand no big issue, we could implement "Logsene Output" to get disk buffers back.
bunyan logger, LA logged to stderr only, to have stdin/stdout free for piping data from/to unix tools in addtion our own logger could print line numbers/calling function etc in debug mode. When installed as service the stderr logs got collected by systemd or upstart. So I will check bunyan settings as well.
explicit GC: a clear indication that something is wrong in the workflow (use streams is the only alternative or wrap event emitters in some streaming lib ... ). One way to limit memory consumption is to limit the input rate of data sources, so the event loop gets not filled with too many pending actions ... clearly a problem of the "event driven" architecture. Limiting input rates is done in the current LA for TCP and stdin, but not for file input (yet).

Funny how I rail against DSL's and kinda created one.

I fully agree, I thought having "JS plugins" in every place, JS could be used to tricky tasks. Or making the config syntax easy for each plugin...
And the "original" aggregation use case with start/stop events cold be solved in the current version with a filter-plugin, which drops the start events, buffers the start time in "config" object and emits a new event with start/end and duration once the stop event arrives.


Reduce array allocations via ary.each_with_object

What problem is this PR intended to solve?
Unnecessary object allocations:

array.map duplicates given Array array
Array#[] generates a new Array instance.

Therefore, Hash[array.map { |e| [e, foo] }] generates multiple interim arrays on each call and iteration.
Have you included adequate test coverage?
Not necessary since this is just refactoring existing implementation.
Does this change affect the C or the Java implementations?
Does not affect C or Java code

Merged manually! Thank you!

(see 63c2b74)


nixos/tt-rss: supply --force-yes to update-schema

This commit fixes the service failing to start for the first time since the update-schema operation requires human interaction (typing 'yes') in order to actually perform the schema upgrade.
Description of changes
This module does not seem to have a NixOS test, so here's a little test I've done:

Status quo

Service fails to start because it requires human interaction:
Oct 07 16:47:03 cobalt systemd[1]: Starting Tiny Tiny RSS feeds update daemon...
<...>
Oct 07 16:47:04 cobalt tt-rss[2323512]: [16:47:04/2323512] Lock: update.lock
# Human interaction required here
Oct 07 16:47:04 cobalt tt-rss[2323512]: [16:47:04/2323512] Type 'yes' to continue.
Oct 07 16:47:04 cobalt systemd[1]: tt-rss.service: Control process exited, code=exited, status=1/FAILURE

This commit
Human interaction is skipped by supplying the --force-yes flag:

# sudo -u tt_rss /nix/store/b2samp9wri1221ymplzcabz7mdfqbnzf-php-with-extensions-8.1.24/bin/php /var/lib/tt-rss/www/update.php --update-schema --force-yes
[16:51:43/2332896] Lock: update.lock
[16:51:43/2332896] Proceeding to update without confirmation.
[16:51:43/2332896] Loading base database schema...
[16:51:45/2332896] Migration finished, current version: 147

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2741


Add the possibility to save and restore checkboxes

Hi !
First of all, thanks for this wonderful plugin.
One nice feature would be to be able to save and restore the state of check boxes (to follow you progress while sourcing components, in example).
The absolute best would be to be able to save the whole html with the checkboxs state embedded by simply hitting Ctrl-S but anything less optimal (using an intermediate file ?) would be nice too.
Best regards,
Julien

Use a decent browser (i.e. not edge) and it will save them for you.
Or are you talking about transferring checkboxes state between different computers?

I’m using up-to-date firefox on Ubuntu and, unless I missed something, it doesn’t save anything.
Moreover it would be nice to be able to share through several computers, but it is not my main point.

I just tried this on latest firefox on debian, it definitely saves the checkboxes and settings (like dark mode etc) across refreshes and restarts of the browser.
Couple questions:

Is html bom openeded from local filesystem (i.e. url starts with file:///) or is it hosted somewhere, even if it's http://localhost ?
Open storage viewer in firefox (shift+f9) and go to localstorage, select opened file. Play with the settings and checkboxes. Do you see values for your file change? Keys are prefixed with "KiCad_HTML_BOM__your_board_name__your_board_rev". When you restart browser, open bom file and open storage viewer again, are the values still there?

@qu1ck
if you store your settings in localstorage, those will be deleted if the browser is set to "Delete cookies and site data when Firefox Developer Edition is closed"
This is a quite common setting for privacy ...
Would it be possible to add an option to save the settings in a '.ini' file beside the html file itself?

@qu1ck HTML BOM is oppened from the local filesystem (file://<stuff> URL).
I will fiddle around with Firefox parameters later today.

@qu1ck For whatever reason, it seems to work fine today.
I still think that it would be interesting to save those data in some actual file, but I'm fine with the current state.

I'll keep export/import of settings and checkboxes as a wishlist item.


The functions for "match the whole word" and "match case sensitive" fail

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
Below is an example for "match case sensitive", where I first pressed Alt+C, and then, by selecting x1 , typedShift+Ctrl+L to match all. But it still matches X1 (which should only match x1 with lowest case).
To show that this is not because that I am already in "case sensitive mode" before pressing Alt+C, I pressed Alt+C again, still, X1 is being matched.
Also from the search panel on the top right, it shows that the icon "Aa" is not activated when I pressed Alt+C.  (Though I can manualy click that icon, but nothing changes but the icon will be highlighted, as if it is activated, but still did not do the right matching)
The same issue happens to Alt+W (match the whole word) too.

This only happens in Jupyter notebook. For example, in normal plain text editor, Alt+C and Alt+W works normally as shown below.

VS Code Version
Version: 1.78.0-insider (user setup) Commit: 7f03ec6f7a48791d8027761b22773b454fc52478 Date: 2023-04-10T15:38:56.561Z Electron: 22.4.4 Chromium: 108.0.5359.215 Node.js: 16.17.1 V8: <IP_ADDRESS>-electron.0 OS: Windows_NT x64 10.0.19045 Sandboxed: Yes
Jupyter Extension Version
v2023.4.1001051011
Jupyter logs
There is no output in Jupyter Server Console when pressing any of the key above.

Coding Language and Runtime Version
No response
Language Extension Version (if applicable)
No response
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Local

The title of the issue is appropiate but I think I can provide a better context

This is related to the commands for toggling case, whole word and regex that exist for the different search widgets, for example these are the commands for toggling the case:

(toggleFindCaseSensitive) -> in-editor widget
(toggleSearchCaseSensitive) -> side panel widget
"Search Editor: Toggle Match Case" (toggleSearchEditorCaseSensitive) -> side panel open in editor
"Terminal: Toggle Find Using Case Sensitive" (workbench.action.terminal.toggleFindCaseSensitive)

So in the previous case displayed in GIFs, when you use Shift+Ctrl+L (default keybind for editor.action.selectHighlights guessing_ ), under the hood it is using the in-editor find widget. So its behavior depends on its current state (leftover from the last time it was used). Moreover, Alt+C (default keybind for toggleFindCaseSensitive) simple toggles it for the in-editor find widget.

The issue
Both find/search widgets inside jupyter notebooks AND interactive windows do NOT respond to any of the aforementioned commands related to toggling case, whole word, regex...
Jupyter does not add any command/keybind related to this so I suppose it should simply react to toggleFindCaseSensitive. This seems related to how the search widget is implemented inside jupyter editors, so the bug might be on vs-jupyter after all, @rebornix ?


XML support TCGC Author doc

I'm not sure at this point if we should have codegen specific page about XML, keeping this issue opened until we figure this out

Thinking about this, it doesn't seem DPG should have a specific doc for this. See https://typespec.io/docs/next/libraries/xml/reference


Miniscript updates

Oh, you've added some inputs for miniscript_string already. Interestingly it doesn't seem to conflict. I still need to merge mines in yours though.

It should never conflict, as the filenames are simply the hash of the file content

Though, it looks like you deleted my inputs? :sweat_smile:

Oh, maybe you moved all inputs (mine + yours) into a new folder and then merged the new folder into an empty folder? In that case not all inputs that were there originally are taken over.

I think i merged your folder into mine. What i'm doing now is merging both yours and mines at the same time in an empty folder:
FUZZ=.... ./empty_folder ./my_miniscript_script ./existing_miniscript_script

Does that sound right?
Also i'm using a branch from https://github.com/bitcoin/bitcoin/pull/24860#discussion_r893376746 that doesn't have the slowness of current master. I guess it shouldn't wreck the coverage, but maybe i should use master instead?

Generally I try to make adding inputs append-only.
You can achieve that by checking out the current main and then merging into the main folders.
For example, FUZZ=miniscript_string ./src/test/fuzz/fuzz -merge=1 ../btc_qa_assets/fuzz_seed_corpus/miniscript_string  ./temp_ms_s/ or ./test/fuzz/test_runner.py -l DEBUG  -j 9 --m_dir ../btc_qa_assets/oss_fuzz_inputs/ ../btc_qa_assets/fuzz_seed_corpus.

Obviously we still need to delete non-reduced inputs occasionally. Though, I am doing that separately: https://github.com/bitcoin-core/qa-assets/pull/82
Not sure if it is better to do both steps in the same pull.

Ok, that's what i was going to ask. :)
I ended up doing essentially what you said with one more step (merging into an empty dir and moving the seeds of the empty dir to fuzz_seed_corpus/miniscript_script/).

This was re-reverted (aka now merged) in aea795691ccb463ea6a6d071bf426acdf889ea5f


GetOccurrences returns wrong the first occurrence

I am using Ical.net version 2.2.13 and there is a bug, that I can reproduce everytime.
When I use Event GetOccurences, for my icalendar definition, I am getting The first occurence of the VEvent, even when it isn't in the range.
This is the code that reproduces the bug
IEvent vEvent;
using (var strm = new MemoryStream(Encoding.UTF8.GetBytes("BEGIN:VEVENT\nSUMMARY:Abraham Lincoln\nRRULE:FREQ=DAILY;INTERVAL=1;BYDAY=MO,TU,WE,TH,FR\nDTSTART:20160901T083000Z\nDTEND:20170701\nEND:VEVENT")))
{
    EventSerializer serializer = new EventSerializer();
    vEvent = (IEvent)serializer.Deserialize(strm, Encoding.UTF8);
}
var cal = new Calendar();
cal.Events.Add(vEvent);
cal.AddTimeZone(new VTimeZone("UTC"));
HashSet<Occurrence> occurrences = cal.GetOccurrences(startTime, endTime);

when I put startTime as September 19 - I get an occurence at September 1st, and that shouldn't be. Other than that, the rest of the occurences are correct.

did you try vEvent.getOccurrences(startTime, endTime) rather than cal.getOccurrences(startTime, endTime)?

I've got a similar issue, but I'm using RecurrenceUtil.GetOccurrences(). Changing that code to vEvent.getOccurrences() also returns incorrect results, but those are wrong in a different way.
I'll try to come up with a test case later.

I have a similar issue of the DtStart date not matching the RRULE and have figured out a work around using the RecurrencePatternEvaluator.
// 3/1/2017 is a wednesday so it shouldn't display in the occurrences
var vEvent = new Event {
 DtStart = new CalDateTime(newDateTime(2017, 3, 1, 9, 0, 0)),
 DtEnd = new CalDateTime(newDateTime(2017, 3, 1, 10, 0, 0))
};
var recurrenceRule = new RecurrencePattern(FrequencyDayType.Weekly, 1) {
 ByDay = new IList<IWeekday> { new WeekDay(DayOfWeek.Thursday) }
};
var recurrenceEvaluator = new RecurrencePatternEvaluator(recurrenceRule);
var searchStart = new DateTime(2017, 3, 1, 0, 0, 0);
var searchStart = new DateTime(2017, 3, 17, 0, 0, 0);
var correctOccurrences = recurrenceEvaluator.Evaluate(vEvent.DtStart, searchStart, searchEnd, false);

Occurrence should equal 3/2/2017, 3/9/2017, and 3/16/2017. By setting the parameter includeReferenceDateInResults to false only adds the DtStart to the occurrences if it matches the RRULE.

same problem.
any solution by the library?
or just through workaround?

I have a similar issue of the DtStart date not matching the RRULE and have figured out a work around using the RecurrencePatternEvaluator.
// 3/1/2017 is a wednesday so it shouldn't display in the occurrences
var vEvent = new Event {
 DtStart = new CalDateTime(newDateTime(2017, 3, 1, 9, 0, 0)),
 DtEnd = new CalDateTime(newDateTime(2017, 3, 1, 10, 0, 0))
};
var recurrenceRule = new RecurrencePattern(FrequencyDayType.Weekly, 1) {
 ByDay = new IList<IWeekday> { new WeekDay(DayOfWeek.Thursday) }
};
var recurrenceEvaluator = new RecurrencePatternEvaluator(recurrenceRule);
var searchStart = new DateTime(2017, 3, 1, 0, 0, 0);
var searchStart = new DateTime(2017, 3, 17, 0, 0, 0);
var correctOccurrences = recurrenceEvaluator.Evaluate(vEvent.DtStart, searchStart, searchEnd, false);

Occurrence should equal 3/2/2017, 3/9/2017, and 3/16/2017. By setting the parameter includeReferenceDateInResults to false only adds the DtStart to the occurrences if it matches the RRULE.

You're resetting the value of searchStart to 17th March, instead of setting searchEnd to it. Anyways, any solution so far?

It can be due to broken RecurrencePatternEvaluator. It returns first wrong occurrence in some cases.
var evaluator = new RecurrencePatternEvaluator(new RecurrencePattern(FrequencyType.Weekly, 2)
{
	ByDay = new List<WeekDay> { new WeekDay { DayOfWeek = DayOfWeek.Monday } }
});

var recurringPeriods = evaluator.Evaluate(
	referenceDate: new CalDateTime(DateTime.Parse("11/01/2021 8:00:00 AM", CultureInfo.InvariantCulture)),
	periodStart: DateTime.Parse("12/01/2021 8:00:00 AM", CultureInfo.InvariantCulture),
	periodEnd: DateTime.Parse("12/30/2021 8:00:00 AM", CultureInfo.InvariantCulture),
	includeReferenceDateInResults: false);

var firstOcurrence = recurringPeriods.First();
// Here we have firstOcurrence.StartTime = '11/29/2021 8:00:00 AM'

This is still happening as of version: 4.2.0 of Ical.NET


added API for opening sign up screen

Coverage increased (+0.2%) to 82.277% when pulling 3cbdca81e4d4a72de6d18c30ec948643285e2757 on label-fix into 512e5c0a3ba936a2d541307d914486ccd0198061 on development.

Coverage increased (+0.2%) to 82.277% when pulling d1a843b3017c6eff0cd8493cc1fd7f2bc9da25a1 on label-fix into 512e5c0a3ba936a2d541307d914486ccd0198061 on development.

Coverage increased (+0.2%) to 82.277% when pulling d1a843b3017c6eff0cd8493cc1fd7f2bc9da25a1 on label-fix into 512e5c0a3ba936a2d541307d914486ccd0198061 on development.


changing page title

Just updated _includes/head.html so the page title shows up correctly. @bdarnell, look right?

LGTM


[Docs] Replace pip with python3 -m pip

Heya,
If I understood https://github.com/tiangolo/fastapi/issues/531 correctly this PR should take care of replacing every occurence of pip with python3 -m pip.
Let me know if you have any feedback.
Thanks!

Thanks for your contribution! :rocket:
The preference of pip install, python -m pip install, python3 -m pip install, pip3 install, is quite subjective and varies a lot depending on personal taste.
So, I prefer to leave the simplest version (pip install), and let everyone chose from that what to use.
That's the same reason why I don't suggest Pipenv, Poetry, venv, etc.

On the other side, the $ <something> are quite inconsistent with the rest of the docs.
I should remove those... If you wanna do it, that would be great :smile: :sparkles:

Alrighty 👍  Thank you for your feedback. I'll create a seperate PR to get rid of the $. Feel free to close this if you prefer not to have these changes :)

here's an argument in favor - https://snarky.ca/why-you-should-use-python-m-pip/

Thanks @sandys , yes, I saw the article by Brett Cannon.
It almost convinced me, but doing python -m something removes any possibility for shell autocompletion, so I'm not sure I want to use it here in the docs.
For now, let's leave it as is.
Thanks @nstapelbroek !


[1.18, Bug] cannot reconnect

So, if you log onto a server, log off, and then log on again, you won't be able to join
You'll just time out instead

seems to be fixed


Fix installing dateutil since it is now a dependency of the GHP import script

This fixes the publication even though still running on Travis CI.
See #280

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @jamesmunns (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see the contribution instructions for more information.

Pinging @adamgreig since you reviewed the same thing for the discovery book


rationalizing logs

updating mapping to more constricted set of logs per
https://github.com/RackHD/RackHD/issues/34
I left notes in the places where I was unsure of the level

:+1:

:+1:


Fixes #5907 convert modules admin page to use css flexbox

Fixes #5907 convert modules admin page to use css flexbox
Summary view now stretches when needed on a per row basis and keeps alignment:

Previously "Depends on" would be cut off for long lists like Blogs.
Detail view is converted to column flexbox as well:

I artificially inflated the Blog dependencies to confirm that it expands correctly.

Just a note that this will be replaced by the new admin theme, using the same technique.

I will try to keep my emotions under control when that happens.


php 7.3.6 version problem

Code:
composer require php-curl-class/php-curl-class
Problem 1
    - mdanter/ecc v0.4.4 requires php >=5.6.0, <7.2 -> your PHP version (7.3.6) does not satisfy that requirement.
    - mdanter/ecc v0.4.4 requires php >=5.6.0, <7.2 -> your PHP version (7.3.6) does not satisfy that requirement.
    - mdanter/ecc v0.4.4 requires php >=5.6.0, <7.2 -> your PHP version (7.3.6) does not satisfy that requirement.
    - Installation request for mdanter/ecc (locked at v0.4.4) -> satisfiable by mdanter/ecc[v0.4.4].

the problem is not related to this!


All Actions Result in Page Expiration

Originally reported on Google Code with ID 267
What steps will reproduce the problem?
1. Set up GitBlit, following the installation instructions. (I've tried 3 times now)
2. Opened Web Browser (Chrome) to both the http and the https pages, and attempted
to create a user/change the admin password/create a repo.
3. All of the above actions result in a page showing "The page has Expired, go back
to the home page", and the action fails to complete.

What is the expected output? What do you see instead?
I expected to be able to create a user/repo or change the password.

What version of the product are you using? On what operating system?
GitBlit 1.3.0, Windows 8 Pro (64-bit), JDK 7 Update 17 (64-bit).

Please provide any additional information below.

Reported by TheWiduX on 2013-07-15 04:45:47

I have the same problem with Chrome on Windows
But it works with firefox windows.
Version gitblit 1.8.8


C# 9 records are not recognized by CodeNav

Hello Samir,
CodeNav doesn't recognize C# 9 records.

In a C# file containing a class and a record, only the class is displayed in CodeNav.

A C# file containing only a record doesn't display CodeNav at all. E.g.
namespace SomeNamespace
{
    /// <summary>
    /// Some comment.
    /// </summary>
    public record MyModel(string Param1, object? Param2);
}

Please add support for records.
CodeNav 2019 v8.8.32 running on VS 2019 v16.11.4.
Best regards,
Didier

Good enhancement!
Should be to difficult to add :)

Hello Samir,
I'm using the latest version (8.9.34 on VS 2019 v16.11.4), but it still doesn't recognize records. No apparent change on that level since v8.8.32.
Please investigate.
Thanks in advance,
Didier

Could you please share a source file?
It looks to be working for me with my simple .cs file:

Hello Samir,
Here's one:
using Elsa.Activities.Http.Models;

namespace ElsaWorkflows.Models
{
    /// <summary>
    /// This model represents the 2 parts of the response of a <see cref="Elsa.Activities.Http.SendHttpRequest"/>
    /// activity.
    /// </summary>
    public record SendHttpRequestResponse(HttpResponseModel Model, object? Body);
}

Note: if you need this file to be compilable, install NuGet package Elsa 2.3 or later.
Best regards,
Didier

My previous post was about v8.9.35.
I've just upgraded to v8.9.38 and the problem is still present.

Could you please test if this version fixes the problem?
Records work in the VS 2022 version of CodeNav, but getting it to work in the VS 2019 version might create some weird nuget package errors.
CodeNav.VS2019.zip


feat: add COMMENT_TYPE env variable to use discussions or notes

This PR adds a new COMMENT_TYPE environment variable, to control the type of comment created when using changesets-gitlab comment:

discussion creates a new discussion (thread). This is the default
note creates a note (comment)

This is useful when a GitLab repo requires all threads to be resolved before merging an MR. We can instead use this new note COMMENT_TYPE which won't block merging.

Just curious, why not just resolve the thread if you want to merge?

It creates an extra step that isn't needed, and makes it impossible to auto-merge MR from bots (e.g renovate).

It creates an extra step that isn't needed, and makes it impossible to auto-merge MR from bots (e.g renovate).

Thanks for explain! I didn't use auto-merge MR from bots in GitLab.
I think this PR is acceptable.


Add ticket status endpoints

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


With spring fox version 2.9.2 Im getting "Correct the classpath of your application so that it contains a single, compatible version of springfox.documentation.builders.RequestHandlerSelectors" following error

Please take the time to search the repository, if your question has already been asked or answered.

[ ] What version of the library are you using? Is it the latest version? The latest released version is

What kind of issue is this?

[ ] Question. Is this a question about how to do a certain thing?

[ ] Bug report. If you’ve found a bug, spend the time to write a failing test. Bugs with tests or
steps to reproduce get fixed faster. Here’s an example: https://gist.github.com/swankjesse/6608b4713ad80988cdc9

[ ] spring xml/java config that is relevant
[ ] springfox specific configuration if it makes sense
[ ] include any output you've received; logs, json snippets etc.; and what the expected output should be
[ ] if you have a repo that demonstrates the issue for bonus points! See this example

[ ] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution
already exists! Don’t send pull requests to implement new features without first getting our
support. Sometimes we leave features out on purpose to keep the project small.

Please do consider starring this repository if you do like this project and find it useful.

@dilipkrish please have a look at this issue. Im new to springfox


Set Windows timeouts to enforce non-blocking read

This is the equivalent of the PR https://github.com/serialport/serialport-rs/pull/79. I would actually recommend completely removing the COMMTIMEOUTS override in mio-serial when/if the patch is accepted on serialport-rs, as this will solve the issue of read not returning available data until buffer is full or timeout. However, without this patch, the fix in serialport-rs does nothing.
This behavior is critical for and request/response protocol. I see 27-30 millisecond minimum latency on reads for data that takes <1ms to transfer, which is unacceptable.
Original change description:
Settings ReadIntervalTimeout and ReadTotalTimeoutMultiplier to MAXDWORD in the COMMTIMEOUTS structure on Windows makes Windows behave as normally expected; a read call to the serial port will return immediately if there is any data available, or if a single byte arrives in the buffer. If no data arrives, it will timeout out after the timeout specified in ReadTotalTimeoutConstant.
This fixes issues where reading from a serial port would take longer than desired when less data arrives that there is room for in the buffer passed to the read function.
See also: https://learn.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-commtimeouts

ping me when the patch is applied to serialport and I'll look at this in detail

Ping me when this is accepted in serialport and I'll give it a detailed look

@estokes this has now been released with serialport-rs 4.4 :)

@larsch I may have encountered an issue with this as I have been testing it. It seems that you get the EOF condition somewhat unexpectedly with the serial port, causing read_exact to terminate with ErrorKind::UnexpectedEof. If you get 0 bytes back upon termination, you can't give 0 bytes to the caller, because the rules for read are that returning 0 bytes indicates an EOF condition. Instead, it needs to ignore the fact that 0 bytes were returned unless the serial port is truly "EOF", whatever that means for a serial port.
I might work up a fix for this if I can.

Just based on my rudimentary understanding, which is not yet complete, I think that we need to detect somewhere (PollEvented or NamedPipeClient in tokio?) whether this 0 bytes occurred due to this timeout we set or whether the real end of file was reached. In the meantime I will be adding a hack somewhere to ignore this end of file condition for the serial port.

Given that the fix is accepted in serialport, would you like to amend this PR to completely remove the override?


Fix a few comments in reconciler-gen

Changes
As per title. A few comment nits.
/assign @n3wscott

/assign @vagababov

/retest


[Navigation-Animation] Fix NavController receiving back events even while AnimatedNavHost is not in composition

Hi folks, I think there's a bug in the library which casues navController to receive back events even if AnimatedNavHost is not in composition. Because of that each time AnimatedNavHost gets recomposed new navController will be registered for receiving back events which can lead to weird issues with back navigation. Please check the MRE  and detailed description in this repo.
Fix is based on androidx.navigation.compose.NavHost which has the same mechanism for disabling back events in onDispose.

As per this comment, all of Accompanist Navigation Animation has already been merged back into Navigation Compose itself, so we won't be updating Accompanist anymore - you'll just be able to use Navigation 2.7.0-alpha01 once it comes out, which, as you've mentioned, already contains this fix.


feat(@vtmn/svelte): add VtmnTabs component

Changes description

Context

Checklist

[ ] Make sure you are requesting to pull a topic/feature/bugfix branch. Please, don't request directly from your main!
[ ] Check commits & PR names matches our requested structure. It must follow the https://www.conventionalcommits.org pattern.
[ ] Check your code additions will fail neither code linting checks.
[ ] I have reviewed the submitted code.
[ ] I have tested on related showcases.
[ ] If it includes design changes, please ask for a review with a core team designer.

Does this introduce a breaking change?

Yes
No

Other information

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Gomez Miguel seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

#1077 fix issue with icon colors


Update web3addicted.toml  for testnet-15

Create web3addicted.toml for namada-public-testnet-15
Description
All previous genesis validators should name their PRs "Update {validator_alias}.toml for tesntet-15" and provide links to previous PRs merged.
#472
#1270
If this is an UPDATE for a previous genesis validator
Checks are made against your net_address. If this has changed since the previous testnet, make sure you provide links of previous prs merged from your previous git username.
Thanks in advance!
Checklist before merging

[ ] Only one toml is added in this PR
[ ] The file being added is indeed a .toml file
[ ] The toml being added is to the correct folder, and only to the correct folder
[ ] The eth_hot_key and eth_cold_key are present
[ ] The email, discord, elements telegram, and twitter fields are present and valid

Links to previous PRs merged.
https://github.com/anoma/namada-testnets/pull/472
https://github.com/anoma/namada-testnets/pull/1270


[Bug] Too long prompt

Context / Scenario
I'm asking question while connected to PostgreSQL
Prompt is 25692 length
CompletionOptions.MaxTokens is 300 (if that matters)
What happened?
I got the following error:
Content:
      {
  "error": {
    "message": "This model's maximum context length is 4096 tokens. However, your messages resulted in 5818 tokens. Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

Importance
edge case
Platform, Language, Versions
.NET/C#/0.26.240116.2
Relevant log output
info: SimpleChat[0]
      Function SimpleChat invoking.
info: Microsoft.SemanticKernel.Connectors.OpenAI.AzureOpenAIChatCompletionService[0]
      Prompt tokens: 616. Completion tokens: 21. Total tokens: 637.
info: Ask[0]
      Function Ask invoking.
fail: Ask[0]
      Function failed. Error: This model's maximum context length is 4096 tokens. However, your messages resulted in 5818 tokens. Please reduce the length of the messages.
      Status: 400 (model_error)
      ErrorCode: context_length_exceeded

      Content:
      {
  "error": {
    "message": "This model's maximum context length is 4096 tokens. However, your messages resulted in 5818 tokens. Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

      Headers:
      Access-Control-Allow-Origin: REDACTED
      X-Content-Type-Options: REDACTED
      x-ratelimit-remaining-requests: REDACTED
      apim-request-id: REDACTED
      x-ratelimit-remaining-tokens: REDACTED
      X-Request-ID: REDACTED
      ms-azureml-model-error-reason: REDACTED
      ms-azureml-model-error-statuscode: REDACTED
      x-ms-client-request-id: dd6bf4cb-0d37-444c-8925-b67e56e5d070
      x-ms-region: REDACTED
      azureml-model-session: REDACTED
      Strict-Transport-Security: REDACTED
      Date: Fri, 26 Jan 2024 10:15:45 GMT
      Content-Length: 281
      Content-Type: application/json

      Azure.RequestFailedException: This model's maximum context length is 4096 tokens. However, your messages resulted in 5818 tokens. Please reduce the length of the messages.
      Status: 400 (model_error)
      ErrorCode: context_length_exceeded

      Content:
      {
  "error": {
    "message": "This model's maximum context length is 4096 tokens. However, your messages resulted in 5818 tokens. Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

      Headers:
      Access-Control-Allow-Origin: REDACTED
      X-Content-Type-Options: REDACTED
      x-ratelimit-remaining-requests: REDACTED
      apim-request-id: REDACTED
      x-ratelimit-remaining-tokens: REDACTED
      X-Request-ID: REDACTED
      ms-azureml-model-error-reason: REDACTED
      ms-azureml-model-error-statuscode: REDACTED
      x-ms-client-request-id: dd6bf4cb-0d37-444c-8925-b67e56e5d070
      x-ms-region: REDACTED
      azureml-model-session: REDACTED
      Strict-Transport-Security: REDACTED
      Date: Fri, 26 Jan 2024 10:15:45 GMT
      Content-Length: 281
      Content-Type: application/json

         at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
         at Azure.AI.OpenAI.OpenAIClient.GetChatCompletionsStreamingAsync(ChatCompletionsOptions chatCompletionsOptions, CancellationToken cancellationToken)
         at Microsoft.KernelMemory.AI.AzureOpenAI.AzureOpenAITextGenerator.GenerateTextAsync(String prompt, TextGenerationOptions options, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.KernelMemory.AI.AzureOpenAI.AzureOpenAITextGenerator.GenerateTextAsync(String prompt, TextGenerationOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.KernelMemory.Search.SearchClient.AskAsync(String index, String question, ICollection`1 filters, Double minRelevance, CancellationToken cancellationToken)
         at Microsoft.KernelMemory.Search.SearchClient.AskAsync(String index, String question, ICollection`1 filters, Double minRelevance, CancellationToken cancellationToken)
         at Microsoft.KernelMemory.MemoryPlugin.AskAsync(String question, String index, Double minRelevance, ILoggerFactory loggerFactory, CancellationToken cancellationToken)
         at Microsoft.SemanticKernel.KernelFunctionFromMethod.<>c.<<GetReturnValueMarshalerDelegate>b__12_4>d.MoveNext()      --- End of stack trace from previous location ---
         at Microsoft.SemanticKernel.KernelFunction.InvokeAsync(Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)
info: Ask[0]
      Function completed. Duration: 103.442043s

hi @rosieks I'm not ure how you're using the code, is it via the service or the embedded serverless memory? in the model configuration, did you set MaxTokenTotal to 4096?

I'm using serverless mode. Actually right now I set MaxTokenTotal to 4096 so instead of error I get: "INFO NOT FOUND"

Looks like setting MaxTokenTotal fixed the problem. About "INFO NOT FOUND" that would be because the search for relevant text finds no results.


[type:Integration Test] add integrated-test of logging-elasticsearch-plugin

Fixes #3589

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #3602 (afb5bce) into master (f69c5ca) will decrease coverage by 0.47%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3602      +/-   ##
============================================
- Coverage     62.57%   62.10%   -0.48%
+ Complexity     5705     5659      -46
============================================
  Files           870      870
  Lines         24069    24067       -2
  Branches       2194     2193       -1
============================================
- Hits          15062    14946     -116
- Misses         7606     7715     +109
- Partials       1401     1406       +5

Impacted Files
Coverage Δ

...plugin/logging/rocketmq/LoggingRocketMQPlugin.java
22.72% <0.00%> (-72.73%)
:arrow_down:

...ging/elasticsearch/LoggingElasticSearchPlugin.java
22.72% <0.00%> (-72.73%)
:arrow_down:

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...nyu/plugin/response/strategy/RPCMessageWriter.java
77.77% <0.00%> (-22.23%)
:arrow_down:

...ogging/rocketmq/body/LoggingServerHttpRequest.java
0.00% <0.00%> (-20.00%)
:arrow_down:

...ch/body/LoggingElasticSearchServerHttpRequest.java
0.00% <0.00%> (-20.00%)
:arrow_down:

...logging/elasticsearch/entity/ShenyuRequestLog.java
86.20% <0.00%> (-13.80%)
:arrow_down:

...h/elasticsearch/ElasticSearchLogCollectClient.java
50.00% <0.00%> (-13.05%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

...ng/rocketmq/rocketmq/RocketMQLogCollectClient.java
68.00% <0.00%> (-8.00%)
:arrow_down:

... and 46 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f69c5ca...afb5bce. Read the comment docs.

please pull code from master, see https://github.com/apache/incubator-shenyu/pull/3611


Prepare to release 3.0.0

Description
Release 3.0.0
Issues Resolved
N/A
Check List

[ ] New functionality includes testing.

[x] All tests pass ($ rake test).
[ ] New resources and/or actions have are included in the matchers.rb and matchers_spec.

[ ] New functionality has been documented in the README if applicable.

[ ] New functionality has been thoroughly documented in examples (please include helpful comments).

[x] Changes documented in the CHANGELOG.

The version in metadata.rb was already set to 3.0.0.