'utf-8' codec can't decode byte 0x8c in position 1708: invalid start byte

Hi Karol,
I am trying to use this library.
When i call "videos_to_frames_pipe()" function i got this error : "'utf-8' codec can't decode byte 0x8c in position 1708: invalid start byte" on my own dataset (which includes some videos of Ucf-101 datasets).
I suppose there is some error in some videos, but I can't figure out which one.
I thank you for your availability.
Bests
Samuele

Hi @SamueledelVescovo
Yeah, I would guess there's something wrong with one of the files.
You can try modifying that videos_to_frames_pipe() function to add something like below to the begining of the pipe:
...
| mp.iter('filename', lambda fn: print('Processing: ', fn))
...

That way you would at least print the filename before it throws exception maybe.

Hi Karol,
thanks for your advice.
I wish you good work.
Bests
Samuele


I tried to add contact with Notes but couldn't

Is it possible to add contacts with Notes or organization? I tried to simulate postalAdress example, but didn't work

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

This is not possible using this library. Try the PHP Google People API instead.


MINOR: [C++] Fix the wrong comment in #45104

Rationale for this change
In my last typo fix in #45104 I made a mistake: https://github.com/apache/arrow/pull/45104/files#diff-a1f24396d20cd510ae48838e684fbc5ba7b23475702ae849a2d13bf48e5251aaR284 I was too careless. The size of a block in swiss table is much more subtle than I thought. It turns out the original 12KB was right.

What changes are included in this PR?
Removing my last wrong change and elaborate a bit about how the number 12KB comes.

Are these changes tested?

Are there any user-facing changes?

Hi @kou would you like to take a look? My apology for my last careless fix.

Also cc @pitrou .


Support for PHP 5.3?

I know PHP 5.3 is not supported and probably never will, but what would be the easiest way to achieve this?
Sometimes I need to run ancient systems in order to analyse an migrate them - e. g. an ancient version of typo3 that will not run on systems with PHP > 5.3.

I've definitely wanted to figure out how to do that, but the packages we use (deb.sury.org) only go to 5.6, and it's not trivial to build older versions. I pursued that quite heavily when sury was going to discontinue 5.6, https://github.com/oerdnj/deb.sury.org/issues/1092
But I think for now the easiest thing to do is to run a virtualbox instance with Ubuntu 12.04 and the basic standard packages that come with it. That's what I've done when forced to run an old site. Also... turn that old site into html!

If it's super important and you can figure out how to build the sury images from that comment, we'll figure out  a way to get old stuff available for ddev.

As a developer, it is often important to go back to old releases to check and see how an older version acted, or to upgrade or migrate. Many sites are running on dangerously outdated PHP and need to be upgraded. Are there possibly archived versions of containers? Possibly only as difficult as locating these copies. Running on a dev box isn't as big a security issue as running on a live site. I would hope developer tools would be able to assist in these cases. (Just my 2cents worth)

I would absolutely love to provide old PHP versions, but it's quite a lot of work, since deb.sury.org doesn't provide these any more. If I could figure out from https://github.com/drud/ddev/issues/1714#issuecomment-512893053 how to build them, I'd even consider doing it, but that wasn't really very descriptive. We almost lost access to 5.6, and will likely in the future (although we have a workaround for that)

I did some poking around for non-Sury.org strategies for old php, and found

a package-based approach (which has real potential). The mentioned PPA repository also has 5.2 and 5.4/5.5 and even 4.x.
a build-from-source approach; I experimented with this but didn't finish the work.

Thanks for exploring this further! I think I can point you to an alternative right in the middle of the two mentioned strategies: http://phpbrew.github.io/phpbrew/
No ready build packages but quite a lot of automation. I believe you especially will like the  phpbrew switch command: This way PHPBrew would allow to build only one image for all the old an upcoming unsupported versions of PHP!

phpbrew looks like it might just be pretty nice. It would probably mean a horrible image build session while building docker image. But way lots of flexibility. And if we used it, it might be possible for people to just go wherever they wanted to go with the versioning. Occasionally people will ask for a minor version.
The downside is that everything is currently built around the very nice sury.org multiple-supported-php-packages approach, and we'd lose that.

You're right: it's a lot of building beforehand - but how about a two-way approach: sury.org for the supported versions and phpbrew for the unsupported versions? The unsupported versions image could also be pre-built, can't it? All those unsupported versions per definition don't move much anymore.

I immediately ran into https://github.com/phpbrew/phpbrew/issues/861, phpbrew completely failed; I actually ran into this same issue trying to build manually. I'm not sure phpbrew is as robust as it wants to be. Of course, it's always a mess trying to build old recipes on newer systems, and that's what's actually going on here.
I do agree that it might be possible to use sury for current versions and phpbrew for older. Or just provide a recipe for people to do their own. But it would have to work better :) If you use Debian or Ubuntu... are you able to use phpbrew to build 5.3 or 5.4?

Yeah, for all its promise, phpbrew is an animal whose complexity is just the complexity of building old PHP on new systems. The biggest problem is openssl, whose current versions are incompatible with the ones that php5.x need. https://github.com/phpbrew/phpbrew/issues/939 gives a strategy for working around this with old libraries, but it's not actually compatible with using the machine for anything else.

What I was hinting at was working with one of the other projects to find old images that might work. Might need older versions from several projects. Or working with them for suggestions. Maybe versions could be created just for DDEV use. I know Docker had a security issue a while back where some images were available that had malware. I believe they were 3rd party and recommended being careful whose images you used, but not sure if any others were compromised. Docker Toolbox uses https://github.com/boot2docker/boot2docker This contains the Linux kernel and base Linux. Possibly use older versions? It would be like using the older containers that remain after a DDEV upgrade, although Docker upgrades the containers to the latest when you upgrade Docker. Just throwing out some ideas, hoping to trigger other ideas.

I tried the libraries from https://github.com/zener05/debian-stretch-php-5-3 with lots of hopes. But too many of the modules wouldn't install, including gd but most importantly mysql, which depends on a library not included in stretch.
Experimented with adding this to the Dockerfile, which hopes of transplanting it later to a .ddev/web-build/Dockerfile recipe:
RUN apt-get update && apt-get install -y logrotate  #libicu55 libmysqlclient18 libreadline6
RUN wget -nv https://launchpad.net/~sergey-dryabzhinsky/+archive/ubuntu/php53/+files/php53-cli_5.3.29-1sergeyd14.4~xenial1_amd64.deb https://launchpad.net/~sergey-dryabzhinsky/+archive/ubuntu/php53/+files/php53-common_5.3.29-1sergeyd14.4~xenial1_all.deb https://launchpad.net/~sergey-dryabzhinsky/+archive/ubuntu/php53/+files/libapache2-mod-php53_5.3.29-1sergeyd14.4~xenial1_amd64.deb https://launchpad.net/~sergey-dryabzhinsky/+archive/ubuntu/php53/+files/php53-fpm_5.3.29-1sergeyd14.4~xenial1_amd64.deb && \
( wget -nv $(for item in bcmath bz2 curl gd intl json mysql pgsql mbstring sqlite3 readline xdebug xml xmlrpc zip; do printf "https://launchpad.net/~sergey-dryabzhinsky/+archive/ubuntu/php53/+files/php53-mod-${item}_5.3.29-1sergeyd14.4~xenial1_amd64.deb "; done)  || true ) && dpkg -i php*cli*.deb php*common*.deb php*fpm*.deb && dpkg -i php53-mod*.deb

For @a8w4  though: The way I've dealt with this in the past seems to still be the easiest way.

Find the version of Ubuntu that matches the PHP version you need.
Find a web recipe showing how to install it on that version of Ubuntu
Spin it up in Virtualbox and set it up
Spelunk in your website.

It's way easier than what I was just trying to do in this issue :)

https://github.com/zener05/debian-stretch-php-5-3/blob/master/README.md

Yeah, that's what I tried above (package-based approach), https://github.com/drud/ddev/issues/1714#issuecomment-515966830

Oh - sry missed that comment

While I would absolutely love to do this, it doesn't seem like it's in the cards. I'll be happy to help anybody set up an old Ubuntu in Virtualbox to get the job done.

Sad 😞 ...but understandable.
Despite its EoL, php5 is still around, and at a big % unfortunately.
...in Backdrop, we support back to 5.3, so our CI is configured to run tests for php53 and php70. Each time that php7 passes, but php5 fails, I am at a dead end, as it requires way much more work to get a php53 environment span up to test things. I eventually get frustrated and give up 😓
I have started an issue to discuss dropping support for php 5.3 entirely here: https://github.com/backdrop/backdrop-issues/issues/3992, but the fact that there are many outdated but still LTS linux distros that are used by hosting companies, makes it hard to decide. It is a pickle 🤷♂

I would dearly love to support earlier versions if it could be an appropriate level of attention, but I haven't figured out a way to do it, as you already know.

maybe it makes sense to contact suri - then debian maintainer for php.

php5.6/                                            09-Aug-2019 18:37                   -
php7.0/                                            09-Aug-2019 10:57                   -
php7.1/                                            09-Aug-2019 15:47                   -
php7.2/                                            10-Aug-2019 00:56                   -
php7.3/                                            09-Aug-2019 21:17                   -
php7.4/

https://packages.sury.org/php/pool/main/p/
He only supports supported versions ...

Absolutely no way to get him to do it :) Have worked on that a bit :)  He almost turned off 5.6 and 7.0 earlier this year until somebody stepped up with security support for those.

@rfay may using a dedicated docker image for fpm is the solution?
This would also ensure, that ddev can launch old environments.

Right, would also mean a separate container per version. Might break the project - as we often also need the cli. Really hard to tackle problem.

The reason I posted this issue was in case what was required was saving a resource that provided the older versions before they were deleted by the project owner, or asking them for older versions. I understand removing them when they are no longer supported, but devs sometimes need them for development work on older projects, as opposed to using them in a production environment, which none of us want to support. I've had to use older versions of Drupal to build a dev environment, and it wouldn't work with the latest PHP versions. Creating an environment that supports older versions is difficult and time consuming. If a project like DDEV made that easier, it would help pull people into using it that might stay permanently, because of how flexible it is. I'd rather suggest it now, when something easier might be done, than in a year, when it will be more difficult. And can we, as developers, really stop supporting something when hosting companies still provide them for their sites? Moving the latest production projects like Drupal off of deprecated dependencies is a good practice, but Dev environments are, by definition, not production.

I'm not even sure if installing older versions of Docker would be possible, or if they would have already deleted the objects that contained the deprecated PHP versions. Maybe just saving those resources and providing a backup source for them is all that is required, before they are gone forever.

This lando article has an interesting approach that could be much simpler in ddev (I think). Basically it would probably require a custom Dockerfile that used a FROM ubuntu:12.04, and then the use of deb.sury.org repos would have to be removed. It's similar to the "Run Ubuntu 12.04 in Virtualbox approach", but might have some standardization value.

This! Sounds like a plan!

I was kinda thinking it might require saving some files before they disappear. Or requesting files that may have been made unavailable. Whether older versions of docker, etc. In other words, something easier than creating it from scratch. Hopefully this is easy and sustainable. It's only for Dev environments temporarily, so security is not as much of an issue. How far back do current hosting allow you to go? I posted this in response to someone in slack looking for a PHP 5.3 environment.

The real problem is that old versions of PHP are not able to interoperate with current versions of openssl, which means that they're not able to build on current Debian, etc. So the key is to use old Debian/Ubuntu versions or images of old stuff, where that stuff plays happily. But, of course, doesn't play happily with current PHP versions.
FYI we have stored away versions of php5.6/7.0 packages, which sury was threatening to discontinue but backed off of. But those are after the key breakage of OpenSSL change.

I have compiled 5.3 successfully on Ubuntu 18.04 LTS recently. There's a lot of work involved, but I wrote out an entire guide on it a while ago so I do not forget in case I need it in the future. It doesn't generate packages, it just compiles it from source. It requires downloading the source for openssl 1.0.1 and the source of an old version of libGL for the main binary to compile, and a few other packages, but most of the modules can compile with OpenSSL 1.1 such as cURL and imap. It is actually a lot easier to compile on 16.04 as it comes with a choice of dev packages for openssl 1.0.1.
Perhaps that could be somewhat useful as a basis for someone to figure out how to package it?

oops, in my pastebin you will probably want to make a change as I did something custom:
look for this in the pastebin:
export APACHE_RUN_USER=httpservice
export APACHE_RUN_GROUP=httpusers
and change it to this:
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data

Thanks @RedScourge - Have you tried this with other versions of PHP on the same machine? The problem has really been the coexistence of different versions of OpenSSL. (The ddev-webserver container has all versions installed on it.)

@rfay - If you compile it how I did, the main binary, which is the only part that seems to need the old version, links the OpenSSL libraries statically into the main binary, thereby making it no longer dependent on the OpenSSL version. However,  then the binary is like 25MB instead of the rather small size you get from whatever process Ondrej uses. I'm not quite sure how he gets it so small, but then again I am not an expert at compiling PHP. The compilation process uses the -dev packages, so I had the OpenSSL 1.1 libraries installed on the machine the whole time, but used the OpenSSL 1.0.1 source to compile against. I made it compile as few modules as possible in the main compilation step, then compiled all the modules which use OpenSSL against OpenSSL 1.1 as needed. Once it's all compiled, the -dev packages and OpenSSL 1.0.1 source can be removed completely and you'd only have 1.1 left on the system.
The only (minor) problem I ran into was that after compile, I was getting warnings in syslog referring to /usr/local/lib/libcrypto.so.1.0.0, as according to the ldd command, apparently on Ubuntu 18.04 the /usr/bin/openssl binary looks for it there first before looking for it in /usr/lib/x86_64-linux-gnu/, though those warnings went away after I deleted the file and no functionality stopped.

A static binary id definitely the way to solve that problem. It would be cool if we could come up with a package that had the static binaries in it.

Indeed, and I am using PHP 5.3 on Ubuntu 16.04 in production and on 18.04 in test and it is very fast despite the large binary, so I know the file size is not a liability. I've never made a package before, so if someone here is good at packaging they may be able to use my guide to make one. I may go through Ondrej's guide at some point and try to do it myself but don't expect that any time soon.
Failing that, people can at least compile it as I did in my guide there. The way I did it, it can coexist with 5.6 and 7.3 on the same system, and allow switching via update-alternatives for cli+cgi, and via a2enmod/a2dismod for the apache module.

Just in case anyone was wondering which extensions my compile method works with, the answer is pretty much all of them. I went back and did a quick scripted "make test" in Ubuntu 18.04 on all of the ones that didn't require custom configure options and gathered the test fail/skip results here to give a little breakdown on that. Here's the commands I used to do that in case anyone was curious:
#to ensure I am compiling them against PHP 5.3
#update-alternatives --set php /usr/lib/php/5.3/bin/php
#update-alternatives --set php-config /usr/lib/php/5.3/bin/php-config
#update-alternatives --set phpize /usr/lib/php/5.3/bin/phpize

export NO_INTERACTION=true

function maketestphpmod(){
  cd /usr/src/php-5.3.29/ext/$1
  phpize
  ./configure
  make -j$(nproc) 2>&1 > build.make.$1.log
  make test > build.maketest.$1.log
  make clean
}

for i in bcmath bz2 calendar ctype dba dom exif fileinfo filter gettext gmp hash json mbstring mcrypt pcntl posix pspell readline ; do maketestphpmod $i; done
for i in recode shmop simplexml snmp soap sockets sysvmsg sysvsem sysvshm tidy tokenizer xmlreader xmlrpc xmlwriter xsl wddx zip ; do maketestphpmod $i; done

fgrep 'Tests failed' /usr/src/php-5.3.29/ext/*/build.maketest.*.log | fgrep -v 'Tests failed    :    0'
fgrep '100.0%' /usr/src/php-5.3.29/ext/*/build.maketest.*.log | grep skipped

The following are built-in when I compile PHP, because either the option to compile them as a separate module did not exist in PHP 5.3, or they require OpenSSL 1.0.x either to work at all or for all their extra options to work, or they're an SQL module which has to be built-in in order for its corresponding PDO module to work, or a combination of the last two:
openssl (has to be 1.0.1, is used by others which must be built-in such as mysqlnd)
mysql (because I do it with mysqlnd which must be built in)
odbc
pdo
pdo_mysql (because I do it with mysqlnd which must be built in)
pdo_odbc
pdo_pgsql
pdo_sqlite
sqlite
pgsql
zlib (must be built in)
The following modules built without any custom configure options:
bcmath bz2 calendar ctype dba dom exif fileinfo filter gettext gmp hash json mbstring mcrypt pcntl posix pspell readline recode shmop simplexml snmp soap sockets sysvmsg sysvsem sysvshm tidy tokenizer xmlreader xmlrpc xmlwriter xsl wddx zip
When I build those modules, here are the ones that fail tests with "make test", and how many test failures:
dom 1
fileinfo 1
pcntl 2 (i think it only works in cli/cgi i think)
readline 2 (make test says "warning: #warning Readline module will NEVER be thread-safe")
soap 2
sockets 1 (i think it only works in cli/cgi)
tidy 10 (did not test this, perhaps may need to compile against an older version of libtidy?)
Here are the ones that skip 100% of the tests:
recode (does not have any tests)
snmp (does not have any tests)
sysvsem 1/1
xml 81/81 (says "reason: xml extension not available", unsure if this indicates a problem)
xsl 35/35 (says "reason: xsl extension not available", unsure if this is a problem)
The following build with custom configure options:
curl (to specify openssl 1.1)
ftp (to specify openssl 1.1) skips 100% of tests, "make test" says "reason: fork not available"
gd (have to download a lot of source packages and specify them) skips 22 tests, fails 8 tests, passes 223 tests
imap (just --with-kerberos --with-imap-ssl)
ldap (--with-ldap=/usr --with-ldap-sasl --with-libdir=lib/x86_64-linux-gnu)
mssql (--with-libdir=lib/x86_64-linux-gnu)
xmlrpc (must export CC and CFLAGS before compile then unset after)
These ones should be installed as PECL packages, or an old version of the source should be downloaded from PECL and compiled by hand as they may have phased out PHP 5.3 support:
intl
imagick
With every single module enabled, PHP did not crash or segfault for me at all or behave buggy as far as I could tell, though I did not test any XML or tidy stuff, though I did test cURL and GD

The long term question is, if the better long term solution would be to have a docker image for every php-fpm version supported, this way ddev can be extremely flexible and support more versions. Maybe also an image per master branch (5.x, 7.x) can be a solution

I'm not familiar with docker images or php-fpm, as most of what I deal with has depended on the apache2/apxs SAPI, but if that involves compiling PHP from source, whoever attempts it will probably find that my little guide there should have enough to give them everything they need. I really need to turn it into a blog post somewhere tbh.
Also just a minor deficiency I just found with my guide there while digging through my syslog, if you want the Ubuntu /usr/lib/php/sessionclean script to also work with it, you'll want to add running ln -s /usr/lib/php/5.3/bin/php /usr/bin/php5.3 to tyhat process, as that script looks for the PHP 5.3 binary at that location

Maybe these are a staring point
https://hub.docker.com/r/bitnami/php-fpm/tags?page=444
Still need tuning for ddev though

Regarding my tests above, the make test tests all being skipped for the xml module with the cryptic message reason: xml extension not available had me worried, so I whipped together a test script to find out if calls to its xml_parser() function work, and it worked.
With the following test script:
xml.php:
<?php

$xml = '
<response>
        <method>test</method>
        <result>this is a test</result>
</response>
';

$parser = xml_parser_create();
$bob = xml_parse( $parser, $xml, true );
if( $bob == 0 )
{
        echo xml_get_error_code( $parser );
        echo xml_error_string( xml_get_error_code( $parser ) );
}
var_dump( $bob );
xml_parser_free( $parser );

?>

I ran it on the command line via /usr/bin/php5.3 xml.php and the return was int(1).
I also ran it with invalid XML and it did in fact go into the if() statement and dumped out an error code and message, so those 3 functions which are defined in the xml module, which should only work if the module is loaded, clearly work. I suspect that whatever caused all the xml module tests to be skipped also caused the same for xsl, and so if xml works xsl probably does too.

Also, have you guys heard of phpbrew (its on GitHub)? It's a shell tool which will help you build and manage several different custom builds of php on the same system through your /home folder.
I tried it and didn't like it because I don't think it was designed to be run as root, but then assumed you had root permissions if you wanted to install the apache2/apxs SAPI, and it messed around with your PEAR/PECL config parameters for your user, but for most people it's probably an easier solution for rolling your own PHP >= 5.3 binaries for testing, especially if you have a separate user account for using phpbrew that you can login to, so that it does not mess with any other account's environment variables.
I ran the following on Ubuntu 18.04 to get it to build me a static PHP 5.3 for just my non-root user:
cd ~
wget https://github.com/phpbrew/phpbrew/archive/master.zip
unzip master.zip
cd phpbrew-master/
./phpbrew init
echo source ~/.phpbrew/bashrc >> ~/.bashrc
source ~/.phpbrew/bashrc
curl -sS https://getcomposer.org/installer | php
php composer.phar install
phpbrew update --old

# export apache variables which Ubuntu writes in /etc/apache2/envvars:
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2/apache2.pid
export APACHE_RUN_DIR=/var/run/apache2
export APACHE_LOCK_DIR=/var/lock/apache2
export APACHE_LOG_DIR=/var/log/apache2

# to build cli SAPI on Ubuntu 18.04 it's easy:
# assuming you have already downloaded source and compiled openssl 1.0.x in /usr/local
# and libcrypto.so.1.0.0 and libssl.so.1.0.0 exist in /usr/local/lib
# note: may want to move/remove/rename those two after, as they
# conflict with package openssh-server
phpbrew install 5.3.29 +everything -fpm -apxs2 +cli -cgi -embed -gcov -- --with-curl=shared,/usr/include/x86_64-linux-gnu/curl

# to build apache2/apxs sapi module you must give phpbrew access to
# make changes to your apache config, and you must have compiled cli first!
sudo chmod a+rw /usr/lib/apache2/modules
sudo chmod a+rw /etc/apache2
sudo chmod a+rw /etc/apache2/mods-available
sudo chmod a+rw /etc/apache2/mods-enabled
phpbrew install 5.3.29 +everything -fpm +apxs2 -cli -cgi -embed -gcov -- --with-curl=shared,/usr/include/x86_64-linux-gnu/curl
# and finally instruct phpbrew to use that newly compiled version assuming it worked:
phpbrew switch php-5.3.29

To examine installed extensions, run phpbrew ext
To have phpbrew compile an extension, for example the imap extension which requires custom parameters, run phpbrew ext install imap -- --with-kerberos --with-imap-ssl
To go back to system php, run phpbrew switch-off
Note that you may want to prefer to compile php with fewer modules and then use the phpbrew commands to add extensions after, especially if you have difficulty compiling on your system. This will become more of an issue as Ubuntu updates libraries and downloading more old versions of those libraries from source and compiling against them becomes necessary.

If you read above you'll find my completely disastrous experiences with PHPBrew. It has the same problems as every other build technique.

therefore i'd opt for going with separated docker images per main version

@rfay - It's true that phpbrew does not resolve the OpenSSL issue, which causes both of those above mentioned phpbrew issues. To resolve them on Ubuntu 16.04, you should only need to issue this command first:
apt-get install libssl-dev=1.0.2*

This works because they released a 1.0.x openssl dev package for 16.04, though it is not installed by default.
On Ubuntu 18.04, they did not release any such package, so you end up having to download and compile your own in /usr/local, create a few symlinks, pass special parameters to the ./configure command if compiling by hand (or the phpbrew install command if doing it through phpbrew) to make it look for openssl in /usr/local, and compile. You'd want to use the process I came up with for downloading and compiling OpenSSL 1.0.1, getting the PHP compile to use it, then adding  -- --with-curl=shared,/usr/include/x86_64-linux-gnu/curl at the end of the phpbrew install command if your phpbrew command includes curl, or the phpbrew ext install curl command if compiling it separately, though I think it makes more sense to do it in the main php compile.

Anyway, sorry for all the blabbing in here about this; I just wanted to make sure there's a record somewhere on the Internet that there is in fact a way to get PHP 5.3 to compile on Ubuntu 16.04 and 18.04, because I had been wanting to do this for years and only figured it out recently,

I spent some more time fiddling with external PHP containers today. It's pretty easy to get an external container like https://hub.docker.com/r/devilbox/php-fpm-5.2 going. (Built from https://github.com:devilbox/docker-php-fpm-5.2)
I created a "php" service with a docker-compose.php.yaml like this:

version: '3.6'

services:
  php:
    # This is the name of the container. It is recommended to follow the same
    # name convention used in the main docker-compose.yml file.
    container_name: ddev-${DDEV_SITENAME}-php
#    image: docker-php-fpm-5.2:latest
    image: devilbox/php-fpm-5.2
    restart: "no"
    ports:
      - 9000
    # These labels ensure this service is discoverable by ddev.
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
    - type: bind
      source: ../
      target: /var/www/html
      consistency: cached
    - ".:/mnt/ddev_config:ro"
    - ddev-global-cache:/mnt/ddev-global-cache

#    user: "$DDEV_UID:$DDEV_GID"
    hostname: $DDEV_HOSTNAME

  web:
    links:
      - php:$DDEV_HOSTNAME

Then I switched to Apache, since older sites are more likely to work with apache, and moved to the oldest mariadb we explicitly support, so config.yaml needed
webserver_type: apache-fpm
mariadb_version: 10.1

Then I used a custom .ddev/apache/apache-site.conf, because it needs to go off-container for the fpm/cgi process:
<VirtualHost *:80>
    # Workaround from<EMAIL_ADDRESS>    # See also https://gist.github.com/nurtext/b6ac07ac7d8c372bc8eb

    RewriteEngine On
    RewriteCond %{HTTP:X-Forwarded-Proto} =https
    RewriteCond    %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -d
    RewriteRule    ^(.+[^/])$           https://%{HTTP_HOST}$1/ [redirect,last]

    SetEnvIf X-Forwarded-Proto "https" HTTPS=on

	ServerAdmin webmaster@localhost
	DocumentRoot $WEBSERVER_DOCROOT
	<Directory "$WEBSERVER_DOCROOT/">
  		AllowOverride All
  		Allow from All
	</Directory>
	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog /dev/stdout
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
	# Simple ddev technique to get a phpstatus
	Alias "/phpstatus" "/var/www/phpstatus.php"

    <FilesMatch ".+\.ph(ar|p|tml)$">
        SetHandler "proxy:fcgi://php:9000"
    </FilesMatch>
    <FilesMatch ".+\.phps$">
        # Deny access to raw php sources by default
        # To re-enable it's recommended to enable access to the files
        # only in specific virtual host or directory
        Require all denied
    </FilesMatch>
    # Deny access to files without filename (e.g. '.php')
    <FilesMatch "^\.ph(ar|p|ps|tml)$">
        Require all denied
    </FilesMatch>
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/master.crt
    SSLCertificateKeyFile /etc/ssl/certs/master.key

    # Workaround from<EMAIL_ADDRESS>    # See also https://gist.github.com/nurtext/b6ac07ac7d8c372bc8eb

    RewriteEngine On
    RewriteCond %{HTTP:X-Forwarded-Proto} =https
    RewriteCond    %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -d
    RewriteRule    ^(.+[^/])$           https://%{HTTP_HOST}$1/ [redirect,last]

    SetEnvIf X-Forwarded-Proto "https" HTTPS=on

	ServerAdmin webmaster@localhost
	DocumentRoot $WEBSERVER_DOCROOT
	<Directory "$WEBSERVER_DOCROOT/">
  		AllowOverride All
  		Allow from All
	</Directory>
	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog /dev/stdout
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
	# Simple ddev technique to get a phpstatus
	Alias "/phpstatus" "/var/www/phpstatus.php"

    <FilesMatch ".+\.ph(ar|p|tml)$">
        SetHandler "proxy:fcgi://php:9000"
    </FilesMatch>
    <FilesMatch ".+\.phps$">
        # Deny access to raw php sources by default
        # To re-enable it's recommended to enable access to the files
        # only in specific virtual host or directory
        Require all denied
    </FilesMatch>
    # Deny access to files without filename (e.g. '.php')
    <FilesMatch "^\.ph(ar|p|ps|tml)$">
        Require all denied
    </FilesMatch>
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

But wow, I still wasn't there. Although a phpinfo() was fine and showed everything I wanted, I ran into other troubles, and wasn't successful doing an install of Drupal5 or Drupal6.
I also tried installing Drupal 4.6... remember? You had to manually load the database, there was no install process. I had to remove TYPE=MyISAM from all the table creation stanzas. Didn't work anyway.
So this approach definitely ought to work, but it's once again a lot of work, with modest results. All failed in my case.
The good news is that the devilbox php 5.2 project does demonstrate how and where to build a statically linked php on debian jesse, and installs what it needs there. As @RedScourge points out, this is a technique that ought be be do-able and repeatable. It's possible that php 5.2 could be built in our web container for special cases.
It's possible that somebody could push this work through; the idea of using an external one-off PHP container certainly should work.

Well, I did do a basic Drupal5 install using an Ubuntu 12.04 VM with Apache/php5.3 using basically just the recipe in https://www.howtoforge.com/installing-apache2-with-php5-and-mysql-support-on-ubuntu-12.04-lts-lamp
I created a user, created a page or two, dumped the database, brought it down to just the setup in https://github.com/drud/ddev/issues/1714#issuecomment-538799770 and it worked. So it may be even though I can't get an install to work in this environment, it actually works for real life.

OK, I wrote this up in ddev-contrib, and it's now available as a pretty flexible recipe, https://github.com/drud/ddev-contrib/blob/master/docker-compose-services/old_php


bug_fix

The pull request is ready to be merged.
The details of the pull request:
Fix a small bug in searcher, and in selecting gpu devices.

Pull Request Test Coverage Report for Build 663

19 of 25 (76.0%)  changed or added relevant lines in 2 files are covered.
3 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-0.2%) to 95.937%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

autokeras/utils.py
14
20
70.0%

Files with Coverage Reduction
New Missed Lines
%

autokeras/bayesian.py
1
97.55%

autokeras/image_supervised.py
1
93.67%

autokeras/utils.py
1
82.5%

Totals

Change from base Build 660:
-0.2%

Covered Lines:
1889

Relevant Lines:
1969

💛  - Coveralls


Don't throw "restartmanager canceled" error for no restart policy container

- What I did
Don't throw "restartmanager canceled" error for no restart policy container
or else if kill a no restart policy container, there is a error log
ERRO[0024] restartmanager canceled
and add the container id to the warning message if a container has restart policy
and make error log to warn log
ping @tonistiigi
- How I did it
- How to verify it
- A picture of a cute animal (not mandatory but encouraged)
Signed-off-by: Lei Jitang<EMAIL_ADDRESS>

@cpuguy83  updated

libcontainerd/container_linux.go:8: imported and not used: "strings"

ping @cpuguy83 @tonistiigi PTAL

LGTM

LGTM


Add container level

Our sectioning macros are

Part of a tutoriel,
Chapter of a tutorial,
Section of a tutorial,
# title,
## title,
### title,
#### title.

But, I think we could distinguish container level and title level, to have the same title macros for any type of document (maybe easier for rebber) and to have better introductions macros (see #29). @pierre-24 @AmarOk1412 @vhf ?

From what I know from the process of turning a content to a PDF (I coded part of it back in ZEP-12), we extend the full text into a single markdown file (with this template), in which the title are "decaled" (it is not English, but that's how the command is called ^^) according to the size of the tutorial, so it matches the current implementation (for example, for article, section titles becomes #, and # titles becomes ## and so all).
Therefore, if we keep this behaviour, I don't think we need to change that since this big md file would then be used by rebber to get a PDF file, in l my mind, and rebber can simply use the number of # to replace by the corresponding \LevelxxxTitle. That was basically the whole point of #6 :)
But of course, if we want to do something else, then your idea may become interesting. So it is up to @vhf to decide whether the pdf will be generated from this "big md file" or something else.

It may also be interesting to have an input from the other ZEP-12 historical coder. I therefore summon @artragis !

Hello,
So what do we do when we publish the tutorial?
First we create the html representation of each container. All titles are shifted (for you pierre) of two levels.
Then we "flatten" the whole markdown and store it in a simple md file. This md file is the very basis for every compilation we provide.
Latex generation will therefore be handled from a "non shifted" basis and you will really have the chapter/part/subpart artchitecture you're custom with.

Latex generation will therefore be handled from a "non shifted" basis and you will really have the chapter/part/subpart artchitecture you're custom with.

This last part seems in opposition to what you explain previously o_O

Here are the two solutions.

\LevelOneTitle for # Title, \LevelTwoTitle for ## Title, etc. \part, \chapter and \section or LevelOneContainer, \LevelTwoContainer and \LevelThreeContainer.
\LevelOneTitle for : tutorial part if big tutorial, chapter if middle tutorial, section if small tutorial.

The two are easy to set up. We just have to choose.

WHat I call "non shifted" is just a "normal heading hierarchy". eg :

# Tutorial title

## Part One title

### Chapter one title

#### extract title

##### extract sub part title

conten
content

content

## Part two

etc.

Ok. So like I said, the number of # is directly related to the level of title, and the template deals with what it mean with the small/medium/big. But if you prefer the solution of @Karnaj, no problem. Just pick what is easier for implementation :)

It doesn't matter to us:

we already have all it takes to shift all titles up or down an arbitrary number of levels
we already have the compilation from markdown titles to latex.

If you're still ok with this syntax, let's settle on it.

I am :)


fix issue Unable to build on newer LLVM #268

Description
This resolves the issue of build error of bpftime with LLVM VERSION > 16.
Fixes #268
Type of change

[✅] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
I've built the tool locally after making the changes.
Test Configuration:
pegasus@pegasus:~$ uname -a
Linux pegasus 6.5.0-26-generic #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar  5 21:19:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

pegasus@pegasus:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 23.10"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=mantic
LOGO=ubuntu-logo

pegasus@pegasus:~$ clang --version
clang version 19.0.0git (https://github.com/llvm/llvm-project.git 815644b4dd882ade2e5649d4f97c3dd6f7aea200)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/pegasus/Documents/llvm-project/build/bin
pegasus@pegasus:~$ llvm-config --version
19.0.0git

Checklist

[✅] My code follows the style guidelines of this project
[✅] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[✅] I have checked my code and corrected any misspellings

@Officeyutong
This is getting built error freely on my local ubuntu 23.10.
Not sure why it's throwing error on Checks.

The tests are using LLVM 15 if I'm not wrong.
So why are they executing the if block, while in my local machine it's working as expected?

@NobinPegasus what are the options you are passing to build it locally?

@NobinPegasus what are the options you are passing to build it locally?
@hp77-creator
I've used

make build-llvm -j

I've also used
make release JOBS=$(nproc) # Build and install the runtime
export PATH=$PATH:~/.bpftime

or
cmake -Bbuild  -DCMAKE_BUILD_TYPE:STRING=Release \
           -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO
cmake --build build --config Release --target install
export PATH=$PATH:~/.bpftime

All works fine

I tried with the same command that you mentioned, I am getting errors.

@Officeyutong This is getting built error freely on my local ubuntu 23.10. Not sure why it's throwing error on Checks.

The tests are using LLVM 15 if I'm not wrong. So why are they executing the if block, while in my local machine it's working as expected?

I'm also confusing. But if you have more time, you may use https://github.com/marketplace/actions/debugging-with-ssh to debug github workflows online

I figured out the issue, seems at some place, you are using simple if where we should use #if directives, there are couple of other places where you have used #if LLVM_MAJOR_VERSION but haven't covered the cases where version is less than 17, you can make those changes and it will work fine then. I will see if I can commit here, else you can do it at your end @NobinPegasus

0001-preprocessor-directive-fix.patch
you can see this @NobinPegasus , apply this patch and add it to your PR. should work. I have also removed <llvm/Support/TargetRegistry> my system was unable to find it and it was not processing correctly. you can see if its required or not and accordingly add or remove it.

you can download the patch and
do git apply 0001-preprocessor-directive-fix.patch in your repo. test it on your system and let us know

@hp77-creator @Officeyutong What can be the probable cause?
It now builds properly. But shows error on TestCLI
AGENT: JIT session error: Symbols not found: [ __lddw_helper_map_by_fd, __lddw_helper_map_val ]
AGENT: [2024-04-11 11:37:16][critical][1325] Unable to find symbol `bpf_main`: Failed to materialize symbols: { (main, { bpf_main }) }
AGENT: terminate called after throwing an instance of 'std::runtime_error'
AGENT:   what():  Unable to link symbol `bpf_main`
AGENT:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/__w/bpftime/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/__w/bpftime/bpftime/.github/script/run_example.py", line 96, in main
    await asyncio.wait_for(expected_str_signal.wait(), AGENT_TIMEOUT)
  File "/usr/lib64/python3.12/asyncio/tasks.py", line [50](https://github.com/eunomia-bpf/bpftime/actions/runs/8645910782/job/23704474603?pr=276#step:10:51)9, in wait_for
    async with timeouts.timeout(timeout):
  File "/usr/lib64/python3.12/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError
Error: Process completed with exit code 1.

Can u try running example programs on ur system after building it? Are you getting above errors on your system or did you observe that in the CI?

Can u try running example programs on ur system after building it? Are you getting above errors on your system or did you observe that in the CI?

pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/opensnoop$ python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "/github/home/.bpftime/bpftime -i /github/home/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
cannot access local variable 'server' where it is not associated with a value
Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 55, in main
    server = await asyncio.subprocess.create_subprocess_exec(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 224, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1742, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 211, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File<EMAIL_ADDRESS>line 820, in _start
    self._proc = subprocess.Popen(
                 ^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File<EMAIL_ADDRESS>line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/github/home/.bpftime/bpftime'

@NobinPegasus
It looks like you are running the command for docker where the install is made inside /github
The default install of bpftime is made in ~/.bpftime/
You can try this instead
python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "~/.bpftime/bpftime -i ~/.bpftime" 1

I'm facing a different error. Don't know why
pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/opensnoop$ python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "/home/pegasus/.bpftime/bpftime -i /home/pegasus/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File<EMAIL_ADDRESS>line 212, in wait
    await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 75, in main
    await asyncio.wait_for(server_started_signal.wait(), SERVER_TIMEOUT)
  File<EMAIL_ADDRESS>line 519, in wait_for
    async with timeouts.timeout(timeout):
  File<EMAIL_ADDRESS>line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

Your process injected with bpftime-server directly exited
SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

Your process injected with bpftime-server directly exited SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

@Officeyutong I've seen the errors. But what might be the probable reason of this? Why is it happening?

SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

Your process injected with bpftime-server directly exited SERVER: [2024-04-12 16:24:32.503] [error] Program exited abnormally: 1

@Officeyutong I've seen the errors. But what might be the probable reason of this? Why is it happening?

It just indicates that your process injected with syscall-server exited abnormally. You may manually run that process with LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so, and execute the corresponding victim process as the example CI does, and check what happened

After running with LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/statsnoop$ python3 $ROOT/.github/script/run_example.py "./statsnoop" "./victim" "victim               0    0    /sys" "/home/pegasus/.bpftime/bpftime -i /home/pegasus/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
cannot access local variable 'server' where it is not associated with a value
Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 55, in main
    server = await asyncio.subprocess.create_subprocess_exec(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 224, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1742, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 211, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File<EMAIL_ADDRESS>line 820, in _start
    self._proc = subprocess.Popen(
                 ^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File<EMAIL_ADDRESS>line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/pegasus/.bpftime/bpftime'

@Officeyutong no improvement happened.
pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/statsnoop$ python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "/home/pegasus/.bpftime/bpftime -i /home/pegasus/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
SERVER: [2024-04-13 21:27:05.820] [error] Program exited abnormally: 1
SERVER:
Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File<EMAIL_ADDRESS>line 212, in wait
    await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 75, in main
    await asyncio.wait_for(server_started_signal.wait(), SERVER_TIMEOUT)
  File<EMAIL_ADDRESS>line 519, in wait_for
    async with timeouts.timeout(timeout):
  File<EMAIL_ADDRESS>line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

@Officeyutong no improvement happened.
pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/statsnoop$ python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "/home/pegasus/.bpftime/bpftime -i /home/pegasus/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
SERVER: [2024-04-13 21:27:05.820] [error] Program exited abnormally: 1
SERVER:
Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File<EMAIL_ADDRESS>line 212, in wait
    await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 75, in main
    await asyncio.wait_for(server_started_signal.wait(), SERVER_TIMEOUT)
  File<EMAIL_ADDRESS>line 519, in wait_for
    async with timeouts.timeout(timeout):
  File<EMAIL_ADDRESS>line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

I think the callback trace gives better overview.
SERVER: [2024-04-12 21:13:27][info][1323] Global shm constructed. shm_open_type 0 for bpftime_maps_shm
SERVER: [2024-04-12 21:13:27][info][1323] Global shm initialized
SERVER: [2024-04-12 21:13:27][info][1323] Enabling helper groups ufunc, kernel, shm_map by default
SERVER: [2024-04-12 21:13:27][info][1323] bpftime-syscall-server started
Callback triggered
Server started!
Warning: 024-04-12 21:13:37.[36](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:37)7] [warning] Unable to determine home directory. You must specify --install-location
AGENT: [2024-04-12 21:13:[37](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:38).370] [info] [agent-transformer.cpp:37] Entering bpftime syscal transformer agent
AGENT: [2024-04-12 21:13:37.370] [info] [agent-transformer.cpp:68] Using agent /github/home/.bpftime/libbpftime-agent.so
AGENT: [2024-04-12 21:13:37.370] [info] [text_segment_transformer.cpp:246] Page zero setted up..
AGENT: [2024-04-12 21:13:37.370] [info] [text_segment_transformer.cpp:274] Rewriting executable segments..
AGENT: [2024-04-12 21:13:37.801] [info] [bpftime_shm_internal.cpp:658] Global shm constructed. shm_open_type 1 for bpftime_maps_shm
AGENT: [2024-04-12 21:13:37.801] [info] [bpftime_shm_internal.cpp:29] Global shm initialized
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_func_arg, index 183
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_func_ret_id, index 184
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_retval, index 186
AGENT: [2024-04-12 21:13:37.801] [info] [agent.cpp:1[48](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:49)] Initializing agent..
AGENT: [2024-04-12 21:13:37][info][1325] Initializing llvm
AGENT: [2024-04-12 21:13:37][info][1325] Attach successfully
AGENT: [2024-04-12 21:13:37][info][1325] Agent syscall trace setup exiting..
AGENT: [2024-04-12 21:13:37.808] [info] [agent-transformer.cpp:98] Transformer exiting, trace will be usable now
AGENT: JIT session error: Symbols not found: [ __lddw_helper_map_by_fd, __lddw_helper_map_val ]
AGENT: [2024-04-12 21:13:37][critical][1325] Unable to find symbol `bpf_main`: Failed to materialize symbols: { (main, { bpf_main }) }
AGENT: terminate called after throwing an instance of 'std::runtime_error'
AGENT:   what():  Unable to link symbol `bpf_main`
AGENT:
Error: Process completed with exit code 1.

But I'm not sure why it's unable to link symbol

@Officeyutong no improvement happened.
pegasus@pegasus:~/Documents/bpftime/example/libbpf-tools/statsnoop$ python3 $ROOT/.github/script/run_example.py "./opensnoop" "./victim" "   0 test.txt" "/home/pegasus/.bpftime/bpftime -i /home/pegasus/.bpftime" 1
/home/pegasus/Documents/bpftime/.github/script/run_example.py:112: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
SERVER: [2024-04-13 21:27:05.820] [error] Program exited abnormally: 1
SERVER:
Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File<EMAIL_ADDRESS>line 212, in wait
    await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 112, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File<EMAIL_ADDRESS>line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pegasus/Documents/bpftime/.github/script/run_example.py", line 75, in main
    await asyncio.wait_for(server_started_signal.wait(), SERVER_TIMEOUT)
  File<EMAIL_ADDRESS>line 519, in wait_for
    async with timeouts.timeout(timeout):
  File<EMAIL_ADDRESS>line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

I think the callback trace gives better overview.
SERVER: [2024-04-12 21:13:27][info][1323] Global shm constructed. shm_open_type 0 for bpftime_maps_shm
SERVER: [2024-04-12 21:13:27][info][1323] Global shm initialized
SERVER: [2024-04-12 21:13:27][info][1323] Enabling helper groups ufunc, kernel, shm_map by default
SERVER: [2024-04-12 21:13:27][info][1323] bpftime-syscall-server started
Callback triggered
Server started!
Warning: 024-04-12 21:13:37.[36](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:37)7] [warning] Unable to determine home directory. You must specify --install-location
AGENT: [2024-04-12 21:13:[37](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:38).370] [info] [agent-transformer.cpp:37] Entering bpftime syscal transformer agent
AGENT: [2024-04-12 21:13:37.370] [info] [agent-transformer.cpp:68] Using agent /github/home/.bpftime/libbpftime-agent.so
AGENT: [2024-04-12 21:13:37.370] [info] [text_segment_transformer.cpp:246] Page zero setted up..
AGENT: [2024-04-12 21:13:37.370] [info] [text_segment_transformer.cpp:274] Rewriting executable segments..
AGENT: [2024-04-12 21:13:37.801] [info] [bpftime_shm_internal.cpp:658] Global shm constructed. shm_open_type 1 for bpftime_maps_shm
AGENT: [2024-04-12 21:13:37.801] [info] [bpftime_shm_internal.cpp:29] Global shm initialized
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_func_arg, index 183
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_func_ret_id, index 184
AGENT: [2024-04-12 21:13:37.801] [info] [bpf_attach_ctx.cpp:170] Register attach-impl defined helper bpf_get_retval, index 186
AGENT: [2024-04-12 21:13:37.801] [info] [agent.cpp:1[48](https://github.com/eunomia-bpf/bpftime/actions/runs/8667912296/job/23772274502#step:10:49)] Initializing agent..
AGENT: [2024-04-12 21:13:37][info][1325] Initializing llvm
AGENT: [2024-04-12 21:13:37][info][1325] Attach successfully
AGENT: [2024-04-12 21:13:37][info][1325] Agent syscall trace setup exiting..
AGENT: [2024-04-12 21:13:37.808] [info] [agent-transformer.cpp:98] Transformer exiting, trace will be usable now
AGENT: JIT session error: Symbols not found: [ __lddw_helper_map_by_fd, __lddw_helper_map_val ]
AGENT: [2024-04-12 21:13:37][critical][1325] Unable to find symbol `bpf_main`: Failed to materialize symbols: { (main, { bpf_main }) }
AGENT: terminate called after throwing an instance of 'std::runtime_error'
AGENT:   what():  Unable to link symbol `bpf_main`
AGENT:
Error: Process completed with exit code 1.

But I'm not sure why it's unable to link symbol

It says Symbols not found: [ __lddw_helper_map_by_fd, __lddw_helper_map_val ]. These symbols are defined by calling hhttps://github.com/NobinPegasus/bpftime/blob/5e701ca5f30c34e38a7148f54f246f83bb4cf5d4/vm/llvm-jit/src/llvm/llvm_jit_context.cpp#L457
Seems you have modified this function and symbols of lddw helpers functions are added into extSymbols, but another map named lddwSymbols are used at https://github.com/NobinPegasus/bpftime/blob/5e701ca5f30c34e38a7148f54f246f83bb4cf5d4/vm/llvm-jit/src/llvm/llvm_jit_context.cpp#L485

@Officeyutong Thanks. How did I do this typo... It's solved now.

Thank you!


Create a Proper Redirectable Navlink For Email In Footer Bar

This Does Not Redirect

Go ahead @swarnade

What is the update @swarnade

Sorry , My laptop Was In Issue For A While

Ok


[Windows] Maximizing window on secondary monitor broken in 0.4.0

I have 4 monitors each with different resolutions and scalings.
Since 0.4.0 maximizing the window on the monitors that are not the main monitor causes the window contents to freeze and don't scale correctly.
Rolling back to 0.3.9 and running flutter clean fixes the issue indicating that commit fa2a96308c80b11f70ad54f1d1fe74148f143c47 introduced the issue.
I am on Flutter stable 3.24.0 with dart 3.5.0
possibly related to #483

Fixed on https://github.com/leanflutter/window_manager/pull/482


⚠ ClamAV has degraded performance

In 5102fb8, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 444 ms

Resolved: ClamAV performance has improved in 52bb6aa.


helix to 22.12

Checklist

[x] Checked the nixpkgs master branch

[x] Checked the nixpkgs pull requests

Project name
nix search name:

current version: 22.08.1
desired version: 22.12
Notify maintainers

maintainers: @danth @yusdacra
Note for maintainers
No PR.

Your desired version is already in nixpkgs. Relevant pr: https://github.com/NixOS/nixpkgs/pull/204931

thanks. apparently it hasn't gotten into 22.11.


how do I install this script

how do I install this script ?
is this a Mobile or Web client / server ?

This repo hasn't been updated in a few years. You can install it, but it's probably not going to be a good experience without a lot of work to update it.
If you want to install it though, you'll need NPM and Composer. It's a PHP app, so you can run it in an Apache server. You'll also need to start the PubSub server separately.


Switch to kotlin gradle dsl and use version catalogs

switches from groovy to kotlin for most gradle files
uses version catalogs
updates gradle version to latest

I changed some annotation imports due to them not being found anymore, may have been an issue with my IDE, not sure.


Add Latency Conf 2018

@leftclickben mentioned Latency conf today and I was going to look it up but couldn't see it here...
Fixed.

Thank you! 🎉


fix cert manager webhooks cert renewal

This PR aims to fix a bug found regarding the web-hook certificate changes brought in this PR https://github.com/nephio-project/porch/pull/53
The web-hook server should use a pointer to the TLS certificate variable rather than the variable itself such that when the renewal of the certificate takes place and the variable is updated without any restart/update of the HTTP server being required.

/approve
/lgtm

/retest

/lgtm


Fix security headers not being set for root url

This ideally should be fixed by historyApiFallback webpack setting instead. However seems it doesn't work with this version of webpack. The original developers have actually set this setting in webpack config and then worked around with the custom middleware serving index.html for non-root urls (ex /namespace)
Changed the order of middlewares making security tokens to be set before the first middleware that serves static files (index.html)

@just-at-uber fyi, if user starts the app from root url /, then the security headers will not be set. Related PR https://github.com/temporalio/web/pull/256


messagebox cannot show button content

I use the control: MessageBox, but the button ok/cancel text not show?

the code:
MessageBox.Show("222", "title", MessageBoxButton.OKCancel, MessageBoxImage.Information);

Hi, for Styles and Translations to work in the WPF libraries you need to add the resource dictionary "ControlStyles.xaml" from Project "MaSch.Presentation.Wpf.Themes" to your app.xaml.
You can see an example of this here: https://github.com/MaSch0212/cura-manager/blob/cdf8632a92c7e157984a889ac5dad374a7ac8ecd/src/CuraManager/App.xaml#L13
You currently also need to add the "MaSch.Presentation.Wpf.Themes" NuGet package to your project in Version 0.1.3 in addition to the "Controls" one, because the dependency in that Controls package is currently using and older version of the Themes package (0.1.2) which contains a bug.

Version 0.1.5 of MaSch.Presentation.Wpf.Controls is published that now uses the correct version of Themes, so an explicit reference to the Themes package is not necessary anymore ;)

I guess this issue is resolved, so I will close it. If you got more issue, please let me know.


Redis bgsave fails and prevents any further persistence

I'm runnin redis as a windows service (with the default settings). From time to time the bgsave command fails. I only can resolve this issue by re-starting the service (and loosing all data since the last successful bgsave :-( ) This is what the redis info command says (the info command has been run after the service restart):
Server
redis_version:2.8.19
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9968db13395be4aa
redis_mode:standalone
os:Windows
arch_bits:64
multiplexing_api:winsock_IOCP
gcc_version:0.0.0
process_id:1208
run_id:de4f7916b1a32b53648cd774ebe3a5f95e64d5a0
tcp_port:6379
uptime_in_seconds:9626
uptime_in_days:0
hz:10
lru_clock:12735354
config_file:C:\Program Files\Redis\redis_store.conf
Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
Memory
used_memory:131435032
used_memory_human:125.35M
used_memory_rss:131383704
used_memory_peak:132169328
used_memory_peak_human:126.05M
used_memory_lua:35840
mem_fragmentation_ratio:1.00
mem_allocator:dlmalloc-2.8
Persistence
loading:0
rdb_changes_since_last_save:16
rdb_bgsave_in_progress:0
rdb_last_save_time:1438798500
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1438798501
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
Stats
total_connections_received:1146
total_commands_processed:19651
instantaneous_ops_per_sec:1
total_net_input_bytes:963058
total_net_output_bytes:0
instantaneous_input_kbps:0.04
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:44
keyspace_misses:1217
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:93601
Replication
role:master
connected_slaves:1
slave0:ip=<IP_ADDRESS>,port=6379,state=online,offset=233779,lag=1
master_repl_offset:233779
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:233778
CPU
used_cpu_sys:11.28
used_cpu_user:2.11
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
Keyspace
db0:keys=375844,expires=0,avg_ttl=0
Logfile
Here is the relevant part of the logfile:
[1716] 05 Aug 06:49:32.088 * Background saving terminated with success
[1716] 05 Aug 06:54:33.045 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 06:54:34.652 # fork operation complete
[1716] 05 Aug 06:54:34.839 * Background saving terminated with success
[1716] 05 Aug 06:59:35.001 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 06:59:36.841 # fork operation complete
[1716] 05 Aug 06:59:37.075 * Background saving terminated with success
[1716] 05 Aug 07:04:38.033 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:04:38.033 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error
[1716] 05 Aug 07:04:38.033 # Can't save in background: fork: Invalid argument
[1716] 05 Aug 07:04:38.142 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:04:39.905 # fork operation complete
[1716] 05 Aug 07:04:40.092 * Background saving terminated with success
[1716] 05 Aug 07:09:41.002 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:09:41.002 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error
[1716] 05 Aug 07:09:41.002 # Can't save in background: fork: Invalid argument
[1716] 05 Aug 07:09:41.111 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:09:41.111 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error
[1716] 05 Aug 07:09:41.111 # Can't save in background: fork: Invalid argument
[1716] 05 Aug 07:09:41.221 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:09:43.030 # fork operation complete
[1716] 05 Aug 07:09:43.217 * Background saving terminated with success
[1716] 05 Aug 07:14:44.034 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:14:44.034 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error
[1716] 05 Aug 07:14:44.034 # Can't save in background: fork: Invalid argument
[1716] 05 Aug 07:14:44.143 * 10 changes in 300 seconds. Saving...
[1716] 05 Aug 07:14:44.143 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error

Hi @JeHoDe,
thank you for reporting the issue.
The problem seems caused by the fact that the system paging file is not large enough or is too fragmented to map the virtual memory of the child process.
Please take the time to read the Memory Configuration wiki page.
Also, consider updating to the latest v2.8.2101 release.
If the problem still persists, please let us know.

Closing this issue since it hasn't been updated in a month.
The problem was originally caused by "The paging file is too small for this operation to complete."
Increasing the system paging file size should prevent the error to occur.
If the error occurs in latest builds, a more informative error message is displayed.


simd: reapply and fix split cursor advancing

This reapplies https://github.com/seanmonstar/httparse/pull/156 with an additional commit on top in order to fix https://github.com/seanmonstar/httparse/issues/172.

Note, cursory benches for this on M1 / arm64 show a ~10% perf regression vs main

@seanmonstar I can take a deeper look at this before reconsidering landing it, haven't looked into it in depth, but potentially there could be gains on avx2 simd from this approach because it avoids reprocessing an entire/last block on SIMD miss, allowing the SWAR or byte-wise code to pick up.
Since avx2 operates on 32B/256bit blocks it possibly "misses" a lot on header-values, short URLs etc...
So definitely a variant of this is worth considering, but I think it should be rigorously benchmarked given those shared in #156 were unclear... @seanmonstar probably worth adding CI benches (which can be noisy, but probably not meaningfully so these pure compute bound flows)

Also a big factor here could be avx2 being inlined or dispatched at runtime (when generically compiled for x64), the inlined version compiled for avx2 will obviously have less overhead.
@lucab Also are all the pub => pub(crate) changes necessary ?

@seanmonstar probably worth adding CI benches (which can be noisy, but probably not meaningfully so these pure compute bound flows)

Did a first pass at CI benches: https://github.com/seanmonstar/httparse/pull/179, this PR should show a measurable delta

@AaronO from https://github.com/seanmonstar/httparse/pull/156#issuecomment-1980339794:

There is a minor cleanup bundled in this PR (marking several functions as pub(crate)) which I did in order to make sure I wasn't changing public APIs. I can split that to a dedicated PR if you prefer.

@AaronO from #156 (comment):

There is a minor cleanup bundled in this PR (marking several functions as pub(crate)) which I did in order to make sure I wasn't changing public APIs. I can split that to a dedicated PR if you prefer.

@lucab Yes I think that should be split out. Especially since the original PR made substantial (AVX2) perf claims, it's best if we can focus on that and remove unrelated "noise". The alt refactor should be judged on its own merits. The focus here should be demonstrating, explaining clear perf benefits. From the benchmarks (local [arm64] & CI) that appears unclear.

@lucab Modern CPUs with speculative execution can counterintuitively be faster "processing the same bytes twice" if both execution paths are independent. There can also be tradeoffs impacting inline-ability if you increase register usage in hot loop, etc...
I touched this code a while back so not 100% sure, but I think I explored your change (if I'm distilling it correctly to returning partial matches) and concluded (based off local M1 benches at the time) that it wasn't a net positive.
For example your original PR #156, explains the improvement with:

This refactors all SIMD modules in order to make the value-matching logic self-contained. Thus, all bytes-cursor manipulations are now grouped and performed once at the end, outside of SIMD logic.

I should double check but IIRC, minimizing the amount of times you update the bytes-cursor sounds like a plausible improvement but in practice it impacts lowering. What actually happens here is that the "stack-allocated" Bytes struct actually gets allocated into 3 registers (start, cursor, end), we're always reading from cursor, with your code I think it get's lowered as basepointer+offset or ends up requiring an extra register for that inner loop.
Long story short, we should really distill this perf improvement hypothesis to its essentials and fully test/understand that.

@lucab Ok looking at the CI benches there does appear to be a beyind-noise improvement for sse42 (not avx2, but possibly I missed something in the CI benches).
It smells like it might boil down to inline-ability of sse42. Would need to double check, I'll take a deeper look tomorrow.
In part it's a perf bug vs purely raising the ceiling, especially for small values sse42 was substantially underperforming swar, possibly due to (not-)inlining (if sse42 is called non-inlined in the loop you pay some painful register init costs).

@AaronO Ack, I'll split the pub cleanup as I was originally open to.
I've marked this PR as a draft, as I think we aren't going to land this as-is anyway.
For context, I wasn't really looking at perf improvements when I opened this, but it happened as a local side effect:

My goal was to (eventually) use the bytes crate in this library, but then I realized that this required SIMD-related groundwork was actually providing performance improvements on its own.


Mods not loading on Steam Deck.

I successfully installed 7th Heaven on my Steam Deck using dotaxis’s script. I installed mainly SYW mods and when I launch the game (from my SD card) the FFNx screen appears but besides that the game doesn’t load any of the mods. If I installed any sound mods, no sound plays even with VGMStream enabled.

Wrong repo


Subscriber maxes out CPU after exactly 60 minutes on GKE

Environment details

OS: GKE
Node.js version: 12.14.1
npm version: –
@google-cloud/pubsub version<EMAIL_ADDRESS>/<EMAIL_ADDRESS>
Steps to reproduce

Start a pod that subscribes to an idle subscription
Wait 60 minutes
Observe node process eat up all available CPU

I have a pod that subscribes to a subscription which is 100% idle (i.e. no messages published on that topic) during the observation period.
Exactly 60 minutes after the pod started it eats up all available CPU until the pod is deleted manually to force a restart.
This behavior does not appear when using the C++ gGRPC bindings as suggested in https://github.com/googleapis/nodejs-pubsub/issues/850#issuecomment-573900907
I have also enabled debug output through
GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

I've put the full debug output (as fetched from Google Cloud Logging) of the entire lifetime of the pod in this gist.
There is no further output after the last line even though the pod kept running for a while as can be seen form the graph above: the bump in CPU usage is the time where the pod eats up all available CPU and doesn't seem to log anything. (Times in the logs correspond to the times in the graph above with 1h timezone difference, so 10:51 in the logs = 11:51 in the graph).
I can reproduce this problem with any of my subscriber pods.
I believe it is very likely that this problem also falls into the category of #868.

same is happening to us. Thought it was a bad implementation...

We are also seeing the same issue. We've been having a lot of strange behavior with this library.

Thanks everyone, especially for that log output! I find this to be particularly suspicious:
2020-02-19T10:10:14.382Z | call_stream | [6] write() called with message of length 0

I'm working on trying to build some kind of repro to pass on to the grpc team.
We have only one Node library that's defaulting to the gRPC C++ bindings at this point (spanner), so there has been some hesitation about switching this one back to it by default, but it also seems to be affecting a lot of people. (Also, the C++ one requires node-gyp, which may not be available in some environments, so adding a hard dependency to it is possibly problematic.) The current thought is that it may have something to do with this library's more extensive use of streaming APIs. So progress is being made there. Thanks for the patience and debugging help!

We are seeing a similar issue where our node process gets killed if a subscription has been idle for 60 minutes - without any logs.

Can you clarify when you restarted the process relative to the final timestamped log message? Was the log cut off because you restarted, or was there a time gap after the last log message and before the manual restart?

We are not manually restarting the process. Its GKE, that finds an unhealthy "health" check and restarts it. There has been no code changes except for upgrading packages. The worker has a PubSub connection and at night is mostly idle (except for running a cron job every hour). It seems that since pretty close after 1 hour the process just gets killed because of unknown reasons (ie we dont see any error in the logs) but we suspect some timeout with the pubsub connection

The log you posted shows that the grpc library establishes a connection and some requests are made, then exactly an hour later the connection drops and a new connection is established (this is a normal thing that the grpc library is designed to handle). Then, right after the new connection is established, the log cuts off, saying that the process was restarted. I'm wondering if you can see timestamps for when GKE restarts the process, and when that is relative to the log lines in the file.
The reason is that I'm wondering if the problem I should be looking for is more like "something weird happened while reconnecting and that triggered GKE to restart it" or "the process entered an infinite loop without logging anything, and GKE eventually detected it".

Can you clarify when you restarted the process relative to the final timestamped log message? Was the log cut off because you restarted, or was there a time gap after the last log message and before the manual restart?

I only restarted about 20 mins after the last log message (see the graph in the original post, where the bump in CPU usage goes down again).
Between the last log message in my gist and the restart there were no further log messages retrievable from kubectl or Google cloud logging and the process consumer all CPU that Kubernetes allowed.

The log you posted shows that the grpc library establishes a connection and some requests are made, then exactly an hour later the connection drops and a new connection is established (this is a normal thing that the grpc library is designed to handle). Then, right after the new connection is established, the log cuts off, saying that the process was restarted. I'm wondering if you can see timestamps for when GKE restarts the process, and when that is relative to the log lines in the file.
The reason is that I'm wondering if the problem I should be looking for is more like "something weird happened while reconnecting and that triggered GKE to restart it" or "the process entered an infinite loop without logging anything, and GKE eventually detected it".

My observation is: after 60 minutes of repeated log output every 30 seconds all log output stops and the node process jumps to max available CPU usage for no apparent reason. The log output in the gist shows everything retrieved from that pod which formally lived about 20 minutes longer than the last line of logs.

Just to make sure we're not missing out any details, our initialization logic is like this:
const pubsub = new PubSub();
const subscription = pubsub.subscription(subscriptionName, {
 flowControl: {
   maxBytes: Math.floor(getenv.int('MEMORY_LIMIT_BYTES') * 0.5),
 },
});
subscription.on('message', handleMessage);

where MEMORY_LIMIT_BYTES comes from the Kubernetes pod spec resource limits.

I have published @grpc/grpc-js version 0.6.17 with a possible mitigation for the issue and some more logging that should hopefully be relevant. Can you try that out and share the resulting logs?

Whoops, I have troubles downloading the logs, there are megabytes of:
E 2020-02-21T20:47:18.959847704Z 2020-02-21T20:47:18.959Z | channel | Failed to start call on picked subchannel <IP_ADDRESS>:443. Retrying pick
E 2020-02-21T20:47:18.930389263Z 2020-02-21T20:47:18.930Z | channel | Pick result: COMPLETE subchannel: <IP_ADDRESS>:443 status: undefined undefined

during the period where the process runs with maxed out CPU. So basically tryPick seems to run in an infinite loop.
I can highly recommend not running @grpc/grpc-js version 0.6.17 with debug flags enabled 😉

@murgatroid99 I managed to download 140MB of logs from Google Cloud Logging and put the part until the infinite loop begins into the gist.

@ctavan Thanks for the continued debug info! This actually is starting to sound like something we were talking about as a possibility for a cause. Basically, after some time (I think it might've been 30 minutes) the stream is closed, and we attempt to reconnect it. If something prevents that connection, it's not supposed to busy loop reconnecting, but that might be what it's doing.
A gRPC fix might be useful if that's what it is (this guy - https://github.com/grpc/grpc-node/pull/1271) but we also talked about putting in a workaround in the nodejs-pubsub library if that proves more elusive.

I had a feeling that the problem was something like that, but I tried to mitigate a slightly different potential issue. I didn't realize it would loop that tightly and generate that many log lines, though I do want to note that in normal circumstances, when you're not encountering this bug, you'll usually get about one extra log line per request.
What's happening here is that a connection has been established, but something is in an invalid state that is preventing it from actually starting a stream, so it's just repeatedly retrying. The reason it only happens on 0.6.16 and above is that it's caused by grpc/grpc-node#1251, which was supposed to fix googleapis/nodejs-firestore#890. I'll try to get a new version out on Monday that fixes this. For now, if you can force it to use 0.6.15 you'll probably be fine.

I have now published @grpc/grpc-js version 0.6.18. I'm pretty sure I've fixed it this time, and if not I added some more relevant logging. Can you try that out?

@murgatroid99 I've upgraded to<EMAIL_ADDRESS>and the specific problem described in this issue is now gone. Great work!
For the sake of completeness I have added the log output of the (still running since 160 minutes) pod to the gist.
I'll leave this issue open in case you want to update the transitive dependencies of @google-cloud/pubsub so that it pulls at least 0.6.18 of @grpc/grpc-js.

@ctavan I will take a look at updating the versions in nodejs-pubsub. Thank you for all the help in testing this stuff!

Had a similar issue and switched to the older C++ grpc client in the meantime. It seems to be behaving.

@jperasmus Did you try out the latest version of the library lately, which pulls in grpc-js 0.6.18? That library released 1.x recently, and I'm planning to pull that in after pubsub has its 2.0 release.

@feywind I did try with the latest version at the time, which was after Christoph posted about 0.6.18. I will try with the latest v1 grpc-js client on one of our test environments after v2 of this library includes it.

@jperasmus Checking back in - have you had better luck with the grpc-js client 1.x? That should be picked up here now (by way of gax updates).

Hi @feywind thanks for following up. I actually moved to a different company where we are not using this package anymore, but before I left, I did run an experiment for almost 2 weeks with the v1 grpc-js client on a test environment and it was substantially more stable than the old grpc client. Basically, where the grpc client would throw at least one connection issue in production per day, the grpc-js client had no connection issues in the test environment under similar, but slighter less, load than that of production. As far as I am aware, grpc-js is running in production now for the old company.

Okay, cool! If anyone is still running into this with grpc-js, feel free to re-open.


Remove support for #[derive(IntoBytes)] on unions?

Progress

[ ] Audit Fuchsia, Google, and other Rust dependencies for use of #[derive(IntoBytes)] on unions
[ ] Put #[derive(IntoBytes)] on unions behind a --cfg

Details
Support for #[derive(IntoBytes)] on unions was added in https://github.com/google/zerocopy/commit/9c19cbe1cf5c2621ae9a63179664db3470146c10 (reviewed at fxrev.dev/639087). This support is sound on the assumption that if every field of a union is IntoBytes and there is no extra padding before or after any field, then no bit-valid instance of that union can have uninitialized bytes.
However, this assumes too much. It is currently up in the air whether this actually holds. It's not clear from reading the history why we were okay adding this implementation in the first place, but it now seems like it may have been premature.
It may eventually become the case that this assumption is guaranteed by Rust, so this may eventually become a non-issue. However, for the time being, it's a violation of our soundness policy, which promises to be sound on all future Rust compilers.
Unfortunately, existing users rely on #[derive(IntoBytes)] support on unions, so removing this support will break users. This is likely not a problem on today's Rust, so forcing users to migrate to something else might be too drastic of a solution. Instead, I propose that we discourage further use by gating #[derive(IntoBytes)] on unions behind a --cfg. The reason for a --cfg instead of a Cargo feature is that Cargo features are footguns in cases like this - it's easy for crate A to enable the feature and then for crate B, which depends on A, to accidentally rely on that feature despite not enabling that feature itself. For something with soundness and stability implications, that's risky. By contrast, a --cfg requires the top-level crate to enable it for all downstream crates.
In parallel, we will try to get Rust to promise what we need in order for #[derive(IntoBytes)] on unions to be guaranteed sound on all future compilers.

Summarizing a discussion with @jswrenn
The situation may be more permissive than I originally thought. In particular, while it might be the case at some point in the future that any union with all uninitialized bytes is considered bit-valid (ie, it is sound to produce such a union value), it is currently the case that creating such a value requires unsafe code. Safe code is only permitted to instantiate or modify a union value using its actual fields.
It is already the case that zerocopy's traits imply both bit validity and safety requirements. For example, FromBytes attests that a type has a particular bit validity, but it also permits users to construct arbitrary instances of a type, which implies that that type must not have certain safety invariants which are inconsistent with constructing a value from arbitrary bytes. We can say that IntoBytes is similar - it adds a new safety invariant to types that they must only be constructed from their fields.
In order for this to continue to be sound, we need the following forwards-compatibility guarantees from Rust:

It will never be safe to construct a union value other than from its fields or by overwriting one of its sub-fields
In a typed copy of a union value, any bytes previously written are preserved so long as those bytes are not written to a byte offset at which there is padding in all fields of the union


Using https proxy as upstream proxy occur Server disconnected

Steps to reproduce the problem:

Using https proxy as upstream proxy as following commands: mitmdump --mode upstream:https://foo.com -p 8081 -s ./proxy/flow_dumper.py

The mode represents as below:
Client <--- http ---> MITMproxy<----HTTPS(Proxy)---> Internet

It shows Server disconnected as below

<IP_ADDRESS>:51409: HTTP protocol error in client request: Server disconnected
<IP_ADDRESS>:51409: clientdisconnect
 << Server disconnected
<IP_ADDRESS>:51408: HTTP protocol error in client request: Server disconnected
<IP_ADDRESS>:51408: clientdisconnect
<IP_ADDRESS>:51412: clientconnect
 << Server disconnected
<IP_ADDRESS>:51412: HTTP protocol error in client request: Server disconnected
<IP_ADDRESS>:51412: clientdisconnect
<IP_ADDRESS>:51419: clientconnect
 << Server disconnected
<IP_ADDRESS>:51419: HTTP protocol error in client request: Server disconnected
<IP_ADDRESS>:51419: clientdisconnect
<IP_ADDRESS>:51429: clientconnect
 << Server disconnected
... ...

Any other comments? What have you tried so far?
I have no idea what options set to mitmproxy. Try several options while it doesn't work.
System information

Mitmproxy: 4.0.4
Python:    3.6.1
OpenSSL:   OpenSSL 1.1.0i  14 Aug 2018
Platform:  Darwin-17.5.0-x86_64-i386-64bit

You probably just want to use "http://". mitmproxy will of course still send HTTPS requests over that proxy.
As per #1775, we currently don't support TLS-over-TLS, which would be required for what the Chromium guys call a "Secure Web Proxy". Most regular HTTP proxies don't work that way and should work just fine.

@mhils Oh sorry I had make a mistake on mode i draw above.  I update the graph
Client <--- http ---> MITMproxy<----HTTPS(Proxy)---> Internet

While I wan to visit the https site like https://www.google.com through http://<IP_ADDRESS>:8081 (mitmproxy). It represent as the forward node to second https proxy.

Sorry, I don't understand your point or what you are trying to achieve.

@mhils I want to use the mitmproxy in upstream proxy mode to forward the traffic to https proxy.

There's no such thing like an "HTTPS proxy". You are probably referring to an HTTP proxy that support TLS after CONNECT (in which case you just use http:// in the upstream specification), or you are referring to a "Secure Web Proxy" - a relative new (and still rare) type of proxy that requires TLS over TLS, which we unfortunately currently don't support.

@mhils Yeah, I mean that it was probably the proxy which could support TLS after Connect. Could mitmproxy support this proxy in upstream mode?

Any updates about this ? I'm searching for this problem too, "https://" in  --mode upstream give me  Upstream proxy myproxy:9702 refused HTTP CONNECT.
Any solution for this ?
Thanks


tracing issue of the plugin system.

add the option directive to enable unstable feature via 'option "feature.plugins" "true"'
handle plugin directives in zhang, to enable the plugin when launching up.
2.1 plugin directive should support meta
plugin directives should support local file(wasm) and https source as source

we decide to use the extism as the base of plugin system


Add Kontent to the headless CMS list

Hey @peduarte,
would you be open in adding Kontent to the list of headless CMS?
I have seen you linked @ondrabus's pull request with our JAMStack report recently, so I think it would be a nice enhancement.

Definitely 💯
Thanks!


[FLINK-11168] Fix LargePlanTest timeout on Travis

What is the purpose of the change
Fix a flaky test on Travis. The test completes in 500ms on my machine. On Travis the resources are much more limited and it fails occasionally.
Brief change log

Increase test timeout to 30 seconds
Reduce complexity of tests by decreasing branching factor

Verifying this change
The test still takes forever to run without the fix in #7276.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature?  no

Thanks for reporting @zentol. This should fix any flakiness.

Thanks!


Theta offset fix

This fix should address the Q-range for PolarizedNeutronProbe not matching the Q-range for the cross-sections in the Probe, when a non-zero theta_offset is present in one of the cross-sections (#114)
Uses a the cached self.calc_Qo if the theta_offset values in the cross-sections do not change.
If all the theta_offset values are the same, it does a minimal calculation to update Q (this is a common situation)
If all the theta_offset values are not all the same, and they change, a full calculation of the probe Q the (measurement_union(xs)) is called.

Tested, seems to show 10% slowdown in fits in worst case (single layer with microstepping disabled to emphasize model stepping overhead)


When deploying Qwen-2VL using Swift, the generated results encounter an early termination issue.

I deployed Qwen-2VL-72B using Swift, but during multi-image content inference, the generated results consistently terminate early. Could you advise on how to resolve this?
The startup script is as follows:
#!/bin/bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 swift deploy \
    --model_type qwen2-vl-72b-instruct \
    --model_id_or_path /local_model_path \
    --infer_backend vllm \
    --limit_mm_per_prompt '{"image":8}' \
    --tensor_parallel_size 8 \
    --max_model_len 32768 \
    --enforce_eager True \
    --port 9782 \
    --host <IP_ADDRESS> \

please help me！！！Thank you very much！！！

i got t it   ！！！
the script needs setting the parameter "--mode_kwargs"
""""
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 swift deploy
--model_type qwen2-vl-72b-instruct
--model_id_or_path /Qwen2-VL-72B-Instruct
--model_kwargs '{"image_factor":28, "resized_height":1024, "resized_width":768}'
--infer_backend vllm
--limit_mm_per_prompt '{"image":8}'
--tensor_parallel_size 8
--max_model_len 32768
--enforce_eager True
--port 9782
--host <IP_ADDRESS>
"""
Besides, i think 'max_pixel' or 'min_pixel' is also useful~

Hello, what issue are you facing? Has it been resolved?


fix: check label list list column type earlier

fixes #2844

make: *** [test] Illegal instruction: 4
python/tests/test_filter.py::test_duckdb

rust linux-arm Error: The operation was canceled.

@westonpace @wjones127 PTAL, thanks!

New version: (0, 17, 1)
File "/home/runner/work/lance/lance/pr/../base/ci/check_versions.py", line 64, in
new_version[1] == last_version[1] + 1
AssertionError: Minor version should have been bumped because there was a breaking change.
Last version: (0, 17, 0)

Thanks for cleaning this up!

Thanks for reviewing and providing good suggestions!


Is it possible to do fast batch processing using a GPU?

I want to use spleeter to clean up a large quantity of audio files (>10M). When running on the GPU, however, there is some bottleneck that is severely constraining the program: I see less than 1% GPU utilization on average.
I can readily reproduce the issue by running this script in a directory with a large number of files and watching the GPU utilization using nvidia-smi: https://github.com/deezer/spleeter/issues/44
If I wire up my dataset to the pytorch variant, I get >100x more performance. The problem is that those implementations have some bug that causes the model to not perform well.
Is there some trick to improving batch performance on TF?

Using Spleeter v2.3.0
Ubuntu
GPU is RTX 3090. Processor is EPYC 7502.

Hi @neonbjb,
Yes, there seems to be troubles with GPU usage on spleeter v2.3.0. For some reasons I can get the processing to be done on GPU. We need to investigate further what's wrong. In the meantime, an older version (such as 2.2.1) should do the trick: using the docker image deezer/spleeter-gpu:3.8 (that uses spleeter 2.2.1) with nvidia-docker, I could process more than 3h audio in less than 2 minutes with the 4stems model on a single nvidia Titan X.

Ok, I've managed to make spleeter v2.3.0 work on GPU.
I just had a CUDA version compatibility issue.
As spleeter v2.3.0 uses tensorflow 2.5, it requires CUDA 11.2 (with proper driver version) + CuDNN 8. So make sure you have a proper CUDA install to make tensorflow 2.5 work.
With this, I could again process more than 3h of audio in less than 2 minutes, with a quite heavy GPU usage.

Hey, I appreciate the debugging. I tried your suggestion but it did not seem to affect performance. I managed to get what I wanted to done with the torch repo, so I no longer need any help. This is honestly probably just due to my naivete with tensorflow, closing.