nwis_client "sqlite3.OperationalError: database is locked"

When running the following code, I get an error. I've noticed this error seems to occur around the ~2500 site mark. That is I'm having trouble retrieving data from more than ~2500 sites at a time. The script raises the error and then freezes until I cancel it with ctrl-c.
Advice?
from hydrotools.nwis_client.iv import IVDataService
from hydrotools.nwm_client.gcp import NWMDataService

model_client = NWMDataService()
xwalk = model_client.crosswalk
xwalk = xwalk[xwalk["usgs_site_code"].str.isdigit()]

obs_client = IVDataService()
data = obs_client.get(sites=xwalk["usgs_site_code"].values)

Traceback (most recent call last):
  File "main.py", line 15, in <module>
    main()
  File "main.py", line 10, in main
    data = obs_client.get(sites=xwalk["usgs_site_code"].values)
  File "~/env/lib64/python3.8/site-packages/hydrotools/nwis_client/_utilities.py", line 93, in wrapper
    return fn(*args, **kwargs)
  File "~/env/lib64/python3.8/site-packages/hydrotools/nwis_client/iv.py", line 236, in get
    raw_data = self.get_raw(
  File "~/env/lib64/python3.8/site-packages/hydrotools/nwis_client/iv.py", line 446, in get_raw
    results = self._restclient.mget(parameters=query_params, headers=self._headers)
  File "~/env/lib64/python3.8/site-packages/forge/_revision.py", line 328, in inner
    return callable(*mapped.args, **mapped.kwargs)
  File "~/env/lib64/python3.8/site-packages/hydrotools/_restclient/_restclient.py", line 162, in mget
    return self._add_to_loop(
  File "~/env/lib64/python3.8/site-packages/hydrotools/_restclient/async_helpers.py", line 23, in _add_to_loop
    return self._loop.run_until_complete(coro)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "~/env/lib64/python3.8/site-packages/forge/_revision.py", line 322, in inner
    return await callable(*mapped.args, **mapped.kwargs)
  File "~/env/lib64/python3.8/site-packages/hydrotools/_restclient/_restclient.py", line 231, in _mget
    return await asyncio.gather(
  File "~/env/lib64/python3.8/site-packages/forge/_revision.py", line 322, in inner
    return await callable(*mapped.args, **mapped.kwargs)
  File "~/env/lib64/python3.8/site-packages/hydrotools/_restclient/_restclient.py", line 184, in _get
    resp = await self._session.get(url, headers=_headers, **kwargs)
  File "~/env/lib64/python3.8/site-packages/forge/_revision.py", line 322, in inner
    return await callable(*mapped.args, **mapped.kwargs)
  File "~/env/lib64/python3.8/site-packages/hydrotools/_restclient/async_client.py", line 49, in _request
    resp = await bound_resp()
  File "~/env/lib64/python3.8/site-packages/forge/_revision.py", line 322, in inner
    return await callable(*mapped.args, **mapped.kwargs)
  File "~/env/lib64/python3.8/site-packages/aiohttp_client_cache/session.py", line 62, in _request
    await self.cache.save_response(new_response, actions.key, actions.expires)
  File "~/env/lib64/python3.8/site-packages/aiohttp_client_cache/backends/base.py", line 169, in save_response
    await self.responses.write(cache_key, cached_response)
  File "~/env/lib64/python3.8/site-packages/aiohttp_client_cache/backends/sqlite.py", line 189, in write
    await super().write(key, sqlite3.Binary(self.serialize(item)))
  File "~/env/lib64/python3.8/site-packages/aiohttp_client_cache/backends/sqlite.py", line 170, in write
    await db.execute(
  File "~/env/lib64/python3.8/site-packages/aiosqlite/core.py", line 190, in execute
    cursor = await self._execute(self._conn.execute, sql, parameters)
  File "~/env/lib64/python3.8/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
  File "~/env/lib64/python3.8/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
sqlite3.OperationalError: database is locked

Hmmm, thanks for reporting this @jarq6c! This is a little surprising given that we are not using multiprocessing. This smells like an upstream bug, but i'll need to do a little digging to verify that.
For a little context sqlite, the db used by nwis_client to cache http requests, uses a multiple reader single writer model where when a writer starts a write, the whole db is locked. This consequently means that any readers or writers that try to access the db while the lock is held error out. Libraries that interact with sqlite will often use a spin lock to get around this.
This error is surprising to me because we are using an async approach to retrieving data instead of threads or multiprocessing. It is surprising because I would expect that writing to the cache is a blocking operation. I suspect what is happening is a lot of async writes to the cache (db) (that might be large in size and take more time) and no logic to enforce sqlite's the single writer constraint. Meaning, you need some way of capturing that a write to the db is occurring. Example solutions are a readwrite lock on the db connection object or a retry backoff strategy.
Ill do some digging later this morning and try to find a way around this and report back. Thanks again, @jarq6c!

Starting to dig now. It looks like others have run into similar problems when using aiosqlite. See https://github.com/omnilib/aiosqlite/issues/234.

In https://github.com/omnilib/aiosqlite/issues/234 (the issue I previously linked) a provided solution was to limit the number of concurrent asyncio.gathers.

aiosqlite does not use a connection pool. This is likely a limit of the underlying sqlite3 implementation. I'd suggest using a concurrency-limited gather, like aioitertools.asyncio.gather(..., limit=X), especially with many concurrent writes, where only one connection would ever be able to write to a shared database anyways.

asyncio.gather is used in the _restclient's mget method to do concurrent request retrieval which nwis_client is using. I am not keen on placing a concurrency limit at the library level like was suggested above because it could hinder performance for some users depending on their hardware/software. I could be talked into adding a optional keyword argument that exposes this low level detail if we keep running into it. However, I think the most viable solution is to increase the number of sites returned in each request. You can pass the max_sites_per_request keyword argument (current default is 20 sites) to IVDataService.get to effectively change the request chunking size. I think increasing this to something like 50 or 75 will reduce the likelihood of running into this problem. @jarq6c, can you try this and see if you continue to run into the same issue?

@aaraney Increasing max_sites_per_request to 50 fixed the issue.
Of note: I haven't been able to get this error to appear on all systems. I expect this is related to the performance of sqlite on different systems? Do you recall how we settled on default number of 20 sites per request?

Great!

Of note: I haven't been able to get this error to appear on all systems. I expect this is related to the performance of sqlite on different systems?

I think it's less about sqlite's performance per say, but instead io speed. So for example, at the hardware level an SSD vs a HDD and at the software level OS userspace / kernel space switching that's done during reading and writing.

Do you recall how we settled on default number of 20 sites per request?

I will have to look if I have the numbers somewhere, but it was a back-of-the-napkin calculation. I requested data at something like 10, 20, 40, 60, 80, 100, 500 sites and looked at request-response full travel times and 20 was on average the fastest.

Barring submission of another issue it's probably safe to close this one. The default of 20 sites per request likely covers most users, so changing it might do more harm than good.
Note the machine on which I got this error was a virtual machine, running on an ancient OS, stored on an HDD, and frequently subject to scans that eat up a significant portion of system resources. So, quite an extraordinary case remedied by a simple solution.
Thanks for the help!


Add End-2-End tests to the core

Missing description. I believe that this is still relevant and don't want to close it, but we need at least a simple use-case description to properly track this / evaluate its scope.


Merge develop branch to main branch and create tag v2.0.1_impl, v2.0.1_test

Merging develop to main branch

[ ] Create pull request to merge develop branch to main branch
[ ] Create a tag v2.0.1_impl with the following details of the open issues in the version 2.
[ ] Create a tag v2.0.1_test for publishing the testsuites for version 2.

Implementation of ApplicationPattern v2.0.1
This release contains,

Open Issues
Report timestamp :
Total run Duration :
Average response time :
Total assertions :
Total failures :
Total skipped test :
Consistent Issues :

Issue Number
assertions failures

[[Issue_number]]
[[assertions_failures]]

Inconsistent Issues :

Issue Number
assertions failures

[[Issue_number]]
[[assertions_failures]]

Closing the issue as the v2.0.1_impl and v2.0.1_test tags are published.


100 images

Hello, loving this program - so useful, thanks for creating it!
I am having trouble setting it up to >100 issues. I installed via CLI and ran setup.py, so am assuming based on your instructions that Selenium is installed. I also followed the geckodriver instructions and that seems to have been successful. I'm running Linux Mint. Any support would be much appreciated!
$ googleimagesdownload -k "Siamese cat, Domestic cat" -l 199 -o "cat images" -f jpg

Item no.: 1 --> Item name = Siamese cat
Evaluating...
Starting Download...

Unfortunately all 199 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Item no.: 2 --> Item name =  Domestic cat
Evaluating...
Starting Download...

Unfortunately all 199 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Everything downloaded!
Total Errors: 0

Total time taken: 0.14542031288146973 Seconds

@freekeys thanks for your feedback :-)
yes, as you might have read in #53 , this is a known issue with Linux OS but works fine on Windows and OS X. I will add this information in the documentation. I will also try implementing the algo using chromedriver instead of geckodriver. But for now this is a won't fix issue.
Closing this issue for now. Feel free to post back if you found a workaround/solution to the geckodriver issue on Linux!

Thanks @hardikvasa!

was working with a sample/dummy code using chromedriver (and chrome browser) to download all the images returned by google search and that seems to work on linux, mac and windows. Let me keep this issue open until i do some more tests and merge it in this repo... :-)

Thanks @jaredjohansen for the feedback.
@jaredjohansen @freekeys the script is now updated to use chromedriver (with chrome browser) instead of geckodriver (with firefox browser).
I have successfully tested this script on Windows, Mac, Linux CentOs and Ubuntu and works perfectly fine. I could download 750+ images per keyword (everything that google returns).
You can follow the one time installation guide of chromedriver and chrome browser in the troubleshooting section of the documentation. Do let me know if this works for you.

@hardikvasa I am trying this but still not working for me- I have chromedriver (v2.9, latest version) in the same directory where I am calling googleimagesdownload. I'm doing
googleimagesdownload --keywords "cute" --limit 200 --chromedriver "./chromedriver"
And then I get the error
Looks like we cannot locate the path the 'chromedriver'. Please use the '--chromedriver' argument to specify the path to the executable.
It get the same error if I call it at a different location-
googleimagesdownload --keywords "cute" --limit 200 --chromedriver "/usr/local/bin/chromedriver"

@andronovhopf what linux OS are you running this from?
this could be due to various reasons:

If the path is correct but the chromedriver file does not have execute permission
can you try with an absolute path /usr/local/bin/chromedriver instead of a relative path? (should not matter but just for testing purposes)
you do not have chrome browser installed in your system. In that case please look into the troubleshooting guide for links to install google chrome stable version

I’m running Ubuntu (xenial).
I’ll check out the permissions on the chrome driver- I got the same error using an absolute path, and I definitely have chrome browser installed on the system.

On Mar 31, 2018, at 1:56 PM, Hardik Vasa<EMAIL_ADDRESS>wrote:
@andronovhopf what linux OS are you running this from?
this could be due to various reasons:
If the path is correct but the chromedriver file does not have execute permission
can you try with an absolute path /usr/local/bin/chromedriver instead of a relative path? (should not matter but just for testing purposes)
you do not have chrome browser installed in your system. In that case please look into the troubleshooting guide for links to install google chrome stable version
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Yep it's in my directory and permissions look good
eobrien@picket:~$ ls -la
total 61016
drwxr-xr-x 61 eobrien eobrien     4096 Mar 31 13:39 .
drwxr-xr-x  4 root    root        4096 Jun 21  2017 ..
-rwxr-xr-x  1 eobrien root    19354520 Dec 16  2013 chromedriver

Tried the following
 sudo googleimagesdownload  --keywords "cute" --limit 200 --chromedriver "/home/eobrien/chromedriver"
and
 googleimagesdownload  --keywords "cute" --limit 200 --chromedriver "./chromedriver"
Failure :/

@andronovhopf thanks for the info. would you mind running the below command again to see if it updates to a stable version?
sudo apt-get install google-chrome-stable
This error could only be thrown if either the chromedriver was not found or the chrome browser was not found. This does not look to me like the issue is with chromedriver.
Did you happen to download the latest version of chromedriver (2.37) ?

@andronovhopf
I did spin up an EC2 instance: ubuntu-xenial-16.04-amd64-server-20180306 (ami-4e79ed36)
And then i followed the exact same steps as mentioned in the accepted answer here to download google-chrome-stable
https://askubuntu.com/questions/510056/how-to-install-google-chrome
and then i used your query and it worked perfectly fine.

Got it! Now it's working. I had to make sure I had chromedriver 2.37, and
then it worked. I must have had a different version.
On Sat, Mar 31, 2018 at 3:59 PM, Hardik Vasa<EMAIL_ADDRESS>wrote:

@andronovhopf https://github.com/andronovhopf
I did spin up an EC2 instance: ubuntu-xenial-16.04-amd64-server-20180306
(ami-4e79ed36)
And then i followed the exact same steps as mentioned in the accepted
answer here to download google-chrome-stable
https://askubuntu.com/questions/510056/how-to-install-google-chrome
and then i used your query and it worked perfectly fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hardikvasa/google-images-download/issues/56#issuecomment-377729400,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcVw9MN1ztbaRH6Az7dcaJm5fBYerFaCks5tkApDgaJpZM4S8Y4Y
.

Sounds good! Closing this issue since this feature has been tested and it works on all major OS.

Just tested it on my machine.  Works for me too!  Thanks hardikvasa!

I have get fine  at first .but get the same error today

Unfortunately all 250 could not be downloaded because some images were not downloadable. 100 is all we got for this search filter!

@xxllp this could be because the qury that you used, Google image search only returned 100 images. Could you search on Google images page directly to see if that's true?


feat(emblem,trading): add url builder for emblem

fix(trading): allow arbitrum assets to have icons in squid
feat(emblem,trading): add url builder for emblem and use it in squid config

Related issues 🔗
Closes #6509
Description ℹ
The experimental Squid cross-chain deposit widget was incorrectly rendering a blank logo for Arbitrum USDT. This was down to a leftover bit of logic that was easy to remove, but also gave us a chance to standardise the logo generation. Now Emblem and squid config use the same function to build image URLs

Add url-builder to emblem
Update squid config to use url-builder
Remove flag that hardcodes arbitrum logos to be blank

Demo 📺

looks good


OpenEXR 3.3.2 crashes when reading tiled files

Has been noticed with GDAL when alpine:edge has upgraded from OpenEXR 3.1.13 to 3.3.2 (https://github.com/OSGeo/gdal/pull/11352)
Can be reproduced with the following sample file temp.exr.zip
and the following procedure:
docker run --rm -it -v $PWD:$PWD alpine:edge
# under Docker
apk add cmake make g++ proj-dev openexr-dev curl
curl https://download.osgeo.org/gdal/3.10.0/gdal-3.10.0.tar.gz --output gdal-3.10.0.tar.gz
tar xvzf gdal-3.10.0.tar.gz
cd gdal-3.10.0
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DGDAL_BUILD_OPTIONAL_DRIVERS=OFF -DOGR_BUILD_OPTIONAL_DRIVERS=OFF -DGDAL_ENABLE_DRIVER_EXR=ON -DACCEPT_MISSING_LINUX_FS_HEADER=ON
make -j$(nproc)
export LD_LIBRARY_PATH=$PWD
export PATH=$PWD/apps:$PATH
apk add valgrind
valgrind gdallocationinfo /path/to/temp.exr 0 0

outputs:
==4165== Memcheck, a memory error detector
==4165== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==4165== Using Valgrind-3.24.0 and LibVEX; rerun with -h for copyright info
==4165== Command: gdallocationinfo /home/even/gdal/gdal/build_ci_alpine/temp.exr 0 0
==4165==
Report:
  Location: (0P,0L)
  Band 1:
==4165== Invalid write of size 8
==4165==    at 0x6B2F833: ??? (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6B2FFEC: ??? (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6B3828A: exr_decoding_run (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x655B75E: ??? (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6558BFF: Imf_3_3::TiledInputFile::readTiles(int, int, int, int, int, int) (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x655B505: Imf_3_3::TiledInputFile::readTile(int, int, int, int) (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x53F7F66: GDALEXRRasterBand::IReadBlock(int, int, void*) (exrdataset.cpp:162)
==4165==    by 0x551A1F7: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:2006)
==4165==    by 0x55FF07D: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (rasterio.cpp:458)
==4165==    by 0x5518A5A: GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (gdalrasterband.cpp:435)
[ ... snip ... ]
==4165==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==4165==
==4165==
==4165== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==4165==  Access not within mapped region at address 0x0
==4165==    at 0x6B2F833: ??? (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6B2FFEC: ??? (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6B3828A: exr_decoding_run (in /usr/lib/libOpenEXRCore-3_3.so.<IP_ADDRESS>)
==4165==    by 0x655B75E: ??? (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x6558BFF: Imf_3_3::TiledInputFile::readTiles(int, int, int, int, int, int) (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x655B505: Imf_3_3::TiledInputFile::readTile(int, int, int, int) (in /usr/lib/libOpenEXR-3_3.so.<IP_ADDRESS>)
==4165==    by 0x53F7F66: GDALEXRRasterBand::IReadBlock(int, int, void*) (exrdataset.cpp:162)
[ ... snip ... ]
==4165==  If you believe this happened as a result of a stack
==4165==  overflow in your program's main thread (unlikely but
==4165==  possible), you can try to increase the size of the
==4165==  main thread stack using the --main-stacksize= flag.
==4165==  The main thread stack size used in this run was 8388608.

The crash also happens without Valgrind.  It occurs around https://github.com/OSGeo/gdal/blob/master/frmts/exr/exrdataset.cpp#L131


No host port mapping after creating container

I noticed when creating a new MySQLContainer like so
protected function setUp(): void
{
      $this->container = MySQLContainer::make();
      $this->container->withMySQLDatabase('pimcore');
      $this->container->withMySQLUser('pimcore_user', 'pimcore');
      try {
          $this->container->run();
      } catch (JsonException $e) {
          $this->fail($e->getMessage());
      }
}

but when checking out the docker container with docker ps and docker inspect there is no port mapped to the host (localhost). And then when I try to connect to it like so
$this->pdo = new PDO(
      sprintf('mysql:host=%s;port=3306', $this->container->getAddress()),
      'pimcore_user',
      'pimcore',
);

host missing when inspecting the container
"Ports": {
      "3306/tcp": null,
      "33060/tcp": null
},

I get this timeout error
1) Tests\App\journey\ContactIntegrationTest::testGetAllAction
PDOException: SQLSTATE[HY000] [2002] Operation timed out

here the container in Docker Desktop

In which environment are you running PHPUnit? We use the internal IP of the container right now, so we don't have to do any port mapping :)

Testcontainer Java uses random ports if it is not set, this way can avoid port conflict.

Do you have an link to the implementation? What do they use as "IP", that would be interesting 🤔, hard-code <IP_ADDRESS> 🤔

I used Testcontainers Java in projects for years, and by default when declaring a container, it will expose all ports, and map to random ports of clients(your machine), it has APIs to get the mapped ports, and first mapping port, etc.
see: https://github.com/hantsy/spring-r2dbc-sample/blob/master/testcontainers/src/test/java/com/example/demo/PostRepositoryWithTestcontainersAnnotationsAndDynamicPropertySourceTest.java#L35


Api users

possibility to get list of users (of the same group)

Do you want be to retrieve the usergroup or you want to specify it in the query? I don't know how you want to user it  @matpuz81  :)

by group_id:
users/group_id=1 gives all users that are in the group with group_id 1


installed_distributions() always uses python 2 executable

When you execute installed_distributions() pip_api performs a python call to list the installed packages.
The used python executable is hard coded to python (if the environment variable PIPAPI_PYTHON_LOCATION is not set). It would be better to check here if the script is executed via python 2 or 3 and then use python respectively python3 depending on the used Python version.
Or alternatively simply use the current python interpreter executable (stored in the variable sys.executable)

Thanks for the report @jpstotz! This is fixed in pip-api==0.0.4.
As an aside, I'm curious what you're using this project for, and how you found it, if you have a second to let me know. Thanks!


IllegalStateException on world creation

Appears to be related to the OpenJ9 JVM?
crash-2020-03-01_03.52.39-server.txt
crash-2020-03-01_04.01.39-server.txt
First is 1.0-Alpha-1, second is breaking/1.0 build 213

openj9 is not supported at all, use hotspot

As said above, we are not testing on OpenJ9 at this time.

just a little doubt: there is 0% probable that OpenJ9 will be supported in the future?

@jdw-mis this issue was fixed by OpenJ9 side, you can get the latests releases here:
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u-2020-12-10-04-05/OpenJDK8U-jdk_x64_windows_openj9_2020-12-10-04-05.zip


Don't attempt to resolve TCP sockets client side

It may be important to have the SSH server perform DNS resolution, as the client often does not use the same DNS server as the server.
We ran into this issue at @MaterializeInc, where we use openssh to set up SSH tunnels via bastion hosts. Our customers often want to establish a tunnel to connect to a host whose name is not resolvable on the client side of the tunnel.

This change is

Codecov Report

Merging #120 (0483d87) into master (a471aa2) will increase coverage by 0.04%.
The diff coverage is 57.14%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files

Impacted Files
Coverage Δ

src/port_forwarding.rs
57.14% <57.14%> (+2.14%)
:arrow_up:

Thanks for the review, @NobodyXu. Sorry for the terse PR description last night; filed in a rush. Added some more context to the description now.

@benesch Please run cargo fmt locally and fix the format.
Thanks!

Whoops, sorry about that! Done.


Update spec for IStio

Based on https://github.com/jhipster/generator-jhipster/pull/8987

Impacts:

istio: should be now true/false since autoinjection will be always available (options no, manualInjection, autoInjection are not needed anymore)
istioRoute: removed

@DanielFran thanks :)

Please update online "deployments" documentation
https://www.jhipster.tech/jdl/deployments


How to pass config when running Docker on Windows

Could you provide an example in the documentation about how you pass a config file to the logstash container when the container is hosted on a windows machine?

I definitely agree that it would be good to add more verbosity to our current docs (https://github.com/docker-library/docs/blob/master/logstash/content.md) -- perhaps an example after that bind-mount that explains using a Dockerfile to COPY and bake a configuration file into the image itself? (which is much more likely to work properly in Windows, and is probably more appropriate for a production deployment as well)


HHH-13764 During enhancement, annotations on the getter are ignored

https://hibernate.atlassian.net/browse/HHH-13764

thanks @DavideD ! just checked it out with @dreab8 ; we just merged it as looks great. funny mistake :)
One question: some comments refer to HHH-13759 , but I take it this fixed only HHH-13764 ?

Also: should we rename the description of HHH-13764 to "During enhancement, annotations on properties are ignored" ?

issue renamed and resolved. Thanks again!


The syntax for mv command under mssparkutils is wrong.

Under the File system utilities, syntax for mv command is mentioned as below.
mv(from: String, to: String, recurse: Boolean = false): Boolean -> Moves a file or directory, possibly across FileSystems
This needs to be fixed as follows,
mv(src: String, dest: String, create_path: Boolean = False, overwrite: Boolean = False): Boolean -> Moves a file or directory, possibly across FileSystems

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c1f7baad-e28d-8e25-2852-b58499cdca8c
Version Independent ID: d88f15f9-7218-3ce2-30bb-388fc63f6ddc
Content: Introduction to Microsoft Spark utilities - Azure Synapse Analytics
Content Source: articles/synapse-analytics/spark/microsoft-spark-utilities.md
Service: synapse-analytics
Sub-service: spark
GitHub Login: @ruixinxu
Microsoft Alias: ruxu

#114356 is raised for this issue

@dheepakg
Thanks for created a PR for this issue once the author will review and approve the changes should go live soon.


Replace regex based system column validation with manual loop

Micro optimization:
Benchmark                                                  (name)  Mode  Cnt     Score    Error  Units
ColumnIdentBenchmark.measureIsSystemColumn                  _name  avgt    6     3.031 ±  0.007  ns/op
ColumnIdentBenchmark.measureIsSystemColumn                 __name  avgt    6     0.991 ±  0.022  ns/op
ColumnIdentBenchmark.measureIsSystemColumn         _system_column  avgt    6     7.275 ±  0.177  ns/op
ColumnIdentBenchmark.measureIsSystemColumn     _no_system_column_  avgt    6     8.005 ±  0.109  ns/op
ColumnIdentBenchmark.measureIsSystemColumnOld               _name  avgt    6    77.019 ±  1.211  ns/op
ColumnIdentBenchmark.measureIsSystemColumnOld              __name  avgt    6    41.822 ±  0.186  ns/op
ColumnIdentBenchmark.measureIsSystemColumnOld      _system_column  avgt    6    82.040 ±  3.363  ns/op
ColumnIdentBenchmark.measureIsSystemColumnOld  _no_system_column_  avgt    6  1351.230 ± 12.000  ns/op

Ran into this while working on https://github.com/crate/crate/pull/14653, where it stood out in a profile:

Benchmark was:
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 2)
@Fork(value = 2)
@Measurement(iterations = 3)
public class ColumnIdentBenchmark {

    private static final Pattern UNDERSCORE_PATTERN = Pattern.compile("^_([a-z][_a-z]*)*[a-z]$");

    @Param({ "_name", "__name", "_system_column", "_no_system_column_" })
    public String name;

    @Benchmark
    public boolean measureIsSystemColumn() {
        return ColumnIdent.isSystemColumn(name);
    }

    @Benchmark
    public boolean measureIsSystemColumnOld() {
        return UNDERSCORE_PATTERN.matcher(name).matches();
    }
}

retest this please
Not sure what's going on, second time this failed today:

/var/lib/jenkins/workspace/CrateDB/crate_on_pr/blackbox/bin/test-decommission: line 24:  3350 Killed                  $DIR/../.venv/bin/python -m unittest -v --failfast test_decommission

> Task :blackbox:gtest FAILED

FAILURE: Build failed with an exception.

* What went wrong:

Execution failed for task ':blackbox:gtest'.

> Process 'command '/var/lib/jenkins/workspace/CrateDB/crate_on_pr/blackbox/bin/test-decommission'' finished with non-zero exit value 137

I'm fairly sure this is unrelated.


not works more for me. you have same problem?

its worked before, but last days i found that code not works as was expected.
get empty response "data".
tried use another settings for
device_id,  fp, iid, openudid
i tried use another server, no success.
please check if your code works?

Hi, same thing here, empty response. If I use a signed url from official android app and make a GET call like /aweme/v1/user/?user_id... via my code then it works. I think they have turned on anti-spam as/mas/cp params.

There are some older version of the app here: https://androidapksfree.com/tik-tok/com-zhiliaoapp-musically/old/
I suspect re-running and capturing traffic patterns from old app might be helpful to troubleshoot

Hi, I have made some additional tests and it seems that device_id and iid parameters are the issue. If you take those two parameters from a real device the API calls will start to work. Can anybody try this this to confirm it?

I took device_id and iid from my Genymotion Android 6.0 emulator. Some API calls can work, such as SearchUsers.
I was not able to get login working, as I haven't implement the anti-spam signing functions yet: https://github.com/szdc/tiktok-api/wiki/Generating-anti-spam-parameters

They have changed a few things since.
As I see they use a secure user id instead of the numeric user id.
Also they added x-khronos and x-gorgon to the header which seerves now as the new antispam parameter. Khronos is just timestamp, but Gorgon is the signature of the url (+ timestamp).

@p4rnicke u figured out how to generate the gorgon token?

Unfortunately not. It is an obfuscated code like previously was. @SebastianWae claimed he did it, but deleted his repo before I could check.
I think it will popup on a chinese page soon.

Oh ok
I'm stuck at leviathan func

Yes, it’s the leviathan...

any updates? it seems impossible to figure out how x-khronos/x-gorgon are generated? We know that the are unique to EVERY request. I can't figure out if x-Khronos/x-gorgon was created specifically by the TikTok app makers or if they are from some public codebase that are used in other applications...

Adding onto @stevensebaweg is there any updates? @p4rnicke

i will pay $1000 for anyone who can make this work. :)

For anyone struggling with generating XGorgon/XKhronos combination:
XGorgon is generated by concatenating four strings:

Hex value of the md5 of all get parameters
X-SS-STUB header value (Which is the hex value of the md5 of the request body)
Hex value of the md5 of the cookie header value
Hex value of the md5 of the session id (which can be found in the cookie header)

If any of the following values don't exist (e.g. session id doesn't exist when you need to register a new device), you simply set the string to a 32 digits of '0'.
Your string should now be 32 * 4 digits long. Set X-Khronos to the current time and pass this value + the string you just generated to the Leviathan encryption algorithm.
This algorithm is extremely complex and can't be easily reverse-engineered. It has lots of security checks to see that your phone isn't rooted, using any emulator, using a proxy/vpn, etc... . It also makes a lot of syscalls which makes it hard to emulate. However, it is possible to reverse with enough determination (Took me around 2 weeks).
The value that is returned by leviathan then just needs to be Hexed, and this is your X-Gorgon value.

@fagenorn how I contact you?  We can offer payment for your services.

Many thanks for @fagenorn , I used dex2jar to get the jar from the apk. Then I noticed I can see the logic for creating XGorgon in the source file using JD-GUI.
I am wondering, apart from reverse engineering leviathan, we might be able to get leviathan working by adopting the approach described in https://github.com/szdc/tiktok-api/wiki/Generating-anti-spam-parameters

@atwleung Definitely possible to emulate a real android device, but do note that the leviathan algorithm checks for many popular emulation techniques (including using a real android device but automating taps and gestures.) and penalizes the user based on this.

https://discord.gg/4fcw8U
We'll let you into the main group after.

@fagenorn, did you know why it doesn't work for every endpoint?
I mean, some endpoints (/aweme/v1/commit/item/digg/,  /aweme/v1/discover/search/...) works flawless, using the leviathan as you described (the only modification is using the "byte value" rather than the hex).
However, some endpoints (/aweme/v1/aweme/post/, /aweme/v1/user/follower/list/) doesn't work, even using the exact same method of the previous one. I notice that the x-gorgon for these endpoints always begins with 03... when I use the mobile app (the TikTok itself). However, calling the leviathan always give me a result starting with 04....
 I could manage to understand some of the parameters, but far from everything, and doesn't work.
Did you have any information about that?

🤫 https://github.com/usualwyy/aweme-algorithm/blob/master/API/units/utils.go

It says that the invite has expired… can you redo?
From: Luke<EMAIL_ADDRESS>Sent: Friday, December 06, 2019 6:30 PM
To: szdc/tiktok-api<EMAIL_ADDRESS>Cc: Steve<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [szdc/tiktok-api] not works more for me. you have same problem? (#120)
https://discord.gg/4fcw8U
We'll let you into the main group after.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/szdc/tiktok-api/issues/120?email_source=notifications&email_token=ADUQAK5L5FHGQKDR5FLO6QDQXLOBDA5CNFSM4JIUO5SKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGFVY2A#issuecomment-562781288 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUQAKYHZYQZEAJHZ7DKLQDQXLOBDANCNFSM4JIUO5SA .

Hello everybody! Who want to sell me solution please contact me by telegram @AlexandrAPI

🤫 https://github.com/usualwyy/aweme-algorithm/blob/master/API/units/utils.go

Didn't work :(

For anyone struggling with generating XGorgon/XKhronos combination:
XGorgon is generated by concatenating four strings:
* Hex value of the md5 of all get parameters

* X-SS-STUB header value (Which is the hex value of the md5 of the request body)

* Hex value of the md5 of the cookie header value

* Hex value of the md5 of the session id (which can be found in the cookie header)

If any of the following values don't exist (e.g. session id doesn't exist when you need to register a new device), you simply set the string to a 32 digits of '0'.
Your string should now be 32 * 4 digits long. Set X-Khronos to the current time and pass this value + the string you just generated to the Leviathan encryption algorithm.
This algorithm is extremely complex and can't be easily reverse-engineered. It has lots of security checks to see that your phone isn't rooted, using any emulator, using a proxy/vpn, etc... . It also makes a lot of syscalls which makes it hard to emulate. However, it is possible to reverse with enough determination (Took me around 2 weeks).
The value that is returned by leviathan then just needs to be Hexed, and this is your X-Gorgon value.

Can we get a better explanation of this, because this is something there can be done..
What do you for example mean by this "X-SS-STUB header value (Which is the hex value of the md5 of the request body)"

Can we get a better explanation of this, because this is something there can be done..
What do you for example mean by this "X-SS-STUB header value (Which is the hex value of the md5 of the request body)"

in fact, it makes no sense to describe how to generate this data. because the most important thing happens in the encryption function to which this data is transmitted and then we get the calculated hash.

So do anyone has a solution ?

shut up jensen
On Sat, Apr 11, 2020, 10:35 Steffan Jensen<EMAIL_ADDRESS>wrote:

So do anyone has a solution ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/szdc/tiktok-api/issues/120#issuecomment-612473143,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2BB3CG2T5PHKLHBIFOT53RMCS6ZANCNFSM4JIUO5SA
.

No just ignore my messages if you don't like them... Looking for a solution for this..

No solution in this repository right now as I haven't had time to update it.

okay great yeah the solution is out there, many has already implemented a solution but they all are "paid solutions"..

hi, where can i buy this paid solution?

hi, where can i buy this paid solution?

Get into contact with me

hi, where can i buy this paid solution?

Get into contact with me

availlable right now for the paid solution

hi, where can i buy this paid solution?

Get into contact with me

availlable right now for the paid solution

pls write to my telegram . I want to buy your solution @leshasotnya

I have contacted you on telegram
On Fri, Jul 24, 2020, 11:52 PM So1omon<EMAIL_ADDRESS>wrote:

hi, where can i buy this paid solution?
Get into contact with me
availlable right now for the paid solution
pls write to my telegram . I want to buy your solution @leshasotnya
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/szdc/tiktok-api/issues/120#issuecomment-663683148,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJMDZSNSXU6NKLEHRKCBMTLR5HJ7LANCNFSM4JIUO5SA
.

I have contacted you on telegram
…
On Fri, Jul 24, 2020, 11:52 PM So1omon @.***> wrote: hi, where can i buy this paid solution? Get into contact with me availlable right now for the paid solution pls write to my telegram . I want to buy your solution @leshasotnya — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#120 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJMDZSNSXU6NKLEHRKCBMTLR5HJ7LANCNFSM4JIUO5SA .

Where? I don't see you in the telegram

I think this your telegram?
On Sat, Jul 25, 2020, 2:15 AM So1omon<EMAIL_ADDRESS>wrote:

I have contacted you on telegram
Where? I don't see you in the telegram
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/szdc/tiktok-api/issues/120#issuecomment-663737392,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJMDZSNOISY7ZIHC54HE2MDR5H2YZANCNFSM4JIUO5SA
.

Can we do it on Skype?
live:ranaabdullah249_1
On Sat, Jul 25, 2020, 2:02 PM Muhammad Abdullah<EMAIL_ADDRESS>wrote:

I think this your telegram?
On Sat, Jul 25, 2020, 2:15 AM So1omon<EMAIL_ADDRESS>wrote:

I have contacted you on telegram
Where? I don't see you in the telegram
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/szdc/tiktok-api/issues/120#issuecomment-663737392,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJMDZSNOISY7ZIHC54HE2MDR5H2YZANCNFSM4JIUO5SA
.

hi, where can i buy this paid solution?

    :D

Paid solution here<EMAIL_ADDRESS>

i have x-gorgon generation script in python. generates x-gorgon starting with 030xxxxxx
contact<EMAIL_ADDRESS>


Chore -> Add CI Pipeline -> Github Actions

I can take this. What we are looking to integrate in the pipeline? @ekas

I can take this. What we are looking to integrate in the pipeline? @ekas

Yes, go on. For now I am looking for adding ESlint and Prettier stages and later on more stages could be added.

Hey, I could do it.

The linter gives me these errors how do i fix them


Sony android 8.0.0 can't not use

Hi Muaz-khan
I try your example app in Sony Xperia XZ F8332.
The Android is 8.0.0.
The Demo Video can't display.

This is the ScreenShot.
And I try the other Sony Xperia XZ1 Compact(The Android is 8.0.0.)
It's the Same problem.
But the HTC M10 (The Android is 8.0.0.)
is working!
Do you have some suggest for me?
Thank you.
Best Regards,
Johnny

Hi there,
I always have best experience using Chrome Beta
Best regards,
viro

Hi @viro24
I know Sony Xperia XZ F8332 can use RTC in the google Chrome.
But I mean use the app.
muaz-khan had made a app on the google play.

You can download and try on your Phone.
*Update
Sony (Android 6.0.0) can use this APP!
best regards
Johnny


Build takes very long time after seed upgrade

After upgrading angular2-seed for my project, I get very slow npm install and builds.
[06:36:46] Requiring external module ts-node/register
[06:37:23] Loading tasks folder C:\project\client\tools\tasks\seed

It takes around 40 seconds between those steps if you look at the time diff.
It fixes when I downgrade ts-node dependency from 0.7.1 to 0.5.5.
I don't see anything suspicious in my setup.
Either it is ts-node itself or the way we use it. Any idea?

It could be ts-node. At this point we don't want to give up static typing so there is nothing much that we can do.
Here's proposal for ts-node 1.0 which will bring a lot of performance improvements https://github.com/TypeStrong/ts-node/issues/44.


[Bug]: Version 2.7.0 extra75 reports default SecurityGroups as unused

What happened?
After upgrading to prowler 2.7.0 existing default SecurityGroups that are not being used are now a failure in extra75 check. Before this version it was only an INFO.
Expected behavior
7.5 [extra75] Ensure there are no Security Groups not being used - ec2 [Informational]
INFO! eu-central-1: sg-XYZ is not being used - default security group
Actual behavior
7.5 [extra75] Ensure there are no Security Groups not being used - ec2 [Informational]
FAIL! eu-central-1: sg-XYZ is not being used!
From where are you running Prowler?
Please, complete the following information:

Resource: EC2 instance
OS: Amazon Linux 2
AWS-CLI Version [aws --version]: 1.22.42
Prowler Version [./prowler -V]: 2.7.0
Shell and version: bash
Others:

I think there is a misunderstanding here, this check has worked that way since it was written. If a SG is not in use it is a FAIL.

OK I found the bug: this commit 2b2814723ffc84495f7af928a5f381bed978dd6d changed the content of the variables, I will create a PR to fix it.


[ISSUE] wrong icon for "themed" smart sockets

First off: I love your tool! If been search for something alike since quite some time, and even configured a workaround solution using the Shortcuts app.
The minor issue I realised is, that sockets, which are configured to be lights (or fans I presume) within the Home app, are still displayed as sockets.

Thank you, your report.
But, right now, I can not find the way to get the event when the icon is changed.
If I can find it, I would like to fix it.

896703ca4a2586798b7707a85fc336df2e941848


Fixed some typos in abort messages

Thanks for contributtion! Can you please sign Scala CLA?

Done!


How to get correct cursor color in Neovide?

Description
I don't think the cursor color is correct In Neovide.
How it looks in Kitty:

How it looks in Neovide:

Neovim version
NVIM v0.10.2
Build type: Release
LuaJIT 2.1.1732813678

Terminal and multiplexer
Neovide NVIM v0.10.2 Build type: Release LuaJIT 2.1.1732813678. No multiuplexer.
Catppuccin version / branch / rev
catppuccin 1.9.0
Steps to reproduce

Use the minimal init.lua
Open Nvim in Neovide

Expected behavior
Peach-colored cursor?
Actual behavior
Black cursor
Repro
-- LazyNvim Setup
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
vim.opt.clipboard = "unnamedplus"
if not (vim.uv or vim.loop).fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "https://github.com/folke/lazy.nvim.git",
    "--branch=stable", -- latest stable release
    lazypath,
  })
end
vim.opt.rtp:prepend(lazypath)
require("lazy").setup({
  {
		"catppuccin/nvim",
		name = "catppuccin",
		priority = 1000,
	},
})

require("catppuccin").setup({
    term_colors = true,
})

vim.cmd.colorscheme "catppuccin-latte"

The cursor in kitty is the one rendered and configured by kitty, which kitty overrides anything related to the terminal cursor. Please check the neovide docs to see if they support customizing cursor color and make issues there.


Some Issues need to be helped！

`#include <stdio.h>
#include <pthread.h>
#include "thpool.h"
int c1 = 0;
int count = 0;
void task1(){
int i;
for (i = 0;i < 1000;i++)
count++;
printf("%d Thread %d working on task1\n",count ,c1++);
}
void task2(){
int i;
for (i = 0;i < 1000;i++)
count++;
printf("%dThread %d working on task2\n",count, c1++);
}
int main(){
puts("Making threadpool with 4 threads");
threadpool thpool = thpool_init(4);

puts("Adding 40 tasks to threadpool");
int i;
for (i=0; i<20; i++){
	thpool_add_work(thpool, (void*)task1, NULL);
	thpool_add_work(thpool, (void*)task2, NULL);
};
thpool_wait(thpool);
puts("Killing threadpool");
thpool_destroy(thpool);

return 0;

}`
the result of this file is variant, I don't know what is going on about this, can someone help me solve this problem? Thanks!

What are the issues? As long as you got the same result in the end, everything is working as expected.

I change the task1() and task2(), the result of the variable count should be  1000*20 = 20000,but the result is less than 20000, and it is variant, I dont know how to make the sub work in the work_queue to work as expected.
the command is: gcc example.c thpool.c -o example.o -lpthread && ./example.o
the result :

Adding 40 tasks to threadpool
1000 Thread 0 working on task1
4000Thread 3 working on task2
5000 Thread 4 working on task1
6000Thread 5 working on task2
7000 Thread 6 working on task1
8000Thread 7 working on task2
9000 Thread 8 working on task1
10000Thread 9 working on task2
11000 Thread 10 working on task1
12000Thread 11 working on task2
13000 Thread 12 working on task1
14000Thread 13 working on task2
2000Thread 1 working on task2
16000Thread 15 working on task2
17000 Thread 16 working on task1
17814Thread 17 working on task2
18355 Thread 18 working on task1
19355Thread 19 working on task2
19819 Thread 20 working on task1
20819Thread 21 working on task2
21376 Thread 22 working on task1
22376Thread 23 working on task2
22880 Thread 24 working on task1
24199 Thread 25 working on task1
25235Thread 27 working on task2
26235 Thread 28 working on task1
27235Thread 29 working on task2
28235 Thread 30 working on task1
29235Thread 31 working on task2
30235 Thread 32 working on task1
31235Thread 33 working on task2
32235 Thread 34 working on task1
33235Thread 35 working on task2
34235 Thread 36 working on task1
35235Thread 37 working on task2
36235 Thread 38 working on task1
37235Thread 39 working on task2
15000 Thread 14 working on task1
24235Thread 26 working on task2
3000 Thread 2 working on task1
Killing threadpool

You're not using the threadpool in a thread-safe way. The c1 and count are shared by both threads so ofcourse the data will be overriden randomly by any thread that first gets access to it.
So you need to figure out on your own how to make the solution thread-safe (either with atomic variables, semaphores or whatever). The threadpool is working properly.


nest: env credentials not working

if env nest credentials are present then json file should not be read.

root cause was old shell opened before env variables were established, this is not a code error, no changes needed other than re-starting the shell


fix footer ui

Details:

Centered Icon and text
Centered links

Before:

After:

probably dont wanna change yarn.lock?

Should i pull this branch and remove the yarn.lock file

lol,I will make the changes with the updated yarn lock file presented on the main branch.

@Fryingpannn I have added the yarn.lock as per the main branch. Let me know your thoughts

seems like there's some conflicts due to changes I pushed for privacy policy. I think the privacy policy and ToS links should be below the social media icons actually. Would be great if u could make that change in order to resolve the conflict.


Please add additional ExchangeInfo filters for market orders to work

I am experiencing a lot of min notional errors with market orders because Binance is now using a filter called avgPricemins along with applyToMarket. Since market orders do not have a price they are using these filters to determine the min notional calculation.
See: https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#min_notional
There are other filters added as of December. See: https://github.com/binance-exchange/binance-official-api-docs/blob/master/CHANGELOG.md
Can we get these filters added please? Great product and thank you!!!!

For now, please use: IBinanceApi.GetAvgPriceAsync() and Symbol.NotionalMinimumValue.


ask for help about low performence in 2021 branch

hello, i'm using the checkpoint your provided to evaluate in the town05, but the performence is not good.

the first is the result of Town05_short, and the other is Town05_long
and the first number is avg, the second number is std
i noticed that the agent has a high probability of blocked.
and here is the config:
`#!/bin/bash
export CARLA_ROOT=~/carla
export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:leaderboard
export PYTHONPATH=$PYTHONPATH:leaderboard/team_code
export PYTHONPATH=$PYTHONPATH:scenario_runner
export LEADERBOARD_ROOT=leaderboard
export CHALLENGE_TRACK_CODENAME=SENSORS
export PORT=2100 # same as the carla server port
export TM_PORT=8100 # port for traffic manager, required when spawning multiple servers/clients
export DEBUG_CHALLENGE=0
export REPETITIONS=3 # multiple evaluation runs
export ROUTES=leaderboard/data/validation_routes/routes_town05_short.xml
export TEAM_AGENT=leaderboard/team_code/transfuser_agent.py # agent
export TEAM_CONFIG=models/transfuser # model checkpoint, not required for expert
export CHECKPOINT_ENDPOINT=results/routes_town05_short_transfuser_evalution.json # results file
export SCENARIOS=leaderboard/data/scenarios/town05_all_scenarios.json
export SAVE_PATH=/media/djy/H/CARLA_DATA/transfuser # path for saving episodes while evaluating
python3 ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py
--scenarios=${SCENARIOS}
--routes=${ROUTES}
--repetitions=${REPETITIONS}
--track=${CHALLENGE_TRACK_CODENAME}
--checkpoint=${CHECKPOINT_ENDPOINT}
--agent=${TEAM_AGENT}
--agent-config=${TEAM_CONFIG}
--debug=${DEBUG_CHALLENGE}
--record=${RECORD_PATH}
--resume=${RESUME}
--port=${PORT}
--trafficManagerPort=${TM_PORT}`
and i run it using single 3080ti gpu
can you give me some suggestions?
Thanks for your attention

The transfuser checkpoint in the 2021 branch is different than the one used for results in the paper. We later updated the checkpoint with the model that was submitted to the leaderboard and that was trained in a different setting than what is mentioned in the paper. I'd suggest shifting to the 2022 branch since it contains all the latest models, checkpoints and contains fixes for several bugs in the 2021 branch.

Thanks for your reply, i will shift to the 2022 branch


Update NDK to 24.0.7856742

Once #423 is implemented, we can update to NDK 23.
Before updating the NDK version, be aware of information in the https://github.com/zcash/zcash-android-wallet-sdk/issues/539.

Before updating the NDK version, be aware of information in the https://github.com/zcash/zcash-android-wallet-sdk/issues/539.


HA restart doesn't update state

Hello there, I was looking if there is any possible way to update the state once home assistant restarts. Right now, it's always set to OFF after a restart even if the LED strip is ON.
Thanks!

What model of Govee device do you own? Some do not support state.


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v4.4.273",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=958cb1078ca60d214826fd90a0961a447fade59a"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "958cb1078ca60d214826fd90a0961a447fade59a",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "isdn: mISDN: netjet: Fix crash in nj_probe:\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.4.273 by commit 958cb1078ca60d214826fd90a0961a447fade59a. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1000827


Rive editor just randomly crashes

My whole Rive project is gone.  I've been working on a Rive project recently and suddenly the app crashes.  I reload the page and click on the project thumbnail in Drafts and nothing shows up.  Any ideas?  This is pretty bad.

I'm able to recover with minimal loss from the revision history!


Automated live deployment + webpack config

@simondevyoutube It might be worth setting up a basic auto-deploy boilerplate for your frontend-only projects.
I put together a little POC for the hell of it here.
It works as follows:

When you push to the main branch on github, a job is triggered (configured by added a yaml file to .github/workflows/ to any github repo). This is an example of one of those jobs
This jobs bundles up the code and sticks it into a gh-pages branch
It's then automatically available for public consumption at the gh-pages url for the repo

Generally speaking, working with all these build tools can be rough though so there is likely a simpler way to do this as well (like just host gh-pages from the main branch as is). However, tools like webpack can be useful if understood and the work is transferable to all the other repos.
Expanding on this could allow for having things like shared libraries and code between all the different projects via npm packages.

Hey, sorry I've been sitting on this. Love the idea, gimme a few days for me to get some time to actually poke through the CL and understand it.

@simondevyoutube  No problem at all. I usually work with React and typescript using create-react-app which does most of this for me. So, this PR specifically isn't exactly best practice or anything.
Mostly just pitching the idea that you can set up your repos to also host up the code. The github action which auto-deploys it when you commit is just a nice extra.

The 13k lines is annoying to look at but that's npm in a nutshell....


Dataflow fails on Transform component in "TFX_Pipeline_for_Bert_Preprocessing"

@hanneshapke
I have tried out TFX_Pipeline_for_Bert_Preprocessing on GCP AI Platform Pipeline while Dataflow option turned on.
However, it failed, and I got some error messages.
tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed

any thoughts?

@deep-diver
Hard to say what the issue is without seeing more details. Which component is failing? Can you maybe paste a longer error message, please?

@hanneshapke
The failure happens during TFX Transform component.
Here is the full message
2021-02-18 09:13:43.360 JSTAn exception was raised when trying to execute the workitem<PHONE_NUMBER>081758712 : Traceback (most recent call last): File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/core.py", line 1616, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 782, in _infer_metadata_from_saved_model return _infer_metadata_from_saved_model_v1(saved_model_dir) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 793, in _infer_metadata_from_saved_model_v1 saved_model_dir, {})) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 426, in partially_apply_saved_transform_internal saved_model_dir, logical_input_map, tensor_replacement_map) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 271, in _partially_apply_saved_transform_impl input_map=input_map) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1462, in import_meta_graph **kwargs)[0] File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1486, in _import_meta_graph_with_return_elements **kwargs)) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/meta_graph.py", line 799, in import_scoped_meta_graph_with_return_elements return_elements=return_elements) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal graph._c_graph, serialized, options) # pylint: disable=protected-access tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 649, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start File "apache_beam/runners/worker/operations.py", line 356, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 703, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 704, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1215, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback raise exc.with_traceback(traceback) File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/core.py", line 1616, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 782, in _infer_metadata_from_saved_model return _infer_metadata_from_saved_model_v1(saved_model_dir) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 793, in _infer_metadata_from_saved_model_v1 saved_model_dir, {})) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 426, in partially_apply_saved_transform_internal saved_model_dir, logical_input_map, tensor_replacement_map) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 271, in _partially_apply_saved_transform_impl input_map=input_map) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1462, in import_meta_graph **kwargs)[0] File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1486, in _import_meta_graph_with_return_elements **kwargs)) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/meta_graph.py", line 799, in import_scoped_meta_graph_with_return_elements return_elements=return_elements) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal graph._c_graph, serialized, options) # pylint: disable=protected-access RuntimeError: tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. [while running 'Analyze/ComputeDeferredMetadata[compat_v1=True]']

In case it is hard to read, here is the screenshot.


Port forwarding not working anymore since update (remote dev)

Type: Bug

Launch a bokeh server or any other process that open a local port
port isn't automatically fowarded anymore.
Browser window/tab isn't launched either.

I've tried changing the port forwarding setting to hybrid, output and process mode with no luck.
Fowarding the port manually then going to localhost:port still work.
Downgrading to previous version resolve the issue.
Please advise.
 SSH? Dev containers?

Sorry for the late reply.
I'm using dev containers. In fact I have about 40 users using remote container dev and we are all experiencing this issue with v1.92.0

Issue is still happening with latest version.
Switching to process mode, and deactivating the auto forward ports fallback resolve the issue. Openpreview/browser doesn't work automatically anymore though.
Problem seems to be with the ouput mode from what i see.
Thanks

@JulienLrx can you share an example of the output that is printed when you execute the program that starts your server?

sure here it is :
Launching server at http://localhost:38181
But your question made me try something :
Problem seems to be related to notebook output, if i export the script as a python script output is correctly detected and portfowarding is working correctly.
Hope that helps.

But your question mades me try something as we mainly use jupyter notebooks : if i export the notebook as a python script and execute it, output is correctly detected and portforwarding is then working correctly.

Thanks @JulienLrx that does explain it. Detecting ports in the output of notebooks isn't something we support.

Ok noted thanks for the answers.
But something has changed with hybrid mode at least. Before (and going back to june release does the same things) It uses to work (auto forward the port + open the browser windows automatically) and now we have to force process mode and set to 0 auto forward ports fallback and even then we have to open browser manually.

I don't think anything relevant in the url finder has changed in years. Launching the browser isn't default behavior when a port is discovered. Maybe the configuration of your project changed:

you could have had a devcontainer.json file that caused the port to be forwarded, rather than waiting for auto port detection to find it
the devcontainer.json could also set the port attributes such that the browser is launched when the port is detected
you could have had remote.portsAttributes or remote.otherPortsAttributes in your settings to cause the browser to be launched


Modify the encoding of AssemblyInfo.cpp

In some language such as Chinese,  the diacritics of comment in file may cause compile warning, which will cause compile error because of the project setting.

Why this pull request is closed with force-pushed, without code merged nor rejected?
Perhaps its a miss-operation?

Sorry I though I'd merged your PR already. I'll reopen it and try to process it tonight.

@xiaoqingxing perhaps you can rebase your commit on top of a clean git tree? Should make this merging easier.

the commit has been cherry-picked as c166e4284fe483846b63e8cb1a342361b454bc83 in the master, closing this PR


Translations update from Weblate

Translations update from Weblate for Wavelog/Main Translation.
It also includes following components:

Wavelog/Installer

Current translation status:

here will be some more changes in the code. keep this in hold


Can't load eslint configs after<EMAIL_ADDRESS>

When using eslint configs e.g.
extends: [
        'eslint:recommended',
]

The configs cannot be found when using<EMAIL_ADDRESS>or later. This PR moved their location so that resolveESLintIntrinsicConfigPath can no longer resolve them.

Thanks to point PRs .

Thanks! Very quick work! @Quramy

Your info helps me so much because I did not know why renovate PR is failed.
Thanks.


Notification: Fix SMTP error handling to avoid useless sending tries

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: jerome-benoit:x: gochaxYou have signed the CLA already but the status is still pending? Let us recheck it.

@jerome-benoit
After the front-end changes I decided that the impact of a simple variable renaming is too vast.
Could you please revert this change?
Thanks.

@jerome-benoit
After the front-end changes I decided that the impact of a simple variable renaming is too vast.
Could you please revert this change?
Thanks.

I've by mistake stacked PRs on ev-dashboard, so if you merge https://github.com/LucasBrazi06/ev-dashboard/pull/2117, that one become more readable. So will become https://github.com/LucasBrazi06/ev-dashboard/pull/2145 after the merge of that one.
Variables, classes, methods, etc. rename is ev-dashboard are harmless. So they are on ev-server. We do that all the times. The rename done here were done automatically by code and just works.
The only rename that have an impact is the one done on the User type because it's a REST API change. And reverting it just change 3 lines in the current PR and make the code less readable.
Reverting it on ev-server change also a few lines and for safety, I can add both fields to the REST API endpoints.
Since there's no hurry on adding SMTP error handling to trigger correctly SMTP backup use, that PRs can be merged only when a full test plan of that feature can be scheduled with Willy first.

I've by mistake stacked PRs on ev-dashboard, so if you merge #2117, that one become more readable. So will become #2145 after the merge of that one.
Variables, classes, methods, etc. rename is ev-dashboard are harmless. So they are on ev-server. We do that all the times. The rename done here were done automatically by code and just works.
The only rename that have an impact is the one done on the User type because it's a REST API change. And reverting it just change 3 lines in the current PR and make the code less readable.
Reverting it on ev-server change also a few lines and for safety, I can add both fields to the REST API endpoints.
Since there's no hurry on adding SMTP error handling to trigger correctly SMTP backup use, that PRs can be merged only when a full test plan of that feature can be scheduled with Willy first.

Ok then you take full responsibility of this choice and will report directly to the boss in case of failure.