No sentinel server available for autodiscovery.

Hi,

Using laravel 5.3, when the replication option is passed to the redis server it gives the above error. If I do not pass this value, I get the message "READONLY: You cannot write against a read only slave".

On the host file if I use the port 26379, it is unable to execute commands SET or GET command through that server.

Also, noticed in the test cases, there is a stub for configuration file that is using port 6379 but in the documentation it is mentioned to use the 26379, which one to use for.

Trying to troubleshoot this issue from past few hours but no avail.
Will appreciate any help.
Thanks

Hi @azhard4int, Sentinel servers typically listen on port 26379 (I'll update tests/stubs/config.php to reflect this, thanks!). Check the config file used to start your Sentinel processes to confirm the port number.
Try running redis-cli -h your-sentinel-host -p 26379 ping to verify that Sentinel is configured properly and accepting connections on that port. If you don't see PONG, there's a problem with the Sentinel server's configuration or connectivity.
The error message in the title indicates that the redis-sentinel driver in config/database.php doesn't contain a connection with a host.  I can try to offer more useful help if you provide that part of the config file.

@cyrossignol  For the ping response:
> redis-cli -h 45.76.XX.XX -p 26379 ping
> PONG

Here is the config file
<?php

return [
    'fetch' => PDO::FETCH_OBJ,
    'default' => env('DB_CONNECTION', 'mongodb'),
    'connections' => [

        'sqlite' => [
            'driver' => 'sqlite',
            'database' => env('DB_DATABASE', database_path('database.sqlite')),
            'prefix' => '',
        ],

        'mysql' => [
            'driver' => 'mysql',
            'host' => env('DB_HOST', 'localhost'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'forge'),
            'username' => env('DB_USERNAME', 'forge'),
            'password' => env('DB_PASSWORD', ''),
            'charset' => 'utf8',
            'collation' => 'utf8_unicode_ci',
            'prefix' => '',
            'strict' => true,
            'engine' => null,
        ],

        'pgsql' => [
            'driver' => 'pgsql',
            'host' => env('DB_HOST', 'localhost'),
            'port' => env('DB_PORT', '5432'),
            'database' => env('DB_DATABASE', 'forge'),
            'username' => env('DB_USERNAME', 'forge'),
            'password' => env('DB_PASSWORD', ''),
            'charset' => 'utf8',
            'prefix' => '',
            'schema' => 'public',
            'sslmode' => 'prefer',
        ],
        'default' => [
                'host' => env('REDIS_HOST',  'tcp://<IP_ADDRESS>'),
                'password' => env('REDIS_PASSWORD', null),
                'port' => env('REDIS_PORT', 6379),
                'database' => 0,
        ],
    ],

    'migrations' => 'migrations',

    'redis' => [

        'cluster' => false,

        'default' => [
            'host' => env('REDIS_HOST', 'localhost'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => 0,
        ],
        'options' => [
            'parameters' => ['password' => env('REDIS_PASSWORD', null)],
        ],

    ],

    'redis-sentinel' => [

        'default' => [
                [
                    'host' => '45.76.XX.XX',
                    'port' => 26379
                ],
                [
                    'host' => '45.32.XX.XX',
                    'port' => 26379
                ],
                [
                    'host' => '45.32.XX.XX',
                    'port' => 26379
                ]
            ],

            'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => 0,
            ],
        ],

],

];

Thanks, @azhard4int. Your configuration looks fine. How are you executing the Redis GET and SET commands? I can't tell for sure, but it seems like the app is using the redis connection instead of the redis-sentinel connection.
If you're using the Redis facade, you'll need to set 'redis' => [ ... 'driver' => 'sentinel', ... ] in config/database.php so the facade uses the Sentinel connection (see overriding the standard Redis API).
You can also use the RedisSentinel facade if you don't want to override Laravel's standard Redis API.

@cyrossignol  thank you for the detailed response.
It was indeed the problem with the network connection, I have increased the sentinel timeout and it works fine now.
One more thing I have noticed is, I have redis-sentinel in the connections array for queue.php, but it picks the default non-sentinel redis drivers.
Below is the connections array:
'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => 'localhost',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => 'your-public-key',
            'secret' => 'your-secret-key',
            'prefix' => 'https://sqs.us-east-1.amazonaws.com/your-account-id',
            'queue' => 'your-queue-name',
            'region' => 'us-east-1',
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 90,
        ],
        'redis-sentinel' => [
            'driver' => 'redis-sentinel',
            'connection' => 'default',
            'queue' => 'default',
            'expire' => 60,
        ],

    ],

And, If I change the driver for the redis key driver value to redis-sentinel it works with the redis sentinel.
Should not it be picking the redis-sentinel key instead of the redis key?

I'm glad you solved the connection issue. To answer your next question:
The config/queue.php file usually contains a 'default' element near the top that tells Laravel which queue connection to use:
'default' => env('QUEUE_DRIVER', 'sync'),

Check the value of QUEUE_DRIVER in .env. You may need to change it from redis to redis-sentinel.

@cyrossignol  I have redis-sentinel in the .env for QUEUE_DRIVER. Also, I have tried changing the  'default'=> env('QUEUE_DRIVER', 'sync'), value directly to the 'default'=>'redis-sentinel' but it is still picking default redis connection.

If you're setting the literal value of 'default' to 'redis-sentinel', then something else must be changing it down the line. You might try looking for any Config::set('queue.default', 'redis') or Queue::setDefaultDriver('redis') calls. What does Queue::getDefaultDriver() show right before your queue logic runs?

@cyrossignol it was onConnection('redis') that was causing this issue.
Thanks a lot man, really appreciate it.

I have the same problem. It happens only in queued jobs. What could be the reason ?

Hi @bdteo,
Do you see the error for every queued job? Or only some of the queued jobs? We can debug the state of the Sentinel connection by adding this temporary code to the boot() method of a service provider like AppServiceProvider:
\Queue::before(function ($event) {
    $lines = [ 'Connection: ' . $event->connectionName ];
    $dumper = new \Symfony\Component\VarDumper\Dumper\CliDumper();
    $dumper->setOutput(function ($line) use (&$lines) {
        $lines[] = $line;
    });

    $dumper->dump((new \Symfony\Component\VarDumper\Cloner\VarCloner())
        ->cloneVar($event->job->getRedisQueue()->getConnection()->client()->getConnection());

    \Log::debug(implode(PHP_EOL, $lines));
});

This will write the state of the Sentinel connection to storage/logs/laravel.log before each job runs so we can verify that it contains the Sentinel servers defined in the REDIS_HOST environment variable. Then, queue up a job for processing and check the output in the log. If you're not sure how to read the output, please post it back here.

Ok, thank you very much for the helpful suggestion.
It seems that for my composer setup  the getConnection method of RedisQueue is protected so this is how I used your code ....
 Queue::before(function ($event) {
            $lines = [ 'Connection: ' . $event->connectionName ];
            $dumper = new \Symfony\Component\VarDumper\Dumper\CliDumper();
            $dumper->setOutput(function ($line) use (&$lines) {
                $lines[] = $line;
            });

            $connectionObjOfInterest = $event->job->getRedisQueue();

            $reflector = new ReflectionObject($connectionObjOfInterest);
            $method = $reflector->getMethod('getConnection');
            $method->setAccessible(true);

            $dumper->dump((new \Symfony\Component\VarDumper\Cloner\VarCloner())
                ->cloneVar($method->invoke($connectionObjOfInterest)->client()->getConnection()));

            Log::debug(implode(PHP_EOL, $lines));
        });

Here is the result ....
call_stack: NULL
  instance: xxxxxxxxxxxx
   channel: pre-production
ip_address: <IP_ADDRESS>
     level: DEBUG
   message: Connection: redis-sentinel
Predis\Connection\Aggregate\SentinelReplication {#1063
#master: Predis\Connection\StreamConnection {#1069
-resource: stream resource {@566
timed_out: false
blocked: true
eof: false
stream_type: "tcp_socket/ssl"
mode: "r+"
unread_bytes: 0
seekable: false
options: []
}
-cachedId: null
#parameters: Predis\Connection\Parameters {#1068
-parameters: array:6 [
"host" => "<IP_ADDRESS>"
"port" => "6379"
"alias" => "master"
"password" => "xxxxxxxxxxxx"
"database" => 0
"scheme" => "tcp"
]
}
#initCommands: array:2 [
0 => Predis\Command\RawCommand {#1070
-slot: null
-commandID: "AUTH"
-arguments: array:1 [
0 => "xxxxxxxxxxxx"
]
}
1 => Predis\Command\RawCommand {#1071
-slot: null
-commandID: "SELECT"
-arguments: array:1 [
0 => 0
]
}
]
}
#slaves: []
#current: Predis\Connection\StreamConnection {#1069}
#service: "servicexxxx"
#connectionFactory: Predis\Connection\Factory {#1065
-defaults: array:2 [
"password" => "xxxxxxxxxxxx"
"database" => 0
]
#schemes: array:6 [
"tcp" => "Predis\Connection\StreamConnection"
"unix" => "Predis\Connection\StreamConnection"
"tls" => "Predis\Connection\StreamConnection"
"redis" => "Predis\Connection\StreamConnection"
"rediss" => "Predis\Connection\StreamConnection"
"http" => "Predis\Connection\WebdisConnection"
]
}
#strategy: Predis\Replication\ReplicationStrategy {#1064
#disallowed: array:11 [
...
"INFO" => true
...
"SLOWLOG" => true
]
#readonly: array:62 [
"EXISTS" => true
....
"ECHO" => true
...
"PFCOUNT" => true
"SORT" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isSortReadOnly"
]
"BITFIELD" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isBitfieldReadOnly"
]
"GEOHASH" => true
"GEOPOS" => true
"GEODIST" => true
"GEORADIUS" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isGeoradiusReadOnly"
]
"GEORADIUSBYMEMBER" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isGeoradiusReadOnly"
]
]
#readonlySHA1: []
}
#sentinels: array:3 [
0 => array:6 [
"host" => "<IP_ADDRESS>"
"port" => "26379"
"database" => null
"password" => null
"timeout" => 0.5
"scheme" => "tcp"
]
1 => array:2 [
"host" => "<IP_ADDRESS>"
"port" => "26379"
]
2 => array:2 [
"host" => "<IP_ADDRESS>"
"port" => "26379"
]
]
#sentinelConnection: Predis\Connection\StreamConnection {#1067
-resource: stream resource {@565
timed_out: false
blocked: true
eof: false
stream_type: "tcp_socket/ssl"
mode: "r+"
unread_bytes: 0
seekable: false
options: []
}
-cachedId: null
#parameters: Predis\Connection\Parameters {#1062
-parameters: array:6 [
"host" => "<IP_ADDRESS>"
"port" => "26379"
"database" => null
"password" => null
"timeout" => 0.5
"scheme" => "tcp"
]
}
#initCommands: []
}
#sentinelTimeout: 0.5
#retryLimit: 20
#retryWait: 1000
#updateSentinels: true
}

   context: []
created_at: 2018-12-20 20:44:45
1 row in set (0.00 sec)

Everything seems to be ok.
I spent almost my whole day reading through the source code of predis, monospice/laravel-redis-sentinel-drivers and laravel.
I came to the suspicion that running the predis client in a Laravel queue worker context may trigger a dangerous bug.  I don't know whether I am right or wrong as of this moment but here is what I mean ....
I found that once I restart my artisan queue:work supervisord workers on all servers then everything seems to be ok (until I deploy my production images again).
Basically said my theory is that Laravel preserves single Predis\Connection\Aggregate\SentinelReplication object for the whole runtime of a queue worker (due to the way Laravel's service container works and the way Laravel workers work).
So ... Single Predis\Connection\Aggregate\SentinelReplication object -  single instance of its property
$sentinels
    /**
     * @var NodeConnectionInterface[]
     */
    protected $sentinels = array();

I suppose this is the culprit
/**
     * Returns the current sentinel connection.
     *
     * If there is no active sentinel connection, a new connection is created.
     *
     * @return NodeConnectionInterface
     */
    public function getSentinelConnection()
    {
        if (!$this->sentinelConnection) {
            if (!$this->sentinels) {
                throw new \Predis\ClientException('No sentinel server available for autodiscovery.');
            }

            $sentinel = array_shift($this->sentinels);
            $this->sentinelConnection = $this->createSentinelConnection($sentinel);
        }

        return $this->sentinelConnection;
    }

When a connection fails then this happens: $this->sentinelConnection = null; .
As far as I see this means that if it happens that all of the connections fail at the same time then $sentinels becomes an empty array for the rest ot the runtime (in a Laravel context).
I am very tired to check this now, so excuse me for my lowered abilities to write reasonable senteces :D I will write tomorrow.
Do you think that this might be the problem ? (I cannot confirm it until I produce the bug forcefully.)

@bdteo Thanks for providing very detailed information. Your configuration appears fine. It looks like you found a bug in Predis, and you correctly identified the culprit:
public function getSentinelConnection()
{
    ...
    $sentinel = array_shift($this->sentinels);  // drops the connection forever
    ...
}

We won't likely see this problem during the short web-request life-cycle because the set of Sentinel connections re-initializes with each request. However, it's just like you said: in a long-running PHP process—like a Laravel queue worker—Predis can eventually drain the set of available Sentinel connections after each connection times-out at some point during the lifetime of the process.
Unfortunately, it doesn't seem like Predis receives regular maintenance right now. I'll start looking into a workaround for this package. In the meantime, try increasing the sentinel_timeout value for the queue service by adding a dedicated connection in config/database.php:
'redis-sentinel' => [
    ...
    'queue' => [
        ...
        'options' => [
            'sentinel_timeout' => 5.0,
        ],
    ],
],

...and change the 'connection' to 'queue' for 'redis-sentinel' in the 'connections' array config/queue.php.
In many cases, a large timeout value here shouldn't significantly impact the response time of the application because the queue workers run in the background. It looks like you've already set update_sentinels to true. This improves the chance that the application will receive a new set of Sentinel connections when Predis drops one. You can also schedule a job to restart the queue workers periodically, or set a custom queue exception handler that restarts workers after catching Predis\ClientException.
In a well-designed Sentinel deployment, a major outage in which every Sentinel becomes unreachable by a client should never happen (if it does, bigger problems exist). When Predis loses all of its Sentinel connections, the config likely contains timeout value too low for a given network. Depending on the application's response-time requirement, we can either increase the timeout value or fix the networking issues that cause this latency. That said, this package should handle brief periods of intermittent connectivity. I'll start working on a solution as soon as possible.

Thank you for the detailed information that you provided.
It will be helpful for sure in future scenarios.
I agree about Sentinels' deployment and the best practices. However this is not always the case.
Often there is a single point of failure - a human being who controls the Redis setup.
This is entirely up to company structure, dev ops and so on ...
So a bug is a bug. It is cool that we can improve the predis project.
I don't have so much knowledge about its general architecture, so can you please submit the bug to their repository :) ?

I got this similar error not able to solve it. I did try to ping all my sentinel server and it is working. below is my configuration:

Laravel version 5.1

"monospice/laravel-redis-sentinel-drivers": "^1.0"

"predis/predis": "1.1"

PHP version 7

I used env file configuration, below is my env file configuration:
CACHE_DRIVER=redis-sentinel
SESSION_DRIVER=redis-sentinel
QUEUE_DRIVER=redis-sentinel
REDIS_DRIVER=redis-sentinel
REDIS_HOST=XX.XXX.XXX.XX, XX.XXX.XXX.XX, XX.XXX.XXX.XX
REDIS_PORT=26379
REDIS_SENTINEL_DISCOVERY=true
REDIS_SENTINEL_SERVICE=mymaster
REDIS_SENTINEL_PASSWORD="xxxxxxxxxx"
REDIS_SENTINEL_TIMEOUT=0.100
May I know what wrong with my configuration?

Hi @pangminfu,
Try increasing REDIS_SENTINEL_TIMEOUT. Predis sets the default timeout value to 0.100 seconds, which may be too short in some environments.
As described in this ticket, we tend to see the "No sentinel server available for autodiscovery" error on slow networks. This might happen when Sentinel servers don't run on the same host or physical network as the application (like in some cloud hosting environments).
Otherwise, your configuration looks fine. One minor suggestion: we can change REDIS_SENTINEL_PASSWORD to REDIS_PASSWORD because it doesn't look like you're using both Sentinel and Laravel's standard Redis API at the same time. In this case, it works the same either way.


staslib: Fix pyudev deadlock waiting for events forever loop.

Signed-off-by: Martin Belanger<EMAIL_ADDRESS>

Thanks!


Transponder starts on test mode when started cold and dark

On the latest version of this project, the transponder starts on "test" mode, not on "stand by" mode, when the aircraft is started cold and dark.
Cheers and thanks for the project!

Just corrected that, thanks. The only file changed was apron.FLT.


[Fix #1899] Mind comments when correcting braced hash

Make sure we don't move a closing parenthesis up to a comment line where it would become part of the comment.

:+1: Looks good!


Help with r2-presigned-url-worker example.

The r2-presigned-url-worker example looks like exactly what I need (thanks!) to implement a worker that retains the existing presigned URL authorization functionality on a private bucket. But I am unable to run it locally via deno serve or publish it example to CloudFlare.
Steps:

Install deno via brew install deno per docs
Install denoflare via deno install --unstable --allow-read --allow-net --allow-env --allow-run --name denoflare --force https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli.ts per docs

I also tried installing https://raw.githubusercontent.com/skymethod/denoflare/master/cli/cli.ts

Clone this repo via git clone https://github.com/skymethod/denoflare.git and cd to denoflare/examples/r2-presigned-url-worker
Create a .denoflare file per docs, but with "path": "worker.ts"  instead of "path": "index.ts", and provide accountId and apiToken
Run denoflare serve hello-local

I get this error:
$ denoflare serve hello-local
Compiling https://raw.githubusercontent.com/skymethod/denoflare/31db528115dc6260ac09ba2abd6c9a5aa8709b8a/cli-webworker/worker.ts into worker contents...
✅ Granted write access to "/var/folders/2j/ymfzssrx66gg87hypqbw1ptr0000gn/T/".
Bundled https://raw.githubusercontent.com/skymethod/denoflare/31db528115dc6260ac09ba2abd6c9a5aa8709b8a/cli-webworker/worker.ts (process) in 15413ms
runScript: worker.ts
Bundled worker.ts (process) in 547ms
worker: start
Error running script TypeError: Cannot set property caches of #<DedicatedWorkerGlobalScope> which has only a getter
    at defineGlobalCaches (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:507:33)
    at defineModuleGlobals (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:490:5)
    at Function.create (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:557:9)
    at Function.start (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:674:78)
    at blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:2561:44
    at RpcChannel.receiveMessage (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:39:42)
    at self.onmessage (blob:null/e6ae24c2-3357-4eff-bd5d-36a8a43a5ebd:2589:30)
    at wrappedHandler (deno:ext/web/02_event.js:1362:14)
    at innerInvokeEventListeners (deno:ext/web/02_event.js:755:9)
    at invokeEventListeners (deno:ext/web/02_event.js:795:7)

I also tried to take an existing worker (kian's s3-to-r2-migration which is already working (but allows public access) and import the r2-presigned-url-worker example so that I could first execute your computeResponse() and fallback to kian's code:
import r2PresignedUrlWorker from "https://raw.githubusercontent.com/skymethod/denoflare/master/examples/r2-presigned-url-worker/worker.ts";

...

export default {
    async fetch(request: Request, env: Env, ctx: EventContext<any, any, any>): Promise<Response> {
        const response = await r2PresignedUrlWorker.fetch(request, env);
        if (response.status == 200) {
            return response;
        }

        ... fallback to kian's code

But when I try to publish this:
$ wrangler publish
 ⛅ wrangler 2.1.10
--------------------
Your worker has access to the following bindings:
- R2 Buckets:
  - R2: ...
- Vars:
  - AWS_DEFAULT_REGION: "auto"
  - AWS_S3_BUCKET: "..."
  - AWS_S3_BUCKET_SCHEME: "https:"
  - AWS_SERVICE: "s3"
Total Upload: 15.42 KiB / gzip: 4.56 KiB

✘ [ERROR] A request to the Cloudflare API (/accounts/.../workers/scripts/s3-to-r2) failed.

  Uncaught Error: No such module
  "https:/raw.githubusercontent.com/skymethod/denoflare/master/examples/r2-presigned-url-worker/worker.ts".
    imported from "index.js"
   [code: 10021]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/wrangler2/issues/new/choose

Can you please provide some guidance on how to run and publish this example as-is, and how I might import and use it within an existing worker?

For what it's worth I'm getting this error even with a simple hello world.

Hmm, the fact that it's failing on caches might indicate it's a problem when running on Deno 1.26+, which was only recently release (I'm still on 1.25.4).
Deno introduced their own global caches implementation in 1.26, which is probably clashing with the polyfill I had to do before.
Try running it on Deno 1.25.4 in the meantime (deno upgrade --version 1.25.4), and I'll try to get a fix for 1.26 out shortly.

I installed deno via brew install deno on macOS, and it doesn't seem to want to let me downgrade:
deno upgrade --version 1.25.4
error: You do not have write permission to "/opt/homebrew/bin/deno"

Same even with sudo (which I believe should not be needed).
I also tried the

can be deployed as is to a custom domain in a zone you own with a single push command

from the 0.5.0 release notes, which fails with an authentication error (but I'm pretty sure my account ID and API token is correct):
push https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/examples/r2-presigned-url-worker/worker.ts \
    --name ... \
    --r2-bucket-binding bucket:$BUCKET \
    --secret-binding credentials:$USER_ACCESS_KEY_ID:$USER_SECRET_ACCESS_KEY \
    --custom-domain $DOMAIN \
    --account-id $CF_ACCOUNT_ID \
    --api-token $CF_API_TOKEN

bundling ... into bundle.js...
bundle finished (process) in 72ms
computed bindings in 0ms
putting module-based worker ... (66.1kb) (14.5kb compressed)
error: Uncaught (in promise) Error: putScript failed: status=400, errors=10000 Authentication error
    { code: 10000, message: "Authentication error" }
    at execute (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/common/cloudflare_api.ts:1063:15)
    at async putScript (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/common/cloudflare_api.ts:96:13)
    at async buildAndPutScript (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_push.ts:83:9)
    at async Object.push [as handler] (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_push.ts:117:5)
    at async CliCommand.routeSubcommand (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_command.ts:104:13)
    at async https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli.ts:33:5

Managed to downgrade to 1.25.4. That fixed the denoflare serve error, but not the single push command (authentication) error.
But I'm now having trouble passing through the config to denoflare serve:
$ denoflare serve https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/examples/r2-presigned-url-worker/worker.ts \
    --r2-bucket-binding "bucket:$BUCKET" \
    --secret-binding "credentials:$USER_ACCESS_KEY_ID:$USER_SECRET_ACCESS_KEY" \
    --account-id "$CF_ACCOUNT_ID" \
    --api-token "$CF_API_TOKEN"

Compiling https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli-webworker/worker.ts into worker contents...
⚠  Deno requests write access to "/var/folders/2j/ymfzssrx66gg87hypqbw1ptr0000gn/T/". Run again with --allow-write to bypass this prompt.
   Allow? [y/n (y = yes allow, n = no deny)]  y
Bundled https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli-webworker/worker.ts (process) in 1383ms
runScript: https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/examples/r2-presigned-url-worker/worker.ts
Bundled https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/examples/r2-presigned-url-worker/worker.ts (process) in 47ms
worker: start
Error running script Error: verifyToken failed: status=400, errors=6003 Invalid request headers
    at execute (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/common/cloudflare_api.ts:1063:15)
    at async verifyToken (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/common/cloudflare_api.ts:576:13)
    at async Function.parseAccountAndCredentials (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/api_r2_bucket.ts:31:29)
    at async computeR2BucketProvider (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/worker_manager.ts:139:44)
    at async WorkerManager.run (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/worker_manager.ts:111:34)
    at async runScript (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_serve.ts:165:21)
    at async createLocalRequestServer (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_serve.ts:171:13)
    at async Object.serve [as handler] (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_serve.ts:188:32)
    at async CliCommand.routeSubcommand (https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli_command.ts:104:13)
    at async https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/cli/cli.ts:33:5

Seems to be the --r2-bucket-binding "bucket:$BUCKET" option that is breaking it. But I've double and triple checked the bucket name, account ID and API token.
Without that option denoflare serve ... does run, but I can't get it to believe that any of my presigned URLs are valid:
isPresignedUrlAuthorized error: Invalid X-Amz-Signature: ...

I'm not sure if it's because of the missing bucket option. But I've tried to generate presigned URLs like so:
$ AWS_ACCESS_KEY_ID="$USER_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$USER_SECRET_ACCESS_KEY" aws s3 presign s3://$BUCKET/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg --endpoint-url "$AWS_S3_ENDPOINT_URL"

which generates a non-local URL in which I change the domain and port to test, and:
$ AWS_ACCESS_KEY_ID="$USER_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$USER_SECRET_ACCESS_KEY" aws s3 presign s3://${BUCKET}/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg --endpoint-url http://localhost:3030

which generates a local URL. But both fail with the same error.

I created a custom API token with these permissions (per docs for r2-public-read worker):

With that change, the --r2-bucket-binding ..."bucket:$BUCKET" option no longer causes an error.
I was able to successfully deploy the worker:
$ denoflare push https://raw.githubusercontent.com/skymethod/denoflare/v0.5.8/examples/r2-presigned-url-worker/worker.ts \
    --name "$WORKER_NAME" \
    --r2-bucket-binding "bucket:$BUCKET" \
    --secret-binding "credentials:$USER_ACCESS_KEY_ID:$USER_SECRET_ACCESS_KEY" \
    --custom-domain "$DOMAIN" \
    --account-id "$CF_ACCOUNT_ID" \
    --api-token "$CF_API_TOKEN"

When generating presigned URLs I also changed the --endpoint-url option to https://$DOMAIN (the domain where the worker is deployed), and it works!
$ AWS_ACCESS_KEY_ID="$USER_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$USER_SECRET_ACCESS_KEY" aws s3 presign "s3://$BUCKET/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg" --endpoint-url "https://$DOMAIN"

But it still does not work locally via denoflare serve and a presigned URL generated with --endpoint-url http://localhost:3030

Also, how do I get rid of the bucket prefix in the resulting presigned URL path?
$ AWS_ACCESS_KEY_ID="$USER_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$USER_SECRET_ACCESS_KEY" aws s3 presign "s3://$BUCKET/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg" --endpoint-url "https://$DOMAIN"
https://$DOMAIN/$BUCKET/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=3173c4beebe4c40c6a96b223b000a6cb%2F20221007%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221007T052945Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=51b461d60fda15e2280630e3cd59b5ff1d7cbf001f15c37388c44826e03232a7

If I literally just remove it from the s3://$BUCKET/... argument, the resulting URL returns a 404.

Alright, so it looks like there are two issues here:

the caches clash, which affects every worker in denoflare serve on Deno 1.26+
Just committed a fix for this, and will publish a new v0.5.9 release shortly that includes it (since it affects everyone using the latest stable version of Deno)
An option on the presigned worker to omit the bucket part of the path-based url.
Since this just affects this worker, it can easily be tweaked post a Denoflare release.  @mrmachine: can you confirm the official Cloudflare presigned url support does not work for you, and you really need Denoflare's version?  It was written prior to CF supporting it natively, but it can still be useful for using creds that are not CF creds.

Thanks for your help. Yes. My use cases are:

I need to implement a progressive migration from S3 to R2 (to reduce bandwidth charges and avoid a window where some files will be unavailable during a full sync migration) for a private bucket that is accessed via presigned URLs.

I also need to implement a "staging" bucket with read-through from a "production" bucket so that writes do not affect production data, and all production data is available to staging environments on-demand via presigned URLs. We currently use S3 replication for this which maintains a full replica (overkill).

Progressive migration is is on CloudFlare's roadmap, but is not yet implemented and it is uncertain how the implementation will work. E.g. when public buckets landed, it is limited to whole buckets. There could be similar unexpected caveats with CF's progressive migration.
Kian's s3-to-r2-migration takes care of the progressive migration, but effectively makes the bucket public.
Kian pointed me to your Denoflare example, and it was the only example I could find of a CF worker validating presigned URLs. (Thanks!)
My plan is to modify your r2-presigned-url-worker example to include a re-implementation of Kian's s3-to-r2 logic as a fallback, and rely on your presigned URL validation.
In testing this, I got Kian's worker and your example both working independently, but I noticed that r2-presigned-url-worker appears to assume the path addressing style. Even though it appears to not actually do anything with the bucket name it extracts from the path (beyond logging), it is still required to get the right key.
This makes it a bit awkward to plan the domain where my buckets (multiple) be available.

I could use a path style at a domain like r2.example.com for multiple distinct buckets, with the bucket name in the path, except for the fact that r2-presigned-url-worker has only a single r2 bucket binding. So I need a different worker on a different domain for each bucket. E.g. bucket1.example.com/bucket1/path/to/file... and bucket2.example.com/bucket2/path/to/file... which seems redundant.
I could use a virtual style (if r2-presigned-url-worker would support it) and two separate domains like bucket1.example.com/path/to/file... and bucket2.example.com/path/to/file....

Even with virtual style addressing, I believe there would still be a limitation where the domain where I deploy the worker must exactly match the bucket name. So I can't have R2 buckets named example-production and example-staging and make them available at r2.example.com and r2.staging.example.com.
I am not sure if the bucket name given in either virtual or path style is actually used in the validation, or if it is just used for routing. Either way, I believe that this limitation could be removed if r2-presigned-url-worker would strip the bucket name from both virtual and path style URLs when it gets the key, and either discard it (if not needed for validation) or using it instead of the actual R2 bucket name in its presigned URL validation.

Also, CF's implementation also has the limitation where credentials grant access to all buckets. With your implementation, we can create multiple buckets (for different clients) and give them different access credentials.

Ah, thanks for the detail - I think I get where you're going.  So ideally you'd deploy the worker to multiple subdomains, each bound to a separate backing bucket, but perhaps want the subdomain to differ from the bucket name.
Then use vhost style presigned urls, using the "vanity" subdomain, not the underlying bucket name.  I wonder, does aws s3 presign even support using custom hostnames?  Iirc even for vhost sigs it's require to hang off of the canonical region.amazonaws.com root.
You can provide an --endpoint-url, perhaps that's actually all we need to check the sig on the backend, not the bucket name per se...
If so, I should be able to support this with a single new optional text binding to the existing worker, called like virtualCustomHostname.  If set, this would be verified against the incoming request url and used in the sig check, but the underlying bucket binding would still be used for serving the data back out.

Indeed, computeExpectedAwsSignature only needs the full url, not the bucket name per se, so this should be totally doable

FWIW 0.5.9 fixed my deployments. Thanks!

FWIW 0.5.9 fixed my deployments. Thanks!

Glad to hear it!  Denoflare is between a rock and a hard place, chasing two relatively moving targets: Deno (breaking changes in every major and sometimes minor releases) and the production Cloudflare runtime (new services in particular can change quite a bit week to week).
So I appreciate anything you see before I do, please do not hesitate to let me know if you see something that doesn't look right.  Often I personally will not upgrade to the .0 Deno releases right away - since I've now been bitten too many times by showstopper bugs in new major releases.

So ideally you'd deploy the worker to multiple subdomains, each bound to a separate backing bucket, but perhaps want the subdomain to differ from the bucket name.
Then use vhost style presigned urls, using the "vanity" subdomain, not the underlying bucket name. I wonder, does aws s3 presign even support using custom hostnames? Iirc even for vhost sigs it's require to hang off of the canonical region.amazonaws.com root.

You got it, exactly.
AWS does allow you to use any DNS compatible bucket name and point that DNS record to S3 via CNAME -- https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingCustomURLs
So in theory I believe it should be possible to generate a presigned URL somehow that points to the vanity CNAME (the whole "DNS compatible" bucket name).
However a Google search reveals that in practice this is not really possible because tools that generate presigned URLs assume virtual addressing style will be a $BUCKET_NAME.s3.amazonaws.com subdomain, rather than a full custom domain, and therefore they disallow any otherwise valid DNS name with a . in it because S3's wildcard certificate will only work for first level *.s3.amazonaws.com subdomains.
That said, AWS does support virtual addressing style with a single level subdomain which is applied as a prefix to the endpoint URL, and as you suggested we can provide a vanity bucket name which is a subdomain to the provided --endpoint-url.
Example:

R2 bucket named acme-production

r2-presigned-url-worker deployed to vanity domain files.acme.com with vanity access key id and secret access key, and vanity bucket name of files and an r2 bucket binding for acme-staging

Generate vanity presigned URLs with:
$ AWS_S3_ADDRESSING_STYLE='virtual' \
    AWS_ACCESS_KEY_ID="$VANITY_ACCESS_KEY_ID" \
    AWS_SECRET_ACCESS_KEY="$VANITY_SECRET_ACCESS_KEY" \
    bash -c 'aws s3 presign "s3://files/path/to/file.ext" --endpoint-url "https://acme.com"'

https://files.acme.com/path/to/file.ext?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=6ff412844208dd86ba8596a7358ec952%2F20221008%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221008T114945Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=ebbf1db08c652436ae6862e05096574a15921879b028a8c44a2007f1846d234e

Worker validates vanity credentials and proxies request to acme-production r2 bucket binding

Generate vanity presigned URLs with:

So I can't seem to generate vhost style urls with aws s3 presign at all.  Using your exact command line (on a mac), I get:
https://acme.com/files/path/to/file.ext?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<elided>%2F20221008%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221008T180703Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=<elided>

Which version of the aws cli are you using?
Mine is: aws-cli/2.5.8 Python/3.9.11 Darwin/21.6.0 exe/x86_64 prompt/off

I'm using aws-cli/2.4.23 Python/3.9.10 Darwin/21.6.0 source/arm64 prompt/off
It seems I had previously enabled virtual addressing style via config:
aws configure set default.s3.addressing_style virtual

When I remove that from my ~/.aws/config I get the same results as you. So it seems the AWS_S3_ADDRESSING_STYLE='virtual' env var from my command was a mistake that did nothing. I'm not sure how to set the on the command line, but the config file seems to work.

Ah ok - too bad there's no equivalent command line option in aws s3 presign.
I just implemented support for this use case in the Denoflare presigned worker and in thedenoflare r2 presign cli command.  Would you mind testing them out before I tag an official release?
You can use the latest commit hash 824beb75c5de1770a7957a517b8e89fb47552d0d to grab the worker code where you were using a version tag before (to test the worker changes), and install denoflare using a commit hash instead of a release tag (to test denoflare r2 presign).
e.g. to install denoflare as of a specific commit:
deno install --unstable --allow-read --allow-net --allow-env --allow-run --name denoflare --force \
https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/cli/cli.ts

The worker now takes a new optional env binding called virtualHostname.  Would set to files.acme.com in your example.
Should work with aws s3 presign with a custom endpoint and vhost-style requests as configured above.
The denoflare r2 presign command now takes three new options: --endpoint-url, --access-key, and --secret-key.  These can be used when pointing to a non-r2 s3 endpoint (like the presigned worker).
Would look something like this in your example:
$ r2 presign files /path/to/file.ext --endpoint-origin https://acme.com \
--access-key $VANITY_ACCESS_KEY_ID \
--secret-key $VANITY_SECRET_ACCESS_KEY

Note the files in the bucket-name argument in this case is your custom subdomain name and doesn't have to match the underlying bucket name.
It should produce presigned urls of the form: https://files.acme.com/path/to/file.ext?X-Amz-Algorithm=... as you're expecting.

Thanks for this! I've just tested it, and found a few issues...

There is a small difference between the URLs produced by aws s3 presign and denoflare r2 presign (the double slash at the beginning of the path):

$ AWS_ACCESS_KEY_ID="$WORKER_ACCESS_KEY_ID" \
    AWS_SECRET_ACCESS_KEY="$WORKER_SECRET_ACCESS_KEY" \
    bash -c 'aws s3 presign "s3://$WORKER_BUCKET_NAME/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg" --endpoint-url "$WORKER_SCHEME://$WORKER_ENDPOINT"'

http://files.lvh.me:3030/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=3173c4beebe4c40c6a96b223b000a6cb%2F20221010%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221010T235717Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d1c86d977d3cfbae5c550e440c58a026d0c3c5947715a13f865ba00e7bcbf145

$ denoflare r2 presign \
    "$WORKER_BUCKET_NAME" \
    /media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg \
    --access-key $WORKER_ACCESS_KEY_ID \
    --endpoint-origin "http://$WORKER_ENDPOINT" \
    --secret-key $WORKER_SECRET_ACCESS_KEY

http://files.lvh.me:3030//media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=3173c4beebe4c40c6a96b223b000a6cb%2F20221011%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20221011T000253Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=5d6902218251023a34e205643c7c17cef323202ada17871a820a0c683d5d0f96

I guess I am supposed to omit the leading / from the key name with denoflare r2 presign? No problem, just something to be aware of coming from aws s3 presign which includes the bucket name and key in one URL. Maybe we could strip the leading /?

virtualHostname is not detected for endpoints on a non-standard port (e.g. denoflare serve).

After running the worker locally via:
$ denoflare serve hello-local
    --account-id "$CF_ACCOUNT_ID" \
    --api-token "$CF_API_TOKEN" \
    --r2-bucket-binding "bucket:$WORKER_R2_BUCKET_NAME" \
    --secret-binding "credentials:$WORKER_ACCESS_KEY_ID:$WORKER_SECRET_ACCESS_KEY" \
    --text-binding "virtualHostname:$WORKER_BUCKET_NAME.$WORKER_ENDPOINT"

When I try to open the denoflare r2 presign URL, both with and without the leading / in the key name, I get this error in the browser:
Error: Unexpected status 403, code=AccessDenied, message=Access Denied
    at throwIfUnexpectedStatus (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/common/r2/r2.ts:147:11)
    at async getOrHeadObject (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/common/r2/get_head_object.ts:36:5)
    at async getObject (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/common/r2/get_head_object.ts:6:12)
    at async ApiR2Bucket.get (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/cli/api_r2_bucket.ts:60:21)
    at async https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/common/rpc_r2_bucket.ts:39:34
    at async RpcChannel.receiveMessage (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/common/rpc_channel.ts:41:36)
    at async Worker.worker.onmessage (https://raw.githubusercontent.com/skymethod/denoflare/824beb75c5de1770a7957a517b8e89fb47552d0d/cli/worker_manager.ts:70:17)

And this in the terminal log:
GET http://files.lvh.me:3030/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=3173c4beebe4c40c6a96b223b000a6cb%2F20221011%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20221011T002829Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d99ce616fb42c3a800968874c2cfcbbe09b90ba446de31bbacad4bb773e7ef38
{"accessKeyIds":["3173c4beebe4c40c6a96b223b000a6cb"],"flags":[]}
request headers:
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate
accept-language: en-AU,en-GB;q=0.9,en;q=0.8,en-US;q=0.7
cf-connecting-ip: <IP_ADDRESS>
cf-ray: 9b8dbd81d7efbaee
connection: keep-alive
dnt: 1
host: files.lvh.me:3030
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/105.0.1343.53
{"style":"path","hostname":"files.lvh.me","bucketName":"media","keyEncoded":"dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg","key":"dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg"}
Authorized request from 3173c4beebe4c40c6a96b223b000a6cb
bucket.get dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg {}

The issue appears to be caused by { hostname, ... } = new URL(url), because URL.hostname does not include the port. When I changed this to { host, ... } = new URL(url) and changed virtualHostname to virtualHost, this issue was fixed.

The aws s3 presign URL does not validate. I get unauthorized in the browser and this terminal log:

GET http://files.lvh.me:3030/media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=3173c4beebe4c40c6a96b223b000a6cb%2F20221011%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221011T004603Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d877c43ea5b3e9f392a3a0883c5d8b13286d7d0d0b2d746c4c8cfa64108fdcd1
{"accessKeyIds":["3173c4beebe4c40c6a96b223b000a6cb"],"flags":[]}
request headers:
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate
accept-language: en-AU,en-GB;q=0.9,en;q=0.8,en-US;q=0.7
cf-connecting-ip: <IP_ADDRESS>
cf-ray: 90b1c7b323ab14c7
connection: keep-alive
dnt: 1
host: files.lvh.me:3030
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/105.0.1343.53
{"style":"vhost","host":"files.lvh.me:3030","bucketName":"files","keyEncoded":"media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg","key":"media/dd/product/images/0aff9ccd1acc3212a8bc3e43cf70a78d.jpg"}
expectedSignature !== xAmzSignature: 54617b87ae025366c572524c41d1738e88c10a1455400a5889bdf830e90b00c5 !== d877c43ea5b3e9f392a3a0883c5d8b13286d7d0d0b2d746c4c8cfa64108fdcd1
isPresignedUrlAuthorized error: Invalid X-Amz-Signature: d877c43ea5b3e9f392a3a0883c5d8b13286d7d0d0b2d746c4c8cfa64108fdcd1

This includes some additional logging added by me, which indicates the expectedSignature !== xAmzSignature.

Both aws s3 presign and denoflare r2 presign URLs return unauthorized in the browser when deployed to CF via:

denoflare push hello-local \
    --account-id "$CF_ACCOUNT_ID" \
    --api-token "$CF_API_TOKEN" \
    --custom-domain "$WORKER_BUCKET_NAME.$WORKER_ENDPOINT" \
    --name "$WORKER_BUCKET_NAME.$WORKER_ENDPOINT-r2-presigned-url-worker" \
    --r2-bucket-binding "bucket:$WORKER_R2_BUCKET_NAME" \
    --secret-binding "credentials:$WORKER_ACCESS_KEY_ID:$WORKER_SECRET_ACCESS_KEY" \
    --text-binding "virtualHost:$WORKER_BUCKET_NAME.$WORKER_ENDPOINT"

Not sure if or how I can access more detailed logging for the deployed worker. I tried the "real-time logs" option in CF Workers page, but I never saw anything come through.

The key name doesn't actually include the leading slash, it's consistent in the denoflare r2 tools, and consistent in what the cf api returns etc, so just something you'll have to translate if you're breaking down a projection url (also "/" is a valid key name/prefix!)

I didn't test non-standard ports, it's a good point, I think virtualHost makes sense, I'll change it.  The logging includes the problem there: {"style":"path"," ie it did not detect it as a vhost-style request.

Maybe the non-standard port thing again?  I'll see if I can repro that.  The aws s3 presign urls worked for me.

Hmm, again hard to say without more logging.  Both worked for me testing from a browser pointing to the new worker deployed to cf.

Check out webtail.denoflare.dev, I built it as an easier way to view those real-time logs coming back from workers.  The Denoflare cli also has a tail command.

Sorry, got distracted. I intend to test this further soon hopefully with more logging.

@johnspurlock-skymethod
I have finally gotten around to testing this. Apologies for the very long delay.
I don't think the change to virtualHost (from virtualHostname) made it into your release. I have local modifications that change virtualHostname to virtualHost (and also const { host, pathname, searchParams } = new URL(url);) and this fixes vhost style detection with denoflare serve ... on port 3030. I could submit a PR, if you'd still like to make this change?
Also, I believe the reason for the Error: Unexpected status 403, code=AccessDenied, message=Access Denied error after the worker successfully authorized the vanity bucket name and credentials was caused by not having assigned sufficient permissions to my Cloudflare API token.
I had already configured permissions per your R2 public read example, but I had to add Workers R2 Storage:Edit as well. I tested the API token permissions with the denoflare r2 list-buckets command which only worked after adding the new permission.
So now, the URL generated by denoflare r2 presign works. I am able to read the file from the backing bucket via the worker with vanity bucket name, access key ID and secret access key. (Yay!)
However, I still get unauthorized for the URL generated with aws s3 presign. My extra logging still says:
isPresignedUrlAuthorized error: Invalid X-Amz-Signature: 7b8f9610d8106b27f97d9f13c35c24923b4dbcb099c784aff538e8bf53ed3481

I tried to deploy to Cloudflare where there would be no port number in the host to confuse things, but I still get unauthorized for both aws s3 presign and denoflare r2 presign URLs when deployed to Cloudflare. Even though denoflare r2 presign works via denoflare serve on port 3030.
I tried using denoflare tail $WORKER_NAME --format pretty but I get no output when I hit the Cloudflare worker's URL:

I also checked the log stream via Cloudflare Workers but it also shows no output:

Should I be seeing the same logging that I see locally from denoflare serve ... when I view the log stream in Cloudflare or use denoflare tail ... ?
I get the same unauthorized response from the worker, so it seems the vanity bucket name and credentials or X-Amz-Signature might be failing rather than a failure to read the R2 backing bucket.

All console.log calls should appear in tailing, yes (sometimes there is lag - cf actually redeploys your worker behind the scenes when toggling tail vs not)
All PRs are welcome! Can you please remind me of the context in the PR as this thread is pretty gigantic at this point


CSS issues with details on Firefox

Firefox

Chrome

Thank you.
The CSS uses the :has() selector. The problem is that FF is the last browser to fully support this selector.
The Issue will be fixed as soon as the selector is officially released.
see also:

The upstream bug
The release plan reference

according to caniuse has() will be fixed with firefox version 121 in Decemeber 2023

Works now with the latest version 121.0.


Provide binary distribution

As we can see in travis log, cargo install cargo-web takes about 10 minutes. It can be much faster if we will just download a compiled binary. It is done by "just" utility, for example.

We now have Travis automatically pushing precompiled binaries on every release, so closing.


MessagingDeviceResult does not include canonicalRegistrationToken

Using Firebase admin SDK 6.1.0.
According to the documentation found here the response of messaging.sendToDevice should contain information about canonical tokens.
A long time ago I've seen this working but it was removed at some point. I don't know if the documentation is outdated or it should be considered a bug in the library.
This data is valuable because if you send messages to say 10 tokens, and 7 were not valid anymore, you can remove those tokens from your database.
Currently I'm relying on the error responses being returned in the same order as the supplied tokens. So if response at index 7 is an error with code messaging/registration-token-not-registered I remove the token that I passed to sendToDevice at index 7.
But this is an assumption I had to make and I don't know if it is completely safe. In any case this should be documented.

@chong-shao can you take a look?
@0x80 any reason why you haven't migrated to the new FCM APIs? The new APIs have clear and well-documented semantics IMO. And if something's missing, we can work on it.

@hiranya911 Last time I looked I concluded that there is no modern API to address individual devices. I need to be able to send notifications to hundreds of devices at once, but topics are simply not an option because I send notifications to a sub-selection of users which is calculated dynamically.
Or did I miss something and the new FCM APIs do include batch sending with registration tokens?

Check the documentation on sendAll() and sendBatch() APIs (released several months ago).

@hiranya911 Aha I missed that. I found your article here
It is unclear to me what the benefits are of this new API compared to the old one. I currently send messages in batches of 500. This new API is limited to 100 tokens at a time, so that doesn't appear to be an improvement.
What would be the main reasons to switch to the new API?

The improvements are in the backend architecture. We can look into increasing the message limit per batch in the future. Also we'd probably deprecate the old API at some point.

Alright, I will plan to update in the near future then. We do get errors occasionally so if the backend architecture is better that would be enough reason to do so.

@hiranya911 I switched to the new API yesterday, but I see no improvements in performance. In fact the v1 legacy API reported an average latency of 63ms and the new FCM API reports 223ms average latency.
Also since I've upgraded firebase-admin from v6 to v8 the messaging API (both legacy and new) seems to be leaking memory. My cloud function responsible for sending the messages is increasingly using more memory and then hits a limit and errors and starts building memory footprint again. The problem only occurs as a result of sending a lot of messages to around 4-5k device tokens (in batches of course). If I cap the maximum at 2k the process seems to remain stable.
I know this is not a proper bug report but I wanted to mention it anyway...

@hiranya911 never mind the latency. I misread the chart. The 223ms was the 99 percentile. The 50 percentile is exactly the same as that of the legacy api.

@0x80 I don't think there are any measurable performance improvements. What I said was there are improvements in the backend architecture -- resulting in better scalability, load balancing and observability at our end.

Close the issue for now. Feel free to reopen if there's more to discuss :)


Fixed unit tests

@SurferJeffAtGoogle looks like tests are failing out of my code.

@SurferJeffAtGoogle looks like tests are failing out of my code.
Yeah, that test was flaky for a spell.  I kicked off a new build.


bug fix in json_serializer: use ordered list of encoders

This fixes a subtle bug.  Sometimes the unknown schema encoder would be called properly, but in other environments, it would not be.  The encoder map was being used like a list, but the order of the entries was random when iterated using _ENCODER_MAP.items().  Changed to be an actual list of pairs, which is the way we need to use it to be able to override a general case with a more specific case that precedes it in the list.

Codecov Report

Merging #356 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master     #356   +/-   ##
=======================================
  Coverage   74.18%   74.18%
=======================================
  Files          63       63
  Lines        5454     5454
=======================================
  Hits         4046     4046
  Misses       1408     1408

Impacted Files
Coverage Δ

opentimelineio/core/json_serializer.py
95.45% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6727d21...8436852. Read the comment docs.


Support for Network Graph Plots

Is your feature request related to a problem? Please describe
Users can add plotly or other library plots to their apps when they need a network graph, but they don't get the benefit of of app-theming or automatic handling of events.
Describe the solution you'd like
Native support for graph datasets with nodes and edges
Additional context
Details to come as needed

where can I find examples of using plotly (or other libraries) that show to use them to make visuals in wave apps?


Include app domain and name in all event tracking calls as properties

@timstackblock , are you setup to QA event tracking stuff with Segment?
If not, I can help show you how to do so, or we can just assign it to Mark. Whatever works for you!

I have not received the invite to segment via email. Could some one please send this to me so I have access. Once this is done I should be able to QA the tracking issues.

I just sent you an email invite to Segment

I'm in thanks Mark

Working as expected


org.bouncycastle.asn1.ASN1TaggedObject#getObject removed， How can i solve it?

When we upgrade bouncycastle from 1.74 to 1.78，the function: org.bouncycastle.asn1.ASN1TaggedObject#getObject has been removed. can anyone give me some suggestions, use getBaseObject instead of?
e.g. code:
private static Optional findByTag(List authorizationList, int tag) {
for (ASN1Encodable entry : authorizationList) {
ASN1TaggedObject taggedEntry = (ASN1TaggedObject) entry;
if (taggedEntry.getTagNo() == tag) {
return Optional.of(taggedEntry.getObject());
}
}
return Optional.empty();
}

Probably getExplicitBaseObject, but if you could tell us what ASN.1 type (e.g. from a specification) you are working with, there might be better options.
You should probably also replace if (taggedEntry.getTagNo() == tag) with if (taggedEntry.hasContextTag(tag) because a tagged object can have one of four tag classes and you are probably expecting only one of them (CONTEXT_SPECIFIC).


Documentation tracker - v0.14.0

Airnode issues that need documentation:

[ ] https://github.com/api3dao/airnode/issues/1506
[ ] https://github.com/api3dao/airnode/issues/1478
[ ] https://github.com/api3dao/airnode/issues/1317

Corresponding docuemntation issues:

[x] https://github.com/api3dao/vitepress-docs/issues/389
[x] https://github.com/api3dao/vitepress-docs/issues/388
[x] https://github.com/api3dao/vitepress-docs/issues/385

Closing this outdated automated doc issue tracking


Azure SQL Server: A Renamed Stored Procedure outside of DBEAVER is not reflected in DDL in DBEAVER after 'refresh'

If I rename a stored procedure using SQL Server Management studio, then pull up DDL in SSMS it  is correct, but DDL in DBEAVER still has previous name even after REFRESH in DBEAVER.
Left file here is DDL from DBEAVER, right file is from SSMS.  It is like the DBEAVER DDL gets stuck with the old name.

Hello @dhorrall
Can you please tell me - where did you use a refresh? In Database Navigator tree? On the Procedure tab? Somewhere else?

It was in the Databasse Navigator tree.  Right click 'REFRESH'.  The name
of the procedure does change.  But the 'Create' in the DDL still has it
'creating' the previous name.
On Tue, Feb 8, 2022 at 3:09 PM Anastasiya @.***> wrote:

Hello @dhorrall https://github.com/dhorrall
Can you please tell me - where did you use a refresh? In Database
Navigator tree? On the Procedure tab? Somewhere else?
—
Reply to this email directly, view it on GitHub
https://github.com/dbeaver/dbeaver/issues/15431#issuecomment-1033015751,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFMEHISYO6QP5HTAGGO6OILU2FZ6HANCNFSM5N3NYJJA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Regards,
Derek Horrall


Looping & Resetting Progress Due To Twitch Bug(?)

I assume this may be a Twitch bug, so there might not be a lot you're able to do here, however, I'm reporting it anyway, as it may be able to be worked around.
Recently, a lot of my drops appear as though they aren't being mined, specifically for me, the Overwatch 2 ones. The miner will watch a channel, progress for some hours and then the timer/progress will reset to zero if the channel is switched or something refreshes the miner.
After some looking around, it turns out I have already earned these drops, however, in my Twitch inventory page, the drops are marked as "Earned 2,023 years ago", and so they appear at the bottom of the page instead of at the top where the recently earned rewards appear.

I think this may be the reason the miner is constantly trying to farm the same drop over and over, as the drops that do this are the ones that show as being "Earned 2,023 years ago". For now, I will have to pay attention to which drops I have earned and exclude the game, but it would be nice if there was some work-around for this.
Thank you :)

Confirming this is also happening to me although I was still using that fork fix up until today. Looks like it's another twitch change in the background.

Hello. Yeah, this definitely looks like some sort of a Twitch bug / visual issue being present. On my side, my last drop says it's been claimed 13 hours ago, and can be found at the top, so everything seems to be working just fine.
Are you using the latest dev version? Please note that I don't/cannot provide much support to you, if you're using a fork instead.

I was actually mining drops on three different accounts, and all three have the same issue. I was using the fork at that time, and when I started noticing things were off, I went back to your version. Same thing happens no matter which one I use

false progress again, in palia and overwatch 2

is it possible that the miner caused this bug?

I was actually mining drops on three different accounts

@iXerxes Multi-account mining is not supported. The issues you're experiencing are most likely caused by Twitch rate-limiting the miner, which causes it to keep artificially increasing the progress, without doing any of the actual mining like it should be.
@matarife123 Are you sure it's the same Twitch account that's logged in the browser and the miner? Because your picture shows that it doesn't seem like it is, and you're looking at the wrong account, or something like that.
Again, just to be clear - for the miner to work, you cannot use it on more than one account at a time, and cannot watch any streams in your browser during the mining process. Otherwise, you'll keep getting false progress and it'll generally not work like you expect.

¿Es posible que el minero haya causado este error?

En realidad, estaba extrayendo drops en tres cuentas diferentes.

@iXerxesNo se admite la minería multicuenta. Los problemas que estás experimentando probablemente se deben a que Twitch limita la velocidad del minero, lo que hace que siga aumentando artificialmente el progreso, sin realizar ninguna minería real como debería ser.
@matarife123¿Estás seguro de que es la misma cuenta de Twitch que está conectada en el navegador y en el minero? Porque tu imagen muestra que no parece ser así y que estás viendo la cuenta equivocada o algo así.
Nuevamente, para que quede claro: para que el minero funcione, no puedes usarlo en más de una cuenta a la vez y no puedes ver ninguna transmisión en tu navegador durante el proceso de minería. De lo contrario, seguirás obteniendo información falsa sobre el progreso y, en general, no funcionará como esperas.

It is my only Twitch account and I am not on another channel, right now it gave me other errors
22:30:00: ERROR: Broadcast settings update for a non-existing channel: 458358256

Broadcast settings update for a non-existing channel

That's related to #110, which has been reported by you already a while ago. I need to finally get around to improving the websocket topics handling.

I have restarted the miner, it says that palia has 2.5% progress but in my Twitch inventory there is no information about the drop. Same for overwatch2

@DevilXD I'm aware it's not supported for multiple accounts, however I have been using it like this for a while now, and it's always worked perfectly fine, up until the slight hiccups with the recent Twitch changes. Even still, I have been mining and getting drops on all accounts as expected.
When I close all instances and leave just one open, the issue still persists. I do actually have the drops for the affected games (2,023 years ago bug), but the miner doesn't seem to detect that, and keeps trying to make some kind of invalid/false progress.
Adding these games to the exclude list fixes the issue - for now - and allows the miner to make progress on the unaffected games, and does so just fine for all account instances running.

I have restarted the miner, it says that palia has 2.5% progress but in my Twitch inventory there is no information about the drop. Same for overwatch2

@matarife123 I recommend you go to the Twitch Inventory page and scroll to the bottom and see if your drops are there. As I've mentioned previously in this issue, some of the drops are marked as being as earned 2,023 years ago, and so they appear at the bottom of the page. For what ever reason, this causes problems for the miner. Also check in game for your drops just to be sure.
For now, you will have to add the affected games to the exclude list in the miner settings, and pay attention to when there are new drops so that you can un-exclude them.

I have been using it like this for a while now, and it's always worked perfectly fine, up until the slight hiccups with the recent Twitch changes.

The thing is, these "recent Twitch changes" have made it so that the exact same pathway as for downloading the stream content to watch it has to be followed. It's likely that Twitch has an already-implemented set of ways to detect a single IP address trying to watch 3 streams at once there, which causes what you're experiencing. That, or it's simply Twitch testing something, and you're getting caught in the process - gradual changes rollouts usually happen to only a selected subset of accounts at first, where it continues to grow as they decide to keep rolling out the update to everyone.
You can't simply downplay it with "but it worked previously", because the previous way of it working literally does not exist anymore. It's an entirely different pathway now, and as I said, it very likely uses the already-existing rate-limiting system Twitch most likely has.

When I close all instances and leave just one open, the issue still persists.

I recommend closing all instances for at least an hour, before concluding that anything "persists", as rate limits can get quite lengthy. A full 24 hours of not doing any mining on any account will let you rule out rate-limiting as the cause, and instead it'll mean it's just Twitch being buggy for you for some reason. Otherwise, we'll never know the real cause.

I recommend closing all instances for at least an hour
I have done this. I closed all instances after finding the bug and forgot about it for a day or two. After which, I loaded up a single instance and the issue still occurred. After adding the affected games to the exclude list, the miner works as expected for the other games.

He reiniciado el minero, dice que palia tiene un 2.5% de progreso pero en mi inventario de Twitch no hay información sobre la caída. Lo mismo para overwatch2

@matarife123Te recomiendo que vayas a la página de Inventario de Twitch , te desplaces hasta el final y veas si tus gotas están allí. Como mencioné anteriormente en esta edición, algunas de las entregas están marcadas como obtenidas hace 2023 años, por lo que aparecen en la parte inferior de la página. Por alguna razón, esto causa problemas al minero. También revisa el juego para ver tus lanzamientos solo para estar seguro.
Por ahora, tendrás que agregar los juegos afectados a la lista de exclusión en la configuración del minero y prestar atención a cuándo hay nuevas caídas para que puedas anular su exclusión.

 I guess this issue can remain open in an off-chance someone else runs into this as well, but other than that, excluding the affected games may help solve the issue for you, and other than that, not much can be done.
I could possibly try to obtain the API response from you, to try and understand what causes the miner to try mining these drops again, but there's currently no way for the miner to save these. I should probably add it as a debug-feature of sorts. I'll try getting to it tomorrow.

I have no real solution to this then, as it appears to be a Twitch-side bug, and I have no way to reproduce this on my side. I guess this issue can remain open in an off-chance someone else runs into this as well, but other than that, excluding the affected games may help solve the issue for you, and other than that, not much can be done.
I could possibly try to obtain the API response from you, to try and understand what causes the miner to try mining these drops again, but there's currently no way for the miner to save these. I should probably add it as a debug-feature of sorts. I'll try getting to it tomorrow.

I think Twitch changed the privacy policy several times, I could see the progress in the profile section like before but it seems like they removed it again. Also thank you very much for your effort and for the great work. ^^

https://github.com/DevilXD/TwitchDropsMiner/commit/f5d5fc3a05b949e2bbec2ef2b79940c30b614fac implements a new launch argument: --dump. Using it will launch the miner in a special mode, where it'll create a dump.dat file containing inventory data the miner uses to build it's internal campaigns "database", and then it'll close afterwards. It should be safe to share this file online, as it doesn't contain any user-related data, besides twitch account connection status to a particular game/campaign (connected or not), and current drop progress as shown on the inventory page.
If you want me to take a look, please attach the dump file here.

I have a

dded palia to the priority list and I have the same problem, it does not recognize that I have the drops

I'll need the dump file to try diagnosing it and provide any further help.

@DevilXD Here is my dump.
Currently, the only bugged drop that I have active is "XDefiant Preseason - W6". The drops are:

Weapon XP Booster - W6
Amber M44 Skin
Pelagic Vector Skin

I have already completed this campaign, owning all of the drops, however, they all appear in my Twitch Inventory as being earned 2,023 years ago and the miner thinks I don't own them.
The miner is currently trying to mine the first drop ("Weapon XP Booster - W6"), but any form of refreshing causes the progress to reset to 3.3%.
https://pastebin.com/SNiw2eYT

@DevilXD Here is my dump.
Currently, the these campaigns are/were bugged for me in some way:

XDefiant Preseason - W6  (This drop is the most relevant to the issue.)

Weapon XP Booster - W6
Amber M44 Skin
Pelagic Vector  Skin

I have already completed this campaign, owning all of the drops, however, they all appear in my Twitch Inventory as being earned 2,023 years ago and the miner thinks I don't own them. The miner is currently trying to mine the first drop ("Weapon XP Booster - W6"), but any form of refreshing causes the progress to reset to 3.3%.

PAYDAY 3

I had no progress towards this campaign and the miner initially couldn't start to make any progress; also resetting to 3.3% upon refresh. After testing to see which of my campaigns were bugged, I was able to start progress on Twitch first, and now the miner is able to progress the drop as normal.

Campaigns that I have confirmed to be making valid progress (during this dump):

Marbles on Stream
Bloodhunt Summer Drops 1

https://pastebin.com/xnizP6Va (Before PAYDAY 3 "Jumpstart")
https://pastebin.com/N5KuzzEg (After PAYDAY 3 "Jumpstart")

Looking at the dump myself, it really doesn't show a whole lot to be honest. XDefiant just shows that I don't have the drops, and I haven't started the campaign, given that it doesn't have the self property.

The most annoying thing about the self edge, is that it's not included when a campaign wasn't started yet, but it's also not included if a drop has already been claimed. And the information about it being claimed is normally in the self edge... So the miner does a little bit of "dancing around" with the info Twitch returns, to figure out if the drop has been claimed or not. One of those dances checks if the claim date on the drop lands within the timeframe of a campaign being active, and if so, it marks it as claimed. If not, it's considered unclaimed, and without a self edge, the progress on it becomes 0%, making it a valid mining target. There's also a catch of benefits vs drops to take care of there somewhere.
If the drop claim date is returned as 2023 years ago, then it's probably been marked as unclaimed, and that's why the miner is trying to mine it. I'll take a look at the dumps later today, once I get out of work.

I've taken a look at the dump file.

For what ever reason, Marbles on Stream exported my Twitch UserID

That's because it was included in the drop instance ID, which is a special ID that's sent to Twitch during drop claiming operation, and is the only way to tell it what you're claiming. If you have a campaign present in your inventory, that has at least one claimed (or ready to claim) drop, but not all of them are claimed yet, then the API will return self edges like that, including the drop instance ID that was used to claim that drop. I've totally forgot that's something that can include user IDs, but just pushed out a commit that ensures those are excluded from the dump: https://github.com/DevilXD/TwitchDropsMiner/commit/da050828c1e054a4197ca57568c78dbd0d1be6bc
As far as inspecting the dump goes, you have 22 drop instances that "were claimed" on the 1st of January 1 A.D, at midnight (0001-01-01 00:00:00+00:00), which is obvious nonsense and missing information. Without this claim date, it's not entirely possible to attribute the drops to any particular campaign, and whatever logic is assumed, it'll always be wrong in some cases:

If we assume that wrong date == always claimed, that'll fix this issue, but introduce a new one where two consecutive campaigns for the same game, having the same benefit in each drop (like Smite, where it's a single benefit repeated x7, reappearing over and over in a new campaign every week), will get marked as finished (all drops claimed), even though the 2nd campaign hasn't been started yet. A user can fix this by manually watching at least one minute of an applicable stream (that'll put the unstarted campaign into inventory, and provide corrent claim status and progress), but it's no longer automatic and will mislabel drops. Basically #418.
If we assume that wrong date == unclaimed, you get what we have here now - the miner trying to mine a campaign that was already mined and claimed. The only way to get around it, is to use the exclude list for the affected campaign, until it expires and can no longer be mined. This may also make you miss out on other campaigns for the same game.

Which one would you like to go with?
There's only one Payday 3 campaign, and it appears you have 15 minutes of progres on it already. There doesn't seem to be any other Payday 3 campaign present in the dump. All other values are as expected in the "before" dump as well. The only reason it wouldn't start on it's own, is the priority list focusing on something else, or no available channels to watch - otherwise it should start normally. If it didn't, then I have no idea why could that be.

Could this have something to do with it?

@matarife123 No, that's just the drop that's currently being mined. This same drop is returned by GQL to check on mining progress, when the websocket decides to not respond for a while. It should generally be the same drop the websocket returns.

The only reason it wouldn't start on it's own, is the priority list focusing on something else, or no available channels to watch - otherwise it should start normally. If it didn't, then I have no idea why could that be.

Yea, I assumed this would be something the dump wouldn't be able to show. Sadly though, I was specifically testing to see which of my drops were working, and had all other available drops in the exclude list to isolate Payday 3. There were definitely people online and streaming it. Everything, from trying to test Payday in the miner, to jumpstarting the drop on Twitch its self, and back again - it all happened within the same 15-20 minutes.

Which one would you like to go with?

lol yea, idk either. These Twitch changes over the past several months have been a killer. I have actually emailed Twitch, however, showing them the glitched "Earned 2,023 years ago" drops, and they have said they're aware of it and are working on some solution. So hopefully this is something you can sit out, and just let it be until it passes :)

There were definitely people online and streaming it.

Was the miner watching a Payday 3 stream though? Or it didn't even attempt to watch it/watched something else?

Was the miner watching a Payday 3 stream though? Or it didn't even attempt to watch it/watched something else?

Yes, it was watching Payday 3. I even had to check a few times because there was some visual glitch where it looked like it was farming another game's drop, before some graphical update. So yes, I'm certain it was watching and attempting to progress Payday 3.

there was some visual glitch where it looked like it was farming another game's drop, before some graphical update.

That actually leads me to believe the miner was trying to do something else, or at least it's internal state lead it to believe it was trying to mine something else, other than Payday 3 campaign.
I'll put some more time into studying the dump file, in hope of finding an answer to this, but for now, it doesn't seem like it's something I can fix in any way. If Payday 3 stream was being watched, the campaign should start one way or another, regardless of what drop progress was being displayed.

this is happening again.

That's not enough information for me to even considered pursuing that further for now. Please provide a more extensive description of the exact issue you're experiencing.

Is this issue still happening to anyone here?

Since there has been no further reports on this, I consider this issue to be over.


feat: Add trigger specification cache to Htmxor

This update introduces a trigger specification cache to the Htmxor library. The new feature improves parsing performance by storing evaluated trigger specifications, albeit at the cost of increased memory usage on client side (htmx's words)
I haven't added all comments or tests yet.
The changes include:

Addition of TriggerSpecificationCache class and related methods in HtmxConfig.cs
Creation of new files HtmxTriggerSpecification.cs, ITriggerBuilder.cs, TriggerBuilder.cs, TriggerModifierBuilder.cs and TriggerQueueOption.cs to handle trigger specifications

The original rationale behind adding a trigger cache to htmx was because if you have a large number of rows with a common trigger there is a sizeable performance impact when htmx has to parse each of the trigger definitions. Htmx added a key/value store to quickly look up the parsed trigger definition from the raw trigger value.
The end result looks like this within the configuration.  Note that because triggers can be complex it gets a bit tricky for developers to get the cached definitions parsed correctly by hand:
{
  "useTemplateFragments": true,
  "selfRequestsOnly": true,
  "triggerSpecsCache": {
    "revealed": [
      {
        "trigger": "revealed"
      }
    ],
    "newContact from:body": [
      {
        "trigger": "newContact",
        "from": "body"
      }
    ],
    "keyup changed delay:500ms, mouseenter once": [
      {
        "trigger": "keyup",
        "changed": true,
        "delay": 500
      },
      {
        "trigger": "mouseenter",
        "once": true
      }
    ],
    "every 30s, click": [
      {
        "trigger": "every",
        "pollInterval": 30000
      },
      {
        "trigger": "click"
      }
    ]
  }
}

This is generated by this code:
builder.Services.AddRazorComponents().AddHtmx(options =>
{
	// Enabled to support out of band updates
	options.UseTemplateFragments = true;

	// Enabled to show use of trigger specification cache
	options.TriggerSpecsCache = new TriggerSpecificationCache (
		Trigger.Revealed(), // Used in InfiniteScroll demo
		Trigger.OnEvent("newContact").From("body"), // Used in TriggeringEvents demo
		Trigger.OnEvent("keyup").Changed().Delay(TimeSpan.FromMilliseconds(500))
			.Or()
			.OnEvent("mouseenter").Once(),  //  Unused, demonstrates complex trigger
		Trigger.Every(TimeSpan.FromSeconds(30)) // Unused, demonstrates use of Every
			.Or()
			.OnEvent("click")
	);
});

The trigger builder builds the complete trigger value as well as the specification used for caching.  It can be added easily to the TriggerSpecificationCache as a comma-separated list of fluent trigger definitions.
Second, the builder can be used directly in code to define triggers.  This is important because it makes sure the exact trigger definition is used in both the cache definition as well as in the markup:
        <HtmxFragment Match=@(req => req.Target == "contacts-table")>
            <tbody id="contacts-table" hx-get hx-trigger=@Trigger.OnEvent("newContact").From("body"<EMAIL_ADDRESS>                @foreach (var contact in Contacts.Data.Values.OrderBy(x => x.Modified).Take(20))
                {
                    <tr>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>                    </tr>
                }
            </tbody>
        </HtmxFragment>

This looks promising. A few quick notes:

do we need both interfaces and public types?
if all cached triggers can be shared on the server, not just in the client, it would save allocations on each request.

This looks promising. A few quick notes:

do we need both interfaces and public types?
if all cached triggers can be shared on the server, not just in the client, it would save allocations on each request.

The trigger builder consists of the base trigger builder and another class to build the modifier chains so the next fluent option is valid for the step you are on.  Because you can end the trigger when in the TriggerBuilder or TriggerModifierBuilder I added ITriggerBuilder to define the build method so either could be used to add cache options.
One of the TriggerSpecificationCache constructors takes a params list of ITriggerBuilder objects.
	public TriggerSpecificationCache(params ITriggerBuilder[] builders)
	{
		builders ??= Array.Empty<ITriggerBuilder>();

		foreach (var triggerBuilder in builders)
		{
			var trigger = triggerBuilder.Build();
			Add(trigger.Key, trigger.Value);
		}
	}

That allows for:
new TriggerSpecificationCache (
		Trigger.Revealed(), // <-- this is a TriggerBuilder
		Trigger.OnEvent("newContact").From("body") // this is a TriggerModifierBuilder
	);

The base TriggerBuilder holds the whole trigger definition list so TriggerModifierBuilder's build method calls the TriggerBuilder build() method.

if all cached triggers can be shared on the server, not just in the client, it would save allocations on each request.

I'm not sure what you are asking.  You can create a proxy object to perhaps download the cache list from the server.  We'd have to set up an endpoint to output the cache and write some code to configure the setting.  I'd put this in the future consideration list of features for now to fully implement

Let me know if you need any additional changes.

Let me know if you need any additional changes.

Taking a look tonight and will push some changes.

I'm not sure what you are asking. You can create a proxy object to perhaps download the cache list from the server. We'd have to set up an endpoint to output the cache and write some code to configure the setting. I'd put this in the future consideration list of features for now to fully implement

I was thinking of a micro-optimization, where we avoid allocating new builder instances when its not needed. Some of the options combinations are static. We can look at this later in the project life cycle.


Continuing Development

My skype id: demutis
Lets talk about getting this back on track.

This is an OLD wallet, there is an updated wallet.  Please link/goto that reposit. https://github.com/mooncoincore/wallet/releases